calendarDataModelItem has getStyleClass method defined. It should return the styleClass name you need to be applied to the date.
There is also just client side solution. You should define dayStyleClass attribute with some your JS function which should get date as parameter and return styleClass name. Then this function will be called every time on popup build.