The is the order the properties are defined. You need to define the append and autoFlush before you define the fileName. If you already booted the server with this configuration you'll also need to fix the logging.properties file as well. You can either delete the reference to your handler or move the handler.VBO_ALL.properties values around to have append and autoFlush at the beginning.
That said in EAP 6.4 there is a periodic-size-rotating-file-handler resource you can use instead of defining it as a custom-handler.
James R. Perkins