Dev Analysis for exposing IO runtime data

Version 2

    User story

    Expose as runtime / statistics data for IO Worker.

    Aim is for users to be able to see as much runtime IO data as possible. Some of this data is already available via jmx, but it needs to be accessible in management model.

    This information is useful when monitoring and diagnosing performance problems.

     

    Issue metadata

    https://issues.jboss.org/browse/EAP7-616  Expose IO worker runtime data / statistics in CLI

    https://issues.jboss.org/browse/WFCORE-1341  Expose IO worker runtime data / statistics

    https://issues.jboss.org/browse/XNIO-268 QueuedNioTcpServer always returns "-1" for "ConnectionCount" attribute in MBean

     

    Hard requirements

    Expose as least as much data as we could get from thread pools in EAP 6

     

    Soft requirements

    Expose any other data that might be available in IO, threading, remoting related subsystems.

     

    Problem with current state

    EAP6 used thread based blocking IO almost everywhere which made it easy to expose runtime data of thread pools that ware in use.

    EAP7 moved to async IO where concept thread pools as they ware known in EAP6 don't exactly map anymore.

    System can provide and expose more runtime data but there is no 1:1 mapping possible

     

    Proposed Solution

     

    Currently XNIO only exposes most of runtime data though mbeans and are not available to server through any other means.

    We need to improve / add XNIO api that will allow us to access this data directly.

     

    There are two kinds of data sets that are available, one for workers and one for "servers"

    dataset for worker is as follows:

     

    boolean isShutdownRequested();
    int getCoreWorkerPoolSize();
    int getMaxWorkerPoolSize();
    int getIoThreadCount();
    int getWorkerQueueSize();
    

     

    Data set for server is:

     

    String getBindAddress();
    int getConnectionCount();
    int getConnectionLimitHighWater();
    int getConnectionLimitLowWater();
    

     

    Each worker can serve multiple servers.

    For example in our default shipped configuration we have only one worker named "default" which is than used/shared by management http interface, remoting and undertow web listeners.

     

    Implementation Ideas

     

    Worker runtime data will be exposed in IO subsystem for each worker under address

    /subsystem=io/worker=*
    

     

    Question is what is the best way to expose server data.

    One could be in IO subsystem with dynamic sub resource of worker which would result in resource address like:

     

    /subsystem=io/worker=default/server=http
    

     

    And another by exposing this data in each subsystem that creates servers separately.

    To aid such exposure IO subsystem would provide helper methods that would register attributes and provide metric handlers.

     

    Implementation details

    worker resource in io subsystem will gain

    • additional attributes with worker runtime metric attributes
    • runtime-only subresources that contain details about "server" consumers.

     

     

     

    Work decomposition

    If we go with first implementation idea work will be contained only to XNIO and wildfly-core.

    But second idea for work in WildFly full would be required as well.

     

    QE

    Testing of this should include looking if data that workers and servers provide are available and if they are correct.