7 Replies Latest reply on Nov 14, 2014 4:24 PM by vietn

    Monitoring a dockerized server managed by Kubernetes

    theute

      I would like to engage a conversation about monitoring a server (Let's use WildFly as an example) that is running inside a docker container managed by Kubernetes. Where to put the agent ?

       

      Some kind of agent is necessary to get metrics from WildFly and push to the monitoring server.

       

      I am still learning about Docker/Kubernetes but AFAIK there would be several options:

      • have the agent within the WildFly docker image (1 agent per container)
      • have the agent as a docker image in the same pod as the WildFly server (1 agent per pod)
      • have a single agent per node instance (Google Cloud Platform actually has an agent per node instance called kublet that monitors the container itself and restart it if needed, I don't know if this is extensible) (1 agent per node)

       

      Last option is not something a PaaS user could install, it would have to be installed by the PaaS owner (OpenShift Enterprise)

       

      Other options ? Comments ?

        • 1. Re: Monitoring a dockerized server managed by Kubernetes
          mfoley

          One difference between option #1 and option #2/3 ... is the "coupling".   So in option #1 ... the coupling occurs when the Docker container (containing both the agent and the middleware product) is made at release time.  This seems a tight coupling.  Nothing wrong, necessarily with that ... but the situation gets a bit more complex when the agent and the middleware product are on different release schedules ...compatibility ... testing ...and the teamwork required by 2 independent teams.  Still do-able ... but if there is a then portfolio of 10 middleware products ... then complexity from this tight coupling might be difficult.   Compared to option #2/3 .... where the coupling between agent and middleware product is in the pod definition ...which is an easily editable json file....so this is a much looser coupling...and might be less problematic from a coupling perspective.

          • 2. Re: Monitoring a dockerized server managed by Kubernetes
            lkrejci

            I guess the answer would depend on what you want to monitor/manage.

             

            Option 1: If the agent is within the container, you can do anything because agent and Wfly are "local" to each other.

            Option 2: If the agent is in the same pod, it shares the IP address with Wfly but NOT storage (necessarily). This might mean that file-based monitoring/management might not be possible (unless the files in question are on a shared volume).

            Option 3: You can monitor/manage remotely and might be able to use docker to "look inside" the containers (using docker-exec, docker-top and the like).

             

            Also, you are increasing the number of stuff the agent is responsible for with each option so there is a scalability concern to this, too.

            • 3. Re: Monitoring a dockerized server managed by Kubernetes
              theute

              Thanks for the insight on option 3, I don't know what's possible exactly. Final answer is probably a mix.

              The other thing to consider is who would manage/install the agent, for option 1 et 2 it can be installed by the owner of the application but option 3 is managed by the owner of the infrastructure.

              • 4. Re: Monitoring a dockerized server managed by Kubernetes
                vietn

                I have been experimenting with Docker and most recently Kubernetes so I thought I should chime in here.

                 

                In light of recent development in container-based OS such as Project Atomic and Core OS perhaps we can come up with a "super agent mode" that can provide management support for the host/bare-metal platform (infrastructure) as well as its container children (applications) as a first class citizen.   Something like rhq-agent -> cadvisor -> containers.  This super mode works well for private/on-premise installation where you have full control of the host. 

                 

                The next flavor, "cloud mode" is suitable for AWS or GCE users where they don't have access to the host/underlying infrastructure.  The agent runs by itself in its own container which can then manage other resources such as providing basic container monitoring via cadvisor, or middleware apps via plugins by means of inter-container networking. File based monitoring is now more of a responsibility of the cloud providers which in a perfect world they should already have super rhqagent running.

                 

                Viet Nguyen

                • 5. Re: Monitoring a dockerized server managed by Kubernetes
                  theute

                  Thank you Viet, that's helpful.

                  We need to figure out how to "expose" the Wildfly metrics (or any other MW piece) to the "super agent"

                  • 6. Re: Monitoring a dockerized server managed by Kubernetes
                    jayshaughnessy

                    Right, that's the part I don't see right now unless an agent runs in the container as in option 1.  It seems cadvisor could give us monitoring of the machine and the containers but no exposure to what's actually being run in the container.  It seems something needs to report from the container to the outside world.

                    • 7. Re: Monitoring a dockerized server managed by Kubernetes
                      vietn

                      Does the RHQ JBoss/Wildfly plugin communicate with Wildfly management interface through localhost:9999?  If so then option #2 should work.