Log Viewer Design

Version 4

Created by crobson on Dec 20, 2013 1:49 PM. Last modified by crobson on Jan 28, 2014 11:49 AM.

Requirements

From John

Provide a log viewer in the admin console that can look at the following logs:

Boot Log (Boot logging is the recording of events that occur while the server is starting up)
Standalone
- Server.log (Contains all server log messages, including server startup messages)
Domain
- host-controller.log (Contains log messages related to the startup of the host controller)
- process-controller.log (Contains log messages related to the startup of the process controller)
- server.log (Contains all server log messages, including server startup messages)

We should be able to view each of these logs in context, respecting what our access permissions allow us to do.

We support many logging frameworks:

JBoss Logging - included with JBoss EAP 6
Apache Commons Logging - http://commons.apache.org/logging/
Simple Logging Facade for Java (SLF4J) - http://www.slf4j.org/
Apache log4j - http://logging.apache.org/log4j/1.2/
Java SE Logging (java.util.logging) - http://download.oracle.com/javase/6/docs/api/java/util/logging/package-summary.html)

Application Logging: we allow applications to create their own logs in addition to the logs we already create.

bstansberry: Should this be an item under "Provide a log viewer in the admin console that can look at the following logs:" above?

Log rotation: we provide several ways that users can rotate their logs based upon time, size, startup, etc. It would be good if we could expose this context.

Formatting: what we output by default as a format can be changed by an admin.

Viewing historical logs should also be accommodated (yesterdays log that we're no longer writing to)

Paging is probably required to ensure the display is responsive.

User Stories/Use Cases

As a user of the log viewer, I need to be able to select which log I want to look at. I should be able to select which type of log (boot, standaloner server, application, etc) as well as which log of that type (historically looking).

bstansberry:

Please note that "type of log" is not really a concept that exists in the software (except for management audit logs). The various logs we create get created due to standard configuration choices that users can heavily alter and that are only loosely tied to notions like "server.log".

1) process-controller.log -- conceptually pretty straightforward -- the log emitted by the PC process. But whether/how the PC logs is entirely based on system properties passed to the JVM launch command (e.g. by domain.sh). The Host Controller process (which would provide the log content to any log viewer) would have to guess at how the PC's logging is configured.

2) host-controller.log -- also conceptually pretty straightforward, and here at least the HC doesn't have to guess so much as to how its logging is configured.

3) boot.log. I hope we can drop this as a category as it's a real corner case since 6.1.

4) server.log -- What makes some handler configured in the logging subsystem the "server log"? Perhaps that it's a handler associated with the root logger?

5) application log -- really nebulous.

6) Management audit log -- this one is pretty certain, as it is produced by a handler that is configured as part of the management audit logging resources.

As a user of the log viewer, I should only be able to see those logs that I am permitted to see via my role/permissions.

As a user of the log viewer, I want to be able to view the details of the log and sort/filter the log to find events that are of interest to me. I typically view the log in chronological order.

As an administrator, I want to be able to modify the information that is shown in the log. TBD - is there a static set of information options - or can this be really anything they want it to be?

(jperkins) This is not possible and won't be possible with a plain text log, which is currently what we use. I don't see this changing any time in the future as it would limit how the user wants the log formatted.

As a user of the log viewer, I want to have the log load quickly, but I should also be able to filter and scrub through the information quickly to find the pertinent information I need.

(hbraun) "Filter" and "scrub": this needs to be defined more precisely. See attached document about extended metric support for some of the design considerations. IMO the kind of supported queries is what ultimately defines the scope of a tool like this.

Needs verification:

As a user of the log viewer, I want to manipulate settings for the log I am working with such as when it rotates and where it is stored.

(jperkins) I'm not sure I like this idea. There is already the ability for an administrator to change these settings in the console. I don't think it needs to or should be done from a log viewer.

(jdoyle) As we move the console forward, we're likely to duplicate some functionality as we create more context driven workflows. Perhaps as an alternative to manipulating the log setting in place, if we can provide some guidance to the user to where the setting for this log is maintained.

Open Questions

Does the server.log capture what we are calling 'audit logging' (user actions on the system)?

A: (jperkins) No the server log does include any audit logging.

It appears Audit logging might be another log. Should this also be viewable through this GUI?

A: (jperkins) I think this is open for debate. I tend to lean towards no, but it depends on where the audit logs are stored on domain servers.

jdoyle- I do think that we should include the audit log in the log viewer. Like the other logs, it would have to respect a user's permissions which would restrict it to only the Auditor role.

(jperkins) Fair enough. As long as we can get security around it I think it's okay. This will require some new operations as the current operations only work on the logging subsystem.

Should users be able to manipulate log rotation/handling via the GUI? Right now you only can through the CLI.

jdoyle-Yes, but that sounds like a different requirement all together.

(jrperkins) If I'm understanding this correctly, this is already done. You can change the rotate suffix pattern or any other handler settings on the web console.

Further Comments:

hpehl:

I think what would make most sense to me and what IMHO is also in the timeline for EAP 6.3, is to view logs which are configured to use file based log handlers. That would include the following handlers (as defined in the management model):

file-handler
periodic-rotating-file-handler
size-rotating-file-handler

Whatever log is configured to use one of these handlers, could be viewed using the console. The log view would be limited to see the last n lines or the log file in total, though. Filtering and searching of the log is something that has to be done using the browser or external tools. IMO any additional features put a risk to the EAP 6.3 timeline.

jperkins:

Only pointing out one thing. If a user uses a custom logging configuration within their deployment, those logs would not be viewable if we only use file handlers in the subsystem. That may not be a huge issue, but it's something to be noted.

We have operations now in the WildFly logging subsystem to read the log files. It already supports last n lines, first n lines or the entire file. One more thing to consider is a log "line" is really not defined. While most formats use a LF or CRLF there is no guarantee that's what the line terminator will be. It think it's fairly safe to assume it's a common terminator, but it's not guaranteed.

With regards to filtering, we don't even want to go there at this point. With a standard pattern-formatter, currently the only one available in EAP, there is no way to parse the line back into kind of object reliably. We allow the users use what ever pattern they want. It's nearly impossible to parse these kinds of things with any real certainty without adding specific rules or log record delimiters.

WildFly Logging subsystem

(Not for the 6.4 release)

Part of the log viewer might be able to view the files from this in the future. Different permissions would need to be enforced. Process controller access allows you to find all servers running and get their logs in a domain environment. In the console, we would need to navigate the domain to figure out which host they are looking at to look at the logs on those hosts.
Should be able to do keyword search at this point as well across the logs. Event driven collector of logs so we can index in the background. Based on Heiko's presentation at the F2F.

From logging meeting on 1/28/14

We can point the users to the settings that change the rotation time of log files in the EAP console.
No current way to delete log files. Need to discuss further - this might be dangerous to provide in the console. Right now log files accumulate on the server. Most users probably have a CRON job or manual way to purge after a certain timeframe.