Integrate IJ Tracer and NTA tool

[zhfeng]

The IronJacamar tracer produces machine-readable codes at a special TRACE log category at key points during the entire interaction of  the ConnectionEventListener making it easy to apply tooling of the data collected. see http://www.ironjacamar.org/doc/roadto12/tracer.html

NTA (Narayana Transaction Analyser) is a tool for helping to diagnose issues with JTA and JTS transactions in WildFly and JBoss EAP 6.

Jeff Zhang thinks

I think NTA and IJTracer have different goal:

1. NTA try to show all transaction event and tell user what's happened if tx has problem.

2. IJTracer is tool around 'ConnectionEventListener' and display invoke flow.

Q1: What's requirement and what do you see if we integrate them?

Now we are trying to understand the IJ Tracer origin data http://www.ironjacamar.org/doc/roadto12/tracer-example.html

IJTRACER-TxLog-3-1400076333382-578a5496-

IJTRACER-TxLog-22-1400076333383-578a5496-

IJTRACER-TxLog-40-1400076333384-578a5496-e0fab91

IJTRACER-TxLog-41-1400076333386-578a5496-e0fab91

IJTRACER-TxLog-32-1400076333387-578a5496-

IJTRACER-TxLog-13-1400076333388-578a5496-

Q2: What's the means of the third field (3, 22,40)?

The format is defined by the TraceEvent class:

ironjacamar/TraceEvent.java at 1.2 · ironjacamar/ironjacamar · GitHub

Q3: The fifth field looks like the object reference of ConnectionListener and is it possible to associate it to a transaction ? Maybe it need to parse the more JCA TRACE logs ?

You need to parse the 2x (ENLIST) + 3x (DELIST) opcodes (TraceEvent.parse(s))

The IJ applications that uses the tracer data are here:

ironjacamar/as/src/main/java/org/jboss/jca/as/tracer at 1.2 · ironjacamar/ironjacamar · GitHub

Q4: How to generate the image by using the java codes ?

The HTML output from tracer-report.sh explain how to run the command.

Q5: integrate the IJ tracer sequence diagram and the NTA txgraph ?

And 2 suggestion for NTA:

1. Use js library instead of JSF. I hear from Amos that we had started to use Angular.js + Rest.
2. Maybe use Java File NIO.2 since Tailer of commons.io use own thread and blocked inside.

Current Works:

1. [NTA-79] write a demo or a unit test with enable the IJ tracer - JBoss Issue Tracker
2. [NTA-80] make sure the demo and tests work with the lastest wildfly 9.0.0.Alpha1 - JBoss Issue Tracker
3. [NTA-81] Investigate the Java File NIO to replace the commons-io - JBoss Issue Tracker

tomjenkinsonjeff.zhang robinpau jalpedersen

[jesper.pedersen]

I think you need to consider having a special logger that logs the required information - turning on TRACE for everything will have too big an impact. Feel free to take a look at the IronJacamar tracer module.

[zhfeng]

yeah, we  still have a jira [NTA-23] Analyse the performance of using the Log Parsing technique

[mmusgrov]

Jesper Pedersen wrote:

I think you need to consider having a special logger that logs the required information - turning on TRACE for everything will have too big an impact. Feel free to take a look at the IronJacamar tracer module.

What's the tracer module. Is it:

- akin to a dedicated logging level (and would we have to update all of relevant trace lines that the NTA tool needs);

- a byte code manipulator that injects log lines into the code;

- ...

[paul.robinson]

I agree that TRACE logging is not ideal. However it has these advantages:

1) It works with already released products. We want to use NTA today, so it needs to work with what our customers are using.

2) It's easy for GSS to ask for a TRACE log from customers.

Going forward, I think we need a more optimal process. For example, the events are sent directly to NTA and kept in memory. NTA can then export this data as a binary dump, to add to be added to a support case.

[tomjenkinson]

I have said for a while we should develop our own log4j appender that intercepts the log messages, processes them and then writes them to the DB (maybe via some UDP transport on a different server?).  I think that should perform as well as any other technique and prevents the intermediary log file being written. The first thing it can do is drop message codes it knows nothing about.

[paul.robinson]

Tom,

Yes I agree that would probably be the implementation route we'd want to go down. But it will only work for new releases, which will take a while to filter through the community.

[mmusgrov]

Paul Robinson wrote:

Tom,

Yes I agree that would probably be the implementation route we'd want to go down. But it will only work for new releases, which will take a while to filter through the community.

But the longer you wait to do it the more releases it won't be available in - if I was on the team I would bite the bullet and apply the JDFI methodology.

[tomjenkinson]

When looking at concrete things we can do. There are a few options:

1. Pick some features we think are blockers

2. Find some blockers that affect our community

I think it would be useful to look at how much of a limiting factor we gauge the performance to be.raggz and jolee might have some input on that?

[jesper.pedersen]

The IronJacamar tracer is a dedicated module that logs at a special TRACE category in a machine readable format, e.g. tools are needed to make it human readable. Currently there is no need to keep the event information in memory, as the tracer is used to track down ConnectionEventListener error scenarios.

Read more here: http://www.ironjacamar.org/doc/roadto12/tracer.html

[jolee]

My use cases are normally after the fact analysis,  but this is based upon TRACE logging having been captured.  So,  currently I don't have performance concerns relating to use of the tool.  I wouldn't recommend using in PROD,  based upon the same rationale that I wouldn't recommend TRACE logging to be left on in PROD.

[tomjenkinson]

Thanks, that makes sense to me.

[mmusgrov]

.Tom Ross raised  [NTA-74] Please add progress bar - JBoss Issue Tracker  so it may be still be an issue

[zhfeng]

Mike,

It takes about 1 hour to analyze the log file with ~1G. The GSS will be more concerned with the "bad" transactions. e.g. the transactions which are rollback due to the exception. So we had considered about a pre-process to filter these logs and this would be a "quick analyze".

[jesper.pedersen]

You may be interested in [JBJCA-1214] XAResource statistics, which adds XAResource statistics at a pool level, if <wrap-xaresource> is true (default). Could be used to display simple values in the web console (through DMR calls to the statistics nodes)