1 2 Previous Next 19 Replies Latest reply on Dec 17, 2009 6:04 PM by deanhiller2000

    warnings in jacorb jboss5.1 all the time...

    deanhiller

      I keep seeing these logs over and over...

      2009-12-03 13:38:38,041 WARN [com.arjuna.ats.jts.logging.loggerI18N] (http-0.0.0.0-8080-7) [co
      m.arjuna.ats.internal.jts.resources.errsavefail] ExtendedResourceRecord.doSave failed. Returnin
      g default value: true
      2009-12-03 13:38:38,043 WARN [jacorb.poa.controller] (RequestController-1) rid: 3186 opname: s
      aveRecord request rejected with exception:
      2009-12-03 13:38:38,050 WARN [jacorb.poa.controller] (RequestController-1) rid: 3188 opname: c
      ommit cannot process request, because object doesn't exist
      2009-12-03 13:38:38,052 WARN [jacorb.poa.controller] (RequestController-1) rid: 3188 opname: c
      ommit request rejected with exception:
      


      Anyway to fix this? In tomcat, we had no warnings so we could easily search on WARN for any problems when launching a new version of our software. With jboss, it sucks because bad problems are lost in the noise with the amount of traffic we get. How do I get around these warnings?

      I am trying to avoid going back to tomcat.
      thanks,
      Dean

        • 1. Re: warnings in jacorb jboss5.1 all the time...
          adinn

           


          Anyway to fix this? In tomcat, we had no warnings so we could easily search on WARN for any problems when launching a new version of our software. With jboss, it sucks because bad problems are lost in the noise with the amount of traffic we get. How do I get around these warnings?


          JBoss is printing these warnings because there is a problem. You may think that warning you about problems sucks melons but mileage does vary on such matters.

          The JBoss error (the one with the errsavefail tag) is being printed because the transaction manager is unable to log details of a transaction to disk when the transaction is being prepared. Specifically, one of the OTS participants registered with the TX throws an error when it is told to persist its state. The transaction manager continues with the commit anyway and jacorb prints further errors. It appears from the jacorb messages that this is because the object handle which has been registered with the transaction manager is invalid or references an object which is invalid. Perhaps you should start looking from there.

          Is there a reason why you are using jacorb and the JTS version of the transaction manager? Were you using an ORB when you weer running with tomcat?


          • 2. Re: warnings in jacorb jboss5.1 all the time...
            mmusgrov

            The messages indicate that you had a remote participant in the transaction which failed during commit. Since you are using JTS the transaction manager is trying to contact the participant using a Corba object reference (which the TM got when the participant first joined the transaction) but the reference is no longer valid - perhaps because the participant is no longer running.

            Do you have any idea which kinds of participants were involved in the distributed transaction?

            • 3. Re: warnings in jacorb jboss5.1 all the time...
              mmusgrov

               

              The messages indicate that you had a remote participant in the transaction


              Correction. The recovery subsystem needs to be able to recover participants and can run independently of the TM. Hence the TM records enough information to enable them to be recovered remotely.

              In other words the participants aren't necessarily remote. Are all the original jars still deployed?


              • 4. Re: warnings in jacorb jboss5.1 all the time...
                deanhiller

                I am actually not sure. I know what we are trying to do is JMS and db transactions so that they are tied and committed together though in testing it was not really working out too well. What should we be using?

                I think my colleague will reply as well as he may have more info.
                thanks,
                Dean

                • 5. Re: warnings in jacorb jboss5.1 all the time...
                  deanhiller

                  oh, and we are not using corba anywhere either. I just assumed JTS was automatically using corba for who knows what reason.

                  • 6. Re: warnings in jacorb jboss5.1 all the time...

                    Since we are in a clustered environment, we followed the directions in
                    jboss-5.1.0.GA/docs/examples/transactions/README.txt
                    to enable JTS for full distributed transaction support. This utilizes jacorb. Other than that, there aren't remote participants in the errors that Dean is talking about. Nor has a failover occurred which would have required distributed completion of in progress transactions...

                    Any other ideas?

                    • 7. Re: warnings in jacorb jboss5.1 all the time...
                      marklittle

                       

                      "deanhiller" wrote:
                      oh, and we are not using corba anywhere either. I just assumed JTS was automatically using corba for who knows what reason.


                      That would be because JTS is the Java language mapping of the OTS, from the OMG, which is the home of CORBA. So if you are using JTS you must be using CORBA.

                      • 8. Re: warnings in jacorb jboss5.1 all the time...
                        mmusgrov

                         

                        In other words the participants aren't necessarily remote. Are all the original jars still deployed?


                        Are all the original jars still deployed?

                        And if they are then try checking with the database and the JMS queues you are using to see which transactions they think are still pending.

                        • 9. Re: warnings in jacorb jboss5.1 all the time...
                          deanhiller

                          yes, all the original jars are still there and the ones we added of course.

                          hmmm, as far as checking for pending transactions. Our test environment is temporarily down, and I was told I can't easily do that in production, though any pending transactions soon become not pending as they are committed. We have not had production run out of connections or anything so I assume transactions are getting committed.

                          I am not sure what to look for in the JMS tables and database.

                          I am very confused here by your statement and how it relates to the warnings I pasted above. I know that there are other warnings on replaying transactions every once in a while as well. This whole replaying transactions thing is a blackbox to me and I would like to understand more so I can dig into those too(and maybe someday understand how to read from those logs if there is a viewer into them).

                          thanks for any pointers here. Also, is there a place to look in the code for who is using jacorb and not using it correctly or something(ie. the "because object doesn't exist" error). I would like to make sure that is not happening if possible.

                          ps. I don't think we were getting these warnings in jboss 5.0.1. and we didn't change our code when porting to 5.1. Is this possibly some new bug? How do I look into this if I want to? Where would I even start?

                          thanks,
                          Dean

                          • 10. Re: warnings in jacorb jboss5.1 all the time...
                            deanhiller

                            mark.little,
                            You had said "if you are using JTS you must be using corba". By this, you were just confirming my statement that JTS uses corba, correct? as that sentence could be taken to mean "you must be using corba" ie. me. We are not using corba "directly" though at all......JTS is. Can you please confirm?

                            If JTS is only for those who use corba though, then what should we use instead of JTS. We have a clustered db and a clustered jboss and JMS. JMS tables are in the same clustered db as all our other data. We would like to commit JMS messages sending along with data into the database at the same time.....though that wasn't working so we hacked something else and had JMS send after data was in the database....not sure why it wasn't working. There seemed to be little documentation on doing it correctly. Thanks for any info here.

                            NONE of our services are using corba.
                            Dean

                            • 11. Re: warnings in jacorb jboss5.1 all the time...
                              marklittle

                              Yes, I'm confirming that if you are using JTS then you are implicitly using CORBA: there is no way to use JTS without CORBA.

                              Try using the other JTA implementation. As long as you don't need distributed transactions you'll be fine.

                              • 12. Re: warnings in jacorb jboss5.1 all the time...
                                deanhiller2000

                                K. so I setup a clean cluster to try to reproduce this issue we are seeing in production.

                                 

                                We would like to use distributed transactions and I ran ant jts to setup the JTS from the example directory.  It is clean so far.  I installed only one of the production apps.  Maybe it takes a two phase commit failure before it would actually work?

                                 

                                If I have a jboss cluster of 2 nodes, and my node 1 goes down while a transaction had started but commit had not been called, there is no recovery for that, correct?  only after commit is called, there may be some 2 phase commit recovery..question there on that recovery though...does the second node handle the rest of the transaction or does it wait for node 1 to come back online and rollback or commit the transaction that was in progress when it went down?

                                 

                                Also, how can I further compare my cluster with production?  I am thinking about wiping out the data directory in test to see if that fixes it once test is back online.

                                 

                                thanks,

                                Dean

                                • 13. Re: warnings in jacorb jboss5.1 all the time...
                                  adinn

                                  {quote}If I have a jboss cluster of 2 nodes, and my node 1 goes down while a transaction had started but

                                  commit had not been called, there is no recovery for that, correct?  only after commit is called,

                                  there may be some 2 phase commit recovery..question there on that recovery though...does the second

                                  node handle the rest of the transaction or does it wait for node 1 to come back online and rollback

                                  or commit the transaction that was in progress when it went down?{quote}


                                  The transaction coordinator (i.e. JBoss) writes a TX log record to disk when all participants have

                                  answered yes to a prepare request. It deletes this log record when the participants have all answered

                                  committed to a commit request. So, a crash in this window will leave a record on disk which is used

                                  by the JBoss recovery thread to roll forward (i.e. resend commit requests).

                                   

                                  At reboot the recovery thread resends commits to every participant even if they already have replied

                                  with committed. It has to becase the log record was written before sending commmits so it doesn't

                                  know whether the participant saw the commit or not. The participant (e.g. a database) ensures that

                                  repated commit requests are ignored.

                                   

                                  The participant writes a TX log record to persistent storage just before replying to a prepare request

                                  with a prepared response. This record contains details of locks taken and data modifications made

                                  during the TX. It delets this record when it receives a  commit request. If a crash happens during

                                  this window then the log record can be used either to roll forward the changes or back them out.

                                   

                                  The recovery thread and resource manager compare notes after a crash to resolve transactions caught

                                  in either of these windows. If the resource manager has a log record for a TX which the recovery

                                  thread does not know about then a crash happened before prepare which means the resource manager is

                                  told to roll back. If the recovery thread has a log record for a TX which the resource manager does

                                  not know about then this means the resource manager must have already been sent a commit so it ignores

                                  a repeated commit request. If they both have a log record then the resource manager has not yet seen

                                  a commit so it still rolls forward the changes. Either way the data can be made consistent, locks dropped

                                  and log records deleted. This works even if a crash happens part way through recovery.


                                   

                                  {quote}Also, how can I further compare my cluster with production?  I am thinking about wiping out the

                                  data directory in test to see if that fixes it once test is back online.{quote}


                                  One thing which might account for the problems you are seeing is that you have left a stale log record

                                  in the JBoss store because you killed JBoss while a TX was in the prepare-commit window. If you reset

                                  your ORB bext time you restared JBoss but did not remove this log record then it could contain a

                                  CORBA object reference which no longer identified an object known to the ORB. If you see the problem

                                  happen again shut JBoss down cleanly and look in the JBOSS_HOME/server/<xxx>/data for a directory called

                                  tx-object-store. It contains a tree of directories used to store TX log records. If any of these

                                  directories contains an actual file then it represents an in-flight TX caught in the window. It might

                                  be there because of a real TX failure or it might just be a left-over from killing JBoss. You can check

                                  the message log in JBOSS_HOME/server/<xxx>/log to rule out the former possibility.

                                  • 14. Re: warnings in jacorb jboss5.1 all the time...
                                    deanhiller2000

                                    It sounds like you are saying node 2 is not involved in recovery at all.  I wonder then why these properties in arjuna have to be unique...

                                     

                                    <property name="com.arjuna.ats.arjuna.xa.nodeIdentifier" value="1"/> in jbossts-properties.xml

                                    and

                                    jacorb.implname=JBoss in jacorb.properties

                                     

                                    It sounds like node 2 and node 1 though do it's own recovery from what you said above?  Why do these properties need to be unique?  Also, what happens if I accidentally did not have it unique(ie. I ran ant jts only and though I was done, but ant jts did not modify that stuff like it should have in the doc/examples/transaction directory.

                                     

                                    thanks,

                                    Dean

                                    1 2 Previous Next