1 2 3 Previous Next 39 Replies Latest reply on Jun 22, 2006 4:37 AM by Martin Welss

    Security Exception

    Martin Welss Newbie

      After 15 or 20 minutes processing messages without any complaints, our application suddenly starts throwing jms security exceptions. There is no visible 'change' in the types of messages that could trigger this, but we have an avarage traffic of about 200 messages per second.
      The effect is on all destinations and it does not recover, so messaging is completely unavailable until jboss is restarted.

      Here are the stacktraces, both exceptions always come together in that order


      2006-05-26 12:46:49,639 DEBUG [org.jboss.jms.server.security.SecurityMetadataStore] No SecurityMetadadata was available for acd-applet-info, using default security config
      2006-05-26 12:46:49,640 ERROR [org.jboss.security.auth.spi.UsersRolesLoginModule] Failed to load users/passwords/role files
      java.io.IOException: No properties file: messaging-users.properties or defaults: defaultUsers.properties found
      at org.jboss.security.auth.spi.Util.loadProperties(Util.java:313)
      at org.jboss.security.auth.spi.UsersRolesLoginModule.loadUsers(UsersRolesLoginModule.java:186)
      at org.jboss.security.auth.spi.UsersRolesLoginModule.createUsers(UsersRolesLoginModule.java:200)





      2006-05-26 12:46:49,640 INFO [STDOUT] [MessageDistributorBean#4] message sending failed: javax.jms.JMSSecurityException: User null is NOT authenticated
      2006-05-26 12:46:49,640 ERROR [STDERR] javax.jms.JMSSecurityException: User null is NOT authenticated
      2006-05-26 12:46:49,640 ERROR [STDERR] at org.jboss.jms.server.security.SecurityMetadataStore.authenticate(SecurityMetadataStore.java:181)
      2006-05-26 12:46:49,640 ERROR [STDERR] at org.jboss.jms.server.container.SecurityAspect.check(SecurityAspect.java:252)
      2006-05-26 12:46:49,640 ERROR [STDERR] at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:148)




        • 1. Re: Security Exception
          Ovidiu Feodorov Master

          Are you using the latest release (CR2)?

          • 2. Re: Security Exception
            Martin Welss Newbie

            Yes, it happens with CR2, too!

            • 3. Re: Security Exception
              Tim Fox Master

              According to your stack trace, it can't find messaging-users.properties:


              Failed to load users/passwords/role files
              java.io.IOException: No properties file: messaging-users.properties or defaults:


              Can you verify this file is in the deploy/jboss-messaging.sar directory and has not become corrupt?

              It should contain the following:

              #
              # user=password
              #
              guest=guest
              


              Also check the file permissions of this file if running on unix/linux.

              • 4. Re: Security Exception
                Martin Welss Newbie

                Both files (messaging-users.properties and messaging-roles.properties) are there and readable.

                The point is: this happens after several thousands of messages have been sucessfully delivered and suddenly its broken. There is no special event in our application and jboss is not running out of memory. So the exceptions seem to be the symptom of some more obscure cause...

                It's jboss 4.0.4GA Patch1 with ejb3 profile, messaging CR2 applied and mysql datasource. Most of the messages are ObjectMessages that are marked as non-persistent.

                • 5. Re: Security Exception
                  Ovidiu Feodorov Master

                   

                  The point is: this happens after several thousands of messages have been sucessfully delivered and suddenly its broken. There is no special event in our application and jboss is not running out of memory. So the exceptions seem to be the symptom of some more obscure cause...


                  That's odd, indeed.

                  Looks to me like a classloading issue.

                  Can you consistently replicate it, so we can attempt to?

                  • 6. Re: Security Exception
                    Martin Welss Newbie

                    Yes. With CR2 it happens always after about 30 minutes.
                    As I mentioned earlier, the traffic is at about 200 messages per second, there are topics, queues and ejb3 mdb involved, but no clustering. OS is Linux and java is Bea jrockit.

                    • 7. Re: Security Exception
                      Tim Fox Master

                      Can you replicate in a test case (or similar) and send to me or Ovidiu so we can investigate further?

                      Thanks

                      • 8. Re: Security Exception
                        Martin Welss Newbie

                        It seems to be a memory leak. When we try it with sun jdk 1.5.0_06 instead of bea jrockit, jboss runs clearly out of memory.
                        Our applications uses lots of ObjectMessages in both queues and topics.

                        • 9. Re: Security Exception
                          Tim Fox Master

                          Ok, a few more questions:

                          You say the "traffic" is 200 msgs/sec. Do you mean you are *sending* 200 msgs/sec, or *consuming* 200 msgs/sec, or perhaps a *throughput* of 200 msgs/sec.

                          If you're sending messages to a queue, but not consuming them, then at some point you are going to run out of memory (unless you configure paging - we'll get to this later).

                          How big are the messages you're sending?

                          Are they all non persistent?

                          Which database are you using?

                          • 10. Re: Security Exception
                            Martin Welss Newbie

                            The throughput is about 200 msgs/s (please don't nail me down on this ;-)

                            On every queue sits an ejb3 mdb, so the messages should get consumed. Furthermore we don't detect any "lost" messages.

                            The size of the messages is several KB.
                            They are all non-persistent.

                            DB is mysql.

                            • 11. Re: Security Exception
                              Tim Fox Master

                              Just because the messages should get consumed doesn't mean they are being consumed.

                              And even if they are being consumed it doesn't mean they are being consumed at the same rate as they are being sent.

                              The most likely cause here is you are sending messages fasting than you are consuming them, consequently they will build up on the queue or subscription, until the system runs out of storage for them.

                              As I say, if there is an underlying problem here then you should be able to replicate it in a test and that will give us something to start from.

                              But first you need to validate you're not sending messages fasting than you're consuming.

                              • 12. Re: Security Exception
                                Martin Welss Newbie

                                timfox wrote:


                                The most likely cause here is you are sending messages fasting than you are consuming them, consequently they will build up on the queue or subscription, until the system runs out of storage for them.


                                Oops, jboss-messaging just let the VM run out of memory?? Is that your idea of good desgin? If that is not a bug, but a feature as you put it, we obviously must rethink our plan to use jboss-messaging!

                                Anyway, we don't see an increasing delay of the messages and the cpu load is low. So I still believe this is a memory leak.



                                • 13. Re: Security Exception
                                  Tim Fox Master

                                   

                                  "mwelss" wrote:
                                  timfox wrote:

                                  Oops, jboss-messaging just let the VM run out of memory?? Is that your idea of good desgin? If that is not a bug, but a feature as you put it, we obviously must rethink our plan to use jboss-messaging!



                                  I said storage, not memory :) If you send messages at a greater rate than you consume them they need to be stored somewhere (memory or on disk most probably).

                                  Assuming storage is finite, then it will someday run out. This will happen with any messaging system, not just jboss messaging.

                                  If configured properly JBoss Messaging will not cause the VM to run out of memory.

                                  Please see section 5.3.2.2 of the getting started guide. http://labs.jboss.com/file-access/default/members/jbossmessaging/freezone/docs/guide-1.0.1.CR2/html_single/index.html#conf.destination.paging

                                  With paging channels, you specify what the maximum in memory size of the queue or subscription is, and jboss messaging will ensure that it does not hold more than that many messages in the queue / subscription in memory at any one time.

                                  It then transparently pages them to from disk as necessary. I've tested this with over 10 million 2K messages, so far and we should be able to go much higher.

                                  How many queues and topics do you have?

                                  Can you please tell me what the fullSize parameter is set to for these queues and topics?

                                  What is the -Xmx setting for the VM?

                                  I am not saying you don't have a memory leak somewhere in the VM, but at the moment it hard for us to know what to do since we haven't been given anything to replicate.

                                  If you can give us instructions of what we need to set up in order to replicate this that would be great and we can make some progress.

                                  • 14. Re: Security Exception
                                    Martin Welss Newbie

                                    I have a testcase that replicates it. The cause seems to be a topic selector whith an "OR" operator.
                                    I will send the exmaple by email to you, it needs a messaging config based on ejb3.

                                    Thanks.

                                    1 2 3 Previous Next