14 Replies Latest reply: Mar 16, 2011 10:32 AM by Ramesh Reddy RSS

    java.io.IOException: Too many open files

    allen li Newbie

      Hi all,

       

                I did a test about deploy VDB to Teiid. I used 20 thread to create VDB. Teiid server throw below log. I want to know why it is and how to avoid it.

      java.io.IOException: Too many open files

              at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

              at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)

              at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:245)

              at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

              at java.lang.Thread.run(Thread.java:636)

       

       

      Thanks

      Best Regards

        • 1. java.io.IOException: Too many open files
          Ramesh Reddy Master

          Allen,

           

          How many client connection do you have. What platform. Teiid certainly does not throw too many files for moderate level of usage. However if you are opening lot of connections and holding on to them and doing the large queries (millions of records) there slight possibility. But you would have describe more to about your use case.

           

          Ramesh..

          • 2. java.io.IOException: Too many open files
            Steven Hawkins Master

            It may help to roughly account for things that may use a file handle:

             

            -Each incoming socket connection

            -Each outgoing socket connection to a source

            -The buffermanager file cache, which is limited to 64 open files by default

            -Each read from a Teiid filestore over the previous limit.  There will never be more concurrent reads than maxActivePlans

            -Any file usage by your sources

             

            We have also seen a similar issue in the past with large jvm sizes (32gigs or more) and some vendor drivers.

             

            Avoiding the issue may include increasing your open file limit, limiting source connections, etc.

             

            Steve

            • 3. java.io.IOException: Too many open files
              allen li Newbie

              Thanks your help.

               

              platform: jboss soa 5.0.1   linux

               

              limit parameter

              core file size          (blocks, -c) unlimited

              data seg size           (kbytes, -d) unlimited

              scheduling priority             (-e) 0

              file size               (blocks, -f) unlimited

              pending signals                 (-i) 135167

              max locked memory       (kbytes, -l) 32

              max memory size         (kbytes, -m) unlimited

              open files                      (-n) 1024

              pipe size            (512 bytes, -p) 8

              POSIX message queues     (bytes, -q) 819200

              real-time priority              (-r) 0

              stack size              (kbytes, -s) 10240

              cpu time               (seconds, -t) unlimited

              max user processes              (-u) 135167

              virtual memory          (kbytes, -v) unlimited

              file locks                      (-x) unlimited

               

              JVM arguments:

               

              -Xms1303m -Xmx3303m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+HeapDumpOnOutOfMemoryError

               

              This case is used 20 concurrent thread to deploy and undeploy VDB

              Duration: 2 hours

               

              Thanks

              Best Regards

              • 4. java.io.IOException: Too many open files
                Ramesh Reddy Master

                Allen,

                 

                Are you saying that you have 20 threads that are constantly deploying and undeploying the vdbs?

                 

                Ramesh..

                • 6. java.io.IOException: Too many open files
                  allen li Newbie

                  Hi Ramesh,

                           

                            Our logic need deploy and undeploy VDB, So we have to pass this case. Do you have any advice ?

                   

                   

                   

                  Thanks

                  Best Regards

                  Allen

                  • 7. java.io.IOException: Too many open files
                    Ramesh Reddy Master

                    Allen,

                     

                    This is a interesting test. Usually any typical applications just deploy a VDB and use it. The queries could be done using multiple threads. Anyway couple questions,

                     

                    1) Are you using 20 different vdbs? or a single vdb.

                    2) Can provide the testcase you are using to test this.

                     

                    Thanks

                     

                    Ramesh..

                    • 8. java.io.IOException: Too many open files
                      allen li Newbie

                      Ramesh,

                       

                                In our architecture, we will used many VDB. This VDB will be deployed and undeployed anytime.

                       

                       

                      Thanks

                      Best Regards

                      Allen

                      • 9. Re: java.io.IOException: Too many open files
                        allen li Newbie

                        Hello Ramesh,

                         

                                  I have tested with deploying  and undelpoying of VDB.

                        I found :

                            When deploy 2 VDB and undeploy 2 VDB, then I execute "lsof" to check. I found teiid server will increase 6 "open file"

                        They are:

                        java    6780 root  395r  IPv4          117035708                TCP 192.168.3.3:59916 (LISTEN)

                        java    6780 root  416u  IPv4          117035704                TCP 192.168.3.3:59915 (LISTEN)

                        java    6780 root  424u   REG              253,0   749935  68519379 /opt/jboss-teiid-5.0.1/jboss-as/server/var/mwm/jboss/server/teiid-server/tmp/teiid/buffer/b_temp-stream_0_4310886018569254268.tmp

                        java    6780 root  426u   REG              253,0   749933  68519380 /opt/jboss-teiid-5.0.1/jboss-as/server/var/mwm/jboss/server/teiid-server/tmp/teiid/buffer/b_temp-stream_0_5857253331971952634.tmp

                        java    6780 root  427u  IPv4          117035721                TCP 192.168.3.3:59917 (LISTEN)

                        java    6780 root  428u  IPv4          117035722                TCP 192.168.3.3:59918 (LISTEN)

                         

                         

                        I found teiid add 2 open file  and 4 listen. Maybe it is caused this issue. Do you think it is leak for teiid?

                        Thanks

                        • 10. java.io.IOException: Too many open files
                          allen li Newbie

                          Hello Ramesh,

                           

                              

                           

                                    I met many question about teiid in our endurance test. I want to know whether it is reasonable. In our architecture, we will used many VDB.Maybe the number will over 100. And under the each VDB, it will have another vdb and an oracle.  I think this architecture is so large for teiid.

                          Do you have any advice? Or do you think how many should teiid in the jboss?

                           

                          Thanks your help.

                           

                          Allen

                          • 11. java.io.IOException: Too many open files
                            Ramesh Reddy Master

                            Allen,

                             

                            This is more of a tuning question, please open another discussion thread for this and post the link here. Simple answer, possible depending upon your VM size and capabilities of the server.

                             

                            Thanks.

                             

                            Ramesh..

                            • 13. java.io.IOException: Too many open files
                              allen li Newbie

                              Dear Ramesh,

                                  

                                             I have tested our test case for one day.

                               

                                        [root@charon test]# lsof -p 6780 | grep java|wc -l

                                        1546

                               

                              We can see. This value has increased to 1546. So I think it is bug.

                               

                               

                              Thanks for your help.

                               

                              Best Regards

                              Allen

                              • 14. java.io.IOException: Too many open files
                                Ramesh Reddy Master

                                Allen,

                                 

                                Yes, but I need to do further testing. Follow the above JIRA for the progress of the issue.

                                 

                                Thanks

                                 

                                Ramesh..