2 Replies Latest reply on Mar 17, 2008 8:26 PM by vashistvishal

    Performance Issue with large chunk of data retreival.

      Guys,

      I would like to get some opinion on performance issue we are facing.

      1. We have a 2/3 tier J2EE application where we are using Struts, SLSB and just JDBC (wrapped in DAO's) in Jboss 4.04 on Windows, to extract data.

      2. There are few reports which requires atleast for 2/3 years of transaction data where every day we add 300-500 records in DB.


      When we run these specific reports we get out of memery error.
      We can see why, it happens at Jdbc layer more often.


      Question is:
      1. First How do you find the bottleneck in a system like this, any recomended specific tool.

      2. How would you change/structure the specific DAO's in question to scale it (getting data of 10,00 records or more)

      3. Any other appraoch, like removing JDBC wrapper DAO and using hibernate or caching..

      Any suggestions...

        • 1. Re: Performance Issue with large chunk of data retreival.

          OK, I am posting this as just me, not who I work for. :)

          First, tell me who the person is that will actually ready that huge report? Must be one of the fastest speed readers in the world.

          OK, So first, reports like these should be run over-night, not as ad-hoc.

          2. There are products out there that generate reports like these, not necessarily in Java (did I just say that). Anyway they are products designed specifically for these types of reports, and therefore are "the right tool for the job" and not trying to pound a square peg into a round hole with a straw.

          3. You might want to handle the data in batches, only bring down enough records at a time to do one page, then release that data to free up memory, and get the next page, and build up the report like that to stay away from out of memory issues. Now this will slow it down, but then again at this point you don't even get the report done.

          Personally, I'd look for a product that does huge reports, that that is what they were built for, rather than forcing a bad solution.

          Thanks

          Mark

          • 2. Re: Performance Issue with large chunk of data retreival.

            Thanks for yr reply.
            It does help.

            What you are proposing is right in terms on intermediate solution.
            some sort of paging mechanism where report data is done in page by, and probably curses approach on DB side.

            anyway what sort of products you have in mind.. anyy insights..

            Cheers