5 Replies Latest reply on May 22, 2002 9:17 AM by bleupen

Design partner for hug amounts of db connections and data?

harold Apr 28, 2002 1:42 PM

I am new to J2EE and I have a question to the more experienced architects here. I would like to build an application based on Jboss-Jetty that will regularly poll multiple databases from different manufactures over the network or the Internet (could be several 1000) and save this data in a central database. This would mean that a full-scale application could receive hug amount of data in bursts or sent on an e2e fashion. For a web shop type of application there is the MVC architecture but is there a design partner for this type of problem where I need to handle hug amounts of db connections and data?
Any help or suggestion would be greatly appreciated.

1. Re: Design partner for hug amounts of db connections and dat

pai_deepak May 2, 2002 10:57 AM (in response to harold)

I do not know what you exactly want to do with the data and why J2EE plays any part in it at all. If you are moving data from one place to another the best solution is to use some replication technology that helps :

* data to be transformed
* partition data
* persist data into central dB
* publish the same data to other related database.

MS SQL Server 2000, PointBase, Oracle and many other databases have very nicely implemented replication that you could use off the shelf. The can also be scheduled as jobs (MS SQL server 2000) or set up for continuous replication.

I hope this help. Or you could give more details so we can discuss it.

Deepak
Actions
2. Re: Design partner for hug amounts of db connections and dat

harold May 3, 2002 2:47 PM (in response to harold)

Thank you for your help, the answer is obvious when you point it out. Oracle seems to be a good choice but is there any shareware about that could do this? I would prefer a low cost solution if it is possible.
Actions
3. Re: Design partner for hug amounts of db connections and dat

jayeshpk May 13, 2002 9:29 PM (in response to harold)

Firebird and FIBRE
http://fibre.sourceforge.net/
Actions
4. Re: Design partner for hug amounts of db connections and dat

davepatterson May 17, 2002 8:42 AM (in response to harold)

If you are working with a number of distributed data sources it may be prudent to look at asynchronous messaging systems. This would be especially true if you use the internet as your transport medium.

JMS is the open standard for java messaging and JbossMQ is based on JMS. There are also a wide range of commercial middleware packages available with an equally wide range of price tags.

Dave Patterson.
Actions
5. Re: Design partner for hug amounts of db connections and dat

bleupen May 22, 2002 9:17 AM (in response to harold)

i'll be your design partner if you give me a hug. :p

b
Actions

Go to original post