Simple search solution architecture
Hello all,
I have an idea but it seams that I don't have the proper ability to communicate to non-technical people. Before going in another round of presentations and explanations, I thought about taking the idea to be discussed with like-minded tech fellows. (Tried to find a better category, as described in https://community.jboss.org/message/798106, but in the lack of such, I'm posting here as the simplest form of the idea would be runnable inside a JBossAS, or even Tomcat actually)
We can see clustering, grid and other interesting things relating to the idea I'm about to describe as briefly as I can, but imagine the following simplest deployment: one box running JBoss, with a simple web application displaying a text field that will trigger lucene queries in indexes from 2 legacy databases and also indexes made by checking a folder that should have some text documents. The box has a properly sized storage for the indexes and they can be updated as soon as changes in the data sources happen or respecting a user defined time window.
The idea is creating a non-intrusive, low cost, low overhead, low deployment effort method/solution for querying organization's existing data sources. As in most fields, existing strategies where created based on the reality and assumptions at the time they where created, resulting in solutions that we place under different categories like EAI or EII. They sometimes bring features that are simply not really needed or perceived as advantage by the user or in worse cases, some features can even be seen as unacceptable drawbacks.
The idea fits especially in scenarios where, for different reasons, it would be unacceptable one of the existing bigger/expensive/full of features (not exactly useful for that specific user) solution.
As I don't believe in one size fits all, I rethought some scenarios and it seams the moment for a new idea is that actually simple, just as a new innovative car is built on top of existing components and concepts like wheels, engines and even bolts.
So, instead of trying to make everything look as being only solvable if using a DW solution or pushing it to the still not completely defined world of big data for instance, the idea is to be as close as possible to the user's data sources and data structures, accessing it instead of accessing copies. Also, not try to be smarter then the experts that should use the solution, but actually deliver just what they asked for.
A solution according to this idea is light and deliver results now (in the sense that it can be deployed and customized quickly), together with the existing infrastructure.
From another point of view, existing solutions tend to ignore the current working database tuning and data retention policies and a salesman try to convince that the possible buyer will someday in the future have the same amount of data as a social network or internet search engine.
Well, I think for a first post to maybe start a discussion this is already more paragraphs than than a fey simple ones which was my intention, but as "for every complex problem there is an answer that is clear, simple, and wrong", most probably too simplistic paragraphs could make the post to be dismissed as nonsense. So, what do you think? Questions? Critics?
