-
1. Re: How to do some action on all the cache enties?
matlach Feb 6, 2013 11:16 PM (in response to szhem)1 of 1 people found this helpfulHi Sergey,
I think you could use the Map Reduce framework which in your Mapper will get called for each Key Value entries of a given Cache.
see : https://docs.jboss.org/author/display/ISPN/Map+Reduce+framework
Hope this helps,
Mat
-
2. Re: How to do some action on all the cache enties?
szhem Feb 7, 2013 1:27 PM (in response to matlach)Hello Mat,
Thanks a lot for the link. I've already tried it.
I understood that I'd like to use the same code for all my environments (unit testing, local, uat, prod, etc.)
As map-reduce works only with distributed caching it's hardly possible to use the same code for different cache configurations in different environments.
As a rule distributed executor service works wito replicated and distributed caches, but not with a local ones.
Regards,
Sergey
-
3. Re: How to do some action on all the cache enties?
matlach Feb 7, 2013 6:46 PM (in response to szhem)Yep sadly MapReduceTask only works with distributed cachev(just tried it with replicated cache and it doesn't work).
I also guess using cache.values() is not an acceptable approach (I dont even know if in either mode the cache store value are returned and if so, if it could turn into an OOME if too much data is returned).
Though, by looking at the source code of MapReduceTask, the MapCombineCommand, ReduceCommand are managed differently when executed locally.
like :
protected Set<KOut> executeMapPhase(boolean useCompositeKeys) throws InterruptedException, ExecutionException { RpcManager rpc = cache.getRpcManager(); MapCombineCommand<KIn, VIn, KOut, VOut> cmd = null; Set<KOut> mapPhasesResult = new HashSet<KOut>(); List<MapTaskPart<Set<KOut>>> futures = new ArrayList<MapTaskPart<Set<KOut>>>(); if (inputTaskKeysEmpty()) { for (Address target : rpc.getMembers()) { if (target.equals(rpc.getAddress())) { // <<<<<<<< if myself build command in a specific way cmd = buildMapCombineCommand(taskId.toString(), clone(mapper), clone(combiner), null, true, useCompositeKeys); } else { cmd = buildMapCombineCommand(taskId.toString(), mapper, combiner, null, true, useCompositeKeys); } MapTaskPart<Set<KOut>> part = createTaskMapPart(cmd, target, true); part.execute(); futures.add(part); }
What would be needed is to enhance the
private void ensureProperCacheState(AdvancedCache<KIn, VIn> cache) throws NullPointerException, IllegalStateException { if (cache.getRpcManager() == null) // throw new IllegalStateException("Can not use non-clustered cache for MapReduceTask"); // do not fail performLocalMapReduce(); if (cache.getStatus() != ComponentStatus.RUNNING) throw new IllegalStateException("Invalid cache state " + cache.getStatus()); if (cache.getDistributionManager() == null) { // throw new IllegalStateException("Cache mode should be DIST, rather than " // + cache.getCacheConfiguration().clustering().cacheModeString()); // do not fail performLocalMapReduce(); // as data is the same accross the cluster } }
That's just my idea, what do you think about it ?
Maybe we should fill a JIRA feature request issue with this ?
-
4. Re: How to do some action on all the cache enties?
szhem Feb 11, 2013 7:10 AM (in response to matlach)Hello Mat,
Thanks a lot for analysis and clarification.
I really like your idea and I suppose the same code must be runnable event if the cache topologies are different (local, replicated, distributed, etc.).
The configuration should not force the developer to write the code in any special way.
I've raise a JIRA - https://issues.jboss.org/browse/ISPN-2812.
-
5. Re: How to do some action on all the cache enties?
matlach Feb 11, 2013 7:46 AM (in response to szhem)just voted for it,
thanks you sergey !
-
6. Re: How to do some action on all the cache enties?
amanukyan Feb 11, 2013 11:18 AM (in response to szhem)Hi Sergey,
For your information, the Distributed Executors support LOCAL caches starting from version 5.2.0 (see the following enhancement - https://issues.jboss.org/browse/ISPN-2192 ).
Regards,
Anna.
-
7. Re: How to do some action on all the cache enties?
szhem Feb 14, 2013 3:37 PM (in response to szhem)Hello Anna,
Many thanks for the notice.
So, Distributed Executors support local, replicated, distributed envs.
Will be great to have the same thing for the map-reduce.
Regards,
Sergey