Skip navigation

If you like graph databases, you probably have tried the Gremlin query language.

After some development, you get to a state when you have quite a large database, and need to query it interactively, mainly for the purpose of learning more complex queries and applying them in your app.

 

On various Gremlin docs sites, you can see code like this:

 

gremlin> g = TitanFactory.open('../server/conf/titan-cassandra-es.properties')
==>standardtitangraph[cassandra:[127.0.0.1]]

 

The problem is that Gremlin, as distributed, can't open Titan graph files - it doesn't contain the libs.

So this is what I had to do to make it work.

It's probably not the best way to do it, quite unsystemic, but got me to a working state, so...

 

1) Copy these into gremlin-groovy-2.5.0/lib:

(I just took the newest I had in my local Maven repo.)

  • .m2/repository/com/google/guava/guava/18.0/guava-18.0.jar
  • .m2/repository/com/thinkaurelius/titan/titan-berkeleyje/0.5.4/titan-berkeleyje-0.5.4.jar

2) Then the missing classes were appearantly from Titan, so I downloaded Titan 0.5.4 and copied

  • titan-0.5.4-hadoop2/lib/je-5.0.73.jar

3) And there were more missing classes, so I just copied everything from titan-0.5.4-hadoop2/lib/

and removed those which seemed to duplicating, only other version:

  • commons-io-2.1.jar
  • commons-beanutils-core-1.7.0.jar
  • concurrentlinkedhashmap-lru-1.3.jar
  • guava-15.0.jar
  • netty-3.2.7.Final.jar
  • stax-api-1.0-2.jar

 

Then it started working:

 

$ gremlin
         \,,,/
         (o o)
-----oOOo-(_)-oOOo-----
gremlin> import com.thinkaurelius.titan.core.TitanFactory
...
gremlin> g = TitanFactory.open("target/reportsW2/WebLogicExamples_wldfdemo-wlnav.war-report/graph/TitanConfiguration.properties");
... -> Few exceptions but finally:
==>titangraph[berkeleyje:/home/ondra/work/Migration/WindupCompar1vs2/target/reportsW2/WebLogicExamples_wldfdemo-wlnav.war-report/graph/titangraph]

gremlin> g.has("w:vertextype", "BaseWindupConfiguration")
--> com.thinkaurelius.titan.graphdb.database.StandardTitanGraph cannot be cast to com.tinkerpop.blueprints.Element

gremlin> g.V.has("w:vertextype", "BaseWindupConfiguration")
19:11:34.155 [main] DEBUG c.t.t.g.transaction.StandardTitanTx - Guava vertex cache size: requested=20000 effective=20000 (min=100)
19:11:34.155 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created dirty vertex map with initial size 32
19:11:34.156 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created vertex cache with max size 20000
19:11:34.160 [main] DEBUG c.t.t.g.b.TitanBlueprintsGraph - Created new thread-bound transaction standardtitantx[null]
19:11:34.174 [main] DEBUG c.t.t.g.transaction.StandardTitanTx - Guava vertex cache size: requested=20000 effective=20000 (min=100)
19:11:34.174 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created dirty vertex map with initial size 32
19:11:34.174 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created vertex cache with max size 20000
19:11:34.208 [main] DEBUG c.t.t.g.transaction.StandardTitanTx - Guava vertex cache size: requested=20000 effective=20000 (min=100)
19:11:34.208 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created dirty vertex map with initial size 32
19:11:34.208 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created vertex cache with max size 20000
19:11:34.214 [main] DEBUG c.t.t.g.d.s.kryo.ThreadLocalKryos - [0x41385767] Returning thread-local Kryo com.esotericsoftware.kryo.Kryo@69ee8449
19:11:34.226 [main] DEBUG c.t.t.g.transaction.StandardTitanTx - Guava vertex cache size: requested=20000 effective=20000 (min=100)
19:11:34.227 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created dirty vertex map with initial size 32
19:11:34.227 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created vertex cache with max size 20000
19:11:34.232 [main] DEBUG c.t.t.g.transaction.StandardTitanTx - Guava vertex cache size: requested=20000 effective=20000 (min=100)
19:11:34.232 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created dirty vertex map with initial size 32
19:11:34.232 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created vertex cache with max size 20000
19:11:34.234 [main] DEBUG c.t.t.g.transaction.StandardTitanTx - Guava vertex cache size: requested=20000 effective=20000 (min=100)
19:11:34.234 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created dirty vertex map with initial size 32
19:11:34.234 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created vertex cache with max size 20000
19:11:34.236 [main] DEBUG c.t.t.g.transaction.StandardTitanTx - Guava vertex cache size: requested=20000 effective=20000 (min=100)
19:11:34.237 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created dirty vertex map with initial size 32
19:11:34.237 [main] DEBUG c.t.t.g.t.v.GuavaVertexCache - Created vertex cache with max size 20000
==>v[256]
gremlin> g.V("w:vertextype", "BaseWindupConfiguration")
==>v[256]

 

Enjoy

Windup 2.3.0.Final was released.

 

What's the news?

 

  • In this release, we only included the reviewed rules - targetted mainly for WebSphere and WebLogic.
  • The reviewed rules have more useful information in them, including links that point to Red Hat Knowledge Base and other sources.
  • Thanks to parallelization of the Java analysis, the run time got reduced significantly.

 

Download here.

Stay tuned, soon you'll see a hands-on video by Brad Davis.

 

And - most important of all - if you're going to Red Hat Summit, make sure to attend the Windup presentation!

ozizka

Pobyt v Raleigh v kostce

Posted by ozizka Jun 4, 2015

Hotwire ne.

Hospody - Pit, BuKu, Speak Easy.

Hotely - Days Inn, Extended Stay America - taxi $15.

Red Hat Tower

Vlak do NY

Windup 2.2 was primarily improving performance. That is still a big priority. For Windup 2.3, we also review the existing rules, to be more useful than just saying "this is bad" - we add links to Red Hat knowledge base, javadoc, and more meaningful messages.

 

Another news is that XML rules development should be easier and more smooth. As a side effect of us working on the rules, we see what's the biggest pain to achieve, what's missing, and what's not working. All that should be improved and fixed until the JBoss summit. Also, there's a new XSD for the Java EE app related rules, which will make it easy for your favorite IDE (provided it supports XSD-based XML code completion).

 

Stay tuned!

ozizka

Windup 2.2.0 Released

Posted by ozizka Mar 27, 2015

A new veresion of Windup was released.

What's new? Windup is faster, rules are shorter, few bugfixes, a bit nicer API.

Download here.

 

Enjoy!

 

 

 

   Bug  WINDUP-517

Application Index page lacks technology list and effort information

   Bug  WINDUP-518

XML Handlers do not support nested rules

   Bug  WINDUP-524

Fix the cyclic dependency between windup and windup-rulesets

   Bug  WINDUP-511

Quickstarts tests fail with a NPE

   Task  WINDUP-512

Rename Implicit to Dependent

   Enhancement  WINDUP-508

WindupRuleProvider should contain a location/origin field (to indicate where it was defined)

   Bug  WINDUP-510

Build fails with BOM missing

   Enhancement  WINDUP-470

Classes in java.lang package may differ in how they are parsed

   Enhancement  WINDUP-506

Identify Archives Ruleset: Improve memory footprint

   Enhancement  WINDUP-409

Allow rules metadata to be specified with annotations - @Rules( phase = ..., ...)

   Feature Request  WINDUP-402

Change ruleset category metadata to tag-like system (make it a set of strings)

   Feature Request  WINDUP-521

Rule Externalization - Create XML Handlers for Rule metadata

   Task  WINDUP-530

Rename the module "utils" to "windup-utils"

   Task  WINDUP-513

Phase pseudo-RuleProviders to have ~Phase suffix

   Bug  WINDUP-516

[Performance] Multiple valueStores even if the rule does not use parameters

   Feature Request  WINDUP-522

Rule Externalization - Create user-interface for selecting source/target platforms

   Enhancement  WINDUP-520

Rule Externalization - Introduce a Rule, RuleProvider and RulesetMetadata API

   Bug  WINDUP-542

Fix severe regression in AST performance

   Bug  WINDUP-544

StackOverflowException when generating Application report

   Feature Request  WINDUP-523

Rule Externalization - Allow filtering of RuleProviders by tags (categories)

   Enhancement  WINDUP-515

Add Severity to Hint

   Sub-task  WINDUP-464

WINDUP-463 Expand all addons with simple layout to use full api/impl/tests structure

   Sub-task  WINDUP-465

WINDUP-463 Review all rules, organize into an external repository, and add useful hint information with links to knowledge articles

Interesting news. If I understood it correctly:

Datastax bought the company behind graph database Titan (last version is 0.5.4).

They announced that they will finish Titan 1.0 and donate it to the Apache Foundation.

Then they will work on newly rewritten implementation of graph database, integrated with their products.

And that will be opensource too.

Not sure about licensing for comercial purposes.

 

http://www.zdnet.com/article/datastax-snaps-up-aurelius-and-its-titan-team-to-build-new-graph-database/

ozizka

Windup just sped up 10x

Posted by ozizka Mar 15, 2015

Good news! Windup does not run for 16 hours anymore... when refactoring, we omitted to sync one constant between addons... and it was a name of indexed graph DB property. As a result, windup ran 3 times slower on SSD disk and 10 times slower on HDD.

 

Now I fixed that and voila - you don't have to wait 20 hours for Windup to finish your 300 MB app

 

So that was [WINDUP-529] Performance issue: Query requires iterating over all vertices [(ClassificationModel:classification = ...)] .

Additionally, I've increased Titan database cache size to 9 % of available heap, which I experimentally ascertained to give best results with our current default heap size.

But my notion is we take wrong approach for some modeling cases and we should use edges rather than indexed properties. That could give some more performance gain, and not insignificant.

That's [WINDUP-532] Performance: Change storing of classification to edges instead of vertex property .

 

Enjoy the recovered performance and stay tuned for more

There are two main technologies from the old ages of internet that are (were) used for mass distribution of assynchronous posts: mailing lists and newsgroups (NNTP).

 

Mailing lists are, well, just a list of recipients that will get a copy of a mail sent to a special mail on a mailing list server.

Newsgroups are "topic IDs" (such like "org.jboss.windup.users") to which you can post, and it gets distributed over NNTP protocol to newsgroups servers all over the world. It's not copied to users' inboxes - it waits until they request it from the server.

 

I hate mailing lists (and I am not alone by far). They are just a poor substitution for newsgroups (NNTP protocol). For many reasons:

 

1) Subscribe/unsubscribe nonsense

Useless overhead. Why should anyone subscribe? Why not just authenticate / get authorized and access the content? Subscribing is a waste of time.

NNTP server, on the other hand, allows authentication and authorization, and "subscription" is done in your client - you simply favorite some groups and read them at will.

 

2) Downloading everything

In a mailing list, you simply get all of the trafic to your inbox. If someone sends 5 MB file, you get 5 MB file. And have to delete it. (IMAP helps a bit, but you still get a copy to your inbox).

NNTP server, on the other hand, passively waits until you _want_ to read the post, and sends it on your request.

Of course, pre-downloading in favorited groups may happen.

 

3) Waste of disk space

Again, if someone sends a 5 MB file, and the mailing list happens to be company-wide list with 4000 recipients, then we get around 40 GB wasted (5 MB attachment is encoded into roughly 10 MB in MIME format). That's really sick. And no - if you doubt the mail servers are so dumb and they only keep one copy of the file somewhere - they really don't, at least not the one we use(d) internally (confirmed by it's admin).

NNTP server, on the other hand, really does keep just one single copy of the post, and sends it to individuals on demand. The same goes for cross-posting.

 

4) Silly sorting to folders using filters

If you are in a company running a lot of mailing lists, you have to create a filter for each mailing list, if you don't want to end up with 1500 mails a day in your inbox. And not just you - everyone has. Typically, everyone needs to go to the impractical web UI of mail server ZYX and do the same stuff, figuring out which headers are the right to filter by. There's usually no sharing of these filters. And yet when finished, it still doesn't work well - sometimes you get stuff you have nothing to do with in your inbox, just because some header goes wrong. Or, very often, when a mail is cross-posted, it goes to one folder, but not to the other. And if it was copied to multiple folders, we're back to #3.

Newsgroups posts, on the other hand, are sorted by definition - each posts goes to one or more groups. And that's where you find them, period. No need for everyone to waste hours to create his/her own filters, almost identical to what others have.

 

5) Cross-posting

With mailing lists, cross-posting is real hell. Okay, the mailing lists server can prevent sending the mail to you twice. But not two servers - they have no idea about each other's internal IDs, so if someone sends a mail to a-list@foo.org and b-list@bar.org, and you happen to be subscribed to both - you end up with 2 mails. Now, to which of them are you going to reply? Both? If not both, some people will not get your reply. And worse: You need to watch both of the copied threads; otherwise, If someone replies to just one of them (i.e. not subscribed to the other), you will miss it.

NNTP protocol, on the other hand, is distributed, the IDs are shared, and one post, no matter to what group or what server, is the same, and you get just one single copy of that post. It _will_ appear in multiple groups, but it will be the same post, and if you reply to it, it will get distributed to all newsgroup servers, and nobody will miss any reply in any of the groups the conversation goes in (cross-posted groups).

 

6) Missing posts when not subscribed

When you're not subscribed, you don't get the posts. And after you subscribe - you still don't get the old posts. That's really one of biggest disadvantages - which other asynchronous channel does this? Forums let you see the old posts. Google groups do. Stack overflow does. Twitter does. Facebook does. Youtube does. Wherever you "subscribe" to some content, you get the old stuff. Except mailing lists.

Of course, there are archives. But how convenient is that? Leave your mail client and go search in the ugly Mailman UI from 90's? Thanks, but no thanks.

NNTP protocol, needless to say, allows you to download whatever you want, just what the server keeps in it's archive (some delete old posts).

 

7) Security reasons

You want to bring a mail to broader audience. But wait! Is the original list private list? Public list? And what about the 2nd list? How the hell should I know? Well, you need to go check the domain name. And even then, you still don't know - different list, different policy. You have to remember. If you forward a mail to another list, nothing prevents you to do so, or better said, nothing reminds you not to do so.

NNTP, on the other hand, supports mechanisms to refuse forwarding of a post based on the ID's of the posts contained in the post's headers. Sure, you could do it with mails, too; but it's up to mail client if it forwards the MIME headers or not. It's not mandatory for the protocol. For NNTP, it is, so a good client should send them, and your SMTP server (often the same instance as your local NNTP server) will refuse to send the post.

 

On par

There are areas where NNTP is just like mails. It's the same format (MIME), after all. And NNTP is quite similar to SMTP.

For example, most mainstream email clients support NNTP: ThunderBird, Evolution, Outlook (even the old Outlook Express did), ...

Basically you work with NNTP the same way as with mails; only, the addresses are group names such like "org.jboss.windup.users".

And you can even mix recipients - a post can go as mail to some users, and to a newsgroup.

 

Let's revive NNTP

With the above said, I really don't get why companies run mailing lists. What advantage mailing lists give? Waste of storage on server, waste of bandwidth, waste of user's disk space, waste of CPU on mail servers, waste of people's time. It's not even simpler or anything. It's just that people are used to it, or rather, don't know about newsgroups at all.

 

Let's spread the word about newsgroups. It's one of the best things that came from internet's stone ages, and should not be forgotten.

Share this article if you agree, and let me know in comments.

For a little while, Windup Quickstarts could not build as they got out of sync with latest API.

 

That's over and now they build. Well, if you skip tests One of rules fails and I had no time to figure out why yet. But at least you can take the quickstarts as a basis for your rule.

 

Go and try: windup/windup-quickstarts · GitHub

 

Here's a script to get everything you need to start playing with Windup rules in Java syntax: Windup lab scripts

Good news!

 

Windup rules, when written in Java syntax (instead of XML), is now much nicer, after we did [WINDUP-409] in PR #380 @Rules( phase = ..., .... by OndraZizka .

 

Before, metadata were provided by callback methods. Combined with our insisting on space-wasting formatting, each piece of metadata required 5 lines in class source.

 

public class SkipArchivesRules extends WindupRuleProvider
{
    private static final Logger log = Logging.get(SkipArchivesRules.class);

    @Override
    public void enhanceMetadata(Context context)
    {
        super.enhanceMetadata(context);
        context.put(RuleMetadata.CATEGORY, "Java");
    }

    @Override
    public List<Class<? extends WindupRuleProvider>> getExecuteAfter()
    {
        return asClassList(IdentifyArchivesRules.class, SkipArchivesLoadConfigRules.class);
    }

    @Override
    public List<Class<? extends WindupRuleProvider>> getExecuteBefore()
    {
        return asClassList(DecompileArchivesRuleProvider.class);
    }

    @Override
    public Class<? extends RulePhase> getPhase()
    {
        return Implicit.class;
    }
...

 

Now it's just one line with annotations:

 

@RuleMetadata(tags = "java", after = {IdentifyArchivesRules.class, SkipArchivesLoadConfigRules.class}, before = {DecompileArchivesRuleProvider.class}, phase = DependentPhase.class)
public class SkipArchivesRules extends AbstractRuleProvider
{
    private static final Logger log = Logging.get(SkipArchivesRules.class);
...

 

Isn't it great improvement?

Congratulations to Vojtěch who wrote self-monitoring subsystem for WildFly 8.

Succesfully defened on Tuesday and got A.

bender-futur/wildfly-self-monitor · GitHub

 

In the meantime, Heiko coded something similar, but hey - still a good job!

With 2 weeks delay in this announcement:

 

Windup 2.0 was released.

 

What's new in 2.0? Well, everything At least considering code - it's a complete rewrite.

The purpose is still to help facilitate the migration from various platforms to JBoss.

The rules (yes, it's rule-based) are now Java in the core, but some subset can be written in easy XML.

 

Go and try

 

If you want to try building it and try write some rule, here's the quick way to get it all (copy and paste one command at time - it uses background wget downloading) :

 

mkdir tmp; cd tmp/
mkdir windup-lab
cd windup-lab/

##  Java 8 download
firefox http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
wget -bc


##  NetBeans IDE
wget -bc http://download.netbeans.org/netbeans/8.0.2/final/bundles/netbeans-8.0.2-javase-linux.sh
chmod +x netbeans-8.0.2-javase-linux.sh
bash ./netbeans-8.0.2-javase-linux.sh

##  Maven 3.2.5+
mvn -v
wget ftp://mirror.reverse.net/pub/apache/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.zip
unzip apache-maven-3.2.5-bin.zip
MVN=$PWD/apache-maven-3.2.5/bin/mvn

##  Windup
firefox https://github.com/windup/windup
git co git@github.com:windup/windup.git
git clone git@github.com:windup/windup.git windup
cd windup/
mvn install -DskipTests
cd ..

##  Windup QuickStarts
firefox https://github.com/windup/windup-quickstarts
git clone git@github.com:windup/windup-quickstarts.git
cd windup-quickstarts/
$MVN install -DskipTests
cd ..


 

If you don't want to install but want to see it in action, watch our presentation from DevConf Brno:

https://www.youtube.com/watch?v=zLSYUzifJbY

 

Progressing with the Windup project, we're introducing Windup BOM (Build of Materials).

 

A BOM is basically a declaration of versions of artifact that you may

need when building Windup 2 rulesets.

By adding it to your ruleset's pom.xml, you avoid setting wrong versions

and maintaining them when upgrading.

 

Usage:

 

<properties>
  ...
  <version.windup>2.0.0.Beta2</version.windup>
</properties>

<dependencyManagement>
  <dependencies>

    <!-- Windup BOM -->
    <dependency>
      <groupId>org.jboss.windup</groupId>
      <artifactId>windup-rulesets-bom</artifactId>
      <version>${version.windup}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>

  </dependencies>

 

See the Windup QuickStarts for an example.

 

Enjoy

Ondra

 

TIP: Parallelized Maven builds: `mvn -T 1C clean install` speeds up esp.

the initial downloading of dependencies.

We're decompiling .class files using Procyon.

Sometimes, it get's stuck, looping infinitely.

 

So what to do with that?

 

private class DecompileExecutor extends Thread {
    public void cancelDecompilation() {
        this.stop(new NullPointerException());
    }
}

 

(16:33:46) ozizka-FN: jsightler: this.stop(new NullPointerException());
(16:33:52) jsightler: yeah
(16:33:57) ozizka-FN: that makes a thread exit from a loop?
(16:34:22) jsightler: ozizka: I think it will make it throw a spurious NPE wherever it is
(16:34:51) LincolnThree: doesn't look like its thrown. what does stop() do with it?
(16:34:54) LincolnThree: reading javadocs..
(16:35:12) jsightler: lincolnthree: Yeah, it should be thrown by the thread itself, I think
(16:35:22) LincolnThree: jsightler: afaik this will do nothing
(16:35:27) ozizka-FN: That must have come from the darkest corners of consulting :)
(16:35:39) jsightler: lincolnthree: Er, it definitely doesn't do nothing (it does stop the thread in my tests)
(16:35:58) jsightler: ozizka: TBH, it comes over from overusing this method in the java 1.1 days (predeprection :) )
(16:36:08) LincolnThree: jsightler: oh wow. this method is actually still implemented
(16:36:26) jsightler: lincolnthree: Yeah, just dangerous. :)

 

Enjoy

 

WINDUP-343: Fixes for discover properties, and some other issues by jsight · Pull Request #305 · windup/windup · GitHub

Some projects need a priming build - mostly to load the deps, otherwise the tests would fail when looking for it.

While it's typically an error of the tests or a wrong build order / dependencies, sometimes the project is kept such as it's not a biggest issue.

 

Here's how to determine if such priming build is needed.

Basically, you pick one artifact and check for it's existence in local repository.

mvn dependency:get -o -Dartifact=G:A:V

The trick is to feed it the right version.

Well, provided you run this in project's directory, you can use ${project.version}.

 

mvn dependency:get -o -Dartifact=org.jboss.windup.graph:windup-graph:\${project.version}

 

This will return 0 code when it exists, 1 code if it does not.

 

In our project, I needed to polish this in to a build script. So here's my result:

 

##  Determine this script's location ("Tools" home dir).
scriptPath="$(cd "${0%/*}" 2>/dev/null; echo "$PWD"/"${0##*/}")"
# For the case when called through a symlink
scriptPath=`readlink -f "$scriptPath"`
scriptDir=`dirname $scriptPath`

SETTINGS="-s $scriptDir/settings.xml"

localRepo=""
for val in "$@"
do
    echo $val
    if [[ "$val" == -Dmaven.repo.local=* ]] ; then localRepo=$val; fi
done

## Priming build to avoid WINDUP-322
if ! mvn $SETTINGS $localRepo dependency:get -o -Dartifact=org.jboss.windup.graph:windup-graph:\${project.version} > /dev/null
then
    echo -e "\n\nRunning a priming build...\n\n"
    mvn $SETTINGS $localRepo install -DskipTests
    echo -e "\n\nPriming build finished.\n\n"
fi


set -x
mvn $SETTINGS clean install $localRepo $@

 

Enjoy!

Filter Blog

By date:
By tag: