25 Mar 2014

How to setup the storm-starter repo

Tags

Just today, I have faced an issue which took up (wasted) several hours of my time in a rather painful way, so I thought I should share my findings so that others do not repeat the same mistake.

I was learning Storm and one of the resources I found was the storm-starter repository. While I was disappointed that there were no instructions for setup using Leiningen (other than a one-liner pointing to the Leiningen repo, more on this towards the end of this article), I proceeded to set up Maven, and ran the WordCountTopology using:

mvn compile exec:java -Dstorm.topology=storm.starter.WordCountTopology

however, this was met with rather cryptic error messages from Maven (which I shall omit here due to my frustration earlier today) and it took me literally hours of googling, trying the command and variations of the command at both the root directory of the repo and the examples/storm-starter directory, appending those directories to CLASSPATH, all of which did not get me any closer to solving the problem.

It turns out that the solution is simple. Instead of using the above storm-starter repository, use this one instead, and follow the same instructions. One thing to take note of is that in this repository, the pom.xml file is named m2-pom.xml . So to execute the WordCountTopology code, you will run the command:

mvn -f m2-pom.xml compile exec:java -Dstorm.topology=storm.starter.WordCountTopology

A kind reader, Jorge Plaza, has pointed out that the following series of commands will work too (Thank you Jorge!). If you can’t run the WordCountTopology with the above command, these should do the trick:

mvn -f m2-pom.xml clean
mvn -f m2-pom.xml package
storm jar target/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.WordCountTopology

A note of caution: this make use of the storm client and if you’ve set up the Storm client to deploy to a production server, that is what the storm jar command will do. However, if you’ve not set up the Storm client to deploy to a production server, the storm jar command will run the WordCountTopology on your local machine.

It makes me wonder why the repository was moved in the first place. Worse, the documentation is not up to date. I don’t mean to criticize anyone, but I personally believe that documentation to do with setting things up should be accurate, especially for a repository like this which is meant for someone new to Storm. Not exactly the best welcome party.

Final notes

Based on this pull request, it seems that Leiningen cannot be used to set up the storm-starter repository.

I have a few more blog posts queueing up (for quite some time), but I decided to work on this one because it’ll be a very quick write up, and the issue described here caused me a lot of frustration.

Disclaimer: Opinions expressed on this blog are solely my own and do not express the views or opinions of my employer(s), past or present.