04 Apr 2014

How to build the Storm client from source

Introduction

This article touches Storm 0.9.0.1 and should be relevant for 0.9.x releases. If you’re facing problems setting up the Storm client (as I did when I first got started), you might find the information here helpful.

NOTE: This covers building the Storm client from its github repository). If you wish to learn how to setup Storm components (Zookeeper, Nimbus, UI, Supervisor, etc) or deploy a Storm topology, then this article is not for you.

About the Storm Client

The Storm client is a command line tool mainly used to control a remote Storm topology. Its various commands can be found here.

A shortcut solution

The Storm website’s download page contains binary releases. These should work but some of them have failed for me. If you encounter problems running the Storm client in those binary releases, do follow the instructions in this article to build the Storm client from source.

System requirements

Maven (ideally Maven 3 or greater)
Python 2.7.x
Git

Ensure that you have the above software installed before you proceed.

Setup

Clone the incubator-storm github repository:

git clone https://github.com/apache/incubator-storm.git

You should see an incubator-storm directory. Go into it:

cd incubator-storm

Important Step:

We’ll checkout the 0.9.0.1 tag:

git checkout 0.9.0.1

If you do an ls, you should see a bin directory, and a executable file named storm inside it. At this point, you might think that our job is done.

The problem

That is, until you try running the storm binary in the bin directory:

bin/storm

You will be greeted with the following:

******************************************

The storm client can only be run from within a release. You appear to be trying to run the client from a checkout of Storm’s source code.

You can download a Storm release at http://storm-project.net/downloads.html

******************************************

When I just got started with Storm, this was the exact same issue I faced when I downloaded the binary release for 0.9.x and expected the bin/storm to just run (after all it’s a binary release, right? However, this issue should have been fixed). Since we’re now building from source, you might not be so surprised by the above message. Sorry for the anticlimax if that’s the case.

Building the Storm client

The more meticulous people among us may have observed that there is a build_release.sh file in the bin folder. It turns out that this is indeed the solution to our woes.

Add executable permissions to build_release.sh and run it:

chmod a+x bin/build_release.sh
bin/build_release.sh

This might take some time to run. There will be some error and warning messages but they should not affect the overall build process.

Once done, you should see 2 new files in the repository: storm-0.9.0.1.tar.gz and storm-0.9.0.1.zip . Let’s copy the storm-0.9.0.1.tar.gz to our $HOME directory and extract it:

cp storm-0.9.0.1.tar.gz ~
cd ~
tar xzf storm-0.9.0.1.tar.gz

You should see a storm-0.9.0.1 directory. Inside it, there is a bin/storm binary. Running it displays the list of commands it supports. You might want to add $HOME/storm-0.9.0.1/bin to your PATH, like so:

echo 'PATH=$PATH:$HOME/storm-0.9.0.1/bin' >> ~/.bashrc

Mission accomplished.

Disclaimer: Opinions expressed on this blog are solely my own and do not express the views or opinions of my employer(s), past or present.