Working with Hadoop under Eclipse

Last week i was setting up development environment for Hadoop under the Eclipse IDE by following the instructions provided by Apache  . But the installation wasn't as smooth as one would hope. Some hassles were there which are not documented (or maybe were invisible to me). Anyways i am writing about the errors faced by me in this blog.

Maven installation

• you need maven to run the mvn command.
• While the documentation at maven download page is given for Windows 2000/XP or UNIX based systems.But it does not map correctly on win 7.
• In Windows 7 one need to make the  M2_HOME and M2 environment variables (see the installation instructions here) as system variables and then they should be added to the path.
• Other instructions are same as documented in release notes by APACHE.

 Now after running the $ mvn install -DskipTests command the building of the projects                started. But one of the projects Apache Hadoop Common Project failed to build and reported MojoFailureException.

• You need to install protobuf (protocol buffers) and add it to your path variable to build the project successfully.Download it from here.

Now the next project to fail in build was hadoop-yarn-common.

• The build process failed because windows was not able to run "saveVersion.sh" file successfully.
• You need to add following patch to the Pom.xml file in hadoop yarn common to allow windows to run the  "saveVersion.sh" file successfully.

After these steps the build process will complete itself successfully.and you can run the next command to create eclipse project.

$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true

Now the projects will be ready to be imported in eclipse.

Hope these points would save other developers some time in installation.:-)