Sunday, March 17, 2013

How to solve Datanode error in Hadoop

Sometimes the datanode daemon process does not start successfully when you run the startall.sh script in hadoop on both the single and multinode clusters.

The Datanode log gives java.io.IOException .Further it explains that namespaceid of namenode and datanode are not compatible.

This problem can be averted by changing the namesapce id of the datanode.

To change the namespace id of datanode follow these steps
  1. Check the value of hadoop.tmp.dir in conf/core-site.xml file. 
  2. As my value is /app/hadoop/tmp i will open that directory.
  3. Now /app/hadoop/tmp/dfs/name/current/VERSION represents the version file of name node which contains its namenode.
  4. /app/hadoop/tmp/dfs/data/current/VERSION contains the namespace id of datanode.
  5. Both of these namespace ids need to match.
  6. Delete the file /app/hadoop/tmp/dfs/data/current/VERSION and restart the namenode.
  7. Now Datanode will also start normally.
The  Version file looks something like this


    

Wednesday, November 7, 2012

Configure eclipse map reduce plugin for hadoop

Hadoop Eclipse Plug-in can be downloaded from this link .

The Hadoop Eclipse Plug-in provides tools to ease the experience of Map/Reduce on Hadoop. Among other things, the plug-in provides support to:
  1. create Mapper, Reducer, Driver classes;
  2. browse and interact with distributed file systems;
  3. submit jobs and monitor their execution.
Now to configure the plugin you need to follow these steps

  1. Move  the jar file you have downloaded in the plugins folder of your eclipse installation.
  2. Restart the Eclipse and switch to  map-reduce perspective from Window-> open perspective ->other 
  3. Now set your DFS location by selecting create new location in Map/Reduce locations area.
  4. Enter the name of your location and specify the ports.
  5. Add the port numbers of JobTracker and DFS as specified in conf/mapred-site.xml and  conf/core-site.xml files.


  6.  Now create new map-reduce project and configure your Hadoop installation directory. 
  7. Now you can start off by creating mapper and reducer classes.
  8. To run your project select run on Hadoop in the run configuration.

Friday, August 24, 2012

Working with Hadoop under Eclipse

Last week i was setting up development environment for Hadoop under the Eclipse IDE by following the instructions provided by Apache  . But the installation wasn't as smooth as one would hope. Some hassles were there which are not documented (or maybe were invisible to me). Anyways i am writing about the errors faced by me in this blog.

Maven installation
  • you need maven to run the mvn command.
  • While the documentation at maven download page is given for Windows 2000/XP or UNIX based systems.But it does not map correctly on win 7.
  • In Windows 7 one need to make the  M2_HOME and M2 environment variables (see the installation instructions here) as system variables and then they should be added to the path.
  • Other instructions are same as documented in release notes by APACHE.
 Now after running the $ mvn install -DskipTests command the building of the projects                started. But one of the projects Apache Hadoop Common Project failed to build and reported MojoFailureException.
  • You need to install protobuf (protocol buffers) and add it to your path variable to build the project successfully.Download it from here.
Now the next project to fail in build was hadoop-yarn-common.
  • The build process failed because windows was not able to run "saveVersion.sh" file successfully.
  • You need to add following patch to the Pom.xml file in hadoop yarn common to allow windows to run the  "saveVersion.sh" file successfully.
After these steps the build process will complete itself successfully.and you can run the next command to create eclipse project.
$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true

Now the projects will be ready to be imported in eclipse.

Hope these points would save other developers some time in installation.:-)

Friday, August 3, 2012

FBcoverprofilecombo

This post explains the Html5 and JavaScript app Create Facebook cover and profile pic combo.The major elements used in this app are Canvas and Drag and drop API of Html5.

Drag and Drop API is used to upload the images from the user.It makes the look and feel of the app trendy and reliable.Following event handler in JavaScript does that dirty work:
 dropzone.on('drop', function(e) {  
           //prevent browser from open the file when drop off  
           e.stopPropagation();  
           e.preventDefault();  
           dropzone.removeClass('hover');  
           //retrieve uploaded files data  
           var files = e.originalEvent.dataTransfer.files;  
           processFiles(files);  
           return false;  
      });  

Now the file is processed to create a cover image with the use of Canvas element of HTML5 in following function.The image settings are set depending on the field chosen (page or profile).Canvas element does the task of resizing the image.

 var getCanvasImage = function(image) {  
           //define canvas  
           var croping = $('input[name=croping]:checked').val();  
         //check for page or profile  
           if(croping == 'crop') {  
                imgWidth = 958;             imgHeight = 450;  
           }else{  
           imgWidth = 958;            imgHeight = 419;  
           }  
           var canvas = document.createElement('canvas');  
           canvas.width = imgWidth;  
           canvas.height = imgHeight;  
           var ctx = canvas.getContext('2d');  
           //draw canvas image       
           ctx.drawImage(image,0,0,image.width,image.height,0,0,imgWidth,imgHeight);   
           //convert canvas to jpeg url  
           return canvas.toDataURL("image/jpeg");  
      }  

After the image is created The above function returns the source of the cover image.Now this image is processed to create profile photo in the following event handler and later appended to the body of document.

 createprofile.on('click',function(){  
           var cw,ch,a,sx,sy;  
      var croping = $('input[name=croping]:checked').val();  
      if(croping == 'crop') {  
                cw=180;ch=180;sx=28;sy=240;  
           }else{  
           cw=190;ch=190;sx=28;sy=220;  
           }  
           var covershhh=document.getElementById("cov");  
           var canvs = document.createElement('Canvas');  
           canvs.width = cw;  
           canvs.height = ch;  
           var ctx = canvs.getContext('2d');  
           //draw canvas image                           
           ctx.drawImage(covershhh, 28, sy, cw, ch, 0, 0, cw, ch);
           var oImg=document.createElement('img');       
           oImg.id="gdfg";                      
           var a=canvs.toDataURL("image/jpeg");  
           oImg.src=a;
           document.getElementById("result2").appendChild(oImg);           
      });  

Note:All the measurements are taken are in pixels.In next post i will share some of the creative timelines created by this app.Till then good bye!!

Don't forget to give your feedback and encouragement.

Friday, July 27, 2012

Create facebook cover and profile pic combo

This post explains the Html5 and JavaScript app "Create Facebook cover and profile pic combo" developed by me this summer.Since the conception of timeline feature in Facebook most people  were initially very intrigued by it but later they were despised by it.Then i saw some creative examples of combining cover and profile pictures which aroused my interest in the timeline again.


So in order to join the cool gang i tried my hand at Photoshop with some disastrous results.Finally i decided to automate the whole process and hence after some smart coding and rigorous testing i created this app.

The best way to make most of this app is to keep your face in the lower left corner of the picture you want to upload on the web-app.This way your face will be visible in your profile pic.

You just need to upload a single image and then you just need to use "press to view your profile pic" button.Download both the images by using right-click of mouse and upload them on Facebook.Align the thumbnail of profile pic as it suits best to you.

My app utilizes the Html 5 Canvas and File API combined with some smart JavaScript. The inclusion of these elements makes sure that old browsers(read Internet Explorer) will not be a part of it.So use this app in the browsers which support HTML5 and JavaScript.


Any criticism or suggestions for the development of the app is always welcome.Mail me at aribachhawat@gmail.com for that.

I will explain the source code of the app in my next post.Till then HASTA la VISTA baby.