I'm working from the Hadoop book, and I'm noticing some gaps. When I ran hadoop using streaming like this, my job hangs:The tracking URL is handy, from there I could see that hadoop streaming wasn't able to run my groovy mapper:$ hadoop jar $HADOOP_INSTALL/contrib/streaming/hadoop-0.20.1-streaming.jar \
-input movies.input.txt -output output -mapper movies.map.groovy \
-reducer movies.reduce.groovy
packageJobJar: [/tmp/hadoop-user/hadoop-unjar2417163781720808364/] [] /var/folders/Ht/HtruzsCeGAukVrRT16Q4+k+++TI/-Tmp-/streamjob7394763723023597419.jar tmpDir=null
09/10/16 21:13:46 INFO mapred.FileInputFormat: Total input paths to process : 1
09/10/16 21:13:47 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-user/mapred/local]
09/10/16 21:13:47 INFO streaming.StreamJob: Running job: job_200910162029_0008
09/10/16 21:13:47 INFO streaming.StreamJob: To kill this job, run:
09/10/16 21:13:47 INFO streaming.StreamJob: /Users/user/Applications/hadoop-0.20.1/bin/../bin/hadoop job -Dmapred.job.tracker=localhost:8021 -kill job_200910162029_0008
09/10/16 21:13:47 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_200910162029_0008
09/10/16 21:13:48 INFO streaming.StreamJob: map 0% reduce 0%
^CI was missing the -file parameter. This now gets me to my next error...Caused by: java.io.IOException: Cannot run program "movies.map.groovy": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:166)
... 19 more$ hadoop jar $HADOOP_INSTALL/contrib/streaming/hadoop-0.20.1-streaming.jar \
-input movies.input.txt -output output -mapper movies.map.groovy \
-reducer movies.reduce.groovy -file movies.map.groovy -file movies.reduce.groovy
Clojure 1.1 is out ... plus videos about new features
-
So, Clojure 1.1 is now available, with lots of cool new features, including
transients, pre & post conditions, futures, promises and a boat load of
other...
7 hours ago

0 comments:
Post a Comment