Hadoop: Example Program:
In this example by using HadoopStreaming you are able to interact with Hadoop. HadooopStreaming is an API that allows you to write code in any language and use a simple text-based record format for the input and output <key, value> pairs.
Hadoop example program.
In the above link there is a tutorial that shows how to write a program in Python and then it shows how to make it run through Hadoop.
When you reach the part in which you need to put your files in HDFS don't follow the instructions in the tutorial. Instead do the following:
$ bin/hadoop fs -mkdir urls
$ bin/hadoop fs -put url1 urls/
$ bin/hadoop fs -put url2 urls/
$ bin/hadoop jar contrib/streaming/hadoop-0.20.2-streaming.jar \
-mapper $HOME/proj/hadoop/multifetch.py \
-reducer $HOME/proj/hadoop/reducer.py \
-input urls/* \
-output titles
Wednesday, 4 January 2012
Hadoop: How to set up a single node:
Simple operations to use Hadoop MapReduce and the Hadoop Distributed File System (HDFS).
Prerequisites
GNU/Linux is supported as a development and production platform. Hadoop has been demonstrated on GNU/Linux clusters with 2000 nodes.
Win32 is supported as a development platform. Distributed operation has not been well tested on Win32, so it is not supported as a production platform.
$ sudo apt-get install ssh
$ sudo apt-get install rsync
Installation
Download a recent stable release from one of the Apache Download Mirrors.
Edit the file conf/hadoop-env.sh and point JAVA_HOME to your JAVA root installation. How to know where is your JAVA ($ echo $JAVA_HOME)
Pseudo-Distributed Operation
Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process. Follow this link.
In one of the steps you will need to ssh to the localhost:
$ ssh localhost
If you get the error
$ ssh: connect to host localhost port 22: Connection refused
Then try:
$ sudo apt-get install openssh-server
Simple operations to use Hadoop MapReduce and the Hadoop Distributed File System (HDFS).
Prerequisites
GNU/Linux is supported as a development and production platform. Hadoop has been demonstrated on GNU/Linux clusters with 2000 nodes.
Win32 is supported as a development platform. Distributed operation has not been well tested on Win32, so it is not supported as a production platform.
$ sudo apt-get install ssh
$ sudo apt-get install rsync
Installation
Download a recent stable release from one of the Apache Download Mirrors.
Edit the file conf/hadoop-env.sh and point JAVA_HOME to your JAVA root installation. How to know where is your JAVA ($ echo $JAVA_HOME)
Pseudo-Distributed Operation
Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process. Follow this link.
In one of the steps you will need to ssh to the localhost:
$ ssh localhost
If you get the error
$ ssh: connect to host localhost port 22: Connection refused
Then try:
$ sudo apt-get install openssh-server
Thursday, 22 December 2011
Hadoop: MapReduce Introduction:
Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.
A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.
Typically the compute nodes and the storage nodes are the same, that is, the MapReduce framework and the Hadoop Distributed File System (see HDFS Architecture Guide) are running on the same set of nodes. This configuration allows the framework to effectively schedule tasks on the nodes where data is already present, resulting in very high aggregate bandwidth across the cluster.
The MapReduce framework consists of a single master JobTracker and one slave TaskTracker per cluster-node. The master is responsible for scheduling the jobs' component tasks on the slaves, monitoring them and re-executing the failed tasks. The slaves execute the tasks as directed by the master.
Minimally, applications specify the input/output locations and supply map and reduce functions via implementations of appropriate interfaces and/or abstract-classes. These, and other job parameters, comprise the job configuration. The Hadoop job client then submits the job (jar/executable etc.) and configuration to the JobTracker which then assumes the responsibility of distributing the software/configuration to the slaves, scheduling tasks and monitoring them, providing status and diagnostic information to the job-client.
Although the Hadoop framework is implemented in JavaTM, MapReduce applications need not be written in Java.
Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.
A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.
Typically the compute nodes and the storage nodes are the same, that is, the MapReduce framework and the Hadoop Distributed File System (see HDFS Architecture Guide) are running on the same set of nodes. This configuration allows the framework to effectively schedule tasks on the nodes where data is already present, resulting in very high aggregate bandwidth across the cluster.
The MapReduce framework consists of a single master JobTracker and one slave TaskTracker per cluster-node. The master is responsible for scheduling the jobs' component tasks on the slaves, monitoring them and re-executing the failed tasks. The slaves execute the tasks as directed by the master.
Minimally, applications specify the input/output locations and supply map and reduce functions via implementations of appropriate interfaces and/or abstract-classes. These, and other job parameters, comprise the job configuration. The Hadoop job client then submits the job (jar/executable etc.) and configuration to the JobTracker which then assumes the responsibility of distributing the software/configuration to the slaves, scheduling tasks and monitoring them, providing status and diagnostic information to the job-client.
Although the Hadoop framework is implemented in JavaTM, MapReduce applications need not be written in Java.
More Research Topics:
MapReduce: A software framework introduced by Google to support distributed computing on large datasets.
Answer Set Programming (ASP): Declarative programming oriented towards difficult (primarily NP-hard) search problems. ASP includes all applications of answer sets to knowledge representation. Answer Set Solvers: smodels, assat, clasp and dlv.
Using these two concepts I am going to start reading and understanding about MapReduce and Answer Set Programming and one of its solvers dlv. The challenge is to produce an implementation using ASP and MapReduce together.
MapReduce: A software framework introduced by Google to support distributed computing on large datasets.
Answer Set Programming (ASP): Declarative programming oriented towards difficult (primarily NP-hard) search problems. ASP includes all applications of answer sets to knowledge representation. Answer Set Solvers: smodels, assat, clasp and dlv.
Using these two concepts I am going to start reading and understanding about MapReduce and Answer Set Programming and one of its solvers dlv. The challenge is to produce an implementation using ASP and MapReduce together.
Monday, 19 December 2011
Research Topics:
After talking to Dr. Kewen Wang he suggested me some research topics. He talked about a new model in the World Wide Web in which Artificial Intelligence and Knowledge Representation is applied. He said, nowadays the Internet is a web of linked documents, but not data. Efforts like DBpedia are trying to make a new Internet. The solution to the myriad of data formats could be Linking Open Data (LOD). He also mentioned about Wikipedia evolution, from WorldNet to Wikipedia and the question is now LOD? Based on what have been done in WorldNet and Wikipedia what could be an innovative approach.
Research Topics:
1. Resolve conflicts
Research Topics:
1. Resolve conflicts
2. Ranking candidate solutions
3. Linked open data
Thursday, 15 December 2011
Firefox extensions and On-line tools for the Semantic Web
Firefox Extensions
Semantic Radar: Displays a status bar icon to indicate presence of Semantic Web (RDF) data in the web page.
More extensions
On-Line Tools
Semantic Query End-Point
Firefox Extensions
Semantic Radar: Displays a status bar icon to indicate presence of Semantic Web (RDF) data in the web page.
More extensions
On-Line Tools
Semantic Query End-Point
Thursday, 8 December 2011
Installing a Semantic Web Environment
On Ubuntu 10.04
I'll be doing some tests on Jena Framework for the Semantic Web. First things first we need to install Jena. As long as Java is correctly configured to install Jena we just need:
1. Download Jena
2. Unzip it in any folder
3. Run the test as said in this instruction tutorial
We will also need a Java IDE, as recommended by this book: Semantic web programming. I have installed Eclipse in my Ubuntu 10.04. It is very easy:
$sudo apt-get install eclipse
On Windows 7
1. Setting up the environment variable:
http://introcs.cs.princeton.edu/java/15inout/windows-cmd.html
2. Java editor: Eclipse
3. Ontology editor: Protégé
4. Semantic web programming framework: Jena
5. Pellet reasoner
That's it!
This is a basic environment installation. I'll be doing more interesting things by following the above book. The book has a web site in which they have all the source code they use.
On Ubuntu 10.04
I'll be doing some tests on Jena Framework for the Semantic Web. First things first we need to install Jena. As long as Java is correctly configured to install Jena we just need:
1. Download Jena
2. Unzip it in any folder
3. Run the test as said in this instruction tutorial
We will also need a Java IDE, as recommended by this book: Semantic web programming. I have installed Eclipse in my Ubuntu 10.04. It is very easy:
$sudo apt-get install eclipse
On Windows 7
1. Setting up the environment variable:
http://introcs.cs.princeton.edu/java/15inout/windows-cmd.html
2. Java editor: Eclipse
3. Ontology editor: Protégé
4. Semantic web programming framework: Jena
5. Pellet reasoner
That's it!
This is a basic environment installation. I'll be doing more interesting things by following the above book. The book has a web site in which they have all the source code they use.
Subscribe to:
Posts (Atom)