Как установить hadoop на windows 10

How to Install Hadoop in Windows 10

Installing Hadoop on Windows 10 can seem daunting, but it’s pretty straightforward if you follow the right steps. First, you need to ensure that Java is installed on your system. Then, you’ll have to download Hadoop and set up some environment variables. Finally, you’ll configure Hadoop and start it up. Let’s break it down into simple steps.

Installing Hadoop on Windows 10 involves several steps that will configure your machine to run Hadoop seamlessly. Here’s how to do it step by step:

Step 1: Install Java

Ensure Java is installed on your computer.

Java is necessary for Hadoop to run. You can download it from the official Oracle website. After downloading, follow the installation wizard to install Java on your system.

Step 2: Set Java Environment Variable

Set the JAVA_HOME environment variable.

Open your system properties and go to the Advanced tab. Click on Environment Variables, then click New under System Variables. Enter JAVA_HOME as the variable name and the path to your Java installation as the variable value.

Step 3: Download Hadoop

Download Hadoop from the official Apache website.

Navigate to the Apache Hadoop official website and download the binary distribution. Choose a stable release to avoid any bugs or issues.

Step 4: Install WinRAR

Install WinRAR to extract Hadoop files.

You will need a tool like WinRAR to unzip the Hadoop binaries you downloaded. Install WinRAR and use it to extract the Hadoop files to a directory of your choice.

Step 5: Configure Hadoop Environment Variables

Set HADOOP_HOME and add it to the PATH.

Go back to Environment Variables in System Properties. Create a new system variable with HADOOP_HOME as the name and the path to your Hadoop directory as the value. Then, edit the PATH variable and add %HADOOP_HOME%bin to it.

Step 6: Edit Hadoop Configuration Files

Configure core-site.xml and hdfs-site.xml.

Navigate to the Hadoop directory and find the conf folder. Edit the core-site.xml and hdfs-site.xml files to set the necessary parameters for your Hadoop setup.

Step 7: Format the Namenode

Open Command Prompt and format the Namenode.

Open Command Prompt as an administrator. Navigate to the bin directory inside your Hadoop folder and run the command «hdfs namenode -format» to format the Namenode.

Step 8: Start Hadoop Services

Start Hadoop services using the command line.

In the same Command Prompt window, start the Hadoop services by running the start-dfs.cmd and start-yarn.cmd scripts. This will start the Hadoop Distributed File System (HDFS) and Yet Another Resource Negotiator (YARN).

After completing these steps, Hadoop should be up and running on your Windows 10 machine, ready to handle big data tasks.

Tips for Installing Hadoop in Windows 10

• Make sure to install the correct version of Java that is compatible with your Hadoop version.
• Always download Hadoop from the official Apache website to avoid security risks.
• If you encounter issues with environment variables, restarting your computer can sometimes solve them.
• Running Command Prompt as an administrator can prevent permission errors.
• Keep your Hadoop configuration files backed up in case you need to revert changes.

FAQ on Installing Hadoop in Windows 10

What is Hadoop?

Hadoop is an open-source framework that allows for the distributed processing of large data sets across clusters of computers.

Do I need to install Java to run Hadoop?

Yes, Java is essential for running Hadoop as it is built on Java.

Can I install Hadoop without admin rights?

You need admin rights to set environment variables and format the Namenode.

How do I check if Hadoop is running?

You can check the running status by accessing the web interfaces of HDFS and YARN, usually available at localhost:50070 and localhost:8088 respectively.

Is it necessary to format the Namenode?

Yes, formatting the Namenode is crucial for initializing the Hadoop file system.

Summary

1. Install Java.
2. Set Java Environment Variable.
3. Download Hadoop.
4. Install WinRAR.
5. Configure Hadoop Environment Variables.
6. Edit Hadoop Configuration Files.
7. Format the Namenode.
8. Start Hadoop Services.

Conclusion

Installing Hadoop in Windows 10 might seem like climbing a mountain, but with these steps, it’s more like a walk in the park. From installing Java to configuring environment variables, each step is crucial to ensuring your Hadoop setup runs smoothly. Once you’ve completed these steps, you can dive into the world of big data with ease. For further reading, you might want to explore more about Hadoop’s ecosystem, including tools like Hive and Pig. So, what are you waiting for? Start your Hadoop journey today!

To Perform setting up and installing Hadoop in the pseudo-distributed mode in Windows 10 using the following steps given below as follows. Let’s discuss one by one.

Step 1: Download Binary Package :

Download the latest binary from the following site as follows.

http://hadoop.apache.org/releases.html

Картинка с сайта: www.geeksforgeeks.org

For reference, you can check the file save to the folder as follows.

C:\BigData

Step 2: Unzip the binary package

Open Git Bash, and change directory (cd) to the folder where you save the binary package and then unzip as follows.

$ cd C:\BigData
MINGW64: C:\BigData
$ tar -xvzf  hadoop-3.1.2.tar.gz

For my situation, the Hadoop twofold is extricated to C:\BigData\hadoop-3.1.2.  

Next, go to this GitHub Repo and download the receptacle organizer as a speed as demonstrated as follows. Concentrate the compress and duplicate all the documents present under the receptacle envelope to C:\BigData\hadoop-3.1.2\bin. Supplant the current records too. 

Картинка с сайта: www.geeksforgeeks.org

Step 3: Create folders for datanode and namenode :

• Goto C:/BigData/hadoop-3.1.2 and make an organizer ‘information’. Inside the ‘information’ envelope make two organizers ‘datanode’ and ‘namenode’. Your documents on HDFS will dwell under the datanode envelope.

Картинка с сайта: www.geeksforgeeks.org

• Set Hadoop Environment Variables
• Hadoop requires the following environment variables to be set.

HADOOP_HOME=” C:\BigData\hadoop-3.1.2”
HADOOP_BIN=”C:\BigData\hadoop-3.1.2\bin”
JAVA_HOME=<Root of your JDK installation>”

• To set these variables, navigate to My Computer or This PC.

Right-click -> Properties -> Advanced System settings -> Environment variables. 

• Click New to create a new environment variable.

Картинка с сайта: www.geeksforgeeks.org

• In the event that you don’t have JAVA 1.8 introduced, at that point you’ll have to download and introduce it first. In the event that the JAVA_HOME climate variable is now set, at that point check whether the way has any spaces in it (ex: C:\Program Files\Java\… ). Spaces in the JAVA_HOME way will lead you to issues. There is a stunt to get around it. Supplant ‘Program Files ‘to ‘Progra~1’in the variable worth. Guarantee that the variant of Java is 1.8 and JAVA_HOME is highlighting JDK 1.8.

Step 4: To make Short Name of Java Home path

Картинка с сайта: www.geeksforgeeks.org

Картинка с сайта: www.geeksforgeeks.org

• Set Hadoop Environment Variables
• Edit PATH Environment Variable

Картинка с сайта: www.geeksforgeeks.org

• Click on New and Add %JAVA_HOME%, %HADOOP_HOME%, %HADOOP_BIN%, %HADOOP_HOME%/sin to your PATH one by one.

Картинка с сайта: www.geeksforgeeks.org

• Now we have set the environment variables, we need to validate them. Open a new Windows Command prompt and run an echo command on each variable to confirm they are assigned the desired values.

echo %HADOOP_HOME%
echo %HADOOP_BIN%
echo %PATH%

• On the off chance that the factors are not instated yet, at that point it can likely be on the grounds that you are trying them in an old meeting. Ensure you have opened another order brief to test them.

Step 5: Configure Hadoop

Once environment variables are set up, we need to configure Hadoop by editing the following configuration files.

hadoop-env.cmd
core-site.xml
hdfs-site.xml
mapred-site.xml
yarn-site.xml
hadoop-env.cmd

First, let’s configure the Hadoop environment file. Open C:\BigData\hadoop-3.1.2\etc\hadoop\hadoop-env.cmd and add below content at the bottom

set HADOOP_PREFIX=%HADOOP_HOME%
set HADOOP_CONF_DIR=%HADOOP_PREFIX%\etc\hadoop
set YARN_CONF_DIR=%HADOOP_CONF_DIR%
set PATH=%PATH%;%HADOOP_PREFIX%\bin

Step 6: Edit hdfs-site.xml 

After editing core-site.xml, you need to set the replication factor and the location of namenode and datanodes. Open C:\BigData\hadoop-3.1.2\etc\hadoop\hdfs-site.xml and below content within <configuration> </configuration> tags.

<configuration>
 <property>
    <name>dfs.replication</name>
    <value>1</value>
 </property>
 <property>
    <name>dfs.namenode.name.dir</name>
    <value>C:\BigData\hadoop-3.2.1\data\namenode</value>
 </property>
 <property>
    <name>dfs.datanode.data.dir</name>
    <value>C:\BigData\hadoop-3.1.2\data\datanode</value>
 </property>
</configuration>

Step 7: Edit core-site.xml

Now, configure Hadoop Core’s settings. Open C:\BigData\hadoop-3.1.2\etc\hadoop\core-site.xml and below content within <configuration> </configuration> tags.

<configuration>
 <property>
   <name>fs.default.name</name>
   <value>hdfs://0.0.0.0:19000</value>
 </property>  
</configuration>

Step 8: YARN configurations

Edit file yarn-site.xml      

Make sure the following entries are existing as follows.

<configuration> <property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value> </property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

Step 9: Edit mapred-site.xml

At last, how about we arrange properties for the Map-Reduce system. Open C:\BigData\hadoop-3.1.2\etc\hadoop\mapred-site.xml and beneath content inside <configuration> </configuration> labels. In the event that you don’t see mapred-site.xml, at that point open mapred-site.xml.template record and rename it to mapred-site.xml

<configuration>
 <property>
    <name>mapreduce.job.user.name</name>    <value>%USERNAME%</value>
 </property>
 <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
 </property>
 <property>
    <name>yarn.apps.stagingDir</name>    <value>/user/%USERNAME%/staging</value>
 </property>
 <property>
    <name>mapreduce.jobtracker.address</name>
    <value>local</value>
 </property>
</configuration>

Check if C:\BigData\hadoop-3.1.2\etc\hadoop\slaves file is present, if it’s not then created one and add localhost in it and save it.

Step 10: Format Name Node :

To organize the Name Node, open another Windows Command Prompt and run the beneath order. It might give you a few admonitions, disregard them.

• hadoop namenode -format

Картинка с сайта: www.geeksforgeeks.org

                       Format Hadoop Name Node

Step 11: Launch Hadoop :

Open another Windows Command brief, make a point to run it as an Administrator to maintain a strategic distance from authorization mistakes. When opened, execute the beginning all.cmd order. Since we have added %HADOOP_HOME%\sbin to the PATH variable, you can run this order from any envelope. In the event that you haven’t done as such, at that point go to the %HADOOP_HOME%\sbin organizer and run the order.

Картинка с сайта: www.geeksforgeeks.org

You can check the given below screenshot for your reference 4 new windows will open and cmd terminals for 4 daemon processes like as follows. 

• namenode
• datanode
• node manager
• resource manager

Don’t close these windows, minimize them. Closing the windows will terminate the daemons. You can run them in the background if you don’t like to see these windows.

Картинка с сайта: www.geeksforgeeks.org

Step 12: Hadoop Web UI

In conclusion, how about we screen to perceive how are Hadoop daemons are getting along. Also you can utilize the Web UI for a wide range of authoritative and observing purposes. Open your program and begin.

Step 13: Resource Manager

Open localhost:8088 to open Resource Manager

Картинка с сайта: www.geeksforgeeks.org

Step 14: Node Manager

Open localhost:8042 to open Node Manager

Картинка с сайта: www.geeksforgeeks.org

Step 15: Name Node :

Open localhost:9870 to check out the health of Name Node

Картинка с сайта: www.geeksforgeeks.org

Step 16: Data Node :

Open localhost:9864 to check out Data Node

Картинка с сайта: www.geeksforgeeks.org

This article provides a step-by-step guide to install Hadoop 3.3.0 on Windows 10 via WSL (Windows Subsystem for Linux). These instructions are also applicable for installing Hadoop on Linux systems.

Hadoop 3.3.0 (released on July 14, 2020) is the most recent release in the Apache Hadoop 3.3 line as of the time of this article.

Prerequisites

Follow the steps mentioned on the page below to first enable Windows Subsystem for Linux and then install a Linux distribution of your choice.

Windows Subsystem for Linux Installation Guide for Windows 10

Initial Linux Configuration (optional)

Create a dedicated Hadoop user

We will create a dedicated Hadoop user for executing Hadoop commands. Although not required, it is a recommended step as it separates the Hadoop installation from other software applications and user accounts running on the same machine.

• Create a user group hadoop

sudo addgroup hadoop

• Create a user hduser

sudo adduser hduser

It will ask for password 2 times followed by user details, just press enter for each prompt and confirm Yes in the end.

• Add user hduser to user group hadoop

sudo  adduser hduser hadoop

• Add hduser to sudoers list so that hduser can do admin tasks

sudo visudo

Append after the line «#Allow member of group sudo to execute any command»

hduser ALL=(ALL) ALL 

Then save and close the file.

• Logout of the current session

logout

• Login again as hduser

su hduser
cd

Configure passphraseless ssh

Hadoop requires SSH access to manage its nodes.

This is a critical step and please make sure you follow the steps

Make sure you can SSH to localhost in Ubuntu:

ssh localhost

If you encounter the error ssh: connect to host localhost port 22: Connection refused, run the following commands:

sudo apt remove openssh-server
sudo apt install openssh-server
sudo service ssh start

If you cannot ssh to localhost without a passphrase, run the following command to initialize your private and public keys:

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

Now ssh localhost should work without any passphrase.

Hadoop Installation

Install Java JDK

Update the package index:

sudo apt update

Check whether Java is already installed:

java -version

If Java isn’t installed, install Java 8 from OpenJDK via the following command:

sudo apt-get install openjdk-8-jdk

Check the version installed:

java -version
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-8u275-b01-0ubuntu1~20.04-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)

Download Hadoop binary

Visit the Hadoop releases page to find a download URL for Hadoop 3.3.0:

Hadoop Releases

Run the following command to download the latest binary for Hadoop 3.3.0:

wget https://downloads.apache.org/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz

Wait until the download is completed.

Unzip Hadoop binary

Create a hadoop folder under the user home folder:

mkdir ~/hadoop

Unzip the downloaded binary package:

tar -xvzf hadoop-3.3.0.tar.gz -C ~/hadoop

After the package is unzipped, change the current directory to the Hadoop folder:

cd ~/hadoop/hadoop-3.3.0/

Setup environment variables

nano ~/.bashrc

Setup Hadoop and Java environment variables at the end of the .bashrc file as below and then save the bash file and close it.

#Set Hadoop-related environment variables
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=~/hadoop/hadoop-3.3.0
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME

For applying all these changes to the current Terminal, execute the source command.

source ~/.bashrc

Make sure that Java and Hadoop have been properly installed on your system and can be accessed through the Terminal by,

java -version
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-8u275-b01-0ubuntu1~20.04-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)

hadoop version
Hadoop 3.3.0
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r aa96f1871bfd858f9bac59cf2a81ec470da649af
Compiled by brahma on 2020-07-06T18:44Z
Compiled with protoc 3.7.1
From source with checksum 5dc29b802d6ccd77b262ef9d04d19c4
This command was run using /home/daftdey/hadoop/hadoop-3.3.0/share/hadoop/common/hadoop-common-3.3.0.jar

Configure the pseudo-distributed mode (Single Node mode)

This page below is the official documentation to configure a single node cluster:

Pseudo-Distributed Operation

All the Hadoop configuration files should be located in the ~/hadoop/hadoop-3.3.0/etc/hadoop directory.

cd ~/hadoop/hadoop-3.3.0/etc/hadoop

The configuration setup steps are as follows:

1. Edit file hadoop-env.sh:

   nano hadoop-env.sh
   

   Set Java environment variable as,

   # Java path
   export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
   

2. Edit file core-site.xml:
   core-site.xml informs the Hadoop daemon where NameNode runs in the cluster. It contains configuration settings of Hadoop core such as I/O settings that are common to HDFS & MapReduce.

   nano core-site.xml
   

   Add the following configuration:

   <configuration>
       <property>
           <name>fs.defaultFS</name>
           <value>hdfs://localhost:9000</value>
       </property>
   </configuration>
   

3. Edit file hdfs-site.xml:
   hdfs-site.xml contains configuration settings of HDFS daemons (i.e. NameNode, DataNode, Secondary NameNode). It also includes the replication factor and block size of HDFS.

   nano hdfs-site.xml
   

   Add the following configuration:

   <configuration>
       <property>
           <name>dfs.replication</name>
           <value>1</value>
       </property>
   </configuration>
   

4. Edit file mapred-site.xml:
   mapred-site.xml contains configuration settings of MapReduce application like the number of JVM that can run in parallel, the size of the mapper and the reducer process, CPU cores available for a process, etc.

   nano mapred-site.xml
   

   Add the following configuration:

   <configuration>
       <property>
           <name>mapreduce.framework.name</name>
           <value>yarn</value>
       </property>
       <property>
           <name>mapreduce.application.classpath</name>
           <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
       </property>
   </configuration>
   

5. Edit file yarn-site.xml:
   yarn-site.xml contains configuration settings of ResourceManager and NodeManager like application memory management size, the operation needed on program & algorithm, etc.

   nano yarn-site.xml
   

   Add the following configuration:

   <configuration>
       <property>
           <name>yarn.nodemanager.aux-services</name>
           <value>mapreduce_shuffle</value>
       </property>
       <property>
           <name>yarn.nodemanager.env-whitelist</name>
           <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
       </property>
   </configuration>
   

Format namenode

Go to the Hadoop home directory and format the Hadoop namenode:

cd ~/hadoop/hadoop-3.3.0
bin/hdfs namenode -format

This formats the HDFS via the NameNode. Formatting the file system means initializing the directory specified by the dfs.name.dir variable.

This command should be executed only for the first time. Never format an up and running Hadoop filesystem. You will lose all your data stored in the HDFS.

Run HDFS daemons

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files stored in the HDFS and tracks all the files stored across the cluster.

On startup, a DataNode connects to the Namenode and it responds to the requests from the Namenode for different operations.

Start NameNode daemon and DataNode daemon

sbin/start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [tars]

Check status via jps command

jps
8544 SecondaryNameNode
8304 DataNode
8149 NameNode
8702 Jps

When the HDFS services are successfully initiated, you should be able to see these four processes running.

View name node portal

Now open the browser and go to the following URL to view the NameNode interface. Scroll down and see the number of live nodes.

http://localhost:9870/dfshealth.html#tab-overview

The web UI looks like this:

Картинка с сайта: dev.to

Data nodes information can be also viewed through the Datanodes menu link:

Картинка с сайта: dev.to

Run YARN daemon

ResourceManager is the master that arbitrates all the available cluster resources and thus helps in managing the distributed applications running on the YARN system. Its work is to manage each NodeManagers and each application’s ApplicationMaster.

The NodeManager in each machine framework is the agent that is responsible for managing containers, monitoring their resource usage, and reporting the same to the ResourceManager.

Start ResourceManager daemon and NodeManager daemon

sbin/start-yarn.sh
Starting resourcemanager
Starting nodemanagers

Check status using the jps command

jps
11831 ResourceManager
11352 DataNode
12347 Jps
11196 NameNode
11981 NodeManager
11597 SecondaryNameNode

Once the YARN services are started, two more processes for NodeManager and ResourceManage can be seen running.

View YARN web portal

Now open the browser and go to the following URL to view the YARN resource manager web UI.

http://localhost:8088/cluster

The web UI looks like this:

Картинка с сайта: dev.to

Execution of all the Hadoop applications can be monitored through this web portal.

Shutdown Hadoop services

After completion of the execution of all Hadoop applications, the HDFS and YARN services can be stopped as follows:

sbin/stop-yarn.sh
sbin/stop-dfs.sh

Verify status via jps command, only the jps service should be running:

jps
13257 Jps

Summary

Congratulations, you have successfully installed a single-node Hadoop cluster in one go on your Linux subsystem of Windows 10. It’s relatively easier compared to installing on Windows as we don’t need to download or build native Hadoop HDFS libraries.

Have fun exploring with Hadoop.

Try out an example application: Hadoop MapReduce computational model to perform analyses on DBLP publication data

If you encounter any issues while following the steps, please post a comment and I will try my best to help.

Further References

Install Hadoop 3.3.0 on Windows 10 using WSL

Install Hadoop: Setting up a Single Node Hadoop Cluster