Hadoop是分布式系统基础平台，主要存储数据挖掘计算任务的中间结果数据。
1、系统环境准备
| 注意 |
|---|
注意：如果服务部署集群，集群内所有服务器均需设置系统环境。 |
1.1防火墙配置
为了便于安装，建议在安装前关闭防火墙。使用过程中，为了系统安全可以选择启用防火墙，但必须启用服务相关端口。
1.关闭防火墙
临时关闭防火墙
| 代码块 | ||
|---|---|---|
| ||
systemctl stop firewalld |
永久关闭防火墙
| 代码块 | ||
|---|---|---|
| ||
systemctl disable firewalld |
查看防火墙状态
| 代码块 | ||
|---|---|---|
| ||
systemctl status firewalld |
2.开启防火墙
相关服务及端口对照表：
| 服务名 | 需要开放端口 |
|---|---|
Hadoop | 50090，50070，9000，50010，50075，50020 |
如果确实需要打开防火墙安装，需要给防火墙放开以下需要使用到的端口
开启端口：50090，50070，9000，50010，50075，50020
| 代码块 | ||
|---|---|---|
| ||
firewall-cmd --zone=public --add-port=50090/tcp --permanent firewall-cmd --zone=public --add-port=50070/tcp --permanent firewall-cmd --zone=public --add-port=9000/tcp --permanent firewall-cmd --zone=public --add-port=50010/tcp --permanent firewall-cmd --zone=public --add-port=50075/tcp --permanent firewall-cmd --zone=public --add-port=50020/tcp --permanent |
配置完以后重新加载firewalld，使配置生效
| 代码块 | ||
|---|---|---|
| ||
firewall-cmd --reload |
查看防火墙的配置信息
| 代码块 | ||
|---|---|---|
| ||
firewall-cmd --list-all |
3.关闭selinux
临时关闭selinux，立即生效，不需要重启服务器。
| 代码块 | ||
|---|---|---|
| ||
setenforce 0 |
永久关闭selinux，修改完配置后需要重启服务器才能生效
| 代码块 | ||
|---|---|---|
| ||
sed -i 's/=enforcing/=disabled/g' /etc/selinux/config |
1.2 安装Java环境
安装包解压到/opt目录
| 代码块 | ||
|---|---|---|
| ||
tar -zxvf jdk8.0.202-linux_x64.tar.gz -C /opt |
配置java环境变量
①在/etc/profile 文件末尾添加，并保存
| 代码块 | ||
|---|---|---|
| ||
export JAVA_HOME=/opt/jdk8.0.202-linux_x64 export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/jre/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH |
②使配置生效
| 代码块 | ||
|---|---|---|
| ||
source /etc/profile |
③查看java版本信息
| 代码块 | ||
|---|---|---|
| ||
java -version |
1.3取消打开文件限制
修改/etc/security/limits.conf文件在文件的末尾加入以下内容：
| 代码块 | ||||
|---|---|---|---|---|
| ||||
vi /etc/security/limits.conf |
在文件的末尾加入以下内容：
| 代码块 | ||||
|---|---|---|---|---|
| ||||
* soft nofile 65536 * hard nofile 65536 * soft nproc 131072 * hard nproc 131072 |
2、Hadoop单机部署
1.修改主机名-添加映射关系
根据部署实际环境，各个服务器主机名不同即可不修改主机名，如需修改，可参考下面的修改方式：
| 代码块 | ||
|---|---|---|
| ||
hostnamectl set-hostname 主机名 |
PS：主机名不能使用下划线
配置主机名和IP的映射关系
| 代码块 | ||
|---|---|---|
| ||
vi /etc/hosts |
内容设置，例如:
| 代码块 | ||
|---|---|---|
| ||
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.10.17.186 hadoop |
2.设置系统免密登陆
生成密钥
| 代码块 | ||
|---|---|---|
| ||
ssh-keygen |
输入ssh-keygen后，连续按三次回车，不用输入其它信息
复制公钥
| 代码块 | ||
|---|---|---|
| ||
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 0600 ~/.ssh/authorized_keys |
测试是否设置成功，例如:
| 代码块 | ||
|---|---|---|
| ||
ssh root@hadoop |
如果不用输入密码，表示成功。
3.安装Hadoop
①创建hadoop相关目录
创建临时目录
| 代码块 | ||
|---|---|---|
| ||
mkdir -p /opt/hdfs/tmp |
创建namenode数据目录
| 代码块 | ||
|---|---|---|
| ||
mkdir -p /opt/hdfs/name |
创建datanode目录
注意这个目录尽量创建在空间比较大的目录，如有多个磁盘，可创建多个目录
| 代码块 | ||
|---|---|---|
| ||
mkdir -p /opt/hdfs/data |
②解压hadoop安装包到指定目录
| 代码块 | ||
|---|---|---|
| ||
tar -zxvf hadoop-2.7.3.tar.gz -C /opt |
③修改配置文件
1.修改hadoop-env.sh
| 代码块 | ||
|---|---|---|
| ||
cd /opt/hadoop-2.7.3/etc/hadoop vi hadoop-env.sh |
找到export JAVA_HOME= ，修改Java安装路径如下所示
| 代码块 | ||
|---|---|---|
| ||
export JAVA_HOME=/opt/jdk8.0.202-linux_x64 |
找到export HADOOP_OPTS，在下面添加一行
| 代码块 | ||
|---|---|---|
| ||
export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC -Xmx4g" |
2、修改core-site.xml配置文件
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/etc/hadoop vi core-site.xml |
内容如下：
| 代码块 | ||||
|---|---|---|---|---|
| ||||
<configuration>
<property>
<name>fs.defaultFS</name>
<!--根据实际情况替换成本机的IP或主机名 -->
<value>hdfs://hadoop:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/opt/hdfs/tmp</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>100800</value>
</property>
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
</configuration> |
3、修改hdfs-site.xml配置文件
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/etc/hadoop vi hdfs-site.xml |
内容如下：
| 代码块 | ||||
|---|---|---|---|---|
| ||||
<configuration>
<property>
<name>dfs.name.dir</name>
<value>file:/opt/hdfs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file:/opt/hdfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.max.transfer.threads</name>
<value>16384</value>
</property>
</configuration> |
| 注意 |
|---|
注意：dfs.data.dir尽量配置在空间比较大的目录，可以配置多个目录，中间用逗号分隔 |
4、修改hadoop-policy.xml
| 代码块 | ||
|---|---|---|
| ||
cd /opt/hadoop-2.7.3/etc/hadoop vi hadoop-policy.xml |
内容如下:
| 代码块 | ||||
|---|---|---|---|---|
| ||||
<configuration>
<property>
<name>security.client.protocol.acl</name>
<value>*</value>
<description>ACL for ClientProtocol, which is used by user code
via the DistributedFileSystem.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<!-- 这里把实验引擎ip, python执行节点ip，spark部署机器ip，hadoop部署机器ip都加上-->
<!-- 如果实验引擎，python执行节点，spark，hadoop这些组件是集群部署，那么所有的IP地址都需要添加进来 -->
<property>
<name>security.client.protocol.hosts</name>
<value>192.168.137.139,192.168.137.140,192.168.137.141</value>
</property>
<property>
<name>security.client.datanode.protocol.acl</name>
<value>*</value>
<description>ACL for ClientDatanodeProtocol, the client-to-datanode protocol
for block recovery.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<!-- 这里把实验引擎ip,python执行节点ip，spark部署机器ip，hadoop部署机器ip都加上-->
<!-- 如果实验引擎，python执行节点，spark，hadoop这些组件是集群部署，那么所有的IP地址都需要添加进来 -->
<property>
<name>security.client.datanode.protocol.hosts</name>
<value>192.168.137.139,192.168.137.140,192.168.137.141</value>
</property>
<property>
<name>security.datanode.protocol.acl</name>
<value>*</value>
<description>ACL for DatanodeProtocol, which is used by datanodes to
communicate with the namenode.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.inter.datanode.protocol.acl</name>
<value>*</value>
<description>ACL for InterDatanodeProtocol, the inter-datanode protocol
for updating generation timestamp.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.namenode.protocol.acl</name>
<value>*</value>
<description>ACL for NamenodeProtocol, the protocol used by the secondary
namenode to communicate with the namenode.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.admin.operations.protocol.acl</name>
<value>*</value>
<description>ACL for AdminOperationsProtocol. Used for admin commands.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.refresh.user.mappings.protocol.acl</name>
<value>*</value>
<description>ACL for RefreshUserMappingsProtocol. Used to refresh
users mappings. The ACL is a comma-separated list of user and
group names. The user and group list is separated by a blank. For
e.g. "alice,bob users,wheel". A special value of "*" means all
users are allowed.</description>
</property>
<property>
<name>security.refresh.policy.protocol.acl</name>
<value>*</value>
<description>ACL for RefreshAuthorizationPolicyProtocol, used by the
dfsadmin and mradmin commands to refresh the security policy in-effect.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.ha.service.protocol.acl</name>
<value>*</value>
<description>ACL for HAService protocol used by HAAdmin to manage the
active and stand-by states of namenode.</description>
</property>
<property>
<name>security.zkfc.protocol.acl</name>
<value>*</value>
<description>ACL for access to the ZK Failover Controller
</description>
</property>
<property>
<name>security.qjournal.service.protocol.acl</name>
<value>*</value>
<description>ACL for QJournalProtocol, used by the NN to communicate with
JNs when using the QuorumJournalManager for edit logs.</description>
</property>
<property>
<name>security.mrhs.client.protocol.acl</name>
<value>*</value>
<description>ACL for HSClientProtocol, used by job clients to
communciate with the MR History Server job status etc.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<!-- YARN Protocols -->
<property>
<name>security.resourcetracker.protocol.acl</name>
<value>*</value>
<description>ACL for ResourceTrackerProtocol, used by the
ResourceManager and NodeManager to communicate with each other.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.resourcemanager-administration.protocol.acl</name>
<value>*</value>
<description>ACL for ResourceManagerAdministrationProtocol, for admin commands.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.applicationclient.protocol.acl</name>
<value>*</value>
<description>ACL for ApplicationClientProtocol, used by the ResourceManager
and applications submission clients to communicate with each other.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.applicationmaster.protocol.acl</name>
<value>*</value>
<description>ACL for ApplicationMasterProtocol, used by the ResourceManager
and ApplicationMasters to communicate with each other.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.containermanagement.protocol.acl</name>
<value>*</value>
<description>ACL for ContainerManagementProtocol protocol, used by the NodeManager
and ApplicationMasters to communicate with each other.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.resourcelocalizer.protocol.acl</name>
<value>*</value>
<description>ACL for ResourceLocalizer protocol, used by the NodeManager
and ResourceLocalizer to communicate with each other.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.job.task.protocol.acl</name>
<value>*</value>
<description>ACL for TaskUmbilicalProtocol, used by the map and reduce
tasks to communicate with the parent tasktracker.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.job.client.protocol.acl</name>
<value>*</value>
<description>ACL for MRClientProtocol, used by job clients to
communciate with the MR ApplicationMaster to query job status etc.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.applicationhistory.protocol.acl</name>
<value>*</value>
<description>ACL for ApplicationHistoryProtocol, used by the timeline
server and the generic history service client to communicate with each other.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
</configuration> |
| 注意 | ||
|---|---|---|
| ||
hadoop-policy.xml配置文件中，security.client.protocol.hosts，security.client.datanode.protocol.hosts 这两个配置项的值，要改成实际部署环境的IP地址 |
④配置hadoop环境变量
添加环境变量。
| 代码块 | ||||
|---|---|---|---|---|
| ||||
vi /etc/profile |
在最底下添加下面内容：
| 代码块 | ||||
|---|---|---|---|---|
| ||||
export HADOOP_HOME=/opt/hadoop-2.7.3 export PATH=$PATH:$HADOOP_HOME/bin |
让配置生效
| 代码块 | ||||
|---|---|---|---|---|
| ||||
source /etc/profile |
4.启动Hadoop
①格式化hadoop
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/ ./bin/hdfs namenode -format |
| 注意 | ||
|---|---|---|
| ||
仅第一次启动时需要执行格式化Hadoop操作，后续启动无需进行此操作 |
②启动hadoop
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/ ./sbin/start-dfs.sh |
③创建中间数据存储目录(数据挖掘实验引擎集群需要使用)
| 代码块 | ||||
|---|---|---|---|---|
| ||||
hdfs dfs -mkdir /mining hdfs dfs -chown mining:mining /mining |
5.验证Hadoop
①在浏览器输入: http://本机ip:50070/dfshealth.html#tab-overview, 检查集群状态
②检查sbimining目录是否创建成功②检查mining目录是否创建成功
| 代码块 | ||||
|---|---|---|---|---|
| ||||
hdfs dfs -ls / #显示创建的/mining即表示创建成功 |
如上显示，表示Hadoop安装成功。
6.运维操作
停止hadoop
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/ ./sbin/stop-dfs.sh |
启动hadoop
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/ ./sbin/start-dfs.sh |
3、Hadoop集群部署
部署服务器：
| IP地址 | 主机名 | 角色 |
|---|---|---|
10.10.17.186 | hadoop | hadoop namenode,hadoop datanode |
10.10.17.127 | hadoop2 | hadoop datanode |
10.10.17.39 | hadoop3 | hadoop datanode |
1.修改主机名-添加映射关系
根据部署实际环境，各个服务器主机名不同即可不修改主机名，如需修改，可参考下面的修改方式：
| 代码块 | ||
|---|---|---|
| ||
hostnamectl set-hostname 主机名 |
PS：主机名不能使用下划线
配置主机名和IP的映射关系
| 代码块 | ||
|---|---|---|
| ||
vi /etc/hosts |
内容设置，例如:
| 代码块 | ||
|---|---|---|
| ||
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.10.17.186 hadoop 10.10.17.127 hadoop2 10.10.17.39 hadoop3 |
2.设置系统免密登陆(集群服务器都执行)
生成密钥
| 代码块 | ||
|---|---|---|
| ||
ssh-keygen |
输入ssh-keygen后，连续按三次回车，不用输入其它信息
复制公钥
| 代码块 | ||
|---|---|---|
| ||
ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop2 ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop3 |
如果以上命令执行失败，可以手工把每个节点 ~/.ssh/id_rsa.pub的数据追加到其它节点的~/.ssh/authorized_keys文件中, 如果~/.ssh/authorized_keys不存在，可以新建一个.
测试是否设置成功，例如:
| 代码块 | ||
|---|---|---|
| ||
ssh root@hadoop ssh root@hadoop2 ssh root@hadoop3 |
如果不用输入密码，表示成功。
3.创建Hadoop相关目录(集群服务器都执行)
创建hadoop相关目录(集群每个节点均需操作)
创建临时目录
| 代码块 | ||
|---|---|---|
| ||
mkdir -p /opt/hdfs/tmp |
创建namenode数据目录
| 代码块 | ||
|---|---|---|
| ||
mkdir -p /opt/hdfs/name |
创建datanode目录
注意这个目录尽量创建在空间比较大的目录，如有多个磁盘，可创建多个目录
| 代码块 | ||
|---|---|---|
| ||
mkdir -p /opt/hdfs/data |
4.在管理节点安装配置Hadoop
例如在hadoop节点执行
①解压hadoop安装包到指定目录
| 代码块 | ||
|---|---|---|
| ||
tar -zxvf hadoop-2.7.3.tar.gz -C /opt |
②修改配置文件
1.修改hadoop-env.sh
| 代码块 | ||
|---|---|---|
| ||
cd /opt/hadoop-2.7.3/etc/hadoop vi hadoop-env.sh |
找到export JAVA_HOME= ，修改Java安装路径如下所示
| 代码块 | ||
|---|---|---|
| ||
export JAVA_HOME=/opt/jdk8.0.202-linux_x64 |
找到export HADOOP_OPTS，在下面添加一行
| 代码块 | ||
|---|---|---|
| ||
export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC -Xmx4g" |
2、修改core-site.xml配置文件
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/etc/hadoop vi core-site.xml |
内容如下：
| 代码块 | ||||
|---|---|---|---|---|
| ||||
<configuration>
<property>
<name>fs.defaultFS</name>
<!--根据实际情况替换成本机的IP或主机名 -->
<value>hdfs://hadoop:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/opt/hdfs/tmp</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>100800</value>
</property>
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
</configuration> |
3、修改hdfs-site.xml配置文件
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/etc/hadoop vi hdfs-site.xml |
内容如下：
| 代码块 | ||||
|---|---|---|---|---|
| ||||
<configuration>
<property>
<name>dfs.name.dir</name>
<value>file:/opt/hdfs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file:/opt/hdfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.max.transfer.threads</name>
<value>16384</value>
</property>
</configuration> |
| 注意 |
|---|
注意：dfs.data.dir尽量配置在空间比较大的目录，可以配置多个目录，中间用逗号分隔 |
4、修改hadoop-policy.xml文件
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/etc/hadoop vi hadoop-policy.xml |
内容如下：
| 代码块 | ||||
|---|---|---|---|---|
| ||||
<configuration>
<property>
<name>security.client.protocol.acl</name>
<value>*</value>
<description>ACL for ClientProtocol, which is used by user code
via the DistributedFileSystem.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<!-- 这里把实验引擎ip, python执行节点ip，spark部署机器ip，hadoop部署机器ip都加上-->
<!-- 如果实验引擎，python执行节点，spark，hadoop这些组件是集群部署，那么所有的IP地址都需要添加进来 -->
<property>
<name>security.client.protocol.hosts</name>
<value>192.168.137.139,192.168.137.140,192.168.137.141,192.168.137.142,192.168.137.143</value>
</property>
<property>
<name>security.client.datanode.protocol.acl</name>
<value>*</value>
<description>ACL for ClientDatanodeProtocol, the client-to-datanode protocol
for block recovery.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<!-- 这里把实验引擎ip,python执行节点ip，spark部署机器ip，hadoop部署机器ip都加上-->
<!-- 如果实验引擎，python执行节点，spark，hadoop这些组件是集群部署，那么所有的IP地址都需要添加进来 -->
<property>
<name>security.client.datanode.protocol.hosts</name>
<value>192.168.137.139,192.168.137.140,192.168.137.141,192.168.137.142,192.168.137.143</value>
</property>
<property>
<name>security.datanode.protocol.acl</name>
<value>*</value>
<description>ACL for DatanodeProtocol, which is used by datanodes to
communicate with the namenode.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.inter.datanode.protocol.acl</name>
<value>*</value>
<description>ACL for InterDatanodeProtocol, the inter-datanode protocol
for updating generation timestamp.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.namenode.protocol.acl</name>
<value>*</value>
<description>ACL for NamenodeProtocol, the protocol used by the secondary
namenode to communicate with the namenode.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.admin.operations.protocol.acl</name>
<value>*</value>
<description>ACL for AdminOperationsProtocol. Used for admin commands.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.refresh.user.mappings.protocol.acl</name>
<value>*</value>
<description>ACL for RefreshUserMappingsProtocol. Used to refresh
users mappings. The ACL is a comma-separated list of user and
group names. The user and group list is separated by a blank. For
e.g. "alice,bob users,wheel". A special value of "*" means all
users are allowed.</description>
</property>
<property>
<name>security.refresh.policy.protocol.acl</name>
<value>*</value>
<description>ACL for RefreshAuthorizationPolicyProtocol, used by the
dfsadmin and mradmin commands to refresh the security policy in-effect.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.ha.service.protocol.acl</name>
<value>*</value>
<description>ACL for HAService protocol used by HAAdmin to manage the
active and stand-by states of namenode.</description>
</property>
<property>
<name>security.zkfc.protocol.acl</name>
<value>*</value>
<description>ACL for access to the ZK Failover Controller
</description>
</property>
<property>
<name>security.qjournal.service.protocol.acl</name>
<value>*</value>
<description>ACL for QJournalProtocol, used by the NN to communicate with
JNs when using the QuorumJournalManager for edit logs.</description>
</property>
<property>
<name>security.mrhs.client.protocol.acl</name>
<value>*</value>
<description>ACL for HSClientProtocol, used by job clients to
communciate with the MR History Server job status etc.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<!-- YARN Protocols -->
<property>
<name>security.resourcetracker.protocol.acl</name>
<value>*</value>
<description>ACL for ResourceTrackerProtocol, used by the
ResourceManager and NodeManager to communicate with each other.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.resourcemanager-administration.protocol.acl</name>
<value>*</value>
<description>ACL for ResourceManagerAdministrationProtocol, for admin commands.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.applicationclient.protocol.acl</name>
<value>*</value>
<description>ACL for ApplicationClientProtocol, used by the ResourceManager
and applications submission clients to communicate with each other.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.applicationmaster.protocol.acl</name>
<value>*</value>
<description>ACL for ApplicationMasterProtocol, used by the ResourceManager
and ApplicationMasters to communicate with each other.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.containermanagement.protocol.acl</name>
<value>*</value>
<description>ACL for ContainerManagementProtocol protocol, used by the NodeManager
and ApplicationMasters to communicate with each other.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.resourcelocalizer.protocol.acl</name>
<value>*</value>
<description>ACL for ResourceLocalizer protocol, used by the NodeManager
and ResourceLocalizer to communicate with each other.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.job.task.protocol.acl</name>
<value>*</value>
<description>ACL for TaskUmbilicalProtocol, used by the map and reduce
tasks to communicate with the parent tasktracker.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.job.client.protocol.acl</name>
<value>*</value>
<description>ACL for MRClientProtocol, used by job clients to
communciate with the MR ApplicationMaster to query job status etc.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
<property>
<name>security.applicationhistory.protocol.acl</name>
<value>*</value>
<description>ACL for ApplicationHistoryProtocol, used by the timeline
server and the generic history service client to communicate with each other.
The ACL is a comma-separated list of user and group names. The user and
group list is separated by a blank. For e.g. "alice,bob users,wheel".
A special value of "*" means all users are allowed.</description>
</property>
</configuration> |
| 注意 | ||
|---|---|---|
| ||
hadoop-policy.xml配置文件中，security.client.protocol.hosts，security.client.datanode.protocol.hosts 这两个配置项的值，要改成实际部署环境的IP地址 |
5、修改slaves配置文件
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/etc/hadoop vi slaves |
把所有的datanode的主机名添加到文件中，例如：
| 代码块 | ||||
|---|---|---|---|---|
| ||||
hadoop hadoop2 hadoop3 |
5.分发Hadoop安装包（管理节点执行）
将hadoop管理节点上的hadoop安装包分发到其他节点：
| 代码块 | ||||
|---|---|---|---|---|
| ||||
scp -r /opt/hadoop-2.7.3 root@hadoop2:/opt scp -r /opt/hadoop-2.7.3 root@hadoop3:/opt |
6.配置Hadoop环境变量（每个节点执行）
添加环境变量。
| 代码块 | ||||
|---|---|---|---|---|
| ||||
vi /etc/profile |
在最底下添加下面内容：
| 代码块 | ||||
|---|---|---|---|---|
| ||||
export HADOOP_HOME=/opt/hadoop-2.7.3 export PATH=$PATH:$HADOOP_HOME/bin |
让配置生效
| 代码块 | ||||
|---|---|---|---|---|
| ||||
source /etc/profile |
7.启动Hadoop集群
①格式化hadoop（管理节点执行）
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/ ./bin/hdfs namenode -format |
| 注意 | ||
|---|---|---|
| ||
仅第一次启动时需要执行格式化Hadoop操作，后续启动无需进行此操作 |
②启动hadoop
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/ ./sbin/start-dfs.sh |
③创建中间数据存储目录(数据挖掘实验引擎集群需要使用)
| 代码块 | ||||
|---|---|---|---|---|
| ||||
hdfs dfs -mkdir /mining hdfs dfs -chown mining:mining /mining |
8.验证Hadoop集群
①在浏览器输入: http://本机ip:50070/dfshealth.html#tab-overview, 检查集群状态
②检查sbimining目录是否创建成功②检查mining目录是否创建成功
| 代码块 | ||||
|---|---|---|---|---|
| ||||
hdfs dfs -ls / #显示/mining目录，表示创建成功 |
如上显示，表示hadoop集群安装成功
9.运维操作
停止hadoop
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/ ./sbin/stop-dfs.sh |
启动hadoop
| 代码块 | ||||
|---|---|---|---|---|
| ||||
cd /opt/hadoop-2.7.3/ ./sbin/start-dfs.sh |
| 目录 |
|---|
