页面历史

数据挖掘引擎V95到Beta版本升级内容如下：

数据挖掘组件	V95版本

V96版本

Beta版本	更新内容
实验引擎	√	√	数据挖掘引擎版本更新
服务引擎	√	√	数据挖掘引擎版本更新
Spark	√	√

无改动

Spark版本由2.4升级到3.0版本
Python执行节点	√	√	数据挖掘引擎版本更新，新增代理程序启动用户。
Hadoop	×	√	新增Hadoop组件，用于节点中间数据存储。

一、安装Hadoop组件

数据挖掘引擎包更新

数据挖掘V96版本增加了加点中间数据存储，所以需要安装Hadoop组件。

获取新版本的数据挖掘引擎安装包。

注意

title	注意事项

注意：原有环境如果装的spark是单机版，那就在spark服务器中，安装hadoop单机版。如果装的是spark集群，那就在spark集群机器中安装hadoop集群

1、系统环境设置

单机或集群部署Hadoop集群，均需设置系统环境

1.开放防火墙端口

服务名需要开放端口

Hadoop

50090，50070，9000，50010，50075，50020

如果确实需要打开防火墙安装，需要给防火墙放开以下需要使用到的端口
开启端口：50090，50070，9000，50010，50075，50020配置完以后重新加载firewalld，使配置生效

新数据挖掘引擎安装包解压缩后；

先备份<数据挖掘安装目录>/engine目录；

再删除<数据挖掘安装目录>/engine目录，然后上传新的engine目录，并重启数据挖掘引擎。

数据挖掘引擎安装包版本要和smartbi的war包版本一致，更新时需要同步更新Python节点中的引擎包。

Spark版本升级

1、停止spark

进入spark安装目录，执行命令停止spark2.4服务

代码块

linenumbers	true

firewall-cmd --zone=public --add-port=50090/tcp --permanent
firewall-cmd --zone=public --add-port=50070/tcp --permanent
firewall-cmd --zone=public --add-port=9000/tcp --permanent
firewall-cmd --zone=public --add-port=50010/tcp --permanent
firewall-cmd --zone=public --add-port=50075/tcp --permanent
firewall-cmd --zone=public --add-port=50020/tcp --permanent

languagebash

cd /data/spark-2.4.0-bin-hadoop2.7/sbin/     #注意进入实际spark部署目录
./stop-all.sh

2、安装spark3.0

上传新的spark3.0安装包到服务器，并解压到指定目录(安装目录可自定义)

代码块

linenumbers	true

tar -zxvf spark-3.0.0-bin-hadoop2.7.tgz -C /opt

3、配置spark3.0

代码块

linenumbers	true

cd /opt/spark-3.0.0-bin-hadoop2.7/conf
cp spark-defaults.conf.template spark-defaults.conf   
vi spark-defaults.conf

在配置文件末尾添加以下内容，保存

代码块

linenumbers	true

firewall-cmd --reload

查看防火墙的配置信息

代码块

linenumbers	true

firewall-cmd --list-all

2.取消打开文件限制

修改/etc/security/limits.conf文件在文件的末尾加入以下内容：

代码块

spark.authenticate true
spark.authenticate.secret kW9y@5yheyJ&IMlD41Dlv#lHFKi7fg7#

Image Added

注意

title	配置解析

其中 kW9y@5yheyJ&IMlD41Dlv#lHFKi7fg7# 是默认spark认证密钥, 支持更改成自定义的字符串(大小写英文字母+数字+英文符号构成)。
如不使用默认值，需要在部署实验引擎时, experiment-application.properties文件修改配置项(如果没有该配置项则需要添加)。
spark.authenticate.secret=你修改的密钥

4.启动Spark3.0

①启动Spark master

代码块

linenumbers	true

vi

cd /

etc/security/limits.conf

在文件的末尾加入以下内容：

代码块languagebash

2、Hadoop单节点安装

1.设置免密登陆

生成密钥

代码块

linenumbers	true

ssh-keygen

输入ssh-keygen后，连续按三次回车，不用输入其它信息
复制公钥

opt/spark-3.0.0-bin-hadoop2.7/sbin
./start-master.sh -h 主机名

例如：主机名为smartbi-spark，则执行：

代码块

linenumbers	true

* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072

测试是否设置成功，例如:

代码块

linenumbers	true

ssh root@hadoop

如果不用输入密码，表示成功。

2.安装Hadoop

创建hadoop相关目录

创建临时目录

代码块

linenumbers	true

mkdir -p /opt/hdfs/tmp

创建namenode数据目录

./start-master.sh -h smartbi-spark

②启动Spark work

注意:参数 -c 为分配给spark work 节点的cpu核数，-m 为分配给spark work节点内存值

代码块

linenumbers	true

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

cd /opt/spark-3.0.0-bin-hadoop2.7/sbin
./start-slave.sh spark://master节点的主机名:7077 -c 配置的cpu数 -m xg 配置的内存数(g为单位)

注意
Work 节点最低配置为 1 核 8G 内存。 cpu 和内存比值建议为 1:8 ，即一个 cpu 配置 8G 的内存

例如：worker节点为8核，64G内存的配置，master的主机名为smartbi-spark，2核16G留给系统跟hadoop，剩下的资源留给spark。则执行:

代码块

linenumbers	true

./start-slave.sh spark://smartbi-spark:7077 -c 6 -m 48g

5.检查Spark

在浏览器中输入：http://master节点的ip:8080，查看集群状态

Image Added

在master节点提交任务，执行以下命令(注意将”节点的IP”替换对应的IP或主机名)

代码块

linenumbers	true

mkdir

cd

-p

/opt/

hdfs/name

创建datanode目录
注意这个目录尽量创建在空间比较大的目录，如有多个磁盘，可创建多个目录

代码块

linenumbers	true

mkdir -p /opt/hdfs/data

解压hadoop安装包到指定目录

代码块

linenumbers	true

tar -zxvf hadoop-2.7.3.tar.gz -C /opt

修改Hadoop配置文件

1.修改hadoop-env.sh

代码块

linenumbers	true

cd /opt/hadoop-2.7.3/etc/hadoop
vi hadoop-env.sh

找到export JAVA_HOME= ，修改Java安装路径如下所示

代码块

linenumbers	true

export JAVA_HOME=/opt/jdk8.0.202-linux_x64

找到export HADOOP_OPTS，在下面添加一行

代码块

linenumbers	true

export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC -Xmx4g"

2、修改core-site.xml配置文件

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/etc/hadoop
vi core-site.xml

内容如下：

代码块

language	xml
linenumbers	true

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <!--根据实际情况替换成本机的IP或主机名 -->
        <value>hdfs://hadoop:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/opt/hdfs/tmp</value>
    </property>
    <property>
        <name>fs.trash.interval</name>
        <value>100800</value>
    </property>
    <property>
        <name>hadoop.security.authorization</name>
        <value>true</value>
    </property>
</configuration>

3、修改hdfs-site.xml配置文件

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/etc/hadoop
vi hdfs-site.xml

内容如下：

代码块

language	xml
linenumbers	true

spark-3.0.0-bin-hadoop2.7/bin
./spark-submit --class org.apache.spark.examples.SparkPi --master spark://节点的ip:7077 /opt/spark-3.0.0-bin-hadoop2.7/examples/jars/spark-examples_2.12-3.0.0.jar 100

Image Added

运行得出圆周率Pi的近似值3.14即部署成功。

安装Hadoop组件

数据挖掘V96版本增加了节点中间数据存储Hadoop，非必选组件，可不安装。

系统环境设置

单机或集群部署Hadoop集群，均需设置系统环境

1.开放防火墙端口

服务名	需要开放端口
Hadoop	50090，50070，9000，50010，50075，50020

如果确实需要打开防火墙安装，需要给防火墙放开以下需要使用到的端口
开启端口：50090，50070，9000，50010，50075，50020

代码块

linenumbers	true

firewall-cmd --zone=public --add-port=50090/tcp --permanent
firewall-cmd --zone=public --add-port=50070/tcp --permanent
firewall-cmd --zone=public --add-port=9000/tcp --permanent
firewall-cmd --zone=public --add-port=50010/tcp --permanent
firewall-cmd --zone=public --add-port=50075/tcp --permanent
firewall-cmd --zone=public --add-port=50020/tcp --permanent

配置完以后重新加载firewalld，使配置生效

代码块

linenumbers	true

firewall-cmd --reload

查看防火墙的配置信息

代码块

linenumbers	true

firewall-cmd --list-all

2.取消打开文件限制

修改/etc/security/limits.conf文件在文件的末尾加入以下内容：

代码块

language	bash
linenumbers	true

vi /etc/security/limits.conf

在文件的末尾加入以下内容：

代码块

language	bash
linenumbers	true

* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072

Hadoop单节点安装

如果部署Hadoop集群，则跳过该章节，查看Hadoop集群部署章节。

1.设置免密登陆

生成密钥

代码块

linenumbers	true

ssh-keygen

输入ssh-keygen后，连续按三次回车，不用输入其它信息
复制公钥

代码块

linenumbers	true

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

测试是否设置成功，例如:

代码块

linenumbers	true

ssh root@hadoop

如果不用输入密码，表示成功。

2.安装Hadoop

创建hadoop相关目录

创建临时目录

代码块

linenumbers	true

mkdir -p /opt/hdfs/tmp

创建namenode数据目录

代码块

linenumbers	true

mkdir -p /opt/hdfs/name

创建datanode目录
注意这个目录尽量创建在空间比较大的目录，如有多个磁盘，可创建多个目录

代码块

linenumbers	true

mkdir -p /opt/hdfs/data

解压hadoop安装包到指定目录

代码块

linenumbers	true

tar -zxvf hadoop-2.7.3.tar.gz -C /opt

修改Hadoop配置文件

1.修改hadoop-env.sh

代码块

linenumbers	true

cd /opt/hadoop-2.7.3/etc/hadoop
vi hadoop-env.sh

找到export JAVA_HOME= ，修改Java安装路径如下所示

代码块

linenumbers	true

export JAVA_HOME=/opt/jdk8.0.202-linux_x64

找到export HADOOP_OPTS，在下面添加一行

代码块

linenumbers	true

export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC -Xmx4g"

2、修改core-site.xml配置文件

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/etc/hadoop
vi core-site.xml

内容如下：

代码块

language	xml
linenumbers	true

<configuration>
    <property>

<name>dfs

<name>fs.

replication<

defaultFS</name>

<value>1<

<!--根据实际情况替换成本机的IP或主机名 -->
        <value>hdfs://hadoop:9000</value>
    </property>
    <property>

<name>dfs

<name>hadoop.

webhdfs

tmp.

enabled<

dir</name>

<value>false</value>

<!-- 注意替换实际目录 -->
        <value>file:/opt/hdfs/tmp</value>
    </property>
    <property>

<name>dfs

<name>fs.

datanode.max.transfer.threads<

trash.interval</name>
        <value>100800</value>
    </property>
    <property>
        <name>hadoop.security.authorization</name>

<value>16384<linenumbers

<value>true</value>
    </property>
</configuration>

注意

title	注意事项

注意：dfs.data.dir尽量配置在空间比较大的目录，可以配置多个目录，中间用逗号分隔

4、修改hadoop-policy.xml

代码块

3、修改hdfs-site.xml配置文件

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/etc/hadoop
vi

hadoop

hdfs-

policy

site.xml

内容如下:

内容如下：

代码块

language	xml
linenumbers	true

<configuration>
    <property>

<name>security.client.protocol.acl<

    <name>dfs.name.dir</name>

   <!-- 注意替换实际目录 -->

<description>ACL

for

ClientProtocol,

which

is

used by user code

 <value>file:/opt/hdfs/name</value>
    </property>

via

the

DistributedFileSystem.

 <property>

The

ACL

is

a

comma-separated list of user and group names. The user and group list is separated by a blank. For e.g. "alice,bob users,wheel".

 <name>dfs.data.dir</name>
       <!-- 注意替换实际目录 -->
        <value>file:/opt/hdfs/data</value>
    </property>

A

<property>

special

value

of

"*"

means

all

users

are

allowed

<name>dfs.

<

replication</

description>

name>

</property>

 <value>1</value>
    <

!--

/property>

如果实验引擎，python执行节点，spark，hadoop这些组件是集群部署，那么所有的IP地址都需要添加进来

-->

   <property>

<name>security

<name>dfs.

client

webhdfs.

protocol.hosts<

enabled</name>

   <value>false</value>
    </property>
    <property>

<name>security

 <name>dfs.

client

datanode.

datanode

max.

protocol

transfer.

acl<

threads</name>

    <value>16384</value>

<description>ACL

for

ClientDatanodeProtocol,

the client-to-datanode protocol for block recovery. The ACL is a comma-separated list of user and group

 </property>
</configuration>

注意

title	注意事项

注意：dfs.data.dir尽量配置在空间比较大的目录，可以配置多个目录，中间用逗号分隔

4、修改hadoop-policy.xml

代码块

linenumbers	true

cd /opt/hadoop-2.7.3/etc/hadoop
vi hadoop-policy.xml

内容如下:

代码块

language	xml
linenumbers	true

<configuration>
  <property>
    <name>security.client.protocol.acl</name>
    <value>*</value>
    <description>ACL for ClientProtocol, which is used by user code
    via the DistributedFileSystem.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>
 
  <!-- 这里把实验引擎ip, python执行节点ip，spark部署机器ip，hadoop部署机器ip都加上--> 
  <!-- 如果实验引擎，python执行节点，spark，hadoop这些组件是集群部署，那么所有的IP地址都需要添加进来 -->
  <!-- 增加以下配置 -->
  <property>
    <name>security.client.

datanode.

protocol.hosts</name>
    <value>192.168.137.139,192.168.137.140,192.168.137.141</value>
  </property>
   <!-- end -->
  <property>
    <name>security.client.datanode.protocol.acl</name>
    <value>*</value>
    <description>ACL for

DatanodeProtocol

ClientDatanodeProtocol,

which is used by datanodes to

the client-to-datanode protocol

communicate

with

for

the

block

namenode

recovery.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

<!-- 这里把实验引擎ip,python执行节点ip，spark部署机器ip，hadoop部署机器ip都加上-->

<name>security.inter.datanode.protocol.acl</name>

<!-- 如果实验引擎，python执行节点，spark，hadoop这些组件是集群部署，那么所有的IP地址都需要添加进来 -->

<!-- 增加以下配置 -->

<description>ACL

<property>

for

InterDatanodeProtocol,

the inter-datanode protocol

 <name>security.client.datanode.protocol.hosts</name>

for updating generation timestamp. The ACL is a comma-separated list of user and group names. The user and group list is separated by a blank. For e.

<value>192.168.137.139,192.168.137.140,192.168.137.141</value>
  </property>
  <!-- end -->
  <property>
    <name>security.datanode.protocol.acl</name>
    <value>*</value>
    <description>ACL for DatanodeProtocol, which is used by datanodes to
    communicate with the namenode.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.

namenode

inter.datanode.protocol.acl</name>
    <value>*</value>
    <description>ACL for

NamenodeProtocol

InterDatanodeProtocol, the inter-datanode protocol

used

by

the

secondary

 for updating generation timestamp.

namenode

to

communicate

with the namenode.

 The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.

admin.operations

namenode.protocol.acl</name>
    <value>*</value>
    <description>ACL for

AdminOperationsProtocol. Used for admin commands

NamenodeProtocol, the protocol used by the secondary
    namenode to communicate with the namenode.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>



 <property>
    <name>security.

refresh

admin.

user.mappings

operations.protocol.acl</name>
    <value>*</value>
    <description>ACL for

RefreshUserMappingsProtocol

AdminOperationsProtocol. Used

to

for

refresh

admin commands.

users

mappings.

The ACL is a comma-separated list of user and

group names. The user and
    group list is separated by a blank. For

e.g. "alice,bob users,wheel".
    A special value of "*" means all

users are allowed.</description>
  </property>

  <property>
    <name>security.refresh.

policy

user.mappings.protocol.acl</name>
    <value>*</value>
    <description>ACL for

RefreshAuthorizationPolicyProtocol, used by the dfsadmin and mradmin commands

RefreshUserMappingsProtocol. Used to refresh

the

security

policy

in-effect.

 users mappings.

The ACL is a comma-separated list of user and
    group names. The user and

group list is separated by a blank. For
    e.g. "alice,bob users,wheel".

  A special value of "*" means all
    users are allowed.</description>
  </property>

  <property>
    <name>security.

ha

refresh.

service

policy.protocol.acl</name>
    <value>*</value>
    <description>ACL for

HAService protocol

RefreshAuthorizationPolicyProtocol, used by

HAAdmin

the

to

manage

the

  dfsadmin and mradmin commands to

active

refresh

and

the

stand-by

security

states of namenode.</description>

policy in-effect.

</property> <property> <name>security.zkfc.protocol.acl</name> <value>*</value> <description>ACL for access to the ZK Failover Controller

  The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.

qjournal

ha.service.protocol.acl</name>
    <value>*</value>
    <description>ACL for HAService

QJournalProtocol,

protocol used by

the NN

HAAdmin to

communicate

manage

with

the

JNs

when

using

active

the

and

QuorumJournalManager

stand-by

for

states

edit

of

logs

namenode.</description>
  </property>

  <property>
    <name>security.

mrhs.client

zkfc.protocol.acl</name>
    <value>*</value>
    <description>ACL for

HSClientProtocol,

access

used

to

by

the

job

ZK

clients

Failover

to

Controller

communciate

</description>

with

the

MR

</property>

History

Server

job

status

<property>

etc.

    <name>security.qjournal.service.protocol.acl</name>

The

ACL

is

a comma-separated list of user and group names. The user and group list is separated by a blank. For e.g. "alice,bob users,wheel".

<value>*</value>
    <description>ACL for QJournalProtocol, used by the NN to communicate with

A

JNs

special

when

value

using

of

the

"*"

QuorumJournalManager

means

for

all users are allowed

edit logs.</description>
  </property>

<!-- YARN

Protocols

-->

<property>
    <name>security.

resourcetracker

mrhs.client.protocol.acl</name>
    <value>*</value>
    <description>ACL for

ResourceTrackerProtocol

HSClientProtocol, used by job clients to

the

    communciate

ResourceManager

with

and

the

NodeManager

MR

to

History

communicate

Server

with

job

each

status

other

etc. 
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <!-- YARN Protocols -->

  <property>
    <name>security.

resourcemanager-administration

resourcetracker.protocol.acl</name>
    <value>*</value>
    <description>ACL for

ResourceManagerAdministrationProtocol, for admin commands

ResourceTrackerProtocol, used by the
    ResourceManager and NodeManager to communicate with each other.

The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.

applicationclient

resourcemanager-administration.protocol.acl</name>
    <value>*</value>
    <description>ACL for

ApplicationClientProtocol

ResourceManagerAdministrationProtocol,

used

for

by

admin

the ResourceManager and applications submission clients to communicate with each other.

commands. 
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.

applicationmaster

applicationclient.protocol.acl</name>
    <value>*</value>
    <description>ACL for

ApplicationMasterProtocol

ApplicationClientProtocol, used by the ResourceManager 
    and

ApplicationMasters

applications submission clients to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.

containermanagement

applicationmaster.protocol.acl</name>
    <value>*</value>
    <description>ACL for

ContainerManagementProtocol protocol

ApplicationMasterProtocol, used by the

NodeManager

ResourceManager 
    and ApplicationMasters to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.

resourcelocalizer

containermanagement.protocol.acl</name>
    <value>*</value>
    <description>ACL for

ResourceLocalizer

ContainerManagementProtocol protocol, used by the NodeManager 
    and

ResourceLocalizer

ApplicationMasters to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.

job.task

resourcelocalizer.protocol.acl</name>
    <value>*</value>
    <description>ACL for ResourceLocalizer

TaskUmbilicalProtocol

protocol, used by the

map

NodeManager

and

reduce

and

tasks

ResourceLocalizer to communicate with

the

each

parent tasktracker

other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.job.

client

task.protocol.acl</name>
    <value>*</value>
    <description>ACL for

MRClientProtocol

TaskUmbilicalProtocol, used by

job clients to

 the map and reduce
    tasks to

communciate

communicate with the

MR ApplicationMaster to query job status etc

parent tasktracker.

The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.job.

applicationhistory

client.protocol.acl</name>
    <value>*</value>
    <description>ACL for

ApplicationHistoryProtocol

MRClientProtocol, used by

the

job clients

timeline

to

server

communciate

and

with the

generic history service client

MR ApplicationMaster to

communicate

query

with

job

each

status

other

etc. 
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by

a blank. For e.g. "alice,bob users,wheel". A special value of "*" means all users are allowed.</description> </property> </configuration>

注意

title	注意事项

hadoop-policy.xml配置文件中，security.client.protocol.hosts，security.client.datanode.protocol.hosts 这两个配置项的值，要改成实际部署环境的IP地址

配置Hadoop环境变量

添加环境变量。在最底下添加下面内容：

 a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.applicationhistory.protocol.acl</name>
    <value>*</value>
    <description>ACL for ApplicationHistoryProtocol, used by the timeline
    server and the generic history service client to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>
</configuration>

注意

title	注意事项

hadoop-policy.xml配置文件中，security.client.protocol.hosts，security.client.datanode.protocol.hosts 这两个配置项的值，要改成实际部署环境的IP地址

配置Hadoop环境变量

添加环境变量。

代码块

language	bash
linenumbers	true

 vi ~/.bash_profile

在最底下添加下面内容：

代码块

language	bash
linenumbers	true

export HADOOP_HOME=/opt/hadoop-2.7.3
export PATH=$PATH:$HADOOP_HOME/bin

让配置生效

代码块

language	bash
linenumbers	true

source ~/.bash_profile

3.启动Hadoop

①格式化hadoop

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/
./bin/hdfs namenode -format

注意

title	注意事项

仅第一次启动时需要执行格式化Hadoop操作，后续启动无需进行此操作

②启动hadoop

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/
./sbin/start-dfs.sh

③创建中间数据存储目录(数据挖掘实验引擎集群需要使用)

代码块

language	bash
linenumbers	true

hdfs dfs -mkdir /mining
hdfs dfs -chown mining:mining /mining

4.验证Hadoop

①在浏览器输入: http://本机ip:50070/dfshealth.html#tab-overview, 检查集群状态

Image Added

②检查mining目录是否创建成功

代码块

language	bash
linenumbers	true

hdfs dfs -ls /    #显示创建的/mining即表示创建成功

如上显示，表示Hadoop安装成功。

5.运维操作

停止hadoop

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/
./sbin/stop-dfs.sh

启动hadoop

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/
./sbin/start-dfs.sh

Hadoop集群安装

如已部署Hadoop单机，则跳过Hadoop集群部署章节，Hadoop非必选组件，可不安装。

例如集群服务器：

机器	主机名	组件实例
192.168.137.141	smartbi-spark	spark master,spark worker,hadoop namenode,hadoop datanode
192.168.137.142	smartbi-spark2	spark worker,hadoop datanode
192.168.137.143	smartbi-spark3	spark worker,hadoop datanode

1.设置免密登陆（集群服务器都执行）

生成密钥

代码块

linenumbers	true

ssh-keygen

输入ssh-keygen后，连续按三次回车，不用输入其它信息
复制公钥

代码块

linenumbers	true

ssh-copy-id -i ~/.ssh/id_rsa.pub root@smartbi-spark
ssh-copy-id -i ~/.ssh/id_rsa.pub root@smartbi-spark2
ssh-copy-id -i ~/.ssh/id_rsa.pub root@smartbi-spark3

测试是否设置成功，例如:

代码块

linenumbers	true

ssh root@smartbi-spark
ssh root@smartbi-spark2
ssh root@smartbi-spark3

如果不用输入密码，表示成功。

创建hadoop相关目录（集群服务器都执行）

创建临时目录

代码块

linenumbers	true

mkdir -p /opt/hdfs/tmp

创建namenode数据目录

代码块

linenumbers	true

mkdir -p /opt/hdfs/name

创建datanode目录

注意这个目录尽量创建在空间比较大的目录，如有多个磁盘，可创建多个目录

代码块

linenumbers	true

mkdir -p /opt/hdfs/data

在管理节点安装配置Hadoop

例如在smartbi-spark节点执行

解压hadoop安装包到指定目录

代码块

linenumbers	true

tar -zxvf hadoop-2.7.3.tar.gz -C /opt

②修改配置文件

1.修改hadoop-env.sh

代码块

linenumbers	true

cd /opt/hadoop-2.7.3/etc/hadoop
vi hadoop-env.sh

找到export JAVA_HOME= ，修改Java安装路径如下所示

代码块

linenumbers	true

export JAVA_HOME=/opt/jdk8.0.202-linux_x64

找到export HADOOP_OPTS，在下面添加一行

代码块

linenumbers	true

export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC -Xmx4g"

2、修改core-site.xml配置文件

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/etc/hadoop
vi core-site.xml

内容如下：

代码块

language	xml
linenumbers	true

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <!--根据实际情况替换成本机的IP或主机名 -->
        <value>hdfs://smartbi-spark:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
       <!-- 注意替换实际目录 -->
        <value>file:/opt/hdfs/tmp</value>
    </property>
    <property>
        <name>fs.trash.interval</name>
        <value>100800</value>
    </property>
    <property>
        <name>hadoop.security.authorization</name>
        <value>true</value>
    </property>
</configuration>

3、修改hdfs-site.xml配置文件

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/etc/hadoop
vi hdfs-site.xml

内容如下：

代码块

language	xml
linenumbers	true

<configuration>
    <property>
        <name>dfs.name.dir</name>
       <!-- 注意替换实际目录 -->
        <value>file:/opt/hdfs/name</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
       <!-- 注意替换实际目录 -->
        <value>file:/opt/hdfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>dfs.datanode.max.transfer.threads</name>
        <value>16384</value>
    </property>
</configuration>

注意
注意：dfs.data.dir尽量配置在空间比较大的目录，可以配置多个目录，中间用逗号分隔

4、修改hadoop-policy.xml文件

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/etc/hadoop
vi hadoop-policy.xml

内容如下：

代码块

language	xml
linenumbers	true

<configuration>
  <property>
    <name>security.client.protocol.acl</name>
    <value>*</value>
    <description>ACL for ClientProtocol, which is used by user code
    via the DistributedFileSystem.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>
 
  <!-- 这里把实验引擎ip, python执行节点ip，spark部署机器ip，hadoop部署机器ip都加上--> 
  <!-- 如果实验引擎，python执行节点，spark，hadoop这些组件是集群部署，那么所有的IP地址都需要添加进来 -->
  <!-- 增加以下配置 -->
  <property>
    <name>security.client.protocol.hosts</name>
    <value>192.168.137.139,192.168.137.140,192.168.137.141,192.168.137.142,192.168.137.143</value>
  </property>
  <!-- end -->
  <property>
    <name>security.client.datanode.protocol.acl</name>
    <value>*</value>
    <description>ACL for ClientDatanodeProtocol, the client-to-datanode protocol
    for block recovery.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <!-- 这里把实验引擎ip,python执行节点ip，spark部署机器ip，hadoop部署机器ip都加上-->
  <!-- 如果实验引擎，python执行节点，spark，hadoop这些组件是集群部署，那么所有的IP地址都需要添加进来 -->
  <!-- 增加以下配置 -->
  <property>
    <name>security.client.datanode.protocol.hosts</name>
    <value>192.168.137.139,192.168.137.140,192.168.137.141,192.168.137.142,192.168.137.143</value>
  </property>
  <!-- end -->
  <property>
    <name>security.datanode.protocol.acl</name>
    <value>*</value>
    <description>ACL for DatanodeProtocol, which is used by datanodes to
    communicate with the namenode.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.inter.datanode.protocol.acl</name>
    <value>*</value>
    <description>ACL for InterDatanodeProtocol, the inter-datanode protocol
    for updating generation timestamp.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.namenode.protocol.acl</name>
    <value>*</value>
    <description>ACL for NamenodeProtocol, the protocol used by the secondary
    namenode to communicate with the namenode.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

 <property>
    <name>security.admin.operations.protocol.acl</name>
    <value>*</value>
    <description>ACL for AdminOperationsProtocol. Used for admin commands.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.refresh.user.mappings.protocol.acl</name>
    <value>*</value>
    <description>ACL for RefreshUserMappingsProtocol. Used to refresh
    users mappings. The ACL is a comma-separated list of user and
    group names. The user and group list is separated by a blank. For
    e.g. "alice,bob users,wheel".  A special value of "*" means all
    users are allowed.</description>
  </property>

  <property>
    <name>security.refresh.policy.protocol.acl</name>
    <value>*</value>
    <description>ACL for RefreshAuthorizationPolicyProtocol, used by the
    dfsadmin and mradmin commands to refresh the security policy in-effect.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.ha.service.protocol.acl</name>
    <value>*</value>
    <description>ACL for HAService protocol used by HAAdmin to manage the
      active and stand-by states of namenode.</description>
  </property>

  <property>
    <name>security.zkfc.protocol.acl</name>
    <value>*</value>
    <description>ACL for access to the ZK Failover Controller
    </description>
  </property>

  <property>
    <name>security.qjournal.service.protocol.acl</name>
    <value>*</value>
    <description>ACL for QJournalProtocol, used by the NN to communicate with
    JNs when using the QuorumJournalManager for edit logs.</description>
  </property>

  <property>
    <name>security.mrhs.client.protocol.acl</name>
    <value>*</value>
    <description>ACL for HSClientProtocol, used by job clients to
    communciate with the MR History Server job status etc. 
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <!-- YARN Protocols -->

  <property>
    <name>security.resourcetracker.protocol.acl</name>
    <value>*</value>
    <description>ACL for ResourceTrackerProtocol, used by the
    ResourceManager and NodeManager to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.resourcemanager-administration.protocol.acl</name>
    <value>*</value>
    <description>ACL for ResourceManagerAdministrationProtocol, for admin commands. 
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.applicationclient.protocol.acl</name>
    <value>*</value>
    <description>ACL for ApplicationClientProtocol, used by the ResourceManager 
    and applications submission clients to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.applicationmaster.protocol.acl</name>
    <value>*</value>
    <description>ACL for ApplicationMasterProtocol, used by the ResourceManager 
    and ApplicationMasters to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.containermanagement.protocol.acl</name>
    <value>*</value>
    <description>ACL for ContainerManagementProtocol protocol, used by the NodeManager 
    and ApplicationMasters to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.resourcelocalizer.protocol.acl</name>
    <value>*</value>
    <description>ACL for ResourceLocalizer protocol, used by the NodeManager 
    and ResourceLocalizer to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.job.task.protocol.acl</name>
    <value>*</value>
    <description>ACL for TaskUmbilicalProtocol, used by the map and reduce
    tasks to communicate with the parent tasktracker.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.job.client.protocol.acl</name>
    <value>*</value>
    <description>ACL for MRClientProtocol, used by job clients to
    communciate with the MR ApplicationMaster to query job status etc. 
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.applicationhistory.protocol.acl</name>
    <value>*</value>
    <description>ACL for ApplicationHistoryProtocol, used by the timeline
    server and the generic history service client to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>
</configuration>

注意

title	注意事项

hadoop-policy.xml配置文件中，security.client.protocol.hosts，security.client.datanode.protocol.hosts 这两个配置项的值，要改成实际部署环境的IP地址

5、修改slaves配置文件

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/etc/hadoop
vi slaves

把所有的datanode的主机名添加到文件中，例如：

代码块

language	bash
linenumbers	true

smartbi-spark
smartbi-spark2
smartbi-spark3

分发Hadoop安装包（管理节点执行）

将hadoop管理节点上的hadoop安装包分发到其他节点：

代码块

language	bash
linenumbers	true

scp -r /opt/hadoop-2.7.3 root@hadoop2:/opt
scp -r /opt/hadoop-2.7.3 root@hadoop3:/opt

配置Hadoop环境变量（每个节点执行）

添加环境变量。

代码块

language	bash
linenumbers	true

vi ~/.bash_profile

在最底下添加下面内容：

代码块

language	bash
linenumbers	true

export HADOOP_HOME=/opt/hadoop-2.7.3
export PATH=$PATH:$HADOOP_HOME/bin

让配置生效

代码块

language	bash
linenumbers	true

source ~/.bash_profile

启动Hadoop集群

①格式化hadoop（管理节点执行）

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/
./bin/hdfs namenode -format

注意

title	注意事项

仅第一次启动时需要执行格式化Hadoop操作，后续启动无需进行此操作

②启动hadoop

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/
./sbin/start-dfs.sh

③创建中间数据存储目录(数据挖掘实验引擎集群需要使用)

代码块

language	bash
linenumbers	true

hdfs dfs -mkdir /mining
hdfs dfs -chown mining:mining /mining

验证Hadoop集群

①在浏览器输入: http://本机ip:50070/dfshealth.html#tab-overview, 检查集群状态

Image Added

②检查mining目录是否创建成功

代码块

language	bash
linenumbers	true

hdfs dfs -ls /    #显示创建的/mining即表示创建成功

如上显示，表示Hadoop安装成功。

运维操作

停止hadoop(管理节点执行)

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/
./sbin/stop-dfs.sh

启动hadoop(管理节点执行)

代码块

language	bash
linenumbers	true

vi /etc/profile

cd /opt/hadoop-2.7.3/
./sbin/start-dfs.sh

设置实验引擎练级中间数据存储URL

管理员登陆Smartbi

系统运维--系统选项--执行引擎–引擎配置

Image Added

注意

title	注意事项

如果是Hadoop集群，上图中节点数据hdfs存储目录需要填写Hadoop管理节点的IP

Python执行节点更新

停止Python服务

进入安装Python计算节点的服务器，进入目录，停止python服务

代码块

language	bash
linenumbers	true

export

cd

HADOOP_HOME=

/opt/

hadoop-2.7.3 export PATH=$PATH:$HADOOP_HOME/bin让配置生效①格式化hadoop

smartbi-mining-engine-bin/engine/sbin
./python-daemon.sh stop

注意

title	注意事项

注意，如果出现无法停止情况，可以通过jps查看python服务进程id，然后 kill -9 进程id

更新引擎包

更新方式，参考实验引擎、服务引擎的更新方式，如果python执行节点跟实验引擎在同台机器，这步骤可以省略

创建执行代理程序启动用户

创建mining用户组组

代码块

language	bash
linenumbers	true

source /etc/profile

启动Hadoop

groupadd mining

创建启动用户(mining-ag)并指定用户组为mining

代码块

language	bash
linenumbers	true

cd

useradd

/opt/hadoop-2.7.3/ ./bin/hdfs namenode -format

注意

title	注意事项

仅第一次启动时需要执行格式化Hadoop操作，后续启动无需进行此操作

②启动hadoop

-g mining mining-ag

设置用户密码

代码块

linenumbers
language	bash

true

cd /opt/hadoop-2.7.3/
./sbin/start-dfs.sh

③创建中间数据存储目录(数据挖掘实验引擎集群需要使用

true

passwd mining-ag

给引擎安装目录附权限(为了使用mining-ag用户启动执行代理程序时候，有权限创建agent-data跟agent-logs目录)

代码块

language	bash
linenumbers	true

hdfs

chgrp

dfs

mining

-mkdir /sbimining hdfs dfs -chown mining:mining /mining

5.验证Hadoop

①在浏览器输入: http://本机ip:50070/dfshealth.html#tab-overview, 检查集群状态

Image Removed

②检查sbimining目录是否创建成功

Image Removed

如上两图显示，表示Hadoop安装成功。

运维操作

停止hadoop

/opt/smartbi-mining-engine-bin

chmod 775 /opt/smartbi-mining-engine-bin

启动Python执行代理

管理员登陆Smartbi

系统运维--引擎设置

Image Added

登陆到部署Python节点服务器，并切换到mining-ag用户

注意

title	注意事项

为了避免出现安全问题，一定要切换到mining-ag用户去启动执行代理服务，不要使用安装用户或带有sudu权限的用户来启动执行代理服务

代码块

language	bash
linenumbers	true

su - mining-ag

切换到引擎启动目录

代码块

language	bash
linenumbers	true

cd /opt/

hadoop-2.7.3/ ./sbin/stop-dfs.sh启动hadooptoc

smartbi-mining-engine-bin/engine/sbin

把拷贝命令粘贴，并执行，例如：

代码块

language	bash
linenumbers	true

cd /opt/hadoop-2.7.3/
./sbin/start-dfs.sh

./agent-daemon.sh start --master http://smartbi-engine:8899 --env python

等待启动成功即可。

页面树结构

页面历史

版本比较

旧版本 4

新版本 当前版本

标识

数据挖掘引擎包更新

1、系统环境设置

1.开放防火墙端口

Spark版本升级

1、停止spark

2、安装spark3.0

3、配置spark3.0

2.取消打开文件限制

4.启动Spark3.0

2、Hadoop单节点安装

1.设置免密登陆

2.安装Hadoop

创建hadoop相关目录

5.检查Spark

解压hadoop安装包到指定目录

修改Hadoop配置文件

安装Hadoop组件

系统环境设置

1.开放防火墙端口

2.取消打开文件限制

Hadoop单节点安装

1.设置免密登陆

2.安装Hadoop

创建hadoop相关目录

解压hadoop安装包到指定目录

修改Hadoop配置文件

配置Hadoop环境变量

配置Hadoop环境变量

3.启动Hadoop

4.验证Hadoop

5.运维操作

Hadoop集群安装

1.设置免密登陆（集群服务器都执行）

创建hadoop相关目录（集群服务器都执行）

创建临时目录

创建namenode数据目录

创建datanode目录

在管理节点安装配置Hadoop

分发Hadoop安装包（管理节点执行）

配置Hadoop环境变量（每个节点执行）

启动Hadoop集群

验证Hadoop集群

运维操作

设置实验引擎练级中间数据存储URL

Python执行节点更新

停止Python服务

更新引擎包

创建执行代理程序启动用户

启动Hadoop

5.验证Hadoop

运维操作

启动Python执行代理

新版本当前版本