| 大数据

CDH 5.13 搭建

记录一次 Cloudera’s Distribution Including Apache Hadoop平台搭建过程。
基本模拟生产环境用于日常测试

准备工作

环境实施

  • 虚拟机搭建
      vmware新建虚拟机比较简单步骤省略。这部分重点记录下网络环境配置信息，因为本地局域网IP不固定，所以我们采用NAT模式配置虚拟机网络 （采用桥接模式直接连接，因为我们IP必须固定，可能导致和局域网其它服务器IP冲突等异常），NAT模式配置如下：
    • 网络规划 ：网段（192.168.64.0） 、网关（192.168.64.1）、3台虚拟机IP (192.168.64.101/102/103)
    • 修改物理机vmnet8配置： ip: 192.168.64.2、掩码:255.255.255.0、网关:192.168.64.1
    • 修改vmware网络：编辑 -》虚拟网络编辑-》更改设置-》vmnet8 -》子网IP: 192.168.64.0 、子网掩码：255.255.255.0 -》NAT设置-》网关IP：192.168.64.1
    • 修改虚拟机网络：
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      vi /etc/sysconfig/network-scripts/ifcfg-ens33
      ##vi begin
      TYPE=Ethernet
      PROXY_METHOD=none
      BROWSER_ONLY=no
      BOOTPROTO=static
      DEFROUTE=yes
      IPV4_FAILURE_FATAL=no
      IPV6INIT=yes
      IPV6_AUTOCONF=yes
      IPV6_DEFROUTE=yes
      IPV6_FAILURE_FATAL=no
      IPV6_ADDR_GEN_MODE=stable-privacy
      NAME=ens33
      UUID=ba0197c3-ef17-41d8-b41c-fbfd12c30b4f
      DEVICE=ens33
      ONBOOT=yes

      IPADDR=192.168.64.101
      PREFIX=24
      GATEWAY=192.168.64.1
      DNS1=114.114.114.114
      ##vi end
      service network restart
    • 修改虚拟机hostname、 /etc/hosts 、修改selinux、关闭防火墙
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      hostnamectl set-hostname cdh01
      hostnamectl set-hostname cdh02
      hostnamectl set-hostname cdh03
      vi /etc/hosts
      ##/etc/hosts
      cdh01 192.168.64.101
      cdh02 192.168.64.102
      cdh03 192.168.64.103
      ##selinux
      vi /etc/selinux/config
      SELINUX=disabled
      ##firewalld
      systemctl stop firewalld
      systemctl disable firewalld
      systemctl status firewalld
    • ssh互信 (把3台机器的id_rsa.pub放一起，分发到3台机器上)
      1
      2
      3
      4
      ssh-keygen -t rsa
      cd /root/.ssh
      cat id_rsa.pub >> authorized_keys
      scp  authorized_keys cdh02:/root/.ssh
    • 配置ntp
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      ##安装ntp服务（所有节点）
      yum install -y ntp 
      ##主节点(cdh01)配置
      vi /etc/ntp.conf 

      # the administrative functions.
      restrict 127.0.0.1 
      restrict ::1
      ​
      # Hosts on local network are less restricted.
      #restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
      ​
      # Use public servers from the pool.ntp.org project.
      # Please consider joining the pool (http://www.pool.ntp.org/join.html).
      # server 0.centos.pool.ntp.org iburst
      # server 1.centos.pool.ntp.org iburst
      # server 2.centos.pool.ntp.org iburst
      # server 3.centos.pool.ntp.org iburst
      # 192.168.64.101 作为server
      server 127.127.1.0
      Fudge 127.127.1.0 stratum 10
      ​
      ##其它节点
      vi /etc/ntp.conf
      # the administrative functions.
      restrict 192.168.64.101 nomodify notrap nopeer noquery
      restrict 127.0.0.1
      restrict ::1
      # Hosts on local network are less restricted.
      #restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
      ​
      # Use public servers from the pool.ntp.org project.
      # Please consider joining the pool (http://www.pool.ntp.org/join.html).
      # server 0.centos.pool.ntp.org iburst
      # server 1.centos.pool.ntp.org iburst
      # server 2.centos.pool.ntp.org iburst
      # server 3.centos.pool.ntp.org iburst
      # 172.28.52.66-172.18.12.69 作为client
      server 192.168.64.101
      Fudge 192.168.64.101 stratum 10
      ##启动ntp服务
      systemctl disable chronyd.service
      systemctl enable ntpd.service
      systemctl start ntpd 
      ntpstat 
      #ntpdate -u  ntp.aliyun.com
  • 安装jdk 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    ##卸载原有JDK，如果有的话
    rpm -qa | grep jdk 
    yum remove -y *jdk*
    yum remove -y *java*
     tar xzvf jdk-8u144-linux-x64.tar.gz
    mv jdk1.8.0_144 /usr/java
    vi  /etc/profile
    export JAVA_HOME=/usr/java/jdk1.8.0_144
    export PATH=$JAVA_HOME/bin:$PATH
    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
    source /etc/profile
    java -version
  • 安装mysql （主节点）
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    yum install net-tools #安装net-tools包，在之前配置网络的时候也需要安装
    yum install perl #安装perl包
    tar -xf mysql-5.7.21-1.el7.x86_64.rpm-bundle.tar
    ##按顺序安装下面rpm
    rpm -ivh mysql-community-common-5.7.21-1.el7.x86_64.rpm
    rpm -ivh mysql-community-libs-5.7.21-1.el7.x86_64.rpm
    rpm -ivh mysql-community-client-5.7.21-1.el7.x86_64.rpm
    rpm -ivh mysql-community-server-5.7.21-1.el7.x86_64.rpm
    rpm -ivh mysql-community-libs-compat-5.7.21-1.el7.x86_64.rpm
    rpm -ivh mysql-community-devel-5.7.21-1.el7.x86_64.rpm
    systemctl start mysqld
    systemctl enable mysqld
    systemctl daemon-reload
    #查看原始root密码
    cat /var/log/mysqld.log |grep password 
    mysql -uroot –p
    set global validate_password_policy=0;
    set global validate_password_length=2;
    set global validate_password_special_char_count=0;
    set global validate_password_mixed_case_count=0;
    set global validate_password_number_count=0;
    set password = password('123');
    GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123' WITH GRANT OPTION;
    flush privileges;
    ##执行下面的建库脚本
    create database metastore default character set utf8;
    CREATE USER 'hive'@'%' IDENTIFIED BY 'hive';
    GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';
    create database hive default character set utf8;
    GRANT ALL PRIVILEGES ON hive. * TO 'hive'@'%';
    create database cm default character set utf8;
    CREATE USER 'cm'@'%' IDENTIFIED BY 'cm';
    GRANT ALL PRIVILEGES ON cm.* TO 'cm'@'%';
    create database am default character set utf8;
    CREATE USER 'am'@'%' IDENTIFIED BY 'am';
    GRANT ALL PRIVILEGES ON am. * TO 'am'@'%';
    create database rm default character set utf8;
    CREATE USER 'rm'@'%' IDENTIFIED BY 'rm';
    GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';
    create database hue default character set utf8;
    CREATE USER 'hue'@'%' IDENTIFIED BY 'hue';
    GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%';
    create database oozie default character set utf8;
    CREATE USER 'oozie'@'%' IDENTIFIED BY 'oozie';
    GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%';
    grant all privileges on *.* to 'root'@'cdh01' identified by '123' with grant option;
    flush privileges;
  • 安装cm
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    tar zxvf cloudera-manager-centos7-cm5.13.0_x86_64.tar.gz
    tar zxvf  mysql-connector-java-5.1.47.tar.gz
    cp /opt/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /usr/java/
    cp /opt/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /opt/cm-5.13.0/share/cmf/lib/
    ##主节点执行
    /opt/cm-5.13.0/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p123 scm scm scm
    vi /opt/cm-5.13.0/etc/cloudera-scm-agent/config.ini
    server_host=cdh01
    scp -r /opt/cm-5.13.0/ chd02:/opt/
    scp -r /opt/cm-5.13.0/ chd03:/opt/
    ##所有节点执行
    useradd --system --home=/opt/cm-5.13.0/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
    ##主节点 /opt/cloudera/parcel-repo/目录下拷贝下面文件
    CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha1
    manifest.json
    CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel
    ##修改sha文件名
    mv  CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha1 CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha
    ##主节点启动 server
    /opt/cm-5.13.0/etc/init.d/cloudera-scm-server start
    ##主节点及其它节点启动 agent
    /opt/cm-5.13.0/etc/init.d/cloudera-scm-agent start
    ## 启动需要点时间，可以查看数据库scm中的hosts表，如果里面出现节点信息表示server启动完成
  • Hadoop集群安装
    • 访问浏览器地址: http://cdh01:7180 (admin/admin)
    • 选择 Cloudera Express版本
    • 选择主机(cdh01、cdh02、chd03)
    • 选择集群版本 （cdh-5.13）
    • 服务器检测，遇到警告执行 ：
      1
      2
      3
      echo 10 > /proc/sys/vm/swappiness 
      echo never >/sys/kernel/mm/transparent_hugepage/defrag 
      echo never > /sys/kernel/mm/transparent_hugepage/enabled
  • 自定义服务安装 参考生产环境
    • HDFS
    • HIVE
    • HUE
    • IMPALA
    • OOZIE
    • SPARK
    • SQOOP
    • YARN
    • ZOOKPPER
  • 集群数据库设置
    • 数据库主机：cdh01
    • 数据库类型：MySql
    • 数据库名称分别对应:hive 、oozie、hue等，hue测试连接可能会有ERROR，处理方法见后面。

其它配置及问题汇总

  • 扩展虚拟机目录
    1
    2
    3
    4
    5
    6
    7
    vmware虚拟机设置-》扩展
    fdisk /dev/sda 
    ##分区之后可能需要重启才能在/dev下看到
    pvcreate /dev/sda3
    vgextend /dev/sda3 centos
    lvextend -L +19G /dev/centos/root
    xfs_growfs /dev/centos/root
  • 修改本地yum源
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    挂载光盘(本例使用)
    mount -t iso9660 /dev/cdrom /mnt/cdrom
    挂载镜像
    mount -o loop /tmp/linux.iso  /mnt/linux
    挂载软盘
    mount /dev/fd0 /mnt/floppy
    vi /etc/yum.repos.d/CentOS-Base.repo
    baseurl=file:///mnt/cdrom
    gpgcheck=0
    enabled=1
  • ERROR1 Error: Rpmdb checksum is invalid: pkg checksums
    1
    2
    3
    yum clean all
    yum makecache
    yum list installed
  • ERROR2 /opt/cm-5.16.2/etc/init.d/cloudera-scm-agent:行109: pstree: 未找到命令
    1
    yum install -y psmisc
  • ERROR3 + echo ‘Failed to find the Apache HTTPD executable.’
    1
    yum -y install httpd
  • ERROR4 Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs found
    1
    yum install cyrus-sasl-plain  cyrus-sasl-devel  cyrus-sasl-gssapi
  • ERROR5 /opt/cm-5.11.0/lib64/cmf/service/client/deploy-cc.sh: line 115: perl: command not found
    1
    yum install -y perl
ESC