hadoop2.4.0 ha 搭建

php中文网

发布时间：2016-06-07 15:43:59

1316人浏览过

来源于php中文网

原创

问题导读：
1、hadoop ha是通过什么配置实现自动切换的？
2、配置中mapred与mapreduce的区别是什么？
3、hadoop ha两个namenode之间的关系是什么？

-- hadoop 版本：2.4.0
-- 安装包名：
hadoop-2.4.0.tar.gz 或者源码版本 hadoop-2.4.0-src.tar.gz(我hadoop、hbase、hive均是用的源码编译安装)

-- 安装参考：
http://www.netfoucs.com/article/book_mmicky/79985.html
http://www.byywee.com/page/M0/S934/934356.html
http://www.itpub.net/thread-1631536-1-1.html
http://demo.netfoucs.com/u014393917/article/details/25913363
http://www.aboutyun.com/thread-8294-1-1.html

-- 找不到本地库
参考：http://www.ercoppa.org/Linux-Com ... -hadoop-library.htm

-- lzo支持，
参考：http://blog.csdn.net/zhangzhaokun/article/details/17595325
http://slaytanic.blog.51cto.com/2057708/1162287/
http://hi.baidu.com/qingchunranzhi/item/3662ed5ed29d37a1adc85709

-- 安装以下RPM包：
yum -y install openssh*
yum -y install man*
yum -y install compat-libstdc++-33*
yum -y install libaio-0.*
yum -y install libaio-devel*
yum -y install sysstat-9.*
yum -y install glibc-2.*
yum -y install glibc-devel-2.* glibc-headers-2.*
yum -y install ksh-2*
yum -y install libgcc-4.*
yum -y install libstdc++-4.*
yum -y install libstdc++-4.*.i686*
yum -y install libstdc++-devel-4.*
yum -y install gcc-4.*x86_64*
yum -y install gcc-c++-4.*x86_64*
yum -y install elfutils-libelf-0*x86_64* elfutils-libelf-devel-0*x86_64*
yum -y install elfutils-libelf-0*i686* elfutils-libelf-devel-0*i686*
yum -y install libtool-ltdl*i686*
yum -y install ncurses*i686*
yum -y install ncurses*
yum -y install readline*
yum -y install unixODBC*
yum -y install zlib
yum -y install zlib*
yum -y install openssl*
yum -y install patch
yum -y install git
yum -y -y install lzo-devel zlib-devel gcc autoconf automake libtool
yum -y install lzop
yum -y install lrzsz
yum -y -y install lzo-devel zlib-devel gcc autoconf automake libtool
yum -y install nc
yum -y install glibc
yum -y install java-1.7.0-openjdk
yum -y install gzip
yum -y install zlib
yum -y install gcc
yum -y install gcc-c++
yum -y install make
yum -y install protobuf
yum -y install protoc
yum -y install cmake
yum -y install openssl-devel
yum -y install ncurses-devel
yum -y install unzip
yum -y install telnet
yum -y install telnet-server
yum -y install wget
yum -y install svn
yum -y install ntpdate

-- hive 安装，参考：http://kicklinux.com/hive-deploy/

5台服务器设计图

IP地址	主机名	NameNode	JournalNode	DataNode	Zookeeper	Hbase	Hive
192.168.117.194	funshion-hadoop194	是	是	否	是	是	否
192.168.117.195	funshion-hadoop195	是	是	否	是	是	否
192.168.117.196	funshion-hadoop196	否	是	是	是	是(Master)	是(Mysql)
192.168.117.197	funshion-hadoop197	否	是	是	是	是	否
192.168.117.198	funshion-hadoop198	否	是	是	是	是	否

-- 配置Linux、安装JDK

--参考：linux(ubuntu)安装Java jdk环境变量设置及小程序测试

-- Step 1. 建立用户hadoop的ssh无密码登陆

--参考：
linux（ubuntu）无密码互通、相互登录高可靠文档
CentOS6.4之图解SSH无验证双向登陆配置

-- Step 2. zookeeper配置(配置奇数台zk集群，我用的5台)
-- 参考：Zookeeper集群环境安装过程详解

-- Step 3. Hadoop集群配置：

-- Step 3.1 vi $HADOOP_HOME/etc/hadoop/slaves

funshion-hadoop196
funshion-hadoop197
funshion-hadoop198

-- Step 3.2 vi $HADOOP_HOME/etc/hadoop/hadoop-env.sh (添加 JAVA_HOME 环境变量、本地library库)

export JAVA_HOME=/usr/java/latest
export LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib/native"

-- 注意：${HADOOP_PREFIX}/lib/native 下的内容如下：
[hadoop@funshion-hadoop194 native]$ pwd
/usr/local/hadoop/lib/native

[hadoop@funshion-hadoop194 native]$ ls -l
total 8640
-rw-r--r--. 1 hadoop hadoop 2850660 Jun 9 14:58 hadoop-common-2.4.0.jar
-rw-r--r--. 1 hadoop hadoop 1509888 Jun 9 14:58 hadoop-common-2.4.0-tests.jar
-rw-r--r--. 1 hadoop hadoop 178637 Jun 9 14:58 hadoop-lzo-0.4.20-SNAPSHOT.jar
-rw-r--r--. 1 hadoop hadoop 145385 Jun 9 14:58 hadoop-nfs-2.4.0.jar
-rw-r--r--. 1 hadoop hadoop 983042 Jun 6 19:36 libhadoop.a
-rw-r--r--. 1 hadoop hadoop 1487284 Jun 6 19:36 libhadooppipes.a
lrwxrwxrwx. 1 hadoop hadoop 18 Jun 6 19:42 libhadoop.so -> libhadoop.so.1.0.0
-rwxr-xr-x. 1 hadoop hadoop 586664 Jun 6 19:36 libhadoop.so.1.0.0
-rw-r--r--. 1 hadoop hadoop 582040 Jun 6 19:36 libhadooputils.a
-rw-r--r--. 1 hadoop hadoop 298178 Jun 6 19:36 libhdfs.a
lrwxrwxrwx. 1 hadoop hadoop 16 Jun 6 19:42 libhdfs.so -> libhdfs.so.0.0.0
-rwxr-xr-x. 1 hadoop hadoop 200026 Jun 6 19:36 libhdfs.so.0.0.0
drwxrwxr-x. 2 hadoop hadoop 4096 Jun 6 20:37 Linux-amd64-64

-- Step 3.5 vi $HADOOP_HOME/etc/hadoop/mapred-site.xml

mapreduce.jobhistory.address
funshion-hadoop194:10020

mapreduce.jobhistory.webapp.address
funshion-hadoop194:19888

mapreduce.map.output.compress
true

mapreduce.map.output.compress.codec
com.hadoop.compression.lzo.LzoCodec

mapred.child.env
LD_LIBRARY_PATH=/usr/local/hadoop/lib/native

mapred.child.java.opts
-Xmx2048m

mapred.reduce.child.java.opts
-Xmx2048m

mapred.map.child.java.opts
-Xmx2048m

mapred.remote.os
Linux
Remote MapReduce framework's OS, can be either Linux or Windows

-- 注意：1、以mapred.开头的形式去指定属性名，都是一种过时的形式，建议使用mapreduce.
比如：mapred.compress.map.output 属性应该对应修改成：mapreduce.map.output.compress
具体可以查阅：http://hadoop.apache.org/docs/r2 ... /mapred-default.xml 文件，
　　　　　　当然，好像还有少量属性名是没有修改的，比如：mapred.child.java.opts、mapred.child.env

-- 注意：/usr/local/hadoop/lib/native 目录下有如下内容：
[hadoop@funshion-hadoop194 sbin]$ ls -l /usr/local/hadoop/lib/native
total 12732
-rw-r--r-- 1 hadoop hadoop 2850900 Jun 20 19:22 hadoop-common-2.4.0.jar
-rw-r--r-- 1 hadoop hadoop 1509411 Jun 20 19:22 hadoop-common-2.4.0-tests.jar
-rw-r--r-- 1 hadoop hadoop 178559 Jun 20 18:38 hadoop-lzo-0.4.20-SNAPSHOT.jar
-rw-r--r-- 1 hadoop hadoop 1407039 Jun 20 19:25 hadoop-yarn-common-2.4.0.jar
-rw-r--r-- 1 hadoop hadoop 106198 Jun 20 18:37 libgplcompression.a
-rw-r--r-- 1 hadoop hadoop 1124124 Jun 20 18:37 libgplcompression.la
-rwxr-xr-x 1 hadoop hadoop 69347 Jun 20 18:37 libgplcompression.so
-rwxr-xr-x 1 hadoop hadoop 69347 Jun 20 18:37 libgplcompression.so.0
-rwxr-xr-x 1 hadoop hadoop 69347 Jun 20 18:37 libgplcompression.so.0.0.0
-rw-r--r-- 1 hadoop hadoop 983042 Jun 20 18:10 libhadoop.a
-rw-r--r-- 1 hadoop hadoop 1487284 Jun 20 18:10 libhadooppipes.a
lrwxrwxrwx 1 hadoop hadoop 18 Jun 20 18:27 libhadoop.so -> libhadoop.so.1.0.0
-rwxr-xr-x 1 hadoop hadoop 586664 Jun 20 18:10 libhadoop.so.1.0.0
-rw-r--r-- 1 hadoop hadoop 582040 Jun 20 18:10 libhadooputils.a
-rw-r--r-- 1 hadoop hadoop 298178 Jun 20 18:10 libhdfs.a
lrwxrwxrwx 1 hadoop hadoop 16 Jun 20 18:27 libhdfs.so -> libhdfs.so.0.0.0
-rwxr-xr-x 1 hadoop hadoop 200026 Jun 20 18:10 libhdfs.so.0.0.0
-rw-r--r-- 1 hadoop hadoop 906318 Jun 20 19:17 liblzo2.a
-rwxr-xr-x 1 hadoop hadoop 929 Jun 20 19:17 liblzo2.la
-rwxr-xr-x 1 hadoop hadoop 562376 Jun 20 19:17 liblzo2.so
-rwxr-xr-x 1 hadoop hadoop 562376 Jun 20 19:17 liblzo2.so.2
-rwxr-xr-x 1 hadoop hadoop 562376 Jun 20 19:17 liblzo2.so.2.0.0

-- Step 3.6 vi $HADOOP_HOME/etc/hadoop/yarn-site.xml

yarn.resourcemanager.connect.retry-interval.ms
60000

yarn.resourcemanager.ha.enabled
true

yarn.resourcemanager.cluster-id
rm-cluster

yarn.resourcemanager.ha.rm-ids
rm1,rm2

yarn.resourcemanager.ha.id
rm1

yarn.resourcemanager.hostname.rm1
funshion-hadoop194

yarn.resourcemanager.hostname.rm2
funshion-hadoop195

yarn.resourcemanager.recovery.enabled
true

yarn.resourcemanager.store.class
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore

yarn.resourcemanager.zk-address
funshion-hadoop194:2181,funshion-hadoop195:2181,funshion-hadoop196:2181,funshion-hadoop197:2181,funshion-hadoop198:2181

yarn.resourcemanager.address.rm1
${yarn.resourcemanager.hostname.rm1}:23140

yarn.resourcemanager.scheduler.address.rm1
${yarn.resourcemanager.hostname.rm1}:23130

yarn.resourcemanager.webapp.https.address.rm1
${yarn.resourcemanager.hostname.rm1}:23189

yarn.resourcemanager.webapp.address.rm1
${yarn.resourcemanager.hostname.rm1}:23188

yarn.resourcemanager.resource-tracker.address.rm1
${yarn.resourcemanager.hostname.rm1}:23125

yarn.resourcemanager.admin.address.rm1
${yarn.resourcemanager.hostname.rm1}:23141

yarn.resourcemanager.address.rm2
${yarn.resourcemanager.hostname.rm2}:23140

yarn.resourcemanager.scheduler.address.rm2
${yarn.resourcemanager.hostname.rm2}:23130

yarn.resourcemanager.webapp.https.address.rm2
${yarn.resourcemanager.hostname.rm2}:23189

yarn.resourcemanager.webapp.address.rm2
${yarn.resourcemanager.hostname.rm2}:23188

yarn.resourcemanager.resource-tracker.address.rm2
${yarn.resourcemanager.hostname.rm2}:23125

yarn.resourcemanager.admin.address.rm2
${yarn.resourcemanager.hostname.rm2}:23141

yarn.resourcemanager.scheduler.class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

yarn.scheduler.fair.allocation.file
${yarn.home.dir}/etc/hadoop/fairscheduler.xml

yarn.nodemanager.local-dirs
/home/hadoop/logs/yarn_local

yarn.nodemanager.log-dirs
/home/hadoop/logs/yarn_log

yarn.nodemanager.remote-app-log-dir
/home/hadoop/logs/yarn_remotelog

yarn.app.mapreduce.am.staging-dir
/home/hadoop/logs/yarn_userstag

mapreduce.jobhistory.intermediate-done-dir
/home/hadoop/logs/yarn_intermediatedone

mapreduce.jobhistory.done-dir
/var/lib/hadoop/dfs/yarn_done

yarn.log-aggregation-enable
true

yarn.nodemanager.resource.memory-mb
2048

yarn.nodemanager.vmem-pmem-ratio
4.2

yarn.nodemanager.resource.cpu-vcores
2

yarn.nodemanager.aux-services
mapreduce_shuffle

yarn.nodemanager.aux-services.mapreduce.shuffle.class
org.apache.hadoop.mapred.ShuffleHandler

Classpath for typical applications.
yarn.application.classpath

$HADOOP_HOME/etc/hadoop,
$HADOOP_HOME/share/hadoop/common/*,
$HADOOP_HOME/share/hadoop/common/lib/*,
$HADOOP_HOME/share/hadoop/hdfs/*,
$HADOOP_HOME/share/hadoop/hdfs/lib/*,
$HADOOP_HOME/share/hadoop/mapreduce/*,
$HADOOP_HOME/share/hadoop/mapreduce/lib/*,
$HADOOP_HOME/share/hadoop/yarn/*,
$HADOOP_HOME/share/hadoop/yarn/lib/*

-- 注意：两个namenode，funshion-hadoop194直接用上面的配置，
-- funshion-hadoop195的话，只需修改一个地方：修改yarn.resourcemanager.ha.id 属性值为 rm2

-- Step 3.7 vi $HADOOP_HOME/etc/hadoop/fairscheduler.xml

1024 mb, 1 vcores
1536 mb, 1 vcores
5
300
1.0
root,yarn,search,hdfs

1024 mb, 1 vcores
1536 mb, 1 vcores

1024 mb, 1 vcores
1536 mb, 1 vcores

##################################################################################

scp -r /usr/local/hadoop/etc/hadoop/* hadoop@funshion-hadoop195:/usr/local/hadoop/etc/hadoop/
scp -r /usr/local/hadoop/etc/hadoop/* hadoop@funshion-hadoop196:/usr/local/hadoop/etc/hadoop/
scp -r /usr/local/hadoop/etc/hadoop/* hadoop@funshion-hadoop197:/usr/local/hadoop/etc/hadoop/
scp -r /usr/local/hadoop/etc/hadoop/* hadoop@funshion-hadoop198:/usr/local/hadoop/etc/hadoop/

-- Step 4. 创建相关目录

mkdir ~/logs
mkdir ~/mydata

-- 备注：mydate目录下的相关子目录会自动生成，不需要创建。
-- 在每台集群机器上创建如上两个目录，并同步 $HADOOP_HOME/etc/hadoop目录下的所有文件到各节点

-- Step 5. 启动Zookeeper、JournalNode、格式化Hadoop集群并启动

-- Step 5.1 启动Zooker (ZK集群是funshion-hadoop194、funshion-hadoop195、funshion-hadoop196、funshion-hadoop197、funshion-hadoop198 五台服务器)

[hadoop@funshion-hadoop194 bin]$ /usr/local/zookeeper/bin/zkServer.sh start
[hadoop@funshion-hadoop195 bin]$ /usr/local/zookeeper/bin/zkServer.sh start
[hadoop@funshion-hadoop196 bin]$ /usr/local/zookeeper/bin/zkServer.sh start
[hadoop@funshion-hadoop197 bin]$ /usr/local/zookeeper/bin/zkServer.sh start
[hadoop@funshion-hadoop198 bin]$ /usr/local/zookeeper/bin/zkServer.sh start

-- 可以如下查看Zookeeper集群各节点的状态：
/usr/local/zookeeper/bin/zkServer.sh status

-- 然后在某一个namenode节点执行如下命令，创建命名空间
[hadoop@funshion-hadoop194 bin]$ cd $HADOOP_HOME
[hadoop@funshion-hadoop194 hadoop]$ ./bin/hdfs zkfc -formatZK

-- 备注：停止zookeeper相关命令类似如下：
/usr/local/zookeeper/bin/zkServer.sh stop
/usr/local/zookeeper/bin/zkServer.sh restart

-- Step 5.2 启动JournalNode进程(在funshion-hadoop194、funshion-hadoop195、funshion-hadoop196、funshion-hadoop197、funshion-hadoop198五台服务器上分别执行)：

[hadoop@funshion-hadoop194 bin]$ $HADOOP_HOME/sbin/hadoop-daemon.sh start journalnode
[hadoop@funshion-hadoop195 bin]$ $HADOOP_HOME/sbin/hadoop-daemon.sh start journalnode
[hadoop@funshion-hadoop196 bin]$ $HADOOP_HOME/sbin/hadoop-daemon.sh start journalnode
[hadoop@funshion-hadoop197 bin]$ $HADOOP_HOME/sbin/hadoop-daemon.sh start journalnode
[hadoop@funshion-hadoop198 bin]$ $HADOOP_HOME/sbin/hadoop-daemon.sh start journalnode

-- Step 5.3 格式化Hadoop集群并启动：

-- 在 funshion-hadoop194 上执行：
[hadoop@funshion-hadoop194 bin]$ $HADOOP_HOME/bin/hdfs namenode -format mycluster
[hadoop@funshion-hadoop194 bin]$ $HADOOP_HOME/sbin/hadoop-daemon.sh start namenode

-- 上步执行完后，在 funshion-hadoop195 上执行：
[hadoop@funshion-hadoop195 bin]$ $HADOOP_HOME/bin/hdfs namenode -bootstrapStandby
[hadoop@funshion-hadoop195 bin]$ $HADOOP_HOME/sbin/hadoop-daemon.sh start namenode

-- 上步执行完后，可以继续在某个 namenode 执行 $HADOOP_HOME/sbin/start-all.sh 启动datanode及yarn相关进程。

-- 因为是配置的自动故障转移，所以不能手工切换namenode的active和stadby角色。

-- 可以通过haadmin查看每个Service的角色状态：
[hadoop@funshion-hadoop194 lab]$ $HADOOP_HOME/bin/hdfs haadmin -getServiceState nn1
standby
[hadoop@funshion-hadoop194 lab]$ $HADOOP_HOME/bin/hdfs haadmin -getServiceState nn2
active
[hadoop@funshion-hadoop194 lab]$

-- 通过hdfs-site.xml中的如下配置，我们知道nn1是在 funshion-hadoop194上的namenode服务，nn2是funshion-hadoop195上的namenode服务

dfs.namenode.rpc-address.mycluster.nn1
funshion-hadoop194:8020

dfs.namenode.rpc-address.mycluster.nn2
funshion-hadoop195:8020

-- 所以，我们可以尝试 kill 掉 nn2(状态为active的namenode进程，然后去查看nn1的角色是否改变：
[hadoop@funshion-hadoop195 bin]$ jps
3199 JournalNode
3001 NameNode
1161 QuorumPeerMain
3364 DFSZKFailoverController
4367 Jps

[hadoop@funshion-hadoop195 bin]$ kill -9 3001
[hadoop@funshion-hadoop195 bin]$ jps
3199 JournalNode
1161 QuorumPeerMain
3364 DFSZKFailoverController
4381 Jps

[hadoop@funshion-hadoop195 bin]$ $HADOOP_HOME/bin/hdfs haadmin -getServiceState nn1
active
[hadoop@funshion-hadoop195 bin]$ $HADOOP_HOME/sbin/hadoop-daemon.sh start namenode
starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-funshion-hadoop195.out
[hadoop@funshion-hadoop195 bin]$ $HADOOP_HOME/bin/hdfs haadmin -getServiceState nn1
active
[hadoop@funshion-hadoop195 bin]$ $HADOOP_HOME/bin/hdfs haadmin -getServiceState nn2
standby

-- 甚至可以直接reboot状态为active的namenode节点(执行操作系统的重启动作，看另一个standby状态的namenode节点是否能正常转换成acitve状态
-- 甚至可以在有作业运行的时候去执行reboot操作系统(namenode的active节点执行)以测试双节点故障转移是否确实健壮

-- 集群相关网页：
-- http://funshion-hadoop194:50070/dfshealth.html#tab-overview

-- ##################################################################################

-- Step 6. 上传测试数据：

-- Step 6.1 安装wget包、创建相关目录及shell上传数据脚本：
[root@funshion-hadoop194 ~]# yum -y install wget
[hadoop@funshion-hadoop194 ~]$
[hadoop@funshion-hadoop194 ~]$ mkdir -p /home/hadoop/datateam/ghh/lab
[hadoop@funshion-hadoop194 ~]$ mkdir -p /home/hadoop/log_catch/down
[hadoop@funshion-hadoop194 ~]$ mkdir -p /home/hadoop/log_catch/put
[hadoop@funshion-hadoop194 ~]$ mkdir -p /home/hadoop/log_catch/zip
[hadoop@funshion-hadoop194 ~]$ vi /home/hadoop/datateam/ghh/lab/log_catch_hour_lzo.sh

#!/bin/bash

function f_show_info()
{
printf "%20s = %s\n" "$1" "$2"
return 0
}

function f_catch_all_day_log()
{
local str_date=""
local year=""
local month=""
local day=""

for(( str_date=${g_start_date};${str_date} do
year=$(date -d "${str_date}" +%Y )
month=$(date -d "${str_date}" +%m )
day=$(date -d "${str_date}" +%d )
f_catch_all_log ${year} ${month} ${day}
done
}

function f_catch_all_log()
{
local year="$1"
local month="$2"
local day="$3"
local hour=""
local date_hour=""
local date_dir=""
local hdfs_dir=""
local g_hdfs_dir=""
local hdfs_file=""
local url=""
local i=0
local nRet=0

for(( i=${g_start_hour};i do
hour=$(printf "%02d" "$i")
date_hour="${year}${month}${day}${hour}"
date_dir="${year}/${month}/${day}"
hdfs_dir="${year}/${month}/${day}/${hour}"
g_hdfs_dir="${g_hdfs_path}/${hdfs_dir}"
hdfs_file="${g_hdfs_path}/${hdfs_dir}/BeiJing_YiZhuang_CTC_${date_hour}.lzo"

url="${g_url}/${date_dir}/BeiJing_YiZhuang_CTC_${date_hour}.gz"
f_show_info "url" "${url}"
f_show_info "hdfs" "${hdfs_file}"
f_catch_log "${url}" "${hdfs_file}" "${g_hdfs_dir}"

hdfs_file="${g_hdfs_path}/${hdfs_dir}/BeiJing_ShangDi_CNC_${date_hour}.lzo"
url="${g_url}/${date_dir}/BeiJing_ShangDi_CNC_${date_hour}.gz"
f_show_info "url" "${url}"
f_show_info "hdfs" "${hdfs_file}"
f_catch_log "${url}" "${hdfs_file}" "${g_hdfs_dir}"
done
return $nRet
}

function f_catch_log()
{
local tmp_name=$( uuidgen | sed 's/-/_/g' )
local local_down_file="${g_local_down_path}/${tmp_name}"
local local_zip_file="${g_local_zip_path}/${tmp_name}"
local local_put_file="${g_local_put_path}/${tmp_name}"
local log_url="$1"
local hdfs_file="$2"
local nRet=0

if [[ 0 == $nRet ]];then
wget -O "${local_down_file}" "${log_url}"
nRet=$?
fi

if [[ 0 == $nRet ]];then
gzip -cd "${local_down_file}" | lzop -o "${local_zip_file}"
nRet=$?
fi

# if [[ 0 == $nRet ]];then
# gzip -cd "${local_down_file}" > "${local_zip_file}"
# nRet=$?
# fi

if [[ 0 == $nRet ]];then
mv "${local_zip_file}" "${local_put_file}"
hdfs dfs -mkdir -p "${g_hdfs_dir}"
hdfs dfs -put "${local_put_file}" "${hdfs_file}"
nRet=$?
fi

if [[ 0 == $nRet ]];then
hadoop jar /usr/local/hadoop/lib/native/hadoop-lzo-0.4.20-SNAPSHOT.jar com.hadoop.compression.lzo.LzoIndexer "${hdfs_file}"
nRet=$?
fi

rm -rf "${local_down_file}" "${local_put_file}" "${local_zip_file}"

return $nRet
}

# shell begins here

g_local_down_path="/home/hadoop/log_catch/down"
g_local_zip_path="/home/hadoop/log_catch/zip"
g_local_put_path="/home/hadoop/log_catch/put"

g_start_date=""
g_end_date=""
g_start_hour=0
g_end_hour=0
g_hdfs_path=""
g_url=""

nRet=0

if [[ 0 == $nRet ]];then
if [[ $# -ne 6 ]];then
f_show_info "cmd format" "sh ./log_catch.sh 'url' 'hdfs_path' 'start_date' 'end_date' 'start_hour' 'end_hour'"
nRet=1
else
g_url="$1"
g_hdfs_path="$2"
g_start_date="$3"
g_end_date="$4"
g_start_hour="$5"
g_end_hour="$6"
fi
fi

if [[ 0 == $nRet ]];then
f_catch_all_day_log
nRet=$?
fi

exit $nRet

-- Step 6.2 调用脚本上传数据：
[hadoop@funshion-hadoop194 ~]$ nohup sh /home/hadoop/datateam/ghh/lab/log_catch_hour_lzo.sh 'http://192.168.116.61:8081/website/pv/2' 'hdfs://mycluster/dw/logs/web/origin/pv/2' 20140524 20140525 0 23 &

-- nohup sh /home/hadoop/datateam/ghh/lab/log_catch_hour_lzo.sh 'http://192.168.116.61:8081/website/pv/2' 'hdfs://mycluster/dw/logs/web/origin/pv/2' 20140525 20140525 3 23 &

-- 上面这些脚本都是取公司的Oxeye的日志数据。（大家可以忽略此步操作）

-- Step 7. Hive安装(安装到196机器) (使用Hive与HBase整合安装；使用源码编译安装)

-- (其实应该先安装hbase，再安装hive可能顺序合理一点)

-- 参考：https://cwiki.apache.org/conflue ... iorto0.13onHadoop23
http://www.hadoopor.com/thread-5470-1-1.html
http://blog.csdn.net/hguisu/article/details/7282050
http://blog.csdn.net/hguisu/article/details/7282050
http://www.micmiu.com/bigdata/hive/hive-hbase-integration/

-- 源码下载编译操作如下：
mkdir -p /opt/software/hive_src
cd /opt/software/hive_src/
svn checkout http://svn.apache.org/repos/asf/hive/trunk/ hive_trunk
cd /opt/software/hive_src/hive_trunk

-- 下载以后，我们检查 hive_trunk目录下的pom.xml文件，发现hadoop-23.version这个变量已经引用了hadoop 2.4.0版本，所以，我们可以什么也不用修改，直接用ant去编译：
2.4.0

-- 或者如果发现版本不正确的话，我们可以这样指定参数执行(也可以修改pom.xml文件中对应正确的hadoop、hbase、zookeeper版本）：
-- 最后我选用的版本相关参数如下：
2.4.0
0.98.3-hadoop1
0.98.3-hadoop2
3.4.6

-- 最后，开始编译：
cd /opt/software/hive_src/hive_trunk
mvn clean package -DskipTests -Phadoop-2,dist

[INFO] Hive .............................................. SUCCESS [ 6.481 s]
[INFO] Hive Ant Utilities ................................ SUCCESS [ 4.427 s]
[INFO] Hive Shims Common ................................. SUCCESS [ 2.418 s]
[INFO] Hive Shims 0.20 ................................... SUCCESS [ 1.284 s]
[INFO] Hive Shims Secure Common .......................... SUCCESS [ 2.466 s]
[INFO] Hive Shims 0.20S .................................. SUCCESS [ 0.961 s]
[INFO] Hive Shims 0.23 ................................... SUCCESS [ 3.247 s]
[INFO] Hive Shims ........................................ SUCCESS [ 0.364 s]
[INFO] Hive Common ....................................... SUCCESS [ 5.259 s]
[INFO] Hive Serde ........................................ SUCCESS [ 7.428 s]
[INFO] Hive Metastore .................................... SUCCESS [ 27.000 s]
[INFO] Hive Query Language ............................... SUCCESS [ 51.924 s]
[INFO] Hive Service ...................................... SUCCESS [ 6.037 s]
[INFO] Hive JDBC ......................................... SUCCESS [ 14.293 s]
[INFO] Hive Beeline ...................................... SUCCESS [ 1.406 s]
[INFO] Hive CLI .......................................... SUCCESS [ 10.297 s]
[INFO] Hive Contrib ...................................... SUCCESS [ 1.418 s]
[INFO] Hive HBase Handler ................................ SUCCESS [ 33.679 s]
[INFO] Hive HCatalog ..................................... SUCCESS [ 0.443 s]
[INFO] Hive HCatalog Core ................................ SUCCESS [ 8.040 s]
[INFO] Hive HCatalog Pig Adapter ......................... SUCCESS [ 1.795 s]
[INFO] Hive HCatalog Server Extensions ................... SUCCESS [ 2.007 s]
[INFO] Hive HCatalog Webhcat Java Client ................. SUCCESS [ 1.548 s]
[INFO] Hive HCatalog Webhcat ............................. SUCCESS [ 11.718 s]
[INFO] Hive HCatalog Streaming ........................... SUCCESS [ 1.845 s]
[INFO] Hive HWI .......................................... SUCCESS [ 1.246 s]
[INFO] Hive ODBC ......................................... SUCCESS [ 0.626 s]
[INFO] Hive Shims Aggregator ............................. SUCCESS [ 0.192 s]
[INFO] Hive TestUtils .................................... SUCCESS [ 0.324 s]
[INFO] Hive Packaging .................................... SUCCESS [01:21 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:53 min
[INFO] Finished at: 2014-06-22T11:58:05+08:00
[INFO] Final Memory: 147M/1064M
[INFO] ------------------------------------------------------------------------

-- 最后，/opt/software/hive_src/hive_trunk/packaging/target　目录下的 apache-hive-0.14.0-SNAPSHOT-bin.tar.gz 文件，就是我们需要的安装包（这个版本还没有正式发布）

-- Step 7.1 My SQL安装(安装到194机器)，并在My SQL中创建名为hive的数据库用以存放hive元数据：

-- 安装如下rpm包
rpm -ivh MySQL-client-5.6.17-1.linux_glibc2.5.x86_64.rpm
rpm -ivh MySQL-devel-5.6.17-1.linux_glibc2.5.x86_64.rpm
rpm -ivh MySQL-embedded-5.6.17-1.linux_glibc2.5.x86_64.rpm
rpm -e --nodeps mysql-libs-5.1.66-2.el6_3.x86_64
rpm -ivh MySQL-server-5.6.17-1.linux_glibc2.5.x86_64.rpm
rpm -ivh MySQL-shared-5.6.17-1.linux_glibc2.5.x86_64.rpm
rpm -ivh MySQL-shared-compat-5.6.17-1.linux_glibc2.5.x86_64.rpm
rpm -ivh MySQL-test-5.6.17-1.linux_glibc2.5.x86_64.rpm

A RANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER !
You will find that password in '/root/.mysql_secret'.

You must change that password on your first connect,
no other statement but 'SET PASSWORD' will be accepted.
See the manual for the semantics of the 'password expired' flag.

Also, the account for the anonymous user has been removed.

In addition, you can run:

/usr/bin/mysql_secure_installation

which will also give you the option of removing the test database.
This is strongly recommended for production servers.

See the manual for more instructions.
Please report any problems at http://bugs.mysql.com/
The latest information about MySQL is available on the web at
http://www.mysql.com
Support MySQL by buying support/licenses at http://shop.mysql.com
New default config file was created as /usr/my.cnf and
will be used by default by the server when you start it.
You may edit this file to change server settings
-- 查看安装生成的root用户随机密码：
[root@funshion-hadoop194 ~]# more /root/.mysql_secret
# The random password set for the root user at Mon Jun 9 18:18:48 2014 (local time): QVkyOjwSlAEiPaeT

-- 登录My SQL数据库并修改root密码，并创建名为hive的数据库与用户：
[root@funshion-hadoop194 ~]# service mysql start
Starting MySQL... SUCCESS!

-- 设置mysql服务自启动
chkconfig mysql on

[root@funshion-hadoop194 ~]# mysql -uroot -pQVkyOjwSlAEiPaeT
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.17

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SET PASSWORD = PASSWORD('bee56915');
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE DATABASE `hive` /*!40100 DEFAULT CHARACTER SET utf8 */;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER 'hive'@'funshion-hadoop196' IDENTIFIED BY password('bee56915');
Query OK, 0 rows affected (0.00 sec)

GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%' Identified by 'bee56915';
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'localhost' Identified by 'bee56915';
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'127.0.0.1' Identified by 'bee56915';
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'funshion-hadoop196' Identified by 'bee56915';

-- Step 7.2 解决hive安装包到/usr/local下，添加hive相关环境变量：

[root@funshion-hadoop194 ~]# cd /opt/software
[root@funshion-hadoop194 software]# ls -l|grep hive
-rw-r--r--. 1 root root 65662469 May 15 14:04 hive-0.12.0-bin.tar.gz
[root@funshion-hadoop194 software]# tar -xvf ./hive-0.12.0-bin.tar.gz
[root@funshion-hadoop194 software]# mv hive-0.12.0-bin /usr/local
[root@funshion-hadoop194 software]# cd /usr/local
[root@funshion-hadoop194 local]# chown -R hadoop.hadoop ./hive-0.12.0-bin
[root@funshion-hadoop194 local]# ln -s hive-0.12.0-bin hive

[hadoop@funshion-hadoop194 local]$ vi ~/.bash_profile
export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/bin

[hadoop@funshion-hadoop194 local]$ source ~/.bash_profile

-- Step 7.3 在My SQL数据库的hive数据库中执行创建hive元数据脚本：

-- Step 7.4 修改hive相关配置文件：

[hadoop@funshion-hadoop194 mysql]$ cd $HIVE_HOME/conf
[hadoop@funshion-hadoop194 conf]$ ls -l
total 92
-rw-rw-r--. 1 hadoop hadoop 81186 Oct 10 2013 hive-default.xml.template
-rw-rw-r--. 1 hadoop hadoop 2378 Oct 10 2013 hive-env.sh.template
-rw-rw-r--. 1 hadoop hadoop 2465 Oct 10 2013 hive-exec-log4j.properties.template
-rw-rw-r--. 1 hadoop hadoop 2870 Oct 10 2013 hive-log4j.properties.template

[hadoop@funshion-hadoop194 conf]$ mv hive-env.sh.template hive-env.sh
[hadoop@funshion-hadoop194 conf]$ mv hive-default.xml.template hive-site.xml

-- 7.4.1 修改 $HIVE_HOME/bin/hive-config.sh 文件，添加如下环境变量：

[hadoop@funshion-hadoop194 conf]$ vi $HIVE_HOME/bin/hive-config.sh

export JAVA_HOME=/usr/java/latest
export HIVE_HOME=/usr/local/hive
export HADOOP_HOME=/usr/local/hadoop

-- 7.4.2 修改 $HIVE_HOME/conf/hive-site.xml 的第2002行：

4.4.报错—请修改hive-site.xml:（vi编辑下： /auth）

-- 原值：
auth

-- 修改为：
auth）

-- 7.4.3 修改 $HIVE_HOME/conf/hive-site.xml 的如下property:

-- 7.4.3.1
-- 原值：

javax.jdo.option.ConnectionURL
jdbc:derby:;databaseName=metastore_db;create=true
JDBC connect string for a JDBC metastore

-- 修改为：

javax.jdo.option.ConnectionURL
jdbc:mysql://funshion-hadoop194:3306/hive?createDatabaseIfNotExist=true
JDBC connect string for a JDBC metastore

-- 7.4.3.2
-- 原值：

javax.jdo.option.ConnectionDriverName
org.apache.derby.jdbc.EmbeddedDriver
Driver class name for a JDBC metastore

-- 修改为：

javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver
Driver class name for a JDBC metastore

-- 7.4.3.3
-- 原值：

javax.jdo.option.ConnectionUserName
APP
username to use against metastore database

-- 修改为：

javax.jdo.option.ConnectionUserName
hive
username to use against metastore database

-- 7.4.3.4
-- 原值：

javax.jdo.option.ConnectionPassword
mine
password to use against metastore database

-- 修改为

javax.jdo.option.ConnectionPassword
bee56915
password to use against metastore database

-- 7.4.3.5
-- 原值：

hive.metastore.warehouse.dir
/user/hive/warehouse
location of default database for the warehouse

-- 修改为：

hive.metastore.warehouse.dir
hdfs://mycluster:8020/user/hive/warehouse
location of default database for the warehouse

-- 7.4.3.6
-- 原值：

hive.exec.scratchdir
/tmp/hive-${user.name}
Scratch space for Hive jobs

-- 修改为：

hive.exec.scratchdir
hdfs://mycluster:8020/tmp/hive-${user.name}
Scratch space for Hive jobs

-- 添加：

hbase.zookeeper.quorum
funshion-hadoop194,funshion-hadoop195,funshion-hadoop196,funshion-hadoop197,funshion-hadoop198

hive.aux.jars.path

file:///usr/local/hive/lib/hive-ant-0.14.0-SNAPSHOT.jar,
file:///usr/local/hive/lib/protobuf-java-2.5.0.jar,
file:///usr/local/hbase/lib/hbase-server-0.98.3-hadoop2.jar,
file:///usr/local/hbase/lib/hbase-client-0.98.3-hadoop2.jar,
file:///usr/local/hbase/lib/hbase-common-0.98.3-hadoop2.jar,
file:///usr/local/hbase/lib/hbase-common-0.98.3-hadoop2-tests.jar,
file:///usr/local/hbase/lib/hbase-protocol-0.98.3-hadoop2.jar,
file:///usr/local/hbase/lib/htrace-core-2.04.jar,
file:///usr/local/hive/lib/zookeeper-3.4.6.jar,
file:///usr/local/hive/lib/guava-11.0.2.jar

-- 上面格式是方便查看，真正使用下面的格式：将所有的jar包放到一行：

hive.aux.jars.path
file:///usr/local/hive/lib/hive-ant-0.14.0-SNAPSHOT.jar,file:///usr/local/hbase/lib/hbase-server-0.98.3-hadoop2.jar,file:///usr/local/hbase/lib/hbase-client-0.98.3-hadoop2.jar,file:///usr/local/hbase/lib/hbase-common-0.98.3-hadoop2.jar,file:///usr/local/hbase/lib/hbase-common-0.98.3-hadoop2-tests.jar,file:///usr/local/hbase/lib/hbase-protocol-0.98.3-hadoop2.jar,file:///usr/local/hbase/lib/htrace-core-2.04.jar,file:///usr/local/hive/lib/zookeeper-3.4.6.jar

-- 首先需要把hive/lib下的hbase包替换成安装的hbase的，需要如下几下：
hbase-client-0.98.2-hadoop2.jar
hbase-common-0.98.2-hadoop2.jar
hbase-common-0.98.2-hadoop2-tests.jar
hbase-protocol-0.98.2-hadoop2.jar
htrace-core-2.04.jar
hbase-server-0.98.2-hadoop2.jar

将hadoop节点添加到hive-site.xml中

hbase.zookeeper.quorum
所有节点

-- 另外，你必须在创建Hive库表前，在HDFS上创建/tmp和/user/hive/warehouse（也称为hive.metastore.warehouse.dir所指定的目录），并且将它们的权限设置为chmod g+w。完成这个操作的命令如下：
$ $HADOOP_HOME/bin/hadoop fs -mkdir /tmp
$ $HADOOP_HOME/bin/hadoop fs -mkdir /user/hive/warehouse
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse

-- Step 7.5 启动并登录hive，并创建hive表

14/06/16 18:58:50 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect. Use hive.hmshandler.retry.* instead

-- 集群启动：
# bin/hive --service hiveserver -hiveconf hbase.zookeeper.quorum=funshion-hadoop194,funshion-hadoop195,funshion-hadoop196,funshion-hadoop197,funshion-hadoop198 &
# bin/hive -hiveconf hbase.zookeeper.quorum=funshion-hadoop194,funshion-hadoop195,funshion-hadoop196,funshion-hadoop197,funshion-hadoop198 &
# bin/hive -hiveconf hive.root.logger=DEBUG,console hbase.master=funshion-hadoop194:60010

# bin/hive -hiveconf hbase.master=funshion-hadoop194:60010 --auxpath /usr/local/hive/lib/hive-ant-0.13.1.jar,/usr/local/hive/lib/protobuf-java-2.5.0.jar,/usr/local/hive/lib/hbase-client-0.98.3-hadoop2.jar, \
/usr/local/hive/lib/hbase-common-0.98.3-hadoop2.jar,/usr/local/hive/lib/zookeeper-3.4.6.jar,/usr/local/hive/lib/guava-11.0.2.jar

#bin/hive -hiveconf hbase.zookeeper.quorum=node1,node2,node3

-- 客户端登录：
$HIVE_HOME/bin/hive -h127.0.0.1 -p10000
$HIVE_HOME/bin/hive -hfunshion-hadoop194 -p10000
$HIVE_HOME/bin/hive -p10000

[hadoop@funshion-hadoop194 lib]$ hive --service hiveserver &

[hadoop@funshion-hadoop194 lib]$ hive
14/06/10 16:56:59 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
14/06/10 16:56:59 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/06/10 16:56:59 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/06/10 16:56:59 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/06/10 16:56:59 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/06/10 16:56:59 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/06/10 16:56:59 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative

Logging initialized using configuration in jar:file:/usr/local/hive-0.12.0-bin/lib/hive-common-0.12.0.jar!/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hive-0.12.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
hive> show databases;
OK
Failed with exception java.io.IOException:java.io.IOException: Cannot create an instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!

-- 如果报类似如上错误，在 ~/.bash_profile 添加环境变量，如下：

export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native/Linux-amd64-64
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_HOME/lib/native/hadoop-lzo-0.4.20-SNAPSHOT.jar

-- hive客户端登录：
[hadoop@funshion-hadoop194 bin]$ $HIVE_HOME/bin/hive -h127.0.0.1 -p10000
14/06/10 17:13:17 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
14/06/10 17:13:17 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/06/10 17:13:17 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/06/10 17:13:17 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/06/10 17:13:17 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/06/10 17:13:17 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/06/10 17:13:17 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative

Logging initialized using configuration in jar:file:/usr/local/hive-0.12.0-bin/lib/hive-common-0.12.0.jar!/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hive-0.12.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[127.0.0.1:10000] hive> create database web;
OK
[127.0.0.1:10000] hive> CREATE EXTERNAL TABLE pv2(
> protocol string,
> rprotocol string,
> time int,
> ip string,
> fck string,
> mac string,
> userid string,
> fpc string,
> version string,
> sid string,
> pvid string,
> config string,
> url string,
> referurl string,
> channelid string,
> vtime string,
> ext string,
> useragent string,
> step string,
> sestep string,
> seidcount string,
> ta string)
> PARTITIONED BY (
> year string,
> month string,