sqoop 1.99 安装配置

abloz 2013-08-20
abloz 2013-08-20

周海汉/文 2013.8.20 http://abloz.com

摘要:

  1. sqoop 1.99的安装配置
  2. client使用
  3. 从HBase,Hive导数据到mysql

版本 sqoop-1.99.2-bin-hadoop100

Sqoop是Hadoop系统数据和RDBMS互导数据的工具。1.99新版的一个包里包含两个部分:客户端和服务器端。必须在集群中安装好一个节点的服务器端。所有客户端和该服务器端相连。服务器端作为mapreduce的client,所以Hadoop必须和Sqoop服务器端装在一起。客户端则不限。这种设计让导数据更灵活。原1.44版则没有区分服务器端和客户端。

服务器端安装

1.确认本机有hadoop hadoop fs -ls

由于hadoop的主版本1.xx和2.xx不兼容,所以sqoop的二进制版本也是分100和200的。如我在hadoop 1.1.2中使用sqoop-1.99.2-bin-hadoop100来进行配套。 解压 tar zxvf sqoop-1.99.2-bin-hadoop100.tar.gz cd sqoop-1.99.2-bin-hadoop100

  1. 安装依赖库和组件

./bin/addtowar.sh -hadoop-auto

[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/addtowar.sh -hadoop-auto

Non of expected directories with Hadoop exists

Usage : addtowar.sh Options: -hadoop-auto Try to guess hadoop version and path -hadoop-version HADOOP_VERSION Specify used version -hadoop-path HADOOP_PATHS Where to find hadoop jars (multiple paths with Hadoop jars separated by ‘:’) -jars JARS_PATH Special jars that should be added (multiple JAR paths separated by ‘:’) -war SQOOP_WAR Target Sqoop war file where all jars should be ingested

由于我的hadoop没有安装到系统路径,所以需修改配置文件。 [hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ vi ./bin/addtowar.sh hadoopPossiblePaths=”/home/hadoop/hadoop-1.1.2 /usr/lib/hadoop /usr/lib/hadoop-mapreduce/ /usr/lib/hadoop-yarn/ /usr/lib/hadoop-hdfs”

[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/addtowar.sh -hadoop-auto ./bin/addtowar.sh: line 126: [: missing `]’ Hadoop version: 1.1.2 Hadoop path: /home/hadoop/hadoop-1.1.2 Extra jars:

Injecting following Hadoop JARs

/home/hadoop/hadoop-1.1.2/hadoop-core-1.1.2.jar /home/hadoop/hadoop-1.1.2/lib/jackson-core-asl-1.8.8.jar /home/hadoop/hadoop-1.1.2/lib/jackson-mapper-asl-1.8.8.jar /home/hadoop/hadoop-1.1.2/lib/commons-configuration-1.6.jar /home/hadoop/hadoop-1.1.2/lib/commons-logging-api-1.0.4.jar /home/hadoop/hadoop-1.1.2/lib/slf4j-api-1.4.3.jar /home/hadoop/hadoop-1.1.2/lib/slf4j-log4j12-1.4.3.jar

Backing up original WAR file to ./bin/../server/webapps/sqoop.war_2013-08-20_09:36:01.263437795

New Sqoop WAR file with added ‘Hadoop JARs’ at ./bin/../server/webapps/sqoop.war

脚本126行]有个空格问题,但不影响结果。改后: [hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ vi ./bin/addtowar.sh [hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/addtowar.sh -hadoop-auto Hadoop version: 1.1.2 Hadoop path: /home/hadoop/hadoop-1.1.2 Extra jars:

Specified Sqoop WAR ‘./bin/../server/webapps/sqoop.war’ already contains Hadoop JAR files

也可以指定,-hadoop-version指定版本,-hadoop-path 指定目录。如果各部分安装在多个目录,则用:分隔。 如: hadoop 1.0 ./bin/addtowar.sh -hadoop-version 1.0 -hadoop-path /usr/lib/hadoop-common:/usr/lib/hadoop-hdfs:/usr/lib/hadoop-mpred

hadoop 2.0 ./bin/addtowar.sh -hadoop-version 2.0 -hadoop-path /usr/lib/hadoop-common:/usr/lib/hadoop-hdfs:/usr/lib/hadoop-yarn

也可以用addtowar.sh的-jars参数来绑定其他的jar文件。 绑定jdbc mysql库: 由于不同协议,sqoop没有自带mysql jdbc库,需下载,并绑定: http://dev.mysql.com/downloads/mirror.php?id=13597 wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.0.8.tar.gz/from/http://cdn.mysql.com/

注意下载的是tar.gz, 里面有源码和jar包。只需解压后使用jar包即可。

[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/addtowar.sh -jars /home/hadoop/hive-0.11.0/lib/mysql-connector-java-5.0.8-bin.jar Hadoop version: Hadoop path: Extra jars: /home/hadoop/hive-0.11.0/lib/mysql-connector-java-5.0.8-bin.jar

Injecting following additional JARs

/home/hadoop/hive-0.11.0/lib/mysql-connector-java-5.0.8-bin.jar

Backing up original WAR file to ./bin/../server/webapps/sqoop.war_2013-08-20_09:49:32.401896012

New Sqoop WAR file with added ‘JARs’ at ./bin/../server/webapps/sqoop.war

3.配置sqoop服务器 server/conf 里面存放服务器的配置文件。包括tomcat等配置。但缺省配置PropertiesConfigurationProvider足够用。如需修改,编辑sqoop_bootstrap.properties的sqoop.config.provider即可。 [hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ cd server/conf [hadoop@hs11 conf]$ ls catalina.policy catalina.properties context.xml logging.properties server.xml sqoop_bootstrap.properties sqoop.properties tomcat-users.xml web.xml [hadoop@hs11 conf]$ cat sqoop_bootstrap.properties sqoop.config.provider=org.apache.sqoop.core.PropertiesConfigurationProvider

sqoop.properties包含其他的一些修改。可能需要微调,以适应环境需要。 #org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/ org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/hadoop/hadoop-1.1.2/conf/

  1. 启动和停止服务器 ./bin/sqoop.sh server start

[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/sqoop.sh server start Sqoop home directory: /home/hadoop/sqoop-1.99.2-bin-hadoop100… Using CATALINA_BASE: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server Using CATALINA_HOME: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server Using CATALINA_TMPDIR: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server/temp Using JRE_HOME: /usr/java/jdk1.6.0_45 Using CLASSPATH: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server/bin/bootstrap.jar

[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/sqoop.sh server stop Sqoop home directory: /home/hadoop/sqoop-1.99.2-bin-hadoop100… Using CATALINA_BASE: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server Using CATALINA_HOME: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server Using CATALINA_TMPDIR: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server/temp Using JRE_HOME: /usr/java/jdk1.6.0_45 Using CLASSPATH: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server/bin/bootstrap.jar

5.客户端安装 客户端无需配置,只需将下载版本解压即可。 [hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/sqoop.sh client Sqoop home directory: /home/hadoop/sqoop-1.99.2-bin-hadoop100… Aug 20, 2013 10:07:54 AM java.util.prefs.FileSystemPreferences$2 run INFO: Created user preferences directory. Sqoop Shell: Type ‘help’ or ‘h’ for help.

sqoop:000> 或执行sqoop脚本: sqoop.sh client /path/to/your/script.sqoop

具体使用在下一篇详述。 6.参考: http://sqoop.apache.org/docs/1.99.2/Installation.html http://sqoop.apache.org/docs/1.99.2/Sqoop5MinutesDemo.html http://sqoop.apache.org/docs/1.99.2/CommandLineClient.html


如非注明转载, 均为原创. 本站遵循知识共享CC协议,转载请注明来源