dataX3.0安装使用手册.doc
-Data*3.0使用手册目录一、data*概述31data*作用32Data*3.0框架设计33Data*3.0插件体系3二、data*安装31创立用户组及用户32环境变量配置33安装python3三、data*使用31生成配置文件模板32配置定时任务33常见问题及处理3四、data*插件开发及安装31.Reader插件开发32.Writter插件开发33.插件部署3一、 data*概述1 data*作用Data* 是一个异构数据源离线同步工具，致力于实现包括关系型数据库(MySQL、Oracle等)、HDFS、Hive、ODPS、HBase、FTP等各种异构数据源之间稳定高效的数据同步功能。为了解决异构数据源同步问题，Data*将复杂的网状的同步链路变成了星型数据链路，Data*作为中间传输载体负责连接各种数据源。当需要接入一个新的数据源的时候，只需要将此数据源对接到Data*，便能跟已有的数据源做到无缝数据同步。2 Data*3.0框架设计Data*本身作为离线数据同步框架，采用Framework + plugin架构构建。将数据源读取和写入抽象成为Reader/Writer插件，纳入到整个同步框架中。Reader：Reader为数据采集模块，负责采集数据源的数据，将数据发送给Framework。Writer： Writer为数据写入模块，负责不断向Framework取数据，并将数据写入到目的端。Framework：Framework用于连接reader和writer，作为两者的数据传输通道，并处理缓冲，流控，并发，数据转换等核心技术问题。3 Data*3.0插件体系类型数据源Reader(读)Writer(写)RDBMS 关系型数据库MySQL Oracle SQL Server PostgreSQL 达梦 通用RDBMS(支持所有关系型数据库)阿里云数仓数据存储Ma*Compute(原ODPS) Analytic DB(原ADS) OSS 云数据库Memcache版(原OCS)HiveNoSQL数据存储Table Store(原OTS) Hbase0.94 Hbase1.1 MongoDB无构造化数据存储T*tFileJsonFile FTP HDFS二、 data*安装1 创立用户组及用户roothmaster-hdfs:/home/ubuntu# groupadd data*roothmaster-hdfs:/home/ubuntu# useradd -g data* data* -m -d /home/data*roothmaster-hdfs:/home/ubuntu# passwd data*Enter new UNI* password: Retype new UNI* password: passwd: password updated successfullyroothmaster-hdfs:/home/ubuntu# usermod -G adm -a data*roothmaster-hdfs:/home/ubuntu# su - data*data*hmaster-hdfs:$ iduid=1004(data*) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),1002(data*)2 环境变量配置e*port JRE_HOME=$JAVA_HOME/jree*port CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/libe*port PATH=$JAVA_HOME/bine*port LANG=zh_CN.UTF-83 安装pythondata*hmaster-hdfs:/home/ubuntu$ pythonPython 2.7.6 (default, Jun 22 2015, 17:58:13) GCC 4.8.2 on linu*2Type "help", "copyright", "credits" or "license" for more information.三、 data*使用1 生成配置文件模板命令：bin/python data*.py -r YOUR_READER -w YOUR_WRITER案例1：mysqlreader-> mysqlwriterdata*hmaster-hdfs:/data*3/data*/bin$ python data*.py -r mysqlreader -w mysqlwriterData* (DATA*-OPENSOURCE-1.0), From Alibaba !Copyright (C) 2010-2015, Alibaba Group. All Rights Reserved.Please refer to the mysqlreader document: s:/github./alibaba/Data*/blob/master/mysqlreader/doc/mysqlreader.md Please refer to the mysqlwriter document: s:/github./alibaba/Data*/blob/master/mysqlwriter/doc/mysqlwriter.mdPlease save the following configuration as a json file and use python DATA*_HOME/bin/data*.py JSON_FILE_NAME.json to run the job. "job": "content": "reader": "name": "mysqlreader", "parameter": "column": , "connection": "jdbcUrl": , "table": , "password": "", "username": "", "where": "" , "writer": "name": "mysqlwriter", "parameter": "column": , "connection": "jdbcUrl": "", "table": , "password": "", "preSql": , "session": , "username": "", "writeMode": "" , "setting": "speed": "channel": "" 案例2：jsonfilereader-> hbasewriterdata*hmaster-hdfs:/data*3/data*/bin$ python data*.py -r jsonfilereader -w hbasewriter Data* (DATA*-OPENSOURCE-1.0), From Alibaba !Copyright (C) 2010-2015, Alibaba Group. All Rights Reserved.Please refer to the jsonfilereader document: s:/github./alibaba/Data*/blob/master/jsonfilereader/doc/jsonfilereader.md Please refer to the hbasewriter document: s:/github./alibaba/Data*/blob/master/hbasewriter/doc/hbasewriter.md Please save the following configuration as a json file and use python DATA*_HOME/bin/data*.py JSON_FILE_NAME.json to run the job. "job": "content": "reader": "name": "jsonfilereader", "parameter": "column": , "compress": "zip", "encoding": "", "path": , "writer": "name": "hbase11*writer", "parameter": "column": , "encoding": "", "hbaseConfig": "hbase.cluster.distributed": "", "hbase.rootdir": "", "hbase.zookeeper.quorum": "" , "mode": "", "rowkeyColumn": , "table": "", "versionColumn": "inde*": "", "value": "" , "setting": "speed": "channel": "" 2 配置定时任务1.1 创立任务列表文件data*hmaster-hdfs:/data*3/data*/task$ vi crontab.tasks.t*t 20,30,40 0-23 * * * /usr/bin/python /home/data*/data*3/data*/bin/data*.py /home/data*/data*3/data*/job/jsonfile2hbase_fullry_log.terminal_log.json >/data/data*/log/data*.crontab.out.log 2>&11.2 crontab加载系统任务列表文件data*hmaster-hdfs:/data*3/data*/task$ crontab crontab.tasks.t*t data*hmaster-hdfs:/data*3/data*/task$ crontab -l20,30,40 0-23 * * * /usr/bin/python /home/data*/data*3/data*/bin/data*.py /home/data*/data*3/data*/job/jsonfile2hbase_fullry_log.terminal_log.json >/data/data*/log/data*.crontab.out.log 2>&13 常见问题及处理1.1 /bin/sh: 1: java: not found原因：/home/data*/data*3/data*/bin/ data*.py未取到JAVA_HOME环境变量值四、 data*插件开发及安装1. Reader插件开发publicclass JsonFileReader e*tends Readerpublicstaticclass Job e*tendsReader.Job Overridepublicvoidinit() / TODO Auto-generated method stubOverridepublicvoiddestroy() / TODO Auto-generated method stubOverridepublic List<Configuration> split(intadviceNumber) / TODO Auto-generated method stubreturnnull;publicstaticclass Task e*tends Reader.TaskOverridepublicvoid init() / TODO Auto-generated method stubOverridepublicvoid destroy() / TODO Auto-generated method stubOverridepublicvoid startRead(RecordSender recordSender) / TODO Auto-generated method stub2. Writter插件开发publicclass JsonFileWriter e*tends Writer publicstaticclass Job e*tends Writer.Job Overridepublicvoid init() / TODO Auto-generated method stubOverridepublicvoid destroy() / TODO Auto-generated method stubOverridepublic List<Configuration> split(intmandatoryNumber) / TODO Auto-generated method stubreturnnull;publicstaticclassTaske*tends Writer.Task Overridepublicvoid init() / TODO Auto-generated method stubOverridepublicvoid destroy() / TODO Auto-generated method stubOverridepublicvoid startWrite(RecordReceiver lineReceiver) / TODO Auto-generated method stub3. 插件部署目录及文件说明data*hmaster-hdfs:/data*3/data*/plugin/reader/jsonfilereader$ lltotal 56drw*r-*r-* 3 data* ubuntu 4096 Jul 6 16:10 ./drw*r-*r-* 21 data* ubuntu 4096 Jun 21 11:05 ./-rw-r-r- 1 data* ubuntu 36348 Jun 22 17:52 jsonfilereader.jardrw*r-*r-* 2 data* ubuntu 4096 May 16 12:46 libs/-rw- 1 data* ubuntu 315 May 16 11:24 plugin.json-rw- 1 data* ubuntu 149 May 15 18:24 plugin_job_template.jsonl jsonfilereader.jar 插件JAR包文件l libs 插件依赖第三方JAR包文件l plugin.json 插件说明文件l plugin_job_template.json 插件配置模板文件plugin.json文件容说明 "name": "jsonfilereader", "class": ".alibaba.data*.plugin.reader.jsonfilereader.JsonFileReader", "description": "useScene: test. mechanism: use data* framework to transport data from json file. warn: The more you know about the data, the less problems you encounter.", "developer": "alibaba"Name：插件名称Class：jsonfilereader.jar文件中全路径Description：功能描述plugin_job_template.json文件容说明 "name": "jsonfilereader", "parameter": "path": , "encoding": "", "column": , "compress":"zip" . z.
