1. 到github下载源码,自己编译。同步数据支持mysql8.0,如果直接用编译好的会遇到各种问题。

https://github.com/alibaba/DataX/blob/master/userGuid.md

idea导入项目,需要先安装好jdk1.8+,python,maven3.x,mysql-connmaven依赖版本号修改:

把mysql-connector-java-5.1.34.jar 修改为mysql-connector-java-8.0.19.jar,这些模块都要修改

  <mysql.driver.version>8.0.19</mysql.driver.version>

DataBaseType.java修改:com.mysql.jdbc.Driver改为:com.mysql.cj.jdbc.Driver

<mysql.driver.version>8.0.19</mysql.driver.version>

MySql("mysql", "com.mysql.cj.jdbc.Driver"),
    Tddl("mysql", "com.mysql.cj.jdbc.Driver"),
    DRDS("drds", "com.mysql.cj.jdbc.Driver"),
    Oracle("oracle", "oracle.jdbc.OracleDriver"),
    SQLServer("sqlserver", "com.microsoft.sqlserver.jdbc.SQLServerDriver"),
    PostgreSQL("postgresql", "org.postgresql.Driver"),
    RDBMS("rdbms", "com.alibaba.datax.plugin.rdbms.util.DataBaseType"),
    DB2("db2", "com.ibm.db2.jcc.DB2Driver"),
    ADS("ads","com.mysql.cj.jdbc.Driver"),
    ClickHouse("clickhouse", "ru.yandex.clickhouse.ClickHouseDriver"),
    KingbaseES("kingbasees", "com.kingbase8.Driver"),
    Oscar("oscar", "com.oscar.Driver"),
    OceanBase("oceanbase", "com.alipay.oceanbase.jdbc.Driver"),
    StarRocks("starrocks", "com.mysql.cj.jdbc.Driver");

将连接属性中的zeroDateTimeBehavior=convertToNull改为zeroDateTimeBehavior=CONVERT_TO_NULL 

suffix = "yearIsDateType=false&zeroDateTimeBehavior=CONVERT_TO_NULL&tinyInt1isBit=false&rewriteBatchedStatements=true";

  重新编译:mvn -U clean package assembly:assembly -Dmaven.test.skip=true

  等待编译完成,实际大概花了18分钟,每个人可能不同。最后在target目录下生产文件。

 

  2. 新建java springboot项目

  datax:刚才编译的项目

导入依赖。datax-common-0.0.1-SNAPSHOT.jar,datax-core-0.0.1-SNAPSHOT.jar

把这2个jar导入本地maven仓库中。在idea,maven命令行窗口执行下面命令:

install:install-file -Dfile=D:\work\temp\datax-common-0.0.1-SNAPSHOT.jar -DgroupId=com.datax.common -DartifactId=common -Dversion=0.0.1 -Dpackaging=jar
install:install-file -Dfile=D:\work\temp\datax-core-0.0.1-SNAPSHOT.jar -DgroupId=com.datax.core -DartifactId=core -Dversion=0.0.1 -Dpackaging=jar

  pom.xml文件添加依赖:

<dependency>
            <groupId>com.datax.core</groupId>
            <artifactId>core</artifactId>
            <version>0.0.1</version>
        </dependency>
        <dependency>
            <groupId>com.datax.common</groupId>
            <artifactId>common</artifactId>
            <version>0.0.1</version>
        </dependency>
        <dependency>
            <groupId>commons-cli</groupId>
            <artifactId>commons-cli</artifactId>
            <version>1.4</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.13</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-io</artifactId>
            <version>1.3.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.12.0</version>
        </dependency>
        <dependency>
            <groupId>commons-lang</groupId>
            <artifactId>commons-lang</artifactId>
            <version>2.6</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.60</version>
        </dependency>

新建test.json文件

{
  "job": {
    "setting": {
      "speed": {
        "channel": 4
      }
    },
    "content": [
      {
        "reader": {
          "name": "mysqlreader",
          "parameter": {
            "username": "root",
            "password": "123456",
            "connection": [
              {
                "jdbcUrl": ["jdbc:mysql://127.0.0.1:3306/test1?serverTimezone=UTC&characterEncoding=utf8&useSSL=true&autoReconnect=true&failOverReadOnly=false"],
                "querySql": ["select t.id,t.name,t.status from user_test t where t.id>=${id}"]
              }
            ]
          }
        },
        "writer": {
          "name": "mysqlwriter",
          "parameter": {
            "username": "root",
            "password": "123456",
            "writeMode": "insert",
            "column": ["id","name","status"],
            "connection": [
              {
                "table": [
                  "user_test"
                ],
                "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/test2?serverTimezone=UTC&characterEncoding=utf8&useSSL=true&autoReconnect=true&failOverReadOnly=false"
              }
            ]
          }
        }
      }
    ]
  }
}

java调用代码:

@Override
    public void syncDataUser() {
        System.setProperty("datax.home","D:\\work\\project\\code2\\demo1106\\datax");
        System.setProperty("id","5");
        String path= commonUtils.getCurrentClasspath();
        String[] datxArgs2 = {"-job", path+"/datax/test.json", "-mode", "standalone", "-jobid", "-1"};
        try {
            Engine.entry(datxArgs2);
        } catch (Throwable e) {
            e.printStackTrace();
        }
    }

datax.home:源码编译后的路径。不能错否则报错。

从mysql test1.user_test 条件:id>=5数据同步到test2.user_test

目标:

  源库表:

 

 执行结果:

  完整项目代码下载:

 链接:https://pan.baidu.com/s/1Ooj-jvqpwa1MSaq0eCJY8g 

提取码:pzpj

 

参考资料:

alibaba/DataX github
https://github.com/alibaba/DataX/blob/master/userGuid.md
java整合datax最详细的教程
https://blog.csdn.net/Ting1king/article/details/120871137
DataX实践趟坑大全
https://blog.csdn.net/superman_wang/article/details/118635976

 

原文地址:http://www.cnblogs.com/xiaozw/p/16884664.html

1. 本站所有资源来源于用户上传和网络,如有侵权请邮件联系站长! 2. 分享目的仅供大家学习和交流,请务用于商业用途! 3. 如果你也有好源码或者教程,可以到用户中心发布,分享有积分奖励和额外收入! 4. 本站提供的源码、模板、插件等等其他资源,都不包含技术服务请大家谅解! 5. 如有链接无法下载、失效或广告,请联系管理员处理! 6. 本站资源售价只是赞助,收取费用仅维持本站的日常运营所需! 7. 如遇到加密压缩包,默认解压密码为"gltf",如遇到无法解压的请联系管理员! 8. 因为资源和程序源码均为可复制品,所以不支持任何理由的退款兑现,请斟酌后支付下载 声明:如果标题没有注明"已测试"或者"测试可用"等字样的资源源码均未经过站长测试.特别注意没有标注的源码不保证任何可用性