Git Product home page Git Product logo

datasourcex's Introduction

数据源插件 common-loader

一、介绍

模拟 Driver 通过不同的 classloader 加载,避免类名冲突导致加载不上的问题,提供大量关系型数据库、非关系型数据库以及像 HDFS、S3、ftp数据源的一些通用方法,简化开发成本:


二、已支持的数据源

2.1 关系型数据库

Database Type Client doc
MySQL5 doc
MySQL8 doc
Polardb_For_MySQL(同mysql5) doc
Oracle doc
SQLServer doc
PostgreSQL doc
DB2 doc
DMDB doc
KINGBASE8 doc
HIVE1.X doc
HIVE2.X doc
SPARK doc
IMPALA doc
INCEPTOR doc
Clickhouse doc
TiDB(同mysql5) doc
CarbonData(同hive2.X) doc
Kudu doc
ADS(同mysql5) doc
Kylin doc
Libra doc
GREENPLUM6 doc
GBase_8a doc
Phoenix4 doc
Phoenix5 doc
oceanbase doc
Vertica doc

2.2 非关系型数据库

Database Type Client doc
REDIS doc
HBASE(hbase1.x、2.x版本) doc
ES(ES6、ES7版本) doc
SOLR(SOLR7.4版本) doc
MONGODB doc
HBASE doc

2.3 文件系统

Database Type Client doc
HDFS doc
S3(此为**移动版本) doc
AWS_S3 doc
FTP doc

2.4 消息队列

Database Type Client doc
KAFKA(0.9、0.10、0.11、1.x版本) doc
EMQ doc

2.5 其他

Database Type Client doc
websocket doc
socket doc
influxdb doc

三、立刻使用

请点击立刻使用


四、kerberos认证

请点击kerberos认证


五、开发步骤说明

  1. 在core模块下com.dtstack.dtcenter.loader.source.DataSourceType中增加新的数据源信息
  2. 在core模块下com.dtstack.dtcenter.loader.dto.source包中创建新数据源对应的sourceDTO,继承对应的抽象类或实现ISourceDTO接口
  3. 在项目下增加子模块并按照其他模块修改项目和子模块对应的pom文件信息
  4. 在新创建的模块中创建client类继承对应抽象类或实现对应接口并重写需要实现的方法
  5. 在Resources/META-INF/services 下增加文件com.dtstack.dtcenter.loader.client.IClient,并在里面补充实现类的引用地址:例如:com.dtstack.dtcenter.common.loader.db2.Db2Client

datasourcex's People

Contributors

anpetrichor avatar hpgd avatar wangchuanpoxiao avatar yangsishu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

datasourcex's Issues

[BUG] pg14 getCreateTableSql Error

数据库版本:14.2 (Debian 14.2-1.pgdg110+1)

postgreSQL 14版本时,postgresql 插件的getCreateTableSql 方法会报错。


com.dtstack.dtcenter.loader.exception.DtLoaderException: SQL execute exception:ERROR: column "consrc" does not exist
  建议:Perhaps you meant to reference the column "c.conkey" or the column "c.conbin".
  位置:143

	at com.dtstack.dtcenter.loader.ClassLoaderCallBackMethod.callbackAndReset(ClassLoaderCallBackMethod.java:36)
	at com.dtstack.dtcenter.loader.client.sql.DataSourceClientProxy.getCreateTableSql(DataSourceClientProxy.java:155)
	at com.dtstack.dtcenter.loader.client.sql.PostgreSQLTest.getCreateTable(PostgreSQLTest.java:355)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:221)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
Caused by: com.dtstack.dtcenter.loader.exception.DtLoaderException: SQL execute exception:ERROR: column "consrc" does not exist
  建议:Perhaps you meant to reference the column "c.conkey" or the column "c.conbin".
  位置:143
	at com.dtstack.dtcenter.common.loader.common.utils.DBUtil.executeQuery(DBUtil.java:123)
	at com.dtstack.dtcenter.common.loader.rdbms.AbsRdbmsClient.executeQuery(AbsRdbmsClient.java:153)
	at com.dtstack.dtcenter.common.loader.rdbms.AbsRdbmsClient.executeQuery(AbsRdbmsClient.java:163)
	at com.dtstack.dtcenter.common.loader.postgresql.PostgresqlClient.getCreateTableSql(PostgresqlClient.java:330)
	at com.dtstack.dtcenter.loader.client.sql.DataSourceClientProxy.lambda$getCreateTableSql$17(DataSourceClientProxy.java:155)
	at com.dtstack.dtcenter.loader.ClassLoaderCallBackMethod.callbackAndReset(ClassLoaderCallBackMethod.java:34)
	... 25 more
Caused by: org.postgresql.util.PSQLException: ERROR: column "consrc" does not exist
  建议:Perhaps you meant to reference the column "c.conkey" or the column "c.conbin".
  位置:143
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
	at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:266)
	at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:95)
	at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
	at com.dtstack.dtcenter.common.loader.common.utils.DBUtil.executeQuery(DBUtil.java:98)
	... 30 more

使用MySQL的下载器的时候报错

com.dtstack.dtcenter.loader.dto.source.Mysql8SourceDTO cannot be cast to com.dtstack.dtcenter.loader.dto.source.Mysql5SourceDTO

目标MySQL版本是8.0.31

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.