google / dwh-migration-tools Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
Update Hive connector to extract Hive table StorageDescriptor details for SerDe and Input/Output formats. This will help in importing hive table metadata into DPMS.
When executing the dwh-migration-tools for Snowflake, the follow error is returned:
********************************************************************
* ERROR
* StatementCallback; uncategorized SQLException for SQL [SELECT CURRENT_DATABASE()]; SQL state [XX000]; error code [200001]; JDBC driver internal error: exception creating result java.lang.ExceptionInInitializerError at net.snowflake.client.jdbc.internal.apache.arrow.memory.UnsafeAllocationManager.<clinit>(UnsafeAllocationManager.java:27).; nested exception is net.snowflake.client.jdbc.SnowflakeSQLException: JDBC driver internal error: exception creating result java.lang.ExceptionInInitializerError at net.snowflake.client.jdbc.internal.apache.arrow.memory.UnsafeAllocationManager.<clinit>(UnsafeAllocationManager.java:27).
* Caused by: JDBC driver internal error: exception creating result java.lang.ExceptionInInitializerError at net.snowflake.client.jdbc.internal.apache.arrow.memory.UnsafeAllocationManager.<clinit>(UnsafeAllocationManager.java:27).
********************************************************************
Issue seems to be related to the following:
https://community.snowflake.com/s/article/JDBC-Driver-Compatibility-Issue-With-JDK-16-and-Later
During testing the migration-tool we were blocked by a strange error message: Error while Calculating GCS size for workflow projects/30322705XXXX/locations/eu/workflows/90ae6fa8-80b7-459f-9f4e-aa24ccd1xxxx: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Max length for bucket name without dots exceeded. Max length='63', actual='101'.
The built source and target paths are: project-id\Translation_HiveQL2BQ-2022-07-26-5484364d-d729-469d-a736-1d8812744008\
It seems that the Windows path.separator is used by the tool and the GCS considers the whole path as bucket name.
Some of the output files still have the "CompilerWorks" term in the name. And I also see them in this open source directory as well. For example:
For legal reasons, maybe we should remove these legal sensitive terms or rename them.
Hi Team,
We are trying to use the Metadata dumper in a secured environment(without internet access).
We are facing below issue, please help us.
Could not resolve all artifacts for configuration ':dumper:app:classpath'.
Could not resolve com.github.jk1:gradle-license-report:1.17.
Required by:
project :dumper:app
> Could not resolve com.github.jk1:gradle-license-report:1.17.
> Could not get resource 'https://plugins.gradle.org/m2/com/github/jk1/gradle-license-report/1.17/gradle-license-report-1.17.pom'.
> Could not GET 'https://plugins.gradle.org/m2/com/github/jk1/gradle-license-report/1.17/gradle-license-report-1.17.pom'.
> Connect to plugins.gradle.org:443 [plugins.gradle.org/104.18.191.9, plugins.gradle.org/104.18.190.9, plugins.gradle.org/2606:4700:0:0:0:0:6812:bf09, plugins.gradle.org/2606:4700:0:0:0:0:6812:be09] failed: Network is unreachable (connect failed)
Could not get unknown property 'installPublishedDist' for task set of type org.gradle.api.internal.tasks.DefaultTaskContainer.
Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
Get more help at https://help.gradle.org
Thanks
When executing the dw-migration-client with metadata generated by dwh-migration-dumper the following errors are produced in the batch_translation_report.csv:
BigQuery does not support INDEX-backed constraints such as PRIMARY KEY and UNIQUE.
BigQuery does not support set tables.
BigQuery cluster-expression 'dataparcel' is of unsupported type VARBINARY
BigQuery cluster-expression 'pnch_in_tim' is of unsupported type TIME
BigQuery cluster-expression 'txn_tim' is of unsupported type TIME
BigQuery does not support more than 4 CLUSTER BY expressions, got X
Unsupported RANGE partition: RANGE partition supported only for Integers. Using auto partitioning for DATE
Ideally these conflicts would be automatically resolved for example:
Removal of Primary and Unique keys definitions
Remove qualifiers for set and multiset in ddl
Reducing clustered fields to the limit
Removing unsupported partition definitions
Calling the dumper with --driver ""
(e.g. due to a bad environment variable on the command line) throws an exception like:
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: 0
at java.base/java.lang.StringLatin1.charAt(StringLatin1.java:47)
at java.base/java.lang.String.charAt(String.java:693)
at com.google.edwmigration.dumper.application.dumper.JsonResponseFile.addResponseFiles(JsonResponseFile.java:61)
at com.google.edwmigration.dumper.application.dumper.MetadataDumper.main(MetadataDumper.java:319)
This error message is unclear what the problem is, making it hard for the user to debug.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.