org.apache.thrift.transport.TTransportException: MaxMessageSize reached about dwh-migration-tools HOT 8 CLOSED

@paolomorandini is this related to thrift frame size? If so is it possible to set a larger default when we build? Or if changing default is not the answer, provide a recipe for how customer can run a build specifying an increase?

from dwh-migration-tools.

Hi @sguruprasad-google, it looks like you're running from a source release - that will need to download the build system (gradle) as well as dependencies.
In absence of public internet access the binary release should be used, the binary release can be found at the following location:

https://github.com/google/dwh-migration-tools/releases/latest

from dwh-migration-tools.

Hi @paolomorandini

Thanks a lot for sharing the binary.
On using the packaged binary, i am facing below issue. Can we please get any suggestions on this.

05:59:11.350 [main] WARN com.google.edwmigration.dumper.application.dumper.MetadataDumper - Task failed: Write functions.csv from get_all_functions(): org.apache.thrift.transport.TTransportException: MaxMessageSize reached
org.apache.thrift.transport.TTransportException: MaxMessageSize reached
at org.apache.thrift.transport.TEndpointTransport.checkReadBytesAvailable(TEndpointTransport.java:81)
at org.apache.thrift.protocol.TBinaryProtocol.checkStringReadLength(TBinaryProtocol.java:443)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:414)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:255)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at com.google.edwmigration.dumper.ext.hive.metastore.thrift.api.superset.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:4785)
at com.google.edwmigration.dumper.ext.hive.metastore.thrift.api.superset.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:4773)
at com.google.edwmigration.dumper.ext.hive.metastore.HiveMetastoreThriftClient_Superset.getFunctions(HiveMetastoreThriftClient_Superset.java:207)
at com.google.edwmigration.dumper.application.dumper.connector.hive.HiveMetadataConnector$FunctionsTask.run(HiveMetadataConnector.java:192)
at com.google.edwmigration.dumper.application.dumper.connector.hive.AbstractHiveConnector$AbstractHiveTask.doRun(AbstractHiveConnector.java:170)
at com.google.edwmigration.dumper.application.dumper.connector.hive.AbstractHiveConnector$AbstractHiveTask.doRun(AbstractHiveConnector.java:154)
at com.google.edwmigration.dumper.application.dumper.task.AbstractTask.run(AbstractTask.java:106)
at com.google.edwmigration.dumper.application.dumper.MetadataDumper.runTask(MetadataDumper.java:135)
at com.google.edwmigration.dumper.application.dumper.MetadataDumper.run(MetadataDumper.java:262)
at com.google.edwmigration.dumper.application.dumper.MetadataDumper.run(MetadataDumper.java:108)
at com.google.edwmigration.dumper.application.dumper.MetadataDumper.main(MetadataDumper.java:337)
05:59:11.491 [main] DEBUG com.google.edwmigration.dumper.application.dumper.MetadataDumper - Dumper wrote 27152 bytes.
05:59:11.492 [main] DEBUG com.google.edwmigration.dumper.application.dumper.MetadataDumper - Dumper took 810.4 ms.

• Task Summary:
• Task FAILED (REQUIRED) Write functions.csv from get_all_functions(): org.apache.thrift.transport.TTransportException: MaxMessageSize reached
• Task SUCCEEDED (INFORMATIONAL) Write compilerworks-arguments.txt from command-line arguments.
• Task SUCCEEDED (REQUIRED) Write compilerworks-metadata.yaml containing dump metadata.
• Task FAILED (REQUIRED) Write schemata.csv from get_all_databases(): org.apache.thrift.transport.TTransportException: MaxMessageSize reached
• Task SUCCEEDED (REQUIRED) Write compilerworks-format.txt containing 'hiveql.dump.zip'.
• Task SUCCEEDED (INFORMATIONAL) Write compilerworks-version.txt from product version information.
• Task FAILED (REQUIRED) Write tables.jsonl from get_all_databases().get_all_tables().[.get_fields(),.get_partition_keys()]: org.apache.thrift.transport.TTransportException: MaxMessageSize reached

• Metadata has been saved to dwh-migration-hiveql-metadata.zip
• To import this file, add the metadata line to your
• transpiler configuration:
• environment("my-environment-name", type: "hiveql") {
  metadata "dwh-migration-hiveql-metadata.zip"
  }

• ERROR: 3 required task[s] failed.
• Output, including debugging information, has been saved to dwh-migration-hiveql-metadata.zip

from dwh-migration-tools.

@genemerrill , @paolomorandini
Yes, please help us with the steps to increase the frame size.

from dwh-migration-tools.

Based on engineering investigating the frame size may not be the issue, FYI we are working on b/235577106 behind the scenes to fix.

from dwh-migration-tools.

@sguruprasad-google can you provide details on the Hive version? And confirm you are using v1.0.4 (or latest) version of metadata dumper release at https://github.com/google/dwh-migration-tools/releases/latest ?

from dwh-migration-tools.

@genemerrill
Below are the details

1. Hive version: 3.1.2

2. We are using the binary dwh-migration-tools-v1.0.3 from the below location:
   https://github.com/google/dwh-migration-tools/releases/latest

from dwh-migration-tools.

Fixed in release v1.0.7

from dwh-migration-tools.