Comments (5)
Thank you for using AWS Glue.
If the tables are missing, did the script create any database in your Glue Data Catalog? You may find the list of databases on Glue console.
Another way to check if the tables are showing up is to use AWS CLI to call Glue service directly. For example, you could try the following shell commands:
$ aws glue get-database --name myHiveMetastore_default
$ aws glue get-table --database-name myHiveMetastore_default --name myHiveMetastore_customers
for instructions to install or update AWS CLI, you may refer to http://docs.aws.amazon.com/cli/latest/userguide/installing.html
The script hive_metastore_migration.py is completely open-source Spark code, so you could directly test and debug the script using Spark local mode. But the script import_into_datacatalog.py include Glue specific library, so you need to run it from a Glue job. You could modify the script to add print line statements to see the job output though.
If you have further questions, you can also use the AWS Glue Forum: https://forums.aws.amazon.com/forum.jspa?forumID=262.
from aws-glue-samples.
Thanks dichenli for prompt response.
So firstly, yes database is also missing. In my previous post, I had attached database and tables files generated by 2nd approach and exported metastore to S3.
Both these things are not visible in Glue catalog after running direct from Hive to Glue.
I ran hive_metastore_migration.py and it is working fine because it created files in S3 using 2nd approach (attached in previous post).
It is import_into_datacatalog.py which is not working and problem is that no error is there. Job succeeds with no error logs but tables and databases are not seen. I have not edited a single char. in the script.
Yes I would be trying now with exhaustive print statements now.
from aws-glue-samples.
I tried to run the migration job from S3 to Glue, with the databases and tables files you provided. It succeeded and the databases and tables correctly show up in my Glue console. Could you check if your job configuration is the same as mine?
Script Path: s3://someBucket/import_into_datacatalog.py
Temporary directory: s3://someBucket
Required connections: empty (for migration from S3)
Python library path: s3://someBucket/hive_metastore_migration.py
Job parameters:
-m: from-s3
-D: s3://someBucket/output_from_previous_job/databases
-T: s3://someBucket/output_from_previous_job/tables
-P: s3://someBucket/output_from_previous_job/partitions
Notice the -m, -D, -T and -P are short names for --mode, --database-input-path, --table-input-path, --partition-input-path, you could use either.
I manually created the s3://someBucket/output_from_previous_job/databases, tables and partitions folders, and uploaded databases.txt and tables.txt files to the respective folders. partitions folder is empty.
from aws-glue-samples.
Hi Anshul,
I'll close the issue. If you are still blocked on the same problem, please feel free to reopen it. If you have any new questions, you may open a new issue, or post on AWS Glue forum https://forums.aws.amazon.com/forum.jspa?forumID=262. Thank you!
from aws-glue-samples.
Hello Anshul and Dichen,
I am having the same issue when I use the scripts to directly migrate from hive metastore to glue data catalog. The job runs successfully in Glue but I don't see anything migrated to Glue data catalog. I am wondering if you guys were able to resolve this issue. Help on this issue is very appreciated.
Thanks,
Kshitij
from aws-glue-samples.
Related Issues (20)
- 'glue/sparkui:latest' missing in Docker hub HOT 3
- Issue with migrating directly from AWS Glue to Hive HOT 2
- Creating AWS- Glue Pipeline using Cloud Formation HOT 3
- Issue migrating directly from Hive Metastore to Glue Data Catalog
- Spark-UI docker container startup issue HOT 4
- hive_metastore_migration.py fails with AttributeError: 'str' object has no attribute '_jdf' HOT 1
- Unable to start Spark-UI docker container from EC2 in China Region HOT 6
- tinyint(1) issue from mysql database
- Issues using Spark_UI/glue-3_0 and Spark_UI/glue-4_0 HOT 2
- Spark UI Glue 4.0 Logging Not Working? HOT 2
- Request to Host Glue Spark UI Images on DockerHub
- Spark UI container is not getting started HOT 3
- Launch AWS Glue Spark UI Filtered to Specific Applications
- EMR Hive Metastore to Glue Migration
- Setup AWS glue
- Wrong escape character in avro.schema.url
- Couldn't resolve host name for Spark UI HOT 4
- writing data to s3 using spark and updating catalog
- Unsupported jdbc driver classname with com.ibm.as400.access.AS400JDBCDriver HOT 1
- Spark history server: README.md to show using AWS_PROFILE HOT 23
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-glue-samples.