Comments (6)
kjudahlookout
commented on July 3, 2024
Please let me know what more info I can provide to debug this issue. Thanks
from aws-glue-samples.
dichenli
commented on July 3, 2024
Hi Kshitij,
Thank you for using the Glue service. Here are some of my suggestions:
- Try to run Hive Metastore -> S3 migration as well, see if any data is generated in the S3 bucket. This helps to narrow down the search space for the error.
- The script hive_metastore_migration.py can migrate your Hive metastore to S3 or a local file system. It is completely open source Spark 2.1 code, you can run the script on your own Spark cluster or a local Spark installation on your laptop, as long as it's connectable to your Hive metastore. Debug locally using this script might be helpful.
- If you are familiar with Spark programming or Python, you may manually add "print()" debugging statement in the script, then run the script on Glue. It will print to the job output logs in CloudWatch.
- Double check if the data is showing up on Glue console. If it shows up on Glue console but not Athena console, you may need to migrate your Athena catalog to Glue: http://docs.aws.amazon.com/athena/latest/ug/glue-upgrade.html
- In your use case, if you are looking for a solution to sync from Hive metastore to Glue DataCatalog in near-realtime, you may consider calling Glue DataCatalog APIs directly on a stream processing workflow. You may find Glue DataCatalog APIs documentation here: http://docs.aws.amazon.com/glue/latest/dg/aws-glue-api.html.
from aws-glue-samples.
kjudahlookout
commented on July 3, 2024
Hello,
Thank you very much for your reply and sorry about the late reply. I was able add some print statements to these Python scripts, run them in Glue and able to see dataframes that is being processed by these scripts. As far as I can see, i didnt see any issue with the dataframes themselves, i was able to print their count and first 10 rows of dataframes for databases and tables and I can see all the data even after the step in which Spark dataframes get converted to Glue's dynamic frames.
However, the last step of loading the dynamic frames into glue catalog seems to be (silently) failing for me. Despite printing all data successfully in logs, the data is not getting loaded into glue by glue APIs.
I am wondering if this is something to do with my old version of hive metastore. My hive metastore version is 1.1.0 which is almost 3 years old. Is it possible that these scripts may have been developed and tested with newer versions of Hive metastore, and may not support old versions such as 1.1.0 as in my case? Pls let me know.
Thanks,
Kshitij
from aws-glue-samples.
dichenli
commented on July 3, 2024
Hmmm... this is really weird. Have you tried to migrate Hive metastore data
into S3? Could you send some sample data in S3 to my email address? Make
sure to mask the confidential fields in the data. I can try to reproduce
the issue, and debug from there.
…On Mon, Dec 11, 2017 at 4:40 PM, kjudahlookout ***@***.***> wrote:
Hello,
Thank you very much for your reply and sorry about the late reply. I was
able add some print statements to these Python scripts, run them in Glue
and able to see dataframes that is being processed by these scripts. As far
as I can see, i didnt see any issue with the dataframes themselves, i was
able to print their count and first 10 rows of dataframes for databases and
tables and I can see all the data even after the step in which Spark
dataframes get converted to Glue's dynamic frames.
However, the last step of loading the dynamic frames into glue catalog
seems to be (silently) failing for me. Despite printing all data
successfully in logs, the data is not getting loaded into glue by glue APIs.
I am wondering if this is something to do with my old version of hive
metastore. My hive metastore version is 1.1.0 which is almost 3 years old.
Is it possible that these scripts may have been developed and tested with
newer versions of Hive metastore, and may not support old versions such as
1.1.0 as in my case? Pls let me know.
Thanks,
Kshitij
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#9 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AIXWASqUdSJou4QB4XlM-M0vHt4b_u2xks5s_ctvgaJpZM4Q7prM>
.
from aws-glue-samples.
dichenli
commented on July 3, 2024
Also, you may send me your AWS account ID and job run ID through email if
you feel comfortable. I can use it to investigate our service log and see
if anything went wrong.
…On Mon, Dec 11, 2017 at 6:26 PM, Dichen Li ***@***.***> wrote:
Hmmm... this is really weird. Have you tried to migrate Hive metastore
data into S3? Could you send some sample data in S3 to my email address?
Make sure to mask the confidential fields in the data. I can try to
reproduce the issue, and debug from there.
On Mon, Dec 11, 2017 at 4:40 PM, kjudahlookout ***@***.***>
wrote:
> Hello,
> Thank you very much for your reply and sorry about the late reply. I was
> able add some print statements to these Python scripts, run them in Glue
> and able to see dataframes that is being processed by these scripts. As far
> as I can see, i didnt see any issue with the dataframes themselves, i was
> able to print their count and first 10 rows of dataframes for databases and
> tables and I can see all the data even after the step in which Spark
> dataframes get converted to Glue's dynamic frames.
>
> However, the last step of loading the dynamic frames into glue catalog
> seems to be (silently) failing for me. Despite printing all data
> successfully in logs, the data is not getting loaded into glue by glue APIs.
>
> I am wondering if this is something to do with my old version of hive
> metastore. My hive metastore version is 1.1.0 which is almost 3 years old.
> Is it possible that these scripts may have been developed and tested with
> newer versions of Hive metastore, and may not support old versions such as
> 1.1.0 as in my case? Pls let me know.
>
> Thanks,
> Kshitij
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <#9 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AIXWASqUdSJou4QB4XlM-M0vHt4b_u2xks5s_ctvgaJpZM4Q7prM>
> .
>
from aws-glue-samples.
kjudahlookout
commented on July 3, 2024
This issue seems to have been resolved for me with using --region option in the migration scripts. Thanks Dichen Li for the help.
from aws-glue-samples.
Related Issues (20)
- 'glue/sparkui:latest' missing in Docker hub HOT 3
- Issue with migrating directly from AWS Glue to Hive HOT 2
- Creating AWS- Glue Pipeline using Cloud Formation HOT 3
- Issue migrating directly from Hive Metastore to Glue Data Catalog
- Spark-UI docker container startup issue HOT 4
- hive_metastore_migration.py fails with AttributeError: 'str' object has no attribute '_jdf' HOT 1
- Unable to start Spark-UI docker container from EC2 in China Region HOT 6
- tinyint(1) issue from mysql database
- Issues using Spark_UI/glue-3_0 and Spark_UI/glue-4_0 HOT 2
- Spark UI Glue 4.0 Logging Not Working? HOT 2
- Request to Host Glue Spark UI Images on DockerHub
- Spark UI container is not getting started HOT 3
- Launch AWS Glue Spark UI Filtered to Specific Applications
- EMR Hive Metastore to Glue Migration
- Setup AWS glue
- Wrong escape character in avro.schema.url
- Couldn't resolve host name for Spark UI HOT 4
- writing data to s3 using spark and updating catalog
- Unsupported jdbc driver classname with com.ibm.as400.access.AS400JDBCDriver HOT 1
- Spark history server: README.md to show using AWS_PROFILE HOT 23
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-glue-samples.