Question: Group by key about aws-glue-samples HOT 1 CLOSED

Thanks for trying AWS Glue. Here is a solution for your problem.

First, you can refer the following blog post on how to partition a dynamic frame into Hive-style s3 partitions based on key values:

https://aws.amazon.com/blogs/big-data/work-with-partitioned-data-in-aws-glue/

glueContext.write_dynamic_frame.from_options(
frame = datasource0,
connection_type = "s3",
connection_options = {"path": "s3://my-result-bucket/sub-bucket", "partitionKeys" : ["key1"]},
format = "json",
transformation_ctx = "datasink2")

Now if you would like to coalesce all files in each of these written Hive-style partition into a single file, you can create another dynamic frame using the Grouping feature in Glue that could read a large number of files from these Hive-style partitions into a group:

https://docs.aws.amazon.com/glue/latest/dg/grouping-input-files.html

datasource1 = glueContext.create_dynamic_frame_from_options("s3", {'paths': ["s3://my-result-bucket/sub-bucket/"], "recurse": True, 'groupFiles': 'inPartition', 'groupSize': 1024 * 1024}, format="json")

Finally, you can write out this dynamic frame datasource1 out to individual s3 files, where each output file would be roughly the same as groupSize. Make sure that your groupSize is larger than the size of each Hive-style partition, so that you have exactly 1 file per Hive-style partition or key value.

from aws-glue-samples.