Lately we have been building a new data warehouse built upon the AWS Infrastructure at work. It has been going quite smoothly, but for the occasional ups and downs. However, one thing would never quite work: Accessing S3 content from a (py)spark job that is run locally.
S3 access from AWS EMR cluster instances was quite easy.
Simply providing the file system context s3a://
sufficed and everything worked smoothly.
The same method on a local machine with a spark installation proved as quite cumbersome and made us encounter errors such as:
AttributeError: 'SparkConf' object has no attribute '_get_object_id'
comment here
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
comment here
java.io.IOException: No FileSystem for scheme: s3a
comment here
java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
bump down aws-java-sdk-x.x.x.jar to 1.7.4, see here
...
We went through some pain and decided to put that aspect on ice for the time being. However, during some slack time I decided to start going at it once again. Luckily I had some luck and my efforts paid off.
Since I have spent some time banging my head, I tought I might make it easier on some people by providing some easy examples and verbose comments.
I have provided examples and readmes both for Scala and Python. I hope this helps some people and wish you luck with your big data infrastructure!