Class clash between AWS libraries on the classpath. about secor HOT 10 CLOSED

Robert,

I think you can exclude the aws-java-sdk-s3 when pulling in
org.apache.hadoop dependency in pom.xml. Take a look at the coming PR:
https://github.com/pinterest/secor/pull/159/files, I think that probably
will fix your problem.

On Thu, Dec 17, 2015 at 4:31 AM, Robert Firek [email protected]
wrote:

I've upgrade Secor to 0.9 and I've switched to s3a file system.

After upgrade I've started to receive the following errors (stacktrace can
be in a little mess, because I've retrieved from our temporary log system) :

INFO: Cleaning up!
INFO: Cleaning up this locks ephemeral node.
Dec 17, 2015 10:30:29 AM com.twitter.common.zookeeper.DistributedLockImpl unlock
Dec 17, 2015 10:30:29 AM com.twitter.common.zookeeper.DistributedLockImpl cleanup
2015-12-17 10:30:29,299 Thread-3 ERROR Thread Thread[Thread-3,5,main] failed
java.lang.RuntimeException: Failed to apply upload policy
at com.pinterest.secor.uploader.HadoopS3UploadManager$1.run(HadoopS3UploadManager.java:57)
at com.pinterest.secor.util.FileUtil.moveToCloud(FileUtil.java:156)
at com.pinterest.secor.util.FileUtil.getFileSystem(FileUtil.java:71)
at com.pinterest.secor.uploader.Uploader.uploadFiles(Uploader.java:95)
at com.pinterest.secor.uploader.Uploader.checkTopicPartition(Uploader.java:186)
at com.pinterest.secor.uploader.Uploader.applyPolicy(Uploader.java:207)
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:112)
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:110)
at com.pinterest.secor.consumer.Consumer.run(Consumer.java:102)
at com.pinterest.secor.uploader.FutureHandle.get(FutureHandle.java:34)
Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:285)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.lang.Thread.run(Thread.java:745)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

Problems is with AWS libs:

• Secor requires com.amazonaws:aws-java-sdk-s3:1.10.2
  (com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold
  method has Long parameter)
• Hadoop lib org.apache.hadoop:hadoop-aws:2.7.0 requires
  com.amazonaws:aws-java-sdk:1.7.4
  (com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold
  method has int parameter)

—
Reply to this email directly or view it on GitHub
#160.

from secor.

Great. I will wait for it.

from secor.

I just merged in that PR, you can give it a try.

On Fri, Dec 18, 2015 at 2:46 AM, Robert Firek [email protected]
wrote:

Great. I will wait for it.

—
Reply to this email directly or view it on GitHub
#160 (comment).

from secor.

Is it possible to release it to Maven Repo?

from secor.

I just cut 0.11, wait for one hour, it should be there.

On Sun, Dec 20, 2015 at 3:38 AM, Robert Firek [email protected]
wrote:

Is it possible to release it to Maven Repo?

—
Reply to this email directly or view it on GitHub
#160 (comment).

from secor.

I'm having the same error, I'm building Secor from source with mvn package then I use the content of the secor-0.X-SNAPSHOT-bin.tar.gz.

from secor.

Which version of secor are you using? Can you try the latest one?

On Fri, Feb 5, 2016 at 7:11 AM, El-Bech [email protected] wrote:

I'm having the same error, I'm building Secor from source with mvn package
then I use the content of the secor-0.X-SNAPSHOT-bin.tar.gz.

—
Reply to this email directly or view it on GitHub
#160 (comment).

from secor.

In the latest version, aws-java-sdk-s3 is excluded in the pom.xml however I still see the exception when I use s3a!

from secor.

I'm hitting the same problem as well with s3a.

s3a://fsm-haas/raw_logs/secor_backup/rawxxx/offset=0/1_1_00000000000000000000
2016-03-16 23:35:30,442 Thread-4 ERROR Thread Thread[Thread-4,5,main] failed
java.lang.RuntimeException: Failed to apply upload policy
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:115)
at com.pinterest.secor.consumer.Consumer.run(Consumer.java:105)
Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.pinterest.secor.uploader.FutureHandle.get(FutureHandle.java:34)
at com.pinterest.secor.uploader.Uploader.uploadFiles(Uploader.java:95)
at com.pinterest.secor.uploader.Uploader.checkTopicPartition(Uploader.java:186)
at com.pinterest.secor.uploader.Uploader.applyPolicy(Uploader.java:207)
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:113)
... 1 more
Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:285)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at com.pinterest.secor.util.FileUtil.getFileSystem(FileUtil.java:71)
at com.pinterest.secor.util.FileUtil.moveToCloud(FileUtil.java:156)
at com.pinterest.secor.uploader.HadoopS3UploadManager$1.run(HadoopS3UploadManager.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

from secor.

I dont think there is a solution for this issue now :-(
Pls check:
https://issues.apache.org/jira/browse/HADOOP-12420
http://deploymentzone.com/2015/12/20/s3a-on-spark-on-aws-ec2/

We will have to wait till hadoop-aws-2.8

from secor.