Support fror s3a hadoop file system is broken about secor HOT 10 CLOSED

Good catch. Can you prepare a PR for this fix?

On Fri, Dec 11, 2015 at 8:59 AM, Robert Firek [email protected]
wrote:

I've tried use s3a file system in my configuration, but I've get error:

java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key
must be specified as the username or password (respectively) of a s3n URL,
or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey
properties (respectively).

After short investigation I've noticed that support for s3a introduced in
commit Support s3a hadoop file system.
e00fac6
was accidentally destroyed by next commit Openstack Swift integration
049379a
.

In the class com.pinterest.secor.uploaderHadoopS3Uploader the method
mConfig.getS3Prefix() was replaced with
com.pinterest.secor.util.FileUtil.getPrefix() which doesn't support
"s3a". Probably only change which have to be made to restore s3a support is
change in the com.pinterest.secor.util.FileUtil.getPrefix() method.

Line 90:

prefix = "s3n://" + config.getS3Bucket() + "/" + config.getS3Path();

should be changed to:

prefix = config.getS3Prefix();

—
Reply to this email directly or view it on GitHub
#153.

from secor.

@HenryCaiHaiying Done "Restore support for s3a file system" #155

How often do you release? I would like to use this feature as soon as possible.

from secor.

@HenryCaiHaiying When do you plan to release the new version with this change?

from secor.

Do you want me to cut a new maven version with this change?

On Mon, Dec 14, 2015 at 2:38 AM, Robert Firek [email protected]
wrote:

@HenryCaiHaiying https://github.com/HenryCaiHaiying When do you plan to
release the new version with this change?

—
Reply to this email directly or view it on GitHub
#153 (comment).

from secor.

In our project we use Jar file from the maven repository. If you can release new version to the maven repository I can avoid creating my own distribution.

from secor.

I cut 0.9 version, it should show up in maven central after one hour.

On Mon, Dec 14, 2015 at 1:55 PM, Robert Firek [email protected]
wrote:

In our project we use Jar file from the maven repository. If you can
release new version to the maven repository I can avoid creating my own
distribution.

—
Reply to this email directly or view it on GitHub
#153 (comment).

from secor.

@HenryCaiHaiying Thank you

from secor.

why this issue is still open? has the fixe not been yet merged?? I'm trying to use s3a (to avoid putting AWS credentials in the properties file and instead use AMI instance role).
I'm seeing this problem No broker partions consumed by consumer thread secor_backup_X for topic Y.

from secor.

Which version of secor are you using? We don't use s3a here, but according
to Robert, the problem is already fixed.

On Fri, Feb 5, 2016 at 2:48 AM, El-Bech [email protected] wrote:

why this issue is still open? has the fixe not been yet merged?? I'm
trying to use s3a (to avoid putting AWS credentials in the properties file
and instead use AMI instance role).

—
Reply to this email directly or view it on GitHub
#153 (comment).

from secor.

@HenryCaiHaiying I was using the latest version with s3a, but when I noticed this issue I thought the support is broken which is not the case.
Now, my problem is No broker partions consumed by consumer thread secor_backup_X for topic Y, And Secor is not archiving in S3. Any hint to fix that?

from secor.