Comments (10)
Good catch. Can you prepare a PR for this fix?
On Fri, Dec 11, 2015 at 8:59 AM, Robert Firek [email protected]
wrote:
I've tried use s3a file system in my configuration, but I've get error:
java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key
must be specified as the username or password (respectively) of a s3n URL,
or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey
properties (respectively).After short investigation I've noticed that support for s3a introduced in
commit Support s3a hadoop file system.
e00fac6
was accidentally destroyed by next commit Openstack Swift integration
049379a
.In the class com.pinterest.secor.uploaderHadoopS3Uploader the method
mConfig.getS3Prefix() was replaced with
com.pinterest.secor.util.FileUtil.getPrefix() which doesn't support
"s3a". Probably only change which have to be made to restore s3a support is
change in the com.pinterest.secor.util.FileUtil.getPrefix() method.Line 90:
prefix = "s3n://" + config.getS3Bucket() + "/" + config.getS3Path();
should be changed to:
prefix = config.getS3Prefix();
—
Reply to this email directly or view it on GitHub
#153.
from secor.
@HenryCaiHaiying Done "Restore support for s3a file system" #155
How often do you release? I would like to use this feature as soon as possible.
from secor.
@HenryCaiHaiying When do you plan to release the new version with this change?
from secor.
Do you want me to cut a new maven version with this change?
On Mon, Dec 14, 2015 at 2:38 AM, Robert Firek [email protected]
wrote:
@HenryCaiHaiying https://github.com/HenryCaiHaiying When do you plan to
release the new version with this change?—
Reply to this email directly or view it on GitHub
#153 (comment).
from secor.
In our project we use Jar file from the maven repository. If you can release new version to the maven repository I can avoid creating my own distribution.
from secor.
I cut 0.9 version, it should show up in maven central after one hour.
On Mon, Dec 14, 2015 at 1:55 PM, Robert Firek [email protected]
wrote:
In our project we use Jar file from the maven repository. If you can
release new version to the maven repository I can avoid creating my own
distribution.—
Reply to this email directly or view it on GitHub
#153 (comment).
from secor.
@HenryCaiHaiying Thank you
from secor.
why this issue is still open? has the fixe not been yet merged?? I'm trying to use s3a (to avoid putting AWS credentials in the properties file and instead use AMI instance role).
I'm seeing this problem No broker partions consumed by consumer thread secor_backup_X for topic Y.
from secor.
Which version of secor are you using? We don't use s3a here, but according
to Robert, the problem is already fixed.
On Fri, Feb 5, 2016 at 2:48 AM, El-Bech [email protected] wrote:
why this issue is still open? has the fixe not been yet merged?? I'm
trying to use s3a (to avoid putting AWS credentials in the properties file
and instead use AMI instance role).—
Reply to this email directly or view it on GitHub
#153 (comment).
from secor.
@HenryCaiHaiying I was using the latest version with s3a, but when I noticed this issue I thought the support is broken which is not the case.
Now, my problem is No broker partions consumed by consumer thread secor_backup_X for topic Y, And Secor is not archiving in S3. Any hint to fix that?
from secor.
Related Issues (20)
- Blocking Issue when running on K8s on GCP HOT 1
- manually assign avro schema to topic HOT 1
- Crashing exception after some time HOT 1
- Cannot parse timestamp with timezone HOT 1
- Racing condition with parquet writer HOT 3
- Cannot make secor work with AWS instance profile HOT 8
- Ignore parsing exceptions HOT 2
- Expose JVM metrics when using Micrometer HOT 3
- Secor uploads (different) files with the same "name" into different "days". HOT 2
- Kuberntes OOM Error. Very High memory usage > 15gb HOT 2
- Is there any alternative to AvroParquetWriter so that we can avoid using IndexedRecord? HOT 1
- ParquetWriter write method is taking too long to write single avro message HOT 1
- A Simple Study about Configuration Java Types
- PartitionFinalizer fails when there is no data produced in the topic eventhough there are other topics with data HOT 3
- Add headers while calling schema registry for decoding messages HOT 1
- Duplicate message when partition revoking or rebalancing HOT 1
- secor-monitor container not getting started. HOT 2
- Tune file size create by Secor
- Secor with zookeerper less Kafka HOT 1
- Secor filling up local disk space with intermittent data
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from secor.