Comments (8)
dovka
commented on July 18, 2024
2
Henry,
thank you for your advice -
switching to s3a and HadoopS3UploadManager did the trick -
secor started implicitly using instance profile :)
from secor.
HenryCaiHaiying
commented on July 18, 2024
1
You want to make sure there is nothing set for aws configs. You might need
to trace the code a little bit for that. You can also try between
HadoopS3UploadManager.java and S3UploadManager.java
…On Mon, Feb 22, 2021 at 5:04 AM David ***@***.***> wrote:
Hi guys,
we are prohibited form using the hardcoded AWS credentials
(that we had been using until now) and trying to make secor work with AWS
instance profile or role.
I didn't find any documentation but from the secor.common.properties
comment
we infer that we need to leave the credentials blank and set the role name.
`# AWS authentication credentials.
"
Leave empty if using IAM role-based authentication with s3a filesystem.
aws.access.key=
aws.secret.key=
Session token only required if using temporary S3 access keys
aws.session.token=
aws.role=
`
We specified the role name (not sure what to do with aws.session.token)
but still get the same error:
2021-02-22 12:55:27,900 [main] (com.pinterest.secor.main.ConsumerMain:87)
ERROR Consumer failed java.lang.RuntimeException: Failed to find required
configuration option 'aws.access.key'. at
com.pinterest.secor.common.SecorConfig.checkProperty(SecorConfig.java:725)
at com.pinterest.secor.common.SecorConfig.getString(SecorConfig.java:731)
at
com.pinterest.secor.common.SecorConfig.getAwsAccessKey(SecorConfig.java:417)
at com.pinterest.secor.util.FileUtil.configure(FileUtil.java:75) at
com.pinterest.secor.main.ConsumerMain.main(ConsumerMain.java:70)
how to make secor use instance profile or role?
thank you for your help!
David
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#1873>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABYJP77F265ADX7NWVKVFFTTAJI5RANCNFSM4YAPPB7A>
.
from secor.
HenryCaiHaiying
commented on July 18, 2024
Glad it worked out. You can also update README on how to setup secor for
AWS IAM roles
…On Tue, Feb 23, 2021 at 5:22 AM David ***@***.***> wrote:
Henry,
thank you for your advice -
switching to s3a and HadoopS3UploadManager did the trick -
secor started implicitly using instance profile :)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1873 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABYJP77QCMK7K6X6RWOK6F3TAOTX7ANCNFSM4YAPPB7A>
.
from secor.
dovka
commented on July 18, 2024
Henry,
I will gladly do update the README file.
How do I do it - via pull request or just attach here?
We put the IAM roles authentication config in production yesterday and since then we see duoble the CPU usage by secor.
We need twice the amount of nodes (had 3 now need 6 or we are falling behind with data being queued up)
The only thing we changed is authentication.
Is it possible the the code checking on re-authentication (AWS token expiration) all the time and causes this CPU drain?
Thank you for your help Henry
from secor.
HenryCaiHaiying
commented on July 18, 2024
You can provide a PR for README file change.
For CPU usage increase, a quick way to verify is usually doing a few thread
dumps while the process is running to see where the CPU is spent most time
on. Otherwise tools like flame graph or some JVM profiling tool (Yourkit?)
can help too.
…On Tue, Mar 2, 2021 at 9:13 AM David ***@***.***> wrote:
Henry,
I will gladly do update the README file.
How do I do it - via pull request or just attach here?
We put the IAM roles authentication config in production yesterday and
since then we see duoble the CPU usage by secor.
We need twice the amount of nodes (had 3 now need 6 or we are falling
behind with data being queued up)
The only thing we changed is authentication.
Is it possible the the code checking on re-authentication (AWS token
expiration) all the time and causes this CPU drain?
Thank you for your help Henry
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1873 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABYJP7ZQ3KHRF7HEW5YJGDDTBUMCLANCNFSM4YAPPB7A>
.
from secor.
dovka
commented on July 18, 2024
Henry,
we run into an issue with secor instance profile in AWS China region.
The S3 code that works in regular AWS breaks in AWS China with this error:
com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: SMY7YNMPSYCKSKNJ
I asked for the clarification from AWS Support, and here is their answer:
"This request is sent to Amazon Global region, not China region;
So let us check the region setting first.
When using resources in China region, we must clearly specify the region information.
I saw you use Amazon Java SDK V1, you could use below code to set the region:
AmazonS3 s3 = AmazonS3ClientBuilder.standard().withRegion(Regions.CN-NORTH-1).build();
"
Would you please check the code,
how easy would it be to patch it up?
Thank you for your help!
David
from secor.
dovka
commented on July 18, 2024
created pull request #2095 with instructions on instance profile setup
from secor.
HenryCaiHaiying
commented on July 18, 2024
I merged your PR, looks like you have at least a workaround.
About code code, the place we are creating S3Client is in:
src/main/java/com/pinterest/secor/uploader/S3UploadManager.java
client = new AmazonS3Client(provider, clientConfiguration);
You can take a look to see whether it's possible to pass in the region
information in clientConfiguration or chaining with that withRegion() call
…On Wed, Jun 2, 2021 at 3:59 AM David ***@***.***> wrote:
created pull request #2095 <#2095>
with instructions on instance profile setup
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1873 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABYJP72MJ2HR2W2FEP2A4OLTQYFI7ANCNFSM4YAPPB7A>
.
from secor.
Related Issues (20)
- Get exception when using micrometer HOT 4
- Blocking Issue when running on K8s on GCP HOT 1
- manually assign avro schema to topic HOT 1
- Crashing exception after some time HOT 1
- Cannot parse timestamp with timezone HOT 1
- Racing condition with parquet writer HOT 3
- Ignore parsing exceptions HOT 2
- Expose JVM metrics when using Micrometer HOT 3
- Secor uploads (different) files with the same "name" into different "days". HOT 2
- Kuberntes OOM Error. Very High memory usage > 15gb HOT 2
- Is there any alternative to AvroParquetWriter so that we can avoid using IndexedRecord? HOT 1
- ParquetWriter write method is taking too long to write single avro message HOT 1
- A Simple Study about Configuration Java Types
- PartitionFinalizer fails when there is no data produced in the topic eventhough there are other topics with data HOT 3
- Add headers while calling schema registry for decoding messages HOT 1
- Duplicate message when partition revoking or rebalancing HOT 1
- secor-monitor container not getting started. HOT 2
- Tune file size create by Secor
- Secor with zookeerper less Kafka HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from secor.