Comments (1)
RobertIndie
commented on July 18, 2024
The configuration field should be sourceSubscriptionPosition instead of subscriptionPosition .
The correct yaml config file should be ：
tenant: "public"
namespace: "default"
name: "cloud-storage-sinktest"
inputSpecs:
persistent://public/sinktest/.*:
regexPattern: true
persistent://public/sinktest2/.*:
regexPattern: true
archive: "builtin://cloud-storage-test"
parallelism: 1
autoAck: true
cleanupSubscription: true
sourceSubscriptionPosition: "Earliest"
configs:
role: "arn:aws:iam::$$$$$$$$$$$$$:role/pulsar-storage-connector-role"
roleSessionName: "name"
provider: "s3v2"
bucket: "$$$$$$$$$$$$$"
region: "us-west-2"
formatType: "json"
pathPrefix: "sinktest/"
partitionerType: "GLOBAL_TIME"
batchSize: 15000
batchTimeMs: 15000
withMetadata: true
useHumanReadableMessageId: trueVerification
I have verified, and it works now.
Follow-up
This could reproduce both in pulsar-admin and pulsarctl. I think it's better to improve the user experience here. If users pass an incorrect field, we should report the error.
from pulsar-io-cloud-storage.
Related Issues (20)
- [BUG] extract schema info from messages without schema version
- [BUG] Time Partitioner not working as expected for GCS HOT 1
- [BUG] failed to extract schema version for AVRO format and causing cast error
- [BUG] error while writing data in Parquet format using JsonSchema as schema format HOT 3
- [FEATURE] Add a new batch triggering mechanism that is based on the batch size
- [FEATURE] Pending Queuesize default to the maxbatch size
- [BUG] Connector does not create output files for each topic when reading multiple topics
- [FEATURE] Azure Blob Storage support
- [BUG] PR template should correctly add doc labels based on description
- [BUG] formatType not working as expected HOT 1
- [BUG] Sink flushes only when batchTimeMs is exceeded for large batchSize
- [BUG] Files generated per topic have fluctuating size instead of `batchSize`
- [BUG] `maxTimeoutMs` should apply at the topic level
- Update master to JDK17 HOT 1
- [BUG] NPE if partitionerType is not set
- [BUG] The connector covert array json bytes failed.
- [FEATURE] add configuration for including the original topic in the metadata when withMetadata is set to true
- [BUG] NPE when schema not exist with useHumanReadableSchemaVersion enabled
- [FEATURE] Add datetime string substitutions to prefix field
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pulsar-io-cloud-storage.