kafkaesque-io / pulsar-beam Goto Github PK
View Code? Open in Web Editor NEWPulsar Beam is a streaming service via HTTP built on Apache Pulsar.
License: Apache License 2.0
Pulsar Beam is a streaming service via HTTP built on Apache Pulsar.
License: Apache License 2.0
The detail is explained by
https://github.community/t/make-secrets-available-to-builds-of-forks/16166/20
We need to rewrite the test cases to avoid using secrets
Hello,
while trying to build your image I encountered the following error:
D:\projects\pulsar\pulsar-beam (master -> origin)
λ docker build -t pulsar-beam .
Sending build context to Docker daemon 237.6kB
Step 1/13 : FROM golang:alpine AS builder
---> 3289bf11c284
Step 2/13 : LABEL maintainer="kafkaesque.io"
---> Using cache
---> b0b2cacac0a7
Step 3/13 : RUN apk --no-cache add build-base git bzr mercurial gcc
---> Running in c4cc97e18153
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
bzr (missing):
required by: world[bzr]
The command '/bin/sh -c apk --no-cache add build-base git bzr mercurial gcc' returned a non-zero code: 1
I tried to check why bazaar is missing - but could not see any quick solution, therefore I tested it without bzr and was able to build the image.
Can you elaborate why bzr is needed and if fix the error?
Need to find where is the best place to publish swagger
The link in README.md is broken, or the repository is private for the Helm chart. https://github.com/kafkaesque-io/pulsar-helm-chart/blob/master/helm-chart-sources/pulsar/templates/beamwh-deployment.yaml
Currently, pulsar-beam does not decompress a HTTP request with header Content-encoding: gzip
before pushing to Pulsar.
By default, REST interface should use Pulsar to store topic/webhook registration data. Using Kafka as a way to store data has been discussed at https://www.confluent.io/blog/okay-store-data-apache-kafka/
I do not like the current Mongo as default data store which introduces external dependency.
this pubsub system base on http,I have a question that if a client subscribe topic named 'A’, and then the client closed or losed, may be it never use,the pulsar system will keep this topic forever? If not a client ,rather than one millon..how to handle this scene?
This is more of a question. If I have multiple webhook configurations and multiple instances of pulsar-beam
brokers are running, how does the assignment of topics happens? Or, do all brokers try to read from all topics?
Hi
This is more of a question about Pulsar Beam registering webhooks
I have a docker instance of Pulsar Standalone, another instance of Pulsar Admin and a docker instance Pulsar Beam running locally.
I create a token with https://pulsar.apache.org/docs/en/administration-pulsar-manager/#set-administrator-account-and-password and use it
curl -v -X POST -H "Authorization: Bearer $TOKEN1" -d ' {"TopicFullName": "persistent://tenant22/namespace2/test-topic3", "PulsarURL": "pulsar+ssl://toronto.do.kafkaesque.io:6651", "Token": "TOKEN2", "Webhooks": [{"URL": "https://webhook.com/webhook1", "InitialPosition": "earliest", "Subscription": "mysub", "WebhookStatus": 1}]}' localhost:8085/v2/topic
When using the API Create-or-Update-Topic https://kafkaesque-io.github.io/pulsar-beam-swagger/#/ for I do get the "Unauthorized" error
it is not clear to me how to get TOKEN1 and TOKEN2.
what configurations should I do in order to allow Pulsar Beam to register webhooks ?
Currently the rate limit is hard coded to 200 which is not proper expectation for everyone. So it is better if the number is configurable or set to zero to disable.
The work load of pushing data to webhooks are within a single process. To scale up, we shall be able to have multiple brokers/workers to share the load. The first iteration could be a predefined number of workers to share based on hash of topic or subscription name.
Pulsar supports properties. the only property added by default is PulsarBeamId. Is it possible to add message properties?
This will enable webhook or Cloud Function to be able to reply its state and acquire the previous state along the data when Pulsar Beam passes.
A typical use case would be word count. The count is passed along the text data by the Beam. The incremented counts can be returned by the Function or Webhook and stored by the Beam.
The plan is to support complex JSON object.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.