kafkaesque-io / pulsar-beam Goto Github PK

The detail is explained by
https://github.community/t/make-secrets-available-to-builds-of-forks/16166/20

We need to rewrite the test cases to avoid using secrets

Hello,

while trying to build your image I encountered the following error:

D:\projects\pulsar\pulsar-beam (master -> origin)
λ docker build -t pulsar-beam .
Sending build context to Docker daemon  237.6kB
Step 1/13 : FROM golang:alpine AS builder
 ---> 3289bf11c284
Step 2/13 : LABEL maintainer="kafkaesque.io"
 ---> Using cache
 ---> b0b2cacac0a7
Step 3/13 : RUN apk --no-cache add build-base git bzr mercurial gcc
 ---> Running in c4cc97e18153
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
  bzr (missing):
    required by: world[bzr]
The command '/bin/sh -c apk --no-cache add build-base git bzr mercurial gcc' returned a non-zero code: 1

I tried to check why bazaar is missing - but could not see any quick solution, therefore I tested it without bzr and was able to build the image.

Can you elaborate why bzr is needed and if fix the error?

Need to find where is the best place to publish swagger

The link in README.md is broken, or the repository is private for the Helm chart. https://github.com/kafkaesque-io/pulsar-helm-chart/blob/master/helm-chart-sources/pulsar/templates/beamwh-deployment.yaml

Currently, pulsar-beam does not decompress a HTTP request with header Content-encoding: gzip before pushing to Pulsar.

By default, REST interface should use Pulsar to store topic/webhook registration data. Using Kafka as a way to store data has been discussed at https://www.confluent.io/blog/okay-store-data-apache-kafka/

I do not like the current Mongo as default data store which introduces external dependency.

this pubsub system base on http,I have a question that if a client subscribe topic named 'A’, and then the client closed or losed, may be it never use,the pulsar system will keep this topic forever? If not a client ,rather than one millon..how to handle this scene?

This is more of a question. If I have multiple webhook configurations and multiple instances of pulsar-beam brokers are running, how does the assignment of topics happens? Or, do all brokers try to read from all topics?

Hi
This is more of a question about Pulsar Beam registering webhooks

I have a docker instance of Pulsar Standalone, another instance of Pulsar Admin and a docker instance Pulsar Beam running locally.

I create a token with https://pulsar.apache.org/docs/en/administration-pulsar-manager/#set-administrator-account-and-password and use it

curl -v -X POST -H "Authorization: Bearer $TOKEN1" -d ' {"TopicFullName": "persistent://tenant22/namespace2/test-topic3", "PulsarURL": "pulsar+ssl://toronto.do.kafkaesque.io:6651", "Token": "TOKEN2", "Webhooks": [{"URL": "https://webhook.com/webhook1", "InitialPosition": "earliest", "Subscription": "mysub", "WebhookStatus": 1}]}' localhost:8085/v2/topic

When using the API Create-or-Update-Topic https://kafkaesque-io.github.io/pulsar-beam-swagger/#/ for I do get the "Unauthorized" error

it is not clear to me how to get TOKEN1 and TOKEN2.
what configurations should I do in order to allow Pulsar Beam to register webhooks ?

Currently the rate limit is hard coded to 200 which is not proper expectation for everyone. So it is better if the number is configurable or set to zero to disable.

The work load of pushing data to webhooks are within a single process. To scale up, we shall be able to have multiple brokers/workers to share the load. The first iteration could be a predefined number of workers to share based on hash of topic or subscription name.

Pulsar supports properties. the only property added by default is PulsarBeamId. Is it possible to add message properties?

This will enable webhook or Cloud Function to be able to reply its state and acquire the previous state along the data when Pulsar Beam passes.

A typical use case would be word count. The count is passed along the text data by the Beam. The incremented counts can be returned by the Function or Webhook and stored by the Beam.

The plan is to support complex JSON object.