[BUG] Messages are ignored with "Message not exist in record" exception about pulsar-io-cloud-storage HOT 5 CLOSED

@artem-garmash thank you for reporting this. Sorry for the late reply! We will take a look at this issue.

from pulsar-io-cloud-storage.

The error is because we can't get the Message instance, because the Source in Function is not a topic message, but postgresql, and cloud-storage expects to get the PulsarRecord message.

During the run, if I use only Record data and not PulsarRecord data, I will not be able to process Schema. So I need to use Pulsar topic as source.

from pulsar-io-cloud-storage.

@artem-garmash pulsar-io-cloud-storage needs to use a schema to hold structural information about the data, so it can only use the Pulsar topic as message input. You need to write your records to the topic and then use pulsar-io-cloud-storage to write them out.

There was sensitive information in the log and I removed it.

from pulsar-io-cloud-storage.

@jianyun8023, thanks for the reply. My pulsar topic had schema definition as you can see in the bug description. I literally followed README instructions 1:1 to test pulsar-io-cloud-storage but it didn't work.
At the end of my attempts I tried to use jdbc connector with postgres sink instead of pulsar-io-cloud-storage just to verify that my topic/setup worked. And it all worked with jdbc connector, consuming the same pulsar topic.

from pulsar-io-cloud-storage.

I cannot reproduce this issue with the latest version, so I assume this issue has been resolved in past commits. Feel free to re-open it if there is any more context.

from pulsar-io-cloud-storage.