Protected names are not sanitized correctly in Data Lake Sink / Influx sink about streampipes HOT 3 CLOSED

Just published a PR! The scope of the sanitization is limited to InfluxDb.

from streampipes.

Hi @bossenti, I have looked into the issue. The event is "correct" ("correct" meaning key value pairs are the same with the file) in frontend, Kafka producer, and Kafka consumer. However, once the code hits Influx Store, the runtime name of some keys are modified. The root cause is that "name" key in the file stream is a keyword of Influx reserved keywords list and Streampipes sanitizes DataLake measure for those keys conflicting with Influx reserved keywords. If you change "name" to something else not in the list everything works.

To fix this, I propose to also sanitize event instead of only sanitizing DataLake measure. Let me know what do you think🙂!

from streampipes.

Hi @muyangye,

you are right, changing the column name makes it work.

As you said, sanitation should prevent naming conflicts and therefore rename the name column. But the expected outcome is then that the values are part of the DataLakeMeasure with the sanitized measure name, which doesn't seem to work here.

Where would you place the sanitation of the event? Sanitation is a specific requirement of the InfluxDb in this case, so I'm a bit hesitant to affect other usages of the event stream as well.

from streampipes.