Git Product home page Git Product logo

trino-event-stream's Introduction

Trino event stream

Development guide

A Trino plugin to stream trino events into a Kafka topic.

It implements the io.trino.spi.eventlistern.EventListener interface.

Install

Run mvn install to build this plugin, then put the plugin file trino-event-stream-352.zip to the plugin folder of trino server.

Configuration

Create new properties file event-listener.properties inside the /etc/trino directory:

event-listener.name=event-stream
bootstrap.servers=broker:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

Avro formatter is added to serialize messages generated from QueryCreatedEvent, QueryCompletedEvent. Avro formatted messages would be read as String using the StringSerializer Then it will emit events to the Kafka topic trino.event.

trino-event-stream's People

Contributors

apreethi13 avatar justineyster avatar meneal avatar shawnzhu avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar

Forkers

apreethi13 cuebiq lucaqy aka-shi usr-av meneal chris-d-edwards ronaldobernardi guanlisheng sumit-gupta-sgt gabrieltrettel ghas-results kazanexpress alexrogalskiy

trino-event-stream's Issues

Be able to use environment variables in "event-listener.properties"

What happened

Trino itself supports injecting secrets into properties file via syntax like ${ENV:env_var_name}, but it looks like doesn't work for event-listener.properties.

Expectation

It can use environment variables in event-listener.properties via the same syntax ☝️

Need to specify a working value serializer

No events will be published b/c of lots of warnings like this:

Failed to publish QueryCreatedEvent for query 20210315_013130_00000_kd5ep
org.apache.kafka.common.errors.SerializationException: Can't convert value of class io.trino.spi.eventlistener.QueryCreatedEvent to class org.apache.kafka.common.serialization.ByteArraySerializer specified in value.serializer
Caused by: java.lang.ClassCastException: class io.trino.spi.eventlistener.QueryCreatedEvent cannot be cast to class [B (io.trino.spi.eventlistener.QueryCreatedEvent is in unnamed module of loader 'app'; [B is in module java.base of loader 'bootstrap')
        at org.apache.kafka.common.serialization.ByteArraySerializer.serialize(ByteArraySerializer.java:19)
        at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
        at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:902)
        at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:862)
        at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:750)
        at io.trino.plugin.eventstream.EventStreamEventListener.queryCreated(EventStreamEventListener.java:43)
        at io.trino.eventlistener.EventListenerManager.queryCreated(EventListenerManager.java:150)
        at io.trino.event.QueryMonitor.queryCreatedEvent(QueryMonitor.java:132)
        at io.trino.dispatcher.LocalDispatchQueryFactory.createDispatchQuery(LocalDispatchQueryFactory.java:120)
        at io.trino.dispatcher.DispatchManager.createQueryInternal(DispatchManager.java:198)
        at io.trino.dispatcher.DispatchManager.lambda$createQuery$0(DispatchManager.java:148)
        at io.trino.$gen.Trino_352____20210315_013051_2.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

Provide a decent value serializer other than StringSerializer

Today

It uses .toString() method to serialize event data, the serialized metadata is not very useful. E.g., query user, query status, time information, etc.

Expectation

Build or reuse a value serializer to emit message represented by either JSON or AVRO, so that the result will be query-able.

Be able to toggle event to send

Now

it emit all 3 events (QueryCreatedEvent, QueryCompletedEvent and SplitCompletedEvent) to kafka topic.

Expectation

It should provide config option like:

event-stream.querycompletedevent=true
event-stream.querycreatedevent=true
event-stream.splitcompletedevent=true

if any of the above value is false, it won't emit the very event into kafka topic.

Make the topic configurable

Today

it uses the topic trino.event. could not configure at all.

Expectation

Expose an option like event-stream.topic=trino.event in event-listener.properties

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.