Comments (5)
I have a working prototype using gRPC. Initial thoughts:
- gRPC is very new and so installation and use is difficult since it requires building lots of pre-released code from source.
- There are not very many languages supported yet. Key languages are the scientific languages. C, Python, R, maybe even Fortran. gRPC says its focus is on mobile, which does not align with our needs. This is an issue yet but it means integration with scientific languages is not likely to happen.
- The streaming is very nice. Specially since the bi-directional streams are independent and the order of reads vs writes doesn't matter.
from kapacitor.
How about adding support for lua runtime in kapacitor?
If we add then the users can define their functions in lua .
Heka https://hekad.readthedocs.org/en/v0.9.2/ allow user to write custom filter in lua and load it to run time. Heka also written in Go.
https://github.com/mozilla-services/lua_sandbox lua snabox is opensoure.
from kapacitor.
@skyrocknroll we thought a lot about using lua initially and it would be able to solve a piece of the problem, but we also decided that it would not solve enough.
The goal of custom functions is not necessarily so that user can write their own functions but rather can use their existing algorithms/code for processing their data. Many users will already have their algorithms already written whatever language C/C++ Python etc and many of them will not be able to rewrite the code effectively in any other language. By creating a method for input and output of Kapacitor users can just create wrappers as needed.
Having multiple language support is important enough that we decided away from Lua.
from kapacitor.
I finished a survey of serialization formats and RPC frameworks. Here are my thoughts:
Serialization:
There are many serialization formats out there and protobufs are one of the most popular. While C/C++, Java, and Python are the only languages officially builtin to the compiler, support for most other languages exists. See https://github.com/google/protobuf/wiki/Third-Party-Add-ons#programming-languages
Protobuf 3 also provides a JSON serialization along with its binary format. This way we can support any language that can also read JSON. This makes it also easier for users to write their own simple functions if they are not overly concerned about performance, without having to consume protobuf libraries. For these reasons we have decided to go with protobuf version 3. Version 3 is still in beta but as Kapacitor is young this should not be an issue.
RPC:
Protobuf provides an RPC system via service
declarations that allows methods to be defined and then through protoc plugins RPC code can be generated for the services. Currently gRPC is the main framework that takes advantage of the RPC system in proto3. But gRPC is limited in the languages is supports and adding language support is non-trivial. Both the complexity of gRPC code generation and limited language support made us decide not to use gRPC.
Proto3's RPC system can be used without gRPC but it requires effectively writing your own version of gRPC. While a simplified socket based gRPC system seems beneficial it would require that we would have to write a protoc plugin to generate code for every language we wanted to support.
Finally protobuf messages can easily be streamed over a socket and communication can be handled directly via sockets without using a RPC framework. This is the approach that we like the best so far. Our needs for communicating with the process are not complex. Basic heartbeat and snapshot messages plus the actual data flow. Using STDIN and STDOUT to send/receive messages from a process is simple and effective. This way all that needs to be done to support a language is to write a code to read and write protobuf messages to a socket. This is trivial code to write if it doesn't exist already for a given language. Writing code for a given language is much simpler and easier to maintain than a code generation program. Since using the protobuf RPC framework requires writing a code generation program we decide not to use it and just use protobuf messages directly and write our own clients/servers in the desired languages.
In summary simplicity wins out for more widespread support and the lower barrier to entry for writing your own custom functions. See this WIP PR for details of the initial proto message definitions #99 .
from kapacitor.
Thanks
from kapacitor.
Related Issues (20)
- Caching of meta user does not seem to work
- Google Hangouts Chat integration - HTTP/Post issue
- impossible to run kapacitor in windows service
- Running kapacitor without connecting to influxdb
- `-dbrp` argument behavior
- Please upgrade flux to 0.191.0 HOT 1
- about deadman rule
- Alerts suddenly start getting inhibited
- kapacitord ignores commands following the `-config` flag HOT 1
- disable a task that generates a lot of alerts takes a very long time HOT 3
- kapacitor memory usage
- Kapacitor 1.7 isn't available through Homebrew HOT 3
- Replace, remove or make configurable - printing non-log output to stdout/stderr
- Group by part of tag
- kapacitor tasks migration
- kapacitor listen 9093
- Alert Data Field Value Integer Type Error HOT 1
- how to configure the deadman function to trigger a warning alert instead of a critical alert in kapacitor
- Failed to link subscription on startup: not implemented: SHOW SUBSCRIPTIONS HOT 1
- Alerts history is empty after creaeting Alert rule .
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kapacitor.