What is an Elastic Input?
Elastic Inputs are binaries that are distributed with and controlled by Elastic Agent. As the name implies they collect data from and on endpoints where they are deployed.
Goal
The goal of this project is to design a developer friendly inputs platform. The system should help people to write new inputs and test them easily. This ticket lists the requirements and defines the different development phases.
You can read about the motivation in the design document below.
Requirements
Input
- Each input must speak the control protocol of Elastic Agent
- Each input must provide a manifest file describing its features
- Each input must run in a separate process
- Each input must log to stdout and stderr in JSON format
- Each input must be able to test a configuration and report errors without starting data collection
- Each input must be able to send events to the shipper
- Each input must provide a reference configuration to aid integration developers
- Each input must accept a logger on init and enrich log events with metadata so the issues can be tracked down elastic/beats#9177
Developer experience
- Testing framework
- to check if the input understands the control protocol
- to check if the input can report its own state, metrics and logs
- Input code generator to let users work on production logic and not waste time on boilerplate
- Write a guide for Elastic developers for migrating existing inputs
Development plan
Phase I.
The goal of the first phase is to lay the foundation for the new developer tooling by developing the first new agent input. The first input developed will be the agent load generator described in elastic/elastic-agent-shipper#57. Developing a completely new input will allow this phase to focus on the basic input structure, developer tooling, and release process.
TODO
TBD.
Dependencies
Packaging
Packaging is done with mage. Every binary is crossbuilt in a Docker container. Then the resulting binaries and generated files are moved to a common folder.
A package contains inputs in separate binaries with their manifest files.
$ tar tvf elastic-agent-inputs-linux-arm64.tar.gz
inputs/
filestream/
filestream
manifest.yml
journald/
journald
manifest.yml
...
If someone wants to create their own inputs package, it must follow the pattern above. Then this package has to be extracted into the folder where Agent can find it.
inputs/
{input_name}/
{input_name}
manifest.yml
The packages could be installed manually by moving the inputs to the appropriate folder. However, we should also provide DEB, RPM and other packages that can be installed along with Elastic Agent.
More details here: elastic/elastic-agent#222
Phase II.
In this phase we focus on making input migration and development accessible to both internal and external developers. In this phase the filestream
and system metrics
inputs will be migrated to the new framework.
TODO
Dependencies
Phase III.
Phase III is going to be the first phase for moving real inputs to the new system. The inputs selected for this phase are easy to move because they already use the inputs v2 architecture. We could delegate/ask other teams to help with inputs under x-pack/filebeat.
Phase IV.
This is the last phase of the development. I expect collaboration from all teams as our team does not have the bandwidth to migrate all existing inputs. In this phase the Data plane team should focus on supporting other developers by reviewing their work, adjusting documentation if needed, etc.
TODO
Documentation
Related