elastic / integrations Goto Github PK
View Code? Open in Web Editor NEWElastic Integrations
Home Page: https://www.elastic.co/integrations
License: Other
Elastic Integrations
Home Page: https://www.elastic.co/integrations
License: Other
I've been working on the fields validation in elastic/package-registry#486 and noticed that the netflow
integration fails the validation due to invalid format.
name: netflow
type: group
description: >
Fields from NetFlow and IPFIX.
fields:
- name: type
type: keyword
description: >
The type of NetFlow record described by this event.
^ this should be an array like this one:
- name: cisco
type: group
There is no need anymore (once already split with package-storage) to apply additional versioning in directories, e.g.:
change dev/packages/beats/aws/0.0.4
to dev/packages/beats/aws
EDIT
I've renamed the issue as it's no longer a discussion about the version tag.
Looking at the expected O365 output, I noticed somethings that may need improvement.
First event.category
is being implemented as a string, but the ECS schema says this field is an array.
https://github.com/elastic/beats/blob/c01dfe680e8d4d810e014c6caa6b0e543c56df57/x-pack/filebeat/module/o365/audit/test/01-exchange-admin.log-expected.json#L5
The following might be limited to the Exchange logs...
Second, two fields seem to be copied but not parsed, and I'm curious is this is the expected ECS result (to leave the content as is)? The server.name
field and user.id
field contain more than just those values, but they are not being parsed.
https://github.com/elastic/beats/blob/c01dfe680e8d4d810e014c6caa6b0e543c56df57/x-pack/filebeat/module/o365/audit/test/01-exchange-admin.log-expected.json#L39-L44
Third, the server.address
field is supposed to be copied to either server.ip
or server.domain
depending on what type of value it is, but it is not. Should this be updated?
When testing system
package, I found out I can not filter using stream.dataset
in Kibana Discover page. Here is what's shown in Kibana index pattern when running system
package:
But with a new agent, new test environment when testing aws package, I was able to use stream.dataset
as a filter in Kibana Discover. Here is what's shown in Kibana index pattern when running aws
package:
I suspect that dbus based datasets should be disabled by default. Otherwise, the docker image for Elastic-Agent fails. I suspect that it will fail also for ordinary environment where there is no dbus available.
2020-06-17T10:18:58.830Z ERROR [centralmgmt.fleet] fleet/manager.go:261 2 errors: Error creating runner from config: 1 error: error connecting to dbus: error getting connection to system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory; Error creating runner from config: 1 error: error connecting to dbus: dial unix /run/systemd/private: connect: no such file or directory
I think we need to select a subset of datasets that is all-OS friendly, because we install system monitoring by default.
Hi, I'm testing in 7.8 BC5 deployed on June 3 and find that the Cisco package is still erroring when I try to install.
I see Ruflin commented in closed pr below as:
@alakahakai Could you open a PR to the package-storage repo with this change as a new version so we can release it?
#36
I'm opening this issue to track it since the issue is closed as is the pr above
Describe the enhancement:
I checked again the existing log types that exist in filebeat because of a test I made with zeek 3.0.
https://docs.zeek.org/en/current/script-reference/log-files.html
These issues
elastic/beats#12724
elastic/beats#12812
elastic/beats#14150
elastic/beats#14404
I did now produce a list of all logs to identify all missing log types:
One special part is extra
Additionally documentation doesn't have much information about how to configure seek module:
https://www.elastic.co/guide/en/beats/filebeat/7.7/filebeat-module-zeek.html
To ensure consistency in style and content between Integration packages, it would be beneficial to our authors if they had some documentation to follow. While packages will vary in content, they should look, feel, and generally read the same. There are some intricacies in the design that might not be obvious to our authors that i'd like to call out. Some examples:
#
. Top level headings create sections that are added to the navigation.# About
--> # Compatability
--> ...
--> # Questions and Contributions
title
and alt
text. The title
gets rendered as a caption below the image, and alt
is used by screen readers.The list goes on... Where should this 'guide' live? a Google doc? a CONTRIBUTING.md file in our codebase?
I'm happy to take lead on this for the design related bits, but I also thought this may be an appropriate place for @Titch990 to weigh in. I'm not sure if we have existing writing guides we can follow, or if we should make one specific to Integration package READMEs that is included in this guide. Regardless, I'd love to work with you and get your input on this effort :)
It would be helpful to have 1-3 examples of real content to work from (I believe @ruflin has these).
There's currently a windows
package containing modules migrated from winlogbeat
. The goal of this issue is to migrate the metrics datasets from the metricbeat windows
module.
This checklist is intended for Devs which create or update a package to make sure they are consistent.
This entry is currently recommended. It will be mandatory once we provide better support for it.
sample_event.json
) existsCurrently, the script aggregates datasource inputs based on "type" instead of stream input.
For the redis integration, it results in having a single datasource input "logs" for, both, application logs and slow logs, but should be two different ones.
Describe the enhancement:
Support for MacOS Unified Logging
Describe a specific use case for the enhancement or feature:
Auditbeat doesn’t provide much valuable information because it is still pulling information from syslog, MacOS is deprecating use of syslog and has moved to Unified Logging. To my knowledge there is no Beat for MacOS that will track login, logout, lock, unlock, or sudo access. It is possible to create custom scripts to grab some (not all) of this info but a Beat would be much easier
mage importBreats is giving the following error:
2020/06/29 08:35:15 x-pack/filebeat okta: module found
2020/06/29 08:35:15 Docs found (path: ../beats/x-pack/filebeat/module/okta/_meta/docs.asciidoc)
2020/06/29 08:35:15 system: dataset found
2020/06/29 08:35:15 system: no docs found (path: ../beats/x-pack/filebeat/module/okta/system/_meta/docs.asciidoc), skipped
2020/06/29 08:35:15 okta: icon not found
2020/06/29 08:35:15 system: dataset found
2020/06/29 08:35:15 ingest-pipeline found: ingest/pipeline.yml
2020/06/29 08:35:15 dashboard found: 749203a0-67b1-11ea-a76f-bf44814e437d.json
2020/06/29 08:35:15 creating from logs source failed: migrating dashboard file failed (path: ../beats/x-pack/filebeat/module/okta/_meta/kibana/7/dashboard/749203a0-67b1-11ea-a76f-bf44814e437d.json): making POST request failed: {"statusCode":400,"error":"Bad Request","message":"[request body.version]: expected value of type [string] but got [undefined]"}
exit status 1
Error: running "go run ./dev/import-beats/ -packages okta *.go" failed with exit code 1
I've read the README in the system
integration and it looks that it needs editing.
There are references to metricsets, but now we're using datasets.
mage UpdatePackageStorage
should be executed on git push to the integrations repository, to make sure we do not skip any package updates.
The script already reacts to integration versions that haven't been pushed yet. It lacks support for assignees and labels.
With #97, empty field group in package-fields.yml files are manually removed. Empty field group causes Kibana to not recognize the field type, for example:
We should adjust the import-beats
script to remove package-fields.yml file if it is empty.
In elastic/package-registry#536 the support for an owner field was added. We should add it to all our package. The value should be the team "owning" the package.
In elastic/beats#19360 we transitioned to use logfile
as an input type for gathering logs from file.
Previously we were using logs
. As for now both options are supported but we will move to only support logfile
soon.
DoD for this issue is having all input.type
replaced from logs
to logfile
Apparently, it slipped through the importing process. There is no fileset
field anymore, hence dashboards may not work.
Currently the docs (README.md) can be generated out of a package during the migration process (mage ImportBeats
). It would be great if it's possible to regenerate docs out of content of existing package.
Use Github API and .githubtoken
to open a PR with changes.
This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.
This entry is currently only recommended. It will be mandatory once we provide better support for it.
sample_event.json
) existsdata.json
exists and an automated way to generate it exists (go test -data
)Add support for the IIS package
PR #138
During the investigation of elastic/beats#8301 issue we identified some patterns that could be added to the initial module implemented for haproxy (#8014):
option tcplog
. elastic/beats#8526 elastic/beats#8637Feb 6 12:12:56 localhost haproxy[14387]: 10.0.1.2:33313 [06/Feb/2009:12:12:51.443] fnt bck/srv1 0/0/5007 212 -- 0/0/0/0/3 0/0
Feb 6 12:12:09 localhost haproxy[14385]: Connect from 10.0.1.2:33312 to 10.0.3.31:8012 (www/HTTP)
Sep 13 15:51:16 debian8-haproxy haproxy[5988]: Server mysvc/myserver01 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Sep 13 15:51:15 debian8-haproxy haproxy[5988]: backend myservers has no server available!
haproxy.mode
. Right now, only HAProxy default format outputs a mode
field pointing if the log line is either HTTP or TCP. But the HTTP and TCP logs doesn't actually show anything like this.Apr 28 16:09:58 ha1.prod.ad.qqqcore.com haproxy[18923]: 119.169.133.47:50040 [28/Apr/2022:16:09:58.167] Advertstream_Log~ Advertstream_Log/log2.prod.ad.qqqcore.com 0/0/2/32/+34 200 +313 - - --VN 116/106/1/1/0 0/0 {|l.qqqcore.com||https://qwersimon.com/} {Apache|57|max-age=||} \\\"GET https://l.qqqcore.com/a/log/view/?c=3vUCAFGazq16o_BmABdcR3-BMkXpE6O-3i1M7PyulK3onD3Z1cjvbl-qUdo_wrcYlXJUe1kU-CD48n-9QWED-lfd2vXLBzp6xQiMOoBSfYfo6Bk9qMGPn901IK2Cs0SHewmpxeNKa7Y4AYMiq9dAb-hSHEsku-ijbNiDmPwh5bAp-NR22OdD6ZlJ-7g0rGPF_mtfW3XWaFuUHLqDeu6mIyMHvbf95aPl0AZt481_2b_ujFh2eTEvK0q_dvjfhWr4P_w1_M24LKm_ipHcmzwmXVjdWzMQGxPFeLVA9YuB1akMuOLwFYneJCVa5foi3WTVyBIvwiMpzbYcSfGl5JVJSNq8VsHh5ZyA9GdqnCBI3V3VcPiBwxQZ0Z1fsCEeo29mj4_WmCPFtEYKUTNJYTTcBaNNUZh_cypX&impid=2327760487204226&&r=&npbk=0&dispatcher=&k=&b=204012&zoneid=232776&siteid=11081&a=ae-d&bidder=goodad&earning=3.4019999999999997¤cy=EUR&auctionId=8739afe7-ad6b-4676-b9ea-05cb68871be6&adId=11954862d0bfbe6&creativeId=0&testId=0&domain=&country=XX&device=DESK&auctions=adaccess-0_adaccess-0_adpone-0.
As discussed in elastic/package-registry#533, currently the os specific configs are not used yet and the final format might differ from what we have today. To make sure we don't have any conflicts in the future if we decide to have a different format, I suggest for now we remove it from the packages and from the generator.
For the goal of getting Cloud Foundry to GA we need to add integration tests. The best option in the case is to use CFDev to bring up a development Cloud Foundry cluster. This will bring enough up to allow apps to be deployed and for filebeat and metricbeat to connect to loggregator.
Checks can be added to ensure that:
To verify that router logs are being retrieved an application would need to be deployed and some traffic generated to ensure that proper router logs are being retrieved.
Modify the behavior of the tool:
Update the package-storage only if the version has been bumped up (which means it's not in the package-storage)
Describe the enhancement:
Enhance Integrations repo CI to call new e2e Agent test when PRs are opened
Files from this repo location should trigger the noted CI:
https://github.com/elastic/integrations/tree/master/packages
I would like the team's confirmation of any other files we wish to use to trigger the test being included, above ^ please.
The e2e test is here:
https://github.com/elastic/e2e-testing/tree/master/e2e/_suites/ingest-manager
It can be called as, from the elastic/e2e-testing repo e2e/_suites/ingest-manager
$ godog -t stand_alone_mode
Note, the above e2e-testing repo is in use for CI testing for Metricbeat already, if it helps to model tests on it or research usage.
Currently, README.md template files are stored in dev/import-beats-resources/<integration>docs/README.md
. The place is aside from the integration, which might be confusing for contributors.
The original assumption was to generate integrations from Beats files. If there is a cut off date in the future to stop generating and modify already generated integrations, the current routine might be a problem.
README.md files are rendered from template during import step:
Questions:
import-beats-resources
?This issue is intended to track the process around and the schedule for when Elastic will include Integration package changes into a given release (either experimental or prod).
It is a place-holder for now as the team hashes out some initial thoughts we can post back. We will be interested in any feedback from contributors and consumers.
orignaly reported by @mtojek in elastic/kibana#67598
I spotted this one while testing the "redis" integration.
solution: elastic/kibana#67598 (comment)
I'm creating this issue to track AWS integration package from recent PRs that made in Filebeat aws
module:
Improve AWS cloudtrail fileset: elastic/beats#18958
Fix improper nesting of session_issuer in aws/cloudtrail: elastic/beats#18915
cloudtrail
data source: https://github.com/elastic/integrations/tree/master/packages/aws/dataset/cloudtrail
cc @leehinman
The dev/update-package-storage
will simplify pushing next changes to the package-storage
.
Requirements:
package-storage
repository.Here are the next steps:
sync-integrations-<timestamp>
package-storage
repository.package-storage
for every single package.Once test resources are added to integrations, the "update-package-storage" should ignore these files and don't copy them to the package-storage
repository.
I have packaged the latest version of the elastic agent and tried it out and I am getting:
app/app.go[214]: unknown error
2020-06-17T16:00:14+02:00 ERROR reporter.go:47 2020-06-17T16:00:14+02:00: type: 'ERROR': sub_type: 'CONFIG' message: Application: metricbeat[912f7464-4612-4b87-8038-139ff8b67054]: application 'metricbeat--8.0.0' crashed: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/core/plugin/app/app.go[214]: unknown error
2020-06-17T16:00:17+02:00 ERROR reporter.go:47 2020-06-17T16:00:17+02:00: type: 'ERROR': sub_type: 'CONFIG' message: Application: metricbeat[912f7464-4612-4b87-8038-139ff8b67054]: application 'metricbeat--8.0.0' crashed: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/core/plugin/app/app.go[214]: unknown error
2020-06-17T16:00:20+02:00 ERROR reporter.go:47 2020-06-17T16:00:20+02:00: type: 'ERROR': sub_type: 'CONFIG' message: Application: metricbeat[912f7464-4612-4b87-8038-139ff8b67054]: application 'metricbeat--8.0.0' crashed: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/core/plugin/app/app.go[214]: unknown error
2020-06-17T16:00:23+02:00 ERROR reporter.go:47 2020-06-17T16:00:23+02:00: type: 'ERROR': sub_type: 'CONFIG' message: Application: metricbeat[912f7464-4612-4b87-8038-139ff8b67054]: application 'metricbeat--8.0.0' crashed: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/core/plugin/app/app.go[214]: unknown error
2020-06-17T16:00:26+02:00 ERROR reporter.go:47 2020-06-17T16:00:26+02:00: type: 'ERROR': sub_type: 'CONFIG' message: Application: metricbeat[912f7464-4612-4b87-8038-139ff8b67054]: application 'metricbeat--8.0.0' crashed: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/core/plugin/app/app.go[214]: unknown error
Are there are any tests or any steps on how to debug this?
No data coming in as well.
It says "Customs logs" and it should say "Custom logs"
Use test logs to verify integration's ingest pipeline.
From developer perspective
The command mage test
or PACKAGES=aws mage test
uses a tool which performs the following steps:
The system dashboard contains a default query for CHANGEME_HOSTNAME
:
This query is on the Beats side replaced with the actual hostname but now in the package this does not work anymore.
A package can contain a changelog. Currently propose format can be found here: https://github.com/elastic/package-registry/blob/master/testdata/package/reference/1.0.0/changelog.yml
I'm proposing to start using the changelog in packages even though we don't show it in the UI yet. This should make it easy when looking at a version of a package to see what changed.
Following comments written here: elastic/beats#10592 ETCD module needs some grooming and polishing to make it follow naming conventions (and probably update some metrics more).
Tasks could be:
etcd.self.recv.pkgrate
are hard to read and doesn't show a metric unit (per_sec
in this case, if it's a rate that occurs every second). Leaves room for misunderstandings.leader
metricset as you can see in this event example:{
"@timestamp": "2017-10-12T08:05:34.853Z",
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"etcd": {
"leader": {
"followers": {
"5a22bdba1efc5b4a": {
"latency": {
"average": 0.0024145817307692323,
"current": 0.001494,
"maximum": 0.061351,
"minimum": 0,
"standardDeviation": 0.0029017970782575734
},
"counts": {
"success": 1248,
"fail": 0
}
},
"639ec377a30542cf": {
"latency": {
"average": 0.0026389089456869013,
"current": 0.001241,
"maximum": 0.233578,
"minimum": 0,
"standardDeviation": 0.00695758066274549
},
"counts": {
"success": 1252,
"fail": 0
}
}
},
"leader": "d3cf079af51fa9a8"
}
},
"event": {
"dataset": "etcd.leader",
"duration": 115000,
"module": "etcd"
},
"metricset": {
"name": "leader"
},
"service": {
"address": "127.0.0.1:2379",
"type": "etcd"
}
}
As you can imagine, the list of followers may be way too long. In this case, I think that each follower should have its own event so that mapping is consistent, something like this:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"etcd": {
"follower": {
"id": "5a22bdba1efc5b4a",
"latency":{
"ms": 0.001494
},
"success_operations": 1248,
"failed_operations": 0,
"leader": "d3cf079af51fa9a8"
},
"event": {
"dataset": "etcd.follower",
"duration": 115000,
"module": "etcd"
},
"metricset": {
"name": "follower"
},
"service": {
"address": "127.0.0.1:2379",
"type": "etcd"
}
}
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"etcd": {
"follower": {
"id": "639ec377a30542cf",
"latency":{
"ms": 0.001241
},
"success_operations": 1252,
"failed_operations": 0,
"leader": "d3cf079af51fa9a8"
},
"event": {
"dataset": "etcd.follower",
"duration": 115000,
"module": "etcd"
},
"metricset": {
"name": "follower"
},
"service": {
"address": "127.0.0.1:2379",
"type": "etcd"
}
}
At the moment in the system package we show the user most configs by default. Instead I would propose that most of the ones we don't expect users to change often, we hide. Taking cpu as an example, I suggest to hide both period
and cpu Metrics
. This will make the system configuration much more compact.
mage importBeats is giving the following error when the module is using the tojson and inList functions in the template.
2020/06/29 08:48:21 x-pack/filebeat okta: module found
2020/06/29 08:48:21 Docs found (path: ../beats/x-pack/filebeat/module/okta/_meta/docs.asciidoc)
2020/06/29 08:48:21 system: dataset found
2020/06/29 08:48:21 system: no docs found (path: ../beats/x-pack/filebeat/module/okta/system/_meta/docs.asciidoc), skipped
2020/06/29 08:48:21 okta: icon not found
2020/06/29 08:48:21 system: dataset found
2020/06/29 08:48:21 ingest-pipeline found: ingest/pipeline.yml
2020/06/29 08:48:21 creating from logs source failed: creating streams failed (datasetPath: ../beats/x-pack/filebeat/module/okta/system): creating log streams failed (modulePath: ../beats/x-pack/filebeat/module/okta, datasetName: system): parsing stream config failed: parsing template failed: template: input-config:20: function "tojson" not defined
exit status 1
Error: running "go run ./dev/import-beats/ -packages okta *.go" failed with exit code 1
The related configurations in beat modules:
pagination: {{ .pagination | tojson }}
rate_limit: {{ .rate_limit | tojson }}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}
The kibanaLogosPath in icons.go (kibana/src/legacy/core_plugins/kibana/public/home/tutorial_resources/logos) is no longer valid. It should be kibana/src/plugins/home/public/assets/logos.
The project should contain a README file describing its content, goal, short HOWTO guide and potential references to other guide or resources.
Please link a CONTRIBUTING guide too.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.