elastic / machinebeat Goto Github PK

This is a beat for collecting data of machines, sensors and PLCs. At the moment this is supporting OPC UA and MQTT. More to add.

Home Page: https://www.elastic.co/

License: Other

Makefile 3.24% Go 94.86% Batchfile 0.69% Dockerfile 0.92% Shell 0.29%

machinebeat's Introduction

About Machinebeat

This Beat is beta. It is used in many different environments without issues for longe periods of time. However if you have issues with this beat you can't ask Elastic support, you will need to open an issue in this repository to get help.

The current version in master of Machinebeat is well working and tested. It will be developed further over time but can already be used. The downloadable binaries are expected to run without any major issues. Every attempt will be made to avoid breaking changes in future releases.

What is a beat? A beat is a lightweight data shipper written in GOLANG developed by Elastic N.V. and the community. It is open source and also implements a framework offered to the community to build their own beats. Those beats may offer special purpose data collection not offered by existing beats. Machinebeat is one of those special purpose beats.

The aim of Machinebeat is to build a lightweight data shipper that is able to pull metrics from machines used in industrial environments and ship it to Elasticsearch, or have Logstash or a message queue like Kafka inbetween, depending on the use case and architecture.

Today Machinebeat is able to connect to OPC-UA servers, MQTT Broker and IoT Cloud Services like AWS IoT core. It can pull data in real-time, subscribing to topics and feeding the data into Elasticsearch for near real-time monitoring purposes. This will enable machine operators to see behavior of their machines and analyse important metrics on the fly.

Machinebeat also supports PLC4X protocols such as Modbus, S7, Ads, Bacnet, CBus, Eip and KNX. The PLC4X module is relatively new, we are looking for some real world users to check that the different protocols work as expected.

The ability to get machine metrics and other related information is a foundation for maintenance, quality assurance and optimization of industry 4.0 environments. This observability can help to identify potential issues early in the production cycle or optimize the use of ingredients for the products produced. There are many other problems that can be solved by monitoring machine metrics once the data is integrated into an analytics platform such as Elastic.

In a future version Machinebeat is envisaged to support additional protocols in order to cover a broader mix of different sensor metrics in the industry.

Let's test it

You don't have an Elastic cluster up and running? Use Elasticsearch Service

Getting Started with Machinebeat

To start quickly and easy use the pre-built binaries shared in this repository. To get started choose the version that fits for your environment.

ARM

Download the latest version of the binary here: https://ela.st/machinebeat-arm

Linux

Download the latest version of the binary here: https://ela.st/machinebeat-linux

Or use wget https://github.com/elastic/Machinebeat/releases/download/v.8.7.0/machinebeat_linux_8.7.0.zip and change the version number 8.7.0 to the latest

Start the beat with the following command in your terminal.

./machinebeat -e -c machinebeat.yml

Use -e to see the log output. Use -c to set a different config file.

Windows

Download the latest version of the binary here: https://ela.st/machinebeat-windows

Start the beat with the following command in your CMD or PowerShell. There is also a PowerShell Script to add the beat as a service.

machinebeat.exe -e -c machinebeat.yml

Use -e to see the log output. Use -c to set a different config file.

Docker

We do not publish images of machinebeat. However we prepared a Dockerfile in order to build your own docker images. Before you build the image you can already change the config files. That way the configuration will be part of the image. Of course you could also mount the config files later.

Just run

docker build . -f docker/Dockerfile -t elastic/machinebeat

In order to get your own docker image. After finishing that process you can start your machinebeat container with

docker/docker-run.sh

Afterwardd you can check that it is working with

docker logs machinebeat

Configurations

OPCUA Module

Machinebeat uses the GOPCUA project to connect and pull data or subscribe from OPS-UA servers. GOPCUA is an implementation of the OPC-UA specification written in GOLANG. More about OPC-UA can be found on Wikipedia.

To enable the OPCUA Module rename the file modules.d/opcua.yml.disabled to modules.d/opcua.yml.

The current version is able to read and subscribe to nodes from an OPC-UA address space specified in the modules.d/opcua.yml file from servers and transfer the collected data to Elasticsearch directly or via Logstash. The nodes and its leaves will be discovered automatically from mentioned YAML file like this:

- module: opcua
  metricsets: ["nodevalue"]
  enabled: true
  period: 2s
  
#The URL of your OPC UA Server
  endpoint: "opc.tcp://opcuaserver.com:48010"

Different nodes can be specified and read by Machinebeat.

MQTT Module

To enable the MQTT Module rename the file modules.d/mqtt.yml.disabled to modules.d/mqtt.yml. Change the configuration based on your needs. It works with every MQTT broker and also IoT cloud services.

Example configuration to collect from AWS IoT core:

- module: mqtt
  metricsets: ["topic"]
  enabled: true
  period: 1s
  host: "tcps://<yourAWSEndpoint>:8883/mqtt"
  clientID: "<yourAWSclientID>"
  topics: ["#"]
  decode_payload: true
  ssl: true
  CA: "<pathToAWSRootCA>"
  clientCert: "<pathToAWSyourIoTCertificate>"
  clientKey: ""<pathToAWSyourIoTPrivateKey>""

Your client id from IoT console -> things:

arn:aws:iot:us-east-2:<AWS account>:thing/<clientID>

Make sure your certificate has the correct policies attached:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:*",
      "Resource": "*"
    }
  ]
}

PLC4X Module

Machinebeat supports PLC4X protocols such as Modbus, S7, Ads, Bacnet, CBus, Eip and KNX. The PLC4X module is relatively new. We are looking for some real world users to check that the different protocols work as expected. To enable the PLC4X Module rename the file modules.d/plc4x.yml.disabled to modules.d/plc4x.yml. Change the configuration based on your needs.

How to build on your own environment

1.) Download all dependencies from go.mod using go get -u 2.) You may need to overwrite some modules with the following versions that do not support go.mod in older versions:

go get k8s.io/[email protected]
go get k8s.io/[email protected]
go get k8s.io/[email protected]

3.) Run go build in the machinebeat repository

machinebeat's People

Contributors

Stargazers

Watchers

machinebeat's Issues

How to build machine beat from source

Machinebeat is failing to connect to OPCUA server. I may need to tweak TCP parameters in GO client.
Where can I find build instructions, so that I can tweak the parameters and rebuild machinebeat?

OPC-UA: Crash after certain amount of time 'panic: close of closed channel'

After a certain amount of time without a value change on the subscribed tag, the following happens:

Machinebeat output:

panic: close of closed channel

goroutine 656 [running]:
github.com/gopcua/opcua/uasc.(*SecureChannel).Close.func1(0xc0003de700)
        C:/Users/felix/Documents/Machinebeat/pkg/mod/github.com/gopcua/[email protected]/uasc/secure_channel.go:842 +0x36
github.com/gopcua/opcua/uasc.(*SecureChannel).Close(0xc0003de700, 0x2e8a2a0, 0xc000094110)
        C:/Users/felix/Documents/Machinebeat/pkg/mod/github.com/gopcua/[email protected]/uasc/secure_channel.go:851 +0x1a5
github.com/gopcua/opcua.(*Client).monitor(0xc00030c7e0, 0x2ecced8, 0xc0001289c0)
        C:/Users/felix/Documents/Machinebeat/pkg/mod/github.com/gopcua/[email protected]/client.go:326 +0x1006
created by github.com/gopcua/opcua.(*Client).Connect.func1
        C:/Users/felix/Documents/Machinebeat/pkg/mod/github.com/gopcua/[email protected]/client.go:216 +0x6a

opcua.yml:

- module: opcua
  metricsets: ["nodevalue"]
  enabled: true
#   period: 1s
  
  #The URL of your OPC UA Server
  endpoint: "MYURL"

  #==========================  Security configuration ============================
  username: "MYUSER"
  password: "MYPASS"
  

  
  #==========================  Data collection configuration ============================

  ecsFields: true
  legacyFields: false  

  subscribe: true

  browse.enabled: false

  
  nodes:
  -  id: "ns=2;s=Path.To.TestBoolean"
     label: "TestBoolean"

Am I doing something wrong?

[Question] Support "Word" data type

Dear Contributor,

Thanks for your work.
I could integrate my OPC UA Server into ELK stack.
But, I figured out that only Bool data type can be added automatically.

2022-02-21T17:44:58.352+0900    INFO    nodevalue/nodevalue.go:130      Browsing is enabled. Data collection will start after discovery. Based on your server and browsing configuration this can take some time.
2022-02-21T17:44:58.352+0900    INFO    nodevalue/client.go:309 [OPCUA] Start browsing node: ns=4;i=1
2022-02-21T17:44:58.352+0900    INFO    nodevalue/client.go:359 Analyse node id ns=4;i=1
2022-02-21T17:44:58.376+0900    INFO    nodevalue/client.go:359 Analyse node id ns=4;i=25
2022-02-21T17:44:58.385+0900    INFO    nodevalue/client.go:381 Add new node to list: ID: ns=4;i=25| Type bool| Name variable1
2022-02-21T17:44:58.394+0900    INFO    nodevalue/client.go:359 Analyse node id ns=4;i=26
2022-02-21T17:44:58.403+0900    INFO    nodevalue/client.go:381 Add new node to list: ID: ns=4;i=26| Type bool| Name variable2
2022-02-21T17:44:58.412+0900    INFO    nodevalue/client.go:359 Analyse node id ns=4;i=27
2022-02-21T17:44:58.431+0900    INFO    nodevalue/client.go:359 Analyse node id ns=4;i=28
2022-02-21T17:44:58.449+0900    INFO    nodevalue/client.go:359 Analyse node id ns=4;i=29

ns=4;i=28 NodeID is WORD type variabls.
Could you anyone help me to add node regarding of "WORD" type?
Here's OPC UA configuration applied.

# Module: opcua
# Docs: https://www.elastic.co/guide/en/beats/machinebeat//machinebeat-module-opcua.html

- module: opcua
  metricsets: ["nodevalue"]
  enabled: true
  period: 1s

  #The URL of your OPC UA Server
  endpoint: "opc.tcp://192.168.1.202:4840"

  #==========================  Security configuration ============================
  #username: ""
  #password: ""

  ##If you do not know which policies and modes are supported: start the beat in the debug mode and you will see which is supported by your server

  #Possible values: None, Basic128Rsa15, Basic256Sha256, Basic256, Aes128Sha256RsaOaep, Aes256Sha256RsaPss
  #policy: ""

  #Possible values: None, Sign, SignAndEncrypt
  #mode: ""

  ##Configuration for certificate based security
  #clientCert: ""
  #clientKey: ""

  ##This needs to be part of the certificates SAN URL and DNS-Name
  #appName: "machinebeat"

  #==========================  Data collection configuration ============================

  ##The subscribe mode collects one event whenever there is a change of the value. This is the default.
  ## All collected events will be send after the configured period
  ## If you turn this off the beat will pull the current value after each period.
  #subscribe: true

  ##The browse mode is enabled at default
  ## This means, if the configured nodes have childs all subscribable nodes will be monitored
  browse.enabled: true
  browse.maxLevel: 100
  browse.maxNodePerParent: 100

  ##How often we should retry to connect to the opcua server when something is failes after the first inital successful connection
  #retryOnError: 5

  ##How many collect threads should be opened. This is only relevant for non subscribe mode.
  ## This limit can be reached when it takes longer to get a value than it is configured with period.
  #maxThreads: 50

  #==========================  Node configuration ============================
  ##If this is not configured the browse will start at root. This is required if browse is set to false
  ## Configure the nodes directly to speed up start up of the beat
  nodes:
  -  id: "ns=4;i=1"
     label: "OPC UA"

Thanks in advance.

publishing 0 new events

Hi Felix,

I discovered a problem with the machinebeat, when I set the period quite low (less than 500ms).
When I'm setting the period to 500ms machinebeat works stable and publishes only a few 0 events.
When I'm setting the period to 100ms machinebeat throws (quite regularly) a too much zero publish attempts error and waits for reconnection.

2021-03-11T15:53:51.904+0100    INFO    nodevalue/nodevalue.go:215      [OPCUA] Publishing 7 new events
2021-03-11T15:53:52.004+0100    INFO    nodevalue/nodevalue.go:215      [OPCUA] Publishing 0 new events
2021-03-11T15:53:52.103+0100    INFO    nodevalue/nodevalue.go:215      [OPCUA] Publishing 0 new events
2021-03-11T15:53:52.203+0100    INFO    nodevalue/nodevalue.go:215      [OPCUA] Publishing 0 new events
2021-03-11T15:53:52.304+0100    INFO    nodevalue/nodevalue.go:215      [OPCUA] Publishing 0 new events
2021-03-11T15:53:52.404+0100    INFO    nodevalue/nodevalue.go:215      [OPCUA] Publishing 0 new events
2021-03-11T15:53:52.404+0100    INFO    nodevalue/nodevalue.go:206      [OPCUA] Too much zero publish attempts.
2021-03-11T15:53:52.503+0100    INFO    nodevalue/nodevalue.go:315      [OPCUA] Lets wait a while before reconnect happe
ns
2021-03-11T15:53:57.533+0100    INFO    nodevalue/client.go:68  [OPCUA] Get all endpoints from opc.tcp://MyOpcServer:4
840
2021-03-11T15:53:57.839+0100    INFO    nodevalue/client.go:77  [OPCUA] Your selected policy: http://opcfoundation.org/U
A/SecurityPolicy#None and security mode: MessageSecurityModeNone
2021-03-11T15:53:57.839+0100    INFO    nodevalue/client.go:87  [OPCUA] Policy URI: http://opcfoundation.org/UA/Security
Policy#None with security mode MessageSecurityModeNone
2021-03-11T15:53:57.839+0100    INFO    nodevalue/client.go:95  [OPCUA] Set ApplicationName to machinebeat
2021-03-11T15:53:57.839+0100    INFO    nodevalue/client.go:97  [OPCUA] Set ApplicationDescription (SAN DNS and SAN URL)
 to machinebeat
2021-03-11T15:53:57.839+0100    INFO    nodevalue/client.go:110 [OPCUA] Set to anonymous login
2021-03-11T15:53:58.228+0100    INFO    nodevalue/client.go:121 [OPCUA] Connection established
2021-03-11T15:53:58.228+0100    INFO    nodevalue/client.go:204 [OPCUA] Starting subscribe process
2021-03-11T15:53:58.228+0100    INFO    nodevalue/client.go:211 [OPCUA] Subscribe process done
2021-03-11T15:53:58.229+0100    INFO    nodevalue/nodevalue.go:215      [OPCUA] Publishing 0 new events

The configuration file looks like

- module: opcua
  metricsets: ["nodevalue"]
  enabled: true
  #period: 500ms
  period: 100ms

  #The URL of your OPC UA Server
  endpoint: "opc.tcp://MyOpcServer:4840"
  policy: "None"
  securityMode: "None"
  subscribe: true
  
  ##How often we should retry to connect to the opcua server
  #retryOnError: 5

  ##How many collect threads should be opened. This is only relevant for non subscription mode.  
  ## This limit can be reached when it takes longer to get a value than it is configured with period. 
  #maxThreads: 50
  
  browse.enabled: true

  nodes:
  -  id: "ns=4;s=var1"
     label: "var1"
  -  id: "ns=4;s=var2"
     label: "var2"
  -  id: "ns=4;s=var3"
     label: "var3"
  -  id: "ns=4;s=var4"
     label: "var4"
  -  id: "ns=4;s=var5"
     label: "var5"
  -  id: "ns=4;s=var6"
     label: "var6"

OPC-UA browsing doesn't work

I reopen this as a new issue after discussing it in another issue. Browsing does not work on our Gateway. Subscribing to explicitly configured nodes works and browsing also works in UAexpert.

Debug log:

2020-04-14T08:02:09.469Z        INFO    instance/beat.go:439    machinebeat start running.
2020-04-14T08:02:09.469Z        DEBUG   [cfgfile]       cfgfile/reload.go:133   Checking module configs from: /home/cleaned/machinebeat-test/modules.d/*.yml
2020-04-14T08:02:09.469Z        DEBUG   [cfgfile]       cfgfile/cfgfile.go:193  Load config from file: /home/cleaned/machinebeat-test/modules.d/opcua.yml
2020-04-14T08:02:09.469Z        DEBUG   [cfgfile]       cfgfile/reload.go:147   Number of module configs found: 1
2020-04-14T08:02:09.469Z        WARN    [cfgwarn]       nodevalue/nodevalue.go:92       BETA: The OPCUA metricset is beta.
2020-04-14T08:02:09.469Z        INFO    nodevalue/client.go:51  [OPCUA] Get all endpoints from opc.tcp://cleaned:cleaned
2020-04-14T08:02:13.162Z        INFO    nodevalue/client.go:64  [OPCUA] Policy URI: http://opcfoundation.org/UA/SecurityPolicy#None with security mode MessageSecurityModeNone
2020-04-14T08:02:13.162Z        INFO    nodevalue/client.go:80  [OPCUA] Set authentication information
2020-04-14T08:02:13.162Z        INFO    nodevalue/client.go:81  [OPCUA] User: cleaned
2020-04-14T08:02:20.725Z        INFO    nodevalue/client.go:95  [OPCUA] Connection established
2020-04-14T08:02:20.725Z        INFO    nodevalue/client.go:297 [OPCUA] Start browsing from node: root
2020-04-14T08:02:21.785Z        INFO    nodevalue/client.go:327 Analyse node id i=85
2020-04-14T08:02:21.785Z        DEBUG   [Get DataType]  nodevalue/client.go:388 Status not okay
2020-04-14T08:02:22.125Z        INFO    cfgfile/reload.go:171   Config reloader started
2020-04-14T08:02:22.125Z        DEBUG   [cfgfile]       cfgfile/reload.go:197   Scan for new config files
2020-04-14T08:02:22.125Z        DEBUG   [cfgfile]       cfgfile/cfgfile.go:193  Load config from file: /home/cleaned/machinebeat-test/modules.d/opcua.yml
2020-04-14T08:02:22.126Z        DEBUG   [cfgfile]       cfgfile/reload.go:216   Number of module configs found: 1
2020-04-14T08:02:22.126Z        DEBUG   [reload]        cfgfile/list.go:62      Starting reload procedure, current runners: 0
2020-04-14T08:02:22.126Z        DEBUG   [reload]        cfgfile/list.go:80      Start list: 1, Stop list: 0
2020-04-14T08:02:22.126Z        WARN    [cfgwarn]       nodevalue/nodevalue.go:92       BETA: The OPCUA metricset is beta.
2020-04-14T08:02:22.126Z        WARN    nodevalue/nodevalue.go:124      A new connection attempt was made. This gets ignored from this module
2020-04-14T08:02:22.126Z        DEBUG   [reload]        cfgfile/list.go:101     Starting runner: opcua [metricsets=1]
2020-04-14T08:02:22.126Z        DEBUG   [module]        module/wrapper.go:120   Starting Wrapper[name=opcua, len(metricSetWrappers)=1]
2020-04-14T08:02:22.126Z        INFO    cfgfile/reload.go:226   Loading of config files completed.
2020-04-14T08:02:22.126Z        DEBUG   [module]        module/wrapper.go:182   Starting metricSetWrapper[module=opcua, name=nodevalue, host=]
2020-04-14T08:02:22.127Z        INFO    nodevalue/nodevalue.go:185      [OPCUA] Publishing 0 new events
2020-04-14T08:02:25.127Z        INFO    nodevalue/nodevalue.go:185      [OPCUA] Publishing 0 new events
2020-04-14T08:02:28.127Z        INFO    nodevalue/nodevalue.go:185      [OPCUA] Publishing 0 new events
2020-04-14T08:02:31.127Z        INFO    nodevalue/nodevalue.go:185      [OPCUA] Publishing 0 new events
2020-04-14T08:02:34.127Z        INFO    nodevalue/nodevalue.go:185      [OPCUA] Publishing 0 new events
2020-04-14T08:02:34.127Z        INFO    nodevalue/nodevalue.go:176      [OPCUA] Too much zero publish attempts.

OPCUA: No boolean values if browse options explictily set

The default opcua.yml looks like this:

  ##The browse mode is enabled at default
  ## This means, if the configured nodes have childs all subscribable nodes will be monitored
  #browse.enabled: true
  #browse.maxLevel: 3
  #browse.maxNodePerParent: 5

If I uncomment the browse options machinebeat still works but is not subscribing or polling data points with opcua.node.DataType "bool"

Code for PLC4x

Dear Contributors,

First of all, Happy New Year!

I'd like to use PLC4x modules, but I cannot find any relevant code for explanation.
Does anyone guide me to use plc4x modules for Machinebeat?

Sincerely,

Nodes not being published

Hello,

I started playing around with machinebeat and found this strange behaviour.

Similar to #20 I find that it does not matter if enable or disable browsing by browse.enable: true or browse.enable: false I always see browsing happening in the debug view:

2020-08-14T14:38:21.487+0200    INFO    nodevalue/nodevalue.go:132      Browsing is enabled. Data collection will start after discovery. Based on your server and browsi
 configuration this can take some time.
2020-08-14T14:38:21.487+0200    INFO    nodevalue/client.go:304 [OPCUA] Start browsing from Objects and Views folder
2020-08-14T14:38:21.487+0200    DEBUG   [Browse]        nodevalue/client.go:341 Start browsing
2020-08-14T14:38:21.487+0200    INFO    nodevalue/client.go:349 Analyse node id i=85
2020-08-14T14:38:21.487+0200    DEBUG   [Get DataType]  nodevalue/client.go:425 This node has no data attached.
2020-08-14T14:38:21.488+0200    DEBUG   [Browse]        nodevalue/client.go:391 Found 6 new nodes for browsing with ref id 0
2020-08-14T14:38:21.488+0200    DEBUG   [Browse]        nodevalue/client.go:341 Start browsing

Furthermore the nodes that I would like to subscribe are not visible in logstash nor in elasticsearch.
The only events I see are the processor information from machinebeat.yml:

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Here is my opcua.yml file:

# Module: opcua
# Docs: https://www.elastic.co/guide/en/beats/machinebeat//machinebeat-module-opcua.html

- module: opcua
  metricsets: ["nodevalue"]
  enabled: true
  period: 1s
  
  #The URL of your OPC UA Server
  endpoint: "opc.tcp://localhost:55105"

  #==========================  Security configuration ============================
  #username: ""
  #password: ""
  
  ##If you do not know which policies and modes are supported: start the beat in the debug mode and you will see which is supported by your server
  
  #Possible values: None, Basic128Rsa15, Basic256Sha256, Basic256, Aes128Sha256RsaOaep, Aes256Sha256RsaPss
  #policy: ""
  
  #Possible values: None, Sign, SignAndEncrypt
  #mode: ""

  ##Configuration for certificate based security
  #clientCert: ""
  #clientKey: ""

  ##This needs to be part of the certificates SAN URL and DNS-Name 
  #appName: "machinebeat"
  
  #==========================  Data collection configuration ============================
  
  ##The subscribe mode collects one event whenever there is a change of the value. This is the default.
  ## All collected events will be send after the configured period
  ## If you turn this off the beat will pull the current value after each period.
  #subscribe: true

  ##The browse mode is enabled at default
  ## This means, if the configured nodes have childs all subscribable nodes will be monitored
  ## This does not work if subscribe: false
  browse.enable: false
  #maxLevel: 3
  #maxNodePerParent: 5
  
  ##How often we should retry to connect to the opcua server when something is failes after the first inital successful connection
  retryOnError: 5

  ##How many collect threads should be opened. This is only relevant for non subscribe mode.  
  ## This limit can be reached when it takes longer to get a value than it is configured with period. 
  #maxThreads: 50
  
  #==========================  Node configuration ============================
  ##If this is not configured the browse will start at root. This is required if browse is set to false
  ## Configure the nodes directly to speed up start up of the beat
 nodes:
-  id: "ns=6;s=S71500ET200MP-Station_1.Z105-CPU"
   label: "z105"

Might it be because of this following debug message:

20-08-14T15:41:15.057+0200    DEBUG   [Subscribe]     nodevalue/client.go:255 [OPCUA] Start listening
20-08-14T15:41:15.058+0200    DEBUG   [Subscribe]     nodevalue/client.go:262 [OPCUA] subscription=0 error=The server has reached the maximum number of queued publish requests. StatusBadTooManyPublishRequests (0x80780000)

Thank you for your help

Yours humbly nacho

kepware opcua connection result inquiry.

Hello. Thank you for developing machinebeat.

I used machinebeat to connect with kepware opcua.

The connection seems to be fine,
The data to be imported is not the node value that I want.

As shown in the image id: "ns=2;s=Accura/Accura2300/Current"
I want to bring the value
Keep getting the value "opcua.node.ID i=2258".

What should I modify to get the correct value?
Please help me.

opcua.yml

result

OPC UA Connection

I am testing machine beat for OPC UA communication with kepware opc UA server. I am successfully transferred one tag data to elastic search. But, If I configure more than one tag only last tag configured in yml file is transferring to elastic. Please help me out to solve this issue. Please find attached yml file.
opcua.zip

segfault with 7.6.2

I got a segfault after ~40 minutes with the new version

2020-04-14T08:06:51.082Z        INFO    nodevalue/nodevalue.go:185      [OPCUA] Publishing 16 new events
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x304b4af]

goroutine 9584 [running]:
github.com/elastic/machinebeat/module/opcua/nodevalue.collectData(0x3872978, 0x9, 0x38c0a46, 0x20, 0x0)
        /root/go_projects/src/github.com/elastic/machinebeat/module/opcua/nodevalue/client.go:116 +0x31f
github.com/elastic/machinebeat/module/opcua/nodevalue.collect(0xc0001cc700, 0x7f54a64c8ce0, 0xc0004a7c50, 0x23c7, 0xc0002679d0)
        /root/go_projects/src/github.com/elastic/machinebeat/module/opcua/nodevalue/nodevalue.go:160 +0x65
github.com/elastic/machinebeat/module/opcua/nodevalue.(*MetricSet).Fetch.func1(0xc0001cc700, 0x7f54a64c8ce0, 0xc0004a7c50)
        /root/go_projects/src/github.com/elastic/machinebeat/module/opcua/nodevalue/nodevalue.go:235 +0x3f
created by github.com/elastic/machinebeat/module/opcua/nodevalue.(*MetricSet).Fetch
        /root/go_projects/src/github.com/elastic/machinebeat/module/opcua/nodevalue/nodevalue.go:234 +0x231

MQTT Broker to run with Machinebeat

I'm trying to configure MQTT to acquire data from a field device. I will be using machinebeat for it. Reading the guide for configuration, It's required to use MQTT broker.

Any recommendations?

Thanks

Panic: runtime error: index out of range [33] with length 20

Hi all

Machinebeat encounter panic runtime error when i let it run for several hours. Below is the error message:

2022-02-09T03:05:34.038+0800	INFO	nodevalue/nodevalue.go:217	[OPCUA] Publishing 0 new events
2022-02-09T03:05:36.038+0800	INFO	nodevalue/nodevalue.go:217	[OPCUA] Publishing 0 new events
2022-02-09T03:05:38.038+0800	INFO	nodevalue/nodevalue.go:217	[OPCUA] Publishing 0 new events
2022-02-09T03:05:40.038+0800	INFO	nodevalue/nodevalue.go:217	[OPCUA] Publishing 0 new events
2022-02-09T03:05:42.038+0800	INFO	nodevalue/nodevalue.go:217	[OPCUA] Publishing 0 new events
2022-02-09T03:05:44.038+0800	INFO	nodevalue/nodevalue.go:217	[OPCUA] Publishing 0 new events
2022-02-09T03:05:46.038+0800	INFO	nodevalue/nodevalue.go:217	[OPCUA] Publishing 0 new events
panic: runtime error: index out of range [33] with length 20

goroutine 1139 [running]:
github.com/elastic/machinebeat/module/opcua/nodevalue.(*Client).subscribeTo(0xc0002ee880)
	C:/Users/FelixRoessel/Documents/Projekte/Machinebeat/src/github.com/elastic/machinebeat/module/opcua/nodevalue/client.go:286 +0xc32
created by github.com/elastic/machinebeat/module/opcua/nodevalue.(*Client).startSubscription
	C:/Users/FelixRoessel/Documents/Projekte/Machinebeat/src/github.com/elastic/machinebeat/module/opcua/nodevalue/client.go:209 +0x74

May I know what is causing this error?

How to get the mqtt message topic?

I'm using machinebeat to monitore some devices, but I need the mqtt topic where the message came from. When I see the generated document on elastic, it shows me this message:
"event": { "module": "mqtt", "creation": "2022-12-06T14:08:01.020Z", "dataset": "mqtt.topic", "provider": "mqtt", "url": "tcps://localhost:8883", "duration": 628392 },

Is there any way to get the message topic?

opcua.nodevalue.value can only handle one data type

I read several numeric values from our OPC-UA gateway which works fine. All values get written on opcua.nodevalue.value and I can filter bei node id or opcua.node.Name.
If add a node which has a non-numerical data type (e.g. string) it won't index the data because the Elastic mapping for opcua.nodevalue.value is already set to type:long.

It would be good to have a way to handle different data types

Noob question: What to do after MQTT reaches Elasticsearch?

Hello there, first, many thanks for this beat it had helped me to retrieve data from my sensors :-). Problem is that I am pretty new in Elasticsearch and really don't know too much about it, I come from an industrial programming, PLC, SCADA programming, etc.
If I go to analytics -> discovery I can actually see the data. My problem is that it arrives in string format, therefore in Kibana I cannot graph my data. My guess is that I've to convert the data from string to number at some point in the machinebeat.yml file, but honestly I don't know how to do it. do you think you can help me?

Thanks in advance,

Regarding anonymous login issue

Hello,

I'm writing to inquire about 'anonymous login' issue.
First of all, thank you for making machinebeat.

It looks a nice tool and I succeed to connect to the demo endpoint using your program without any problem.

demo endpoint: opc.tcp://opcuaserver.com:48010

However, I have a problem when I connect to my OPC UA server.
My local endpoint information like below and I checked the OPC UA server has no problem with another OPC/UA client tools

local endpoint: opc.tcp://host.docker.internal:53530/OPCUA/SimulationServer

Is there some configuration what I can do?
Here is disconnection full log information and thank you in advanced.

2020-08-11T17:44:35.246+0900    INFO    instance/beat.go:622    Home path: [C:\Users\dwku\OneDrive\바탕 화면\machinebeat] Config path: [C:\Users\dwku\OneDrive\바탕 화면\machinebeat] Data path: [C:\Users\dwku\OneDrive\바탕 화면\machinebeat\data] Logs path: [C:\Users\dwku\OneDrive\바탕 화면\machinebeat\logs]
2020-08-11T17:44:35.307+0900    INFO    instance/beat.go:630    Beat ID: d51d9d17-d145-489f-b43d-f9953bc4d771
2020-08-11T17:44:35.355+0900    INFO    [beat]  instance/beat.go:958    Beat info       {"system_info": {"beat": {"path": {"config": "C:\\Users\\dwku\\OneDrive\\바탕 화면\\machinebeat", "data": "C:\\Users\\dwku\\OneDrive\\바탕 화면\\machinebeat\\data", "home": "C:\\Users\\dwku\\OneDrive\\바탕 화면\\machinebeat", "logs": "C:\\Users\\dwku\\OneDrive\\바탕 화면\\machinebeat\\logs"}, "type": "machinebeat", "uuid": "d51d9d17-d145-489f-b43d-f9953bc4d771"}}}
2020-08-11T17:44:35.356+0900    INFO    [beat]  instance/beat.go:967    Build info      {"system_info": {"build": {"commit": "unknown", "libbeat": "7.6.1", "time": "1754-08-30T22:43:41.128Z", "version": "7.6.1"}}}
2020-08-11T17:44:35.356+0900    INFO    [beat]  instance/beat.go:970    Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.12.4"}}}
2020-08-11T17:44:35.358+0900    INFO    add_cloud_metadata/add_cloud_metadata.go:89     add_cloud_metadata: hosting provider type not detected.
2020-08-11T17:44:35.402+0900    INFO    [beat]  instance/beat.go:974    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-08-11T13:44:01.31+09:00","name":"SEOUL-DKU","ip":["fe80::cc0f:37ad:34f1:3009/64","169.254.48.9/16","fe80::90e8:fdde:a2e0:8875/64","169.254.136.117/16","fe80::49f8:3301:4b72:98ad/64","192.168.0.6/24","fe80::a4fd:5499:5ae7:de0a/64","169.254.222.10/16","::1/128","127.0.0.1/8","fe80::4928:1421:1cf2:2d4b/64","172.25.129.17/28"],"kernel_version":"10.0.17134.1610 (WinBuild.160101.0800)","mac":["92:32:4b:1c:90:47","d2:32:4b:1c:90:47","90:32:4b:1c:90:47","90:32:4b:1c:90:48","72:15:91:d4:83:ca"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"17134.1610"},"timezone":"KST","timezone_offset_sec":32400,"id":"e47d409c-3cca-409f-a573-66904f0b01e3"}}}
2020-08-11T17:44:35.405+0900    INFO    [beat]  instance/beat.go:1003   Process info    {"system_info": {"process": {"cwd": "C:\\Users\\dwku\\OneDrive\\바탕  화면\\machinebeat", "exe": "C:\\Users\\dwku\\OneDrive\\\ufffd\ufffd\ufffd\ufffd ȭ\ufffd\ufffd\\machinebeat\\machinebeat.exe", "name": "machinebeat.exe", "pid": 22124, "ppid": 22836, "start_time": "2020-08-11T17:44:34.360+0900"}}}
2020-08-11T17:44:35.406+0900    INFO    instance/beat.go:298    Setup Beat: machinebeat; Version: 7.6.1
2020-08-11T17:44:35.406+0900    INFO    [index-management]      idxmgmt/std.go:182      Set output.elasticsearch.index to 'machinebeat-7.6.1' as ILM is enabled.
2020-08-11T17:44:35.407+0900    INFO    elasticsearch/client.go:174     Elasticsearch url: http://localhost:9200
2020-08-11T17:44:35.407+0900    INFO    [publisher]     pipeline/module.go:110  Beat name: SEOUL-DKU
2020-08-11T17:44:35.408+0900    INFO    instance/beat.go:439    machinebeat start running.
2020-08-11T17:44:35.408+0900    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-08-11T17:44:35.410+0900    WARN    [cfgwarn]       nodevalue/nodevalue.go:92       BETA: The OPCUA metricset is beta.
2020-08-11T17:44:35.410+0900    INFO    nodevalue/client.go:54  [OPCUA] Get all endpoints from opc.tcp://host.docker.internal:53530/OPCUA/SimulationServer
2020-08-11T17:44:35.432+0900    INFO    nodevalue/client.go:68  [OPCUA] Policy URI: http://opcfoundation.org/UA/SecurityPolicy#None with security mode MessageSecurityModeNone
2020-08-11T17:44:35.432+0900    INFO    nodevalue/client.go:88  [OPCUA] Set to anonymous login
2020-08-11T17:44:36.449+0900    ERROR   nodevalue/nodevalue.go:165      [OPCUA] Tried to connect to OPC UA server 1 time(s). Without success.
2020-08-11T17:44:36.462+0900    INFO    [monitoring]    log/log.go:153  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":156,"time":{"ms":156}},"total":{"ticks":187,"time":{"ms":187},"value":187},"user":{"ticks":31,"time":{"ms":31}}},"handles":{"open":236},"info":{"ephemeral_id":"bd0cdfe6-5384-4376-9e79-4223acf6bc67","uptime":{"ms":2072}},"memstats":{"gc_next":5973680,"memory_alloc":3132680,"memory_total":6589824,"rss":34734080},"runtime":{"goroutines":17}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":8}}}}}
2020-08-11T17:44:36.462+0900    INFO    [monitoring]    log/log.go:154  Uptime: 2.0728753s
2020-08-11T17:44:36.463+0900    INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2020-08-11T17:44:36.464+0900    INFO    instance/beat.go:445    machinebeat stopped.
2020-08-11T17:44:36.465+0900    ERROR   instance/beat.go:933    Exiting: 1 error: Connection was not possible
Exiting: 1 error: Connection was not possible

MQTT module: SSL connection fails

Sorry, not sure if this is the right place to post this. I have added a topic to the Elastic forum on this issue here: https://discuss.elastic.co/t/machinebeat-mqtt-module-ssl-connection-to-broker-fails/206369 . Please let me know if this might be a bug in the module.

How to run machinebeat as service

Guys,
I have machinebeat set up and up and running. The only problem is that it's tied to the user logins. My question is how to run it as a service?

Thanks,

log4j vulnerability clarifications

Hi,

I am not sure if this is right way of getting this information.
But is this component affected by log4j vulnerability? And if so will you be making a release just to patch up the issue?

Regards
PK

Using Putty Terminal to install Machinebeat and collect data over OPC/MQTT into Logstach

I just started exploring elastic search and was trying to install machinebeat Linus/windows version via putty. Are there any documentation on how to proceed (steps) and set of line commands to accomplish this?

Thanks

Documentation link not valid

I am looking at the mqtt.yml.disabled file and the link:
https://www.elastic.co/guide/en/beats/machinebeat/machinebeat-module-mqtt.html

Does not work.

cannot find suitable endpoint error

Hi,

I am trying to connect to an OPC-UA Server, but without success. When I try to connect via (string) Hostname, I get an 'cannot find suitable endpoint' error:

PS C:\machinebeat\colvistec> .\machinebeat.exe -e -c machinebeat.yml -d "*"
2019-12-16T08:54:49.539+0100    INFO    instance/beat.go:606    Home path: [C:\machinebeat\colvistec] Config path: [C:\machinebeat\colvistec] Data path: [C:\machinebeat\colvistec\data] Logs path: [C:\machinebeat\colvistec\logs]
2019-12-16T08:54:49.542+0100    DEBUG   [beat]  instance/beat.go:658    Beat metadata path: C:\machinebeat\colvistec\data\meta.json
2019-12-16T08:54:49.544+0100    INFO    instance/beat.go:614    Beat ID: 4db98739-3d2b-4be2-bc75-0449e2469e59
2019-12-16T08:54:49.819+0100    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:164    add_cloud_metadata: starting to fetch metadata, timeout=3s
2019-12-16T08:54:49.911+0100    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:196    add_cloud_metadata: received disposition for qcloud after 85.1225ms. result=[provider:qcloud, error=failed requesting qcloud metadata: Get http://metadata.tencentyun.com/meta-data/placement/zone: dial tcp: lookup metadata.tencentyun.com: no such host, metadata={}]
2019-12-16T08:54:49.993+0100    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:196    add_cloud_metadata: received disposition for ecs after 166.8587ms. result=[provider:ecs, error=failed with http status code 401, metadata={}]
2019-12-16T08:54:52.833+0100    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:196    add_cloud_metadata: received disposition for digitalocean after 3.0063162s. result=[provider:digitalocean, error=failed requesting digitalocean metadata: Get http://169.254.169.254/metadata/v1.json: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), metadata={}]
2019-12-16T08:54:52.834+0100    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:203    add_cloud_metadata: timed-out waiting for all responses
2019-12-16T08:54:52.834+0100    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:167    add_cloud_metadata: fetchMetadata ran for 3.0073111s
2019-12-16T08:54:52.834+0100    INFO    add_cloud_metadata/add_cloud_metadata.go:347    add_cloud_metadata: hosting provider type not detected.
2019-12-16T08:54:52.835+0100    DEBUG   [processors]    processors/processor.go:101     Generated new processors: add_host_metadata=[netinfo.enabled=[false], cache.ttl=[5m0s]], add_cloud_metadata=null
2019-12-16T08:54:52.838+0100    DEBUG   [seccomp]       seccomp/seccomp.go:96   Syscall filtering is only supported on Linux
2019-12-16T08:54:52.839+0100    INFO    [beat]  instance/beat.go:902    Beat info       {"system_info": {"beat": {"path": {"config": "C:\\machinebeat\\colvistec", "data": "C:\\machinebeat\\colvistec\\data", "home": "C:\\machinebeat\\colvistec", "logs": "C:\\machinebeat\\colvistec\\logs"}, "type": "machinebeat", "uuid": "4db98739-3d2b-4be2-bc75-0449e2469e59"}}}
2019-12-16T08:54:52.844+0100    INFO    [beat]  instance/beat.go:911    Build info      {"system_info": {"build": {"commit": "unknown", "libbeat": "8.0.0", "time": "1754-08-30T22:43:41.128Z", "version": "8.0.0"}}}
2019-12-16T08:54:52.907+0100    INFO    [beat]  instance/beat.go:914    Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":2,"version":"go1.12.4"}}}
2019-12-16T08:54:52.914+0100    INFO    [beat]  instance/beat.go:918    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-04-01T08:47:25.03+02:00","name":"myBeat","ip":["fe80::b878:f783:db9:a3df/64","172.30.1.115/16","::1/128","127.0.0.1/8","fe80::5efe:ac1e:173/128"],"kernel_version":"6.3.9600.19153 (winblue_ltsb.180908-0600)","mac":["00:50:56:91:07:1c","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Datacenter","version":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19155"},"timezone":"CET","timezone_offset_sec":3600,"id":"3a43f2c3-68d2-47b0-bd20-bbc5831cf6c0"}}}
2019-12-16T08:54:52.919+0100    INFO    [beat]  instance/beat.go:947    Process info    {"system_info": {"process": {"cwd": "C:\\machinebeat\\colvistec", "exe": "C:\\machinebeat\\colvistec\\machinebeat.exe", "name": "machinebeat.exe", "pid": 79688, "ppid": 22244, "start_time": "2019-12-16T08:54:47.800+0100"}}}
2019-12-16T08:54:52.920+0100    INFO    instance/beat.go:292    Setup Beat: machinebeat; Version: 8.0.0
2019-12-16T08:54:52.921+0100    DEBUG   [beat]  instance/beat.go:318    Initializing output plugins
2019-12-16T08:54:52.922+0100    INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'machinebeat-8.0.0' as ILM is enabled.
2019-12-16T08:54:52.923+0100    INFO    elasticsearch/client.go:170     Elasticsearch url: http://svrehelk01.rehau.org:9200
2019-12-16T08:54:52.927+0100    DEBUG   [publisher]     pipeline/consumer.go:137        start pipeline event consumer
2019-12-16T08:54:52.928+0100    INFO    [publisher]     pipeline/module.go:97   Beat name: myBeat
2019-12-16T08:54:52.929+0100    DEBUG   [modules]       beater/metricbeat.go:121        Available modules and metricsets: Register [ModuleFactory:[beat, docker, elasticsearch, kibana, logstash, mongodb, mysql, postgresql, system, uwsgi, windows], MetricSetFactory:[aerospike/namespace, apache/status, beat/state, beat/stats, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/monitor_health, ceph/osd_df, ceph/osd_tree, ceph/pool_disk, consul/agent, couchbase/bucket, couchbase/cluster, couchbase/node, couchdb/server, docker/container, docker/cpu, docker/diskio, docker/event, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/ccr, elasticsearch/cluster_stats, elasticsearch/index, elasticsearch/index_recovery, elasticsearch/index_summary, elasticsearch/ml_job, elasticsearch/node, elasticsearch/node_stats, elasticsearch/pending_tasks, elasticsearch/shard, envoyproxy/server, etcd/leader, etcd/metrics, etcd/self, etcd/store, golang/expvar, golang/heap, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/stats, kibana/status, kubernetes/apiserver, kubernetes/container, kubernetes/controllermanager, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/proxy, kubernetes/scheduler, kubernetes/state_container, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_statefulset, kubernetes/system, kubernetes/volume, kvm/dommemstat, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mongodb/metrics, mongodb/replstatus, mongodb/status, mqtt/topic, munin/node, mysql/galera_status, mysql/status, nats/connections, nats/routes, nats/stats, nats/subscriptions, nginx/stubstatus, opcua/nodevalue, php_fpm/pool, php_fpm/process, postgresql/activity, postgresql/bgwriter, postgresql/database, postgresql/statement, prometheus/collector, rabbitmq/connection, rabbitmq/exchange, rabbitmq/node, rabbitmq/queue, redis/info, redis/key, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/memory, system/network, system/process, system/process_summary, system/raid, system/socket_summary, system/uptime, traefik/health, uwsgi/status, vsphere/datastore, vsphere/host, vsphere/virtualmachine, windows/perfmon, windows/service, zookeeper/connection, zookeeper/mntr, zookeeper/server]]
2019-12-16T08:54:52.936+0100    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2019-12-16T08:54:52.936+0100    INFO    instance/beat.go:421    machinebeat start running.
2019-12-16T08:54:52.936+0100    DEBUG   [service]       service/service_windows.go:72   Windows is interactive: true
2019-12-16T08:54:52.938+0100    DEBUG   [cfgfile]       cfgfile/reload.go:133   Checking module configs from: C:\machinebeat\colvistec/modules.d/colvistec.yml
2019-12-16T08:54:52.941+0100    DEBUG   [cfgfile]       cfgfile/cfgfile.go:175  Load config from file: C:\machinebeat\colvistec/modules.d/colvistec.yml
2019-12-16T08:54:52.943+0100    DEBUG   [cfgfile]       cfgfile/reload.go:147   Number of module configs found: 1
2019-12-16T08:54:52.944+0100    WARN    [cfgwarn]       nodevalue/nodevalue.go:71       BETA: The OPCUA metricset is beta.
2019-12-16T08:54:52.947+0100    INFO    nodevalue/client.go:41  [OPCUA] Get all endpoints from opc.tcp://Colvistec-PC:59595
2019-12-16T08:54:56.226+0100    ERROR   nodevalue/nodevalue.go:119      Tried to connect to OPC UA server 1 time(s). Without success.
2019-12-16T08:54:56.270+0100    INFO    [monitoring]    log/log.go:153  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":234,"time":{"ms":234}},"total":{"ticks":296,"time":{"ms":296},"value":296},"user":{"ticks":62,"time":{"ms":62}}},"handles":{"open":266},"info":{"ephemeral_id":"3fbd4a42-90b0-46ec-88d4-db62914d11e9","uptime":{"ms":7768}},"memstats":{"gc_next":4194304,"memory_alloc":3230760,"memory_total":4780720,"rss":27131904},"runtime":{"goroutines":16}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":2}}}}}
2019-12-16T08:54:56.273+0100    INFO    [monitoring]    log/log.go:154  Uptime: 7.8114225s
2019-12-16T08:54:56.273+0100    INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2019-12-16T08:54:56.274+0100    INFO    instance/beat.go:431    machinebeat stopped.
2019-12-16T08:54:56.275+0100    ERROR   instance/beat.go:877    Exiting: 1 error: [OPCUA] Failed to find suitable endpoint
Exiting: 1 error: [OPCUA] Failed to find suitable endpoint

And when I try to connect via IP I get

PS C:\machinebeat\MyMachine> .\machinebeat.exe -e -c machinebeat.yml -d "*"
2019-12-16T08:55:10.389+0100    INFO    instance/beat.go:606    Home path: [C:\machinebeat\MyMachine] Config path: [C:\machinebeat\MyMachine] Data path: [C:\machinebeat\MyMachine\data] Logs path: [C:\machinebeat\MyMachine\logs]
2019-12-16T08:55:10.392+0100    DEBUG   [beat]  instance/beat.go:658    Beat metadata path: C:\machinebeat\MyMachine\data\meta.json
2019-12-16T08:55:10.394+0100    INFO    instance/beat.go:614    Beat ID: 4db98739-3d2b-4be2-bc75-0449e2469e59
2019-12-16T08:55:10.412+0100    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:164    add_cloud_metada
ta: starting to fetch metadata, timeout=3s
2019-12-16T08:55:10.420+0100    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:196    add_cloud_metadata: received disposition for qcloud after 7.3431ms. result=[provider:qcloud, error=failed requesting qcloud metadata: Get http://metadata.tencentyun.com/meta-data/instance-id: dial tcp: lookup metadata.tencentyun.com: no such host, metadata={}]
2019-12-16T08:55:10.499+0100    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:196    add_cloud_metadata: received disposition for ecs after 85.9353ms. result=[provider:ecs, error=failed with http status code 401, metadata={}]
2019-12-16T08:55:13.416+0100    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:203    add_cloud_metadata: timed-out waiting for all responses
2019-12-16T08:55:13.418+0100    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:167    add_cloud_metadata: fetchMetadata ran for 3.004532s
2019-12-16T08:55:13.418+0100    INFO    add_cloud_metadata/add_cloud_metadata.go:347    add_cloud_metadata: hosting provider type not detected.
2019-12-16T08:55:13.418+0100    DEBUG   [processors]    processors/processor.go:101     Generated new processors: add_host_metadata=[netinfo.enabled=[false], cache.ttl=[5m0s]], add_cloud_metadata=null
2019-12-16T08:55:13.420+0100    DEBUG   [seccomp]       seccomp/seccomp.go:96   Syscall filtering is only supported on Linux2019-12-16T08:55:13.422+0100    INFO    [beat]  instance/beat.go:902    Beat info       {"system_info": {"beat": {"path": {"config": "C:\\machinebeat\\MyMachine", "data": "C:\\machinebeat\\MyMachine\\data", "home": "C:\\machinebeat\\MyMachine", "logs": "C:\\machinebeat\\MyMachine\\logs"}, "type": "machinebeat", "uuid": "4db98739-3d2b-4be2-bc75-0449e2469e59"}}}
2019-12-16T08:55:13.425+0100    INFO    [beat]  instance/beat.go:911    Build info      {"system_info": {"build": {"commit": "unknown", "libbeat": "8.0.0", "time": "1754-08-30T22:43:41.128Z", "version": "8.0.0"}}}
2019-12-16T08:55:13.427+0100    INFO    [beat]  instance/beat.go:914    Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":2,"version":"go1.12.4"}}}
2019-12-16T08:55:13.442+0100    INFO    [beat]  instance/beat.go:918    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-04-01T08:47:25.02+02:00","name":"myBeat","ip":[","fe80::5efe:ac1e:173/128"],"kernel_version":"6.3.9600.19153 (winblue_ltsb.180908-0600)","mac":["00:50:56:91:07:1c","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Datacenter","version":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19155"},"timezone":"CET","timezone_offset_sec":3600,"id":"3a43f2c3-68d2-47b0-bd20-bbc5831cf6c0"}}}
2019-12-16T08:55:13.446+0100    INFO    [beat]  instance/beat.go:947    Process info    {"system_info": {"process": {"cwd": "C:\\machinebeat\\MyMachine", "exe": "C:\\machinebeat\\MyMachine\\machinebeat.exe", "name": "machinebeat.exe", "pid": 67832, "ppid": 22244, "start_time": "2019-12-16T08:55:09.458+0100"}}}
2019-12-16T08:55:13.450+0100    INFO    instance/beat.go:292    Setup Beat: machinebeat; Version: 8.0.0
2019-12-16T08:55:13.451+0100    DEBUG   [beat]  instance/beat.go:318    Initializing output plugins
2019-12-16T08:55:13.453+0100    INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'machinebeat-8.0.0' as ILM is enabled.
2019-12-16T08:55:13.457+0100    INFO    elasticsearch/client.go:170     Elasticsearch url: http://myElasticStack:9200
2019-12-16T08:55:13.458+0100    DEBUG   [publisher]     pipeline/consumer.go:137        start pipeline event consumer
2019-12-16T08:55:13.458+0100    INFO    [publisher]     pipeline/module.go:97   Beat name: myBeat
2019-12-16T08:55:13.460+0100    DEBUG   [modules]       beater/metricbeat.go:121        Available modules and metricsets: Register [ModuleFactory:[beat, docker, elasticsearch, kibana, logstash, mongodb, mysql, postgresql, system, uwsgi, windows], MetricSetFactory:[aerospike/namespace, apache/status, beat/state, beat/stats, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/monitor_health, ceph/osd_df, ceph/osd_tree, ceph/pool_disk, consul/agent, couchbase/bucket, couchbase/cluster, couchbase/node, couchdb/server, docker/container, docker/cpu, docker/diskio, docker/event, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/ccr, elasticsearch/cluster_stats, elasticsearch/index, elasticsearch/index_recovery, elasticsearch/index_summary, elasticsearch/ml_job, elasticsearch/node, elasticsearch/node_stats, elasticsearch/pending_tasks, elasticsearch/shard, envoyproxy/server, etcd/leader, etcd/metrics, etcd/self, etcd/store, golang/expvar, golang/heap, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/stats, kibana/status, kubernetes/apiserver, kubernetes/container, kubernetes/controllermanager, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/proxy, kubernetes/scheduler, kubernetes/state_container, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_statefulset, kubernetes/system, kubernetes/volume, kvm/dommemstat, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mongodb/metrics, mongodb/replstatus, mongodb/status, mqtt/topic, munin/node, mysql/galera_status, mysql/status, nats/connections, nats/routes, nats/stats, nats/subscriptions, nginx/stubstatus, opcua/nodevalue, php_fpm/pool, php_fpm/process, postgresql/activity, postgresql/bgwriter, postgresql/database, postgresql/statement, prometheus/collector, rabbitmq/connection, rabbitmq/exchange, rabbitmq/node, rabbitmq/queue, redis/info, redis/key, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/memory, system/network, system/process, system/process_summary, system/raid, system/socket_summary, system/uptime, traefik/health, uwsgi/status, vsphere/datastore, vsphere/host, vsphere/virtualmachine, windows/perfmon, windows/service, zookeeper/connection, zookeeper/mntr, zookeeper/server]]
2019-12-16T08:55:13.470+0100    INFO    instance/beat.go:421    machinebeat start running.
2019-12-16T08:55:13.470+0100    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2019-12-16T08:55:13.470+0100    DEBUG   [service]       service/service_windows.go:72   Windows is interactive: true
2019-12-16T08:55:13.472+0100    DEBUG   [cfgfile]       cfgfile/reload.go:133   Checking module configs from: C:\machinebeat\MyMachine/modules.d/MyMachine.yml
2019-12-16T08:55:13.479+0100    DEBUG   [cfgfile]       cfgfile/cfgfile.go:175  Load config from file: C:\machinebeat\MyMachine/modules.d/MyMachine.yml
2019-12-16T08:55:13.482+0100    DEBUG   [cfgfile]       cfgfile/reload.go:147   Number of module configs found: 1
2019-12-16T08:55:13.483+0100    WARN    [cfgwarn]       nodevalue/nodevalue.go:71       BETA: The OPCUA metricset is beta.
2019-12-16T08:55:13.484+0100    INFO    nodevalue/client.go:41  [OPCUA] Get all endpoints from opc.tcp://1.2.3.4:59595
2019-12-16T08:55:13.512+0100    INFO    nodevalue/client.go:52  [OPCUA] Policy URI: http://opcfoundation.org/UA/SecurityPolicy#None with security mode MessageSecurityModeNone
2019-12-16T08:55:13.514+0100    INFO    nodevalue/client.go:63  User:
2019-12-16T08:55:13.515+0100    INFO    nodevalue/client.go:68  [OPCUA] Set to anonymous login
2019-12-16T08:55:16.783+0100    ERROR   nodevalue/nodevalue.go:119      Tried to connect to OPC UA server 1 time(s). Without success.
2019-12-16T08:55:16.786+0100    INFO    [monitoring]    log/log.go:153  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":62,"time":{"ms":62}},"total":{"ticks":155,"time":{"ms":155},"value":155},"user":{"ticks":93,"time":{"ms":93}}},"handles":{"open":265},"info":{"ephemeral_id":"07ded5b5-b36b-4fef-ae73-4d598c546602","uptime":{"ms":7297}},"memstats":{"gc_next":4769760,"memory_alloc":2479048,"memory_total":5119048,"rss":27840512},"runtime":{"goroutines":16}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":2}}}}}
2019-12-16T08:55:16.788+0100    INFO    [monitoring]    log/log.go:154  Uptime: 7.301856s
2019-12-16T08:55:16.788+0100    INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2019-12-16T08:55:16.789+0100    INFO    instance/beat.go:431    machinebeat stopped.
2019-12-16T08:55:16.791+0100    ERROR   instance/beat.go:877    Exiting: 1 error: could not resolve address MyMachine-PC:59595
Exiting: 1 error: could not resolve address MyMachine-PC:59595

The yml file contains

- module: opcua
  metricsets: ["nodevalue"]
  enabled: true
  period: 2s

  #The URL of your OPC UA Server
  endpoint: "opc.tcp://MyMachine-PC:59595"
  #endpoint: "opc.tcp://1.2.3.4:59595"
  policy: ""

and the node settings with IDs and labels.

Via UaExpert the connection is successful. Is this an issue or a config problem?

Kind regards,

Markus

No reconnect after connection to OPC server is lost

Machinebeat does not reconnect to the OPC nodes when the connection to the OPC server is interrupted. It states "Published 0 events" forever.

Linux machinebeat file not working

I am able to start the machinebeat in windows but linux one is not working

Issue with Node Configurations in OPCUA.yml

Hello,

I am attempting to collect data from specific nodes from a Kepware OPC server. I have added specific tags to the node configuration section and I have tried to turn browsing off. Attached is the opcua.yml config file and some debug info. When I try to start machinebeat, it appears that it tries to browse anyway and other information is being sent to Elastic.

Thanks for any help!

log.txt

opcua.txt

Multiple endpoints - connection issues and result data errors

I need to monitor some nodes from bunch of different endpoints. The issue arises when more than one endpoint is configured. When multiple endpoint are configured in opcua.yml, it is high chance to connect only to first endpoint. Also, there is no information in log file, that we are connected to another endpoint (only the first one).

Here is the opcua.yml config file.

# ----
# Endpoint 1
# ----
- module: opcua
  metricsets: ["nodevalue"]
  enabled: true
  period: 1s

  endpoint: "opc.tcp://<endpoint_IP_1:Port>"

  policy: "None"
  mode: "None"

  subscribe: true
  browse:
  - enabled: true
    maxLevel: 3
    maxNodePerParent: 5

  retryOnError: 5
  maxThreads: 50

  nodes:
  - id: "ns=3;s=Lebensbit[0]"
    label: "Lbit EP1"
  - id: "ns=3;s=Tag[0]"
    label: "Tag EP1"


# ----
# Endpoint 2
# ----
- module: opcua
  metricsets: ["nodevalue"]
  enabled: true
  period: 1s

  endpoint: "opc.tcp://<endpoint_IP_2:Port>"
  policy: "None"
  mode: "None"

  subscribe: true
  browse:
  - enabled: true
    maxLevel: 3
    maxNodePerParent: 5

  retryOnError: 5
  maxThreads: 50

  nodes:
  - id: "ns=3;s=Lebensbit[0]"
    label: "Lbit EP2"
  - id: "ns=3;s=Tag[0]"
    label: "Tag EP2"

Also, if those endpoints are connected, node labels on other endpoints are overriden by node labels from first endpoint.

Below is partial JSON data, first from EP1 (correct), second from EP2 (wrong)

{
  "opcua": {
    "nodevalue": {
      "state": "OK",
      "created": "2020-06-25 06:03:15.0713383 +0000 UTC",
      "uint32": 1
    },
    "node": {
      "Label": "Lbit EP1",
      "Name": "1000 Lebensbit",
      "DataType": "uint32",
      "ID": "ns=3;s=Lebensbit[0]"
    },
    "endpoint": "opc.tcp://<IP_Endpoint_1:Port>"
  }
}

{
  "opcua": {
    "nodevalue": {
      "created": "2020-06-25 06:03:16.0872965 +0000 UTC",
      "uint32": 1,
      "state": "OK"
    },
    "node": {
      "Name": "1000 Lebensbit",
      "DataType": "uint32",
      "ID": "ns=3;s=Lebensbit[0]",
      "Label": "LBit EP1"
    },
    "endpoint": "opc.tcp://<IP_Endpoint_2:Port>"
  }
}

If I'm wrong, please tell me, how should look the correct configuration file for multiple endpoint.

send data to elastic index

Hi Felix,

one more problem on my side:
I'm not able to write the collected data to an index differing from machinebeat-*

I've tried:

setup.template.pattern: "my_index-*"
setup.template.name: "my_index"

with a created index mapping:

{
  "index_patterns": [
    "my_index-*"
  ],
  "mappings": {
    "_meta": {
      "beat": "my_index",
      "version": "7.10.1"
    },
    "_source": {
      "enabled": false
    },
    "date_detection": false
  },
  "order": 1,
  "settings": {
    "index": {
      "lifecycle": {
        "name": "my_index_rollover",
        "rollover_alias": "my_index_rollover"
      },
      "number_of_shards": 1
    },
      "number_of_shards": 1
  }
}

and also with the standard setting of

setup.template.pattern: "machinebeat-*"
setup.template.name: "machinebeat"

my output setting is

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["my.elastic.server:9200"]
  index: "my_index-%{[agent.version]}-%{+yyyy_MM}"

  username: "uname"
  password: "pwd"

What am I doing wrong?

Thank you :D

opcua module label not set correctly

  nodes: 
  -  id: "ns=1;i=176000"
     label: "Zentrale - Summe der Wirkleistung"
  -  id: "ns=1;i=176080"
     label: "Zentrale - Temp_Umgebung_1"
  -  id: "ns1;i=176048"
     label: "Zentrale - Wirkarbeit"

Thats my node configuration in opcua module because the discovery does not work. However, only the last label is applied to all of the nodes. Is this a bug?

Kind regards
Conrad

username and password in opcua.yml

Hi,
I am trying to ingest data from OPC-UA to Elasticsearch. I wonder how to pass username and password in the opcua.yml. I tried by giving simply username and password, however it is not working. Please let me know if it is possible

module: opcua
metricsets: ["nodevalue"]
enabled: true
period: 2s
endpoint: "opc.tcp://{server}:port"
username: "username"
password: "password"

Future support/logstash alternative

Hello

As said in the first lines of the README, Machinebeat is experimental & not recommended to use in production. But my client would like to get the data from their OPC UA server into ElasticSearch.

As an alternative, I was thinking about using Logstash to read directly from the OPC UA server. Is this possible via some kind of logstash plugin? How is this done in other applications that use OPC UA?

Thx!

Failed to connect to backoff: Get EOF

machinebeat is running on my computer

IP: 192.168.137.120
OS: Windows11 21H2

I am sure that I can connect to the server from my computer: I use Elasticvue and it works well.

I also checked with curl:

> curl http://192.168.137.121:9200/


StatusCode        : 200
StatusDescription : OK
Content           : {
                      "name" : "server1",
                      "cluster_name" : "cpps-server",
                      "cluster_uuid" : "Cdy1jqpYTyWRWsHjU6NebQ",
                      "version" : {
                        "number" : "7.17.2",
                        "build_flavor" : "default",
                        "build_type" : "rpm"...
RawContent        : HTTP/1.1 200 OK
                    X-elastic-product: Elasticsearch
                    Warning: 299 Elasticsearch-7.17.2-de7261de50d90919ae53b0eff9413fd7e5307301 "Elast
                    icsearch built-in security features are not enabled. Without authent...
Forms             : {}
Headers           : {[X-elastic-product, Elasticsearch], [Warning, 299 Elasticsearch-7.17.2-de7261de5
                    0d90919ae53b0eff9413fd7e5307301 "Elasticsearch built-in security features are not
                     enabled. Without authentication, your cluster could be accessible to anyone. See
                     https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-se
                    tup.html to enable security."], [Content-Length, 535], [Content-Type, application
                    /json; charset=UTF-8]}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 535

I even have turned off all firewalls on server and my computer.

machinebeat configuration machinebeat.yml:

#==========================  Modules configuration ============================

machinebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  
#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["192.168.137.121"]

#================================ Processors =====================================
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

#================================ Logging =====================================
logging.level: debug

test:

> .\machinebeat.exe test output -c .\machinebeat.yml
elasticsearch: http://192.168.137.121:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.137.121
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... ERROR Get "http://192.168.137.121:9200": EOF

During this time, wireshark with tcp.port eq 9200 captures:

machinebeat run log

2022-04-08T22:12:28.557+0800	ERROR	[publisher_pipeline_output]	pipeline/output.go:154	Failed to connect to backoff(elasticsearch(http://192.168.137.121:9200)): Get "http://192.168.137.121:9200": EOF
2022-04-08T22:12:28.557+0800	INFO	[publisher_pipeline_output]	pipeline/output.go:145	Attempting to reconnect to backoff(elasticsearch(http://192.168.137.121:9200)) with 1 reconnect attempt(s)
2022-04-08T22:12:28.557+0800	DEBUG	[esclientleg]	eslegclient/connection.go:260	ES Ping(url=http://192.168.137.121:9200)
2022-04-08T22:12:28.557+0800	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2022-04-08T22:12:28.557+0800	INFO	[publisher]	pipeline/retry.go:223	  done
2022-04-08T22:12:28.557+0800	DEBUG	[esclientleg]	transport/logging.go:41	Completed dialing successfully	{"network": "tcp", "address": "127.0.0.1:1080"}
2022-04-08T22:12:28.558+0800	DEBUG	[esclientleg]	eslegclient/connection.go:264	Ping request failed with: Get "http://192.168.137.121:9200": EOF
2022-04-08T22:12:29.532+0800	INFO	nodevalue/nodevalue.go:217	[OPCUA] Publishing 15 new events

During running wireshark captures nothing with tcp.port eq 9200

I have been struggling with this for a long time, but have not found the reason, and I hope you can help me. Thank you!

Windows version of machinebeat has a malicious file

The windows version of the machinebeat has a malicious file flagged by Jiangmin scanner software. Please note that two other sites came up with the same result as well.

A new compiled version is required in order to use machinebeat.

security configuration with Aspentech IP21

I am trying to connect to Aspentech IP21 OPC UA server.
They sent certificate file as "xxxx.der". but I don't know how to use it in machinebeat.
Can i get some guide about it?

ARM build

We would like to run Machinebeat on a Raspberry Pi 4. Is it possible to cross compile for ARM8?

OPC UA Questions

Felix,

First off thank you for the OPC UA beat and the time and effort put forth. I am completely new to Elastic, Kibana, Beats etc.

I am a Automation/Robotics, and Windows developer, and have not really done other languages in depth. I installed the OPC UA beat and I have it working with one of our Skye Controls OPCUA IoT data tools.

We current have a set of IoT tools for the automation world.

As you can see by the picture there are 2 sessions active. Is this normal?

Also here is my OPCUA.yml file contents.

Module: opcua

Docs: https://www.elastic.co/guide/en/beats/machinebeat//machinebeat-module-opcua.html

module: opcua
metricsets: ["nodevalue"]
enabled: true
period: 1s

#The URL of your OPC UA Server
#endpoint: "opc.tcp://milo.digitalpetri.com:62541/milo"
#The URL of your OPC UA Server
endpoint: "opc.tcp://xx.xx.xx.xx:51310/SkyeDataHub/DataAccess"

#========================== Security configuration ============================
#username: ""
#password: ""

##If you do not know which policies and modes are supported: start the beat in the debug mode and you will see which is supported by your server

#Possible values: None, Basic128Rsa15, Basic256Sha256, Basic256, Aes128_Sha256_RsaOaep, Aes256_Sha256_RsaPss
#policy: ""

#Possible values: None, Sign, SignAndEncrypt
#securityMode: ""

##Configuration for certificate based security
#clientCert: ""
#clientKey: ""

##This needs to be part of the certificates SAN URL and DNS-Name
#appName: "machinebeat"

#========================== Data collection configuration ============================

##Machinebeat now introduces Elastic Common Schema(ECS) compliant fields accross all data sources. For migration you can choose to include or exclude fields
#ecsFields: true
#legacyFields: false

##The subscribe mode collects one event whenever there is a change of the value. This is the default.

All collected events will be send after the configured period

If you turn this off the beat will pull the current value after each period.

#subscribe: true

##The browse mode is enabled at default

This means, if the configured nodes have childs all subscribable nodes will be monitored

#browse.enabled: true
#browse.maxLevel: 3
#browse.maxNodePerParent: 5

##How often we should retry to connect to the opcua server when something is failes after the first inital successful connection
#retryOnError: 5

##How many collect threads should be opened. This is only relevant for non subscribe mode.

This limit can be reached when it takes longer to get a value than it is configured with period.

#maxThreads: 50

#========================== Node configuration ============================
##If this is not configured the browse will start at root. This is required if browse is set to false

Configure the nodes directly to speed up start up of the beat

#nodes:
#- id: "ns=2;s=Dynamic/RandomDouble"

label: "Random Double"

My next question is, on our Skye OPCUA/DA tool we can browse and configure the ndoes as expose by the server. See picture.

How can I do the same within the OPCUA beat?

I am assuming I need to add something within the .yml file.

Also how many OPC tags can the beat handle, we have some applications that could have 2000 OPC Tags for just one location.

We also have the ability in our OPCUA server to return "BaseDataType" to a UA Client reading an "Any" datapoint. BaseDataType is a valid UA datatype and it is capable of storing any value. Can the OPCUA beat handle a "BaseDataType"

Thank you so much for all your help in advance.

Regards,
Skye Controls

sensor.label and sensor.name having same value in Kibana

Hi,

I am not sure if this is expected but since there is an option of providing the label value in the opcua.yml I thought that label should be configurable by the user. Or it might be that I am looking at wrong field in Kibana.

Here is the opcua.yml:

- module: opcua
  metricsets: ["nodevalue"]
  enabled: true
  period: 1s
  
  #The URL of your OPC UA Server
  endpoint: "opc.tcp://thecompanyserver:49320"

  #browse.enabled: true

  nodes:
  -  id: "ns=2;s=Asset1.Block.Oven"
     label: "Oven"
  -  id: "ns=2;s=Asset1.Block.Robot"
     label: "Robot"

There are many tags (in Kepware speak) or variables or nodes (I am a newbie not familiar with OPC UA language) under both Asset1.Block.Oven as well as Asset1.Block.Robot node.

Project
  -- Connections
        -- Asset1
              -- Block
                     -- Oven
                         -- TC1
                         -- TC2
                         -- TC3
                     -- Robot
                         -- Angle1
                         -- Angle2
                         -- Angle3

Suppose there is a variable or tag (in Kepware speak) under Asset.One.Block.Oven called TC1.

My expectation was that I will see in Kibana:
sensor.name: TC1
sensor.label: Oven
value.value_float64: 123.23

However in Kibana I see:
sensor.name: TC1
sensor.label: TC1
value.value_float64: 123.23

Is it because there are nested elements/tags/variables under the nodes I have specified? I was hoping that they will be auto discovered and will have labels on them.

Machinebeat Version: v7.17.1, v7.17.0
Kibana: 7.16.3
Logstash: 7.13.2

Regards
Pankaj K

Apache Kafka Output possible?

The README mentions that Apache Kafka can be used to process the message queue instead of connecting directly to an Elasticsearch or Logstash instance. Now how would one go about configuring a Kafka output? The machinebeat.yml configuration file has entries for the Elasticsearch and Logstash outputs but none for Kafka.

I've successfully configured the OPC input module but how would one go about configuring the Kafka output module?

Also I noticed that the official link to the documentation in the OPC-UA module configuration file is no longer valid.

make setup does not seem to work

When it tries to run create-metricset it calls mage in the wrong location resulting in:

2020/06/18 15:55:32 Found Elastic Beats dir at /home/mpdreamz/projects/go/src/github.com/elastic/Machinebeat/vendor/github.com/elastic/beats
Unknown target specified: createMetricset

If I update the make call so that the pwd is scoped to metricbeat.

--- a/Makefile
+++ b/Makefile
@@ -16,7 +16,7 @@ MAGE_IMPORT_PATH=${BEAT_PATH}/vendor/github.com/magefile/mage
 setup: copy-vendor git-init
        #call make recursively so we can reload the above include.
        #Only needed during the first setup phase, before /vendor exists
-       $(MAKE) create-metricset collect git-add
+       $(MAKE) -C $(ES_BEATS)/metricbeat create-metricset collect git-add

It will succeed to call mage createMetricet but then none of the file references seem to resolve:

make -C ./vendor/github.com/elastic/beats/metricbeat create-metricset collect git-add
make[1]: Entering directory '/home/mpdreamz/projects/go/src/github.com/elastic/Machinebeat/vendor/github.com/elastic/beats/metricbeat'
mage createMetricset
ERROR: Could not open requirements file: [Errno 2] No such file or directory: '/home/mpdreamz/projects/go/src/github.com/elastic/Machinebeat/libbeat/tests/system/requirements.txt'
Error: running "/home/mpdreamz/projects/go/src/github.com/elastic/Machinebeat/build/ve/linux/bin/pip install --quiet -Ur /home/mpdreamz/projects/go/src/github.com/elastic/Machinebeat/libbeat/tests/system/requirements.txt" failed with exit code 1

make --version

GNU Make 4.2.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

mage --version

Mage Build Tool v1.9.0-12-g310e198
Build Date: 2020-06-18T14:34:58+02:00
Commit: 310e198
built with: go1.14.4

go --version

go version go1.14.4 linux/amd64

Machinebeat logs having statement: "is something else entirely"

Hi,
I am reading this data off Kepware.
When I run the machinebeat (I have not yet configured a elasticsearch details into the machinebeat.yml) to see if it is able to read data in the first place. I see this:

2022-02-22T11:08:53.965+1100 INFO nodevalue/nodevalue.go:217 [OPCUA] Publishing 15 new events
21.30145263671875 is something else entirely
25 is something else entirely
21.40155029296875 is something else entirely
21.601648330688477 is something else entirely
21.40155029296875 is something else entirely
21.501453399658203 is something else entirely
21.101451873779297 is something else entirely
21.40155029296875 is something else entirely
21.40155029296875 is something else entirely
21.301549911499023 is something else entirely
17.165603637695312 is something else entirely

This behaviour is seen in the latest verstion 7.17.0

In version 7.13.2 this is not seen. It shows something like this:
2022-02-22T11:46:22.113+1100 INFO nodevalue/nodevalue.go:214 [OPCUA] Publishing 1 new events
2022-02-22T11:46:23.112+1100 INFO nodevalue/nodevalue.go:214 [OPCUA] Publishing 2 new events
2022-02-22T11:46:24.118+1100 INFO nodevalue/nodevalue.go:214 [OPCUA] Publishing 1 new events
2022-02-22T11:46:25.111+1100 INFO nodevalue/nodevalue.go:214 [OPCUA] Publishing 2 new events
2022-02-22T11:46:26.115+1100 INFO nodevalue/nodevalue.go:214 [OPCUA] Publishing 0 new events

Changes in opcua.yml are:
#The URL of your OPC UA Server
endpoint: "opc.tcp://ourserver:49320"

nodes:

id: "ns=2;s=demo.fixture.readings"
label: "fixture"

opcua: subscribe=false leads to ES error

We need the data time continuously through time so I set subscribe to false. When Logstash tries to write the data to ES I get an error.

[2020-04-08T11:18:56,291][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"machinebeat-opcua-2020.04", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x450bf4f9>], :response=>{"index"=>{"_index"=>"machinebeat-opcua-2020.04", "_type"=>"_doc", "_id"=>"aT6DWXEBX62O6uTftSed", "status"=>400, " error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"field name cannot be an empty string"}}}}}

I wrote the logstash events to a text file but can't spot any errors

linux error

hello,
when i run machinebeat on linux:
./machinebeat -e -c machinebeat.yml
I get the following error message:
Exiting: error loading config file: config file ("machinebeat.yml") must be owned by the user identifier (uid=0) or root
best regards, Víctor.

Ingest data via Modbus and BACnet communication protocols to Elastic

Hello Team,

Trying to ingest data from Modbus TCP and BACnet IP devices into Logstash. Seems like there is no driver so far to accomplish this since machinebeat only communicates over OPC UA.

Any help would be appreciated

Thanks

Panic while starting machinebeat

Hi !

I'm testing Machinebeat with an OPC Server emulator : https://www.prosysopc.com/products/opc-ua-simulation-server/

I just created a user "bdt" on this server.

Then, I updated the opcua.yml file with the URL, the user and the password.

Finally, I started the machinebeat.exe and had the following logs :

2020-06-29T17:20:58.951+0200	INFO	instance/beat.go:622	Home path: [D:\Projets\Elasticsearch\opc\machinebeat] Config path: [D:\Projets\Elasticsearch\opc\machinebeat] Data path: [D:\Projets\Elasticsearch\opc\machinebeat\data] Logs path: [D:\Projets\Elasticsearch\opc\machinebeat\logs]
2020-06-29T17:20:58.986+0200	INFO	instance/beat.go:630	Beat ID: 6b8a6085-8c43-4bc0-9a0b-458778cb2ff0
2020-06-29T17:20:59.069+0200	INFO	[beat]	instance/beat.go:958	Beat info	{"system_info": {"beat": {"path": {"config": "D:\\Projets\\Elasticsearch\\opc\\machinebeat", "data": "D:\\Projets\\Elasticsearch\\opc\\machinebeat\\data", "home": "D:\\Projets\\Elasticsearch\\opc\\machinebeat", "logs": "D:\\Projets\\Elasticsearch\\opc\\machinebeat\\logs"}, "type": "machinebeat", "uuid": "6b8a6085-8c43-4bc0-9a0b-458778cb2ff0"}}}
2020-06-29T17:20:59.070+0200	INFO	[beat]	instance/beat.go:967	Build info	{"system_info": {"build": {"commit": "unknown", "libbeat": "7.6.1", "time": "1754-08-30T22:43:41.128Z", "version": "7.6.1"}}}
2020-06-29T17:20:59.070+0200	INFO	[beat]	instance/beat.go:970	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.12.4"}}}
2020-06-29T17:20:59.070+0200	INFO	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2020-06-29T17:20:59.151+0200	INFO	[beat]	instance/beat.go:974	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-06-28T18:25:01.66+02:00","name":"BDT-10","ip":["fe80::c8f9:901b:268:4fbd/64","169.254.79.189/16","fe80::5c6f:1f32:38e6:607e/64","192.168.56.1/24","fe80::2cb5:c6ee:7f33:a1b7/64","169.254.161.183/16","fe80::3c67:74ce:ed22:e856/64","169.254.232.86/16","fe80::c930:cdf5:1426:1f82/64","192.168.125.10/30","fe80::751c:f339:36cd:f708/64","169.254.247.8/16","fe80::959d:f53c:7924:95f0/64","192.168.229.1/24","fe80::9c8a:74e7:1859:7a48/64","192.168.211.1/24","fe80::d520:6ad8:7d59:91/64","192.168.0.24/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.18362.900 (WinBuild.160101.0800)","mac":["e4:e7:49:43:ce:e1","0a:00:27:00:00:18","1c:1b:b5:10:94:5c","1e:1b:b5:10:94:5b","00:ff:60:49:f5:80","00:ff:6e:80:17:95","00:50:56:c0:00:01","00:50:56:c0:00:08","1c:1b:b5:10:94:5b"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"18363.900"},"timezone":"CEST","timezone_offset_sec":7200,"id":"cd71d502-a2d2-48ca-9c2a-d9adb85e1121"}}}
2020-06-29T17:20:59.154+0200	INFO	[beat]	instance/beat.go:1003	Process info	{"system_info": {"process": {"cwd": "D:\\Projets\\Elasticsearch\\opc\\machinebeat", "exe": "D:\\Projets\\Elasticsearch\\opc\\machinebeat\\machinebeat.exe", "name": "machinebeat.exe", "pid": 19436, "ppid": 19188, "start_time": "2020-06-29T17:20:57.774+0200"}}}
2020-06-29T17:20:59.154+0200	INFO	instance/beat.go:298	Setup Beat: machinebeat; Version: 7.6.1
2020-06-29T17:20:59.154+0200	INFO	[index-management]	idxmgmt/std.go:182	Set output.elasticsearch.index to 'machinebeat-7.6.1' as ILM is enabled.
2020-06-29T17:20:59.154+0200	INFO	elasticsearch/client.go:174	Elasticsearch url: http://localhost:9200
2020-06-29T17:20:59.155+0200	INFO	[publisher]	pipeline/module.go:110	Beat name: BDT-10
2020-06-29T17:20:59.155+0200	INFO	instance/beat.go:439	machinebeat start running.
2020-06-29T17:20:59.155+0200	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-06-29T17:20:59.155+0200	WARN	[cfgwarn]	nodevalue/nodevalue.go:92	BETA: The OPCUA metricset is beta.
2020-06-29T17:20:59.155+0200	INFO	nodevalue/client.go:54	[OPCUA] Get all endpoints from opc.tcp://BDT-10.GROUPE-SPC.local:53530/OPCUA/SimulationServer
2020-06-29T17:20:59.167+0200	INFO	nodevalue/client.go:68	[OPCUA] Policy URI: http://opcfoundation.org/UA/SecurityPolicy#None with security mode MessageSecurityModeNone
2020-06-29T17:20:59.167+0200	INFO	nodevalue/client.go:84	[OPCUA] Set authentication information
2020-06-29T17:20:59.167+0200	INFO	nodevalue/client.go:85	[OPCUA] User: bdt
2020-06-29T17:20:59.321+0200	INFO	nodevalue/client.go:99	[OPCUA] Connection established
2020-06-29T17:20:59.321+0200	INFO	nodevalue/nodevalue.go:132	Browsing is enabled. Data collection will start after discovery. Based on your server and browsing configuration this can take some time.
2020-06-29T17:20:59.322+0200	INFO	nodevalue/client.go:304	[OPCUA] Start browsing from Objects and Views folder
2020-06-29T17:20:59.322+0200	INFO	nodevalue/client.go:349	Analyse node id i=85
2020-06-29T17:20:59.345+0200	INFO	nodevalue/client.go:349	Analyse node id i=2253
2020-06-29T17:20:59.349+0200	INFO	[monitoring]	log/log.go:153	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":109,"time":{"ms":109}},"total":{"ticks":171,"time":{"ms":171},"value":171},"user":{"ticks":62,"time":{"ms":62}}},"handles":{"open":263},"info":{"ephemeral_id":"5f59b0df-151f-4ad9-8045-3f993b15eeb6","uptime":{"ms":1548}},"memstats":{"gc_next":5337456,"memory_alloc":4127496,"memory_total":8028656,"rss":35614720},"runtime":{"goroutines":18}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":8}}}}}
2020-06-29T17:20:59.349+0200	INFO	[monitoring]	log/log.go:154	Uptime: 1.5492895s
2020-06-29T17:20:59.349+0200	INFO	[monitoring]	log/log.go:131	Stopping metrics logging.
2020-06-29T17:20:59.349+0200	INFO	runtime/panic.go:522	machinebeat stopped.
2020-06-29T17:20:59.394+0200	FATAL	[machinebeat]	instance/beat.go:163	Failed due to panic.	{"panic": "runtime error: index out of range", "stack": "github.com/elastic/beats/libbeat/cmd/instance.Run.func1.1\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:164\nruntime.gopanic\n\tC:/Go/src/runtime/panic.go:522\nruntime.panicindex\n\tC:/Go/src/runtime/panic.go:44\ngithub.com/elastic/machinebeat/module/opcua/nodevalue.browse\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/machinebeat/module/opcua/nodevalue/client.go:359\ngithub.com/elastic/machinebeat/module/opcua/nodevalue.browse\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/machinebeat/module/opcua/nodevalue/client.go:369\ngithub.com/elastic/machinebeat/module/opcua/nodevalue.startBrowse\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/machinebeat/module/opcua/nodevalue/client.go:316\ngithub.com/elastic/machinebeat/module/opcua/nodevalue.New\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/machinebeat/module/opcua/nodevalue/nodevalue.go:135\ngithub.com/elastic/beats/metricbeat/mb.initMetricSets\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/beats/metricbeat/mb/builders.go:139\ngithub.com/elastic/beats/metricbeat/mb.NewModule\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/beats/metricbeat/mb/builders.go:65\ngithub.com/elastic/beats/metricbeat/mb/module.NewWrapper\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:84\ngithub.com/elastic/beats/metricbeat/mb/module.(*Factory).CheckConfig\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/beats/metricbeat/mb/module/factory.go:72\ngithub.com/elastic/beats/libbeat/cfgfile.(*Reloader).Check\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/beats/libbeat/cfgfile/reload.go:157\ngithub.com/elastic/beats/metricbeat/beater.(*Metricbeat).Run\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:255\ngithub.com/elastic/beats/libbeat/cmd/instance.(*Beat).launch\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:445\ngithub.com/elastic/beats/libbeat/cmd/instance.Run.func1\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:188\ngithub.com/elastic/beats/libbeat/cmd/instance.Run\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:189\nmain.main\n\tC:/Users/felix/projekte/machinebeat/src/github.com/elastic/machinebeat/main.go:21\nruntime.main\n\tC:/Go/src/runtime/proc.go:200"}

Can you tell me what went wrong ?

Thanks.

error

hello,

I get the following error:

2020-08-15T21:19:52.537Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://localhost:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

how can I solve it?
best regards, Víctor.

Issue sending OPC data to logstash from machinebeat

I'm working on sending OPC data to a docker ELK stack and running into an error that seems to be related to the machinebeat input into logstash. The attached screenshot is a portion of the logstash logs that seem to be pointing to the pipeline having an issue with input. I'm not sure why the connection is being reset, or what the pipeline error could be. I censored the remote address a bit. That is the host running the OPC server, with machinebeat forwarding data to the docker logstash instance.

Any thoughts on what this could be?

Can't build Machinebeat from source

First I changed beats version in go.mod from v7.6.1 to v7.6.2+incompatible because it couldn't find 7.6.1
I also had to create a "v7" folder under /src.../beats and copy the libbeat folder into it because it couldn't find the sources.
I'm now stuck here (btw: what to enter for module name and metricset?)

vendor/github.com/elastic/beats/metricbeat/Makefile:84: Warnung: Die Befehle für das Ziel „integration-tests“ werden überschrieben
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:216: Warnung: Alte Befehle für das Ziel „integration-tests“ werden ignoriert
mkdir -p vendor/github.com/elastic
cp -R /src/github.com/elastic/beats vendor/github.com/elastic/
ln -sf /home/pi/Machinebeat/vendor/github.com/elastic/beats/metricbeat/scripts/generate_imports_helper.py /home/pi/Machinebeat/vendor/github.com/elastic/beats/script/generate_imports_helper.py
rm -rf vendor/github.com/elastic/beats/.git vendor/github.com/elastic/beats/x-pack
mkdir -p vendor/github.com/magefile
cp -R /src/github.com/elastic/beats/vendor/github.com/magefile/mage vendor/github.com/magefile
git init
Bestehendes Git-Repository in /home/pi/Machinebeat/.git/ neuinitialisiert
#call make recursively so we can reload the above include.
#Only needed during the first setup phase, before /vendor exists
make create-metricset collect git-add
make[1]: Verzeichnis „/home/pi/Machinebeat“ wird betreten
vendor/github.com/elastic/beats/metricbeat/Makefile:84: Warnung: Die Befehle für das Ziel „integration-tests“ werden überschrieben
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:216: Warnung: Alte Befehle für das Ziel „integration-tests“ werden ignoriert
Module name:
Metricset name:
Module already exists. Skipping creating module
Metricset already exists. Skipping creating metricset
go run -mod=vendor ./vendor/github.com/elastic/beats/metricbeat/scripts/assets/assets.go ./vendor/github.com/elastic/beats/metricbeat/module
build github.com/elastic/beats/metricbeat/scripts/assets: cannot find module for path github.com/pkg/errors
make[1]: *** [vendor/github.com/elastic/beats/metricbeat/Makefile:77: assets] Fehler 1
make[1]: Verzeichnis „/home/pi/Machinebeat“ wird verlassen
make: *** [Makefile:19: setup] Fehler 2

OPCUA Int64 problem

Hello,

we have issue with recognition of Int64 datatype.

024-03-28T13:00:41.446+0100 INFO nodevalue/client.go:426 Analyse node id ns=2;s=50221101.production.actParts
2024-03-28T13:00:41.474+0100 DEBUG [Get DataType] nodevalue/client.go:526 Was not able to detect DataType

Is it possible to enable this kind of datatype?

ps. We found that this datatype is missing in client.go, func getDataType

func getDataType(value *ua.DataValue) string { switch err := value.Status; err { case ua.StatusOK: switch v := value.Value.NodeID().IntID(); v { case id.DateTime: return "time.Time" case id.Boolean: return "bool" case id.SByte: return "int8" case id.Int16: return "int16" case id.Int32: return "int32" case id.Byte: return "byte" case id.UInt16: return "uint16" case id.UInt32: return "uint32" case id.UtcTime: return "time.Time" case id.String: return "string" case id.Float: return "float32" case id.Double: return "float64" }

elastic / machinebeat Goto Github PK

machinebeat's Introduction

About Machinebeat

Let's test it

Getting Started with Machinebeat

ARM

Linux

Windows

Docker

Configurations

OPCUA Module

MQTT Module

PLC4X Module

How to build on your own environment

machinebeat's People

Contributors

Stargazers

Watchers

Forkers

machinebeat's Issues

Module: opcua

Docs: https://www.elastic.co/guide/en/beats/machinebeat//machinebeat-module-opcua.html

All collected events will be send after the configured period

If you turn this off the beat will pull the current value after each period.

This means, if the configured nodes have childs all subscribable nodes will be monitored

This limit can be reached when it takes longer to get a value than it is configured with period.

Configure the nodes directly to speed up start up of the beat

label: "Random Double"

Recommend Projects

Recommend Topics

Recommend Org