[Elastic-Agent] [Docker] Discuss: accessing logs from different container about elastic-agent HOT 14 OPEN

Hi,

Just move to elastic-agent with Fleet management.
And yes, that's something missin the "autodiscovery" feature to retrive all logs from docker and at same time having an option in different integrations to map the discovered logs to them (thinking MySQL, Traefik, Apache, Nginx ...)
I think an integration "Docker Logs" with the autodiscovery options configurable in the fleet management will be a starting point and ability for any integration that request a log file to use this instead as index.

from elastic-agent.

Hi all.
Sorry for resurrecting such an old thread, but I think it's not resolved yet.
I think it could be a good idea to be able to call modules as if they where processors.
Something like this:

- id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}
  type: filestream
  use_output: default
  meta:
    package:
      name: kubernetes
      version: 1.29.2
  data_stream:
    namespace: default
  streams:
    - id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}
      data_stream:
        dataset: kubernetes.container_logs
        type: logs
      prospector.scanner.symlinks: true
      paths:
        - /var/log/containers/*${kubernetes.container.id}.log
      processors:
        - module:
            name: nginx
            when: ${kubernetes.labels.app} == "nginx"      

from elastic-agent.

Pinging @elastic/ingest-management (Team:Ingest Management)

from elastic-agent.

/cc @ruflin @michalpristas @ph

from elastic-agent.

I don't think we are quite there yet with the Docker/Kubernetes story with Agent. This really comes down to dynamic configurations and the ability for Agent to interact directly with Docker and Kubernetes.

from elastic-agent.

This really comes down to dynamic configurations and the ability for Agent to interact directly with Docker and Kubernetes.

This is the effort being tracked in elastic/beats#19225, yes?

from elastic-agent.

I understand. Thanks for responding! The reason why am I asking is because we'll need to somehow fetch logs from containers and wouldn't like to enforce a nasty workaround if there is a sophisticated method available.

EDIT:

There were few options on the table (rather quick wins, workarounds):

1. Install an agent on top of the other Docker image (with product like nginx, apache, etc.) or merge two images.
2. Expose via FUSE logs between Docker containers.

from elastic-agent.

@ycombinator elastic/beats#19255 is the one.

At the moment in 7.9 there is no sophisticated method.

from elastic-agent.

elastic/beats#19225 is part of the solution. @mtojek I assume we just beats, you solve this with autodiscovery? If you want all logs, I would assume the "old" trick around mounting volumes should still work? https://www.elastic.co/guide/en/beats/filebeat/master/running-on-docker.html#_volume_mounted_configuration It assumes the nginx logs are not stored inside the container but written to file by docker.

from elastic-agent.

Not sure if I follow your idea.

The Elastic Agent runs as binary in a Docker container, together with filebeat and metricbeat. My question is: how can I expose directories with logs to these processes?

According to what @blakerouse confirmed, there is no specific method, hence I'm asking for some official recommendation :)

from elastic-agent.

The way it is done today (see link I provided) is that the Container in which Filebeat is running (in your case the Agent), it mounts volumes from the Docker Host where these logs are stored. Now Filebeat (Agent) must be pointed to these directories with the logs and tail them. The above follows the assumption that you can the default json-file logging driver and nginx container writes to it (not the log file inside your container).

I think I miss something on why what is possible with Filebeat should not be possible with Agent?

from elastic-agent.

I think I miss something on why what is possible with Filebeat should not be possible with Agent?

Most likely that's possible, but I didn't measure performance here. I'm not sure if there were similar tests executed. I'm looking for the method which causes less problems, e.g. docker container restarts, suddenly unmounted volumes, missing permissions.

There are couple of follow up questions:
Which container exposes the volume (host vs guest)? What if Elastic Agent or Nginx got restarted?

from elastic-agent.

As this is a way we recommended for quite some time I would expect it to work fairly well. @exekias Perhaps you can chime in here?

For your follow up questions:

• Nginx does not need any volume as the logging driver writes it to the right place
• Agent container needs to mount the volume from the host machine. It is not that the 2 containers share volumes with each other
• This should also solve the problem around restarting. All container logs are shipped, so if it is a new container, it has a new id and on disk a new directory, but agent will just ship it too.

from elastic-agent.

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

from elastic-agent.