Problem with dnstap logger - forwarded data isn't recorded in Loki since version 0.30.0 about go-dnscollector HOT 6 CLOSED

A regression has been introduced in the v0.30.0 in the sniffer with the add of the new feature "IP defrag and TCP reassembly support" => The timestamp is outdated so ignored by loki.

A new beta release has been generated https://github.com/dmachard/go-dns-collector/releases/tag/v0.31.0-beta4 , can you test-it ?

By the way since 0.31.0, a new buffer has been introduced in the dnstap logger to avoid memory leak in some cases, the buffer can be configured (flush-interval and buffer-size )

    - name: dnstap
      dnstap:
        remote-address: 192.168.1.210
        remote-port: 6000
        retry-interval: 5
        flush-interval: 5
        buffer-size: 100

and the new config of the sniffer (with transformers)

multiplexer:
  collectors:
    - name: sniff
      afpacket-sniffer:
        port: 53
        device:
      transforms:
        latency:
          measure-latency: true
          queries-timeout: 2

I will add more tests to avoid this type of regression in the future

from go-dnscollector.

Thanks to sharing this issue and use case.
I will try to reproduce in my side

from go-dnscollector.

Just tested 0.31.0-beta4 and data is now getting to Loki as it used to with versions =< 0.29.0, thanks!

You said that the issue has been introduced in the sniffer and not in the DNStap logger as I previously thought. I've run tests using a sniffer collector and a Loki logger and I could swear the data was reaching its destination. I'm not overruling the possibility of experimenting error though. I can run some more tests if you want.

from go-dnscollector.

Thank for the quick test.
The issue occurs only with sniffer collector+dnstap logger , a specific function ToDnstap is used by the dnstap logger to send messages - witch is not the case with the Loki logger.

from go-dnscollector.

I get it, thanks for the clarification.

from go-dnscollector.

The release v0.31.0 is out, enjoy.

from go-dnscollector.