Hi,

My application outputs JSON formatted logs, which are do not have a guaranteed order of keys.
I'd like to capture key/value pairs by having capturing groups that look like:
/\\\"name\\\":\\\"(?P<application>[A-Za-z\-]+)/

Is there a mechanism to allow executing multiple matches against a single line in order to aggregate capturing groups?

thank you!

Unfortunately regexp parsing doesn't work super-well for JSON logs. I'm trying to parse the Suricata eve.json logs and things work fine until you get to escaped characters. I don't think there's a workable way to unescape escaped characters. And regardless, I'd rather not use regular expressions to parse JSON logs in the first place. Something like JMESPath with be vastly preferable for extracting JSON values from a log encoded in that format.

I was testing some modifications to a program, and I noticed that mtail reloaded it immediately, without me having the chance to run it with -compile_only. Moreover, something in the program made it crash completely:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x57cc71]

goroutine 1 [running]:
panic(0x88b600, 0xc82000e0e0)
    /usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
github.com/google/mtail/vm.(*compiler).compile(0xc8201f7668, 0x7a3a40, 0xc82007ba20)
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/vm/compiler.go:194 +0x5411
github.com/google/mtail/vm.(*compiler).compile(0xc8201f7668, 0x7a36e0, 0xc8201405a0)
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/vm/compiler.go:121 +0x1864
github.com/google/mtail/vm.(*compiler).compile(0xc8201f7668, 0x7a3800, 0xc820140630)
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/vm/compiler.go:83 +0xf07
github.com/google/mtail/vm.(*compiler).compile(0xc8201f7668, 0x7a3f20, 0xc82007ba00)
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/vm/compiler.go:68 +0x14a
github.com/google/mtail/vm.(*compiler).compile(0xc8201f7668, 0x7a3800, 0xc8201409f0)
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/vm/compiler.go:91 +0x105b
github.com/google/mtail/vm.(*compiler).compile(0xc8201f7668, 0x7a3f20, 0xc82007a2e0)
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/vm/compiler.go:68 +0x14a
github.com/google/mtail/vm.Compile(0xc82007a28b, 0x14, 0x7fcdc416e4a8, 0xc82007e020, 0xc8201400c0, 0x101, 0x91bcf8, 0x0, 0x0)
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/vm/compiler.go:43 +0x41f
github.com/google/mtail/vm.(*Loader).CompileAndRun(0xc820078420, 0xc82007a28b, 0x14, 0x7fcdc416e4a8, 0xc82007e020, 0x0, 0x0)
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/vm/loader.go:98 +0x8f
github.com/google/mtail/vm.(*Loader).LoadProg(0xc820078420, 0xc82007a280, 0x1f, 0x0, 0x0)
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/vm/loader.go:89 +0x3c0
github.com/google/mtail/vm.(*Loader).LoadProgs(0xc820078420, 0x7fffa7fc4888, 0xa, 0x0, 0x0)
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/vm/loader.go:67 +0x5ed
github.com/google/mtail/mtail.(*Mtail).InitLoader(0xc820148000, 0x0, 0x0)
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/mtail/mtail.go:121 +0x180
github.com/google/mtail/mtail.New(0x7fffa7fc4888, 0xa, 0xc82013e0e0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x91b6c8, 0x4, ...)
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/mtail/mtail.go:164 +0x175
main.main()
    /build/mtail-0.0+git20160704.35c4023/build/src/github.com/google/mtail/main.go:66 +0x5d2

Is there any way to have an action happen if a pattern /doesn't/ match? Go's regex library doesn't support negative-look-ahead, but i can't find any syntax in the mtail language which allows me to work around this.

What i'm trying to do is something like this:

/pattern1/ {
  do pattern1 stuff
}
/pattern2/ {
  do pattern2 stuff
}
otherwise {
  do other stuff
}

In my case the otherwise bit happens in the absence of the first two. Ideally, being able to chain if/else would be great, but in a pinch a simple 'else' keyword would allow this to be represented as:

/pattern1/ {
  do pattern1 stuff
} else {
  /pattern2/ {
    do pattern2 stuff
  } else {
    do other stuff
  }
}

If a key is missing when a metric is used in an action, mtail stops with a runtime error.

i.e.
prog:

counter n by method, time

/(?P<method>.*) (?P<time>\d+\.\d+)/ {
    n[$time]++
}

output:

panic: runtime error: index out of range

goroutine 23 [running]:
github.com/google/mtail/vm.(*VM).execute(0xc42019a280, 0xc42008a640, 0x1a, 0x7b34a0, 0xc42014c5f8)
        /home/test/go/src/github.com/google/mtail/vm/vm.go:600 +0x44a
github.com/google/mtail/vm.(*VM).processLine(0xc42019a280, 0xc420017cb0, 0xa)
        /home/test/go/src/github.com/google/mtail/vm/vm.go:676 +0x17a
github.com/google/mtail/vm.(*VM).Run(0xc42019a280, 0x74657374, 0xc420080780, 0xc4200807e0)
        /home/test/go/src/github.com/google/mtail/vm/vm.go:692 +0x138
created by github.com/google/mtail/vm.(*Loader).CompileAndRun

It would nice catching this error at compile time. Do you think it would be possible?

Hi,
we've observed mtail stop updating its exported metrics (and e.g. stop pushing to graphite) after a parse error has been encountered, in this form:

I1108 04:32:57.161927   16726 vm.go:176] Runtime error: conversion of "kernel: [1614712.454448] EDAC MC1: 1 CE" to numeric failed: strconv.ParseInt: parsing "kernel: [1614712.454448] EDAC MC1: 1 CE": invalid syntax
I1108 04:32:57.162025   16726 vm.go:177] VM stack:
goroutine 23 [running]:
runtime/debug.Stack(0x0, 0x0, 0x0)
/usr/lib/go-1.6/src/runtime/debug/stack.go:24 +0x80
github.com/google/mtail/vm.(*VM).errorf(0xc8201bebe0, 0x926478, 0x2, 0xc820205bd8, 0x1, 0x1)
/home/filippo/.go/src/github.com/google/mtail/vm/vm.go:177 +0xca
github.com/google/mtail/vm.(*VM).execute(0xc8201bebe0, 0xc820b76af0, 0xd, 0x0, 0x0)
/home/filippo/.go/src/github.com/google/mtail/vm/vm.go:382 +0x2c69
github.com/google/mtail/vm.(*VM).processLine(0xc8201bebe0, 0xc820cbcc00, 0xfd)
/home/filippo/.go/src/github.com/google/mtail/vm/vm.go:612 +0x1d1
github.com/google/mtail/vm.(*VM).Run(0xc8201bebe0, 0xc8200818c0, 0xc820081920)
/home/filippo/.go/src/github.com/google/mtail/vm/vm.go:626 +0x1a2
created by github.com/google/mtail/vm.(*Loader).CompileAndRun
/home/filippo/.go/src/github.com/google/mtail/vm/loader.go:130 +0x956
I1108 04:32:57.162040   16726 vm.go:178] Dumping vm state
I1108 04:32:57.162049   16726 vm.go:179] Name: kernel.mtail
I1108 04:32:57.162061   16726 vm.go:180] Input: "Nov  8 04:32:57 mw2213 kernel: [1614712.454448] EDAC MC1: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#0 (channel:1 slot:0 page:0x9a0b7a offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c2 socket:1 ha:0 channel_mask:2 rank:0)"
I1108 04:32:57.162081   16726 vm.go:181] Regexes:
I1108 04:32:57.162090   16726 vm.go:183] #011   0 (?P<date>(?P<legacy_date>\w+\s+\d+\s+\d+:\d+:\d+)|(?P<rfc3339_date>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d+[+-]\d{2}:\d{2}))\s+(?:\w+@)?(?P<hostname>[\w\.-]+)\s+(?P<application>[\w\.-]+)(?:\[(?P<pid>\d+)\])?:\s+(?P<message>.*)
I1108 04:32:57.162111   16726 vm.go:183] #011   1 kernel: \[[\d.]+\] mce: .*Machine check events logged
I1108 04:32:57.162136   16726 vm.go:183] #011   2 kernel: \[[\d.]+\] .* above threshold, cpu clock throttled
I1108 04:32:57.162145   16726 vm.go:183] #011   3 kernel: \[[\d.]+\] EDAC
I1108 04:32:57.162153   16726 vm.go:183] #011   4 kernel: \[[\d.]+\] EDAC MC\d+: (\d+) CE
I1108 04:32:57.162162   16726 vm.go:183] #011   5 kernel: \[[\d.]+\] EDAC MC\d+: (\d+) UE
I1108 04:32:57.162171   16726 vm.go:183] #011   6 kernel: \[[\d.]+\] init: (?P<service>[\S]+) .*process ended, respawning
I1108 04:32:57.162180   16726 vm.go:183] #011   7 kernel: \[[\d.]+\] (?P<binary>[\S]+) invoked oom-killer:.*
I1108 04:32:57.162189   16726 vm.go:183] #011   8 kernel: \[[\d.]+\] (?P<binary>[^\[]+)\[\d+\]: segfault at.*
I1108 04:32:57.162198   16726 vm.go:185] Strings:
I1108 04:32:57.162205   16726 vm.go:187] #011   0 "Jan _2 15:04:05"
I1108 04:32:57.162215   16726 vm.go:187] #011   1 "2006-01-02T03:04:05-0700"
I1108 04:32:57.162224   16726 vm.go:189] Thread:
I1108 04:32:57.162232   16726 vm.go:190] #011PC 57
I1108 04:32:57.162240   16726 vm.go:191] #011Match true
I1108 04:32:57.162248   16726 vm.go:192] #011Matched false
I1108 04:32:57.162255   16726 vm.go:193] #011Matches map[2:[] 3:[kernel: [1614712.454448] EDAC ] 4:[kernel: [1614712.454448] EDAC MC1: 1 CE 1] 0:[Nov  8 04:32:57 mw2213 kernel: [1614712.454448] EDAC MC1: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#0 (channel:1 slot:0 page:0x9a0b7a offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c2 socket:1 ha:0 channel_mask:2 rank:0) Nov  8 04:32:57 Nov  8 04:32:57  mw2213 kernel  [1614712.454448] EDAC MC1: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#0 (channel:1 slot:0 page:0x9a0b7a offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c2 socket:1 ha:0 channel_mask:2 rank:0)] 1:[]]
I1108 04:32:57.162353   16726 vm.go:194] #011Timestamp 2016-11-08 04:32:57 +0000 UTC
I1108 04:32:57.162381   16726 vm.go:195] #011Stack [0@0 0@0]
I1108 04:32:57.162402   16726 vm.go:196] Program:
I1108 04:32:57.162413   16726 vm.go:204]   #011   0        match 0
I1108 04:32:57.162427   16726 vm.go:204]   #011   1          jnm 100
I1108 04:32:57.162451   16726 vm.go:204]   #011   2   setmatched false
I1108 04:32:57.162462   16726 vm.go:204]   #011   3         push 0
I1108 04:32:57.162471   16726 vm.go:204]   #011   4       capref 2
I1108 04:32:57.162481   16726 vm.go:204]   #011   5       length 1
I1108 04:32:57.162490   16726 vm.go:204]   #011   6         push 0
I1108 04:32:57.162499   16726 vm.go:204]   #011   7          cmp 1
I1108 04:32:57.162511   16726 vm.go:204]   #011   8          jnm 15
I1108 04:32:57.162520   16726 vm.go:204]   #011   9   setmatched false
I1108 04:32:57.162530   16726 vm.go:204]   #011  10         push 0
I1108 04:32:57.162540   16726 vm.go:204]   #011  11       capref 1
I1108 04:32:57.162549   16726 vm.go:204]   #011  12          str 0
I1108 04:32:57.162558   16726 vm.go:204]   #011  13     strptime 2
I1108 04:32:57.162568   16726 vm.go:204]   #011  14   setmatched true
I1108 04:32:57.162577   16726 vm.go:204]   #011  15         push 0
I1108 04:32:57.162586   16726 vm.go:204]   #011  16       capref 3
I1108 04:32:57.162595   16726 vm.go:204]   #011  17       length 1
I1108 04:32:57.162604   16726 vm.go:204]   #011  18         push 0
I1108 04:32:57.162614   16726 vm.go:204]   #011  19          cmp 1
I1108 04:32:57.162623   16726 vm.go:204]   #011  20          jnm 27
I1108 04:32:57.162632   16726 vm.go:204]   #011  21   setmatched false
I1108 04:32:57.162641   16726 vm.go:204]   #011  22         push 0
I1108 04:32:57.162664   16726 vm.go:204]   #011  23       capref 3
I1108 04:32:57.162674   16726 vm.go:204]   #011  24          str 1
I1108 04:32:57.162683   16726 vm.go:204]   #011  25     strptime 2
I1108 04:32:57.162693   16726 vm.go:204]   #011  26   setmatched true
I1108 04:32:57.162703   16726 vm.go:204]   #011  27        match 1
I1108 04:32:57.162713   16726 vm.go:204]   #011  28          jnm 34
I1108 04:32:57.162722   16726 vm.go:204]   #011  29   setmatched false
I1108 04:32:57.162732   16726 vm.go:204]   #011  30        mload 0
I1108 04:32:57.162742   16726 vm.go:204]   #011  31        dload 0
I1108 04:32:57.162752   16726 vm.go:204]   #011  32          inc <nil>
I1108 04:32:57.162762   16726 vm.go:204]   #011  33   setmatched true
I1108 04:32:57.162772   16726 vm.go:204]   #011  34        match 2
I1108 04:32:57.162783   16726 vm.go:204]   #011  35          jnm 41
I1108 04:32:57.162793   16726 vm.go:204]   #011  36   setmatched false
I1108 04:32:57.162803   16726 vm.go:204]   #011  37        mload 1
I1108 04:32:57.162813   16726 vm.go:204]   #011  38        dload 0
I1108 04:32:57.162822   16726 vm.go:204]   #011  39          inc <nil>
I1108 04:32:57.162832   16726 vm.go:204]   #011  40   setmatched true
I1108 04:32:57.162841   16726 vm.go:204]   #011  41        match 3
I1108 04:32:57.162851   16726 vm.go:204]   #011  42          jnm 48
I1108 04:32:57.162861   16726 vm.go:204]   #011  43   setmatched false
I1108 04:32:57.162870   16726 vm.go:204]   #011  44        mload 4
I1108 04:32:57.162880   16726 vm.go:204]   #011  45        dload 0
I1108 04:32:57.162889   16726 vm.go:204]   #011  46          inc <nil>
I1108 04:32:57.162899   16726 vm.go:204]   #011  47   setmatched true
I1108 04:32:57.162909   16726 vm.go:204]   #011  48        match 4
I1108 04:32:57.162933   16726 vm.go:204]   #011  49          jnm 60
I1108 04:32:57.162945   16726 vm.go:204]   #011  50   setmatched false
I1108 04:32:57.162968   16726 vm.go:204]   #011  51        mload 2
I1108 04:32:57.162977   16726 vm.go:204]   #011  52        dload 0
I1108 04:32:57.162987   16726 vm.go:204]   #011  53        mload 2
I1108 04:32:57.162996   16726 vm.go:204]   #011  54        dload 0
I1108 04:32:57.163005   16726 vm.go:204]   #011  55         push 4
I1108 04:32:57.163015   16726 vm.go:204]   #011  56       capref 0
I1108 04:32:57.163024   16726 vm.go:204]  *#011  57          add <nil>
I1108 04:32:57.163033   16726 vm.go:204]   #011  58          set <nil>
I1108 04:32:57.163042   16726 vm.go:204]   #011  59   setmatched true
I1108 04:32:57.163052   16726 vm.go:204]   #011  60        match 5
I1108 04:32:57.163061   16726 vm.go:204]   #011  61          jnm 72
I1108 04:32:57.163072   16726 vm.go:204]   #011  62   setmatched false
I1108 04:32:57.163082   16726 vm.go:204]   #011  63        mload 3
I1108 04:32:57.163091   16726 vm.go:204]   #011  64        dload 0
I1108 04:32:57.163100   16726 vm.go:204]   #011  65        mload 3
I1108 04:32:57.163109   16726 vm.go:204]   #011  66        dload 0
I1108 04:32:57.163119   16726 vm.go:204]   #011  67         push 5
I1108 04:32:57.163128   16726 vm.go:204]   #011  68       capref 0
I1108 04:32:57.163137   16726 vm.go:204]   #011  69          add <nil>
I1108 04:32:57.163147   16726 vm.go:204]   #011  70          set <nil>
I1108 04:32:57.163156   16726 vm.go:204]   #011  71   setmatched true
I1108 04:32:57.163165   16726 vm.go:204]   #011  72        match 6
I1108 04:32:57.163174   16726 vm.go:204]   #011  73          jnm 81
I1108 04:32:57.163183   16726 vm.go:204]   #011  74   setmatched false
I1108 04:32:57.163192   16726 vm.go:204]   #011  75         push 6
I1108 04:32:57.163202   16726 vm.go:204]   #011  76       capref 1
I1108 04:32:57.163211   16726 vm.go:204]   #011  77        mload 5
I1108 04:32:57.163221   16726 vm.go:204]   #011  78        dload 1
I1108 04:32:57.163230   16726 vm.go:204]   #011  79          inc <nil>
I1108 04:32:57.163239   16726 vm.go:204]   #011  80   setmatched true
I1108 04:32:57.163249   16726 vm.go:204]   #011  81        match 7
I1108 04:32:57.163258   16726 vm.go:204]   #011  82          jnm 90
I1108 04:32:57.163267   16726 vm.go:204]   #011  83   setmatched false
I1108 04:32:57.163277   16726 vm.go:204]   #011  84         push 7
I1108 04:32:57.163286   16726 vm.go:204]   #011  85       capref 1
I1108 04:32:57.163295   16726 vm.go:204]   #011  86        mload 6
I1108 04:32:57.163304   16726 vm.go:204]   #011  87        dload 1
I1108 04:32:57.163314   16726 vm.go:204]   #011  88          inc <nil>
I1108 04:32:57.163325   16726 vm.go:204]   #011  89   setmatched true
I1108 04:32:57.163334   16726 vm.go:204]   #011  90        match 8
I1108 04:32:57.163344   16726 vm.go:204]   #011  91          jnm 99
I1108 04:32:57.163353   16726 vm.go:204]   #011  92   setmatched false
I1108 04:32:57.163362   16726 vm.go:204]   #011  93         push 8
I1108 04:32:57.163371   16726 vm.go:204]   #011  94       capref 1
I1108 04:32:57.163380   16726 vm.go:204]   #011  95        mload 7
I1108 04:32:57.163389   16726 vm.go:204]   #011  96        dload 1
I1108 04:32:57.163398   16726 vm.go:204]   #011  97          inc <nil>
I1108 04:32:57.163407   16726 vm.go:204]   #011  98   setmatched true
I1108 04:32:57.163416   16726 vm.go:204]   #011  99   setmatched true
I1108 04:33:02.043236   16726 export.go:119] pushing to graphite-in.eqiad.wmnet:2003
I1108 04:33:02.043908   16726 export.go:104] Sent 54 bytes

Note that the pushing to graphite keeps happening but graphite on the end reports all null, ditto for prometheus metrics: values reported are stuck to those before the error.

Hello,

I really like the simplicity of mtail, but for my usecase i need to report timings from log-files, and i saw this in the code

Is there any plans for adding explicit timer metrics, or is there some workaround that's usable?

Hi,

I am trying to monitor an application that makes some metrics available in an XML document and the rest I have to get by parsing logs. mtail looked like a good fit, but I wanted to also take the metrics and from the XML document and add them to /metrics. I thought I'll just write my own main.go and add the extra bits there. Here's where I ran into some trouble:

mtail does not use the official prometheus go client. If I want to add some metrics to /metrics, I can't use m.Run() or m.Serve(). There's a workaround - m.StartTailing() Then I can call m.WriteMetrics() and pip that json into a decoder, then decode it into a metrics struct.
This works, but it's not very convenient. I'm taking the unexported metrics store from m.Mtail, encoding it into json, then decoding it back into another struct of the same type.

I'd like to see one of these two changes in mtail if possible:

• rewrite the prometheus exporter to use the official prometheus client. This would be ideal as it would allow mtail to be easily extended with additional metrics that don't come from a log.
• change the Mtail struct to export Metrics directly without first encoding/decoding from JSON.

I'd be glad to submit a pull request with a prometheus exporter that uses the official client library if you're not opposed to such a change.

https://groups.google.com/d/topic/mtail-users/hdXIyiluDxc/discussion talks about mapping logs to programs and points out that running the apache program over the nginx logs is a waste of CPU. It really is.

This bug acknowledges the feature request, and hopefully comes up with a solution that avoids the issues I'm worried about:

• increasing maintenance cost of this mapping as the number of programs increases
• ensuring distribution and OS independence across log filename patterns

The Prometheus native data format is float64, but it appears mtail only stores metrics as int. It would be nice to be able to capture float values, for example 3.14 as a value in seconds.

It would be great to be able to read from systemd's journal directly, instead of going through a regular log file.

The C functions one would use are:

• sd_journal_open(3) or sd_journal_open_directory(3) (preferably make it configurable to specify the directory, for postfix running in containers),
• sd_journal_add_match(3) to match for the right unit,
• sd_journal_seek_head(3) (or sd_journal_seek_cursor(3) if you want to jump to a previous position),
• sd_journal_next(3) and sd_journal_get_data(3) to get the raw data.

More complex filters are certainly possible, but I don't know whether they are needed.

Hi!
I thought initially this was similar to #50, though there are no logs from mtail itself this time.
We are running a simple mtail instance to tail three log files (ranging from 300 lines/s to 1.2k lines/s roughly) on a central syslog host. The three files are rotated by logrotate daily, we see this in mtail logs at rotation time:

Jun 05 06:34:27 lithium mtail[28177]: I0605 06:34:27.635592   28177 tail.go:144] read /srv/syslog/syslog.log: file already closed
Jun 05 06:34:27 lithium mtail[28177]: I0605 06:34:27.670146   28177 tail.go:144] read /srv/syslog/swift.log: file already closed
...
Jun 05 06:34:27 lithium mtail[28177]: I0605 06:34:27.704315   28177 tail.go:144] read /srv/syslog/syslog.log: file already closed
Jun 05 06:34:27 lithium mtail[28177]: I0605 06:34:27.707416   28177 tail.go:315] Tailing /srv/syslog/syslog.log
Jun 05 06:34:27 lithium mtail[28177]: I0605 06:34:27.708463   28177 tail.go:315] Tailing /srv/syslog/swift.log
Jun 05 06:34:27 lithium mtail[28177]: I0605 06:34:27.708490   28177 tail.go:315] Tailing /srv/syslog/apache2.log

Though after about a week of mtail running (from git 9ae83e2) no more logs are emitted and mtail is effectively hung as far as metrics reporting is concerned (datapoints not pushed to graphite, asking prometheus metrics on /metrics just hangs)

I've dumped the goroutines while mtail is hanging at https://phabricator.wikimedia.org/P5569 and the respective heap at https://phabricator.wikimedia.org/P5568

Happy to do more testing if needed, hope this helps!

Hi, when metrics are exported by mtail, I notice that it automatically adds a reference to the mtail program as a label for the metric, e.g.: mymetric{prog="foo.mtail",instance="276d82f2092c"} 1

I was just wondering if it's possible to disable or remove this label via a configuration option. I ask because I am writing a generic exporter and I don't think it's relevant to expose the internals of how my mtail programs are set up to external consumers (or the fact that I am using mtail at all - as great as it is!). In my situation, there is no scenario where a user would want to query by the "progs" label.

Cheers.

The histogram examples that create Prometheus-compatible metrics (using an le label for distinguish buckets) should use an inclusive upper bound.

For example:

  $time_us < 5000 {
    apache_http_request_time_microseconds["5000"][$server_port][$handler][$request_method][$request_status][$request_protocol]++
  }

...should be:

  $time_us <= 5000 {
    apache_http_request_time_microseconds["5000"][$server_port][$handler][$request_method][$request_status][$request_protocol]++
  }

See: https://prometheus.io/docs/concepts/metric_types/#histogram

/cc @SuperQ

Hi,
Happy to accept I might be missing something here, but this commit bac6bb4#diff-7ddfb3e035b42cd70649cc33393fe32cR76 seems to have broken -oneshot.

mtail -one_shot -one_shot_metrics -logtostderr --progs ./progs/ --logs ./logs/else.log
Gives output like this:

I0705 03:01:07.568566    5802 main.go:54] mtail version  git revision  go version go1.7.4
I0705 03:01:07.584662    5802 loader.go:122] Loaded program else.mtail
I0705 03:01:07.584698    5802 loader.go:133] Program else.mtail has thread ID 656c7365
I0705 03:01:07.584819    5802 loader.go:122] Loaded program linecount.mtail
I0705 03:01:07.584830    5802 loader.go:133] Program linecount.mtail has thread ID 6c696e65

Removing the if !*oneShot { in main.go gives the expected output:

Oneshot results:
I0705 03:03:15.773862    5836 mtail.go:47] Oneshot "./logs/else.log"
./logs/else.log: 2 MAXPROCS, 2 CPUs, I0705 03:03:15.774151    5836 vm.go:576] Starting program linecount.mtail
I0705 03:03:15.774239    5836 vm.go:576] Starting program else.mtail
5 lines, 1.515613ms total time, 303.123 µs/line
Metrics store:{
  "line_count": [
    {
      "Name": "line_count",
 ...

A sigterm doesn't always cleanly terminate mtail. The process just hangs on 23:53:07.163583 25177 mtail.go:255] Shutdown requested.

While reading the sample programs, I noticed that there were features not described in the wiki, most notably, decorators and function calling.

Decorators seem self-evident (except for the "next" action), but I have no clue which functions are available, or even what actions are allowed. I wonder -for example- if I can overwrite a match group, or use metrics in conditionals.

Thanks!

Original issue 1 created by jaqx0r on 2012-01-17T03:35:28.000Z:

What steps will reproduce the problem?

1. Go to project website
2. Try and work out how to use tool
3. Fail to do so.

What is the expected output?

Some documentation.

What do you see instead?

Nothing, expect code.

Would be very helpful if /var/logs/*-request.log style paths were supported

thanks!

Below is crash dump, the program was run with:

$ mtail --logs timings.txt --progs timers --alsologtostderr=true -metric_push_interval_seconds=5 -statsd_hostport=localhost:8125

dump:

I0729 20:40:24.908709    1647 tail.go:297] Tailing /Users/david/Projects/go/src/github.com/google/mtail/timings.txt
I0729 20:40:24.909750    1647 export.go:154] Started metric push.
I0729 20:40:24.909797    1647 mtail.go:209] Listening on port 3903
panic: interface conversion: interface is nil, not time.Time

goroutine 19 [running]:
github.com/google/mtail/exporter.(*Exporter).WriteMetrics(0xc20802c7d0)
    /Users/david/Projects/go/src/github.com/google/mtail/exporter/export.go:131 +0x78
github.com/google/mtail/exporter.func·002()
    /Users/david/Projects/go/src/github.com/google/mtail/exporter/export.go:160 +0x74
created by github.com/google/mtail/exporter.(*Exporter).StartMetricPush
    /Users/david/Projects/go/src/github.com/google/mtail/exporter/export.go:163 +0x1d3

goroutine 1 [select]:
github.com/google/mtail/mtail.(*Mtail).shutdownHandler(0xc20807c0e0)
    /Users/david/Projects/go/src/github.com/google/mtail/mtail/mtail.go:232 +0x3e0
github.com/google/mtail/mtail.(*Mtail).Serve(0xc20807c0e0)
    /Users/david/Projects/go/src/github.com/google/mtail/mtail/mtail.go:215 +0x3d8
github.com/google/mtail/mtail.(*Mtail).Run(0xc20807c0e0)
    /Users/david/Projects/go/src/github.com/google/mtail/mtail/mtail.go:263 +0x42
main.main()
    /Users/david/Projects/go/src/github.com/google/mtail/main.go:70 +0x852

goroutine 6 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x5eab40)
    /Users/david/Projects/go/src/github.com/golang/glog/glog.go:879 +0x78
created by github.com/golang/glog.init·1
    /Users/david/Projects/go/src/github.com/golang/glog/glog.go:410 +0x2a7

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/Cellar/go/1.4.2/libexec/src/runtime/asm_amd64.s:2232 +0x1

goroutine 8 [syscall]:
os/signal.loop()
    /usr/local/Cellar/go/1.4.2/libexec/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /usr/local/Cellar/go/1.4.2/libexec/src/os/signal/signal_unix.go:27 +0x35

goroutine 9 [syscall]:
syscall.Syscall6(0x16b, 0x3, 0x0, 0x0, 0xc208018690, 0xa, 0x5f3570, 0x5761, 0xc2080422bc, 0x0)
    /usr/local/Cellar/go/1.4.2/libexec/src/syscall/asm_darwin_amd64.s:44 +0x5
syscall.kevent(0x3, 0x0, 0x0, 0xc208018690, 0xa, 0x5f3570, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.4.2/libexec/src/syscall/zsyscall_darwin_amd64.go:200 +0x81
syscall.Kevent(0x3, 0x0, 0x0, 0x0, 0xc208018690, 0xa, 0xa, 0x5f3570, 0x0, 0x0, ...)
    /usr/local/Cellar/go/1.4.2/libexec/src/syscall/syscall_bsd.go:480 +0x9b
gopkg.in/fsnotify%2ev1.read(0x3, 0xc208018690, 0xa, 0xa, 0x5f3570, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/david/Projects/go/src/gopkg.in/fsnotify.v1/kqueue.go:453 +0xab
gopkg.in/fsnotify%2ev1.(*Watcher).readEvents(0xc208042300)
    /Users/david/Projects/go/src/gopkg.in/fsnotify.v1/kqueue.go:262 +0x1a2
created by gopkg.in/fsnotify%2ev1.NewWatcher
    /Users/david/Projects/go/src/gopkg.in/fsnotify.v1/kqueue.go:61 +0x31f

goroutine 10 [chan receive]:
github.com/google/mtail/watcher.(*LogWatcher).run(0xc20802aab0)
    /Users/david/Projects/go/src/github.com/google/mtail/watcher/log_watcher.go:46 +0xf6
created by github.com/google/mtail/watcher.NewLogWatcher
    /Users/david/Projects/go/src/github.com/google/mtail/watcher/log_watcher.go:29 +0xf0

goroutine 11 [chan receive]:
github.com/google/mtail/vm.(*Loader).processEvents(0xc208042480)
    /Users/david/Projects/go/src/github.com/google/mtail/vm/loader.go:198 +0xde
created by github.com/google/mtail/vm.NewLoader
    /Users/david/Projects/go/src/github.com/google/mtail/vm/loader.go:184 +0x44b

goroutine 12 [chan receive]:
github.com/google/mtail/vm.(*Loader).processLines(0xc208042480, 0xc208042060)
    /Users/david/Projects/go/src/github.com/google/mtail/vm/loader.go:218 +0x71
created by github.com/google/mtail/vm.NewLoader
    /Users/david/Projects/go/src/github.com/google/mtail/vm/loader.go:185 +0x472

goroutine 13 [syscall]:
syscall.Syscall6(0x16b, 0x6, 0x0, 0x0, 0xc20801c690, 0xa, 0x5f3570, 0x5761, 0xc208042bbc, 0x0)
    /usr/local/Cellar/go/1.4.2/libexec/src/syscall/asm_darwin_amd64.s:44 +0x5
syscall.kevent(0x6, 0x0, 0x0, 0xc20801c690, 0xa, 0x5f3570, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.4.2/libexec/src/syscall/zsyscall_darwin_amd64.go:200 +0x81
syscall.Kevent(0x6, 0x0, 0x0, 0x0, 0xc20801c690, 0xa, 0xa, 0x5f3570, 0x0, 0x0, ...)
    /usr/local/Cellar/go/1.4.2/libexec/src/syscall/syscall_bsd.go:480 +0x9b
gopkg.in/fsnotify%2ev1.read(0x6, 0xc20801c690, 0xa, 0xa, 0x5f3570, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/david/Projects/go/src/gopkg.in/fsnotify.v1/kqueue.go:453 +0xab
gopkg.in/fsnotify%2ev1.(*Watcher).readEvents(0xc208042c00)
    /Users/david/Projects/go/src/gopkg.in/fsnotify.v1/kqueue.go:262 +0x1a2
created by gopkg.in/fsnotify%2ev1.NewWatcher
    /Users/david/Projects/go/src/gopkg.in/fsnotify.v1/kqueue.go:61 +0x31f

goroutine 14 [chan receive]:
github.com/google/mtail/watcher.(*LogWatcher).run(0xc20802b860)
    /Users/david/Projects/go/src/github.com/google/mtail/watcher/log_watcher.go:46 +0xf6
created by github.com/google/mtail/watcher.NewLogWatcher
    /Users/david/Projects/go/src/github.com/google/mtail/watcher/log_watcher.go:29 +0xf0

goroutine 15 [chan receive]:
github.com/google/mtail/tailer.(*Tailer).run(0xc208042cc0)
    /Users/david/Projects/go/src/github.com/google/mtail/tailer/tail.go:306 +0x9e
created by github.com/google/mtail/tailer.New
    /Users/david/Projects/go/src/github.com/google/mtail/tailer/tail.go:86 +0x507

goroutine 16 [chan receive]:
github.com/google/mtail/watcher.func·001()
    /Users/david/Projects/go/src/github.com/google/mtail/watcher/log_watcher.go:41 +0x6b
created by github.com/google/mtail/watcher.(*LogWatcher).run
    /Users/david/Projects/go/src/github.com/google/mtail/watcher/log_watcher.go:45 +0x9b

goroutine 18 [chan receive]:
github.com/google/mtail/watcher.func·001()
    /Users/david/Projects/go/src/github.com/google/mtail/watcher/log_watcher.go:41 +0x6b
created by github.com/google/mtail/watcher.(*LogWatcher).run
    /Users/david/Projects/go/src/github.com/google/mtail/watcher/log_watcher.go:45 +0x9b

goroutine 20 [IO wait]:
net.(*pollDesc).Wait(0xc2080103e0, 0x72, 0x0, 0x0)
    /usr/local/Cellar/go/1.4.2/libexec/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2080103e0, 0x0, 0x0)
    /usr/local/Cellar/go/1.4.2/libexec/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208010380, 0x0, 0x6b1d08, 0xc20802ab90)
    /usr/local/Cellar/go/1.4.2/libexec/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc20803a058, 0xdf384, 0x0, 0x0)
    /usr/local/Cellar/go/1.4.2/libexec/src/net/tcpsock_posix.go:234 +0x4e
net/http.tcpKeepAliveListener.Accept(0xc20803a058, 0x0, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.4.2/libexec/src/net/http/server.go:1976 +0x4c
net/http.(*Server).Serve(0xc2080425a0, 0x6b3998, 0xc20803a058, 0x0, 0x0)
    /usr/local/Cellar/go/1.4.2/libexec/src/net/http/server.go:1728 +0x92
net/http.(*Server).ListenAndServe(0xc2080425a0, 0x0, 0x0)
    /usr/local/Cellar/go/1.4.2/libexec/src/net/http/server.go:1718 +0x154
net/http.ListenAndServe(0xc20802ab88, 0x5, 0x0, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.4.2/libexec/src/net/http/server.go:1808 +0xba
github.com/google/mtail/mtail.func·001()
    /Users/david/Projects/go/src/github.com/google/mtail/mtail/mtail.go:210 +0x163
created by github.com/google/mtail/mtail.(*Mtail).Serve
    /Users/david/Projects/go/src/github.com/google/mtail/mtail/mtail.go:214 +0x3c5

It would be useful if you could do divisions (and multiplication if that's missing too), so you can perform simple calculations yourself. As an example, keeping an average, with something like below

counter requests_total by vhost
gauge total_request_time by vhost
gauge avg_request_time by vhost

/(?P<vhost>\S+) (?P<latency>\d+)/ {
  requests_total[$vhost]++
  total_request_time[$vhost] += $latency
  avg_request_time[$vhost] = $total_request_time[$vhost] / $requests_total[$vhost]
}

Input would look like:

example.com 1231
foobar.com 123

And so on.

Not sure if it's intended to be able to do these calculations yourself in mtail itself, or it's best left up to exporters to calculate these types of metrics.

https://groups.google.com/d/topic/mtail-users/7h1AVuY4YCE/discussion

The request is to annotate the exported metrics with some labels at a larger scope than just the log line.

The log filename could contain metadata about the log line, i.e. the hostname it arrived from, perhaps. This also raises the question, should you be able to add labels at higher levels too: the whole program, and mtail itself?

(The short answer to the latter, I think, is no: if you want to export a whole program label, you can do that already with constant values; if you want to apply a label globally, you can do that with the collecting system, e.g. with Prometheus, use the labels syntax to annotate with static labels, or use relabel_config: https://prometheus.io/docs/operating/configuration/#<relabel_config>)

The mtail program currently does not get to see the metadata of the log, such as its filename. The language assumes that a match is done solely against the log line.

This bug should figure out how to modify the language to pass the log filename to the program so that it can be matched upon, as well. If we discover other log metadata that should be available to the program, let's add it here too.

Checking the prometheus logs, I see this error:
text format parsing error in line 1: expected '"' at start of label value, found '2'

The first line is:

http_requests_total{code=200,content_type=application/vnd.google.protobuf,method=get,vhost=some.host,prog=nginx.mtail,instance=some.instance} 20

Is there a reason why mtail isn't quoting the values of labels?

I'm using the prometheus endpoint to collect metrics and have a rule with:

/^\w+\s+\d+\s+\d+:\d+:\d+\..+Retrieving pluginfacts$/ {
    puppet_last_run_timestamp = timestamp()
}

Sometimes this works, other times I'm getting a negative result:

root@bass00:/etc/mtail.d# curl http://localhost:3903/metrics
# TYPE puppet_last_run_timestamp gauge
puppet_last_run_timestamp{prog="puppet.mtail",instance="bass00"} -62135596800

Hi,

we monitor an exim log file using mtail that gets regularly rotated.

And sadly, frequently (but not always) mtail fails and fills its own log file with:

I0402 10:43:06.053308    2930 tail.go:139] read /var/log/exim4/mainlog: bad file descriptor

It there anything we can do here or is that some bug in mtail? Anyone else experiencing this?

Hello.

I would like to count on evergrowing set of metric indices (i.e. relay hosts from mail logs or similar case)
but it's clear that free memory on the host is likely to be exhausted after some time which makes mtail
less useful in general.

A solution to this "issue" may be to have the possibility to erase metrices that were successfuly
scraped.

This may be achieved by either allowing marking them with special keyword when declaring them (i.e. [hidden] session counter <metric> ...) and erasing these automatically after successful scrape or by extending the del functionality by extending the syntax with something like scrape del <metric> (ugly?) or with sort of section like awk does have (awk has BEGIN { ... } and END { ... } sections, mtail can have at least the END { ... } section to place anything right after every successful scrape, i.e. END { del <metric> } - I would prefare this one personally).

The maintenance of the metrices with evergrowing indices would be moved to a scraper this way, and on the scraper-side data retention policy could be set, some calculations done, new metrices created, etc.

I'm writing an mtail prog to extract metrics from strongSwan VPN daemon log (charon), I would like to measure the average duration it takes the server to reply to an IKE INIT request.

In essence, I want to measure the avg duration between two events "IKE_SA_INIT request" and "IKE_SA_INIT response".

Below's the block that takes care of that, it does the actual extraction very well, however this consumes a lot of CPU (within a few minutes consumed an entire core on the 8-core system and seemed like it would get worse if left running). I suspect that this is due to high amount of events (we have ~20-40 IKE INIT requests per second) so perhaps the two hidden gauges just grow in length out of control (?)

Since there could be duplicate requests for the same session, I'm also using ike_init_req_dup to make sure for the same pid I only count the request once.

counter strongswan_ike_init_request
counter strongswan_ike_init_response
counter strongswan_ike_init_time_total

hidden gauge ike_init_time by pid
hidden gauge ike_init_req_dup by pid

/^(?P<date>\d+-\d+-\d+ \d+:\d+:\d+\.\d+).*\<(?P<pid>\d+)\>/ {
  strptime($date, "2006-01-02 15:04:05.000")

  /IKE_SA_INIT request/ {
    ike_init_req_dup[$pid] != 1 {
      strongswan_ike_init_request++
      ike_init_time[$pid] = timestamp()
      ike_init_req_dup[$pid] = 1
    }
  }

  /IKE_SA_INIT response/ {
    strongswan_ike_init_response++
    strongswan_ike_init_time_total += timestamp() - ike_init_time[$pid]
  }
}

I'm not familiar with how to profile Go application, but I used perf top to get the breakdown on where mtail process spends its CPU cycles, here's what I got:

Samples: 8K of event 'cpu-clock', Event count (approx.): 1830781615                                                                                                                                 
Overhead  Shared Object     Symbol                                                                                                                                                                  
  21.20%  mtail             [.] runtime.memeqbody
  19.81%  mtail             [.] github.com/google/mtail/metrics.(*Metric).findLabelValueOrNil
   4.25%  mtail             [.] runtime.scanobject
   3.86%  mtail             [.] runtime.mallocgc
   3.42%  [kernel]          [k] _raw_spin_unlock_irqrestore
   2.53%  mtail             [.] runtime.rawstring
   2.30%  [kernel]          [k] xfrm_policy_match
   1.94%  mtail             [.] regexp.(*machine).tryBacktrack
   1.78%  mtail             [.] runtime.concatstrings
   1.65%  mtail             [.] runtime.greyobject
   1.47%  mtail             [.] runtime.heapBitsForObject
   1.28%  mtail             [.] github.com/google/mtail/tailer.(*Tailer).read
   1.24%  [kernel]          [k] queue_work_on
   1.15%  mtail             [.] runtime.eqstring
   1.13%  [kernel]          [k] 0x00007fff8171b9c8
   1.08%  mtail             [.] runtime.memmove

When I add a timer metric for a HAProxy log tailer, Prometheus can't scrape the /metrics endpoint.

text format parsing error in line 13: unknown metric type "timer"

The timer metric works well with the StatsD exporter, but to expose this kind of information to the pull-based Prometheus they use summaries for metrics representing request durations or response sizes.

Things that could be done

1. Don't publish timer metrics via Prometheus export
2. Add an explicit summary metric type
3. Back a timer with a summary metric, but only expose it fully during Prometheus export

Items 2. and 3. may or may not include bringing in the Prometheus packages.

It would be super-userful (particularly in conjunction with #21) to have the filename the line came from available as a variable.
An example use case - a directory full of access logs named by virtualhost where i'd like the vhost as a label. Being able to extract a substring from the filename would be a nice cherry on top.

Rather than as a builtin function, my initial thought was an option to have the filename prepended to the line - it could then be partially captured and used in the action?

It would be nice to have support for writing into OpenTSDB - it's a pretty widely used time series database.

hi,
thanks for mtail, looks really useful! Is reading from stdin something supported or considered? the use case I have in mind would be programs that don't log to file but in a ring memory buffer (e.g. varnish)

thanks!

the singe -port option is implemented in such a way as to make it impossible to selectively bind a specific interface.

It takes a string as an argument, but then that string is passed to http.ListenAndServe with a bare colon prepended to cause a default 0.0.0.0 binding.

https://github.com/google/mtail/blob/master/mtail/mtail.go#L288

I propose that either the code be modified to allow for a full address:port being passed from the -port flag OR adding an additional interface/address flag.

The use case is for exporting metrics on a multihomed machine where it is undesirable to have an http listener on every interface.

Hello,

We are currently having problems similar #50 where mtails remains unresponsive, if we check out commit 39de9dd things fail but still mtail is able to keep working.

My questions is why:

• 39de9dd fixes it
• and immediately c6b320f reverts the change?

Am I missing something? The latest commit still fails for me making mtail unusable as it will run for only a minute or so and then die.

Cheers

This partially relates to #40, although the phrasing of that ticket makes me think that the author may be after something else.

As noted here, mtail doesn't export a zero'd counter until it's incremented- ie, it's non zero. This is suboptimal for the following reasons:

1. the absence of a log line match is a metric. We have a pattern watching for buffer IO errors for example- a counter that should be zero 99.9-whatever% of the time. Via the metric not being fully exported to prometheus, the only way we can confirm that mtail is actually being scraped is checking other counters we collect via mtail.
2. Via not exporting zero'd counters, prometheus's rollover logic can misinterpret the value. Say I've got a counter of 'io_errors_total' that is currently at 6, and we're scraping every minute. We restart the mtail process (zero'ing the counter), 10 minutes later >6 matches come in- say 8. Prometheus should interpret that as the counter incrementing by 8- instead, since prometheus never saw the intermediate zero's (which the rate() logic would account for as a rollover), prometheus thinks the counter has increased by 2, rather than by 8.

Offhand, I couldn't find the exact spot that needed adjustment to force zero'd counters to be exported. What's a bit odd is that the type information is being exported, I just couldn't quickly spot where in EmitLabelSets that it was doing the filtering.

As discussed on https://groups.google.com/forum/#!topic/mtail-users/72_-p1wiPsY, would like to request support for being able to store key-value pairs for later use with the key being a string value.

The problem is that I'm trying to translate the log of an ftp service to metrics. The logs from the used 3rd software logs data at various intervals like below:

2017-04-28 12:00:00 123 127.0.0.1 Ftp session opened
2017-04-28 12:00:01 123 127.0.0.1 User 'A' logged in 
2017-04-28 12:00:02 123 127.0.0.1 12 bytes downloaded
2017-04-28 12:00:04 123 127.0.0.1 8 bytes downloaded
2017-04-28 12:00:05 123 127.0.0.1 Ftp session closed
....
2017-04-28 12:00:09 123 0.0.0.0 Ftp session opened
2017-04-28 12:00:10 123 0.0.0.0 User 'B' logged in 

Would like to expose metrics based on how long a user session lasted, as well as how many bytes were downloaded in total by a user. The log above is in the format

   <timestamp> <pid> <ip> <message>

Some pseudocode in mtail I would be looking to use:

    counter bytes_downloaded by user,ip
hidden counter tmp_bytes_downloaded by pid
map pid_to_user pid int->user string

/<timestamp> <pid> <ip>/ {
  strptime(<timestamp>)

	/FTP session opened/ {
		...
	}

	/<user> logged in / {
		len($user) > 0 {
			map[$pid] = $user
		}
	}

	/<bytes> downloaded/ {
		bytes_downloaded[$pid] += bytes
	}
 
	/FTP session closed/ {
		len(map[$pid]) > 0 {
			bytes_downloaded[map[$pid]][$ip] = bytes_downloaded[$pid]
		}
		del map[$pid]
		del tmp_bytes_downloaded[$pid]
	}
}

A oneliner explaining what this thing is doing would be fine; it's been >2 minutes and I still don't have a clue what this thing is about.

We're using mtail with a local ruleset to process logs from the Exim MTA. This has been running since Mar 30th and is now using 1.8G of RAM. The ruleset is a fairly simple set of regexp incrementing some counters - will attach examples and a /debug/pprof

Any thoughts?

I tried to build mtail for the first time this morning and got this error:

M00973008: /Users/x0ox/Dropbox/ActiveDev/go/src/github.com/google/mtail> make
# use prompt formatted as <hostname>: <path>/>
go generate -x ./vm
goyacc -v y.output -o parser.go -p mtail parser.y
vm/compiler.go:5: running "goyacc": exec: "goyacc": executable file not found in $PATH

I was able to workaround the error by:

1. creating the following alias:
   alias goyacc="go tool yacc"
2. doing a go build in the vm directory
3. re-running make

Our alerts started to fire on more and more machines about low available disk space. While that was clearly visible in df, nothing showed up when using du. Eventually I was able to track this down to mtail, restarting mtail (which was running for >1 month) freed up all the disk space immediately. I'm still investigating this issue.

• the log file is a svlogd controlled "current" symlink file
• no open file descriptors for (deleted) files are hold by mtail (the typical scenario for df/du differences)
• The log file show many Unexpected event: {"...."} lines. I believe because the DeleteEvent isn't handled and svlogd will briefly delete the symlink before creating it again.

Additionally it seems I can't kill mtail on some instances with a sigterm, only a hard kill worked. On some other nodes the metrics were stale.

While the last version was built on march 26th from master, I can't guarantee that version is actually used on all nodes, given a sigterm sometimes doesn't work.

Support for copytruncate when log rotating would be nice.

copytruncate
Truncate the original log file in place after creating a copy,
instead of moving the old log file and optionally creating a new
one. It can be used when some program cannot be told to close
its logfile and thus might continue writing (appending) to the
previous log file forever. Note that there is a very small time
slice between copying the file and truncating it, so some log-
ging data might be lost. When this option is used, the create
option will have no effect, as the old log file stays in place.

I posted an issue on prometheus prometheus/prometheus#1853 , i got a problem with the counter metrics that doesnt count from 0,

well, if i write mtail progs it would a long progs and the progs wouldn't be dynamic, any solution for this?

Thank You, 💘

Hi,
AFAICS changes in github.com/jaqx0r/afero have been merged upstream into github.com/spf13/afero, mtail could be switched back to that repo now?
I'm asking mostly because of Debian packaging of mtail, Debian has spf13/afero ATM.

I found a panic backtrace in mtail's logs which I've not been able to reproduce:

Jul 11 10:29:08 dev mtail: panic: runtime error: invalid memory address or nil pointer dereference
Jul 11 10:29:08 dev mtail: [signal 0xb code=0x1 addr=0x58 pc=0x574fb8]
Jul 11 10:29:08 dev mtail: goroutine 39 [running]:
Jul 11 10:29:08 dev mtail: panic(0x88c880, 0xc82002c060)
Jul 11 10:29:08 dev mtail:     /usr/local/go/src/runtime/panic.go:481 +0x3e6
Jul 11 10:29:08 dev mtail: github.com/google/mtail/tailer.(*Tailer).startNewFile(0xc82004ac40, 0x0, 0x0, 0x1, 0x0, 0x0)
Jul 11 10:29:08 dev mtail:     /build/mtail/tmp/build/gopath/src/github.com/google/mtail/tailer/tail.go:279 +0x48
Jul 11 10:29:08 dev mtail: github.com/google/mtail/tailer.(*Tailer).openLogPath(0xc82004ac40, 0xc820128740, 0x11, 0x1)
Jul 11 10:29:08 dev mtail:     /build/mtail/tmp/build/gopath/src/github.com/google/mtail/tailer/tail.go:269 +0x37c
Jul 11 10:29:08 dev mtail: github.com/google/mtail/tailer.(*Tailer).handleLogCreate.func1(0xc82004ac40, 0xc820128740, 0x11)
Jul 11 10:29:08 dev mtail:     /build/mtail/tmp/build/gopath/src/github.com/google/mtail/tailer/tail.go:221 +0x1f8
Jul 11 10:29:08 dev mtail: created by github.com/google/mtail/tailer.(*Tailer).handleLogCreate
Jul 11 10:29:08 dev mtail:     /build/mtail/tmp/build/gopath/src/github.com/google/mtail/tailer/tail.go:222 +0x765

It appears that this is due the fact that the file returned by os.NewFile on this line is not checked for a nil value.

There may be an issue with the numerical expressions or the mtail's type system as such:

$ cat progs/test.mtail
counter t by le
counter t_sum

/^(?P<v>\d+(\.\d+)?)/ {
    $v < 0.5 {
            t["0.5"]++
    }
    $v < 1 {
            t["1"]++
    }
    t["inf"]++
    t_sum += $v
}

$ touch test.log
$ mtail -progs progs -logs test.log

$ curl -s localhost:3903/metrics
# TYPE t counter
# TYPE t_sum counter
t_sum{prog="test.mtail",instance="test"} 0

$ echo 0.1 >>test.log
$ echo 1 >>test.log
$ echo 1.765 >>test.log

$ curl -s localhost:3903/metrics
# TYPE t counter
# TYPE t_sum counter
t_sum{prog="test.mtail",instance="test"} 0

Q1: Am I missing something? (BTW: mtail itself produces neither warning nor error and simply does nothing in such a configuration.)

Q2: How do I cope with mixed integer and floting-point values in logs (such as the delay= and delays= values in postfix logs)?

Sidenote:

I would like to suggest mtail behaves similarly to awk. Please refer to for instance gawk(1) and look up the "Variable Typing And Conversion" paragraph to know more on this topic. The awk's way is good-enough (and smart-enough) and would provide much more power to mtail users as well as may be compatible with most existing mtail programs (not sure).

Although I see these constants are defined in vm/paser.go which seems to be generated by parser.y, compilation fails for me with:

../github.com/google/mtail/vm/lexer.go:19: undefined: EOF
../github.com/google/mtail/vm/lexer.go:20: undefined: INVALID
../github.com/google/mtail/vm/lexer.go:21: undefined: LCURLY
../github.com/google/mtail/vm/lexer.go:22: undefined: RCURLY
../github.com/google/mtail/vm/lexer.go:23: undefined: LPAREN
../github.com/google/mtail/vm/lexer.go:24: undefined: RPAREN
../github.com/google/mtail/vm/lexer.go:25: undefined: LSQUARE
../github.com/google/mtail/vm/lexer.go:26: undefined: RSQUARE
../github.com/google/mtail/vm/lexer.go:27: undefined: COMMA
../github.com/google/mtail/vm/lexer.go:28: undefined: INC
../github.com/google/mtail/vm/lexer.go:28: too many errors
make: *** [.dep-stamp] Error 2

Hi,

I want to tail /var/log/exim4/mainlog so I run mtail using runit like this:

#!/bin/sh -e

export LC_ALL="en_US.UTF-8"

exec 2>&1 \
    chpst -u Debian-exim:Debian-exim \
        mtail -progs /etc/mtail -logs /var/log/exim4/mainlog -logtostderr

That seems to work fine, except that my log is spammed with the following message multiple times per second:

@40000000563b39770fc60854 I1105 12:11:41.264553   20836 tail.go:130] No file descriptor found for "/var/log/exim4/rejectlog

At no point did I tell mtail to care about the reject log nor do I want to tail it since it would taint the metrics. What is going wrong here?

Why does mtail think it should tail an unrelated file that happens to be in the same directory and what can I do about it? It is generating quite a bit of log data…

I try to make mtail on docker machine but after two days without success. Please, can somene help me with some step by step instruction or does it exist binary for mtail.
Linux ubuntu is. I setup golang and all path. I was installed some Prometheus exporter in go and I don't have problem.
Please, help me.
Thank you.

Hey, it would be nice to have official releases of mtail when things are in good feature release points.

Also add travis/circleci type auto-builds as well, as we do for various Prometheus components.

Hello there!
Trying to install mtail, but i got some trouble with command make
Makefile:120: recipe for target '.dep-stamp' failed
make: *** [.dep-stamp] Error 1
OS Fedora 23 4.8.13 x86_64
[mrviii@madify go]$ go version
go version go1.5.4 linux/amd64
[mrviii@madify go]$ make

Install all dependencies, ensuring they're updated

go get -u -v archive/zip bufio bytes compress/bzip2 compress/flate compress/gzip compress/zlib container/heap container/list context crypto crypto/aes crypto/cipher crypto/des crypto/dsa crypto/ecdsa crypto/elliptic crypto/hmac crypto/md5 crypto/rand crypto/rc4 crypto/rsa crypto/sha1 crypto/sha256 crypto/sha512 crypto/subtle crypto/tls crypto/x509 crypto/x509/pkix debug/elf encoding encoding/asn1 encoding/base64 encoding/binary encoding/gob encoding/hex encoding/json encoding/pem encoding/xml errors expvar flag fmt github.com/golang/glog github.com/kr/fs github.com/pkg/errors github.com/pkg/sftp github.com/spf13/afero github.com/spf13/afero/mem go/ast go/build go/constant go/doc go/format go/importer golang.org/x/crypto/acme golang.org/x/crypto/acme/autocert golang.org/x/crypto/blowfish golang.org/x/crypto/cast5 golang.org/x/crypto/chacha20poly1305/internal/chacha20 golang.org/x/crypto/curve25519 golang.org/x/crypto/ed25519 golang.org/x/crypto/ed25519/internal/edwards25519 golang.org/x/crypto/nacl/secretbox golang.org/x/crypto/openpgp/armor golang.org/x/crypto/openpgp/elgamal golang.org/x/crypto/openpgp/errors golang.org/x/crypto/openpgp/packet golang.org/x/crypto/openpgp/s2k golang.org/x/crypto/pbkdf2 golang.org/x/crypto/pkcs12/internal/rc2 golang.org/x/crypto/poly1305 golang.org/x/crypto/salsa20/salsa golang.org/x/crypto/ssh golang.org/x/crypto/ssh/agent golang.org/x/crypto/ssh/terminal golang.org/x/net/bpf golang.org/x/net/context golang.org/x/net/html golang.org/x/net/html/atom golang.org/x/net/http2 golang.org/x/net/http2/hpack golang.org/x/net/idna golang.org/x/net/internal/iana golang.org/x/net/internal/socket golang.org/x/net/internal/timeseries golang.org/x/net/ipv4 golang.org/x/net/ipv6 golang.org/x/net/lex/httplex golang.org/x/net/webdav/internal/xml golang.org/x/net/websocket golang.org/x/sys/unix golang.org/x/text/cases golang.org/x/text/collate golang.org/x/text/encoding golang.org/x/text/encoding/charmap golang.org/x/text/encoding/htmlindex golang.org/x/text/encoding/internal golang.org/x/text/encoding/internal/identifier golang.org/x/text/encoding/japanese golang.org/x/text/encoding/korean golang.org/x/text/encoding/simplifiedchinese golang.org/x/text/encoding/traditionalchinese golang.org/x/text/encoding/unicode golang.org/x/text/internal golang.org/x/text/internal/colltab golang.org/x/text/internal/format golang.org/x/text/internal/gen golang.org/x/text/internal/stringset golang.org/x/text/internal/tag golang.org/x/text/internal/utf8internal golang.org/x/text/language golang.org/x/text/runes golang.org/x/text/secure/bidirule golang.org/x/text/transform golang.org/x/text/unicode/bidi golang.org/x/text/unicode/cldr golang.org/x/text/unicode/norm golang.org/x/text/width golang.org/x/tools/benchmark/parse golang.org/x/tools/blog golang.org/x/tools/blog/atom golang.org/x/tools/cmd/guru/serial golang.org/x/tools/container/intsets golang.org/x/tools/cover golang.org/x/tools/go/ast/astutil golang.org/x/tools/go/buildutil golang.org/x/tools/go/callgraph golang.org/x/tools/go/callgraph/cha golang.org/x/tools/go/callgraph/rta golang.org/x/tools/go/callgraph/static golang.org/x/tools/godoc golang.org/x/tools/godoc/analysis golang.org/x/tools/godoc/redirect golang.org/x/tools/godoc/static golang.org/x/tools/godoc/util golang.org/x/tools/godoc/vfs golang.org/x/tools/godoc/vfs/gatefs golang.org/x/tools/godoc/vfs/httpfs golang.org/x/tools/godoc/vfs/mapfs golang.org/x/tools/godoc/vfs/zipfs golang.org/x/tools/go/gcimporter15 golang.org/x/tools/go/internal/gccgoimporter golang.org/x/tools/go/loader golang.org/x/tools/go/pointer golang.org/x/tools/go/ssa golang.org/x/tools/go/ssa/interp golang.org/x/tools/go/ssa/ssautil golang.org/x/tools/go/types/typeutil golang.org/x/tools/imports golang.org/x/tools/playground golang.org/x/tools/playground/socket golang.org/x/tools/present golang.org/x/tools/refactor/eg golang.org/x/tools/refactor/importgraph golang.org/x/tools/refactor/rename golang.org/x/tools/refactor/satisfy go/parser gopkg.in/fsnotify.v1 go/printer go/scanner go/token go/types hash hash/crc64 hash/fnv html html/template image image/jpeg index/suffixarray io io/ioutil log math math/big math/rand mime net net/http net/http/cookiejar net/http/httptest net/http/httputil net/http/pprof net/textproto net/url os os/exec os/signal os/user path path/filepath reflect regexp regexp/syntax runtime runtime/debug runtime/pprof runtime/trace sort strconv strings sync sync/atomic syscall testing text/scanner text/tabwriter text/template time unicode unicode/utf16 unicode/utf8 unsafe
import "context": import path does not begin with hostname
package context: unrecognized import path "context"
github.com/golang/glog (download)
github.com/kr/fs (download)
github.com/pkg/errors (download)
github.com/pkg/sftp (download)
Fetching https://golang.org/x/crypto/ssh?go-get=1
Parsing meta tags from https://golang.org/x/crypto/ssh?go-get=1 (status code 200)
get "golang.org/x/crypto/ssh": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/ssh?go-get=1
get "golang.org/x/crypto/ssh": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
golang.org/x/crypto (download)
Fetching https://golang.org/x/crypto/curve25519?go-get=1
Parsing meta tags from https://golang.org/x/crypto/curve25519?go-get=1 (status code 200)
get "golang.org/x/crypto/curve25519": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/curve25519?go-get=1
get "golang.org/x/crypto/curve25519": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/ed25519?go-get=1
Parsing meta tags from https://golang.org/x/crypto/ed25519?go-get=1 (status code 200)
get "golang.org/x/crypto/ed25519": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/ed25519?go-get=1
get "golang.org/x/crypto/ed25519": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/ed25519/internal/edwards25519?go-get=1
Parsing meta tags from https://golang.org/x/crypto/ed25519/internal/edwards25519?go-get=1 (status code 200)
get "golang.org/x/crypto/ed25519/internal/edwards25519": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/ed25519/internal/edwards25519?go-get=1
get "golang.org/x/crypto/ed25519/internal/edwards25519": verifying non-authoritative meta tag
github.com/spf13/afero (download)
Fetching https://golang.org/x/text/transform?go-get=1
Parsing meta tags from https://golang.org/x/text/transform?go-get=1 (status code 200)
get "golang.org/x/text/transform": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/transform?go-get=1
get "golang.org/x/text/transform": verifying non-authoritative meta tag
Fetching https://golang.org/x/text?go-get=1
Parsing meta tags from https://golang.org/x/text?go-get=1 (status code 200)
golang.org/x/text (download)
Fetching https://golang.org/x/text/unicode/norm?go-get=1
Parsing meta tags from https://golang.org/x/text/unicode/norm?go-get=1 (status code 200)
get "golang.org/x/text/unicode/norm": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/unicode/norm?go-get=1
get "golang.org/x/text/unicode/norm": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/acme?go-get=1
Parsing meta tags from https://golang.org/x/crypto/acme?go-get=1 (status code 200)
get "golang.org/x/crypto/acme": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/acme?go-get=1
get "golang.org/x/crypto/acme": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/acme/autocert?go-get=1
Parsing meta tags from https://golang.org/x/crypto/acme/autocert?go-get=1 (status code 200)
get "golang.org/x/crypto/acme/autocert": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/acme/autocert?go-get=1
get "golang.org/x/crypto/acme/autocert": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/blowfish?go-get=1
Parsing meta tags from https://golang.org/x/crypto/blowfish?go-get=1 (status code 200)
get "golang.org/x/crypto/blowfish": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/blowfish?go-get=1
get "golang.org/x/crypto/blowfish": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/cast5?go-get=1
Parsing meta tags from https://golang.org/x/crypto/cast5?go-get=1 (status code 200)
get "golang.org/x/crypto/cast5": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/cast5?go-get=1
get "golang.org/x/crypto/cast5": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/chacha20poly1305/internal/chacha20?go-get=1
Parsing meta tags from https://golang.org/x/crypto/chacha20poly1305/internal/chacha20?go-get=1 (status code 200)
get "golang.org/x/crypto/chacha20poly1305/internal/chacha20": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/chacha20poly1305/internal/chacha20?go-get=1
get "golang.org/x/crypto/chacha20poly1305/internal/chacha20": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/nacl/secretbox?go-get=1
Parsing meta tags from https://golang.org/x/crypto/nacl/secretbox?go-get=1 (status code 200)
get "golang.org/x/crypto/nacl/secretbox": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/nacl/secretbox?go-get=1
get "golang.org/x/crypto/nacl/secretbox": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/poly1305?go-get=1
Parsing meta tags from https://golang.org/x/crypto/poly1305?go-get=1 (status code 200)
get "golang.org/x/crypto/poly1305": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/poly1305?go-get=1
get "golang.org/x/crypto/poly1305": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/salsa20/salsa?go-get=1
Parsing meta tags from https://golang.org/x/crypto/salsa20/salsa?go-get=1 (status code 200)
get "golang.org/x/crypto/salsa20/salsa": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/salsa20/salsa?go-get=1
get "golang.org/x/crypto/salsa20/salsa": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/openpgp/armor?go-get=1
Parsing meta tags from https://golang.org/x/crypto/openpgp/armor?go-get=1 (status code 200)
get "golang.org/x/crypto/openpgp/armor": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/openpgp/armor?go-get=1
get "golang.org/x/crypto/openpgp/armor": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/openpgp/errors?go-get=1
Parsing meta tags from https://golang.org/x/crypto/openpgp/errors?go-get=1 (status code 200)
get "golang.org/x/crypto/openpgp/errors": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/openpgp/errors?go-get=1
get "golang.org/x/crypto/openpgp/errors": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/openpgp/elgamal?go-get=1
Parsing meta tags from https://golang.org/x/crypto/openpgp/elgamal?go-get=1 (status code 200)
get "golang.org/x/crypto/openpgp/elgamal": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/openpgp/elgamal?go-get=1
get "golang.org/x/crypto/openpgp/elgamal": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/openpgp/packet?go-get=1
Parsing meta tags from https://golang.org/x/crypto/openpgp/packet?go-get=1 (status code 200)
get "golang.org/x/crypto/openpgp/packet": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/openpgp/packet?go-get=1
get "golang.org/x/crypto/openpgp/packet": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/openpgp/s2k?go-get=1
Parsing meta tags from https://golang.org/x/crypto/openpgp/s2k?go-get=1 (status code 200)
get "golang.org/x/crypto/openpgp/s2k": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/openpgp/s2k?go-get=1
get "golang.org/x/crypto/openpgp/s2k": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/pbkdf2?go-get=1
Parsing meta tags from https://golang.org/x/crypto/pbkdf2?go-get=1 (status code 200)
get "golang.org/x/crypto/pbkdf2": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/pbkdf2?go-get=1
get "golang.org/x/crypto/pbkdf2": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/pkcs12/internal/rc2?go-get=1
Parsing meta tags from https://golang.org/x/crypto/pkcs12/internal/rc2?go-get=1 (status code 200)
get "golang.org/x/crypto/pkcs12/internal/rc2": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/pkcs12/internal/rc2?go-get=1
get "golang.org/x/crypto/pkcs12/internal/rc2": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/ssh/agent?go-get=1
Parsing meta tags from https://golang.org/x/crypto/ssh/agent?go-get=1 (status code 200)
get "golang.org/x/crypto/ssh/agent": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/ssh/agent?go-get=1
get "golang.org/x/crypto/ssh/agent": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto/ssh/terminal?go-get=1
Parsing meta tags from https://golang.org/x/crypto/ssh/terminal?go-get=1 (status code 200)
get "golang.org/x/crypto/ssh/terminal": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/ssh/terminal?go-get=1
get "golang.org/x/crypto/ssh/terminal": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/bpf?go-get=1
Parsing meta tags from https://golang.org/x/net/bpf?go-get=1 (status code 200)
get "golang.org/x/net/bpf": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/bpf?go-get=1
get "golang.org/x/net/bpf": verifying non-authoritative meta tag
Fetching https://golang.org/x/net?go-get=1
Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
golang.org/x/net (download)
Fetching https://golang.org/x/net/context?go-get=1
Parsing meta tags from https://golang.org/x/net/context?go-get=1 (status code 200)
get "golang.org/x/net/context": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/context?go-get=1
get "golang.org/x/net/context": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/html?go-get=1
Parsing meta tags from https://golang.org/x/net/html?go-get=1 (status code 200)
get "golang.org/x/net/html": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/html?go-get=1
get "golang.org/x/net/html": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/html/atom?go-get=1
Parsing meta tags from https://golang.org/x/net/html/atom?go-get=1 (status code 200)
get "golang.org/x/net/html/atom": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/html/atom?go-get=1
get "golang.org/x/net/html/atom": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/http2?go-get=1
Parsing meta tags from https://golang.org/x/net/http2?go-get=1 (status code 200)
get "golang.org/x/net/http2": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/http2?go-get=1
get "golang.org/x/net/http2": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/http2/hpack?go-get=1
Parsing meta tags from https://golang.org/x/net/http2/hpack?go-get=1 (status code 200)
get "golang.org/x/net/http2/hpack": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/http2/hpack?go-get=1
get "golang.org/x/net/http2/hpack": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/idna?go-get=1
Parsing meta tags from https://golang.org/x/net/idna?go-get=1 (status code 200)
get "golang.org/x/net/idna": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/idna?go-get=1
get "golang.org/x/net/idna": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/secure/bidirule?go-get=1
Parsing meta tags from https://golang.org/x/text/secure/bidirule?go-get=1 (status code 200)
get "golang.org/x/text/secure/bidirule": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/secure/bidirule?go-get=1
get "golang.org/x/text/secure/bidirule": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/unicode/bidi?go-get=1
Parsing meta tags from https://golang.org/x/text/unicode/bidi?go-get=1 (status code 200)
get "golang.org/x/text/unicode/bidi": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/unicode/bidi?go-get=1
get "golang.org/x/text/unicode/bidi": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/lex/httplex?go-get=1
Parsing meta tags from https://golang.org/x/net/lex/httplex?go-get=1 (status code 200)
get "golang.org/x/net/lex/httplex": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/lex/httplex?go-get=1
get "golang.org/x/net/lex/httplex": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/internal/iana?go-get=1
Parsing meta tags from https://golang.org/x/net/internal/iana?go-get=1 (status code 200)
get "golang.org/x/net/internal/iana": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/internal/iana?go-get=1
get "golang.org/x/net/internal/iana": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/internal/socket?go-get=1
Parsing meta tags from https://golang.org/x/net/internal/socket?go-get=1 (status code 200)
get "golang.org/x/net/internal/socket": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/internal/socket?go-get=1
get "golang.org/x/net/internal/socket": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/internal/timeseries?go-get=1
Parsing meta tags from https://golang.org/x/net/internal/timeseries?go-get=1 (status code 200)
get "golang.org/x/net/internal/timeseries": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/internal/timeseries?go-get=1
get "golang.org/x/net/internal/timeseries": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/ipv4?go-get=1
Parsing meta tags from https://golang.org/x/net/ipv4?go-get=1 (status code 200)
get "golang.org/x/net/ipv4": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/ipv4?go-get=1
get "golang.org/x/net/ipv4": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/ipv6?go-get=1
Parsing meta tags from https://golang.org/x/net/ipv6?go-get=1 (status code 200)
get "golang.org/x/net/ipv6": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/ipv6?go-get=1
get "golang.org/x/net/ipv6": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/webdav/internal/xml?go-get=1
Parsing meta tags from https://golang.org/x/net/webdav/internal/xml?go-get=1 (status code 200)
get "golang.org/x/net/webdav/internal/xml": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/webdav/internal/xml?go-get=1
get "golang.org/x/net/webdav/internal/xml": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/websocket?go-get=1
Parsing meta tags from https://golang.org/x/net/websocket?go-get=1 (status code 200)
get "golang.org/x/net/websocket": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/websocket?go-get=1
get "golang.org/x/net/websocket": verifying non-authoritative meta tag
Fetching https://golang.org/x/sys/unix?go-get=1
Parsing meta tags from https://golang.org/x/sys/unix?go-get=1 (status code 200)
get "golang.org/x/sys/unix": found meta tag main.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys/unix?go-get=1
get "golang.org/x/sys/unix": verifying non-authoritative meta tag
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
golang.org/x/sys (download)
Fetching https://golang.org/x/text/cases?go-get=1
Parsing meta tags from https://golang.org/x/text/cases?go-get=1 (status code 200)
get "golang.org/x/text/cases": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/cases?go-get=1
get "golang.org/x/text/cases": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/internal?go-get=1
Parsing meta tags from https://golang.org/x/text/internal?go-get=1 (status code 200)
get "golang.org/x/text/internal": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/internal?go-get=1
get "golang.org/x/text/internal": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/language?go-get=1
Parsing meta tags from https://golang.org/x/text/language?go-get=1 (status code 200)
get "golang.org/x/text/language": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/language?go-get=1
get "golang.org/x/text/language": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/internal/tag?go-get=1
Parsing meta tags from https://golang.org/x/text/internal/tag?go-get=1 (status code 200)
get "golang.org/x/text/internal/tag": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/internal/tag?go-get=1
get "golang.org/x/text/internal/tag": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/collate?go-get=1
Parsing meta tags from https://golang.org/x/text/collate?go-get=1 (status code 200)
get "golang.org/x/text/collate": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/collate?go-get=1
get "golang.org/x/text/collate": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/internal/colltab?go-get=1
Parsing meta tags from https://golang.org/x/text/internal/colltab?go-get=1 (status code 200)
get "golang.org/x/text/internal/colltab": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/internal/colltab?go-get=1
get "golang.org/x/text/internal/colltab": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/encoding?go-get=1
Parsing meta tags from https://golang.org/x/text/encoding?go-get=1 (status code 200)
get "golang.org/x/text/encoding": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/encoding?go-get=1
get "golang.org/x/text/encoding": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/encoding/internal/identifier?go-get=1
Parsing meta tags from https://golang.org/x/text/encoding/internal/identifier?go-get=1 (status code 200)
get "golang.org/x/text/encoding/internal/identifier": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/encoding/internal/identifier?go-get=1
get "golang.org/x/text/encoding/internal/identifier": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/encoding/charmap?go-get=1
Parsing meta tags from https://golang.org/x/text/encoding/charmap?go-get=1 (status code 200)
get "golang.org/x/text/encoding/charmap": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/encoding/charmap?go-get=1
get "golang.org/x/text/encoding/charmap": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/encoding/internal?go-get=1
Parsing meta tags from https://golang.org/x/text/encoding/internal?go-get=1 (status code 200)
get "golang.org/x/text/encoding/internal": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/encoding/internal?go-get=1
get "golang.org/x/text/encoding/internal": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/encoding/htmlindex?go-get=1
Parsing meta tags from https://golang.org/x/text/encoding/htmlindex?go-get=1 (status code 200)
get "golang.org/x/text/encoding/htmlindex": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/encoding/htmlindex?go-get=1
get "golang.org/x/text/encoding/htmlindex": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/encoding/japanese?go-get=1
Parsing meta tags from https://golang.org/x/text/encoding/japanese?go-get=1 (status code 200)
get "golang.org/x/text/encoding/japanese": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/encoding/japanese?go-get=1
get "golang.org/x/text/encoding/japanese": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/encoding/korean?go-get=1
Parsing meta tags from https://golang.org/x/text/encoding/korean?go-get=1 (status code 200)
get "golang.org/x/text/encoding/korean": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/encoding/korean?go-get=1
get "golang.org/x/text/encoding/korean": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/encoding/simplifiedchinese?go-get=1
Parsing meta tags from https://golang.org/x/text/encoding/simplifiedchinese?go-get=1 (status code 200)
get "golang.org/x/text/encoding/simplifiedchinese": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/encoding/simplifiedchinese?go-get=1
get "golang.org/x/text/encoding/simplifiedchinese": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/encoding/traditionalchinese?go-get=1
Parsing meta tags from https://golang.org/x/text/encoding/traditionalchinese?go-get=1 (status code 200)
get "golang.org/x/text/encoding/traditionalchinese": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/encoding/traditionalchinese?go-get=1
get "golang.org/x/text/encoding/traditionalchinese": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/encoding/unicode?go-get=1
Parsing meta tags from https://golang.org/x/text/encoding/unicode?go-get=1 (status code 200)
get "golang.org/x/text/encoding/unicode": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/encoding/unicode?go-get=1
get "golang.org/x/text/encoding/unicode": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/internal/utf8internal?go-get=1
Parsing meta tags from https://golang.org/x/text/internal/utf8internal?go-get=1 (status code 200)
get "golang.org/x/text/internal/utf8internal": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/internal/utf8internal?go-get=1
get "golang.org/x/text/internal/utf8internal": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/runes?go-get=1
Parsing meta tags from https://golang.org/x/text/runes?go-get=1 (status code 200)
get "golang.org/x/text/runes": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/runes?go-get=1
get "golang.org/x/text/runes": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/internal/format?go-get=1
Parsing meta tags from https://golang.org/x/text/internal/format?go-get=1 (status code 200)
get "golang.org/x/text/internal/format": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/internal/format?go-get=1
get "golang.org/x/text/internal/format": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/internal/gen?go-get=1
Parsing meta tags from https://golang.org/x/text/internal/gen?go-get=1 (status code 200)
get "golang.org/x/text/internal/gen": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/internal/gen?go-get=1
get "golang.org/x/text/internal/gen": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/unicode/cldr?go-get=1
Parsing meta tags from https://golang.org/x/text/unicode/cldr?go-get=1 (status code 200)
get "golang.org/x/text/unicode/cldr": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/unicode/cldr?go-get=1
get "golang.org/x/text/unicode/cldr": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/internal/stringset?go-get=1
Parsing meta tags from https://golang.org/x/text/internal/stringset?go-get=1 (status code 200)
get "golang.org/x/text/internal/stringset": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/internal/stringset?go-get=1
get "golang.org/x/text/internal/stringset": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/width?go-get=1
Parsing meta tags from https://golang.org/x/text/width?go-get=1 (status code 200)
get "golang.org/x/text/width": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/width?go-get=1
get "golang.org/x/text/width": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/benchmark/parse?go-get=1
Parsing meta tags from https://golang.org/x/tools/benchmark/parse?go-get=1 (status code 200)
get "golang.org/x/tools/benchmark/parse": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/benchmark/parse?go-get=1
get "golang.org/x/tools/benchmark/parse": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
golang.org/x/tools (download)
Fetching https://golang.org/x/tools/blog?go-get=1
Parsing meta tags from https://golang.org/x/tools/blog?go-get=1 (status code 200)
get "golang.org/x/tools/blog": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/blog?go-get=1
get "golang.org/x/tools/blog": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/blog/atom?go-get=1
Parsing meta tags from https://golang.org/x/tools/blog/atom?go-get=1 (status code 200)
get "golang.org/x/tools/blog/atom": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/blog/atom?go-get=1
get "golang.org/x/tools/blog/atom": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/present?go-get=1
Parsing meta tags from https://golang.org/x/tools/present?go-get=1 (status code 200)
get "golang.org/x/tools/present": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/present?go-get=1
get "golang.org/x/tools/present": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/cmd/guru/serial?go-get=1
Parsing meta tags from https://golang.org/x/tools/cmd/guru/serial?go-get=1 (status code 200)
get "golang.org/x/tools/cmd/guru/serial": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/cmd/guru/serial?go-get=1
get "golang.org/x/tools/cmd/guru/serial": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/container/intsets?go-get=1
Parsing meta tags from https://golang.org/x/tools/container/intsets?go-get=1 (status code 200)
get "golang.org/x/tools/container/intsets": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/container/intsets?go-get=1
get "golang.org/x/tools/container/intsets": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/cover?go-get=1
Parsing meta tags from https://golang.org/x/tools/cover?go-get=1 (status code 200)
get "golang.org/x/tools/cover": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/cover?go-get=1
get "golang.org/x/tools/cover": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/ast/astutil?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/ast/astutil?go-get=1 (status code 200)
get "golang.org/x/tools/go/ast/astutil": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/ast/astutil?go-get=1
get "golang.org/x/tools/go/ast/astutil": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/buildutil?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/buildutil?go-get=1 (status code 200)
get "golang.org/x/tools/go/buildutil": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/buildutil?go-get=1
get "golang.org/x/tools/go/buildutil": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/callgraph?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/callgraph?go-get=1 (status code 200)
get "golang.org/x/tools/go/callgraph": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/callgraph?go-get=1
get "golang.org/x/tools/go/callgraph": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/ssa?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/ssa?go-get=1 (status code 200)
get "golang.org/x/tools/go/ssa": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/ssa?go-get=1
get "golang.org/x/tools/go/ssa": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/types/typeutil?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/types/typeutil?go-get=1 (status code 200)
get "golang.org/x/tools/go/types/typeutil": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/types/typeutil?go-get=1
get "golang.org/x/tools/go/types/typeutil": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/callgraph/cha?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/callgraph/cha?go-get=1 (status code 200)
get "golang.org/x/tools/go/callgraph/cha": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/callgraph/cha?go-get=1
get "golang.org/x/tools/go/callgraph/cha": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/ssa/ssautil?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/ssa/ssautil?go-get=1 (status code 200)
get "golang.org/x/tools/go/ssa/ssautil": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/ssa/ssautil?go-get=1
get "golang.org/x/tools/go/ssa/ssautil": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/loader?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/loader?go-get=1 (status code 200)
get "golang.org/x/tools/go/loader": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/loader?go-get=1
get "golang.org/x/tools/go/loader": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/callgraph/rta?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/callgraph/rta?go-get=1 (status code 200)
get "golang.org/x/tools/go/callgraph/rta": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/callgraph/rta?go-get=1
get "golang.org/x/tools/go/callgraph/rta": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/callgraph/static?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/callgraph/static?go-get=1 (status code 200)
get "golang.org/x/tools/go/callgraph/static": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/callgraph/static?go-get=1
get "golang.org/x/tools/go/callgraph/static": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/godoc?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc?go-get=1 (status code 200)
get "golang.org/x/tools/godoc": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc?go-get=1
get "golang.org/x/tools/godoc": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/godoc/analysis?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/analysis?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/analysis": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/analysis?go-get=1
get "golang.org/x/tools/godoc/analysis": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/pointer?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/pointer?go-get=1 (status code 200)
get "golang.org/x/tools/go/pointer": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/pointer?go-get=1
get "golang.org/x/tools/go/pointer": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/godoc/util?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/util?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/util": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/util?go-get=1
get "golang.org/x/tools/godoc/util": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/godoc/vfs?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/vfs?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/vfs": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/vfs?go-get=1
get "golang.org/x/tools/godoc/vfs": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/godoc/vfs/httpfs?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/vfs/httpfs?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/vfs/httpfs": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/vfs/httpfs?go-get=1
get "golang.org/x/tools/godoc/vfs/httpfs": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/godoc/redirect?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/redirect?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/redirect": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/redirect?go-get=1
get "golang.org/x/tools/godoc/redirect": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/godoc/static?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/static?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/static": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/static?go-get=1
get "golang.org/x/tools/godoc/static": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/godoc/vfs/gatefs?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/vfs/gatefs?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/vfs/gatefs": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/vfs/gatefs?go-get=1
get "golang.org/x/tools/godoc/vfs/gatefs": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/godoc/vfs/mapfs?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/vfs/mapfs?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/vfs/mapfs": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/vfs/mapfs?go-get=1
get "golang.org/x/tools/godoc/vfs/mapfs": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/godoc/vfs/zipfs?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/vfs/zipfs?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/vfs/zipfs": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/vfs/zipfs?go-get=1
get "golang.org/x/tools/godoc/vfs/zipfs": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/gcimporter15?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/gcimporter15?go-get=1 (status code 200)
get "golang.org/x/tools/go/gcimporter15": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/gcimporter15?go-get=1
get "golang.org/x/tools/go/gcimporter15": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/internal/gccgoimporter?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/internal/gccgoimporter?go-get=1 (status code 200)
get "golang.org/x/tools/go/internal/gccgoimporter": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/internal/gccgoimporter?go-get=1
get "golang.org/x/tools/go/internal/gccgoimporter": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/go/ssa/interp?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/ssa/interp?go-get=1 (status code 200)
get "golang.org/x/tools/go/ssa/interp": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/ssa/interp?go-get=1
get "golang.org/x/tools/go/ssa/interp": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/imports?go-get=1
Parsing meta tags from https://golang.org/x/tools/imports?go-get=1 (status code 200)
get "golang.org/x/tools/imports": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/imports?go-get=1
get "golang.org/x/tools/imports": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/playground?go-get=1
Parsing meta tags from https://golang.org/x/tools/playground?go-get=1 (status code 200)
get "golang.org/x/tools/playground": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/playground?go-get=1
get "golang.org/x/tools/playground": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/playground/socket?go-get=1
Parsing meta tags from https://golang.org/x/tools/playground/socket?go-get=1 (status code 200)
get "golang.org/x/tools/playground/socket": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/playground/socket?go-get=1
get "golang.org/x/tools/playground/socket": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/refactor/eg?go-get=1
Parsing meta tags from https://golang.org/x/tools/refactor/eg?go-get=1 (status code 200)
get "golang.org/x/tools/refactor/eg": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/refactor/eg?go-get=1
get "golang.org/x/tools/refactor/eg": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/refactor/importgraph?go-get=1
Parsing meta tags from https://golang.org/x/tools/refactor/importgraph?go-get=1 (status code 200)
get "golang.org/x/tools/refactor/importgraph": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/refactor/importgraph?go-get=1
get "golang.org/x/tools/refactor/importgraph": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/refactor/rename?go-get=1
Parsing meta tags from https://golang.org/x/tools/refactor/rename?go-get=1 (status code 200)
get "golang.org/x/tools/refactor/rename": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/refactor/rename?go-get=1
get "golang.org/x/tools/refactor/rename": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools/refactor/satisfy?go-get=1
Parsing meta tags from https://golang.org/x/tools/refactor/satisfy?go-get=1 (status code 200)
get "golang.org/x/tools/refactor/satisfy": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/refactor/satisfy?go-get=1
get "golang.org/x/tools/refactor/satisfy": verifying non-authoritative meta tag
Fetching https://gopkg.in/fsnotify.v1?go-get=1
Parsing meta tags from https://gopkg.in/fsnotify.v1?go-get=1 (status code 200)
get "gopkg.in/fsnotify.v1": found meta tag main.metaImport{Prefix:"gopkg.in/fsnotify.v1", VCS:"git", RepoRoot:"https://gopkg.in/fsnotify.v1"} at https://gopkg.in/fsnotify.v1?go-get=1
gopkg.in/fsnotify.v1 (download)
Makefile:120: recipe for target '.dep-stamp' failed
make: *** [.dep-stamp] Error 1

When using /metrics, the server returns the timestamp for the last time a counter value was updated. This is not the correct thing to respond with, as it causes stale metrics to vanish when doing evaluation.

Here's an example of the output:

# TYPE response counter
response{status_code="200",prog="TEST.mtail",instance="X"} 55948 1434722695891
response{status_code="503",prog="TEST.mtail",instance="X"} 1 1434722073677

The 503 counter is ~622 seconds behind the 200 counter. This means if I look at the current rate over 5min of metrics, 503 will vanish from my dashboard.

The correct timestamp response should be the timestamp in which a counter is read from memory. It's also recommended for simple read-from-memory counters to not return any timestamp and the server will assign one.