Git Product home page Git Product logo

Comments (11)

Knappek avatar Knappek commented on May 20, 2024 1

Thank you @sidhantg2 for reporting this issue. It might be related to what has been reported in #43. I'll try to look into it very soon and try to reproduce this error. I'll keep you posted.

Sent with GitHawk

from prometheus-msteams.

sidhantg2 avatar sidhantg2 commented on May 20, 2024 1

Hi @Knappek , I have found the issue. It was not able to parse this field {\"description\":\"172.17.0.11:8081 of hobject job \\\"kube-api-scrape\\\" has been down for more than 1 minute.\"} because of having double quotes in the text (""). I have removed the double quotes and it works fine.

from prometheus-msteams.

Knappek avatar Knappek commented on May 20, 2024 1

@HuJake can you please post your complete template file? And please make sure to properly use the code snippet syntax, otherwise I am not able to use your template and test it by myself.

from prometheus-msteams.

Knappek avatar Knappek commented on May 20, 2024 1

@HuJake I have opened a PR to improve the log output when enabling debug mode. I have tested your template against this code using a test alert request json. This provides me more information why you are getting an error where the raw teams message card that will be attempted to send to teams is:

{
  "@type": "MessageCard",
  "@context": "http://schema.org/extensions",
  "themeColor": "FFA500",
  "summary": "Server High Memory usage",
  "title": "Prometheus (high_memory_load) Alert (FIRING)",
  "sections": [
    {
      "activityTitle": "[10.80.40.11 reported high memory usage with 23.28%.](http://docker.for.mac.host.internal:9093)",
      "facts": [
        description
        summary
        {
          "name": "alertname",
          "value": "high\\_memory\\_load"
        },
        {
          "name": "instance",
          "value": "instance-with-hyphen\\_and\\_underscore"
        },
        {
          "name": "job",
          "value": "docker\\_nodes"
        },
        {
          "name": "monitor",
          "value": "master"
        },
        {
          "name": "severity",
          "value": "warning"
        }
      ],
      "markdown": true
    }
  ]
}

As you can see in this example, there is something not working correctly right after facts where description and summary are not formatted correctly. The correct format should be

        {
          "name": "description",
          "value": "10.80.40.11 reported high memory usage with 23.28%."
        },
        {
          "name": "summary",
          "value": "Server High Memory usage"
        },

I haven't yet investigated how to fix your template. It is late right now, maybe I can investigate further tomorrow.
I ask you to try out the new code from the PR and test it similarly by enabling debug mode. Please let me know whether this helps finding the root cause in your case.

from prometheus-msteams.

Knappek avatar Knappek commented on May 20, 2024 1

@HuJake glad to help you solve the problem. I will soon merge the PR and make it available on version v1.1.3.

Regarding text style, you can do that, but apparently only for $value and not for $key (don't ask me why ;)). So, this example prints your values in red:

"value": "<span style=\"color:red\">{{ reReplaceAll "_" "\\\\_" $value }}</span>"

In general you should be able to use Markdown font style syntax, which is basically html style syntax (here is an example: https://answers.squarespace.com/questions/107013/how-to-change-font-style-in-markdown-editing.html).

Hope this helps.

from prometheus-msteams.

jangrewe avatar jangrewe commented on May 20, 2024

@Knappek We're also getting that error with v1.1.2 and the following (unescaped) JSON:

{
  "receiver": "teams",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "TestAlert",
        "cluster_env": "dev",
        "instance": "prometheus:80",
        "job": "prometheus",
        "k8s_cluster": "clstr",
        "severity": "test"
      },
      "annotations": {
        "description": "This is a test alert.\n*Instance:* prometheus:80",
        "link": "<https://grafana-monitoring.clstr.example.com/dashboard/db/host-info?var-hostname=|:grafana:> <https://dashboard-kube-system.clstr.example.com/#!/deployment?namespace=monitoring|:kubernetes:>",
        "summary": "TestAlert"
      },
      "startsAt": "2019-07-16T13:31:14.670231818Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://prometheus-monitoring.clstr.example.com/graph?g0.expr=up%7Bjob%3D%22prometheus%22%7D+%3D%3D+1&g0.tab=1"
    }
  ],
  "groupLabels": {
    "alertname": "TestAlert",
    "k8s_cluster": "clstr",
    "severity": "test"
  },
  "commonLabels": {
    "alertname": "TestAlert",
    "cluster_env": "dev",
    "instance": "prometheus:80",
    "job": "prometheus",
    "k8s_cluster": "clstr",
    "severity": "test"
  },
  "commonAnnotations": {
    "description": "This is a test alert.\n*Instance:* prometheus:80",
    "link": "<https://grafana-monitoring.clstr.example.com/dashboard/db/host-info?var-hostname=|:grafana:> <https://dashboard-kube-system.clstr.example.com/#!/deployment?namespace=monitoring|:kubernetes:>",
    "summary": "TestAlert"
  },
  "externalURL": "https://alertmanager-monitoring.clstr.example.com",
  "version": "4",
  "groupKey": "{}/{}:{alertname=\"TestAlert\", k8s_cluster=\"clstr\", severity=\"test\"}"
}

The full log output:

time="2019-07-16T13:31:44Z" level=info msg="/alertmanager received a request"
time="2019-07-16T13:31:44Z" level=debug msg="{\"receiver\":\"teams\",\"status\":\"firing\",\"alerts\":[{\"status\":\"firing\",\"labels\":{\"alertname\":\"TestAlert\",\"cluster_env\":\"dev\",\"instance\":\"prometheus:80\",\"job\":\"prometheus\",\"k8s_cluster\":\"clstr\",\"severity\":\"test\"},\"annotations\":{\"description\":\"This is a test alert.\\n*Instance:* prometheus:80\",\"link\":\"\\u003chttps://grafana-monitoring.clstr.example.com/dashboard/db/host-info?var-hostname=|:grafana:\\u003e \\u003chttps://dashboard-kube-system.clstr.example.com/#!/deployment?namespace=monitoring|:kubernetes:\\u003e\",\"summary\":\"TestAlert\"},\"startsAt\":\"2019-07-16T13:31:14.670231818Z\",\"endsAt\":\"0001-01-01T00:00:00Z\",\"generatorURL\":\"https://prometheus-monitoring.clstr.example.com/graph?g0.expr=up%7Bjob%3D%22prometheus%22%7D+%3D%3D+1\\u0026g0.tab=1\"}],\"groupLabels\":{\"alertname\":\"TestAlert\",\"k8s_cluster\":\"clstr\",\"severity\":\"test\"},\"commonLabels\":{\"alertname\":\"TestAlert\",\"cluster_env\":\"dev\",\"instance\":\"prometheus:80\",\"job\":\"prometheus\",\"k8s_cluster\":\"clstr\",\"severity\":\"test\"},\"commonAnnotations\":{\"description\":\"This is a test alert.\\n*Instance:* prometheus:80\",\"link\":\"\\u003chttps://grafana-monitoring.clstr.example.com/dashboard/db/host-info?var-hostname=|:grafana:\\u003e \\u003chttps://dashboard-kube-system.clstr.example.com/#!/deployment?namespace=monitoring|:kubernetes:\\u003e\",\"summary\":\"TestAlert\"},\"externalURL\":\"https://alertmanager-monitoring.clstr.example.com\",\"version\":\"4\",\"groupKey\":\"{}/{}:{alertname=\\\"TestAlert\\\", k8s_cluster=\\\"clstr\\\", severity=\\\"test\\\"}\"}"
time="2019-07-16T13:31:44Z" level=debug msg="Size of message is 0 Bytes (~0 KB)"
time="2019-07-16T13:31:44Z" level=error msg="Failed to parse json with key 'sections': Key path not found"

from prometheus-msteams.

Knappek avatar Knappek commented on May 20, 2024

@jangrewe thanks for reporting it and the detailed information. I'll have a look into this and keep you posted.

Sent with GitHawk

from prometheus-msteams.

Knappek avatar Knappek commented on May 20, 2024

Hey @jangrewe I was able to reproduce the issue. The problem is the "\n" that you have in the two description fields.
In order to produce a newline, you have to use "<br>" instead. So your alert should look like this:

{
  "receiver": "teams",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "TestAlert",
        "cluster_env": "dev",
        "instance": "prometheus:80",
        "job": "prometheus",
        "k8s_cluster": "clstr",
        "severity": "test"
      },
      "annotations": {
        "description": "This is a test alert.<br>*Instance:* prometheus:80",
        "link": "<https://grafana-monitoring.clstr.example.com/dashboard/db/host-info?var-hostname=|:grafana:> <https://dashboard-kube-system.clstr.example.com/#!/deployment?namespace=monitoring|:kubernetes:>",
        "summary": "TestAlert"
      },
      "startsAt": "2019-07-16T13:31:14.670231818Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://prometheus-monitoring.clstr.example.com/graph?g0.expr=up%7Bjob%3D%22prometheus%22%7D+%3D%3D+1&g0.tab=1"
    }
  ],
  "groupLabels": {
    "alertname": "TestAlert",
    "k8s_cluster": "clstr",
    "severity": "test"
  },
  "commonLabels": {
    "alertname": "TestAlert",
    "cluster_env": "dev",
    "instance": "prometheus:80",
    "job": "prometheus",
    "k8s_cluster": "clstr",
    "severity": "test"
  },
  "commonAnnotations": {
    "description": "This is a test alert.<br>*Instance:* prometheus:80",
    "link": "<https://grafana-monitoring.clstr.example.com/dashboard/db/host-info?var-hostname=|:grafana:> <https://dashboard-kube-system.clstr.example.com/#!/deployment?namespace=monitoring|:kubernetes:>",
    "summary": "TestAlert"
  },
  "externalURL": "https://alertmanager-monitoring.clstr.example.com",
  "version": "4",
  "groupKey": "{}/{}:{alertname=\"TestAlert\", k8s_cluster=\"clstr\", severity=\"test\"}"
}

Hope this solved your problem, otherwise feel free to reopen.

from prometheus-msteams.

HuJake avatar HuJake commented on May 20, 2024

@Knappek

I tried to change default-message-card.tmpl

I want the field that the filter wants ($key)
But I found that I always appear after the change

=> Level=error msg="Failed to parse json with key 'sections': Key path not found"

Is the judgment condition of my increase incorrect?
Below is my change~

``
"title": "Prometheus ({{ .CommonLabels.alertname }}) Alert ({{ .Status | toUpper }})",
"sections": [ {{$externalUrl := .ExternalURL}}
{{- range $index, $alert := .Alerts }}{{- if $index }},{{- end }}
{
"activityTitle": "[{{ $alert.Annotations.description }}]({{ $externalUrl }})",
"facts": [
{{- range $key, $value := $alert.Annotations }}
{{ reReplaceAll "" "\\" $key }}
{{- if eq $key "namespace" "container_name" -}}
{
"name": "{{- if eq $key "namespace" -}}FF0000
{{- else if eq $key "container_name" -}}FF0000
{{- else -}}FF00FF
{{- end -}}",
"value": "{{ reReplaceAll "" "\\" $value }}"
},
{{- end -}}
{{- end -}}
{{$c := counter}}{{ range $key, $value := $alert.Labels }}{{if call $c}},{{ end }}
{
"name": "{{ reReplaceAll "" "\\" $key }}",
"value": "{{ reReplaceAll "" "\\" $value }}"
}
{{- end }}
],
"markdown": true
}
{{- end }}
],

`

from prometheus-msteams.

HuJake avatar HuJake commented on May 20, 2024

@HuJake can you please post your complete template file? And please make sure to properly use the code snippet syntax, otherwise I am not able to use your template and test it by myself.

@Knappek thanks your help, below it's template file,

{{ define "teams.card" }}
{
  "@type": "MessageCard",
  "@context": "http://schema.org/extensions",
  "themeColor": "{{- if eq .Status "resolved" -}}53FF53
                 {{- else if eq .Status "firing" -}}
                    {{- if eq .CommonLabels.severity "critical" -}}FF0000
                    {{- else if eq .CommonLabels.severity "warning" -}}FFA500
                    {{- else -}}FF00FF{{- end -}}
                 {{- else -}}FF00FF{{- end -}}",
  "summary": "{{- if eq .CommonAnnotations.summary "" -}}
                  {{- if eq .CommonAnnotations.message "" -}}
                    {{- .CommonLabels.alertname -}}
                  {{- else -}}
                    {{- .CommonAnnotations.message -}}
                  {{- end -}}
              {{- else -}}
                  {{- .CommonAnnotations.summary -}}
              {{- end -}}",
  "title": "Prometheus ({{ .CommonLabels.alertname }}) Alert ({{ .Status | toUpper }})",
  "sections": [ {{$externalUrl := .ExternalURL}}
  {{- range $index, $alert := .Alerts }}{{- if $index }},{{- end }}
    {
      "activityTitle": "[{{ $alert.Annotations.description }}]({{ $externalUrl }})",
      "facts": [
        {{- range $key, $value := $alert.Annotations }}
        {{ reReplaceAll "_" "\\\\_" $key }}
        {{- if eq $key "namespace" "container_name" -}}
        {
          "name": "{{- if eq $key "namespace" -}}FF0000
                   {{- else if eq $key "container_name" -}}FF0000
                   {{- else -}}FF00FF
                   {{- end -}}",
          "value": "{{ reReplaceAll "_" "\\\\_" $value }}"
        },
        {{- end -}}
        {{- end -}}
        {{$c := counter}}{{ range $key, $value := $alert.Labels }}{{if call $c}},{{ end }}
        {
          "name": "{{ reReplaceAll "_" "\\\\_" $key }}",
          "value": "{{ reReplaceAll "_" "\\\\_" $value }}"
        }
        {{- end }}
      ],
      "markdown": true
    }
    {{- end }}
  ],
  "potentialAction": [
    {
        "@context": "http://schema.org",
        "@type": "ViewAction",
        "name": "Runbook",
        "target": [
            "{{ reReplaceAll "_" "\\\\_" .CommonAnnotations.runbook_url }}"
        ]
    },
    {
        "@context": "http://schema.org",
        "@type": "ViewAction",
        "name": "A",
        "target": [
            "http://testA.com"
        ]
    },
    {
        "@context": "http://schema.org",
        "@type": "ViewAction",
        "name": "B",
        "target": [
            "http://testB.com"
        ]
    }
  ]
}
{{ end }}

from prometheus-msteams.

HuJake avatar HuJake commented on May 20, 2024

@Knappek
Thanks for your new code from the PR, this helps me a lot.

In addition, I tried to modify the template file and found that there is a format error in the composition of "facts": [...].

{{ define "teams.card" }}
{
  "@type": "MessageCard",
  "@context": "http://schema.org/extensions",
  "themeColor": "{{- if eq .Status "resolved" -}}53FF53
                 {{- else if eq .Status "firing" -}}
                    {{- if eq .CommonLabels.severity "critical" -}}FF0000
                    {{- else if eq .CommonLabels.severity "warning" -}}FFA500
                    {{- else -}}FF00FF{{- end -}}
                 {{- else -}}FF00FF{{- end -}}",
  "summary": "{{- if eq .CommonAnnotations.summary "" -}}
                  {{- if eq .CommonAnnotations.message "" -}}
                    {{- .CommonLabels.alertname -}}
                  {{- else -}}
                    {{- .CommonAnnotations.message -}}
                  {{- end -}}
              {{- else -}}
                  {{- .CommonAnnotations.summary -}}
              {{- end -}}",
  "title": "Prometheus ({{ .CommonLabels.alertname }}) Alert ({{ .Status | toUpper }})",
  "sections": [ {{$externalUrl := .ExternalURL}}
  {{- range $index, $alert := .Alerts }}{{- if $index }},{{- end }}
    {
      "activityTitle": "[{{ $alert.Annotations.description }}]({{ $externalUrl }})",
      "facts": [
        {{- range $key, $value := $alert.Annotations }}
        {{- if eq $key "instance" "container_name" -}}
        {{ reReplaceAll "_" "\\\\_" $key }}
        {
          "name": "{{- if eq $key "instance" -}}instance
                   {{- else if eq $key "container_name" -}}container_name
                   {{- else -}}
                   {{- end -}}",
          "value": "{{ reReplaceAll "_" "\\\\_" $value }}"
        },
        {{- end -}}
        {{- end -}}
        {{$c := counter}}{{ range $key, $value := $alert.Labels }}{{if call $c}},{{ end }}
        {{- if eq $key "instance" "container_name" -}}
        {{ reReplaceAll "_" "\\\\_" $key }}
        {
          "name": "{{- if eq $key "instance" -}}instance
                   {{- else if eq $key "container_name" -}}container_name
                   {{- else -}}
                   {{- end -}}",       
          "value": "{{ reReplaceAll "_" "\\\\_" $value }}"
        }
        {{- end -}}
        {{- end -}}
      ],
      "markdown": true
    }
    {{- end }}
  ],
  "potentialAction": [
    {
        "@context": "http://schema.org",
        "@type": "ViewAction",
        "name": "Runbook",
        "target": [
            "{{ reReplaceAll "_" "\\\\_" .CommonAnnotations.runbook_url }}"
        ]
    },
    {
        "@context": "http://schema.org",
        "@type": "ViewAction",
        "name": "A",
        "target": [
            "http://testA.com"
        ]
    },
    {
        "@context": "http://schema.org",
        "@type": "ViewAction",
        "name": "B",
        "target": [
            "http://testB.com"
        ]
    }
  ]
}
{{ end }}

And I also want to ask can change the color or font in the fact[""] component ?
(I want to change the color for "$key" and "$value")

Thank you very much, if you can help investigate :-)

many thanks!

from prometheus-msteams.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.