How are webhook payloads published to the service bus? about opensource-management-portal HOT 4 CLOSED

Great question. So there are 3 different ways to get through to the "firehose" for eventually consistency to work, + 1 if you don't:

1. In theory, you could point your app's webhooks at the app's /webhook endpoint. We've disabled this after having issues with validating webhook signatures due to some company-specific deployment goo. Less resilient since the processing happens on the web backend.
2. Use your own ingestion service or tech or serverless between webhooks and Service Bus. We're using Azure Logic Apps for this, happy to share how we have that configured.
3. Use your own ingestion thing, we again use Logic Apps, to place the payloads into an Azure Storage Queue. This is more cost-effective probably than Service Bus. We're using this for a similar but different app to validate the codepaths.
4. Slower eventual consistency: as long as you don't have billions of repos or teams or things, the Conditional Request logic is designed to slowly build up caches of objects over time. So if you visit a list of teams, you may get a cached version, but it will kick-off a background refresh of this data. You may get some rate-limiting, but it'll eventually be OK.

Are you deploying on Azure or a different provider?

from opensource-management-portal.

Just sharing some from our logic app in the form of screenshots for now...

Yes this really is the world's most basic logic app, could probably be a serverless or even super simple ingress app. However we love knowing that logic apps is on the hook for resiliency here.

Basic mapping of some headers into message properties

Incoming things

from opensource-management-portal.

Thanks for the quick reply! The notes and screenshots are very helpful. I've gotten everything set up except for the properties mappings which look like they are required as I get this error when consuming webhooks from the queue: Error: Missing event properties - delivery, signature, and/or event. Could you provide the configuration for the send message properties in the screenshot?

from opensource-management-portal.

Initially getting past this with the configuration below. Still working through the codebase to understand which of the other properties are required. Also note, a webhook secret must be set on the GitHub app side for the signature properties to exist on the incoming payload.

{
  "definition": {
    "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
    "actions": {
      "Send_message": {
        "inputs": {
          "body": {
            "ContentData": "@{base64(triggerBody())}",
            "ContentType": "application/json",
            "Properties": {
              "delivery": "@coalesce(triggerOutputs()['headers']['X-GitHub-Delivery'])",
              "event": "@coalesce(triggerOutputs()['headers']['X-GitHub-Event'])",
              "signature": "@coalesce(triggerOutputs()['headers']['X-Hub-Signature'])",
              "signature256": "@coalesce(triggerOutputs()['headers']['X-Hub-Signature-256'])",
              "started": "@utcNow()"
            }
          },
          "host": {
            "connection": {
              "name": "@parameters('$connections')['servicebus']['connectionId']"
            }
          },
          "method": "post",
          "path": "/@{encodeURIComponent(encodeURIComponent('events'))}/messages",
          "queries": {
            "systemProperties": "Run Details"
          }
        },
        "runAfter": {},
        "type": "ApiConnection"
      }
    },
    "contentVersion": "1.0.0.0",
    "outputs": {},
    "parameters": {
      "$connections": {
        "defaultValue": {},
        "type": "Object"
      }
    },
    "triggers": {
      "manual": {
        "inputs": {},
        "kind": "Http",
        "type": "Request"
      }
    }
  },
  "parameters": {
    "$connections": {
      "value": {
        "servicebus": {
          "connectionId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/[RESOURCEGROUPNAME]/providers/Microsoft.Web/connections/servicebus",
          "connectionName": "servicebus",
          "id": "/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Web/locations/eastus/managedApis/servicebus"
        }
      }
    }
  }
}

from opensource-management-portal.