AlertManager offers the possibility to configure receivers to forward grouped alerts to multiple notification integrations.
I recently needed to send the notifications to a webhook receiver. The required configuration is well documented.
The Openshift Monitoring stack offers the possibility to add some custom configuration to Prometheus and Alertmanager components. In this case, the AlertManager needed to be configured and Openshift offers that possibilty, for more details check the documentation.
But the problem I had was that the front endpoint that would receive the alerts had some strict requirements:
- A TLS client certificate.
- A custom HTTP header needed by the gateway.
The first issue was that the Alertmanager is controlled, configured and managed by the Openshift Monitoring Operator and it does not allow to add volumes to the Alertmanager StatefulSet because the controller will force its state back, and this is a problem because the TLS configuration requires the location of the certificate and the key, since I cannot mount a Secret as volume in the filesystem of the pod then I cannot provide the credentials.
The second problem was the HTTP header. Even if I have the full control of the configuration of Alertmanager I cannot add HTTP headers or even URL params to the URL of the host that receives the notifications.
The only solution I found was to use an intermediary proxy to receive the payload sent by Alertmanager and re-send the request to the desired receiver with the custom TLS configuration and the custom header. I had to add the webhook receiver to the configuration and provide the name of the service that would dispatch the notifications and forward them to the special receiver.
This pattern is more or less what a Service Mesh would do with sidecar proxies since they intercept the HTTP requests and rework them if necessary.