Flowhook: Gitlab pipelines in Grafana

How to share CI/CD results in your most beloved dashboard

Hey there. In this article I will demonstrate how easy you can share your Gitlab events like pipeline results in Grafana by using Flowhook in between.

Flowhook is a small (and free) service that consumes your webhook payloads, converts them and allows Grafana to display the result in a standard Log panel.

Lets look at data flow from left to right:

Gitlab is the source that issues the Webhook for the pipeline. The Flowhook service on Heroku will consume it and map it to a compact model.

The Flowhook datasource in Grafana connects to the Flowhook service via Websockets, consumes the mapped data and prepares it for the Grafana Explorer / Log panel.

That’s about it. Setting up the relevant bits and pieces is almost trivial. The real fun starts with the mapping of the payload. Let’s create our pipeline example.

Pipeline example

Create a new Flowhook

We will start right in the middle and create a new Flowhook at https://flowhook.herokuapp.com/. No need to sign up, you will be identified via cookie to manage your Flowhooks.

Select “Create new from scratch” and enter the source (Gitlab) and resource (pipeline) and you will be provided a new Flowhook ID which will be used to map inbound and outbound connection as well a storing the conversion mapping.

Link the webhook source

Looking now at Gitlab, we will find the webhooks setup in the settings and can now point to our new Flowhook using our ID: https://flowhook.herokuapp.com/api/v1/webhook?flowhook=vh7fd1vol232

Now send a couple of pipeline events to Flowhook so we can start mapping them.

Payload Mapping

Let’s look at the Flowhook UI in detail. You see the posted webhooks in request sampler (left side), the mapping editor (in the middle) and the later mapping result (right-hand side).

The last webhook we posted is already shown in the payload editor.

We now can edit the mapping fields in the middle and use JsonPath for selecting the bits and pieces from the source payload we are interested in.

Some fields are conditional and require us to build expressions, other fields simply let us select content.

The main work is now to experiment with the right expressions, mapping and previewing them and alter the payload. You will find more details on the mapping options and a couple of examples on the help page.

In the screenshot below you can see a simple mapping I designed to check the pipeline build state, mapping some labels and extracting a bunch of metadata from Gitlab’s webhook payload.

The result of the mapping is visible on the right-hand side. This is what will be transported to Grafana. Let’s attach Grafana next.

The datasource plugin

Grafana uses datasources to ingest data for its visualizations. We install and configure the Flowhook Datasource plugin.

Installation currently needs to be done by downloading it from Github https://github.com/verticle-io/flowhook-datasource and placing the plugin into Grafana’s data/plugins dir.

You now can create a new datasource and configure our Flowhook ID, other settings left as default.

The Explorer

Setup and mapping is done. We now can shoot some webhooks from Gitlab again with the same URL and their mapped payload will pop up in the explorer view:

Done!

Flowhook still is in a very early stage but quite usable already. I plan to add couple of additional features in the next weeks. If you have questions, ideas or want to contribute have a chat with us on out gitter channel: https://gitter.im/verticle-io/flowhook

Hope you like it — don’t forget to clap ;)