Grafana is one of, if not the most common tool for visualization of telemetry data from your applications. I myself use Grafana both at work, and for my own personal projects to plot all sorts of data, from my Plex server usage stats, to my houses power usage. Generally I use the WYSIWYG interface for building my dashboards, but in modern software development the push seems to be to Gitops everything, so I went looking for solutions - here’s what I found. Note that this isn’t a debate on whether dashboards are actually useful, that’s a topic for another day.

Why Gitops things at all?

This was a question I wrestled with for a while - what’s the point of using Gitops to manage dashboards? What are we actually gaining? What are we losing? I think there’s a few main benefits to managing Dashboards with Git, mostly revolving around Observability Data living with the service that it is linked with (This allows your dashboards to be updated and version controlled as your application changes). Using Git also allows easy peer review on changes, that the Grafana interface itself doesn’t offer. On the other hand by managing dashboards as text (as is basically required in any git backed system), you lose the Graphical nature of the dashboard - how useful is peer review if I can’t play around with the dashboard, look at the visualizations and play around with the data? For the most part though, I think in collaborative teams the good outweighs the bad when it comes to Gitops.

Why I don’t like Grafana DSLs

Once we’ve decided to Gitops our Grafana dashboards, the logical next question is how to handle the Dashboards to Text conversion so that the dashboard can be stored in a Git repository. A quick google finds many different libraries for doing this, in just about every language you can think of (Kotlin, Python, Javascript to name a few). Most of these provide some sort of language specific instrumentation to build up your dashboard, and a way to output it in a JSON blob that Grafana can understand. This is the crux of what confuses me - Why do these exist?

I guess that could simply come down to “Other people think differently than me” which is clearly true and a justifiable reason, but my question to those people is: What benefit do you get out of a DSL over just the raw dashboard? Playing devils advocate I could come up with a few:

  • Reusable components and the ability to template dashboards for like services
  • Probably easier to review text changes than raw JSON
  • Probably easier to CI validate than raw JSON
  • You don’t have to munge the JSON (update the version etc)

But on the other hand, why don’t we just gitops the raw dashboard JSON?

  • You can use Grafana templating to make reusable dashboards
  • Raw JSON means that it probably came from somewhere so you can actually look at the dashboard to see what it looks like

And that last point is probably the big thing to me - to me, Grafana is a visual tool - reviewing text seems counter productive to the point of Grafana. I don’t actually want to peer review the text, I want to review the dashboard so writing text to change a dashboard seems super weird to me.

Do you use them?

Do you use a Grafana DSL? If so, I’d love to hear what your usecase is

I'm on Twitter: @sinkingpoint and BlueSky: @colindou.ch. Come yell at me!