Skip to content

Grafana → SignalFx Cutover

Table of Contents

  1. Overview
  2. Step 1: Verify dashboards and charts
  3. Step 2: Verify alerts and fix extrapolation
  4. Step 3: Enable PagerDuty on each alert
  5. Step 4: Disable the equivalent Grafana alert
  6. Optional: going beyond Prometheus

Overview

We are required to fully cut over from Grafana to SignalFx by the end of September. The minimum ask for this cutover is to migrate using our existing Prometheus metrics — this doc assumes Prometheus scraping/metrics are already flowing into SignalFx for your service, and focuses purely on the steps needed to safely retire Grafana in favour of SignalFx:

  1. Make sure all dashboards/charts work as expected in SignalFx. Dashboard variables may need adjusting as they currently default to PRD.
  2. Make sure all alerts look correct in SignalFx, fixing extrapolation along the way.
  3. Enable PagerDuty on those alerts.
  4. Disable the equivalent alert in Grafana (unless happy to be double alerted).

Moving to the OTel collector plugin (rather than relying on Prometheus) is desirable but not part of the minimum ask — see Optional: going beyond Prometheus at the end if your team wants to go further.

Step 1: Verify dashboards and charts

Go through every dashboard/chart your team currently relies on in Grafana and confirm the equivalent exists and renders correctly in SignalFx: - Check that every panel has data. If you have time, you can double check the values look sane compared to Grafana for the same time range, although this should have already been done. - Check any panel that uses a custom metric particularly carefully — these are the ones most likely to need a query rewritten rather than a straight import. - If a query needs translating from a Grafana function to a SignalFx one (e.g. Rate, Increase, Drop/Replace Non-numeric Values), see the Translating queries from Grafana to SignalFx section of the Observability doc. - Once a dashboard is correct in the SignalFx UI, don't forget to save it as code by following the Saving and Importing process — otherwise your changes will be lost on the next redeploy.

Step 2: Verify alerts and fix extrapolation

Every alert currently firing from Grafana needs a working equivalent in SignalFx before Grafana's alerting can be switched off. - All alerts should already be in SignalFc, but if the alert doesn't have a SignalFx detector yet, create one following Alerts and Detectors in the Observability doc. - While you're in there, change the extrapolation policy from "Zero" or "Null" to "Last Value" (via the cog next to the query). With "Zero"/"Null", a gap in incoming data (e.g. a metric temporarily not reporting) gets treated as a real zero/missing value and can trigger a false alert; "Last Value" instead holds the last known value during a gap, which prevents alerts caused purely by missing data rather than an actual problem.

Step 3: Enable PagerDuty on each alert

Once you're confident an alert is working correctly in SignalFx, add PagerDuty as a notification recipient on it, alongside (or instead of) the Slack channel it currently alerts to — our PagerDuty integration is already configured in SignalFx, so this is just a case of adding it as a recipient on each detector/alert rule.

Step 4: Disable the equivalent Grafana alert

Once the SignalFx alert + PagerDuty is confirmed working, disable (or delete) the equivalent alert in Grafana. Do this after, not before, confirming Step 2 and Step 3 both work — unless you're happy to be alerted twice in the meantime.

Optional: going beyond Prometheus

The steps above cover the minimum ask (cutting over using our existing Prometheus metrics). If your team wants to go further and move to instrumenting via the OTel collector directly instead of Prometheus, see Adding OTel collector.