Skip to content

Otel collector plugin migration

Testing the collector plugin

Local testing

  1. To test the collector plugin locally, you can add the following docker file to the base of the repo and name it otel-compose.yaml:
services:
    grafana-lgtm:
        image: 'grafana/otel-lgtm:latest'
        ports:
            - '3000:3000' # UI
            - '4317:4317'
            - '4318:4318'

  1. Run docker-compose -f otel-compose.yaml up --force-recreate to start the application.

  2. Build and start your service as usual. Make sure the otel endpoint property is pointing to http://localhost:4317 or http://localhost:4318.

  3. Visit http://localhost:3000 to access the Grafana UI. There you can test if both the non-custom and custom metrics are being collected as expected. These will have the prometheus ("_") nomenclature rather than the standard otel one.

If you see repeated Failed to export ... Connection refused: localhost/127.0.0.1:4317 warnings in your logs while running locally, this just means no collector is currently listening on that port — either ignore them or start the otel-compose.yaml above.

Adding OTEL collector plugin to a Spring Boot application (GraalVM native image, Spring Boot 4.x, see des-router as an example)

These instructions cover services built as a GraalVM native image, on Spring Boot 4.0+ (des-router is on 4.0.6). Native images can't use the OTel Java agent (no bytecode instrumentation at runtime), so instrumentation is done manually via a dependency instead. If your service is still on Spring Boot 3.x (e.g. bcg-service, atc-service, fred-service), see the non-native section below instead.

Service Repository Changes

  1. Add the OTel dependency to the pom.xml:
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-opentelemetry</artifactId>
</dependency>

This is Spring Boot's all-in-one OTel starter — it alone provides both metrics and tracing export over OTLP, and no extra Micrometer/OTel dependencies are needed alongside it (see Troubleshooting below).

  1. If the service includes the Protobuf dependency, you may need to bump its version to 4.28.3 to make it compatible with the OTel/Micrometer dependencies.

Config Repository Changes

Adapt the property values to your TLA:

    env:
      - name: OTEL_TRACES_SAMPLER
        value: always_on
      - name: OTEL_TRACES_EXPORTER
        value: otlp
      - name: OTEL_SERVICE_NAME
        value: des-router
      - name: OTEL_RESOURCE_ATTRIBUTES
        value: deployment.environment=nxt
      - name: MANAGEMENT_OTLP_METRICS_EXPORT_ENABLED
        value: 'true'
      - name: MANAGEMENT_OPENTELEMETRY_TRACING_EXPORT_OTLP_ENDPOINT
        value: http://otel-collector-agent.otel-splunk.svc:4318/v1/traces
      - name: MANAGEMENT_OTLP_METRICS_EXPORT_URL
        value: http://otel-collector-agent.otel-splunk.svc:4318/v1/metrics
      - name: MANAGEMENT_OTLP_METRICS_EXPORT_STEP
        value: 10s

Verification

  1. Go to SignalFx and make sure both custom and non-custom metrics are coming through for the service with the dot annotation rather than the underscore, indicating the metrics are coming through the Otel collector. If unsure, you can disable Prometheus scraping.
  2. Check that the service is findable in APM (traces).
  3. Make sure to change dashboards and alerts to the new metric naming — see Metric Renaming and Dashboard Changes below.

Adding OTEL collector to a non-native Spring Boot application (Micrometer + OTLP, Spring Boot 3.x, see atc-service and fred-service as an example)

These instructions cover regular (non-native/JVM) Spring Boot services on Spring Boot 3.x (atc-service is on 3.2.2, fred-service and bcg-service are both on 3.5.15).

If/when a service upgrades to Spring Boot 4.x, follow the native section above instead — the property names differ between the two Spring Boot versions (see Troubleshooting for details).

Service Repository Changes

Add exactly these 4 dependencies to the pom.xml (this is the proven set from ATC and FRED — no explicit version pins are needed, they resolve via Spring Boot's existing dependency management):

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-otlp</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
  • micrometer-registry-otlp exports Micrometer MeterRegistry metrics (including hand-written custom metrics) over OTLP.
  • micrometer-tracing + micrometer-tracing-bridge-otel bridge Spring's tracing abstraction to OTel, replacing what the Java agent used to provide via bytecode instrumentation.
  • opentelemetry-exporter-otlp is the actual OTLP wire exporter used by the tracing bridge.

Do not try spring-boot-starter-opentelemetry here — ATC tried this first, but that starter only exists for Spring Boot ≥4.0.0 and fails to resolve on 3.x (see Troubleshooting).

Config Repository Changes

  1. Mount a ConfigMap file at /var/app/otel and reference it via SPRING_CONFIG_IMPORT, instead of setting individual OTEL_*/MANAGEMENT_OTLP_* env vars. This consolidates all OTel config into one file and avoids scattering related settings across many env vars:
    volumeMounts:
      <name>-otel-config:
        mountPath: /var/app/otel
  ...
    volumes:
      <name>-otel-config:
        configMap:
          name: '{{ template "application.name" $ }}-otel'
  ...
    env:
      SPRING_CONFIG_IMPORT:
        value: optional:file:/var/app/otel/application-otel.yml
  1. Add the application-otel.yml content itself under configMap.files, adapting service.name/deployment.environment to your TLA/environment:
  configMap:
    files:
      otel:
        application-otel.yml: |-
          management:
            tracing:
              enabled: 'true'
              sampling:
                probability: '1.0'
            otlp:
              tracing:
                endpoint: http://otel-collector-agent.otel-splunk.svc:4318/v1/traces
              metrics:
                export:
                  enabled: 'true'
                  url: http://otel-collector-agent.otel-splunk.svc:4318/v1/metrics
                  step: 10s
            opentelemetry:
              resource-attributes:
                service.name: bcg
                deployment.environment: nxtpp

Port matters: use 4318 (HTTP OTLP), not 4317 (gRPC OTLP). management.otlp.* properties use the HTTP exporter by default. Using 4317 here will silently fail to export (see Troubleshooting).

Note the exact volume/configMap schema depends on your Helm chart — adapt accordingly. FRED's config repo skips this ConfigMap pattern entirely and instead bakes minimal OTLP metrics-only config directly into application.properties in the service repo (FRED has not enabled tracing) — either approach is valid, but the ConfigMap pattern above (used by ATC and BCG) is recommended as it keeps config-repo-only environments (nxt/qa/drk/prd) fully self-contained without a service-repo change per environment.

Verification

  1. Go to SignalFx and make sure both custom and non-custom metrics are coming through for the service with the dot annotation rather than the underscore, indicating the metrics are coming through the Otel collector. If unsure, you can disable Prometheus scraping.
  2. Check that the service is findable in APM (traces).
  3. Make sure to change dashboards and alerts to the new metric naming — see Metric Renaming and Dashboard Changes below.

Adding OTEL collector to a Quarkus application (GraalVM native image, see des-recommendations as an example)

These instructions cover services built as a GraalVM native image.

Service Repository Changes

  1. Add the following dependencies to the pom.xml:
<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-micrometer-opentelemetry</artifactId>
</dependency>

quarkus-micrometer-opentelemetry alone is enough to bridge Micrometer metrics and traces to OTLP — keep quarkus-micrometer-registry-prometheus alongside it only if you still want Prometheus scraping to coexist (see Troubleshooting below).

  1. Add the following properties (adjusting values for your service) to application.properties:
quarkus.otel.metrics.enabled=true
quarkus.otel.logs.enabled=true
quarkus.otel.traces.enabled=true
quarkus.otel.exporter.otlp.enabled=true
quarkus.otel.traces.sampler=always_on
quarkus.otel.traces.sampler.arg=0.0  # Traces disabled by default, set to 1.0 to enable
quarkus.otel.service.name=des-recommendations
quarkus.otel.exporter.otlp.endpoint=http://localhost:4317
quarkus.application.name=recommendations-service
quarkus.otel.metric.export.interval=5s

Config Repository Changes

Adapt the property values to your TLA: 

    QUARKUS_OTEL_SERVICE_NAME: recommendations-decision
    QUARKUS_OTEL_RESOURCE_ATTRIBUTES: deployment.environment=prdpp
    QUARKUS_OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector-agent.otel-splunk.svc:4317
    QUARKUS_OTEL_TRACES_SAMPLER_ARG: 0.0d  # Traces disabled by default, set to 1.0 to enable

Verification

  1. Go to SignalFx and make sure both custom and non-custom metrics are coming through for the service with the dot annotation rather than the underscore, indicating the metrics are coming through the Otel collector. If unsure, you can disable Prometheus scraping.
  2. Check that the service is findable in APM (traces).
  3. Make sure to change dashboards and alerts to the new metric naming — see Metric Renaming and Dashboard Changes below.

Note: when filtering dashboards, for some reason you will not be able to filter both by service and namespace in most metrics, as this will stop the data from coming through.

Metric Renaming

The semantic conventions repo is the source of truth for custom metrics within the company. It contains Copilot instructions as a guide to migrate custom metrics from their old naming to Otel compliant naming, and it should be updated when we use a metric name or attribute that is not in there (this shoul ALWAYS be done by Copilot and not manually)

For a swift metric naming migration, follow these instructions:

  1. Clone the semantic conventions repository: https://github.com/Flutter-Global/observability-custom-semantic-conventions

  2. Then, open a copilot terminal on a base folder and give it the path for both the above repo and the repo you want the metric naming change on.

  3. Tell Copilot: "Read the instructions in the observability-custom-semantic-conventions repository and use it to update all custom metrics in the {{ REPO NAME HERE }} repository. You will also need to update the metrics documentation and README in the {{ REPO NAME HERE }} following the example in fred-service and all docs in the semantics repo if any new metrics or attributes have been added. Please follow the copilot instructions in the semantics repo to make these changes as well".

  4. Make sure all metrics are used (remove them if not) and check if they can be swapped for an auto-instrumented one. Copilot will often pick up on these things, but it gets missed often. You may also need to go through the semantic conventions repo yourself to see if any namespace exists that makes sense for a particular metric (namespace will be the first word on the metric name)

  5. Raise a service PR and make sure the team is happy the new metric names (most times, a discussion will be needed)

  6. Make sure to get Copilot to update the semantics-convention repo PR with the latest version of the metric naming. Once happy, raise the semantics-convention repo PR for approval with the Observability team.

Note: this process may take several days, to make your life easier, it is useful to ask Copilot to make notes of where things were left and the reasoning for the metric changes so it can pick it up from where it was left

Dashboard Changes

Dashboards will need updating from the old Prometheus name (with underscores) to the new Otel way (with dots). Some metric names will also vary slightly (for example, having a _count at the end). If custom metric names have been changed, or if we need to substitute them with non-custom metrics, this will also need updating. Please make sure all charts are working as expected after the changes.

One trick that can help with updating dashboards:

  1. Export the existing dashboard (see image below - Use JSON)

How to export a dashboard

  1. (Optional) Export Grafana's dashboard JSON

  2. Get copilot to update the dashboard with the new metrics and the Otel naming convention, use Grafana's dashboard if you want more accuracy or if some charts are missing from SignalFx

  3. Import the updated dashboard to SignalFx (remember to change the name to something like {{TLA}}-Otel to not have them duplicated, you can do this on the JSON file itself). Remember to double check that all metrics are picked up correctly, as Copilot can make mistakes, then import this new dashboard to the repo once you are done.

How to export a dashboard

Troubleshooting / Lessons Learned

  • Spring Boot native — don't add extra OTel/Micrometer dependencies alongside the starter. spring-boot-starter-opentelemetry already bundles OTLP metrics export and tracing. Adding micrometer-registry-otlp, micrometer-tracing-bridge-otel, or opentelemetry-exporter-otlp on top of it is redundant — des-router tried all three and removed them once the starter alone proved sufficient.
  • Quarkus native — don't add quarkus-opentelemetry or quarkus-micrometer separately. quarkus-micrometer-opentelemetry alone already covers both metrics and tracing bridging; des-recommendations tried the split-out dependencies and consolidated back down to just this one (plus quarkus-micrometer-registry-prometheus, kept for Prometheus/OTel coexistence).
  • spring-boot-starter-opentelemetry does not exist for Spring Boot 3.x. ATC tried it first and the dependency failed to resolve — it's only published from Spring Boot 4.0.0 onwards. On 3.x, use the 4-dependency Micrometer + OTLP set instead (see the non-native section above).
  • Wrong OTLP port silently no-ops. management.otlp.* properties use the HTTP OTLP exporter, which listens on 4318, not the gRPC OTLP port 4317. BCG's original pilot environment config used 4317 and this was very likely why several early rollout attempts never got metrics or traces flowing, despite no errors being thrown.
  • Consolidate OTel config into a single mounted file rather than many env vars. Both ATC and BCG now use a SPRING_CONFIG_IMPORT: optional:file:/var/app/otel/application-otel.yml env var pointing at a ConfigMap-mounted YAML file, instead of setting management.tracing.*/ management.otlp.*/management.opentelemetry.* individually as separate env vars. This is easier to review/diff per environment and avoids partially-applied config during rollout.
  • Spring Boot 3.x vs. 4.x — the traces OTLP endpoint property name changed. Boot 3.x uses management.otlp.tracing.endpoint (env var form: MANAGEMENT_OTLP_TRACING_ENDPOINT); Boot 4.x renamed it to MANAGEMENT_OPENTELEMETRY_TRACING_EXPORT_OTLP_ENDPOINT. Using the wrong one for your service's Boot version silently no-ops (traces just won't export) rather than failing loudly — see the native (4.x)/non-native (3.x) sections above for the version-correct config, and double check spring.boot.version in the parent POM before copying config from another service. The metrics export properties (management.otlp.metrics.export.*) are unaffected and identical across both versions.
  • Local dev "connection refused" log noise: see the note under "Testing the collector" above — harmless if you're not running a local collector.
  • Protobuf version conflicts: if the service has an explicit Protobuf dependency, you may need to bump it (e.g. to 4.28.3) for compatibility with the OTel/Micrometer dependencies. This has so far only been needed for the native/GraalVM path (des-router) — the standard Micrometer + OTLP dependency set on 3.x (ATC/FRED/BCG) has not required any protobuf version bump.
  • Newly-registered metrics only appear once their code path has actually run. Lazily-created Micrometer meters (e.g. Counter.builder(...).register(...), Timer.builder(...).register(...)) only exist in SignalFx once they've been recorded at least once — e.g. a duration metric tied to a specific message source will only appear after a message from that source has been processed. Gauges registered eagerly at bean construction time, by contrast, appear immediately (even at value 0). If a custom metric seems to be "missing" right after a config change, check whether its code path has actually been exercised yet before assuming the OTel config is wrong.