Adding OTEL collector to a SpringBoot application and enabling custom metrics

Config Repository Changes

1. Under additionalPodAnnotations, add:

       additionalPodAnnotations:
         instrumentation.opentelemetry.io/inject-java: otel-splunk/otel-collector
   

2. Under env, add:

       env:
         OTEL_SERVICE_NAME:
           value: '{ template "application.name" $ }'
         OTEL_RESOURCE_ATTRIBUTES:
           value: 'deployment.environment= { for example, nxtpp }'
   

3. If the application has the following property JAVA_TOOL_OPTIONS: -Xms512m -Xmx2g -XX:+UseG1GC -XX:NewRatio=1 in the configMap, you may need to change it to JDK_JAVA_OPTIONS: -Xms512m -Xmx2g -XX:+UseG1GC -XX:NewRatio=1, as the OTEL java agent will override the JAVA_TOOL_OPTIONS and the memory settings will not be applied. This way, both values are picked up appropriately.

4. For the micrometer OTLP exporter, please add the following properties to the configMap in each environment's values.yaml file:

       MANAGEMENT_OTLP_METRICS_EXPORT_URL: http://otel-collector:4318/v1/metrics
       MANAGEMENT_OTLP_METRICS_EXPORT_STEP: 15s
   

Service Repository Changes

1. If the service includes the Protobuf dependency, you may need to update the version to 4.28.3 to make it compatible with the micrometer OTLP dependency.

2. Add the following dependency for our custom micrometer metrics to be exported through OTLP:

       <dependency>
         <groupId>io.micrometer</groupId>
         <artifactId>micrometer-registry-otlp</artifactId>
       </dependency>
   

   This exporter should be able to coexist with the existing Prometheus one (although metrics may be duplicated), but if you want to remove or disable Prometheus in the config repo, please also remove the below dependency to prevent duplication:

       <dependency>
         <groupId>io.micrometer</groupId>
         <artifactId>micrometer-registry-prometheus</artifactId>
       </dependency>
   

3. Some custom metric names may need to change or be replaced by non-custom metrics. Please, visit https://github.com/Flutter-Global/uki-copilot-signalfx/blob/main/knowledge/tenants/platformservices/AUTO-INSTRUMENTATION-PLAN.md for a list of our TLAs and affected metrics. Tests may need updating after these changes.

Dashboard Changes

Dashboards will likely need updating if custom metric names have been changed, or if we need to substitute them with non-custom metrics. Please make sure all charts are working as expected after the changes.