Skip to content

FRED - alerts

Pst - FRED-Detector

This detector monitors FRED metrics only.

  • A Slack alert is triggered when the DES ERROR status count is greater than 10 in each data point over 5 minutes regardless the DES request.
  • A Slack alert is triggered per Kafka topic when the executor service thread pool queue remains above 100 per pod (maximum queue is 100 * number of pods) for 5 minutes.
  • A PagerDuty alert is triggered when more than 10 FRED messages are dropped in each data point for 80% of a 1-minute duration.
  • An alert is triggered when more than 10 HTTP responses from all clients, including DES and BCG, have a status other than 200.

Pst - K8S Detector

This detector monitors the Sstk8sPRD Kubernetes cluster.

  • A Slack alert is triggered when CPU usage for any pod exceeds 95% for 5 minutes per pod.
  • A Slack alert is triggered when memory usage for any pod exceeds 95% for 5 minutes per pod.
  • A Slack alert is triggered per Kubernetes namespace when a container status reason count is not 0 for 5 minutes. Monitored reasons include ImagePullBackOff, CrashLoopBackOff, ContainerCannotRun, OOMKilled, CreateContainerConfigError, Error, and ErrImagePull.
  • A Slack alert is triggered per namespace when the unavailable replica count is greater than 0 for 5 minutes.