Skip to content

Breaking Change

Some releases require updating both DEC and TIM, and errors can occur in TIM if the release process is not followed correctly. This document provides an example to demonstrate how to release breaking changes that update DEC and ultimately cause issues in TIM.

Example: Incompatible Serialization Scenario

Both DEC and TIM use the Google GSON library to serialize and deserialize the CppActivity object, while DES uses Jackson’s ObjectMapper for deserialization. This mismatch causes the DES response to contain incorrect data. To resolve this issue, both DEC and TIM must switch from GSON to ObjectMapper. However, releasing DEC changes with ObjectMapper before TIM is updated will cause errors, since TIM’s GSON cannot deserialize a CppActivity object serialized by ObjectMapper

Release steps

The following release steps can be used to avoid above example error. Please note, the steps are based on release in dec/tim qa envrionment.

Stop the Flink Job and create a savepoint in dec-cpp-activity-consumer-qapp. This can be done by calling Flink job stop api:

curl -X POST 'https://dec-cpp-activity-consumer-qabf.platformservices-dev.aws.private/jobs/ced1d19ae412edfd04b89d2c1833166f/stop' \
  -H 'Content-Type: application/json' \
  -d '{
    "drain": true,
    "targetDirectory": "s3://dec-qa-flink-state/cpp-activity-consumer-savepoints-qapp"
  }'

The curl command returns {"request-id":"c6105f1beba2f85733c1fd77610fb102"} if the request is handled by Flink successfully.

The ced1d19ae412edfd04b89d2c1833166f is the flink job ID, while s3://dec-qa-flink-state/cpp-activity-consumer-savepoints-qapp/ is the AWS S3 bucket to store the save point. The savepoint path can be found by in the Flink UI.

The next step is to markdown the new savepoints path in S3 butcket: * Search Splunk by using keyword savepoints * Or the savepoint path can be found in Flink job UI * Or login to AWS S3 to locate the folder using in the target-directory and then find out the latest savepoints

Flink job is stopped alt text

Savepoint is created: alt text

TIM consumes all messages

This is step is to monitor the dashboards to ensure TIM has consumed all messages.

  • The Records Emitted by Opertors and Total number of input/out records should be flat in this Grafana Dashboard
  • Ensure no message is published by DEC by checking kafbat UI

alt text

Release TIM

The TIM release can start after ensuring no message is consumed in TIM Flink job, for example tim-reward-processor-qapp. Ensure the TIM release is successful.

DEC is healthy alt text

Release DEC

In DEC PR, add initialSavepointPath and savepointRedeployNonce to the jobConfig, the savepointRedeployNonce can be any number if it doesn't exit in configuration number, or increase by 1 if savepointRedeployNonce can be found. The initialSavepointPath is the savepoint path created in first step

      initialSavepointPath:  s3://dec-nxt-flink-state/cpp-activity-consumer-savepoints-nxtpp/savepoint-218735-d20543b7ab64
      savepointRedeployNonce: 1

Here is an example PR

Release DEC change and check no error is found in dec and tim:

DEC is healthy alt text