The problem updating StatefulSets on Kubernetes platforms

Once a StatefulSet is deployed to Kubernetes and/or Red Hat OpenShift many of the attributes cannot be updated anymore.

Trying to update settings which are not updateable via the Kubernetes platform results in an error message like the following:

Error: UPGRADE FAILED: cannot patch "dx-blue-persistence-node" with kind StatefulSet: StatefulSet.apps "dx-blue-persistence-node" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Note: In our case dx-blue-persistence-node is the name of the StatefulSet we are going to update.

But over time it might be required to update the StatefulSet definition - for example the PVC template via the volumeClaimTemplates. This might be necessary e.g. if the related PVs need to be extended. Although existing PVs can be extended using the Kubernetes / OpenShift platform (depending on the storage driver) the definition of the StatefulSet can’t be updated on the Kubernetes platform by applying an updated YAML. This actually results in the situation that the PVC definiton on the Kubernetes platform differs from the volumeClaimTemplates definiton in the StatefulSet. A situation which we definitely don’t want - right? Furthermore this stuation bears the risk that scaling up the StatefulSet results in different PV sizes of the existing PVs and the newly created PV!

Therefore we’ll show how we can update a StatefulSet without downtime using the volumeClaimTemplates attribute as an example in the next sections.

Approach how to fix the problem

So how can we avoid that the current PVC definition in Kubernetes is different to the volumeClaimTemplates setting in the StatefulSet?

Sure, we could simply delete the StatefulSet and have it recreated. This would however cause an outage of our application as the pods would be deleted as well. Fortunately the Kubernetes CLIs (kubectl and/or oc) expose an APIs option to delete the StatefulSet without deleting the dependent pods at the same time, namely –cascade=orphan.

We can solve the problem by performing the following steps:

1. Update the existing PVC definition on the Kubernetes platform
2. Update the YAML definition on the StatefulSet to reflect the desired volumeClaimTemplates definiton
3. Delete the existing StatefulSet using the –cascade=orphan option like for example: oc --context [<context>] -n [<namespace>] delete statefulset <statefulset-name> --cascade=orphan
4. Apply the updated YAML for the StatefulSet
5. Verify the currently applied StatefulSet on Kubernetes