What You Didn't Know About kubectl apply

1 kubectl apply Cannot Remove Annotations

I usually prefer deploying applications using YAML, and recently, when using kubectl apply, I encountered an issue. When I use kubectl rollout restart to restart an application, kubectl adds kubectl.kubernetes.io/restartedAt: <current time> to spec.template.metadata.annotations. Then, when I update the YAML file and use kubectl apply, it doesn’t remove the kubectl.kubernetes.io/restartedAt: "2022-07-26T11:44:32+08:00" from the annotations.

Object after kubectl rollout restart:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#kubectl get -f nginx-deployment.yaml -o yaml
....
spec:
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2022-07-26T11:44:32+08:00"
      labels:
        app: nginx
...

No kubectl.kubernetes.io/restartedAt in the file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# cat nginx-deployment.yaml
...
spec:
  template:
    metadata:
      labels:
        app: nginx

# kubectl apply -f nginx-deployment.yaml
deployments.apps/nginx configured

# kubectl get -f nginx-deployment.yaml -o yaml
....
spec:
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2022-07-26T11:44:32+08:00"
      labels:
        app: nginx
...

Why can’t kubectl apply remove the annotation? kubectl apply, as part of Declarative Management of Kubernetes Objects, should ideally reflect what I have defined. To find the answer to this problem, let’s first understand how kubectl apply works.

2 Working Principle of kubectl apply -f

The kubectl apply command has some default command-line parameters, and the important ones here are --validate=true --override=true --dry-run=none --server-side=false. In other words, kubectl apply -f file is equivalent to kubectl apply -f file --validate=true --override=true --dry-run=none --server-side=false.

Here’s an explanation of how kubectl apply -f file works:

Overall process:

1. Get the openapi schema.
2. Read the file.
3. Validate the fields.
4. Get the meta.RESTMapping of the object.
5. Retrieve the object from the apiserver.
6. Generate a patch and make a patch request, or make a create request.
7. Display the final output.

2.1 Get the OpenAPI Schema

Retrieve the schema definitions for all resources (built-in objects, CRD objects, and aggregate apiserver objects) from the /openapi/v2 endpoint of the apiserver.

2.2 Read the File

kubectl apply supports content formats in JSON or YAML.

If the parameter following -f is a file, it is read directly.

If the parameter following -f is a directory, it reads files with extensions “.json,” “.yaml,” or “.yml” in that directory. If -R, --recursive=true is set, it recursively reads files with extensions “.json,” “.yaml,” or “.yml” in subdirectories.

If the parameter following -f is “-”, it reads from standard input.

The -f parameter also supports addresses starting with “http://” and “https://” fetching content from these addresses.

2.3 Validate Fields

Next, it validates whether the types of various fields are valid or unknown and whether any required fields are missing. This is client-side field validation.

Starting from Kubernetes version 1.24, kubectl supports ServerSideFieldValidation, where the –validate parameter defaults to “strict.” This means that field validation is performed on the apiserver, and if the server doesn’t support it, client-side validation is used. Specific implementations are added to CreateOptions, PatchOptions, and UpdateOptions through the fieldValidation field. For more details, visit the design document.

For example, for type errors:

1

error: error validating "nginx-deployment.yaml": error validating data: ValidationError(Deployment.spec.replicas): invalid type for io.k8s.api.apps.v1.DeploymentSpec.replicas: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false

For nonexistent field errors:

1

error: error validating "nginx-deployment.yaml": error validating data: ValidationError(Deployment.spec): unknown field "notexist" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false

For missing required field errors:

1

error: error validating "nginx-deployment.yaml": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false

2.4 Obtaining the meta.RESTMapping of the Object

After successfully validating the fields, the discovery client is used to obtain the meta.RESTMapping of the object. This information is used to determine whether the object is namespaced and the object’s Resource (which determines the path to access the apiserver).

2.5 Retrieving Objects from the apiserver

A GET request is made to the apiserver to retrieve the object.

2.6 Performing a Patch or Create Operation

Different operations are taken based on whether the object exists. When the object exists, a patch calculation is performed. If the patch is not empty (indicating changes), a patch request is made. If the object does not exist, a create request is made.

2.6.1 Patching When the Object Exists

Concepts

Last Applied Configuration from the Previous Apply: kubectl apply stores the content from the previous apply operation in the object’s annotations as kubectl.kubernetes.io/last-applied-configuration. This is stored in the annotations of the object in the apiserver.

Content of the Current Apply Operation: The content of the file that was read is added to the annotations with the key kubectl.kubernetes.io/last-applied-configuration, and its value is set to the content of the file (the previous kubectl.kubernetes.io/last-applied-configuration in annotations is removed). Afterward, it is encoded into bytes using JSON encoding.

Current Object Content: Refers to the content of the object in the apiserver.

2.6.2 Field Diff Algorithm

Field Type Categories (This list only includes types parsed from JSON to Golang and does not cover all Golang types, such as struct, which is represented as a map type):

• primitive：string, float64, bool, int64, nil
• list：slice, array
• map：map, struct

2.6.3 JSON Merge Patch

For non-built-in type objects (CRD and aggregator resources), the patch is calculated using CreateThreeWayJSONMergePatch, with the patch type being json merge patch. JSON merge patch is a standard protocol defined in RFC7386.

The calculation steps are as follows:

• Compare the “Current Object Content” with the “Content of the Current Apply Operation” to calculate “added and updated fields.” Exclude fields in “added and updated fields” with null values.
• Compare the “Last Applied Configuration from the Previous Apply” with the “Content of the Current Apply Operation” to calculate “deleted fields” (including fields added and updated with null values).
• Combine “deleted fields” and “added and updated fields” into the final patch.
• Check that “apiVersion,” “kind,” and “metadata.name” cannot be modified in the patch (i.e., these fields cannot be changed in the patch).

2.6.4 Strategic Merge Patch

For built-in objects, the patch is calculated using CreateThreeWayMergePatch, with the patch type being strategic merge patch.

Strategic merge patch is a patch type defined by Kubernetes itself. It is generally used when dealing with slices of fields that are of struct type. Instead of directly replacing field values, it aggregates elements when merging. To achieve this, a merge strategy, patchStrategy, needs to be defined, and a field is required to distinguish (similar to a unique key in a database) whether two elements are the same. This field is referred to as patchMergeKey.

Here’s an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

    // Current Condition of persistent volume claim. If underlying persistent volume is being
	// resized then the Condition will be set to 'ResizeStarted'.
	// +optional
	// +patchMergeKey=type
	// +patchStrategy=merge
	Conditions []PersistentVolumeClaimCondition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type" protobuf:"bytes,4,rep,name=conditions"`

// PersistentVolumeClaimCondition contails details about state of pvc
type PersistentVolumeClaimCondition struct {
	Type   PersistentVolumeClaimConditionType `json:"type" protobuf:"bytes,1,opt,name=type,casttype=PersistentVolumeClaimConditionType"`
	Status ConditionStatus                    `json:"status" protobuf:"bytes,2,opt,name=status,casttype=ConditionStatus"`
	// Last time we probed the condition.
	// +optional
	LastProbeTime metav1.Time `json:"lastProbeTime,omitempty" protobuf:"bytes,3,opt,name=lastProbeTime"`
	// Last time the condition transitioned from one status to another.
	// +optional
	LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty" protobuf:"bytes,4,opt,name=lastTransitionTime"`
	// Unique, this should be a short, machine understandable string that gives the reason
	// for condition's last transition. If it reports "ResizeStarted" that means the underlying
	// persistent volume is being resized.
	// +optional
	Reason string `json:"reason,omitempty" protobuf:"bytes,5,opt,name=reason"`
	// Human-readable message indicating details about last transition.
	// +optional
	Message string `json:"message,omitempty" protobuf:"bytes,6,opt,name=message"`
}

In this example, the Conditions field has a type of []PersistentVolumeClaimCondition, and it is marked with +patchMergeKey=type and +patchStrategy=merge. This indicates that this field should use the merge strategy when aggregating, and it distinguishes elements based on the type field’s value.

Strategic merge patch also has other special keys, such as "$patch," "$retainKeys," "$deleteFromPrimitiveList," and "$setElementOrder." For more details, you can visit the strategic-merge-patch proposal and Preserve Order in Strategic Merge Patch.

If the apiserver has an openapi schema, the patchMergeKey and patchStrategy for fields are obtained from the schema. If the apiserver does not have an openapi interface, the information is extracted from the JSON tags of the fields in the struct, which contain “patchStrategy” and “patchMergeKey” values. Based on this information, the patch is generated.

The patch generation process:

• Compare the “Current Object Content” with the “Content of the Current Apply Operation” to calculate “added and updated fields.”
• Compare the “Last Applied Configuration from the Previous Apply” with the “Content of the Current Apply Operation” to calculate “deleted fields.”
• Combine “deleted fields” and “added and updated fields” into the “final patch.”
• If –override is set to false, check for conflicts between the differences between the “Last Applied Configuration from the Previous Apply” and the “Current Object Content” and the “final patch” (i.e., if both attempt to modify the same key simultaneously).

For example, if the “Last Applied Configuration from the Previous Apply” is "{k: a}", the “Current Object Content” is "{k: b}", and the “Content of the Current Apply Operation” is "{k: c}", where "{k: b}" was not modified by kubectl apply, and it is now being modified based on the “Last Applied Configuration from the Previous Apply” to become "{k: c}", a conflict will arise.

2.6.5 Creating the Object When It Doesn’t Exist

If the object does not exist in the apiserver, it is created as a new resource.

3 Displaying the Final Output

When the creation is successful, the output will be:

1

deployment.apps/nginx-deployment created

If the patch is not empty and the patching is successful, the output will be:

1

deployment.apps/nginx-deployment configured

If the patch is empty, meaning there are no changes, the output will be:

1

deployment.apps/nginx-deployment unchanged

You can specify the output format using -o or --output, for example, -o name will output:

1

deployment.apps/nginx-deployment

4 Reasons for Unable to Remove Annotations

Returning to the previous question, the reason why annotations cannot be removed is that the deleted fields are calculated based on the differences between the “Last Applied Configuration from the Previous Apply” and the “Content of the Current Apply Operation.” However, the kubectl.kubernetes.io/restartedAt field in spec.template.metadata.annotations is not present in the “Last Applied Configuration from the Previous Apply,” so it won’t be removed.

4.1 How to Remove this Annotation

Use a Patch Request to Remove the Annotation

You can use kubectl edit to remove it, or use a patch request like kubectl patch deployment nginx-deployment -p '{"spec":{"template":{"metadata":{"annotations": {"kubectl.kubernetes.io/restartedAt": null}}}}}'.

Utilize kubectl Apply Mechanism for Removal

When “Last Applied Configuration from the Previous Apply” has spec.template.metadata.annotations as null: Set spec.template.metadata.annotations to null in the YAML file, and then apply it using kubectl.

1
2
3
4
5
6
7

....
spec:
  template:
    metadata:
      annotations: null
      labels:
        app: nginx

In typical cases where “Last Applied Configuration from the Previous Apply” has spec.template.metadata.annotations as non-null: This operation is a bit more involved.

1. Add kubectl.kubernetes.io/restartedAt to the YAML file and then apply it using kubectl to record this annotation in the kubectl.kubernetes.io/last-applied-configuration field in annotations.

   1 2 3 4 5 6 7 8 9 10

   .... spec: template: metadata: # add next two line annotations: a: b kubectl.kubernetes.io/restartedAt: "2022-07-26T11:44:32+08:00" labels: app: nginx

2. Remove this annotation from the YAML file, and then apply it using kubectl.

Use an Update Request to Remove It

Use kubectl replace -f nginx-deployment.yaml --save-config=true to remove it.

5 Other Important kubectl Apply Command Line Parameters

5.1 Warnings for Applying to Objects Not Managed by kubectl Apply

If the object in the apiserver does not have the kubectl.kubernetes.io/last-applied-configuration annotation in its annotations, executing kubectl apply will output a warning similar to the following:

1

Warning: resource deployments/nginx-deployment is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.

5.2 server side apply

All the mechanisms mentioned above are client-side apply, and they have several issues:

• When multiple different clients operate on the same object, it becomes difficult to manage fields added by non-kubectl apply operations.
• Conflicts can occur when different clients operate on the same field, and it’s challenging to determine which client made the change.
• Strategic merge patch is not a standard protocol, and any modifications require changes to both the client and server, with considerations for version compatibility.

Due to the issues mentioned above (among others), the Kubernetes community has introduced server-side apply. Here, we won’t delve into it, but if you want to understand the design intent, please visit the design document.

You can specify --server-side=true to use server-side apply. When using server-side apply, a patch request is made directly with the content read as the request body, and the patch type is ApplyPatch. This eliminates the need for the client to calculate the patch (i.e., it does not need to get the object first and then calculate the patch by diffing). The calculation of the patch and merge is done on the apiserver side.

The successful execution will output deployment.app/nginx-deployment serverside-applied.

5.2.1 Conflict Resolution

Server-side apply records which client manages each field in the managedFields field.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

 - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubectl.kubernetes.io/last-applied-configuration: {}
    manager: kubectl
    operation: Update
    time: "2023-10-04T10:54:28Z"
    
 - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:template:
          f:metadata:
            f:annotations:
              .: {}
              f:kubectl.kubernetes.io/restartedAt: {}
    manager: kubectl-rollout
    operation: Update
    time: "2023-10-05T03:11:36Z"

When two different managers modify the same field, conflicts can arise. Here, “manager” refers to the FieldManager parameter specified by the client when making Patch, Update, or Create requests.

Three Ways to Resolve Conflicts:

Seize Control: If the --force-conflicts flag in kubectl apply is set to true, it will seize control of the field and make it its own.

Give Up Control: Remove the conflicting field from the file’s content.

Share Control: Add the current value of the field from the apiserver to the file’s content, and then use kubectl apply --server-side to gain shared control over the field.

How to Use Server Side Apply to Remove the annotation kubectl.kubernetes.io/restartedAt

Currently, there is no straightforward way to remove this annotation using server-side apply. Related issue: https://github.com/kubernetes/kubernetes/issues/99003

However, if you insist on using server-side apply for removal, there is a way:

1. Remove the control of kubectl-rollout over the annotation, i.e., remove the manager with the element kubectl-rollout from the managedFields.

   1	kubectl patch deployment nginx-deployment --type=json -p '[{ "op": "remove", "path": "/metadata/managedFields/3"}]'

2. In the nginx-deployment.yaml file, add kubectl.kubernetes.io/restartedAt: "2023-10-05T11:11:36+08:00" to spec.template.metadata.annotations, then execute kubectl apply -f nginx-deployment.yaml --server-side to regain control over the field.

   1 2 3 4 5

   spec: template: metadata: annotations: kubectl.kubernetes.io/restartedAt: "2022-07-26T11:44:32+08:00"

3. Remove this annotation from the nginx-deployment.yaml file, and then apply it again using kubectl apply -f nginx-deployment.yaml --server-side.

While server-side apply solves some problems, it does not work seamlessly with the existing client-side apply when multiple managers simultaneously control a field, preventing the deletion of that field.

5.3 Dry Run

By default, --dry-run is set to none, meaning the command will actually take effect.

If --dry-run is set to client, it won’t execute Create and Patch requests, including calculating patches. The output will resemble deployment.apps/nginx-deployment configured (dry run) or deployment.apps/nginx-deployment created (dry run).

If --dry-run is set to server, it first validates whether the object’s Patch request supports dry run. If it doesn’t support dry run, an error like xxx doesn't support dry-run is reported. If it does support it, the apiserver will only execute the request without saving the modified object to etcd.

5.4 –prune

This is a relatively less used feature. Its purpose is to delete resources that meet the following conditions:

• The resource type is in the --prune-whitelist list. If --prune-whitelist is empty, the default resource type list is used (ConfigMap, Endpoints, Namespace, PersistentVolumeClaim, PersistentVolume, Pod, ReplicationController, Secret, Service, Job, CronJob, Ingress, DaemonSet, Deployment, ReplicaSet, StatefulSet).
• The resource is in the namespace of the object being operated on by kubectl apply.
• The resource is managed by kubectl apply, meaning it has kubectl.kubernetes.io/last-applied-configuration in its annotations.
• The resource is not the object being operated on by kubectl apply this time.

This is a potentially dangerous command, so use it with caution.

5.5 –force

When making a patch request with client-side apply, if the apiserver returns a Conflict error, it will retry 5 times. Other errors will not be retried.

After the retry attempts have been exhausted without success or in case of other errors (no further retries), and if the apiserver returns a Conflict error or an IsInvalid error, and if the --force flag is set, it will delete the object and then create it again.

First, it attempts to create the object based on the “Content of the Current Apply Operation.” If this fails, it attempts to create the object based on the “Last Applied Configuration from the Previous Apply.”

5.6 –overwrite

When using strategic merge patch to calculate a patch, if the field being modified has been modified by a non-kubectl apply operation (i.e., it differs from the “Last Applied Configuration”), and --overwrite is set to false (default is true), a conflict error will occur.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

warning: error calculating patch from openapi spec: patch:
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      .....
spec:
  template:
    metadata:
      annotations: null

conflicts with changes made from original to current:
metadata:
  annotations:
    deployment.kubernetes.io/revision: "10"
    ...
error: error when applying patch:

to:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "nginx-deployment", Namespace: "default"
for: "nginx-deployment.yaml": error when creating patch with:
original:
....
modified:
....
current:
....
for: "nginx-deployment.yaml": patch:
metadata:
  annotations:
  ....

5.7 Subcommands

kubectl apply includes three subcommands:

• view-last-applied
• edit-last-applied
• set-last-applied

view-last-applied: View the value of kubectl.kubernetes.io/last-applied-configuration in the object’s annotations.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

# kubectl apply view-last-applied deployments.apps nginx-deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  name: nginx-deployment
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:1.18
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

set-last-applied: Set the value of kubectl.kubernetes.io/last-applied-configuration in the object’s annotations to the content of the file, without modifying other fields.

So, the method to remove an annotation as mentioned earlier can also be performed as follows:

1. Add kubectl.kubernetes.io/restartedAt to nginx-deployment.yaml.
2. Execute kubectl apply set-last-applied -f nginx-deployment.yaml --create-annotation=true.
3. Then, remove kubectl.kubernetes.io/restartedAt from nginx-deployment.yaml.
4. Execute kubectl apply again.

edit-last-applied: Edit the kubectl.kubernetes.io/last-applied-configuration value in the object’s annotations, similar to kubectl edit.

6 Reference

server-side field validation

openapi-v3-field-validation-ga

[Remove unexisting annotations in kubectl apply -f](https://stackoverflow.com/questions/60988558/remove-unexisting-annotations-in-kubectl-apply-f)

how-apply-calculates-differences-and-merges-changes

How to clear server-defaulted fields or fields set by other writers