How to Reduce Disk Space Used by Container Images? Practical Tips for Efficient Storage Configuration in containerd

xiaoqing included in category kubernetes

2024-09-18 2024-09-18 605 words 3 minutes

Contents

The previous article discussed the principles of containerd’s image storage, where images are stored in two forms in containerd: one as the original image files (manifest, config, blob), and the other as snapshots of the decompressed layers. When running containers, only the decompressed layer snapshots are used, which leads to wasted space for the original image blobs in scenarios where image pushing is not needed. In environments with limited disk space, it is easy for images to consume excessive disk space. So how can we save space?

This article is based on containerd v1.7.15.

1 How to Reduce Image Storage Usage

For containers running via CRI, such as when containerd is used as the runtime for Kubernetes or when using the crictl command to run containers:

In the containerd configuration file, set discard_unpacked_layers to true, which allows the original image blob in the content store to be cleaned up by garbage collection (GC). Note that this option only applies to newly pulled images.

1
2
3
4
5
6
[plugins]
....
[plugins."io.containerd.grpc.v1.cri"]
.....
[plugins."io.containerd.grpc.v1.cri".containerd]
discard_unpacked_layers = true

For images imported using the ctr image import command, use the --discard-unpacked-layers option.

2 Considerations and Risks

However, be aware that enabling discard_unpacked_layers will prevent you from exporting and pushing images. Use this feature with caution.

3 Situations Where It’s Not Recommended

Regular Image Repository Cleanup: If the image repository is regularly cleaned, and a running container’s image is deleted from the repository, the image cannot be re-pushed to the repository or run on other nodes.
Sufficient Disk Space: If you have ample disk space.

4 Situations Where It’s Recommended

Limited Disk Space: When disk space is tight.
No Need to Push Images: If you don’t need to push images.

5 Underlying Mechanism

In the output of the ctr -n k8s.io content ls command, a containerd.io/gc.ref.content.l.<index> label will appear on the manifest. This label is used to mark which layers belong to this manifest. If a layer is not referenced by any manifest’s containerd.io/gc.ref.content.l.<index> label, it will be cleaned up by garbage collection (GC).

Below is an example of the containerd.io/gc.ref.content.l.<index> label in the manifest:

1
2
3
4
containerd.io/gc.ref.content.l.0=sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405
containerd.io/gc.ref.content.l.1=sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312
containerd.io/gc.ref.content.l.2=sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b
containerd.io/gc.ref.content.l.3=sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76

After enabling discard_unpacked_layers, the label containerd.io/gc.ref.content.l.<index> will not be added to the manifest during image pulls or imports. This means that the image blob will not be mapped to the manifest, allowing the image blob to be cleaned up by garbage collection (GC).

Relevant code:

https://github.com/containerd/containerd/blob/926c9586fe4a6236699318391cd44976a98e31f1/images/mediatypes.go#L189-L215

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
// IsLayerType returns true if the media type is a layer
func IsLayerType(mt string) bool {
	if strings.HasPrefix(mt, "application/vnd.oci.image.layer.") {
		return true
	}

	// Parse Docker media types, strip off any + suffixes first
	switch base, _ := parseMediaTypes(mt); base {
	case MediaTypeDockerSchema2Layer, MediaTypeDockerSchema2LayerGzip,
		MediaTypeDockerSchema2LayerForeign, MediaTypeDockerSchema2LayerForeignGzip:
		return true
	}
	return false
}
.....

// ChildGCLabels returns the label for a given descriptor to reference it
func ChildGCLabels(desc ocispec.Descriptor) []string {
	mt := desc.MediaType
	if IsKnownConfig(mt) {
		return []string{"containerd.io/gc.ref.content.config"}
	}

	switch mt {
	case MediaTypeDockerSchema2Manifest, ocispec.MediaTypeImageManifest:
		return []string{"containerd.io/gc.ref.content.m."}
	}

	if IsLayerType(mt) {
		return []string{"containerd.io/gc.ref.content.l."}
	}

	return []string{"containerd.io/gc.ref.content."}
}

// ChildGCLabelsFilterLayers returns the labels for a given descriptor to
// reference it, skipping layer media types
func ChildGCLabelsFilterLayers(desc ocispec.Descriptor) []string {
	if IsLayerType(desc.MediaType) {
		return nil
	}
	return ChildGCLabels(desc)
}