How Containerd Stores Images?

xiaoqing included in category kubernetes

2024-09-18 2024-09-20 2351 words 12 minutes

When we execute nerdctl pull, crictl pull, or nerdctl load, have you ever wondered how the images are stored in the system?

This article will introduce how containerd stores images to help you understand the image storage mechanism in containerd.

This article is based on containerd v1.7.15.

1 Image Storage Process

1.1 Image Overview

An image consists of the following three parts:

manifest: Describes the metadata of the image, which helps the client or container runtime verify the integrity and contents of the image.
config: Contains the instructions used during the image build process (for example, each line of a DockerFile), along with information such as the image’s history and runtime environment.
blob: The compressed files for each layer of the image, usually packaged with gzip.

1.2 containerd Image Storage Process

The containerd image storage process consists of two steps: Saving the original image and Unpacking the image to create snapshots.

The diagram below shows the key steps and file storage locations in the containerd image storage process, where the content store is containerd’s content storage area, and snapshots represent the snapshot storage area. A snapshot can be understood as a decompressed layer of the image.

1.2.1 Saving the Original Image

First, the manifest, config, and blob are stored in the content store, the default path is:
1
/var/lib/containerd/io.containerd.content.v1.content/blobs/
At the same time, the image metadata (digest, size) and content metadata (label, size) are stored in the meta.db file. This file is in the format of a bbolt database, the path is:
1
/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db

1.2.2 Unpacking the Image and Creating Snapshots

Each layer of the blob is unpacked, and its content is stored in the snapshots directory. The Snapshotter manages the image’s layered file system, with the default Snapshotter being overlayfs. The files are stored at:
1
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/
The metadata of the snapshots (parent, child, digest) is saved in both /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db and io.containerd.snapshotter.v1.overlayfs/metadata.db.

When a container is generated, a snapshot is created based on the top layer of the image to serve as the container’s file system.

2 image content in the disk

Here, we take the kindest/kindnetd:v20240202-8f1494ea image as an example to explain how it is stored in a Linux environment.

2.1 Storage Directory

The default storage directory for containerd is /var/lib/containerd, which stores data related to the content, CRI, introspection, metadata, runtime, and snapshotter modules.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# tree /var/lib/containerd/ -L 2 -F
/var/lib/containerd//
|-- io.containerd.content.v1.content/
|   |-- blobs/
|   `-- ingest/
|-- io.containerd.grpc.v1.cri/
|   |-- containers/
|   `-- sandboxes/
|-- io.containerd.grpc.v1.introspection/
|   `-- uuid
|-- io.containerd.metadata.v1.bolt/
|   `-- meta.db
|-- io.containerd.runtime.v1.linux/
|-- io.containerd.runtime.v2.task/
|   `-- k8s.io/
|-- io.containerd.snapshotter.v1.blockfile/
|-- io.containerd.snapshotter.v1.native/
|   `-- snapshots/
|-- io.containerd.snapshotter.v1.overlayfs/
|   |-- metadata.db
|   `-- snapshots/
`-- tmpmounts/

18 directories, 3 files

2.2 Image Metadata

Regardless of whether the image is imported or pulled, the raw image content is ultimately stored in the io.containerd.content.v1.content directory, while the image metadata is stored in the io.containerd.metadata.v1.bolt/meta.db file.

Using the ctr command to view the image list, this data is read from the io.containerd.metadata.v1.bolt/meta.db file:

1
2
3
4
# ctr -n k8s.io image ls
REF        TYPE      DIGEST      SIZE      PLATFORMS        LABELS
docker.io/kindest/kindnetd:v20240202-8f1494ea           application/vnd.docker.distribution.manifest.list.v2+json sha256:61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 26.5 MiB  linux/amd64,linux/arm64                        io.cri-containerd.image=managed
docker.io/kindest/local-path-helper:v20230510-486859a6      application/vnd.docker.distribution.manifest.list.v2+json sha256:d8a6cc3b66ff253e3adb1ed3017bd1556301a76bcb3a0d2a4db3c8eeb7f0fc0a 2.9 MiB   linux/amd64,linux/arm64      io.cri-containerd.image=managed

Opening the io.containerd.metadata.v1.bolt/meta.db file with boltbrowser, the images stored under the v1 → k8s.io → images path include labels, digest, and media type for each image. The digest for kindest/kindnetd:v20240202-8f1494ea is 61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988.

CAUTION
Be sure to stop the containerd process before opening the io.containerd.metadata.v1.bolt/meta.db file, as only one process can access it in read-write mode. Since containerd opens the file in read-write mode, this is detailed in issue 228.

2.3 Image Content in the `content` Directory

The image data is stored in the /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/{digest} path.

2.3.1 Manifest Index

The manifest index for the image is stored at /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988.

NOTE
If the image’s type is application/vnd.docker.distribution.manifest.list.v2+json, it indicates that the image supports multiple platforms, and its digest is the manifest index (containing multiple manifest entries).
If the type is application/vnd.docker.distribution.manifest.v2+json, the image supports a single platform, and its digest points to the manifest.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac",
      "size": 1092,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:fde0f6062db0a3b3323d76a4cde031f0f891b5b79d12be642b7e5aad68f2836f",
      "size": 1092,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    }
  ]
}

# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988  | sha256sum
61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988  -

The content is identical to what is retrieved from Docker Hub, and 61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 is the value of the docker-content-digest header in the HTTP response.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# token=$(https_proxy=127.0.0.1:7890 curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:kindest/kindnetd:pull" | jq '.token' -r)

# https_proxy=127.0.0.1:7890 curl -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Authorization: Bearer $token" https://registry-1.docker.io/v2/kindest/kindnetd/manifests/v20240202-8f1494ea -sv
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac",
      "size": 1092,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:fde0f6062db0a3b3323d76a4cde031f0f891b5b79d12be642b7e5aad68f2836f",
      "size": 1092,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    }
  ]
}

< HTTP/1.1 200 OK
< content-length: 685
< content-type: application/vnd.docker.distribution.manifest.list.v2+json
< docker-content-digest: sha256:61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988
< docker-distribution-api-version: registry/2.0
< etag: "sha256:61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988"
< date: Mon, 16 Sep 2024 07:35:47 GMT
< strict-transport-security: max-age=31536000
< docker-ratelimit-source: x.x.x.x

2.3.2 Manifest

Since it’s on the linux/amd64 platform, the manifest used is bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "digest": "sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5",
    "size": 1261
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405",
      "size": 7732034
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312",
      "size": 20002230
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b",
      "size": 17686
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76",
      "size": 269
    }
  ]
}

# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac | sha256sum
bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac  -

This is identical to what is retrieved from Docker Hub, as the docker-content-digest in the HTTP response header is also bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
$ https_proxy=127.0.0.1:7890 curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer $token" https://registry-1.docker.io/v2/kindest/kindnetd/manifests/sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac -sv
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "digest": "sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5",
    "size": 1261
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405",
      "size": 7732034
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312",
      "size": 20002230
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b",
      "size": 17686
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76",
      "size": 269
    }
  ]
}
< HTTP/1.1 200 OK
< content-length: 1092
< content-type: application/vnd.docker.distribution.manifest.v2+json
< docker-content-digest: sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac
< docker-distribution-api-version: registry/2.0
< etag: "sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac"
< date: Mon, 16 Sep 2024 07:37:49 GMT
< strict-transport-security: max-age=31536000
< docker-ratelimit-source: x.x.x.x

Now, the different parts of the image stored in content can be identified:

1
2
3
4
5
6
7
8
# ls /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/
61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 # manifest index
bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac # linux/amd64 platform manifest
4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5 # config
b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405 # layer 0
d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312 # layer 1
1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b # layer 2
c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 # layer 3

2.3.3 Config

The content of the config file:

1
2
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5
{"architecture":"amd64","config":{"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/kindnetd"],"WorkingDir":"/","ArgsEscaped":true,"OnBuild":null},"created":"2024-02-02T00:11:55.219470298Z","history":[{"created":"2023-04-06T00:03:08.397579916Z","created_by":"COPY / / # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2024-02-02T00:11:55.130996095Z","created_by":"COPY ./go/src/kindnetd /bin/kindnetd # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2024-02-02T00:11:55.184280737Z","created_by":"COPY /_LICENSES/* /LICENSES/ # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2024-02-02T00:11:55.219470298Z","created_by":"COPY files/LICENSES/* /LICENSES/* #

2.3.4 blob

The content of each layer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# file /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405
/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405: gzip compressed data, original size modulo 2^32 19633664
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405 | sha256sum
b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405  -
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312 | sha256sum
d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312  -
# file /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312
/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312: gzip compressed data, original size modulo 2^32 45286400
# file /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b
/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b: gzip compressed data, original size modulo 2^32 365056
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b |sha256sum
1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b  -
# file /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76
/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76: gzip compressed data, original size modulo 2^32 2560
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 | sha256sum
c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76  -

2.4 Content label

In order to find the corresponding file through the image, containerd maintains the relationship between the directory and the image. This is the metadata of content. The metadata of content contains the size of the mirror image content, creation time, and label.

The label of content stores the correlation relationship of various components of the image, the decompressed digest of the blob, and the corresponding relationship between the image and the snapshot. You can use ctr to view the label of each component of the image in content. This data is also read from the io.containerd.metadata.v1.bolt/meta.db file.

1
2
3
4
5
6
7
8
9
# ctr -n k8s.io content ls
DIGEST									SIZE	AGE		LABELS
sha256:61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988	685B	4 months	containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/gc.ref.content.m.0=sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac,containerd.io/gc.ref.content.m.1=sha256:fde0f6062db0a3b3323d76a4cde031f0f891b5b79d12be642b7e5aad68f2836f
sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac	1.092kB	4 months	containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/gc.ref.content.config=sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5,containerd.io/gc.ref.content.l.0=sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405,containerd.io/gc.ref.content.l.1=sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312,containerd.io/gc.ref.content.l.2=sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b,containerd.io/gc.ref.content.l.3=sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76
sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5	1.261kB	4 months	containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/gc.ref.snapshot.overlayfs=sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951
sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405	7.732MB	4 months	containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/uncompressed=sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19
sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312	20MB	4 months	containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/uncompressed=sha256:252acc7615fcb660d6ba94f6a47e47e6878c37545ba0b9bff9c5218320cdc86f
sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b	17.69kB	4 months	containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/uncompressed=sha256:3880c8fa363b674787663c7d18b91c38c0c57330850d986d816d2034ccc46728
sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76	269B	4 months	containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/uncompressed=sha256:dbdbd0938e462e6b62b1dd8d7539cea8db0fa1cc4864b01ca74bb35d8c7103e1

The image storage metadata in the io.containerd.metadata.v1.bolt/meta.db file is located at the path: v1 → k8s.io → content → blob.

content-blob-in-bolt-metadata — content blob info

2.4.1 Public Label

containerd.io/distribution.source.docker.io=kindest/kindnetd indicates that this content belongs to the kindest/kindnetd image on Docker Hub. This label is only added if the content was pulled from a registry. If the image was imported, it will not have this label. The format of the label is containerd.io.distribution.source.<registry>=[<repo/1>,<repo/2>]. If multiple images share the same digest, the image names are separated by commas, such as containerd.io/distribution.source.docker.io=kindest/kindnetd,public/kindnetd.

2.4.2 Manifest Index Label

containerd.io/gc.ref.content.m.<index> represents the digest of the manifest content for each platform. In this case, there are two platforms: linux/amd64 and linux/arm64.

1
2
containerd.io/gc.ref.content.m.0=sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac
containerd.io/gc.ref.content.m.1=sha256:fde0f6062db0a3b3323d76a4cde031f0f891b5b79d12be642b7e5aad68f2836f

2.4.3 Manifest Label

containerd.io/gc.ref.content.config represents the digest of the config content.

containerd.io/gc.ref.content.l.<index> represents the digest of each image layer.

The platform in this case is linux/amd64, so the labels for the manifest with digest sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac are as follows:

1
2
3
4
5
containerd.io/gc.ref.content.config=sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5
containerd.io/gc.ref.content.l.0=sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405
containerd.io/gc.ref.content.l.1=sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312
containerd.io/gc.ref.content.l.2=sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b
containerd.io/gc.ref.content.l.3=sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76

2.4.4 Config Label

containerd.io/gc.ref.snapshot.overlayfs represents the digest of the snapshot for the last image layer.

1
sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5	1.261kB	4 months	containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/gc.ref.snapshot.overlayfs=sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951

2.4.5 Layer Label

containerd.io/uncompressed represents the digest value after the content of the layer is decompressed. The digest is calculated with the command cat <file> | gunzip - | sha256sum.

For example, the final layer content sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 has the following label:

1
containerd.io/uncompressed=sha256:dbdbd0938e462e6b62b1dd8d7539cea8db0fa1cc4864b01ca74bb35d8c7103e1

This corresponds to the uncompressed digest.

1
2
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 | gunzip| sha256sum
dbdbd0938e462e6b62b1dd8d7539cea8db0fa1cc4864b01ca74bb35d8c7103e1  -

2.5 Snapshot

Since each image layer is compressed and immutable, it cannot be directly used by a container. The image must be mounted as a filesystem for the container to use it. This process is called “snapshotting,” and in containerd, each layer of the image is a snapshot, with each lower layer serving as the parent of the layer above. The default snapshotter used is overlayfs.

The snapshot’s digest is calculated by merging the data from the parent layer. Therefore, the first layer’s digest is the value of containerd.io/uncompressed, while the other layers’ snapshot digests will differ from the corresponding containerd.io/uncompressed values. The digest of the topmost layer’s snapshot is stored in the label containerd.io/gc.ref.snapshot.overlayfs in the config.

A snapshot’s kind can be Committed (immutable) or Active (read-write).

Since this image has four layers, there are also four snapshots. The bottom snapshot has the digest sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19, which matches the label value of containerd.io/uncompressed for the first image layer (sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405), and its parent is empty.

The topmost snapshot, sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951, is recorded in the config under the label containerd.io/gc.ref.snapshot.overlayfs.

1
2
3
4
5
6
# ctr -n k8s.io snapshot ls
KEY                                    PARENT                     KIND
sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 sha256:7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 Committed
sha256:7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 sha256:1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 Committed
sha256:1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 Committed
sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19                                                                         Committed 

The output of the ctr command is retrieved from the file io.containerd.metadata.v1.bolt/meta.db, under the path v1 → k8s.io → snapshots → overlayfs. It records information such as children, name, and parent.

snapshots-in-bolt-metadata — snapshots info in metadata

The value of name is used as the key in the path io.containerd.snapshotter.v1.overlayfs/metadata.db under snapshots.

snapshots-in-overlayfs-metadata — snapshots info in overlayfs metadata

The id here represents the snapshot’s path in the filesystem. The IDs for each layer snapshot are as follows:

1
2
3
4
98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 # 71
7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 # 70
1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 # 69
e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 # 68

1
2
3
4
5
6
7
8
# ls /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/71/fs/
LICENSES
# ls /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/70/fs/
LICENSES
# ls /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/69/fs/bin/
kindnetd
# ls /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/68/fs/
bin  boot  dev  etc  go-runner  home  lib  lib64  proc  root  run  sbin  sys  tmp  usr  var

3 Container Creation

When a container is created, an Active snapshot is generated, with its parent being the final image snapshot.

In this example, there are two containers, resulting in two snapshots. The snapshot name corresponds to the container ID, and both containers have sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 as their parent.

1
2
3
4
5
6
7
8
# ctr -n k8s.io snapshot ls
KEY                                                                     PARENT                                                                  KIND
95552d484ecc82490a233d1dce0fc9d6bb04472d4855edb16d563036d8f05965        sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 Active
17bb90bf08d8be9c4e4bf36dae6239bbe0322a735899ece4193b76fe9715a98f        sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 Active
sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 sha256:7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 Committed
sha256:7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 sha256:1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 Committed
sha256:1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 Committed
sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19                                                                         Committed

For the container 17bb90bf08d8be9c4e4bf36dae6239bbe0322a735899ece4193b76fe9715, a snapshot directory /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/174 was created as the upperdir and workdir for overlay.

1
2
3
4
5
# mount |grep 17bb90bf08d8be9c4e4bf36dae6239bbe0322a735899ece4193b76fe9715
overlay on /run/containerd/io.containerd.runtime.v2.task/k8s.io/17bb90bf08d8be9c4e4bf36dae6239bbe0322a735899ece4193b76fe9715a98f/rootfs type overlay (rw,relatime,lowe
rdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/71/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/70/fs:/var/lib/co
ntainerd/io.containerd.snapshotter.v1.overlayfs/snapshots/69/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/68/fs,upperdir=/var/lib/container
d/io.containerd.snapshotter.v1.overlayfs/snapshots/174/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/174/work)

4 Summary

The storage of containerd images includes the original image content, the decompressed image files, and related metadata.

The original image content (pulled or imported images) is stored in the io.containerd.content.v1.content directory.
The decompressed image content is stored in the io.containerd.snapshotter.v1.overlayfs directory (with overlay being the default storage driver on Linux).
Image-related metadata (including images, contents, and snapshots) is stored in the io.containerd.metadata.v1.bolt/meta.db file.
The metadata for each image layer snapshot (directory ID, kind, size, parent, etc.) is stored in the io.containerd.snapshotter.v1.overlayfs/metadata.db file.
When a container is created, a new snapshot is generated on top of the topmost image layer snapshot, which is used as the container’s filesystem.

5 Reference

重学容器09: Containerd是如何存储容器镜像和数据的

https://github.com/richlander/container-registry-api

metadata tree

Containerd 的 image 儲存機制到 content, snapshot 的概念介紹

containerd Content Flow

containerd Architecture

Docker Registry v2 get manifest and push\pull

Image Manifest V 2, Schema 2

How to get manifests using HTTP API v2?

Contents

How Containerd Stores Images?

1 Image Storage Process

1.1 Image Overview

1.2 containerd Image Storage Process

1.2.1 Saving the Original Image

1.2.2 Unpacking the Image and Creating Snapshots

2 image content in the disk

2.1 Storage Directory

2.2 Image Metadata

2.3 Image Content in the `content` Directory

2.3.1 Manifest Index

2.3.2 Manifest

2.3.3 Config

2.3.4 blob

2.4 Content label

2.4.1 Public Label

2.4.2 Manifest Index Label

2.4.3 Manifest Label

2.4.4 Config Label

2.4.5 Layer Label

2.5 Snapshot

3 Container Creation

4 Summary

5 Reference

Related Content

Contents

How Containerd Stores Images?

1 Image Storage Process

1.1 Image Overview

1.2 containerd Image Storage Process

1.2.1 Saving the Original Image

1.2.2 Unpacking the Image and Creating Snapshots

2 image content in the disk

2.1 Storage Directory

2.2 Image Metadata

2.3 Image Content in the content Directory

2.3.1 Manifest Index

2.3.2 Manifest

2.3.3 Config

2.3.4 blob

2.4 Content label

2.4.1 Public Label

2.4.2 Manifest Index Label

2.4.3 Manifest Label

2.4.4 Config Label

2.4.5 Layer Label

2.5 Snapshot

3 Container Creation

4 Summary

5 Reference

Related Content

How to Reduce Disk Space Used by Container Images? Practical Tips for Efficient Storage Configuration in containerd

Is cloud native infrastructure dead?

2.3 Image Content in the `content` Directory