xiaoqing blogHow Containerd Stores Images?
When we execute nerdctl pull, crictl pull, or nerdctl load, have you ever wondered how the images are stored in the system?
This article will introduce how containerd stores images to help you understand the image storage mechanism in containerd.
This article is based on containerd v1.7.15.
1 Image Storage Process
1.1 Image Overview
An image consists of the following three parts:
- manifest: Describes the metadata of the image, which helps the client or container runtime verify the integrity and contents of the image.
- config: Contains the instructions used during the image build process (for example, each line of a DockerFile), along with information such as the image’s history and runtime environment.
- blob: The compressed files for each layer of the image, usually packaged with gzip.
1.2 containerd Image Storage Process
The containerd image storage process consists of two steps: Saving the original image and Unpacking the image to create snapshots.
The diagram below shows the key steps and file storage locations in the containerd image storage process, where the content store is containerd’s content storage area, and snapshots represent the snapshot storage area. A snapshot can be understood as a decompressed layer of the image.
1.2.1 Saving the Original Image
First, the manifest, config, and blob are stored in the
content store, the default path is:/var/lib/containerd/io.containerd.content.v1.content/blobs/At the same time, the image metadata (digest, size) and content metadata (label, size) are stored in the
meta.dbfile. This file is in the format of a bbolt database, the path is:/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db
1.2.2 Unpacking the Image and Creating Snapshots
Each layer of the blob is unpacked, and its content is stored in the
snapshotsdirectory. The Snapshotter manages the image’s layered file system, with the default Snapshotter being overlayfs. The files are stored at:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/The metadata of the snapshots (parent, child, digest) is saved in both
/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.dbandio.containerd.snapshotter.v1.overlayfs/metadata.db.
When a container is generated, a snapshot is created based on the top layer of the image to serve as the container’s file system.
2 image content in the disk
Here, we take the kindest/kindnetd:v20240202-8f1494ea image as an example to explain how it is stored in a Linux environment.
2.1 Storage Directory
The default storage directory for containerd is /var/lib/containerd, which stores data related to the content, CRI, introspection, metadata, runtime, and snapshotter modules.
# tree /var/lib/containerd/ -L 2 -F
/var/lib/containerd//
|-- io.containerd.content.v1.content/
| |-- blobs/
| `-- ingest/
|-- io.containerd.grpc.v1.cri/
| |-- containers/
| `-- sandboxes/
|-- io.containerd.grpc.v1.introspection/
| `-- uuid
|-- io.containerd.metadata.v1.bolt/
| `-- meta.db
|-- io.containerd.runtime.v1.linux/
|-- io.containerd.runtime.v2.task/
| `-- k8s.io/
|-- io.containerd.snapshotter.v1.blockfile/
|-- io.containerd.snapshotter.v1.native/
| `-- snapshots/
|-- io.containerd.snapshotter.v1.overlayfs/
| |-- metadata.db
| `-- snapshots/
`-- tmpmounts/
18 directories, 3 files2.2 Image Metadata
Regardless of whether the image is imported or pulled, the raw image content is ultimately stored in the io.containerd.content.v1.content directory, while the image metadata is stored in the io.containerd.metadata.v1.bolt/meta.db file.
Using the ctr command to view the image list, this data is read from the io.containerd.metadata.v1.bolt/meta.db file:
# ctr -n k8s.io image ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/kindest/kindnetd:v20240202-8f1494ea application/vnd.docker.distribution.manifest.list.v2+json sha256:61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 26.5 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
docker.io/kindest/local-path-helper:v20230510-486859a6 application/vnd.docker.distribution.manifest.list.v2+json sha256:d8a6cc3b66ff253e3adb1ed3017bd1556301a76bcb3a0d2a4db3c8eeb7f0fc0a 2.9 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managedOpening the io.containerd.metadata.v1.bolt/meta.db file with boltbrowser, the images stored under the v1 → k8s.io → images path include labels, digest, and media type for each image. The digest for kindest/kindnetd:v20240202-8f1494ea is 61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988.
CAUTION
Be sure to stop the containerd process before opening the
io.containerd.metadata.v1.bolt/meta.dbfile, as only one process can access it in read-write mode. Since containerd opens the file in read-write mode, this is detailed in issue 228.
2.3 Image Content in the content Directory
The image data is stored in the /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/{digest} path.
2.3.1 Manifest Index
The manifest index for the image is stored at /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988.
NOTE
If the image’s type is
application/vnd.docker.distribution.manifest.list.v2+json, it indicates that the image supports multiple platforms, and its digest is the manifest index (containing multiple manifest entries).If the type is
application/vnd.docker.distribution.manifest.v2+json, the image supports a single platform, and its digest points to the manifest.
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac",
"size": 1092,
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:fde0f6062db0a3b3323d76a4cde031f0f891b5b79d12be642b7e5aad68f2836f",
"size": 1092,
"platform": {
"architecture": "arm64",
"os": "linux"
}
}
]
}
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 | sha256sum
61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 -The content is identical to what is retrieved from Docker Hub, and 61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 is the value of the docker-content-digest header in the HTTP response.
# token=$(https_proxy=127.0.0.1:7890 curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:kindest/kindnetd:pull" | jq '.token' -r)
# https_proxy=127.0.0.1:7890 curl -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Authorization: Bearer $token" https://registry-1.docker.io/v2/kindest/kindnetd/manifests/v20240202-8f1494ea -sv
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac",
"size": 1092,
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:fde0f6062db0a3b3323d76a4cde031f0f891b5b79d12be642b7e5aad68f2836f",
"size": 1092,
"platform": {
"architecture": "arm64",
"os": "linux"
}
}
]
}
< HTTP/1.1 200 OK
< content-length: 685
< content-type: application/vnd.docker.distribution.manifest.list.v2+json
< docker-content-digest: sha256:61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988
< docker-distribution-api-version: registry/2.0
< etag: "sha256:61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988"
< date: Mon, 16 Sep 2024 07:35:47 GMT
< strict-transport-security: max-age=31536000
< docker-ratelimit-source: x.x.x.x2.3.2 Manifest
Since it’s on the linux/amd64 platform, the manifest used is bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac.
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5",
"size": 1261
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405",
"size": 7732034
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312",
"size": 20002230
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b",
"size": 17686
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76",
"size": 269
}
]
}
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac | sha256sum
bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac -This is identical to what is retrieved from Docker Hub, as the docker-content-digest in the HTTP response header is also bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac.
$ https_proxy=127.0.0.1:7890 curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer $token" https://registry-1.docker.io/v2/kindest/kindnetd/manifests/sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac -sv
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5",
"size": 1261
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405",
"size": 7732034
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312",
"size": 20002230
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b",
"size": 17686
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76",
"size": 269
}
]
}
< HTTP/1.1 200 OK
< content-length: 1092
< content-type: application/vnd.docker.distribution.manifest.v2+json
< docker-content-digest: sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac
< docker-distribution-api-version: registry/2.0
< etag: "sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac"
< date: Mon, 16 Sep 2024 07:37:49 GMT
< strict-transport-security: max-age=31536000
< docker-ratelimit-source: x.x.x.xNow, the different parts of the image stored in content can be identified:
# ls /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/
61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 # manifest index
bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac # linux/amd64 platform manifest
4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5 # config
b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405 # layer 0
d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312 # layer 1
1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b # layer 2
c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 # layer 32.3.3 Config
The content of the config file:
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5
{"architecture":"amd64","config":{"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/kindnetd"],"WorkingDir":"/","ArgsEscaped":true,"OnBuild":null},"created":"2024-02-02T00:11:55.219470298Z","history":[{"created":"2023-04-06T00:03:08.397579916Z","created_by":"COPY / / # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2024-02-02T00:11:55.130996095Z","created_by":"COPY ./go/src/kindnetd /bin/kindnetd # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2024-02-02T00:11:55.184280737Z","created_by":"COPY /_LICENSES/* /LICENSES/ # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2024-02-02T00:11:55.219470298Z","created_by":"COPY files/LICENSES/* /LICENSES/* #2.3.4 blob
The content of each layer:
# file /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405
/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405: gzip compressed data, original size modulo 2^32 19633664
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405 | sha256sum
b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405 -
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312 | sha256sum
d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312 -
# file /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312
/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312: gzip compressed data, original size modulo 2^32 45286400
# file /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b
/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b: gzip compressed data, original size modulo 2^32 365056
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b |sha256sum
1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b -
# file /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76
/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76: gzip compressed data, original size modulo 2^32 2560
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 | sha256sum
c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 -2.4 Content label
In order to find the corresponding file through the image, containerd maintains the relationship between the directory and the image. This is the metadata of content. The metadata of content contains the size of the mirror image content, creation time, and label.
The label of content stores the correlation relationship of various components of the image, the decompressed digest of the blob, and the corresponding relationship between the image and the snapshot. You can use ctr to view the label of each component of the image in content. This data is also read from the io.containerd.metadata.v1.bolt/meta.db file.
# ctr -n k8s.io content ls
DIGEST SIZE AGE LABELS
sha256:61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 685B 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/gc.ref.content.m.0=sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac,containerd.io/gc.ref.content.m.1=sha256:fde0f6062db0a3b3323d76a4cde031f0f891b5b79d12be642b7e5aad68f2836f
sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac 1.092kB 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/gc.ref.content.config=sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5,containerd.io/gc.ref.content.l.0=sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405,containerd.io/gc.ref.content.l.1=sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312,containerd.io/gc.ref.content.l.2=sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b,containerd.io/gc.ref.content.l.3=sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76
sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5 1.261kB 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/gc.ref.snapshot.overlayfs=sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951
sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405 7.732MB 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/uncompressed=sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19
sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312 20MB 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/uncompressed=sha256:252acc7615fcb660d6ba94f6a47e47e6878c37545ba0b9bff9c5218320cdc86f
sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b 17.69kB 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/uncompressed=sha256:3880c8fa363b674787663c7d18b91c38c0c57330850d986d816d2034ccc46728
sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 269B 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/uncompressed=sha256:dbdbd0938e462e6b62b1dd8d7539cea8db0fa1cc4864b01ca74bb35d8c7103e1The image storage metadata in the io.containerd.metadata.v1.bolt/meta.db file is located at the path: v1 → k8s.io → content → blob.
2.4.1 Public Label
containerd.io/distribution.source.docker.io=kindest/kindnetd indicates that this content belongs to the kindest/kindnetd image on Docker Hub. This label is only added if the content was pulled from a registry. If the image was imported, it will not have this label. The format of the label is containerd.io.distribution.source.<registry>=[<repo/1>,<repo/2>]. If multiple images share the same digest, the image names are separated by commas, such as containerd.io/distribution.source.docker.io=kindest/kindnetd,public/kindnetd.
2.4.2 Manifest Index Label
containerd.io/gc.ref.content.m.<index> represents the digest of the manifest content for each platform. In this case, there are two platforms: linux/amd64 and linux/arm64.
containerd.io/gc.ref.content.m.0=sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac
containerd.io/gc.ref.content.m.1=sha256:fde0f6062db0a3b3323d76a4cde031f0f891b5b79d12be642b7e5aad68f2836f2.4.3 Manifest Label
containerd.io/gc.ref.content.config represents the digest of the config content.
containerd.io/gc.ref.content.l.<index> represents the digest of each image layer.
The platform in this case is linux/amd64, so the labels for the manifest with digest sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac are as follows:
containerd.io/gc.ref.content.config=sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5
containerd.io/gc.ref.content.l.0=sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405
containerd.io/gc.ref.content.l.1=sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312
containerd.io/gc.ref.content.l.2=sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b
containerd.io/gc.ref.content.l.3=sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e762.4.4 Config Label
containerd.io/gc.ref.snapshot.overlayfs represents the digest of the snapshot for the last image layer.
sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5 1.261kB 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/gc.ref.snapshot.overlayfs=sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b9512.4.5 Layer Label
containerd.io/uncompressed represents the digest value after the content of the layer is decompressed. The digest is calculated with the command cat <file> | gunzip - | sha256sum.
For example, the final layer content sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 has the following label:
containerd.io/uncompressed=sha256:dbdbd0938e462e6b62b1dd8d7539cea8db0fa1cc4864b01ca74bb35d8c7103e1This corresponds to the uncompressed digest.
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 | gunzip| sha256sum
dbdbd0938e462e6b62b1dd8d7539cea8db0fa1cc4864b01ca74bb35d8c7103e1 -2.5 Snapshot
Since each image layer is compressed and immutable, it cannot be directly used by a container. The image must be mounted as a filesystem for the container to use it. This process is called “snapshotting,” and in containerd, each layer of the image is a snapshot, with each lower layer serving as the parent of the layer above. The default snapshotter used is overlayfs.
The snapshot’s digest is calculated by merging the data from the parent layer. Therefore, the first layer’s digest is the value of containerd.io/uncompressed, while the other layers’ snapshot digests will differ from the corresponding containerd.io/uncompressed values. The digest of the topmost layer’s snapshot is stored in the label containerd.io/gc.ref.snapshot.overlayfs in the config.
A snapshot’s kind can be Committed (immutable) or Active (read-write).
Since this image has four layers, there are also four snapshots. The bottom snapshot has the digest sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19, which matches the label value of containerd.io/uncompressed for the first image layer (sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405), and its parent is empty.
The topmost snapshot, sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951, is recorded in the config under the label containerd.io/gc.ref.snapshot.overlayfs.
# ctr -n k8s.io snapshot ls
KEY PARENT KIND
sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 sha256:7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 Committed
sha256:7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 sha256:1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 Committed
sha256:1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 Committed
sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 Committed The output of the ctr command is retrieved from the file io.containerd.metadata.v1.bolt/meta.db, under the path v1 → k8s.io → snapshots → overlayfs. It records information such as children, name, and parent.
The value of name is used as the key in the path io.containerd.snapshotter.v1.overlayfs/metadata.db under snapshots.
The id here represents the snapshot’s path in the filesystem. The IDs for each layer snapshot are as follows:
98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 # 71
7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 # 70
1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 # 69
e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 # 68# ls /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/71/fs/
LICENSES
# ls /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/70/fs/
LICENSES
# ls /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/69/fs/bin/
kindnetd
# ls /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/68/fs/
bin boot dev etc go-runner home lib lib64 proc root run sbin sys tmp usr var3 Container Creation
When a container is created, an Active snapshot is generated, with its parent being the final image snapshot.
In this example, there are two containers, resulting in two snapshots. The snapshot name corresponds to the container ID, and both containers have sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 as their parent.
# ctr -n k8s.io snapshot ls
KEY PARENT KIND
95552d484ecc82490a233d1dce0fc9d6bb04472d4855edb16d563036d8f05965 sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 Active
17bb90bf08d8be9c4e4bf36dae6239bbe0322a735899ece4193b76fe9715a98f sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 Active
sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 sha256:7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 Committed
sha256:7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 sha256:1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 Committed
sha256:1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 Committed
sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 CommittedFor the container 17bb90bf08d8be9c4e4bf36dae6239bbe0322a735899ece4193b76fe9715, a snapshot directory /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/174 was created as the upperdir and workdir for overlay.
# mount |grep 17bb90bf08d8be9c4e4bf36dae6239bbe0322a735899ece4193b76fe9715
overlay on /run/containerd/io.containerd.runtime.v2.task/k8s.io/17bb90bf08d8be9c4e4bf36dae6239bbe0322a735899ece4193b76fe9715a98f/rootfs type overlay (rw,relatime,lowe
rdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/71/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/70/fs:/var/lib/co
ntainerd/io.containerd.snapshotter.v1.overlayfs/snapshots/69/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/68/fs,upperdir=/var/lib/container
d/io.containerd.snapshotter.v1.overlayfs/snapshots/174/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/174/work)4 Summary
The storage of containerd images includes the original image content, the decompressed image files, and related metadata.
- The original image content (pulled or imported images) is stored in the
io.containerd.content.v1.contentdirectory. - The decompressed image content is stored in the
io.containerd.snapshotter.v1.overlayfsdirectory (with overlay being the default storage driver on Linux). - Image-related metadata (including images, contents, and snapshots) is stored in the
io.containerd.metadata.v1.bolt/meta.dbfile. - The metadata for each image layer snapshot (directory ID, kind, size, parent, etc.) is stored in the
io.containerd.snapshotter.v1.overlayfs/metadata.dbfile. - When a container is created, a new snapshot is generated on top of the topmost image layer snapshot, which is used as the container’s filesystem.
5 Reference
重学容器09: Containerd是如何存储容器镜像和数据的
https://github.com/richlander/container-registry-api
Containerd 的 image 儲存機制到 content, snapshot 的概念介紹