xiaoqing blogContainerd是如何存储镜像？
当我们执行 nerdctl pull、crictl pull 或 nerdctl load 时，镜像是如何保存到系统中的呢？本文将介绍 containerd 如何储存镜像，帮助大家理解镜像在containerd中的存储机制。
本文基于containerd v1.7.15版本
1 镜像存储流程
1.1 镜像介绍
镜像由以下三部分组成：
- manifest：描述了镜像的元数据，用于帮助客户端或 container runtime 确认镜像的完整性和内容。
- config：包含了镜像构建时候的指令（比如DockerFile里的每一行指令），还包括镜像历史、运行时环境等信息。
- blob：是镜像每一层文件的打包，一般使用gzip进行压缩。
1.2 containerd镜像保存流程
“containerd 镜像保存流程分为两个步骤：保存原始镜像、解压镜像创建快照”
下图展示了 containerd 镜像存储流程中的各个关键步骤和文件存储位置，其中content store是containerd 的内容存储区，snapshots是快照存储区，快照简单理解为一个快照就是一层解压后的镜像。
1.2.1 保存原始镜像
将 manifest、config 和 blob 存储到
content store中，路径为：/var/lib/containerd/io.containerd.content.v1.content/blobs/同时将镜像的元数据（如 digest 和大小）和内容元数据（如标签、大小）存储到
meta.db文件中。该文件是基于 bbolt 数据库 的存储格式，路径为：/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db
1.2.2 镜像解压创建快照
将每层 blob 解压，并将其内容存储到
snapshots中。Snapshotter 负责管理镜像的分层文件系统，默认Snapshotter为overlayfs。文件存储路径为：为/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/将snapshots的元数据（parent、child、digest）保存到/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db和io.containerd.snapshotter.v1.overlayfs/metadata.db文件中。
容器生成时候会创建基于镜像的最上层的快照，作为容器的文件系统。
2 磁盘中的镜像内容
下面以kindest/kindnetd:v20240202-8f1494ea镜像为例，讲解它在linux环境里是如何保存的。
2.1 存储目录
containerd的存储目录默认为/var/lib/containerd，这个路径保存了content、cri、introspection、metadata、runtime、snapshotter模块的相关数据。
# tree /var/lib/containerd/ -L 2 -F
/var/lib/containerd//
|-- io.containerd.content.v1.content/
| |-- blobs/
| `-- ingest/
|-- io.containerd.grpc.v1.cri/
| |-- containers/
| `-- sandboxes/
|-- io.containerd.grpc.v1.introspection/
| `-- uuid
|-- io.containerd.metadata.v1.bolt/
| `-- meta.db
|-- io.containerd.runtime.v1.linux/
|-- io.containerd.runtime.v2.task/
| `-- k8s.io/
|-- io.containerd.snapshotter.v1.blockfile/
|-- io.containerd.snapshotter.v1.native/
| `-- snapshots/
|-- io.containerd.snapshotter.v1.overlayfs/
| |-- metadata.db
| `-- snapshots/
`-- tmpmounts/
18 directories, 3 files2.2 镜像元数据
无论是导入的镜像还是拉取镜像，最后原始镜像内容都保存在io.containerd.content.v1.content目录，镜像的元数据保存到io.containerd.metadata.v1.bolt/meta.db文件里。
ctr命令查看镜像列表，这个数据是从io.containerd.metadata.v1.bolt/meta.db文件读取出来的
# ctr -n k8s.io image ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/kindest/kindnetd:v20240202-8f1494ea application/vnd.docker.distribution.manifest.list.v2+json sha256:61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 26.5 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
docker.io/kindest/local-path-helper:v20230510-486859a6 application/vnd.docker.distribution.manifest.list.v2+json sha256:d8a6cc3b66ff253e3adb1ed3017bd1556301a76bcb3a0d2a4db3c8eeb7f0fc0a 2.9 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed使用boltbrowser打开io.containerd.metadata.v1.bolt/meta.db文件，在v1 →k8s.io →images 下面保存了节点上的镜像列表，每个镜像下面包含labels、digest、mediatype。kindest/kindnetd:v20240202-8f1494ea的digest为61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988
注意：需要先停止containerd进程，再打开io.containerd.metadata.v1.bolt/meta.db文件。因为只有读写模式只能有一个进程打开，containerd是以读写模式打开这个文件，详见issue 228。
2.3 content中镜像内容
镜像数据按照这个/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/{digest}路径进行保存。
2.3.1 manifest index
在/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 保存了镜像的manifests index内容。
如果镜像的Type是application/vnd.docker.distribution.manifest.list.v2+json，代表镜像支持多平台，镜像的digest是manifest index（多个manifest信息）。
如果Type为application/vnd.docker.distribution.manifest.v2+json，代表镜像只支持一个平台，镜像的digest是manifest。
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac",
"size": 1092,
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:fde0f6062db0a3b3323d76a4cde031f0f891b5b79d12be642b7e5aad68f2836f",
"size": 1092,
"platform": {
"architecture": "arm64",
"os": "linux"
}
}
]
}
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 | sha256sum
61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 -它的内容跟docker hub里获取的一样，同时61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988是HTTP响应header的docker-content-digest的值
# token=$(https_proxy=127.0.0.1:7890 curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:kindest/kindnetd:pull" | jq '.token' -r)
$ https_proxy=127.0.0.1:7890 curl -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Authorization: Bearer $token" https://registry-1.docker.io/v2/kindest/kindnetd/manifests/v20240202-8f1494ea -sv
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac",
"size": 1092,
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:fde0f6062db0a3b3323d76a4cde031f0f891b5b79d12be642b7e5aad68f2836f",
"size": 1092,
"platform": {
"architecture": "arm64",
"os": "linux"
}
}
]
}
< HTTP/1.1 200 OK
< content-length: 685
< content-type: application/vnd.docker.distribution.manifest.list.v2+json
< docker-content-digest: sha256:61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988
< docker-distribution-api-version: registry/2.0
< etag: "sha256:61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988"
< date: Mon, 16 Sep 2024 07:35:47 GMT
< strict-transport-security: max-age=31536000
< docker-ratelimit-source: x.x.x.x2.3.2 manifest
由于是linux/amd64平台，所以使用mainifest–bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/s
a256/bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5",
"size": 1261
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405",
"size": 7732034
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312",
"size": 20002230
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b",
"size": 17686
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76",
"size": 269
}
]
}
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac | sha256sum
bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac -这个跟从docker hub获取到的一样，HTTP响应的header的docker-content-digest值也是bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac
$ https_proxy=127.0.0.1:7890 curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer $token" https://registry-1.docker.io/v2/kindest/kindnetd/manifests/sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac -sv
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5",
"size": 1261
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405",
"size": 7732034
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312",
"size": 20002230
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b",
"size": 17686
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76",
"size": 269
}
]
}
< HTTP/1.1 200 OK
< content-length: 1092
< content-type: application/vnd.docker.distribution.manifest.v2+json
< docker-content-digest: sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac
< docker-distribution-api-version: registry/2.0
< etag: "sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac"
< date: Mon, 16 Sep 2024 07:37:49 GMT
< strict-transport-security: max-age=31536000
< docker-ratelimit-source: x.x.x.x现在可以推断出镜像在content存储中的各个部分
# ls /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/
61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 # mainifest index
bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac # linux/amd64 platform mainifest
4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5 # config
b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405 # layer 0
d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312 # layer 1
1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b # layer 2
c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 # layer 32.3.3 config
config文件内容
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5
{"architecture":"amd64","config":{"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/kindnetd"],"WorkingDir":"/","ArgsEscaped":true,"OnBuild":null},"created":"2024-02-02T00:11:55.219470298Z","history":[{"created":"2023-04-06T00:03:08.397579916Z","created_by":"COPY / / # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2024-02-02T00:11:55.130996095Z","created_by":"COPY ./go/src/kindnetd /bin/kindnetd # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2024-02-02T00:11:55.184280737Z","created_by":"COPY /_LICENSES/* /LICENSES/ # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2024-02-02T00:11:55.219470298Z","created_by":"COPY files/LICENSES/* /LICENSES/* # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2024-02-02T00:11:55.219470298Z","created_by":"CMD [\"/bin/kindnetd\"]","comment":"buildkit.dockerfile.v0","empty_layer":true}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19","sha256:252acc7615fcb660d6ba94f6a47e47e6878c37545ba0b9bff9c5218320cdc86f","sha256:3880c8fa363b674787663c7d18b91c38c0c57330850d986d816d2034ccc46728","sha256:dbdbd0938e462e6b62b1dd8d7539cea8db0fa1cc4864b01ca74bb35d8c7103e1"]}}
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5 |sha256sum
4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5 -2.3.4 blob
镜像各层的数据
# file /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405
/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405: gzip compressed data, original size modulo 2^32 19633664
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405 | sha256sum
b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405 -
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312 | sha256sum
d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312 -
# file /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312
/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312: gzip compressed data, original size modulo 2^32 45286400
# file /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b
/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b: gzip compressed data, original size modulo 2^32 365056
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b |sha256sum
1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b -
# file /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76
/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76: gzip compressed data, original size modulo 2^32 2560
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 | sha256sum
c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 -2.4 content label
为了能够通过image找到对应的文件，contianerd维护了目录与镜像对应的关系，这个就是content的元数据。content的元数据包含了镜像内容的大小、创建时间、label。content的label保存了镜像各个组件的关联关系、blob的解压后的digest、镜像与快照对应关系。
可以使用ctr查看content里镜像各个组件的label，这个数据也是从io.containerd.metadata.v1.bolt/meta.db文件里读取出来的
# ctr -n k8s.io content ls
DIGEST SIZE AGE LABELS
sha256:61f9956af8019caf6dcc4d39b31857b868aaab80521432ddcc216b805c4f7988 685B 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/gc.ref.content.m.0=sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac,containerd.io/gc.ref.content.m.1=sha256:fde0f6062db0a3b3323d76a4cde031f0f891b5b79d12be642b7e5aad68f2836f
sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac 1.092kB 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/gc.ref.content.config=sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5,containerd.io/gc.ref.content.l.0=sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405,containerd.io/gc.ref.content.l.1=sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312,containerd.io/gc.ref.content.l.2=sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b,containerd.io/gc.ref.content.l.3=sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76
sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5 1.261kB 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/gc.ref.snapshot.overlayfs=sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951
sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405 7.732MB 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/uncompressed=sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19
sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312 20MB 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/uncompressed=sha256:252acc7615fcb660d6ba94f6a47e47e6878c37545ba0b9bff9c5218320cdc86f
sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b 17.69kB 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/uncompressed=sha256:3880c8fa363b674787663c7d18b91c38c0c57330850d986d816d2034ccc46728
sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 269B 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/uncompressed=sha256:dbdbd0938e462e6b62b1dd8d7539cea8db0fa1cc4864b01ca74bb35d8c7103e1io.containerd.metadata.v1.bolt/meta.db文件里储存镜像元数据，路径是 v1 →k8s.io →content →blob
2.4.1 公共label
containerd.io/distribution.source.docker.io=kindest/kindnetd，代表了这个内容属于docker hub上的kindest/kindnetd镜像。只有通过拉取方式这个内容才有这个label，通过导入镜像方式是没有这个label。label的格式containerd.io.distribution.source.<registry>=[<repo/1>,<repo/2>]，如果多个镜像的digest一样，则镜像名使用逗号分隔，比如containerd.io/distribution.source.docker.io=kindest/kindnetd,public/kindnetd。
2.4.2 manifest index label
containerd.io/gc.ref.content.m.<index>，代表了每种运行平台的manifest内容的digest。这里有两个运行平台linux/amd64、linux/arm64。
containerd.io/gc.ref.content.m.0=sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac
containerd.io/gc.ref.content.m.1=sha256:fde0f6062db0a3b3323d76a4cde031f0f891b5b79d12be642b7e5aad68f2836f2.4.3 manifest label
containerd.io/gc.ref.content.config，代表了config内容的digest。
containerd.io/gc.ref.content.l.<index>，代表了镜像各层的内容的digest。
运行平台是linux/amd64，所以下面是sha256:bdddbe20c61d325166b48dd517059f5b93c21526eb74c5c80d86cd6d37236bac的manifest的label。
containerd.io/gc.ref.content.config=sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5
containerd.io/gc.ref.content.l.0=sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405
containerd.io/gc.ref.content.l.1=sha256:d3515376e10b73f645c97cbad9258641916af76ae518be17e163d3bc3bd64312
containerd.io/gc.ref.content.l.2=sha256:1fcc5fefb39fa5244e9673a2cad2f2d4dbc89cb0fbaacdca36c05a90e02af18b
containerd.io/gc.ref.content.l.3=sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e762.4.4 config label
containerd.io/gc.ref.snapshot.overlayfs，它代表镜像最后一层快照的digest。
sha256:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5 1.261kB 4 months containerd.io/distribution.source.docker.io=kindest/kindnetd,containerd.io/gc.ref.snapshot.overlayfs=sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b9512.4.5 layer label
containerd.io/uncompressed，代表这层内容解压后的digest值，digest计算方式cat <file> | gunzip - | sha256sum 。
比如最后一层内容sha256:c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76
containerd.io/uncompressed=sha256:dbdbd0938e462e6b62b1dd8d7539cea8db0fa1cc4864b01ca74bb35d8c7103e1对应的解压后的digest
# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/c2420332d0c7405cccc405b98ff4a1867bd103cbe571f03ac04b7e1cc7408e76 | gunzip| sha256sum
dbdbd0938e462e6b62b1dd8d7539cea8db0fa1cc4864b01ca74bb35d8c7103e1 -2.5 快照
由于镜像的每层数据都是压缩且不可修改，所以无法被container直接使用，需要将镜像映射成文件系统才能被容器所使用。将镜像映射成文件系统的方式叫做快照，在containerd里镜像的每一层都是一个快照，下面的层是上一层的parent。默认使用overlayfs作为snapshotters。
快照的digest是基于上一层数据合并后计算出的digest，所以第一层的digest为containerd.io/uncompressed的值，其他层快照的digest与该层的containerd.io/uncompressed的值是不一样。最后一层（最上面一层）快照的digest值会保存在config的label的containerd.io/gc.ref.snapshot.overlayfs里。
快照的kind为Committed代表不可修改，Active代表可以读写
由于这里镜像有四层，所以快照也有4个。最底下是第一层快照，它的digest为sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 ，这个值是第一层镜像sha256:b6425c1785a5de9285e14b515ddb6135b93ca5dd9edb744e1d1916a7a3687405的containerd.io/uncompressed的label值，而且它的parent为空。
最后一层快照sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 ，记录在config的containerd.io/gc.ref.snapshot.overlayfs的label里。
# ctr -n k8s.io snapshot ls
KEY PARENT KIND
sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 sha256:7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 Committed
sha256:7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 sha256:1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 Committed
sha256:1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 Committed
sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 Committed ctr命令结果是从io.containerd.metadata.v1.bolt/meta.db文件里读取出来的，路径是v1 →k8s.io →snapshots →overlayfs，它里面记录children、name、parent信息
这个name的值作为io.containerd.snapshotter.v1.overlayfs/metadata.db文件里snapshots路径下的key
这里面的id代表这个snapshot在文件系统的路径，上图由于boltbrowser使用ASCII码将数字转成字符显示。
各层快照对应的id
98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 # 71
7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 # 70
1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 # 69
e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 # 68# ls /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/71/fs/
LICENSES
# ls /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/70/fs/
LICENSES
# ls /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/69/fs/bin/
kindnetd
# ls /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/68/fs/
bin boot dev etc go-runner home lib lib64 proc root run sbin sys tmp usr var3 容器创建
容器生成时候会创建一个Active状态的快照，它的parent是镜像的最后一层快照。
这里我有两个容器，所以有两个快照，其中容器快照名字也是容器的id。这里两个容器的parent都是 sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951
# ctr -n k8s.io snapshot ls
KEY PARENT KIND
95552d484ecc82490a233d1dce0fc9d6bb04472d4855edb16d563036d8f05965 sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 Active
17bb90bf08d8be9c4e4bf36dae6239bbe0322a735899ece4193b76fe9715a98f sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 Active
sha256:98b2b8dceda4d797430d2160aff66d6ad2eaaee440072f10c3fbbc56a338b951 sha256:7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 Committed
sha256:7009f869326b4d361c87487dd5129546209e412dc3fc3726a3eaa59c488556c4 sha256:1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 Committed
sha256:1498834b7cb9b98acb24792ae65282e70875dd6753ca3ebf77f6cd7e127aae33 sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 Committed
sha256:e4a5933ff9603ec98b5df28cf7c07c7be52fc15146020cabafb94e0dbb844e19 Committed容器17bb90bf08d8be9c4e4bf36dae6239bbe0322a735899ece4193b76fe9715创建了一个快照目录/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/174，作为overlay的upperdir和workdir。
# mount |grep 17bb90bf08d8be9c4e4bf36dae6239bbe0322a735899ece4193b76fe9715
overlay on /run/containerd/io.containerd.runtime.v2.task/k8s.io/17bb90bf08d8be9c4e4bf36dae6239bbe0322a735899ece4193b76fe9715a98f/rootfs type overlay (rw,relatime,lowe
rdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/71/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/70/fs:/var/lib/co
ntainerd/io.containerd.snapshotter.v1.overlayfs/snapshots/69/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/68/fs,upperdir=/var/lib/container
d/io.containerd.snapshotter.v1.overlayfs/snapshots/174/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/174/work)4 总结
containerd镜像储存包括原始镜像内容和镜像文件解压后的内容和相关的元数据。
镜像的原始内容（拉取或导入的镜像）保存在io.containerd.content.v1.content目录。
镜像解压后保存在io.containerd.snapshotter.v1.overlayfs目录（linux下默认的存储插件是overlay）。
镜像相关的元数据（包括images和contents、snapshots）保存在io.containerd.metadata.v1.bolt/meta.db文件。
镜像每一层快照的元数据（目录id、Kind、大小、parent等）保存在io.containerd.snapshotter.v1.overlayfs/metadata.db文件。
容器生成时候会在镜像最上面一层快照之上，创建一个新的快照，作为容器的文件系统来使用。
5 Reference
重学容器09: Containerd是如何存储容器镜像和数据的
https://github.com/richlander/container-registry-api
Containerd 的 image 儲存機制到 content, snapshot 的概念介紹