各种折腾

我的 homelab(5): nerdctl：docker 的升级版 / 镜像拉取缓存和加速

摘要

本文讲述了博主使用 nerdctl，通过添加 containerd 的配置文件，和魔改 harbor 的配置文件，达到了一个 harbor 缓存加速所有镜像仓库的故事。

目标

首先，还是要重申一下我的目标。

因为准备折腾 k8s，中间可能会重装无数次，而每次重装和配置应用都需要拉取镜像，所以需要一种方便的镜像缓存的方案，节约时间和国际流量。

背景

在前一天的折腾中，我们认识到 docker 仅支持给 dockerhub 添加镜像。而我们需要的很多镜像并不在 dockerhub 上，所以我们只能舍弃 docker 的方案，寻求他路。

隆重介绍：nerdctl

nerdctl

contaiNERD CTL - Docker-compatible CLI for containerd, with support for Compose, Rootless, eStargz, OCIcrypt, IPFS, …

通过官方的介绍，我们了解到 nerdctl 是一个和 docker 的 ui/ux 兼容的，containerd 的命令行工具。

简单的来说，nerdctl 提供了几乎和 docker 命令一模一样的界面来直接操作 contianerd。和 ctr 以及 crictl 命令相比，nerdctl 支持更多的功能。和 docker 命令相比，nerdctl 同样支持构建镜像，同样支持配置网络，还添加了很多新的功能。虽然官方没有这个说法，但我认为 nerdctl 是 docker 更好的替代。

最最重要的是，我发现 nerdctl 会去读取/etc/containerd/certs.d/docker.io/hosts.toml 里面的配置。也就是说，他所有源都支持走镜像！支持走镜像！支持走镜像！

又因为，nerdctl 提供了和 docker 基本兼容的命令行，那，我只要把 docker 替换成 nerdctl，那不就万事大吉了。😁

不过总感觉开发者在骂人。

安装

release 有两种版本，一种是只含 nerdctl 命令行的，需要自己手动安装 containerd 以及其他组件。另一种 full版本，是包括了依赖组件的。我这里使用full 版本安装。

目前似乎还不支持包管理器安装，我写了一段简单的命令来安装：

1
2
3
4

wget https://github.com/containerd/nerdctl/releases/download/v1.7.6/nerdctl-full-1.7.6-linux-amd64.tar.gz -O /root/nerdctl.tar.gz
tar -xvf /root/nerdctl.tar.gz -C /usr/local/
rm /root/nerdctl.tar.gz
systemctl enable --now containerd

其实就是下载下来之后解压到 /usr/local，然后再启用服务就行了。

配置镜像

具体可以参考 这篇文章，

1
2
3
4
5
6
7

mkdir -p /etc/containerd/certs.d/docker.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://cr.suyiiyii.top"]
  capabilities = ["pull", "resolve"]

EOF

然后我们尝试拉取，我是用 --debug 选项查看详细信息，可以看到顺利的从我们指定的服务器上面拉取了镜像。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

root@ubuntu-99:~# nerdctl --debug pull nginx
DEBU[0000] verifying process skipped
DEBU[0000] Found hosts dir "/etc/containerd/certs.d"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs".
DEBU[0000] fetching                                      image="docker.io/library/nginx:latest"
DEBU[0000] loading host directory                        dir=/etc/containerd/certs.d/docker.io
DEBU[0000] resolving                                     host=cr.suyiiyii.top
DEBU[0000] do request                                    host=cr.suyiiyii.top request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=HEAD url="https://cr.suyiiyii.top/v2/library/nginx/manifests/latest?ns=docker.io"
DEBU[0000] fetch response received                       host=cr.suyiiyii.top response.header.connection=keep-alive response.header.content-length=152 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 20 Jun 2024 13:18:43 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx response.header.set-cookie="sid=c439d809f23d19d9ae22429d816b251b; Path=/; HttpOnly" response.header.www-authenticate="Bearer realm=\"https://cr.suyiiyii.top/service/token\",service=\"harbor-registry\",scope=\"repository:library/nginx:pull\"" response.header.x-request-id=e14f136a-81a7-41dc-9c03-db84b7f8c1ba response.status="401 Unauthorized" url="https://cr.suyiiyii.top/v2/library/nginx/manifests/latest?ns=docker.io"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://cr.suyiiyii.top/service/token\",service=\"harbor-registry\",scope=\"repository:library/nginx:pull\"" host=cr.suyiiyii.top
DEBU[0000] do request                                    host=cr.suyiiyii.top request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=HEAD url="https://cr.suyiiyii.top/v2/library/nginx/manifests/latest?ns=docker.io"
DEBU[0000] fetch response received                       host=cr.suyiiyii.top response.header.connection=keep-alive response.header.content-length=1778 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Thu, 20 Jun 2024 13:18:43 GMT" response.header.docker-content-digest="sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498\"" response.header.server=nginx response.header.set-cookie="sid=a7f8d23eebefe5589a90da2d2d547be6; Path=/; HttpOnly" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.header.x-request-id=c58fbc9e-7dbf-417a-a868-f39d31677f2c response.status="200 OK" url="https://cr.suyiiyii.top/v2/library/nginx/manifests/latest?ns=docker.io"
DEBU[0000] resolved                                      desc.digest="sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498" host=cr.suyiiyii.top
DEBU[0000] loading host directory                        dir=/etc/containerd/certs.d/docker.io
DEBU[0000] fetch                                         digest="sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1778
DEBU[0000] do request                                    digest="sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=GET size=1778 url="https://cr.suyiiyii.top/v2/library/nginx/manifests/sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498?ns=docker.io"
DEBU[0000] fetch response received                       digest="sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.connection=keep-alive response.header.content-length=152 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 20 Jun 2024 13:18:43 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx response.header.set-cookie="sid=dfa854d66d85b03c169b9c18bcd8b529; Path=/; HttpOnly" response.header.www-authenticate="Bearer realm=\"https://cr.suyiiyii.top/service/token\",service=\"harbor-registry\",scope=\"repository:library/nginx:pull\"" response.header.x-request-id=7a7f431f-87b6-449a-aaba-73c9e591da0e response.status="401 Unauthorized" size=1778 url="https://cr.suyiiyii.top/v2/library/nginx/manifests/sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498?ns=docker.io"
DEBU[0000] Unauthorized                                  digest="sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498" header="Bearer realm=\"https://cr.suyiiyii.top/service/token\",service=\"harbor-registry\",scope=\"repository:library/nginx:pull\"" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1778
DEBU[0000] do request                                    digest="sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=GET size=1778 url="https://cr.suyiiyii.top/v2/library/nginx/manifests/sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498?ns=docker.io"
DEBU[0000] fetch response received                       digest="sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.connection=keep-alive response.header.content-length=1778 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Thu, 20 Jun 2024 13:18:43 GMT" response.header.docker-content-digest="sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498\"" response.header.server=nginx response.header.set-cookie="sid=4a936e040c1dea8883bec90701358115; Path=/; HttpOnly" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.header.x-request-id=7df0824c-fbb2-4e5e-86d9-46d2e519c71e response.status="200 OK" size=1778 url="https://cr.suyiiyii.top/v2/library/nginx/manifests/sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498?ns=docker.io"
DEBU[0000] fetch                                         digest="sha256:dde0cca083bc75a0af14262b1469b5141284b4399a62fef923ec0c0e3b21f5bc" mediatype=application/vnd.docker.container.image.v1+json size=7155
DEBU[0000] do request                                    digest="sha256:dde0cca083bc75a0af14262b1469b5141284b4399a62fef923ec0c0e3b21f5bc" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=GET size=7155 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:dde0cca083bc75a0af14262b1469b5141284b4399a62fef923ec0c0e3b21f5bc?ns=docker.io"
DEBU[0000] fetch response received                       digest="sha256:dde0cca083bc75a0af14262b1469b5141284b4399a62fef923ec0c0e3b21f5bc" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=7155 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/octet-stream response.header.date="Thu, 20 Jun 2024 13:18:43 GMT" response.header.docker-content-digest="sha256:dde0cca083bc75a0af14262b1469b5141284b4399a62fef923ec0c0e3b21f5bc" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:dde0cca083bc75a0af14262b1469b5141284b4399a62fef923ec0c0e3b21f5bc\"" response.header.server=nginx response.header.set-cookie="sid=ba1f78c0cc5466d6923532e54129ba6b; Path=/; HttpOnly" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.header.x-request-id=57a6d4ef-c52c-42f3-9d52-a93a2a1758ac response.status="200 OK" size=7155 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:dde0cca083bc75a0af14262b1469b5141284b4399a62fef923ec0c0e3b21f5bc?ns=docker.io"
docker.io/library/nginx:latest:                                                   resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498: downloading    |--------------------------------------|    0.0 B/1.7 KiB
elapsed: 0.1 s                                                                    total:   0.0 B (0.0 B/s)
DEBU[0000] fetch                                         digest="sha256:ca6fb48c6db48342a3905bf65037e97543080a052a5f169b4b40b8c83b850f41" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1404
DEBU[0000] fetch                                         digest="sha256:9571e65a55a3fd4ccd461b4fbaf5e8e38242317add94cb088268b70d6d7d08b2" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=627
DEBU[0000] fetch                                         digest="sha256:928cc9acedf0354de565f85d9df9d519e44a29a585d6c19a37a8aeb02e25212c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1214
DEBU[0000] fetch                                         digest="sha256:2cc3ae149d28a36d28d4eefbae70aaa14a0c9eab588c3790f7979f310b893c44" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=29150430
DEBU[0000] fetch                                         digest="sha256:0b432cb2d95eea3d638db7e7cfb51eb7d7828f87c31d7a8c40ac5bb0278ca118" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=959
DEBU[0000] fetch                                         digest="sha256:24436676f2decbc5ed11c2e5786faa3dd103bc0fc738a2033b2f1aaab57226ad" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=398
DEBU[0000] fetch                                         digest="sha256:a97f9034bc9b7e813d93db97482046e20f581e1a80ddeda9b331c3ec6ed1cd8b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=41829706
DEBU[0000] do request                                    digest="sha256:2cc3ae149d28a36d28d4eefbae70aaa14a0c9eab588c3790f7979f310b893c44" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=GET size=29150430 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:2cc3ae149d28a36d28d4eefbae70aaa14a0c9eab588c3790f7979f310b893c44?ns=docker.io"
DEBU[0000] do request                                    digest="sha256:9571e65a55a3fd4ccd461b4fbaf5e8e38242317add94cb088268b70d6d7d08b2" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=GET size=627 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:9571e65a55a3fd4ccd461b4fbaf5e8e38242317add94cb088268b70d6d7d08b2?ns=docker.io"
DEBU[0000] do request                                    digest="sha256:928cc9acedf0354de565f85d9df9d519e44a29a585d6c19a37a8aeb02e25212c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=GET size=1214 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:928cc9acedf0354de565f85d9df9d519e44a29a585d6c19a37a8aeb02e25212c?ns=docker.io"
DEBU[0000] do request                                    digest="sha256:0b432cb2d95eea3d638db7e7cfb51eb7d7828f87c31d7a8c40ac5bb0278ca118" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=GET size=959 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:0b432cb2d95eea3d638db7e7cfb51eb7d7828f87c31d7a8c40ac5bb0278ca118?ns=docker.io"
DEBU[0000] fetch response received                       digest="sha256:2cc3ae149d28a36d28d4eefbae70aaa14a0c9eab588c3790f7979f310b893c44" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=29150430 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/octet-stream response.header.date="Thu, 20 Jun 2024 13:18:43 GMT" response.header.docker-content-digest="sha256:2cc3ae149d28a36d28d4eefbae70aaa14a0c9eab588c3790f7979f310b893c44" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:2cc3ae149d28a36d28d4eefbae70aaa14a0c9eab588c3790f7979f310b893c44\"" response.header.server=nginx response.header.set-cookie="sid=a3cc96d2663a9d61d818186cce0bf863; Path=/; HttpOnly" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.header.x-request-id=54897a2d-6a2c-4e63-973d-b7610c1869d6 response.status="200 OK" size=29150430 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:2cc3ae149d28a36d28d4eefbae70aaa14a0c9eab588c3790f7979f310b893c44?ns=docker.io"
DEBU[0000] do request                                    digest="sha256:a97f9034bc9b7e813d93db97482046e20f581e1a80ddeda9b331c3ec6ed1cd8b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=GET size=41829706 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:a97f9034bc9b7e813d93db97482046e20f581e1a80ddeda9b331c3ec6ed1cd8b?ns=docker.io"
DEBU[0000] fetch response received                       digest="sha256:9571e65a55a3fd4ccd461b4fbaf5e8e38242317add94cb088268b70d6d7d08b2" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=627 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/octet-stream response.header.date="Thu, 20 Jun 2024 13:18:43 GMT" response.header.docker-content-digest="sha256:9571e65a55a3fd4ccd461b4fbaf5e8e38242317add94cb088268b70d6d7d08b2" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:9571e65a55a3fd4ccd461b4fbaf5e8e38242317add94cb088268b70d6d7d08b2\"" response.header.server=nginx response.header.set-cookie="sid=24242acfa26a11aaed65b80b8fdc051a; Path=/; HttpOnly" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.header.x-request-id=0fa63c04-8672-4199-bb7a-0a6456e61cf8 response.status="200 OK" size=627 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:9571e65a55a3fd4ccd461b4fbaf5e8e38242317add94cb088268b70d6d7d08b2?ns=docker.io"
DEBU[0000] do request                                    digest="sha256:24436676f2decbc5ed11c2e5786faa3dd103bc0fc738a2033b2f1aaab57226ad" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=GET size=398 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:24436676f2decbc5ed11c2e5786faa3dd103bc0fc738a2033b2f1aaab57226ad?ns=docker.io"
DEBU[0000] fetch response received                       digest="sha256:928cc9acedf0354de565f85d9df9d519e44a29a585d6c19a37a8aeb02e25212c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=1214 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/octet-stream response.header.date="Thu, 20 Jun 2024 13:18:43 GMT" response.header.docker-content-digest="sha256:928cc9acedf0354de565f85d9df9d519e44a29a585d6c19a37a8aeb02e25212c" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:928cc9acedf0354de565f85d9df9d519e44a29a585d6c19a37a8aeb02e25212c\"" response.header.server=nginx response.header.set-cookie="sid=c41bf3d54deb044f937f4e384b0cc3a2; Path=/; HttpOnly" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.header.x-request-id=b2a1d490-343b-45a7-bd24-67ce824c28f4 response.status="200 OK" size=1214 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:928cc9acedf0354de565f85d9df9d519e44a29a585d6c19a37a8aeb02e25212c?ns=docker.io"
DEBU[0000] do request                                    digest="sha256:ca6fb48c6db48342a3905bf65037e97543080a052a5f169b4b40b8c83b850f41" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=GET size=1404 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:ca6fb48c6db48342a3905bf65037e97543080a052a5f169b4b40b8c83b850f41?ns=docker.io"
DEBU[0000] fetch response received                       digest="sha256:a97f9034bc9b7e813d93db97482046e20f581e1a80ddeda9b331c3ec6ed1cd8b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=41829706 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/octet-stream response.header.date="Thu, 20 Jun 2024 13:18:43 GMT" response.header.docker-content-digest="sha256:a97f9034bc9b7e813d93db97482046e20f581e1a80ddeda9b331c3ec6ed1cd8b" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:a97f9034bc9b7e813d93db97482046e20f581e1a80ddeda9b331c3ec6ed1cd8b\"" response.header.server=nginx response.header.set-cookie="sid=03aceb1b643cc0dd82b9b1012b1a852c; Path=/; HttpOnly" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.header.x-request-id=e0a8f09d-b700-4a91-9caf-ec46b238d52a response.status="200 OK" size=41829706 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:a97f9034bc9b7e813d93db97482046e20f581e1a80ddeda9b331c3ec6ed1cd8b?ns=docker.io"
DEBU[0000] fetch response received                       digest="sha256:0b432cb2d95eea3d638db7e7cfb51eb7d7828f87c31d7a8c40ac5bb0278ca118" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=959 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/octet-stream response.header.date="Thu, 20 Jun 2024 13:18:43 GMT" response.header.docker-content-digest="sha256:0b432cb2d95eea3d638db7e7cfb51eb7d7828f87c31d7a8c40ac5bb0278ca118" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:0b432cb2d95eea3d638db7e7cfb51eb7d7828f87c31d7a8c40ac5bb0278ca118\"" response.header.server=nginx response.header.set-cookie="sid=acae287452dee1fa6b945cfc234a3168; Path=/; HttpOnly" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.header.x-request-id=69542e57-1da5-45bd-8c2e-ddce6fa80776 response.status="200 OK" size=959 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:0b432cb2d95eea3d638db7e7cfb51eb7d7828f87c31d7a8c40ac5bb0278ca118?ns=docker.io"
DEBU[0000] fetch response received                       digest="sha256:24436676f2decbc5ed11c2e5786faa3dd103bc0fc738a2033b2f1aaab57226ad" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=398 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/octet-stream response.header.date="Thu, 20 Jun 2024 13:18:43 GMT" response.header.docker-content-digest="sha256:24436676f2decbc5ed11c2e5786faa3dd103bc0fc738a2033b2f1aaab57226ad" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:24436676f2decbc5ed11c2e5786faa3dd103bc0fc738a2033b2f1aaab57226ad\"" response.header.server=nginx response.header.set-cookie="sid=1ebc9b8ec35b775e5d1b36dd26a0c791; Path=/; HttpOnly" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.header.x-request-id=8aff8fd7-b288-4944-b5cd-0bca1a623557 response.status="200 OK" size=398 url="https://cr.suyiiyii.top/v2/library/nginx/blobs/sha256:24436676f2decbc5ed11c2e5786faa3dd103bc0fc738a2033b2f1aaab57226ad?ns=docker.io"
DEBU[0000] fetch response received                       digest="sha256:ca6fb48c6db48342a3905bf65037e97543080a052a5f169b4b40b8c83b850f41" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=1404 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/octet-stream response.header.date="Thu, 20 Jun 2024 13:18:43 GMT" response.header.docker-content-digest="sha256:ca6fb48c6db48342a3905bf65037e97543080a052a5f169b4b40b8c83b850f41" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:ca6fb48docker.io/library/nginx:latest:                                                   resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/nginx:latest:                                                   resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/nginx:latest:                                                   resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:dde0cca083bc75a0af14262b1469b5141284b4399a62fef923ec0c0e3b21f5bc:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:ca6fb48c6db48342a3905bf65037e97543080a052a5f169b4b40b8c83b850f41:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9571e65a55a3fd4ccd461b4fbaf5e8e38242317add94cb088268b70d6d7d08b2:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0b432cb2d95eea3d638db7e7cfb51eb7d7828f87c31d7a8c40ac5bb0278ca118:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:24436676f2decbc5ed11c2e5786faa3dd103bc0fc738a2033b2f1aaab57226ad:    done           |++++++++++++++++++++++++++++++++++++++|
docker.io/library/nginx:latest:                                                   resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:dde0cca083bc75a0af14262b1469b5141284b4399a62fef923ec0c0e3b21f5bc:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:ca6fb48c6db48342a3905bf65037e97543080a052a5f169b4b40b8c83b850f41:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9571e65a55a3fd4ccd461b4fbaf5e8e38242317add94cb088268b70d6d7d08b2:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0b432cb2d95eea3d638db7e7cfb51eb7d7828f87c31d7a8c40ac5bb0278ca118:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:24436676f2decbc5ed11c2e5786faa3dd103bc0fc738a2033b2f1aaab57226ad:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:928cc9acedf0354de565f85d9df9d519e44a29a585d6c19a37a8aeb02e25212c:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:2cc3ae149d28a36d28d4eefbae70aaa14a0c9eab588c3790f7979f310b893c44:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:a97f9034bc9b7e813d93db97482046e20f581e1a80ddeda9b331c3ec6ed1cd8b:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 3.2 s                                                                    total:  39.9 M (12.5 MiB/s)

harbor

创建拉取缓存仓库

先直接给结论吧，harbor 这边需要手动添加你要缓存的仓库（harbor 可能需要魔法）。

提供者全部选择 Docker Registry，因为我们没有其他平台的账号，所以全部作为普通的 registry 匿名拉取。

然后目标 url 就是你要缓存的仓库。

Untitled.png

然后去新建项目，填的信息都差不多，把刚刚镜像的仓库公开出去。

需要设定一个项目名，也就是请求的路径，不然 harbor 没办法区分到底要返回哪一个镜像仓库，最好和仓库地址一样。

Untitled.png

但是，目前 harbor 只支持前缀的代理形式，也就是这样：

1

docker pull <harbor_server_name>/<proxy_project_name>/goharbor/harbor-core:dev

那就又要手动替换 tag 了，这不是我期望的。

配置镜像

不管了，我们先试一下拉取镜像

1

nerdctl pull cr.suyiiyii.top/registry.k8s.io/kube-apiserver:v1.30.0

如果是我期望的样子的话，那拉取的命令应该是这样子的

1

nerdctl pull registry.k8s.io/kube-proxy:v1.30.0

emmmmmmm

再仔细看一下上面的配置文件（为了方便理解做了一点小替换）

1
2
3

server = "https://registry.k8s.io"
[host."https://cr.suyiiyii.top"]
  capabilities = ["pull", "resolve"]

emmmm

🤔

好像，是不是就是给地址的前面的部分做了一个重写。💡！

那我把 registry.k8s.io 给全部换成cr.suyiiyii.top/registry.k8s.io，不就可以了。😲

1
2
3

server = "https://registry.k8s.io"
[host."https://cr.suyiiyii.top/registry.k8s.io"]
  capabilities = ["pull", "resolve"]

我们来试一下直接拉取，我感觉成功就在眼前了。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


root@ubuntu-98:~# nerdctl pull registry.k8s.io/kube-proxy:v1.30.0
DEBU[0000] verifying process skipped
DEBU[0000] Found hosts dir "/etc/containerd/certs.d"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs".
DEBU[0000] fetching                                      image="registry.k8s.io/kube-proxy:v1.30.0"
DEBU[0000] loading host directory                        dir=/etc/containerd/certs.d/registry.k8s.io
DEBU[0000] resolving                                     host=cr.suyiiyii.top
DEBU[0000] do request                                    host=cr.suyiiyii.top request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=HEAD url="https://cr.suyiiyii.top/registry.k8s.io/v2/kube-proxy/manifests/v1.30.0?ns=registry.k8s.io"
DEBU[0000] fetch response received                       host=cr.suyiiyii.top response.header.accept-ranges=bytes response.header.cache-control="no-store, no-cache, must-revalidate" response.header.connection=keep-alive response.header.content-length=785 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=text/html response.header.date="Thu, 20 Jun 2024 13:47:19 GMT" response.header.etag="\"665eee4b-311\"" response.header.last-modified="Tue, 04 Jun 2024 10:36:59 GMT" response.header.server=nginx response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.status="200 OK" url="https://cr.suyiiyii.top/registry.k8s.io/v2/kube-proxy/manifests/v1.30.0?ns=registry.k8s.io"
DEBU[0000] no Docker-Content-Digest header, fetching manifest instead  host=cr.suyiiyii.top
DEBU[0000] do request                                    host=cr.suyiiyii.top request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=GET url="https://cr.suyiiyii.top/registry.k8s.io/v2/kube-proxy/manifests/v1.30.0?ns=registry.k8s.io"
DEBU[0000] fetch response received                       host=cr.suyiiyii.top response.header.accept-ranges=bytes response.header.cache-control="no-store, no-cache, must-revalidate" response.header.connection=keep-alive response.header.content-length=785 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=text/html response.header.date="Thu, 20 Jun 2024 13:47:19 GMT" response.header.etag="\"665eee4b-311\"" response.header.last-modified="Tue, 04 Jun 2024 10:36:59 GMT" response.header.server=nginx response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.status="200 OK" url="https://cr.suyiiyii.top/registry.k8s.io/v2/kube-proxy/manifests/v1.30.0?ns=registry.k8s.io"
DEBU[0000] resolved                                      desc.digest="sha256:fe29cb63644f0d85a0e8abd7c494c1a350d652a0f2962fbf1b12102ac6cf75b6" host=cr.suyiiyii.top
DEBU[0000] loading host directory                        dir=/etc/containerd/certs.d/registry.k8s.io
DEBU[0000] fetch                                         digest="sha256:fe29cb63644f0d85a0e8abd7c494c1a350d652a0f2962fbf1b12102ac6cf75b6" mediatype=text/html size=785
WARN[0000] reference for unknown type: text/html         digest="sha256:fe29cb63644f0d85a0e8abd7c494c1a350d652a0f2962fbf1b12102ac6cf75b6" mediatype=text/html size=785
DEBU[0000] do request                                    digest="sha256:fe29cb63644f0d85a0e8abd7c494c1a350d652a0f2962fbf1b12102ac6cf75b6" mediatype=text/html request.header.accept="text/html, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=GET size=785 url="https://cr.suyiiyii.top/registry.k8s.io/v2/kube-proxy/blobs/sha256:fe29cb63644f0d85a0e8abd7c494c1a350d652a0f2962fbf1b12102ac6cf75b6?ns=registry.k8s.io"
DEBU[0000] fetch response received                       digest="sha256:fe29cb63644f0d85a0e8abd7c494c1a350d652a0f2962fbf1b12102ac6cf75b6" mediatype=text/html response.header.accept-ranges=bytes response.header.cache-control="no-store, no-cache, must-revalidate" response.header.connection=keep-alive response.header.content-length=785 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=text/html response.header.date="Thu, 20 Jun 2024 13:47:19 GMT" response.header.etag="\"665eee4b-311\"" response.header.last-modified="Tue, 04 Jun 2024 10:36:59 GMT" response.header.server=nginx response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.status="200 OK" size=785 url="https://cr.suyiiyii.top/registry.k8s.io/v2/kube-proxy/blobs/sha256:fe29cb63644f0d85a0e8abd7c494c1a350d652a0f2962fbf1b12102ac6cf75b6?ns=registry.k8s.io"
DEBU[0000] encountered unknown type text/html; children may not be fetched
FATA[0000] failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:fe29cb63644f0d85a0e8abd7c494c1a350d652a0f2962fbf1b12102ac6cf75b6: not found

但是很遗憾，拉取失败了。

我们仔细看一下日志，他说在访问

https://cr.suyiiyii.top/registry.k8s.io/v2/kube-proxy/manifests/v1.30.0?ns=registry.k8s.io

这个地址的时候报错了。

感觉没有什么问题，我们再找个正常的拉取过程看看。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

root@ubuntu-98:~# nerdctl pull cr.suyiiyii.top/registry.k8s.io/kube-apiserver:v1.30.0
DEBU[0000] verifying process skipped
DEBU[0000] Found hosts dir "/etc/containerd/certs.d"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs".
DEBU[0000] fetching                                      image="cr.suyiiyii.top/registry.k8s.io/kube-apiserver:v1.30.0"
DEBU[0000] resolving                                     host=cr.suyiiyii.top
DEBU[0000] do request                                    host=cr.suyiiyii.top request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=HEAD url="https://cr.suyiiyii.top/v2/registry.k8s.io/kube-apiserver/manifests/v1.30.0"
DEBU[0000] fetch response received                       host=cr.suyiiyii.top response.header.connection=keep-alive response.header.content-length=152 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 20 Jun 2024 13:51:02 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx response.header.set-cookie="sid=7c726572e44d8a8a188a68c5a99ab9be; Path=/; HttpOnly" response.header.www-authenticate="Bearer realm=\"https://cr.suyiiyii.top/service/token\",service=\"harbor-registry\",scope=\"repository:registry.k8s.io/kube-apiserver:pull\"" response.header.x-request-id=58fcd38e-88d9-45f0-9085-7ee90858227f response.status="401 Unauthorized" url="https://cr.suyiiyii.top/v2/registry.k8s.io/kube-apiserver/manifests/v1.30.0"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://cr.suyiiyii.top/service/token\",service=\"harbor-registry\",scope=\"repository:registry.k8s.io/kube-apiserver:pull\"" host=cr.suyiiyii.top
DEBU[0000] do request                                    host=cr.suyiiyii.top request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.16+unknown request.method=HEAD url="https://cr.suyiiyii.top/v2/registry.k8s.io/kube-apiserver/manifests/v1.30.0"
cr.suyiiyii.top/registry.k8s.io/kube-apiserver:v1.30.0: resolving      |--------------------------------------|
elapsed: 7.8 s                                          total:   0.0 B (0.0 B/s)
DEBU[0008] fetch response received                       host=cr.suyiiyii.top response.header.connection=keep-alive response.header.content-length=1363 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Thu, 20 Jun 2024 13:51:10 GMT" response.header.docker-content-digest="sha256:6b8e197b2d39c321189a475ac755a77896e34b56729425590fbc99f3a96468a3" response.header.etag="sha256:6b8e197b2d39c321189a475ac755a77896e34b56729425590fbc99f3a96468a3" response.header.server=nginx response.header.set-cookie="sid=47d495d91f0408a836c708db0dedfe51; Path=/; HttpOnly" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-frame-options=DENY response.header.x-request-id=cba32369-dfb6-46c6-b464-c2ccde8738ab response.status="200 OK" url="https://cr.suyiiyii.top/v2/registry.k8s.io/kube-apiserver/manifests/v1.30.0"
DEBU[0008] resolved                                      desc.digest="sha256:6b8e197b2d39c321189a475ac755a77896e34b56729425590fbc99f3a96468a3" host=cr.suyiiyii.top
cr.suyiiyii.top/registry.k8s.io/kube-apiserver:v1.30.0: resolving      |--------------------------------------|
elapsed: 7.9 s                                          total:   0.0 B (0.0 B/s)
DEBU[0008] fetch                                         digest="sha256:31282cf15b67192cd35f847715a9571f5dd4ac0e130290a408a866bd040bcd81" mediatype=application/vnd.docker.distribution.manifest.v2+json size=3024
DEBU[0008] fetch                                         digest="sha256:c42f13656d0b2e905ee7977f67ea7a17715b24fae9daca1fcfb303cdb90728f0" mediatype=application/vnd.docker.container.image.v1+json size=2797
DEBU[0008] layer unpacked                                duration="637.59µs" layer="sha256:286c61c9a31ace5fa0b8832c8e8e30d66bf32138f2f787463235aa0071f714ea"
DEBU[0008] layer unpacked                                duration="422.427µs" layer="sha256:2bdf44d7aa71bf3a0da2de0563ad0e3882948d699b4991edf8c0ab44e7f26ae3"
DEBU[0008] layer unpacked                                duration="358.593µs" layer="sha256:452e9eed7ecfd0c2b44ac6fda20cee66ab98aec38ba30aa868e02445be7c8bb0"
DEBU[0008] layer unpacked                                duration="396.345µs" layer="sha256:b6824ed73363f94b3b2b44084c51c31bc32af77a96861d49e16f91e3ab6bed71"
DEBU[0008] layer unpacked                                duration="492.841µs" layer="sha256:7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212"
DEBU[0008] layer unpacked                                duration="381.644µs" layer="sha256:33e068de264953dfdc9f9ada207e76b61159721fd64a4820b320d05133a55fb8"
DEBU[0008] layer unpacked                                duration="328.271µs" layer="sha256:5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd"
DEBU[0008] layer unpacked                                duration="437.488µs" layer="sha256:27be814a09ebd97fac6fb7b82d19f117185e90601009df3fbab6f442f85cd6b3"
DEBU[0008] layer unpacked                                duration="493.432µs" layer="sha256:4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f"
DEBU[0008] layer unpacked                                duration="390.224µs" layer="sha256:3f4e2c5863480125882d92060440a5250766bce764fee10acdbac18c872e4dc7"
DEBU[0008] layer unpacked                                duration="400.147µs" layer="sha256:80a8c047508ae5cd6a591060fc43422cb8e3aea1bd908d913e8f0146e2297fea"
DEBU[0008] layer unpacked                                duration="398.355µs" layer="sha256:6625e293fdbf7cfdd3272c9e871714e6de754bc859bf9e0ec84e4dd0a860b631"
DEBU[0008] layer unpacked                                duration="419.697µs" layer="sha256:0c6c1ee970856d051398b9bdfbd829083b7fa79aeb59435f4dafae3837ba8948"
cr.suyiiyii.top/registry.k8s.io/kube-apiserver:v1.30.0:                           resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:6b8e197b2d39c321189a475ac755a77896e34b56729425590fbc99f3a96468a3:    exists         |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:31282cf15b67192cd35f847715a9571f5dd4ac0e130290a408a866bd040bcd81: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:c42f13656d0b2e905ee7977f67ea7a17715b24fae9daca1fcfb303cdb90728f0:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 8.1 s                                                                    total:   0.0 B (0.0 B/s)

好像，他一开始访问的是

https://cr.suyiiyii.top/v2/registry.k8s.io/kube-apiserver/manifests/v1.30.0

emmm，仔细看看，可以发现是 v2 的位置不同。

两个放一起做对比：

https://cr.suyiiyii.top/registry.k8s.io/v2/kube-proxy/manifests/v1.30.0?ns=registry.k8s.io

https://cr.suyiiyii.top/v2/registry.k8s.io/kube-apiserver/manifests/v1.30.0

这个 v2 的话，大胆猜测，应该是 api 的路径。

就是说，api 的路径永远都是 cr.suyiiyii.top/v2/ 在这个路径下的，而我们做替换，把 v2 换到后面去了，自然就拉取失败了。

反向代理重写路径

Caddy

我首先想到的是用 caddy 去重写路径。

直接说结论吧，我没搞好，放弃了。

nginx

但是想起来，harbor 的组件里面好像有 nginx，也就是说，harbor 自己就有用 nginx 做反向代理，那我干嘛舍近求远，再加一层，直接改 harbor 的 nginx 配置不就行了吗。

我的需求是，添加配置，当访问/{path}/v2/xxx 的时候，路由到/v2/{path}/xxx，其他的不变。

在 ai 的帮助下，我把这段配置添加到 harbor 的 nginx 配置里面了。

1
2
3
4
5
6
7
8
9

location ~ ^/(?<path>.+)/v2/(.*)$ {
  proxy_pass http://core/v2/$path/$2;
  proxy_set_header Host $http_host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $x_forwarded_proto;
  proxy_buffering off;
  proxy_request_buffering off;
}

最终结果

最后，见证奇迹的时刻到了

1
2
3
4
5
6
7
8
9

root@ubuntu-98:~# nerdctl pull openjdk:17-slim
docker.io/library/openjdk:17-slim:                                                resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:aaa3b3cb27e3e520b8f116863d0580c438ed55ecfa0bc126b41f68c3f62f9774:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:779635c0c3d23cc8dbab2d8c1ee4cf2a9202e198dfc8f4c0b279824d9b8e0f22: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:37cb44321d0423bc57266a3bff658daf00478e4cdf2d3b8091f785310534256d:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6ce99fdf16e86bd02f6ad66a0e1334878528b5a4b5487850a76e0c08a7a27d56:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:1fe172e4850f03bb45d41a20174112bc119fbfec42a650edbbd8491aee32e3c3:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:44d3aa8d076675d49d85180b0ced9daef210fe4fdff4bdbb422b9cf384e591d0:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 77.8s

虽然有一点慢，但我们还是很顺利的把镜像拉取下来了。

同时，如果我们删除镜像后再次拉取，就可享受到缓存的加速效果，效果还是非常明显的。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

root@ubuntu-98:~# docker image rm openjdk:17-slim
Untagged: docker.io/library/openjdk:17-slim@sha256:aaa3b3cb27e3e520b8f116863d0580c438ed55ecfa0bc126b41f68c3f62f9774
Deleted: sha256:9c1b6dd6c1e6be9fdd2b1987783824670d3b0dd7ae8ad6f57dc3cea5739ac71e
Deleted: sha256:13a34b6fff7804cf7f6e8f52a4cf25ceb2e32fc35a6f39e8158074c64831ebf0
Deleted: sha256:6be690267e47ddcfd965449d2af70a9eca9879f9436948ee83d7f4ad473b8e64
root@ubuntu-98:~# docker pull openjdk:17-slim
docker.io/library/openjdk:17-slim:                                                resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:aaa3b3cb27e3e520b8f116863d0580c438ed55ecfa0bc126b41f68c3f62f9774:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:779635c0c3d23cc8dbab2d8c1ee4cf2a9202e198dfc8f4c0b279824d9b8e0f22: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:37cb44321d0423bc57266a3bff658daf00478e4cdf2d3b8091f785310534256d:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6ce99fdf16e86bd02f6ad66a0e1334878528b5a4b5487850a76e0c08a7a27d56:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:1fe172e4850f03bb45d41a20174112bc119fbfec42a650edbbd8491aee32e3c3:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:44d3aa8d076675d49d85180b0ced9daef210fe4fdff4bdbb422b9cf384e591d0:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 16.5s                                                                    total:  209.1  (12.7 MiB/s)

收尾

最后，我们还要手动给每一个可能会遇到的镜像站镜像配置。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

mkdir -p /etc/containerd/certs.d/docker.io && cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://cr.suyiiyii.top/docker.io"]
  capabilities = ["pull", "resolve"]
EOF
mkdir -p /etc/containerd/certs.d/registry.k8s.io && cat > /etc/containerd/certs.d/registry.k8s.io/hosts.toml << EOF
server = "https://registry.k8s.io"
[host."https://cr.suyiiyii.top/registry.k8s.io"]
  capabilities = ["pull", "resolve"]
EOF
mkdir -p /etc/containerd/certs.d/gcr.io && cat > /etc/containerd/certs.d/gcr.io/hosts.toml << EOF
server = "https://gcr.io"
[host."https://cr.suyiiyii.top/gcr.io"]
  capabilities = ["pull", "resolve"]
EOF
mkdir -p /etc/containerd/certs.d/ghcr.io && cat > /etc/containerd/certs.d/ghcr.io/hosts.toml << EOF
server = "https://ghcr.io"
[host."https://cr.suyiiyii.top/ghcr.io"]
  capabilities = ["pull", "resolve"]
EOF
mkdir -p /etc/containerd/certs.d/k8s.gcr.io && cat > /etc/containerd/certs.d/k8s.gcr.io/hosts.toml << EOF
server = "https://k8s.gcr.io"
[host."https://cr.suyiiyii.top/k8s.gcr.io"]
  capabilities = ["pull", "resolve"]
EOF
mkdir -p /etc/containerd/certs.d/quay.io && cat > /etc/containerd/certs.d/quay.io/hosts.toml << EOF
server = "https://quay.io"
[host."https://cr.suyiiyii.top/quay.io"]
  capabilities = ["pull", "resolve"]
EOF

我不会 bash，所以写的真的很冗长，能用就行了。😝

总结

不管怎么说，很爽的一次折腾。从开始遇到问题，再从各种角度想办法去解决，不断遇到问题再解决问题，最后达到自己想要的效果。成就感满满。

那就算是把镜像的缓存搞好了，应该不会再受到网络环境的影响了。

可以开始好好玩 k8s 了。💪

Homelab
Licensed under CC BY-NC-SA 4.0