Can not successfully docker pull the v1.1 version image

[Zirui]

I docker pull the newest version, it failed with the following error message.

$ docker pull intellabs/intel_quantum_sdk:latest
latest: Pulling from intellabs/intel_quantum_sdk
99de9192b4af: Pull complete 
609a09d951fc: Pull complete 
4c3fc2dbbd1c: Pull complete 
677f22243eb5: Pull complete 
8c35c1e4e7d9: Pull complete 
33b4e133a31d: Pull complete 
eb1afe458086: Pull complete 
9d286ab772e8: Extracting [==================================================>]  979.5MB/979.5MB
30387de74455: Download complete 
64b1dbf32957: Download complete 
f8d720b89301: Download complete 
a96bf0b96e4c: Download complete 
8978314a77d0: Download complete 
failed to register layer: failed to Lchown "/opt/intel/quantum-sdk/docker-intel_quantum_sdk_1.1.0.2024-04-06T19_31_56+00_00" for UID 1285492, GID 2222 (try increasing the number of subordinate IDs in /etc/subuid and /etc/subgid): lchown /opt/intel/quantum-sdk/docker-intel_quantum_sdk_1.1.0.2024-04-06T19_31_56+00_00: invalid argument

Then I tried to docker pull the v1.0 version, and it succeeded. So I guess there are some bugs in the v1.1 version image?

[KevinR_Intel]

Hi Zirui,

It seems to be working on several platforms:  we are able to successfully pull the image from Ubuntu 22.04, 24.04, Debian Trixie, and Podman Desktop on Win11.

Our initial suggestion is to make sure the container software is up-to-date, and to try a command like`docker system prune -a` to start with a clean container install.

If the image pull fails after that, would you mind sharing with us which OS and container software you're using?

[Zirui]

Thanks for your reply! I tried `docker system prune -a` but it still failed after that.

I'm using a server.

The OS version of the server is Ubuntu 22.04.3 LTS.

The docker version: Docker version 25.0.2, build 29cf629

I can successfully pull the v1.0 image though. When I pulled the v1.1 image, it seemed to try to access the `/opt`. But I don't have the write access to `/opt`. By the way, I can only use Docker as a non-root user and store data in some limited directory. Is that the reason the pull failed on my server?

[Frisco]

Hello Zirui,

I have seen this kind of problem before in a few different container offerings. I know of a couple of things that can be explored to figure out what is going on. One is that storing/running and image on an NFS filesystem can cause this kind of problem as NFS just can't handle the id mappings. Two is that cloud servers are often just containers themselves and running on bare metal or at least a full VM can solve the limited (G)UID problems. Happy to help you dig into this further if you like?

--HTH

[Zirui]

Hi Frisco,

I appreciate your help! I think the first situation matches mine. I'm using the server with an NFS filesystem. What should I do next?

[Frisco]

Hi Zirui,

The fix is usually to redirect the storage of the image to somewhere other than your $home or system path on NFS. Where really depends on what is available...  local storage can be places like /tmp (goes away on restarts) or in many HPC style setups you might have a local disk mounted on /scratch or /data. Do you have any admin privileges or settings that allow you to designate/use local storage capabilities?

I would start by understanding your needs: https://docs.docker.com/engine/security/rootless/ then adapt your engine startup path https://forums.docker.com/t/how-do-i-change-the-docker-image-installation-directory/1169 based on what is available.

Let me know how you get on.

--HTH

[Zirui]

Hi Frisco,

I have access to some local storage, the directory is `/freespace/local/zirui/docker`. I think all the images or data are stored under this directory.

[Frisco]

Hi Zirui,

If that is really local and not an NFS mount and if your images are already stored there then your containers would not be stored on an NFS share. Can you determine for sure that the storage path you have available is not an NFS mount? 

FYI, based on the information so far it seems that you may need to move to bare metal or at least full VM server instance - do you know what the offerings of your server provider are?

[Zirui]

Hi  Frisco,

I checked that the directory `/freespace/local/zirui/docker` doesn't exist on other machines. I think it indicates that the directory is in a local file system of each machine, not NFS.

And I don't think my server provider offers any bare metal or full VM server services.

Also, I can docker pull the v1.0 image. I don't know how to explain that if the setting of my docker is wrong. Thanks for your help again!

[Frisco]

Hi Zirui,

Containers, their contents and the tools to build them have been updated with new utility and greater security. Due to server administration security concerns it is almost never advisable to use outdated software. Security concerns prompted the need for Docker rootless mode development - both at the application layer(quick)  and in the OS(slow). Moving to a fully updated OS infrastructure is the best option for both utility and security. 

[KevinR_Intel]

Hi Zirui,

Based on the explanations from Frisco and yourself, it sounds like the server provided to you is configured such that it won't allow the container image to run as it needs to. It's possible the system's admin does have a workaround to provide support for your use case. As an alternative, I suggest you try using the container from a machine where you are the admin (e.g., a laptop or desktop that you control) or perhaps visit the Intel Developer's Cloud as they have a working installation of the Intel Quantum SDK.

[Zirui]

Thank you and also thank Frisco for explaining all these details to me! I'll work it out using an admin account.

[Zirui]

I'm grateful for your time and effort today. I have one more question:

in the error message, the docker wants to access `/opt` for UID 1285492, GID 2222. I checked by doing the command `id -nu  1285492` and found no such user. It seems the uid gid is from the image. Does the uid gid belong to the creator of the image or any user from your side? I'm not in a hurry to use the image so please take your time.

[Frisco]

Hi Zirui,

Here is a rough outline of what is happening. The /opt access problem you see is the message from an interim container fs being constructed by the container engine, that itself runs in a specially constructed container. This automated (G)UID remapping is due to rootless execution in an intermediate layer. It does not correspond directly to any actual user assigned in the image construction or the executed container. The issue is that the kernel APIs exposed in a container engine are actually wrappers around the host kernel API (which itself is very likely a wrapper around the hypervisor kernel APIs). Since one or more of the intervening kernel APIs lack dynamic (G)UID allocation above a set range then the image reconstruction fails. This problem is mitigated (for instance) in a typical systemd >= v251 based OS. This is also a problem for NFS since there is no (currently deployed) facility to map the service appliance (G)UID space to the requested client based (G)UID set of a container user. By running docker in root-full mode or using a VM (or bare metal) you decouple (G)UID handling from any external influence and process the mapping with full permissions directly in the current OS environment.

Hopefully this (rather cursory) description can help point you in the right direction in your adventure of becoming a security specialist, system administrator, network architect, systems programmer, software engineer and quantum computing expert.