Thursday, September 1, 2022

[SOLVED] error when podman push image to AWS registry

Issue

I'm using fedora with podman to push a image to a aws registry

But I get this error,

Error: writing blob: Post "https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/uploads/": EOF

No issues withe the login:

aws ecr get-login-password --region sa-east-1 | podman login --username AWS --password-stdin XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com

Login Succeeded!

But when I try to push

podman --log-level debug  push XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx

output:

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman --log-level debug push XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx) 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/bruno/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/bruno/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/bruno/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/bruno/.local/share/containers/storage/volumes 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /home/bruno/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 37             
DEBU[0000] Looking up image "XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx" in local containers storage 
DEBU[0000] Trying "XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx" ... 
DEBU[0000] Trying "XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx:latest" ... 
DEBU[0000] parsed reference into "[overlay@/home/bruno/.local/share/containers/storage+/run/user/1000/containers]@9aca0d1dcdb72d44b90f4c02889c307d1c372078866121238bc2606f7e3a3f2b" 
DEBU[0000] Found image "XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx" as "XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx:latest" in local containers storage 
DEBU[0000] Found image "XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx" as "XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx:latest" in local containers storage ([overlay@/home/bruno/.local/share/containers/storage+/run/user/1000/containers]@9aca0d1dcdb72d44b90f4c02889c307d1c372078866121238bc2606f7e3a3f2b) 
DEBU[0000] Pushing image XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx to XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx 
DEBU[0000] Copying source image [overlay@/home/bruno/.local/share/containers/storage+/run/user/1000/containers]@9aca0d1dcdb72d44b90f4c02889c307d1c372078866121238bc2606f7e3a3f2b to destination image //XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx:latest 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Found credentials for XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com in credential helper containers-auth.json in file /run/user/1000/containers/auth.json 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/containers/sigstore  
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com 
DEBU[0000] Using blob info cache at /home/bruno/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image containers-storage:[overlay@/home/bruno/.local/share/containers/storage]@9aca0d1dcdb72d44b90f4c02889c307d1c372078866121238bc2606f7e3a3f2b 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking /v2/nginx/blobs/sha256:6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f 
DEBU[0000] GET https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/ 
DEBU[0000] Ping https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/ status 401 
DEBU[0000] HEAD https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/sha256:6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f 
DEBU[0000] ... not present                              
DEBU[0000] exporting filesystem layer "6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f" without compression for blob "sha256:6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f" 
DEBU[0001] No compression detected                      
DEBU[0001] Compressing blob on the fly                  
DEBU[0001] Uploading /v2/nginx/blobs/uploads/           
DEBU[0001] POST https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/uploads/ 
Copying blob 6973a448ae32 [--------------------------------------] 8.0b / 137.4MiB
WARN[0001] failed, retrying in 1s ... (1/3). Error: writing blob: Post "https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/uploads/": EOF 
DEBU[0002] Found credentials for XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com in credential helper containers-auth.json in file /run/user/1000/containers/auth.json 
DEBU[0002] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0002]  Using "default-docker" configuration        
DEBU[0002]   Using file:///var/lib/containers/sigstore  
DEBU[0002] Looking for TLS certificates and private keys in /etc/docker/certs.d/XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com 
DEBU[0002] Using blob info cache at /home/bruno/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0002] IsRunningImageAllowed for image containers-storage:[overlay@/home/bruno/.local/share/containers/storage]@9aca0d1dcdb72d44b90f4c02889c307d1c372078866121238bc2606f7e3a3f2b 
DEBU[0002]  Using default policy section                
DEBU[0002]  Requirement 0: allowed                      
DEBU[0002] Overall: allowed                             
Getting image source signatures
DEBU[0002] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0002] ... will first try using the original manifest unmodified 
DEBU[0002] Checking /v2/nginx/blobs/sha256:6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f 
DEBU[0002] GET https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/ 
DEBU[0002] Ping https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/ status 401 
DEBU[0002] HEAD https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/sha256:6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f 
DEBU[0002] ... not present                              
DEBU[0002] exporting filesystem layer "6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f" without compression for blob "sha256:6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f" 
DEBU[0003] No compression detected                      
DEBU[0003] Compressing blob on the fly                  
DEBU[0003] Uploading /v2/nginx/blobs/uploads/           
DEBU[0003] POST https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/uploads/ 
Copying blob 6973a448ae32 [--------------------------------------] 8.0b / 137.4MiB
WARN[0004] failed, retrying in 1s ... (2/3). Error: writing blob: Post "https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/uploads/": EOF 
DEBU[0005] Found credentials for XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com in credential helper containers-auth.json in file /run/user/1000/containers/auth.json 
DEBU[0005] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0005]  Using "default-docker" configuration        
DEBU[0005]   Using file:///var/lib/containers/sigstore  
DEBU[0005] Looking for TLS certificates and private keys in /etc/docker/certs.d/XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com 
DEBU[0005] Using blob info cache at /home/bruno/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0005] IsRunningImageAllowed for image containers-storage:[overlay@/home/bruno/.local/share/containers/storage]@9aca0d1dcdb72d44b90f4c02889c307d1c372078866121238bc2606f7e3a3f2b 
DEBU[0005]  Using default policy section                
DEBU[0005]  Requirement 0: allowed                      
DEBU[0005] Overall: allowed                             
Getting image source signatures
DEBU[0005] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0005] ... will first try using the original manifest unmodified 
DEBU[0005] Checking /v2/nginx/blobs/sha256:6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f 
DEBU[0005] GET https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/ 
DEBU[0005] Ping https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/ status 401 
DEBU[0005] HEAD https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/sha256:6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f 
DEBU[0005] ... not present                              
DEBU[0005] exporting filesystem layer "6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f" without compression for blob "sha256:6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f" 
DEBU[0006] No compression detected                      
DEBU[0006] Compressing blob on the fly                  
DEBU[0006] Uploading /v2/nginx/blobs/uploads/           
DEBU[0006] POST https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/uploads/ 
Copying blob 6973a448ae32 [--------------------------------------] 8.0b / 137.4MiB
WARN[0006] failed, retrying in 1s ... (3/3). Error: writing blob: Post "https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/uploads/": EOF 
DEBU[0007] Found credentials for XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com in credential helper containers-auth.json in file /run/user/1000/containers/auth.json 
DEBU[0007] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0007]  Using "default-docker" configuration        
DEBU[0007]   Using file:///var/lib/containers/sigstore  
DEBU[0007] Looking for TLS certificates and private keys in /etc/docker/certs.d/XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com 
DEBU[0007] Using blob info cache at /home/bruno/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0007] IsRunningImageAllowed for image containers-storage:[overlay@/home/bruno/.local/share/containers/storage]@9aca0d1dcdb72d44b90f4c02889c307d1c372078866121238bc2606f7e3a3f2b 
DEBU[0007]  Using default policy section                
DEBU[0007]  Requirement 0: allowed                      
DEBU[0007] Overall: allowed                             
Getting image source signatures
DEBU[0007] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0007] ... will first try using the original manifest unmodified 
DEBU[0007] Checking /v2/nginx/blobs/sha256:6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f 
DEBU[0007] GET https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/ 
DEBU[0007] Ping https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/ status 401 
DEBU[0007] HEAD https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/sha256:6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f 
DEBU[0007] ... not present                              
DEBU[0007] exporting filesystem layer "6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f" without compression for blob "sha256:6973a448ae3253e9c587bc4d29346bf77ae9fa350693a81f1ee28cd07c13d92f" 
DEBU[0008] No compression detected                      
DEBU[0008] Compressing blob on the fly                  
DEBU[0008] Uploading /v2/nginx/blobs/uploads/           
DEBU[0008] POST https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/uploads/ 
Copying blob 6973a448ae32 [--------------------------------------] 8.0b / 137.4MiB
DEBU[0008] Looking up image "XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx" in local containers storage 
DEBU[0008] Trying "XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx" ... 
DEBU[0008] Trying "XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx:latest" ... 
DEBU[0008] parsed reference into "[overlay@/home/bruno/.local/share/containers/storage+/run/user/1000/containers]@9aca0d1dcdb72d44b90f4c02889c307d1c372078866121238bc2606f7e3a3f2b" 
DEBU[0008] Found image "XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx" as "XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/nginx:latest" in local containers storage 
Error: writing blob: Post "https://XXXXXXXXXXXX.dkr.ecr.sa-east-1.amazonaws.com/v2/nginx/blobs/uploads/": EOF

Solution

I got a similar issue, and resolve it by creating the empty ECR repository from the console and then to podman push command works



Answered By - Ariel Jalil
Answer Checked By - Mary Flores (WPSolving Volunteer)