Thursday, November 18, 2021

[SOLVED] Docker (compose): connection refused without sudo

Issue

All was fine until "something". After that "something", I could use docker and docker-compose with sudo but without it the docker would hang indefinitely and the docker-compose would return connection stack traces.

Doesn't work:

  • Running docker without sudo hangs indefinitely. Temporarily fixed by exporting DOCKER_HOST=tcp://127.0.0.1:2375.
  • docker-compose without sudo returns with connection refused. It's not about connecting to specific container but for all basic commands such as docker-compose ps.

Proof that user belongs to docker group:

kretyn@junk$ groups kretyn
kretyn : kretyn adm cdrom sudo dip plugdev lpadmin lxd sambashare docker

Solution hints

I had a docker context activated that was pointing at existing but inactive host. Removing that context and unsetting DOCKER_HOST allowed the docker command to work correctly. That's a surprise since I reinstalled (purged everything I could find) the docker before writing this question.

Removing context isn't sufficient. Docker Compose required explicit docker context use default otherwise it wouldn't understand.

The problem with docker-compose ps is still present.

Stack trace on docker-compose

kretyn@junk$ docker-compose ps
Traceback (most recent call last):
  File "urllib3/connection.py", line 159, in _new_conn
  File "urllib3/util/connection.py", line 84, in create_connection
  File "urllib3/util/connection.py", line 74, in create_connection
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 670, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1255, in request
  File "http/client.py", line 1301, in _send_request
  File "http/client.py", line 1250, in endheaders
  File "http/client.py", line 1010, in _send_output
  File "http/client.py", line 950, in send
  File "urllib3/connection.py", line 187, in connect
  File "urllib3/connection.py", line 171, in _new_conn
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f762c4e3e20>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "requests/adapters.py", line 439, in send
  File "urllib3/connectionpool.py", line 726, in urlopen
  File "urllib3/util/retry.py", line 446, in increment
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=2375): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f762c4e3e20>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docker/api/client.py", line 214, in _retrieve_server_version
  File "docker/api/daemon.py", line 181, in version
  File "docker/utils/decorators.py", line 46, in inner
  File "docker/api/client.py", line 237, in _get
  File "requests/sessions.py", line 543, in get
  File "requests/sessions.py", line 530, in request
  File "requests/sessions.py", line 643, in send
  File "requests/adapters.py", line 516, in send
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=2375): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f762c4e3e20>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 80, in main
  File "compose/cli/main.py", line 189, in perform_command
  File "compose/cli/command.py", line 60, in project_from_options
  File "compose/cli/command.py", line 152, in get_project
  File "compose/cli/docker_client.py", line 41, in get_client
  File "compose/cli/docker_client.py", line 170, in docker_client
  File "docker/api/client.py", line 197, in __init__
  File "docker/api/client.py", line 221, in _retrieve_server_version
docker.errors.DockerException: Error while fetching server API version: HTTPConnectionPool(host='127.0.0.1', port=2375): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f762c4e3e20>: Failed to establish a new connection: [Errno 111] Connection refused'))
[46145] Failed to execute script docker-compose

Versions/builds

Linux distro

kretyn@junk$ uname -a
Linux junk 5.8.0-43-generic #49~20.04.1-Ubuntu SMP Fri Feb 5 09:57:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Docker

kretyn@junk$ sudo docker system info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 9
  Running: 8
  Paused: 0
  Stopped: 1
 Images: 44
 Server Version: 20.10.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.8.0-43-generic
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.29GiB
 Name: meland
 ID: 3PE6:4AWU:LGRR:HOBJ:XUCJ:ER4H:ZEF5:MWUJ:COEG:WPK5:YH2U:FKUF
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio weight support
WARNING: No blkio weight_device support

Docker-compose

kretyn@junk$ sudo docker-compose version
docker-compose version 1.28.0, build d02a7b1a
docker-py version: 4.4.1
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Solution

The problem was a dead context entry.

Solution

Removed the dead context named remote using docker context rm remote.

Even though docker context ls was pointing default as default, it was necessary to do docker context use default for the context to be properly updated.

As a partial fix I had previously set DOCKER_HOST to tcp://127.0.0.1:2375. After updating context the env variable had to be unset, i.e. unset DOCKER_HOST.

Lazy commands, assuming context remote:

$ docker context rm remote
$ docker context use default
$ unset DOCKER_HOST

Background

Had a remote context that was based on ssh endpoint. My ssh config was set that the name points at my domain but recently the host behind it was removed.

Interestingly, I've been "fixing" this for a while and one of the steps was to completely purge docker from my local Ubuntu host, including apt, /var/lib/docker, /etc/docker and random scripts in ~/.local/. The context, however, wasn't cleaned.



Answered By - Dawid Laszuk
at