[SOLVED] How to run docker-image on ec2 instance with public ipv4 address

Issue

Hello I created EC2 instance then updated packages and installed docker and docker-compose. After that I have created docker-compose.yml file like that:

version: '3.7'
volumes:
  shared-workspace:
    name: "hadoop-distributed-file-system"
    driver: local
services:
    jupyter:
        image: 'jupyter/datascience-notebook:latest'
        hostname: jupyterlab
        ports:
            - 4888:4888
        volumes:
          - shared-workspace:/opt/workspace
        container_name: jupyter
        environment:
        - JUPYTER_TOKEN=...

and moved the file to instance and run in terminal commands:

sudo systemctl start docker
docker-compose up

...
jupyter  | [I 2023-10-12 14:04:31.021 ServerApp] Jupyter Server 2.7.3 is running at:
jupyter  | [I 2023-10-12 14:04:31.021 ServerApp] http://jupyterlab:8888/lab?token=...
jupyter  | [I 2023-10-12 14:04:31.021 ServerApp] http://127.0.0.1:8888/lab?token=...
...

My question is why this is running on ip http://127.0.0.1:8888/lab?token=... while I expecting and wanted to have it on public IPv4 address from ec2? (http://ipv4_address:4888/lab?).I have security group set to all traffic. What am I doing wrong?

Solution

My question is why this is running on ip http://127.0.0.1:8888/lab?token=...

From Jupyter's point of view, it's listening on port 8888 on two addresses: localhost and the one jupyterlab resolves to.

Docker substitutes /etc/hosts when running a container and adds an entry which maps the hostname to the address of its virtual interface. So effectively Jupyter listens on localhost and whatever address the Docker virtual network interface has.

Since a Docker container is an isolated environment and has its own network subsystem, it does not know anything about your host's network stack.

What happens when you specify ports: -4888:4888 in your docker-compose.yml, is that Docker starts forwarding requests coming to your host's interfaces on port 4888 to Docker's virtual interface, also on port 4888. It happens either through iptables or through user-space processes called docker-proxy. Either way, your requests reach the Docker's virtual network interface, as long as their destination port is 4888.

Since your Jupyter process listens on 8888 (as you can see here on line 41), but nothing is being forwarded to port 8888 (because you haven't set up the forwarding for it in docker-compose.yml), Jupyter cannot receive any requests.

To answer your question more directly:

1. Jupyter listens on 127.0.0.1 (and on jupyterlab) because it's set up this way in the Docker image you're using, and honestly reports it in the logs.

2. Even though Jupyter doesn't know about the external host's IP and doesn't show it in the logs, it will get the requests coming to http://[external-ip]:[external-port] as soon as you set correct port mapping: ports: - [external port]:8888. You can set the external port to whatever you want, but the internal port has to be 8888.

Answered By - Quassnoi

Answer Checked By - Mary Flores (WPSolving Volunteer)