[SOLVED] Establish SSH connection from Jenkins container to SSH server container, I can establish with password login but can't establish with private key

Issue

The Story:

I am following a Udemy course for learning Jenkins with docker. The lab is that I setup a SSH server container & a Jenkins container, put them in the same docker network. Then, I go into the jenkins container to establish ssh connection to the SSH server container.

The issue:

I can establish SSH connection from jenkins container to the SSH server container with password login, however, it fails to establish the SSH -i with the key pair I generated.

The overview of my directory and files:

Under my working space of my local machine, I have:

• a docker-compose.yml (details see below)
• a directory named centos/,
• Inside centos/ I have a Dockerfile for building the SSH server image.

The steps I tried and their results:

Step 1, I firstly created the docker-compose.yml for the two containers

version: '3'
services:
  jenkins:
    container_name: jenkins
    image: jenkins/jenkins
    ports:
      - "8080:8080"
    volumes:
      - $PWD/jenkins_home:/var/jenkins_home
    networks:
      - net
  remote_host:
    container_name: remote_host
    image: remote-host
    build: 
      context: centos7
    networks:
      - net
networks:
  net:

As you can see above, the remote_host service refers to the SSH server image which is built from a Dockerfile under centos/ directory.

Step 2, so here is my Dockerfile under centos/ :

FROM centos

RUN yum -y install openssh-server

RUN useradd remote_user && \
    echo remote_user:1234 | chpasswd && \
    mkdir /home/remote_user/.ssh && \
    chmod 700 /home/remote_user/.ssh

COPY remote-key.pub /home/remote_user/.ssh/authorized_keys

RUN chown remote_user:remote_user -R /home/remote_user/.ssh/ && \
    chmod 600 /home/remote_user/.ssh/authorized_keys

RUN ssh-keygen -A
RUN rm -rf /run/nologin
CMD /usr/sbin/sshd -D

Step 3. As you can see in above Dockerfile, I COPY a public key remote-key.pub from my local host to the container and named it authorized_keys. I generated the key pair under the centos/ folder of my local host by command:

ssh-keygen -t rsa -m PEM -f remote-key

Step 4. After all above are done. I run command docker-compose build to build images.

Step 5. Finally I run docker-compose up to bring up containers.

Both jenkins & remote_host(the ssh server) containers are up and running successfully.

Step 6. Then I go inside the jenkins container by docker exec -it jenkins bash, in the bash terminal of the container, I do ssh connection:

jenkins@7551f2fa441d:/$ ssh remote_user@remote_host
remote_user@remote_host's password: 

After input the passowrd 1234 (Defined in the Dockerfile), I successfully established the SSH connection from jenkins container to the SSH server container. But I would like to establish the connection via key pairs instead of password login.

Step 7, since the SSH server container has the public key copied, so, I copied the private key from my local host to the jenkins container by command:

docker cp remote-key jenkins:/tmp/

So, now the jenkins container's /tmp/ directory contains the private key.

Step 8, Now I try to use the private key to establish the SSH connection to the ssh server container. I go inside the jenkins by docker exec -it jenkins bash, then, I run command:

jenkins@7551f2fa441d:/$ ssh -i /tmp/remote-key remote_user@remote_host
Load key "/tmp/remote-key": Permission denied
remote_user@remote_host's password: 

As you can see above, inside jenkins container, the bash prompt tells me the key "/tmp/remote-key": Permission denied. Right after that it asks me to input password, then I input password 1234, and the SSH connection is established.

So, why I can't use the private key to establish the SSH connection from the jenkins container to the ssh server container but only login with password works?

==== Update =====

Inspired by the comment from @agentsmith under his answer, I checked the ownership of the /tmp/remote-key inside my jenkins container:

drwxr-xr-x 2 jenkins   jenkins       4096 Sep 20 13:01 hsperfdata_jenkins
drwxr-xr-x 2 root      root          4096 Feb  2  2020 hsperfdata_root
drwxr-xr-x 2 jenkins   jenkins       4096 Sep 20 13:01 jetty-0_0_0_0-8080-war-_-any-190970179478026794.dir
drwxr-xr-x 2 jenkins   jenkins       4096 Sep 18 20:55 jetty-0_0_0_0-8080-war-_-any-878046537266404011.dir
-rw------- 1 245867976 1349604816    1679 Sep 18 20:53 remote-key
-rw-r--r-- 1 jenkins   jenkins    3167976 Sep 18 20:55 winstone3001500689590881345.jar
-rw-r--r-- 1 jenkins   jenkins    3167976 Sep 20 13:01 winstone8218655308653013358.jar

As you can see above, all other files have user and group ownership of jenkins (one file with root) except the remote-key file. So I tried to change the ownership by:

jenkins@7551f2fa441d:/$ chown jenkins:jenkins /tmp/remote-key 
chown: changing ownership of '/tmp/remote-key': Operation not permitted 

However the chown command tells me operation not permitted. :( What should I do now?

Solution

-rw------- 1 245867976 1349604816    1679 Sep 18 20:53 remote-key

jenkins@7551f2fa441d:/$ chown jenkins:jenkins /tmp/remote-key
chown: changing ownership of '/tmp/remote-key': Operation not permitted 

However the chown command tells me operation not permitted. :( What should I do now?

Looks like permission on your SSH key are indeed not set properly. You cannot change the file permission as it's owned by 245867976 but your bash session is ran by jenkins user.

You should change the file ownership key using root:

docker exec -it -u root jenkins bash
$ chown jenkins:jenkins /tmp/remote-key 
$ exit

-u flag will run bash in jenkins container as root user, with which you should be able to change your file permission. Then try again:

docker exec -it jenkins bash
$ ssh -i /tmp/remote-key remote_user@remote_host

---

Note: you can also directly run ssh with root without needing to change permissions

docker exec -it -u root jenkins bash
$ ssh -i /tmp/remote-key remote_user@remote_host

Answered By - Pierre B.