Issue
When using gitlab-ci to deploy a file to a server using scp, the scp command fails only when the pipeline job is triggered from a tag
image: registry.gitlab.com/ubuntu-1804-qt5-builder:master
stages:
- debian-package
- deploy
variables:
RELEASE_NAME: ${CI_PROJECT_NAME}_${CI_COMMIT_REF_NAME}.${CI_PIPELINE_ID}
before_script:
## Install ssh-agent if not already installed, it is required by Docker.
## (change apt-get to yum if you use an RPM-based image)
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
## Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
## We're using tr to fix line endings which makes ed25519 keys work
## without extra base64 encoding.
## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
## Create the SSH directory and give it the right permissions
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
debian-package:
stage: debian-package
before_script:
- apt-get update
- apt-get install -y -f debhelper
script:
- 'export DEB_PACKAGE=${CI_PROJECT_NAME}_$(echo ${CI_COMMIT_REF_NAME} | sed "s/master/0.0/g")-${CI_PIPELINE_ID}_all.deb'
- 'echo BUILDING: ${DEB_PACKAGE}'
- ./prepdebian.sh ${CI_PROJECT_NAME} "$(echo ${CI_COMMIT_REF_NAME} | sed 's/master/0.0/g')" "${CI_PIPELINE_ID}" "${GITLAB_USER_NAME} <${GITLAB_USER_EMAIL}>" "${CI_COMMIT_SHA}"
- dpkg-buildpackage -us -uc -b
- cp ../${DEB_PACKAGE} .
artifacts:
name: ${RELEASE_NAME}
paths:
- ${CI_PROJECT_NAME}_$(echo ${CI_COMMIT_REF_NAME} | sed 's/master/0.0/g')-${CI_PIPELINE_ID}_all.deb
deploy-staging:
stage: deploy
dependencies:
- debian-package
script:
- 'export DEB_PACKAGE=${CI_PROJECT_NAME}_$(echo ${CI_COMMIT_REF_NAME} | sed "s/master/0.0/g")-${CI_PIPELINE_ID}_all.deb'
- 'echo DEPLOYING TO STAGING: ${DEB_PACKAGE}'
- scp -vvv ${DEB_PACKAGE} [email protected]:incomming/debs/
only:
- tags
I have set the scp command to provide verbose output (-vvv) The output from scp shows
Executing: program /usr/bin/ssh host server.com, user user, command scp -v -t incomming/debs/
OpenSSH_7.6p1 Ubuntu-4, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "server.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to server.com [11.22.33.44] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4
debug1: match: OpenSSH_7.6p1 Ubuntu-4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to server.com:22 as 'user'
debug3: hostkeys_foreach: reading file "/user/.ssh/known_hosts"
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ooGb0IBrr6wGPwJo23g+eMW9S6lkIOQ5lx0XL4PReeg
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug1: read_passphrase: can't open /dev/tty: No such device or address
Host key verification failed.
lost connection
If you replace the pipeline job trigger from
only:
- tags
to
except:
- tags
Then the scp command succeeds
This does not make any sense, could some one shed some light on why the deployment fails only when using only: tags
Solution
I am answering my own question as I have finally resolved the issue.
I set the variable CI_DEBUG_TRACE: "true". This allowed a lot more information about the environment to be output. When doing this I noticed that the SSH_PRIVATE_KEY was being set; however, the SSH_KNOWN_HOSTS was not. This was only the case if the following was used.
only:
- tags
The variables SSH_KNOWN_HOSTS and SSH_PRIVATE_KEY where CI variables.
The variable SSH_KNOWN_HOSTS was mistakenly set to be protected. The strange thing which I still do not understand, event though the variable was set as protected , the environment was set to All environments (*)
Answered By - Simon Black