Issue
We have two GKE private clusters, where access is only possible via ssh proxy.
In the local environment this works this way:
Open a ssh connection with port 8888 forwarding to the bastion host
gcloud compute ssh dev-cluster-bastion --project client-dev --zone xxxx -- -L 8888:127.0.0.1:8888
In another session
HTTPS_PROXY=localhost:8888 kubectl get pods
This returns the list of the running pods.
When we do this inside a bitbucket pipeline, then the ssh connects, but then closes and the kubectl
call fails.
The message from the ssh connection is:
Pseudo-terminal will not be allocated because stdin is not a terminal
So the portforwarding is closed too.
Adding -fN
to the ssh start does not help, the port forwarding is not working.
gcloud compute ssh dev-cluster-bastion --project client-dev --zone xxxx -- -fN -L 8888:127.0.0.1:8888
ssh then tells me client_loop: send disconnect: Connection reset by peer
Any ideas how to open the port 8888 tcp tunnel inside a bitbucket pipeline, so we can send the kubectl commands to the cluster?
Solution
The recommended way to use Google Cloud APIs in a non-interactive setting (like Bitbucket pipelines, GitHub actions, etc.) is to:
- Create a Service Account , and refer to this link for service account creation.
- Create a key file for this service account and download it. Kindly refer to the link for file creation.
- Grant the needed IAM permissions to this account to be able to SSH into the Bastion host
- Make the key file available to the pipeline environment securely (most CI/CD systems allow a way to encrypt secret files or sensitive environment variables)
- In the pipeline, use
gcloud auth activate-service-account --key-file=...
to makegcloud
use the service account - Now run
gcloud compute ssh
in the pipeline and use SSH port forwarding as usual.
Note that:
- steps 1.--4. are one time only;
- steps 5. and 6. are part of the pipeline and will run every time the pipeline is run
- The only difference being the presence of the
-fN
flags, which execute SSH in the background and make it run no command on the remote host (i.e. only do port forwarding).
You can also refer to Service account creation and Get started with Bitbucket-pipeline for more information.
Answered By - Ramesh kollisetty