[SOLVED] CyberArk ITATS004E Authentication failure for User in python script

Issue

I'm trying to implement a python script that executes local bash scripts or simple commands on remote CyberArk machines. Here is my code:

if __name__ == '__main__':
    for ip in IP_LIST:
        bash_cmd = f"ssh -o stricthostkeychecking=no {USER}%{LOCAL_USER}%{ip}@{PROXY} 'bash -s' < {BASH_SCRIPT}"
        exit_code = subprocess.call(bash_cmd, shell=True)
        print(exit_code)
        bash_cmd = f"scp {USER}%{LOCAL_USER}%{ip}@{PROXY}:server_info_PY.txt ."
        exit_code = subprocess.call(bash_cmd, shell=True)
        print(exit_code)

The main problem is that i get this CyberArk authentication error most of the times, but not always, so it's kind of random and i don't know why:

PSPSD072E Perform session error occurred. Reason: PSPSD033E Error receiving PSM For SSH server 
response (Extra information: [289E [4426b00e-cc44-11ec-bca1-005056b74f99] Failed to impersonate as 
user <user>. Error: [ITATS004E Authentication failure for User <user>.

In this case the ssh exit code is 255, but if i check sshd service logs on the remote machine, there are no errors. I even tried with the os library to execute bash commands, but I got same result.
I was thinking of multiple ssh sessions hanging after executing this script a lot of times, but on the remote machine i only find the one i'm using.
Could someone explain what is happening or do you have any ideas?

Notes: I don't have any access to the PSM server, that is stored in the variable PROXY

Edit 1: I tried to use Paramiko library to create the ssh connection, but i get an authentication error related to Paramiko and not related to CyberArk. I also tried Fabric library which is based on Paramiko, so it didn't work.
If i try to run the same ssh command manually from my terminal it works and i can see that it first connects to the PROXY and then to the ip of the remote machine. From the script side it looks like he can't even connect to the PROXY because of the CyberArk authentication error.

Edit 2: I logged some informations about all commands running when executing the python script and i found out that the first command which is launched is /bin/sh/ -c plus the ssh string:

/bin/sh -c ssh <user>@<domain>

Could be this the main problem? The prepending of /bin/sh -c? Or it's a normal behaviour when using subprocess library? There is a way to just execute the ssh command without this prepend?

Edit 3: I removed shell=True but got same auhtentication error. So, if i execute manually the ssh command i get no error, but if it is executed from the python script i get the error, but i can't find any contradiction at proccess level using ps aux in both cases.

Solution

Since the authentication error is kind of random, I just added a while loop that resets known_hosts file and runs the ssh command for n retries.

succeeded_cmd_exec = False
retries = 5
while not succeeded_cmd_exec:
    if retries == 0:
        break
    bash_cmd = f'ssh-keygen -f "{Configs.KNOWN_HOSTS}" -R "{Configs.PROXY}"'
    _, _, exit_code = exec_cmd(bash_cmd)
    if exit_code == 0:
        radius_password = generate_password(Configs.URI, Configs.PASSWORD)
        bash_cmd = f"sshpass -p \"{radius_password}\" ssh -o stricthostkeychecking=no {Configs.USER}%{Configs.LOCAL_USER}%{ip}@{Configs.PROXY} 'ls'"
        stdout, stderr, exit_code = exec_cmd(bash_cmd)
        if exit_code == 0:
            print('Output from SSH command:\n')
            print(stdout)
            succeeded_cmd_exec = True
        else:
            retries = retries - 1
            print(stdout)
            print('SSH command failed, retrying ... ')
            print('Sleeping 15 seconds')
            time.sleep(15)
    else:
        print('Reset known hosts files failed, retrying ...')
if retries == 0 and not succeeded_cmd_exec:
    print(f'Failed processing IP {ip}')

The exec_cmd function is defined like this:

def exec_cmd(bash_cmd: str):
    process = subprocess.Popen(bash_cmd, shell=True, executable='/bin/bash', stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()
    process.wait()
    return stdout.decode('utf-8'), stderr.decode('utf-8'), process.returncode

Answered By - Alexandru Ropotica

Answer Checked By - Clifford M. (WPSolving Volunteer)