Issue
I wrote a script that checks the kubernetes pods and in case of hanging it must write pod logs and delete it. Kube master is on another server and I try to connect to it by ssh. It's working good when I try to start script locally on kube master server, but when I try to enter the same command as ssh argument, I get pod deleted, but this pod's logs weren't written and there's no errors, so I don't understand what I do wrong. I use python 3.6, access permissions 777 for scripts and directories in which they are located.
I know about possibility of doing this using k8s api but I don't know how to do it now. I am still learning and will try to realize it a bit later but the system should work now.
PS. The reason that I do this is that I want to add 'check_radio' to cron and forget about hanging pods... PPS. I am not master-pro at python, just a regular sysadmin that wants to become a devops, so if you have some ideas how to optimize my work you're welcome.
Here is the script that is on kube master (kube_reload.py):
#!/bin/python3
import sys
import os
import re
import subprocess as sp
from contextlib import redirect_stdout as r_stdout
marker = False
pod_nums = ''
pod_nums_int = []
for_reload = []
get_pods = sp.check_output(['kubectl', 'get', 'pods'])
get_pods = str(get_pods)
get_pods = get_pods.split()
pods = []
'''write garbage pod names in array'''
for word in get_pods:
if 'radio-hls' in word:
pods.append(word)
'''make usual pod names from garbage names'''
for num in range(len(pods)):
while marker == False:
if pods[num][:5] != 'radio':
pods[num] = pods[num][1:]
if pods[num][:5] == 'radio':
break
'''Function that lists all pods'''
def pod_list():
sp.run('clear')
print('PODS LIST:\n')
for num in range(len(pods)):
print(num+1, '.\t', pods[num])
print(len(pods)+1, '.\t Reload all')
'''Function for recursion in try-except statement in 'input_nums()' function'''
def return_for_except():
pod_list()
print('Error input. Try again\n\n')
input_nums()
'''Function that asks which pods you want to reload'''
def input_nums():
pod_nums = str(input('Select which pods you want to reload (type all numbers with a space):\nExample: 1 2 3 10\n\n'))
pod_nums = pod_nums.split()
try:
global pod_nums_int
pod_nums_int = [eval(i) for i in pod_nums]
except:
return return_for_except()
'''Function to write pod logs to a file'''
def write_logs():
global for_reload
if len(pods)+1 in pod_nums_int:
all_reload = input('You selected "Reload all". To continue, press Enter\n')
if all_reload == '':
for i in range(len(pods)):
with open (f'{pods[i-1][10:-17]}.log', 'w') as pod_log:
sp.run(['kubectl', 'logs', f'{pods[i-1]}'], stdout=pod_log)
print(f'{pods[i-1]} logged successfully')
for_reload.append(i)
else:
print('Something went wrong')
quit()
else:
for i in pod_nums_int:
with open (f'{pods[i-1][10:-17]}.log', 'w') as pod_log:
sp.run(['kubectl', 'logs', f'{pods[i-1]}'], stdout=pod_log)
print(f'{pods[i-1]} logged successfully')
for_reload.append(i)
'''Function that reloads streams'''
def reload_pods(for_reload, pods):
for i in for_reload:
sp.run(['kubectl', 'delete', 'pod', f'{pods[i-1]}'])
print(f'{pods[i-1]}', 'deleted')
'''Start'''
'''manual (with no arguments)'''
if len(sys.argv) == 1:
pod_list()
input_nums()
write_logs()
reload_pods(for_reload, pods)
sp.run(['kubectl', 'get', 'pods'])
print()
sp.run(['ls', '-lah'])
'''auto (from nginx srver)'''
if len(sys.argv) > 1:
for arg in sys.argv:
if arg == './kube_reload.py':
continue
else:
for pod in pods:
if arg in pod:
#write logs
with open (f'{arg}.log', 'w') as log:
sp.run(['kubectl', 'logs', f'{pod}'], stdout=log)
#reload pods
sp.run(['kubectl', 'delete', 'pod', f'{pod}'])
else:
continue
Here is the script from another server (check_radio):
#!/bin/python3
import requests as r
import subprocess as sp
import sys
'''IN CASE OF ADDING ADDITIONAL STREAM ADD IT TO "streams" '''
streams = [
'gold',
'tophits',
'worldchart',
'ukraine',
'rock',
'chill',
'rap',
'retromusic',
'elektro',
'sport',
'radionv',
'edyninov',
'hromadske'
]
'''IF IF THERE IS NO NEED TO CHECK SOME STREAMS YOU CAN ADD IT TO "streams_not_to_check" '''
streams_not_to_check = [
'radionv',
'edyninov',
'hromadske'
]
streams_to_delete = []
#CLUSTER_API = 'https://host_ip:6443'
#auth_header = 'Authorization: '
for stream in streams:
check_stream = r.get(f'https://host.host/stream/{stream}/status/health').json()
if check_stream['metadata'] == 'UNHEALTHY':
streams_to_delete.append(stream)
for stream in streams_not_to_check:
if stream in streams_to_delete:
streams_to_delete.remove(stream)
print(streams_to_delete)
if len(streams_to_delete) >= 1:
for stream in streams_to_delete:
sp.Popen(f'ssh developer@radio1 python3 ~/deekly/kube_reload.py {stream}', shell=True).communicate()
I try this from nginx server
./check_radio
and get this
[developer@radio-lb1 deekly]$ ./check_radio
['rap', 'rock']
pod "radio-hls-rap-f4b86bd77-jpmr4" deleted
pod "radio-hls-rock-57fc8fcd64-m54k5" deleted
[developer@radio-lb1 deekly]$
and this on kube server
[developer@radio1 deekly]$ ls -la
total 8
drwxrwxrwx 2 developer developer 28 Aug 4 20:46 .
drwx------ 14 developer developer 4096 Aug 4 20:26 ..
-rwxrwxrwx 1 developer developer 2850 Aug 4 19:51 kube_reload.py
[developer@radio1 deekly]$
but when I try this from kube master
./kube_reload.py rap rock
or this (doesn't matter)
python3 kube_reload.py rap rock
I get this
[developer@radio1 deekly]$ ./kube_reload.py rap rock
pod "radio-hls-rap-f4b86bd77-4bvcs" deleted
pod "radio-hls-rock-57fc8fcd64-g5j96" deleted
[developer@radio1 deekly]$ ls -la
total 32
drwxrwxrwx 2 developer developer 59 Aug 5 16:38 .
drwx------ 14 developer developer 4096 Aug 4 20:26 ..
-rwxrwxrwx 1 developer developer 2850 Aug 4 19:51 kube_reload.py
-rw-rw-r-- 1 developer developer 8303 Aug 5 16:38 rap.log
-rw-rw-r-- 1 developer developer 8345 Aug 5 16:38 rock.log
Solution
Your kube_reload.py script writes its logs to its current working directory:
with open (f'{arg}.log', 'w') as log:
...
When you're running kube_reload.py interactively, you're running it from this "deekly" directory, so "deekly" is the process's working directory and the logs are written there:
[developer@radio1 deekly]$ ./kube_reload.py rap rock
...
[developer@radio1 deekly]$ ls -la
...
-rw-rw-r-- 1 developer developer 8303 Aug 5 16:38 rap.log
-rw-rw-r-- 1 developer developer 8345 Aug 5 16:38 rock.log
When this check_radio invokes kube_reload.py through ssh, the kube_reload.py process has this "developer" user's home directory as its working directory, not the "deekly" directory:
sp.Popen(f'ssh developer@radio1 python3 ~/deekly/kube_reload.py {stream}', shell=True)...
kube_reload.py will presumably write its log files to this "developer" user's home directory, not the "deekly" subdirectory.
If you want the logs to be written to this "deekly" directory, you should do one of three things:
Modify
kube_reload.pyto put the logs where you want them, instead of its current working directory.Modify
kube_reload.pyto change its working directory the desired directory before opening the logs.Modify
check_radioto invokekube_reload.pyon the remote host with the correct working directory. I'm not a python programmer so I can't give you the exact python syntax. But a command such as the following should do it:ssh developer@radio1 'cd deekly && python3 ./kube_reload.py args...'
You will have to do whatever is necessary in python to escape those single quotes, so that they're present in the command actually being executed.
Answered By - Kenster Answer Checked By - Mildred Charles (WPSolving Admin)