[SOLVED] How to parse list using shell and update set of yaml files

Issue

I have a list of values in a text file like so:

service-1 | a /
service-1 | b /path2/
service-2 | b /path3/
service-2 | b /path4/

I also have a YAML for each of the services like service-1.yaml, service-2.yaml, etc. Each one has an ingress path where I need to put the path:

ingress: 
  instances: 
    - auth: a
      path: 
        <to be filled in>
        <to be filled in>
    - auth: b
      path: 
        <to be filled in>
        <to be filled in>

I want to parse the text file in Shell and then update the appropriate fields in the YAML. I know I can parse the first and second parts using cut, e.g.,

echo "service-1 | a /" | cut -d "|" -f1 or echo "service-1 | a /" | cut -d "|" -f2, but how can I determine which service is which and which paths go where in the YAML? Any suggestions are appreciated.

Solution

I would solve it with the YAML processor kislyuk/yq. The following code shows the solution for just one file. You can put it in a bash loop that iterates over all files.

SERVICE='service-1'   # or 'service-2'
PATHS='servicePaths.txt'
FILE_IN="$SERVICE.yaml"
FILE_OUT="$SERVICE.out.yaml"

yq -y --arg service $SERVICE --rawfile paths $PATHS '
def getPath($auth):
  $paths / "\n"                                                   # split lines by "\n"
  | map(. / "|"                                                   # split each line by "|"
  | map(sub("^\\s+";"") | sub("\\s+$";"")))                       # trim strings
  | map(select(any))                                              # remove empty arrays (caused by empty lines in file servicePaths.txt)
  | map([.[0], (.[1] / " ")[0], (.[1] / " ")[1]])                 # split auth/path by " "
  | del(.[] | select(.[0] == $service and .[1] == $auth | not))   # remove definition that does not match $service and $auth
  | .[0][2] // "path undefined";                                  # return path of first match

.ingress.instances |= map(.path = getPath(.auth))' "$FILE_IN" > "$FILE_OUT"

File service-1.out.yaml

ingress:
  instances:
    - auth: a
      path: /
    - auth: b
      path: /path2/

File service-2.out.yaml

ingress:
  instances:
    - auth: a
      path: path undefined
    - auth: b
      path: /path3/

Remarks

• yq offers an option -i for inplace editing of files, if you want to replace the original template file instead of using $FILE_OUT.
• the function getPath($auth) build up a lookup for the paths from the file servicePaths.txt and then selects the correct path for $service and $path
• .ingress.instances |= map(...) in the last line updates the paths of all services (|= is the update operator)
• the solution is fail safe:
  • if no path for service, auth is defined in servicePaths.txt then "path undefined" is inserted.
  • if more than one path is defined, the first path from servicePaths.txt is inserted.

Variation

If it is valid to have multiple definitions of paths for the same service/auth, use this little modification:

SERVICE='service-1'   # or 'service-2'
PATHS='servicePaths.txt'
FILE_IN="$SERVICE.yaml"
FILE_OUT="$SERVICE.out.yaml"

yq -y --arg service $SERVICE --rawfile paths $PATHS '
def getPath($auth):
  $paths / "\n"
  | map(. / "|"
  | map(sub("^\\s+";"") | sub("\\s+$";"")))
  | map(select(any))
  | map([.[0], (.[1] / " ")[0], (.[1] / " ")[1]])
  | del(.[] | select(.[0] == $service and .[1] == $auth | not))
  | map(.[2]);                               # <-- line changed

.ingress.instances |= map(.path = getPath(.auth))' "$FILE_IN" > "$FILE_OUT"

File service-1.out.yaml

ingress:
  instances:
    - auth: a
      path:
        - /
    - auth: b
      path:
        - /path2/

File service-2.out.yaml

ingress:
  instances:
    - auth: a
      path: []
    - auth: b
      path:
        - /path3/
        - /path4/

Answered By - jpseng

Answer Checked By - Senaida (WPSolving Volunteer)