[SOLVED] Extract and group by type env variables from k8s deployments with jq

Issue

I am trying to extract the list of k8s deployment environment variables and merge the environment variables into a group of categories using jq and regex.

Example

k8s deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test
  name: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - name: server
        env:
        - name: DB_MAINDB_HOST
          value: maindb.example.com
        - name: DB_MAINDB_DATABASE
          value: test
        - name: DB_MAINDB_USERNAME
          value: username
        - name: DB_MAINDB_PASSWORD
          value: password
        - name: ES_HOST
          value: es-client.example.com
        - name: ES_INDEX
          value: test
        - name: REDIS_HOST
          value: redis.example.com
        image: bustbox

Desired output

{
  "namespace": "default",
  "name": "test",
  "databases": [
    {
      "name": "DB_MAINDB_DATABASE",
      "value": "test"
    },
    {
      "name": "DB_MAINDB_HOST",
      "value": "maindb.example.com"
    },
    {
      "name": "DB_MAINDB_PASSWORD",
      "value": "password"
    },
    {
      "name": "DB_MAINDB_USERNAME",
      "value": "username"
    }
  ],
  "redis": [
    {
      "name": "REDIS_HOST",
      "value": "redis.example.com"
    }
  ],
  "elasticsearch": [
    {
      "name": "ES_HOST",
      "value": "es-client.example.com"
    },
    {
      "name": "ES_INDEX",
      "value": "test"
    }
  ]
}

I tried to do this with the following command:

kubectl get deployments test -o json | jq -r '. |
        {namespace: .metadata.namespace,
        name: .metadata.name,
        databases: (.spec.template.spec.containers[].env | [ map(.) | .[] | select(.name | contains ("DB"))] | sort_by(.name)),
        redis: (.spec.template.spec.containers[].env | [ map(.) | .[] | select(.name | contains ("REDIS"))] | sort_by(.name)),
        elasticsearch: (.spec.template.spec.containers[].env | [ map(.) | .[] | select(.name | test("^(ES_).") or contains ("ELASTIC"))] | sort_by(.name)),
        }'

I am looking for a slightly more elegant solution to my problem and, in addition, want to have the ability to manage a mapping of the categories with the relevant regex as additional input to the command.

Thanks!

Solution

Here's one rewrite with several duplicates factored out. Decide for yourself if this meets your requirement for elegance:

(
  .metadata | {namespace, name}
) + (
  .spec.template.spec.containers[].env | {
    databases: map(select(.name | contains ("DB"))),
    redis: map(select(.name | contains ("REDIS"))),
    elasticsearch: map(select(.name | test("^(ES_).") or contains ("ELASTIC")))
  }
  | map_values(sort_by(.name))
)

Additionally, you could bring the sort up front to sort the unfiltered array just once:

(
  .metadata | {namespace, name}
) + (
  .spec.template.spec.containers[].env | sort_by(.name) | {
    databases: map(select(.name | contains ("DB"))),
    redis: map(select(.name | contains ("REDIS"))),
    elasticsearch: map(select(.name | test("^(ES_).") or contains ("ELASTIC")))
  }
)

Answered By - pmf

Answer Checked By - Mary Flores (WPSolving Volunteer)