[SOLVED] Remove mutiline string after equal sign

Issue

I am trying to get IAM policies from AWS and add it to a text file in the below specific format. I want to delete everything after policy= in the file before end of the "}" bracket.

This is the text file sample I have. But the original file can have multiple instances of example_policy in the same file.

"example1_policy" {
    name="example"
    policy=jsonencode(
        {
            Statement=[
                {
                    Action=[
                        s3:*
                    ]
                    Effect="Allow"
                },
            ]
            Version="2012-10-17"
        }
    )
}
"example2_policy" {
    name="example2"
    policy=jsonencode(
        {
            Statement=[
                {
                    Action=[
                        s3:*
                    ]
                    Effect="Allow"
                },
            ]
            Version="2012-10-17"
        }
    )
}
"example3_policy" {
    name="example3"
    policy=jsonencode(
        {
            Statement=[
                {
                    Action=[
                        s3:*
                    ]
                    Effect="Allow"
                },
            ]
            Version="2012-10-17"
        }
    )
}

Expected Output:

"example1_policy" {
    name="example1"
    policy=
}
"example2_policy" {
    name="example2"
    policy=
}
"example3_policy" {
    name="example3"
    policy=
}

or

"example1_policy" {
    name="example1"
    policy=<placeholder>
}
"example2_policy" {
    name="example2"
    policy=<placeholder>
}
"example3_policy" {
    name="example3"
    policy=<placeholder>
}

As per @Wiktor's comment I tried out this command

sed -i '/policy=/,/^\s*)\s*$/d' test.txt

Output: policy= should remain intact.

"example_policy" {
    name="example"
}

Solution

You can use the following GNU sed command:

sed -i '/policy=/,/^\s*)\s*$/{/policy=/!d};s/\(policy=\).*/\1<placehlder>/' file

See the online demo. Details:

• /policy=/,/^\s*)\s*$/ - finds blocks of lines between a line with policy= and a line that contains only a ) char enclosed with zero or more whitespaces
• {/policy=/!d} - prevents the first line in the found block to be removed and removed the other line(s)
• s/\(policy=\).*/\1<placehlder>/ - replaces all after policy= with <placeholder>.

If there is a need to match policy=, then any chars up to ( and then up to the next balanced ) char, you can use perl command like

perl -0777 -i -pe 's/^(\h*policy=)[^()]*(\((?:[^()]++|(?2))*\))/$1<placehlder>/gm' file

Details:

• -0777 - read the file into a single string so that line breaks and the lines
• -i - inline file change
• ^ - start of a line (due to m flag)
• (\h*policy=) - Group 1 ($1): zero or more horizontal whitespaces
• [^()]* - zero or more chars other than ( and )
• (\((?:[^()]++|(?2))*\)) - Group 2 (for recursion): \( - a ( char, (?:[^()]++|(?2))* - zero or more sequences of one or more chars other than ( and ) or the whole Group 2 pattern recursed, and \) matches a ) char
• $1<placehlder> - the replacement is Group 1 + <placeholder> string

See the online demo and the regex demo:

#!/bin/bash
s='"example1_policy" {
    name="example"
    policy=jsonencode(
        {
            Statement=[
                {
                    Action=[
                        s3:*
                    ]
                    Effect="Allow"
                },
            ]
            Version="2012-10-17"
        }
    )
}
"example2_policy" {
    name="example2"
    policy=jsonencode(
        {
            Statement=[
                {
                    Action=[
                        s3:*
                    ]
                    Effect="Allow"
                },
            ]
            Version="2012-10-17"
        }
    )
}
"example3_policy" {
    name="example3"
    policy=jsonencode(
        {
            Statement=[
                {
                    Action=[
                        s3:*
                    ]
                    Effect="Allow"
                },
            ]
            Version="2012-10-17"
        }
    )
}'
perl -0777 -pe 's/^(\h*policy=)[^()]*(\((?:[^()]++|(?2))*\))/$1<placehlder>/gm' <<< "$s"

Output:

"example1_policy" {
    name="example"
    policy=<placehlder>
}
"example2_policy" {
    name="example2"
    policy=<placehlder>
}
"example3_policy" {
    name="example3"
    policy=<placehlder>
}

Answered By - Wiktor Stribiżew

Answer Checked By - Pedro (WPSolving Volunteer)