[SOLVED] JSON Parsing with awk

Issue

i have huge chunk of server logs. The structure as below. I am trying to extract number unique requestIds values from the json file. requestId is a key and there could be many number of same requestId values. I want to extract unique requestIds. The order if the fields can be different in the the inputfile.

input.txt

{
   "timestamp":"2021-05-06T20:33:47.986Z",
   "logger_name":"c.a.p.r.engine.Taxengine",
   "thread_name":"http-nio-8080-exec-18",
   "level":"INFO",
   "serviceArchPath":"rb.amxp.Taxfiler",
   "process":"NA",
   "message":"Deployment successfully",
   "requestId":"CFLGftt2hfrBhr6Ltyjdx"
},
{
   "newField":"tempfield", 
   "timestamp":"2021-05-06T20:33:47.986Z",
   "logger_name":"c.a.p.r.engine.Taxengine",
   "thread_name":"http-nio-8080-exec-18",
   "level":"INFO",
   "serviceArchPath":"rb.amxp.Taxfiler",
   "process":"NA",
   "message":"Deployment successfully",
   "requestId":"DELGongDKGbehr6Lsak"
},
{
   "timestamp":"2021-05-06T20:33:47.986Z",
   "logger_name":"c.a.p.r.engine.Taxengine",
   "thread_name":"http-nio-8080-exec-18",
   "level":"INFO",
   "serviceArchPath":"rb.amxp.Taxfiler",
   "process":"NA",
   "message":"Deployment successfully",
   "requestId":"CFLGftt2hfrBhr6Ltyjdx"
},

Can i get help to Log number of requestIds and display them. Thanks for your help.

Solution

Try the following:

awk 'BEGIN { FS = "\"" } /requestId/ { print $(NF - 1) }' input.txt

This basically uses " as the field separator and extract the penultimate field if it sees requestId.

Ouptut:

CFLGftt2hfrBhr6Ltyjdx
DELGongDKGbehr6Lsak
CFLGftt2hfrBhr6Ltyjdx

You can then pipe it to sort --unique if needed.

But I am not quite sure whether awk is the tool for this task. As the comment suggests, other simpler tools or json-targeted tools might be better.

Answered By - j1-lee