[SOLVED] Fluend does not automatically add the current system time in Json Parser

Issue

Fluentd Experts and Users!

Currently we have met an issue in using Fluentd to parse json format log. Fluentd does not automatically add the current system time to the parsing result, although I have configured time_key and keep_time_key according to the documentation.

The example of our log is, {"host": "204.48.112.175", "user-identifier": "-", "method": "POST", "request": "/synthesize/initiatives/integrated", "protocol": "HTTP/2.0", "status": 502, "bytes": 10272} and you can see that there is no time field in it.

But there is no system current time in the parsed log output (the output is in stdout (debug mode) )：

loghub_s3: {"host":"204.48.112.175","user-identifier":"-","method":"POST","request":"/synthesize/initiatives/integrated","protocol":"HTTP/2.0","status":502,"bytes":10272,"referer":"http://www.centralenable.name/user-centric/reintermediate/synergistic/e-business","s3_bucket":"loghub-logs-691546483958","s3_key":"json/json-notime.json"}

And my config file is:

<system>
  log_level debug
</system>

<match loghub_s3>
  @type stdout
  @id output_stdout
</match>

<source>
  @type s3
  tag loghub_s3

  s3_bucket loghub-logs-691546483958
  s3_region us-east-1
  store_as json
  add_object_metadata true
  <instance_profile_credentials>
    ip_address 169.254.169.254
    port       80
  </instance_profile_credentials>

  <sqs>
    queue_name loghub-fluentd-dev
  </sqs>
  
  <parse>
    @type json
    time_type string
    time_format %d/%b/%Y:%H:%M:%S %z
    time_key time
    keep_time_key true
  </parse>
</source>

Other informations:

• Fluentd version: 1.14.3
• TD Agent version: 4.3.0
• fluent-plugin-s3 version: 1.6.1
• Operating system: Amazon Linux2
• Kernel version: 5.10.102-99.473.amzn2.x86_64

And we have used the s3-input-plugin: https://github.com/fluent/fluent-plugin-s3

Can anyone help us to check if our configuration is wrong. And I’m not sure if this is a Fluentd issue, or Plugin issue.

Thanks a lot in advance!

Solution

As mentioned in the comments, fluentd does not create a time/timestamp field unless configured otherwise. You can inject this field under filter or match section.

Here's an example with the sample input and stdout output plugins:

fluentd: 1.12.3

fluent.conf

<source>
  @type sample
  @id in_sample
  sample {"k":"v"}
  tag sample
</source>

<match sample>
  @type stdout
  @id out_stdout
  <inject>
    time_key timestamp
    time_type string
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </inject>
</match>

Run fluentd:

fluentd -c ./fluent.conf

fluentd logs

2022-04-10 08:46:26.053278947 +0500 sample: {"k":"v","timestamp":"2022-04-10T08:46:26.053278947Z"}
2022-04-10 08:46:27.056770340 +0500 sample: {"k":"v","timestamp":"2022-04-10T08:46:27.056770340Z"}
2022-04-10 08:46:28.059998159 +0500 sample: {"k":"v","timestamp":"2022-04-10T08:46:28.059998159Z"}

Answered By - Azeem

Answer Checked By - Mary Flores (WPSolving Volunteer)