[SOLVED] SIGTERM not trapped by AWS ECS docker container

Issue

I have the following ECS container definition

ContainerDefinitions:
        - Name: myfluentd
          Essential: true
          Image: !Ref DockerImage
          MountPoints:
            - SourceVolume: efs-volume
              ContainerPath: '/fluentd/buffers'
              ReadOnly: false
          PortMappings:
            - ContainerPort: 24224
              HostPort: 24224
              Protocol: tcp
          EntryPoint:
            - "/var/lib/twistlock/defender.sh"
            - "defender"
            - "entrypoint"
            - "tini"
            - "--"
            - "/bin/entrypoint.sh"
          Command:
            - fluentd
          VolumesFrom:
            - ReadOnly: false
              SourceContainer: "TwistlockDefender"

The ENTRYPOINT is

["/var/lib/twistlock/fargate/fargate_defender.sh","fargate","entrypoint","tini","--","/bin/entrypoint.sh"]

and the contents of entrypoint.sh

#!/bin/sh

#source vars if file exists
DEFAULT=/etc/default/fluentd

if [ -r $DEFAULT ]; then
    set -o allexport
    . $DEFAULT
    set +o allexport
fi

# If the user has supplied only arguments append them to `fluentd` command
if [ "${1#-}" != "$1" ]; then
    set -- fluentd "$@"
fi

# If user does not supply config file or plugins, use the default
if [ "$1" = "fluentd" ]; then
    if ! echo $@ | grep ' \-c' ; then
       set -- "$@" -c /fluentd/etc/${FLUENTD_CONF}
    fi

    if ! echo $@ | grep ' \-p' ; then
       set -- "$@" -p /fluentd/plugins
    fi
fi

. /bin/env.sh

task_cleanup() {
    echo "Deleting files"
    rm -rf /tmp/*
    exit
}
trap task_cleanup SIGTERM SIGINT

exec "$@"

When I stop a ECS task it issues the SIGTERM to the container and I expect it to executed the task_cleanup(), however the signal is not caught.

I logged into the container to check the processes

PID   USER     TIME  COMMAND
    1 fluent    0:00 sh /var/lib/twistlock/defender.sh defender entrypoint tini -- /bin/entrypoint.sh fluentd
    7 fluent    0:00 /var/lib/twistlock/defender defender entrypoint tini -- /bin/entrypoint.sh fluentd
   16 root      0:00 /managed-agents/execute-command/amazon-ssm-agent
   22 fluent    0:00 /sbin/tini -- /bin/entrypoint.sh fluentd
   29 fluent    0:02 {fluentd} /usr/bin/ruby /usr/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins
   72 root      0:00 /managed-agents/execute-command/ssm-agent-worker
   90 fluent    1:06 /usr/bin/ruby -Eascii-8bit:ascii-8bit /usr/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins --under-supervisor
  546 root      0:00 /managed-agents/execute-command/ssm-session-worker ecs-execute-command-03931001c6df08625
  556 root      0:00 /bin/bash
  870 root      0:00 ps aux 

What am I missing? why is the signal not getting trapped?

UPDATE: The ENTRYPOINT kicks off 2 containers and I think the signal is only sent to the first container - when I removed the first container, I was able to see the cleanup working. Any thoughts how to handle this?

Solution

Per the POSIX documentation for exec (bolding mine):

If exec is specified with command, it shall replace the shell with command without creating a new process. ...

Your shell process and the signal handler you created with trap no longer exists once you call exec "$@".

This will probably do what you want, but it's totally untested and I haven't analyzed what the string "$@" that you've constructed:

   .
   .
   .
task_cleanup() {
    echo "Deleting files"
    rm -rf /tmp/*
}
trap task_cleanup SIGTERM SIGINT

# background the task
"$@" &
pid=$!

wait $pid

kill $pid

exit

Instead of replacing the shell with the command held in "$@", this would run the command as a child process of the shell.

Answered By - Andrew Henle

Answer Checked By - Marilyn (WPSolving Volunteer)