[SOLVED] AWS ECS Scheduled task not running when released by CI/CD

Issue

I'm experiencing a very annoying problem. I created a CI/CD pipelines using AWS CodePipeline and CloudFormation.

This is the template.yml used by CloudFormation to create a ScheduledTask on ECS.

AWSTemplateFormatVersion: "2010-09-09"
Description: Template for deploying a ECR image on ECS

Resources:
  VPC:
    Type: "AWS::EC2::VPC"
    Properties:
      CidrBlock: "10.0.0.0/16"
      EnableDnsSupport: true
      EnableDnsHostnames: true
      InstanceTenancy: default

  Subnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [0, !GetAZs ""]
      CidrBlock: !Sub "10.0.0.0/20"
      MapPublicIpOnLaunch: true

  Subnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [1, !GetAZs ""]
      CidrBlock: !Sub "10.0.32.0/20"
      MapPublicIpOnLaunch: true

  InternetGateway:
    Type: "AWS::EC2::InternetGateway"

  VPCGatewayAttachment:
    Type: "AWS::EC2::VPCGatewayAttachment"
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref VPC

  RouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VPC

  RouteTableAssociation1:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      SubnetId: !Ref Subnet1
      RouteTableId: !Ref RouteTable

  RouteTableAssociation2:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      SubnetId: !Ref Subnet2
      RouteTableId: !Ref RouteTable

  InternetRoute:
    Type: "AWS::EC2::Route"
    DependsOn: VPCGatewayAttachment
    Properties:
      GatewayId: !Ref InternetGateway
      RouteTableId: !Ref RouteTable
      DestinationCidrBlock: "0.0.0.0/0"

  ECSCluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: "SLAComputation"

  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: ecs-services
      Subnets:
        - !Ref "Subnet1"
        - !Ref "Subnet2"
      SecurityGroups:
        - !Ref LoadBalancerSecurityGroup

  LoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn: !Ref LoadBalancer
      Protocol: HTTP
      Port: 80
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref DefaultTargetGroup

  LoadBalancerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for loadbalancer to services on ECS
      VpcId: !Ref "VPC"
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          IpProtocol: -1

  DefaultTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Name: default
      VpcId: !Ref "VPC"
      Protocol: "HTTP"
      Port: "80"

  CloudWatchLogsGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: "sla_computation"
      RetentionInDays: 1

  ContainerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      VpcId: !Ref "VPC"
      GroupDescription: for ecs containers
      SecurityGroupIngress:
        - SourceSecurityGroupId: !Ref "LoadBalancerSecurityGroup"
          IpProtocol: -1

  Task:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: apis
      Cpu: 1024
      Memory: 2048
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn: !Ref ECSTaskExecutionRole
      ContainerDefinitions:
        - Name: ass001
          Image: !Sub 649905970782.dkr.ecr.eu-west-1.amazonaws.com/ass001:latest
          Cpu: 1024
          Memory: 2048
          HealthCheck:
            Command: [ "CMD-SHELL", "exit 0" ]
            Interval: 30
            Retries: 5
            Timeout: 10
            StartPeriod: 30
          PortMappings:
            - ContainerPort: 8080
              Protocol: tcp
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: "sla_computation"
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: "ass001"

  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Name: employee-tg
      VpcId: !Ref VPC
      Port: 80
      Protocol: HTTP
      Matcher:
        HttpCode: 200-299
      TargetType: ip

  ListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      ListenerArn: !Ref LoadBalancerListener
      Priority: 2
      Conditions:
        - Field: path-pattern
          Values:
            - /*
      Actions:
        - TargetGroupArn: !Ref TargetGroup
          Type: forward

  ECSTaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: [ecs-tasks.amazonaws.com]
            Action: ["sts:AssumeRole"]
      Path: /
      Policies:
        - PolicyName: AmazonECSTaskExecutionRolePolicy
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  # ECS Tasks to download images from ECR
                  - "ecr:GetAuthorizationToken"
                  - "ecr:BatchCheckLayerAvailability"
                  - "ecr:GetDownloadUrlForLayer"
                  - "ecr:BatchGetImage"
                  # ECS tasks to upload logs to CloudWatch
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Resource: "*"

  TaskSchedule:
    Type: AWS::Events::Rule
    Properties:
      Description: SLA rule ass001
      Name: ass001
      ScheduleExpression: cron(0/5 * * * ? *)
      State: ENABLED
      Targets:
        - Arn:
            !GetAtt ECSCluster.Arn
          Id: dump-data-ecs-task
          RoleArn:
            !GetAtt ECSTaskExecutionRole.Arn
          EcsParameters:
            TaskDefinitionArn:
              !Ref Task
            TaskCount: 1
            LaunchType: FARGATE
            PlatformVersion: LATEST
            NetworkConfiguration:
              AwsVpcConfiguration:
                AssignPublicIp: ENABLED
                SecurityGroups: 
                  - sg-07db5ae6616a8c5fc
                Subnets: 
                  - subnet-031d0787ad492c1c4

  TaskSchedule:
    Type: AWS::Events::Rule
    Properties:
      Description: SLA rule ass002
      Name: ass002
      ScheduleExpression: cron(0/5 * * * ? *)
      State: ENABLED
      Targets:
        - Arn:
            !GetAtt ECSCluster.Arn
          Id: dump-data-ecs-task
          RoleArn:
            !GetAtt ECSTaskExecutionRole.Arn
          EcsParameters:
            TaskDefinitionArn:
              !Ref Task
            TaskCount: 1
            LaunchType: FARGATE
            PlatformVersion: LATEST
            NetworkConfiguration:
              AwsVpcConfiguration:
                AssignPublicIp: ENABLED
                SecurityGroups:
                  - sg-07db5ae6616a8c5fc
                Subnets:
                  - subnet-031d0787ad492c1c4

Outputs:
  ApiEndpoint:
    Description: Employee API Endpoint
    Value: !Join ["", ["http://", !GetAtt LoadBalancer.DNSName, "/employees"]]
    Export:
      Name: "EmployeeApiEndpoint"

The ScheduledTask is created successfully but it is not running actually. Very strange. But the strangest thing is that the ScheduledTask starts working when I click on "Edit" from the AWS console and (without making any change) I save.

Solution

The main issue I see is that you are using wrong role for your scheduled rule. It can't be !GetAtt ECSTaskExecutionRole.Arn. Instead you should create new role (or edit existing one) which has AmazonEC2ContainerServiceEventsRole AWS Managed policy.

It works after you edit in console, because AWS console will probably create the correct role in the background and use it instead of yours.

Answered By - Marcin