EC2 Container not adding in ECS

[SibghatullahSheikh]

Dear Pahud,

I have been trying to get mixed instance policy to work. My EC2 Instace/Container is not adding in ECS. It starts outside the Ecs. I have tried to connect public subnet to it but still no success. Sharing Config. Please guide.

  ECSAutoScalingGroup:
    DependsOn: ECSCluster
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      MixedInstancesPolicy:
        InstancesDistribution:
          OnDemandAllocationStrategy: prioritized
          OnDemandBaseCapacity: 0
          OnDemandPercentageAboveBaseCapacity: 50
          SpotInstancePools: 20
          SpotAllocationStrategy: lowest-price          
        LaunchTemplate:
          LaunchTemplateSpecification: 
            LaunchTemplateId: !Ref ECSLaunchConfiguration
            Version: 1
          Overrides: 
            - InstanceType: !Select [0, !Split [ ",", !Ref InstanceTypesOverride ] ]
            - InstanceType: !Select [1, !Split [ ",", !Ref InstanceTypesOverride ] ]
      VPCZoneIdentifier: 
        !Ref Subnets
      MinSize: 2
      MaxSize: 5
      DesiredCapacity: 2
      HealthCheckGracePeriod: 90
      Tags:
        - Key: Name
          Value: Value
          PropagateAtLaunch: true
    CreationPolicy:
      ResourceSignal:
        Timeout: PT15M
    UpdatePolicy:
      AutoScalingRollingUpdate:
        MinInstancesInService: 1
        MaxBatchSize: 1
        PauseTime: PT15M
        SuspendProcesses:
          - HealthCheck
          - ReplaceUnhealthy
          - AZRebalance
          - AlarmNotification
          - ScheduledActions
        WaitOnResourceSignals: true
  
  ECSLaunchConfiguration:
    Type: AWS::EC2::LaunchTemplate
    Properties:
      LaunchTemplateData:
        ImageId: !Ref ECSAMI
        InstanceType: !Ref InstanceType
        IamInstanceProfile:
          Name: !Ref ECSInstanceProfile
          Arn: !GetAtt ECSInstanceProfile.Arn
        NetworkInterfaces:  
          -  
            DeviceIndex: 0 
            AssociatePublicIpAddress: true               
            SubnetId: !Select [0, !Ref Subnets] 
            Groups:  
              - !Ref SecurityGroup
        UserData:
          "Fn::Base64": !Sub |
            #!/bin/bash
            yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
            yum install -y https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
            yum install -y aws-cfn-bootstrap hibagent 
            echo ECS_CLUSTER=ECSCluster >> /etc/ecs/ecs.config
            echo ECS_DISABLE_IMAGE_CLEANUP=false >> /etc/ecs/ecs.config
            iid=$(curl -s http://169.254.169.254/latest/meta-data/instance-id) 
            ilc=$(aws ec2 describe-instances --instance-ids  $iid  --query 'Reservations[0].Instances[0].InstanceLifecycle' --output text) 
            if [ "$ilc" == "spot" ]; then 
              echo ECS_INSTANCE_ATTRIBUTES='{"instance-purchase-option":"spot"}' >> /etc/ecs/ecs.config 
            else 
              echo ECS_INSTANCE_ATTRIBUTES='{"instance-purchase-option":"ondemand"}' >> /etc/ecs/ecs.config 
            fi 
            /opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSLaunchConfiguration
            /opt/aws/bin/cfn-signal -e 0 --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSAutoScalingGroup
            /usr/bin/enable-ec2-spot-hibernation              

    Metadata:
      AWS::CloudFormation::Init:
        config:
          packages:
            yum:
              collectd: []

          commands:
#            01_add_instance_to_cluster:
#              command: !Sub echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config
            01_enable_cloudwatch_agent:
              command: !Sub /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:${ECSCloudWatchParameter} -s
#            03_Add_tags:
#              command: !Sub echo ECS_CONTAINER_INSTANCE_PROPAGATE_TAGS_FROM=Value >> /etc/ecs/ecs.config            
          files:
            /etc/cfn/cfn-hup.conf:
              mode: 000400
              owner: root
              group: root
              content: !Sub |
                [main]
                stack=${AWS::StackId}
                region=${AWS::Region}

            /etc/cfn/hooks.d/cfn-auto-reloader.conf:
              content: !Sub |
                [cfn-auto-reloader-hook]
                triggers=post.update
                path=Resources.ECSLaunchConfiguration.Metadata.AWS::CloudFormation::Init
                action=/opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSLaunchConfiguration

          services:
            sysvinit:
              cfn-hup:
                enabled: true
                ensureRunning: true
                files:
                  - /etc/cfn/cfn-hup.conf
                  - /etc/cfn/hooks.d/cfn-auto-reloader.conf

  # This IAM Role is attached to all of the ECS hosts. It is based on the default role
  # published here:
  # http://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html
  #
  # You can add other IAM policy statements here to allow access from your ECS hosts
  # to other AWS services. Please note that this role will be used by ALL containers
  # running on the ECS host.
  
  ECSInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref ECSRole

  ECSRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      RoleName: !Sub ${EnvironmentName}-ECSRole-${AWS::Region}
      AssumeRolePolicyDocument: |
        {
            "Statement": [{
                "Action": "sts:AssumeRole",
                "Effect": "Allow",
                "Principal": {
                    "Service": "ec2.amazonaws.com"
                }
            }]
        }
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
        - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
        - arn:aws:iam::aws:policy/AmazonEC2FullAccess
      Policies:
        - PolicyName: ecs-service
          PolicyDocument: |
            {
                "Statement": [{
                    "Effect": "Allow",
                    "Action": [
                        "ecs:CreateCluster",
                        "ecs:DeregisterContainerInstance",
                        "ecs:RegisterContainerInstance", 
                        "ecs:DiscoverPollEndpoint",
                        "ecs:Poll",
                        "ecs:StartTelemetrySession",
                        "ecs:Submit*",
                        "ecr:BatchCheckLayerAvailability",
                        "ecr:BatchGetImage",
                        "ecr:GetDownloadUrlForLayer",
                        "ecr:GetAuthorizationToken",
                        "logs:CreateLogStream",
                        "logs:PutLogEvents"
                    ],
                    "Resource": "*"
                }]
            }

  ECSServiceAutoScalingRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          Action:
            - "sts:AssumeRole"
          Effect: Allow
          Principal:
            Service:
              - application-autoscaling.amazonaws.com
      Path: /
      Policies:
        - PolicyName: ecs-service-autoscaling
          PolicyDocument:
            Statement:
              Effect: Allow
              Action:
                - application-autoscaling:*
                - cloudwatch:DescribeAlarms
                - cloudwatch:PutMetricAlarm
                - ecs:DescribeServices
                - ecs:UpdateService
              Resource: "*"

[SibghatullahSheikh]

Dear Pahud. In your service you are using image id
Type: String
Default: /aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id

This gives an error.
My Issue got solved by using
Type: AWS::SSM::Parameter::ValueAWS::EC2::Image::Id
Default: /aws/service/ecs/optimized-ami/amazon-linux/recommended/image_id

Linux-2 does not work for me. can you please tell me why its happening if you have clue regarding that.

Also now i can see that my docker service is not working
task exited for container (Service Events)
unable to create state change event for container ( cloudwatch)

Please guide if you can help.

Thanks

[wayneforrest]

I also have the same error when using Linux 2.

[pahud]

Hi,

Unfortunately I am not be able to maintain the cfn templates anymore as most of my workloads are now composed in CDK.

Have you tried to deploy it with CDK. Does it work?