Month: June 2020

How to update ECS Capacity Provider

The issue

I wanted to use autoscaling for the underlying EC2 instances of an ECS Cluster. In december 2019 AWS announced the availability of the ECS capacity provider (https://aws.amazon.com/about-aws/whats-new/2019/12/amazon-ecs-capacity-providers-now-available/)

The ECS capacity provider makes it easier to set up autoscaling of the underlying EC2 instances as ECS natively only supports scaling of the ECS tasks (!= ECS/EC2 instances)

I created an ECS capacity provider via the ECS web console as you can see in the following screenshot:

ECS capacity provider shown in the ECS web console

As you can see there are only buttons for creating or deactivating capacity providers which means you can’t update an already existing capacity provider.

As I wanted to avoid creating another ECS capacity provider (I don’t like redundancy 😉 ) I was looking for a way to update ECS capacity provider. In my case I wanted to change the threshold when autoscaling should happen. The corresponding value is “Target capacity”.

My first attempt was: just click on the create-Button and try to create a capacity provider with the same name and hope this overwrites the current one. Unfortunately this didn’t work as you can see in the following screenshot containing the error message “There was an error creating the capacity provider. Fix the following error and try again. The specified capacity provider already exists. To change the configuration of an existing capacity provider, update the capacity provider“:

“To change the configuration of an existing capacity provider, update the capacity provider.”

Ok, so I asked google how to update ECS capacity provider.

I found several github issues. One of them is contained in the official containers-roadmap git repository from AWS. The issue can be found at: https://github.com/aws/containers-roadmap/issues/688 which points to https://github.com/aws/containers-roadmap/issues/633

AWS is already working on implementing the API command for updating an ECS capacity provider. Currently the second issue mentioned above is in the state “Coming soon”.

As I needed a solution now I investigated how the ECS capacity provider works behind the scenes and finally found a solution which I present you below:

The solution

As far as I know the ECS capacity provider creates an EC2 Autoscaling tracking policy which you can see when looking at the associated EC2 autoscaling group of your already existing ECS capacity provider. I marked the target capacity threshold red in the following screenshot. As I only created a screenshot after updating the policy I had to place the old value 75 manually in the image:

Target tracking policy of the EC2 autoscaling group associated with the ECS capacity provier

So my next step was to switch to CloudWatch console and change the tracking policy. Unfortunately this is not possible (at least via the console). This is the error message you get when trying to change a tracking policy:

Error message “Cannot edit TargetTracking … as it is an Auto Scaling alarm with target tracking scaling policies.

Fortunately updating an target tracking policy via API call is possible. So let’s come to the solution.

The solution

The API command aws autoscaling put-scaling-policy (https://awscli.amazonaws.com/v2/documentation/api/latest/reference/autoscaling/put-scaling-policy.html) is made for creating or updating an autoscaling policy.

You need three parameters in this case to update the tracking policy:

  • –auto-scaling-group-name: use the name of the already exisiting EC2 autoscaling group
  • –policy-name: use the name of the target tracking policy you want to change
  • –policy-type: use the type TargetTrackingScaling
  • –target-tracking-configuration: specify a JSON file containing the changes you want to achieve

So your command should look similar to mine:

aws autoscaling  put-scaling-policy --auto-scaling-group-name ecsCluster-ECSAutoScalingGroup-12345678 --policy-name AutoScaling-ECSManagedAutoScalingPlan-12345678 --policy-type TargetTrackingScaling --target-tracking-configuration file://target.json

The tracking configuration file which I named target.json looks like:

{
    "TargetValue":100.0,
    "CustomizedMetricSpecification":{
       "MetricName":"CapacityProviderReservation",
       "Namespace":"AWS/ECS/ManagedScaling",
       "Dimensions":[
          {
             "Name":"CapacityProviderName",
             "Value":"Cap-provider-ecs-12345678"
          },
          {
              "Name":"ClusterName",
              "Value":"ecs-xEV-1234"
          }
       ],
       "Statistic":"Average",
       "Unit":"Percent"
    }
 }

You have to change the CloudWatch dimensions in the JSON file as they could be named differently than mine. The first item “TargetValue” was the attribute I changed in my case.

In the following screenshot you can see the changed value of target capacity:

I hope you liked this post and helped you using the ECS target capacity provider.

Related ressources: