Introduction

Recently, as in a few weeks ago, AWS finally announced CloudFormation for StackSets.

StackSets are an awesome idea when you approach AWS from a purely Infrastructure-as-Code mindset. However, you had to create and manage them manually. But not anymore!

Unsure what StackSets are? The tl;dr version is that they allow you to deploy the same CloudFormation Stack across multiple regions and/or AWS accounts.

For me there is no better way to learn than by doing; so, I dusted off my old suite of AWS accounts and dove into deploying my very first CloudFormation StackSet by CloudFormation.

This post explains how I got everything working in my accounts (the oldest account being created circa 2015) and the (mis)steps I took along the way. A follow up piece on “what I deployed” will be released soon.

Let’s dive in!

Diving In

  1. My first task was to ensure my accounts were managed via AWS Organisations and that all features were enabled (rather than consolidated billing). One ✅ for me as I had already enabled all features sometime in the distant past: Screenshot of full permissions

  2. The next item to do was to enable CloudFormation StackSets for my organisation: Screenshot before stacksets enabled I just clicked the Enabled access button, and ta-daa! Screenshot after stacksets enabled Screenshot after stacksets enabled

  3. From here, I created a new Organisational Unit (OU) and moved all my child accounts under it. I did this as I knew that StackSets can automatically deploy stack instances to all accounts under an OU, which suited me perfectly for my handful of accounts.

  4. From here, I tried to deploy my StackSet and encountered a weird error. I also needed to enable StackSets via the CloudFormation console: Screenshot before stacksets enabled Screenshot after stacksets enabled

  5. One more try and it’s still not working. If you have ever worked with StackSets before, you’ll know that the error reporting is basically non-existent. You are forced to use the command-line and guesswork to try and uncover what is going on. What follows is a command line tale of frustration and woe…

    # First, lets list the StackSets
aws cloudformation list-stack-sets
{
    "Summaries": [
        {
            "StackSetName": "MyStackSet",
            "StackSetId": "MyStackSet:00000000-0000-0000-0000-000000000000",
            "Description": "MyStackSet Description",
            "Status": "DELETED",
            "DriftStatus": "NOT_CHECKED"
        }
    ]
}

# Not super helpful, but it gives me the StackSetId which I will need laster. Deleted stacks can only be described by the full id.
aws cloudformation describe-stack-set --stack-set-name "MyStackSet:00000000-0000-0000-0000-000000000000"
{
    "StackSet": {
        "StackSetName": "MyStackSet",
        "StackSetId": "MyStackSet:00000000-0000-0000-0000-000000000000",
        "Description": "MyStackSet Description",
        "Status": "DELETED",
        "TemplateBody": "AWSTemplateFormatVersion: 2010-09-09\n ...",
        "Parameters": [],
        "Capabilities": [
            "CAPABILITY_NAMED_IAM"
        ],
        "Tags": [],
        "StackSetARN": "arn:aws:cloudformation:ap-southeast-2:000000000000:stackset/MyStackSet:00000000-0000-0000-0000-000000000000",
        "AdministrationRoleARN": "arn:aws:iam::000000000000:role/aws-service-role/stacksets.cloudformation.amazonaws.com/AWSServiceRoleForCloudFormationStackSetsOrgAdmin",
        "ExecutionRoleName": "stacksets-exec-deadbeefdeadbeefdeadbeefdeadbeef",
        "StackSetDriftDetectionDetails": {
            "DriftStatus": "NOT_CHECKED",
            "TotalStackInstancesCount": 0,
            "DriftedStackInstancesCount": 0,
            "InSyncStackInstancesCount": 0,
            "InProgressStackInstancesCount": 0,
            "FailedStackInstancesCount": 0
        }
    }
}

# Not very helpful, so I dig in further
aws cloudformation list-stack-set-operations --stack-set-name "MyStackSet:00000000-0000-0000-0000-000000000000"
{
    "Summaries": [
        {
            "OperationId": "22222222-2222-2222-2222-222222222222",
            "Action": "DELETE",
            "Status": "SUCCEEDED",
            "CreationTimestamp": "2020-09-22T01:58:27.847Z",
            "EndTimestamp": "2020-09-22T01:58:52.502Z"
        },
        {
            "OperationId": "11111111-1111-1111-1111-111111111111",
            "Action": "CREATE",
            "Status": "FAILED",
            "CreationTimestamp": "2020-09-22T01:55:01.515Z",
            "EndTimestamp": "2020-09-22T01:58:07.163Z"
        }
    ]
}

# Getting some more details
aws cloudformation describe-stack-set-operation --stack-set-name "MyStackSet:00000000-0000-0000-0000-000000000000" --operation-id "11111111-1111-1111-1111-111111111111"
{
    "StackSetOperation": {
        "OperationId": "11111111-1111-1111-1111-111111111111",
        "StackSetId": "MyStackSet:00000000-0000-0000-0000-000000000000",
        "Action": "CREATE",
        "Status": "FAILED",
        "OperationPreferences": {
            "RegionOrder": [],
            "FailureToleranceCount": 1,
            "MaxConcurrentCount": 1
        },
        "AdministrationRoleARN": "arn:aws:iam::000000000000:role/aws-service-role/stacksets.cloudformation.amazonaws.com/AWSServiceRoleForCloudFormationStackSetsOrgAdmin",
        "ExecutionRoleName": "stacksets-exec-deadbeefdeadbeefdeadbeefdeadbeef",
        "CreationTimestamp": "2020-09-22T01:55:01.515Z",
        "EndTimestamp": "2020-09-22T01:58:07.163Z"
    }
}

# Grr, not helpful at all! Maybe there is detail in the stack instances
aws cloudformation list-stack-instances --stack-set-name "MyStackSet:00000000-0000-0000-0000-000000000000"
{
    "Summaries": []
}

# Makes sense, they'd be in the child accounts.
aws --profile child-account cloudformation list-stack-sets
{
    "Summaries": []
}

  6. At this point, I gave up and just tried redeploying while watching the AWS Console to see if any errors would appear. I got super lucky as I saw this error appear!

    ResourceLogicalId:RbacDeployerRole, ResourceType:AWS::IAM::Role, ResourceStatusReason:1 validation error detected: Value '900' at 'maxSessionDuration' failed to satisfy constraint: Member must have value greater than or equal to 3600 (Service: AmazonIdentityManagement; Status Code: 400; Error Code: ValidationError; Request ID: 44444444-4444-4444-4444-444444444444; Proxy: null).

  7. Changed the erroneous value, commit, push and voila! It all deploys smoothly and without further errors!

Conclusion

One last item to keep in mind is the difference in behaviour between TemplateURL and TemplateBody. Initially I used TemplateURL to keep the template that StackSets was deploying separate from the CloudFormation template that deployed the StackSet itself. This meant that when the referenced template changes, the StackSet pipeline doesn’t detect a change and so won’t deploy anything. My solution was to embed the template within the StackSet definition (using TemplateBody) to ensure that changes would always be detected and deployed.

The caveat, of course, is that I lost linting support. I’m happy with this trade-off as I’m the only person in my accounts. I’ll cut the existing template out and paste into a temporary local file. I’ll make the changes there and then paste them back into once I’m done. I’m not expecting to deploy many changes in the future so I can live this this workaround.

  StackSet:
    Type: AWS::CloudFormation::StackSet
    Properties:
      ...
      # TemplateURL: !Sub 'https://${ParamBucketName}.s3-ap-southeast-2.amazonaws.com/template.yml'
      TemplateBody: |
        AWSTemplateFormatVersion: 2010-09-09
        Parameters:
          ...
        Resources:
          ...
        Outputs:
          ...

In future post I’ll be covering how I use CloudFormation StackSets to manage roll-based access control to all my accounts.

Bonus content - StackSet announcement timeline