Graduate Program KB

What is CloudFormation?

  • CloudFormation is a declarative way of outlining your AWS infrastructure for any resources
  • Ex. Specify security group, EC2 instances using the group, elastic IPs for the instances, S3 bucket, load balancers, etc.
  • Executes configuration for you in the specified order
  • Benefits:
    • Infrastructure as code
      • No resources manually created
      • Versioning
      • Infrastructure changes reviewed through code
    • Cost
      • Resources within the stack are tagged
      • Estimate cost of resources with templates
      • Automate deletion and recreation of templates to suit needs
    • Productivity
      • Quickly destroy and recreate infrastructure
      • Automate diagram generation of templates
      • Declarative programming (don't need to consider ordering)
    • Separation of concerns
      • VPC, network, app stacks
    • Don't re-invent the wheel
      • Leverage existing templates and documentation

Introduction

  • Templates must be uploaded in S3 then referenced in CloudFormation
  • To update a template, you can't edit previous ones. A new version of the template will need to be re-uploaded
  • Stacks are identified by a name
  • Deleting a stack deletes all artifacts created by CloudFormation
  • Ex. of template, create EC2 instance, add security group 
    Resources:
MyInstance:
    Type: AWS::EC2::Instance
    Properties:
    AvailabilityZone: us-east-1a
    ImageId: ami-0742b4e673072066f
    InstanceType: t2.micro

Getting started

  • Can use YAML and JSON for CloudFormation templates
  • Tool for converting JSON to YAML
  • YAML features:
    • Key-value pairs
    • Nested objects
    • Supports arrays (-)
    • Multi-line strings (|)
    • Can include comments (#)

S3

  • S3 CloudFormation documentation

  • Ex. create S3 bucket

    Resources:
    MyS3Bucket:
        Type: AWS::S3::Bucket
        Properties: {}

  • Update behaviour:

    • Resources update based on differences between current and submitted template
    • Method depends on which property updated
      • Update with no interruption: No disruption to resource operations & without changing physical ID. Ex. Updating IAM instance profile
      • Update with some interruption: Ex. updating EC2 instance type
      • Replacement: Recreating resource with new physical ID. Creates new resource, change references from other resources to new resource then delete old resource
Resources:
    MyS3Bucket:
        Type: AWS::S3::Bucket
        Properties:
            # Adding access control
            AccessControl: PublicRead
            # Changing name
            BucketName: "insert-some-random-string-here-12345233"
  • Delete behaviour:
    • Can't delete non-empty S3 bucket, must delete all objects inside first

Template options

  • Common parameters to any template:
    • Tags
    • Permissions
    • Notification options
    • Timeout
    • Rollback on failure
    • Rollback configuration (monitoring time & CloudWatch alarm)
    • Stack policy
    • Termination protection
    • Quick-start link

AWS Application Composer

  • Visually design and build serverless applications quickly on AWS
  • Deploy AWS infrastructure code without much expertise
  • Configure interaction between resources
  • Generates infrastructure as code using CloudFormation
  • Can import existing CloudFormation / SAM templates for visualisation

Building blocks

  • AWSTemplateFormatVersion: IDentifies capabilities of template ("2010-09-09")
  • Description: Template comments
  • Transform: Specifies macros used to process template
  • Metadata
  • Resources: AWS resources declared in the template (mandatory)
  • Parameters: Dynamic inputs for your template
  • Mappings: Static variables for your template
  • Outputs: References to what's been created
  • Conditionals: List of conditions to perform resource creation
  • Rules: Validate parameters during stack creation and update

Parameters

  • Parameters are a way to provide inputs to your CloudFormation template
  • Reuse templates and specify uncertain inputs at the time of creation
  • Powerful, controlled and prevent errors due to type checking
  • Can be cross-validated using rules

Types

  • String

  • Number

  • CommaDelimitedList

  • List<Number>

  • AWS-Specific Parameter (help catch invalid values and match against existing values in AWS account)

  • List<AWS-Specific Parameter>

  • SSM Parameter (get parameter value from SSM Parameter store)

    • Specify SSM parameter key as the value
    • CloudFormation always fetches the lastest value
    • CloudFormation doesn't store secure string values
    • Validation is done on SSM parameter keys but not values
    • Supported SSM Parameter Types:
      • AWS::SSM::Parameter::Name
      • AWS::SSM::Parameter::Value<String>
      • AWS::SSM::Paramter::Value<List<String>>
      • AWS::SSM::Paramter::Value<CommaDelimitedList>
      • AWS::SSM::Paramter::Value<AWS-Specific Parameter>
      • AWS::SSM::Parameter::Value<List<AWS-Specific Paramter>>

  • Description

  • ConstraintDescription (String)

  • MinLength / MaxLength

  • MinValue / MaxValue

  • Default

  • AllowedValues (array)

  • AllowedPattern (regex)

  • NoEcho (boolean)

Referencing a parameter

  • Fn::Ref function can be leveraged to reference parameters, the shorthand in YAML is !Ref

  • Parameters can be used anywhere in a template except:

    • AWSTemplateFormatVersion
    • Description
    • Transform
    • Mappings
    Parameters:
    InstanceType:
        Description: WebServer EC2 instance type
        Type: AWS::SSM::Parameter::Value<String>
        Default: /dev/ec2/instanceType
    
    ImageId:
        Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
        Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
    
    Resources:
        MyEC2Instance:
            Type: AWS::EC2::Instance
            Properties:
                InstanceType: !Ref InstanceType
                ImageId: !Ref ImageId

Resources

  • Resources are mandatory
  • Represent different AWS Components that'll be created and configured
  • Resources are declared and can reference each other
  • AWS manages out createSolutionBuilder, updates and deletes of resources for us
  • Over 700 types of resources
  • Resource types identifiers are of the form: AWS::aws-product-name::data-type-name

Attributes for resources

  • Optional
  • DependsOn: Useful to draw dependency between two resources. Ex. create an ECS cluster after creating an ASG
  • DeletionPolicy: Protect resources from deletion even if the CloudFormation stack is deleted. Ex. an RDS database
  • UpdateReplacePolicy: Protect resources from replacement during a CloudFormation update
  • CreationPolicy: Details in CFN Init section
  • UpdatePolicy: Applies to some specific resources such as ASG
  • Metadata: Details in CFN Init section

DependsOn

  • Specify creation of specific resources in a certain order (resources depend on another)
  • When the attribute is added to a resource, it's created only after the resource specified by DependsOn is created
  • Automatically applied when using !Ref and !GetAtt
  • Can be used with any resource
  • In the following example, MyEC2Instance is created after MyDB as it depends on that resource 
    Resources:
    MyEC2Instance:
        Type: AWS::EC2::Instance
        Properties:
            ImageId: ami-0742b4e673072066f
        DependsOn: MyDB
    MyDB:
        Type: AWS::RDS::DBInstance
        Properties:
            DBInstanceClass: db.t2.small
            Engine: MySQL

DeletionPolicy

  • Control events when the CloudFormation template is deleted or when a resource is directly removed from it
  • DeletionPolicy=Retain: Specify on resources to preserve / backup in case of deletion (specify Retain)
  • DeletionPolicy=Snapshot: Ex. EBS volume, ElastiCache cluster / ReplicationGroup, RDS DBInstance / DBCluster, Redshift cluster, Neptune DBCluster
  • DeletionPolicy=Delete: Default behaviour. Exceptions for AWS::RDS::DBCluster resources (default is Snapshot) and AWS::RDS:DBInstance resources (default is Snapshot for resources that don't specify DBClusterIdentifier property)
  • Can be used with any resource

UpdateReplacePolicy

  • Control events to a resource if you update a property whose update behaviour is Replacement
  • UpdateReplacePolicy=Delete: Default behaviour. CloudFormation deletes the old resource and creates a new one with a new physical ID
  • UpdateReplacePolicy=Retain: Keeps the resource but removed from the CloudFormation's scope
  • DeletionPolicy=Snapshot: Ex. EBS volume, ElastiCache cluster / ReplicationGroup, RDS DBInstance / DBCluster, Redshift cluster, Neptune DBCluster
  • Can be used with any resource

UpdateReplacePolicy vs DeletionPolicy

  • UpdateReplacePolicy only applies to resources replaced during stack updates
  • DeletionPolicy applies to resources deleted when a stack is deleted and when a resource definition is deleted from the template as part of the update 
    Parameters:
    BucketName:
        Type: String

Resources:
    MyS3Bucket:
        Type: AWS::S3::Bucket
        Properties:
            BucketName: !Ref BucketName
        DeletionPolicy: Retain
        UpdateReplacePolicy: Retain

Mappings

  • Fixed variables (hardcoded values) within a CloudFormation template
  • Differentiate between environments (developer vs production), AWS regions, AMI types, etc.
  • Use when you know in advance all values that can be taken and deduced from variables such as region, AZ, AWS account, environment, etc.
  • Safer control over template
  • Parameters on the other hand, are used when values are really user specific (variable to change)

Accessing mapping values

  • Fn::FnInMap is a function that returns a named value from a specific key
  • Ex. !FindInMap [MapName, TopLevelKey, SecondLevelKey] 
    AWSTemplateFormatVersion: "2010-09-09"
Mappings:
    RegionMap:
        us-east-1:
            HVM01: ami-0000000000000000
            HVM02: ami-0000000000000001
        us-west-1:
            HVM03: ami-0000000000000002
            HVM04: ami-0000000000000003
        eu-west-1:
            HVM05: ami-0000000000000004
            HVM06: ami-0000000000000005
Resources:
    MyEC2Instance:
        Type: AWS::EC2::Instance
        Properties:
            ImageId: !FindInMap [RegionMap, !Ref "AWS::Region", HVM03]
            InstanceType: m1.small

Concept of pseudo parameters

  • AWS offers pseudo parameters in any template, they can be used anytime and are enabled by default
  • List of pseudo parameters:
    • AWS::AccountId
    • AWS::Region
    • AWS::StackId
    • AWS::StackName
    • AWS::NotificationARNs
    • AWS::NoValue
    • AWS::Partition
    • AWS::URLSuffix

Outputs

  • Optional output values that can be imported into other stacks (if you export them first using the Export property)
  • Useful for cross stack collaboration
  • Can view outputs in Console or using the CLI
  • Ex. define a network CloudFormation and output variables such as the VPC ID and Subnet IDs

Cross stack

  • Provided an output value from a template, a second template that can leverage it...
  • Use the Fn::ImportValue function
  • Note, the underlying stack can't be deleted until all references are deleted as well
  • Ex. SSHSecurityGroup was exported from the first template and imported into the following template 
    Resources:
    MySecureInstance:
        Type: AWS::EC2::Instance
        Properties:
            AvailabilityZone: us-east-1a
            ImageId: ami-0742b4e673072066f
            InstanceType: t2.micro
            SecurityGroups:
                - !ImportValue SSHSecurityGroup

Conditions

  • Conditions control the creation of resources or outputs based on a condition

  • Common conditions are environment (dev / test / prod), region, any parameter value, etc.

  • Each condition can reference another condition, parameter value or mapping

  • Logical functions:

    • Fn::And
    • Fn::Equals
    • Fn::If
    • Fn::Not
    • Fn::Or
    Conditions:
    CreateProdResources: !Equals [!Ref EnvType, prod]

Get attribute function

  • Fn::GetAtt
  • Attributes are attached to any resources you create, the documentation provides more information on it (ex. the AZ on an EC2 machine) 
    Resources:
    MyEC2Instance:
        Type: AWS::EC2::Instance
        Properties:
            ImageId: ami-0742b4e673072066f
            InstanceType: t2.micro
    NewVolume:
        Type: AWS::EC2::Volume
        Condition: CreateProdResources
        Properties:
            Size: 100
            AvailabilityZone: !GetAtt MyEC2Instance.AvailabilityZone

Rules

  • Rules are used to perform parameter validations based on values of other parameters (cross-parameter validation)
  • Ex. ensure all selected subnets are within the same VPC
  • Defining a rule:
    • RuleCondition (optional): Determines when a rule takes effect (only one condition)
    • Assertions: Describes what values are allowed for a particular parameter, one or more assertions allowed
    • If a rule condition is not defined, assertions will take effect with every create and update operation
  • Ex. Enforce users to provide an ACM certificate ARN if configuring an SSL listener on an ALB 
    Rules:
    IsSSLCertificate:
        RuleCondition: !Equals
            - !Ref UseSSL
            - Yes
        Assertions:
            - Assert: !Not
                - !Equals
                    - !Ref ALBSSLCertificateARN
                    - ''
              AssertDescription: 'ACM certificate value can not be empty if SSL is required'

Rule-specific intrinsic functions

  • Used for defining a rule condition and assertions
  • Only used in rules section
  • Functions can be nested but result of a rule condition or assertion must be true or false
  • Supported functions:
    • Fn::And
    • Fn::Contains
    • Fn::EachMemberEquals
    • Fn::EachMemberIn
    • Fn::Equals
    • Fn::If
    • Fn::Not
    • Fn::Or

Metadata

  • Metadata is optional
  • Can include arbitrary YAML, providing details about the template or resource 
    Metadata:
    Instances:
        Description: "Information about the instances"
    Databases:
        Description: "Information about the databases"

Special metadata keys

  • AWS::CloudFormation::Interface: Define grouping and ordering of input parameters when displayewd in the Console
  • AWS::CloudFormation::Authentication: Used to specify authentication credentials for files or sources you specify in AWS::CloudFormation::Init
  • AWS::CloudFormation::Init: Define configuration tasks for cfn-init (most powerful usage of metadata)

CFN-Init and EC2 user data

  • CloudFormation is about provisioning computing resources in your Cloud such as EC2 instances, ASGs, etc.
    • These instances usually want to be configured to perform jobs upon creation, CloudFormation Init can automate your EC2 fleet state
    • The manual way is to create a user data script which is executed at the first boot of an instance
  • Can write the same EC2 user data script in a CloudFormation template, passing the entire script through the function Fn::Base64
  • The user data script log will be located in /var/log/cloud-init-output.log 
    Resources:
WebServer:
    Type: AWS::EC2::Instance
    Properties:
    ImageId: !Ref ImageId
    InstanceType: t2.micro
    KeyName: !Ref KeyName
    SecurityGroups:
        - !Ref WebServerSecurityGroup
    UserData:
        Fn::Base64: |
        #!/bin/bash
        yum update -y
        amazon-linux-extras install -y lamp-mariadb10.2-php7.2 php7.2
        yum install -y httpd mariadb-server
        systemctl start httpd
        systemctl enable httpd
        usermod -a -G apache ec2-user
        chown -R ec2-user:apache /var/www
        chmod 2775 /var/www
        find /var/www -type d -exec sudo chmod 2775 {} \;
        find /var/www -type f -exec sudo chmod 0664 {} \;
        echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php

Problems with EC2 user data

  • Very large instance configuration
  • Evolve the state of EC2 instance without terminating and creating a new one
  • Make EC2 user data script more readable
  • How to know or signal that our EC2 user data script is completed successfully

CloudFormation helper scripts

  • 4 Python scripts available directly on Amazon Linux 2 AMI or installed using yum on non-Amazon Linux AMIs
  • cfn-init: Used to retrieve and interpret resource metadata, installing packages, creating files and starting services
  • cfn-signal: Simple wrapper to signal with a CreationPolicy or WaitCondition, enables you to synchronise other resources in the stack with the application being ready
  • cfn-get-metadata: Wrapper script for easy retrieval of all metadata defined for a resource / path to a specific key / subtree of a resource metadata
  • cfn-hup: A daemon to check for updates to metadata and execute custom hooks when changes are detected
  • Usual flow: cfn-init --> cfn-signal --> cfn-hup (optional)

AWS::CloudFormation::Init

  • A config contains the following, executed in that order:

    • Packages: Used to download and install pre-packaged apps
    • Groups: Define user groups
    • Users: Define users and which group they belong to
    • Sources: Download files, archives and place them on the EC2 instance
    • Files: Create files on the EC2 instance, using inline text or can be pulled from a URL
    • Commands: Run a series of commands
    • Services: Launch a list of sysvinit

  • An AWS::CloudFormation::Init can have multiple configs

  • Can create configSets with multiple configs

  • configSets can be invoked from the EC2 user data script

Packages

  • Install packages from repositories: apt, msi, python, rpm, rubygems and yum
  • Packages processed in the following order: rpm, yum / apt then rubygems / python
  • Can specify a version or no version (empty array means latest version)

Groups and Users

  • Ex. multiple users and groups (with optional gid) 
    groups:
    groupOne: {}
    groupTwo:
        gid: "45"

users:
    myUser:
        groups:
            - "groupOne"
            - "groupTwo"
        uid: "50"
        homeDir: "/tmp"

Sources

  • Download whole compressed archives from the web and unpack them on the EC2 instance directly
  • Ex. useful if you have a set of standardised scripts for your EC2 instances that you store in S3, or you want to download a whole GitHub project on an EC2 instance
  • Supported formats: tar, tar+gzip, tar+bz2, zip

Files

  • Files provide full control over your content
  • Can come from a specific URL or written inline
  • Additional attributes for files:
    • source: URL to load file from
    • authentication: Name of authentication method to use. Ex. for files protected by username and password

AWS::CloudFormation::Authentication

  • Specify authentication credentials for files or sources in AWS::CloudFormation::Init
  • Two types:
    • basic: Used when the source is a URL
    • S3: Used when the source is an S3 bucket
  • Prefer using roles instead of access keys for EC2 instances

Function Fn::Sub

  • Used to substitute variables from a text, useful for customising templates
  • Can combine Fn::Sub with References or AWS Pseudo variables
  • String must contain ${VariableName} and it will substitute them

Commands

  • Can run commands one at a time in alphabetical order
  • Can set a directory from which that command is run
  • Can provide a test to control whether the command is executed or not. Ex. call the echo command if file doesn't exist

Services

  • Launch many services at EC2 instance launch
  • Ensure services are started when files are changed or packages are updated by cfn-init

CFN Init Scripts

  • The cfn-init script makes complex EC2 configurations readable
  • The EC2 instance queries the CloudFormation service to retrieve init data
  • Logs go to /var/log/cfn-init.log
  • The cfn-signal script can be run after cfn-init to inform CloudFormation service the resource creation was successful or not
  • Can block the template until it receives a signal from cfn-signal by defining a WaitCondition
    • Attach a CreationPolicy
    • Can define a Count for cases with multiple signals

cfn-hup

  • Used to tell EC2 instance to look for metadata changes every 15 minutes, then apply the metadata configuration again
  • Relies on cfn-hup configuration, see /etc/cfn/cfn-hup.conf and /etc/cfn/hooks.d/cfn-auto-reloader.conf

CreationPolicy

  • Prevent resource's status from reaching CREATE_COMPLETE until CloudFormation receives a specified number of success signals or timeout period is exceeded
  • Use cfn-signal helper script to signal a resource
  • Supported resources:
    • AWS::EC2::Instance
    • AWS::CloudFormation::WaitCondition
    • AWS::AutoScaling::AutoScalingGroup
    • AWS::AppStream::Fleet
  • Ex. wait until applications installed and configured on your EC2 instance

CFN Init troubleshooting

  • Wait condition didn't receive the required number of signals from an EC2 instance
    • Ensure AMI has CloudFormation helper scripts installed
    • Verify cfn-init and cfn-signal command was successfully run on the instance, can also view logs for debugging
    • Retrieve logs by logging into the instance but must disable rollback on failure, else CloudFormation deletes the instance after stack fails to create
    • Verify the instance has connection to the Internet. If in VPC, instance can connect through a NAT device if in a private subnet, or through an Internet gateway if in a public subnet

User data vs CloudFormation::Init vs helper scripts

  • EC2 user data is an imperative way to provision an EC2 instance using Shell syntax
  • AWS::CloudFormation::Init is a declarative way to provision an EC2 instance using JSON or YAML syntax. It's useful if not triggered by a script within the EC2 user data
  • Triggering AWS::CloudFormation::Init inside an EC2 user data script is done by using cfn-init or cfn-hup

CloudFormation Drift

  • CloudFormation doesn't protect you against manual configuration changes
  • Drift provides a report to alert us if resources have drifted
    • Detect drift on an entire stack / individual resources within a stack
    • Can resolve stack / resource drift by using Resource Import
    • Not all resources are supported yet

Nested stacks

  • Nested stacks are stacks part of other stacks, allowing you to isolate repeated patterns / common components in separate stacks and call them from other stacks
  • Considered best practice
  • To update a nested stack, always update the parent / root stack
  • Ex. re-using load balancer configuration or security group

Nested stacks update

  • Ensure updated nested stacks are uploaded to S3 first
  • Re-upload your root stack after that

Nested stacks delete

  • Never delete / apply changes to the nested stack, always do changes from the top-level stack

Nested stacks vs cross stacks vs exported stack output values

  • Cross stacks are helpful when stacks have different lifecycles, use outputs and imports or when you need to pass export values to many stacks
  • Nested stacks are helpful for re-using components and if the nested stack is not shared but only important to a higher-level stack
  • Exported stack output values are helpful if you have a central resource shared by many stacks and stacks need to be updated immediately if the central resource is updated

StackSets

  • Create, update or delete stacks across multiple accounts and regions with a single operation / template
  • An administrator account is used to create StackSets
  • Target accounts can create, update and delete stack instances from StackSets
  • When a stack set is updated, all associated instanceds are updated throughout all accounts and regions
  • Operations:
    • Create StackSet: Provide template and target accounts / regions
    • Update StackSSet: Updates affect all stacks
    • Delete Stacks: Delete stack and its resources from target accounts / regions, delete stack from your StackSet (stack will still run independently) or delete all stacks from StackSet
    • Delete StackSet: Must delete all stacks within StackSet first before deleting it

StackSets deployment options

  • Deployment Order: Order of regions where stacks are deployed, operations are performed one region at a time
  • Maximum Concurrent Accounts: Maximum number / percentage of target accounts per region which you can deploy stacks at a time
  • Failure Tolerance: Maximum number / percentage of target accounts per region of stack operation failures that can occur before CloudFormation stops operation in all regions
  • Region Concurrency: Deploy StackSet into regions Sequential (default) or Parallel
  • Retain Stacks: Use when deleting StackSet to keep stacks and their resources running once removed

Permission models

  • Self-managed permissions
    • Create IAM roles in both administrator and target accounts
    • Deploy to any target account in which you have permissions to create an IAM role
  • Service-managed permissions
    • Deploy to accounts managed by AWS Organizations
    • StackSets create IAM roles on your behalf (must enable trusted access with AWS Organizations)
    • Automatic deployments to future accounts added

Drift detection

  • Drift detection identifies unmanaged changes outside CloudFormation
    • Changes made through CloudFormation to a stack directly but not at a StackSet level, aren't considered drift
  • Perform drift detection on the stack associated with each stack instance in the StackSet
    • Can also stop drift detection on a StackSet
  • If the state of a resource in a stack has changed:
    • The stack is considered drifted
    • The stack instance that the stack is associated with is considered drifted
    • The StackSet is considered drifted

CloudFormation deployment options

ChangeSets

  • When updating a stack, the user needs to know what changes will occur before applying them
  • ChangeSets provides a view of these changes
  • It does not give information whether the update will be successful

Stack creation failures

  • ROLLBACK_COMPLETE: Status when a CloudFormation stack creation fails
    • Occurs when one resource creation fails, causing CloudFormation to roll back all the previously created resources before this point
    • To resolve, delete the failed stack and create a new one. You can't update, validate or change-set on a create failed stack

Rollback triggers

  • Enables CloudFormation to rollback stack create / update operation if it triggers CloudWatch Alarm
  • The alarms are monitored during stack create / update and some monitoring period after resources are deployed (default of 0 minutes, maximum of 180)
  • If any alarms go to ALARM state, the entire stack operation is rolled back
  • If a monitoring time is set but rollback triggers aren't specified, CloudFormation still waits out the period before cleaning up old resources for update operations

Continue rolling back an update

  • UPDATE_ROLLBACK_FAILED: Status when CloudFormation can't roll back all changes during an update
    • A resource can't return to its original state, causing rollback to fail\
    • Stacks in this state can't be updated
    • To resolve, fix errors manually outside of CloudFormation then continue update rollback on the stack. Otherwise, skip resources that can't rollback successfully, CloudFormation will mark them as UPDATE_COMPLETE

Stack policy

  • A JSON document defining the update actions allowed on stack resources
  • Prevent stack resources from being unintentionally updated / deleted during a stack update
  • By default, all update actions are allowed on all resources
  • By default, all resources are protected when stack policy is enabled
  • List of actions:
    • Update:Modify
    • Update:Replace
    • Update:Delete
    • Update:*
  • The Principal only supports the wildcard (*)
  • To update protected resources:
    • Create a temporary policy overriding the stack policy
    • The override policy doesn't permanently change the stack policy
    {
    "Statement": [
        {
            "Effect": "Deny_or_Allow",
            "Action": "update_actions",
            "Principal": "*",
            "Resource": "LogicalResourceId/resource_logical_ID",
            "Condition": {
                "StringEquals_or_StringLike": {
                    "ResourceType": [resource_type, ...]
                }
            }
        }
    ]
}

Termination protection on stacks

  • Prevent accidental deletes of stacks, using the action cloudformation:UpdateTerminationProtection
  • Also applies to any nested stacks
  • Add explicit Deny on some user groups in IAM policies

Service role

  • An IAM role allowing CloudFormation to create / update / delete stack resources on your behalf
  • By default, CloudFormation uses a temporary session that is generates from your user credentials
  • Allows users without permissions to work with the stack's resources to create / update / delete them
  • Useful for achieving least privilege principle and not wanting to give user all the required permissions to create stack resources
  • Custom URLs used to launch stacks quickly from Console
  • Reduce number of wizard pages and amount of user input required
  • CloudFormation ignores parameters that don't exist in the template and are defined with the NoEcho property set to true
  • Ex. create multiple URLs specifying different values for the same template

Continuous delivery with CodePipeline

  • Use CodePipeline to build a continuous delivery workflow for CloudFormation stacks
  • Rapidly and reliably make changes to AWS infrastructure
  • Automatically build and test changes to templates before promoting them to production stacks
  • Ex. build test stack when template is submitted to a code repository, can test the stack and decided whether to push changes to the production stack

Custom resources

  • Can write custom provision logic in templates that runs whenever a stack is created, updated or deleted
  • Defined in the template using AWS::CloudFormation::CustomResource or Custom::MyCustomResourceTypeName (recommended)
  • Two types:
    • Amazon SNS-backed custom resources
    • AWS Lambda-backed custom resources
  • Useful for AWS resources that aren't supported yet, an on-premises resource, running a Lambda function to empty an S3 bucket before deleting it or fetching an AMI ID, etc.

How to define it?

  • ServiceToken property specifies where CloudFormation sends requests to, such as Lambda ARN or SNS ARN (required + must be same region)
  • Input data parameters are optional 
    Resources:
    LogicalResourceName:
        Type: Custom::MyCustomResourceTypeName
        Properties:
            ServiceToken: service_token

WaitCondition

  • Make CloudFormation pause the creation of a stack and wait for a signal before continuing to create it
  • Ex. start creating another resource after fully / partially configuring an application on an EC2 instance
  • Properties:
    • Count: Number of success signals required to continue stack creation (default is 1)
    • Timeout: Time to wait for the number of signals (max. of 43200 seconds / 12 hours)
    • Handle: Reference to AWS::CloudFormation::WaitConditionHandle, a pre-signed URL enabling you to send a signal (success / failure) to a WaitCondition
  • For only EC2 and ASG, it's recommended to use CreationPolicy

How to use a WaitCondition?

  • Use the WaitConditionHandle pre-signed URL to signal the WaitCondition
    • Use cfn-signal helper script
    • Make a HTTP PUT request

Dynamic references

  • Reference external values stored in SSM Parameter Store and AWS Secrets Manager within templates
  • CloudFormation retrieves the value of a specified reference during stack and change set operations
  • Ex. retrieve RDS DB instance master password from Secrets Manager
  • Supports:
    • ssm: For plaintext values stored in SSM Parameter Store
    • ssm-secure: For secure strings stored in SSM Parameter Store
    • secretsmanager: For secret values stored in AWS Secrets Manager
  • Can have up to 60 dynamic references in a template
  • '{{resolve:service-name:reference-key}}'

ssm

  • Reference values stored in SSM Parameter Store of type String and StringList
  • Latest version is used if not specified
  • Doesn't support public SSM parameters (ex. Amazon Linux 2 AMI)
  • '{{resolve:ssm:parameter-name:version}}'

ssm-secure

  • Reference values stored in SSM Parameter Store of type SecureString
  • Ex. passwords, license keys, etc.
  • CloudFormation never stores the actual parameter value
  • Latest version is used if not specified
  • Only use with supported resources
  • '{{resolve:ssm-secure:parameter-name:version}}'

secretsmanager

  • Retrieve entire secrets or secret values stored in Secrets Manager
  • Ex. database credentials, passwords, API keys, etc.
  • To update a secret, you must update the resource containing the secretsmanager dynamic reference
  • '{{resolve:secretsmanager:secret-id:secret-string:json-key:version-stage:version-id}}'

UpdatePolicy

  • Specify how CloudFormation handles updates to the following resources:
    • AWS::AppStream::Fleet
    • AWS::AutoScaling::AutoScalingGroup
      • AutoScalingRollingUpdate
      • AutoScalingReplacingUpdate
      • AutoScalingScheduledAction
    • AWS::ElastiCache::ReplicationGroup: Modify replication group's shards by adding or removing shards, rather than replacing a resource
    • AWS::Elasticsearch::Domain: Update an Elasticsearch domain to a new version, rather than replacing a resource
    • AWS::Lambda::Alias: CloudFormation performs CodeDeploy deployment when the version changes on the alias

AutoScalingRollingUpdate

  • Specify whether CloudFormation updates instances in an ASG in batches or all at once
  • During stack update rollback operation, the old template's UpdatePolicy is used before the new one

AutoScalingReplacingUpdate

  • Specify whether CloudFormation replaces an ASG with a new one or replaces the instances in the ASG
  • CloudFormation retains old ASG until new one is finished creating
  • If update fails, CloudFormation rolls back to the old ASG and deletes the new ASG

AutoScalingScheduledAction

  • Specify how CloudFormation handles updates for group size properties (MinSize, MaxSize, DesiredCapacity) of an ASG with a scheduled action
  • Group size properties of an ASG can change at any time
  • During a stack update, the group size property values of your ASG are always set to values defined in your ASG resource, even with a scheduled action in effect
  • Useful for preventing CloudFormation from changing group size property values when a scheduled action is in effect, unless you modify values in your template

Registry

  • Contains private and public extensions (resource types and modules)
  • Extensions are artifacts that augments functionality of CloudFormation resources and properties
    • Registered in the CloudFormation Registry
    • Can be written by Amazon, APN Partners, Marketplace sellers and the community
  • Extension types:
    • Private extensions: You created or are shared with you
    • Public extensions: Provided by AWS (ex. AWS::DynamoDB::Table)
  • Use CLI to create extensions

Resource types

  • Model and provisions resources using CloudFormation
  • Ex. create a custom resource that doesn't exist in CloudFormation
  • Follow the structure Organization::Service::Resource
  • Resource type package consists of a JSON schema defining your type and handlers that perform the required actions (create, update, delete, read, list)
  • Steps to create:
    • Model: Create and validate schema that serves as the definition of your resource type
    • Develop: Write a handler that defines five core operations (create, read, update, delete, list) on your resource type and test locally
    • Register: Register the resource type with CloudFormation so that it can be used in your templates
  • Handlers can be written in Python, Java, TypeScript and Go

CloudFormation CLI

  • Develop and test AWS / 3rd party extensions
  • Register extensions for use in CloudFormation
  • Supports Java, Go, Python and TypeScript to write your own extensions

3rd Party resource types

  • 3rd party vendors create resource types using CloudFormation CLI
  • Can be downloaded an added to your account via the CloudFormation Registry

Custom resources vs resource types

  • Custom resource
    • Operations: Create, update, delete
    • Languages: Any languages that Lambda supports
    • Execution location: In your account (Lambda function)
    • Billing: Lambda function invocations
    • Does not use registry and not integrated with drift detection and change sets
  • Resource type
    • Operations: Create, update, delete, read, list
    • Languages: Python, Java, Go, TypeScript
    • Execution location: By AWS
    • Billing: Handler operations per month
    • Uses registry and integrates with drift detection and change sets

Modules

  • Package resources and configurations for use across stack templates
  • Useful for keeping resource configurations aligned with best practices and using code written by experts
  • Contains:
    • Template sections: Resources, outputs, etc.
    • Module parameters: Input custom values to your module
  • Should follow the structure Organization::Service::Resource::MyModule 
    Parameters:
    VersioningConfigurationParam:
        Type: String
        Description: "Versioning configuration"
        AllowedValues: ['Enabled', 'Suspended']

Resources:
    MyBucket:
        Type: AWS::S3::Bucket
        DeletionPolicy: Retain
        Properties:
            AccessControl: Private
            VersioningConfiguration:
                Status: !Ref VersioningConfigurationParam

Reference resources in a module

  • Resources in a module can be referenced by logical names
  • A complete logical name follows the format ModuleLogicalName.ResourceLogicalName or ModuleLogicalNameResourceLogicalName
  • Use GetAtt and Ref intrinsic functions to access property values

Resource import

  • Import existing resources into existing / new stacks
  • Don't need to delete and re-create resources as part of a stack
  • During the import operation, you need:
    • A template describing the entire stack (original resources and target import resources)
    • A unique identifier for each target resource
  • Each resource to import must have a DeletionPolicy attribute and Identifier
  • CloudFormation performs the following validation:
    • Check if resource to import exists
    • Properties and configuration values adhere to the resource schema
    • Resource's required properties are specified
    • Resource to import doesn't belong to another stack
  • CloudFormation doesn't check the template configuration matches the actual configuration
  • Recommended to run drift detection on imported resources after operation
  • Use cases:
    • Create new stack from existing resources
    • Import existing resources into existing stack
    • Move resources between stacks
    • Remove resource from a stack
    • Remediate a detected drift
    • Moving nested stack from parent stack and import it into another parent stack
    • Nesting an existing stack

AWS SAM

  • Serverless Application Model (SAM) is a framework for developing and deploying serverless applications
  • All configuration is YAML code
  • Generate complex CloudFormation from a simple SAM YAML file
  • Supports features from CloudFormation: Outputs, Mappings, Parameters, Resources, etc.
  • Only two commands to deploy to AWS
  • SAM can also use CodeDeploy to deploy Lambda functions and help you run Lambda, API Gateway, DynamoDB locally
  • Contains:
    • Transform: 'AWS::Serverless-2016-10-31'
    • Code:
      • AWS::Serverless::Function
      • AWS::Serverless::Api
      • AWS::Serverless::SimpleTable
    • Package and deploy:
      • aws cloudformation package / sam package
      • aws cloudformation deploy / sam deploy

AWS Cloud Development Kit (CDK)

  • Define cloud infrastructure using a programming language (JS, TS, Python, Java, .NET)
  • Contains high level components called "constructs"
  • Code is "compiled" into a CloudFormation template, allowing you to deploy infrastructure and application runtime code together
  • Useful for Lambda functions and Docker containers in ECS / EKS
  • Can also import or migrate a CloudFormation template into AWS CDK

SAM vs CDK

  • SAM:
    • Serverless
    • Write template declaratively in JSON or YAML
    • Quickly started with Lambda
    • Leverages CloudFormation
  • CDK:
    • All AWS services
    • Write infrastructure in a programming language
    • Leverages CloudFormation

Macros

  • Perform custom processing on templates (ex. find-and-replace, transformations)

  • Ex. AWS::Serverless transforming a SAM template into a compliant CloudFormation template

  • Defining a macro:

    • Lambda function to perform the template processing
    • A resource of type AWS::CloudFormation::Macro (create a stack containing this resource)

  • Can process the entire template (reference the macro in Transform section of template) or a snippet of a template (reference the macro in Fn::Transform)

  • CloudFormation generates a ChangeSet that includes the processed template

    Transform: MyMacro

Resources:
    MyBucket:
        Type: AWS::S3::Bucket
        Fn::Transform:
            Name: MyMacro
            Parameters:
                Key: Value

  • Extra notes on macros:

    • Can't use a macro in the same template you register it in
    • Can't include macros within macros
    • Not supported in StackSets
    • Macros are evaluated in order, processed from deepest to shallowest

Other intrinsic functions

  • Fn::Join
    • Join valeus with a delimiter
    !Join [:, [a, b, c]]
  • Fn::Cidr
    • Returns an array of CIDR address blocks
    • Parameters:
      • ipBlock: CIDR address block to be split
      • count: Number of CIDRs to generate (1 - 256)
      • cidrBits: Number of subnet bits of the CIDR, inverse of the subnet mask (32 - subnet_mask)
    !Cidr ['192.168.0.0/24', 6, 5]
  • Fn::GetAZs
    • Returns an array of AZ in a region in alphabetical order
    !GetAZs region
  • Fn::Select
    • Returns a single object from an array of objects by index
    !Select ['1', ['apples', 'grapes', 'oranges', 'mangoes']]
  • Fn::Split
    • Split a string into a set of string values
    !Split [ '|', 'a|b|c']

CloudFormation best practices

  • Layered architecture (horizontal) vs. service-oriented architecture (vertical)
  • Use cross-stack references to reference a VPC or subnet for example
  • Template should be environment agnostic, dev / test / prod and cross regions / accounts
  • Use nested stacks to reuse common template patterns
  • Don't embed credentials into templates, use parameters with NoEcho or Dynamic references
  • Use cfn-init and other helper scripts
  • Validate templates
  • Use specific parameter types and constraints
  • Automate elements of the stack to prevent state mismatch
  • Verify changes with ChangeSets
  • Use stack policies to prevent critical components from being deleted after creation
  • Use code reviews and revision controls to manage templates