AWS CloudFormation

Section 4: CloudFormation

• Can use YAML or JSON for CloudFormation templates

Update Behaviour

• Updates resources based on differences between initial and current template
• Update with no interruption:
  • Without disruption resource operation & no changing physical ID
• Update with Some interruption:
  • Updating EC2 instance type for example
• Replacement:
  • Recreating the resource with new Physical ID

Application Composer

• Visually design and build sever-less applications quickly
• Deploy AWS infrastructure code as a non-expert
• Generates IaC using CloudFormation
• Good to quickly draft a CloudFormation template

Deploying CloudFormation Templates

Manual Way:

• Editing templates in application composer/code editor
• Use console to input parameters elastic

Automated Way:

• Editing templates in YAML file
• Use CLI or CD (Continuous Delivery) tool
• Recommended for fully automating work flow

Section 5: Parameters

• A way to provide inputs to your template
• Important to know about if you want to reuse templates, or some inputs cannot be determined ahead of time

Settings

Can be controlled by these settings:

• Type
  • String
  • Number
  • CommaDelimitedList
  • List(Number)
  • AWS-Specific Parameter
  • SSM Parameter
• Description
• ConstraintDescription
• Min/MaxLength
• Default
• AllowedValues
• AllowedPattern
• NoEcho

The Fn:Ref function can be leveraged to reference parameters. The shorthand for this in YAML is !Ref

SSM Parameter Type

• Reference parameters in Systems Manager Parameter Store
• Specify SSM parameter key as the value
• CloudFormation always fetches the latest value
• CloudFormation doesn't store secure string values Type: AWS::SSM::Parameter::Value(String) is an example Default: /dev/ec2/instanceType

Section 6: Resources

• Only mandatory section in the template
• Resources are declared and can reference each other
• AWS figures out creation, updates and deletes of resources for us
• There are over 700 types of resources
• Resource types identifiers are of the form: AWS::aws-product-name::data-type-name

Optional Attributes for Resources

• DependsON
  • Useful to draw a dependency between two resources
• DeletionPolicy
  • Prevents resources from being deleted even if the CloudFormation stack is deleted
• UpdateReplacePolicy
  • Protect resources from being replaced during a CloudFormation update
• CreationPolicy
• UpdatePolicy
• Metadata
  • Anything you want

DependsOn

• Specify that the creation of a specific resource follows another
• A resource with this is created after the DependsOn resource
• Applied automatically when using !Ref and !GetAtt

DeletionPolicy

• Control what happens when the template is deleted
• Use with any resource
• DeletionPolicy=Retain
• DeletionPolicy=Snapshot
• DeletionPolicy=Delete (Default Behaviour)

Update Replace Policy

• Only applies to resources replaced during stack updates

Section 7: Mappings

• Fixed variables within the template

• Handy to differentiate between different environments, regions, AMI types etc

• All the values are hardcoded within the template

• Great when you know in advance all the values that can be taken and that they can be deduced from variables like:

  • Region
  • AZ
  • AWS account
  • Environment

• They allow safer control over the template

Accessing Mapping Values

• We use Fn::FindInMap to return a named value from a specific key
• !FindInMap [ MapName, TopLevelKey, SecondLevelKey ]

Pseudo Parameters

• AWS::AccountId is an example, will return the account id
• Many others like AWS::Region

Section 8: Outputs

• Declares optional output values that we can import into other stacks
• Can view the outputs in the Console or CLI
• Best way to perform some collaboration cross stack

Outputs:
  StackSSHSecurityGroup:
    Description: The SSH Security Group for our Company
    Value: !Ref MyCompanyWideSSHSecurityGroup
    Export:
      Name: SSH Security Group

We can reference another stacks output value by using:

• !ImportValue 'SomeValue'
• SomeValue will be a globally unique value, and we won't know which stack exported it
• If you are importing a value from another stack, you will not be able to delete the original stack until you get rid of the dependency

Section 9: Conditions

• Used to control the creation of resources or outputs based on a condition

Conditions:
  CreateProdResources: !Equals [!Ref EnvType, prod]

• Conditions can be applied to resources/outputs/etc

Resources:
  MountPoint:
    Type: AWS::EC2::VolumeAttachment
    Condition: CreateProdResources

Fn:GetAtt

• Attributes are attached to any resource you create
• Different resources have different attributes
• For example, the AZ of an EC2 machine

Section 10: Rules

• Rules are used to perform parameter validations based on the values of other parameters
• For example: Ensure that all subnets selected are within the same VPC
• Each rule consists of a rule condition and an assertion
• If you don't define a rule condition, the rule's assertions will take effect with every create/update operation

Example

Enforce users to provide an ACM certificate ARN if they configure an SSL listener on an Application Load Balancer

Rules:
    IsSSLCertificate:
        RuleCondition: !Equals
            - !Ref UseSSL
            - Yes
        Assertions:
            - Assert: !Not
               - !Equals
                - !Ref ALBSSLCertificateARN
                - ''
            AssertDescription: 'ACM certificate value cannot be empty if SSL is required'

Rule-Specific Intrinsic Functions

• Used to define a Rule condition and assertions
• Can only be used in the Rules section
• Functions can be nested, but the result of a rule condition or assertion must be either true or false

Section 11: Metadata

• Optional metadata to include arbitrary YAML which provides details about the template or resource
• Three metadata keys which have special meaning
  • Interface: Define group and ordering of input parameters in console
  • Authentication: Specify authentication credentials for files or sources
  • Init: Define configuration tasks for cfn-init

Section 12: CFN Init and EC2 User Data

• Typically want instances to be self-configured so that they cna perform the job they're supposed to perform
• Can fully automate your EC2 fleet state with CloudFormation init

EC2 User Data

• Customise instance at runtime
• In a template, must pass the script through the Fn:Base64 function

CloudFormation Helper Scripts

• 4 python scripts directly on Amazon Linux 2 AMI, (or can be installed manually)
  • cfn-init: Retrieve/interpret resource metadata, installing packages and creating files, must be triggered by a script within EC2 user data
  • cfn-signal: Simple wrapper to signal with a CreationPolicy or WaitCondition, enabling you to synchronise other resources in the stack with the application being ready
  • cfn-get-metadata: A wrapper, makes it easy to retrieve all metadata defined for a resource, or path to a key or subtree of resource data
  • cfn-hup: Daemon to check for updates to metadata and execute custom hooks when changes are detected

AWS::CloudFormation::Init

A config contains the following and is executed in that order

1. Packages: Install packages from different repositories
2. Groups: When you want multiple groups within your EC2 instance
3. Users: Add users to specific groups, can specify uid and home directory
4. Sources: Download whole compressed archives and unpack on the EC2 instance
5. Files: Full control over any content you want. Can come from specific URL or written inline
6. Commands: Run one at a time alphabetically, can provide a test to control if the command is executed
7. Services: Launch services at EC2 instance launch

AWS::CloudFormation::Authentication

• Used to specify authentication credentials for files or sources in AWS::CloudFormation::init
• Either basic for URL or S3 for bucket

Fn:Sub

• !Sub for shorthand, used to substitute variables from a text
• String must contain $VariableName

Creation Policy

• Prevents resource's status from reaching CREATE_COMPLETE until CloudFormation receives either a number of success signals, or a timeout occurs

Section 13: CloudFormation Drift

• CloudFormation does not protect you against manual configuration changes in the account
• Resources have 'drifted' if they have been modified since the original template
• CloudFormation drift can be used to determine what has been changed
• Can detect drift on an entire stack, or just a single resource
• Can resolve stack/resource with Resource Import

Section 14: CloudFormation Nested Stacks

• Stacks as part of other stacks
• Isolate repeated patterns / common components in separate stacks and call them from other stacks
• Example: Load balancer configuration that is re-used
• Nested stacks are considered best practice

SSHSecurityGroupStack:
  Type: AWS::CloudFormation::Stack
  Properties:
    # ...

• Then in an EC2 instance for example, we would reference the securityGroupId from the output of the other stack
• To update, ensure the updated nested stacks are uploaded onto S3 first, then re-upload your root stack
• To delete, you need to apply changes from the top level stack

**Cross Stacks vs Nested Stacks

Cross Stacks

• Helpful when stacks have different lifecycles
• Use outputs Export and Fn::ImportValue
• Useful when you need to pass export values to many stacks (VPC Id, etc)

Nested Stacks

• Helpful when components must be re-used
• Example: Re-use properly configured Application Load LoadBalancer
• The nested stack is only important to the higher-level stack (it's not shared)

Section 15: Stack Sets

• Create, update or delete stacks across multiple accounts and regions with a single operation/template
• Administrator account can create these
• Target accounts to create, update, and delete stack instances from StackSets
• When you update a stack set, all associated stack instances are updated throughout all accounts and regions
• Regional service
• Can be applied to all accounts of an AWS organisation

StackSet Operations

Create StackSet

• Provide template + target accounts/regions

Update StackSet

• Updates always affect all stacks (can't selectively update some stacks in the StackSet but not others)

Delete Stacks

• Delete stack and its resources from target accounts/regions
• Delete stack from your StackSet (stack will continue to run independently)
• Delete all stacks from your StackSet (prepare for StackSet deletion)

Delete StackSet

• Must delete all stack instances within StackSet to delete it

StackSet Deployment Options

Deployment Order

• Order of regions where stacks are deployed
• Operations performed one region at a time

Maximum Concurrent Accounts

• Max number/percentage of target accounts per region to which you can deploy stacks at one time

Failure Tolerance

• Max number/percentage (target accounts per region) of stack operation failures that can occur before CloudFormation stops operation in all regions

Region Concurrency

• Whether StackSet deployed into regions Sequential (Default) or Parallel

Retain Stacks

• Used when deleting StackSet to keep stacks and their resources running when removed from StackSet

Permission Models for StackSet

Self-Managed Permissions

• Create the IAM roles in both admin and target accounts
• Deploy to any target account in which you have permissions to create IAM role

Service-Managed Permissions

• Deploy to accounts managed by AWS organisations
• StackSets create the IAM roles on your behalf (enable trusted access with AWS organisations)
• Must enable all features in AWS organisations
• Ability to deploy accounts added to your organisation in the future (automatic deployments)

StackSets with AWS Organisations

• Ability to automatically deploy Stack instances to new Accounts in an Organisation
• Can delegate StackSets administration to member accounts in AWS organisation
• Trusted access with AWS organisations must be enabled before delegated administrators can deploy to accounts managed by organisations

StackSet Drift Detection

• Performs drift detection on the stack associated with each stack instance in the StackSet
• Will identify unmanaged changes
• Change made through CloudFormation aren't considered a drift, however this is not recommended

StackSet Deletion

• First need to delete all stacks within the StackSet, must select retain as you do this

Section 16: Deployment Options

ChangeSets

• When you update a stack, you need to know what changes will happen before it applying them for greater confidence
• ChangeSets won't say if the update will be successful
• For Nested Stacks, you will see the changes across all stacks

Stack Creation Failures

• If a stack creation fails, you will get the status ROLLBACK_COMPLETE
• This means:
  1. CloudFormation tried to create some resources
  2. One resource creation failed
  3. CloudFormation rolled back the resources
  4. The stack is in failed created ROLLBACK_COMPLETE state
• To resolve the error, you must delete the failed stack and create a new stack

Rollback Triggers

• Enables CloudFormation to rollback stack create/update operation if that operation triggers CloudWatch Alarm
• CloudFormation monitors the specified CloudWatch alarms during:
  • Stack create/update
  • The monitoring period (after all resources have been deployed), 0 to 180 minutes
• If any alarm is triggered, CloudFormation rolls back the entire stack operation
• If a monitoring time is set, but no rollback triggers, CloudFormation still waits the period before cleaning up old resources
• Up to 5 CloudWatch alarms

Continue Rolling Back an Update

• A stack goes into the UPDATE_ROLLBACK_FAILED state when CloudFormation can't roll back all changes during an update
• A resource can't return to its original state, causing the rollback to fail
• Example: Roll back to an old database instance that was deleted outside CloudFormation
• Solutions:
  • Fix the errors manually outside of CloudFormation and then continue update rollback the stack
  • Skip the resources that can't be rollback successfully (CloudFormation will mark the failed resources as UPDATE_COMPLETE)
• You can't update a stack in this state
• For nested stacks, rolling back the parent stack will attempt to roll back all child stacks as well

Stack Policy

• JSON doc which defines the updates actions allowed on stack resources
• Prevent stack resources from being unintentionally update/deleted during a stack update
• By default all update actions are allowed on all resources
• When enabled, all resources protected by default
• Principal supports only the wildcard (*)
• To update protected resources:
  • Create a temporary policy that overrides the stack policy
  • The override policy doesn't permanently change the stack policy
• Once created, can't be deleted (edit to allow all update actions on all resources)

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "Update:*",
      "Principal": "*",
      "Resource": "*"
    },
    {
      "Effect": "Deny",
      "Action": ["Update:Replace", "Update:Delete"],
      "Principal": "*",
      "Resource": "LogicalResourceId/MyDDBTable",
      "Condition": {
        "StringEquals": {
          "ResourceType": ["AWS::DynamoDB::Table"]
        }
      }
    }
  ]
}

Termination Protection on Stacks

• Prevents accidental deletes of CloudFormation
• Applied to any nested stacks
• Tighten your IAM policies (on UpdateTerminationProtection, can guarantee no accidental deletions)

Service Role

• IAM role which allows CloudFormation to create/update/delete on behalf of you
• By default, uses a temp session which is generated from your user credentials (will have the same permissions as you)
• Use cases:
  • Want to achieve least privilege principle
  • Don't want to give user all required permissions to create the stack resources

Quick-Create Links for Stacks

• Custom URLs that used to launch CloudFormation stacks quickly from AWS Console
• Reduce the number of wizard pages and the amount of user input that's required
• For example: Create multiple URLs that specify different values for the same template
• CloudFormation ignores parameters:
  • That don't exist in the template
  • That are defined with the NoEcho property set to true

Section 17: Continuous Delivery

• Use CodePipeline to build a continuous delivery workflow (building a pipeline for CloudFormation stacks)
• Rapidly and reliably make changes to your AWS infrastructure
• Automatically build and test changes to your CloudFormation templates before promoting them to production stacks
• For example:
  • Create a workflow that automatically builds a test stack when you submit a CloudFormation template to a code repository

Section 18: Resource Advanced - Custom Resources, Registry, Modules

Custom Resources

• Enable you to write custom provision login in templates that AWS CloudFormation runs anytime you create, update or delete stacks
• Defined the template using AWS::CloudFormation::CustomResource or Custom::MyCustomResourceTypeName (recommended)
• Either Amazon SNS-backed Custom Resource or AWS Lambda-backed Custom Resource

How to Define a Custom Resource

• ServiceToken specifies where CloudFormation sends requests to, such as Lambda ARN or SNS ARN
• Input data parameters (options)

Resources:
  LogicalResourceName:
    Type: Custom::MyCustomResourceTypeName
    Properties;
      ServiceToken: service_token

Wait Condition

• Makes CloudFormation pause the creation of a stack and wait for a signal before it continues to create the stack
• Example: Start the creation of another resource after an application on an EC2 instance is fully/partially configured
• Properties:
  • Count: Number of success signals required to continue stack creation (1 is default)
  • Timeout: Time to wait for the number of signals
  • Handle: Reference to AWS::CloudFormation::WaitConditionHandle

Resources:
  WaitConditionLogicalId:
    Type: AWS::CloudFormation::WaitCondition
    Properties:
      Count: Integer
      Handle: String
      Timeout: String

AWS::CloudFormation::WaitConditionHandle

• A presigned URL enables you to send a signal to a WaitCondition

Resources:
  WaitConditionHandleLogicalId:
    Type: AWS::CloudFormation:::WaitConditionHandle
    Properties:
      # ...

How to use a WaitCondition

• To signal the WaitCondition, you use the WaitConditionHandle pre-signed URL
  • Using the cfn-signal helper script
  • Make a HTTP PUT request

Dynamic References

• References external values stored in SSM Parameter Store and AWS secrets manager within CloudFormation templates
• CloudFormation retrieves the value of the specified reference during stack and change set operations
• For Example: Retrieve RDS DB instance master password from AWS secrets manager
• Supports:
  • ssm: For plaintext values
  • ssm-secure: For secure strings
  • secretsmanager: For secret values in AWS secrets manager

ssm

• Reference values stored in SSM parameter store of type String and StringList
• If no version specified, CloudFormation uses the latest version
• Doesn't support public SSM parameters

Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: '{{resolve:ssm:S3AccessControl:2}}'

ssm-secure

• References values stored in SSM Parameter Store of type SecureString
• Like passwords, license keys, etc
• CloudFormation never stores the actual parameter values
• Only can be used with supported resources

secretsmanager

• Retrieve entire secret or secret values stored in AWS Secrets Manager
• Like database credentials, passwords etc
• To update a secret, you must update the resource containing the secretsmanager dynamic reference (one of the resource properties)
• 'resolve:secretsmanager:secret-it:secret-string:json-key:version-stage:version-id'

UpdatePolicy

• Specify how CloudFormation handles updates for:
• AWS::AppStream::Fleet
• AWS::AutoScaling::AutoScalingGroup
  • RollingUpdate
  • ReplacingUpdate
  • ScheduledAction
• AWS::ElastiCache::ReplicationGroup
• AWS::Elasticsearch::Domain
• AWS::Lambda::Alias

AutoScalingRollingUpdate

• Specify whether CloudFormation updates instances that are in an ASG in batches or all at once
• During stack update rollback operation, CloudFormation uses the UpdatePolicy in the old template before the current stack update operation

AutoScalingReplacingUpdate

• Specify whether CloudFormation replaces an ASG with a new one, or replaces the instances within the ASG
• CloudFormation retains the old ASG until it finishes creating the new one
• If update fails, it will rollback to the old ASG

AutoScalingScheduledAction

• Specify how CloudFormation handles updates for the group size properties of an ASG that has a scheduled action
• With scheduled actions, the Group size properties of an ASG can change at any time
• During stack update, CloudFormation always sets the group size property values of your ASG to values defined in ASG resource, even a scheduled action is in effect
• Usage: To prevent CloudFormation from changing group size property values when you have scheduled action in effect, unless you modify the values in your template

CloudFormation Registry

• Contains private and public extensions (Resource Types and Modules)
• Extensions are artifacts that augments the functionality of CloudFormation resources and properties
• Extensions registered in CloudFormation Registry
• Extensions can be written by Amazon, APN partners, Marketplace sellers and the community
• Extension types:
  • Private extensions: You created or shared with out
  • Public extensionsL: Provided by AWS (Ex: AWS::DynamoDB:Table)

Resource Types

• Model and provisions resource using CloudFormation
• For example, create a custom resource that doesn't exist in CloudFormation
• Follows the structure - Organisation::Service::Resource

CloudFormation CLI

• Register extensions for use in CloudFormation
• Supports Java, Go, Python, TypeScript to write your own extensions

Custom Resources are Different from Resource Types

Modules

• Package resources and their configurations for use across stack templates
• Different to nested stacks
• Use cases:
  • Keep resource configurations aligned with best practices
  • Use code written by experts
• Module contains
  • Template sections: Resources, outputs, ...
  • Module parameters: Input custom values to your module
• It should follow the structure Organisation::Service::Resource::MODULE
• Registered in CloudFormation registry as private extensions
• Modules are versioned and can contain nested modules

Module Parameters

• Enables you to input custom values to you module from the template/module that contains it
• Defined the same as template parameters
• You can pass template (parent) parameters to module parameters
• You can't perform constraint checking on Modules Parameters (AllowedValues, AllowedPattern etc)

How to Define a Module

# My::S3::SampleBucket::Module
Parameters:
  BucketName:
    Type: String
    Description: Name for you sample bucket

Resources:
  MyBucket:
    Type: My::S3::SampleBucketPrivate::MODULE
    Properties:
      BucketName: !Ref BucketName
      AccessControl: 'Private'

Reference Resources in a Module

• Resources in a Module can be referenced by logical names
• The fully qualifies logical name
  • ModuleLogicalName.ResourceLogicalName
  • ModuleLogicalNameResourceLogicalName
• Use GetAtt and Ref intrinsic functions to access property values as usual

Section 19: Generating CloudFormation Templates: Imports, SAM, CDK & Macros

Resource Imports

• Allows imports of existing resource into new/existing stacks
• You don't need to delete and re-create the resource as part of a CloudFormation stack
• During import operation, you'll need
  • A template that describes the entire stack (original stack resources & target resources to import)
  • A unique identifier for each target resource
• Each resource to import must have a DeletionPolicy attribute (any value) & Identifier
• Can't import the same resource into multiple stacks

Resource Import

• CloudFormation performs the following validations:
  • The resource to import exists
  • Properties and configuration values adhere to the resource schema
  • The resource's required properties are specified
  • The resource to import doesn't belong to another stack
• CloudFormation doesn't check that the template configuration matches the actual configuration
• Recommended to run Drift Detection on imported resources after the operation
• Use Cases:
  • Create a new stack from existing resources
  • Existing resources imported into existing stack
  • Move resources between stacks
  • Remediate a detected drift
  • Nesting an existing stack

AWS SAM

• SAM = Sever-less Application Model
• Framework for developing and deploying serverless applications
• All the configuration is YAML code
• Generate complex CloudFormation from simple SAM YAML file
• Supports anything from CloudFormation: Outputs, Mappings, Parameters, Resources
• Only two commands to deploy to AWS

AWS SAM - Recipe

• Transform Header indicates it's SAM template:
  • Transform: 'AWS::Serverless-2016-10-31'
• Write Code
  • AWS::Serverless::Function
  • AWS::Serverless::Api
  • AWS::Serverless::SimpleTable
• Package & Deploy
  • aws cloudformation package / sam package
  • aws cloudformation deploy / sam deploy

AWS Cloud Development Kit (CDK)

• Define your cloud infrastructure using a familiar language like JS, Python or Java
• Contains high level components known as constructs
• The code is 'compiled' into a CloudFormation template (JSON/YAML)
• You can therefore deploy infrastructure and application runtime code together
  • Great for Lambda functions
  • Great for Docker containers in ECS/EKS
• Can import/migrate a CloudFormation template into/to AWS CDK

CloudFormation Macros

• Perform custom processing of CloudFormation template
• For example, AWS::Serverless which takes an entire template written in SAM syntax and transforms it into a compliant CloudFormation template
• To define a macro, you need:
  • Lambda function: perform the template processing
  • A resource of type AWS::CloudFormation::Macro
• You can process
  • The entire template
  • A snippet of a template

Resources:
  MacroDefinition:
    Type: AWS::CloudFormation::Macro
    Properties:
      Name: MyMacro
      Description: MyMacro Description
      FunctionName: lambda_function_ARN

Resources:
  MyMacro:
    Type: AWS::CloudFormation::Macro
    Properties:
      Name: MyMacro
      FunctionName: arn:aws:lambda:us-east-1:123456789012:function:MyFunction

Section 20: Open-Source and 3rd Part Project for CloudFormation

Using the AWS CLI

• CLI can be used to create, update or delete CloudFormation templates
• It's very convenient when you start automating your deployments
• It's also great for managing parameters in a file