Graduate Program KB

  1. CloudFormation Introduction
  2. Getting Started with CloudFormation
  3. Parameters
    1. SSM Parameter Type
  4. Resources
    1. DependsOn
    2. DeletionPolicy
    3. UpdateReplacePolicy
  5. Mappings
    1. Pseudo Parameters
  6. Outputs
  7. Conditions
    1. Fn::GetAtt
  8. Rules
  9. Metadata
  10. Cloudformation Init and EC2 User Data
    1. AWS::CloudFormation::Authentication
  11. CloudFormation Drift
  12. Nested Stacks
  13. StackSets

CloudFormation Introduction

  • CloudFormation - A declarative way of outlining your AWS Infrastructure for most available resources. CloudFormation creates these for you in the right order with the exact configuration you specified.

Benefits of AWS CloudFormation:

  • Infrastructure as Code (IaC)
    • No resources are manually created, excellent for control
    • Code can be version controlled using Git
    • Changes to the infrastructure are reviewed through code
  • Cost
    • Each resource within the stack is tagged with an identifier so you can see how much a stack costs you
    • You can estimate the costs of your resources using the CloudFormation template
    • You could automate creation/deletion to save costs
  • Productivity
    • Ability to destroy and re-create on the fly
    • Automated generation of diagram for your templates
    • Declarative programming (no need to order or orchestrate tasks)
  • Separation of concerns - Create many stacks for many apps and many layers
  • Don't reinvent the wheel - You can leverage existing templates

How does CloudFormation work?

  1. Templates are uploaded to S3
  2. CloudFormation references template in S3 to create stack
  3. Stack creates AWS resources
  • To update a template you must re-upload the new version to S3
  • Stacks are identified by a name
  • Deleting a stack deletes every single artifact that was created by CloudFormation

Getting Started with CloudFormation

  • YAML and JSON are languages you can use for CloudFormation, with YAML being preferred
  • YAML supports:
    • Key-value pairs
    • Nested objects
    • Support Arrays
    • Multi-line strings
    • Comments

To create an S3 bucket (full list of properties here):

Resources:
  BucketName:
    Type: AWS::S3::Bucket
    Properties: {}

CloudFormation update behavior depends on which property you update for a resource:

  • Update with No Interruption - Updates without disrupting resources' operation & without changing physical ID, eg. updating the IAM instance profile of an EC2 instance
  • Update with Some Interruption - eg. updating an EC2 instance type
  • Replacement - Recreating the resource with a new physical ID. Creates the new resources, changes references in other resources to the new resource, deletes the old resource

CloudFormation Template Options:

  • Tags - Passed onto every resource in stack
  • Permissions - IAM role used to create stack
  • Notification Options - SNS
  • Timeout - How long to wait before CloudFormation operation is seen as a failure
  • Rollback on Failure
  • Rollback Configuration (Monitoring time & CloudWatch Alarm)
  • Stack Policy
  • Termination Protection
  • Quick-start Link

  • AWS Application Composer - Visually design and build serverless applications quickly in AWS
    • Generates IaC using CloudFormation
    • Allows visualization of existing CloudFormation code

CloudFormation Template Components

  1. AWSTemplateFormatVersion - "2010-09-09"
  2. Description - Comments about the template
  3. Transform - Specifies one or more macros used to process the template
  4. Metadata
  5. Resources - The AWS resources you wish to specify
  6. Parameters - The dynamic inputs for your template
  7. Mappings - Static variables for your template
  8. Outputs - References to what has been created
  9. Conditionals - List of conditions to perform resource creation
  10. Rules - Validate a parameter during stack creation/update

Deploying CloudFormation templates

  • Manual: Edit templates in Application Composer/code editor, use console to input parameters, etc
  • Automatic: Edit templates in YAML file, use the AWS CLI to deploy the templates or use a Continuous Deliver (CD) tool.

Parameters

  • Parameters are a way to provide inputs to your CF template
  • They're important to know about if:
    • You want to reuse your templates across the company
    • Some inputs can not be determined ahead of time
  • Parameters are extremely powerful, controlled and can prevent errors thanks to types
  • Parameters can be cross-validated using Rules
  • Use a parameter if the CF resource is likely to change in the future.
  • You don't have to re-upload templates to change parameter values
  • Parameter settings:
    • Type:
      • String
      • Number
      • CommaDelimitedList
      • List<Number>
      • AWS-Specific Parameter (matches with existing values in the AWS account. Supported types listed here)
      • List<AWS-Specific Parameter>
      • SSM Parameter (get parameter value from SSM Parameter store)
    • Description
    • ConstraintDescription
    • Min/MaxLength
    • Min/MaxValue
    • Default
    • AllowedValues (array)
    • AllowedPattern (regex)
    • NoEcho (Boolean)
  • Parameters can be used anywhere in a template except:
    • AWSTemplateFormatVersion
    • Description
    • Transform
    • Mappings
  • Reference Parameters using Fn::Ref or YAML's shorthand !Ref <parameter-name>
  • Use !Select[<index>, <parameter-name>] to select from a List parameter

SSM Parameter Type

  • CloudFormation always fetches the latest value
  • CloudFormation doesn't store Secure String values
  • Validation done on SSM parameter keys but not values Example:
Parameters:
  InstanceType:
    Description: WebServer EC2 instance type
    Type: AWS::SSM::Parameter::Value<String>
    Default: /dev/ec2/instanceType

  ImageId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2

Resources

  • Resources are mandatory in CloudFormation templates. They represent the different AWS components that will be created and configured.
  • Resources are declared and can reference each other.
  • There are over 700 types of resources
  • Resource type identifiers are of the form AWS::<product-name>::<data-type-name>
  • You can create a dynamic number of resources using CloudFormation Macros and Transform
  • Almost every AWS Service is supported, but a few still aren't. You can work around these using Custom Resources Optional attributes for Resources include:
    • DependsOn - Creates a dependency between resources, e.g. ensure an ECS cluster is only created after creating an ASG
    • DeletionPolicy - Protect resources from being deleted even if the CloudFormation stack is deleted
    • UpdateReplacePolicy - Protect resources from being replaced during a CloudFormation update
    • CreationPolicy
    • UpdatePolicy
    • Metadata - Anything you want

DependsOn

Resources:
  MyS3Bucket:
    Type: AWS::S3::Bucket

  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref ImageId
      InstanceType: t2.micro
    DependsOn: MyS3Bucket

DeletionPolicy

  • Can be used with any resource
  • Options:
    • Retain - Specify what resources should be preserved/backed up in case of CloudFormation deletion
    • Snapshot - EBS Volume, ElastiCache Cluster, ElastiCache ReplicationGroup, RDS DBInstance, RDS DBCluster, Redshift Cluster, Neptune DBCluster
    • Delete - (default behaviour)

UpdateReplacePolicy

  • Controls what happens to a resource if you update a property whose update behaviour is Replacement
  • Can be used with any resource
  • Options:
    • Delete - Deletes the old resource and creates a new one with a physical ID (default behaviour)
    • Retain - Keeps the resource
    • Snapshot - EBS Volume, ElastiCache Cluster, ElastiCache ReplicationGroup, RDS DBInstance, RDS DBCluster, Redshift Cluster, Neptune DBCluster

Mappings

  • Mappings are fixed variables hardcoded within your CloudFormation template
  • Handy to differentiate between different environments, regions, AMI types, etc
  • Mappings are great when you know in advance all the values that can be taken and can be deduced from variables like Region, Availability Zone, AWS Account, Environment, etc. Use parameters when values cannot be predicted ahead of time.
  • To Access Map Values:
    • Fn::FindInMap, or
    • !FindInMap[MapName, TopLevelKey, SecondLevelKey]

Pseudo Parameters

  • Pseudo-Parameters are variables that can be used in any CloudFormation template at any time and are enabled by default
    Reference ValueExample Returned Value
    AWS::AccountId123456789012
    AWS::Regionus-east-1
    AWS::StackId

    arn:aws:cloudformation:us-east-1:123456789012:stack/MyStack/1c2fa620-982a-11e3-aff7-50e2416294e0

    AWS::StackNameMyStack
    AWS::NotificationARNs[arn:aws:sns:us-east-1:123456789012:MyTopic]
    AWS::NoValueDoesn't return a value
    AWS::Partitionaws, aws-cn, aws-us-gov
    AWS::URLSuffixamazonaws.com / amazonaws.com.cn

Outputs

  • The Outputs section declares optional outputs that can be imported into other stacks Example:
Outputs:
  StackSSHSecurityGroup:
    Description: The SSH Security Group for our company
    Value: !Ref MyCompanyWideSSHSecurityGroup
    Export:
      Name: SSHSecurityGroup
  • Outputs can be viewed in the AWS Console or using the AWS CLI
  • Outputs are the best way to perform cross-stack collaboration, eg.
resources:
  MySecureInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref ImageId
      InstanceType: t2.micro
      SecurityGroups:
        - !ImportValue SSHSecurityGroup

Conditions

  • Conditions are used to control the creation of resources or outputs based on a condition
  • Can be whatever you want them to be but common ones are:
    • Environment (dev/test/prod)
    • AWS Region
    • Any parameter value

Example usage

Conditions:
  # Defining a condition
  # True if parameter 'EnvType' is equal to the word 'prod'
  CreateProdResources: !Equals [!Ref EnvType, prod]

Resources:
  MountPoint:
    Type: AWS::EC2::VolumeAttachment
    # Using a condition
    # 'MountPoint' will only be created if condition is true
    Condition: CreateProdResources

The intrinstic function can be any of the following (use with either the Fn:: prefix or !):

  • !And
  • !Equals
  • !If
  • !Not
  • !Or

Fn::GetAtt

  • Attributes are attached to any resources you create
  • You can reference attributes of resources in your template with !GetAtt <resource-name>.<attribute>

Rules

  • Rules are used to perform parameter validations based on the values of other parameters (cross-parameter validation)
  • Each Rule consists of:
    • Condition (optional) under which a rule will be applied
    • Assertions - Describes what values are allowed for a particular parameter. Can contain one or more assertions

  • If you don't define a Rule Condition the rule's assertions will take effect with every create/update operation Example:
Rules:
  IsSSLCertificate:
  RuleCondition: !Equals
    - !Ref UseSSL
    - Yes
  Assertions:
    - Assert: !Not
        - !Equals
          - !Ref ALBSSLCertificateARN
          - ''
      AssertDescription: 'ACM certificate value can not be empty if SSL is required'
  • Supported functions:
    • Fn::And
    • Fn::Contains
    • Fn::EachMemberEquals
    • Fn::EachMemberIn
    • Fn::Equals
    • Fn::If
    • Fn::Not
    • Fn::Or
    • Fn::RefAll
    • Fn::ValueOf
    • Fn::ValueOfAll

Metadata

  • You can use the optional metadata section to include arbitrary YAML that provides details about the template or its resources
  • Meaningful Metadata keys:
    • Interface - Define grouping and ordering of input parameters when they're displayed in the AWS Console
    • Authentication - Used to specify authentication credentials for files or sources that you specify in Init
    • Init - Define configuration tasks for cfn-lint.

Example Interface usage:

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: 'Network Configuration'
        Parameters:
          - SubnetID
          - SecurityGroupID
      - Label:
          default: 'Amazon EC2 Configuration'

        Parameters:
          - InstanceType
          -

Cloudformation Init and EC2 User Data

  • CloudFormation Init and EC2 user data both enable you to fully automate your EC2 fleet state
  • UserData is defines the commands EC2 instances run on startup. Scripts can be passed through using the Function Fn::Base64, eg.
Resources:
  WebServer:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref ImageId
      InstanceType: t2.micro
      KeyName: !Ref KeyName
      SecurityGroups:
        - !Ref WebServerSecurityGroup
      UserData:
        Fn::Base64: |
          #! /bin/bash
          echo "Hello world!"
          apt install ...

  WebServerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable HTTP access via port 80 + SSH access
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          IpProtocol: tcp
          FromPort: 80
          ToPort: 80
        - CidrIp: !Ref SSHLocation
          IpProtocol: tcp
          FromPort: 22
          ToPort: 22

  • User data script logs are stored at /var/log/cloud-init-output.log

  • Issues with EC2 User Data

    • What if we want to have a very large instance configuration?
    • What if we want to evolve the state of the EC2 instance without terminating it and creating a new one?
    • How do we make EC2 user-data more readable?
    • How do we know or signal that our EC2 user-data script completed successfully?

  • There are 4 cloudformation helper scripts that come with Amazon Linux 2 AMIs or can be installed using yum

    • cfn-init - Used to retrieve and interpret the resource metadata, installing packages, creating files and starting services. Logs to /var/log/cfn-init.log
    • cfn-signal - A simple wrapper to signal a CreationPolicy or WaitCondition to CloudFormation, enabling you to synchronize other resources in the stack with the application being ready. Logs to /var/log/cfn-signal
    • cfn-get-metadata - A wrapper script making it easy to retrieve either all metadata defined for a resource or path to a specific key or subtree of the resource metadata. Relies on configuration files at /etc/cfn/cfn-hyp.conf and /etc/cfn/hooks.d/cfn-auto-reloader.conf
    • cfn-hup - A daemon to check for updates to the metadata and execute custom hooks when the changes are detected

  • Usual script flow: cfn-init -> cfn-signal -> (optionally) cfn-hup



  • A CloudFormation::Init config contains the following (executed in this order);

    • Packages - Used to download an install pre-packaged apps and components on Linux/Windows. Specify the package manager and a list of packages, along with their versions.
    • Groups - Define user groups
    • Users - Define users and which group they belong to
    • Sources - Download files and archives and place them on the EC2 instance
    • Files - Create files on the EC2 isntance using inline or can be pulled from a URL
    • Commands - Run a series of commands
    • Services - Launch a list of sysvinit services. Services will be started when files are changed or relevant packages are updated by cfn-init
AWS::CloudFormation::Init:
  config:
    packages:
      yum:
        node: ['21.1.2']
        httpd: [] # empty array = get latest
    groups:
      groupOne: {}
      groupTwo:
        gid: '45'
    users:
      myUser:
        groups:
          - 'groupOne'
          - 'groupTwo'
        uid: '50'
        homeDir: '/tmp'
    sources:
      /etc/myapp: https://s3.amazonaws.com/mybucket/myapp.tar.gz
      /etc/puppet: https://github.com/user1/cfn-demo/tarball/main
    files:
      /tmp/example.txt:
        content: Hello!
        mode: 000644
        owner: root
        group: root
        # authentication (optional; tied to AWS::CloudFormation::Authentication)
        # source (optional)
    commands:
      test:
        command: echo "${MAGIC}" > test.txt
        env:
          MAGIC: I come from the environment!
        cwd: '~' # working directory to run command in
        test: test ! -e ~/test.txt # tests if test.txt was created
        ignoreErrors: false
    services:
      sysvinit:
        nginx:
          enable: true
          ensureRunning: true
          files:
            - '/etc/nginx/nginx.conf'
          sources:
            - /var/www/html
          php-fastcgi:
            enabled: true
            ensureRunning: true
            packages:
              yum:
                - php
                - spawn-fcgi
  • You can have multiple configs in your Init by creating configSets invoked from your EC2 user data

AWS::CloudFormation::Authentication

Example:

AWS::CloudFormation::Authentication:
  S3AccessCreds:
  type: S3
  buckets:
    - !Sub ${MyS3BucketName}
  roleName: !Ref InstanceRole
InstanceRole:
  Type: AWS::IAM::Role
  Properties:
  AssumeRolePolicyDocument:
  Statement:
    - Action: 'sts:AssumeRole'
      Principal:
        Service: ec2.amazonaws.com
      Effect: Allow
      Sid: ''
  Policies:
    - PolicyName: AuthenticatedS3GetObjects
      PolicyDocument:
        Version 2012-10-17
        Statement:
          - Action:
              - s3:GetObject
            Resource: !Sub 'arn:aws:s3:::${MyS3BucketName}/*'
            Effect: Allow
  • Fn::Sub allows you to substitute variables from a text, for example with references or AWS Pseudo variables

CloudFormation Drift

  • Drifting - Resource configurations being manually changed after creation. The resource is 'drifting' from the specification.
  • CloudFormation Drift is a service that can detect drift for an entire stack or on individual stack resources.
  • Stack/resource drift can be resolved by using Resource Import

Nested Stacks

  • Nested Stacks are stacks that are part of other stacks.
    • They allow you to isolate repeated patterns / common components in separate stacks and call them from other stacks
    • Helpful when stack components must be re-used
  • To update a nested stack, ensure the updated nested stacks are uploaded onto S3, then re-upload your root stack template
  • To delete a nested stack, delete the root stack.
  • If your stacks have different lifecycles or import/export values from other stacks, use cross stacks instead

StackSets

  • StackSets allow you to create, update or delete stacks across multiple accounts and regions with a single operation/template
    • Admins create StackSets, target accounts create, update, delete stack instances from StackSets
    • When you update a StackSet, all associated stack instances are updated throughout all accounts and regions
    • Can be applied into all accounts of an AWS organization
  • StackSet operations:
    • Create - Provide template + target accounts/regions
    • Update - Updates always affect all stacks
    • Delete Stacks - Delete stack and its resources from target accounts/regions or from its stack
    • Delete StackSet - Must delete all stack instances within StackSet to delete it
  • StackSet deployment options:
    • Deployment Order
    • Maximum concurrent accounts
    • Failure tolerance