Infrastructure as Code (IaC)
Infrastructure as Code is a practice that takes writing custom scripts to automate tasks related to managing configurations, provisioning, and deployments to a new level. It leverages software development practices for infrastructure management.
IaC uses software development practices like version control, testing, and continuous integration/continuous deployment (CI/CD) pipelines for infrastructure. Examples of IaC tooling include CloudFormation, Terraform, Chef, and Puppet.
This practice allows for:
* **Cost reduction:** Less time spent by people and effort.
* **Speed:** Automation-enabled speed through fast execution with more visibility.
* **Less risk:** Reduced chance of human error and manual misconfiguration, which increases reliability, decreases downtime, and improves security.
The process starts with a template, which acts as a blueprint of the desired state of infrastructure. This template is processed by tooling (e.g., CloudFormation), which parses it and determines the necessary infrastructure changes. The tooling then makes API calls to the targeted environment to provision or modify servers. It also calculates the delta between the current and desired states, allowing focus on the template's end state rather than procedural scripting or change calculations.
CloudFormation enables provisioning AWS resources using Infrastructure as Code (IaC). It provides a repeatable process for provisioning resources and works with most AWS services. A real-world example is automating the infrastructure-provisioning process for EC2 servers.
CloudFormation Fundamentals - Terminology
Understanding key terminology is critical for comprehending how CloudFormation operates.
A template is a JSON or YAML formatted text file that acts as a blueprint for building AWS resources. It is provided to the CloudFormation service as input and describes the desired end state of the AWS infrastructure. CloudFormation then orchestrates the provisioning of this infrastructure based on the template.
When CloudFormation executes a template, it creates a stack containing the resources defined in that template. A stack represents a collection of related resources managed as a single unit. In essence, the stack is a provisioned instance of the template. To update resources through CloudFormation, the stack must be updated with a new template.
Before updating a stack, a change set can be generated to preview how proposed changes will impact existing resources. This is crucial for live systems, as some modifications might delete or recreate resources, potentially causing data loss (e.g., renaming an RDS Instance creates a new one and deletes the old one). Change sets provide visibility into the actions CloudFormation will perform.
CloudFormation Template Anatomy - Top-Level Sections
CloudFormation templates are structured documents with several top-level sections, though only the Resources section is mandatory.
The template sections define different aspects of your infrastructure and its deployment.
AWSTemplateFormatVersion
Defines the CloudFormation template format version. This is optional. There is only one valid value at this stage: '2010-09-09'.
valid_value
'2010-09-09'
optional
Yes
Use Cases:
- Defining template version
Description
A text string that describes the template. This section is optional.
Use Cases:
- Documenting template purpose
Metadata
A section for objects that provide additional information about the template. This is also optional.
Use Cases:
- Adding supplementary template information
Parameters
A section that allows you to pass values into your template at runtime, for example, at stack creation. This section is optional.
Use Cases:
- Customizing stack deployments without modifying the template file
Mappings
A set of keys and values which can operate as a lookup table. This section is optional. Mappings enable you to use an input value to determine another value, such as determining an AMI ID based on the region.
Use Cases:
- Dynamically selecting values based on conditions (e.g., region-specific AMIs)
Conditions
A section where you can place conditions on whether certain resources are created or properties defined. This section is optional.
Use Cases:
- Conditionally creating resources based on input parameters or other logic
Transform
Where you can specify transforms that CloudFormation will use to process your template. This is an optional section. For example, it can include external snippets into a template or specify a version of the Serverless Application Model (SAM).
optional
Yes
examples
['AWS::Serverless (SAM version)', 'AWS::Include (include code snippets)']
Use Cases:
- Utilizing macros or SAM transformations for template processing
Resources
This is the main section and the only required section in a CloudFormation template. It defines the stack resources and their properties.
Use Cases:
- Declaring all AWS resources to be provisioned
Outputs
This section allows you to return values from your stack. These values can be imported to other stacks, returned to describe stack calls, or displayed on the console.
Use Cases:
- Sharing information between stacks (cross-stack references)
- Displaying key resource attributes upon stack completion
CloudFormation Template Languages - JSON and YAML
CloudFormation templates can be written in either JSON (JavaScript Object Notation) or YAML (YAML Ain’t Markup Language), each with distinct characteristics.
YAML is a human-readable data serialization language and a superset of JSON, meaning JSON can be parsed with a YAML parser. It is often considered more readable than JSON due to its conciseness and use of indentation for structure, avoiding noisy curly brackets. YAML allows inline comments. Whitespaces and lines act as delimiters. It supports complex data types.
**Key/Value Pair Example:**
```yaml
name : Craig
course : CloudFormation Deep Dive level: intermediate
```
**Objects (List with 2 elements each) Example:**
```yaml
Department:
- finance : 12
IT : 8
- pre-sales : 1
marketing : 8
```
**Array Formats (Inline and Indented) Example:**
```yaml
# Inline Array
Instance:
[t2.micro, t2.small, t2. large]
# Indented Array
Instance
- t2. micro
- t2.small
- t2. large
```
**Key Takeaways for YAML:**
* Indentation is a key part of YAML and represents relationships between layers of data.
* Colons separate key/value pairs.
* Dashes represent lists of items (single dash followed by a space).
* Do not use tabs; use spaces for indentation.
**CloudFormation Specific YAML Examples:**
* `Type: AWS::EC2::SecurityGroup`
* `Type: AWS::EC2::Instance`
**Pros:** More compact than JSON, allows inline comments, used in tools like Ansible Playbooks, Docker Compose, Kubernetes, supports complex data types.
**Cons:** May not appeal to the Javascript crowd.
JSON is a lightweight data interchange format used throughout AWS (e.g., CLI, IAM Policies). It is based on Javascript and resembles Java and C#.
**Key Components:**
* **Key/Value Pairs:** Contain data (e.g., `"name" : "Craig"`).
* **CloudFormation Specific Examples:** `"Type" : "AWS::EC2::SecurityGroup"`, `"Type" : "AWS::EC2::Instance"`
* **Commas:** Separate data (e.g., `"Firstname": "Craig", "LastName": "Arcuri"`).
* **Curly Braces `{}`:** Hold objects.
* **CloudFormation Example:**
```json
{
"Resources" : {
"HelloBucket" : {
"Type" : "AWS::S3::Bucket"
}
}
}
```
* **Square Brackets `[]`:** Hold arrays.
**Pros:** Resembles popular languages, widely used in AWS.
**Cons:** Does NOT allow inline comments.
CloudFormation Template Sections - Advanced Configuration
Beyond the basic structure, CloudFormation templates offer advanced sections for dynamic configuration, reusability, and conditional logic.
These sections allow for more sophisticated and flexible infrastructure definitions.
Parameters
Parameters allow values to be passed into the template at stack creation or update time. A default parameter can be defined, along with a list of allowed values and a description.
example_yaml
```yaml
Parameters:
InstanceTypeParameter:
Type: String
Default: t2.micro
Allowed Values:
- t2.micro
- m1.small
- m1.large
Description: Enter t2.micro, m1.small, or m1 .large.
```
Use Cases:
- Customizing instance types, database names, or other resource properties during deployment.
Mappings
Mappings match keys to corresponding name-value pairs, acting as a lookup table. They are useful for determining values like AMI IDs based on the region. The `Fn::FindInMap` intrinsic function is used to retrieve values from mappings. AMIs are region-specific, so ensure they are copied to the target region.
**Example:**
```yaml
Mappings:
RegionMap:
us-east-1:
AMI: ami-76f0061f
us-west-1:
AMI: ami-655a0a20
Resources:
Ec2Instance:
Type: 'AWS::EC2::Instance’
Properties:
ImageId: !FindInMap
- RegionMap
- !Ref 'AWS::Region’
- AMI
```
Use Cases:
- Automating selection of region-specific resources (e.g., AMIs, endpoints).
Transforms
The `Transform` section specifies macros that CloudFormation uses to perform custom processing on parts of a template. This can include using `AWS::Serverless` for the Serverless Application Model (SAM) or `AWS::Include` to embed external code snippets from S3.
supported_transforms
['AWS::Serverless', 'AWS::Include']
example_json
```json
Resources:
MyBucket:
Type: AWS:S3::Bucket
Properties:
Fn:: Transform:
- Name: AWS::Include
Parameters:
Location: s3://bucket/myBucketName.json
- Name: AWS::Include
Parameters:
Location: s3://bucket/myBucketAcl.json
```
Use Cases:
- Simplifying serverless application deployments
- Modularizing templates with external snippets.
Conditions
Conditions define when a resource is created or a property is defined. They allow for conditional logic within the template, such as creating resources only for a 'prod' environment type using `Fn::Equals` and `Ref` pseudo-parameters.
**Example:**
```json
Parameters:
EnvType:
Description: Environment type.
Default: test
Type: String
AllowedValues:
- prod
- test
ConstraintDescription: must specify prod or test.
Conditions:
CreateProdResources:
Fn::Equals:
- Ref: EnvType
- prod
Resources:
EC2Instance:
Type: AWS::EC2::Instance
Properties:
Imageld:
Fn::FindInMap:
- RegionMap
- Ref: AWS::Region
- AMI
MountPoint:
Type: AWS::EC2::VolumeAttachment
Condition: CreateProdResources
Properties:
Instanceld:
Ref: EC2Instance
Volumeld:
Ref: NewVolume
Device: "/dev/sdh"
```
Use Cases:
- Deploying different resources based on environment (e.g., dev/prod)
- Enabling/disabling features based on specific criteria.
Resources (Mandatory Section)
The `Resources` section is the only mandatory part of a CloudFormation template. It declares the AWS resources to be included in the stack, each with a logical identifier, a `Type` (identifying the resource type), and `Properties` appropriate for that resource type. The logical ID is used for internal template references, distinct from the physical ID assigned by AWS upon provisioning.
**Example EC2 Instance (YAML):**
```yaml
AWSTemplateFormatVersion: '2010-09-09'
Resources:
MyEC2Instance:
Type: AWS::EC2::Instance
Properties:
ImageId: ami-0c55b159cbfafe1f0 # Linux AMI
InstanceType: t2.micro
```
**Example S3 Bucket (YAML):**
```yaml
Resources:
MyS3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: HelloWorldWebsite
AccessControl: PublicRead
WebsiteConfiguration:
IndexDocument: index.html
```
**Example Security Group (YAML):**
```yaml
Resources:
MySecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Enable SSH access via port 22
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '22'
ToPort: '22'
CidrIp: 0.0.0.0/0
```
Use Cases:
- Defining all infrastructure components of a stack.
Outputs
The `Outputs` section declares output values that can be imported to other stacks (creating cross-stack references), returned to describe stack calls, or displayed on the console. Outputs provide a way to retrieve important information about the created resources.
**Example (JSON):**
```json
Outputs:
BackupLoadBalancer DNSName:
Description: The DNSName of the backup load balancer
Value:
Fn::GetAtt:
- BackupLoadBalancer
- DNSName
InstancelD:
Description: The Instance ID
Value:
Ref: EC2Instance
```
Use Cases:
- Facilitating communication between different CloudFormation stacks
- Displaying key resource identifiers (e.g., DNS names, IDs) after deployment.
CloudFormation Intrinsic Functions
Intrinsic functions are built-in functions in CloudFormation that help you manage your stacks by performing operations like concatenating strings, retrieving resource attributes, or conditionally creating resources.
These functions are processed by CloudFormation before the stack is created or updated, allowing for dynamic and flexible template definitions.
Fn::Base64
Returns the Base64 representation of a string.
Use Cases:
- Encoding UserData for EC2 instances.
Fn::Cidr
Returns an array of CIDR address blocks. The number of CIDR blocks depends on the `count` parameter.
json_syntax
{ "En::Cidr": [ipBlock, count, cidrBits]}
yaml_syntax
Fn.:Cidr - ipBlock - count - cidrBits
yaml_short_form
!Cidr [ ipBlock, count, cidrBits ]
ip_block
The user-specified CIDR address block to be split into smaller CIDR blocks.
count
The number of CIDRs to generate. Valid range is between 1 and 256.
cidr_bits
The number of subnet bits for the CIDR. E.g., '8' for a mask of '/24'.
Use Cases:
- Dynamically creating subnet CIDR blocks.
Fn::FindInMap
Returns the value corresponding to keys in a two-level map that is declared in the Mappings section.
json_syntax
{"En::FindInMap" : [ "MapName", "TopLevelKey", "SecondLevelKey"] }
yaml_syntax
Fn::FindInMap: [ MapName, TopLevelKey, SecondLevelKey ]
yaml_short_form
!FindInMap [ MapName, TopLevelKey, SecondLevelKey ]
Use Cases:
- Retrieving region-specific AMI IDs, or other configuration values from mappings.
Fn::GetAtt
Returns the value of an attribute from a resource in the template.
json_syntax
{"En::GetAtt" : [ "logicalNameOfResource", "attributeName" ]}
yaml_syntax
Fn::GetAtt: [ logicalNameOfResource, attributeName ]
yaml_short_form
!GetAtt logicalNameOfResource.attributeName
example_ec2_public_dns_json
```json
{"Fn::GetAtt" : [ "WebServerInstance", "PublicDnsName" ]}
```
example_ec2_public_dns_yaml
```yaml
Fn::GetAtt: [ WebServerName, PublicDnsName ]
or
!GetAtt WebServerName.PublicDnsName
```
Use Cases:
- Retrieving dynamically generated resource attributes (e.g., DNS names, ARN, IDs).
Fn::GetAZs
Returns an array that lists Availability Zones for a specified region. This function enables template authors to write templates that adapt to the calling user's access, as customers have access to different Availability Zones. Specifying an empty string is equivalent to specifying `AWS::Region`.
json_syntax
{"Fn::GetAZs": "region" }
yaml_syntax
Fn::GetAZs: region
yaml_short_form
!GetAZs region
valid_inputs_us_east_1
{ "Fn::GetAZs" : ''' }
valid_inputs_aws_region
{"Fn::GetAZs" : {"Ref" : "AWS::Region" } }
valid_inputs_explicit_region
{"Fn::GetAZs" : "us-east-1" }
Use Cases:
- Dynamically deploying resources across available Availability Zones.
Fn::ImportValue
Returns the value of an output exported by another stack. This function is typically used to create cross-stack references.
json_syntax
{"En:ImportValue" : sharedValueToImport }
yaml_syntax
Fn::ImportValue: sharedValueToImport
yaml_short_form
!ImportValue sharedValueTolmport
example_json
```json
{ "Fn:ImportValue" : {"Fn::Sub":"${NetworkStack}-SubnetID" } }
```
example_yaml
```yaml
Fn::ImportValue:
!Sub "${NetworkStack}-SubnetID"
```
Use Cases:
- Referencing resources created in a separate infrastructure stack (e.g., VPC, Subnet IDs)
Fn::Join
Appends a set of values into a single value, separated by the specified delimiter. If the delimiter is an empty string, the values are concatenated with no delimiter.
json_syntax
{"Fn:Join" : [ "delimiter", [ comma-delimited list of values ]]}
yaml_syntax
Fn:Join: [ delimiter, [ comma-delimited list of values ]]
yaml_short_form
!Join [ delimiter, [ comma-delimited list of values ]]
example_json
```json
"Fn:Join" : ["-", [ "CloudFormation", "Deep", "Dive" ]]
```
example_yaml
```yaml
Fn::Join: [-, [ CloudFormation, Deep, Dive ]]
or
Join [- [ CloudFormation, Deep, Dive 1]
```
Use Cases:
- Constructing strings like ARN values, file paths, or resource names.
Fn::Length
Returns the number of elements in an array or the number of characters in a string.
Use Cases:
- Determining the size of a list or string length for conditional logic.
Fn::Select
Returns a single object from a list of objects by index. The index must be a value from zero to N-1, where N is the number of elements in the list.
json_syntax
{ "F::Select" : [ index, listOfObjects ]}
yaml_syntax
Fn::Select: [ index, listOfObjects ]
yaml_short_form
!Select [ index, listOfObjects ]
example_json
```json
{ "Fn::Select": ["2", [ "VPC", "IGW", "Route", "Key" ]] }
```
example_yaml
```yaml
En::Select: [2, "VPC", "IGW", "Route", "Key" ]
or
!Select ["2", [ "VPC", "IGW", "Route", "Key" ]]
```
Use Cases:
- Extracting specific values from an array, such as a particular subnet from a list of subnets.
Fn::Split
Splits a string into a list of string values using a specified delimiter. The `Fn::Select` function can then be used to pick a specific element from the resulting list.
json_syntax
{ "Fn::Split" : [ "delimiter", "source string" ]}
yaml_syntax
Fn::Split: [ delimiter, source string ]
yaml_short_form
!Split [delimiter, source string ]
example_splitting_imported_value_json
```json
{ "Fn:Select" : ["2", {"Fn::Split":['.',{"Fn:ImportValue": "AccountSubnetIDs"}]}] }
```
Use Cases:
- Parsing values from strings (e.g., extracting components from an ARN).
Fn::Sub
Substitutes variables in an input string with values that you specify. This function can be used to construct command-line strings or outputs that include values only available after stack creation or update.
json_syntax
{"Fn::Sub" : [ String, { Var1Name: Var1Value, Var2Name: Var2Value } ] }
yaml_syntax
Fn::Sub:
- String
- {Var1 Name: Var1Value, Var2Name: Var2Value }
yaml_short_form
!Sub
- String
- { Var1Name: Var1Value, VarName: Var2Value }
Use Cases:
- Creating dynamic strings based on runtime values or resource attributes.
Fn::ToJsonString
Converts an object into a JSON string.
Use Cases:
- Passing complex objects as string values, for instance, in UserData.
Fn::Transform (Macro)
Specifies a macro to perform custom processing on part of a stack template.
json_syntax
{"Fn::Transform" : { "Name" : macro name, "Parameters" : {key: value,}}}
yaml_syntax
Fn::Transform Name: macro name Parameters: Key : value
yaml_short_form
!Transform { "Name" : macro name, "Parameters" : {key : value, .. } }
Use Cases:
- Applying Serverless Application Model (SAM) transformations or including external snippets.
Ref
Returns the value of the specified parameter or resource. If you specify a parameter's logical name, it returns the value of the parameter. If you specify a resource's logical name, it returns a value typically used to refer to that resource, such as a physical ID.
json_syntax
{"Ref" : "logicalName" }
yaml_syntax
Ref: logicalName
yaml_short_form
!Ref logicalName
Use Cases:
- Referencing parameter values or resource logical IDs within a template.
CloudFormation Condition Functions
CloudFormation provides condition functions to conditionally create stack resources or define properties based on logical evaluations.
The following functions are used to conditionally create stack resources:
* `And`
* `Equals`
* `If`
* `Not`
* `Or`
CloudFormation Pseudo Parameters
Pseudo parameters are predefined by CloudFormation, acting similarly to environment variables, and can be reliably referenced within templates.
These parameters are automatically set and maintained by CloudFormation, ensuring consistency and correctness. They are referenced using the `Ref` Intrinsic Function.
* `AWS::AccountId`: Returns the AWS account ID of the account.
* `AWS::NotificationARNs`: Returns the list of notification ARNs for the current stack.
* `AWS::StackId`: Returns the ID of the stack.
* `AWS::StackName`: Returns the name of the stack.
* `AWS::Region`: Returns a string representing the AWS Region in which the resource is being created.
**Example: Setting a Tag with Region using !Join and !Ref**
```yaml
Tags:
- Key: "Name"
Value: !Join
- ""
- - "EC2 Instance for "
- !Ref AWS::Region
```
CloudFormation Tools and Helper Scripts
CloudFormation provides tools like Designer for visual template creation and helper scripts for instance bootstrapping to enhance infrastructure management.
CloudFormation Designer is a visual tool that provides a drag-and-drop interface for adding resources to templates. It supports both JSON and YAML formats.
The `UserData` property for an EC2 instance allows scripts to be executed on the instance during its first boot cycle. This data is Base64 encoded and works for both Linux and Windows. It's important to be aware of the time it takes for these scripts to execute.
While procedural scripting is often not ideal, CloudFormation provides Python-based helper scripts to optimize instance bootstrapping. These scripts come preinstalled on Amazon Linux but are not executed automatically. They are updated periodically and can be updated using `yum install -y aws-cfn-bootstrap`.
**Available Helper Scripts:**
* `cfn-init`: Reads and interprets Metadata to execute `AWS::CloudFormation::Init`.
* `cfn-signal`: Used to signal when a resource or application is ready.
* `cfn-get-metadata`: Used to retrieve metadata based on a specific key.
* `cfn-hup`: Used to check for updates to metadata and execute custom hooks.
CloudFormation in Operational Excellence & Disaster Recovery
CloudFormation plays a key role in achieving operational excellence through automation and is a vital component in disaster recovery strategies.
Within the AWS Well-Architected Framework, 'Operational Excellence' emphasizes performing operations as code. CloudFormation is explicitly listed as a tool for automating deployments using infrastructure as code (IaC) templates. It is also given as an example for automated deployments. (source_page: 3, 9)
CloudFormation can be a vital part of disaster recovery, though the specific mechanisms are covered in more advanced topics, such as the course wrap-up in a deep-dive course. (source_page: 4)