What is Infrastructure as Code and AWS Cloudformation?
- Infrastructure as Code (IaC) allows the management, provision and deployment of servers or infrastrucuture via a template rather than through manual hardware deployment (click-ops) or custom scripts, which are more error-prone, less reliable and less secure.
- Cloudformation is the AWS version of Infrastructure as Code.
- It manages the lifecycle of AWS infrastructure, including the creation and provisioning of infrastructure and the deletion and cleanup as well.
- To create the desired infrastructure within the AWS cloud environment, we can create a Cloudformation template (JSON or YAML file), pass this to Cloudformation orchestrator within our AWS environment, which will then make the necessary API calls to achieve the desired end state.
- Any examples that are provided will be in YAML, since that is the preferred method for our company.
Terminology
- Templates: the blueprint JSON or YAML file that instructs the Cloudformation orchestrator.
- Stacks: all the resource(s) that are created by the template.
- Change sets: Updating a previous template and directly sending it to the Cloudformation orchestrator can create unexpected changes, so we upload the template to the change set first to see what the impacts are beforehand.
Template Anatomy
- Cloudformation templates have the following anatomy or structure:
AWSTemplateFormatVersion (optional) Description (optional) Metadata (optional) Parameters (optional) Rules (optional) Mappings (optional) Conditions (optional) Transform (optional) Resources (required) Outputs (optional) - The AWSTemplateFormatVersion is a literal string value for the latest version of Cloudformation which is “2010-09-09”. It is not important.
- All categories are optional except for the resource section, but we will explore all the sections of a Cloudformation template as this can help provision the ideal infrastructure that we need.
Resources
- This section specifies the type of resource that we want to create, as well as its properties.
```yaml
AWSTemplateFormatVersion: “2010-09-09”
Resources:
SecurityDNSCanary: # Logical name (within the template)
Type: AWS::EC2::Instance # Type of resource
Properties:
InstanceType: t2.micro
ImageId: ami-43874721 # AWS Linux AMI in ap-southeast-2
Tags:
- Key: Name Value: Security DNS Canary ```
- Here we created an EC2 instance that is a “t2-micro” using the AWS Linux AMI. We then added a name tag that is “Security DNS Canary”.
Intrinsic Functions
- Cloudformation has built-in functions to manage the stack more effectively. This allows assigning values to certain properties that happen at runtime.
- To call an intrinsic function in YAML format we can use Fn::[command] or the short-form ![command].
- Fn::Join - appends strings with delimiter
AWSTemplateFormatVersion: "2010-09-09"
Resources:
SecurityDNSCanary: # Logical name (within the template)
Type: AWS::EC2::Instance # Type of resource
Properties:
InstanceType: t2.micro
ImageId: ami-43874721 # AWS Linux AMI in ap-southeast-2
Tags:
- Key: Name
Value: !Join [ " ", [ Security, DNS, Canary ] ]
Multiple Resources
- We can create multiple resources that have dependencies on each other by using the reference function (!Ref)
- For example, to create an EC2 instance with a security group (virtual firewall), we need to create the security group first and then reference this value in EC2
AWSTemplateFormatVersion: "2010-09-09" Resources: SecurityDNSCanary: # Logical name (within the template) Type: AWS::EC2::Instance # Type of resource Properties: InstanceType: t2.micro ImageId: ami-1853ac65 # AWS Linux AMI in us-east-1 Tags: - Key: Name Value: !Join [ " ", [ Security, DNS, Canary ] ] SecurityGroups: - !Ref SecurityDNSCanarySecurityGroup SecurityDNSCanarySecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: GroupDescription: Enable DNS access via UDP + TCP port 53 SecurityGroupIngress: - IpProtocol: tcp FromPort: '53' ToPort: '53' CidrIp: 0.0.0.0/0 - IpProtocol: udp FromPort: '53' ToPort: '53' CidrIp: 0.0.0.0/0
Psuedo Parameters
- Are similar to environment variables
- The parameters are predefined by AWS and not declared within the cloudformation template
- Also uses the !Ref function to call the values
- Examples:
- AWS::AccountId
- AWS::Region
- AWS::StackId
- AWS::StackName
```yaml
AWSTemplateFormatVersion: “2010-09-09”
Resources:
SecurityDNSCanary: # Logical name (within the template)
Type: AWS::EC2::Instance # Type of resource
Properties:
InstanceType: t2.micro
ImageId: ami-1853ac65 # AWS Linux AMI in us-east-1
Tags:
- Key: Name
Value: !Join
- ””
-
- “Security DNS Canary in “
- !Ref AWS::Region # us-east-1 SecurityGroups:
- !Ref SecurityDNSCanarySecurityGroup SecurityDNSCanarySecurityGroup: Type: ‘AWS::EC2::SecurityGroup’ Properties: GroupDescription: Enable DNS access via UDP + TCP port 53 SecurityGroupIngress:
- IpProtocol: tcp FromPort: ‘53’ ToPort: ‘53’ CidrIp: 0.0.0.0/0
- IpProtocol: udp FromPort: ‘53’ ToPort: ‘53’ CidrIp: 0.0.0.0/0 ```
- Key: Name
Value: !Join
Mappings
- Allows the calling of self-defined key-value pairings that is located at the beginning of the cloudformation template
- We can use the Fn::FindInMap function to then retrieve the value
AWSTemplateFormatVersion: "2010-09-09" Mappings: CanaryImageRegion: # nested key-value pairings us-east-1: AMI: ami-1853ac65 us-west-1: AMI: ami-bf5540df eu-west-1: AMI: ami-3bfab942 ap-southeast-1: AMI: ami-e2adf99e ap-southeast-2: AMI: ami-43874721 Resources: SecurityDNSCanary: # Logical name (within the template) Type: AWS::EC2::Instance # Type of resource Properties: InstanceType: t2.micro ImageId: !FindInMap - CanaryImageRegion - !Ref AWS::Region - AMI Tags: - Key: Name Value: !Join - "" - - "Security DNS Canary in " - !Ref AWS::Region SecurityGroups: - !Ref SecurityDNSCanarySecurityGroup SecurityDNSCanarySecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: GroupDescription: Enable DNS access via UDP + TCP port 53 SecurityGroupIngress: - IpProtocol: tcp FromPort: '53' ToPort: '53' CidrIp: 0.0.0.0/0 - IpProtocol: udp FromPort: '53' ToPort: '53' CidrIp: 0.0.0.0/0
Input Parameters
- We can add custom parameters to the stack
- Each parameter must have an assigned value at runtime, although it can take a specified default value
```yaml
AWSTemplateFormatVersion: “2010-09-09”
Parameters:
CanaryInstanceTypeParameter:
Type: String
Default: t2.micro
AllowedValues:
- t2.micro
Description: Instance type of DNS canary is restricted to t2.micro
Mappings:
CanaryImageRegion:
us-east-1:
AMI: ami-1853ac65
us-west-1:
AMI: ami-bf5540df
eu-west-1:
AMI: ami-3bfab942
ap-southeast-1:
AMI: ami-e2adf99e
ap-southeast-2:
AMI: ami-43874721
Resources:
SecurityDNSCanary: # Logical name (within the template)
Type: AWS::EC2::Instance # Type of resource
Properties:
InstanceType: t2.micro
ImageId: !FindInMap
- CanaryImageRegion
- !Ref AWS::Region
- AMI Tags:
- Key: Name
Value: !Join
- ””
-
- “Security DNS Canary in “
- !Ref AWS::Region # us-east-1 SecurityGroups:
- !Ref SecurityDNSCanarySecurityGroup SecurityDNSCanarySecurityGroup: Type: ‘AWS::EC2::SecurityGroup’ Properties: GroupDescription: Enable DNS access via UDP + TCP port 53 SecurityGroupIngress:
- IpProtocol: tcp FromPort: ‘53’ ToPort: ‘53’ CidrIp: 0.0.0.0/0
- IpProtocol: udp FromPort: ‘53’ ToPort: ‘53’ CidrIp: 0.0.0.0/0 ```
- t2.micro
Description: Instance type of DNS canary is restricted to t2.micro
Mappings:
CanaryImageRegion:
us-east-1:
AMI: ami-1853ac65
us-west-1:
AMI: ami-bf5540df
eu-west-1:
AMI: ami-3bfab942
ap-southeast-1:
AMI: ami-e2adf99e
ap-southeast-2:
AMI: ami-43874721
Resources:
SecurityDNSCanary: # Logical name (within the template)
Type: AWS::EC2::Instance # Type of resource
Properties:
InstanceType: t2.micro
ImageId: !FindInMap
Outputs
- We can receive the outputs of various resources from the stack.
- We can use the Fn::GetAtt function to retrieve the attributes for any particular resource.
- For example we can retrieve some properties in AWS::EC2::Instance
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
CanaryInstanceTypeParameter:
Type: String
Default: t2.micro
AllowedValues:
- t2.micro
Description: Instance type of DNS canary is restricted to t2.micro
Mappings:
CanaryImageRegion:
us-east-1:
AMI: ami-1853ac65
us-west-1:
AMI: ami-bf5540df
eu-west-1:
AMI: ami-3bfab942
ap-southeast-1:
AMI: ami-e2adf99e
ap-southeast-2:
AMI: ami-43874721
Resources:
SecurityDNSCanary: # Logical name (within the template)
Type: AWS::EC2::Instance # Type of resource
Properties:
InstanceType: t2.micro
ImageId: !FindInMap
- CanaryImageRegion
- !Ref AWS::Region
- AMI
Tags:
- Key: Name
Value: !Join
- ""
- - "Security DNS Canary in "
- !Ref AWS::Region # us-east-1
SecurityGroups:
- !Ref SecurityDNSCanarySecurityGroup
SecurityDNSCanarySecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupDescription: Enable DNS access via UDP + TCP port 53
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '53'
ToPort: '53'
CidrIp: 0.0.0.0/0
- IpProtocol: udp
FromPort: '53'
ToPort: '53'
CidrIp: 0.0.0.0/0
Outputs:
CanaryPublicIp:
Value: !GetAtt
- SecurityDNSCanary
- PublicIp
UserData, Cloudformation Helper Scripts (Init)
- Userdata: A property of the AWS::EC2::Instance that allows us to hook into the instance and execute commands on startup
- This must be base64 encoded (which we can use the Fn::base64
- For Linux, this runs as root, does not run interactively (therefore we need to -y everything)
- #!/bin/bash
- The problem is that this method of procedually scripting partially defeats the point of using Infrastructure as Code, as it can become inefficient and messy over time
- There are python-based helper scripts for Cloudformation that can solve this (pre-installed in Amazon Linux)
- There are four helper scripts but we only care about cfn-init script - which takes metadata from Cloudformation template and allows us to use AWS::Cloudformation::init, which contains config keys and config sections that help us set up our EC2 instance. These are:
- packages: download and install packages
- groups: create linux groups
- users: create linux users
- sources: downloads archive files and unpack it
- files: downloads a file to EC2 instance
- commands: executes any command
- services: enables/disables and launches services
- Therefore to setup EC2 instances we can: userdata –> cloudformation init script –> use AWS::Cloudformation::init
Creating DNS Canary
- We need to create the following:
- Create a VPC
- Create a subnet within VPC
- Create an EC2 instance in public subnet (private subnet optional)
- Create a security group (virtual firewall) for EC2 instance
- Create an elastic IP address and attach to EC2 instance
- Allocate key pairing for SSH access (optional)
- Allocate IAM permissions for SSM access
- Within EC2 instance
- Use Cloudformation init script to accept metadata into AWS::Cloudformation:init
- Update Amazon Linux instance and install Docker
- Ensure python script file is also on the EC2 instance
- Execute Docker command to get it up and running
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
NameOfService:
Default: Test DNS Canary
Type: String
Description: "The name of the service this stack is to be used for."
KeyName:
Description: Name of an existing EC2 KeyPair to enable SSH access into the server
Type: AWS::EC2::KeyPair::KeyName
Mappings:
CanaryImageRegion:
us-east-1:
AMI: ami-04823729c75214919
us-west-1:
AMI: ami-bf5540df
eu-west-1:
AMI: ami-3bfab942
ap-southeast-1:
AMI: ami-e2adf99e
ap-southeast-2:
AMI: ami-43874721
Resources:
TestDNSCanary:
Type: AWS::EC2::Instance
Metadata:
AWS::CloudFormation::Init:
config:
packages:
yum:
docker: []
files:
/home/ec2-user/test-dns-canary/test.py:
content: |
file_content = "It works!"
file_path = "/root/complete.txt"
try:
with open(file_path, "w") as file:
file.write(file_content)
except Exception as e:
print(f"Error occurred: {str(e)}")
services:
sysvinit:
docker:
enabled: true
ensureRunning: true
Properties:
InstanceType: t2.micro
ImageId: !FindInMap
- CanaryImageRegion
- !Ref AWS::Region
- AMI
Tags:
- Key: Name
Value: !Join
- ""
- - "Test DNS Canary in "
- !Ref AWS::Region
SecurityGroups:
- !Ref TestDNSCanarySecurityGroup
KeyName: !Ref KeyName
UserData:
Fn::Base64:
!Sub |
#!/bin/bash -xe
# Adding directories to install and run the DNS canary
mkdir -p /home/ec2-user/test-dns-canary
# Ensure AWS CFN Bootstrap is the latest
yum install -y aws-cfn-bootstrap
# Install the files and packages from the metadata
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource TestDNSCanary --region ${AWS::Region}
python3 /home/ec2-user/test-dns-canary/test.py
TestDNSCanarySecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupDescription: Enable DNS access via UDP + TCP port 53 and SSH access on port 22
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '53'
ToPort: '53'
CidrIp: 0.0.0.0/0
- IpProtocol: udp
FromPort: '53'
ToPort: '53'
CidrIp: 0.0.0.0/0
- IpProtocol: tcp
FromPort: '22'
ToPort: '22'
CidrIp: 0.0.0.0/0
Outputs:
CanaryPublicIp:
Value: !GetAtt
- TestDNSCanary
- PublicIp
Test
Test
Test
UserData:
Fn::Base64:
!Sub |
#!/bin/bash -xe
mkdir /home/ssm-user/${AWS-Region}
echo "luke" >> /home/ssm-user/${AWS-Region}/blocklist
