In this post I will go over the installation and usage of the AWS CLI to deploy EC2 machines, also combined with AWS user-data to automate actions/scripts that will run on the EC2 machines at install time.

Install the AWS CLI
You can install the AWS CLI in many ways (e.g. zip file, brew, PIP, etc) for details you can follow http://docs.aws.amazon.com/cli/latest/userguide/installing.html
I will install via the Python package manager (PIP) inside a virtual environment

Create virtual environment

$ virtualenv env
New python executable in env/bin/python2.7
Also creating executable in env/bin/python
Installing setuptools, pip…done.

Activate virtual environment

$ source env/bin/activate

Install the awscli

$ pip install awscli
Collecting awscli
Downloading awscli-1.10.53-py2.py3-none-any.whl (970kB)
100% |################################| 970kB 582kB/s
Collecting botocore==1.4.43 (from awscli)
Downloading botocore-1.4.43-py2.py3-none-any.whl (2.5MB)
100% |################################| 2.5MB 206kB/s
Collecting s3transfer<0.2.0,>=0.1.0 (from awscli)
Downloading s3transfer-0.1.1-py2.py3-none-any.whl (49kB)
100% |################################| 49kB 1.4MB/s
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
100% |################################| 49kB 6.2MB/s
Collecting colorama<=0.3.7,>=0.2.5 (from awscli)
Downloading colorama-0.3.7-py2.py3-none-any.whl
Collecting docutils>=0.10 (from awscli)
Downloading docutils-0.12.tar.gz (1.6MB)
100% |################################| 1.6MB 312kB/s
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.4.43->awscli)
Downloading jmespath-0.9.0-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.4.43->awscli)
Downloading python_dateutil-2.5.3-py2.py3-none-any.whl (201kB)
100% |################################| 204kB 1.9MB/s
Collecting futures<4.0.0,>=2.2.0 (from s3transfer<0.2.0,>=0.1.0->awscli)
Downloading futures-3.0.5-py2-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
Downloading pyasn1-0.1.9-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.4.43->awscli)
Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, pyasn1, futures, python-dateutil, jmespath, docutils, colorama, rsa, s3transfer, botocore, awscli
Running setup.py install for docutils
changing mode of build/scripts-2.7/rst2html.py from 644 to 755
changing mode of build/scripts-2.7/rst2s5.py from 644 to 755

Successfully installed awscli-1.10.53 botocore-1.4.43 colorama-0.3.7 docutils-0.12 futures-3.0.5 jmespath-0.9.0 pyasn1-0.1.9 python-dateutil-2.5.3 rsa-3.4.2 s3transfer-0.1.1 six-1.10.0

Verify installation (and review help documentation)

$ aws help

Configure the AWS CLI to use AWS credentials
You will need to provide an AWS Access Key ID and AWS secreate Access Key, these will map to a AWS user and the privileges this user have. The user can (probably should have) been created from the AWS console.

$ aws configure
AWS Access Key ID [****************XYYY]: XXXXXXXXYYY
AWS Secret Access Key [****************ZOOO]:ZZZZZZZOOO
Default region name [us-east-1]:us-east-1
Default output format [None]: json

The above will create a default configuration and set of credentials under ~/.aws/{config,credentials}

Launching an EC2 Instance from awscli

To connect to the EC2 instance you will need at least a SSH keypair and allow access to SSH into the EC2 instance

Create key pair

$ aws ec2 create-key-pair –key-name john-keypair –query ‘KeyMaterial’ –output text > ~/.ssh/john-keypair.pem

$ chmod 400 ~/.ssh/john-keypair.pem

Verify key existence

$ aws ec2 describe-key-pairs –key-name john-keypair
KEYPAIRS 03:eb:3a:d3:13:ba:d3:e3:03:13:b3:f1:43:83:cc:03:ec:d8:4b:43 john-keypair

Create a Security group that allows ingress SSH (tcp 22)

$ aws ec2 create-security-group –group-name allow_tcp_22 –description “Allow SSH”

sg-b1740cd5

// TAG it, I cannot stress enough how important it is to tag every resource with at least a Name

$ aws ec2 create-tags –resources sg-b1740cd5 –tags Key=Name,Value=allow_tcp_22

Add rules, for example: allow inbound (ingress) tcp port 22 to all (0.0.0.0/0)

$ aws ec2 authorize-security-group-ingress –group-id sg-b1740cd5 –protocol tcp –port 22 –cidr 0.0.0.0/0

Select the AWS AMI (aka template) you will use to clone and create your EC2 instance.

See your images (For example I have two Encrypted Ubuntu images) These are using output=text in ~/.aws/config, just to show you what that looks like as oposed to JSON.

$ aws ec2 describe-images –owners self
IMAGES x86_64 2016-07-30T13:16:39.000Z xen ami-a64403c2 /Encrypted Ubuntu Linux 14.04 (HVM) machine Encrypted Red Hat Enterprise Linux (HVM) False /dev/sda1 ebs simple available hvm
BLOCKDEVICEMAPPINGS /dev/sda1
EBS True True snap-bc158f41 10 gp2
IMAGES x86_64 2016-07-30T13:22:03.000Z xen ami-a74403c3 /Encrypted Ubuntu Linux 16.04 (HVM) machine Encrypted Red Hat Enterprise Linux (HVM) False /dev/sda1 ebs simple available hvm
BLOCKDEVICEMAPPINGS /dev/sda1
EBS True True snap-900aa0a1 10 gp2

Or see available images from Redhat

$ aws ec2 describe-images –owners 309956199498 –filters “Name=architecture,Values=x86_64” | grep RHEL-7.1
IMAGES x86_64 2015-02-26T16:27:33.000Z Provided by Red Hat, Inc. xen ami-a540a5e1 309956199498/RHEL-7.1_HVM_GA-20150225-x86_64-1-Hourly2-GP2 machine RHEL-7.1_HVM_GA-20150225-x86_64-1-Hourly2-GP2 309956199498 True /dev/sda1 ebs available hvm
IMAGES x86_64 2015-08-04T17:22:47.000Z Provided by Red Hat, Inc. xen ami-c1996685 309956199498/RHEL-7.1_HVM-20150803-x86_64-1-Hourly2-GP2 machine RHEL-7.1_HVM-20150803-x86_64-1-Hourly2-GP2309956199498 True /dev/sda1 ebs simple available hvm

Or even the images from Amazon, which are FREE – this is the one I will use

$ aws ec2 describe-images –owners amazon –filters “Name=root-device-type,Values=ebs” “Name=architecture,Values=x86_64” | grep ‘Amazon Linux AMI 2016’

IMAGES x86_64 2016-06-22T08:08:12.000Z Amazon Linux AMI 2016.03.3 x86_64 HVM GP2 xen ami-31490d51 amazon/amzn-ami-hvm-2016.03.3.x86_64-gp2 amazon machine amzn-ami-hvm-2016.03.3.x86_64-gp2 137112412989 True /dev/xvda ebs simple available hvm

Launch/Run the instance based on the chosen AMI (e.g. ami-31490d51)

$ aws ec2 run-instances –image-id ami-31490d51 –count 1 –instance-type m3.medium –key-name john-keypair –security-group-ids sg-b1740cd5

{
    "OwnerId": "XXXX",
    "ReservationId": "r-XXc3852a",
    "Groups": [],
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            },
            "PublicDnsName": "",
            "RootDeviceType": "ebs",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "EbsOptimized": false,
            "LaunchTime": "2016-07-30T23:44:30.000Z",
            "PrivateIpAddress": "172.31.13.82",
            "ProductCodes": [],
            "VpcId": "vpc-d491d9b1",
            "StateTransitionReason": "",
            "InstanceId": "i-e1aeaf54",
...
...

TAG IT

$ aws ec2 create-tags –resources i-e1aeaf54 –tags Key=Name,Value=JohnInstance1

Check its status

$ aws ec2 describe-instance-status –instance-ids i-e1aeaf54

Check its details

$ aws ec2 describe-instances –instance-ids i-e1aeaf54

Retrive from its details specifc information using jq (jq is a lightweight and flexible command-line JSON processor)

$ aws ec2 describe-instances –instance-ids i-e1aeaf54 | jq ‘.Reservations[].Instances[].PublicDnsName’

“ec2-54-183-59-200.us-west-1.compute.amazonaws.com”

Connect to the EC2 instance using your SSH key-pair

ssh -i ~/.ssh/john-keypair.pem ec2-user@ec2-54-183-59-200.us-west-1.compute.amazonaws.com

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/
10 package(s) needed for security, out of 22 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-31-13-82 ~]$

Destroy EC2 instance

$ aws ec2 terminate-instances –instance-ids i-e1aeaf54

{
    "TerminatingInstances": [
        {
            "InstanceId": "i-e1aeaf54",
            "CurrentState": {
                "Code": 32,
                "Name": "shutting-down"
            },
            "PreviousState": {
                "Code": 16,
                "Name": "running"
            }
        }
    ]
}


Create EC2 instance with user data script

Now let’s create an EC2 instance that will be ready to serve a website, we will do that by adding user data content that installs a webserver and adds some content.

Notes:
Amazon EC2 limits the size of user-data to 16KB (This limit applies to the data in raw form, not base64-encoded form.)
You can download a larger script and run it, from S3 for example.
You can run any language that supports the shabang(#!) (e.g. Bash, Python, Ruby, Perl)

//user_data.sh

#!/bin/bash
set -e -x
yum install httpd -y
chkconfig httpd on
service httpd start
INSTANCEID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
echo "Hello from ${INSTANCEID}" > /var/www/html/index.html

Launch/run instance with user-data:

$ aws ec2 run-instances –image-id ami-31490d51 –count 1 –instance-type m3.medium –key-name john-keypair –security-group-ids sg-b1740cd5 –user-data file://user_data.sh

{
    "OwnerId": "XXXX",
    "ReservationId": "r-7ed096cc",
    "Groups": [],
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            },
            "PublicDnsName": "",
            "RootDeviceType": "ebs",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "EbsOptimized": false,
            "LaunchTime": "2016-07-30T22:34:29.000Z",
            "PrivateIpAddress": "172.31.6.82",
            "ProductCodes": [],
            "VpcId": "vpc-d491d9b1",
            "StateTransitionReason": "",
            "InstanceId": "i-6b6a6ade",
...

TAG it

$ aws ec2 create-tags –resources i-e1aeaf54 –tags Key=Name,Value=JohnInstance

1

Get its public DNS name

$ aws ec2 describe-instances –instance-ids i-e974745c | jq ‘.Reservations[].Instances[].PublicDnsName’

“ec2-52-53-165-15.us-west-1.compute.amazonaws.com”

Connect to it

$ ssh -i ~/.ssh/john-keypair.pem ec2-user@ec2-52-53-165-15.us-west-1.compute.amazonaws.com

Verify that httpd is running and the right content is displayed

[ec2-user@ip-172-31-9-131 ~]$ curl http://localhost

Hello from i-e974745c

Troubleshooting: Check the /var/log/cloud-init-output.log file

[ec2-user@ip-172-31-9-131 ~]$ sudo tail /var/log/cloud-init-output.log

  apr-util-ldap.x86_64 0:1.4.1-4.17.amzn1 httpd-tools.x86_64 0:2.2.31-1.8.amzn1

Complete!
+ chkconfig httpd on
+ service httpd start
Starting httpd: [  OK  ]
++ curl -s http://169.254.169.254/latest/meta-data/instance-id
+ INSTANCEID=i-e974745c
+ echo 'Hello from i-e974745c'
Cloud-init v. 0.7.6 finished at Thu, 18 Aug 2016 19:51:46 +0000. Datasource DataSourceEc2.  Up 63.66 seconds

Extra:
To allow access to it from the public you will need to add a security group that allows ports 80 and 443.
You can create one as we did for the SSH access, or you can see if you already have a group.

Look at the descriptions of you security groups (this is why it is important to add good descriptions)
“allow SSH access”
“Allow SSH”
“Allow Web tcp 80 and 443 from 0.0.0.0/0”

Looks like “Allow Web tcp 80 and 443 from 0.0.0.0/0” will do the trick, let’s verify

$ aws ec2 describe-security-groups | jq ‘.SecurityGroups[2]’

{
  "IpPermissionsEgress": [
    {
      "IpProtocol": "-1",
      "IpRanges": [
        {
          "CidrIp": "0.0.0.0/0"
        }
      ],
      "UserIdGroupPairs": [],
      "PrefixListIds": []
    }
  ],
  "Description": "Allow Web tcp 80 and 443 from 0.0.0.0/0",
  "Tags": [
    {
      "Value": "allow_tcp_web",
      "Key": "Name"
    }
  ],
  "IpPermissions": [
    {
      "PrefixListIds": [],
      "FromPort": 80,
      "IpRanges": [
        {
          "CidrIp": "0.0.0.0/0"
        }
      ],
      "ToPort": 80,
      "IpProtocol": "tcp",
      "UserIdGroupPairs": []
    },
    {
      "PrefixListIds": [],
      "FromPort": 443,
      "IpRanges": [
        {
          "CidrIp": "0.0.0.0/0"
        }
      ],
      "ToPort": 443,
      "IpProtocol": "tcp",
      "UserIdGroupPairs": []
    }
  ],
  "GroupName": "allow_tcp_web",
  "VpcId": "vpc-XXX",
  "OwnerId": "XXXXXXX",
  "GroupId": "sg-a3672ac7"
}

Let’s add this security group (sg-a3672ac7) to our EC2 instance (i-e974745c)
// This does not append, you have to specify all groups, in this case I am adding the SSH and WEB security groups to the instance

$ aws ec2 modify-instance-attribute –instance-id i-e974745c –groups sg-b1740cd5 sg-a3672ac7

Verify that the instance has both security groups

$ aws ec2 describe-instances –instance-ids i-e974745c | jq ‘.Reservations[].Instances[].SecurityGroups[]’

{
  "GroupName": "allow_tcp_web",
  "GroupId": "sg-a3672ac7"
}
{
  "GroupName": "allow_tcp_22",
  "GroupId": "sg-b1740cd5"
}

You can now verify that the content is publicly available
awscli_userdata1