Technologist

Tech stuff about Cloud, DevOps, SysAdmin, Virtualization, SAN, Hardware, Scripting, Automation and Development

Browsing Posts in aws

In this guide I go over how to use Vagrant with AWS and in the process have an automated way to install Puppet Enterprise.
I am separating data and code by having a generic Vagrantfile with the code and have a servers.yaml file with all the data that will change from user to user.
For installing the Puppet Enterprise server I am including the automated provisioning script I am using with Vagrant and using AWS Tags to set the hostname of the launched server.

Pre-requisites:

  • Vagrant
  • vagrant-aws plug-in:
  • $ vagrant plugin install vagrant-aws
    
  • AWS pre-requisites:
  • While you can add your AWS credentials to the Vagrantfile, it is not recommended. A better way is to have the AWS CLI tools installed and configured

    $ aws configure
    AWS Access Key ID [****************XYYY]: XXXXXXXXYYY
    AWS Secret Access Key [****************ZOOO]:ZZZZZZZOOO
    Default region name [us-east-1]:us-east-1
    Default output format [None]: json
    

    TL;DR To get started right away you can download the project from github vagrant-aws-puppetserver, otherwise follow the guide below.

    Create Vagrantfile

    The below Vagrantfile utilizes a yaml file (servers.yaml) to provide the data, it allows you to control data using the yaml file and not have to modify the Vagrantfile code – separating code and data.

    // Vagrantfile

    # -*- mode: ruby -*-
    # vi: set ft=ruby :
    
    # Specify minimum Vagrant version and Vagrant API version
    Vagrant.require_version ">= 1.6.0"
    VAGRANTFILE_API_VERSION = "2"
    
    # Require YAML module
    require 'yaml'
    
    # Read YAML file with box details
    servers = YAML.load_file('servers.yaml')
    
    Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
    
        # Iterate through entries in YAML file
        servers.each do |server|
        
            config.vm.define server['name'] do |srv|
    
                srv.vm.box = server['box']
                srv.vm.box_url = server['box_url']
    
                srv.vm.provider :aws do |aws, override|
    
                    ### Dont do these here, better to use awscli with a profile
                    #aws.access_key_id = "YOUR KEY"
                    #aws.secret_access_key = "YOUR SECRET KEY"
                    #aws.session_token = "SESSION TOKEN"
    
                    aws.region = server["aws_region"] if server["aws_region"]
                    aws.keypair_name = server["aws_keypair_name"] if server["aws_keypair_name"]
                    aws.subnet_id = server["aws_subnet_id"] if server["aws_subnet_id"]
                    aws.associate_public_ip = server["aws_associate_public_ip"] if server["aws_associate_public_ip"]
                    aws.security_groups = server["aws_security_groups"] if server["aws_security_groups"]
                    aws.iam_instance_profile_name = server['aws_iam_role'] if server['aws_iam_role']
                    aws.ami = server["aws_ami"] if server["aws_ami"]
                    aws.instance_type = server["aws_instance_type"] if server["aws_instance_type"]
                    aws.tags = server["aws_tags"] if server["aws_tags"]
                    aws.user_data = server["aws_user_data"] if server["aws_user_data"]
    
                    override.ssh.username = server["aws_ssh_username"] if server["aws_ssh_username"]
                    override.ssh.private_key_path = server["aws_ssh_private_key_path"] if server["aws_ssh_private_key_path"]           
    
                    config.vm.synced_folder ".", "/vagrant", type: "rsync"
    
                    config.vm.provision :shell, path: server["provision"] if server["provision"]
    
                end  
            end
        end
    end
    

    Create servers.yaml

    This file contains the information that will be used by the Vagrantfile, this includes
    AWS region: Which region will this EC2 server run
    AWS keypair: Key used to connect to your launched EC2 instance
    AWS subnet id: Where will this EC2 instance sit in the AWS network
    AWS associate public ip: Do you need a public IP? true or false
    AWS security group: What AWS security group should be associated, should allow Puppetserver needed ports and whatever else you need (ssh, etc)
    AWS ami: Which AMI will you be using I am using a CentOS7
    AWS instance type: Puppetserver needs enough CPU/RAM, during my testing m3.xlarge was appropriate
    AWS SSH username: The EC2 instance user (depends on which AMI you choose), the CentOS AMI expects ec2-user
    AWS SSH private key path: The local path to the SSH key pair
    AWS User Data: I am adding user data which will execute a bash script that allows Vagrant to interact with the launched EC2 instance
    AWS Tags: This is not required for Vagrant and AWS/EC2, but in my provision script I am using the AWS Name Tag to be the system’s hostname, the other 2 tags are there for demonstration purposes
    provision: This is a provisioning script that will be run on the EC2 instance – this is the script that install the Puppet Enterprise server
    AWS IAM Role: You don’t need to add a role when working with Vagrant and AWS/EC2, but I am using a specific IAM role to allow the launched EC2 instance to be able to get information about its AWS Tags, so it is important that you provide it with a Role that allows DescribeTags, see below IAM policy:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "XXXXXXX",
                "Action": [
                    "ec2:DescribeTags"
                ],
                "Effect": "Allow",
                "Resource": "*"
            }
        ]
    }

    Now use your data in the servers.yaml file
    // servers.yaml

    ---
    - name: puppet4
      box: dummy
      box_url: https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
      aws_region: "us-east-1"
      aws_keypair_name: "john-key"
      aws_subnet_id: "subnet-0001" 
      aws_associate_public_ip: false
      aws_security_groups: ['sg-0001'] 
      aws_ami: "ami-0001"
      aws_instance_type: m3.xlarge
      aws_ssh_username: "ec2-user"
      aws_iam_role: "iam-able-to-describe-tags"
      aws_ssh_private_key_path: "/Users/john/.ssh/john-key.pem"
      aws_user_data: "#!/bin/bash\nsed -i -e 's/^Defaults.*requiretty/# Defaults requiretty/g' /etc/sudoers"
      aws_tags: 
        Name: 'puppet4'
        tier: 'stage'
        application: 'puppetserver'
      update: false
      provision: install_puppetserver.sh
    

    At this point you can spin up EC2 instances using the above Vagrantfile and servers.yaml file. If you add provision: install_puppetserver.sh to the servers.yaml file as I did and add the below script you will have a Puppet Enterprise server ready to go.

    // install_puppetserver.sh

    #!/bin/bash -xe
    
    # Ensure pip is installed
    if ! which pip; then
    	curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
    	python get-pip.py
    fi
    
    # Upgrade pip 
    /bin/pip install --upgrade pip
    
    # Upgrade awscli tools (They are installed in the AMI)
    /bin/pip install --upgrade awscli
    
    # Get hostname from AWS Name Tag (requires the EC2 instance to have an IAM role that allows DescribeTags)
    AWS_INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
    AWS_REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | grep region | cut -d\" -f4)
    AWSHOSTNAME=$(aws ec2 describe-tags --region ${AWS_REGION} --filters "Name=resource-id,Values=${AWS_INSTANCE_ID}" --query "Tags[?Key=='Name'].Value[] | [0]" | cut -d\" -f2)
    
    # Set hostname (use the AWS Name Tag)
    hostnamectl set-hostname ${AWSHOSTNAME}.cpg.org
    
    # Update system and install wget, git
    yum update -y
    yum install wget git -y
    
    # Set puppet.cpg.org as hostname in hosts file
    echo "$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4) puppet puppet.cpg.org" >> /etc/hosts
    
    # Download puppet 
    wget "https://pm.puppetlabs.com/cgi-bin/download.cgi?dist=el&rel=7&arch=x86_64&ver=2016.4.0" -O puppet.2016.4.0.tar.gz
    tar -xvzf puppet.2016.4.0.tar.gz
    cd puppet-enterprise*
    
    # Create pe.conf file
    touch pe.conf
    echo '{' >> pe.conf
    echo '"console_admin_password": "puppet"' >> pe.conf
    echo '"puppet_enterprise::puppet_master_host": "%{::trusted.certname}"' >> pe.conf
    echo '}'  >> pe.conf
    
    echo "Install Puppetserver"
    ./puppet-enterprise-installer -c pe.conf
    
    echo "Adding * to autosign.conf"
    cat >> /etc/puppetlabs/puppet/autosign.conf <<'AUTOSIGN'
    *
    AUTOSIGN
    
    # Run puppet agent
    /usr/local/bin/puppet agent -t
    

In this post I will go over the installation and usage of the AWS CLI to deploy EC2 machines, also combined with AWS user-data to automate actions/scripts that will run on the EC2 machines at install time.

Install the AWS CLI
You can install the AWS CLI in many ways (e.g. zip file, brew, PIP, etc) for details you can follow http://docs.aws.amazon.com/cli/latest/userguide/installing.html
I will install via the Python package manager (PIP) inside a virtual environment

Create virtual environment

$ virtualenv env
New python executable in env/bin/python2.7
Also creating executable in env/bin/python
Installing setuptools, pip…done.

Activate virtual environment

$ source env/bin/activate

Install the awscli

$ pip install awscli
Collecting awscli
Downloading awscli-1.10.53-py2.py3-none-any.whl (970kB)
100% |################################| 970kB 582kB/s
Collecting botocore==1.4.43 (from awscli)
Downloading botocore-1.4.43-py2.py3-none-any.whl (2.5MB)
100% |################################| 2.5MB 206kB/s
Collecting s3transfer<0.2.0,>=0.1.0 (from awscli)
Downloading s3transfer-0.1.1-py2.py3-none-any.whl (49kB)
100% |################################| 49kB 1.4MB/s
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
100% |################################| 49kB 6.2MB/s
Collecting colorama<=0.3.7,>=0.2.5 (from awscli)
Downloading colorama-0.3.7-py2.py3-none-any.whl
Collecting docutils>=0.10 (from awscli)
Downloading docutils-0.12.tar.gz (1.6MB)
100% |################################| 1.6MB 312kB/s
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.4.43->awscli)
Downloading jmespath-0.9.0-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.4.43->awscli)
Downloading python_dateutil-2.5.3-py2.py3-none-any.whl (201kB)
100% |################################| 204kB 1.9MB/s
Collecting futures<4.0.0,>=2.2.0 (from s3transfer<0.2.0,>=0.1.0->awscli)
Downloading futures-3.0.5-py2-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
Downloading pyasn1-0.1.9-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.4.43->awscli)
Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, pyasn1, futures, python-dateutil, jmespath, docutils, colorama, rsa, s3transfer, botocore, awscli
Running setup.py install for docutils
changing mode of build/scripts-2.7/rst2html.py from 644 to 755
changing mode of build/scripts-2.7/rst2s5.py from 644 to 755

Successfully installed awscli-1.10.53 botocore-1.4.43 colorama-0.3.7 docutils-0.12 futures-3.0.5 jmespath-0.9.0 pyasn1-0.1.9 python-dateutil-2.5.3 rsa-3.4.2 s3transfer-0.1.1 six-1.10.0

Verify installation (and review help documentation)

$ aws help

Configure the AWS CLI to use AWS credentials
You will need to provide an AWS Access Key ID and AWS secreate Access Key, these will map to a AWS user and the privileges this user have. The user can (probably should have) been created from the AWS console.

$ aws configure
AWS Access Key ID [****************XYYY]: XXXXXXXXYYY
AWS Secret Access Key [****************ZOOO]:ZZZZZZZOOO
Default region name [us-east-1]:us-east-1
Default output format [None]: json

The above will create a default configuration and set of credentials under ~/.aws/{config,credentials}

Launching an EC2 Instance from awscli

To connect to the EC2 instance you will need at least a SSH keypair and allow access to SSH into the EC2 instance

Create key pair

$ aws ec2 create-key-pair –key-name john-keypair –query ‘KeyMaterial’ –output text > ~/.ssh/john-keypair.pem

$ chmod 400 ~/.ssh/john-keypair.pem

Verify key existence

$ aws ec2 describe-key-pairs –key-name john-keypair
KEYPAIRS 03:eb:3a:d3:13:ba:d3:e3:03:13:b3:f1:43:83:cc:03:ec:d8:4b:43 john-keypair

Create a Security group that allows ingress SSH (tcp 22)

$ aws ec2 create-security-group –group-name allow_tcp_22 –description “Allow SSH”

sg-b1740cd5

// TAG it, I cannot stress enough how important it is to tag every resource with at least a Name

$ aws ec2 create-tags –resources sg-b1740cd5 –tags Key=Name,Value=allow_tcp_22

Add rules, for example: allow inbound (ingress) tcp port 22 to all (0.0.0.0/0)

$ aws ec2 authorize-security-group-ingress –group-id sg-b1740cd5 –protocol tcp –port 22 –cidr 0.0.0.0/0

Select the AWS AMI (aka template) you will use to clone and create your EC2 instance.

See your images (For example I have two Encrypted Ubuntu images) These are using output=text in ~/.aws/config, just to show you what that looks like as oposed to JSON.

$ aws ec2 describe-images –owners self
IMAGES x86_64 2016-07-30T13:16:39.000Z xen ami-a64403c2 /Encrypted Ubuntu Linux 14.04 (HVM) machine Encrypted Red Hat Enterprise Linux (HVM) False /dev/sda1 ebs simple available hvm
BLOCKDEVICEMAPPINGS /dev/sda1
EBS True True snap-bc158f41 10 gp2
IMAGES x86_64 2016-07-30T13:22:03.000Z xen ami-a74403c3 /Encrypted Ubuntu Linux 16.04 (HVM) machine Encrypted Red Hat Enterprise Linux (HVM) False /dev/sda1 ebs simple available hvm
BLOCKDEVICEMAPPINGS /dev/sda1
EBS True True snap-900aa0a1 10 gp2

Or see available images from Redhat

$ aws ec2 describe-images –owners 309956199498 –filters “Name=architecture,Values=x86_64” | grep RHEL-7.1
IMAGES x86_64 2015-02-26T16:27:33.000Z Provided by Red Hat, Inc. xen ami-a540a5e1 309956199498/RHEL-7.1_HVM_GA-20150225-x86_64-1-Hourly2-GP2 machine RHEL-7.1_HVM_GA-20150225-x86_64-1-Hourly2-GP2 309956199498 True /dev/sda1 ebs available hvm
IMAGES x86_64 2015-08-04T17:22:47.000Z Provided by Red Hat, Inc. xen ami-c1996685 309956199498/RHEL-7.1_HVM-20150803-x86_64-1-Hourly2-GP2 machine RHEL-7.1_HVM-20150803-x86_64-1-Hourly2-GP2309956199498 True /dev/sda1 ebs simple available hvm

Or even the images from Amazon, which are FREE – this is the one I will use

$ aws ec2 describe-images –owners amazon –filters “Name=root-device-type,Values=ebs” “Name=architecture,Values=x86_64” | grep ‘Amazon Linux AMI 2016’

IMAGES x86_64 2016-06-22T08:08:12.000Z Amazon Linux AMI 2016.03.3 x86_64 HVM GP2 xen ami-31490d51 amazon/amzn-ami-hvm-2016.03.3.x86_64-gp2 amazon machine amzn-ami-hvm-2016.03.3.x86_64-gp2 137112412989 True /dev/xvda ebs simple available hvm

Launch/Run the instance based on the chosen AMI (e.g. ami-31490d51)

$ aws ec2 run-instances –image-id ami-31490d51 –count 1 –instance-type m3.medium –key-name john-keypair –security-group-ids sg-b1740cd5

{
    "OwnerId": "XXXX",
    "ReservationId": "r-XXc3852a",
    "Groups": [],
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            },
            "PublicDnsName": "",
            "RootDeviceType": "ebs",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "EbsOptimized": false,
            "LaunchTime": "2016-07-30T23:44:30.000Z",
            "PrivateIpAddress": "172.31.13.82",
            "ProductCodes": [],
            "VpcId": "vpc-d491d9b1",
            "StateTransitionReason": "",
            "InstanceId": "i-e1aeaf54",
...
...

TAG IT

$ aws ec2 create-tags –resources i-e1aeaf54 –tags Key=Name,Value=JohnInstance1

Check its status

$ aws ec2 describe-instance-status –instance-ids i-e1aeaf54

Check its details

$ aws ec2 describe-instances –instance-ids i-e1aeaf54

Retrive from its details specifc information using jq (jq is a lightweight and flexible command-line JSON processor)

$ aws ec2 describe-instances –instance-ids i-e1aeaf54 | jq ‘.Reservations[].Instances[].PublicDnsName’

“ec2-54-183-59-200.us-west-1.compute.amazonaws.com”

Connect to the EC2 instance using your SSH key-pair

ssh -i ~/.ssh/john-keypair.pem ec2-user@ec2-54-183-59-200.us-west-1.compute.amazonaws.com

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/
10 package(s) needed for security, out of 22 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-31-13-82 ~]$

Destroy EC2 instance

$ aws ec2 terminate-instances –instance-ids i-e1aeaf54

{
    "TerminatingInstances": [
        {
            "InstanceId": "i-e1aeaf54",
            "CurrentState": {
                "Code": 32,
                "Name": "shutting-down"
            },
            "PreviousState": {
                "Code": 16,
                "Name": "running"
            }
        }
    ]
}


Create EC2 instance with user data script

Now let’s create an EC2 instance that will be ready to serve a website, we will do that by adding user data content that installs a webserver and adds some content.

Notes:
Amazon EC2 limits the size of user-data to 16KB (This limit applies to the data in raw form, not base64-encoded form.)
You can download a larger script and run it, from S3 for example.
You can run any language that supports the shabang(#!) (e.g. Bash, Python, Ruby, Perl)

//user_data.sh

#!/bin/bash
set -e -x
yum install httpd -y
chkconfig httpd on
service httpd start
INSTANCEID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
echo "Hello from ${INSTANCEID}" > /var/www/html/index.html

Launch/run instance with user-data:

$ aws ec2 run-instances –image-id ami-31490d51 –count 1 –instance-type m3.medium –key-name john-keypair –security-group-ids sg-b1740cd5 –user-data file://user_data.sh

{
    "OwnerId": "XXXX",
    "ReservationId": "r-7ed096cc",
    "Groups": [],
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            },
            "PublicDnsName": "",
            "RootDeviceType": "ebs",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "EbsOptimized": false,
            "LaunchTime": "2016-07-30T22:34:29.000Z",
            "PrivateIpAddress": "172.31.6.82",
            "ProductCodes": [],
            "VpcId": "vpc-d491d9b1",
            "StateTransitionReason": "",
            "InstanceId": "i-6b6a6ade",
...

TAG it

$ aws ec2 create-tags –resources i-e1aeaf54 –tags Key=Name,Value=JohnInstance

1

Get its public DNS name

$ aws ec2 describe-instances –instance-ids i-e974745c | jq ‘.Reservations[].Instances[].PublicDnsName’

“ec2-52-53-165-15.us-west-1.compute.amazonaws.com”

Connect to it

$ ssh -i ~/.ssh/john-keypair.pem ec2-user@ec2-52-53-165-15.us-west-1.compute.amazonaws.com

Verify that httpd is running and the right content is displayed

[ec2-user@ip-172-31-9-131 ~]$ curl http://localhost

Hello from i-e974745c

Troubleshooting: Check the /var/log/cloud-init-output.log file

[ec2-user@ip-172-31-9-131 ~]$ sudo tail /var/log/cloud-init-output.log

  apr-util-ldap.x86_64 0:1.4.1-4.17.amzn1 httpd-tools.x86_64 0:2.2.31-1.8.amzn1

Complete!
+ chkconfig httpd on
+ service httpd start
Starting httpd: [  OK  ]
++ curl -s http://169.254.169.254/latest/meta-data/instance-id
+ INSTANCEID=i-e974745c
+ echo 'Hello from i-e974745c'
Cloud-init v. 0.7.6 finished at Thu, 18 Aug 2016 19:51:46 +0000. Datasource DataSourceEc2.  Up 63.66 seconds

Extra:
To allow access to it from the public you will need to add a security group that allows ports 80 and 443.
You can create one as we did for the SSH access, or you can see if you already have a group.

Look at the descriptions of you security groups (this is why it is important to add good descriptions)
“allow SSH access”
“Allow SSH”
“Allow Web tcp 80 and 443 from 0.0.0.0/0”

Looks like “Allow Web tcp 80 and 443 from 0.0.0.0/0” will do the trick, let’s verify

$ aws ec2 describe-security-groups | jq ‘.SecurityGroups[2]’

{
  "IpPermissionsEgress": [
    {
      "IpProtocol": "-1",
      "IpRanges": [
        {
          "CidrIp": "0.0.0.0/0"
        }
      ],
      "UserIdGroupPairs": [],
      "PrefixListIds": []
    }
  ],
  "Description": "Allow Web tcp 80 and 443 from 0.0.0.0/0",
  "Tags": [
    {
      "Value": "allow_tcp_web",
      "Key": "Name"
    }
  ],
  "IpPermissions": [
    {
      "PrefixListIds": [],
      "FromPort": 80,
      "IpRanges": [
        {
          "CidrIp": "0.0.0.0/0"
        }
      ],
      "ToPort": 80,
      "IpProtocol": "tcp",
      "UserIdGroupPairs": []
    },
    {
      "PrefixListIds": [],
      "FromPort": 443,
      "IpRanges": [
        {
          "CidrIp": "0.0.0.0/0"
        }
      ],
      "ToPort": 443,
      "IpProtocol": "tcp",
      "UserIdGroupPairs": []
    }
  ],
  "GroupName": "allow_tcp_web",
  "VpcId": "vpc-XXX",
  "OwnerId": "XXXXXXX",
  "GroupId": "sg-a3672ac7"
}

Let’s add this security group (sg-a3672ac7) to our EC2 instance (i-e974745c)
// This does not append, you have to specify all groups, in this case I am adding the SSH and WEB security groups to the instance

$ aws ec2 modify-instance-attribute –instance-id i-e974745c –groups sg-b1740cd5 sg-a3672ac7

Verify that the instance has both security groups

$ aws ec2 describe-instances –instance-ids i-e974745c | jq ‘.Reservations[].Instances[].SecurityGroups[]’

{
  "GroupName": "allow_tcp_web",
  "GroupId": "sg-a3672ac7"
}
{
  "GroupName": "allow_tcp_22",
  "GroupId": "sg-b1740cd5"
}

You can now verify that the content is publicly available
awscli_userdata1