Technologist

Tech stuff about Cloud, DevOps, SysAdmin, Virtualization, SAN, Hardware, Scripting, Automation and Development

Browsing Posts tagged automation

In this post I will go over the installation and usage of the AWS CLI to deploy EC2 machines, also combined with AWS user-data to automate actions/scripts that will run on the EC2 machines at install time.

Install the AWS CLI
You can install the AWS CLI in many ways (e.g. zip file, brew, PIP, etc) for details you can follow http://docs.aws.amazon.com/cli/latest/userguide/installing.html
I will install via the Python package manager (PIP) inside a virtual environment

Create virtual environment

$ virtualenv env
New python executable in env/bin/python2.7
Also creating executable in env/bin/python
Installing setuptools, pip…done.

Activate virtual environment

$ source env/bin/activate

Install the awscli

$ pip install awscli
Collecting awscli
Downloading awscli-1.10.53-py2.py3-none-any.whl (970kB)
100% |################################| 970kB 582kB/s
Collecting botocore==1.4.43 (from awscli)
Downloading botocore-1.4.43-py2.py3-none-any.whl (2.5MB)
100% |################################| 2.5MB 206kB/s
Collecting s3transfer<0.2.0,>=0.1.0 (from awscli)
Downloading s3transfer-0.1.1-py2.py3-none-any.whl (49kB)
100% |################################| 49kB 1.4MB/s
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
100% |################################| 49kB 6.2MB/s
Collecting colorama<=0.3.7,>=0.2.5 (from awscli)
Downloading colorama-0.3.7-py2.py3-none-any.whl
Collecting docutils>=0.10 (from awscli)
Downloading docutils-0.12.tar.gz (1.6MB)
100% |################################| 1.6MB 312kB/s
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.4.43->awscli)
Downloading jmespath-0.9.0-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.4.43->awscli)
Downloading python_dateutil-2.5.3-py2.py3-none-any.whl (201kB)
100% |################################| 204kB 1.9MB/s
Collecting futures<4.0.0,>=2.2.0 (from s3transfer<0.2.0,>=0.1.0->awscli)
Downloading futures-3.0.5-py2-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
Downloading pyasn1-0.1.9-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.4.43->awscli)
Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, pyasn1, futures, python-dateutil, jmespath, docutils, colorama, rsa, s3transfer, botocore, awscli
Running setup.py install for docutils
changing mode of build/scripts-2.7/rst2html.py from 644 to 755
changing mode of build/scripts-2.7/rst2s5.py from 644 to 755

Successfully installed awscli-1.10.53 botocore-1.4.43 colorama-0.3.7 docutils-0.12 futures-3.0.5 jmespath-0.9.0 pyasn1-0.1.9 python-dateutil-2.5.3 rsa-3.4.2 s3transfer-0.1.1 six-1.10.0

Verify installation (and review help documentation)

$ aws help

Configure the AWS CLI to use AWS credentials
You will need to provide an AWS Access Key ID and AWS secreate Access Key, these will map to a AWS user and the privileges this user have. The user can (probably should have) been created from the AWS console.

$ aws configure
AWS Access Key ID [****************XYYY]: XXXXXXXXYYY
AWS Secret Access Key [****************ZOOO]:ZZZZZZZOOO
Default region name [us-east-1]:us-east-1
Default output format [None]: json

The above will create a default configuration and set of credentials under ~/.aws/{config,credentials}

Launching an EC2 Instance from awscli

To connect to the EC2 instance you will need at least a SSH keypair and allow access to SSH into the EC2 instance

Create key pair

$ aws ec2 create-key-pair –key-name john-keypair –query ‘KeyMaterial’ –output text > ~/.ssh/john-keypair.pem

$ chmod 400 ~/.ssh/john-keypair.pem

Verify key existence

$ aws ec2 describe-key-pairs –key-name john-keypair
KEYPAIRS 03:eb:3a:d3:13:ba:d3:e3:03:13:b3:f1:43:83:cc:03:ec:d8:4b:43 john-keypair

Create a Security group that allows ingress SSH (tcp 22)

$ aws ec2 create-security-group –group-name allow_tcp_22 –description “Allow SSH”

sg-b1740cd5

// TAG it, I cannot stress enough how important it is to tag every resource with at least a Name

$ aws ec2 create-tags –resources sg-b1740cd5 –tags Key=Name,Value=allow_tcp_22

Add rules, for example: allow inbound (ingress) tcp port 22 to all (0.0.0.0/0)

$ aws ec2 authorize-security-group-ingress –group-id sg-b1740cd5 –protocol tcp –port 22 –cidr 0.0.0.0/0

Select the AWS AMI (aka template) you will use to clone and create your EC2 instance.

See your images (For example I have two Encrypted Ubuntu images) These are using output=text in ~/.aws/config, just to show you what that looks like as oposed to JSON.

$ aws ec2 describe-images –owners self
IMAGES x86_64 2016-07-30T13:16:39.000Z xen ami-a64403c2 /Encrypted Ubuntu Linux 14.04 (HVM) machine Encrypted Red Hat Enterprise Linux (HVM) False /dev/sda1 ebs simple available hvm
BLOCKDEVICEMAPPINGS /dev/sda1
EBS True True snap-bc158f41 10 gp2
IMAGES x86_64 2016-07-30T13:22:03.000Z xen ami-a74403c3 /Encrypted Ubuntu Linux 16.04 (HVM) machine Encrypted Red Hat Enterprise Linux (HVM) False /dev/sda1 ebs simple available hvm
BLOCKDEVICEMAPPINGS /dev/sda1
EBS True True snap-900aa0a1 10 gp2

Or see available images from Redhat

$ aws ec2 describe-images –owners 309956199498 –filters “Name=architecture,Values=x86_64” | grep RHEL-7.1
IMAGES x86_64 2015-02-26T16:27:33.000Z Provided by Red Hat, Inc. xen ami-a540a5e1 309956199498/RHEL-7.1_HVM_GA-20150225-x86_64-1-Hourly2-GP2 machine RHEL-7.1_HVM_GA-20150225-x86_64-1-Hourly2-GP2 309956199498 True /dev/sda1 ebs available hvm
IMAGES x86_64 2015-08-04T17:22:47.000Z Provided by Red Hat, Inc. xen ami-c1996685 309956199498/RHEL-7.1_HVM-20150803-x86_64-1-Hourly2-GP2 machine RHEL-7.1_HVM-20150803-x86_64-1-Hourly2-GP2309956199498 True /dev/sda1 ebs simple available hvm

Or even the images from Amazon, which are FREE – this is the one I will use

$ aws ec2 describe-images –owners amazon –filters “Name=root-device-type,Values=ebs” “Name=architecture,Values=x86_64” | grep ‘Amazon Linux AMI 2016’

IMAGES x86_64 2016-06-22T08:08:12.000Z Amazon Linux AMI 2016.03.3 x86_64 HVM GP2 xen ami-31490d51 amazon/amzn-ami-hvm-2016.03.3.x86_64-gp2 amazon machine amzn-ami-hvm-2016.03.3.x86_64-gp2 137112412989 True /dev/xvda ebs simple available hvm

Launch/Run the instance based on the chosen AMI (e.g. ami-31490d51)

$ aws ec2 run-instances –image-id ami-31490d51 –count 1 –instance-type m3.medium –key-name john-keypair –security-group-ids sg-b1740cd5

{
    "OwnerId": "XXXX",
    "ReservationId": "r-XXc3852a",
    "Groups": [],
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            },
            "PublicDnsName": "",
            "RootDeviceType": "ebs",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "EbsOptimized": false,
            "LaunchTime": "2016-07-30T23:44:30.000Z",
            "PrivateIpAddress": "172.31.13.82",
            "ProductCodes": [],
            "VpcId": "vpc-d491d9b1",
            "StateTransitionReason": "",
            "InstanceId": "i-e1aeaf54",
...
...

TAG IT

$ aws ec2 create-tags –resources i-e1aeaf54 –tags Key=Name,Value=JohnInstance1

Check its status

$ aws ec2 describe-instance-status –instance-ids i-e1aeaf54

Check its details

$ aws ec2 describe-instances –instance-ids i-e1aeaf54

Retrive from its details specifc information using jq (jq is a lightweight and flexible command-line JSON processor)

$ aws ec2 describe-instances –instance-ids i-e1aeaf54 | jq ‘.Reservations[].Instances[].PublicDnsName’

“ec2-54-183-59-200.us-west-1.compute.amazonaws.com”

Connect to the EC2 instance using your SSH key-pair

ssh -i ~/.ssh/john-keypair.pem ec2-user@ec2-54-183-59-200.us-west-1.compute.amazonaws.com

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/
10 package(s) needed for security, out of 22 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-31-13-82 ~]$

Destroy EC2 instance

$ aws ec2 terminate-instances –instance-ids i-e1aeaf54

{
    "TerminatingInstances": [
        {
            "InstanceId": "i-e1aeaf54",
            "CurrentState": {
                "Code": 32,
                "Name": "shutting-down"
            },
            "PreviousState": {
                "Code": 16,
                "Name": "running"
            }
        }
    ]
}


Create EC2 instance with user data script

Now let’s create an EC2 instance that will be ready to serve a website, we will do that by adding user data content that installs a webserver and adds some content.

Notes:
Amazon EC2 limits the size of user-data to 16KB (This limit applies to the data in raw form, not base64-encoded form.)
You can download a larger script and run it, from S3 for example.
You can run any language that supports the shabang(#!) (e.g. Bash, Python, Ruby, Perl)

//user_data.sh

#!/bin/bash
set -e -x
yum install httpd -y
chkconfig httpd on
service httpd start
INSTANCEID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
echo "Hello from ${INSTANCEID}" > /var/www/html/index.html

Launch/run instance with user-data:

$ aws ec2 run-instances –image-id ami-31490d51 –count 1 –instance-type m3.medium –key-name john-keypair –security-group-ids sg-b1740cd5 –user-data file://user_data.sh

{
    "OwnerId": "XXXX",
    "ReservationId": "r-7ed096cc",
    "Groups": [],
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            },
            "PublicDnsName": "",
            "RootDeviceType": "ebs",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "EbsOptimized": false,
            "LaunchTime": "2016-07-30T22:34:29.000Z",
            "PrivateIpAddress": "172.31.6.82",
            "ProductCodes": [],
            "VpcId": "vpc-d491d9b1",
            "StateTransitionReason": "",
            "InstanceId": "i-6b6a6ade",
...

TAG it

$ aws ec2 create-tags –resources i-e1aeaf54 –tags Key=Name,Value=JohnInstance

1

Get its public DNS name

$ aws ec2 describe-instances –instance-ids i-e974745c | jq ‘.Reservations[].Instances[].PublicDnsName’

“ec2-52-53-165-15.us-west-1.compute.amazonaws.com”

Connect to it

$ ssh -i ~/.ssh/john-keypair.pem ec2-user@ec2-52-53-165-15.us-west-1.compute.amazonaws.com

Verify that httpd is running and the right content is displayed

[ec2-user@ip-172-31-9-131 ~]$ curl http://localhost

Hello from i-e974745c

Troubleshooting: Check the /var/log/cloud-init-output.log file

[ec2-user@ip-172-31-9-131 ~]$ sudo tail /var/log/cloud-init-output.log

  apr-util-ldap.x86_64 0:1.4.1-4.17.amzn1 httpd-tools.x86_64 0:2.2.31-1.8.amzn1

Complete!
+ chkconfig httpd on
+ service httpd start
Starting httpd: [  OK  ]
++ curl -s http://169.254.169.254/latest/meta-data/instance-id
+ INSTANCEID=i-e974745c
+ echo 'Hello from i-e974745c'
Cloud-init v. 0.7.6 finished at Thu, 18 Aug 2016 19:51:46 +0000. Datasource DataSourceEc2.  Up 63.66 seconds

Extra:
To allow access to it from the public you will need to add a security group that allows ports 80 and 443.
You can create one as we did for the SSH access, or you can see if you already have a group.

Look at the descriptions of you security groups (this is why it is important to add good descriptions)
“allow SSH access”
“Allow SSH”
“Allow Web tcp 80 and 443 from 0.0.0.0/0”

Looks like “Allow Web tcp 80 and 443 from 0.0.0.0/0” will do the trick, let’s verify

$ aws ec2 describe-security-groups | jq ‘.SecurityGroups[2]’

{
  "IpPermissionsEgress": [
    {
      "IpProtocol": "-1",
      "IpRanges": [
        {
          "CidrIp": "0.0.0.0/0"
        }
      ],
      "UserIdGroupPairs": [],
      "PrefixListIds": []
    }
  ],
  "Description": "Allow Web tcp 80 and 443 from 0.0.0.0/0",
  "Tags": [
    {
      "Value": "allow_tcp_web",
      "Key": "Name"
    }
  ],
  "IpPermissions": [
    {
      "PrefixListIds": [],
      "FromPort": 80,
      "IpRanges": [
        {
          "CidrIp": "0.0.0.0/0"
        }
      ],
      "ToPort": 80,
      "IpProtocol": "tcp",
      "UserIdGroupPairs": []
    },
    {
      "PrefixListIds": [],
      "FromPort": 443,
      "IpRanges": [
        {
          "CidrIp": "0.0.0.0/0"
        }
      ],
      "ToPort": 443,
      "IpProtocol": "tcp",
      "UserIdGroupPairs": []
    }
  ],
  "GroupName": "allow_tcp_web",
  "VpcId": "vpc-XXX",
  "OwnerId": "XXXXXXX",
  "GroupId": "sg-a3672ac7"
}

Let’s add this security group (sg-a3672ac7) to our EC2 instance (i-e974745c)
// This does not append, you have to specify all groups, in this case I am adding the SSH and WEB security groups to the instance

$ aws ec2 modify-instance-attribute –instance-id i-e974745c –groups sg-b1740cd5 sg-a3672ac7

Verify that the instance has both security groups

$ aws ec2 describe-instances –instance-ids i-e974745c | jq ‘.Reservations[].Instances[].SecurityGroups[]’

{
  "GroupName": "allow_tcp_web",
  "GroupId": "sg-a3672ac7"
}
{
  "GroupName": "allow_tcp_22",
  "GroupId": "sg-b1740cd5"
}

You can now verify that the content is publicly available
awscli_userdata1

One of the most important things you should do to your systems is to ensure they have the right time.
In this post I will show how to check and ensure your systems have the correct time using PowerCli.

==> Login to vCenter:

$admin = Get-Credential –Credential EXAMPLE\john
Connect-VIServer -Server vc.example.com -Credential $admin

==> Check time settings:

Get-VMHost | Sort Name | Select Name, `
   @{N="NTPServer";E={$_ |Get-VMHostNtpServer}}, `
   Timezone, `
   @{N="CurrentTime";E={(Get-View $_.ExtensionData.ConfigManager.DateTimeSystem) | Foreach {$_.QueryDateTime().ToLocalTime()}}}, `
   @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq "ntpd"}).Running}}, `
   @{N="StartUpPolicy";E={(Get-VMHostService -VMHost $_ |Where-Object {$_.Key -eq "ntpd"}).Policy}}, `
   @{N="FirewallException";E={$_ | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Select-Object –ExpandProperty Enabled}} `
   | Format-Table -AutoSize

Output:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMHost | Sort Name | Select Name, `
>>    @{N="NTPServer";E={$_ |Get-VMHostNtpServer}}, `
>>    Timezone, `
>>    @{N="CurrentTime";E={(Get-View $_.ExtensionData.ConfigManager.DateTimeSystem) | Foreach {$_.QueryDateTime().ToLocalTime()}}}, `
>>    @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq "ntpd"}).Running}}, `
>>    @{N="StartUpPolicy";E={(Get-VMHostService -VMHost $_ |Where-Object {$_.Key -eq "ntpd"}).Policy}}, `
>>    @{N="FirewallException";E={$_ | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Select-Object -ExpandProperty Enabled}} `
>>    | Format-Table -AutoSize
>>

Name                 NTPServer 					TimeZone CurrentTime         ServiceRunning StartUpPolicy FirewallException
----                 --------- 					-------- -----------         -------------- ------------- -----------------
esx1.example.com           					UTC      6/7/2015 3:25:39 PM          False off                       False
esx2.example.com           					UTC      6/7/2015 3:25:40 PM          False off                       False
esx3.example.com 	{192.168.10.1,192.168.11.1}	        UTC      6/7/2015 3:25:42 PM          False off                       False
esx4.example.com 	192.168.11.1 				UTC      6/7/2015 3:25:43 PM          False off                       False

==> Set time to correct time:

// Get time from the machine running PowerCli
$currentTime = Get-Date

// Update time on ESX hosts
$hosts_time = Get-VMHost | %{ Get-View $_.ExtensionData.ConfigManager.DateTimeSystem }
$hosts_time.UpdateDateTime((Get-Date($currentTime.ToUniversalTime()) -format u))

==> Remove old NTP servers (if any):

$old_ntp_server = '192.168.10.1'
Get-VMHost | Remove-VmHostNtpServer -NtpServer $old_ntp_server -Confirm

Output:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMHost | Sort Name | Select Name, `
>>    @{N="NTPServer";E={$_ |Get-VMHostNtpServer}}, `
>>    Timezone, `
>>    @{N="CurrentTime";E={(Get-View $_.ExtensionData.ConfigManager.DateTimeSystem) | Foreach {$_.QueryDateTime().ToLocalTime()}}}, `
>>    @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq "ntpd"}).Running}}, `
>>    @{N="StartUpPolicy";E={(Get-VMHostService -VMHost $_ |Where-Object {$_.Key -eq "ntpd"}).Policy}}, `
>>    @{N="FirewallException";E={$_ | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Select-Object -ExpandProperty Enabled}} `
>>    | Format-Table -AutoSize
>>

Name                 NTPServer 	TimeZone CurrentTime         ServiceRunning StartUpPolicy FirewallException
----                 --------- 	-------- -----------         -------------- ------------- -----------------
esx1.example.com           	UTC      6/7/2015 3:25:39 PM          False off                       False
esx2.example.com           	UTC      6/7/2015 3:25:40 PM          False off                       False
esx3.example.com 		UTC      6/7/2015 3:25:42 PM          False off                       False
esx4.example.com		UTC      6/7/2015 3:25:43 PM          False off                       False

==> Change NTP to desired configuration:

$ntp_server = '192.168.10.1'
Get-VMHost | Add-VMHostNtpServer $ntp_server
Get-VMHost | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Set-VMHostFirewallException -Enabled:$true
Get-VMHost | Get-VmHostService | Where-Object {$_.key -eq "ntpd"} | Start-VMHostService
Get-VMhost | Get-VmHostService | Where-Object {$_.key -eq "ntpd"} | Set-VMHostService -policy "automatic"

Output:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> $ntp_server = '192.168.10.1'
PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMHost | Add-VMHostNtpServer $ntp_server
192.168.10.1
192.168.10.1
192.168.10.1
192.168.10.1

==> Enable Firewall Exception

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMHost | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Set-VMHostFirewallException -Enabled:$true

Name                 Enabled IncomingPorts  OutgoingPorts  Protocols  ServiceRunning
----                 ------- -------------  -------------  ---------  --------------
NTP Client           True                   123            UDP        True
NTP Client           True                   123            UDP        True
NTP Client           True                   123            UDP        False
NTP Client           True                   123            UDP        False

==> Start NTPd service

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMHost | Get-VmHostService | Where-Object {$_.key -eq "ntpd"} | Start-VMHostService

Key                  Label                          Policy     Running  Required
---                  -----                          ------     -------  --------
ntpd                 NTP Daemon                     on         True     False
ntpd                 NTP Daemon                     on         True     False
ntpd                 NTP Daemon                     off        True     False
ntpd                 NTP Daemon                     off        True     False

==> Ensure NTPd service starts automatically (via policy)

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMhost | Get-VmHostService | Where-Object {$_.key -eq "ntpd"} | Set-VMHostService -policy "automatic"

Key                  Label                          Policy     Running  Required
---                  -----                          ------     -------  --------
ntpd                 NTP Daemon                     automatic  True     False
ntpd                 NTP Daemon                     automatic  True     False
ntpd                 NTP Daemon                     automatic  True     False
ntpd                 NTP Daemon                     automatic  True     False

==> Verify all is set the way you expected

Get-VMHost | Sort Name | Select Name, `
   @{N="NTPServer";E={$_ |Get-VMHostNtpServer}}, `
   Timezone, `
   @{N="CurrentTime";E={(Get-View $_.ExtensionData.ConfigManager.DateTimeSystem) | Foreach {$_.QueryDateTime().ToLocalTime()}}}, `
   @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq "ntpd"}).Running}}, `
   @{N="StartUpPolicy";E={(Get-VMHostService -VMHost $_ |Where-Object {$_.Key -eq "ntpd"}).Policy}}, `
   @{N="FirewallException";E={$_ | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Select-Object –ExpandProperty Enabled}} `
   | Format-Table -AutoSize

Output:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMHost | Sort Name | Select Name, `
>>    @{N="NTPServer";E={$_ |Get-VMHostNtpServer}}, `
>>    Timezone, `
>>    @{N="CurrentTime";E={(Get-View $_.ExtensionData.ConfigManager.DateTimeSystem) | Foreach {$_.QueryDateTime().ToLocalTime()}}}, `
>>    @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq "ntpd"}).Running}}, `
>>    @{N="StartUpPolicy";E={(Get-VMHostService -VMHost $_ |Where-Object {$_.Key -eq "ntpd"}).Policy}}, `
>>    @{N="FirewallException";E={$_ | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Select-Object -ExpandProperty Enabled}} `
>>    | Format-Table -AutoSize
>>

Name                 NTPServer  TimeZone CurrentTime         ServiceRunning StartUpPolicy FirewallException
----                 ---------  -------- -----------         -------------- ------------- -----------------
esx1.example.com 192.168.10.1 UTC      6/7/2015 3:34:49 PM           True automatic                  True
esx2.example.com 192.168.10.1 UTC      6/7/2015 3:34:51 PM           True automatic                  True
esx3.example.com 192.168.10.1 UTC      6/7/2015 3:34:52 PM           True automatic                  True
esx4.example.com 192.168.10.1 UTC      6/7/2015 3:34:54 PM           True automatic                  True