DevOps | Technologist

Browsing Posts in DevOps

Installing Puppet Enterprise on AWS EC2 using Vagrant

February 6, 2017 John No comments

In this guide I go over how to use Vagrant with AWS and in the process have an automated way to install Puppet Enterprise.
I am separating data and code by having a generic Vagrantfile with the code and have a servers.yaml file with all the data that will change from user to user.
For installing the Puppet Enterprise server I am including the automated provisioning script I am using with Vagrant and using AWS Tags to set the hostname of the launched server.

Pre-requisites:

Vagrant
vagrant-aws plug-in:

$ vagrant plugin install vagrant-aws

1	$ vagrant plugin install vagrant-aws

AWS pre-requisites:

While you can add your AWS credentials to the Vagrantfile, it is not recommended. A better way is to have the AWS CLI tools installed and configured

$ aws configure
AWS Access Key ID [****************XYYY]: XXXXXXXXYYY
AWS Secret Access Key [****************ZOOO]:ZZZZZZZOOO
Default region name [us-east-1]:us-east-1
Default output format [None]: json

$ aws configure

AWS Access Key ID [****************XYYY]: XXXXXXXXYYY

AWS Secret Access Key [****************ZOOO]:ZZZZZZZOOO

Default region name [us-east-1]:us-east-1

Default output format [None]: json

TL;DR To get started right away you can download the project from github vagrant-aws-puppetserver, otherwise follow the guide below.

Create Vagrantfile

The below Vagrantfile utilizes a yaml file (servers.yaml) to provide the data, it allows you to control data using the yaml file and not have to modify the Vagrantfile code – separating code and data.

// Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Specify minimum Vagrant version and Vagrant API version
Vagrant.require_version ">= 1.6.0"
VAGRANTFILE_API_VERSION = "2"

# Require YAML module
require 'yaml'

# Read YAML file with box details
servers = YAML.load_file('servers.yaml')

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

    # Iterate through entries in YAML file
    servers.each do |server|
    
        config.vm.define server['name'] do |srv|

            srv.vm.box = server['box']
            srv.vm.box_url = server['box_url']

            srv.vm.provider :aws do |aws, override|

                ### Dont do these here, better to use awscli with a profile
                #aws.access_key_id = "YOUR KEY"
                #aws.secret_access_key = "YOUR SECRET KEY"
                #aws.session_token = "SESSION TOKEN"

                aws.region = server["aws_region"] if server["aws_region"]
                aws.keypair_name = server["aws_keypair_name"] if server["aws_keypair_name"]
                aws.subnet_id = server["aws_subnet_id"] if server["aws_subnet_id"]
                aws.associate_public_ip = server["aws_associate_public_ip"] if server["aws_associate_public_ip"]
                aws.security_groups = server["aws_security_groups"] if server["aws_security_groups"]
                aws.iam_instance_profile_name = server['aws_iam_role'] if server['aws_iam_role']
                aws.ami = server["aws_ami"] if server["aws_ami"]
                aws.instance_type = server["aws_instance_type"] if server["aws_instance_type"]
                aws.tags = server["aws_tags"] if server["aws_tags"]
                aws.user_data = server["aws_user_data"] if server["aws_user_data"]

                override.ssh.username = server["aws_ssh_username"] if server["aws_ssh_username"]
                override.ssh.private_key_path = server["aws_ssh_private_key_path"] if server["aws_ssh_private_key_path"]           

                config.vm.synced_folder ".", "/vagrant", type: "rsync"

                config.vm.provision :shell, path: server["provision"] if server["provision"]

            end  
        end
    end
end

# -*- mode: ruby -*-

# vi: set ft=ruby :

# Specify minimum Vagrant version and Vagrant API version

Vagrant.require_version ">= 1.6.0"

VAGRANTFILE_API_VERSION = "2"

# Require YAML module

require 'yaml'

# Read YAML file with box details

servers = YAML.load_file('servers.yaml')

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

# Iterate through entries in YAML file

servers.each do |server|

config.vm.define server['name'] do |srv|

srv.vm.box = server['box']

srv.vm.box_url = server['box_url']

srv.vm.provider :aws do |aws, override|

### Dont do these here, better to use awscli with a profile

#aws.access_key_id = "YOUR KEY"

#aws.secret_access_key = "YOUR SECRET KEY"

#aws.session_token = "SESSION TOKEN"

aws.region = server["aws_region"] if server["aws_region"]

aws.keypair_name = server["aws_keypair_name"] if server["aws_keypair_name"]

aws.subnet_id = server["aws_subnet_id"] if server["aws_subnet_id"]

aws.associate_public_ip = server["aws_associate_public_ip"] if server["aws_associate_public_ip"]

aws.security_groups = server["aws_security_groups"] if server["aws_security_groups"]

aws.iam_instance_profile_name = server['aws_iam_role'] if server['aws_iam_role']

aws.ami = server["aws_ami"] if server["aws_ami"]

aws.instance_type = server["aws_instance_type"] if server["aws_instance_type"]

aws.tags = server["aws_tags"] if server["aws_tags"]

aws.user_data = server["aws_user_data"] if server["aws_user_data"]

override.ssh.username = server["aws_ssh_username"] if server["aws_ssh_username"]

override.ssh.private_key_path = server["aws_ssh_private_key_path"] if server["aws_ssh_private_key_path"]

config.vm.synced_folder ".", "/vagrant", type: "rsync"

config.vm.provision :shell, path: server["provision"] if server["provision"]

end

Create servers.yaml

This file contains the information that will be used by the Vagrantfile, this includes
AWS region: Which region will this EC2 server run
AWS keypair: Key used to connect to your launched EC2 instance
AWS subnet id: Where will this EC2 instance sit in the AWS network
AWS associate public ip: Do you need a public IP? true or false
AWS security group: What AWS security group should be associated, should allow Puppetserver needed ports and whatever else you need (ssh, etc)
AWS ami: Which AMI will you be using I am using a CentOS7
AWS instance type: Puppetserver needs enough CPU/RAM, during my testing m3.xlarge was appropriate
AWS SSH username: The EC2 instance user (depends on which AMI you choose), the CentOS AMI expects ec2-user
AWS SSH private key path: The local path to the SSH key pair
AWS User Data: I am adding user data which will execute a bash script that allows Vagrant to interact with the launched EC2 instance
AWS Tags: This is not required for Vagrant and AWS/EC2, but in my provision script I am using the AWS Name Tag to be the system’s hostname, the other 2 tags are there for demonstration purposes
provision: This is a provisioning script that will be run on the EC2 instance – this is the script that install the Puppet Enterprise server
AWS IAM Role: You don’t need to add a role when working with Vagrant and AWS/EC2, but I am using a specific IAM role to allow the launched EC2 instance to be able to get information about its AWS Tags, so it is important that you provide it with a Role that allows DescribeTags, see below IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "XXXXXXX",
            "Action": [
                "ec2:DescribeTags"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "XXXXXXX",

"Action": [

"ec2:DescribeTags"

"Effect": "Allow",

"Resource": "*"

}

]

}

Now use your data in the servers.yaml file
// servers.yaml

---
- name: puppet4
  box: dummy
  box_url: https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
  aws_region: "us-east-1"
  aws_keypair_name: "john-key"
  aws_subnet_id: "subnet-0001" 
  aws_associate_public_ip: false
  aws_security_groups: ['sg-0001'] 
  aws_ami: "ami-0001"
  aws_instance_type: m3.xlarge
  aws_ssh_username: "ec2-user"
  aws_iam_role: "iam-able-to-describe-tags"
  aws_ssh_private_key_path: "/Users/john/.ssh/john-key.pem"
  aws_user_data: "#!/bin/bash\nsed -i -e 's/^Defaults.*requiretty/# Defaults requiretty/g' /etc/sudoers"
  aws_tags: 
    Name: 'puppet4'
    tier: 'stage'
    application: 'puppetserver'
  update: false
  provision: install_puppetserver.sh

---

- name: puppet4

box: dummy

box_url: https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box

aws_region: "us-east-1"

aws_keypair_name: "john-key"

aws_subnet_id: "subnet-0001"

aws_associate_public_ip: false

aws_security_groups: ['sg-0001']

aws_ami: "ami-0001"

aws_instance_type: m3.xlarge

aws_ssh_username: "ec2-user"

aws_iam_role: "iam-able-to-describe-tags"

aws_ssh_private_key_path: "/Users/john/.ssh/john-key.pem"

aws_user_data: "#!/bin/bash\nsed -i -e 's/^Defaults.*requiretty/# Defaults requiretty/g' /etc/sudoers"

aws_tags:

Name: 'puppet4'

tier: 'stage'

application: 'puppetserver'

update: false

provision: install_puppetserver.sh

At this point you can spin up EC2 instances using the above Vagrantfile and servers.yaml file. If you add provision: install_puppetserver.sh to the servers.yaml file as I did and add the below script you will have a Puppet Enterprise server ready to go.

// install_puppetserver.sh

#!/bin/bash -xe

# Ensure pip is installed
if ! which pip; then
	curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
	python get-pip.py
fi

# Upgrade pip 
/bin/pip install --upgrade pip

# Upgrade awscli tools (They are installed in the AMI)
/bin/pip install --upgrade awscli

# Get hostname from AWS Name Tag (requires the EC2 instance to have an IAM role that allows DescribeTags)
AWS_INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
AWS_REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | grep region | cut -d\" -f4)
AWSHOSTNAME=$(aws ec2 describe-tags --region ${AWS_REGION} --filters "Name=resource-id,Values=${AWS_INSTANCE_ID}" --query "Tags[?Key=='Name'].Value[] | [0]" | cut -d\" -f2)

# Set hostname (use the AWS Name Tag)
hostnamectl set-hostname ${AWSHOSTNAME}.cpg.org

# Update system and install wget, git
yum update -y
yum install wget git -y

# Set puppet.cpg.org as hostname in hosts file
echo "$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4) puppet puppet.cpg.org" >> /etc/hosts

# Download puppet 
wget "https://pm.puppetlabs.com/cgi-bin/download.cgi?dist=el&rel=7&arch=x86_64&ver=2016.4.0" -O puppet.2016.4.0.tar.gz
tar -xvzf puppet.2016.4.0.tar.gz
cd puppet-enterprise*

# Create pe.conf file
touch pe.conf
echo '{' >> pe.conf
echo '"console_admin_password": "puppet"' >> pe.conf
echo '"puppet_enterprise::puppet_master_host": "%{::trusted.certname}"' >> pe.conf
echo '}'  >> pe.conf

echo "Install Puppetserver"
./puppet-enterprise-installer -c pe.conf

echo "Adding * to autosign.conf"
cat >> /etc/puppetlabs/puppet/autosign.conf <<'AUTOSIGN'
*
AUTOSIGN

# Run puppet agent
/usr/local/bin/puppet agent -t

#!/bin/bash -xe

# Ensure pip is installed

if ! which pip; then

curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"

python get-pip.py

# Upgrade pip

/bin/pip install --upgrade pip

# Upgrade awscli tools (They are installed in the AMI)

/bin/pip install --upgrade awscli

# Get hostname from AWS Name Tag (requires the EC2 instance to have an IAM role that allows DescribeTags)

AWS_INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)

AWS_REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | grep region | cut -d\" -f4)

AWSHOSTNAME=$(aws ec2 describe-tags --region ${AWS_REGION} --filters "Name=resource-id,Values=${AWS_INSTANCE_ID}" --query "Tags[?Key=='Name'].Value[] | [0]" | cut -d\" -f2)

# Set hostname (use the AWS Name Tag)

hostnamectl set-hostname ${AWSHOSTNAME}.cpg.org

# Update system and install wget, git

yum update -y

yum install wget git -y

# Set puppet.cpg.org as hostname in hosts file

echo "$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4) puppet puppet.cpg.org" >> /etc/hosts

# Download puppet

wget "https://pm.puppetlabs.com/cgi-bin/download.cgi?dist=el&rel=7&arch=x86_64&ver=2016.4.0" -O puppet.2016.4.0.tar.gz

tar -xvzf puppet.2016.4.0.tar.gz

cd puppet-enterprise*

# Create pe.conf file

touch pe.conf

echo '{' >> pe.conf

echo '"console_admin_password": "puppet"' >> pe.conf

echo '"puppet_enterprise::puppet_master_host": "%{::trusted.certname}"' >> pe.conf

echo '}' >> pe.conf

echo "Install Puppetserver"

./puppet-enterprise-installer -c pe.conf

echo "Adding * to autosign.conf"

cat >> /etc/puppetlabs/puppet/autosign.conf <<'AUTOSIGN'

AUTOSIGN

# Run puppet agent

/usr/local/bin/puppet agent -t

aws, Cloud, DevOps, Puppet, Scripting, Vagrant aws, devops, puppet, puppetserver, vagrant

Docker Registry with Authentication and SSL/TLS

September 25, 2016 John 2 comments

In this post I will go over the installation of Docker Registry with Basic Authentication and over SSL/TLS (self-signed cert for demonstration purposes). I will also do this on Docker for Mac, which has some interesting things to note.

Let me start by saying that you can use the Docker Hub which is a freely available Hosted registry, but there may be instances where you want to host your own registry to control where your images are being stored, you can also opt to use the commercial Docker Trusted Registry, but I will go over the free open-source Registry.

Create project structure
Password file to support Basic Auth
Certificates to support TLS
Location to store the images
Create Docker Registry container
Using the Registry
Registry API

Create project structure

mac$ mkdir -p registry/{volumes/{auth,certs,data},docker-registry}
mac$ cd registry
mac$ tree
.
└── registry
    ├── docker-registry // WhereI will store the docker-compose.yml file
    └── volumes 
        ├── auth  // Volume for Basic Auth password file 
        ├── certs // Volume for TLS certs
        └── data  // Volumes for images storage

mac$ mkdir -p registry/{volumes/{auth,certs,data},docker-registry}

mac$ cd registry

mac$ tree

└── registry

├── docker-registry // WhereI will store the docker-compose.yml file

└── volumes

├── auth // Volume for Basic Auth password file

├── certs // Volume for TLS certs

└── data // Volumes for images storage

Password file to support Basic Auth

Let’s create a username and password

mac$ htpasswd -Bbn <USER> <PASSWORD> > volumes/auth/htpasswd

mac$ cat volumes/auth/htpasswd
user1:$2y$05$xOuwA8oaflSZ5wpiJto2/OBOIMTSpZrH69wsaQOsnaKt07Vzu1sBW

mac$ htpasswd -Bbn <USER> <PASSWORD> > volumes/auth/htpasswd

mac$ cat volumes/auth/htpasswd

user1:$2y$05$xOuwA8oaflSZ5wpiJto2/OBOIMTSpZrH69wsaQOsnaKt07Vzu1sBW

Certificates to support TLS

Create a root CA key, which will sign the actual certificates

mac$ openssl genrsa -out volumes/certs/rootCA.key 2048

1	mac$ openssl genrsa -out volumes/certs/rootCA.key 2048

Create root CA certificate which will need to be installed on the systems that will use the Registry

mac$ openssl req -x509 -new -nodes -key volumes/certs/rootCA.key -days 365 -out volumes/certs/rootCA.crt

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:NY
Locality Name (eg, city) []:NY
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Technologist
Organizational Unit Name (eg, section) []:Tech
Common Name (e.g. server FQDN or YOUR name) []:Technologist CA
Email Address []:johnATtechnologist

mac$ openssl req -x509 -new -nodes -key volumes/certs/rootCA.key -days 365 -out volumes/certs/rootCA.crt

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

-----

Country Name (2 letter code) [AU]:US

State or Province Name (full name) [Some-State]:NY

Locality Name (eg, city) []:NY

Organization Name (eg, company) [Internet Widgits Pty Ltd]:Technologist

Organizational Unit Name (eg, section) []:Tech

Common Name (e.g. server FQDN or YOUR name) []:Technologist CA

Email Address []:johnATtechnologist

Now that you have a CA, you can sign certificates.

Create a key for the Registry system

mac$ openssl genrsa -out volumes/certs/host.key 2048

1	mac$ openssl genrsa -out volumes/certs/host.key 2048

Create a Certificate Signing Request (CSR) for the CA to sign

mac$ openssl req -new -key volumes/certs/host.key -out volumes/certs/host.csr

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:NY
Locality Name (eg, city) []:NY
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Technologist
Organizational Unit Name (eg, section) []:Tech
Common Name (e.g. server FQDN or YOUR name) []:registry.example.com
Email Address []:johnATtechnologist

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

mac$ openssl req -new -key volumes/certs/host.key -out volumes/certs/host.csr

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

-----

Country Name (2 letter code) [AU]:US

State or Province Name (full name) [Some-State]:NY

Locality Name (eg, city) []:NY

Organization Name (eg, company) [Internet Widgits Pty Ltd]:Technologist

Organizational Unit Name (eg, section) []:Tech

Common Name (e.g. server FQDN or YOUR name) []:registry.example.com

Email Address []:johnATtechnologist

Please enter the following 'extra' attributes

to be sent with your certificate request

A challenge password []:

An optional company name []:

Finally have the root CA sign the CSR

mac$ openssl x509 -req -in volumes/certs/host.csr -CA volumes/certs/rootCA.crt -CAkey volumes/certs/rootCA.key -CAcreateserial -out volumes/certs/host.crt -days 365

Signature ok
subject=/C=US/ST=NY/L=NY/O=Technologist/OU=Tech/CN=registry.example.com/emailAddress=johnATtechnologist
Getting CA Private Key

mac$ openssl x509 -req -in volumes/certs/host.csr -CA volumes/certs/rootCA.crt -CAkey volumes/certs/rootCA.key -CAcreateserial -out volumes/certs/host.crt -days 365

Signature ok

subject=/C=US/ST=NY/L=NY/O=Technologist/OU=Tech/CN=registry.example.com/emailAddress=johnATtechnologist

Getting CA Private Key

Location to store the images

The tree structure should look like below, having the data folder be the image repository

mac $ tree
.
├── docker-registry
│   └── docker-compose.yml
└── volumes
    ├── auth
    │   └── htpasswd
    ├── certs
    │   ├── host.crt
    │   ├── host.csr
    │   ├── host.key
    │   ├── rootCA.crt
    │   ├── rootCA.key
    │   └── rootCA.srl
    └── data

mac $ tree

├── docker-registry

│ └── docker-compose.yml

└── volumes

├── auth

│ └── htpasswd

├── certs

│ ├── host.crt

│ ├── host.csr

│ ├── host.key

│ ├── rootCA.crt

│ ├── rootCA.key

│ └── rootCA.srl

└── data

Create Docker Registry container

Run your container using docker command

mac$ docker run -d -p 5000:5000 --restart=always --name registry \
  -v /Users/john/docker/blog/registry/volumes/auth:/auth \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  -v /Users/john/docker/blog/registry/volumes/certs:/certs \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/host.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/host.key \
  registry:2

mac$ docker run -d -p 5000:5000 --restart=always --name registry \

-v /Users/john/docker/blog/registry/volumes/auth:/auth \

-e "REGISTRY_AUTH=htpasswd" \

-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \

-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \

-v /Users/john/docker/blog/registry/volumes/certs:/certs \

-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/host.crt \

-e REGISTRY_HTTP_TLS_KEY=/certs/host.key \

registry:2

Or even better, use docker-compose:

mac$ cd docker-registry
mac$ cat docker-compose.yml

1 2	mac$ cd docker-registry mac$ cat docker-compose.yml

registry:
  restart: always
  image: registry:2
  ports:
    - 5000:5000
  environment:
    REGISTRY_HTTP_TLS_CERTIFICATE: /certs/host.crt
    REGISTRY_HTTP_TLS_KEY: /certs/host.key
    REGISTRY_AUTH: htpasswd
    REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
    REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
  volumes:
    - /Users/john/docker/blog/registry/volumes/data:/var/lib/registry
    - /Users/john/docker/blog/registry/volumes/certs:/certs
    - /Users/john/docker/blog/registry/volumes/auth:/auth

registry:

restart: always

image: registry:2

ports:

- 5000:5000

environment:

REGISTRY_HTTP_TLS_CERTIFICATE: /certs/host.crt

REGISTRY_HTTP_TLS_KEY: /certs/host.key

REGISTRY_AUTH: htpasswd

REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd

REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm

volumes:

- /Users/john/docker/blog/registry/volumes/data:/var/lib/registry

- /Users/john/docker/blog/registry/volumes/certs:/certs

- /Users/john/docker/blog/registry/volumes/auth:/auth

Start up container based on docker-compose.yml

mac$ docker-compose up -d
Creating dockerregistry_registry_1

1 2	mac$ docker-compose up -d Creating dockerregistry_registry_1

See it running

mac$ docker-compose ps
          Name                         Command               State           Ports
-------------------------------------------------------------------------------------------
dockerregistry_registry_1   /entrypoint.sh /etc/docker ...   Up      0.0.0.0:5000->5000/tcp

mac$ docker-compose ps

Name Command State Ports

-------------------------------------------------------------------------------------------

dockerregistry_registry_1 /entrypoint.sh /etc/docker ... Up 0.0.0.0:5000->5000/tcp

At this point the Docker Registry is running

Using the Registry

Download an image from Docker Hub (or you can test with your own images built from Dockerfile)

mac$ docker pull ubuntu

1	mac$ docker pull ubuntu

Tag the image to point to your new Registry

mac$ docker tag ubuntu registry.example.com:5000/john/ubuntu

1	mac$ docker tag ubuntu registry.example.com:5000/john/ubuntu

Push image to your Registry (see failure due to ‘unknow CA authority’)

mac$ docker push registry.example.com:5000/john/ubuntu
The push refers to a repository [registry.example.com:5000/john/ubuntu]
Get https://registry.example.com:5000/v1/_ping: x509: certificate signed by unknown authority

mac$ docker push registry.example.com:5000/john/ubuntu

The push refers to a repository [registry.example.com:5000/john/ubuntu]

Get https://registry.example.com:5000/v1/_ping: x509: certificate signed by unknown authority

At this point, you need to add the root CA cert to your trusted certificates.
On the machine that will pull or push to the registry, you will need to install the rootCA.crt CA certificate created earlier:

docker-engine$ mkdir /etc/docker/certs.d/registry.example.com:5000/ 
docker-engine$ cp rootCA.crt /etc/docker/certs.d/registry.example.com:5000/rootCA.crt

1 2	docker-engine$ mkdir /etc/docker/certs.d/registry.example.com:5000/ docker-engine$ cp rootCA.crt /etc/docker/certs.d/registry.example.com:5000/rootCA.crt

On the Mac, you are probably using Docker for Mac, which actually runs a small hypervisor xhyve that virtualizes the docker engine. That means we need to place the rootCA.crt CA certificate inside the xhyve VM running the Docker Engine. There is no simple way of doing it and I am using the following method (which must be done everytime you restart the Docker service).

mac$ docker run -v /Users/john/docker/blog/registry/volumes/certs:/macdata -v /etc/docker:/hostdata ubuntu bash -c "mkdir -p /hostdata/certs.d/registry.example.com:5000/;cat /macdata/rootCA.crt > /hostdata/certs.d/registry.example.com:5000/ca.crt"

1	mac$ docker run -v /Users/john/docker/blog/registry/volumes/certs:/macdata -v /etc/docker:/hostdata ubuntu bash -c "mkdir -p /hostdata/certs.d/registry.example.com:5000/;cat /macdata/rootCA.crt > /hostdata/certs.d/registry.example.com:5000/ca.crt"

Now let’s try to Push the image to the Registry again

linux$ docker push registry.example.com:5000/john/ubuntu
The push refers to a repository [registry.example.com:5000/john/ubuntu]
4f9d527ff23f: Preparing
cc8df16ebbaf: Preparing
c3c1fd75be70: Preparing
5c42a1311e7e: Preparing
c8305edc6321: Preparing
unauthorized: authentication required

linux$ docker push registry.example.com:5000/john/ubuntu

The push refers to a repository [registry.example.com:5000/john/ubuntu]

4f9d527ff23f: Preparing

cc8df16ebbaf: Preparing

c3c1fd75be70: Preparing

5c42a1311e7e: Preparing

c8305edc6321: Preparing

unauthorized: authentication required

Good, we are getting somewhere, TLS is working, and it now asks for Authentication
Authenticate to the Docker Registry

linux$ docker login registry.example.com:5000
Username (testuser): user1
Password:
Login Succeeded

linux$ docker login registry.example.com:5000

Username (testuser): user1

Password:

Now let’s successfully push our image

linux$ docker push registry.example.com:5000/john/ubuntu
The push refers to a repository [registry.example.com:5000/john/ubuntu]
4f9d527ff23f: Pushed
cc8df16ebbaf: Pushed
c3c1fd75be70: Pushed
5c42a1311e7e: Pushed
c8305edc6321: Pushed
latest: digest: sha256:35dca5fc91c4890f787a266a057ad76b98ff3af43c75aa7e10962d06f06fd6e5 size: 1357

linux$ docker push registry.example.com:5000/john/ubuntu

The push refers to a repository [registry.example.com:5000/john/ubuntu]

4f9d527ff23f: Pushed

cc8df16ebbaf: Pushed

c3c1fd75be70: Pushed

5c42a1311e7e: Pushed

c8305edc6321: Pushed

latest: digest: sha256:35dca5fc91c4890f787a266a057ad76b98ff3af43c75aa7e10962d06f06fd6e5 size: 1357

At this point you can share your Docker image, or incorporate into a development or integration workflow by pulling the image
From any Docker engine system:

linux$ docker login registry.example.com:5000
Username: user1
Password:
Login Succeeded

linux$ docker login registry.example.com:5000

Username: user1

Password:

linux$ docker pull registry.example.com:5000/john/ubuntu
Using default tag: latest
latest: Pulling from john/ubuntu
Digest: sha256:35dca5fc91c4890f787a266a057ad76b98ff3af43c75aa7e10962d06f06fd6e5
Status: Downloaded newer image for registry.example.com:5000/john/ubuntu:latest

linux$ docker pull registry.example.com:5000/john/ubuntu

Using default tag: latest

latest: Pulling from john/ubuntu

Digest: sha256:35dca5fc91c4890f787a266a057ad76b98ff3af43c75aa7e10962d06f06fd6e5

Status: Downloaded newer image for registry.example.com:5000/john/ubuntu:latest

Registry API

You can use the Docker Registry API https://docs.docker.com/registry/spec/api/ to interact with your Registry.

For example to see a catalog of your images:

$ curl --user user1:password1 --cacert rootCA.crt -X GET https://registry.example.com:5000/v2/_catalog

{"repositories":["john/ubuntu"]}

$ curl --user user1:password1 --cacert rootCA.crt -X GET https://registry.example.com:5000/v2/_catalog

{"repositories":["john/ubuntu"]}

DevOps containers, docker, registry, tls

AWS EC2 automation using AWS CLI and User-Data

August 22, 2016 John No comments

In this post I will go over the installation and usage of the AWS CLI to deploy EC2 machines, also combined with AWS user-data to automate actions/scripts that will run on the EC2 machines at install time.

Install the AWS CLI
You can install the AWS CLI in many ways (e.g. zip file, brew, PIP, etc) for details you can follow http://docs.aws.amazon.com/cli/latest/userguide/installing.html
I will install via the Python package manager (PIP) inside a virtual environment

Create virtual environment

$ virtualenv env
New python executable in env/bin/python2.7
Also creating executable in env/bin/python
Installing setuptools, pip…done.

Activate virtual environment

$ source env/bin/activate

Install the awscli

$ pip install awscli
Collecting awscli
Downloading awscli-1.10.53-py2.py3-none-any.whl (970kB)
100% |################################| 970kB 582kB/s
Collecting botocore==1.4.43 (from awscli)
Downloading botocore-1.4.43-py2.py3-none-any.whl (2.5MB)
100% |################################| 2.5MB 206kB/s
Collecting s3transfer<0.2.0,>=0.1.0 (from awscli)
Downloading s3transfer-0.1.1-py2.py3-none-any.whl (49kB)
100% |################################| 49kB 1.4MB/s
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
100% |################################| 49kB 6.2MB/s
Collecting colorama<=0.3.7,>=0.2.5 (from awscli)
Downloading colorama-0.3.7-py2.py3-none-any.whl
Collecting docutils>=0.10 (from awscli)
Downloading docutils-0.12.tar.gz (1.6MB)
100% |################################| 1.6MB 312kB/s
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.4.43->awscli)
Downloading jmespath-0.9.0-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.4.43->awscli)
Downloading python_dateutil-2.5.3-py2.py3-none-any.whl (201kB)
100% |################################| 204kB 1.9MB/s
Collecting futures<4.0.0,>=2.2.0 (from s3transfer<0.2.0,>=0.1.0->awscli)
Downloading futures-3.0.5-py2-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
Downloading pyasn1-0.1.9-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.4.43->awscli)
Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, pyasn1, futures, python-dateutil, jmespath, docutils, colorama, rsa, s3transfer, botocore, awscli
Running setup.py install for docutils
changing mode of build/scripts-2.7/rst2html.py from 644 to 755
changing mode of build/scripts-2.7/rst2s5.py from 644 to 755
…
Successfully installed awscli-1.10.53 botocore-1.4.43 colorama-0.3.7 docutils-0.12 futures-3.0.5 jmespath-0.9.0 pyasn1-0.1.9 python-dateutil-2.5.3 rsa-3.4.2 s3transfer-0.1.1 six-1.10.0

Verify installation (and review help documentation)

$ aws help

Configure the AWS CLI to use AWS credentials
You will need to provide an AWS Access Key ID and AWS secreate Access Key, these will map to a AWS user and the privileges this user have. The user can (probably should have) been created from the AWS console.

$ aws configure
AWS Access Key ID [****************XYYY]: XXXXXXXXYYY
AWS Secret Access Key [****************ZOOO]:ZZZZZZZOOO
Default region name [us-east-1]:us-east-1
Default output format [None]: json

The above will create a default configuration and set of credentials under ~/.aws/{config,credentials}

Launching an EC2 Instance from awscli

To connect to the EC2 instance you will need at least a SSH keypair and allow access to SSH into the EC2 instance

Create key pair

$ aws ec2 create-key-pair –key-name john-keypair –query ‘KeyMaterial’ –output text > ~/.ssh/john-keypair.pem

$ chmod 400 ~/.ssh/john-keypair.pem

Verify key existence

$ aws ec2 describe-key-pairs –key-name john-keypair
KEYPAIRS 03:eb:3a:d3:13:ba:d3:e3:03:13:b3:f1:43:83:cc:03:ec:d8:4b:43 john-keypair

Create a Security group that allows ingress SSH (tcp 22)

$ aws ec2 create-security-group –group-name allow_tcp_22 –description “Allow SSH”

sg-b1740cd5

// TAG it, I cannot stress enough how important it is to tag every resource with at least a Name

$ aws ec2 create-tags –resources sg-b1740cd5 –tags Key=Name,Value=allow_tcp_22

Add rules, for example: allow inbound (ingress) tcp port 22 to all (0.0.0.0/0)

$ aws ec2 authorize-security-group-ingress –group-id sg-b1740cd5 –protocol tcp –port 22 –cidr 0.0.0.0/0

Select the AWS AMI (aka template) you will use to clone and create your EC2 instance.

See your images (For example I have two Encrypted Ubuntu images) These are using output=text in ~/.aws/config, just to show you what that looks like as oposed to JSON.

$ aws ec2 describe-images –owners self
IMAGES x86_64 2016-07-30T13:16:39.000Z xen ami-a64403c2 /Encrypted Ubuntu Linux 14.04 (HVM) machine Encrypted Red Hat Enterprise Linux (HVM) False /dev/sda1 ebs simple available hvm
BLOCKDEVICEMAPPINGS /dev/sda1
EBS True True snap-bc158f41 10 gp2
IMAGES x86_64 2016-07-30T13:22:03.000Z xen ami-a74403c3 /Encrypted Ubuntu Linux 16.04 (HVM) machine Encrypted Red Hat Enterprise Linux (HVM) False /dev/sda1 ebs simple available hvm
BLOCKDEVICEMAPPINGS /dev/sda1
EBS True True snap-900aa0a1 10 gp2

Or see available images from Redhat

$ aws ec2 describe-images –owners 309956199498 –filters “Name=architecture,Values=x86_64” | grep RHEL-7.1
IMAGES x86_64 2015-02-26T16:27:33.000Z Provided by Red Hat, Inc. xen ami-a540a5e1 309956199498/RHEL-7.1_HVM_GA-20150225-x86_64-1-Hourly2-GP2 machine RHEL-7.1_HVM_GA-20150225-x86_64-1-Hourly2-GP2 309956199498 True /dev/sda1 ebs available hvm
IMAGES x86_64 2015-08-04T17:22:47.000Z Provided by Red Hat, Inc. xen ami-c1996685 309956199498/RHEL-7.1_HVM-20150803-x86_64-1-Hourly2-GP2 machine RHEL-7.1_HVM-20150803-x86_64-1-Hourly2-GP2309956199498 True /dev/sda1 ebs simple available hvm

Or even the images from Amazon, which are FREE – this is the one I will use

$ aws ec2 describe-images –owners amazon –filters “Name=root-device-type,Values=ebs” “Name=architecture,Values=x86_64” | grep ‘Amazon Linux AMI 2016’
…
IMAGES x86_64 2016-06-22T08:08:12.000Z Amazon Linux AMI 2016.03.3 x86_64 HVM GP2 xen ami-31490d51 amazon/amzn-ami-hvm-2016.03.3.x86_64-gp2 amazon machine amzn-ami-hvm-2016.03.3.x86_64-gp2 137112412989 True /dev/xvda ebs simple available hvm
…

Launch/Run the instance based on the chosen AMI (e.g. ami-31490d51)

$ aws ec2 run-instances –image-id ami-31490d51 –count 1 –instance-type m3.medium –key-name john-keypair –security-group-ids sg-b1740cd5

{
    "OwnerId": "XXXX",
    "ReservationId": "r-XXc3852a",
    "Groups": [],
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            },
            "PublicDnsName": "",
            "RootDeviceType": "ebs",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "EbsOptimized": false,
            "LaunchTime": "2016-07-30T23:44:30.000Z",
            "PrivateIpAddress": "172.31.13.82",
            "ProductCodes": [],
            "VpcId": "vpc-d491d9b1",
            "StateTransitionReason": "",
            "InstanceId": "i-e1aeaf54",
...
...

{

"OwnerId": "XXXX",

"ReservationId": "r-XXc3852a",

"Groups": [],

"Instances": [

{

"Monitoring": {

"State": "disabled"

"PublicDnsName": "",

"RootDeviceType": "ebs",

"State": {

"Code": 0,

"Name": "pending"

"EbsOptimized": false,

"LaunchTime": "2016-07-30T23:44:30.000Z",

"PrivateIpAddress": "172.31.13.82",

"ProductCodes": [],

"VpcId": "vpc-d491d9b1",

"StateTransitionReason": "",

"InstanceId": "i-e1aeaf54",

...

TAG IT

$ aws ec2 create-tags –resources i-e1aeaf54 –tags Key=Name,Value=JohnInstance1

Check its status

$ aws ec2 describe-instance-status –instance-ids i-e1aeaf54

Check its details

$ aws ec2 describe-instances –instance-ids i-e1aeaf54

Retrive from its details specifc information using jq (jq is a lightweight and flexible command-line JSON processor)

$ aws ec2 describe-instances –instance-ids i-e1aeaf54 | jq ‘.Reservations[].Instances[].PublicDnsName’

“ec2-54-183-59-200.us-west-1.compute.amazonaws.com”

Connect to the EC2 instance using your SSH key-pair

ssh -i ~/.ssh/john-keypair.pem ec2-user@ec2-54-183-59-200.us-west-1.compute.amazonaws.com

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/
10 package(s) needed for security, out of 22 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-31-13-82 ~]$

ssh -i ~/.ssh/john-keypair.pem ec2-user@ec2-54-183-59-200.us-west-1.compute.amazonaws.com

__| __|_ )

_| ( / Amazon Linux AMI

___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/

10 package(s) needed for security, out of 22 available

Run "sudo yum update" to apply all updates.

[ec2-user@ip-172-31-13-82 ~]$

Destroy EC2 instance

$ aws ec2 terminate-instances –instance-ids i-e1aeaf54

{
    "TerminatingInstances": [
        {
            "InstanceId": "i-e1aeaf54",
            "CurrentState": {
                "Code": 32,
                "Name": "shutting-down"
            },
            "PreviousState": {
                "Code": 16,
                "Name": "running"
            }
        }
    ]
}

{

"TerminatingInstances": [

{

"InstanceId": "i-e1aeaf54",

"CurrentState": {

"Code": 32,

"Name": "shutting-down"

"PreviousState": {

"Code": 16,

"Name": "running"

}

]

}

Create EC2 instance with user data script
Now let’s create an EC2 instance that will be ready to serve a website, we will do that by adding user data content that installs a webserver and adds some content.

Notes:
Amazon EC2 limits the size of user-data to 16KB (This limit applies to the data in raw form, not base64-encoded form.)
You can download a larger script and run it, from S3 for example.
You can run any language that supports the shabang(#!) (e.g. Bash, Python, Ruby, Perl)

//user_data.sh

#!/bin/bash
set -e -x
yum install httpd -y
chkconfig httpd on
service httpd start
INSTANCEID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
echo "Hello from ${INSTANCEID}" > /var/www/html/index.html

#!/bin/bash

set -e -x

yum install httpd -y

chkconfig httpd on

service httpd start

INSTANCEID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)

echo "Hello from ${INSTANCEID}" > /var/www/html/index.html

Launch/run instance with user-data:

$ aws ec2 run-instances –image-id ami-31490d51 –count 1 –instance-type m3.medium –key-name john-keypair –security-group-ids sg-b1740cd5 –user-data file://user_data.sh

{
    "OwnerId": "XXXX",
    "ReservationId": "r-7ed096cc",
    "Groups": [],
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            },
            "PublicDnsName": "",
            "RootDeviceType": "ebs",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "EbsOptimized": false,
            "LaunchTime": "2016-07-30T22:34:29.000Z",
            "PrivateIpAddress": "172.31.6.82",
            "ProductCodes": [],
            "VpcId": "vpc-d491d9b1",
            "StateTransitionReason": "",
            "InstanceId": "i-6b6a6ade",
...

{

"OwnerId": "XXXX",

"ReservationId": "r-7ed096cc",

"Groups": [],

"Instances": [

{

"Monitoring": {

"State": "disabled"

"PublicDnsName": "",

"RootDeviceType": "ebs",

"State": {

"Code": 0,

"Name": "pending"

"EbsOptimized": false,

"LaunchTime": "2016-07-30T22:34:29.000Z",

"PrivateIpAddress": "172.31.6.82",

"ProductCodes": [],

"VpcId": "vpc-d491d9b1",

"StateTransitionReason": "",

"InstanceId": "i-6b6a6ade",

...

TAG it

$ aws ec2 create-tags –resources i-e1aeaf54 –tags Key=Name,Value=JohnInstance

Get its public DNS name

$ aws ec2 describe-instances –instance-ids i-e974745c | jq ‘.Reservations[].Instances[].PublicDnsName’

“ec2-52-53-165-15.us-west-1.compute.amazonaws.com”

Connect to it

$ ssh -i ~/.ssh/john-keypair.pem ec2-user@ec2-52-53-165-15.us-west-1.compute.amazonaws.com

Verify that httpd is running and the right content is displayed

[ec2-user@ip-172-31-9-131 ~]$ curl http://localhost

Hello from i-e974745c

Troubleshooting: Check the /var/log/cloud-init-output.log file

[ec2-user@ip-172-31-9-131 ~]$ sudo tail /var/log/cloud-init-output.log

  apr-util-ldap.x86_64 0:1.4.1-4.17.amzn1 httpd-tools.x86_64 0:2.2.31-1.8.amzn1

Complete!
+ chkconfig httpd on
+ service httpd start
Starting httpd: [  OK  ]
++ curl -s http://169.254.169.254/latest/meta-data/instance-id
+ INSTANCEID=i-e974745c
+ echo 'Hello from i-e974745c'
Cloud-init v. 0.7.6 finished at Thu, 18 Aug 2016 19:51:46 +0000. Datasource DataSourceEc2.  Up 63.66 seconds

apr-util-ldap.x86_64 0:1.4.1-4.17.amzn1 httpd-tools.x86_64 0:2.2.31-1.8.amzn1

Complete!

+ chkconfig httpd on

+ service httpd start

Starting httpd: [ OK ]

++ curl -s http://169.254.169.254/latest/meta-data/instance-id

+ INSTANCEID=i-e974745c

+ echo 'Hello from i-e974745c'

Cloud-init v. 0.7.6 finished at Thu, 18 Aug 2016 19:51:46 +0000. Datasource DataSourceEc2. Up 63.66 seconds

Extra:
To allow access to it from the public you will need to add a security group that allows ports 80 and 443.
You can create one as we did for the SSH access, or you can see if you already have a group.

Look at the descriptions of you security groups (this is why it is important to add good descriptions)
“allow SSH access”
“Allow SSH”
“Allow Web tcp 80 and 443 from 0.0.0.0/0”
…

Looks like “Allow Web tcp 80 and 443 from 0.0.0.0/0” will do the trick, let’s verify

$ aws ec2 describe-security-groups | jq ‘.SecurityGroups[2]’

{
  "IpPermissionsEgress": [
    {
      "IpProtocol": "-1",
      "IpRanges": [
        {
          "CidrIp": "0.0.0.0/0"
        }
      ],
      "UserIdGroupPairs": [],
      "PrefixListIds": []
    }
  ],
  "Description": "Allow Web tcp 80 and 443 from 0.0.0.0/0",
  "Tags": [
    {
      "Value": "allow_tcp_web",
      "Key": "Name"
    }
  ],
  "IpPermissions": [
    {
      "PrefixListIds": [],
      "FromPort": 80,
      "IpRanges": [
        {
          "CidrIp": "0.0.0.0/0"
        }
      ],
      "ToPort": 80,
      "IpProtocol": "tcp",
      "UserIdGroupPairs": []
    },
    {
      "PrefixListIds": [],
      "FromPort": 443,
      "IpRanges": [
        {
          "CidrIp": "0.0.0.0/0"
        }
      ],
      "ToPort": 443,
      "IpProtocol": "tcp",
      "UserIdGroupPairs": []
    }
  ],
  "GroupName": "allow_tcp_web",
  "VpcId": "vpc-XXX",
  "OwnerId": "XXXXXXX",
  "GroupId": "sg-a3672ac7"
}

{

"IpPermissionsEgress": [

{

"IpProtocol": "-1",

"IpRanges": [

{

"CidrIp": "0.0.0.0/0"

}

"UserIdGroupPairs": [],

"PrefixListIds": []

}

"Description": "Allow Web tcp 80 and 443 from 0.0.0.0/0",

"Tags": [

{

"Value": "allow_tcp_web",

"Key": "Name"

}

"IpPermissions": [

{

"PrefixListIds": [],

"FromPort": 80,

"IpRanges": [

{

"CidrIp": "0.0.0.0/0"

}

"ToPort": 80,

"IpProtocol": "tcp",

"UserIdGroupPairs": []

{

"PrefixListIds": [],

"FromPort": 443,

"IpRanges": [

{

"CidrIp": "0.0.0.0/0"

}

"ToPort": 443,

"IpProtocol": "tcp",

"UserIdGroupPairs": []

}

"GroupName": "allow_tcp_web",

"VpcId": "vpc-XXX",

"OwnerId": "XXXXXXX",

"GroupId": "sg-a3672ac7"

}

Let’s add this security group (sg-a3672ac7) to our EC2 instance (i-e974745c)
// This does not append, you have to specify all groups, in this case I am adding the SSH and WEB security groups to the instance

$ aws ec2 modify-instance-attribute –instance-id i-e974745c –groups sg-b1740cd5 sg-a3672ac7

Verify that the instance has both security groups

$ aws ec2 describe-instances –instance-ids i-e974745c | jq ‘.Reservations[].Instances[].SecurityGroups[]’

{
  "GroupName": "allow_tcp_web",
  "GroupId": "sg-a3672ac7"
}
{
  "GroupName": "allow_tcp_22",
  "GroupId": "sg-b1740cd5"
}

{

"GroupName": "allow_tcp_web",

"GroupId": "sg-a3672ac7"

}

{

"GroupName": "allow_tcp_22",

"GroupId": "sg-b1740cd5"

}

You can now verify that the content is publicly available

aws, Cloud, DevOps automation, aws, awscli, cloud, ec2, user-data

Infrastructure As Code: Create Linux (RHEL/CentOS) base Images Using Packer

January 25, 2016 John 3 comments

In a previous post I wrote about how to Create a custom Vagrant Box from scratch.

In this post I will walk through the usage of Packer to automate the creation of a CentOS 7 image that can be used in Vagrant or even vSphere.

Packer is a cool Hashicorp tool that helps you automate the creation of machine images.

You can download the project/git repo at https://github.com/parcejohn/packer-centos7

From their website:
“Packer is easy to use and automates the creation of any type of machine image. It embraces modern configuration management by encouraging you to use automated scripts to install and configure the software within your Packer-made images. Packer brings machine images into the modern age, unlocking untapped potential and opening new opportunities.”

I use Packer to continue the journey to Infrastructure As Code, where even my golden images/templates are automated and source controlled (on Git).

Requirements:

* Packer // I installed on Mac using $ brew install packer
* vagrant // I installed on Mac using $ brew install vagrant
* vmware fusion // I installed using $ brew cask install vmware-fusion
* CentOS 7 ISO file

Packer uses a JSON template file to orchestrate the image creation and there are different stages in the process, I will concentrate on the below three, but you should familiarize with all of them (https://www.packer.io/docs/basics/terminology.html)

Builders: Packer component to create a machine image for a single platform, in this case I will be using the VMware builder.
Provisioners: Packer component that installs and configures software within a running machine prior to that machine being turned into a static image. Example provisioners include shell scripts, Chef, Puppet, etc. I will be using the shell provisioner.
Post-Processors: Packer component that takes the result of the builder or another post-processor and process that to create a new artifact. Examples of post-processors are compress to compress artifacts, upload to upload artifacts, etc. I will be creating a Vagrant box as the artifact, and also demonstrate how to upload the image to vSphere.

It’s time to walk through the process of creating a CentOS 7 image using Packer.

1) Create directory structure
The http folder will host a kickstart file, Packer will use its built in web server to serve this kickstart file
The scripts folder will host the provisioning scripts that will define the machine

centos/
├── http
└── scripts

2) Populate with the following configuration files and scripts (the contents of these files are shown in the next steps)

centos/
├── centos-7.1-x64-vmware.json
├── http
│   └── ks.cfg
├── scripts
│   ├── base.sh
│   ├── cleanup.sh
│   ├── hgfs.sh
│   ├── vmware.sh
│   └── zerodisk.sh
├── template.json
└── vagrant_rsa_key

3) Create a SSH key pair, you will need this to login to the server to complete install and configuration

The Public Key will be injected to the ‘vagrant’ user as part of the kickstart, it will then be used in the Packer JSON template to allow Packer to login to the machine
The Private Key will stay on your system to allow Packer and Vagrant to login to the created machine.

$ ssh-keygen -t rsa -b 4096 -C "vagrant" -N '' -q -f ./vagrant_rsa_key

1	$ ssh-keygen -t rsa -b 4096 -C "vagrant" -N '' -q -f ./vagrant_rsa_key

4) Create a Packer JSON Template file (template.json)

{
  "variables": {
    "vm_name": "centos-7.1-vmware",
    "iso_url": "{{env `ISO_URL`}}",
    "iso_sha256": "f90e4d28fa377669b2db16cbcb451fcb9a89d2460e3645993e30e137ac37d284"
  },
  "builders": [
    {
      "headless": true,
      "type": "vmware-iso",
      "boot_command": [
        "<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg<enter><wait>"
      ],
      "boot_wait": "10s",
      "disk_size": 8192,
      "guest_os_type": "centos-64",
      "http_directory": "http",
      "iso_url": "{{user `iso_url`}}",
      "iso_checksum_type": "sha256",
      "iso_checksum": "{{user `iso_sha256`}}",
      "ssh_username": "vagrant",
      "ssh_private_key_file": "vagrant_rsa",
      "ssh_port": 22,
      "ssh_wait_timeout": "10000s",
      "shutdown_command": "echo '/sbin/halt -h -p' > /tmp/shutdown.sh; echo 'vagrant'|sudo -S sh '/tmp/shutdown.sh'",
      "tools_upload_flavor": "linux",
      "tools_upload_path": "/tmp/vmware_tools_{{.Flavor}}.iso",
      "vmx_data": {
        "memsize": "1024",
        "numvcpus": "1",
        "cpuid.coresPerSocket": "1"
      }
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "execute_command": "echo 'vagrant'|sudo -S sh '{{.Path}}'",
      "override": {
        "vmware-iso": {
          "scripts": [
            "scripts/base.sh",
            "scripts/vmware.sh",
            "scripts/hgfs.sh",
            "scripts/cleanup.sh",
            "scripts/zerodisk.sh"
          ]
        }
      }
    }
  ],
  "post-processors": [
    {
      "type": "vagrant",
      "override": {
        "vmware": {
          "output": "centos-7.1-x64-vmware.box"
        }
      }
    }
  ]
}

{

"variables": {

"vm_name": "centos-7.1-vmware",

"iso_url": "{{env `ISO_URL`}}",

"iso_sha256": "f90e4d28fa377669b2db16cbcb451fcb9a89d2460e3645993e30e137ac37d284"

"builders": [

{

"headless": true,

"type": "vmware-iso",

"boot_command": [

"<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg<enter><wait>"

"boot_wait": "10s",

"disk_size": 8192,

"guest_os_type": "centos-64",

"http_directory": "http",

"iso_url": "{{user `iso_url`}}",

"iso_checksum_type": "sha256",

"iso_checksum": "{{user `iso_sha256`}}",

"ssh_username": "vagrant",

"ssh_private_key_file": "vagrant_rsa",

"ssh_port": 22,

"ssh_wait_timeout": "10000s",

"shutdown_command": "echo '/sbin/halt -h -p' > /tmp/shutdown.sh; echo 'vagrant'|sudo -S sh '/tmp/shutdown.sh'",

"tools_upload_flavor": "linux",

"tools_upload_path": "/tmp/vmware_tools_{{.Flavor}}.iso",

"vmx_data": {

"memsize": "1024",

"numvcpus": "1",

"cpuid.coresPerSocket": "1"

}

"provisioners": [

{

"type": "shell",

"execute_command": "echo 'vagrant'|sudo -S sh '{{.Path}}'",

"override": {

"vmware-iso": {

"scripts": [

"scripts/base.sh",

"scripts/vmware.sh",

"scripts/hgfs.sh",

"scripts/cleanup.sh",

"scripts/zerodisk.sh"

]

}

"post-processors": [

{

"type": "vagrant",

"override": {

"vmware": {

"output": "centos-7.1-x64-vmware.box"

}

]

}

Packer Components/Sections:

Variables:
This is just to centralize variables that will be used by other components (e.g. Builder, provisioner, etc)
I can also set a variable to use Environment variables or command line provided values:
User provided values:
“iso_url”: “{{user iso_url}}”,

Environment variable values:
“iso_url”: “{{env ISO_URL}}”,

Builders:
Here is an array of builders and the needed parameters for each builder, in this case the only builder is of type vmware-iso.
Parameters and some comments about them.

      "headless": true, // Means not to open the VMware Fusion Console
      "type": "vmware-iso", // VMware type builder
      "boot_command": [
        "<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg<enter><wait>"  // Send these key combination to the VMWare Fusion Console
      ],
      "boot_wait": "10s",
      "disk_size": 8192,
      "guest_os_type": "centos-64",
      "http_directory": "http",  // Directory where the kickstart file is placed
      "iso_url": "{{user `iso_url`}}", // Location of ISO image, defined at runtime, more on that later
      "iso_checksum_type": "sha256", 
      "iso_checksum": "{{user `iso_sha256`}}",
      "ssh_username": "vagrant",  // Packer will log in to resulting machine for provisioning, this user must exist (user created from the Kickstart) 
      "ssh_private_key_file": "vagrant_rsa", // Packer will log in using ssh key created earlier
      "ssh_port": 22,
      "ssh_wait_timeout": "10000s",
      "shutdown_command": "echo '/sbin/halt -h -p' > /tmp/shutdown.sh; echo 'vagrant'|sudo -S sh '/tmp/shutdown.sh'",
      "tools_upload_flavor": "linux",
      "tools_upload_path": "/tmp/vmware_tools_{{.Flavor}}.iso", 
      "vmx_data": {
        "memsize": "1024",
        "numvcpus": "1",
        "cpuid.coresPerSocket": "1"

"headless": true, // Means not to open the VMware Fusion Console

"type": "vmware-iso", // VMware type builder

"boot_command": [

"<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg<enter><wait>" // Send these key combination to the VMWare Fusion Console

"boot_wait": "10s",

"disk_size": 8192,

"guest_os_type": "centos-64",

"http_directory": "http", // Directory where the kickstart file is placed

"iso_url": "{{user `iso_url`}}", // Location of ISO image, defined at runtime, more on that later

"iso_checksum_type": "sha256",

"iso_checksum": "{{user `iso_sha256`}}",

"ssh_username": "vagrant", // Packer will log in to resulting machine for provisioning, this user must exist (user created from the Kickstart)

"ssh_private_key_file": "vagrant_rsa", // Packer will log in using ssh key created earlier

"ssh_port": 22,

"ssh_wait_timeout": "10000s",

"shutdown_command": "echo '/sbin/halt -h -p' > /tmp/shutdown.sh; echo 'vagrant'|sudo -S sh '/tmp/shutdown.sh'",

"tools_upload_flavor": "linux",

"tools_upload_path": "/tmp/vmware_tools_{{.Flavor}}.iso",

"vmx_data": {

"memsize": "1024",

"numvcpus": "1",

"cpuid.coresPerSocket": "1"

Provisioners:
This section uses the shell provisioner and runs all of those scripts listed, which are located in the scripts folder.

      "type": "shell",
      "execute_command": "echo 'vagrant'|sudo -S sh '{{.Path}}'",
      "override": {
        "vmware-iso": {
          "scripts": [
            "scripts/base.sh",
            "scripts/vmware.sh",
            "scripts/hgfs.sh",
            "scripts/cleanup.sh",
            "scripts/zerodisk.sh"

"type": "shell",

"execute_command": "echo 'vagrant'|sudo -S sh '{{.Path}}'",

"override": {

"vmware-iso": {

"scripts": [

"scripts/base.sh",

"scripts/vmware.sh",

"scripts/hgfs.sh",

"scripts/cleanup.sh",

"scripts/zerodisk.sh"

Post-processor:
This section states that I want a vagrant box and I am giving it the name.

      "type": "vagrant",
      "override": {
        "vmware": {
          "output": "centos-7.1-x64-vmware.box"

"type": "vagrant",

"override": {

"vmware": {

"output": "centos-7.1-x64-vmware.box"

5) Kickstart file (should be http/ks.cfg based on the template.json file)
This is a minimal install of CentOS 7
Note: The private key (XXXX) given to the vagrant user, that is the key that we created in a previous step

install
cdrom
lang en_US.UTF-8
keyboard us
network --onboot yes --device eth0 --bootproto dhcp --noipv6
rootpw  --plaintext vagrant
firewall --enabled --service=ssh
authconfig --enableshadow --passalgo=sha512
selinux --disabled
timezone --utc America/New_York
bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet"

text
skipx
zerombr

clearpart --all --initlabel
autopart

auth  --useshadow  --enablemd5
firstboot --disabled
reboot

%packages --nobase --ignoremissing 
@core
bzip2
kernel-devel
kernel-headers
-ipw2100-firmware
-ipw2200-firmware
-ivtv-firmware
%end

%post
# Install SUDO
/usr/bin/yum -y install sudo

# Create vagrant user
/usr/sbin/useradd vagrant
/bin/mkdir /home/vagrant/.ssh
/bin/chmod 700 /home/vagrant/.ssh
cat >           /home/vagrant/.ssh/authorized_keys <<'VAGRANT_RSA'
ssh-rsa XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX vagrant
VAGRANT_RSA

/bin/chmod 600 /home/vagrant/.ssh/authorized_keys
/bin/chown -R vagrant /home/vagrant/.ssh

# Add vagrant user to SUDO
echo "vagrant        ALL=(ALL)       NOPASSWD: ALL" >> /etc/sudoers.d/vagrant
echo "Defaults:vagrant !requiretty"                 >> /etc/sudoers.d/vagrant
chmod 0440 /etc/sudoers.d/vagrant
%end

install

cdrom

lang en_US.UTF-8

keyboard us

network --onboot yes --device eth0 --bootproto dhcp --noipv6

rootpw --plaintext vagrant

firewall --enabled --service=ssh

authconfig --enableshadow --passalgo=sha512

selinux --disabled

timezone --utc America/New_York

bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet"

text

skipx

zerombr

clearpart --all --initlabel

autopart

auth --useshadow --enablemd5

firstboot --disabled

reboot

%packages --nobase --ignoremissing

@core

bzip2

kernel-devel

kernel-headers

-ipw2100-firmware

-ipw2200-firmware

-ivtv-firmware

%end

%post

# Install SUDO

/usr/bin/yum -y install sudo

# Create vagrant user

/usr/sbin/useradd vagrant

/bin/mkdir /home/vagrant/.ssh

/bin/chmod 700 /home/vagrant/.ssh

cat > /home/vagrant/.ssh/authorized_keys <<'VAGRANT_RSA'

ssh-rsa XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX vagrant

VAGRANT_RSA

/bin/chmod 600 /home/vagrant/.ssh/authorized_keys

/bin/chown -R vagrant /home/vagrant/.ssh

# Add vagrant user to SUDO

echo "vagrant ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers.d/vagrant

echo "Defaults:vagrant !requiretty" >> /etc/sudoers.d/vagrant

chmod 0440 /etc/sudoers.d/vagrant

%end

6 ) Provisioning Scrips
Packer calls these scripts (from the JSON template) on the created machine before making it a template/image.

base.sh // Install basic packages

#!/usr/bin/env bash
set -x

sed -i "s/^.*requiretty/#Defaults requiretty/" /etc/sudoers
yum -y install gcc make gcc-c++ kernel-devel-`uname -r` perl

#!/usr/bin/env bash

set -x

sed -i "s/^.*requiretty/#Defaults requiretty/" /etc/sudoers

yum -y install gcc make gcc-c++ kernel-devel-`uname -r` perl

vmware.sh // Install VMTools, I separated this from base as I may want to call different scripts on a VirtualBox/etc builder.

#!/usr/bin/env bash
set -x

yum install -y fuse-libs open-vm-tools

#!/usr/bin/env bash

set -x

yum install -y fuse-libs open-vm-tools

hgfs.sh // For vagrant to work properly with folders

#!/usr/bin/env bash
set -x

VMWARE_ISO=/tmp/vmware_tools_linux.iso
VMWARE_MNTDIR=$(mktemp --tmpdir=/tmp -q -d -t vmware_mnt_XXXXXX)
VMWARE_TMPDIR=$(mktemp --tmpdir=/tmp -q -d -t vmware_XXXXXX)

# Extract tools
mount -o loop $VMWARE_ISO $VMWARE_MNTDIR
tar zxf $VMWARE_MNTDIR/VMwareTools*.tar.gz -C $VMWARE_TMPDIR
umount $VMWARE_MNTDIR

# Install tools
$VMWARE_TMPDIR/vmware-tools-distrib/vmware-install.pl -d

# Clean up
rm -f $VMWARE_ISO
rm -rf $VMWARE_MNTDIR
rm -rf $VMWARE_TMPDIR

#!/usr/bin/env bash

set -x

VMWARE_ISO=/tmp/vmware_tools_linux.iso

VMWARE_MNTDIR=$(mktemp --tmpdir=/tmp -q -d -t vmware_mnt_XXXXXX)

VMWARE_TMPDIR=$(mktemp --tmpdir=/tmp -q -d -t vmware_XXXXXX)

# Extract tools

mount -o loop $VMWARE_ISO $VMWARE_MNTDIR

tar zxf $VMWARE_MNTDIR/VMwareTools*.tar.gz -C $VMWARE_TMPDIR

umount $VMWARE_MNTDIR

# Install tools

$VMWARE_TMPDIR/vmware-tools-distrib/vmware-install.pl -d

# Clean up

rm -f $VMWARE_ISO

rm -rf $VMWARE_MNTDIR

rm -rf $VMWARE_TMPDIR

cleanup.sh // Clean up before converting to image

#!/usr/bin/env bash
set -x

yum -y erase gtk2 libX11 hicolor-icon-theme avahi freetype bitstream-vera-fonts
rpm --rebuilddb
yum -y clean all

#!/usr/bin/env bash

set -x

yum -y erase gtk2 libX11 hicolor-icon-theme avahi freetype bitstream-vera-fonts

rpm --rebuilddb

yum -y clean all

zerodisk.sh // This is so that the resulting box is as small as possible

#!/usr/bin/env bash
set -x

dd if=/dev/zero of=/EMPTY bs=1M
rm -f /EMPTY

#!/usr/bin/env bash

set -x

dd if=/dev/zero of=/EMPTY bs=1M

rm -f /EMPTY

7) Run Packer against the template.json file created earlier and watch it build the machine image

$ packer build -var ‘iso_url=/Users/john/iso/CentOS-7-x86_64-Minimal-1511.iso’ -only=vmware-iso template.json
vmware-iso output will be in this color.

==> vmware-iso: Downloading or copying ISO
vmware-iso: Downloading or copying: file:///Users/john/iso/CentOS-7-x86_64-Minimal-1511.iso
==> vmware-iso: Creating virtual machine disk
==> vmware-iso: Building and writing VMX file
==> vmware-iso: Starting HTTP server on port 8101
==> vmware-iso: Starting virtual machine…
vmware-iso: The VM will be run headless, without a GUI. If you want to
vmware-iso: view the screen of the VM, connect via VNC without a password to
vmware-iso: 127.0.0.1:5989
==> vmware-iso: Waiting 10s for boot…
==> vmware-iso: Connecting to VM via VNC
==> vmware-iso: Typing the boot command over VNC…
==> vmware-iso: Waiting for SSH to become available…
==> vmware-iso: Connected to SSH!
==> vmware-iso: Uploading the ‘linux’ VMware Tools
==> vmware-iso: Provisioning with shell script: scripts/base.sh
vmware-iso: + sed -i ‘s/^.*requiretty/#Defaults requiretty/’ /etc/sudoers
…
…
…
==> vmware-iso: Provisioning with shell script: scripts/zerodisk.sh
vmware-iso: + dd if=/dev/zero of=/EMPTY bs=1M
vmware-iso: dd: error writing ‘/EMPTY’: No space left on device
vmware-iso: 5488+0 records in
vmware-iso: 5487+0 records out
vmware-iso: 5754265600 bytes (5.8 GB) copied, 5.27091 s, 1.1 GB/s
vmware-iso: + rm -f /EMPTY
==> vmware-iso: Gracefully halting virtual machine…
vmware-iso: Waiting for VMware to clean up after itself…
==> vmware-iso: Deleting unnecessary VMware files…
vmware-iso: Deleting: output-vmware-iso/564de113-2fc5-2010-8e5d-63fec92f12f7.vmem
vmware-iso: Deleting: output-vmware-iso/packer-vmware-iso.plist
vmware-iso: Deleting: output-vmware-iso/vmware.log
==> vmware-iso: Cleaning VMX prior to finishing up…
vmware-iso: Unmounting floppy from VMX…
vmware-iso: Detaching ISO from CD-ROM device…
vmware-iso: Disabling VNC server…
==> vmware-iso: Compacting the disk image
==> vmware-iso: Running post-processor: vagrant
==> vmware-iso (vagrant): Creating Vagrant box for ‘vmware’ provider
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s001.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s002.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s003.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.nvram
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmsd
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmx
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmxf
vmware-iso (vagrant): Compressing: Vagrantfile
vmware-iso (vagrant): Compressing: disk-s001.vmdk
vmware-iso (vagrant): Compressing: disk-s002.vmdk
vmware-iso (vagrant): Compressing: disk-s003.vmdk
vmware-iso (vagrant): Compressing: disk.vmdk
vmware-iso (vagrant): Compressing: metadata.json
vmware-iso (vagrant): Compressing: packer-vmware-iso.nvram
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmsd
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmx
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmxf
Build ‘vmware-iso’ finished.

==> Builds finished. The artifacts of successful builds are:
–> vmware-iso: ‘vmware’ provider box: centos-7.1-x64-vmware.box

8) Add box to Vagrant

$ vagrant box add –name centos-7.1-010-x64-vmware.box centos-7.1-x64-vmware.box
==> box: Adding box ‘centos-7.1-010-x64-vmware.box’ (v0) for provider:
box: Downloading: file:///Users/john/packer/centos/centos-7.1-x64-vmware.box
==> box: Successfully added box ‘centos-7.1-010-x64-vmware.box’ (v0) for ‘vmware_desktop’!

The above is all you need to have your Infrastructure be Code that can be versioned and automatically create your images and use them on Vagrant.
The next item will show you how to use Packer to move the same image to your vSphere environment.

9) Sending Image to vSphere
To send the image created by Packer to vSphere you will need to add another post-provisioning entry to the JSON array:

    {
      "type": "vsphere",
      "host": "{{user `vm_host`}}",
      "username": "{{user `vm_user`}}",
      "password": "{{user `vm_pass`}}",
      "datacenter": "{{user `vm_dc`}}",
      "cluster": "{{user `vm_cluster`}}",
      "resource_pool": " ",
      "datastore": "{{user `vm_datastore`}}",
      "vm_folder": "{{user `vm_folder`}}",
      "vm_name": "{{user `vm_name`}}", 
      "vm_network": "{{user `vm_network`}}",
      "insecure" : "true"
    }

{

"type": "vsphere",

"host": "{{user `vm_host`}}",

"username": "{{user `vm_user`}}",

"password": "{{user `vm_pass`}}",

"datacenter": "{{user `vm_dc`}}",

"cluster": "{{user `vm_cluster`}}",

"resource_pool": " ",

"datastore": "{{user `vm_datastore`}}",

"vm_folder": "{{user `vm_folder`}}",

"vm_name": "{{user `vm_name`}}",

"vm_network": "{{user `vm_network`}}",

"insecure" : "true"

}

As you can see I am using variables to make this portable, so that means you have to add variables to the variables section as well, below is the new template to do both vagrant and vSphere image provisioning.

{
  "variables": {
    "vm_name": "centos-7.1-vmware",
    "iso_url": "{{env `ISO_URL`}}",
    "iso_sha256": "f90e4d28fa377669b2db16cbcb451fcb9a89d2460e3645993e30e137ac37d284",
    "vm_host": "{{ user `vm_host` }}",
    "vm_user": "{{ user `vm_user` }}",
    "vm_pass": "{{ env `vm_pass` }}",
    "vm_dc":   "{{ user `vm_dc` }}",
    "vm_cluster": "{{user `vm_cluster`}}",
    "vm_datastore": "{{user `vm_datastore`}}",
    "vm_folder": "{{user `vm_folder`}}",
    "vm_name": "{{user `vm_name`}}", 
    "vm_network": "{{user `vm_network`}}"
  },
  "builders": [
    {
      "headless": true,
      "type": "vmware-iso",
      "boot_command": [
        "<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg<enter><wait>"
      ],
      "boot_wait": "10s",
      "disk_size": 8192,
      "guest_os_type": "centos-64",
      "http_directory": "http",
      "iso_url": "{{user `iso_url`}}",
      "iso_checksum_type": "sha256",
      "iso_checksum": "{{user `iso_sha256`}}",
      "ssh_username": "vagrant",
      "ssh_private_key_file": "vagrant_rsa",
      "ssh_port": 22,
      "ssh_wait_timeout": "10000s",
      "shutdown_command": "echo '/sbin/halt -h -p' > /tmp/shutdown.sh; echo 'vagrant'|sudo -S sh '/tmp/shutdown.sh'",
      "tools_upload_flavor": "linux",
      "tools_upload_path": "/tmp/vmware_tools_{{.Flavor}}.iso",
      "vmx_data": {
        "memsize": "1024",
        "numvcpus": "1",
        "cpuid.coresPerSocket": "1"
      }
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "execute_command": "echo 'vagrant'|sudo -S sh '{{.Path}}'",
      "override": {
        "vmware-iso": {
          "scripts": [
            "scripts/base.sh",
            "scripts/vmware.sh",
            "scripts/hgfs.sh",
            "scripts/cleanup.sh",
            "scripts/zerodisk.sh"
          ]
        }
      }
    }
  ],
  "post-processors": [
    {
      "type": "vagrant",
      "override": {
        "vmware": {
          "output": "centos-7.1-x64-vmware.box"
        }
      }
    },
    {
      "type": "vsphere",
      "host": "{{user `vm_host`}}",
      "username": "{{user `vm_user`}}",
      "password": "{{user `vm_pass`}}",
      "datacenter": "{{user `vm_dc`}}",
      "cluster": "{{user `vm_cluster`}}",
      "resource_pool": " ",
      "datastore": "{{user `vm_datastore`}}",
      "vm_folder": "{{user `vm_folder`}}",
      "vm_name": "{{user `vm_name`}}", 
      "vm_network": "{{user `vm_network`}}",
      "insecure" : "true"
    }
  ]
}

{

"variables": {

"vm_name": "centos-7.1-vmware",

"iso_url": "{{env `ISO_URL`}}",

"iso_sha256": "f90e4d28fa377669b2db16cbcb451fcb9a89d2460e3645993e30e137ac37d284",

"vm_host": "{{ user `vm_host` }}",

"vm_user": "{{ user `vm_user` }}",

"vm_pass": "{{ env `vm_pass` }}",

"vm_dc": "{{ user `vm_dc` }}",

"vm_cluster": "{{user `vm_cluster`}}",

"vm_datastore": "{{user `vm_datastore`}}",

"vm_folder": "{{user `vm_folder`}}",

"vm_name": "{{user `vm_name`}}",

"vm_network": "{{user `vm_network`}}"

"builders": [

{

"headless": true,

"type": "vmware-iso",

"boot_command": [

"<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg<enter><wait>"

"boot_wait": "10s",

"disk_size": 8192,

"guest_os_type": "centos-64",

"http_directory": "http",

"iso_url": "{{user `iso_url`}}",

"iso_checksum_type": "sha256",

"iso_checksum": "{{user `iso_sha256`}}",

"ssh_username": "vagrant",

"ssh_private_key_file": "vagrant_rsa",

"ssh_port": 22,

"ssh_wait_timeout": "10000s",

"shutdown_command": "echo '/sbin/halt -h -p' > /tmp/shutdown.sh; echo 'vagrant'|sudo -S sh '/tmp/shutdown.sh'",

"tools_upload_flavor": "linux",

"tools_upload_path": "/tmp/vmware_tools_{{.Flavor}}.iso",

"vmx_data": {

"memsize": "1024",

"numvcpus": "1",

"cpuid.coresPerSocket": "1"

}

"provisioners": [

{

"type": "shell",

"execute_command": "echo 'vagrant'|sudo -S sh '{{.Path}}'",

"override": {

"vmware-iso": {

"scripts": [

"scripts/base.sh",

"scripts/vmware.sh",

"scripts/hgfs.sh",

"scripts/cleanup.sh",

"scripts/zerodisk.sh"

]

}

"post-processors": [

{

"type": "vagrant",

"override": {

"vmware": {

"output": "centos-7.1-x64-vmware.box"

}

{

"type": "vsphere",

"host": "{{user `vm_host`}}",

"username": "{{user `vm_user`}}",

"password": "{{user `vm_pass`}}",

"datacenter": "{{user `vm_dc`}}",

"cluster": "{{user `vm_cluster`}}",

"resource_pool": " ",

"datastore": "{{user `vm_datastore`}}",

"vm_folder": "{{user `vm_folder`}}",

"vm_name": "{{user `vm_name`}}",

"vm_network": "{{user `vm_network`}}",

"insecure" : "true"

}

]

}

10) Re-Run Packer against the template.json file and watch it build the machine image an

$ packer build \
> -var ‘iso_url=/Users/john/iso/CentOS-7-x86_64-Minimal-1511.iso’ \
> -var ‘vm_host=vc.example.com’ \
> -var ‘vm_user=john@example.com’ \
> -var ‘vm_pass=XXXXXXX’ \
> -var ‘vm_dc=vDC’ \
> -var ‘vm_cluster=Folder/Cluster’ \
> -var ‘vm_datastore=store’ \
> -var ‘vm_folder=Images’ \
> -var ‘vm_name=centos71’ \
> -var ‘vm_network=dvs-net1’ \
> -only=vmware-iso template.json
vmware-iso output will be in this color.

==> vmware-iso: Downloading or copying ISO
vmware-iso: Downloading or copying: file:///Users/john/iso/CentOS-7-x86_64-Minimal-1511.iso
==> vmware-iso: Creating virtual machine disk
…
…
==> vmware-iso: Running post-processor: vagrant
==> vmware-iso (vagrant): Creating Vagrant box for ‘vmware’ provider
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s001.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s002.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s003.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.nvram
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmsd
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmx
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmxf
vmware-iso (vagrant): Compressing: Vagrantfile
vmware-iso (vagrant): Compressing: disk-s001.vmdk
vmware-iso (vagrant): Compressing: disk-s002.vmdk
vmware-iso (vagrant): Compressing: disk-s003.vmdk
vmware-iso (vagrant): Compressing: disk.vmdk
vmware-iso (vagrant): Compressing: metadata.json
vmware-iso (vagrant): Compressing: packer-vmware-iso.nvram
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmsd
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmx
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmxf
==> vmware-iso: Running post-processor: vsphere
vmware-iso (vsphere): Uploading output-vmware-iso/packer-vmware-iso.vmx to vSphere
vmware-iso (vsphere): Opening VMX source: output-vmware-iso/packer-vmware-iso.vmx
vmware-iso (vsphere): Opening VI target: vi://john%40example.com@vc.exmaple.com:443/vDC/host/Cluster/Resources/
vmware-iso (vsphere): Deploying to VI: vi://john%40example.com@vc.example.com:443/vDC/host/Cluster/Resources/
Transfer Completed
vmware-iso (vsphere): Completed successfully
vmware-iso (vsphere):
Build ‘vmware-iso’ finished.

==> Builds finished. The artifacts of successful builds are:
–> vmware-iso: ‘vmware’ provider box: centos-7.1-x64-vmware.box
–> vmware-iso:

Now you should have the same image running on both Vagrant and vSphere.

DevOps, Packer, Vagrant centos, devops, infrastructure, linux, packer, rhel, vagrant, vsphere

Using Python Virtual Environments with Sublime Text

August 21, 2015 John No comments

When developing in Python I enjoy using Sublime Text, especially its ability to run/build right from Sublime by simply pressing COMMAND+B. Meaning that I dont have to go out of the editor to the terminal to run the Python program.

Sample Python Program:

#!/usr/bin/env python

print "Hello World"

#!/usr/bin/env python

print "Hello World"

In sublime, I can program, run the program and see the result, increasing my productivity:

But what if I want to run a Python Virtual Environment and its benefits (see developing-in-python-using-python-virtual-environments).
Once the virtual environment is activated your terminal will use the Python Virtual Environment just fine, but Sublime Text will not, it will continue to use the system-wide Python environment by default.

Example: Having a Python module (e.g. IPy) installed in your virtual environment, but not system-wide, and having the Python virtual environment active:

# Python code:

#!/usr/bin/env python

import IPy

print "Hello World"

#!/usr/bin/env python

import IPy

print "Hello World"

# Running from terminal – (OK):

$ python hello.py
Hello World

1 2	$ python hello.py Hello World

# Running from Sublime – (NOT OK):

To make Sublime Text use the virtual environment you need to create a Sublime project and edit its properties to use the Virtual Env.

# Create Sublime Project
Go to Project -> Save Project As…
// I named mine blog.sublime-project

# Edit project properties by editing the file in sublime to be as follows:
Note:
Remember the name: label as this is how you will use the build with… option
The shell_cmd: label is the command that will run when you build. It is far from perfect as you have to hardcode the Python program in the configuration, but at least works.

{
	"folders":
	[
		{
			"path": "."
		}
	],
	"virtualenv": "env",
	"build_systems": [
	  {
	  "name": "PYTHON_VENV",
	  "shell_cmd": "env/bin/python hello.py"
	  }
 	] 
}

{

"folders":

[

{

"path": "."

}

"virtualenv": "env",

"build_systems": [

{

"name": "PYTHON_VENV",

"shell_cmd": "env/bin/python hello.py"

}

]

}

# Prepare Sublime to use your new build system:
Tools -> Build Systems -> PYTHON_VENV

# Now you can now continue to code and use COMMAND+B to build/run your programs without leaving your Sublime Text editor:

Development, DevOps, Python python, sublime, virtualenv

Developing in Python using Python Virtual Environments

August 10, 2015 John No comments

When developing in Python you most likely will need to install Python modules that will provide some type of functionality, the process is very simple:

Install the Python package manager:

sudo yum install python-pip -y

1	sudo yum install python-pip -y

Install modules, for example ‘yaml’, this is without virtual environments, this is basically installing a Python module system-wide:

sudo pip install pyaml

1	sudo pip install pyaml

The above will install Python modules for the system, but to do so you need ‘root’ privileges (e.g. sudo), and there is a possibility that developers don’t have ‘root’ privileges.
Another important scenario to consider is that installing/updating/modifying system-wide Python modules could affect other existing Python projects in the same system.

A way to overcome the system-wide python dependencies is to work with your own virtual environment, which allows you to install Python modules in a Python virtual environment and without the need for ‘root’ privileges.

What this accomplishes is to have your Python projects have their specific dependencies met without affecting anyone else.

To develop using Python virtual environments, the system should have the virtualenv Python package first:

[vagrant@vagrant-box ~]$  sudo yum install python-virtualenv

1	[vagrant@vagrant-box ~]$ sudo yum install python-virtualenv

Now as a regular user (no root or sudo needed):
# Create (or cd into) your Python project folder

$ mkdir my_python_project
$ cd my_python_project

1 2	$ mkdir my_python_project $ cd my_python_project

# Create Python Virtual Environment, you can name it what you want, I chose to name it ‘env’

[vagrant@vagrant-box my_python_project]$ virtualenv env
New python executable in env/bin/python
Installing Setuptools..............................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.

[vagrant@vagrant-box my_python_project]$ virtualenv env

New python executable in env/bin/python

Installing Setuptools..............................................................................................................................................................................................................................done.

Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.

# Activating the virtual environment
To use the virtual environment (as opposed to the system-wide Python environment) you need to activate it first. After it is activated you will see it in the left your command prompt.

# System-wide Python

[vagrant@vagrant-box my_python_project]$ which python
/usr/local/bin/python

1 2	[vagrant@vagrant-box my_python_project]$ which python /usr/local/bin/python

# Activate Virtual Env

[vagrant@vagrant-box my_python_project]$ source env/bin/activate

1	[vagrant@vagrant-box my_python_project]$ source env/bin/activate

# Ready to use Python Virtual Env

(env)[vagrant@vagrant-box my_python_project]$ which python
/vagrant/my_python_project/env/bin/python

1 2	(env)[vagrant@vagrant-box my_python_project]$ which python /vagrant/my_python_project/env/bin/python

# Installing modules in the virtual environment

[vagrant@vagrant-box my_python_project]$ pip install pyvmomi
Downloading/unpacking pyvmomi
  Downloading pyvmomi-5.5.0.2014.1.1.tar.gz (198kB): 198kB downloaded
  Running setup.py egg_info for package pyvmomi
Downloading/unpacking requests>=2.3.0 (from pyvmomi)
  Downloading requests-2.5.1.tar.gz (443kB): 443kB downloaded
  Running setup.py egg_info for package requests
Downloading/unpacking six>=1.7.3 (from pyvmomi)
  Downloading six-1.9.0.tar.gz
  Running setup.py egg_info for package six
    no previously-included directories found matching 'documentation/_build'
Installing collected packages: pyvmomi, requests, six
  Running setup.py install for pyvmomi
  Running setup.py install for requests
  Running setup.py install for six
    no previously-included directories found matching 'documentation/_build'
Successfully installed pyvmomi requests six
Cleaning up...

[vagrant@vagrant-box my_python_project]$ pip install pyvmomi

Downloading/unpacking pyvmomi

Downloading pyvmomi-5.5.0.2014.1.1.tar.gz (198kB): 198kB downloaded

Running setup.py egg_info for package pyvmomi

Downloading/unpacking requests>=2.3.0 (from pyvmomi)

Downloading requests-2.5.1.tar.gz (443kB): 443kB downloaded

Running setup.py egg_info for package requests

Downloading/unpacking six>=1.7.3 (from pyvmomi)

Downloading six-1.9.0.tar.gz

Running setup.py egg_info for package six

no previously-included directories found matching 'documentation/_build'

Installing collected packages: pyvmomi, requests, six

Running setup.py install for pyvmomi

Running setup.py install for requests

Running setup.py install for six

no previously-included directories found matching 'documentation/_build'

Successfully installed pyvmomi requests six

Cleaning up...

# Uninstalling modules in the virtual environment

[vagrant@vagrant-box my_python_project]$pip uninstall pyvmomi
Uninstalling pyvmomi:
  /vagrant/my_python_project/env/LICENSE.txt
  /vagrant/my_python_project/env/MANIFEST.in
  /vagrant/my_python_project/env/NOTICE.txt
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVim/__init__.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVim/__init__.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVim/connect.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVim/connect.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Cache.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Cache.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/CoreTypes.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/CoreTypes.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Differ.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Differ.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/DynamicTypeManagerHelper.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/DynamicTypeManagerHelper.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Iso8601.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Iso8601.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/ManagedMethodExecutorHelper.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/ManagedMethodExecutorHelper.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/ServerObjects.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/ServerObjects.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/SoapAdapter.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/SoapAdapter.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/StubAdapterAccessorImpl.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/StubAdapterAccessorImpl.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Version.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Version.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/VmomiSupport.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/VmomiSupport.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/__init__.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/__init__.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/pyVmomiSettings.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/pyVmomiSettings.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyvmomi-5.5.0.2014.1.1-py2.6.egg-info
  /vagrant/my_python_project/env/setup.cfg
  /vagrant/my_python_project/env/setup.py
  /vagrant/my_python_project/env/tox.ini
Proceed (y/n)? y
  Successfully uninstalled pyvmomi

[vagrant@vagrant-box my_python_project]$pip uninstall pyvmomi

Uninstalling pyvmomi:

/vagrant/my_python_project/env/LICENSE.txt

/vagrant/my_python_project/env/MANIFEST.in

/vagrant/my_python_project/env/NOTICE.txt

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVim/__init__.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVim/__init__.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVim/connect.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVim/connect.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Cache.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Cache.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/CoreTypes.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/CoreTypes.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Differ.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Differ.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/DynamicTypeManagerHelper.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/DynamicTypeManagerHelper.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Iso8601.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Iso8601.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/ManagedMethodExecutorHelper.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/ManagedMethodExecutorHelper.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/ServerObjects.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/ServerObjects.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/SoapAdapter.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/SoapAdapter.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/StubAdapterAccessorImpl.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/StubAdapterAccessorImpl.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Version.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Version.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/VmomiSupport.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/VmomiSupport.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/__init__.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/__init__.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/pyVmomiSettings.py

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/pyVmomiSettings.pyc

/vagrant/my_python_project/env/lib/python2.6/site-packages/pyvmomi-5.5.0.2014.1.1-py2.6.egg-info

/vagrant/my_python_project/env/setup.cfg

/vagrant/my_python_project/env/setup.py

/vagrant/my_python_project/env/tox.ini

Proceed (y/n)? y

Successfully uninstalled pyvmomi

# Install multiple python modules from a requirements file
If your project has multiple Python modules required, it is better to create a requirements.txt file with the list of Python modules.

[vagrant@vagrant-box my_python_project]$ cat requirements.txt
argparse
pyaml

[vagrant@vagrant-box my_python_project]$ cat requirements.txt

argparse

pyaml

Then you can install all the listed Python modules in your virtual environment as follows:

[vagrant@vagrant-box my_python_project]$ pip install -r requirements.txt

1	[vagrant@vagrant-box my_python_project]$ pip install -r requirements.txt

# Deactivate virtual environment (go back to using system-wide Python)

[vagrant@vagrant-box my_python_project]$ deactivate

1	[vagrant@vagrant-box my_python_project]$ deactivate

Development, DevOps, Python python, virtualenv

Showing Differences Between two Git Commits

August 3, 2015 John No comments

From time to time I get questions about how to check differences between two Git commits or two branches, I will answer those questions in this post.

How to check differences between two branches:

Example:
you have a dev branch and a master branch. You develop in the dev branch and would like to know if you were to merge the dev changes to master, what will those changes be:
Please note the order of the branch this states differences that will be added to master from dev.

$ git diff master dev

1	$ git diff master dev

How to check differences between two commits:

Example:
When you do git log you will see your different commits over time, you need to check the differences between two commits.

commit eab54d0ec21a1d7e351fee4c67139ada740e7e6b
Author: John
Date: Wed Jul 8 22:18:05 2015 -0400

Add feature 2

commit b722f1650b9fb33e0990beec027c097526c61478
Author: John
Date: Wed Jul 8 22:31:18 2015 -0400

Add feature 1

// Remember the order the first argument is the point in time, and the second item is what is added to the first argument’s commit

$ git diff b722f1650b9fb33e0990beec027c097526c61478  eab54d0ec21a1d7e351fee4c67139ada740e7e6b
diff --git a/test/script.sh b/test/script.sh
index 33f9e16..cc40434 100755
--- a//test/script.sh
+++ b//test/script.sh
@@ -14,7 +14,7 @@ fi

...
-blah1
+blah2
...

$ git diff b722f1650b9fb33e0990beec027c097526c61478 eab54d0ec21a1d7e351fee4c67139ada740e7e6b

diff --git a/test/script.sh b/test/script.sh

index 33f9e16..cc40434 100755

--- a//test/script.sh

+++ b//test/script.sh

@@ -14,7 +14,7 @@ fi

...

-blah1

+blah2

...

How to check differences between the HEAD (current state/latest revision) and a previous commit:

$ git diff HEAD b722f1650b9fb33e0990beec027c097526c61478 
diff --git a/test/script.sh b/test/script.sh
index 33f9e16..cc40434 100755
--- a//test/script.sh
+++ b//test/script.sh
@@ -14,7 +14,7 @@ fi

...
-blah1
+blah2
...

$ git diff HEAD b722f1650b9fb33e0990beec027c097526c61478

diff --git a/test/script.sh b/test/script.sh

index 33f9e16..cc40434 100755

--- a//test/script.sh

+++ b//test/script.sh

@@ -14,7 +14,7 @@ fi

...

-blah1

+blah2

...

Or you can specify the last commit or a number of commits:

// Last commit

$ git diff HEAD~1
diff --git a/test/script.sh b/test/script.sh
index 33f9e16..cc40434 100755
--- a//test/script.sh
+++ b//test/script.sh
@@ -14,7 +14,7 @@ fi

...
-blah1
+blah2
...

$ git diff HEAD~1

diff --git a/test/script.sh b/test/script.sh

index 33f9e16..cc40434 100755

--- a//test/script.sh

+++ b//test/script.sh

@@ -14,7 +14,7 @@ fi

...

-blah1

+blah2

...

//Last 2 commits

$ git diff HEAD~2
diff --git a/test/script2.pp b/test/script2.pp
index 8def87e..be66b71 100644
--- a/test/script2.pp
+++ b/test/script2.pp
@@ -4,11 +4,10 @@

 class test {
-  stage { "first": before => Stage["main"] }

-  class { "repos": stage => "first" }
+  class { "test": stage => "first" }

-  class repos {
+  class test {
...
diff --git a/test/script.sh b/test/script.sh
index 33f9e16..cc40434 100755
--- a//test/script.sh
+++ b//test/script.sh
@@ -14,7 +14,7 @@ fi

...
-blah1
+blah2
...

$ git diff HEAD~2

diff --git a/test/script2.pp b/test/script2.pp

index 8def87e..be66b71 100644

--- a/test/script2.pp

+++ b/test/script2.pp

@@ -4,11 +4,10 @@

class test {

- stage { "first": before => Stage["main"] }

- class { "repos": stage => "first" }

+ class { "test": stage => "first" }

- class repos {

+ class test {

...

diff --git a/test/script.sh b/test/script.sh

index 33f9e16..cc40434 100755

--- a//test/script.sh

+++ b//test/script.sh

@@ -14,7 +14,7 @@ fi

...

-blah1

+blah2

...

How to check what changed since yesterday or a specific date:

git diff @{yesterday}
diff --git a/.gitignore b/.gitignore
index 4327d2e..8fa1531 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,3 @@
-.svn
-.project
+.com_puppetlabs_geppetto_pptp_target/
+.metadata/
+Geppetto.AutoFileSystemLinked/

git diff @{yesterday}

diff --git a/.gitignore b/.gitignore

index 4327d2e..8fa1531 100644

--- a/.gitignore

+++ b/.gitignore

@@ -1,2 +1,3 @@

-.svn

-.project

+.com_puppetlabs_geppetto_pptp_target/

+.metadata/

+Geppetto.AutoFileSystemLinked/

// OR a specific date

git diff @{2015-05-01}
diff --git a/.gitignore b/.gitignore
index 4327d2e..8fa1531 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,3 @@
-.svn
-.project
+.com_puppetlabs_geppetto_pptp_target/
+.metadata/
+Geppetto.AutoFileSystemLinked/

git diff @{2015-05-01}

diff --git a/.gitignore b/.gitignore

index 4327d2e..8fa1531 100644

--- a/.gitignore

+++ b/.gitignore

@@ -1,2 +1,3 @@

-.svn

-.project

+.com_puppetlabs_geppetto_pptp_target/

+.metadata/

+Geppetto.AutoFileSystemLinked/

How to check all changes to a specific file:

$ git log --follow /path/to/file
commit eab54d0ec21a1d7e351fee4c67139ada740e7e6b
Author: John <john@example.org>
Date:   Wed Jul 8 22:18:05 2015 -0400

    Add feature 2

commit b722f1650b9fb33e0990beec027c097526c61478
Author: John <john@example.org>
Date:   Wed Jul 8 22:31:18 2015 -0400

    Add feature 1

$ git log --follow /path/to/file

commit eab54d0ec21a1d7e351fee4c67139ada740e7e6b

Author: John <john@example.org>

Date: Wed Jul 8 22:18:05 2015 -0400

Add feature 2

commit b722f1650b9fb33e0990beec027c097526c61478

Author: John <john@example.org>

Date: Wed Jul 8 22:31:18 2015 -0400

Add feature 1

How to check size differences between two trees

I found this one in stackoverflow and has been very useful:

Reference Source: http://stackoverflow.com/questions/10845051/git-show-total-file-size-difference-between-two-commits/10847242#10847242

“git cat-file -s will output the size in bytes of an object in git. git diff-tree can tell you the differences between one tree and another. Putting this together into a script called git-file-size-diff located somewhere on your PATH will give you the ability to call git file-size-diff . Putting this together we can try something like the following” – Stackoverflow

$ cat /usr/local/bin/git-file-size-diff
#!/bin/sh
. git sh-setup
args=$(git rev-parse --sq "$@")
eval "git diff-tree -r $args" | {
  total=0
  while read A B C D M P
  do
    case $M in
      M) bytes=$(( $(git cat-file -s $D) - $(git cat-file -s $C) )) ;;
      A) bytes=$(git cat-file -s $D) ;;
      D) bytes=-$(git cat-file -s $C) ;;
      *)
        echo >&2 warning: unhandled mode $M in \"$A $B $C $D $M $P\"
        continue
        ;;
    esac
    total=$(( $total + $bytes ))
    printf '%d\t%s\n' $bytes "$P"
  done
  echo total $total Bytes
  echo total $(($total/1000)) KBytes
}

$ cat /usr/local/bin/git-file-size-diff

#!/bin/sh

. git sh-setup

args=$(git rev-parse --sq "$@")

eval "git diff-tree -r $args" | {

total=0

while read A B C D M P

case $M in

M) bytes=$(( $(git cat-file -s $D) - $(git cat-file -s $C) )) ;;

A) bytes=$(git cat-file -s $D) ;;

D) bytes=-$(git cat-file -s $C) ;;

echo >&2 warning: unhandled mode $M in \"$A $B $C $D $M $P\"

continue

;;

esac

total=$(( $total + $bytes ))

printf '%d\t%s\n' $bytes "$P"

done

echo total $total Bytes

echo total $(($total/1000)) KBytes

}

// Using it to check size differences between two branches:

$ git file-size-diff master dev
66	.gitignore
239	modules/test/Modulefile
171	modules/test/README
167	modules/test/files/file.txt
1425	modules/test/manifests/init.pp
total 2068 Bytes
total 2 KBytes

$ git file-size-diff master dev

66 .gitignore

239 modules/test/Modulefile

171 modules/test/README

167 modules/test/files/file.txt

1425 modules/test/manifests/init.pp

total 2068 Bytes

total 2 KBytes

// Using it to check size differences between the local master branch and the remote master branch, this is useful to know how much data will be downloaded when doing a ‘git pull’:

$ git file-size-diff master origin/master
633	modules/test/manifests/config.pp
326	modules/test/manifests/service.pp
60	modules/test/manifests/params.pp
573	modules/test/manifests/install.pp
13	modules/test/manifests/init.pp
86	modules/mod/manifests/params.pp
total 1691 Bytes
total 1 KBytes

$ git file-size-diff master origin/master

633 modules/test/manifests/config.pp

326 modules/test/manifests/service.pp

60 modules/test/manifests/params.pp

573 modules/test/manifests/install.pp

13 modules/test/manifests/init.pp

86 modules/mod/manifests/params.pp

total 1691 Bytes

total 1 KBytes

Development, DevOps development, diff, git, linux

Generate MySQL Password Hash using Python

June 26, 2015 John 1 comment

In this post I am showing how to use generate a MySQL 5 password-hash that can be used to create MySQL GRANTS using a hash instead of a password.

To use a password-hash to create GRANTs:

GRANT ALL ON *.* to user@% identified by PASSWORD '<PASS-HASH>';

1	GRANT ALL ON . to user@% identified by PASSWORD '<PASS-HASH>';

A good use case is the Puppet puppetlabs-mysql module to automate the MySQL environment, You can automate/define USER and GRANT creation by using the code below, but notice that it requires a password-hash instead of a password:

users => {
  'someuser@localhost' => {
    ensure                   => 'present',
    max_connections_per_hour => '0',
    max_queries_per_hour     => '0',
    max_updates_per_hour     => '0',
    max_user_connections     => '0',
    password_hash            => '*F3A2A51A9B0F2BE2468926B4132313728C250DBF',
  },
}

users => {

'someuser@localhost' => {

ensure => 'present',

max_connections_per_hour => '0',

max_queries_per_hour => '0',

max_updates_per_hour => '0',

max_user_connections => '0',

password_hash => '*F3A2A51A9B0F2BE2468926B4132313728C250DBF',

}

OR:

mysql_user { 'root@127.0.0.1':
  ensure                   => 'present',
  max_connections_per_hour => '0',
  max_queries_per_hour     => '0',
  max_updates_per_hour     => '0',
  max_user_connections     => '0',
  password_hash            => '*F3A2A51A9B0F2BE2468926B4132313728C250DBF',
}

mysql_user { 'root@127.0.0.1':

ensure => 'present',

max_connections_per_hour => '0',

max_queries_per_hour => '0',

max_updates_per_hour => '0',

max_user_connections => '0',

password_hash => '*F3A2A51A9B0F2BE2468926B4132313728C250DBF',

}

They recommend using mysql_password() for creating such a hash. But that means you need to have a MySQL server available.
In this post I am writing about getting those hashes using Python, I wrote a program/script to get the password-hash programatically.

The Python program/script can be found at:
https://github.com/parcejohn/mysql_password_hash

Usage

$ ./mysql_password_hash -h
usage: mysql_password_hash [-h] [-p PASSWORD | -r] [-l PASSWORD_LENGTH]

MySQL Password Hash Generator

optional arguments:
  -h, --help            show this help message and exit
  -p PASSWORD, --password PASSWORD
                        Enter a password
  -r, --generate_random
                        Generate a random password
  -l PASSWORD_LENGTH, --password_length PASSWORD

$ ./mysql_password_hash -h

usage: mysql_password_hash [-h] [-p PASSWORD | -r] [-l PASSWORD_LENGTH]

MySQL Password Hash Generator

optional arguments:

-h, --help show this help message and exit

-p PASSWORD, --password PASSWORD

Enter a password

-r, --generate_random

Generate a random password

-l PASSWORD_LENGTH, --password_length PASSWORD

# Using Command line arguments – User provided password (e.g. ‘secret’)

$ mysql_password_hash -p secret
PASSWORD: secret
HASH: *14e65567abdb5135d0cfd9a70b3032c179a49ee7

$ mysql_password_hash -p secret

PASSWORD: secret

HASH: *14e65567abdb5135d0cfd9a70b3032c179a49ee7

# Using Command line arguments – Random password with length=20 (default length=12)

$ mysql_password_hash -r -l 20
PASSWORD: gnlrn96^g18jcblmssa6
HASH: *e3cbe60709e8abe2082c92cc5e72a762d5f18e22

$ mysql_password_hash -r -l 20

PASSWORD: gnlrn96^g18jcblmssa6

HASH: *e3cbe60709e8abe2082c92cc5e72a762d5f18e22

# interactive mode (no arguments)

mysql_password_hash

1	mysql_password_hash

Development, DevOps, Puppet, Python none

Versioning and Update Notifications for your Vagrant boxes

April 18, 2015 John No comments

In a previous post I showed how to create a vagrant box from scratch and add it to Vagrant: Creating a custom Vagrant Box

Traditionally you will place the Vagrant Box in a web server, and then use it in a Vagrantfile as follows:

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "vagrant_rhel6_vanilla"
  config.vm.box_url = "http://vagrant.example.com/vagrant/vagrant_rhel6_vanilla.box"
...

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

config.vm.box = "vagrant_rhel6_vanilla"

config.vm.box_url = "http://vagrant.example.com/vagrant/vagrant_rhel6_vanilla.box"

...

In this post I will show you how to add versioning to your vagrant boxes, so that whenever the base box is changed/updated/patched your end-users/developers will be automatically notified and will be able to get the latest vagrant box with a simple ‘vagrant box update’ command.

1) Create the updated Vagrant Box
(e.g. vagrant_rhel6_vanilla_010.box for version 0.1.0) – you can use the procedure: Creating a custom Vagrant Box

2) Get a checksum of the box – You will need it later for the JSON Box catalog

$ shasum vagrant_rhel6_vanilla_010.box
b42efdb1e789976a45ffc26a321f00a42932f792  vagrant_rhel6_vanilla_010.box

1 2	$ shasum vagrant_rhel6_vanilla_010.box b42efdb1e789976a45ffc26a321f00a42932f792 vagrant_rhel6_vanilla_010.box

$ openssl sha1  vagrant_rhel6_vanilla_010.box
SHA1(vagrant_rhel6_vanilla_010.box)= b42efdb1e789976a45ffc26a321f00a42932f792

1 2	$ openssl sha1 vagrant_rhel6_vanilla_010.box SHA1(vagrant_rhel6_vanilla_010.box)= b42efdb1e789976a45ffc26a321f00a42932f792

3) Upload the Box to the Vagrant Box web server repository:

I recommend to use the following web directory structure:
../vagrant/boxes serves the actual boxes/VMs
../vagrant serves the Box metadata JSON file (I explain what this is below)

vagrant/
|-- boxes
|   |-- vagrant_rhel6_vanilla_010.box
`-- vagrant_rhel6_vanilla.json

vagrant/

|-- boxes

| |-- vagrant_rhel6_vanilla_010.box

`-- vagrant_rhel6_vanilla.json

Also, ensure the web server returns json as the Content-Type.
For Apache you need to check/add that the following configuration is present:
AddType application/json .json

To verify if your webserver is returning the correct Content-Type:

$ curl -sI  http://vagrant.example.com/vagrant/vagrant_rhel6_vanilla.json | grep Content-Type
Content-Type: application/json

1 2	$ curl -sI http://vagrant.example.com/vagrant/vagrant_rhel6_vanilla.json \| grep Content-Type Content-Type: application/json

4) Create a Box catalog to be served by the web server:

The box catalog is a JSON metadata file allows you to server multiple versions of a box and enable update status notification.
Fields:
name: Name of the Vagrant Box, this is what you will put under config.vm.box on the Vagrantfile
Versions: This is an array/list/collection where you will put the different versions you are hosting/serving
url: This is the actual location of the box, this is what you will put under config.vm.box_url on the Vagrantfile

$ cat vagrant_rhel6_vanilla.json

{
  "name": "vagrant_rhel6_vanilla",
  "description": "This box contains RHEL 64-bit.",
  "versions": [
    {
      "version": "0.1.0",
      "providers": [
        {
          "name": "vmware_desktop",
          "url": "http://vagrant.example.com/vagrant/boxes/vagrant_rhel6_vanilla_010.box",
          "checksum_type": "sha1",
          "checksum": "b42efdb1e789976a45ffc26a321f00a42932f792"
        }
      ]
    }
  ]
}

$ cat vagrant_rhel6_vanilla.json

{

"name": "vagrant_rhel6_vanilla",

"description": "This box contains RHEL 64-bit.",

"versions": [

{

"version": "0.1.0",

"providers": [

{

"name": "vmware_desktop",

"url": "http://vagrant.example.com/vagrant/boxes/vagrant_rhel6_vanilla_010.box",

"checksum_type": "sha1",

"checksum": "b42efdb1e789976a45ffc26a321f00a42932f792"

}

]

}

]

}

5) Modify the Vagrantfile to use the json box catalog file, instead of the actual box name

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "vagrant_rhel6_vanilla"
  config.vm.box_url = "http://vagrant.example.com/vagrant/vagrant_rhel6_vanilla.json"
...

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

config.vm.box = "vagrant_rhel6_vanilla"

config.vm.box_url = "http://vagrant.example.com/vagrant/vagrant_rhel6_vanilla.json"

...

The first time you move from traditional box to versioned box, you need to delete the traditional box that is cached on your system.
This is only needed if you have a traditional box cached.

$ vagrant box list
vagrant_rhel6_vanilla      (vmware_desktop, 0)  <font color='red'><-- Traditional Box</font>

$ vagrant box remove vagrant_rhel6_vanilla
Removing box 'vagrant_rhel6_vanilla' (v0) with provider 'vmware_desktop'...

$ vagrant box list

vagrant_rhel6_vanilla (vmware_desktop, 0) <-- Traditional Box

$ vagrant box remove vagrant_rhel6_vanilla

Removing box 'vagrant_rhel6_vanilla' (v0) with provider 'vmware_desktop'...

6) Vagrant up and see Vagrant download the versioned Box

$ vagrant up
Bringing machine 'default' up with 'vmware_fusion' provider...
==> default: Box 'vagrant_rhel6_vanilla' could not be found. Attempting to find and install...
    default: Box Provider: vmware_desktop, vmware_fusion, vmware_workstation
    default: Box Version: >= 0
<font color='red'>==> default: Loading metadata for box 'http://vagrant.example.com/vagrant/vagrant_rhel6_vanilla.json'
    default: URL: http://vagrant.example.com/vagrant/vagrant_rhel6_vanilla.json
==> default: Adding box 'vagrant_rhel6_vanilla' (v0.1.0) for provider: vmware_desktop
    default: Downloading: http://vagrant.example.com/vagrant/boxes/vagrant_rhel6_vanilla_010.box
    default: Calculating and comparing box checksum...</font>
==> default: Successfully added box 'vagrant_rhel6_vanilla' (v0.1.0) for 'vmware_desktop'!
==> default: Cloning VMware VM: 'vagrant_rhel6_vanilla'. This can take some time...
==> default: Checking if box 'vagrant_rhel6_vanilla' is up to date...
==> default: Verifying vmnet devices are healthy...
...

$ vagrant up

Bringing machine 'default' up with 'vmware_fusion' provider...

==> default: Box 'vagrant_rhel6_vanilla' could not be found. Attempting to find and install...

default: Box Provider: vmware_desktop, vmware_fusion, vmware_workstation

default: Box Version: >= 0

==> default: Loading metadata for box 'http://vagrant.example.com/vagrant/vagrant_rhel6_vanilla.json'

default: URL: http://vagrant.example.com/vagrant/vagrant_rhel6_vanilla.json

==> default: Adding box 'vagrant_rhel6_vanilla' (v0.1.0) for provider: vmware_desktop

default: Downloading: http://vagrant.example.com/vagrant/boxes/vagrant_rhel6_vanilla_010.box

default: Calculating and comparing box checksum...

==> default: Successfully added box 'vagrant_rhel6_vanilla' (v0.1.0) for 'vmware_desktop'!

==> default: Cloning VMware VM: 'vagrant_rhel6_vanilla'. This can take some time...

==> default: Checking if box 'vagrant_rhel6_vanilla' is up to date...

==> default: Verifying vmnet devices are healthy...

...

7) Verify cached Vagrant Box is the correct version

$ vagrant box list
vagrant_rhel6_vanilla      (vmware_desktop, 0.1.0)

1 2	$ vagrant box list vagrant_rhel6_vanilla (vmware_desktop, 0.1.0)

At this point you have a Vagrant environment that supports versions and update status notifications, let’s create a new version of the Box and see it in action.

8) The Vagrant administrator creates an updated box
(e.g. vagrant_rhel6_vanilla_010.box for version 0.1.1) – you can use the procedure: Creating a custom Vagrant Box

9) The Vagrant administrator places the Vagrant Box in the web server

vagrant/
|-- boxes
|   |-- vagrant_rhel6_vanilla_010.box
|   `-- vagrant_rhel6_vanilla_011.box  <font color='red'><-- New Vagrant box</font>
`-- vagrant_rhel6_vanilla.json

vagrant/

|-- boxes

| |-- vagrant_rhel6_vanilla_010.box

| `-- vagrant_rhel6_vanilla_011.box <-- New Vagrant box

`-- vagrant_rhel6_vanilla.json

10) The Vagrant administrator adds the new Vagrant Box to the Box manifest Catalog

$ cat vagrant_rhel6_vanilla.json

{
  "name": "vagrant_rhel6_vanilla",
  "description": "This box contains RHEL 64-bit.",
  "versions": [
    {
      "version": "0.1.0",
      "providers": [
        {
          "name": "vmware_desktop",
          "url": "http://vagrant.example.com/vagrant/boxes/vagrant_rhel6_vanilla_010.box",
          "checksum_type": "sha1",
          "checksum": "b42efdb1e789976a45ffc26a321f00a42932f792"
        }
      ]
    },
    {
      "version": "0.1.1",
      "providers": [
        {
          "name": "vmware_desktop",
          "url": "http://vagrant.example.com/vagrant/boxes/vagrant_rhel6_vanilla_011.box",
          "checksum_type": "sha1",
          "checksum": "8ca8207afced5ca4ba387547fe9ef2b79a7d2b5b"
        }
      ]
    }
  ]
}

$ cat vagrant_rhel6_vanilla.json

{

"name": "vagrant_rhel6_vanilla",

"description": "This box contains RHEL 64-bit.",

"versions": [

{

"version": "0.1.0",

"providers": [

{

"name": "vmware_desktop",

"url": "http://vagrant.example.com/vagrant/boxes/vagrant_rhel6_vanilla_010.box",

"checksum_type": "sha1",

"checksum": "b42efdb1e789976a45ffc26a321f00a42932f792"

}

]

{

"version": "0.1.1",

"providers": [

{

"name": "vmware_desktop",

"url": "http://vagrant.example.com/vagrant/boxes/vagrant_rhel6_vanilla_011.box",

"checksum_type": "sha1",

"checksum": "8ca8207afced5ca4ba387547fe9ef2b79a7d2b5b"

}

]

}

]

}

That’s it! from a Vagrant administrator perspective

11) End user/developer experience: User gets notified on the next Vagrant up and can easily update:

$ vagrant up
Bringing machine 'default' up with 'vmware_fusion' provider...
<font color='red'>==> default: Checking if box 'vagrant_rhel6_vanilla' is up to date...
==> default: A newer version of the box 'vagrant_rhel6_vanilla' is available! You currently
==> default: have version '0.1.0'. The latest is version '0.1.1'. Run
==> default: `vagrant box update` to update.</font>
==> default: Verifying vmnet devices are healthy...
...

$ vagrant up

Bringing machine 'default' up with 'vmware_fusion' provider...

==> default: Checking if box 'vagrant_rhel6_vanilla' is up to date...

==> default: A newer version of the box 'vagrant_rhel6_vanilla' is available! You currently

==> default: have version '0.1.0'. The latest is version '0.1.1'. Run

==> default: `vagrant box update` to update.

==> default: Verifying vmnet devices are healthy...

...

Everytime there is an update the end user gets notified and he/she can upgrade by simply:
vagrant box update

$ vagrant box update
==> default: Checking for updates to 'vagrant_rhel6_vanilla'
    default: Latest installed version: 0.1.0
    default: Version constraints:
    default: Provider: vmware_desktop
<font color='red'>==> default: Updating 'vagrant_rhel6_vanilla' with provider 'vmware_desktop' from version
==> default: '0.1.0' to '0.1.1'...
==> default: Loading metadata for box 'http://vagrant.example.com/vagrant/vagrant_rhel6_vanilla.json'
==> default: Adding box 'vagrant_rhel6_vanilla' (v0.1.1) for provider: vmware_desktop
    default: Downloading: http://vagrant.example.com/vagrant/boxes/vagrant_rhel6_vanilla_011.box
    default: Calculating and comparing box checksum...</font>
==> default: Successfully added box 'vagrant_rhel6_vanilla' (v0.1.1) for 'vmware_desktop'!

$ vagrant box update

==> default: Checking for updates to 'vagrant_rhel6_vanilla'

default: Latest installed version: 0.1.0

default: Version constraints:

default: Provider: vmware_desktop

==> default: Updating 'vagrant_rhel6_vanilla' with provider 'vmware_desktop' from version

==> default: '0.1.0' to '0.1.1'...

==> default: Loading metadata for box 'http://vagrant.example.com/vagrant/vagrant_rhel6_vanilla.json'

==> default: Adding box 'vagrant_rhel6_vanilla' (v0.1.1) for provider: vmware_desktop

default: Downloading: http://vagrant.example.com/vagrant/boxes/vagrant_rhel6_vanilla_011.box

default: Calculating and comparing box checksum...

==> default: Successfully added box 'vagrant_rhel6_vanilla' (v0.1.1) for 'vmware_desktop'!

To verify, check cached Vagrant Boxes:

$ vagrant box list
vagrant_rhel6_vanilla      (vmware_desktop, 0.1.0)
vagrant_rhel6_vanilla      (vmware_desktop, 0.1.1) <font color='red'><-- New Vagrant Box will be used</font>

$ vagrant box list

vagrant_rhel6_vanilla (vmware_desktop, 0.1.0)

vagrant_rhel6_vanilla (vmware_desktop, 0.1.1) <-- New Vagrant Box will be used

To delete older cached Vagrant Box no longer needed:

$ vagrant box remove vagrant_rhel6_vanilla --box-version '0.1.0'
Removing box 'vagrant_rhel6_vanilla' (v0.1.0) with provider 'vmware_desktop'...

1 2	$ vagrant box remove vagrant_rhel6_vanilla --box-version '0.1.0' Removing box 'vagrant_rhel6_vanilla' (v0.1.0) with provider 'vmware_desktop'...

This improves the user/developer experience because they just need to do ‘vagrant box update‘ anytime there is an updated box from the Vagrant administrator.

DevOps, Vagrant none

Remove objects from Git, but keep them in the filesystem

October 9, 2014 John No comments

Recently while working on a project I found that we were keeping .svn metadata in our Git repo.

It was not desirable to keep that in the Git repo but we needed to keep those .svn object in the local filesystem.

The below steps allowed me to do that:

1) Find all the .svn objects

find . -name .svn | xargs -n1

1	find . -name .svn \| xargs -n1

2) Remove them from Git
The (–cache) flag keeps them on disk and (-r) does it recursively

find . -name .svn | xargs -n1 echo git rm --cache -r

1	find . -name .svn \| xargs -n1 echo git rm --cache -r

3) Add a .gitignore file with contents below to prevent tracking of .svn objects

.svn

.svn

4) Commit changes

git commit -m "Delete .svn metadata from project"

1	git commit -m "Delete .svn metadata from project"

5)Verify

git status
git diff

1 2	git status git diff

Development, DevOps none

Older Entries

Technologist

Browsing Posts in DevOps

Installing Puppet Enterprise on AWS EC2 using Vagrant

Pre-requisites:

Create Vagrantfile

Create servers.yaml

Docker Registry with Authentication and SSL/TLS

Create project structure

Password file to support Basic Auth

Certificates to support TLS

Location to store the images

Create Docker Registry container

Using the Registry

Registry API

AWS EC2 automation using AWS CLI and User-Data

Infrastructure As Code: Create Linux (RHEL/CentOS) base Images Using Packer

Using Python Virtual Environments with Sublime Text

Developing in Python using Python Virtual Environments

Showing Differences Between two Git Commits

Generate MySQL Password Hash using Python

Versioning and Update Notifications for your Vagrant boxes

Remove objects from Git, but keep them in the filesystem

Recent Posts

Browse by Tags

Categories

Meta

Related Links