Technologist

Tech stuff about Cloud, DevOps, SysAdmin, Virtualization, SAN, Hardware, Scripting, Automation and Development

Browsing Posts in DevOps

In this guide I go over how to use Vagrant with AWS and in the process have an automated way to install Puppet Enterprise.
I am separating data and code by having a generic Vagrantfile with the code and have a servers.yaml file with all the data that will change from user to user.
For installing the Puppet Enterprise server I am including the automated provisioning script I am using with Vagrant and using AWS Tags to set the hostname of the launched server.

Pre-requisites:

  • Vagrant
  • vagrant-aws plug-in:
  • AWS pre-requisites:
  • While you can add your AWS credentials to the Vagrantfile, it is not recommended. A better way is to have the AWS CLI tools installed and configured

    TL;DR To get started right away you can download the project from github vagrant-aws-puppetserver, otherwise follow the guide below.

    Create Vagrantfile

    The below Vagrantfile utilizes a yaml file (servers.yaml) to provide the data, it allows you to control data using the yaml file and not have to modify the Vagrantfile code – separating code and data.

    // Vagrantfile

    Create servers.yaml

    This file contains the information that will be used by the Vagrantfile, this includes
    AWS region: Which region will this EC2 server run
    AWS keypair: Key used to connect to your launched EC2 instance
    AWS subnet id: Where will this EC2 instance sit in the AWS network
    AWS associate public ip: Do you need a public IP? true or false
    AWS security group: What AWS security group should be associated, should allow Puppetserver needed ports and whatever else you need (ssh, etc)
    AWS ami: Which AMI will you be using I am using a CentOS7
    AWS instance type: Puppetserver needs enough CPU/RAM, during my testing m3.xlarge was appropriate
    AWS SSH username: The EC2 instance user (depends on which AMI you choose), the CentOS AMI expects ec2-user
    AWS SSH private key path: The local path to the SSH key pair
    AWS User Data: I am adding user data which will execute a bash script that allows Vagrant to interact with the launched EC2 instance
    AWS Tags: This is not required for Vagrant and AWS/EC2, but in my provision script I am using the AWS Name Tag to be the system’s hostname, the other 2 tags are there for demonstration purposes
    provision: This is a provisioning script that will be run on the EC2 instance – this is the script that install the Puppet Enterprise server
    AWS IAM Role: You don’t need to add a role when working with Vagrant and AWS/EC2, but I am using a specific IAM role to allow the launched EC2 instance to be able to get information about its AWS Tags, so it is important that you provide it with a Role that allows DescribeTags, see below IAM policy:

    Now use your data in the servers.yaml file
    // servers.yaml

    At this point you can spin up EC2 instances using the above Vagrantfile and servers.yaml file. If you add provision: install_puppetserver.sh to the servers.yaml file as I did and add the below script you will have a Puppet Enterprise server ready to go.

    // install_puppetserver.sh

In this post I will go over the installation of Docker Registry with Basic Authentication and over SSL/TLS (self-signed cert for demonstration purposes). I will also do this on Docker for Mac, which has some interesting things to note.

Let me start by saying that you can use the Docker Hub which is a freely available Hosted registry, but there may be instances where you want to host your own registry to control where your images are being stored, you can also opt to use the commercial Docker Trusted Registry, but I will go over the free open-source Registry.

Create project structure
Password file to support Basic Auth
Certificates to support TLS
Location to store the images
Create Docker Registry container
Using the Registry
Registry API

Create project structure

Password file to support Basic Auth

Let’s create a username and password

Certificates to support TLS

Create a root CA key, which will sign the actual certificates

Create root CA certificate which will need to be installed on the systems that will use the Registry

Now that you have a CA, you can sign certificates.

Create a key for the Registry system

Create a Certificate Signing Request (CSR) for the CA to sign

Finally have the root CA sign the CSR

Location to store the images

The tree structure should look like below, having the data folder be the image repository

Create Docker Registry container

Run your container using docker command

Or even better, use docker-compose:

Start up container based on docker-compose.yml

See it running

At this point the Docker Registry is running

Using the Registry

Download an image from Docker Hub (or you can test with your own images built from Dockerfile)

Tag the image to point to your new Registry

Push image to your Registry (see failure due to ‘unknow CA authority’)

At this point, you need to add the root CA cert to your trusted certificates.
On the machine that will pull or push to the registry, you will need to install the rootCA.crt CA certificate created earlier:

On the Mac, you are probably using Docker for Mac, which actually runs a small hypervisor xhyve that virtualizes the docker engine. That means we need to place the rootCA.crt CA certificate inside the xhyve VM running the Docker Engine. There is no simple way of doing it and I am using the following method (which must be done everytime you restart the Docker service).

Now let’s try to Push the image to the Registry again

Good, we are getting somewhere, TLS is working, and it now asks for Authentication
Authenticate to the Docker Registry

Now let’s successfully push our image

At this point you can share your Docker image, or incorporate into a development or integration workflow by pulling the image
From any Docker engine system:

Registry API

You can use the Docker Registry API https://docs.docker.com/registry/spec/api/ to interact with your Registry.

For example to see a catalog of your images:

In this post I will go over the installation and usage of the AWS CLI to deploy EC2 machines, also combined with AWS user-data to automate actions/scripts that will run on the EC2 machines at install time.

Install the AWS CLI
You can install the AWS CLI in many ways (e.g. zip file, brew, PIP, etc) for details you can follow http://docs.aws.amazon.com/cli/latest/userguide/installing.html
I will install via the Python package manager (PIP) inside a virtual environment

Create virtual environment

$ virtualenv env
New python executable in env/bin/python2.7
Also creating executable in env/bin/python
Installing setuptools, pip…done.

Activate virtual environment

$ source env/bin/activate

Install the awscli

$ pip install awscli
Collecting awscli
Downloading awscli-1.10.53-py2.py3-none-any.whl (970kB)
100% |################################| 970kB 582kB/s
Collecting botocore==1.4.43 (from awscli)
Downloading botocore-1.4.43-py2.py3-none-any.whl (2.5MB)
100% |################################| 2.5MB 206kB/s
Collecting s3transfer<0.2.0,>=0.1.0 (from awscli)
Downloading s3transfer-0.1.1-py2.py3-none-any.whl (49kB)
100% |################################| 49kB 1.4MB/s
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
100% |################################| 49kB 6.2MB/s
Collecting colorama<=0.3.7,>=0.2.5 (from awscli)
Downloading colorama-0.3.7-py2.py3-none-any.whl
Collecting docutils>=0.10 (from awscli)
Downloading docutils-0.12.tar.gz (1.6MB)
100% |################################| 1.6MB 312kB/s
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.4.43->awscli)
Downloading jmespath-0.9.0-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.4.43->awscli)
Downloading python_dateutil-2.5.3-py2.py3-none-any.whl (201kB)
100% |################################| 204kB 1.9MB/s
Collecting futures<4.0.0,>=2.2.0 (from s3transfer<0.2.0,>=0.1.0->awscli)
Downloading futures-3.0.5-py2-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
Downloading pyasn1-0.1.9-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.4.43->awscli)
Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, pyasn1, futures, python-dateutil, jmespath, docutils, colorama, rsa, s3transfer, botocore, awscli
Running setup.py install for docutils
changing mode of build/scripts-2.7/rst2html.py from 644 to 755
changing mode of build/scripts-2.7/rst2s5.py from 644 to 755

Successfully installed awscli-1.10.53 botocore-1.4.43 colorama-0.3.7 docutils-0.12 futures-3.0.5 jmespath-0.9.0 pyasn1-0.1.9 python-dateutil-2.5.3 rsa-3.4.2 s3transfer-0.1.1 six-1.10.0

Verify installation (and review help documentation)

$ aws help

Configure the AWS CLI to use AWS credentials
You will need to provide an AWS Access Key ID and AWS secreate Access Key, these will map to a AWS user and the privileges this user have. The user can (probably should have) been created from the AWS console.

$ aws configure
AWS Access Key ID [****************XYYY]: XXXXXXXXYYY
AWS Secret Access Key [****************ZOOO]:ZZZZZZZOOO
Default region name [us-east-1]:us-east-1
Default output format [None]: json

The above will create a default configuration and set of credentials under ~/.aws/{config,credentials}

Launching an EC2 Instance from awscli

To connect to the EC2 instance you will need at least a SSH keypair and allow access to SSH into the EC2 instance

Create key pair

$ aws ec2 create-key-pair –key-name john-keypair –query ‘KeyMaterial’ –output text > ~/.ssh/john-keypair.pem

$ chmod 400 ~/.ssh/john-keypair.pem

Verify key existence

$ aws ec2 describe-key-pairs –key-name john-keypair
KEYPAIRS 03:eb:3a:d3:13:ba:d3:e3:03:13:b3:f1:43:83:cc:03:ec:d8:4b:43 john-keypair

Create a Security group that allows ingress SSH (tcp 22)

$ aws ec2 create-security-group –group-name allow_tcp_22 –description “Allow SSH”

sg-b1740cd5

// TAG it, I cannot stress enough how important it is to tag every resource with at least a Name

$ aws ec2 create-tags –resources sg-b1740cd5 –tags Key=Name,Value=allow_tcp_22

Add rules, for example: allow inbound (ingress) tcp port 22 to all (0.0.0.0/0)

$ aws ec2 authorize-security-group-ingress –group-id sg-b1740cd5 –protocol tcp –port 22 –cidr 0.0.0.0/0

Select the AWS AMI (aka template) you will use to clone and create your EC2 instance.

See your images (For example I have two Encrypted Ubuntu images) These are using output=text in ~/.aws/config, just to show you what that looks like as oposed to JSON.

$ aws ec2 describe-images –owners self
IMAGES x86_64 2016-07-30T13:16:39.000Z xen ami-a64403c2 /Encrypted Ubuntu Linux 14.04 (HVM) machine Encrypted Red Hat Enterprise Linux (HVM) False /dev/sda1 ebs simple available hvm
BLOCKDEVICEMAPPINGS /dev/sda1
EBS True True snap-bc158f41 10 gp2
IMAGES x86_64 2016-07-30T13:22:03.000Z xen ami-a74403c3 /Encrypted Ubuntu Linux 16.04 (HVM) machine Encrypted Red Hat Enterprise Linux (HVM) False /dev/sda1 ebs simple available hvm
BLOCKDEVICEMAPPINGS /dev/sda1
EBS True True snap-900aa0a1 10 gp2

Or see available images from Redhat

$ aws ec2 describe-images –owners 309956199498 –filters “Name=architecture,Values=x86_64” | grep RHEL-7.1
IMAGES x86_64 2015-02-26T16:27:33.000Z Provided by Red Hat, Inc. xen ami-a540a5e1 309956199498/RHEL-7.1_HVM_GA-20150225-x86_64-1-Hourly2-GP2 machine RHEL-7.1_HVM_GA-20150225-x86_64-1-Hourly2-GP2 309956199498 True /dev/sda1 ebs available hvm
IMAGES x86_64 2015-08-04T17:22:47.000Z Provided by Red Hat, Inc. xen ami-c1996685 309956199498/RHEL-7.1_HVM-20150803-x86_64-1-Hourly2-GP2 machine RHEL-7.1_HVM-20150803-x86_64-1-Hourly2-GP2309956199498 True /dev/sda1 ebs simple available hvm

Or even the images from Amazon, which are FREE – this is the one I will use

$ aws ec2 describe-images –owners amazon –filters “Name=root-device-type,Values=ebs” “Name=architecture,Values=x86_64” | grep ‘Amazon Linux AMI 2016’

IMAGES x86_64 2016-06-22T08:08:12.000Z Amazon Linux AMI 2016.03.3 x86_64 HVM GP2 xen ami-31490d51 amazon/amzn-ami-hvm-2016.03.3.x86_64-gp2 amazon machine amzn-ami-hvm-2016.03.3.x86_64-gp2 137112412989 True /dev/xvda ebs simple available hvm

Launch/Run the instance based on the chosen AMI (e.g. ami-31490d51)

$ aws ec2 run-instances –image-id ami-31490d51 –count 1 –instance-type m3.medium –key-name john-keypair –security-group-ids sg-b1740cd5

TAG IT

$ aws ec2 create-tags –resources i-e1aeaf54 –tags Key=Name,Value=JohnInstance1

Check its status

$ aws ec2 describe-instance-status –instance-ids i-e1aeaf54

Check its details

$ aws ec2 describe-instances –instance-ids i-e1aeaf54

Retrive from its details specifc information using jq (jq is a lightweight and flexible command-line JSON processor)

$ aws ec2 describe-instances –instance-ids i-e1aeaf54 | jq ‘.Reservations[].Instances[].PublicDnsName’

“ec2-54-183-59-200.us-west-1.compute.amazonaws.com”

Connect to the EC2 instance using your SSH key-pair

Destroy EC2 instance

$ aws ec2 terminate-instances –instance-ids i-e1aeaf54


Create EC2 instance with user data script

Now let’s create an EC2 instance that will be ready to serve a website, we will do that by adding user data content that installs a webserver and adds some content.

Notes:
Amazon EC2 limits the size of user-data to 16KB (This limit applies to the data in raw form, not base64-encoded form.)
You can download a larger script and run it, from S3 for example.
You can run any language that supports the shabang(#!) (e.g. Bash, Python, Ruby, Perl)

//user_data.sh

Launch/run instance with user-data:

$ aws ec2 run-instances –image-id ami-31490d51 –count 1 –instance-type m3.medium –key-name john-keypair –security-group-ids sg-b1740cd5 –user-data file://user_data.sh

TAG it

$ aws ec2 create-tags –resources i-e1aeaf54 –tags Key=Name,Value=JohnInstance

1

Get its public DNS name

$ aws ec2 describe-instances –instance-ids i-e974745c | jq ‘.Reservations[].Instances[].PublicDnsName’

“ec2-52-53-165-15.us-west-1.compute.amazonaws.com”

Connect to it

$ ssh -i ~/.ssh/john-keypair.pem ec2-user@ec2-52-53-165-15.us-west-1.compute.amazonaws.com

Verify that httpd is running and the right content is displayed

[ec2-user@ip-172-31-9-131 ~]$ curl http://localhost

Hello from i-e974745c

Troubleshooting: Check the /var/log/cloud-init-output.log file

[ec2-user@ip-172-31-9-131 ~]$ sudo tail /var/log/cloud-init-output.log

Extra:
To allow access to it from the public you will need to add a security group that allows ports 80 and 443.
You can create one as we did for the SSH access, or you can see if you already have a group.

Look at the descriptions of you security groups (this is why it is important to add good descriptions)
“allow SSH access”
“Allow SSH”
“Allow Web tcp 80 and 443 from 0.0.0.0/0”

Looks like “Allow Web tcp 80 and 443 from 0.0.0.0/0” will do the trick, let’s verify

$ aws ec2 describe-security-groups | jq ‘.SecurityGroups[2]’

Let’s add this security group (sg-a3672ac7) to our EC2 instance (i-e974745c)
// This does not append, you have to specify all groups, in this case I am adding the SSH and WEB security groups to the instance

$ aws ec2 modify-instance-attribute –instance-id i-e974745c –groups sg-b1740cd5 sg-a3672ac7

Verify that the instance has both security groups

$ aws ec2 describe-instances –instance-ids i-e974745c | jq ‘.Reservations[].Instances[].SecurityGroups[]’

You can now verify that the content is publicly available
awscli_userdata1

In a previous post I wrote about how to Create a custom Vagrant Box from scratch.

In this post I will walk through the usage of Packer to automate the creation of a CentOS 7 image that can be used in Vagrant or even vSphere.

Packer is a cool Hashicorp tool that helps you automate the creation of machine images.

You can download the project/git repo at https://github.com/parcejohn/packer-centos7

From their website:
“Packer is easy to use and automates the creation of any type of machine image. It embraces modern configuration management by encouraging you to use automated scripts to install and configure the software within your Packer-made images. Packer brings machine images into the modern age, unlocking untapped potential and opening new opportunities.”

I use Packer to continue the journey to Infrastructure As Code, where even my golden images/templates are automated and source controlled (on Git).

Requirements:

* Packer // I installed on Mac using $ brew install packer
* vagrant // I installed on Mac using $ brew install vagrant
* vmware fusion // I installed using $ brew cask install vmware-fusion
* CentOS 7 ISO file

Packer uses a JSON template file to orchestrate the image creation and there are different stages in the process, I will concentrate on the below three, but you should familiarize with all of them (https://www.packer.io/docs/basics/terminology.html)

Builders: Packer component to create a machine image for a single platform, in this case I will be using the VMware builder.
Provisioners: Packer component that installs and configures software within a running machine prior to that machine being turned into a static image. Example provisioners include shell scripts, Chef, Puppet, etc. I will be using the shell provisioner.
Post-Processors: Packer component that takes the result of the builder or another post-processor and process that to create a new artifact. Examples of post-processors are compress to compress artifacts, upload to upload artifacts, etc. I will be creating a Vagrant box as the artifact, and also demonstrate how to upload the image to vSphere.

It’s time to walk through the process of creating a CentOS 7 image using Packer.

1) Create directory structure
The http folder will host a kickstart file, Packer will use its built in web server to serve this kickstart file
The scripts folder will host the provisioning scripts that will define the machine

centos/
├── http
└── scripts

2) Populate with the following configuration files and scripts (the contents of these files are shown in the next steps)

centos/
├── centos-7.1-x64-vmware.json
├── http
│   └── ks.cfg
├── scripts
│   ├── base.sh
│   ├── cleanup.sh
│   ├── hgfs.sh
│   ├── vmware.sh
│   └── zerodisk.sh
├── template.json
└── vagrant_rsa_key

3) Create a SSH key pair, you will need this to login to the server to complete install and configuration

The Public Key will be injected to the ‘vagrant’ user as part of the kickstart, it will then be used in the Packer JSON template to allow Packer to login to the machine
The Private Key will stay on your system to allow Packer and Vagrant to login to the created machine.

4) Create a Packer JSON Template file (template.json)

Packer Components/Sections:

Variables:
This is just to centralize variables that will be used by other components (e.g. Builder, provisioner, etc)
I can also set a variable to use Environment variables or command line provided values:
User provided values:
“iso_url”: “{{user iso_url}}”,

Environment variable values:
“iso_url”: “{{env ISO_URL}}”,

Builders:
Here is an array of builders and the needed parameters for each builder, in this case the only builder is of type vmware-iso.
Parameters and some comments about them.

Provisioners:
This section uses the shell provisioner and runs all of those scripts listed, which are located in the scripts folder.

Post-processor:
This section states that I want a vagrant box and I am giving it the name.

5) Kickstart file (should be http/ks.cfg based on the template.json file)
This is a minimal install of CentOS 7
Note: The private key (XXXX) given to the vagrant user, that is the key that we created in a previous step

6 ) Provisioning Scrips
Packer calls these scripts (from the JSON template) on the created machine before making it a template/image.

base.sh // Install basic packages

vmware.sh // Install VMTools, I separated this from base as I may want to call different scripts on a VirtualBox/etc builder.

hgfs.sh // For vagrant to work properly with folders

cleanup.sh // Clean up before converting to image

zerodisk.sh // This is so that the resulting box is as small as possible

7) Run Packer against the template.json file created earlier and watch it build the machine image

$ packer build -var ‘iso_url=/Users/john/iso/CentOS-7-x86_64-Minimal-1511.iso’ -only=vmware-iso template.json
vmware-iso output will be in this color.

==> vmware-iso: Downloading or copying ISO
vmware-iso: Downloading or copying: file:///Users/john/iso/CentOS-7-x86_64-Minimal-1511.iso
==> vmware-iso: Creating virtual machine disk
==> vmware-iso: Building and writing VMX file
==> vmware-iso: Starting HTTP server on port 8101
==> vmware-iso: Starting virtual machine…
vmware-iso: The VM will be run headless, without a GUI. If you want to
vmware-iso: view the screen of the VM, connect via VNC without a password to
vmware-iso: 127.0.0.1:5989
==> vmware-iso: Waiting 10s for boot…
==> vmware-iso: Connecting to VM via VNC
==> vmware-iso: Typing the boot command over VNC…
==> vmware-iso: Waiting for SSH to become available…
==> vmware-iso: Connected to SSH!
==> vmware-iso: Uploading the ‘linux’ VMware Tools
==> vmware-iso: Provisioning with shell script: scripts/base.sh
vmware-iso: + sed -i ‘s/^.*requiretty/#Defaults requiretty/’ /etc/sudoers



==> vmware-iso: Provisioning with shell script: scripts/zerodisk.sh
vmware-iso: + dd if=/dev/zero of=/EMPTY bs=1M
vmware-iso: dd: error writing ‘/EMPTY’: No space left on device
vmware-iso: 5488+0 records in
vmware-iso: 5487+0 records out
vmware-iso: 5754265600 bytes (5.8 GB) copied, 5.27091 s, 1.1 GB/s
vmware-iso: + rm -f /EMPTY
==> vmware-iso: Gracefully halting virtual machine…
vmware-iso: Waiting for VMware to clean up after itself…
==> vmware-iso: Deleting unnecessary VMware files…
vmware-iso: Deleting: output-vmware-iso/564de113-2fc5-2010-8e5d-63fec92f12f7.vmem
vmware-iso: Deleting: output-vmware-iso/packer-vmware-iso.plist
vmware-iso: Deleting: output-vmware-iso/vmware.log
==> vmware-iso: Cleaning VMX prior to finishing up…
vmware-iso: Unmounting floppy from VMX…
vmware-iso: Detaching ISO from CD-ROM device…
vmware-iso: Disabling VNC server…
==> vmware-iso: Compacting the disk image
==> vmware-iso: Running post-processor: vagrant
==> vmware-iso (vagrant): Creating Vagrant box for ‘vmware’ provider
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s001.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s002.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s003.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.nvram
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmsd
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmx
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmxf
vmware-iso (vagrant): Compressing: Vagrantfile
vmware-iso (vagrant): Compressing: disk-s001.vmdk
vmware-iso (vagrant): Compressing: disk-s002.vmdk
vmware-iso (vagrant): Compressing: disk-s003.vmdk
vmware-iso (vagrant): Compressing: disk.vmdk
vmware-iso (vagrant): Compressing: metadata.json
vmware-iso (vagrant): Compressing: packer-vmware-iso.nvram
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmsd
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmx
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmxf
Build ‘vmware-iso’ finished.

==> Builds finished. The artifacts of successful builds are:
–> vmware-iso: ‘vmware’ provider box: centos-7.1-x64-vmware.box

8) Add box to Vagrant

$ vagrant box add –name centos-7.1-010-x64-vmware.box centos-7.1-x64-vmware.box
==> box: Adding box ‘centos-7.1-010-x64-vmware.box’ (v0) for provider:
box: Downloading: file:///Users/john/packer/centos/centos-7.1-x64-vmware.box
==> box: Successfully added box ‘centos-7.1-010-x64-vmware.box’ (v0) for ‘vmware_desktop’!

The above is all you need to have your Infrastructure be Code that can be versioned and automatically create your images and use them on Vagrant.
The next item will show you how to use Packer to move the same image to your vSphere environment.

9) Sending Image to vSphere
To send the image created by Packer to vSphere you will need to add another post-provisioning entry to the JSON array:

As you can see I am using variables to make this portable, so that means you have to add variables to the variables section as well, below is the new template to do both vagrant and vSphere image provisioning.

10) Re-Run Packer against the template.json file and watch it build the machine image an

$ packer build \
> -var ‘iso_url=/Users/john/iso/CentOS-7-x86_64-Minimal-1511.iso’ \
> -var ‘vm_host=vc.example.com’ \
> -var ‘vm_user=john@example.com’ \
> -var ‘vm_pass=XXXXXXX’ \
> -var ‘vm_dc=vDC’ \
> -var ‘vm_cluster=Folder/Cluster’ \
> -var ‘vm_datastore=store’ \
> -var ‘vm_folder=Images’ \
> -var ‘vm_name=centos71’ \
> -var ‘vm_network=dvs-net1’ \
> -only=vmware-iso template.json

vmware-iso output will be in this color.

==> vmware-iso: Downloading or copying ISO
vmware-iso: Downloading or copying: file:///Users/john/iso/CentOS-7-x86_64-Minimal-1511.iso
==> vmware-iso: Creating virtual machine disk


==> vmware-iso: Running post-processor: vagrant
==> vmware-iso (vagrant): Creating Vagrant box for ‘vmware’ provider
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s001.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s002.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s003.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.nvram
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmsd
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmx
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmxf
vmware-iso (vagrant): Compressing: Vagrantfile
vmware-iso (vagrant): Compressing: disk-s001.vmdk
vmware-iso (vagrant): Compressing: disk-s002.vmdk
vmware-iso (vagrant): Compressing: disk-s003.vmdk
vmware-iso (vagrant): Compressing: disk.vmdk
vmware-iso (vagrant): Compressing: metadata.json
vmware-iso (vagrant): Compressing: packer-vmware-iso.nvram
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmsd
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmx
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmxf
==> vmware-iso: Running post-processor: vsphere
vmware-iso (vsphere): Uploading output-vmware-iso/packer-vmware-iso.vmx to vSphere
vmware-iso (vsphere): Opening VMX source: output-vmware-iso/packer-vmware-iso.vmx
vmware-iso (vsphere): Opening VI target: vi://john%40example.com@vc.exmaple.com:443/vDC/host/Cluster/Resources/
vmware-iso (vsphere): Deploying to VI: vi://john%40example.com@vc.example.com:443/vDC/host/Cluster/Resources/
Transfer Completed
vmware-iso (vsphere): Completed successfully
vmware-iso (vsphere):
Build ‘vmware-iso’ finished.

==> Builds finished. The artifacts of successful builds are:
–> vmware-iso: ‘vmware’ provider box: centos-7.1-x64-vmware.box
–> vmware-iso:

Now you should have the same image running on both Vagrant and vSphere.

When developing in Python I enjoy using Sublime Text, especially its ability to run/build right from Sublime by simply pressing COMMAND+B. Meaning that I dont have to go out of the editor to the terminal to run the Python program.

Sample Python Program:

In sublime, I can program, run the program and see the result, increasing my productivity:

sublime_venv1

But what if I want to run a Python Virtual Environment and its benefits (see developing-in-python-using-python-virtual-environments).
Once the virtual environment is activated your terminal will use the Python Virtual Environment just fine, but Sublime Text will not, it will continue to use the system-wide Python environment by default.

Example: Having a Python module (e.g. IPy) installed in your virtual environment, but not system-wide, and having the Python virtual environment active:

# Python code:

# Running from terminal – (OK):

# Running from Sublime – (NOT OK):

sublime_venv2

To make Sublime Text use the virtual environment you need to create a Sublime project and edit its properties to use the Virtual Env.

# Create Sublime Project
Go to Project -> Save Project As…
// I named mine blog.sublime-project

# Edit project properties by editing the file in sublime to be as follows:
Note:
Remember the name: label as this is how you will use the build with… option
The shell_cmd: label is the command that will run when you build. It is far from perfect as you have to hardcode the Python program in the configuration, but at least works.

# Prepare Sublime to use your new build system:
Tools -> Build Systems -> PYTHON_VENV

# Now you can now continue to code and use COMMAND+B to build/run your programs without leaving your Sublime Text editor:

sublime_venv3

When developing in Python you most likely will need to install Python modules that will provide some type of functionality, the process is very simple:

Install the Python package manager:

Install modules, for example ‘yaml’, this is without virtual environments, this is basically installing a Python module system-wide:

The above will install Python modules for the system, but to do so you need ‘root’ privileges (e.g. sudo), and there is a possibility that developers don’t have ‘root’ privileges.
Another important scenario to consider is that installing/updating/modifying system-wide Python modules could affect other existing Python projects in the same system.

A way to overcome the system-wide python dependencies is to work with your own virtual environment, which allows you to install Python modules in a Python virtual environment and without the need for ‘root’ privileges.

What this accomplishes is to have your Python projects have their specific dependencies met without affecting anyone else.

To develop using Python virtual environments, the system should have the virtualenv Python package first:

Now as a regular user (no root or sudo needed):
# Create (or cd into) your Python project folder

# Create Python Virtual Environment, you can name it what you want, I chose to name it ‘env’

# Activating the virtual environment
To use the virtual environment (as opposed to the system-wide Python environment) you need to activate it first. After it is activated you will see it in the left your command prompt.

# System-wide Python

# Activate Virtual Env

# Ready to use Python Virtual Env

# Installing modules in the virtual environment

# Uninstalling modules in the virtual environment

# Install multiple python modules from a requirements file
If your project has multiple Python modules required, it is better to create a requirements.txt file with the list of Python modules.

Then you can install all the listed Python modules in your virtual environment as follows:

# Deactivate virtual environment (go back to using system-wide Python)

From time to time I get questions about how to check differences between two Git commits or two branches, I will answer those questions in this post.

How to check differences between two branches:

Example:
you have a dev branch and a master branch. You develop in the dev branch and would like to know if you were to merge the dev changes to master, what will those changes be:
Please note the order of the branch this states differences that will be added to master from dev.

How to check differences between two commits:

Example:
When you do git log you will see your different commits over time, you need to check the differences between two commits.

commit eab54d0ec21a1d7e351fee4c67139ada740e7e6b
Author: John
Date: Wed Jul 8 22:18:05 2015 -0400

Add feature 2

commit b722f1650b9fb33e0990beec027c097526c61478
Author: John
Date: Wed Jul 8 22:31:18 2015 -0400

Add feature 1

// Remember the order the first argument is the point in time, and the second item is what is added to the first argument’s commit

How to check differences between the HEAD (current state/latest revision) and a previous commit:

Or you can specify the last commit or a number of commits:

// Last commit

//Last 2 commits

How to check what changed since yesterday or a specific date:

// OR a specific date

How to check all changes to a specific file:

How to check size differences between two trees

I found this one in stackoverflow and has been very useful:

Reference Source: http://stackoverflow.com/questions/10845051/git-show-total-file-size-difference-between-two-commits/10847242#10847242

“git cat-file -s will output the size in bytes of an object in git. git diff-tree can tell you the differences between one tree and another. Putting this together into a script called git-file-size-diff located somewhere on your PATH will give you the ability to call git file-size-diff . Putting this together we can try something like the following” – Stackoverflow

// Using it to check size differences between two branches:

// Using it to check size differences between the local master branch and the remote master branch, this is useful to know how much data will be downloaded when doing a ‘git pull’:

In this post I am showing how to use generate a MySQL 5 password-hash that can be used to create MySQL GRANTS using a hash instead of a password.

To use a password-hash to create GRANTs:

A good use case is the Puppet puppetlabs-mysql module to automate the MySQL environment, You can automate/define USER and GRANT creation by using the code below, but notice that it requires a password-hash instead of a password:

OR:

They recommend using mysql_password() for creating such a hash. But that means you need to have a MySQL server available.
In this post I am writing about getting those hashes using Python, I wrote a program/script to get the password-hash programatically.

The Python program/script can be found at:
https://github.com/parcejohn/mysql_password_hash

Usage

# Using Command line arguments – User provided password (e.g. ‘secret’)

# Using Command line arguments – Random password with length=20 (default length=12)

# interactive mode (no arguments)

In a previous post I showed how to create a vagrant box from scratch and add it to Vagrant: Creating a custom Vagrant Box

Traditionally you will place the Vagrant Box in a web server, and then use it in a Vagrantfile as follows:

In this post I will show you how to add versioning to your vagrant boxes, so that whenever the base box is changed/updated/patched your end-users/developers will be automatically notified and will be able to get the latest vagrant box with a simple ‘vagrant box update’ command.

1) Create the updated Vagrant Box
(e.g. vagrant_rhel6_vanilla_010.box for version 0.1.0) – you can use the procedure: Creating a custom Vagrant Box

2) Get a checksum of the box – You will need it later for the JSON Box catalog

OR

3) Upload the Box to the Vagrant Box web server repository:

I recommend to use the following web directory structure:
../vagrant/boxes serves the actual boxes/VMs
../vagrant serves the Box metadata JSON file (I explain what this is below)

Also, ensure the web server returns json as the Content-Type.
For Apache you need to check/add that the following configuration is present:
AddType application/json .json

To verify if your webserver is returning the correct Content-Type:

4) Create a Box catalog to be served by the web server:

The box catalog is a JSON metadata file allows you to server multiple versions of a box and enable update status notification.
Fields:
name: Name of the Vagrant Box, this is what you will put under config.vm.box on the Vagrantfile
Versions: This is an array/list/collection where you will put the different versions you are hosting/serving
url: This is the actual location of the box, this is what you will put under config.vm.box_url on the Vagrantfile

5) Modify the Vagrantfile to use the json box catalog file, instead of the actual box name

The first time you move from traditional box to versioned box, you need to delete the traditional box that is cached on your system.
This is only needed if you have a traditional box cached.

6) Vagrant up and see Vagrant download the versioned Box

7) Verify cached Vagrant Box is the correct version

At this point you have a Vagrant environment that supports versions and update status notifications, let’s create a new version of the Box and see it in action.

8) The Vagrant administrator creates an updated box
(e.g. vagrant_rhel6_vanilla_010.box for version 0.1.1) – you can use the procedure: Creating a custom Vagrant Box

9) The Vagrant administrator places the Vagrant Box in the web server

10) The Vagrant administrator adds the new Vagrant Box to the Box manifest Catalog

That’s it! from a Vagrant administrator perspective

11) End user/developer experience: User gets notified on the next Vagrant up and can easily update:

Everytime there is an update the end user gets notified and he/she can upgrade by simply:
vagrant box update

To verify, check cached Vagrant Boxes:

To delete older cached Vagrant Box no longer needed:

This improves the user/developer experience because they just need to do ‘vagrant box update‘ anytime there is an updated box from the Vagrant administrator.

Recently while working on a project I found that we were keeping .svn metadata in our Git repo.

It was not desirable to keep that in the Git repo but we needed to keep those .svn object in the local filesystem.

The below steps allowed me to do that:

1) Find all the .svn objects

2) Remove them from Git
The (–cache) flag keeps them on disk and (-r) does it recursively

3) Add a .gitignore  file with contents below to prevent tracking of .svn objects

4) Commit changes

5)Verify