Technologist

Tech stuff about Cloud, DevOps, SysAdmin, Virtualization, SAN, Hardware, Scripting, Automation and Development

Browsing Posts in Development

In this guide I go over how to use Vagrant with AWS and in the process have an automated way to install Puppet Enterprise.
I am separating data and code by having a generic Vagrantfile with the code and have a servers.yaml file with all the data that will change from user to user.
For installing the Puppet Enterprise server I am including the automated provisioning script I am using with Vagrant and using AWS Tags to set the hostname of the launched server.

Pre-requisites:

  • Vagrant
  • vagrant-aws plug-in:
  • $ vagrant plugin install vagrant-aws
    
  • AWS pre-requisites:
  • While you can add your AWS credentials to the Vagrantfile, it is not recommended. A better way is to have the AWS CLI tools installed and configured

    $ aws configure
    AWS Access Key ID [****************XYYY]: XXXXXXXXYYY
    AWS Secret Access Key [****************ZOOO]:ZZZZZZZOOO
    Default region name [us-east-1]:us-east-1
    Default output format [None]: json
    

    TL;DR To get started right away you can download the project from github vagrant-aws-puppetserver, otherwise follow the guide below.

    Create Vagrantfile

    The below Vagrantfile utilizes a yaml file (servers.yaml) to provide the data, it allows you to control data using the yaml file and not have to modify the Vagrantfile code – separating code and data.

    // Vagrantfile

    # -*- mode: ruby -*-
    # vi: set ft=ruby :
    
    # Specify minimum Vagrant version and Vagrant API version
    Vagrant.require_version ">= 1.6.0"
    VAGRANTFILE_API_VERSION = "2"
    
    # Require YAML module
    require 'yaml'
    
    # Read YAML file with box details
    servers = YAML.load_file('servers.yaml')
    
    Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
    
        # Iterate through entries in YAML file
        servers.each do |server|
        
            config.vm.define server['name'] do |srv|
    
                srv.vm.box = server['box']
                srv.vm.box_url = server['box_url']
    
                srv.vm.provider :aws do |aws, override|
    
                    ### Dont do these here, better to use awscli with a profile
                    #aws.access_key_id = "YOUR KEY"
                    #aws.secret_access_key = "YOUR SECRET KEY"
                    #aws.session_token = "SESSION TOKEN"
    
                    aws.region = server["aws_region"] if server["aws_region"]
                    aws.keypair_name = server["aws_keypair_name"] if server["aws_keypair_name"]
                    aws.subnet_id = server["aws_subnet_id"] if server["aws_subnet_id"]
                    aws.associate_public_ip = server["aws_associate_public_ip"] if server["aws_associate_public_ip"]
                    aws.security_groups = server["aws_security_groups"] if server["aws_security_groups"]
                    aws.iam_instance_profile_name = server['aws_iam_role'] if server['aws_iam_role']
                    aws.ami = server["aws_ami"] if server["aws_ami"]
                    aws.instance_type = server["aws_instance_type"] if server["aws_instance_type"]
                    aws.tags = server["aws_tags"] if server["aws_tags"]
                    aws.user_data = server["aws_user_data"] if server["aws_user_data"]
    
                    override.ssh.username = server["aws_ssh_username"] if server["aws_ssh_username"]
                    override.ssh.private_key_path = server["aws_ssh_private_key_path"] if server["aws_ssh_private_key_path"]           
    
                    config.vm.synced_folder ".", "/vagrant", type: "rsync"
    
                    config.vm.provision :shell, path: server["provision"] if server["provision"]
    
                end  
            end
        end
    end
    

    Create servers.yaml

    This file contains the information that will be used by the Vagrantfile, this includes
    AWS region: Which region will this EC2 server run
    AWS keypair: Key used to connect to your launched EC2 instance
    AWS subnet id: Where will this EC2 instance sit in the AWS network
    AWS associate public ip: Do you need a public IP? true or false
    AWS security group: What AWS security group should be associated, should allow Puppetserver needed ports and whatever else you need (ssh, etc)
    AWS ami: Which AMI will you be using I am using a CentOS7
    AWS instance type: Puppetserver needs enough CPU/RAM, during my testing m3.xlarge was appropriate
    AWS SSH username: The EC2 instance user (depends on which AMI you choose), the CentOS AMI expects ec2-user
    AWS SSH private key path: The local path to the SSH key pair
    AWS User Data: I am adding user data which will execute a bash script that allows Vagrant to interact with the launched EC2 instance
    AWS Tags: This is not required for Vagrant and AWS/EC2, but in my provision script I am using the AWS Name Tag to be the system’s hostname, the other 2 tags are there for demonstration purposes
    provision: This is a provisioning script that will be run on the EC2 instance – this is the script that install the Puppet Enterprise server
    AWS IAM Role: You don’t need to add a role when working with Vagrant and AWS/EC2, but I am using a specific IAM role to allow the launched EC2 instance to be able to get information about its AWS Tags, so it is important that you provide it with a Role that allows DescribeTags, see below IAM policy:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "XXXXXXX",
                "Action": [
                    "ec2:DescribeTags"
                ],
                "Effect": "Allow",
                "Resource": "*"
            }
        ]
    }

    Now use your data in the servers.yaml file
    // servers.yaml

    ---
    - name: puppet4
      box: dummy
      box_url: https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
      aws_region: "us-east-1"
      aws_keypair_name: "john-key"
      aws_subnet_id: "subnet-0001" 
      aws_associate_public_ip: false
      aws_security_groups: ['sg-0001'] 
      aws_ami: "ami-0001"
      aws_instance_type: m3.xlarge
      aws_ssh_username: "ec2-user"
      aws_iam_role: "iam-able-to-describe-tags"
      aws_ssh_private_key_path: "/Users/john/.ssh/john-key.pem"
      aws_user_data: "#!/bin/bash\nsed -i -e 's/^Defaults.*requiretty/# Defaults requiretty/g' /etc/sudoers"
      aws_tags: 
        Name: 'puppet4'
        tier: 'stage'
        application: 'puppetserver'
      update: false
      provision: install_puppetserver.sh
    

    At this point you can spin up EC2 instances using the above Vagrantfile and servers.yaml file. If you add provision: install_puppetserver.sh to the servers.yaml file as I did and add the below script you will have a Puppet Enterprise server ready to go.

    // install_puppetserver.sh

    #!/bin/bash -xe
    
    # Ensure pip is installed
    if ! which pip; then
    	curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
    	python get-pip.py
    fi
    
    # Upgrade pip 
    /bin/pip install --upgrade pip
    
    # Upgrade awscli tools (They are installed in the AMI)
    /bin/pip install --upgrade awscli
    
    # Get hostname from AWS Name Tag (requires the EC2 instance to have an IAM role that allows DescribeTags)
    AWS_INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
    AWS_REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | grep region | cut -d\" -f4)
    AWSHOSTNAME=$(aws ec2 describe-tags --region ${AWS_REGION} --filters "Name=resource-id,Values=${AWS_INSTANCE_ID}" --query "Tags[?Key=='Name'].Value[] | [0]" | cut -d\" -f2)
    
    # Set hostname (use the AWS Name Tag)
    hostnamectl set-hostname ${AWSHOSTNAME}.cpg.org
    
    # Update system and install wget, git
    yum update -y
    yum install wget git -y
    
    # Set puppet.cpg.org as hostname in hosts file
    echo "$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4) puppet puppet.cpg.org" >> /etc/hosts
    
    # Download puppet 
    wget "https://pm.puppetlabs.com/cgi-bin/download.cgi?dist=el&rel=7&arch=x86_64&ver=2016.4.0" -O puppet.2016.4.0.tar.gz
    tar -xvzf puppet.2016.4.0.tar.gz
    cd puppet-enterprise*
    
    # Create pe.conf file
    touch pe.conf
    echo '{' >> pe.conf
    echo '"console_admin_password": "puppet"' >> pe.conf
    echo '"puppet_enterprise::puppet_master_host": "%{::trusted.certname}"' >> pe.conf
    echo '}'  >> pe.conf
    
    echo "Install Puppetserver"
    ./puppet-enterprise-installer -c pe.conf
    
    echo "Adding * to autosign.conf"
    cat >> /etc/puppetlabs/puppet/autosign.conf <<'AUTOSIGN'
    *
    AUTOSIGN
    
    # Run puppet agent
    /usr/local/bin/puppet agent -t
    

When developing in Python I enjoy using Sublime Text, especially its ability to run/build right from Sublime by simply pressing COMMAND+B. Meaning that I dont have to go out of the editor to the terminal to run the Python program.

Sample Python Program:

#!/usr/bin/env python

print "Hello World"

In sublime, I can program, run the program and see the result, increasing my productivity:

sublime_venv1

But what if I want to run a Python Virtual Environment and its benefits (see developing-in-python-using-python-virtual-environments).
Once the virtual environment is activated your terminal will use the Python Virtual Environment just fine, but Sublime Text will not, it will continue to use the system-wide Python environment by default.

Example: Having a Python module (e.g. IPy) installed in your virtual environment, but not system-wide, and having the Python virtual environment active:

# Python code:

#!/usr/bin/env python

import IPy

print "Hello World"

# Running from terminal – (OK):

$ python hello.py
Hello World

# Running from Sublime – (NOT OK):

sublime_venv2

To make Sublime Text use the virtual environment you need to create a Sublime project and edit its properties to use the Virtual Env.

# Create Sublime Project
Go to Project -> Save Project As…
// I named mine blog.sublime-project

# Edit project properties by editing the file in sublime to be as follows:
Note:
Remember the name: label as this is how you will use the build with… option
The shell_cmd: label is the command that will run when you build. It is far from perfect as you have to hardcode the Python program in the configuration, but at least works.

{
	"folders":
	[
		{
			"path": "."
		}
	],
	"virtualenv": "env",
	"build_systems": [
	  {
	  "name": "PYTHON_VENV",
	  "shell_cmd": "env/bin/python hello.py"
	  }
 	] 
}

# Prepare Sublime to use your new build system:
Tools -> Build Systems -> PYTHON_VENV

# Now you can now continue to code and use COMMAND+B to build/run your programs without leaving your Sublime Text editor:

sublime_venv3

When developing in Python you most likely will need to install Python modules that will provide some type of functionality, the process is very simple:

Install the Python package manager:

sudo yum install python-pip -y

Install modules, for example ‘yaml’, this is without virtual environments, this is basically installing a Python module system-wide:

sudo pip install pyaml

The above will install Python modules for the system, but to do so you need ‘root’ privileges (e.g. sudo), and there is a possibility that developers don’t have ‘root’ privileges.
Another important scenario to consider is that installing/updating/modifying system-wide Python modules could affect other existing Python projects in the same system.

A way to overcome the system-wide python dependencies is to work with your own virtual environment, which allows you to install Python modules in a Python virtual environment and without the need for ‘root’ privileges.

What this accomplishes is to have your Python projects have their specific dependencies met without affecting anyone else.

To develop using Python virtual environments, the system should have the virtualenv Python package first:

[vagrant@vagrant-box ~]$  sudo yum install python-virtualenv

Now as a regular user (no root or sudo needed):
# Create (or cd into) your Python project folder

$ mkdir my_python_project
$ cd my_python_project

# Create Python Virtual Environment, you can name it what you want, I chose to name it ‘env’

[vagrant@vagrant-box my_python_project]$ virtualenv env
New python executable in env/bin/python
Installing Setuptools..............................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.

# Activating the virtual environment
To use the virtual environment (as opposed to the system-wide Python environment) you need to activate it first. After it is activated you will see it in the left your command prompt.

# System-wide Python

[vagrant@vagrant-box my_python_project]$ which python
/usr/local/bin/python

# Activate Virtual Env

[vagrant@vagrant-box my_python_project]$ source env/bin/activate

# Ready to use Python Virtual Env

(env)[vagrant@vagrant-box my_python_project]$ which python
/vagrant/my_python_project/env/bin/python

# Installing modules in the virtual environment

[vagrant@vagrant-box my_python_project]$ pip install pyvmomi
Downloading/unpacking pyvmomi
  Downloading pyvmomi-5.5.0.2014.1.1.tar.gz (198kB): 198kB downloaded
  Running setup.py egg_info for package pyvmomi
Downloading/unpacking requests>=2.3.0 (from pyvmomi)
  Downloading requests-2.5.1.tar.gz (443kB): 443kB downloaded
  Running setup.py egg_info for package requests
Downloading/unpacking six>=1.7.3 (from pyvmomi)
  Downloading six-1.9.0.tar.gz
  Running setup.py egg_info for package six
    no previously-included directories found matching 'documentation/_build'
Installing collected packages: pyvmomi, requests, six
  Running setup.py install for pyvmomi
  Running setup.py install for requests
  Running setup.py install for six
    no previously-included directories found matching 'documentation/_build'
Successfully installed pyvmomi requests six
Cleaning up...

# Uninstalling modules in the virtual environment

[vagrant@vagrant-box my_python_project]$pip uninstall pyvmomi
Uninstalling pyvmomi:
  /vagrant/my_python_project/env/LICENSE.txt
  /vagrant/my_python_project/env/MANIFEST.in
  /vagrant/my_python_project/env/NOTICE.txt
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVim/__init__.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVim/__init__.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVim/connect.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVim/connect.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Cache.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Cache.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/CoreTypes.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/CoreTypes.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Differ.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Differ.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/DynamicTypeManagerHelper.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/DynamicTypeManagerHelper.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Iso8601.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Iso8601.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/ManagedMethodExecutorHelper.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/ManagedMethodExecutorHelper.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/ServerObjects.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/ServerObjects.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/SoapAdapter.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/SoapAdapter.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/StubAdapterAccessorImpl.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/StubAdapterAccessorImpl.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Version.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/Version.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/VmomiSupport.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/VmomiSupport.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/__init__.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/__init__.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/pyVmomiSettings.py
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyVmomi/pyVmomiSettings.pyc
  /vagrant/my_python_project/env/lib/python2.6/site-packages/pyvmomi-5.5.0.2014.1.1-py2.6.egg-info
  /vagrant/my_python_project/env/setup.cfg
  /vagrant/my_python_project/env/setup.py
  /vagrant/my_python_project/env/tox.ini
Proceed (y/n)? y
  Successfully uninstalled pyvmomi

# Install multiple python modules from a requirements file
If your project has multiple Python modules required, it is better to create a requirements.txt file with the list of Python modules.

[vagrant@vagrant-box my_python_project]$ cat requirements.txt
argparse
pyaml

Then you can install all the listed Python modules in your virtual environment as follows:

[vagrant@vagrant-box my_python_project]$ pip install -r requirements.txt

# Deactivate virtual environment (go back to using system-wide Python)

[vagrant@vagrant-box my_python_project]$ deactivate

From time to time I get questions about how to check differences between two Git commits or two branches, I will answer those questions in this post.

How to check differences between two branches:

Example:
you have a dev branch and a master branch. You develop in the dev branch and would like to know if you were to merge the dev changes to master, what will those changes be:
Please note the order of the branch this states differences that will be added to master from dev.

$ git diff master dev

How to check differences between two commits:

Example:
When you do git log you will see your different commits over time, you need to check the differences between two commits.

commit eab54d0ec21a1d7e351fee4c67139ada740e7e6b
Author: John
Date: Wed Jul 8 22:18:05 2015 -0400

Add feature 2

commit b722f1650b9fb33e0990beec027c097526c61478
Author: John
Date: Wed Jul 8 22:31:18 2015 -0400

Add feature 1

// Remember the order the first argument is the point in time, and the second item is what is added to the first argument’s commit

$ git diff b722f1650b9fb33e0990beec027c097526c61478  eab54d0ec21a1d7e351fee4c67139ada740e7e6b
diff --git a/test/script.sh b/test/script.sh
index 33f9e16..cc40434 100755
--- a//test/script.sh
+++ b//test/script.sh
@@ -14,7 +14,7 @@ fi

...
-blah1
+blah2
...

How to check differences between the HEAD (current state/latest revision) and a previous commit:

$ git diff HEAD b722f1650b9fb33e0990beec027c097526c61478 
diff --git a/test/script.sh b/test/script.sh
index 33f9e16..cc40434 100755
--- a//test/script.sh
+++ b//test/script.sh
@@ -14,7 +14,7 @@ fi

...
-blah1
+blah2
...

Or you can specify the last commit or a number of commits:

// Last commit

$ git diff HEAD~1
diff --git a/test/script.sh b/test/script.sh
index 33f9e16..cc40434 100755
--- a//test/script.sh
+++ b//test/script.sh
@@ -14,7 +14,7 @@ fi

...
-blah1
+blah2
...

//Last 2 commits

$ git diff HEAD~2
diff --git a/test/script2.pp b/test/script2.pp
index 8def87e..be66b71 100644
--- a/test/script2.pp
+++ b/test/script2.pp
@@ -4,11 +4,10 @@

 class test {
-  stage { "first": before => Stage["main"] }

-  class { "repos": stage => "first" }
+  class { "test": stage => "first" }

-  class repos {
+  class test {
...
diff --git a/test/script.sh b/test/script.sh
index 33f9e16..cc40434 100755
--- a//test/script.sh
+++ b//test/script.sh
@@ -14,7 +14,7 @@ fi

...
-blah1
+blah2
...

How to check what changed since yesterday or a specific date:

git diff @{yesterday}
diff --git a/.gitignore b/.gitignore
index 4327d2e..8fa1531 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,3 @@
-.svn
-.project
+.com_puppetlabs_geppetto_pptp_target/
+.metadata/
+Geppetto.AutoFileSystemLinked/

// OR a specific date

git diff @{2015-05-01}
diff --git a/.gitignore b/.gitignore
index 4327d2e..8fa1531 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,3 @@
-.svn
-.project
+.com_puppetlabs_geppetto_pptp_target/
+.metadata/
+Geppetto.AutoFileSystemLinked/

How to check all changes to a specific file:

$ git log --follow /path/to/file
commit eab54d0ec21a1d7e351fee4c67139ada740e7e6b
Author: John 
Date:   Wed Jul 8 22:18:05 2015 -0400

    Add feature 2

commit b722f1650b9fb33e0990beec027c097526c61478
Author: John 
Date:   Wed Jul 8 22:31:18 2015 -0400

    Add feature 1

How to check size differences between two trees

I found this one in stackoverflow and has been very useful:

Reference Source: http://stackoverflow.com/questions/10845051/git-show-total-file-size-difference-between-two-commits/10847242#10847242

“git cat-file -s will output the size in bytes of an object in git. git diff-tree can tell you the differences between one tree and another. Putting this together into a script called git-file-size-diff located somewhere on your PATH will give you the ability to call git file-size-diff . Putting this together we can try something like the following” – Stackoverflow


$ cat /usr/local/bin/git-file-size-diff
#!/bin/sh
. git sh-setup
args=$(git rev-parse --sq "$@")
eval "git diff-tree -r $args" | {
  total=0
  while read A B C D M P
  do
    case $M in
      M) bytes=$(( $(git cat-file -s $D) - $(git cat-file -s $C) )) ;;
      A) bytes=$(git cat-file -s $D) ;;
      D) bytes=-$(git cat-file -s $C) ;;
      *)
        echo >&2 warning: unhandled mode $M in \"$A $B $C $D $M $P\"
        continue
        ;;
    esac
    total=$(( $total + $bytes ))
    printf '%d\t%s\n' $bytes "$P"
  done
  echo total $total Bytes
  echo total $(($total/1000)) KBytes
}

// Using it to check size differences between two branches:

$ git file-size-diff master dev
66	.gitignore
239	modules/test/Modulefile
171	modules/test/README
167	modules/test/files/file.txt
1425	modules/test/manifests/init.pp
total 2068 Bytes
total 2 KBytes

// Using it to check size differences between the local master branch and the remote master branch, this is useful to know how much data will be downloaded when doing a ‘git pull’:

$ git file-size-diff master origin/master
633	modules/test/manifests/config.pp
326	modules/test/manifests/service.pp
60	modules/test/manifests/params.pp
573	modules/test/manifests/install.pp
13	modules/test/manifests/init.pp
86	modules/mod/manifests/params.pp
total 1691 Bytes
total 1 KBytes

While working on my video library I noticed that I had to rename a bunch of files to have correct file names that include Show name and Episode name and number.

I have about 275 files named from 000-274, but I needed to rename them to specify the season and episode they correspond to.

For example, files from 125 – 150 are from season 5 of said show, and that is what I will use for this post.

An example of a file in the 125 – 150 range:
‘SHOW – 135 – Some Episode _www.unneededStuff.mp4’

As you can see the filename has the following structure (which I will modify later in this post)

(ShowName) – (filename #) – (EpisodeName)_(Extra unneeded stuff).(extension)

I will be using the Linux/Mac rename command to perform the renaming of these files (125-150).
The end result should be that these files are renamed to have numbers from 501- 526 (for Season 5), so I will use arithmetic manipulate the numbers to be what I want.
I will also clean up the name a little bit so that it does not include the ‘extra unneeded stuff’.
The name of a file should be of form:
‘SHOW – 511 – Some Episode.mp4’

First make sure you work with the specific files you want to modify and nothing else:

$ ls SHOW\ -\ 1{2{5-9},3,4,50}*

// The above will show only files between 125 – 150 will be touched/modified.
// Using Bash brace expansion will help create the list between 125 – 150
// the brace expansion will use anything that has numbers 125-129, and any 13*, 14*, 150

Use the rename command with a RegEx expression that will modify the name the way you want.
I have the following regex expression:

's/(^.+)\s-\s(\d\d\d)(.+)_www.+(\..+$)/$1." - ".($2+376).$3.$4/e'

Explanation:
// (^.+)\s-\s is the Show name plus ‘ – ‘, I am grouping just the name of the show to use it later
// (\d\d\d) is the incorrect episode number used in the filename
// (.+)_www.+ is the Episode name plus ‘unneeded stuff’, I am grouping just the episode name to use it later
// (\..+$) is the file extension, I am grouping it to use it later

// Now what was found with the above Regex will be substituted as follows:
$1 will have the Show name
“ – ” I am concatenating literal “ – “ after the show name
($2+376) I am adding the episode number + 376, which will give me the correct episode number. For example for 135, now it will be 511 (which is Season 5 episode 11)
$3 The Episode name captured from the previous regex
$4 The file extension captured from the previous regex

Run rename with -n (dry run) using the regex and file list created above to verify it will give you what you want/expect.

$ rename -n 's/(^.+)\s-\s(\d\d\d)(.+)_www.+(\..+$)/$1." - ".($2+376).$3.$4/e' SHOW\ -\ 1{2{5..9},3,4,50}*

Output:

‘SHOW – 135 – Some Episode_www.unneededStuff.mp4’ would be renamed to ‘SHOW – 511 – Some Episode.mp4’

Now run it again without -n:

$ rename  's/(^.+)\s-\s(\d\d\d)(.+)_www.+(\..+$)/$1." - ".($2+376).$3.$4/e' SHOW\ -\ 1{2{5..9},3,4,50}*

In this post I am showing how to use generate a MySQL 5 password-hash that can be used to create MySQL GRANTS using a hash instead of a password.

To use a password-hash to create GRANTs:

GRANT ALL ON *.* to user@% identified by PASSWORD '';

A good use case is the Puppet puppetlabs-mysql module to automate the MySQL environment, You can automate/define USER and GRANT creation by using the code below, but notice that it requires a password-hash instead of a password:

users => {
  'someuser@localhost' => {
    ensure                   => 'present',
    max_connections_per_hour => '0',
    max_queries_per_hour     => '0',
    max_updates_per_hour     => '0',
    max_user_connections     => '0',
    password_hash            => '*F3A2A51A9B0F2BE2468926B4132313728C250DBF',
  },
}

OR:

mysql_user { 'root@127.0.0.1':
  ensure                   => 'present',
  max_connections_per_hour => '0',
  max_queries_per_hour     => '0',
  max_updates_per_hour     => '0',
  max_user_connections     => '0',
  password_hash            => '*F3A2A51A9B0F2BE2468926B4132313728C250DBF',
}

They recommend using mysql_password() for creating such a hash. But that means you need to have a MySQL server available.
In this post I am writing about getting those hashes using Python, I wrote a program/script to get the password-hash programatically.

The Python program/script can be found at:
https://github.com/parcejohn/mysql_password_hash

Usage

$ ./mysql_password_hash -h
usage: mysql_password_hash [-h] [-p PASSWORD | -r] [-l PASSWORD_LENGTH]

MySQL Password Hash Generator

optional arguments:
  -h, --help            show this help message and exit
  -p PASSWORD, --password PASSWORD
                        Enter a password
  -r, --generate_random
                        Generate a random password
  -l PASSWORD_LENGTH, --password_length PASSWORD

# Using Command line arguments – User provided password (e.g. ‘secret’)

$ mysql_password_hash -p secret
PASSWORD: secret
HASH: *14e65567abdb5135d0cfd9a70b3032c179a49ee7

# Using Command line arguments – Random password with length=20 (default length=12)

$ mysql_password_hash -r -l 20
PASSWORD: gnlrn96^g18jcblmssa6
HASH: *e3cbe60709e8abe2082c92cc5e72a762d5f18e22

# interactive mode (no arguments)

mysql_password_hash

One of the most important things you should do to your systems is to ensure they have the right time.
In this post I will show how to check and ensure your systems have the correct time using PowerCli.

==> Login to vCenter:

$admin = Get-Credential –Credential EXAMPLE\john
Connect-VIServer -Server vc.example.com -Credential $admin

==> Check time settings:

Get-VMHost | Sort Name | Select Name, `
   @{N="NTPServer";E={$_ |Get-VMHostNtpServer}}, `
   Timezone, `
   @{N="CurrentTime";E={(Get-View $_.ExtensionData.ConfigManager.DateTimeSystem) | Foreach {$_.QueryDateTime().ToLocalTime()}}}, `
   @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq "ntpd"}).Running}}, `
   @{N="StartUpPolicy";E={(Get-VMHostService -VMHost $_ |Where-Object {$_.Key -eq "ntpd"}).Policy}}, `
   @{N="FirewallException";E={$_ | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Select-Object –ExpandProperty Enabled}} `
   | Format-Table -AutoSize

Output:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMHost | Sort Name | Select Name, `
>>    @{N="NTPServer";E={$_ |Get-VMHostNtpServer}}, `
>>    Timezone, `
>>    @{N="CurrentTime";E={(Get-View $_.ExtensionData.ConfigManager.DateTimeSystem) | Foreach {$_.QueryDateTime().ToLocalTime()}}}, `
>>    @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq "ntpd"}).Running}}, `
>>    @{N="StartUpPolicy";E={(Get-VMHostService -VMHost $_ |Where-Object {$_.Key -eq "ntpd"}).Policy}}, `
>>    @{N="FirewallException";E={$_ | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Select-Object -ExpandProperty Enabled}} `
>>    | Format-Table -AutoSize
>>

Name                 NTPServer 					TimeZone CurrentTime         ServiceRunning StartUpPolicy FirewallException
----                 --------- 					-------- -----------         -------------- ------------- -----------------
esx1.example.com           					UTC      6/7/2015 3:25:39 PM          False off                       False
esx2.example.com           					UTC      6/7/2015 3:25:40 PM          False off                       False
esx3.example.com 	{192.168.10.1,192.168.11.1}	        UTC      6/7/2015 3:25:42 PM          False off                       False
esx4.example.com 	192.168.11.1 				UTC      6/7/2015 3:25:43 PM          False off                       False

==> Set time to correct time:

// Get time from the machine running PowerCli
$currentTime = Get-Date

// Update time on ESX hosts
$hosts_time = Get-VMHost | %{ Get-View $_.ExtensionData.ConfigManager.DateTimeSystem }
$hosts_time.UpdateDateTime((Get-Date($currentTime.ToUniversalTime()) -format u))

==> Remove old NTP servers (if any):

$old_ntp_server = '192.168.10.1'
Get-VMHost | Remove-VmHostNtpServer -NtpServer $old_ntp_server -Confirm

Output:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMHost | Sort Name | Select Name, `
>>    @{N="NTPServer";E={$_ |Get-VMHostNtpServer}}, `
>>    Timezone, `
>>    @{N="CurrentTime";E={(Get-View $_.ExtensionData.ConfigManager.DateTimeSystem) | Foreach {$_.QueryDateTime().ToLocalTime()}}}, `
>>    @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq "ntpd"}).Running}}, `
>>    @{N="StartUpPolicy";E={(Get-VMHostService -VMHost $_ |Where-Object {$_.Key -eq "ntpd"}).Policy}}, `
>>    @{N="FirewallException";E={$_ | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Select-Object -ExpandProperty Enabled}} `
>>    | Format-Table -AutoSize
>>

Name                 NTPServer 	TimeZone CurrentTime         ServiceRunning StartUpPolicy FirewallException
----                 --------- 	-------- -----------         -------------- ------------- -----------------
esx1.example.com           	UTC      6/7/2015 3:25:39 PM          False off                       False
esx2.example.com           	UTC      6/7/2015 3:25:40 PM          False off                       False
esx3.example.com 		UTC      6/7/2015 3:25:42 PM          False off                       False
esx4.example.com		UTC      6/7/2015 3:25:43 PM          False off                       False

==> Change NTP to desired configuration:

$ntp_server = '192.168.10.1'
Get-VMHost | Add-VMHostNtpServer $ntp_server
Get-VMHost | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Set-VMHostFirewallException -Enabled:$true
Get-VMHost | Get-VmHostService | Where-Object {$_.key -eq "ntpd"} | Start-VMHostService
Get-VMhost | Get-VmHostService | Where-Object {$_.key -eq "ntpd"} | Set-VMHostService -policy "automatic"

Output:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> $ntp_server = '192.168.10.1'
PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMHost | Add-VMHostNtpServer $ntp_server
192.168.10.1
192.168.10.1
192.168.10.1
192.168.10.1

==> Enable Firewall Exception

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMHost | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Set-VMHostFirewallException -Enabled:$true

Name                 Enabled IncomingPorts  OutgoingPorts  Protocols  ServiceRunning
----                 ------- -------------  -------------  ---------  --------------
NTP Client           True                   123            UDP        True
NTP Client           True                   123            UDP        True
NTP Client           True                   123            UDP        False
NTP Client           True                   123            UDP        False

==> Start NTPd service

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMHost | Get-VmHostService | Where-Object {$_.key -eq "ntpd"} | Start-VMHostService

Key                  Label                          Policy     Running  Required
---                  -----                          ------     -------  --------
ntpd                 NTP Daemon                     on         True     False
ntpd                 NTP Daemon                     on         True     False
ntpd                 NTP Daemon                     off        True     False
ntpd                 NTP Daemon                     off        True     False

==> Ensure NTPd service starts automatically (via policy)

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMhost | Get-VmHostService | Where-Object {$_.key -eq "ntpd"} | Set-VMHostService -policy "automatic"

Key                  Label                          Policy     Running  Required
---                  -----                          ------     -------  --------
ntpd                 NTP Daemon                     automatic  True     False
ntpd                 NTP Daemon                     automatic  True     False
ntpd                 NTP Daemon                     automatic  True     False
ntpd                 NTP Daemon                     automatic  True     False

==> Verify all is set the way you expected

Get-VMHost | Sort Name | Select Name, `
   @{N="NTPServer";E={$_ |Get-VMHostNtpServer}}, `
   Timezone, `
   @{N="CurrentTime";E={(Get-View $_.ExtensionData.ConfigManager.DateTimeSystem) | Foreach {$_.QueryDateTime().ToLocalTime()}}}, `
   @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq "ntpd"}).Running}}, `
   @{N="StartUpPolicy";E={(Get-VMHostService -VMHost $_ |Where-Object {$_.Key -eq "ntpd"}).Policy}}, `
   @{N="FirewallException";E={$_ | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Select-Object –ExpandProperty Enabled}} `
   | Format-Table -AutoSize

Output:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VMHost | Sort Name | Select Name, `
>>    @{N="NTPServer";E={$_ |Get-VMHostNtpServer}}, `
>>    Timezone, `
>>    @{N="CurrentTime";E={(Get-View $_.ExtensionData.ConfigManager.DateTimeSystem) | Foreach {$_.QueryDateTime().ToLocalTime()}}}, `
>>    @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq "ntpd"}).Running}}, `
>>    @{N="StartUpPolicy";E={(Get-VMHostService -VMHost $_ |Where-Object {$_.Key -eq "ntpd"}).Policy}}, `
>>    @{N="FirewallException";E={$_ | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Select-Object -ExpandProperty Enabled}} `
>>    | Format-Table -AutoSize
>>

Name                 NTPServer  TimeZone CurrentTime         ServiceRunning StartUpPolicy FirewallException
----                 ---------  -------- -----------         -------------- ------------- -----------------
esx1.example.com 192.168.10.1 UTC      6/7/2015 3:34:49 PM           True automatic                  True
esx2.example.com 192.168.10.1 UTC      6/7/2015 3:34:51 PM           True automatic                  True
esx3.example.com 192.168.10.1 UTC      6/7/2015 3:34:52 PM           True automatic                  True
esx4.example.com 192.168.10.1 UTC      6/7/2015 3:34:54 PM           True automatic                  True

Recently while working on a project I found that we were keeping .svn metadata in our Git repo.

It was not desirable to keep that in the Git repo but we needed to keep those .svn object in the local filesystem.

The below steps allowed me to do that:

1) Find all the .svn objects

find . -name .svn | xargs -n1

2) Remove them from Git
The (–cache) flag keeps them on disk and (-r) does it recursively

find . -name .svn | xargs -n1 echo git rm --cache -r

3) Add a .gitignore  file with contents below to prevent tracking of .svn objects

.svn

4) Commit changes

git commit -m "Delete .svn metadata from project"

5)Verify

git status
git diff

In this post I will build a very simple RPM, this RPM will contain a very useful program/shell script.
With this information you can build complex RPMs later on.

Set up your build environment

In this case I am using a RHEL 6.5 64bit system

[root@rpmbuild ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.5 (Santiago) 

Install the tools:
rpm-build: is what you need to build RPMs
rpmdevtools: is not required but it is very helpful because it helps you create the directory tree and base SPEC file

[root@rpmbuild ~]# yum install rpm-build rpmdevtools

Create a non-privileged user to build the RPMs

[root@rpmbuild ~]# useradd rpmbuilder
[root@rpmbuild ~]# su - rpmbuilder
[rpmbuilder@rpmbuild ~]$

Create the directory tree using rpmdev-setuptree

[rpmbuilder@rpmbuild ~]$ rpmdev-setuptree
[rpmbuilder@rpmbuild ~]$ tree rpmbuild/
rpmbuild/
├── BUILD
├── RPMS
├── SOURCES
├── SPECS
└── SRPMS

Package Application

Work on packaging your application/program (e.g. very_useful_script.sh)
The folder and the archive naming is important for later when they get unarchived, the rpm tools will by default use name-version (e.g. name=very-useful-script, version=1.0, that is why the folder/archive was named very-useful-script-1.0/ ).
That is the default and can be easily changed in the SPEC file.

[rpmbuilde@rpmbuild ~]$ ls
rpmbuild very_useful_script.sh
[rpmbuilde@rpmbuild ~]$ mkdir very-useful-script-1.0
[rpmbuilde@rpmbuild ~]$ mv very_useful_script.sh very-useful-script-1.0/
[rpmbuilde@rpmbuild ~]$ tar cvzf very-useful-script-1.0.tgz very-useful-script-1.0/
[rpmbuilde@rpmbuild ~]$ ls
rpmbuild  very-useful-script-1.0  very_useful_script.sh  very-useful-script-1.0.tgz

Move your packaged application to the SOURCES directory under rpmbuild/

[rpmbuilde@rpmbuild ~]$ mv very-useful-script-1.0.tgz ~/rpmbuild/SOURCES/

Now it is time to create the SPEC

Create a skeleton spec file

[rpmbuilde@rpmbuild ~]$ rpmdev-newspec 
Skeleton specfile (minimal) has been created to "newpackage.spec".

Move it to your directory tree

[rpmbuilde@rpmbuild ~]$ mv newpackage.spec ~/rpmbuild/SPECS/very-useful-script.spec

This is how your directory tree should look like

[rpmbuilde@rpmbuild ~]$ tree rpmbuild/
rpmbuild/
├── BUILD
├── RPMS
├── SOURCES
│   └── very-useful-script.tgz
├── SPECS
│   └── very-useful-script.spec
└── SRPMS

5 directories, 2 files

SPEC file:

Name:           very-useful-script
Version:        1.0
Release:        1%{?dist}
Summary:        This is a very useful script
Group:          Applications/System
License:        MIT
URL:            http://example.com
Source:         very-useful-script-1.0.tgz

%description
This is a very useful script

%prep
%setup -q

%install
rm -rf $RPM_BUILD_ROOT
install -d $RPM_BUILD_ROOT/usr/local/bin/
install -m 755 very_useful_script.sh $RPM_BUILD_ROOT/usr/local/bin/very_useful_script.sh

%clean
rm -rf $RPM_BUILD_ROOT

%files
%dir /usr/local/bin
%defattr(-,root,root,-)
%doc
/usr/local/bin/very_useful_script.sh

%changelog

Dissecting the SPEC file
The below is header information and just descriptive data

Name:           very-useful-script           
Version:        1.0
Release:        1%{?dist}
Summary:        This is a very useful script
Group:          Applications/System
License:        MIT 
URL:            http://example.com
Source:         very-useful-script-1.0.tgz

%description
This is a very useful script

The below is where we prepare our sources to be packaged into RPM
%prep is a section where we can execute commands or use macros.
%setup is a macro that unarchives the original sources.
Earlier I was discussing the importance of naming the folder and archive as name-version, this is because the %setup macro expects that by default, but you can overwrite the default by specifying the folder name (e.g. %setup -q -n very-useful-script-1.0-john-x86)

%prep
%setup -q 

OR

%prep
%setup -q -n very-useful-script-1.0-john-x86

The below removes previous remains of the files in the buildroot
Then creates a folder /usr/local/bin/ in the buildroot
Then puts our very_useful_script.sh in /usr/loca/bin with mode 755

%install
rm -rf $RPM_BUILD_ROOT
install -d $RPM_BUILD_ROOT/usr/local/bin/
install -m 755 very_useful_script.sh $RPM_BUILD_ROOT/usr/local/bin/very_useful_script.sh

The below just cleans the buildroot

%clean
rm -rf $RPM_BUILD_ROOT

The below specifies all the files that will be installed by the RPM
You need to list them all, or use wildcards

%files
%dir /usr/local/bin
%defattr(-,root,root,-)
%doc
/usr/local/bin/very_useful_script.sh

Build the RPM using the SPEC file

[rpmbuilde@rpmbuild rpmbuild]$ rpmbuild -ba ~/rpmbuild/SPECS/very-useful-script.spec 

After the RPM has been successfully been built, you can find it under:

[root@rpmbuild ~]# ls /home/rpmbuilde/rpmbuild/RPMS/x86_64/

Install it (need to be root)

[root@rpmbuild ~]# rpm -ivh /home/rpmbuilde/rpmbuild/RPMS/x86_64/very-useful-script-1.0-1.el6.x86_64.rpm

Hopefully this guide will help you when building RPMs.

Snapshots are a great feature, probably one of the coolest in virtualization. They can become a problem if they are not used appropriately, unfortunately sometimes we let them grow to an unmanageable size, which can bring performance issues and give us headaches when we need to delete them.

In this post, I will show you how to find out what snapshots are present in your environment, along with some other useful information, like size.

To run the commands below you will need to install PowerCLI (on windows), which is a way to manage a VMware environment programmatically using PowerShell scripting.

To get PowerCLI, go to: www.vmware.com/go/powercli

1) Once you have PowerCLI, open it up, a command prompt will appear:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Connect-VIServer -Server vcenter.example.com -User john

Name                           Port  User
----                           ----  ----
vcenter.example.com             443   john

// At this point you have a session open with your vCenter

2) Query your vCenter to find out what snapshots are present:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VM | Get-Snapshot | Format-list vm,name,sizeGB,create,powerstate

VM         : vm1
Name       : Before_upgrade
SizeGB     : 16.38431124389171600341796875
PowerState : PoweredOn

VM         : vm2
Name       : Before_package_install
SizeGB     : 12.368686250410974025726318359
PowerState : PoweredOn

Let me explain what is going on:
Get-VM‘ asks for the VMs that are running on your vCenter, PowerCLI returns an object for each VM and you then asks for the snapshots of each returned VM object by using ‘Get-Snapshot‘, then you take that output and format it by using ‘Format-list‘, but you are only asking for the information about ‘vm,name,sizeGB,create,powerstate

You can request any of the following:
Description
Created
Quiesced
PowerState
VM
VMId
Parent
ParentSnapshotId
ParentSnapshot
Children
SizeMB
SizeGB
IsCurrent
IsReplaySupported
ExtensionData
Id
Name
Uid

3) The above will give you the info you want, but I prefer CSV reports that I can share with the team or management. To get a good CSV report run the following:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> Get-VM | Get-Snapshot | Select-Object vm,name,sizeGB,create,powerstate | Export-Csv C:\vm_snapshots.csv

I recommend taking a look at VMware’s best practices around snapshots:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1025279