Technologist

Tech stuff about Cloud, DevOps, SysAdmin, Virtualization, SAN, Hardware, Scripting, Automation and Development

Browsing Posts in Development

Installing Puppet Enterprise on AWS EC2 using Vagrant

February 6, 2017 No comments

In this guide I go over how to use Vagrant with AWS and in the process have an automated way to install Puppet Enterprise.
I am separating data and code by having a generic Vagrantfile with the code and have a servers.yaml file with all the data that will change from user to user.
For installing the Puppet Enterprise server I am including the automated provisioning script I am using with Vagrant and using AWS Tags to set the hostname of the launched server.

Pre-requisites:

  • Vagrant
  • vagrant-aws plug-in:

  • AWS pre-requisites:

    • While you can add your AWS credentials to the Vagrantfile, it is not recommended. A better way is to have the AWS CLI tools installed and configured

    TL;DR To get started right away you can download the project from github vagrant-aws-puppetserver, otherwise follow the guide below.

    Create Vagrantfile

    The below Vagrantfile utilizes a yaml file (servers.yaml) to provide the data, it allows you to control data using the yaml file and not have to modify the Vagrantfile code – separating code and data.

    // Vagrantfile

    Create servers.yaml

    This file contains the information that will be used by the Vagrantfile, this includes
    AWS region: Which region will this EC2 server run
    AWS keypair: Key used to connect to your launched EC2 instance
    AWS subnet id: Where will this EC2 instance sit in the AWS network
    AWS associate public ip: Do you need a public IP? true or false
    AWS security group: What AWS security group should be associated, should allow Puppetserver needed ports and whatever else you need (ssh, etc)
    AWS ami: Which AMI will you be using I am using a CentOS7
    AWS instance type: Puppetserver needs enough CPU/RAM, during my testing m3.xlarge was appropriate
    AWS SSH username: The EC2 instance user (depends on which AMI you choose), the CentOS AMI expects ec2-user
    AWS SSH private key path: The local path to the SSH key pair
    AWS User Data: I am adding user data which will execute a bash script that allows Vagrant to interact with the launched EC2 instance
    AWS Tags: This is not required for Vagrant and AWS/EC2, but in my provision script I am using the AWS Name Tag to be the system’s hostname, the other 2 tags are there for demonstration purposes
    provision: This is a provisioning script that will be run on the EC2 instance – this is the script that install the Puppet Enterprise server
    AWS IAM Role: You don’t need to add a role when working with Vagrant and AWS/EC2, but I am using a specific IAM role to allow the launched EC2 instance to be able to get information about its AWS Tags, so it is important that you provide it with a Role that allows DescribeTags, see below IAM policy:

    Now use your data in the servers.yaml file
    // servers.yaml

    At this point you can spin up EC2 instances using the above Vagrantfile and servers.yaml file. If you add provision: install_puppetserver.sh to the servers.yaml file as I did and add the below script you will have a Puppet Enterprise server ready to go.

    // install_puppetserver.sh

aws, Cloud, DevOps, Puppet, Scripting, Vagrant , , , ,
Read more

Using Python Virtual Environments with Sublime Text

August 21, 2015 No comments

When developing in Python I enjoy using Sublime Text, especially its ability to run/build right from Sublime by simply pressing COMMAND+B. Meaning that I dont have to go out of the editor to the terminal to run the Python program.

Sample Python Program:

In sublime, I can program, run the program and see the result, increasing my productivity:

sublime_venv1

But what if I want to run a Python Virtual Environment and its benefits (see developing-in-python-using-python-virtual-environments).
Once the virtual environment is activated your terminal will use the Python Virtual Environment just fine, but Sublime Text will not, it will continue to use the system-wide Python environment by default.

Example: Having a Python module (e.g. IPy) installed in your virtual environment, but not system-wide, and having the Python virtual environment active:

# Python code:

# Running from terminal – (OK):

# Running from Sublime – (NOT OK):

sublime_venv2

To make Sublime Text use the virtual environment you need to create a Sublime project and edit its properties to use the Virtual Env.

# Create Sublime Project
Go to Project -> Save Project As…
// I named mine blog.sublime-project

# Edit project properties by editing the file in sublime to be as follows:
Note:
Remember the name: label as this is how you will use the build with… option
The shell_cmd: label is the command that will run when you build. It is far from perfect as you have to hardcode the Python program in the configuration, but at least works.

# Prepare Sublime to use your new build system:
Tools -> Build Systems -> PYTHON_VENV

# Now you can now continue to code and use COMMAND+B to build/run your programs without leaving your Sublime Text editor:

sublime_venv3

Development, DevOps, Python , ,
Read more

Developing in Python using Python Virtual Environments

August 10, 2015 No comments

When developing in Python you most likely will need to install Python modules that will provide some type of functionality, the process is very simple:

Install the Python package manager:

Install modules, for example ‘yaml’, this is without virtual environments, this is basically installing a Python module system-wide:

The above will install Python modules for the system, but to do so you need ‘root’ privileges (e.g. sudo), and there is a possibility that developers don’t have ‘root’ privileges.
Another important scenario to consider is that installing/updating/modifying system-wide Python modules could affect other existing Python projects in the same system.

A way to overcome the system-wide python dependencies is to work with your own virtual environment, which allows you to install Python modules in a Python virtual environment and without the need for ‘root’ privileges.

What this accomplishes is to have your Python projects have their specific dependencies met without affecting anyone else.

To develop using Python virtual environments, the system should have the virtualenv Python package first:

Now as a regular user (no root or sudo needed):
# Create (or cd into) your Python project folder

# Create Python Virtual Environment, you can name it what you want, I chose to name it ‘env’

# Activating the virtual environment
To use the virtual environment (as opposed to the system-wide Python environment) you need to activate it first. After it is activated you will see it in the left your command prompt.

# System-wide Python

# Activate Virtual Env

# Ready to use Python Virtual Env

# Installing modules in the virtual environment

# Uninstalling modules in the virtual environment

# Install multiple python modules from a requirements file
If your project has multiple Python modules required, it is better to create a requirements.txt file with the list of Python modules.

Then you can install all the listed Python modules in your virtual environment as follows:

# Deactivate virtual environment (go back to using system-wide Python)

Development, DevOps, Python ,
Read more

Showing Differences Between two Git Commits

August 3, 2015 No comments

From time to time I get questions about how to check differences between two Git commits or two branches, I will answer those questions in this post.

How to check differences between two branches:

Example:
you have a dev branch and a master branch. You develop in the dev branch and would like to know if you were to merge the dev changes to master, what will those changes be:
Please note the order of the branch this states differences that will be added to master from dev.

How to check differences between two commits:

Example:
When you do git log you will see your different commits over time, you need to check the differences between two commits.

commit eab54d0ec21a1d7e351fee4c67139ada740e7e6b
Author: John
Date: Wed Jul 8 22:18:05 2015 -0400

Add feature 2

commit b722f1650b9fb33e0990beec027c097526c61478
Author: John
Date: Wed Jul 8 22:31:18 2015 -0400

Add feature 1

// Remember the order the first argument is the point in time, and the second item is what is added to the first argument’s commit

How to check differences between the HEAD (current state/latest revision) and a previous commit:

Or you can specify the last commit or a number of commits:

// Last commit

//Last 2 commits

How to check what changed since yesterday or a specific date:

// OR a specific date

How to check all changes to a specific file:

How to check size differences between two trees

I found this one in stackoverflow and has been very useful:

Reference Source: http://stackoverflow.com/questions/10845051/git-show-total-file-size-difference-between-two-commits/10847242#10847242

“git cat-file -s will output the size in bytes of an object in git. git diff-tree can tell you the differences between one tree and another. Putting this together into a script called git-file-size-diff located somewhere on your PATH will give you the ability to call git file-size-diff . Putting this together we can try something like the following” – Stackoverflow

// Using it to check size differences between two branches:

// Using it to check size differences between the local master branch and the remote master branch, this is useful to know how much data will be downloaded when doing a ‘git pull’:

Development, DevOps , , ,
Read more

Rename Multiple Files in Linux Using Regex and perform Arithmetic

July 26, 2015 No comments

While working on my video library I noticed that I had to rename a bunch of files to have correct file names that include Show name and Episode name and number.

I have about 275 files named from 000-274, but I needed to rename them to specify the season and episode they correspond to.

For example, files from 125 – 150 are from season 5 of said show, and that is what I will use for this post.

An example of a file in the 125 – 150 range:
‘SHOW – 135 – Some Episode _www.unneededStuff.mp4’

As you can see the filename has the following structure (which I will modify later in this post)

(ShowName) – (filename #) – (EpisodeName)_(Extra unneeded stuff).(extension)

I will be using the Linux/Mac rename command to perform the renaming of these files (125-150).
The end result should be that these files are renamed to have numbers from 501- 526 (for Season 5), so I will use arithmetic manipulate the numbers to be what I want.
I will also clean up the name a little bit so that it does not include the ‘extra unneeded stuff’.
The name of a file should be of form:
‘SHOW – 511 – Some Episode.mp4’

First make sure you work with the specific files you want to modify and nothing else:

// The above will show only files between 125 – 150 will be touched/modified.
// Using Bash brace expansion will help create the list between 125 – 150
// the brace expansion will use anything that has numbers 125-129, and any 13*, 14*, 150

Use the rename command with a RegEx expression that will modify the name the way you want.
I have the following regex expression:

Explanation:
// (^.+)\s-\s is the Show name plus ‘ – ‘, I am grouping just the name of the show to use it later
// (\d\d\d) is the incorrect episode number used in the filename
// (.+)_www.+ is the Episode name plus ‘unneeded stuff’, I am grouping just the episode name to use it later
// (\..+$) is the file extension, I am grouping it to use it later

// Now what was found with the above Regex will be substituted as follows:
$1 will have the Show name
“ – ” I am concatenating literal “ – “ after the show name
($2+376) I am adding the episode number + 376, which will give me the correct episode number. For example for 135, now it will be 511 (which is Season 5 episode 11)
$3 The Episode name captured from the previous regex
$4 The file extension captured from the previous regex

Run rename with -n (dry run) using the regex and file list created above to verify it will give you what you want/expect.

Output:
…
‘SHOW – 135 – Some Episode_www.unneededStuff.mp4’ would be renamed to ‘SHOW – 511 – Some Episode.mp4’
…

Now run it again without -n:

Linux, Mac, Scripting none
Read more

Generate MySQL Password Hash using Python

June 26, 2015 1 comment

In this post I am showing how to use generate a MySQL 5 password-hash that can be used to create MySQL GRANTS using a hash instead of a password.

To use a password-hash to create GRANTs:

A good use case is the Puppet puppetlabs-mysql module to automate the MySQL environment, You can automate/define USER and GRANT creation by using the code below, but notice that it requires a password-hash instead of a password:

OR:

They recommend using mysql_password() for creating such a hash. But that means you need to have a MySQL server available.
In this post I am writing about getting those hashes using Python, I wrote a program/script to get the password-hash programatically.

The Python program/script can be found at:
https://github.com/parcejohn/mysql_password_hash

Usage

# Using Command line arguments – User provided password (e.g. ‘secret’)

# Using Command line arguments – Random password with length=20 (default length=12)

# interactive mode (no arguments)

Development, DevOps, Puppet, Python none
Read more

Managing Time in vSphere with PowerCli

June 19, 2015 No comments

One of the most important things you should do to your systems is to ensure they have the right time.
In this post I will show how to check and ensure your systems have the correct time using PowerCli.

==> Login to vCenter:

$admin = Get-Credential –Credential EXAMPLE\john
Connect-VIServer -Server vc.example.com -Credential $admin

==> Check time settings:

Output:

==> Set time to correct time:

==> Remove old NTP servers (if any):

Output:

==> Change NTP to desired configuration:

Output:

==> Enable Firewall Exception

==> Start NTPd service

==> Ensure NTPd service starts automatically (via policy)

==> Verify all is set the way you expected

Output:

Development, PowerCLI, PowerShell, Scripting, Virtualization, VMware , , , , , ,
Read more

Remove objects from Git, but keep them in the filesystem

October 9, 2014 No comments

Recently while working on a project I found that we were keeping .svn metadata in our Git repo.

It was not desirable to keep that in the Git repo but we needed to keep those .svn object in the local filesystem.

The below steps allowed me to do that:

1) Find all the .svn objects

2) Remove them from Git
The (–cache) flag keeps them on disk and (-r) does it recursively

3) Add a .gitignore  file with contents below to prevent tracking of .svn objects

4) Commit changes

5)Verify

Development, DevOps none
Read more

Building RPMs for RHEL6 or CentOS6

January 7, 2014 No comments

In this post I will build a very simple RPM, this RPM will contain a very useful program/shell script.
With this information you can build complex RPMs later on.

Set up your build environment

In this case I am using a RHEL 6.5 64bit system

Install the tools:
rpm-build: is what you need to build RPMs
rpmdevtools: is not required but it is very helpful because it helps you create the directory tree and base SPEC file

Create a non-privileged user to build the RPMs

Create the directory tree using rpmdev-setuptree

Package Application

Work on packaging your application/program (e.g. very_useful_script.sh)
The folder and the archive naming is important for later when they get unarchived, the rpm tools will by default use name-version (e.g. name=very-useful-script, version=1.0, that is why the folder/archive was named very-useful-script-1.0/ ).
That is the default and can be easily changed in the SPEC file.

Move your packaged application to the SOURCES directory under rpmbuild/

Now it is time to create the SPEC

Create a skeleton spec file

Move it to your directory tree

This is how your directory tree should look like

SPEC file:

Dissecting the SPEC file
The below is header information and just descriptive data

The below is where we prepare our sources to be packaged into RPM
%prep is a section where we can execute commands or use macros.
%setup is a macro that unarchives the original sources.
Earlier I was discussing the importance of naming the folder and archive as name-version, this is because the %setup macro expects that by default, but you can overwrite the default by specifying the folder name (e.g. %setup -q -n very-useful-script-1.0-john-x86)

OR

The below removes previous remains of the files in the buildroot
Then creates a folder /usr/local/bin/ in the buildroot
Then puts our very_useful_script.sh in /usr/loca/bin with mode 755

The below just cleans the buildroot

The below specifies all the files that will be installed by the RPM
You need to list them all, or use wildcards

Build the RPM using the SPEC file

After the RPM has been successfully been built, you can find it under:

Install it (need to be root)

Hopefully this guide will help you when building RPMs.

Development, Linux, Systems none
Read more

Find VM snapshots in your VMware environment

February 21, 2013 No comments

Snapshots are a great feature, probably one of the coolest in virtualization. They can become a problem if they are not used appropriately, unfortunately sometimes we let them grow to an unmanageable size, which can bring performance issues and give us headaches when we need to delete them.

In this post, I will show you how to find out what snapshots are present in your environment, along with some other useful information, like size.

To run the commands below you will need to install PowerCLI (on windows), which is a way to manage a VMware environment programmatically using PowerShell scripting.

To get PowerCLI, go to: www.vmware.com/go/powercli

1) Once you have PowerCLI, open it up, a command prompt will appear:

// At this point you have a session open with your vCenter

2) Query your vCenter to find out what snapshots are present:

Let me explain what is going on:
‘Get-VM‘ asks for the VMs that are running on your vCenter, PowerCLI returns an object for each VM and you then asks for the snapshots of each returned VM object by using ‘Get-Snapshot‘, then you take that output and format it by using ‘Format-list‘, but you are only asking for the information about ‘vm,name,sizeGB,create,powerstate‘

You can request any of the following:
Description
Created
Quiesced
PowerState
VM
VMId
Parent
ParentSnapshotId
ParentSnapshot
Children
SizeMB
SizeGB
IsCurrent
IsReplaySupported
ExtensionData
Id
Name
Uid

3) The above will give you the info you want, but I prefer CSV reports that I can share with the team or management. To get a good CSV report run the following:

I recommend taking a look at VMware’s best practices around snapshots:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1025279

Development, PowerCLI, PowerShell, Scripting, Virtualization, VMware , , ,
Read more
RSS

Recent Posts

Browse by Tags

apache automation aws cisco development devops encryption git gpg ldap linux multipathing netapp network powercli powershell python rhel scripting security ssh ssl sun vagrant virtualenv virtualization vmware vsphere vum wan

Categories

Meta

Related Links

Powered by WordPress Web Design by SRS Solutions © 2023 Technologist Design by SRS Solutions