Technologist

Tech stuff about Cloud, DevOps, SysAdmin, Virtualization, SAN, Hardware, Scripting, Automation and Development

Browsing Posts tagged linux

In a previous post I wrote about how to Create a custom Vagrant Box from scratch.

In this post I will walk through the usage of Packer to automate the creation of a CentOS 7 image that can be used in Vagrant or even vSphere.

Packer is a cool Hashicorp tool that helps you automate the creation of machine images.

You can download the project/git repo at https://github.com/parcejohn/packer-centos7

From their website:
“Packer is easy to use and automates the creation of any type of machine image. It embraces modern configuration management by encouraging you to use automated scripts to install and configure the software within your Packer-made images. Packer brings machine images into the modern age, unlocking untapped potential and opening new opportunities.”

I use Packer to continue the journey to Infrastructure As Code, where even my golden images/templates are automated and source controlled (on Git).

Requirements:

* Packer // I installed on Mac using $ brew install packer
* vagrant // I installed on Mac using $ brew install vagrant
* vmware fusion // I installed using $ brew cask install vmware-fusion
* CentOS 7 ISO file

Packer uses a JSON template file to orchestrate the image creation and there are different stages in the process, I will concentrate on the below three, but you should familiarize with all of them (https://www.packer.io/docs/basics/terminology.html)

Builders: Packer component to create a machine image for a single platform, in this case I will be using the VMware builder.
Provisioners: Packer component that installs and configures software within a running machine prior to that machine being turned into a static image. Example provisioners include shell scripts, Chef, Puppet, etc. I will be using the shell provisioner.
Post-Processors: Packer component that takes the result of the builder or another post-processor and process that to create a new artifact. Examples of post-processors are compress to compress artifacts, upload to upload artifacts, etc. I will be creating a Vagrant box as the artifact, and also demonstrate how to upload the image to vSphere.

It’s time to walk through the process of creating a CentOS 7 image using Packer.

1) Create directory structure
The http folder will host a kickstart file, Packer will use its built in web server to serve this kickstart file
The scripts folder will host the provisioning scripts that will define the machine

centos/
├── http
└── scripts

2) Populate with the following configuration files and scripts (the contents of these files are shown in the next steps)

centos/
├── centos-7.1-x64-vmware.json
├── http
│   └── ks.cfg
├── scripts
│   ├── base.sh
│   ├── cleanup.sh
│   ├── hgfs.sh
│   ├── vmware.sh
│   └── zerodisk.sh
├── template.json
└── vagrant_rsa_key

3) Create a SSH key pair, you will need this to login to the server to complete install and configuration

The Public Key will be injected to the ‘vagrant’ user as part of the kickstart, it will then be used in the Packer JSON template to allow Packer to login to the machine
The Private Key will stay on your system to allow Packer and Vagrant to login to the created machine.

$ ssh-keygen -t rsa -b 4096 -C "vagrant" -N '' -q -f ./vagrant_rsa_key

4) Create a Packer JSON Template file (template.json)

{
  "variables": {
    "vm_name": "centos-7.1-vmware",
    "iso_url": "{{env `ISO_URL`}}",
    "iso_sha256": "f90e4d28fa377669b2db16cbcb451fcb9a89d2460e3645993e30e137ac37d284"
  },
  "builders": [
    {
      "headless": true,
      "type": "vmware-iso",
      "boot_command": [
        " text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg"
      ],
      "boot_wait": "10s",
      "disk_size": 8192,
      "guest_os_type": "centos-64",
      "http_directory": "http",
      "iso_url": "{{user `iso_url`}}",
      "iso_checksum_type": "sha256",
      "iso_checksum": "{{user `iso_sha256`}}",
      "ssh_username": "vagrant",
      "ssh_private_key_file": "vagrant_rsa",
      "ssh_port": 22,
      "ssh_wait_timeout": "10000s",
      "shutdown_command": "echo '/sbin/halt -h -p' > /tmp/shutdown.sh; echo 'vagrant'|sudo -S sh '/tmp/shutdown.sh'",
      "tools_upload_flavor": "linux",
      "tools_upload_path": "/tmp/vmware_tools_{{.Flavor}}.iso",
      "vmx_data": {
        "memsize": "1024",
        "numvcpus": "1",
        "cpuid.coresPerSocket": "1"
      }
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "execute_command": "echo 'vagrant'|sudo -S sh '{{.Path}}'",
      "override": {
        "vmware-iso": {
          "scripts": [
            "scripts/base.sh",
            "scripts/vmware.sh",
            "scripts/hgfs.sh",
            "scripts/cleanup.sh",
            "scripts/zerodisk.sh"
          ]
        }
      }
    }
  ],
  "post-processors": [
    {
      "type": "vagrant",
      "override": {
        "vmware": {
          "output": "centos-7.1-x64-vmware.box"
        }
      }
    }
  ]
}

Packer Components/Sections:

Variables:
This is just to centralize variables that will be used by other components (e.g. Builder, provisioner, etc)
I can also set a variable to use Environment variables or command line provided values:
User provided values:
“iso_url”: “{{user `iso_url`}}”,

Environment variable values:
“iso_url”: “{{env `ISO_URL`}}”,

Builders:
Here is an array of builders and the needed parameters for each builder, in this case the only builder is of type vmware-iso.
Parameters and some comments about them.

      "headless": true, // Means not to open the VMware Fusion Console
      "type": "vmware-iso", // VMware type builder
      "boot_command": [
        " text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg"  // Send these key combination to the VMWare Fusion Console
      ],
      "boot_wait": "10s",
      "disk_size": 8192,
      "guest_os_type": "centos-64",
      "http_directory": "http",  // Directory where the kickstart file is placed
      "iso_url": "{{user `iso_url`}}", // Location of ISO image, defined at runtime, more on that later
      "iso_checksum_type": "sha256", 
      "iso_checksum": "{{user `iso_sha256`}}",
      "ssh_username": "vagrant",  // Packer will log in to resulting machine for provisioning, this user must exist (user created from the Kickstart) 
      "ssh_private_key_file": "vagrant_rsa", // Packer will log in using ssh key created earlier
      "ssh_port": 22,
      "ssh_wait_timeout": "10000s",
      "shutdown_command": "echo '/sbin/halt -h -p' > /tmp/shutdown.sh; echo 'vagrant'|sudo -S sh '/tmp/shutdown.sh'",
      "tools_upload_flavor": "linux",
      "tools_upload_path": "/tmp/vmware_tools_{{.Flavor}}.iso", 
      "vmx_data": {
        "memsize": "1024",
        "numvcpus": "1",
        "cpuid.coresPerSocket": "1"

Provisioners:
This section uses the shell provisioner and runs all of those scripts listed, which are located in the scripts folder.

      "type": "shell",
      "execute_command": "echo 'vagrant'|sudo -S sh '{{.Path}}'",
      "override": {
        "vmware-iso": {
          "scripts": [
            "scripts/base.sh",
            "scripts/vmware.sh",
            "scripts/hgfs.sh",
            "scripts/cleanup.sh",
            "scripts/zerodisk.sh"

Post-processor:
This section states that I want a vagrant box and I am giving it the name.

      "type": "vagrant",
      "override": {
        "vmware": {
          "output": "centos-7.1-x64-vmware.box"

5) Kickstart file (should be http/ks.cfg based on the template.json file)
This is a minimal install of CentOS 7
Note: The private key (XXXX) given to the vagrant user, that is the key that we created in a previous step

install
cdrom
lang en_US.UTF-8
keyboard us
network --onboot yes --device eth0 --bootproto dhcp --noipv6
rootpw  --plaintext vagrant
firewall --enabled --service=ssh
authconfig --enableshadow --passalgo=sha512
selinux --disabled
timezone --utc America/New_York
bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet"

text
skipx
zerombr

clearpart --all --initlabel
autopart

auth  --useshadow  --enablemd5
firstboot --disabled
reboot

%packages --nobase --ignoremissing 
@core
bzip2
kernel-devel
kernel-headers
-ipw2100-firmware
-ipw2200-firmware
-ivtv-firmware
%end

%post
# Install SUDO
/usr/bin/yum -y install sudo

# Create vagrant user
/usr/sbin/useradd vagrant
/bin/mkdir /home/vagrant/.ssh
/bin/chmod 700 /home/vagrant/.ssh
cat >           /home/vagrant/.ssh/authorized_keys <<'VAGRANT_RSA'
ssh-rsa XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX vagrant
VAGRANT_RSA

/bin/chmod 600 /home/vagrant/.ssh/authorized_keys
/bin/chown -R vagrant /home/vagrant/.ssh

# Add vagrant user to SUDO
echo "vagrant        ALL=(ALL)       NOPASSWD: ALL" >> /etc/sudoers.d/vagrant
echo "Defaults:vagrant !requiretty"                 >> /etc/sudoers.d/vagrant
chmod 0440 /etc/sudoers.d/vagrant
%end

6 ) Provisioning Scrips
Packer calls these scripts (from the JSON template) on the created machine before making it a template/image.

base.sh // Install basic packages

#!/usr/bin/env bash
set -x

sed -i "s/^.*requiretty/#Defaults requiretty/" /etc/sudoers
yum -y install gcc make gcc-c++ kernel-devel-`uname -r` perl

vmware.sh // Install VMTools, I separated this from base as I may want to call different scripts on a VirtualBox/etc builder.

#!/usr/bin/env bash
set -x

yum install -y fuse-libs open-vm-tools

hgfs.sh // For vagrant to work properly with folders

#!/usr/bin/env bash
set -x

VMWARE_ISO=/tmp/vmware_tools_linux.iso
VMWARE_MNTDIR=$(mktemp --tmpdir=/tmp -q -d -t vmware_mnt_XXXXXX)
VMWARE_TMPDIR=$(mktemp --tmpdir=/tmp -q -d -t vmware_XXXXXX)

# Extract tools
mount -o loop $VMWARE_ISO $VMWARE_MNTDIR
tar zxf $VMWARE_MNTDIR/VMwareTools*.tar.gz -C $VMWARE_TMPDIR
umount $VMWARE_MNTDIR

# Install tools
$VMWARE_TMPDIR/vmware-tools-distrib/vmware-install.pl -d

# Clean up
rm -f $VMWARE_ISO
rm -rf $VMWARE_MNTDIR
rm -rf $VMWARE_TMPDIR

cleanup.sh // Clean up before converting to image

#!/usr/bin/env bash
set -x

yum -y erase gtk2 libX11 hicolor-icon-theme avahi freetype bitstream-vera-fonts
rpm --rebuilddb
yum -y clean all

zerodisk.sh // This is so that the resulting box is as small as possible

#!/usr/bin/env bash
set -x

dd if=/dev/zero of=/EMPTY bs=1M
rm -f /EMPTY

7) Run Packer against the template.json file created earlier and watch it build the machine image

$ packer build -var ‘iso_url=/Users/john/iso/CentOS-7-x86_64-Minimal-1511.iso’ -only=vmware-iso template.json
vmware-iso output will be in this color.

==> vmware-iso: Downloading or copying ISO
vmware-iso: Downloading or copying: file:///Users/john/iso/CentOS-7-x86_64-Minimal-1511.iso
==> vmware-iso: Creating virtual machine disk
==> vmware-iso: Building and writing VMX file
==> vmware-iso: Starting HTTP server on port 8101
==> vmware-iso: Starting virtual machine…
vmware-iso: The VM will be run headless, without a GUI. If you want to
vmware-iso: view the screen of the VM, connect via VNC without a password to
vmware-iso: 127.0.0.1:5989
==> vmware-iso: Waiting 10s for boot…
==> vmware-iso: Connecting to VM via VNC
==> vmware-iso: Typing the boot command over VNC…
==> vmware-iso: Waiting for SSH to become available…
==> vmware-iso: Connected to SSH!
==> vmware-iso: Uploading the ‘linux’ VMware Tools
==> vmware-iso: Provisioning with shell script: scripts/base.sh
vmware-iso: + sed -i ‘s/^.*requiretty/#Defaults requiretty/’ /etc/sudoers



==> vmware-iso: Provisioning with shell script: scripts/zerodisk.sh
vmware-iso: + dd if=/dev/zero of=/EMPTY bs=1M
vmware-iso: dd: error writing ‘/EMPTY’: No space left on device
vmware-iso: 5488+0 records in
vmware-iso: 5487+0 records out
vmware-iso: 5754265600 bytes (5.8 GB) copied, 5.27091 s, 1.1 GB/s
vmware-iso: + rm -f /EMPTY
==> vmware-iso: Gracefully halting virtual machine…
vmware-iso: Waiting for VMware to clean up after itself…
==> vmware-iso: Deleting unnecessary VMware files…
vmware-iso: Deleting: output-vmware-iso/564de113-2fc5-2010-8e5d-63fec92f12f7.vmem
vmware-iso: Deleting: output-vmware-iso/packer-vmware-iso.plist
vmware-iso: Deleting: output-vmware-iso/vmware.log
==> vmware-iso: Cleaning VMX prior to finishing up…
vmware-iso: Unmounting floppy from VMX…
vmware-iso: Detaching ISO from CD-ROM device…
vmware-iso: Disabling VNC server…
==> vmware-iso: Compacting the disk image
==> vmware-iso: Running post-processor: vagrant
==> vmware-iso (vagrant): Creating Vagrant box for ‘vmware’ provider
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s001.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s002.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s003.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.nvram
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmsd
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmx
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmxf
vmware-iso (vagrant): Compressing: Vagrantfile
vmware-iso (vagrant): Compressing: disk-s001.vmdk
vmware-iso (vagrant): Compressing: disk-s002.vmdk
vmware-iso (vagrant): Compressing: disk-s003.vmdk
vmware-iso (vagrant): Compressing: disk.vmdk
vmware-iso (vagrant): Compressing: metadata.json
vmware-iso (vagrant): Compressing: packer-vmware-iso.nvram
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmsd
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmx
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmxf
Build ‘vmware-iso’ finished.

==> Builds finished. The artifacts of successful builds are:
–> vmware-iso: ‘vmware’ provider box: centos-7.1-x64-vmware.box

8) Add box to Vagrant

$ vagrant box add –name centos-7.1-010-x64-vmware.box centos-7.1-x64-vmware.box
==> box: Adding box ‘centos-7.1-010-x64-vmware.box’ (v0) for provider:
box: Downloading: file:///Users/john/packer/centos/centos-7.1-x64-vmware.box
==> box: Successfully added box ‘centos-7.1-010-x64-vmware.box’ (v0) for ‘vmware_desktop’!

The above is all you need to have your Infrastructure be Code that can be versioned and automatically create your images and use them on Vagrant.
The next item will show you how to use Packer to move the same image to your vSphere environment.

9) Sending Image to vSphere
To send the image created by Packer to vSphere you will need to add another post-provisioning entry to the JSON array:

    {
      "type": "vsphere",
      "host": "{{user `vm_host`}}",
      "username": "{{user `vm_user`}}",
      "password": "{{user `vm_pass`}}",
      "datacenter": "{{user `vm_dc`}}",
      "cluster": "{{user `vm_cluster`}}",
      "resource_pool": " ",
      "datastore": "{{user `vm_datastore`}}",
      "vm_folder": "{{user `vm_folder`}}",
      "vm_name": "{{user `vm_name`}}", 
      "vm_network": "{{user `vm_network`}}",
      "insecure" : "true"
    }

As you can see I am using variables to make this portable, so that means you have to add variables to the variables section as well, below is the new template to do both vagrant and vSphere image provisioning.

{
  "variables": {
    "vm_name": "centos-7.1-vmware",
    "iso_url": "{{env `ISO_URL`}}",
    "iso_sha256": "f90e4d28fa377669b2db16cbcb451fcb9a89d2460e3645993e30e137ac37d284",
    "vm_host": "{{ user `vm_host` }}",
    "vm_user": "{{ user `vm_user` }}",
    "vm_pass": "{{ env `vm_pass` }}",
    "vm_dc":   "{{ user `vm_dc` }}",
    "vm_cluster": "{{user `vm_cluster`}}",
    "vm_datastore": "{{user `vm_datastore`}}",
    "vm_folder": "{{user `vm_folder`}}",
    "vm_name": "{{user `vm_name`}}", 
    "vm_network": "{{user `vm_network`}}"
  },
  "builders": [
    {
      "headless": true,
      "type": "vmware-iso",
      "boot_command": [
        " text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg"
      ],
      "boot_wait": "10s",
      "disk_size": 8192,
      "guest_os_type": "centos-64",
      "http_directory": "http",
      "iso_url": "{{user `iso_url`}}",
      "iso_checksum_type": "sha256",
      "iso_checksum": "{{user `iso_sha256`}}",
      "ssh_username": "vagrant",
      "ssh_private_key_file": "vagrant_rsa",
      "ssh_port": 22,
      "ssh_wait_timeout": "10000s",
      "shutdown_command": "echo '/sbin/halt -h -p' > /tmp/shutdown.sh; echo 'vagrant'|sudo -S sh '/tmp/shutdown.sh'",
      "tools_upload_flavor": "linux",
      "tools_upload_path": "/tmp/vmware_tools_{{.Flavor}}.iso",
      "vmx_data": {
        "memsize": "1024",
        "numvcpus": "1",
        "cpuid.coresPerSocket": "1"
      }
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "execute_command": "echo 'vagrant'|sudo -S sh '{{.Path}}'",
      "override": {
        "vmware-iso": {
          "scripts": [
            "scripts/base.sh",
            "scripts/vmware.sh",
            "scripts/hgfs.sh",
            "scripts/cleanup.sh",
            "scripts/zerodisk.sh"
          ]
        }
      }
    }
  ],
  "post-processors": [
    {
      "type": "vagrant",
      "override": {
        "vmware": {
          "output": "centos-7.1-x64-vmware.box"
        }
      }
    },
    {
      "type": "vsphere",
      "host": "{{user `vm_host`}}",
      "username": "{{user `vm_user`}}",
      "password": "{{user `vm_pass`}}",
      "datacenter": "{{user `vm_dc`}}",
      "cluster": "{{user `vm_cluster`}}",
      "resource_pool": " ",
      "datastore": "{{user `vm_datastore`}}",
      "vm_folder": "{{user `vm_folder`}}",
      "vm_name": "{{user `vm_name`}}", 
      "vm_network": "{{user `vm_network`}}",
      "insecure" : "true"
    }
  ]
}

10) Re-Run Packer against the template.json file and watch it build the machine image an

$ packer build \
> -var ‘iso_url=/Users/john/iso/CentOS-7-x86_64-Minimal-1511.iso’ \
> -var ‘vm_host=vc.example.com’ \
> -var ‘vm_user=john@example.com’ \
> -var ‘vm_pass=XXXXXXX’ \
> -var ‘vm_dc=vDC’ \
> -var ‘vm_cluster=Folder/Cluster’ \
> -var ‘vm_datastore=store’ \
> -var ‘vm_folder=Images’ \
> -var ‘vm_name=centos71’ \
> -var ‘vm_network=dvs-net1’ \
> -only=vmware-iso template.json

vmware-iso output will be in this color.

==> vmware-iso: Downloading or copying ISO
vmware-iso: Downloading or copying: file:///Users/john/iso/CentOS-7-x86_64-Minimal-1511.iso
==> vmware-iso: Creating virtual machine disk


==> vmware-iso: Running post-processor: vagrant
==> vmware-iso (vagrant): Creating Vagrant box for ‘vmware’ provider
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s001.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s002.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk-s003.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/disk.vmdk
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.nvram
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmsd
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmx
vmware-iso (vagrant): Copying: output-vmware-iso/packer-vmware-iso.vmxf
vmware-iso (vagrant): Compressing: Vagrantfile
vmware-iso (vagrant): Compressing: disk-s001.vmdk
vmware-iso (vagrant): Compressing: disk-s002.vmdk
vmware-iso (vagrant): Compressing: disk-s003.vmdk
vmware-iso (vagrant): Compressing: disk.vmdk
vmware-iso (vagrant): Compressing: metadata.json
vmware-iso (vagrant): Compressing: packer-vmware-iso.nvram
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmsd
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmx
vmware-iso (vagrant): Compressing: packer-vmware-iso.vmxf
==> vmware-iso: Running post-processor: vsphere
vmware-iso (vsphere): Uploading output-vmware-iso/packer-vmware-iso.vmx to vSphere
vmware-iso (vsphere): Opening VMX source: output-vmware-iso/packer-vmware-iso.vmx
vmware-iso (vsphere): Opening VI target: vi://john%40example.com@vc.exmaple.com:443/vDC/host/Cluster/Resources/
vmware-iso (vsphere): Deploying to VI: vi://john%40example.com@vc.example.com:443/vDC/host/Cluster/Resources/
Transfer Completed
vmware-iso (vsphere): Completed successfully
vmware-iso (vsphere):
Build ‘vmware-iso’ finished.

==> Builds finished. The artifacts of successful builds are:
–> vmware-iso: ‘vmware’ provider box: centos-7.1-x64-vmware.box
–> vmware-iso:

Now you should have the same image running on both Vagrant and vSphere.

From time to time I get questions about how to check differences between two Git commits or two branches, I will answer those questions in this post.

How to check differences between two branches:

Example:
you have a dev branch and a master branch. You develop in the dev branch and would like to know if you were to merge the dev changes to master, what will those changes be:
Please note the order of the branch this states differences that will be added to master from dev.

$ git diff master dev

How to check differences between two commits:

Example:
When you do git log you will see your different commits over time, you need to check the differences between two commits.

commit eab54d0ec21a1d7e351fee4c67139ada740e7e6b
Author: John
Date: Wed Jul 8 22:18:05 2015 -0400

Add feature 2

commit b722f1650b9fb33e0990beec027c097526c61478
Author: John
Date: Wed Jul 8 22:31:18 2015 -0400

Add feature 1

// Remember the order the first argument is the point in time, and the second item is what is added to the first argument’s commit

$ git diff b722f1650b9fb33e0990beec027c097526c61478  eab54d0ec21a1d7e351fee4c67139ada740e7e6b
diff --git a/test/script.sh b/test/script.sh
index 33f9e16..cc40434 100755
--- a//test/script.sh
+++ b//test/script.sh
@@ -14,7 +14,7 @@ fi

...
-blah1
+blah2
...

How to check differences between the HEAD (current state/latest revision) and a previous commit:

$ git diff HEAD b722f1650b9fb33e0990beec027c097526c61478 
diff --git a/test/script.sh b/test/script.sh
index 33f9e16..cc40434 100755
--- a//test/script.sh
+++ b//test/script.sh
@@ -14,7 +14,7 @@ fi

...
-blah1
+blah2
...

Or you can specify the last commit or a number of commits:

// Last commit

$ git diff HEAD~1
diff --git a/test/script.sh b/test/script.sh
index 33f9e16..cc40434 100755
--- a//test/script.sh
+++ b//test/script.sh
@@ -14,7 +14,7 @@ fi

...
-blah1
+blah2
...

//Last 2 commits

$ git diff HEAD~2
diff --git a/test/script2.pp b/test/script2.pp
index 8def87e..be66b71 100644
--- a/test/script2.pp
+++ b/test/script2.pp
@@ -4,11 +4,10 @@

 class test {
-  stage { "first": before => Stage["main"] }

-  class { "repos": stage => "first" }
+  class { "test": stage => "first" }

-  class repos {
+  class test {
...
diff --git a/test/script.sh b/test/script.sh
index 33f9e16..cc40434 100755
--- a//test/script.sh
+++ b//test/script.sh
@@ -14,7 +14,7 @@ fi

...
-blah1
+blah2
...

How to check what changed since yesterday or a specific date:

git diff @{yesterday}
diff --git a/.gitignore b/.gitignore
index 4327d2e..8fa1531 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,3 @@
-.svn
-.project
+.com_puppetlabs_geppetto_pptp_target/
+.metadata/
+Geppetto.AutoFileSystemLinked/

// OR a specific date

git diff @{2015-05-01}
diff --git a/.gitignore b/.gitignore
index 4327d2e..8fa1531 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,3 @@
-.svn
-.project
+.com_puppetlabs_geppetto_pptp_target/
+.metadata/
+Geppetto.AutoFileSystemLinked/

How to check all changes to a specific file:

$ git log --follow /path/to/file
commit eab54d0ec21a1d7e351fee4c67139ada740e7e6b
Author: John 
Date:   Wed Jul 8 22:18:05 2015 -0400

    Add feature 2

commit b722f1650b9fb33e0990beec027c097526c61478
Author: John 
Date:   Wed Jul 8 22:31:18 2015 -0400

    Add feature 1

How to check size differences between two trees

I found this one in stackoverflow and has been very useful:

Reference Source: http://stackoverflow.com/questions/10845051/git-show-total-file-size-difference-between-two-commits/10847242#10847242

“git cat-file -s will output the size in bytes of an object in git. git diff-tree can tell you the differences between one tree and another. Putting this together into a script called git-file-size-diff located somewhere on your PATH will give you the ability to call git file-size-diff . Putting this together we can try something like the following” – Stackoverflow


$ cat /usr/local/bin/git-file-size-diff
#!/bin/sh
. git sh-setup
args=$(git rev-parse --sq "$@")
eval "git diff-tree -r $args" | {
  total=0
  while read A B C D M P
  do
    case $M in
      M) bytes=$(( $(git cat-file -s $D) - $(git cat-file -s $C) )) ;;
      A) bytes=$(git cat-file -s $D) ;;
      D) bytes=-$(git cat-file -s $C) ;;
      *)
        echo >&2 warning: unhandled mode $M in \"$A $B $C $D $M $P\"
        continue
        ;;
    esac
    total=$(( $total + $bytes ))
    printf '%d\t%s\n' $bytes "$P"
  done
  echo total $total Bytes
  echo total $(($total/1000)) KBytes
}

// Using it to check size differences between two branches:

$ git file-size-diff master dev
66	.gitignore
239	modules/test/Modulefile
171	modules/test/README
167	modules/test/files/file.txt
1425	modules/test/manifests/init.pp
total 2068 Bytes
total 2 KBytes

// Using it to check size differences between the local master branch and the remote master branch, this is useful to know how much data will be downloaded when doing a ‘git pull’:

$ git file-size-diff master origin/master
633	modules/test/manifests/config.pp
326	modules/test/manifests/service.pp
60	modules/test/manifests/params.pp
573	modules/test/manifests/install.pp
13	modules/test/manifests/init.pp
86	modules/mod/manifests/params.pp
total 1691 Bytes
total 1 KBytes

Running on runlevel 5 is not a good idea for a server, I try to run servers on runlevel 3 with as minimal packages as needed. But sometimes you need a graphical application or a browser for some reason or another. You can use VNC to connect to the server and do it over SSH to make sure the communication is encrypted.

This guide is to enable VNC over SSH on a Linux Server. I will use CentOS 5.2 server for this guide.

I will allow only user john to be able to VNC/SSH to the server. Since I will be testing GUI based stuff, I will need a graphical environment installed. I do not need to run my server in graphical mode, but I need to have the proper packages installed.

1) Install the X Window System group

[root@server ~]# yum groupinstall “X Window System”

2) Install the vncserver

[root@server ~]# yum install vnc-server

3) As the user that will use VNC, create VNC password

[john@server ~]$ vncpasswd

4) Modify VNC configuration to allow X (/home/john/.vnc/xstartup)

 #!/bin/sh

# Uncomment the following two lines for normal desktop:
unset SESSION_MANAGER
exec /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
twm &

5) Start the vncserver as the user who will have access, use a display number that you will remember, here I am using 2. Also use the following arguments to make sure it ONLY listens to the localhost.

[john@server ~]$ vncserver :2 -geometry 1024×768 -nolisten tcp -nohttpd -localhost

6) Check that the VNC service is only listening locally

[john@server ~]$ netstat -ntlp

tcp 0 0 127.0.0.1:5902 0.0.0.0:* LISTEN 7927/Xvnc

7) Ok, now lets connect using VNC over SSH
From a remote station you will start an SSH session and forward an arbitrary port(e.g. 5544) to the vnc server’s localhost address on port 5092. (It ends with 2 because you started the vnc server with :2)
That means that whenever you are on Server2 and you send packets to localhost on port 5544, those packets will be forwarded through the SSH tunnel to the vnc Server localhost on port 5902.

[alex@server2 ~]$ ssh -L5544:localhost:5902 john@server.example.com

8 ) Now on Server2 start a VNC client/viewer and on the server address, enter:

localhost:5544

9) Enjoy your secure VNC session!

VNC over SSH

VNC over SSH

You have added a new disk or increased the size of your LUN, or increased the size of the virtual disk in case of virtual machines, and now you need to increase the partition, the Logical Volume and the filesystem in order to be able to use the new space.

In this post I go through the steps necessary to make this happen in a RHEL 5.3 system.

The LUN I will increase has 20GB and it had an LVM partition. I decided to increase the LUN size to 72GB and this is how it looks now.

[root@server~]# fdisk -lu
Disk /dev/sdb: 77.3 GB, 77309411328 bytes
255 heads, 63 sectors/track, 9399 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 2611 20971488 8e Linux LVM

I need to perform the following steps in order to be able to use the new space.

1. Increase the size of the partition using fdisk

[root@server ~]# fdisk /dev/sdb

Command (m for help): u //Change the display to sectors
Changing display/entry units to sectors
Command (m for help): p //Print the current partition table for that drive
Disk /dev/sdb: 77.3 GB, 77309411328 bytes
255 heads, 63 sectors/track, 9399 cylinders, total 150994944 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 * 64 41943039 20971488 8e Linux LVM
Command (m for help): d //Delete the partition information, we will recreate
Selected partition 1
Command (m for help): n //Create partition
Command action
e extended
p primary partition (1-4)
p //In this case it is primary
Partition number (1-4): 1 // In this case it is the first partition on the drive
First sector (63-150994943, default 63): 64 //Align partition if used on NetApp
Last sector or +size or +sizeM or +sizeK (64-150994943, default 150994943):
Using default value 150994943
Command (m for help): t //Change type from Linux(default) to Linux LVM
Selected partition 1
Hex code (type L to list codes): 8e //Linux LVM partition type
Changed system type of partition 1 to 8e (Linux LVM)
Command (m for help): p //Print again to double check
Disk /dev/sdb: 77.3 GB, 77309411328 bytes
255 heads, 63 sectors/track, 9399 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 9400 75497440 8e Linux LVM
Command (m for help): w //Write the partition table
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

2. You need to reboot for the changes to take place, or just run

server# partprobe

3. Make LVM acknowledge the new space

[root@server ~]# pvresize /dev/sdb1

4. Check that the Volume group shows the new space

[root@server ~]# vgs
VG #PV #LV #SN Attr VSize VFree
vg0 1 2 0 wz–n- 71.97G 52.00G

5. Extend the logical volume:
make it total of 28G in this example

[root@server~]# lvresize -L 28G /dev/mapper/vg0-lvhome
Extending logical volume lvhome to 28.00 GB
Logical volume lvswap successfully resized

You can also take all the free space available

[root@server ~]# lvresize -l +100%FREE /dev/mapper/vg0-lvhome
Extending logical volume lvhome to 67.97 GB
Logical volume lvhome successfully resized

6. Use the rest for whatever partition you want

[root@server~]# lvcreate -l 100%FREE -n lvdata vg0

7. Resize the Filesystem

[root@server~]# resize2fs /dev/mapper/vg0-lvhome
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/mapper/vg0-lvhome is mounted on /home; on-line resizing required
Performing an on-line resize of /dev/mapper/vg0-lvhome to 9953280 (4k) blocks.
The filesystem on /dev/mapper/vg0-lvhome is now 9953280 blocks long.