Technologist

Tech stuff about Cloud, DevOps, SysAdmin, Virtualization, SAN, Hardware, Scripting, Automation and Development

Browsing Posts in Hardware

VMware ESXi can take advantage of Flash/local SSDs in multiple ways:

  • Host swap cache (since 5.0):  ESXi will use part of the an SSD datastore as swap space shared by all VMs.  This means that when there is ESX memory swapping, the ESXi server will use the SSD drives, which is faster than HDD, but still slower than RAM.
  • Virtual SAN (VSAN) (since 5.5 with VSAN licensing): You can combine  the local HDD and local SSD on each host and basically create a distributed storage platform.  I like to think of it as a RAIN(Redundant Array of Independent Nodes).
  • Virtual Flash/vFRC (since 5.5 with Enterprise Plus): With this method the SSD is formatted with VFFS and can be configured as read and write through cache for your VMs, it allows ESXi to locally cache virtual machine read I/O and survives VM migrations as long as the destination ESXi host has Virtual Flash enabled. To be able to use this feature VMs HW version needs to be 10.

Check if the SSD drives were properly detected by ESXi

From vSphere Web Client

Select the ESXi host with Local SSD drives -> Manage -> Storage -> Storage Devices

See if it shows as SSD or Non-SSD, for example:

flash1

 

From CLI:

~ # esxcli storage core device list
...
naa.60030130f090000014522c86152074c9
 Display Name: Local LSI Disk (naa.60030130f090000014522c86199898)
 Has Settable Display Name: true
 Size: 94413
 Device Type: Direct-Access
 Multipath Plugin: NMP
 Devfs Path: /vmfs/devices/disks/naa.60030130f090000014522c86199898
 Vendor: LSI
 Model: MRSASRoMB-8i
 Revision: 2.12
 SCSI Level: 5
 Is Pseudo: false
 Status: on
 Is RDM Capable: false
 Is Local: true
 Is Removable: false
 Is SSD: false  <-- Not recognized as SSD
 Is Offline: false
 Is Perennially Reserved: false
 Queue Full Sample Size: 0
 Queue Full Threshold: 0
 Thin Provisioning Status: unknown
 Attached Filters:
 VAAI Status: unsupported
 Other UIDs: vml.020000000060030130f090000014522c86152074c94d5253415352
 Is Local SAS Device: false
 Is Boot USB Device: false
 No of outstanding IOs with competing worlds: 32
...

To enable the SSD option on the SSD drive

At this point you should put your host in maintenance mode because it will need to be rebooted.

If the SSD is not properly detected you need to use storage claim rules to force it to be type SSD. (This is also useful if you want to fake a regular drive to be SSD for testing purposes)

# esxcli storage nmp device list
...
naa.60030130f090000014522c86152074c9   <-- Take note of this device ID for the command below
 Device Display Name: Local LSI Disk (naa.60030130f090000014522c86152074c9)
 Storage Array Type: VMW_SATP_LOCAL
 Storage Array Type Device Config: SATP VMW_SATP_LOCAL does not support device configuration.
 Path Selection Policy: VMW_PSP_FIXED
 Path Selection Policy Device Config: {preferred=vmhba2:C2:T0:L0;current=vmhba2:C2:T0:L0}
 Path Selection Policy Device Custom Config:
 Working Paths: vmhba2:C2:T0:L0
 Is Local SAS Device: false
 Is Boot USB Device: false
...

Add a PSA claim rule to mark the device as SSD (if it is not local (e.g. SAN))

# esxcli storage nmp satp rule add --satp=<SATP_TYPE> --device=<device ID> --option="enable_ssd"

For example (in case this was a SAN attached LUN)

# esxcli storage nmp satp rule add --satp=VMW_SATP_XXX --device=naa.60030130f090000014522c86152074c9  --option="enable_ssd"

 

Add a PSA claim rule to mark the device as Local and SSD at the same time (if the SSD drive is local)

# esxcli storage nmp satp rule add –-satp=VMW_SATP_LOCAL –-device=<device ID> --option="enable_local enable_ssd"

For the device in my example it would be:

# esxcli storage nmp satp rule add --satp=VMW_SATP_LOCAL --device=naa.60030130f090000014522c86152074c9 --option="enable_local enable_ssd"

Reboot your ESXi host for the changes to take effect.

 

To remove the rule (for whatever reason, including testing and going back)

esxcli storage nmp satp rule remove --satp VMW_SATP_LOCAL --device <device ID> --option=enable_ssd
esxcli storage nmp satp list |grep ssd
esxcli storage core claiming reclaim -d <device ID>
esxcli storage core device list --device=<device ID>

Once the ESXi server is back online verify that the SSD option is OK

From vSphere Web Client

Select the ESXi host with Local SSD drives -> Manage -> Storage -> Storage Devices

See if it shows as SSD or Non-SSD, for example:

flash2

From CLI:

~ # esxcli storage core device list
...
naa.60030130f090000014522c86152074c9
 Display Name: Local LSI Disk (naa.60030130f090000014522c86152074c9)
 Has Settable Display Name: true
 Size: 94413
 Device Type: Direct-Access
 Multipath Plugin: NMP
 Devfs Path: /vmfs/devices/disks/naa.60030130f090000014522c86152074c9
 Vendor: LSI
 Model: MRSASRoMB-8i
 Revision: 2.12
 SCSI Level: 5
 Is Pseudo: false
 Status: on
 Is RDM Capable: false
 Is Local: true
 Is Removable: false
 Is SSD: true  <-- Now it is true
 Is Offline: false
 Is Perennially Reserved: false
 Queue Full Sample Size: 0
 Queue Full Threshold: 0
 Thin Provisioning Status: unknown
 Attached Filters:
 VAAI Status: unsupported
 Other UIDs: vml.020000000060030130f090000014522c86152074c94d5253415352
 Is Local SAS Device: false
 Is Boot USB Device: false
 No of outstanding IOs with competing worlds: 32
...

Exit Maintenance mode.

Do the same on ALL hosts in the cluster.

Configure Virtual Flash

Now that the ESXi server recognize the SSD drives we can enable Virtual Flash.

You need to perform the below steps from the vSphere Web Client on all ESX hosts

ESXi host -> Manage -> Settings -> Virtual Flash -> Virtual Flash Resource Management -> Add Capacity…

flash3

You will see that the SSD device has been formatted using the VFFS filesystem, it can be used to allocate space for virtual flash host swap cache or to configure virtual Flash Read Cache for virtual disks.

flash4

 

Configure Virtual Flash Host Swap

One of the options you have is to use the Flash/SSD as Host Swap Cache, to do this:

ESXi host -> Manage -> Settings -> Virtual Flash -> Virtual Flash Host Swap Cache Configuration -> Edit…

// Enable and select the size of the cache in GB

flash5

 

Configure Flash Read Cache

Flash read cache is configured on a per-vm basis, per vmdk basis. VMs need to be at virtual hardware version 10 in order to use vFRC.

To enable vFRC on a VM’s harddrive:

VM -> Edit Settings -> Expand Hard Disk -> Virtual Flash Read Cache

Enter the size of the cache in GB (e.g. 20)

You can start conservative and increase if needed, I start with 10% of the VMDK size. Below, in the monitor vFRC section, you will see tips to rightsize your cache.

flash6

 

If you click on Advanced, you can configure/change the specific block-size (default is 8k) for the Read Cache, this allows you to optimize the cache for the specific workload the VM is running.

flash7

The default block size is 8k, but you may want to rightsize this based on the application/workload to be able to efficiently use the cache.

If you dont size the block-size of the cache you could potentially be affecting the efficiency of the cache:

  • If the workload has block sizes larger than the configured block-size then you will have increased cache misses.
  • If the workload has block sizes smaller than the configured block-size then you will be wasting precious cache.

Size correctly the block-size of your Cache

To correctly size the block-size of your cache you need to determine the correct I/O length/size for cache block size:

Login to the ESX host running the workload/VM for which you want to enable vFRC

 

Find world ID of each device

~ # /usr/lib/vmware/bin/vscsiStats -l
Virtual Machine worldGroupID: 44670, Virtual Machine Display Name: myvm, Virtual Machine Config File: /vmfs/volumes/523b4bff-f2f2c400-febe-0025b502a016/myvm/myvm.vmx, {
 Virtual SCSI Disk handleID: 8194 (scsi0:0)
 Virtual SCSI Disk handleID: 8195 (scsi0:1)
 }
...

 

Start gathering statistics on World ID // Give it some time while it captures statistics

~ # /usr/lib/vmware/bin/vscsiStats -s -w 44670
 vscsiStats: Starting Vscsi stats collection for worldGroup 44670, handleID 8194 (scsi0:0)
 Success.
 vscsiStats: Starting Vscsi stats collection for worldGroup 44670, handleID 8195 (scsi0:1)
 Success.

Get the IO length histogram to find the most dominant IO length

You want the IO length for the harddisk you will enable vFRC, in this case scsi0:1

(-c means compressed output)

~ # /usr/lib/vmware/bin/vscsiStats -p ioLength -c -w 44670
...
Histogram: IO lengths of Write commands,virtual machine worldGroupID,44670,virtual disk handleID,8195 (scsi0:1)
 min,4096
 max,409600
 mean,21198
 count,513
 Frequency,Histogram Bucket Limit
 0,512
 0,1024
 0,2048
 0,4095
 174,4096
 0,8191
 6,8192
 1,16383
 311,16384
 4,32768
 1,49152
 0,65535
 2,65536
 1,81920
 1,131072
 1,262144
 11,524288
 0,524288
...

As you can see, in this specific case,  16383(16k) is the most dominant IO length, and this is what you should use in the Advance options.

flash8

Now you are using a Virtual Flash Read Cache on that VM’s harddisk, which should improve the performance.

Monitor your vFRC

Login to the ESX host running the workload/VM for which you enabled vFRC, in the example below it is a 24GB Cache with 4K block-size:

# List physical Flash devices
 ~ # esxcli storage vflash device list
 Name Size Is Local Is Used in vflash Eligibility
 -------------------- ----- -------- ----------------- ---------------------------------
 naa.500a07510c06bf6c 95396 true true It has been configured for vflash
 naa.500a0751039c39ec 95396 true true It has been configured for vflash
# Show virtual disks configured for vFRC. You will find the vmdk name for the virtual disk in the cache list:
 ~ # esxcli storage vflash cache list
 vfc-101468614-myvm_2
# Get Statistics about the cache
~ # esxcli storage vflash cache stats get -c vfc-101468614-myvm_2
   Read:
         Cache hit rate (as a percentage): 60
         Total cache I/Os: 8045314
         Mean cache I/O latency (in microseconds): 3828
         Mean disk I/O latency (in microseconds): 13951
         Total I/Os: 13506424
         Mean IOPS: 249
         Max observed IOPS: 1604
         Mean number of KB per I/O: 627
         Max observed number of KB per I/O: 906
         Mean I/O latency (in microseconds): 4012
         Max observed I/O latency (in microseconds): 6444
   Evict:
         Last I/O operation time (in microseconds): 0
         Number of I/O blocks in last operation: 0
         Mean blocks per I/O operation: 0
   Total failed SSD I/Os: 113
   Total failed disk I/Os: 1
   Mean number of cache blocks in use: 5095521

There is a lot of important information here:
The Cache hit rate shows you the percentage of how much the cache is being used. A high number is better because it means that hits use the cache more frequently.
Other important items are IOPs and latency.

This stats also show information that can help you right size your cache, if you see a high number of cache evictions, Evict->Mean blocks per I/O operation, it could be an indication that your cache size is small or that the block-size of the cache is incorrectly configured.

To calculate available block in the cache, do the following:
SizeOfCache(in bytes) / BlockSizeOfCache(in bytes) = #ofBlocksInvFRC

For the example: A 24GB cache with 4k block-size, will have 6291456 blocks in the vFRC, see:
25769803776
/
4096
=
6291456

 

In the stats above we see 5095521 as the Mean number of cache blocks in use, and no evictions which indicates that 24GB cache with 4k seems to be a correctly sized cache.

Keep monitoring your cache to gain as much performance as you can from your Flash/SSD devices.

I have been playing with my raspberry Pi for a while now, while connected to a TV or a monitor(), but the below guide shows how to connect to its console using a serial cable, this allows me to take it on the road and connect to it from my laptop. Also the serial cable provides power to it so I only need to take my Raspberry Pi and my usb-serial cable (as well as the SD card and WIFI adapter) and I have a complete system on the road.

You will need a usb to serial cable like the: USB to TTL Serial Cable – Debug / Console Cable for Raspberry Pi

I am using my Macbook Air and needed to get the following drives for my usb-serial cable:
http://prolificusa.com/pl-2303hx-drivers/
OR
http://changux.co/osx-installer-to-pl2303-serial-usb-on-osx-lio

Install the drivers and reboot.
Once you plug in your device, open your terminal and you will see the drivers under /dev

// Driver
$sudo dmesg | grep -i "PL-2303"
Password:
PL-2303/X V1.5.0 start, Prolific
// Device
$ls -lrt /dev/tty.usbserial
crw-rw-rw- 1 root wheel 33, 4 Apr 21 15:53 /dev/tty.usbserial

Connect the usb-serial cable to the Raspberry Pi like this:
raspberry_pi_console_cable

Ok, now that Raspberry Pi is connected to the usb-serial cable and the this to the computer, let’s start our terminal.

$ screen /dev/tty.usbserial 115200

You should see the Raspberry Pi starting up and its console messages:
Raspberry Pi Console

Now you can log in and start using your Raspberry Pi, like when it was connected to a TV (of course on command line mode).
Screen Shot 2013-12-05 at 9.36.02 AM

 

This is guide is for the first steps to get started with the Raspberry Pi, I will be working with the Raspberry Pi Model B (http://www.raspberrypi.org/wp-content/uploads/2011/07/RaspiModelB.png)

OS Config:

The first thing you should do, while waiting for the Raspberry Pi to be shipped is to burn the OS in a SD card. When you get a SD card or any piece of hardware or peripheral for the Raspberry Pi be sure to check http://elinux.org/RPi_VerifiedPeripherals

Download an OS to install on the SD card from http://www.raspberrypi.org/downloads
For this guide I am using Raspbian ‘Wheezy’ – which is based on Debian.

To burn the OS on the SD card, on a Mac:
Insert the SD card and find its devide node:

john@mac.local:~/Downloads/raspberry_pi$diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *251.0 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:          Apple_CoreStorage                         250.1 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Macintosh HD           *249.8 GB   disk1
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *4.0 GB     disk2
   1:                 DOS_FAT_32 NO NAME                 4.0 GB     disk2s1

# Unmount the ‘automatically’ mounted partition (e.g. disk2s1)
john@mac.local:~/Downloads/raspberry_pi$sudo diskutil unmount /dev/disk2s1
Volume NO NAME on disk2s1 unmounted

# Unzip OS img
john@mac.local:~/Downloads/raspberry_pi$unzip 2013-05-25-wheezy-raspbian.zip
Archive:  2013-05-25-wheezy-raspbian.zip
  inflating: 2013-05-25-wheezy-raspbian.img

# Install image into SD card using ‘dd’ - go watch a movie, it takes a while...
john@mac.local:~/Downloads/raspberry_pi$sudo dd bs=1m if=~/Downloads/raspberry_pi/2013-05-25-wheezy-raspbian.img of=/dev/rdisk2
1850+0 records in
1850+0 records out
1939865600 bytes transferred in 107.348874 secs (18070666 bytes/sec)

# Eject the card
john@mac.local:~/Downloads/raspberry_pi$sudo diskutil eject /dev/rdisk2
Password:
Disk /dev/rdisk2 ejected

At this point you should have the OS ready on the SD card, if you will be using a TV with HDMI as the display for your Raspberry Pi, skip the next section(HDMI-VGA Adapter Display + Sound Config), otherwise see the ‘HDMI-VGA Adapter Display + Sound Config’ section for how to configure a non-HDMI display

HDMI-VGA Adapter Display + Sound Config::

The raspberry Pi conveniently comes with an HDMI port and also an RCA video port to allow to use a TV to display its output. That is great, but I wanted to use my PC monitor (or one I had lying around) to play with my raspberry Pi.

For those of you who have older monitors and want to use one, know this:
1) If the monitor has a HDMI port, you are all set
2) If the monitor has DVI, you can buy a HDMI to DVI cable
3) If the monitor has VGA, you CANNOT buy a HDMI to VGA cable. This was my situation and what I wanted to do, I wanted to use a small DELL monitor that only had a VGA port.

So, I needed to buy a HDMI to VGA adapter, there are several out there that look like a regular HDMI to VGA cable and they indeed do work with the raspberry Pi, but keep in mind that it will use power to feed the digital to analog conversion, and power is a scarce resource in the Raspberry Pi.

I am using the following adapter, which comes with external power, from Amazon:
Generic PC DVD HDMI to VGA & Audio For HDTV CRT Video Converter Box Adapter 1080P New

To be able to use a HDMI-VGA adapter, you will need to edit a config file, most likely on another computer because the Pi is not available yet due to display not available.

Put the SD card on another computer and edit config.txt

# To enable display through the HDMI-VGA adapter:
#disable_overscan=1
disable_overscan=0

# To enable sound through the HDMI-VGA adapter
hdmi_drive=2

*** To test sound once the Raspberry Pi starts up, plug in speakes to the HDMI-VGA adapter, then type:

pi@raspberrypi ~ $ aplay /usr/share/sounds/alsa/Front_Center.wav

Keyboard/Mouse Config:

The Raspberry Pi comes with 2 USB ports, which will be taken by the keyboard and mouse, but I needed another one for the Wifi. So I got the following keyboard which includes a TouchPad, so I don’t need an extra mouse and works perfectly with the Pi and Raspbian:
Smart Touch Mini USB TouchPad Keyboard

Network (WIFI) Config:

On the other USB port I plugged a Edimax EW-7811Un 150 Mbps Wireless 11n Nano Size USB Adapter which works out of the box with Raspbian. You can use the graphical interface to scan and connect, or edit the config file:

pi@raspberrypi ~ $ cat /etc/network/interfaces
auto lo

iface lo inet loopback
iface eth0 inet dhcp

allow-hotplug wlan0
iface wlan0 inet manual
wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
iface default inet dhcp

pi@raspberrypi ~ $ sudo cat /etc/wpa_supplicant/wpa_supplicant.conf
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
	ssid="mywlan"
	psk="strongpassword"
	proto=WPA
	key_mgmt=WPA-PSK
	pairwise=TKIP
	auth_alg=OPEN
}

Start Up:

After booting your Raspberry Pi you will see a menu, the important things to do are:
1. Expand Filesystem so that the whole SD card is available
2. Change User Password
3. Enable Boot to Desktop – THIS DEPENDS IF YOU THINK YOU WILL BE USING THE DESKTOP MOST OF THE TIME, otherwise dont enable it, and know that from the command line you can start the graphical environment by executing ‘startx’
4. Internationalisation Options
I1 Change Locale – Change to en_US.UTF-8
I2 Change Timezone – US/Eastern
I3 Change Keyboard Layout – English (US)

8. Advanced Options
A2 Hostname
A4 SSH – if you need to enable SSH

Select

// If later you want to go back to this menu, from a terminal type: ‘raspi-config’

You will be left in a command prompt, you can type startx and you will be taken to the graphical interface

That is it, you are all set, enjoy your Raspberry Pi

For a while now, I have been playing with the Soekris SingleBoard Computers (SBCs), they are compact, low-power, low-cost, advanced communication computers based on an up to 500 Mhz 586 class processor. In this particular model 5501, I installed ubunuty JeOS, and it has been my always on computer for a while.

Boot the Soekris while connected via serial. I am using my Macbook Air and needed to get the following drives for my usb-serial cable:
http://prolificusa.com/pl-2303hx-drivers/

Install the drivers and reboot.
Once you plug in your device you will see it under /dev

// Driver
$sudo dmesg | grep -i "PL-2303"
Password:
PL-2303/X V1.5.0 start, Prolific
// Device
$ls -lrt /dev/tty.usbserial
crw-rw-rw- 1 root wheel 33, 4 Apr 21 15:53 /dev/tty.usbserial

I had previously configured my soekris to have console speed of 57600 (default baud rate is 19200)

When connecting via console to a device, I usually prefer to use minicom, but you alternatives:

$ screen /dev/tty.usbserial 57600

But in this case, because I will be using XMODEM to transfer the bios image I downloaded from Soekris I will use ‘cu’

Download the latest BIOS from soekris’ website:
http://soekris.com/downloads.html

Connect to the serial using ‘cu’

$ sudo cu -l /dev/tty.usbserial -s 57600

// It will show you the BIOS version, in this case 1.32
// If the below does not show, it is very likely that you need to set ConMute=Disabled, after pressing Ctrl+P to get to the Menu

...
> set ConMute=Disabled
> reboot
...
comBIOS ver. 1.32 20080626 Copyright (C) 2000-2008 Soekris Engineering.

net5501

0512 Mbyte Memory CPU Geode LX 500 Mhz

Pri Sla WDC WD800BEVT-60ZCT1 LBA Xlt 1024-255-63 78 Gbyte

Slot Vend Dev ClassRev Cmd Stat CL LT HT Base1 Base2 Int
-------------------------------------------------------------------
0:01:2 1022 2082 10100000 0006 0220 08 00 00 A0000000 00000000 10
0:06:0 1106 3053 02000096 0117 0210 08 40 00 0000E101 A0004000 11
0:07:0 1106 3053 02000096 0117 0210 08 40 00 0000E201 A0004100 05
0:08:0 1106 3053 02000096 0117 0210 08 40 00 0000E301 A0004200 09
0:09:0 1106 3053 02000096 0117 0210 08 40 00 0000E401 A0004300 12
0:20:0 1022 2090 06010003 0009 02A0 08 40 80 00006001 00006101
0:20:2 1022 209A 01018001 0005 02A0 08 00 00 00000000 00000000
0:21:0 1022 2094 0C031002 0006 0230 08 00 80 A0005000 00000000 15
0:21:1 1022 2095 0C032002 0006 0230 08 00 00 A0006000 00000000 15

// Enter Ctrl+P to get to the menu

>

// Download the image (In my mac I downloaded to /tmp/b5501_133c.bin
> download -

// Enter/paste ~+lsz -X /tmp/b5501_133c.bin
Start sending file using XMODEM/CRC protocol.
~+lsz -X /tmp/b5501_133c.bin
Sending /tmp/b5501_133c.bin, 784 blocks: Give your local XMODEM receive command now.
Bytes Sent: 100352 BPS:5198

Transfer complete

// Now update the flash with the downloaded image:

> flashupdate
Updating BIOS Flash ,,,,,,,,,,,,,,,,,,,,,,,,,,,,..,,,,.... Done.

// That’s it not reboot and you will see the new version

> reboot
...
comBIOS ver. 1.33c 20080626 Copyright (C) 2000-2008 Soekris Engineering.

net5501

0512 Mbyte Memory CPU Geode LX 500 Mhz
...