Rebel IT Mumblings of a crazy programmer…

2Nov/134

ESXi 5.5 Free Edition vSphere Client and Version 10 VMs

With the release of ESXi 5.5 comes some exciting new features.  The 32GB RAM physical memory limit has been lifted! Version 10 VMs now allow the mapping of USB 3.0 ports, larger than 2TB virtual drives, expanded vGPU features, and vSphere Flash Read Cache.  However you may run into a potential issue that can leave you in a conundrum.  After first installing the beloved vSphere Client you are presented with an interesting note,

vSphere_Client

The new version 10 virtual machine hardware requires the new vCenter web client to edit even the most basic settings. You will be completely locked out of making changes to your VM via the client with very few options left.  Keep in mind that by default the vSphere 5.5 client creates VMs using version 8 but can then be upgraded to version 10.  On 5.1 they were created in version 8 and upgraded to version 9 (vmx-09).

It seems however that they won't be changing their mind on this anytime soon.  Every step of the way you are greeted with the notice about this change.  Upgrading a VM via the vCenter client will produce this message,

VM_Upgrade_Warning

On the official VMware knowledge base for upgrading a VMs hardware version they note,

Note: In a vSphere 5.5 environment, you must use the vSphere Web Client to upgrade the virtual hardware to version 10. However, when the virtual machine is using hardware version 10, you cannot edit the settings of the virtual machine if you connect directly to an ESXi with the vSphere Client. You must use the vSphere Web Client or connect to vCenter Server with the vSphere Client. If you connect directly to an ESXi host using the vSphere Client and try to edit virtual machine settings, you see the error:

VM_Edit_Settings_Warning

There are however a few options to keep this at bay at least for a little while.

Deploying the vCenter Appliance will allow you to make the necessary modifications although there is no fully free option available.  This will at least give you 60 days to get yourself out of "accidentally" upgrading to version 10.  However prepare for another problem.  The vCenter evaluation version doesn't like to play nice with an ESXi 5.5 free license.  Trying to license a host in the web console running in evaluation version mode using a free license doesn't work either and complains that you should turn off the "vCenter agent for VMware host" .  An error when trying to add a host already using a free license returns,

License not available to perform the operation.
The host is licensed with VMware vSphere 5 Hypervisor. The license edition of vCenter Server does not support VMware vSphere 5 Hypervisor.

Another way around this is using VMware Workstation 10 (also not free) to manage your VMs by connecting to your ESXi 5.5 host.  There are some limitations though that make it more useless than helpful.  VM Boot/Advanced Options are missing, the new Host USB Device/PCI Device Passthrough cannot be added, and the Network Adapter Type/MAC Address settings are also missing when creating or editing a network adapter.

Currently the best method is to use the vCenter Standalone Converter to downgrade the VM to the desired version.  You are also given the option to use version 9 (vmx-09) again as well.  I would recommend avoiding the Upgrade Virtual Hardware option in the vSphere client all together until VMware either disable the option to upgrade straight to version 10 or they come up with a better solution for their free license users.

Test Environment:
ESXi 5.5.0 1331820
vCenter Appliance 5.5.0 1369380
vSphere Client 5.5.0 1281650
VMware Workstation 10.0.1 1379776
VMs:
Windows Server 2012 (64-bit)
Windows 7 (64-bit)
Debian GNU/Linux 6 (64-bit)
Debian GNU/Linux 7 (64-bit) *Provided by VMware Workstation 10
Filed under: VMware 4 Comments
7Apr/130

Debian Squeeze/Wheezy ZFS Storage Pool

ZFS packages are available directly from http://zfsonlinux.org/ for Debian as well as other supporting documentation.
This also requires a 64bit version of Debian.

ZFS Setup
Login as root and run the following.

wget http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_1%7Ewheezy_all.deb
dpkg -i zfsonlinux_1~wheezy_all.deb
aptitude update
aptitude install gcc make linux-headers-$(uname -r)
aptitude install debian-zfs

It will take a while to install.  Once completed reboot.

Use parted to create/format your disks for the pool.  If you don't parted the disk first you can use the -f flag during pool creation however you may end up with a small leftover partition on each drive.  Not that the size is noticeable but for completeness lets make a partition that covers 100% of the disk, , "0% 100%" specifies the full disk.

parted /dev/sdb
mklabel gpt
mkpart zfs 0% 100%
quit

Next we need to decide who should access the mount point prior to creating the pool.  If you plan on using Samba to share out the pool space and setup user authentication using groups you will need to make the mount dir and set permissions prior to creating the storage pool.  If not then when mounting the pool it will create /storage as root:root.  You could always do this later as well.

mkdir /storage
chown root:storage /storage
chmod 0775 /storage

Next we need to create a storage pool using zpool, storage is your pool name and -m /storage is your mount point.  Each disk is listed afterwards.

For testing purposes we can use the drive path, ie. "/dev/sdb1".  For production I suggest you use the disk IDs to create the storage pool.  Make sure you keep a space between each disk you are adding to the pool.  Specifying a raid type is done here as well.  The following is a list of each one,

mirror - Raid1
raidz - Raid5
raidz2 - 2 parity disks (Raid6)
raidz3 - 3 parity disks

You can also nest raid levels as well.  This option goes into the create command.  Leaving this blank creates a stripe set spanning across all added disks.

zpool create storage -m /storage /dev/sdb1 /dev/sdc1
zpool create storage -m /storage mirror /dev/sdb1 /dev/sdc1
zpool create storage -m /storage raidz /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

Finding your disk IDs,

cd /dev/disk/by-id
ls -l

This should list out all of your disk IDs with where they are pointed.

Disk by-id

If you get an error about the by-id folder being missing and are running this as a VM on ESXi, edit the VMs config (or VMX file) and add the line,

disk.EnableUUID = "TRUE"

The IDs that contain -part1 are the partitions we created.

Copy down these strings.  Using Putty if you have SSH is extremely handy here as you can copy the IDs right from the terminal.

Once we have the by-id value we need to replace the disks in the above zpool create command with the proper disk paths.

/dev/sdb1 would become /dev/disk/by-id/scsi-36000c29381ac32eb8dfb4346534ce88e-part1

Your storage pool should now be created.  You can check the status using,

zpool status

zpool status

Lets also check using the zfs command as well as verifying our mount point,

zfs list
mountpoint /storage

zfs and mountpoint

I'm using two 30GB disks so my total available storage space is 58.6GB as expected.  cd into the /storage directory and you are now in your zfs storage pool.

Next we need to handle mounting the storage pool on boot.  You can manually mount the pool using the following command,

zfs mount -a

To setup automatically mounting the pool at boot we need to edit the /etc/default/zfs settings,

nano -w /etc/default/zfs

Edit ZFS_MOUNT='no' to 'yes' and ZFS_UNMOUNT='no' to 'yes'.  Ctrl+X then y and enter to save changes.  Reboot the machine.  When it's back up check the storage pool again.

zfs list
mountpoint /storage

If everything came back the same as before the zfs pool is mounted to /storage after your reboot.

Next we need to decide how we want to mount your new storage pool for access.  If you are on a windows based environment I suggest Samba.  You can also use NFS directly from zfs to share out the pool.

Setting up Samba
If you don't have it installed yet run the following,

aptitude install samba

Once it's done installing we need to edit the config.

nano -w /etc/samba/smb.conf

Find the line security = user and uncomment it.  Just below add the following,

map to guest = Bad User

Scroll to the end of your file and create your share,

[storage]
comment = zfs share
path = /storage
browsable = yes
guest account = guest
read list = guest
write list = username
create mask = 0775
directory mask = 0775
force group = storage

Restart Samba for the changes to take effect.

/etc/init.d/samba restart

Next we need to create some users and a group to access the share.

groupadd storage
useradd guest

Remove the password from the guest account and add the user to Samba,

passwd -d guest
smbpasswd -a guest

Enter twice to use nothing as the password.  Now add your write user,

useradd -G storage username
passwd username
smbpasswd -a username

Use the password for the windows account for both the linux and Samba accounts.

Restart Samba for the settings to take effect,

/etc/init.d/samba restart

Now you should be able to login as guest and have read only permissions as well as login as your username and have write permissions.  Don't forget to add other users if you want as well.

Final Notes and Extras
You should now be able to reboot, your ZFS share should automatically mount, and Samba will share it out!

If you have any scripts that create folders if not found you should modify them to wait for the ZFS share to mount.  If you don't your mount will fail and your storage pool mount point will appear empty or have the generated files in it.  Don't panic!  The files are all still located on the storage pool, the mount just failed because another script created it as a folder instead.  To verify run the following,

mountpoint /storage

If it does not return as a mount point you will need to figure out what is causing the folder to be generated before the storage pool mounts.  Many scripts can cause this.  A simple fix is to sleep the startup of the script until the mount point is returned as mounted.

start_service() {
  if mountpoint -q /storage; then
      echo "ZFS mountpoint found: Starting servicename"
      *start service code
  else
      echo "ZFS mountpoint not found: Sleeping $DESC"
      sleep 10
      start_service
  fi
}

Depending on how the script is written you may need small modifications or large modifications but the principle is all the same.  Check for the mount point using the following,

if mountpoint -q /storage; then
     start
else
     sleep
fi

Your script should pause and wait for the ZFS storage pool to mount before starting and giving you a heart attack.

You can also mount the zfs pool using /etc/fstab to avoid the above loop.  Simply set your pool to legacy mode using the following,

zfs set mountpoint=legacy storage

Then you need to set your mount in /etc/fstab,

nano /etc/fstab

Insert the following line at the bottom,

storage  /storage  zfs  defaults  0  0

Adding Disks to the Pool

Adding disks to the storage pool after creation is a simple process.  Start by preparing your disk with parted like we did when creating the pool.  Once done find the ID of the disk.

zpool add -f storage /dev/disk/by-id/scsi-36000c29b4f7f2f3a8f6373f36c9fee85-part1

Verify the disk was added successfully,

zpool status

zpool status add

Check again using zfs to verify the increase is pool size,

zfs list

zfs list add

That's all there is to it.  Enjoy the added storage space!

Moving a Pool
Before moving the pool export it,

zpool export storage

You may need to force it if errors are returned.  I suggest rebooting first and trying the export again before forcing it with the "-f" flag.  Once moved after installing ZFS import it,

zpool import storage

This is handy if the host machine needs re-installed for whatever reason.

Maintenance Tasks
You should also setup some maintenance tasks on your storage pool to check for problems.  The following cron job can be created to scan the pool.

crontab -e

Add the following line,

50 23 * * 6 /sbin/zpool scrub storage

This will run a scrub on the pool every Saturday at 11:50 pm.  Adjust to your needs.  Take note that this will increase disk usage so you should scheduled it for a time when no one will notice.

Update Linux Kernel
spl-dkms and zfs-dkms are compiled for your current kernel build if you update you will need to reconfigure.

dpkg-reconfigure spl-dkms
dpkg-reconfigure zfs-dkms
Filed under: Linux No Comments
23Mar/130

ESXi 5.x SSD Host Cache Configuration “Swap to Host Cache”

A new feature of ESXi 5.x is the ability to use an SSD Datastore to swap to a faster medium than a standard HDD when memory overcomitting.  This is a second to "last hit" method when your host starts to run out of available memory.  The VMs swap file does not reside directly on the SSD.

ESXi uses five memory management mechanisms to handle overcommitting memory.  Page Sharing, Balooning, Memory Compression, Swap to Host Cache, and Regular Swapping.  Instead of directly swapping to disk the ESXi host will use the SSD in between for faster swapping.  Yea can read all about it in the Performance Best Practices for VMWare vSphere 5.1 document from VMWare.

To set this feature up connect an SSD to your system.  Depending on the connection extra steps may need to be taken to get vCenter to see it as an SSD.

Select the Configuration tab in vCenter and under Software to the left is a new option "Host Cache Configuration".  If you have already created the datastore on your host your drive should be listed.  If not select Add Storage and create a new datastore selecting the SSD.

Here is where you may get stuck if vCenter cannot recognize the new datastore as being an SSD.  To work around this we need to connect to the host via SSH.  To start SSH select Security Profile just above Host Cache Configuration.  Under services select Properties.  Select SSH and click the Options button.  Click start and OK your way back to the main vCenter window.

SSH into your ESXi host and run the following command,

esxcli storage nmp device list

In the returned results you should see all of your attached drives.  Match the drive with the naa id from the vCenter GUI under Storage.

vCenter Datastore NAA ID

ESXi nmp device list

Take note of the "Storage Array Type:" as you will need this in a moment.  List out this device in further detail using the following command replacing your naa id,

esxcli storage core device list -d naa.6001e4f01550370018dff84814c293a6

In the returned section you should find "Is SSD:"

SSD Details

To fix this we need to setup a rule to trick vCenter into seeing it as an SSD.  This should work for any type of drives you are trying to use.

esxcli storage nmp satp rule add -s VMW_SATP_LOCAL –d naa.6001e4f01550370018dff84814c293a6 –o enable_ssd

Change the section after -s to match your drive as well as inserting your own naa id.  To verify the rule was added run the following command,

esxcli storage nmp satp rule list | grep enable_ssd

Now reclaim your drive to apply the new rule,

esxcli storage core claiming reclaim -d naa.6001e4f01550370018dff84814c293a6

Note: If you get an error trying to reclaim the datastore check to make sure that the ESXi host did not automatically use it for other functions such as Datastore Heatbeating.

Reload and run the claim rules,

esxcli storage core claimrule load
esxcli storage core claimrule run

List out the device again and the "Is SSD:" should be set to true now.  You may need to refresh the Host Cache Configuration screen to get the drive to show up properly.  Once you see your datastore right click and select properties.

ESXi SSD Host Cache Setup

Select the size you want to allow from the datastore for host swap caching and click OK.  You should now see the drive use up the select space and fill it full of vswap files.  If not click refresh and it should update.

Browsing the datastore we see a new folder and a 1GB swp file.  Inside the folder should be a number of 1GB vswap files depending on how big you allocated the datastore.

SSD Cache Drive Datastore

SSD Cache Drive Datastore 2

 

 

Credits to lamw here for the initial posting on this new feature.

Filed under: VMware No Comments
15Dec/120

Debian 6 VMWare info: mpt raid status change

If you are running Debian as a guest your regular user is probably getting spammed with mails about the mpt raid status.

From r...@debian.local Sat Dec 06 19:39:42 2012
Envelope-to: r...@debian.local
Delivery-date: Sat, 06 Dec 2012 19:39:42 -0600
Date: Sat, 06 Dec 2012 19:39:42 -0600
To: r...@debian.local
Subject: info: mpt raid status change on debian
From: root <r...@debian.local>
This is a RAID status update from mpt-statusd. The mpt-status
program reports that one of the RAIDs changed state:
Report from /etc/init.d/mpt-statusd on debian

There will also be errors in /var/log/messages.  For some reason mpt-statusd is installed even though the guest is running in a non raid environment.  To fix this issue run the following commands,

/etc/init.d/mpt-statusd stop

update-rc.d-insserv -f mpt-statusd remove

You should no longer get these messages.

Filed under: Linux No Comments
4Dec/120

Installing/Updating VMWare Tools on Debian

In order to gracefully shutdown Debian running on an ESXi host you must have VMWare Tool installed and running otherwise the guest will hard power down during the shutdown process.

First make sure the Debian guest is up to date,

aptitude update

If any updates are available install them using the following command,

aptitude safe-upgrade

You also need to have gcc, make, and the kernel headers installed/updated in order for the install to go smoothly,

aptitude install gcc make linux-headers-$(uname -r)

Next edit the guest and mount the VMWare Tools ISO located on the host.  It should be located under Datastores\vmimages\tools-isoimages.  Select the linux.iso and make sure that you have the Connected check box checked.  This can be done after the guest has already been booted up.

Configure the CD/DVD drive settings.

If you don't have SSH installed on the guest I suggest you do so.  This will allow you to copy/paste commands to get this completed.  To do so run the following command,

aptitude install ssh

Once installed SSH into the machine and login as root.

Next we need to mount the VMWare Tools CD so that Debian can access it.

mount -t iso9660 /dev/cdrom /media/cdrom

You should see a mounting read-only message, this is normal.

Next we need to get the file name of the compressed VMWare Tools archive from the iso.  Browse to the mounted drive,

cd /media/cdrom

To get the archive name simply run the following command,

ls -l

You should see something similar returned,

If using SSH highlight and copy the file name or write it down and save it for the next step.

Browse to the /tmp folder and extract the archive,

cd /tmp

tar -zxf /media/cdrom/VMwareTools-9.0.0-782409.tar.gz

Next run the VMWare Tools intaller,

/tmp/vmware-tools-distrib/vmware-install.pl

It should ask you several questions.  Most of these can be left as default.  Once the basic settings are configured it will ask to run the vmware-config-tools.pl script.  Press Enter to do so.

If you wish to use VMWare FileSystem Sync Driver press Y and press Enter.

After this step you may encounter a question asking to locate gcc on the system.  If you get the following,

The path "" is not valid path to the gcc binary.

You are missing gcc and skipped the install step at the beginning of the post.  You probably missed installing make and the kernel headers as well.

Ctrl+C out of the install and run the following,

aptitude install gcc make linux-headers-$(uname -r)

Let the above install and once completed run the setup again,

/tmp/vmware-tools-distrib/vmware-install.pl

Most of the beginning questions will be skipped this time since we already answered them earlier.  Keep pressing enter until you get back to the vmware-config-tools.pl.

If you installed gcc before running the installed you should see the following,

The path "/usr/bin/gcc" appears to be a valid path to the gcc binary.

No is the default option since it does not need any changes, press Enter unless you installed gcc elsewhere.

Press Enter to the kernal headers question.  If you get an error about being unable to locate make see above.  Press Enter again.  If you get an error about being unable to locate the kernel headers see above.  Press Enter again.

The script will now begin compiling.  It should only take a few moments.  Once done you will be asked if you want to use the VMWare Host-Guest Filesystem.  Answer accordingly.  Default is no.

Next it will ask if you want to use vmblock.  Again answer accordingly.  Default is no.

It should then ask if you want to enable VMWare automatic kernel modules.  This is experimental.  Leave this as no, press Enter.

A few more things should run.  If you are using X it will configure it and will need restarted when the script has completed.  The final setup directions should be listed at the end of the script.  Take action based on your setup.  If you are running strictly command line you can ignore them.

To verify that it was installed successfully you can load the vCenter client and select the guest.  Click the Summary tab and you should see the following under General,

All that is left is cleanup.  Unmount the VMWare Tools iso,

umount /media/cdrom

You should still be in /tmp.  Delete the extracted files,

 rm -fr vmware-tools-distrib

That should be it!  Reboot the guest and make sure it is still running once it comes back up.  Now when running shutdown commands to the host if configured properly to power down the VMs Debian will gracefully shutdown.

Filed under: Linux, VMware No Comments
3Dec/128

ESXi 5.1 Dell PERC 6i Health Status Monitoring

Monitoring the Dell PERC 6i on an ESXi 5.1 host is much the same as monitoring on a 5.0 host.  There are however a few changes to get things working properly.  LSI has also released their latest Offline VIB bundle 00.32.V0.02.  During testing on my hosts I found that while it does allow you to monitor the PERC 6i Health Status it shows some discrepancies between vCenter and the cards Raid Manager.  Most notably all of my drives in ONLINE status were now showing as UNCONFIGURED GOOD and the Virtual Disks were no longer showing the list of attached drives.  This happened on both 5.0 and 5.1 hosts with a mix of PERC 6i cards running either firmware 6.3.0-0001 or 6.3.1-0003.  I was still able to boot my test VMs and view the contents of the datastore without issues but it defeated the purpose of monitoring if it's not going to show the proper drive status.  To fix this I removed the VIB bundle from the host and reinstalled the previous version I had saved in the datastore.  You can get the older versions from lsi.com located here (500.04.V0.24 or 500.04.V0.30).

You will need VMware vSphere CLI installed on a machine.  The most notable changes to ESXi 5.1 and the new vSphere CLI 5.1 are the commands to shutdown/restart the host and put it into Maintenance Mode.  It also seems that the last set of updates for 5.0 disabled the ability to use the vicfg-hostops command to put the host in Maintenance Mode on the free version of ESXi.  Previous to the updates I had no issues using the commands.  Since vCLI is supposed to be disabled on the free version the fact that it worked prior to the updates seems to have been a bug.  The newer esxcli commands do work on the free version using ESXi 5.1 and CLI 5.1 however.

The update requires Maintenance Mode and a host reboot so if you are using a vMA make sure it's on another physical host. Using CLI on my Windows desktop machine I first copied the extracted VIB file to the root of the ESXi host datastore via the vSphere Client. Then on the machine with CLI installed I opened command prompt and browsed to the folder C:\Program Files (x86)\VMware\VMware vSphere CLI\bin.

I put the ESXi host in maintenance mode using the following command replacing x.x.x.x with your hosts IP Address,

esxcli -s x.x.x.x system maintenanceMode set -e yes

Note: If you get any weird errors returned try typing the command rather than copy/paste.

After the server was in maintenance mode I verified the status by running the following command,

esxcli -s x.x.x.x system maintenanceMode get

Once the host was in maintenance mode I ran the following command to install the VIB bundle,

esxcli -s x.x.x.x software vib install -v /vmfs/volumes/datastore/vmware-esx-provider-lsiprovider.vib --no-sig-check

If you are using the offline bundle the command is slightly different,

esxcli -s x.x.x.x software vib install -d /vmfs/volumes/datastore/VMW-ESX-5.0.0-LSIProvider-500.04.V0.24-261033-offline_bundle-456178.zip

Notes: If there is spaces in your datastore name you will need to encase the vib/zip path in quotes.  Take note of the slightly different switch -v to install the VIB file instead of the normal -d to use the zip file as the depot.  If you use the -d flag against the VIB bundle instead you will see a MetadataDownloadError.  Also note that the --no-sig-check flag is required otherwise you will get unsigned errors when trying to install the VIB bundle.  The older offline bundle doesn't require this flag.

When running the command and supplying credentials CLI sat at a flashing cursor for a few minutes. If it's going to throw an error it will do it right away, otherwise it's installing and you should leave it alone. There are no status updates until the install has completed.  According to VMWare if you manage to Ctrl+C out of the command it will still continue to run to completion on the host.  I would avoid touching anything during the update process just to be on the safe side.

Once the install was complete the following was returned,

Now you need to restart the ESXi host in order for the changes to work. You can also do this with CLI running the following command,

esxcli -s x.x.x.x system shutdown reboot -r "Reboot Reason"

After the host was done rebooting I logged in with the vSphere Client and checked the Health Status. It took around 5 minutes for the sensors to pull their data and I was then able to view my Health Status.  You can reset the sensors to speed up the process.  The host now shows the Storage category and displays all of the information related to my Dell PERC 6i including battery status.

Remove the host from maintenance mode,

esxcli -s x.x.x.x system maintenanceMode set -e no

I then powered all of the test VMs on without any issues.

You can also check via esxcli if the VIB was installed properly,

esxcli -s x.x.x.x software vib list

Generally the lsiprovider VIB shows at the top of the list,

Note: The eAccepted status on the VIB bundle as opposed to the eCertified from the --no-sig-check flag used earlier.  The older offline bundle will list as eCertified.

If you mess up or something goes wrong you can remove the VIB bundle by name.  Make sure it's installed using the above list command and then run the following while in Maintenance Mode,

esxcli -s x.x.x.x software vib remove --vibname=name

In this case the name of the VIB bundle is "lsiprovider" so you would use the command,

esxcli -s x.x.x.x software vib remove --vibname=lsiprovider

Once it has completed the host must be rebooted.  I reboot twice when changing the VIB bundles to make sure that the secondary bootbank doesn't take over.  List out the installed VIBs again to make sure it was removed after the reboot.  I wouldn't suggest removing and installing a VIB bundle without at least a single reboot in between to keep from causing further issues.

If you have any questions please feel free to let me know!  I've done this on around 6 hosts now without any issues.  I hope this helps any users out there upgrading with a PERC 6i RAID controller that want to retain the ability to monitor their storage array.

Filed under: VMware 8 Comments
24Sep/122

Fail2ban to DD-WRT permanent ban after X attempts

Using Fail2ban to block brute force login attacks can be helpful but what happens if it's just an authorized user who forgot their password?  Setting the ban time to -1 blocks them permanently but if you block for a small time frame it allows them back in without a permanent ban. If you want to permanently ban the hackers and script kiddies after say 3 bans you can setup Fail2ban to monitor it's own log file and send the IP address to your DD-WRT router to block them before they even get in.

First download the attached file at the end of the post.  Using SCP or WGET transfer the package to your Linux machine. I used /home/username to avoid permission issues.

Browse to the folder you transferred the files to and extract fail2ban2ddwrt.tar.gz from the console window.

tar -xzvf fail2ban2ddwrt.tar.gz
 cd fail2ban2ddwrt

There are three files in the archive, the filter.d/fail2ban.conf filter file, the action.d/fail2ban.conf ban action file, and the script to send the banned IP to the DD-WRT router via SSH.

Move the two fail2ban.conf files into their respective folders and set owner/permissions,

mv filter.d/fail2ban.conf /etc/fail2ban/filter.d/fail2ban.conf
mv action.d/fail2ban.conf /etc/fail2ban/action.d/fail2ban.conf
chown root /etc/fail2ban/filter.d/fail2ban.conf
chown root /etc/fail2ban/action.d/fail2ban.conf
chmod 644 /etc/fail2ban/filter.d/fail2ban.conf
chmod 644 /etc/fail2ban/action.d/fail2ban.conf

Now create a copy of the jail.conf file if you have not already,

cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
nano -w /etc/fail2ban/jail.local

Find the jails section and add the following,

[fail2ban]

enabled = true
filter = fail2ban
action = fail2ban
logpath = /var/log/fail2ban.log
maxretry = 3
# Find-time: 1 day
findtime = 86400

Adjust your settings as needed.  The above will run the Fail2ban action after 3 previous bans in a 24 hour period.

Next you will need to configure the script to send the IP to your DD-WRT router.  This requires that SSH management be enabled on your router.

Under the Services tab -> Services, toward the bottom of the page you will find the option to enable SSH management.

DD-WRT SSHd Option

Also make sure you have do not have SSH remote access enabled unless you set it that way intentionally,

Administration tab -> Management, toward the top of the page you will find the option to enable/disable remote management.

DD-WRT Remote Management Options

Once you have your settings changed you can test ssh access via linux command line using the following,

ssh xxx.xxx.xxx.xxx

Now that SSH is enabled we need to install sshpass,

aptitude install sshpass

Now that we have everything in place edit the rban.sh file to enter your router credentials,

nano -w rban.sh

Enter your routers username, password, and IP Address, save the file using Ctrl+X, and press Y to save changes.  Now move the rban.sh file into it's proper place and change owner/permissions,

mv rban.sh /etc/fail2ban/rban.sh
chown root /etc/fail2ban/rban.sh
chmod 700 /etc/fail2ban/rban.sh

Once you have the two config files moved, the jail added to jail.conf, and the rban.sh file edited and moved restart Fail2ban.

/etc/init.d/fail2ban restart

Now when someone tries to brute force their way into your server they get banned for the default application setting, ex. 15 minutes, and after 3 bans the rban.sh script connects to your router and adds their IP to the firewall script.  It gathers the current firewall list and adds the IP to the end of it without removing any firewall settings you may already have.  The nvram is then committed so changes are saved even if the router is rebooted.

Fail2ban2DDWRT
Filed under: Linux 2 Comments
19Sep/1118

VMWare ESXi 4.x/5.x Health Status Email Alert Monitoring

While looking for a good solution to monitor the storage array on my Dell PERC 6i RAID cards I came across a simple script that sends an email alert based on the hosts health status. Only having a few ESXi whitebox servers and no access to a vCenter server to manage my hosts this seemed like the perfect solution.  When setup properly the script will only send an email when something has changed since the last alert report was generated.  This allows me to keep a constant eye on my servers health without having to login to the vCenter Client on a regular basis.  I put together a bash script to allow the changing of options and use crontab to execute the script every hour on a vMA I already have setup for PCNS to monitor all of my whiteboxes.  This should work on both paid and free license ESXi servers.

Note: The script only reports what is listed in the Health Status screen in the vCenter Client.  If the item does not showing in Health Status it will not show in the email alert or be monitored by the script.  CPU usage, Memory usage, and Datastore space remaining should always show if they are above their set parameters in the script.

First download the attached files at the end of the post and upload them to the vMA using SCP or WGET.   I used /home/vi-admin to avoid permission issues.

Connect to the vMA using a telnet client and browse to the folder you transferred the files to.  Run the following commands,

sudo mkdir /opt/ESXiH
sudo cp ESXiH.sh /opt/ESXiH
sudo cp esx-health.pl /opt/ESXiH

Now set the proper permissions and ownership for the files,

sudo chmod 700 /opt/ESXiH/ESXiH.sh
sudo chmod 700 /opt/ESXiH/esx-health.pl
sudo chown root /opt/ESXiH/ESXiH.sh
sudo chown root /opt/ESXiH/esx-health.pl

Once the files are in place edit your settings to your linking,

sudo vim /opt/ESXiH/ESXiH.sh

Press "i" for insert mode and edit the setting to how you want them.  You can find an explanation of each switch here or in the ESXiH.sh file itself.  Once you have everything set press "Esc" to exit insert mode, type ":wq", and press Enter to save and quit.

Now setup a cron job to run the script on a regular basis,

sudo crontab -e

I set mine up for every hour and to output the results to the run.log file.  Again press "i" to enter insert mode.

0 * * * * /opt/ESXiH/ESXiH.sh>/opt/ESXiH/run.log

Once you have your crontab command entered press "Esc" to exit insert mode, type ":wq", and press Enter to save and quit.  You can test by setting the line to,

* * * * * /opt/ESXiH/ESXiH.sh>/opt/ESXiH/run.log

Which will run the script every minute.

That should be it.  If you set it up using the "warnonchange" flag the script will not email you when it runs the first time or anytime after until there is a change in the health status on the host.  If you want to test the email alerts right away change "warnonchange" to no and run the ESXiH.sh file from command line or wait for cron to execute the script again.

sudo sh /opt/ESXiH/ESXiH.sh

SMTP Authentication

To add SMTP Auth to the script you will need to first download Authen::SASL from below,  Note: this is not fully tested as my mail host accepts relay with and without SMTP Auth.
http://search.cpan.org/~gbarr/Authen-SASL-2.15/lib/Authen/SASL.pod - Link is on the right side of the page.

Transfer to your vMA using WGET or SCP and extract,

tar -xvf Authen-SASL-2.15.tar.gz

Browse to the extracted folder and run,

sudo perl Makefile.PL
 sudo make
 sudo make install

Now edit the esx-health.pl file and at line 58 add,

use Authen::SASL;

Right below "use Net::SMTP;".  Then at line 696 right above "$smtp->mail($EMAIL_FROM);" but below the "}" add,

$smtp->auth ( "smtpauthuser", "smtpauthpass" ) or die "Could not authenticate $!";

Put your smtp username and password in and save.

Monitoring Multiple Hosts

If you want to monitor multiple hosts the easiest way is to make folders inside the ESXiH folder for each host.  Copy the ESXiH.sh and esx-health.pl script into each folder.  Edit the settings for each host in their respective folders.  Edit the ESXiH.sh files in each folder and add their folder names to the last two lines of the file,

cd /opt/ESXiH/HERE
 /usr/bin/perl /opt/ESXiH/HERE/esx-health.pl ...

Add a line for each script to the crontab and save.  For my three hosts I have,

0 * * * * /opt/ESXiH/S01/ESXiH.sh>/opt/ESXiH/S01/run.log
 0 * * * * /opt/ESXiH/S02/ESXiH.sh>/opt/ESXiH/S02/run.log
 0 * * * * /opt/ESXiH/S03/ESXiH.sh>/opt/ESXiH/S03/run.log

Update (03/23/13): It appears that the latest 5.x updates require some modification to the script process otherwise the following error is produced,

Server version unavailable at 'https://10.0.0.1:443/sdk/vimService.wsdl' at /usr/lib/perl5/5.10.0/VMware/VICommon.pm line 545.

The universal fix for any of the perl scripts producing this error is to add the following line to the top of whatever script is causing you problems,

$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;

This should get you past the error if you are getting it.  I have also updated the download to include the fix.

ESXi Health Monitoring Script

Credits to William Lam for the script and to James Pearce for the modifications.

Filed under: VMware 18 Comments
15Sep/1171

VMWare PCNS ESXi 5.x Free License shutdown with APC PCNS 3.0.1 and vMA 5.x

For PCNS 3.1 users please see the this comment.

If you are running ESXi 4.1 and are using vMA 4 please see here.

A warning about ESXi 5.0 Update 1.  There is a known issue with the autostart and autoshutdown feature for free license users.  This severely impacts the ability to properly use PCNS on your VMs.  A patch was released to resolve the issue and details can be found here.

To get SOAP functionality first download the latest vMA virtual machine from the VMWare website.  Currently the latest version is vMA-5.1.0.0-782391.

Once the OVF template has been deployed, boot the vMA machine, setup the network settings, and change the vi-admin password from the console window.

Note: vMA 5.x has stronger password requirements that require a non alphanumeric character.  If you want to change the password to something less complex set a password during setup and once the vMA finishes loading run the command "sudo passwd vi-admin".  You will still get the error "BAD PASSWORD: is too simple" but it will still allow you to change the password and override the requirements.

Login to the vMA using vi-admin as the user and the password you just made.  Reboot the vMA for any hostname changes to take effect.

sudo reboot

Enter your password and wait for the vMA to finish the reboot.  At this point I suggest you close out of console and use an SSH client to connect to the vMA so you can copy/paste commands and avoid mistakes.  If you are unable to connect to the vMA via SSH or SCP check your hostname configuration.

Download the ESXi PCNS 3.0.1 package from the APC website. You can find it by going to their website and selecting the Software & Firmware link under Popular Links to the left of the page. Once the page loads select the "Software Upgrades - PowerChute Network Shutdown" from the top drop down and click submit. Under the first 3.0.1 section click the download button to the right. Select the download button for the ESXi package, click the next download button, login to the APC site if you want to receive update notifications, or click Download to get the package.

Using SCP or WGET transfer the ESXi PCNS 3.0.1 package to the new vMA machine. I used /home/vi-admin to avoid permission issues.

Browse to the folder you transferred the files to, extract pcns301ESXi.tar.gz from the console window, and browse into the directory.

tar -xzvf pcns301ESXi.tar.gz
cd ESXi

Run the install file,

sudo ./install_en.sh

Enter your vi-admin password and agree to the license terms.

Press Enter for default install path.

Type Yes and press Enter to install.

Now press Enter to install the included java package.

Next it should ask for the IP and credentials of your ESXi.  You can press "q" to skip.  If you put your IP in it will still work but you will get an error that it wasn't able to add and that you should manually add your host using vifp addserver. This is for vCLI functionality only and is unimportant when using SOAP.  Just ignore it.

Edit the shutdown.pl file attached to the end of this post and enter your ESXi host credentials.  If using a different user than root make sure they are a member of a role that has both the host and guest power options.
Host: All Privileges -> Host -> Configuration -> Power
Guest: Virtual machine -> Interaction -> Power Off

Use SCP or WGET to transfer the shutdown.pl and shutdown files to the vMA machine.

Still connected the the terminal change to the directory where you transferred the files, if same location as the PCNS package use,

cd ..

Copy the shutdown.pl and shutdown files to the default setup of PCNS,

sudo cp shutdown.pl /opt/APC/PowerChute/group1/bin
sudo cp shutdown /opt/APC/PowerChute/group1/bin

Create a host file and add the ESXi host IP(s).  For multiple hosts use new lines for each IP.  All hosts must use the same username and password.  If this is not an options please see the more complicated option at the bottom of the post.

sudo vim /opt/APC/PowerChute/group1/bin/host

Press "i" to enter insert mode, type the IP address in, press Esc to exit insert mode, and type :wq to save and quit.

Contents should look similar,

10.0.0.4
10.0.0.5

Make sure the files are owned by root and set chmod 0744.

sudo chown root /opt/APC/PowerChute/group1/bin/shutdown
sudo chown root /opt/APC/PowerChute/group1/bin/shutdown.pl
sudo chmod 744 /opt/APC/PowerChute/group1/bin/shutdown
sudo chmod 744 /opt/APC/PowerChute/group1/bin/shutdown.pl

Next configure the ESXi shutdown order. Access the vSphere client, select the ESXi host, then the configuration tab.

Click on the Virtual Machine Startup/Shutdown link. Click properties at the top right and use "Move Up" to move the VMs in the order you want leaving the vMA VM at the top. Note that the startup order is the only one displayed. Shutdown uses the startup list in reverse. If you want the VMs to start on their own move them all the way up to Automatic Startup still leaving the vMA VM at the top.

ESXi Guest Shutdown Settings

Change the Shutdown action at the top right of the window to "Guest Shutdown" and click OK.  Several times after clicking OK I had duplicates of my VMs in the list.  A simple refresh (F5) cleans up the list.

Next we need to install VMWare Tools on each VM for the graceful shutdown to occur.  Once installed edit VM to make sure its settings for shutdown are correct. Right click the VM -> Edit Settings -> Options Tab -> VMWareTools.  Under Power Controls Stop should be set to "Shut Down Guest".

ESXi VM Shutdown Settings

Connect to the IP of the vMA machine ex. https://10.0.0.9:6547/login and setup your connection and events as needed. Make sure you remove the check box in the "Turn off UPS" box under Configure Shutdown unless you have the "Restart when power is restored" option set on the APC.  To fire the script configure the UPS: On Battery event by clicking the Shut Down System box (not Run Command File).  Check the box to shut down the PCNS operating system and enter the time in seconds before the event fires.

To test you can either unplug utility power to your APC and use the "UPS: On Battery" event or run the following command from the vMA console window,

cd /opt/APC/PowerChute/group1/bin
sudo ./shutdown.pl host

This will gracefully shutdown the VMs in the order you set before telling the ESXi host to power down. You should see the shutdown commands in the log start when connected to the ESXi host with the vSphere client. If the VM fails to gracefully shutdown check to make sure VMWare Tools is running on the VM under its Summary tab in vCenter.

VMWare Tools Running on Debian

As a side note make sure you give each VM ample time to power off before your battery power runs out.

If you are having problems with the virtual machines not shutting down gracefully please see this VMWare article here.  Also a note that PCNS and most management cards do NOT accept special characters in the password.  Please see this article on the APC site for details.

If you have multiple hosts and having the same username and password across the board is not an option the shutdown file will need to be modified.  You will need to add a new line as well as copies of the shutdown.pl and host files to point to each host.  As an example,

perl ./bin/shutdown.pl ./bin/host
perl ./bin/shutdown2.pl ./bin/host2

In the example host #1 uses root and abc123.  The shutdown.pl file will need these credentials entered in and host #1's IP will need to be entered in the host file.  Host #2 uses root and dog1234.  The shutdown2.pl file will need host #2's credentials and the host2 file will need host #2's IP Address.  Running each shutdown.pl file against 1 host file with all the IPs for all of your hosts is not suggested as it generates bad password attempts on the hosts and can cause the shutdown to timeout.

shutdownPCNS3.0.1.zip

Credits to lamw for the SOAP script here.

Filed under: VMware 71 Comments
14Sep/1146

ESXi 5.0 Dell PERC 6i Health Status Monitoring

After upgrading my ESXi whitebox server using the official ESXi 5.0 install DVD I noticed that the health status monitoring for my PERC 6i RAID card was not showing up anymore. Everything else went smoothly during the upgrade and the test VMs all powered on from the datastore on the PERC 6i without issues. When checking health status only the processors and software components were listed. As it turns out VMware has removed the vendor specific VIBs for health monitoring in ESXi 5.0.

In order to restore health monitoring for the PERC 6i to the health status screen you will need the latest LSI offline bundle VIB for ESXi 5.0. I tried using the Dell OpenManage offline bundle but it stopped displaying all monitoring after the reboot and the system would not reset the sensors. After removing the installed OpenManage VIB and after a few hours of scouring the internet I managed to find the solution. The Dell PERC 6i cards use the LSI MegaRAID chipset for their controller.

LSI’s latest offline bundle package supports a variety of cards. After finding the proper version (500.04.V0.24) I was able to locate the download on one of the other controller card pages. Doing a search for "LSI 500.04.V0.24 site:lsi.com" on Google brought up several results. I selected the first result for the MegaRAID SAS 9260CV-4i - LSI and scrolled down to the Management Tools section. Here you will find VIB downloads for 4.x and 5.x. Download the file for ESXi 5.x from any of the listed card pages. You will need to extract the offline bundle from the archive otherwise it will not install and you will get errors about being unable to load the index.xml file.

You will need VMware vSphere CLI installed on a machine. The update requires maintenance mode and a host reboot so if you are using a vMA make sure it's on another physical host. Using CLI on my Windows desktop machine I first copied the extracted offline bundle zip to the root of the ESXi host datastore via the vSphere Client. Then on the machine with CLI installed I opened command prompt and browsed to the folder C:\Program Files (x86)\VMware\VMware vSphere CLI\bin.

I put the ESXi host in maintenance mode using the following command,

vicfg-hostops.pl -server x.x.x.x -operation enter

Note: Several times CLI returned connection errors or said that operation is a mandatory argument. I found that pasting the command was the culprit and manually typing in each command solved the issue. Also note that the VMs must be powered off to enter maintenance mode.

Update: If you have the latest patches installed on your ESXi 5.0 host you may get a cryptic error when trying to put the host in Maintenance Mode from CLI,

Fault string: fault.RestrictedVersion.summary
Fault detail: RestrictedVersionFault

This appears to be a bug fix done to fully disable CLI on free hosts.  Use the vCenter client to put the host in Maintenance Mode.

After the server was in maintenance mode I verified the status by running the following command,

vicfg-hostops.pl -server x.x.x.x -operation info

Once the host was in maintenance mode I ran the following command to install the vib offline bundle,

esxcli.exe -s x.x.x.x software vib install -d [datastore]VMW-ESX-5.0.0-LSIProvider-500.04.V0.24-261033-offline_bundle-456178.zip

When running the command and supplying credentials CLI sat at a flashing cursor for a few minutes. If it's going to throw an error it will do it right away, otherwise it's installing and you should leave it alone. There are no status updates until the install has completed.

Once the install was complete the following was returned,

Now you need to restart the ESXi host in order for the changes to work. You can also do this with CLI running the following command,

vicfg-hostops.pl -server x.x.x.x -operation reboot

After the host was done rebooting I logged in with the vSphere Client and checked the Health Status. It now shows the Storage category and displays all of the information related to my Dell PERC 6i including battery status.

I removed the host from maintenance mode and powered all of the test VMs on without any issues. I hope this helps any users out there upgrading with a PERC 6i RAID controller that want to retain the ability to monitor their storage array.

Filed under: VMware 46 Comments