sysinfo

If anyone sees something I’ve left out, and I’m sure there’s a lot, please leave a comment.

The script sends an email to the alias diskusage. You can easily change that to an actual email address if you like.

Limitations of the script (minus indents which seem to have been lost by the editor):

• It only checks usage for ext3 file systems, but that’s easy to change
• At the moment the script uses the same thresholds for all file systems – maybe I’ll change that in a future version

Anyway, here’s the script:

#!/bin/sh
#
# Check disk usage and if over specified thresholds send an email
#
# Author: Me
# Date: 20071105
#
 
while true
do
  df -t ext3
  sleep 60
done | awk '
function SendEmail(HostName, FileSystem, PercentUsed)
{
  Message="File system " FileSystem " on " HostName
  Message=Message " is " PercentUsed "% full"
  Subject="Low Disk Space Alert for " HostName
  SysCommand="echo " Message "| mail -s \"" Subject
  SysCommand=SysCommand "\" diskusage"
  # Send the email and return the status of the system call
  return system(SysCommand)
}
 
function CheckUsage(FileSystem,PercentUsed,Threshold)
{
  OverThreshold="no"
  if ( PercentUsed > Threshold ) {
    OverThreshold="yes"
    if ( FirstTime[FileSystem,Threshold] == "no" )
      next
    else {
      if ( ! SendEmail(HostName, FileSystem, PercentUsed) ) {
        # If we had no errors sending the email then dont send
        # any more alerts
        print "yes"
        FirstTime[FileSystem,Threshold]="no"
      }
    }
  } else {
    # Make sure we toggle to send alerts in future
    FirstTime[FileSystem,Threshold]="yes"
  }
  # If we are over the threshold return "yes"
  return OverThreshold
}
/^\// {
  FileSystem=$6
  # Remove the % sign and convert to numeric
  PercentUsed=substr($5,1,length($5)-1) * 1
  # Check if over the threshold. Check the highest
  # threshold first, there is no point in checking the
  # lower threshold if the the higher one is exceeded
  if (CheckUsage(FileSystem,PercentUsed,99)=="no")
    if (CheckUsage(FileSystem,PercentUsed,95)=="no")
      CheckUsage(FileSystem,PercentUsed,90)
} ' HostName=$HOSTNAME

Linux is not Windows. You should not need to reinstall the operating system to restore it. This document describes a way to boot a Linux system from the Red Hat install CD so it can then be restored from a CommVault backup. It does not document how to do the actual restore within CommVault. It was tested on a system running Red Hat Enterprise Linux AS 3 Update 9 and CommVault 7 with Service Pack 1.

This document only examines how to recover the operating system residing on local disks. It does not consider devices that might need additional kernel modules to be loaded. Nor does it consider external file systems such as those mounted via NFS or from a SAN. However, in the latter case that should not be an obstacle – it should be possible to restore the OS and then, using that, restore everything else.

Assumptions

• The system to be restored is completely gone – it’s OK to overwrite the old disks
• The hardware is identical
• The server is using Grub
• The local disks can be accessed using a Red Hat install CD without additional kernel modules being needed

Prerequisites

You need the following:

• Details of all partitions, volume groups, logical volumes and file systems. Also keep a copy of /etc/fstab
  • Use the commands:
    • fdisk –l /dev/sda
    • vgdisplay –v
    • pvdisplay /dev/sda5

• Network details including hostname, IP addresses, default gateway, contents of /etc/hosts and /etc/resolv.conf.
  • Use the commands:
    • ifconfig –a
    • cat /etc/hosts
    • cat /etc/resolv.conf
    • netstat –rn
    • hostname

• A backup of the CommVault client
  • I included the following in my backup of the CommVault client:
    • /etc/CommVaultRegistry
    • /etc/rc.d/rc3.d/S99Galaxy
    • /etc/rc.d/rc3.d/K99Galaxy
    • /var/log/galaxy
    • /opt/galaxy
  • If you don’t have a copy of the client but it’s backed up in CommVault try restoring it to a different location and then copying it across.
  • I was unable to install the CommVault client on to a machine that booted from the Red Hat CD. If you don’t have a backup of the client then you’re on your own.

• A copy of awk
• Somewhere on the local server to copy the CommVault client to. I normally set up a separate /tmp file system. So, in the example below I used that space for the CommVault client. I’m sure you could use NFS or smbmount to mount a remote file system as an alternative. Another option might be to use the partition reserved for swap. Initially set it up with ext3. Once the restore is completed unmount it and convert it to swap.
• And of course a CommVault backup of the server

Recovering the System

Boot up off the Red Hat install CD

At the “boot:” prompt type:

            linux rescue

When prompted select the language to use and then the keyboard type.

Select “Yes” to start the network interfaces on the system.

Enter the network configuration recorded above.

Skip searching for and mounting the file systems.

Configure the remaining network settings that will be needed:

• Set the hostname using the hostname command. For example:

            hostname srvlnx05

• Put the correct entries in /etc/hosts. For example:

            127.0.0.1   localhost localhost.localdomain
            192.168.1.5 srvlnx05.mycompany.com srvlnx05

• Put the DNS search path on /etc/resolv.conf. For example:

            search mycompany.com

• Set any required routes

At the command prompt recreate the partitions using the information recorded above.

Recreate any logical volumes. First do a vgscan, then create the physical volume, then the logical volumes. For example:

            vgscan
            pvcreate /dev/sda5
            vgcreate LocalVG /dev/sda5
            lvcreate –l 64 --name tmpLV LocalVG
            lvcreate -l 32 --name varLV LocalVG
            lvcreate -l 14 --name testLV LocalVG

Once the partitions are created, recreate the file systems. Note that if labels were used in /etc/fstab you will need to create the file systems with the appropriate labels.

For example:

            mkfs.ext3 –L /boot /dev/sda1
            mkfs.ext3 –L / /dev/sda2
            mkfs.ext3 /dev/LocalVG/tmpLV
            mkfs.ext3 /dev/LocalVG/varLV
            mkfs.ext3 /dev/LocalVG/testLV

Format any swap partitions. For example:

            mkswap /dev/sda3

In the example below I will be using the file system reserved for /tmp to store the CommVault client. As an alternative you could use space reserved for swap. Just convert it to a partition type of 83 and add a file system to it.

Mount the file system. For example:

            mkdir /tmp/cv-client
            mount /dev/LocalVG/tmpLV /tmp/cv-client

Copy the CommVault client. I had a tar archive on another server. For example:

            cd /tmp/cv-client
            ssh user@srvlnx02 “cat commvault.tar.gz” | tar -zxf -

Create symbolic links so that the CommVault client will be accessible from the correct paths. For example:

            ln -s /tmp/cv-client/opt /opt
            ln -s /tmp/cv-client/etc/rc.d /etc/rc.d
            ln -s /tmp/cv-client/etc/CommVaultRegistry /etc/CommVaultRegistry
            ln -s /tmp/cv-client/var/log /var/log/

The CommVault start up script needs awk. Copy awk to /bin. For example:

            scp user@srvlnx02:/bin/awk /bin/.

Start CommVault. For example:

            cd /
            /etc/rc.d/rc3.d/S99Galaxy start

Mount the file systems. For example:

            mkdir/tmp/cv-restore/
            mount /dev/sda2 /tmp/cv-restore
            mkdir /tmp/cv-restore/boot
            mount /dev/sda1 /tmp/cv-restore/boot
            mkdir /tmp/cv-restore/var
            mount /dev/LocalVG/varLV /tmp/cv-restore/var
            mkdir /tmp/cv-restore/test
            mount /dev/LocalVG/testLV /tmp/cv-restore/test

Restore the files using the CommVault management server. Specify the alternate location that you have mounted the file systems on (e.g. /tmp/cv-restore). You might want to exclude from the restore the following:

• If you’re using a file system on the local disk to temporarily hold the CommVault client you might want to exclude it. In the example I’m using that would be /tmp.
• Any file systems that you need to load kernel modules to access. For example, if you have file systems residing on SAN you might want to exclude them from the restore. Once you have the system up you can then restore them if need be.

Once the restore has finished we need to make the system bootable. On my test system I’m using Grub. I used http://www.whoopis.com/howtos/howto_restore_mbr_grub.php to guide me in this. I did a chroot to the root of the local disk before doing this step. You need to determine the root device. I looked in the restored grub.conf (boot/grub/grub.conf) for this information. In my case it was (hd0,0). Next run grub, specify the root device and then setup. For example:

            chroot /tmp/cv-restore
            grub
            root (hd0,0)
            setup (hd0)
            quit
            exit

You will also need to make sure there’s a /proc file system on the local disk – the system won’t boot correctly without it. For example:

            mkdir /tmp/cv-restore/proc
            chmod 555 /tmp/cv-restore/proc

Stop the CommVault client. For example:

            /etc/rc.d/rc3.d/K99Galaxy stop

Unmount the file systems. For example:

            cd /
            umount /tmp/cv-restore/boot
            umount /tmp/cv-restore/var
            umount /tmp/cv-restore/test
            umount /tmp/cv-restore

Unmount the file system that held the CommVault client:

            umount /tmp/cv-client

If you used a swap partition for your temporary file system then convert it to swap and format it. For example:

            sfdisk –change-id /dev/sda 3 82
            mkswap /dev/sda3

Restart the server.

tar -cf - sourcedir | "cd destdirparent && tar -xf -"

The -f - option tells tar to write the archive to stdout or read the archive from stdin as the case may be.

Placing the command after the pipe in quotes causes it to run in a subshell. This means that the two halves to the command can operate in different directories. The && means that if the cd command fails the tar won’t run either.

For example:

cd /home/fred
tar -cf - Maildir | "cd /d02/backup/fred && tar -xf -"

tar cf - sourcedir | ssh username@destination "cd destdirparent && tar xf -"