Skip to content

Recovering a Linux System using CommVault

Scope

Linux is not Windows. You should not need to reinstall the operating system to restore it. This document describes a way to boot a Linux system from the Red Hat install CD so it can then be restored from a CommVault backup. It does not document how to do the actual restore within CommVault. It was tested on a system running Red Hat Enterprise Linux AS 3 Update 9 and CommVault 7 with Service Pack 1.

This document only examines how to recover the operating system residing on local disks. It does not consider devices that might need additional kernel modules to be loaded. Nor does it consider external file systems such as those mounted via NFS or from a SAN. However, in the latter case that should not be an obstacle – it should be possible to restore the OS and then, using that, restore everything else.

Assumptions

  • The system to be restored is completely gone – it’s OK to overwrite the old disks
  • The hardware is identical
  • The server is using Grub
  • The local disks can be accessed using a Red Hat install CD without additional kernel modules being needed

Prerequisites

You need the following:

  • Details of all partitions, volume groups, logical volumes and file systems. Also keep a copy of /etc/fstab
    • Use the commands:
      • fdisk –l /dev/sda
      • vgdisplay –v
      • pvdisplay /dev/sda5
  • Network details including hostname, IP addresses, default gateway, contents of /etc/hosts and /etc/resolv.conf.
    • Use the commands:
      • ifconfig –a
      • cat /etc/hosts
      • cat /etc/resolv.conf
      • netstat –rn
      • hostname
  • A backup of the CommVault client
    • I included the following in my backup of the CommVault client:
      • /etc/CommVaultRegistry
      • /etc/rc.d/rc3.d/S99Galaxy
      • /etc/rc.d/rc3.d/K99Galaxy
      • /var/log/galaxy
      • /opt/galaxy
    • If you don’t have a copy of the client but it’s backed up in CommVault try restoring it to a different location and then copying it across.
    • I was unable to install the CommVault client on to a machine that booted from the Red Hat CD. If you don’t have a backup of the client then you’re on your own.
  • A copy of awk
  • Somewhere on the local server to copy the CommVault client to. I normally set up a separate /tmp file system. So, in the example below I used that space for the CommVault client. I’m sure you could use NFS or smbmount to mount a remote file system as an alternative. Another option might be to use the partition reserved for swap. Initially set it up with ext3. Once the restore is completed unmount it and convert it to swap.
  • And of course a CommVault backup of the server

Recovering the System

Boot up off the Red Hat install CD

At the “boot:” prompt type:

            linux rescue

When prompted select the language to use and then the keyboard type.

Select “Yes” to start the network interfaces on the system.

Enter the network configuration recorded above.

Skip searching for and mounting the file systems.

Configure the remaining network settings that will be needed:

  • Set the hostname using the hostname command. For example:

            hostname srvlnx05

  • Put the correct entries in /etc/hosts. For example:

            127.0.0.1   localhost localhost.localdomain
            192.168.1.5 srvlnx05.mycompany.com srvlnx05

  • Put the DNS search path on /etc/resolv.conf. For example:

            search mycompany.com

  • Set any required routes

At the command prompt recreate the partitions using the information recorded above.

Recreate any logical volumes. First do a vgscan, then create the physical volume, then the logical volumes. For example:

            vgscan
            pvcreate /dev/sda5
            vgcreate LocalVG /dev/sda5
            lvcreate –l 64 --name tmpLV LocalVG
            lvcreate -l 32 --name varLV LocalVG
            lvcreate -l 14 --name testLV LocalVG

Once the partitions are created, recreate the file systems. Note that if labels were used in /etc/fstab you will need to create the file systems with the appropriate labels.

For example:

            mkfs.ext3 –L /boot /dev/sda1
            mkfs.ext3 –L / /dev/sda2
            mkfs.ext3 /dev/LocalVG/tmpLV
            mkfs.ext3 /dev/LocalVG/varLV
            mkfs.ext3 /dev/LocalVG/testLV

Format any swap partitions. For example:

            mkswap /dev/sda3

In the example below I will be using the file system reserved for /tmp to store the CommVault client. As an alternative you could use space reserved for swap. Just convert it to a partition type of 83 and add a file system to it.

Mount the file system. For example:

            mkdir /tmp/cv-client
            mount /dev/LocalVG/tmpLV /tmp/cv-client

Copy the CommVault client. I had a tar archive on another server. For example:

            cd /tmp/cv-client
            ssh user@srvlnx02 “cat commvault.tar.gz” | tar -zxf -

Create symbolic links so that the CommVault client will be accessible from the correct paths. For example:

            ln -s /tmp/cv-client/opt /opt
            ln -s /tmp/cv-client/etc/rc.d /etc/rc.d
            ln -s /tmp/cv-client/etc/CommVaultRegistry /etc/CommVaultRegistry
            ln -s /tmp/cv-client/var/log /var/log/

The CommVault start up script needs awk. Copy awk to /bin. For example:

            scp user@srvlnx02:/bin/awk /bin/.

Start CommVault. For example:

            cd /
            /etc/rc.d/rc3.d/S99Galaxy start

Mount the file systems. For example:

            mkdir/tmp/cv-restore/
            mount /dev/sda2 /tmp/cv-restore
            mkdir /tmp/cv-restore/boot
            mount /dev/sda1 /tmp/cv-restore/boot
            mkdir /tmp/cv-restore/var
            mount /dev/LocalVG/varLV /tmp/cv-restore/var
            mkdir /tmp/cv-restore/test
            mount /dev/LocalVG/testLV /tmp/cv-restore/test

Restore the files using the CommVault management server. Specify the alternate location that you have mounted the file systems on (e.g. /tmp/cv-restore). You might want to exclude from the restore the following:

  • If you’re using a file system on the local disk to temporarily hold the CommVault client you might want to exclude it. In the example I’m using that would be /tmp.
  • Any file systems that you need to load kernel modules to access. For example, if you have file systems residing on SAN you might want to exclude them from the restore. Once you have the system up you can then restore them if need be.

Once the restore has finished we need to make the system bootable. On my test system I’m using Grub. I used http://www.whoopis.com/howtos/howto_restore_mbr_grub.php to guide me in this. I did a chroot to the root of the local disk before doing this step. You need to determine the root device. I looked in the restored grub.conf (boot/grub/grub.conf) for this information. In my case it was (hd0,0). Next run grub, specify the root device and then setup. For example:

            chroot /tmp/cv-restore
            grub
            root (hd0,0)
            setup (hd0)
            quit
            exit

You will also need to make sure there’s a /proc file system on the local disk – the system won’t boot correctly without it. For example:

            mkdir /tmp/cv-restore/proc
            chmod 555 /tmp/cv-restore/proc

Stop the CommVault client. For example:

            /etc/rc.d/rc3.d/K99Galaxy stop

Unmount the file systems. For example:

            cd /
            umount /tmp/cv-restore/boot
            umount /tmp/cv-restore/var
            umount /tmp/cv-restore/test
            umount /tmp/cv-restore

Unmount the file system that held the CommVault client:

            umount /tmp/cv-client

If you used a swap partition for your temporary file system then convert it to swap and format it. For example:

            sfdisk –change-id /dev/sda 3 82
            mkswap /dev/sda3

Restart the server.

 

SharePoint Calculated Column Formula to calculate CIDR

This formula takes a column called Mask and calculates the CIDR. It assumes that the subnet mask is valid. Note that if the column is called Subnet Mask this formula won’t work – it’s probably too long.

=IF(Mask="","",IF(Mask="0.0.0.0",0,0)+IF(Mask="128.0.0.0",1,0)+IF(Mask="192.0.0.0",2,0)+IF(Mask="224.0.0.0",3,0)+IF(Mask="240.0.0.0",4,0)+IF(Mask="248.0.0.0",5,0)+IF(Mask="252.0.0.0",6,0)+IF(Mask="254.0.0.0",7,0)+IF(Mask="255.0.0.0",8,0)+IF(Mask="255.128.0.0",9,0)+IF(Mask="255.192.0.0",10,0)+IF(Mask="255.224.0.0",11,0)+IF(Mask="255.240.0.0",12,0)+IF(Mask="255.248.0.0",13,0)+IF(Mask="255.252.0.0",14,0)+IF(Mask="255.254.0.0",15,0)+IF(Mask="255.255.0.0",16,0)+IF(Mask="255.255.128.0",17,0)+IF(Mask="255.255.192.0",18,0)+IF(Mask="255.255.224.0",19,0)+IF(Mask="255.255.240.0",20,0)+IF(Mask="255.255.248.0",21,0)+IF(Mask="255.255.252.0",22,0)+IF(Mask="255.255.254.0",23,0)+IF(Mask="255.255.255.0",24,0)+IF(Mask="255.255.255.128",25,0)+IF(Mask="255.255.255.192",26,0)+IF(Mask="255.255.255.224",27,0)+IF(Mask="255.255.255.240",28,0)+IF(Mask="255.255.255.248",29,0)+IF(Mask="255.255.255.252",30,0)+IF(Mask="255.255.255.254",31,0)+IF(Mask="255.255.255.255",32,0))

Note that if I was to do this again, I would probably just create a new list with the valid subnets and their CIDR values and then do a lookup on it.

My mnemonic for the seven layer OSI model

All Application
People Presentation
Should Session
Taste Transport
Nobby’s Network
Delicious Data Link
Peanuts Physical

Show Drive Letters First via Group Policy

Ever noticed how Windows Explorer, by default, shows the drive letter after the label. For example:

\\longservername.subdomain.domain\AllOurData M:

The problem with this is that in some dialogs you can’t see the drive letter. Wouldn’t it be better if it did it the other way around:

M: \\longservername.subdomain.domain\AllOurData

Here’s how you can change that using group policy. First you need a custom Group Policy template file. Group Policy template files have a .ADM extension. Please see the text at the bottom of this post for an example.

To add the template do the following:

  • Edit the Group Policy to be modified (or create a new one)
  • Expand Computer Configuration
  • Right click Administrative Templates and select Add/Remove Templates
  • Select the template file to add

The template will add an entry under Administrative Templates called Customisations. Under that you will see another called Explorer Settings. If you click on that you won’t see anything in the right hand pane. That’s because you need to change a filter to view this setting. Do the following:

  • On the menu select View and then Filtering
  • Untick “Only show policy settings that can be fully managed”

You should now see an entry in the right hand pane. Select the option you want (for example, “Show all drive letters before the label”).

If you don’t want to use Group Policy then you will need to come up with some other way of setting the entry in the registry. The registry key is
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer. The setting is a DWORD called ShowDriveLettersFirst. Values are:

0
1
2
4

Setting 0 gives letters after the label (the default)
Setting 1 gives network letters before the lable
Setting 2 will mean no drive letters
Setting 4 will give all drive letters before the label

You can download the administrative template from here –> ShowDriveLettersFirst.zip

Configuring L2TP Support on a Cisco ISR Router

Instructions
I consulted a number of sites on the Internet when I was attempting to configure L2TP on a Cisco router. I found many people with the same issues I had but no complete solution. In the end, my problem was that I needed an access list (see access-list 130 below).

Below is an example of a working configuration on a Cisco 2801 router with IOS Version 12.4(13b), RELEASE SOFTWARE (fc3). Note that authentication uses PAP, in most cases you probably want to use a more secure form of authentication. The router this config snippet came from uses RADIUS to authenticate the user. This configuration works with certificates. The change needed to allow pre-shared keys is quite small. I might add it later when I can get access to a non production router.

I’m using Fastethernet 0/1 as the external interface with IP address 123.123.123.123. The DHCP pool is in the 192.168.100.0 subnet. The DNS server is 192.168.200.1. I’ve called the crypto map VPNMAP.

vpdn enable
!
vpdn-group 1
! Default L2TP VPDN group
accept-dialin
protocol l2tp
virtual-template 1
no l2tp tunnel authentication
ip mtu adjust

crypto isakmp policy 20
encr 3des
hash sha
authentication pre-share
group 2

crypto isakmp policy 30
encr 3des
hash md5
group 2
!

crypto ipsec transform-set TRANSESP3DESMD5 esp-3des esp-md5-hmac
mode transport
!
!
!
crypto dynamic-map DYNMAP 1
set nat demux
set transform-set TRANSESP3DESMD5
match address 130
!
!
crypto map VPNMAP 65000 ipsec-isakmp dynamic DYNMAP
!
!
!
interface Virtual-Template1
ip unnumbered FastEthernet0/1
ip mroute-cache
peer default ip address pool VPN_CLIENT_POOL
ppp authentication pap
ppp ipcp dns 192.168.200.1
!
!
ip local pool VPN_CLIENT_POOL 192.168.100.1 192.168.100.99
!
access-list 130 remark Allow L2TP access
access-list 130 permit udp host 123.123.123.123 eq 1701 any
!
interface FastEthernet0/1
crypto map VPNMAP

I don’t think the “authentication pre-shared” is needed. I might remove it later when I have a chance to test it. The ISR seems to allow certificate authentication by default. Note that in my testing I found that the ISR would support both shared key and certificate authentication at the same time.

The router concerned also has a L2L (LAN to LAN) IPSec VPN tunnel configured. I might post the entire config at some later stage.

Copying a directory tree in Unix or Linux

cd to the parent of the source directoryUse the following command:

tar -cf - sourcedir | "cd destdirparent && tar -xf -"

where:
sourcedir is the directory to copy
destdirparent is the parent directory to copy the directory to

The -f - option tells tar to write the archive to stdout or read the archive from stdin as the case may be.

Placing the command after the pipe in quotes causes it to run in a subshell. This means that the two halves to the command can operate in different directories. The && means that if the cd command fails the tar won’t run either.

For example:

cd /home/fred
tar -cf - Maildir | "cd /d02/backup/fred && tar -xf -"

Note also that the source directory could be the current directory.

Using ssh to copy a directory tree from one host to another

From the source host:cd to the parent directory
Use the following command:

tar cf - sourcedir | ssh username@destination "cd destdirparent && tar xf -"

where:
sourcedir is the directory to copy
username is your username on the remote (destination) host
destdirparent is the parent directory on the remote (destination) host