Cloning Linux drives (notes for sio)

[siosios]

Automating process and Case studies:
One of the primary reason for using dd and netcat way of cloning OS instead of using commercial software such as Ghost is we have a liberty to automate process as we like. Following scripts may help in automating cloning process.
Case [1]: Script for Slave machine (netcat and dd cloning) on the fly.

Make sure you have netcat command available either /mnt/floppy or /mnt/sources/mystuff area.
=================================================
cloneme.sh :: Shell script for slave machine.
=================================================
#!/bin/sh
############### Edit variables below ######################
FLOPPY_PATH=/mnt/floppy
MYSTUFF_PATH=/mnt/sources/mystuff

# Uncomment only One of the options below.
#### OPTION ==> 1 if using floppy ################
#NC=$FLOPPY_PATH/nc
#### OPTION ==> 2 if using mystuff/ on CD #########
NC=$MYSTUFF_PATH/nc

LPORT=9000
DEST=/dev/sda
SRC=$DEST
############# No need to edit after this in general ###########

if [ $# -eq 1 ]
then
IPADDR=$1
echo "################################################# ##############"
echo " If there are no errors here. You need to run following"
echo " command on Master Box."
echo ""
echo "dd if=$SRC | nc $IPADDR $LPORT"
echo "################################################# ##############"

echo ""
echo "##>> Preparing /etc/hosts ##"
rm /etc/hosts
echo "127.0.0.1 localhost" > /etc/hosts
echo "$IPADDR fakehost" >> /etc/hosts

echo "#================================================ ===================="
echo "NOTE:: If you need to create routes"
echo " #route add -net <DEST_NET> netmask 255.255.255.0 gw $IPADDR metric 0"
echo "#================================================ ===================="

echo "##>> Preparing interfaces lo and eth0 ##"
ifconfig lo 127.0.0.1 up
ifconfig eth0 $IPADDR up

echo ""
echo ">>> Now start listening(at $LPORT) for traffic from Master "
echo "$NC -l -p $LPORT | dd of=$DEST"
$NC -l -p $LPORT | dd of=$DEST

echo ""
echo "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%"
echo " Cloning Process completed..... Reboot Now"
echo "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%"

else
echo "Usage:: cloneme <IP_ADDR_OF_THIS_MACHINE>"
fi




Case [2] Saving Disk Images (Export Image for later use):

Although you can clone running machine over the network anytime. But it is sometimes desirable to store base installation as a reference image and you may want to clone from this pristine image later. With the help of dd you can image disks also. But let's discuss some issues first.
Most 32 bit operating system (Linux for IA32 , Windows etc.) will have physical limitation on max file size. In general practical limit is 2GB as a max. file size. 64 bit OS (Solaris8, HPUX 11.0, Linux for IA64, etc.) will not have this limitation. So if you use dd to copy harddrive image you can maximum image 2GB harddrive. That is pretty useless these days. Fortunately dd can image in chunks and you can specify start and end blocks, skip blocks etc. So idea here is to image your big harddrives in chunks of approx. 2GB files over network. Although I noticed RedHat 7.1 with Linux 2.4.x kernels will allow fie size even bigger than 4GB on ext2 FS.
Also if you want to store images in compressed format (to save space) it is desirable to have each image file size not too large.
Following perl script (export-image.pl) can be used to image local Linux harddrive /dev/hda to remote machine over NFS using dd. If you are not running NFS you can implement same thing using dd and netcat. For now that would be a manual process. If somebody knows a better way to run netcat and transfer multiple files automatically between two machines please let me know and I will cook up some automation script here.

This perl script is actually use dd command something as described below. This is imaging your big harddrive into chunks of 1950 MB files named (1, 2, 3, 4, .....) over NFS to remote machine.
($NFS is NFS destination directory on another server having plenty of space)

For 1st Image:
dd if=/dev/hda of=$NFS/1 bs=1024k count=1950 skip=0
For 2nd image: (Skipping the part of harddrive used for 1st image.)
dd if=/dev/hda of=$NFS/2 bs=1024k count=1950 skip=1950
For 3rd image: (Skipping the part of Harddrive used for 1st+2nd image)
dd if=/dev/hda of=$NFS/3 bs=1024k count=1950 skip=3900
and so on.


*more below*

[siosios]

In case you want to use netcat you can simply pipe above dd commands manually to netcat and listen using netcat and dd on remote machine, just like we used netcat and dd to clone hardisks above. For example imaging harddrive on machineA and saving image on machineB.

For 1st image:
machineB% nc -l -p 9000 | dd of=1
machineA(master)% dd if=/dev/hda bs=1024k count=1950 skip=0 | nc machineB 9000
For 2nd image:
machineB% nc -l -p 9001 | dd of=2
machineA(master)% dd if=/dev/hda bs=1024k count=1950 skip=1950 | nc machineB 9001
For 3rdimage:
machineB% nc -l -p 9002 | dd of=2
machineA(master)% dd if=/dev/hda bs=1024k count=1950 skip=3900 | nc machineB 9002
and so on.

Once you have images (1, 2, 3, 4 ....) stored on network then you can boot your slave Linux box using bootable CD and pull these images to slave box as described in case [3].

================================================== ======
export-image.pl :: Perl script to image big harddrive using dd and NFS.
================================================== ======
#!/usr/bin/perl
################################################## ###
#This script will run dd command (in serial) and dump
#1950 blocks (1.9GB) file for each.
#Run script as perl export-image.pl
################################################## ###

################ Edit variables below #########################
#device is raw device name for harddrive to be cloned (imaged).
$device="/dev/hda";
#mount NFS file system with large space available which can hold images.
$nfs_path="/nfs/remote/home/tmp";
#Image name (read from user) (Make sure you have $nfs_path/$image directory)
#on remote machine.
$image="ob6000";
################################################## ##########
$dd="/bin/dd";
#For compressing image
$bzip2="/usr/bin/bzip2";
$suffix=".bz2";
############## No need to edit after this #########################

$bs="1024k";
$block_count=1950;
$image_dir="$nfs_path/$image";
$compress=$bzip2;

$proceed=0;

if(!(-d $image_dir) )
{ die "\nOops!! Image Directory $image_dir must exist with chmod 777 permission\n"; }

system("clear");
print <<MSG1;
################################################## #########
NOTE:: COMPRESSION TAKE TOO MUCH TIME(Many HOURS) OVER NFS.
So better compress manually latter on server itself.
################################################## #########
\n\n Do you want to compress images using $compress [y/n] (Default n) = \t
MSG1

$compress_flag=<STDIN>;
if(($compress_flag eq "y") or ($compress_flag eq "Y"))
{ $compress_flag=1; }
else
{ $compress_flag=0; }

print "\n\n";
print "************************************************* **\n";
print " Local Device = $device [SOURCE] \n";
print " Image Dir = $image_dir [TARGET] \n";
print "************************************************* **\n\n\n";
print "Dude! I hope you understand what are you doing by pressing [y/Y] here \n";
print " Press [y/Y] if you want to continue .. ";
$con=<STDIN>; chomp($con);


if(($con eq "y") or ($con eq "Y"))
{
$i=0;
$image_size=1; #Some fake value greater than zero.


print "\n\nDisk Imaging starts...\n";
system("date");
while($image_size > 0)
{
$image_name="$image_dir/$i";
print "##############################################\n" ;
print "Creating Image $image_name\n";
print "##############################################\n" ;
$skip=$i*$block_count;
print "$dd if=$device of=$image_name bs=$bs count=$block_count skip=$skip \n";
system("$dd if=$device of=$image_name bs=$bs count=$block_count skip=$skip");
if($compress_flag)
{
print "Compressing Image: $bzip2 $image_name => $image_name$suffix\n";
system("$bzip2 $image_name");
$image_name .= "$suffix";
}
++$i;
$image_size=(stat($image_name))[7];
system("date");
}
}
else
{
print "Bye Bye ...\n";
}




*more below*

[siosios]

Case [3] Importing Disk Images (1, 2, 3, 4 ...) created in Case [2] using netcat, dd and cat

This part is little tricky in the sense we want all images (1, 2, 3, 4, ...) to be imported on slave machine and use dd to write these images serially on slave drive. A very simple set of commands can be used as below.

On Slave machine: (booted through linux rescue). Run following netcat command to capture incoming data stream.

machineC(slave)% nc -l -p 9000 | dd of=/dev/hda

On machineB machine: (where images 1, 2, 3, 4 .... are stored). Run following cat and netcat command. Make sure you cat images in the same sequence as they were imported in case [2]. cat command will simply join these images and throw data stream to netcat which slave machine will pick up and copy bit by bit on slave harddrive.

machineB% cat 1 2 3 4 .... | nc machineC 9000



Case [4] Importing Disk images created in Case[2]:
Most likely 'linux rescue' system won't have NFS support. Which means when you boot slave box using such method you can not access resources over NFS. But if you cook your own CD and that has NFS support and perl the following perl script can be used to fetch images stored earlier from machineB using NFS. This script is actually doing:
($NFS is NFS source directory on another server machineB where you have images 1, 2, 3, 4, ... stored earlier)

For image 1:
dd if=$NFS/1 of=/dev/hda bs=1024k conv=notrunc seek=0
For image 2:
dd if=$NFS/2 of=/dev/hda bs=1024k conv=notrunc seek=1950
For image 3:
dd if=$NFS/3 of=/dev/hda bs=1024k conv=notrunc seek=3900

In any case if you are interested in using perl script below (if you have perl and NFS client support on slave linux box).

================================================== ======
import-image.pl
================================================== ======

#!/usr/bin/perl
################################################## ###
#This script will run dd command (in serial) and dump
#and import image.
################################################## ###

################################################## ############################
#device is target raw device name for harddrive to be cloned.
$device="/dev/hda";
#mount NFS file system with large space available which can hold images.
$nfs_path="/mnt/images";
#Image name (read from user)
$image="ob6000";
################################################## #############################
$dd="/bin/dd";
#$bzcat="/usr/bin/bzcat";
#$suffix=".bz2";

$bs="1024k";
$block_count=1950;
################################################## #############################
$image_dir="$nfs_path/$image";

$proceed=0;

if(!(-d $image_dir) )
{ die "\nOops!! No Image Directory $image_dir\n"; }

system("clear");
print "************************************************* **\n";
print " Local Device = $device [TARGET]\n";
print " Image Dir = $image_dir [SOURCE]\n";
print "************************************************* **\n\n\n";
print "Dude! I hope you understand what are you doing by pressing [y/Y] here \n";
print " Press [y/Y] if you want to continue .. ";
$con=<STDIN>; chomp($con);
print " Once Again!!! Press [y/Y] if you want to continue .. ";
$con=<STDIN>; chomp($con);

system("date");
if(($con eq "y") or ($con eq "Y"))
{
print "\n\nDisk Imaging import starts...\n";

$i=0;
$image_name="$image_dir/$i";
while(-f $image_name )
{
print "##############################################\n" ;
print "Importing Image $image_name\n";
print "##############################################\n" ;
$seek=$i*$block_count;
print "##############################################\n" ;
$seek=$i*$block_count;
print "$dd if=$image_name of=$device bs=$bs conv=notrunc seek=$seek \n";
#system("$bzcat $image_name | $dd of=$device bs=$bs conv=notrunc seek=$seek");
system("$dd if=$image_name of=$device bs=$bs conv=notrunc seek=$seek");
++$i;
$image_name="$image_dir/$i";
system("date");
}
}
else
{
print "Bye Bye ...\n";
}



Other Operating Systems Tips:
You can pretty much do same in other operating systems also. This section quickly list few tips that may be useful.

Windows:

You can find GNU utilities ( http://unxutils.sourceforge.net/ ) for Win32 platforms which includes dd.exe command.
dd.exe syntax is similar to as you use on Linux side. For physical partition you may have to use devicename something like \\.\PhysicalDrive0 etc. For example
dd if=\\.\PhysicalDrive0 of=<target>
You can download netcat for windows ( http://www.l0pht.com/~weld/netcat/ )
If you are cloning WinNT/2K system , you need to change SID for the new cloned system if machine is participating in Windows domain. You can use Ghostwalker program from Ghost distribution or can use newsid.exe from http://www.sysinternals.com

Solaris:

Burning bootable CDROM for Solaris. ( http://www.lka.ch/projects/solcdburn/solcdburn.html )
Sun Blue print : http://www.sun.com/software/solution.../BuildBoot.pdf
To Make disk bootable (just like putting MBR in Linux) use command installboot(1M)
Others: (Make disk bootable)
Irix : dvhtool
HP-UX: mkboot
Tru64: disklabel


Conclusion:

Few possible uses of netcat and dd shown in this document. Although methods presented here are very simple and easy to use but have few pros and cons also. This technique is very good for on the fly OS cloning. When we image the whole drive we need the equivalent harddrive space on other machine. This may not be very much practical. You can try compressing those images which will save lots of space. I noticed dd image can be compressed upto 30-80% depending upon real data on the drive using gzip/compress program. This cloning and imaging method may be very effective in forensic analysis where sometimes you need an exact snapshot of harddrive including swap space partitions. You can always break your images in small pieces (may be compress them) transfer over network to somewhere else and reproduce data. As mentioned above one of the great advantage here is to custom your own cloning scheme.


References:
GNU utilities for Win32. http://unxutils.sourceforge.net/
netcat for Windows. http://www.l0pht.com/~weld/netcat
First Attempt at Creating a Bootable Live Filesystem on a CDROM http://www.linuxgazette.com/issue54/nielsen.html
Good Site for Windows utilities such as newsid.exe: http://www.sysinternals.com
Modifying ISO image http://www.winiso.com
Solaris Bootable CD creation: http://www.lka.ch/projects/solcdburn/solcdburn.html
Sun Blueprint: http://www.sun.com/software/solution.../BuildBoot.pdf
Linux on Floppy: http://www.toms.net/rb/
Static binaries for Linux.