Every single IT person will tell you to maintain your Windows security updates. It is critical, we are told, so that we can keep our computers safe. While I don’t completely agree with the statement (I’ve personally experienced an “update” that has completely killed a mission critical server), it is important to be able to update your system with the required patches.

Most people use Windows Update and many IT administrators use Windows Software Update Services (WSUS) to deploy these patches. While both works well, I personally prefer AutoPatcher. And here’s why:

1. It uses way less bandwidth than Windows Update. You download the patches once and you are done. You can now easily deploy all of the update to several computers, even computers without Internet access.
2. AutoPatcher does more than just update Microsoft applications. It is completely customizable and you can use it to install pretty much anything you want.
3. You can deploy AutoPatcher updates over a network without a bunch of registry hacks. Unless you are running Active Directory, you need to perform a number of registry hacks to get WSUS to work.
4. You do not need to install the Windows Genuine Advantage “critical” update in order to install other updates. You can keep your information to yourself.
5. AutoPatcher can be run unattended over a network or from a log-on script.

If you have never used this amazing tool, I would really encourage you to give it a try. It’s freeware so all it will cost you is your time.

Trackback link - http://www.DailyCupOfTech.com/2008/05/16/why-autopatcher-beats-windows-update-and-wsus/trackback/

If you purchased any Microsoft operating system over the past number of years, you were probably inundated with offers to purchase Microsoft Plus! This product promised to give you even more than was originally provided in your OS.

Personally, I thought it was Microsoft’s attempt to make you part with even more of your hard earned cash than you had already. What Plus! provided was essentially eye candy and it added very little in the way of extra functionality.

But, it looks like Microsoft will no longer me making this product available to the public. According to their website, they believe that they have now built all of the Plus! functionality into their products. My theory is that the public realized that the product wasn’t worth the money and voted with their feet.

Have you had any experience with the Microsoft Plus! products? Did you like them or not? Are you going to miss them? Were they worth the money?

Trackback link - http://www.DailyCupOfTech.com/2008/05/15/plus-subtracted/trackback/

Every few months, I like to completely blow away my Windows system and reinstall it. This is primarily because I install so much junk on the system that it just starts to clog everything up. A freshly installed copy of Windows always runs so much quicker and has that “new OS smell”!

I firmly believe that more people would do the same if it wasn’t such a pain to do! Most people figure that you simply need to set aside a full day to do the job right.

So, I’ve put together a two part checklist of things that you should do before and after you completely reformat your system to start over.

Before You Reformat

1. Move all of your data off the computer and put it on a different system or drive. The most important folder to backup is the user profile (%USERPROFILE%) folder. This will usually back up 95% of what you want including your My Documents, music, pictures, favorites (for Internet Explorer), e-mail, etc.
2. Backup all of the device drivers that are being used on the system and store them somewhere else. I use a freeware program called DriverMax but I am sure that there are several other good ones out there.
3. Make sure you have a copy of all the software installed on your computer along with all of the license keys. A quick way to get a list of all installed software is to use something like Belarc Advisor. If you do not have the license keys, a program like Magical Jelly Bean Keyfinder might be of help. This includes operating system and software that you bought by download. I usually start making this list a few weeks before I decide to rebuild my system.
4. Download AutoPatcher and prepare a Windows/Office update CD/DVD.
5. Create an image of the system just in case you missed/forgot something. There are a number of free imaging programs available. This way, even if there is something that you forgot to backup, you will still be able to get it back from the image.
6. Perform a thorough check of your hard drive. If your drive has errors or problems (r you just want a bigger hard drive), now would be a really good time to replace the drive. Some tools include:
• Seagate SeaTools
• Hitachi Drive Fitness Test
• Western Digital Data Lifeguard Diagnostic
7. Make sure that you have all of your account information for your Internet provider, including e-mail. Have their phone number handy in case you run into trouble.
8. If you do not have all of your online accounts information memorized, now is a good time to write record them so that you do not destroy the information when you reformat your drive.
9. Make sure that you have your wireless network information recorded and available.

If you have lost your passwords that are stored on your computer, here are some tools that may be able to help you out:

• MessenPass
• Mail PassView
• IE PassView
• Protected Storage PassView
• Dialupass
• Asterisk Logger
• AsterWin IE
• Network Password Recovery
• SniffPass Password Sniffer
• PstPassword
• WirelessKeyView
• Remote Desktop PassView
• PocketAsterisk
• RemotePocketAsterisk
• Unhide Passwords (Firefox)

After You Reformat

Well, you’ve reformatted your disk so there is no turning back. Here is a general overview of the process:

1. Make sure only the bare necessaries are plugged into the computer (monitor, keyboard, and mouse).
2. Boot from the Windows CD and install Windows.
3. Install all of the drivers that you backed up earlier.
4. Plug in all of your peripherals and ensure that they are all working correctly.
5. Install Office if you had it on your computer.
6. Install all of the updates from your AutoPatcher CD/DVD.
7. Run a disk clean and defragment on the drive.
8. Reconnect your systems to the Internet.
9. Install all of the applications that you want installed on the system.
10. Migrate all of your files to your newly built system.
11. Setup all of your accounts again.

I’m sure that there are a few things that I have forgotten but this is a really good start. Let me know what I’ve missed or if there is something that you do that I do not and I will add it to the list.

Trackback link - http://www.DailyCupOfTech.com/2008/05/10/windows-reinstall-simple-easy-and-quick/trackback/

Many of you are familiar with the concept of computer imaging. Essentially, you take a system, configure it the way that you would like, and then duplicate the hard drive to all of the other systems, saving yourself all of the same steps that you did on the first system.

Now, there are a lot of great commercial software programs out there that can do this for you. Acronis and Ghost come to mind. But, not everyone can afford these options. For those of you find yourself in this situation, I present you with a list of options that are available for free or are open source. (All descriptions are copied from their respective websites.)

Partimage is a Linux utility which saves partitions having a supported filesystem to an image file. Most Linux and Windows filesystems are supported. The image file can be compressed with the gzip / bzip2 programs to save disk space, and they can be splitted into multiple files to be copied on CDs / DVDs, … Partitions can also be saved across the network since version 0.6.0 using the partimage network support, or using Samba / NFS. If you don’t want to install Partimage, you can download and burn SystemRescueCd. It’s a livecd that allows to use Partimage immediately even if your computer has no operating system installed (useful to restore an image), and it allows to save an image on a DVD on the fly.

PING (Partimage Is Not Ghost)

PING is a live Linux ISO, based on the excellent Linux From Scratch (LFS) documentation. It can be burnt on a CD and booted, or integrated into a PXE / RIS environment. Several tools have been added and written, so to make this ISO the perfect choice to backup and restore whole partitions, an easy way. It sounds like Symantec Ghost(tm), but has even better features, and is totally free.

Clonezilla

Clonezilla, based on DRBL, Partition Image, ntfsclone, and udpcast, allows you to do bare metal backup and recovery. Two types of Clonezilla are available, Clonezilla live and Clonezilla server edition. Clonezilla live is suitable for single machine backup and restore. While Clonezilla server edition is for massive deployment, it can clone many (40 plus!) computers simultaneously. Clonezilla saves and restores only used blocks in the harddisk. This increases the clone efficiency. At the NCHC’s Classroom C, Clonezilla server edition was used to clone 41 computers simultaneously. It took only about 10 minutes to clone a 5.6 GBytes system image to all 41 computers via multicasting!

IRestorer

Drive Snapshot (IRestorer) creates an exact drive image of your system into a file, including the operating system, installed programs, your data and all security attributes, while Windows is running and you continue to work. If disaster strikes (a virus, worm, trojan, or hard disk crash), a complete and exact restoration of your system will be possible in only few minutes. The image file can be mounted as a virtual drive, so you can easily restore (or use) single files or directories, using the Windows Explorer, or other programs. Restart to DOS is only necessary, if you must restore your system partition. Otherwise there is no need to reboot, not even during installation. Drive Snapshot saves only the used disk space, compresses the data and is extremely fast, a P4 - 3.0 GHz will save approx. 2.5 GB/Min on a modern hard disk!

Linbox Rescue Server

The Linbox Rescue Server is an asset management software including 5 modules :

• system backup for emergency crash recovery, hard disk cloning or deployment using a
  PXE network boot. This module was fully developped by Linbox FAS.
• file backup, based on the famous BackupPC, to which we have added a configuration interface,
• inventory, based on ocs-inventory agents, and on an agent which runs during the PXE network boot,
• Software deployment module which works with Linux, MacOS X and MS Windows clients, or any Un*x system running OpenSSH,
• remote control, based on TightVNC.

HDClone

Copies smaller drives onto larger ones for migrating or backing up complete installations as well as for data rescue. The download package contains a program to easily create a bootable floppy disk or CD/DVD under Windows with only few clicks. A floppy image, ISO image, and manual as PDF are also enclosed.

DiscWizard

If you are setting up your new drive, you can use the install instructions to assist in mounting the drive, then use the tools built into your operating system to prepare the drive.

G4L

G4L is a hard disk and partition imaging and cloning tool. The created images are optionally compressed and transferred to an FTP server instead of cloning locally.

MaxBlast 5

If you are setting up your new drive, you can use the install instructions to assist in mounting the drive, then use the tools built into your operating system to prepare the drive.

DriveImage XML Backup Software

DriveImage XML is an easy to use and reliable program for imaging and backing up partitions and logical drives.

The program allows you to:

Backup logical drives and partitions to image files

Browse these images, view and extract files

Restore these images to the same or a different drive

Copy directly from drive to drive

Schedule automatic backups with your Task Scheduler

Image creation uses Microsoft’s Volume Shadow Services (VSS), allowing you to create safe “hot images” even from drives currently in use.Images are stored in XML files, allowing you to process them with 3rd party tools. Never again be stuck with a useless backup!

Restore images to drives without having to reboot.

DriveImage XML runs under Windows XP, Windows Server 2003 and Windows Vista only. The program will backup, image and restore drives formatted with FAT 12, 16, 32 and NTFS.

DuBaron DiskImage

DiskImage is the ultimate tool to backup and restore partitions, entire harddisks, USB keys, floppy drives and (read-only) optical media. DiskImage features built-in zip compression, it’s own compression method for data
> 2Gb, MD5 and SHA1 checksumming, a hex editor, and various tools like drive speed and seek testing.

Dumping and restoring partitions, USB sticks, floppies, optical media, diskimage is the swiss army knife.

WinDD - Disk Dump for Windows

WinDD - Disk Dump for Windows! Windows XP version of Unix ‘dd’ command. Safe, effort-free backup for FAT, FAT32, NTFS, ext2, ext3 partitions.

Partition Saving

Partition Saving is a DOS and Windows program that is used to save, restore and copy hard-drive, partitions, floppy disk and DOS or Windows devices.

With this program you could save all data on a partition to a file (such as you could save this file on a CD for example). Then if something goes wrong, you can completely restore the partition from the backup file. You no longer have to reinstall every piece of software from scratch. All you have to do is restore the partition from the backup file and then update any software that was modified since the backup was created.

NFGDump

nfgdump (Ntfs-Fat-Generic-dump) is a Win32/Linux tool that dumps/restores NTFS (including 3.1=XP), COMPRESSED NTFS, FAT16, FAT32 and arbitrary (generic) partitions. Dump files support compression, encryption, splits, page file content removal, etc.

Forensic Acquisition Utilities

This is a collection of utilities and libraries intended for forensic or forensic-related investigative use in a modern Microsoft Windows environment. The components in this collection are intended to permit the investigator to sterilize media for forensic duplication, discover where logical volume information is located and to collect the evidence from a running computer system while at the same time ensuring data integrity (e.g. with a cryptographic checksums) and while minimizing distortive alterations to the subject system. The components of this package are not intended to preclude changes to the subject system while the evidence collection process is under way. A third party hardware or software write blocker should be employed in those circumstances where it is deemed necessary to guarantee that no changes occur to the subject volume prior to and after the imaging process.

Self Image

SelfImage is the little hard drive utility with big aspirations.SelfImage is capable of making an image file of a hard disk or hard disk partition, and can restore an image back to any drive or partition that doesn’t have open files. Useful for making backups. Unlike dd for Windows (or cygwin), SelfImage is capable of creating an image of a partition that is currently in use.

SelfImage can even create images of partitions that Windows doesn’t recognize (partitions that Windows doesn’t have mounted on a drive letter). This is perfect for the dual-boot system, you can create an image backup of a Linux partition directly from Windows.

PC inspector clone maxx

PC INSPECTOR™ clone maxx is the new professional hard drive copying program from CONVAR. Using the new direct DMA support, data can be copied from hard drives in high speed mode with speeds up to 3.3 GB per minute.This high speed is achieved by PC INSPECTOR™ clone maxx through intelligent inspection of the PC motherboard and the hard drive controller. The software independently determines the fastest data transmission rate and achieves these exceptional speeds through optimal utilization of the DMA chipset on the motherboard of your PC.

PC INSPECTOR™ clone maxx can be started directly from a boot diskette. The copying process is always based on the physical drive and is independent of the file system (e.g. FAT12, FAT16, FAT32, HPFS, NTFS, Ext2, Reiser, etc.) or the number of partitions.

g4u - Harddisk Image Cloning for PCs

g4u (”ghosting for unix”) is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common setup on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported.

FOG :: A Ghost-like Computer Cloning Solution

Fog is a Linux-based, free and open source computer imaging solution for Windows XP and Vista that ties together a few open-source tools with a php-based web interface. Fog doesn’t use any boot disks, or CDs; everything is done via TFTP and PXE. Also with fog many drivers are built into the kernel, so you don’t really need to worry about drivers (unless there isn’t a linux kernel module for it). Fog also supports putting an image that came from a computer with a 80GB partition onto a machine with a 40GB hard drive as long as the data is
less than 40GB.

Fog also includes a graphical Windows service that is used to change the hostname of the PC, restart the computer if a task is created for it, and auto import hosts into the FOG database. The service also installs printers, and does simple snap-ins.

Each has its own pros and cons but all have a purpose. I am personally really liking FOG and am finding a lot of different places to put it to work.

Trackback link - http://www.DailyCupOfTech.com/2008/05/06/free-or-open-source-imaging-solutions/trackback/

If you are running a small office of a couple dozen employees and you have Windows 2003 R2, you can easily centralize your contacts for your entire office along with give everyone the ability to see the company’s corporate calendar without having to install Exchange in your environment.

With Windows 2003 R2, you also have a copy of Sharepoint services. Sharepoint allows you to create workspaces for your different working groups. It is generally designed to work for either relatively large companies or working groups that are widely distributed. But, if you are neither, you have this technology at your disposal and you might as well put it to good use.

By creating a central contacts and calendar in the Sharepoint services, your users can connect to then using Outlook. This way, is is really easy for them to find contact information for anyone in the company or see what is happening on the corporate calendar right from a tool they already use.

You may even want to expand things a bit more and share your own calendar using Sharepoint so that people can see when you are busy or not and allow them to better select meeting times. Or, instead of just putting in company contacts in Sharepoint, you can also put in vendors and clients so that this information is readily available for everyone in the company to use.

One other nice aspect about doing things this way is that it is also useful for the road warrior. Outlook caches a copy of all the contacts and calendar information locally on your hard drive so that when you are on the road with your laptop, all of the contacts and calendars are still available, even though you may not be able to connect with your Sharepoint server.

Now, if you do not have access to a Sharepoint server, you do have some other options, most free or open source.  Here are just a few:

• eGroupware
• phpGroupware
• OpenGroupware.org
• PHProjekt
• horde
• Simple Groupware

What are you using for online collaboration?  Let us know what works and what doesn’t work in the comments.

Trackback link - http://www.DailyCupOfTech.com/2008/01/28/centralized-contacts-and-corporate-calendar-without-exchange/trackback/

There are a number of Debian based Linux distributions out there. My personal favorite is Ubuntu. I will typically use the desktop or server version for most of my Linux needs.

Each time I build a new system, there are a number of steps that I typically follow which I seem to forget and need to research again. So, both to document my actions and for your benefit, here are the first few things that I do for any new installation.

The Starting Point

This guide assumes that you have just completed the base installation and have access to a command/terminal prompt. You will also need to have the root password but are logged in as a typical user.

Edit Sources

 sudo nano /etc/apt/sources.list

Except for the very first line, remove all of the # in front of each line the begins with # deb and save the file.

Update Distribution

sudo apt-get update
sudo apt-get upgrade

Assign a Static IP

sudo nano /etc/network/interfaces

In this file, find the line that says iface eth0 inet dhcp. Comment out this line (put a # in front of it) and then add after it on the next lines:

iface eth0 inet static
        address 192.168.0.100
        netmask 255.255.255.0
        network 192.168.0.0
        broadcast 192.168.0.255
        gateway 192.168.0.1

Make sure that the network information you enter here is accurate for your network. Once you have saved this file, then restart the network:

sudo /etc/init.d/networking restart

Next, update your host file to reflect your new static IP address:

sudo nano /etc/hosts

Find the line starting with 127.0.1.1 (usually the second line) and change 127.0.1.1 to the IP address you desire. From the example above, it would be 192.168.0.100. Save the file when you are done.

Install the SSH Server

sudo apt-get install ssh openssh-server

Other Useful Tasks

These are some other things that I have found to be useful from time to time. They are not necessarily a part of my default setup but they do come in handy

Determine Your Static IP Address

ifconfig | grep Bcast

Enable root

sudo passwd root

Trackback link - http://www.DailyCupOfTech.com/2008/01/11/first-steps-on-debian-based-distros/trackback/

If you are having a hard time convincing your manager that you need to start using encrypted e-mail or to go away from POP3 e-mail access, here is a nifty little trick!

1. Install Wireshark on a computer that is off the network.
2. Install a managed switch that supports port mirroring or port spanning between your Internet connection and your firewall.
3. Mirror the port that your firewall is plugged into to another port andplug your computer with Wireshark into that port.
4. Capture the information with Wireshark for about an hour (or however long you want) using the following filter:

   pop.request.command == "USER" || pop.request.command == "PASS"

5. Show your manager everyone’s username and password that you captured from outside the network on the Internet!

You can also find all of the FTP passwords using the same method and this filter:

ftp.request.command == "USER" || ftp.request.command == "PASS"

Trackback link - http://www.DailyCupOfTech.com/2007/10/25/e-mail-password-discovery-trick/trackback/

It is always important to know what is going on in your network. And, from time to time, you should perform a simple audit of your network just to determine what is exactly happening.

This is something that I do on a regular basis using an inexpensive network auditor that I built.  (Actually, all I did was install a custom Linux distribution designed for security.)

What You Need

Here are all the tools that you need

• An old laptop (if you want it to be portable) or desktop
• BackTrack Linux Distribution

It’s pretty easy to find a computer that will work and BackTrack is available for free as an ISO.  This is something that probably everyone could do with what they have available.

Build The Analyzer

The steps involved to build the analyzer are pretty simple.

1. Download the ISO.
2. Burn the ISO to CD.
3. Boot the computer from the CD.
4. Log into the BackTrack system with username root and password toor.
5. Create partitions on the hard drive (Be aware that this will destroy the hard drive so make sure that the drive has no needed data.) using fdisk.
6. Write the file system using mke2fs.
7. Mount the devices using mount.
8. Run the GUI installer (K → System → Backtrack Installer).
9. Click Install.
10. Wait for the installation to complete (There is a long pause at about 80% so please be patient.).
11. Remove the CD and reboot to the new system.

If you are looking for a more complete step by step document, there is one available.

Explore

There is a lot to discover in BackTracks.  Dig through the system and see what you can find, both on the system and in your network.

Trackback link - http://www.DailyCupOfTech.com/2007/09/07/build-an-inexpensive-network-security-auditor/trackback/

I can’t tell you how sick I am of people asking me the same question over and over. And, typically, this is a question that I probably shouldn’t really be having to deal with in the first place. Do some of these questions sound familiar to you?

“How do I make a template in Word?”

“Where are my rows in Excel and why does it skip row numbers?”

“Can I change all of the PowerPoint slides at once?”

To be honest, I am not paid to answer these questions. These are basic function questions that have nothing to do with how the systems work or providing the people with the tools that are necessary to do their jobs. It is not a part of IT’s mandate to train or tech people on how to use the tools for their job.

But, we work in a small company and this is not uncommon to see. In fact, in all of my IT jobs, this exact thing happens. So, how to you prevent yourself from becoming overwhelmed by the avalanche of non-technical questions?

The solution that I have discovered which any shop can implement is application champions. What is an application champion? An application champion is someone whom is considered to be more highly skilled at using a specific computer application or program than the average user. They are not a member of the IT department but are skilled in the normal use of a specific application. They need not be technical in the IT sense.

There are three steps that you need to take when setting up your company to use application champions:

1. Identify Application Champions. In the company that I now work for, I sent out an e-mail to everyone in each office with a list of all the regular applications that are used. I then asked them to nominate a person for each application as being an advanced user of this application. Make sure that you spell out clearly that they can not nominate members of the IT department. Make sure that you do not use the same person for all of the champion positions so that if one person leaves the company, then you will not lose all of your extra talent!
2. Enable Application Champions. Get buy in from your new champions and then get them trained! Make sure that they are able to answer most of the questions that they will be inundated with and provide them with the resources that they need to take on this responsibility.
3. Use Application Champions. Make a big production about introducing the application champions once they are ready to be put to use. Send out a list of all the application champions and the applications that they are “certified” to help with. Encourage users to work with these people and help each other out.

Trackback link - http://www.DailyCupOfTech.com/2007/09/06/reduce-your-it-workload/trackback/

I have been putting some thought around creating a distributed file archive system with redundancy lately and I think that I have come up with a viable proof of concept. The entire process is manual at this point but with a bit of work, I think that I could automate it and make it usable.

What Is It?

The whole idea came to me from a comment left on a tumblog post. Essentially, JD asked about whether or not someone could point him in the right direction for something like this. I gave it some thought and I think I have a viable model.

Essentially, the question was asked whether or not we could use all of the unused storage on all of the workstations and laptops in a small enterprise environment as a backup or archive solution. To me, this seemed like a logical use of resources, especially for a small IT shop where the budgets are small or for a home with a now common one computer per person setup.

On the surface, this seemed like a wonderful idea but there were issues.

No Redundancy

The biggest issue that I saw with a solution that uses this concept is the hard drive. Workstations are typically single drive systems. There is rarely any redundancy in place for these drives. If that drive fails, your data is gone.

Now, if this is a simple backup solution, this may be less of an issue because, since the data is a copy to begin with, you already have a copy of the data. Things get a bit more risky when we are talking about an archive system.

The purpose of an archive is to move data to a storage location for later access. By definition, you do not have a copy back where the original was located. Now what should we do?

Parchive to the Rescue

The answer to this problem is to use parchive files for redundancy. What are parchive files? Here is what the Parchive Project says about parchive files:

The original idea behind this project was to provide a tool to apply the data-recovery capability concepts of RAID-like systems to the posting and recovery of multi-part archives on Usenet. We accomplished that goal. Our new goal with version 2.0 of the specification is to improve. It extends the idea of version 1.0 and takes the recovery process beyond the file-level barrier. This allows for more effective protection with less recovery data, and removes some previous limitations on the number of recoverable parts.

How The System Would Work

Let’s use a common scenario to examine how to use parchive files to create a redundant archive storage grid.

Let’s say, for example, that you have six computers on your network, your computer and five others. Your connections to these computers would look something like this:

Let’s also assume that you have write access to a share on each of these computers.

Now, you want to archive your data by distributing it on each of the systems. For our example, we are going to assume that you have a 697 MB file called ubuntu.iso that you want to archive and each system has 150 MB of free disk space.

You compress the file to save disk space. You now have a file ubuntu.zip that is 681 MB in size.

You now split the Data.zip file into five equally sized files. You are now left with the following files:

• ubuntu.zip.001
• ubuntu.zip.002
• ubuntu.zip.003
• ubuntu.zip.004
• ubuntu.zip.005

Each file is 136 MB in size.

You place one file on each computer. So:

• ubuntu.zip.001 on Computer 1
• ubuntu.zip.002 on Computer 2
• ubuntu.zip.003 on Computer 3
• ubuntu.zip.004 on Computer 4
• ubuntu.zip.005 on Computer 5

This creates a total of 681 MB of used storage.

Accounting for Hard Drive Failure

This scenario works well as long as nothing goes wrong! But, if you were to lose the hard drive on just one of the workstations, all of the data in ubuntu.iso is gone!

One option would be to put duplicate files on each system. So, you could do the following:

• ubuntu.zip.001 and ubuntu.zip.002 on Computer 1
• ubuntu.zip.002 and ubuntu.zip.003 on Computer 2
• ubuntu.zip.003 and ubuntu.zip.004 on Computer 3
• ubuntu.zip.004 and ubuntu.zip.005 on Computer 4
• ubuntu.zip.005 and ubuntu.zip.001 on Computer 5

This would require 1,362 MB of storage to ensure that if one of the systems crashed, you would be able to recover all of your data.

But, if we were to create parchive files, the amount of data that we would have to store would become significantly less. In our example, we would need to create five parchive files with a redundancy of 25%. One parchive volume file and the main par file would accompany each file. The file distribution would look like this:

• ubuntu.zip.001, ubuntu.zip.vol000+94.PAR2, and ubuntu.zip.par2 on Computer 1
• ubuntu.zip.002, ubuntu.zip.vol094+94.PAR2, and ubuntu.zip.par2 on Computer 2
• ubuntu.zip.003, ubuntu.zip.vol188+93.PAR2, and ubuntu.zip.par2 on Computer 3
• ubuntu.zip.004, ubuntu.zip.vol281+93.PAR2, and ubuntu.zip.par2 on Computer 4
• ubuntu.zip.005, ubuntu.zip.vol374+93.PAR2, and ubuntu.zip.par2 on Computer 5

The total required amount of disk space would be approximately 854 MB! This is 508 MB less disk storage than the previous solution, a savings of 37.3%!

The More, The Merrier

The nice thing about this solution is that the more workstations that you have, the less redundant overhead that you require. See the table below:

The Math

There are a lot of calculation that are being made for these configurations. All of these configurations are based on the number of archive locations. For these calculations, let’s assume that the number of archive locations is represented by a and the compressed file size in bytes is represented by z.

The number of files (f) equals the number of archive locations (a). This should be used for both splitting the compressed file and determining the number of parchive files to create.

We also need to plan how redundant we want our system to be. So, the number of locations that can be dead is represented by d. Please note that is it is very important that d < a (i.e. the number of archive locations must be greater than the number of dead locations).

Redundancy

The percentage of redundancy (r%) required can be calculated as follows:

r% = d / (a - d) * 100

Total Storage Required

The total storage (s) required for an individual file:

s = z * r% + z

Split File Size

Size of each file in bytes (b) when the compressed file is split:

b = z / f

Using the Calculations in QuickPar

I use QuickPar to create the parchive files. Here is a screenshot to show you where these calculations come in place in the QuickPar application.

Perform Your Own Manual Proof of Concept

Here is how you can do your own proof of concept for this type of a system:

Archiving

1. Download the software that you will work with. I use QuickPar to create parchive files and 7-Zip for file compression and splitting. I use these because they are freely available on the Internet.
2. Create archive locations. Since this is a proof of concept, they can be locations on remote systems or different folders on your own computer.
3. Compress your file using 7-Zip.
4. Determine the size of each file for splitting (b) and spit the file using 7-Zip.
5. Create the parchive files using QuickPar and the calculations provided above.
6. Move the files to your archive locations as indicated in the example above.
7. Move (or if your are interested in living dangerously, delete) the original file and the compressed file.

Recovery

1. Copy all but one location’s worth of the split and parchive files back into the original location. This will simulate the failure of one system.
2. Open up the main parchive file (this is the smallest file) in QuickPar.
3. Rebuild the lost/damaged files in QuickPar.
4. Recombine the files in 7-Zip.

Conclusion

Once again, this is a proof of concept just to show how a system like this would work. My next step would be to get AutoIt fired up and use command line versions of 7-Zip and QuickPar to automate the entire process.

So, what do you think of this idea? How could you use it in your environment? Let me know in the comments.

Trackback link - http://www.DailyCupOfTech.com/2007/08/16/distributed-file-archive-proof-of-concept/trackback/