Daily Cup of Tech

A few weeks ago, I ran across a couple of articles about a university in Holland that was having difficulty getting all of their Microsoft patches out to all of their systems. What did they decide to do?

Buy more servers? Nope!

Increase bandwidth? Nope!

Use a program that has been essentially outlawed in North American universities? Yep!

The university decided, on the advice of their IT consultant, that they would use bittorrent to deploy the patches.

This really got me thinking about how I would go about creating my own U.Holl server (as I like to call it). These are some of the thoughts that I have come up with.

The Server

Just to add a really strong sense of irony, I would probably build the U.Holl server on Linux, most likely Ubuntu with a typical LAMP installation. Then, for ease of management, I would install a tracker such as phpMyBittorrent or something similar.

In the tracker software, I would create several different categories based on the target systems that I would use. For example, there would be Windows XP, Windows Vista, Office 2007, etc. categories. Each category would have its own RSS feed.

The Clients

Each workstation or server that was to receive updates using bittorrent would have a copy of uTorrent running in the background. This would be critical. You would also need to tweak uTorrent so that it automatically ran an installation script r program each time it finished downloading a new patch. This script would also schedule a reboot at 3:00 AM (or whatever time you wanted) since it seems almost every hotfix from Microsoft needs to reboot the system.

A potential install script might look like this:

The trick to automatically downloading files is to subscribe each uTorrent client to the proper RSS feeds given out by the server. This way, each system only gets the patches it needs.

Potential

I think that this type of a delivery system has great potential. You could even use it across slow bandwidth links or to deploy other software, not just patches.

This is a project that I am hoping to spend some time on over the next while but I am looking for your input/suggestions now so that I can hopefully avoid some pitfalls later. Any input would be greatly appreciated!

Trackback link - http://www.dailycupoftech.com/2008/06/12/diy-bittorrent-patch-server/trackback/

It has been long time since I have watched television like “normal” people. Ever since I found bittorrent, there was no turning back!

The problem that I have is that I am a pack rat! Once I download a TV episode, I don’t want to get rid of it. After all, I used up all that bandwidth!

So, I’ve started organizing my television shows into three categories:

1. Downloaded - these are shows that have just completed downloading and are in their original form from the Internet
2. Unsorted - these shows are in in one central location and should be in playable format. These are all my new shows that I probably have not watched yet
3. Sorted - once I have watched a show, the shows get moved into appropriate folders based on series and season name

The issue that I have run into in the past is that once an episode is downloaded, it may not be in playable format since many shows are archived, typically into RAR files. This means that I needed to manually extract these files or move the playable files to the Unsorted folder so that I could watch them.

I finally got fed up with this and I thought that there had to be a better way. It took me a couple of evenings work but I finally came up with a batch files that does the work for me! I simply have Windows programmed to run the batch file on a regular basis and it extracts all of the RAR files to my preferred location and moves all of the AVI files there as well (without copying the samples).

Here is the contents of that batch file:

@echo off
setlocal
cls

rem Set program variables below:
rem sourcedir - where files are saved once they have been downloaded
set sourcedir=C:\Downloaded

rem destinationdir - where you want all of the playable files to be placed
set destinationdir=C:\Unsorted

rem winrarpath - location of the unrar program
set winrarpath=C:\Program Files\WinRAR\

rem filelist - the name of the file which will store all of the RAR and AVI file names that have been processed
set filelist=FoundFiles.txt

rem tempfile - name of the temporary file that will be created and then deleted during the cleanup process
set tempfile=TempFile.txt

set path="%winrarpath%";%path%

echo Extracting new RAR files...
for /R "%sourcedir%" %%R IN (*.rar) DO find /C "%%R" %filelist% > NUL & if errorlevel 1 call unrar e -o- -y "%%R" *.* %destinationdir%\ & echo %%R>>%filelist%

echo Adding sample AVI files to no copy list...
for /R "%sourcedir%" %%R IN (*sample*.avi) DO find /C "%%R" %filelist% > NUL & if errorlevel 1 echo %%R>>%filelist%

echo Moving new AVI files...
for /R "%sourcedir%" %%R IN (*.avi) DO find /C "%%R" %filelist% > NUL & if errorlevel 1 echo Copying %%R... & copy "%%R" %destinationdir%\ & echo %%R>>%filelist%

echo Cleaning up %filelist%...
for /F %%R IN (%filelist%) DO if exist "%%R" echo %%R>>%tempfile%
del %filelist%
rename %tempfile% %filelist%

All you need to do is copy this file and save it as TorMove.cmd.  Then edit the three variables:

• sourcedir
• destinationdir
• winrarpath
• filelist
• tempfile

Only the first three are critical.  You just need to make sure that the last two do not clash with other files.

Also, you do not need to use WinRAR.  If you do not have this program, UnRAR for Windows will work just fine.

Once you have modified your file, save it and then schedule Windows to run it as often as you want!

Trackback link - http://www.dailycupoftech.com/2008/05/14/automatic-torrent-extractormover-script/trackback/

Many of you are familiar with the concept of computer imaging. Essentially, you take a system, configure it the way that you would like, and then duplicate the hard drive to all of the other systems, saving yourself all of the same steps that you did on the first system.

Now, there are a lot of great commercial software programs out there that can do this for you. Acronis and Ghost come to mind. But, not everyone can afford these options. For those of you find yourself in this situation, I present you with a list of options that are available for free or are open source. (All descriptions are copied from their respective websites.)

Partimage is a Linux utility which saves partitions having a supported filesystem to an image file. Most Linux and Windows filesystems are supported. The image file can be compressed with the gzip / bzip2 programs to save disk space, and they can be splitted into multiple files to be copied on CDs / DVDs, … Partitions can also be saved across the network since version 0.6.0 using the partimage network support, or using Samba / NFS. If you don’t want to install Partimage, you can download and burn SystemRescueCd. It’s a livecd that allows to use Partimage immediately even if your computer has no operating system installed (useful to restore an image), and it allows to save an image on a DVD on the fly.

PING (Partimage Is Not Ghost)

PING is a live Linux ISO, based on the excellent Linux From Scratch (LFS) documentation. It can be burnt on a CD and booted, or integrated into a PXE / RIS environment. Several tools have been added and written, so to make this ISO the perfect choice to backup and restore whole partitions, an easy way. It sounds like Symantec Ghost(tm), but has even better features, and is totally free.

Clonezilla

Clonezilla, based on DRBL, Partition Image, ntfsclone, and udpcast, allows you to do bare metal backup and recovery. Two types of Clonezilla are available, Clonezilla live and Clonezilla server edition. Clonezilla live is suitable for single machine backup and restore. While Clonezilla server edition is for massive deployment, it can clone many (40 plus!) computers simultaneously. Clonezilla saves and restores only used blocks in the harddisk. This increases the clone efficiency. At the NCHC’s Classroom C, Clonezilla server edition was used to clone 41 computers simultaneously. It took only about 10 minutes to clone a 5.6 GBytes system image to all 41 computers via multicasting!

IRestorer

Drive Snapshot (IRestorer) creates an exact drive image of your system into a file, including the operating system, installed programs, your data and all security attributes, while windows is running and you continue to work. If disaster strikes (a virus, worm, trojan, or hard disk crash), a complete and exact restoration of your system will be possible in only few minutes. The image file can be mounted as a virtual drive, so you can easily restore (or use) single files or directories, using the Windows Explorer, or other programs. Restart to DOS is only necessary, if you must restore your system partition. Otherwise there is no need to reboot, not even during installation. Drive Snapshot saves only the used disk space, compresses the data and is extremely fast, a P4 - 3.0 GHz will save approx. 2.5 GB/Min on a modern hard disk!

Linbox Rescue Server

The Linbox Rescue Server is an asset management software including 5 modules :

• system backup for emergency crash recovery, hard disk cloning or deployment using a
  PXE network boot. This module was fully developped by Linbox FAS.
• file backup, based on the famous BackupPC, to which we have added a configuration interface,
• inventory, based on ocs-inventory agents, and on an agent which runs during the PXE network boot,
• Software deployment module which works with Linux, MacOS X and MS Windows clients, or any Un*x system running OpenSSH,
• remote control, based on TightVNC.

HDClone

Copies smaller drives onto larger ones for migrating or backing up complete installations as well as for data rescue. The download package contains a program to easily create a bootable floppy disk or CD/DVD under Windows with only few clicks. A floppy image, ISO image, and manual as PDF are also enclosed.

DiscWizard

If you are setting up your new drive, you can use the install instructions to assist in mounting the drive, then use the tools built into your operating system to prepare the drive.

G4L

G4L is a hard disk and partition imaging and cloning tool. The created images are optionally compressed and transferred to an FTP server instead of cloning locally.

MaxBlast 5

If you are setting up your new drive, you can use the install instructions to assist in mounting the drive, then use the tools built into your operating system to prepare the drive.

DriveImage XML Backup Software

DriveImage XML is an easy to use and reliable program for imaging and backing up partitions and logical drives.

The program allows you to:

Backup logical drives and partitions to image files

Browse these images, view and extract files

Restore these images to the same or a different drive

Copy directly from drive to drive

Schedule automatic backups with your Task Scheduler

Image creation uses Microsoft’s Volume Shadow Services (VSS), allowing you to create safe “hot images” even from drives currently in use.Images are stored in XML files, allowing you to process them with 3rd party tools. Never again be stuck with a useless backup!

Restore images to drives without having to reboot.

DriveImage XML runs under Windows XP, Windows Server 2003 and Windows Vista only. The program will backup, image and restore drives formatted with FAT 12, 16, 32 and NTFS.

DuBaron DiskImage

DiskImage is the ultimate tool to backup and restore partitions, entire harddisks, usb keys, floppy drives and (read-only) optical media. DiskImage features built-in zip compression, it’s own compression method for data
> 2Gb, MD5 and SHA1 checksumming, a hex editor, and various tools like drive speed and seek testing.

Dumping and restoring partitions, usb sticks, floppies, optical media, diskimage is the swiss army knife.

WinDD - Disk Dump for Windows

WinDD - Disk Dump for Windows! Windows XP version of Unix ‘dd’ command. Safe, effort-free backup for FAT, FAT32, NTFS, ext2, ext3 partitions.

Partition Saving

Partition Saving is a DOS and Windows program that is used to save, restore and copy hard-drive, partitions, floppy disk and DOS or Windows devices.

With this program you could save all data on a partition to a file (such as you could save this file on a CD for example). Then if something goes wrong, you can completely restore the partition from the backup file. You no longer have to reinstall every piece of software from scratch. All you have to do is restore the partition from the backup file and then update any software that was modified since the backup was created.

NFGDump

nfgdump (Ntfs-Fat-Generic-dump) is a Win32/Linux tool that dumps/restores NTFS (including 3.1=XP), COMPRESSED NTFS, FAT16, FAT32 and arbitrary (generic) partitions. Dump files support compression, encryption, splits, page file content removal, etc.

Forensic Acquisition Utilities

This is a collection of utilities and libraries intended for forensic or forensic-related investigative use in a modern Microsoft Windows environment. The components in this collection are intended to permit the investigator to sterilize media for forensic duplication, discover where logical volume information is located and to collect the evidence from a running computer system while at the same time ensuring data integrity (e.g. with a cryptographic checksums) and while minimizing distortive alterations to the subject system. The components of this package are not intended to preclude changes to the subject system while the evidence collection process is under way. A third party hardware or software write blocker should be employed in those circumstances where it is deemed necessary to guarantee that no changes occur to the subject volume prior to and after the imaging process.

Self Image

SelfImage is the little hard drive utility with big aspirations.SelfImage is capable of making an image file of a hard disk or hard disk partition, and can restore an image back to any drive or partition that doesn’t have open files. Useful for making backups. Unlike dd for Windows (or cygwin), SelfImage is capable of creating an image of a partition that is currently in use.

SelfImage can even create images of partitions that Windows doesn’t recognize (partitions that Windows doesn’t have mounted on a drive letter). This is perfect for the dual-boot system, you can create an image backup of a Linux partition directly from Windows.

PC inspector clone maxx

PC INSPECTOR™ clone maxx is the new professional hard drive copying program from CONVAR. Using the new direct DMA support, data can be copied from hard drives in high speed mode with speeds up to 3.3 GB per minute.This high speed is achieved by PC INSPECTOR™ clone maxx through intelligent inspection of the PC motherboard and the hard drive controller. The software independently determines the fastest data transmission rate and achieves these exceptional speeds through optimal utilization of the DMA chipset on the motherboard of your PC.

PC INSPECTOR™ clone maxx can be started directly from a boot diskette. The copying process is always based on the physical drive and is independent of the file system (e.g. FAT12, FAT16, FAT32, HPFS, NTFS, Ext2, Reiser, etc.) or the number of partitions.

g4u - Harddisk Image Cloning for PCs

g4u (”ghosting for unix”) is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common setup on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported.

FOG :: A Ghost-like Computer Cloning Solution

Fog is a Linux-based, free and open source computer imaging solution for Windows XP and Vista that ties together a few open-source tools with a php-based web interface. Fog doesn’t use any boot disks, or CDs; everything is done via TFTP and PXE. Also with fog many drivers are built into the kernel, so you don’t really need to worry about drivers (unless there isn’t a linux kernel module for it). Fog also supports putting an image that came from a computer with a 80GB partition onto a machine with a 40GB hard drive as long as the data is
less than 40GB.

Fog also includes a graphical Windows service that is used to change the hostname of the PC, restart the computer if a task is created for it, and auto import hosts into the FOG database. The service also installs printers, and does simple snap-ins.

Each has its own pros and cons but all have a purpose. I am personally really liking FOG and am finding a lot of different places to put it to work.

Trackback link - http://www.dailycupoftech.com/2008/05/06/free-or-open-source-imaging-solutions/trackback/

An old computer these days is no longer a completely useless system. In fact, even though I might upgrade my main computer every couple of years, I usually take ten to fifteen years before I completely remove it from my home network and send it to that great LAN in the sky!

Here are ten twelve uses that I have discovered for that more “experienced” computer system!

And once you have read through the article and decided which exciting project you want to do, make sure you let us know by voting in the poll!

Read the rest of the story…

Trackback link - http://www.dailycupoftech.com/2007/09/17/put-that-old-computer-to-good-use/trackback/

If you have ever done a trace route on where information goes when it travels over the Internet to your computer, it is actually pretty staggering to see how far it goes. It is pretty funny sometimes to see an e-mail travels half way around the world and back just to get to your neighbor!

I got thinking about this and an idea came to mind that could change how we look at file sharing.

How It’s Done Now

Let’s say, for example, I want to share a 600 Mb ISO file with my neighbor. There are several different ways of doing this but let’s say that the easiest way to do this would be to transfer the file via a peer to peer program.

This would cost both of us about 600 MB of bandwidth that we would have to pay each of our Internet provider’s for.

WiFi Transfer

Now, my neighbor gets smart when he realizes that both of us have WiFi networks available to our systems. So we connect our two WiFi networks and decide to transfer the file over our WiFi networks.

We’ve now transfered the file between the two of us and it didn’t use any of our Internet bandwidth. In fact, we were able to transfer that file much faster because we were not limited by the upload speed of our Internet connections (Upload speeds are usually less that download speeds.).

WiFi Hops

Now, let’s say that I want to get a file from a second neighbor. But, he is unfortunately outside the range of my WiFi signal. But, my first neighbor, who is located between us, is in range of both of our WiFi signals. He connect to both of our WiFi networks and acts like a relay connection between our two networks.

So, we have now transfered a file using WiFi between two systems that are outside of each other’s WiFi signal range.

Spreading the Quilt

Theoretically, the more people we connect together with this WiFi quilt, the more access we have to information and data.

Each one of these systems would have access to information on each of the other systems, allowing for data to be shared freely without the need or restrictions of their Internet connections.

Stitching Together the Quilt

Just like a quilt is made up of separate pieces of cloth, each metro WiFi network will be limited to the range of its WiFi area. This is where we use the Internet to make data available to each of the metro areas.

Whenever a system needs to go outside of its metro area to access information via the Internet, it can maintain a cache of the information so that it now becomes available to the rest of the metro area.

Concerns and Issues

This idea is not without its problems. For example, I do not think I would give just anyone unfettered access to my home computer or network. Rather, I would probably want to segment my network so that only a certain computer works on the shared.

Another problem is coverage. In order for this idea to work well, there needs to be a relatively large percentage of area covered to see a benefit. But, this might be one of those things where you do not see a lot of benefit at the beginning of the project but, as time goes on, it becomes more and more feasible and useful

The other issue is with this project is just how do people find the information that they want and how does the system go about determining which is the best route to access the information?

Discussion

Even though there are some definite challenges with this idea, I still think that it would be a really interesting thing to try and set up. It could even include some contributions from things like the OpenWRT project, DIY WISP, and cantennas.

But, I’ve talked enough.  Time for you to add your two cents.  What do you think of this type of an idea?

Trackback link - http://www.dailycupoftech.com/2007/09/10/the-wifi-quilt-project/trackback/