notes on javascript, linux, and more

10.02.2008

unpacking tarballs efficiently onto flash devices

I recently installed NetBSD/hpcmips onto an old Windows CE based handheld PC. I tried installing pkgsrc, NetBSD's package management system, but the tarball was nowhere near done extracting to the CompactFlash card, even after leaving tar -xzf running all night. This is expected, given the incredibly low specs of the handheld PC, but I wanted to do something faster.

I fired up a blank virtual machine in VMWare Server, allocating a single gigabyte of disk space and a half gigabyte of RAM. I also added a USB controller to access my CF disk with later. I booted the machine from a NetBSD installation ISO image, and proceeded with a minimal installationg of NetBSD/i386. NetBSD is incredibly optmized and minimalist, so this took less than 5 minutes. I then rebooted the VM, and ran these commands:

mount /dev/sd-my-cf-card's-netbsd-partition /mnt/
tar -xzf pkgsrc.tar.gz -b 52000 -C /mnt/usr/

This finished a lot more quickly. The -b parameter sets the block size. I set this to approximately 26MB, which is the size of the pkgsrc tarball, meaning that the tarball was unpacked completely to RAM before being written to disk. This is a lot faster than:

mount /dev/sd-my-cf-card's-netbsd-partition /mnt/
tar -xzf pkgsrc.tar.gz -C /mnt/usr/

Try it and see! If working with dd, use the bs parameter to set the block size and see that you will get things done a lot faster as well. You'll have to play around with numbers; obviously you don't want to make the block size any bigger than your available RAM or your operating system will start swapping to disk anyway. My general rule of thumb is to use half of my available RAM as the blocksize, which should usually provide maximum efficiency and speed. Cheers.

No comments:

About Me

My photo
chicago, il, United States
I'm a software engineer by profession.

Labels