Streaming Disk Images Over the Network

In 2009, a customer needed a way to stream disk images over the network when imaging new machines. Since the machines were new, the images were mostly empty space, which made them good candidates for compression. CPU was a bigger bottleneck than network bandwidth, and he decided the existing compression options, including gzip, were too slow. He also wanted to use udpcast to image multiple machines at once.

Another developer and I worked on a proof of concept, passing code back and forth through the customer. The other developer wrote the first version, and I found several large speed gains.

We chose to leave the data uncompressed and compress only the empty blocks.

What I Did

• Disk prep. Asked the customer to defragment the source disk and zero its unused space.

• Faster reads. Read the disk in 1 MB chunks instead of byte by byte.

• Faster empty-block checks. Used long long to compare 8 bytes at a time instead of one. A 64-bit build checks each value in one step, and a 32-bit build took two.

• Large file support. Made sure the program could handle the 80 GB disk images on 32-bit systems.

• Cleanup and fixes. Added comments, counted sent bytes accurately, and fixed a bug that cut the output short.

• Standard input and output. Changed the program to send its compressed output to stdout, and to read from stdin when decompressing. The customer could still save images to a file, and could now pipe them straight into udpcast to image several machines at once:

  # ./scomp <source-disk> | udp-sender
  # udp-receiver | ./scomp -r <target-disk>
  

The program was written in plain C. I knew SSE2 and was comfortable mixing assembler into C, but the user and system times I was reviewing with the customer didn’t point to enough of a gain to justify the effort.

The Numbers

I don’t have a clean timing of the first version. Early on, we were testing CPU, memory, disk write and network speeds on the customer’s systems, and those tests can’t be compared with the ones below, which I ran on the final sets of changes.

Compressing and decompressing a test file on one machine, with no network involved:

• gzip and gunzip benchmark: 8.613 seconds (5.568 CPU, 3.045 blocking).
• My changes: 5.274 seconds (1.556 CPU, 3.718 blocking).
• The other developer’s second round of changes: 5.671 seconds (1.472 CPU, 4.199 blocking).
• Both, combined: 4.468 seconds (0.872 CPU, 3.596 blocking).

The combined changes took a little over half of gzip’s time. Most of their time went to waiting on I/O. They also beat my changes alone while doing slightly more processing of the data.

The customer’s tests, in his own environment, showed the program still sent the same amount of data as gzip: about 5 GB for an 80 GB disk.

In that same environment, my changes alone brought network transfers down from 20 minutes with gzip to 7 minutes, 15 seconds. The customer didn’t report final numbers for the combined version.