Troubleshooters.Com and T.C Linux Library Present

Backing Up to Blu-Ray

Copyright (C) 2012 by Steve Litt, All rights reserved. Material provided as-is, use at your own risk. 


Contents

Disclaimer

Getting a restorable backup of the right things is always a challenge. This page is an introduction to backing up to Blu-Ray, -- not an authoritative text. This page may have errors -- in fact it probably does. DO NOT trust what you see on this page without verifying it for yourself. I am not responsible for any data loss, damage or injury caused by your use of this document, or caused by errors and/or omissions in this document. If that's not acceptable to you, you may not use this document. By using this document you are accepting this disclaimer.

Underlying Backup Philosophy

This is just a how-to for getting your backup data onto a Blu-Ray disc. You can actually use it to get 25GB of pretty much anything onto a Blu-Ray disc.

If you want to know my underlying backup philosophy, please read these three documents:
If you read those, you'll see that my backup philosophy evolved to meet the hugely increasing volumes of material needed to be backed up. In 1989, all my data would back up to about 20 floppies. That's about 30 MEGAbytes. Today my data fills 24/25 of a Blu-Ray. That's 24 GIGAbytes. Different procedures, and different media, must be used.

What About Cloud Backup?

If you think a "cloud backup service" is what you need, I guess life is easy, and you don't need this document. But unless you have a very specific contract with the vendor, written by your lawyers rather than the vendor's, outlining what happens if the vendor loses, reveals, or misuses your data, then before you go down the cloud-backup road, here are some reasons I stayed a million miles away from it:
Cloud based backups have a couple advantages:
The easier offsite advantage is negated if you have a good, trusted friend with whom to leave your discs. If you can maybe twice a year ship discs to such a friend who is out-of-region, so much the better -- if a hurricane hits one of you, it will miss the other. This isn't covered here, but you can individually encrypt individual files on your backup discs so that if they fall into the wrong hands, there's not a problem.

The benefit of no need to set up your system is partially negated by the setup you need to do for the backup vendor. And of course, given the disadvantages of cloud backup, I prefer to do a little extra work and know my backups are secure and ready.

Hardware

The hardware you need in order to burn data to Blu-Ray as outlined in this document consists of:

The Blu-Ray Recording Process

From the high level, here's the process:
  1. Create the UDF file
  2. Format the UDF file
  3. Mount the UDF file to make a UDF file system
  4. Copy the material to be backed up to the UDF file system in whatever form you want
  5. Unmount the UDF file system in order to make sure everything's written
  6. Record the UDF file onto a Blu-Ray disc
  7. Test the accuracy of the Blu-Ray's contents
A couple points to consider: The old ISO9660 format isn't appropriate for Blu-Ray recording. First, it has a very small maximum size for an individual file. There are other reasons that a UDF filesystem is better for Blu-Ray than an ISO9660 file system.

The First Three Steps

Here's how you do the first three steps, assuming your UDF file is at /bigdisk/mybr.udf, and your desired mount point for the UDF filesystem is /mnt/test.
truncate --size=25GB /bigdisk/mybr.udf
mkudffs /bigdisk/mybr.udf
sudo mount -oloop,rw /bigdisk/mybr.udf /mnt/test
sudo chown slitt.slitt /mnt/test

Discussion

Use truncate to make the UDF file. Never use dd -- it will take forever. The truncate command takes about a second, versus many minutes or more for dd, no matter how fast the device you're dding it from. Use the size identifier "GB", never just "G". You want this thing to be exactly 25,000,000,000 bytes long, not 25x230 bytes.

Depending on your situation, you might or might not want to use additional arguments for mkudffs. If you're copying a directory structure with tens of thousands of small files, you'll probably want to use a blocksize argument of 1024 instead of the default 2048 bytes. If you're using huge files, you might do better with a blocksize of 4096 bytes. Absent a strong reason to mess with the blocksize, I leave it at its default. It's possible you might need to mess with the --media-type, especially if you'll be writing to a random access type disc, but my experience with going straight one track write-once Blu-Ray is I don't need a --media-type argument. If you need non-ascii characters in your filenames, there are Unicode and UTF-8 options. The mkudffs command has some really exotic arguments -- see the man page if you perceive a need. Unfortunately, as of this writing there appears to be no way to encrypt a UDF filesystem in-toto. However, you always have the option of encrypting the individual files that go on it. If you do what I do and make whole tarball backups, this is actually practical.

The final two commands loopback-mount the UDF file and set it to a user who can copy to it without sudo. If you're going to be repeatedly doing these same operations with the same UDF file and the same mountpoint (and I'd certainly recommend this), you might be better off with a line in /etc/fstab to get rid of the need for sudo. The following is one possible /etc/fstab entry:
/bigdisk/mybr.udf /mnt/test  udf  loop,noauto,user,noatime,nodiratime   0  0
Looking at the option list, loop means you're mounting a file, not a device -- it's a loopback mount. noauto means it won't automatically mount on boot -- you don't want that happening. user means an ordinary user, like you, can mount it, and once mounted, only that same user can unmount it. noatime and nodiratime mean that access times aren't recorded, reducing disk writes and situations where rsync re-copies because the access time made it think there was a change. The 0 on the end means don't fsck check it -- you're probably going to be recreating the filesystem each time -- it's a lot faster than mass deletion.

Once you have your /etc/fstab entry correctly entered, mounting should be as simple as:
mount /bigdisk/mybr.udf
and unmounting will be:
umount /mnt/test
Neither requires sudo because you used the user option in /etc/fstab. Note that the umount command must use the mountpoint, not the UDF file, as its argument. If you'd used the UDF file as its argument, it would have refused on the ground that "/dev/loop0 is not in the fstab". Likewise, don't use the mountpoint in the mount command, in case you have two devices that can mount to the same mountpoint. For instance, after you're all done making the UDF file, you could mount the Blu-Ray recorder to that same mountpoint to accuracy-check your recording.

Copying Your Data

Once your UDF file is mounted, its mount point becomes a proxy for your Blu-Ray disc. Whatever files, whatever directories, whatever structures (within reason) you copy to the mount point, are viewable on the final Blu-Ray in the same form. Be aware that 25GB of data, especially if contained in small files, is going to take a loooong time to copy. If there's a way you can do this copy while you're asleep, that's an excellent idea. If it's likely that the copy will be interrupted, rsync is your friend -- it copies only files differing on the source and destination. You may be able to kill two birds with one stone by using tar to compress a directory structure directly to the UDF file instead of first putting it somewhere, and then copying it onto your mountpoint. If you wonder if the copy is "stuck", you can cd to the mountpoint, and perform a few df . commands to view the progress. Use df, not du, because the former is much faster.

All filesystems have limits. I'm not sure what the limits of the UDF filesystem are -- I only know they easily accommodate my 10 tarballs with 10 md5 files and 10 file lists. If you're laying down an unusually large number of files in a given directory, or are going unusually deep with your directory structure, or have very long filenames, you might need to research these limitations.

When all your data copying is done, it is ABSOLUTELY VITAL that you unmount the UDF file. That's what makes sure the data is actually written inside the UDF file, which is now an image for your Blu-Ray recording.

The Final Three Steps

  1. Unmount the UDF file system in order to make sure everything's written
  2. Record the UDF file onto a Blu-Ray disc
  3. Test the accuracy of the Blu-Ray's contents
Unmounting is vital because only unmounting guarantees that every bit of data is written to your UDF file, instead of hanging out in RAM somewhere.

Recording the UDF

Most GUI recording programs haven't caught on to UDF yet, so you'll probably need to use this simple growisofs command:
growisofs -speed=1 -Z /dev/dvd=/bigdisk/mybr.udf
Be sure /bigdisk/mybr.udf isn't mounted when you do this: You're using it as a Blu-Ray image, not as a filesystem. If you have Blu-Ray media that are known to perform well at elevated speeds, you can use a higher speed. This document is being written in 2012. In a few years Blu-Ray media will record at higher speeds, but for now, if you have no-name Blu-Rays, use a speed of 1.

Note that this recording will take a looong time. Try to do it overnight when you won't need your computer. You'd really prefer not to be using the computer heavily while it's recording, because you could cause a bad recording if you loaded the machine up to the point where the recording process ran out of buffer.

Testing the Blu-Ray

How you test the Blu-Ray depends on what you put on it. For my backups, I have tarballs of each tree I backed up, and for each tarball, I have a .md5 file made when the tarball was made, before the tarball ever got transferred to the DVD. That way, once both the tarball and the .md5 are on the Blu-Ray, you can match them to confirm that you indeed still have a good tarball. I've made the following automated shellscript to do it. You might be able to improve on it -- please do -- it's licensed with the MIT license.
#!/bin/bash

# Copyright (C) 2012 by Steve Litt
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation files
# (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
# BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.


function check_useage(){
let errcount=0
if test "`whoami`" != "root"; then
echo
echo Error: Must be root to use this program. Use sudo.
echo
let errcount=$errcount+1
fi

num_args=$1
if test "$num_args" != "2"; then
echo
echo "Error: Must have 2 arguments. You had $num_args"
echo
let errcount=$errcount+1
fi
#echo errcount is $errcount
if test $errcount -ne 0; then
echo USAGE:
echo sudo ./verify_disc bluray_device mountpoint
echo
exit 1
fi
}


function comp_one(){
echo
echo
fmd5=$1

ftgz=`echo $fmd5 | sed -e "s/\.md5$/.tgz/"`

echo -n Testing $fmd5 and $ftgz ...
echo

md5stored=`cut -b1-32 < $fmd5`
md5live=`md5sum $ftgz | cut -b1-32`
# md5live="simulateerror"

echo Results, stored vs live: $md5stored $md5live

echo -n `date "+%Y%m%d at %k%M%S"` >> $logg
if test "$md5live" == "$md5stored"; then
echo " Md5sum OK for $fmd5" >> $logg
echo " Md5 is $md5stored" >> $logg
echo " Md5sum OK for $fmd5"
else
echo " Md5sum !!ERROR!! for $fmd5" >> $logg
echo " Stored md5=$md5stored, live=$md5live" >> $logg
echo " Md5sum !!ERROR!! for $fmd5"
let numerrors=$numerrors+1
fi
}

##### MAIN PROGRAM STARTS HERE #####
check_useage $#

mount $1 $2
cd $2

let numerrors=0
export numerrors
logg=/home/slitt/verify_disc.log
export logg

echo >> $logg

### VERIFY RUN HEADER
echo -n "======= " >> $logg
echo -n `date "+%Y%m%d at %k%M%S"` >> $logg
echo -n " Starting new disc verify." >> $logg
echo " =======" >> $logg

### LOOP THROUGH DISC
shopt -s nullglob
let md5count=0
for i in *.md5; do
comp_one $i
let md5count=$md5count+1
done

### VERIFY RUN FOOTER
echo -n "======= " >> $logg
echo -n `date "+%Y%m%d at %k%M%S"` >> $logg
echo -n " Disc verify finished. " >> $logg
if test $numerrors -eq 0; then
echo -n "Congrats, all files, $md5count total, verified." >> $logg
else
echo -n "ERROR count is " >> $logg
echo -n $numerrors >> $logg
fi
echo " =======" >> $logg
echo >> $logg
echo

cd -
umount $1
You run this script like this:
sudo ./verify_disc bluray_device    mountpoint
The script iterates through every .md5 file on the mountpoint, and for each one it does an md5sum on the corresponding .tgz file. If their signatures match, the file's OK, otherwise it has problems. A log file is created (be sure to change this log file location to something relevant to your computer).

Summary

So that's it. The process looks like this:
  1. Create the UDF file
  2. Format the UDF file
  3. Mount the UDF file to make a UDF file system
  4. Copy the material to be backed up to the UDF file system in whatever form you want
  5. Unmount the UDF file system in order to make sure everything's written
  6. Record the UDF file onto a Blu-Ray disc
  7. Test the accuracy of the Blu-Ray's contents
This article contains the details.



Back to Troubleshooters.Com * Back to Linux Library