In 1998 I woke up and my PC was making a lot of noise… I've been a System Engineer ever since.

Greg Maxwell at Xiph.org has a great article where he compares Ogg/Theora to x264 video at YouTube video sizes and bit rates.  My goal here is to provide a step by step guide to reproduce Greg Maxwell’s results so you can prove it for yourself.  Greg outlined the following steps which we will reproduce using a fresh installation of Ubuntu Karmic 9.10 i386.

• Obtain the lossless 640×360 Big Buck Bunny source PNGs and FLACs from media.xiph.org.
• Resample the images to 480×270 using ImageMagick’s convert utility.
• Use gstreamer’s jpegenc, produce a quality=100 mjpeg + PCM audio stream. The result is around 1.5Gbytes with a bitrate of around 20Mbit/sec.
• Truncate the file to fit under the YouTube 1Gbyte limit, resulting in input_mjpeg.avi (706MiB).
• Upload the file to YouTube and wait for it to transcode.
• Download the FLV and H.264 files produced by YouTube using one of the many web downloading services. (I used KeepVid)
• Using libtheora 1.1a2 and Vorbis aoTuv 5.7 produce a file of comparable bitrate to the youtube 499kbit/sec from the same file uploaded to YouTube (input_mjpeg.avi).
• Resample the file uploaded to YouTube to 400×226.
• Using libtheora 1.1a2 and Vorbis aoTuv 5.7 produce a file of comparable bitrate to the youtube 327kbit/sec from the 400×226 downsampled copy of input_mjpeg.avi

Step 1 – Prepare the Environment:

1. Install Ubuntu 9.10 Karmic and install all available updates.
2. Download BBB-360-png.tar and BigBuckBunny-stereo.flac from one of the following locations and save to your Desktop:

   http://media.xiph.org/BBB/BigBuckBunny-stereo.flac

   http://gregfaust.net/files/BigBuckBunny-stereo.flac
   http://media.xiph.org/BBB/BBB-360-png.tar
   http://gregfaust.net/files/BBB-360-png.tar

3. Install the packages we will need: sudo apt get install vorbis-tools imagemagick gstreamer-tools mplayer-nogui automake libtool
4. Extract and prepare the files for encoding:
   cd Desktoptar -xvf BBB-360-png.tar

   mkdir BBB-270-png

   mkdir BBB-226-png

   cd BBB-360-png

   ls *.png | xargs -I FILE convert FILE -resize 480×270\! ../BBB-270-png/FILE

   ls *.png | xargs -I FILE convert FILE -resize 400×226\! ../BBB-226-png/FILE

   cd ../BBB-270-png

   gst-launch multifilesrc location=big_buck_bunny_%05d.png index=1 caps=”image/png,framerate=24/1″ ! pngdec ! ffmpegcolorspace ! jpegenc quality=100 ! queue ! mux. filesrc location=../BigBuckBunny-stereo.flac ! flacdec ! audioconvert ! audioresample ! queue ! mux. avimux name=mux ! filesink location=input270_mjpeg.avi

   cd ../BBB-226-png

   gst-launch multifilesrc location=big_buck_bunny_%05d.png index=1 caps=”image/png,framerate=24/1″ ! pngdec ! ffmpegcolorspace ! jpegenc quality=100 ! queue ! mux. filesrc location=../BigBuckBunny-stereo.flac ! flacdec ! audioconvert ! audioresample ! queue ! mux. avimux name=mux ! filesink location=input226_mjpeg.avi

   cd ..

   mplayer input1_mjpeg.avi -ao pcm -vo yuv4mpeg

   mv stream.yuv stream1.yuv

   mplayer input2_mjpeg.avi -ao pcm -vo yuv4mpeg

   mv stream.yuv stream2.yuv

5. The libtheora reference encoder accepts (lossless) yuv4mpeg video and PCM audio.  The yuv4mpeg video we are supplying was created from a lossy .avi source (mjpeg quality=100) because YouTube does not accept lossless input and this is as close as we can get to lossless.  To be fair we are starting with the same lossy input for libtheora, but if we were doing this for real, we would want to go convert directly from png to yuv4mpeg and skip the step where we convert to mjpeg. Note that we are using the reference Vorbis encoder for simplicity and not the tuned AoTuV Vorbis encoder (which in my opinion is better), maybe I’ll do another blog post on AoTuV in the future. cd libtheora/examples
   
   ./encoder_example –output ../../output1_theora.ogv -A64 -V345 –two-pass –keyframe-freq 250 ../../stream1.yuv ../../audiodump.wav ./encoder_example –output ../../output2_theora.ogv -A64 -V261 –two-pass –keyframe-freq 250 ../../stream2.yuv ../../audiodump.wav
6. Now you have a file named output1_theora.ogv which contains 480×270 video at approx 499kbit/sec and audio at 64kbit/sec.  You will also have a file named output2_theora.ogv which contains 400×226 video at approx 325kbit/sec and audio at 64kbit/sec.  In my opinion, this renders better video and audio quality than YouTube.  To compare you YouTuve, you will need to have a YouTube account and upload the files input1_mjpeg.avi and intput2_mjpeg.avi.UPDATE:  It looks like Google may open source the On2 VP8 codec it acquired along with On2 in February.  An open source VP8 could eliminate concerns about H.264 licensing and Theora quality.

Execute the command (replace the IP address with the IP address of your target server):

On Linux, to determine the device name for your source disk, use the following command:

On OpenSolaris or Solaris, to determine the device name for your source disk, use the following command:

Now that you have your lightning fast ZFS iSCSI storage target online, lets connect it to Windows 7 64-Bit and do some testing.

First you will need to configure networking so that your iSCSI target will be available.  In my machine, I have 2 network adapters.  One is connected to the Internet via DHCP and the other is manually configured as displayed below and connected to my iSCSI target machine via an Ethernet crossover cable.  The iSCSI target IP is 192.168.2.1 which is on the same subnet as the iSCSI initiator IP 192.168.2.2.

Next you will need to configure the iSCSI Initiator Service.

Navigate to Control Panel –> System and Security –> Administrative Tools –> iSCSI

You will be prompted to start the iSCSI service and to create a Windows Firewall exception, to be successful, you will want to perform both of these actions.

Next you will need to “discover” your iSCSI target.  On the “Discovery” tab, then click “Discover Portal” and enter the IP address of your iSCSI target.

Next click on the “Targets” tab, highlight your target and click “Connect”.

You will be prompted to make this a persistent target.  In my case, I am experimenting so I do not want it to be persistent.  If I was configuring clustering or something else, I might want my target to be persistent.

Next, you will need to initialize your newly connected iSCSI disk.  Right Click on “Computer” select “Manage”, then expand “Storage” and click on “Disk Management”.

You will immediately be prompted to initialize the new disk.  In my machine, I decided to use an MBR partition because I will not have a volume larger than 2TB and I would like maximum compatibility.

Once the disk is initialized, you will be prompted to format the disk and select a drive letter.  For maximum performance, it is important to choose a 64K “Allocation Unit  Size” when you format the disk.  This “Allocation Unit Size” coincides the with the 64K block size that we used when creating the ZFS volume that backs our iSCSI target.  Aligning the partitions can give a significant performance boost.  Here is a .pdf document that describes why we want to alight the partitions and what kind of performance boost we can expect.  Although this document is written in regards to VMFS, the same holds true for other file systems such as NTFS.

Now we have a new drive letter.  As we test further, it is important to remember that ZFS properties such as compression=on and dedup=on are not retro-active.  This means that when we turn on ZFS properties, they will apply to new data as it is written to the ZFS volume, but it will not compress data that is already in place on the volume.  In the previous post where we created the iSCSI target, I turned on ZFS compression at the time the volume was created.  This way, all of my data will be compressed.  Unfortunately, my personal data does not compress or dedup very well (not at all).  I guess this makes sense since it is mostly .jpg photos, home videos, and AAC music.  However, your results will vary.  In the office we managed to get a 1.54x dedup ratio.  You will not want to use dedup in all cases because it might not provide an benefit and will create some overhead because it has to calculate a SHA256 hash for every block of data.  On my personal data that doesn’t seem to dedup at all, it also seems that dedup thrashes my disks causing additional slowness on top of the SHA256 hashing.  This is likely because the SHA256 hash is too large to fit into RAM and my machine does not have L2ARC cache. Jeff Bonwick at Oracle/Sun has a great writeup on the inner workings of ZFS deduplication. Depending on your data, you may be able to use compression and dedup together.  Experiment with your own data using the commands below:

admin@opensolaris:/etc# zfs set compression=on pool1/vol1

admin@opensolaris:/etc# zfs set dedup=on pool1/vol1

admin@opensolaris:/etc# zfs get compressratio pool1/vol1

admin@opensolaris:/etc# zfs set compression=on pool1/vol1

admin@opensolaris:/etc# zpool get dedupratio pool1

I hope this has been informative and I would like to thank you for viewing.  Stay tuned for the next experiment which will use the same ZFS iSCSI server as shared storage to cluster Microsoft Hyper-V 2008 R2.

Thanks to the folks over at genunix.org, I have a freshly burned CD-ROM of OpenSolaris 2010.03 preview build 133.  I manage a Sun 7410 Unified Storage Appliance (SAN) at work and I’m very interested in the inner workings of the COMSTAR framework used by this appliance.  Unfortunately, using the command prompt on the Sun 7410 (without having Sun Support on the phone) voids the support contract.  Luckily, the Amber Road GUI is feature rich and I haven’t found anything I need the command prompt for other than tinkering.

To learn more about COMSTAR my plan is to install OpenSolaris on my personal machine and configure it as an iSCSI target backed by a ZFS file system.  If the experiment goes well, we may decide to use similar configurations to expand our SAN at work.

I am especially interested to test an iSCSI target backed by a ZFS volume running deduplication and find out what kind of ratio I can get.  I’m afraid that the files will not line up on block boundaries and there might be poor deduplication performance…  lets find out if I’m wrong.

Lets talk requirements for this experiment:

1. ZFS mirror volume for boot and /
2. ZFS stripe volume for data
3. Maximum Performance

Hardware List:

• Motherboard: Gigabyte GA-EP35-DS3R
• CPU: Intel Xeon E3110 (equivalent to C2D E8400)
• RAM: 8GB Mushkin DDR2 (PC2 6400) Dual Channel
• 2x Seagate 320GB SATA II Disks (boot and /)
• 2x Seagate 500GB SATA II Disks (data)
• ATI Radeon HD 4850 (512MB GDDR3)
• IDE DVD-ROM from an old dell desktop

Step 1: Install OpenSolaris

Use the GUI, have the partitioner use an entire disk for the installation (installing ZFS on a slice huts performance because the drive’s cache is not used effectively).  Currently, OpenSolaris 2010.03 build 133 does not have a text installer so you will need to use the graphical installer.

Step 2: Configure Mirroring

Prepare the 2nd disk to be used for the root mirror using the format command.  Delete any existing partitions, create a new Solaris2 partition using the entire disk.  For my 320GB SATA disk the partition looks like this when it’s ready.

Partition   Status    Type          Start   End   Length    %
=========   ======    ============  =====   ===   ======   ===
1                 Solaris2          1  38912    38912    100

Step 3: Copy the Volume Table of Contents from the disk used by the GUI Installer to the disk you just partitioned.

admin@opensolaris:/etc# prtvtoc /dev/rdsk/c6d0s2 | fmthard -s – /dev/rdsk/c7d0s2
fmthard:  New volume table of contents now in place.

Step 4: Attach the 2nd disk to the root mirror

admin@opensolaris:/etc# zpool attach -f rpool c6d0s0 c7d0s0
Please be sure to invoke installgrub(1M) to make ‘c7d0s0′ bootable.
Make sure to wait until resilver is done before rebooting.

Step 5: Install Grub on the disk you just added to the root mirror

admin@opensolaris:/etc# installgrub -m /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c7d0s0

Step 6: You’re basically done with root, but…

Make sure the new root mirror finishes resilvering before you try to reboot.  In the case of my 320GB disk resilvering took 10 minutes and I was able to install grub while the resilvering was running.

admin@opensolaris:/etc# zpool status
pool: rpool
state: ONLINE
scrub: resilver completed after 0h10m with 0 errors on Mon Feb 22 21:42:48 2010
config:

NAME        STATE     READ WRITE CKSUM
rpool       ONLINE       0     0     0
mirror-0  ONLINE       0     0     0
c6d0s0  ONLINE       0     0     0
c7d0s0  ONLINE       0     0     0  7.68G resilvered

errors: No known data errors

Step 7: Configure Networking

Use the GUI application located at System –> Administration –> Network to manually configure networking.  Once the GUI settings are complete, execute the following commands to use DNS for Internet name resolution.  I have two NICS, i will use DHCP on one for Internet access and static on the other for iSCSI access.

admin@opensolaris:~# cd /etc
admin@opensolaris:/etc# cp nsswitch.conf nsswitch.conf.original
admin@opensolaris:/etc# cp nsswitch.dns nsswitch.conf
admin@opensolaris:/etc# pkg install storage-server

Step 8: Enable the STMF Service

admin@opensolaris:/etc# svcadm enable stmf

Step 9: Create a new ZFS volume

replace the device names with the names of your disks from Step 2 above.  I am using a 64k block size to optimize VMware performance.

admin@opensolaris:/etc# zpool create -f pool1 c6d1 c7d1

admin@opensolaris:/etc# zfs create -b 64k  -V 885G pool1/vol1

admin@opensolaris:/etc# zfs set compression=on pool1/vol1

Note: you probably want compression unless your CPU is old and you have a ton of disks.  ZFS uses LZJB compression by default and most likely your CPU has more compression bandwidth than your disks have platter bandwidth.  Multiple benchmarks are available online.  Here is one that is well written and uses ZFS and compression for database storage.

Step 10: Create an iSCSI Logical Unit using the volume.

admin@opensolaris:/etc# sbdadm create-lu /dev/zvol/rdsk/pool1/vol1

Created the following LU:

GUID                    DATA SIZE           SOURCE
——————————–  ——————-  —————-
600144f01db0830000004b83463c0001  950261514240         /dev/zvol/rdsk/pool1/vol1

Step 11: Make the Logical Unit available to all hosts

substitute the GUID from the previous step

admin@opensolaris:/etc# stmfadm add-view 600144F01DB0830000004B83463C0001

Step 12: Enable an iSCSI Target Port

admin@opensolaris:/etc# svcadm enable -r svc:/network/iscsi/target:default
svcadm: svc:/milestone/network depends on svc:/network/physical, which has multiple instances.

Step 13: Configure the iSCSI target for discovery

admin@opensolaris:/etc# devfsadm -i iscsi

That’s it for the OpenSolaris Side.  In my next post, we’ll configure the Windows iSCSI initiator and test deduplication & compression performance for NTFS on iSCSI.  If you’re really hardcore and want maximum performance, you can disable the OpenSolaris GUI with the following command.

admin@opensolaris:/etc# svcadm disable gdm

Thanks to DarkStar for guidance on how to configure a root mirror with a GUI installer, this is not well documented.