Software RAID and Linux

Part of my day to day activities is to support RedHat Enterprise Linux on the desktop. To demo software that runs natively on Linux, I firmly believe that the only bottleneck should be the system itself: no virtualization, no networking, all self contained.

Recently, I’ve been banging my head with a new deployment of desktops. Whenever I tried to format the internal drive, it would come back and say that it was in use. I could partition it, but not format it. I have never seen this type of issue before.

Some PCs ship with RAID enabled on the motherboard. Most, if not all, PCs do not ship with hardware based RAID.

Hardware based RAID is a beautiful thing. Extra silicon is on the system that manages the configuration and operation of multiple disks, making them all appear as a device that is defined by the system administrator. From the perspective of Linux, it simply looks like disk drive, while it may be many, with redundancy. The computer’s CPU is not bothered with RAID operations, and is left to do other things.

Software based RAID is completely different. It is a RAID system, but still requires the host’s CPU to do work, via a driver, thereby gobbling up resources that may be required elsewhere.

Oddly, in this situation, the solution was in part knowing what the problem was. Most search engine queries resulted in similar problems, but none related to this issue.

[root@localhost ~]# fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          62      497983+  83  Linux
/dev/sda2              63        2495    19543072+  82  Linux swap / Solaris
/dev/sda3            2496      121601   956718945   8e  Linux LVM

Disk /dev/sdb: 20.0 GB, 20014718976 bytes
255 heads, 63 sectors/track, 2433 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Looks fine. But see when I try to make a filesystem on /dev/sda1:

[root@localhost ~]# mke2fs -j -m 0 -L BOOT /dev/sda1
mke2fs 1.39 (29-May-2006)
/dev/sda1 is apparently in use by the system; will not make a filesystem here!

I know the disk is not in use because the system is PXE booted via ethernet.

Looking at the partitions:

[root@localhost ~]# cat /proc/partitions
major minor  #blocks  name

   8     0  976762584 sda
   8     1     497983 sda1
   8     2   19543072 sda2
   8     3  956718945 sda3
   8    16   19545624 sdb
 253     0   19503104 dm-0
 253     1  976759808 dm-1
 253     2     102400 dm-2
 253     3   67108864 dm-3
 253     4    7247872 dm-4
 253     5     102400 dm-5

So I have some devices with major number 253 and minor numbers 0 through 5. Checking to see what else is assigned with these numbers in /dev:

[root@localhost ~]# find /dev | xargs ls -ald | grep 253
brw-rw----  1 root disk  253,    0 Jan 22 01:22 /dev/mapper/isw_cehgfbiihe_Cache
brw-rw----  1 root disk  253,    1 Jan 22 01:22 /dev/mapper/isw_dccbeigfai_CACHEVOL
brw-rw----  1 root disk  253,    2 Jan 22 01:22 /dev/mapper/isw_dccbeigfai_CACHEVOLp1
brw-rw----  1 root disk  253,    3 Jan 22 01:22 /dev/mapper/isw_dccbeigfai_CACHEVOLp2
brw-rw----  1 root disk  253,    4 Jan 22 01:22 /dev/mapper/isw_dccbeigfai_CACHEVOLp3
brw-rw----  1 root disk  253,    5 Jan 22 01:22 /dev/mapper/isw_dccbeigfai_CACHEVOLp4

So the device mapper is involved here. But why? Thankfully, this post shed some light on the situation:

So dmraid is involved. If this is the case, then there is some software RAID involved. Issuing the dmraid command with the -l option (from man page, list all available metadata format handlers with their names and descriptions):

[root@localhost ~]# dmraid -l
asr     : Adaptec HostRAID ASR (0,1,10)
ddf1    : SNIA DDF1 (0,1,4,5,linear)
hpt37x  : Highpoint HPT37X (S,0,1,10,01)
hpt45x  : Highpoint HPT45X (S,0,1,10)
isw     : Intel Software RAID (0,1,5,01)
jmicron : JMicron ATARAID (S,0,1)
lsi     : LSI Logic MegaRAID (0,1,10)
nvidia  : NVidia RAID (S,0,1,10,5)
pdc     : Promise FastTrack (S,0,1,10)
sil     : Silicon Image(tm) Medley(tm) (0,1,10)
via     : VIA Software RAID (S,0,1,10)
dos     : DOS partitions on SW RAIDs

Intel Software RAID appears to be the issue, confirmed by the driver listed in /dev/mapper. Though the storage options were configure as AHCI (not RAID or IDE) in the system’s BIOS, the metadata on the disks still throws Linux into a tizzy.

The solution (and it’s painfully easy):

[root@localhost ~]# dmraid -rE
Do you really want to erase "isw" ondisk metadata on /dev/sda ? [y/n] :y
Do you really want to erase "isw" ondisk metadata on /dev/sdb ? [y/n] :y
[root@localhost ~]# reboot

After the power cycle, the disks are available.

Converting Data to QR Code Video

One of the things I enjoy about computing is trying something silly, just to see what happens. Ever wonder what a live well of rainbow trout looks like as a QR code video? Well, look no further!


The tools involved are QREncode, ImageMagick, and MPlayer.

#!/bin/env bash

export PREFIX

cat $1 | base64 -w 0| split -a 4 -b 2048 - ${PREFIX}.

for file in ${PREFIX}.????
  echo Converting $file...
  cat $file | qrencode -s 4 -o - | convert - ${file}.jpg

mencoder                                \
  mf://${PREFIX}*.jpg -mf fps=30        \
  -ovc x264                             \
  -x264encopts "threads=auto:crf=30:me=umh:me_range=16:subq=2:trellis=0:global_header:level_idc=41:force_cfr"           \
  -o output.mp4

echo Cleaning up...
rm ${PREFIX}.???? ${PREFIX}.????.jpg

Clearly you are wondering why I chose to break up the source file into smaller files 2048 bytes in size? According to the QR Code Wikipedia page, the maximum binary data a QR code can hold is 2,953 bytes. The qrencode utility sqawked with some byte sizes above 2048, so picking a nice binary number seems to play well.

The resulting video file looks like:

The real question is can we fetch sane data from the video file? The answer is yes, and more tools, zbar, and FFMpeg.

The decoding script looks something like:


export PREFIX

ffmpeg -i $1 -r 30 ${PREFIX}-%06d.jpg

for x in ${PREFIX}-??????.jpg
  zbarimg -q --raw $x
done | base64 -d - > DATA

Put this under the “why computers are fun” file.

Google Fiber

There has been much hoopla regarding Google Fiber coming to Austin, TX. Yeah! One gigabit download and upload speeds! Not so fast…as with anything digital, the devil is in the details. From the Kansas City implementation, Google offers a choice: Gigabit Internet at $70/month or Gigabit Internet + TV at $120/month. Both of these plans waive the $300 installation fee. So far, so good. $120/month is on par with cable companies.

Focusing in Internet service alone, the first thing you’ll receive is a Google Fiber Network Box. It is billed as a gigabit router, high performance Wi-Fi, and a Gigabit Firewall. In the tech business, a “black box” system is one where you don’t know what is going on inside. Google’s offer here is literal, too:

So, what are the contracts required for service? Moving to their “Privacy and Terms” page for Google Fiber and selecting “Terms of Service”, there is:

You agree not to misuse the Services. […]. A list of examples of prohibited activities appears here.

So, let’s see what Google thinks is improper:

Your Google Fiber account is for your use and the reasonable use of your guests. Unless you have a written agreement with Google Fiber permitting you do so, you should not host any type of server using your Google Fiber connection, use your Google Fiber account to provide a large number of people with Internet access, or use your Google Fiber account to provide commercial services to third parties (including, but not limited to, selling Internet access to third parties).

Respect copyright. Upload and download only content that you are authorized to use or access.

Do not circumvent, disable, or otherwise modify any security features or other limitations Google Fiber places on any services it provides.

Comply with all applicable laws, rules, and regulations when utilizing Google Fiber’s services.

No servers?! That’s a real shame considering a subscriber has a 1 GbE up-link. And why is running a server lumped in with infringing on rights of others? There are countless server programs that respect rights and simply provide utility. If Google truly valued innovation, it should embrace allowing customers to host their own server programs.

These notices, along with the black box give Google tremendous power of their customers. We already know that Google was accused and admitted to snooping and storing unencrypted WiFi signals everywhere it could. What does the black box really do? It’s fair to say that it may analyze usage to ensure the terms of service are met. Yet…where do these analyses go, and for how long?

A real test of Google’s honesty in providing Internet access is if you can buy and manage your own router/firewall. Cable companies do this. You are welcome to buy your own cable-modem instead of leasing one from your provider. This proves that a cable-modem is nominal in its value to the cable companies. For Google Fiber, the black box is everything. Since it is acting as a firewall, it knows of every device you have in your home, with which parties they communicate, when, and for how long. Can it intercept and decrypt SSL traffic?

Can Google be trusted?

Don’t be evil?

Yeah, right.

CentOS 6 in Amazon Cloud

From the CentOS Announce mailing list, we now have public CentOS 6 images in Amazon!

CentOS-6 x86_64 with updates :
CentOS-6 i386 with updates :
CentOS-6.3 x86_64 without updates :
CentOS-6.3 i386 without updates :

Nice job CentOS Team!

Legality of Copying CDs

After posting about using rip to copy audio CDROMs to a computer, it got me thinking. Is it legal to copy audio CDROMs? After all, having rights granted to someone else’s work usually requires a contract. Why are audio CDROMs different? This is is what the RIAA says about copying CDROMs:

Copying CDs

  • It’s okay to copy music onto an analog cassette, but not for commercial purposes.
  • It’s also okay to copy music onto special Audio CD-R’s, mini-discs, and digital tapes (because royalties have been paid on them) – but, again, not for commercial purposes.
  • Beyond that, there’s no legal “right” to copy the copyrighted music on a CD onto a CD-R. However, burning a copy of CD onto a CD-R, or transferring a copy onto your computer hard drive or your portable music player, won’t usually raise concerns so long as:
    • The copy is made from an authorized original CD that you legitimately own
    • The copy is just for your personal use. It’s not a personal use – in fact, it’s illegal – to give away the copy or lend it to others for copying.
  • The owners of copyrighted music have the right to use protection technology to allow or prevent copying.
  • Remember, it’s never okay to sell or make commercial use of a copy that you make.

I like the phrase “won’t usually raise concerns”. It’s my guess a decent IP lawyer could successfully argue that all CDROM copying is illegal. If they are successful, how could the music business make money based on assumed legal actions? Well, Windows Media Player and iTunes know what you have imported from CD. Since copyright lasts so long, I can see a day where if you have X percent of your audio files imported from CDROM, you will get a notice to pay up. Or even better, you could have a surcharge on future purchases until you are deemed paid in full. Why not?

Pressed CDROMs will go away soon. This will benefit the music business in two ways – low cost of distribution (and therefore more profit), and license agreements. Licensing will end this assumed right of ripping audio, while placing controls of what you can and cannot do with music.

Cheapo Discs

What A Music Store Looks Like. Cheapo Discs. Austin TX. 2011.


Encoding CDROM Audio

Encoding CDs is dated, but has still has merit, falling into the preservation-of-formats category. The most recent encoded CD in my house was from August, 2009.

Taking a trip down memory lane, re-installing:

rip – A command-line based audio CD track ripper and MP3, FLAC, or Ogg Vorbis encoder

The installation is easy.  Download the latest (today, it is 1.07 – 2003 vintage).  The archive will extract into rip-1.07.

Opinions on how and where to install freeware vary, but for this post, I’ll say I put rip in /applications/rip-1.07/bin.  Rip also include two perl modules to make CDDB and MP3 tagging possible: CDDB_get-1.66.tar.gz and MP3-Info-0.91.tar.gz.  Extracting these two archives, and running the following in their respective directories sets things up properly:

perl Makefile.PL PREFIX=/applications/rip-1.07
make install

Once installed, if you set PERL5LIB to /applications/rip-1.07/share/perl5, and add /applications/rip-1.07/bin to your PATH, you are ready to run rip.

My preference is to name MP3 files as [TRACK]_[SONG NAME].mp3 while encoding at 256 Kbps. Therefore, issuing this command completed the deal:

rip -c -b 256 -f “%N_%S”

More rip help:

$ rip -help

USAGE: rip [option(s)] 


  -b   --bitrate NUM          set bitrate for encoding to NUM kbps
  -B   --bladeenc             use bladeenc for MP3 encoding
  -c   --cddb                 use CDDB to rename output files (must be online)
  -C   --comment              add your comments to each track via file tagging
  -d   --dev DEV              force input to come from device DEV
  -D   --debug                leave debug info in the file /tmp/rip-stderr
  -e   --eject                eject CD tray after doing everything else
  -f   --format FORMAT        use FORMAT when naming the output file

                                      %A   artist name
                                      %T   title of album
                                      %S   song name
                                      %N   track number

                              example:   rip -f "%A - %T - %N - %S"
                              example:   rip -f "%A_-_%S"

  -F   --flac                 use FLAC for lossless compression
  -g   --generate DIR/NAME    playlist NAME created in DIR (DIR may be omitted)
  -G   --gogo                 use GOGO for MP3 encoding
  -h   --help                 print this help to the screen
  -l   --lazy                 uses -t, -c, -g, -b/q, -m, and -e implicitly,
                              creates a playlist in $PWD in a common format,
                              moves output audio files to $PWD/ArtistName/
  -L   --lame                 use LAME for MP3 encoding
  -m   --move DIR             place all of the ripped/encoded tracks in DIR
  -M   --many                 rip many CDs one after the other (loop thru script)
  -n   --nounderscore         do NOT use an underscore '_' in filenames
  -N   --notlame              use notlame for MP3 encoding
  -O   --oggenc               use oggenc and encode to Ogg Vorbis files
  -p   --paranoia             use 160 kbps and do not accept skips on rip
  -P   --play                 play finished files in XMMS during rip
  -q   --quality NUM          use variable bitrate encoding with quality NUM
                              only works when LAME or oggenc is the encoder
                              see that encoder's documentation when choosing NUM
  -Q   --quiet                rip and encode tracks without any visible output
  -r   --rename               ask for proper name of all tracks before ripping
  -s   --speed NUM            force CD-ROM device to read at speed NUM
  -S   --superlazy            same functionality as -l/--lazy except that all
                              output is to $PWD/ArtistName/AlbumTitle/ instead
  -t   --trayclose            close CD tray before doing anything else
  -T   --tag                  tag the output file with artist/song/album info
  -v   --verbose              print lots of info about what script is doing
  -V   --version              print rip's version information
  -w   --wav                  rip CD tracks to WAV files and no further
  -Y   --cdparanoia           rip CDs using cdparanoia (this is the default)
  -Z   --cdda2wav             rip CDs using cdda2wav instead of cdparanoia


  rip 1 3-5 7 9-11 12 14      rip tracks 1 3 4 5 7 9 10 11 12 and 14 to MP3
  rip 6                       rip track 6 to MP3 using default encoder
  rip -c 6                    rip track 6 and rename its MP3 file via CDDB
  rip -c -d /dev/dvd 6        rip using /dev/dvd as the input device
  rip -c -m ~/mp3 6           simliar: moves output MP3 files to dir ~/mp3


  rip -P                      rip entire CD to MP3 and play tracks as we go
  rip -Pet                    similar: also closes and ejects CD tray
  rip -S                      laziness at its best IMHO (must be online)
  rip -S 1 3-5 7 12           superlazy rip specific tracks
  rip -TOPS                   similar: also tags, uses oggenc, and plays

  If you do not specify which track numbers to rip, then all tracks are ripped.
  You may now specify a subset of tracks to rip during a lazy rip

  The default output directory is the Present Working Directory ($PWD).
  Use -m/--move to alter/change the $PWD to a different directory, DIR.
  If the DIR you give -m/--move doesn't exists, rip will create it for you.

  If no specific encoder is flagged, rip uses the first encoder it finds.
  Encoders are searched in this order: gogo, lame, bladeenc, oggenc, flac, notlame.


  Copyright (C) 2003 by Gregory J. Smethells and licensed under the GPL

If you prefer loss-less audio files, then you should encode to FLAC, not MP3:

rip -c -F -f “%N_%S”

Cryptsetup and LUKS

Ever have a hard drive go bad, yet hesitate to RMA it? That’s understandable. Who knows what people on the other end will do, once it is repaired. Once simple solution is to encrypt the contents of the hard disk. This is made easy using cryptsetup.

For example, I just plugged in a 1TB external USB drive. Looking at my dmesg output, I see it is /dev/sde.

# dmesg
[710452.205068] sd 9:0:0:0: Attached scsi generic sg4 type 0
[710452.206899] sd 9:0:0:0: [sde] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[710452.208131] sd 9:0:0:0: [sde] Write Protect is off
[710452.208361] sd 9:0:0:0: [sde] Mode Sense: 00 38 00 00
[710452.208364] sd 9:0:0:0: [sde] Assuming drive cache: write through
[710452.210121] sd 9:0:0:0: [sde] Assuming drive cache: write through
[710452.226260]  sde:
[710452.228245] sd 9:0:0:0: [sde] Assuming drive cache: write through
[710452.228472] sd 9:0:0:0: [sde] Attached SCSI disk

To install encryption, I use the cryptsetup command:

# cryptsetup --verbose --verify-passphrase luksFormat /dev/sde
This will overwrite data on /dev/sde irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: 
Verify passphrase: 
Command successful.

Now, I need to install assign a device to the encrypted device:

# cryptsetup luksOpen /dev/sde BACKUP
Enter passphrase for /dev/sde:

Finally, I’ll make a filesystem on it and mount it:

# mke2fs -j -m 0 -L BACKUP /dev/mapper/BACKUP
mke2fs 1.41.12 (17-May-2010)
Filesystem label=BACKUP
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
61054976 inodes, 244190134 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
7453 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
	102400000, 214990848

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
# mount /dev/mapper/BACKUP /mnt
# df -k /mnt
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/BACKUP   961432608    204568 961228040   1% /mnt

Being able to unplug this USB drives means I need to issue the following:

# umount /mnt
# cryptsetup luksClose /dev/mapper/BACKUP

There are many nice features in cryptsetup. One is that you can have multiple passwords.
To add a pass phrase to my USB drive I first have to authenticate using an original pass phrase, then I can add a new one:

# cryptsetup luksAddKey /dev/sde 
Enter any passphrase: 
Enter new passphrase for key slot: 
Verify passphrase:

The Marginalization of Adobe Flash

If you remember, it was not that long ago that Steve Jobs categorically rejected Adobe Flash on the iDevice platform. Instead of Flash, Apple promoted H.264 for video. At the time, that seemed like a crazy idea. Adobe Flash has enjoyed great success, and it seemed short sighted to lock out Flash on any platform. Android phones, when they were first introduced, touted as being Flash enabled, while iDevices were not.

How a couple of years can make a difference. HTML 5 is now a part of most web browsers, and in that is the <video> tag. Web browsers now have native video playback capabilities. The clear winner is H.264, having support in Internet Explorer, Safari, and Chrome. The free and open standard WebM comes in second, supported by Firefox and Chrome.

I have used FlowPlayer in the past, but now it is obsolete. Instead of placing a Flash player on a web site, you can now simply use HTML to provide video. It is as simple as this:

<video width="480" height="270" controls="controls">
  <source src="" type="video/webm" />
  <source src="" type="video/mp4" />
  Your browser does not support the video tag.

If your browser desires to download the content instead of playing it, you may need to update a server directive. Consider this .htaccess file:

AddType video/webm .webm 

What is particularly powerful about the <video> tag is that you can specify multiple sources. If a browser does not support a source format, it tries the next.

UPDATE – 20120222 – From the category of not surprising:

For Flash Player releases after 11.2, the Flash Player browser plug-in for Linux will only be available via the “Pepper” API as part of the Google Chrome browser distribution and will no longer be available as a direct download from Adobe. Adobe will continue to provide security updates to non-Pepper distributions of Flash Player 11.2 on Linux for five years from its release.

Adobe Flash Roadmap (URL)

Roadmap PDF

MythTV, UPnP, and CentOS 6

MythTV is certainly one of the outstanding open source projects available. From

MythTV is a Free Open Source software digital video recorder (DVR) project distributed under the terms of the GNU GPL. It has been under heavy development since 2002, and now contains most features one would expect from a good DVR (and many new ones that you soon won’t be able to live without).

It is a little tricky to get going, but once you do, you will have both a fantastic and heartbreaking experience with Linux and multimedia. In this space, you will learn why free and open source software is so important, along with what barriers have been put in place to inhibit it.

One exciting aspect about MythTV is its UPnP functionality. I have a Bluray player that understands this protocol, so out of the box, it had instant access to my 0.21 version of MythTV content. This is why open standards are nice!

CentOS 6 ships with QT 4.6.2, but without QtWebKit. A quick search reveals that you can install it via the Extra Packages for Enterprise Linux (or EPEL) repo from Fedora. I tend to shy away from installing random packages from random people, so the first step will be to compile a different version of QT. I started with 4.7.4 from Nokia’s FTP server. The options are pretty simple for the configure script:

sh ./configure                                                          

gmake clean
gmake install

Your choice of ${PREFIX} is up to you. Now QT 4.7.4 is installed in ${PREFIX}, with the libraries in ${PREFIX}/lib64. After extracting the MythTV source file, I made the following addition to the file:

> EXTRALIBS += -L${PREFIX}/lib64

I fed the following options to the configure script:

export PATH

sh ./configure                                                          

Notice I have enabled Ogg Vorbis, Theora, and GSM. These libraries are required if you want to be able to play those media types. The libraries are stuffed in the -L${PREFIX}/lib64 area mentioned above.

After extracting the plugins source file, I made the following addition to the file:

< EXTRALIBS += -L${PREFIX}/lib64
< EXTRA_LIBS += -lfreetype -lmp3lame $$EXTRALIBS
> EXTRA_LIBS += -lfreetype -lmp3lame

To build and install the plugins, I ran the above configure script without the –enable lines. After the build is complete, all I have to do is add ${PREFIX}/bin to my PATH, along with ${PREFIX}/lib64 to my LD_LIBRARY_PATH. I normally set the runtime library search path to point to ${PREFIX}/lib64 so LD_LIBRARY_PATH is not needed, but somewhere the LDFLAGS variable gets lost in the qmake build.

After upgrading my 0.21 MythTV MySQL database to the 0.24 MythTV Schema (and thus starting the backend), the mythfrontend can now invoke:

MythTV 0.24.1 Initial Screen

MythTV is one of the many successes of Open Source software!