One question that often arises is, look at this e-mail I received…have I been hacked? The answer is not quite as straightforward as one might think. The protocol that defines Internet e-mail was written a long, long, time ago, and the Internet was a much different place than what it is today. The protocol itself, by today’s standards, is pretty weak allowing for much mayhem. It is pretty easy to send e-mail as someone else, and the only resolution is to closely inspect the mail delivery headers – a record of which machines have handled the message.

The most common type of attack is a phishing attack, where a recipient is lured into opening and perhaps clicking on a link in the body of the message.
These messages can contain various means of corrupting the e-mail program itself, allowing for code to be installed on the recipient’s computer (obviously the worst case scenario), or simple theft of data. Most purveyors of software know this game now and are sensitive to it, but as long as programs are big and complex, the opportunities will still exist, with new ones presenting themselves with every update. Now, with plenty of free e-mail services, cloud storage, and mobile devices, new attack vectors have risen, with bigger payoffs, making it nearly impossible to thwart.

There are essentially two camps of spammers: one that simply sends bulk e-mail in the hopes that a very small percentage actually buy products the spam is selling, and the other, that hopes actually control the system of the target. Both rely on massive lists of real e-mail addresses in order to function. The simplest way to gather e-mail addresses is to exploit known weaknesses of the e-mail client and/or the operating system on which it runs, often resulting in the export of the target’s address book. Once this knowledge is gained, it is possible to send e-mail as each person in the address book to all people in the address book, resulting in more successful hacks and address book losses. Put another way, if someone else has you as a contact and they don’t give a flip about good computing practices, password strength, or encryption, then most likely there will be spam sent as you to your colleagues. This is why a friend may say you have been hacked after receiving a message from you, when really it was a friend of a friend of a friend. The hack may have happened long ago, it may have been recent. It’s also quite possible that all of these stolen address books are aggregated, analyzed, and traded in the black market.

But why are there so many spams? The answer lies in the latter group, and it involves money, extradition laws, and botnets. If an attacker (spammer no longer applies here) can insert code on a target’s computer and then control the computer, they can build a much more efficient, disperse, spamming operation. When this happens on a large scale, it is called a botnet. The opportunities for illegal monetary gain increase greatly with botnets, one being extortion through a threat of a Distributed Denial of Service (DDOS) attack. Think of a DDOS attack as tens of thousands of computers flooding a targets network with useless requests, causing all of the infrastructure to overload and miss legitimate business activity. Perpetrators often live in countries that have no extradition treaties with their victims, essentially giving them a safe haven.

Businesses that generate online revenue are typically the victim. Since there is no real good way to fight large scale (botnet controlled) DDOS attacks, they often pay quietly, leaving no record of the extortion. Occasionally one will, and it makes headlines. Do a search of “ddos extortion attacks” and a clear picture starts to form.

So what can be done? Well, you’re in the same boat now with many large companies, and they have much deeper pockets. It’s probably best to analyze e-mail headers to look at the point of origin of the spam. Make sure that the spams do not originate from your IP address, or any IP address you use to send email. Also, look at the payload. Are there images or hyperlinks in the spam? If so, then most likely it is an attempt on your security, using your contacts as additional leverage.

All of this, stemming from a simple mail transfer protocol (SMTP) written at the dawn of the Internet. That’s right – SMTP is the actual name of the protocol. If only it were not so simple.

Requiem for Minecraft

Below is a post I’ve had in my Drafts for a while now. Sadly, Microsoft is buying Mojang without the creative minds behind it.

As any parent knows, MineCraft is white hot. What makes this game popular? The game is constructed in such a way to avoid violence as a principal motivator. A player can farm, mine, build, or fish. Games within games can be created. This opens the door to both genders resulting in exciting interactive play with friends. Whether the game developers realize it or not, MineCraft draws upon the experience of the early days in computing, only using modern computing power.

What technology drives MineCraft? Java. The promises of Java yesterday have come to fruition today. Platform support includes Windows, Mac OS X, and drum roll….Linux! Write once, run anywhere. As a person whose desktop is Linux only, it is nice this program work so well, including 3D OpenGL graphics. The author of MineCraft, Mojang, offers a free to use server version.

Below is a Planet Money segment on MineCraft.

The above segment talks about the business model of MineCraft, and folks, it’s simple. Buy it, and play it forever, without advertising, marketing, or in-app purchases. How quaint! It just shows how we have lost our way when driven to maximize profits. The world of mobile is a very dirty, creepy place when it comes to big data.

Mojang did none of this and created a game for all people of all ages and both genders, open ended…let the brain explore just for the sake of it, and look what happened. Profits followed for Mojang, with the lead developer owning a majority of Mojang, and therefore getting a majority of a 2.5 billion dollar buy out. It should be a lesson to the people hell bent on squeezing every penny out of our children, but it won’t. The big data way is the easy way. It’s easy to sell investors when you have all sorts of data surreptitiously taken from your customers that can be shaped, repacked, and sold to anyone willing to buy. It’s disgusting.

Job well done, Mojang. Thank you for the fun. After reading why you sold it, it is completely understandable.

What can we expect for MineCraft now that Microsoft owns it? If history is any guide, then a game written in a language that Microsoft did not invent running on operating systems that Microsoft does not sell…well, you get the idea. Microsoft is a big data company, too. Now, how can they increase market share and profits for their investors with MineCraft? The ways are innumerable, and also counter to what MineCraft was. Yes, it’s the past tense now, because it will never be the same. The first comment on the Planet Money link sums it up the best:

Game Over


Markus Persson, Mojang
New York Times
NPR’s Planet Money
NPR’s Planet Money Transcript

Follow the Baytrail

The first Intel Baytrail (low power) PCs are out, and mighty interesting. Shuttle has produced a system administrator’s PC that can pretty much do anything, running with only a 65 watt power supply. The specs are impressive. While it ships with no memory or hard drive (a good thing), it can support up to 16GB of RAM. Folks, 16GB of RAM for a 65 watt system is wonderful and insane at the same time. It also has three network interfaces, one of them WiFi B/G/N. HDMI, USB3, SD, DVI – all for $200 – oh my!

The Facade Crumbles

It has been over a year since the revelations of mass surveillance were exposed. As expected, the people of the United States have gone back to work, placing issues of privacy deep into their subconscious. When a new revelation is reported, we think, oh yeah, that security thing again. We bury the thought that all of our decisions are logged and indexed into a database for someone to sell, analyze, or subpoena. We are told, over and over, that if we’re not doing anything wrong, we have nothing to worry about. On the surface, this is true. It is what I would tell my elementary school age children. But my children will grow, and I have to wonder what world we will leave them.

TN-Noonan-20130816-Privacy Peggy Noonan wrote some wonderful words on the matter back in August of 2013 (unfortunately now behind the WSJ paywall). She writes, “Privacy is connected to personhood. It has to do with intimate things—the innards of your head andcheart, the workings of your mind—and the boundary between those things and the world outside. A loss of the expectation of privacy in communications is a loss of something personal and intimate, and it will have broader implications.” She continues, making the suggestion that when the Fourth Amendment’s balance of power shifts away from the individual to the State, the First Amendment becomes far less important. We self-censor. We fear.

TN-PJ_Forced_Exposure.jpgUnfortunately, this is now all too true. During the summer of 2013, Pamela Jones of Groklaw, shuttered her blog. Her work from 2003 to 2013 was invaluable to the Linux community, providing comfort in the wake of a very scary copyright infringement lawsuit brought by SCO against IBM that threatened the very existence of Linux. As a paralegal, she carefully documented all the details of the case, providing daily insight to the technical community, making the legal world bearable. She was there to say, do not panic, the assertions are only assertions, there will be an end to it all. Pamela Jones, not her real name, was an anonymous blogger. Her work existed in a mine field. What we learned from the SCO case is anyone can bring suit, and it is costly to defend. Had she ever been outed, unmerited defamation and slander suits would surely follow. So, farewell Pamela Jones. We miss your thoughts and your right to free speech.

TN-PJ_Forced_Exposure.jpg What makes the shuttering of Groklaw a terrific story is its relationship to another shuttered site, Lavabit. Lavabit provided secure e-mail services. Edward Snowden was alleged to be a customer. The founder and CEO, Ladar Levison, received a national security letter, not just asking for one SSL private key, but all keys. Lavabit’s foundation is based on trust. Once broken, there is nothing left. Realizing this, Ladar Levison literally pulled the plug. He took everything down, and eventually turned over encryption keys (albeit in a 4pt font on printed paper). Ironically, the NSA may have already had the keys to Lavabit’s kingdom, if Lavabit was using rev 1.0.x of OpenSSL. During this time, OpenSSL was (unknowingly to some) suffering from heartbleed, allowing for remote decryption. Farewell Ladar Levison.

It does seem bleak. While Edward Snowden was talking about government surveillance at SXSW 2014, tech companies were introducing all of their wearable smart devices. There is no better example of this contradiction. We want privacy, but are perfectly willing to ignore license agreements and contracts in order to run our gadgets. With the Internet of Things moving into high gear, options for surveillance will blossom horrifically. Keep in mind, too that surveillance is not limited to communications. If you use plastic to purchase anything, data brokers purchase the right to analyze your spending patterns down to the item level, and sell that information. This scenario is by far the most frightening, because it zeroes in on the core of economics. The Summer of 2013 may be remembered as the good old days.

Is there anything that can be done? Absolutely, but it is up to the individual to care, to cherish the freedoms not yet lost. Now that chess table is set, we can see what it takes to play. This game will be difficult, but it is a game that can be played. To start, every person who cares about free thought and speech should take a position that this inevitability is not certain, and that there are countermeasures, that good people do win. Technology meets the needs of those who desire it, and it also is cyclical.

Peggy Noonan: What We Lose if We Give Up Privacy – August 16th, 2013
Pamela Jones Farewell Notice – August 20th, 2013
Lavabit Farewell Notice – August 8th, 2013
Ladar Levison Battle with FBI (NPR) – October 3rd, 2013
On the Media, New Security Standard for Journalists – August 16th, 2013

Goodbye Freecode

Sadly, on June 18th, 2014, (fromerly became static. This web site has been one of the most useful for system administrators. It aggregated software projects and provided a list of subsequent updates. Think of it a searchable RSS feed for all free and open source software on the Internet.

I consider it a site that helped system administrators today (and yesterday) craft some of our modern platforms that we enjoy. Thank you for all of these years!!!

The Magic of Rsync

Without question, rsync is one of a handful tools required for any systems administrator to keep their heads above water. It hooks in wonderfully with other tools to allow synchronization of files across a wide variety of media.

COW, unionfs, aufs – they all rely on a read-only source, with any changes getting written to a separate area. From the perspective if the user, both areas appear to be a single source, all writable. For an operating system, a builder can create a master copy, allowing different types of hardware adjust where necessary: one copy, infinite combinations.

While rsync does not provide real-time access as described above, it does allow for a system administrator to look at two given trees and see what has changed over time.

For a basic example, look at the following:

example $ mkdir foo
example $ touch foo/{1,2,3,4,5}
example $ ls foo/
1  2  3  4  5

There are five files in $PWD/foo, 1, 2, 3, 4, 5. Now, using rsync to back them up:

example $ rsync -av foo/ backup/
sending incremental file list

sent 273 bytes  received 110 bytes  766.00 bytes/sec
total size is 0  speedup is 0.00
example $ ls backup/
1  2  3  4  5

Here, a perfect copy of the files 1, 2, 3, 4, and 5 are made from the source (foo) to the destination (backup).

For the next example, $PWD/foo gets new data, and only the delta should be placed in a new directory:

example $ touch foo/{a,b,c,d,e}
example $ ls foo
1  2  3  4  5  a  b  c	d  e
example $ rsync -av --compare-dest=$PWD/backup foo/ delta/
sending incremental file list

sent 323 bytes  received 110 bytes  866.00 bytes/sec
total size is 0  speedup is 0.00
example $ ls foo
1  2  3  4  5  a  b  c	d  e
example $ ls backup
1  2  3  4  5
example $ ls delta
a  b  c  d  e

Nice! If I overlay delta/ on top of backup/ then it will equal foo/. This is a pretty basic example, but it is apparent that if this is combined with cron, some pretty fancy things can happen.

For example, if you needed to know any file changes (and what they were) from any point in the past, this process would allow it.

Software RAID and Linux

Part of my day to day activities is to support RedHat Enterprise Linux on the desktop. To demo software that runs natively on Linux, I firmly believe that the only bottleneck should be the system itself: no virtualization, no networking, all self contained.

Recently, I’ve been banging my head with a new deployment of desktops. Whenever I tried to format the internal drive, it would come back and say that it was in use. I could partition it, but not format it. I have never seen this type of issue before.

Some PCs ship with RAID enabled on the motherboard. Most, if not all, PCs do not ship with hardware based RAID.

Hardware based RAID is a beautiful thing. Extra silicon is on the system that manages the configuration and operation of multiple disks, making them all appear as a device that is defined by the system administrator. From the perspective of Linux, it simply looks like disk drive, while it may be many, with redundancy. The computer’s CPU is not bothered with RAID operations, and is left to do other things.

Software based RAID is completely different. It is a RAID system, but still requires the host’s CPU to do work, via a driver, thereby gobbling up resources that may be required elsewhere.

Oddly, in this situation, the solution was in part knowing what the problem was. Most search engine queries resulted in similar problems, but none related to this issue.

[root@localhost ~]# fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          62      497983+  83  Linux
/dev/sda2              63        2495    19543072+  82  Linux swap / Solaris
/dev/sda3            2496      121601   956718945   8e  Linux LVM

Disk /dev/sdb: 20.0 GB, 20014718976 bytes
255 heads, 63 sectors/track, 2433 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Looks fine. But see when I try to make a filesystem on /dev/sda1:

[root@localhost ~]# mke2fs -j -m 0 -L BOOT /dev/sda1
mke2fs 1.39 (29-May-2006)
/dev/sda1 is apparently in use by the system; will not make a filesystem here!

I know the disk is not in use because the system is PXE booted via ethernet.

Looking at the partitions:

[root@localhost ~]# cat /proc/partitions
major minor  #blocks  name

   8     0  976762584 sda
   8     1     497983 sda1
   8     2   19543072 sda2
   8     3  956718945 sda3
   8    16   19545624 sdb
 253     0   19503104 dm-0
 253     1  976759808 dm-1
 253     2     102400 dm-2
 253     3   67108864 dm-3
 253     4    7247872 dm-4
 253     5     102400 dm-5

So I have some devices with major number 253 and minor numbers 0 through 5. Checking to see what else is assigned with these numbers in /dev:

[root@localhost ~]# find /dev | xargs ls -ald | grep 253
brw-rw----  1 root disk  253,    0 Jan 22 01:22 /dev/mapper/isw_cehgfbiihe_Cache
brw-rw----  1 root disk  253,    1 Jan 22 01:22 /dev/mapper/isw_dccbeigfai_CACHEVOL
brw-rw----  1 root disk  253,    2 Jan 22 01:22 /dev/mapper/isw_dccbeigfai_CACHEVOLp1
brw-rw----  1 root disk  253,    3 Jan 22 01:22 /dev/mapper/isw_dccbeigfai_CACHEVOLp2
brw-rw----  1 root disk  253,    4 Jan 22 01:22 /dev/mapper/isw_dccbeigfai_CACHEVOLp3
brw-rw----  1 root disk  253,    5 Jan 22 01:22 /dev/mapper/isw_dccbeigfai_CACHEVOLp4

So the device mapper is involved here. But why? Thankfully, this post shed some light on the situation:

So dmraid is involved. If this is the case, then there is some software RAID involved. Issuing the dmraid command with the -l option (from man page, list all available metadata format handlers with their names and descriptions):

[root@localhost ~]# dmraid -l
asr     : Adaptec HostRAID ASR (0,1,10)
ddf1    : SNIA DDF1 (0,1,4,5,linear)
hpt37x  : Highpoint HPT37X (S,0,1,10,01)
hpt45x  : Highpoint HPT45X (S,0,1,10)
isw     : Intel Software RAID (0,1,5,01)
jmicron : JMicron ATARAID (S,0,1)
lsi     : LSI Logic MegaRAID (0,1,10)
nvidia  : NVidia RAID (S,0,1,10,5)
pdc     : Promise FastTrack (S,0,1,10)
sil     : Silicon Image(tm) Medley(tm) (0,1,10)
via     : VIA Software RAID (S,0,1,10)
dos     : DOS partitions on SW RAIDs

Intel Software RAID appears to be the issue, confirmed by the driver listed in /dev/mapper. Though the storage options were configure as AHCI (not RAID or IDE) in the system’s BIOS, the metadata on the disks still throws Linux into a tizzy.

The solution (and it’s painfully easy):

[root@localhost ~]# dmraid -rE
Do you really want to erase "isw" ondisk metadata on /dev/sda ? [y/n] :y
Do you really want to erase "isw" ondisk metadata on /dev/sdb ? [y/n] :y
[root@localhost ~]# reboot

After the power cycle, the disks are available.

Converting Data to QR Code Video

One of the things I enjoy about computing is trying something silly, just to see what happens. Ever wonder what a live well of rainbow trout looks like as a QR code video? Well, look no further!


The tools involved are QREncode, ImageMagick, and MPlayer.

#!/bin/env bash

export PREFIX

cat $1 | base64 -w 0| split -a 4 -b 2048 - ${PREFIX}.

for file in ${PREFIX}.????
  echo Converting $file...
  cat $file | qrencode -s 4 -o - | convert - ${file}.jpg

mencoder                                \
  mf://${PREFIX}*.jpg -mf fps=30        \
  -ovc x264                             \
  -x264encopts "threads=auto:crf=30:me=umh:me_range=16:subq=2:trellis=0:global_header:level_idc=41:force_cfr"           \
  -o output.mp4

echo Cleaning up...
rm ${PREFIX}.???? ${PREFIX}.????.jpg

Clearly you are wondering why I chose to break up the source file into smaller files 2048 bytes in size? According to the QR Code Wikipedia page, the maximum binary data a QR code can hold is 2,953 bytes. The qrencode utility sqawked with some byte sizes above 2048, so picking a nice binary number seems to play well.

The resulting video file looks like:

The real question is can we fetch sane data from the video file? The answer is yes, and more tools, zbar, and FFMpeg.

The decoding script looks something like:


export PREFIX

ffmpeg -i $1 -r 30 ${PREFIX}-%06d.jpg

for x in ${PREFIX}-??????.jpg
  zbarimg -q --raw $x
done | base64 -d - > DATA

Put this under the “why computers are fun” file.

Google Fiber

There has been much hoopla regarding Google Fiber coming to Austin, TX. Yeah! One gigabit download and upload speeds! Not so fast…as with anything digital, the devil is in the details. From the Kansas City implementation, Google offers a choice: Gigabit Internet at $70/month or Gigabit Internet + TV at $120/month. Both of these plans waive the $300 installation fee. So far, so good. $120/month is on par with cable companies.

Focusing in Internet service alone, the first thing you’ll receive is a Google Fiber Network Box. It is billed as a gigabit router, high performance Wi-Fi, and a Gigabit Firewall. In the tech business, a “black box” system is one where you don’t know what is going on inside. Google’s offer here is literal, too:

So, what are the contracts required for service? Moving to their “Privacy and Terms” page for Google Fiber and selecting “Terms of Service”, there is:

You agree not to misuse the Services. […]. A list of examples of prohibited activities appears here.

So, let’s see what Google thinks is improper:

Your Google Fiber account is for your use and the reasonable use of your guests. Unless you have a written agreement with Google Fiber permitting you do so, you should not host any type of server using your Google Fiber connection, use your Google Fiber account to provide a large number of people with Internet access, or use your Google Fiber account to provide commercial services to third parties (including, but not limited to, selling Internet access to third parties).

Respect copyright. Upload and download only content that you are authorized to use or access.

Do not circumvent, disable, or otherwise modify any security features or other limitations Google Fiber places on any services it provides.

Comply with all applicable laws, rules, and regulations when utilizing Google Fiber’s services.

No servers?! That’s a real shame considering a subscriber has a 1 GbE up-link. And why is running a server lumped in with infringing on rights of others? There are countless server programs that respect rights and simply provide utility. If Google truly valued innovation, it should embrace allowing customers to host their own server programs.

These notices, along with the black box give Google tremendous power of their customers. We already know that Google was accused and admitted to snooping and storing unencrypted WiFi signals everywhere it could. What does the black box really do? It’s fair to say that it may analyze usage to ensure the terms of service are met. Yet…where do these analyses go, and for how long?

A real test of Google’s honesty in providing Internet access is if you can buy and manage your own router/firewall. Cable companies do this. You are welcome to buy your own cable-modem instead of leasing one from your provider. This proves that a cable-modem is nominal in its value to the cable companies. For Google Fiber, the black box is everything. Since it is acting as a firewall, it knows of every device you have in your home, with which parties they communicate, when, and for how long. Can it intercept and decrypt SSL traffic?

Can Google be trusted?

Don’t be evil?

Yeah, right.

CentOS 6 in Amazon Cloud

From the CentOS Announce mailing list, we now have public CentOS 6 images in Amazon!

CentOS-6 x86_64 with updates :
CentOS-6 i386 with updates :
CentOS-6.3 x86_64 without updates :
CentOS-6.3 i386 without updates :

Nice job CentOS Team!