Time Capsule As a File Server

Posted on by October 25, 2015

With so many Apple devices now interacting with my family, I made the decision to install a Time Capsule as the principal wifi presence. I have always believed in heterogeneous networks. Yet, other vendors of wifi routers have resulted in performance variability for the Apple products. So, now the onus is on Apple. Guess what? Apple clients work much better with the Time Capsule. Really I should not be surprised, but it does make me wonder what goes on behind the scenes that makes the difference. Hmmmm.

The Time Capsule (TC) has a USB 3 port on it, so we now have a 2TB external drive connected to it. This disk, apart from the internal drive on Time Capsule, is used for central storage. How can one access it via Linux? The TC shares this disk via CIFS, so Windows and Linux computers can utilize the export as well. To get a list of what is available on the TC:

# smbclient -U "USER NAME" -L IP ADDRESS|hostname
Enter USER NAME's password: 
Domain=[WORKGROUP] OS=[Apple Base Station] Server=[CIFS 4.32]

	Sharename       Type      Comment
	---------       ----      -------
	USER NAME       Disk      
	IPC$            IPC       
	Data            Disk      
	USB Drive       Disk      
Domain=[WORKGROUP] OS=[Apple Base Station] Server=[CIFS 4.32]

	Server               Comment
	---------            -------

	Workgroup            Master
	---------            -------

And to mount it, the option that provides the secret sauce is the option “sec=ntlm”:

# mount -t cifs -o user="USER NAME",sec=ntlm --verbose //[IP ADDRESS|hostname]/"USB Drive" /path/to/mount

Now that the mount command can be issued manually, it can then be integrated with the Linux automounter. Now that Linux has access to the drive, the doors are open.

Cryptsetup

Posted on by December 31, 2014

By now, most companies are employing disk encryption. Linux has for a long time supported disk encryption, and using it is imperative for mobile devices in case of loss or theft. Upon occasion, replacing the decryption key is a necessity, and thankfully, LUKS (Linux Unified Key Setup) allows this.

To create a test filesystem with encryption:

# cd /tmp
# dd if=/dev/zero of=test.sparse bs=1 count=0 seek=100M
0+0 records in
0+0 records out
0 bytes (0 B) copied, 2.5627e-05 s, 0.0 kB/s

# ls -als test.sparse
0 -rw-r--r-- 1 root root 104857600 Dec 31 04:14 test.sparse

# losetup /dev/loop0 /tmp/test.sparse
# losetup -a
/dev/loop0: [fe00]:122882 (/tmp/test.sparse)

# cryptsetup luksFormat /dev/loop0

WARNING!
========
This will overwrite data on /dev/loop4 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: FOO
Verify passphrase: FOO

# cryptsetup luksOpen /dev/loop0 TEST-ENCRYPTED
Enter passphrase for /dev/loop0:

# mke2fs -j /dev/mapper/TEST-ENCRYPTED 
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
25168 inodes, 100352 blocks
5017 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
13 block groups
8192 blocks per group, 8192 fragments per group
1936 inodes per group
Superblock backups stored on blocks: 
	8193, 24577, 40961, 57345, 73729

Writing inode tables: done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 31 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

# mount /dev/mapper/TEST-ENCRYPTED /tmp/mount
# df -k /tmp/mount
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/TEST-ENCRYPTED
                         97167      5663     86487   7% /tmp/mount

At this point we have an encrypted filesystem, 100 MB in size, mountable via the loopback filesystem (loopfs). Now, to examine the details of of test.sparse (remembering it could be any storage partition, logical volume, raid set, etc).

# cryptsetup luksDump /dev/loop0
LUKS header information for /dev/loop0

Version:       	1
Cipher name:   	aes
Cipher mode:   	cbc-essiv:sha256
Hash spec:     	sha1
Payload offset:	4096
MK bits:       	256
MK digest:     	88 e2 08 fb 6c 1f b3 cf 31 36 d6 b8 33 e5 26 e0 9e 00 87 3b 
MK salt:       	e9 9c 9b 35 f8 9f 72 f4 db 4d d7 aa 6d 6e 7e a3 
               	85 38 41 65 b5 35 0a 88 08 c9 66 ee ad ba 77 30 
MK iterations: 	66500
UUID:          	0b7452c4-1c2f-43d4-8768-fcf168d990a4

Key Slot 0: ENABLED
	Iterations:         	266167
	Salt:               	66 dc d0 4c dd 22 a4 48 b7 9b 2e bd b2 6d af 9d 
	                      	c5 5c 43 f8 25 f3 d4 86 36 8d 78 28 75 7a 52 a5 
	Key material offset:	8
	AF stripes:            	4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

There are a lot of goodies here. One thing to note are the key slots – eight total. Isn’t that nice? Eight decryption keys can support one device. This means you can place an additional key in case one is lost, a back door. The next step is to add a key to a slot, and for this example, it is slot 7:

# cryptsetup --key-slot=7 luksAddKey /dev/loop0
Enter any passphrase: FOO
Enter new passphrase for key slot: BAR 
Verify passphrase: BAR

# cryptsetup luksDump /dev/loop0
LUKS header information for /dev/loop0

Version:       	1
Cipher name:   	aes
Cipher mode:   	cbc-essiv:sha256
Hash spec:     	sha1
Payload offset:	4096
MK bits:       	256
MK digest:     	88 e2 08 fb 6c 1f b3 cf 31 36 d6 b8 33 e5 26 e0 9e 00 87 3b 
MK salt:       	e9 9c 9b 35 f8 9f 72 f4 db 4d d7 aa 6d 6e 7e a3 
               	85 38 41 65 b5 35 0a 88 08 c9 66 ee ad ba 77 30 
MK iterations: 	66500
UUID:          	0b7452c4-1c2f-43d4-8768-fcf168d990a4

Key Slot 0: ENABLED
	Iterations:         	266167
	Salt:               	66 dc d0 4c dd 22 a4 48 b7 9b 2e bd b2 6d af 9d 
	                      	c5 5c 43 f8 25 f3 d4 86 36 8d 78 28 75 7a 52 a5 
	Key material offset:	8
	AF stripes:            	4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: ENABLED
	Iterations:         	255589
	Salt:               	5e 49 e4 72 6f 27 7a 76 64 b9 df ae 4e 92 bc 1e 
	                      	d9 0e 55 87 61 ba e6 13 af d8 a5 4d 5f 6e 02 14 
	Key material offset:	1800
	AF stripes:            	4000

Now we have two keys for decryption, one in slot zero, the other in slot seven. To confirm the filesystem can be mounted with both passwords:

# umount /tmp/mount
# cryptsetup luksClose /dev/mapper/TEST-ENCRYPTED 
# cryptsetup luksOpen /dev/loop0 TEST-ENCRYPTED
Enter passphrase for /dev/loop0: FOO
# mount /dev/mapper/TEST-ENCRYPTED /tmp/mount
# df -k /tmp/mount
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/TEST-ENCRYPTED
                         97167      5663     86487   7% /tmp/mount

# umount /tmp/mount
# cryptsetup luksClose /dev/mapper/TEST-ENCRYPTED 
# cryptsetup luksOpen /dev/loop4 TEST-ENCRYPTED
Enter passphrase for /dev/loop4: BAR
# mount /dev/mapper/TEST-ENCRYPTED /tmp/mount
# df -k /tmp/mount
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/TEST-ENCRYPTED
                         97167      5663     86487   7% /tmp/mount

Everything works as expected. We now have two keys that can mount the encrypted loopback filesystem. The next step is to remove slot 0, and verify that its key no longer works.

# umount /tmp/mount
# cryptsetup luksClose /dev/mapper/TEST-ENCRYPTED 
# cryptsetup luksKillSlot /dev/loop0 0
Enter any remaining LUKS passphrase: BAR

# cryptsetup luksDump /dev/loop0
LUKS header information for /dev/loop0

Version:       	1
Cipher name:   	aes
Cipher mode:   	cbc-essiv:sha256
Hash spec:     	sha1
Payload offset:	4096
MK bits:       	256
MK digest:     	88 e2 08 fb 6c 1f b3 cf 31 36 d6 b8 33 e5 26 e0 9e 00 87 3b 
MK salt:       	e9 9c 9b 35 f8 9f 72 f4 db 4d d7 aa 6d 6e 7e a3 
               	85 38 41 65 b5 35 0a 88 08 c9 66 ee ad ba 77 30 
MK iterations: 	66500
UUID:          	0b7452c4-1c2f-43d4-8768-fcf168d990a4

Key Slot 0: DISABLED
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: ENABLED
	Iterations:         	255589
	Salt:               	5e 49 e4 72 6f 27 7a 76 64 b9 df ae 4e 92 bc 1e 
	                      	d9 0e 55 87 61 ba e6 13 af d8 a5 4d 5f 6e 02 14 
	Key material offset:	1800
	AF stripes:            	4000

Slot zero is now removed, and trying to use FOO confirms it:

cryptsetup luksOpen /dev/loop0 TEST-ENCRYPTED
Enter passphrase for /dev/loop0: FOO
No key available with this passphrase.
Enter passphrase for /dev/loop0: FOO
No key available with this passphrase.
Enter passphrase for /dev/loop0: FOO
No key available with this passphrase.

To tidy things up, the last steps are to add new key to slot zero, and remove slot seven.

# cryptsetup --key-slot=0 luksAddKey /dev/loop0 
Enter any passphrase: BAR
Enter new passphrase for key slot: NEW 
Verify passphrase: NEW 
# cryptsetup luksKillSlot /dev/loop0 7
Enter any remaining LUKS passphrase: NEW

# cryptsetup luksOpen /dev/loop0 TEST-ENCRYPTED
Enter passphrase for /dev/loop0: 
# mount /dev/mapper/TEST-ENCRYPTED /tmp/mount
# df -k /tmp/mount
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/TEST-ENCRYPTED
                         97167      5663     86487   7% /tmp/mount

# cryptsetup luksDump /dev/loop0
LUKS header information for /dev/loop0

Version:       	1
Cipher name:   	aes
Cipher mode:   	cbc-essiv:sha256
Hash spec:     	sha1
Payload offset:	4096
MK bits:       	256
MK digest:     	88 e2 08 fb 6c 1f b3 cf 31 36 d6 b8 33 e5 26 e0 9e 00 87 3b 
MK salt:       	e9 9c 9b 35 f8 9f 72 f4 db 4d d7 aa 6d 6e 7e a3 
               	85 38 41 65 b5 35 0a 88 08 c9 66 ee ad ba 77 30 
MK iterations: 	66500
UUID:          	0b7452c4-1c2f-43d4-8768-fcf168d990a4

Key Slot 0: ENABLED
	Iterations:         	251733
	Salt:               	4d a1 48 b7 26 15 a3 1c 53 e2 14 a4 75 7d f9 02 
	                      	8b 0f 2c 3d e3 1f 34 05 fa 21 92 15 ea d0 1b a8 
	Key material offset:	8
	AF stripes:            	4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

This completes the key change. Looking at the above dump, one interesting item is the cipher. The default cipher can be changed at compile time for cryptsetup. From the man page (1.2.0 was used in this example), a few key points:

NOTES ON SUPPORTED CIPHERS, MODES, HASHES AND KEY SIZES

The available combinations of ciphers, modes, hashes and key sizes
depend on kernel support. See /proc/crypto for a list of available
options. You might need to load additional kernel crypto modules in
order to get more options.

For –hash option all algorithms supported by gcrypt library are avail-
able.

NOTES ON PASSWORDS

Mathematics can’t be bribed. Make sure you keep your passwords safe.
There are a few nice tricks for constructing a fallback, when suddenly
out of (or after being) blue, your brain refuses to cooperate. These
fallbacks are possible with LUKS, as it’s only possible with LUKS to
have multiple passwords.

Thank you to the folks who wrote cryptsetup. It is available here:

http://code.google.com/p/cryptsetup/

Spectrum

Posted on by December 24, 2014

spectrum-640

Here are the frequency allocations in all of their glory, available from here. It’s interesting that the U.S. government auctions off spectrum for mobile carriers, who then pass this cost back on to the very people who “own” it. But wait, there are some frequencies that you can use gratis, even commercially. It just so happens, though, they don’t go through walls well. Can you find this needle in the haystack?

Large, yet Small and Inexpensive

Posted on by December 22, 2014

With the end of the year upon us, it is time to see what goodies there are give to friends and family. In looking at technology, it appears SanDisk has a 128GB micro-SD card for $110. SanDisk is able to accomplish this feat using a 19nm manufacturing process.

It really is amazing to see so much produced on so little silicon. As EDA zeros in on the possibility of sub-nanometer circuitry, the industry is also started producing chips with FinFet, moving away from planer chips. Every year, the question is, how can it get better? Well, it’s my guess that we will not see the mass results of these advances for a year or two. When they are here, though, today will seem like the good old days of being off the grid. Small, perhaps unnoticed IP enabled devices will be pervasive in all aspects of our lives.

This opens many doors, some of them we wish we could keep closed. Just ask Sony. There will be an overwhelming flood of technology, and it will be the tiny little consumer that is washed out to sea, naked, alone, and scared. Not to stray too far from the benefits, but we should warn our friends and families about the consequences of ignoring the obvious.

SPAM

Posted on by December 7, 2014

One question that often arises is, look at this e-mail I received…have I been hacked? The answer is not quite as straightforward as one might think. The protocol that defines Internet e-mail was written a long, long, time ago, and the Internet was a much different place than what it is today. The protocol itself, by today’s standards, is pretty weak allowing for much mayhem. It is pretty easy to send e-mail as someone else, and the only resolution is to closely inspect the mail delivery headers – a record of which machines have handled the message.

The most common type of attack is a phishing attack, where a recipient is lured into opening and perhaps clicking on a link in the body of the message.
These messages can contain various means of corrupting the e-mail program itself, allowing for code to be installed on the recipient’s computer (obviously the worst case scenario), or simple theft of data. Most purveyors of software know this game now and are sensitive to it, but as long as programs are big and complex, the opportunities will still exist, with new ones presenting themselves with every update. Now, with plenty of free e-mail services, cloud storage, and mobile devices, new attack vectors have risen, with bigger payoffs, making it nearly impossible to thwart.

There are essentially two camps of spammers: one that simply sends bulk e-mail in the hopes that a very small percentage actually buy products the spam is selling, and the other, that hopes actually control the system of the target. Both rely on massive lists of real e-mail addresses in order to function. The simplest way to gather e-mail addresses is to exploit known weaknesses of the e-mail client and/or the operating system on which it runs, often resulting in the export of the target’s address book. Once this knowledge is gained, it is possible to send e-mail as each person in the address book to all people in the address book, resulting in more successful hacks and address book losses. Put another way, if someone else has you as a contact and they don’t give a flip about good computing practices, password strength, or encryption, then most likely there will be spam sent as you to your colleagues. This is why a friend may say you have been hacked after receiving a message from you, when really it was a friend of a friend of a friend. The hack may have happened long ago, it may have been recent. It’s also quite possible that all of these stolen address books are aggregated, analyzed, and traded in the black market.

But why are there so many spams? The answer lies in the latter group, and it involves money, extradition laws, and botnets. If an attacker (spammer no longer applies here) can insert code on a target’s computer and then control the computer, they can build a much more efficient, disperse, spamming operation. When this happens on a large scale, it is called a botnet. The opportunities for illegal monetary gain increase greatly with botnets, one being extortion through a threat of a Distributed Denial of Service (DDOS) attack. Think of a DDOS attack as tens of thousands of computers flooding a targets network with useless requests, causing all of the infrastructure to overload and miss legitimate business activity. Perpetrators often live in countries that have no extradition treaties with their victims, essentially giving them a safe haven.

Businesses that generate online revenue are typically the victim. Since there is no real good way to fight large scale (botnet controlled) DDOS attacks, they often pay quietly, leaving no record of the extortion. Occasionally one will, and it makes headlines. Do a search of “ddos extortion attacks” and a clear picture starts to form.

So what can be done? Well, you’re in the same boat now with many large companies, and they have much deeper pockets. It’s probably best to analyze e-mail headers to look at the point of origin of the spam. Make sure that the spams do not originate from your IP address, or any IP address you use to send email. Also, look at the payload. Are there images or hyperlinks in the spam? If so, then most likely it is an attempt on your security, using your contacts as additional leverage.

All of this, stemming from a simple mail transfer protocol (SMTP) written at the dawn of the Internet. That’s right – SMTP is the actual name of the protocol. If only it were not so simple.

Requiem for Minecraft

Posted on by December 3, 2014

Below is a post I’ve had in my Drafts for a while now. Sadly, Microsoft is buying Mojang without the creative minds behind it.

As any parent knows, MineCraft is white hot. What makes this game popular? The game is constructed in such a way to avoid violence as a principal motivator. A player can farm, mine, build, or fish. Games within games can be created. This opens the door to both genders resulting in exciting interactive play with friends. Whether the game developers realize it or not, MineCraft draws upon the experience of the early days in computing, only using modern computing power.

What technology drives MineCraft? Java. The promises of Java yesterday have come to fruition today. Platform support includes Windows, Mac OS X, and drum roll….Linux! Write once, run anywhere. As a person whose desktop is Linux only, it is nice this program work so well, including 3D OpenGL graphics. The author of MineCraft, Mojang, offers a free to use server version.

Below is a Planet Money segment on MineCraft.

The above segment talks about the business model of MineCraft, and folks, it’s simple. Buy it, and play it forever, without advertising, marketing, or in-app purchases. How quaint! It just shows how we have lost our way when driven to maximize profits. The world of mobile is a very dirty, creepy place when it comes to big data.

Mojang did none of this and created a game for all people of all ages and both genders, open ended…let the brain explore just for the sake of it, and look what happened. Profits followed for Mojang, with the lead developer owning a majority of Mojang, and therefore getting a majority of a 2.5 billion dollar buy out. It should be a lesson to the people hell bent on squeezing every penny out of our children, but it won’t. The big data way is the easy way. It’s easy to sell investors when you have all sorts of data surreptitiously taken from your customers that can be shaped, repacked, and sold to anyone willing to buy. It’s disgusting.

Job well done, Mojang. Thank you for the fun. After reading why you sold it, it is completely understandable.

What can we expect for MineCraft now that Microsoft owns it? If history is any guide, then a game written in a language that Microsoft did not invent running on operating systems that Microsoft does not sell…well, you get the idea. Microsoft is a big data company, too. Now, how can they increase market share and profits for their investors with MineCraft? The ways are innumerable, and also counter to what MineCraft was. Yes, it’s the past tense now, because it will never be the same. The first comment on the Planet Money link sums it up the best:

Game Over

Links:

Markus Persson, Mojang
New York Times
NPR’s Planet Money
NPR’s Planet Money Transcript

Follow the Baytrail

Posted on by December 12, 2014

The first Intel Baytrail (low power) PCs are out, and mighty interesting. Shuttle has produced a system administrator’s PC that can pretty much do anything, running with only a 65 watt power supply. The specs are impressive. While it ships with no memory or hard drive (a good thing), it can support up to 16GB of RAM. Folks, 16GB of RAM for a 65 watt system is wonderful and insane at the same time. It also has three network interfaces, one of them WiFi B/G/N. HDMI, USB3, SD, DVI – all for $200 – oh my!

The Facade Crumbles

Posted on by July 10, 2014

It has been over a year since the revelations of mass surveillance were exposed. As expected, the people of the United States have gone back to work, placing issues of privacy deep into their subconscious. When a new revelation is reported, we think, oh yeah, that security thing again. We bury the thought that all of our decisions are logged and indexed into a database for someone to sell, analyze, or subpoena. We are told, over and over, that if we’re not doing anything wrong, we have nothing to worry about. On the surface, this is true. It is what I would tell my elementary school age children. But my children will grow, and I have to wonder what world we will leave them.

TN-Noonan-20130816-Privacy Peggy Noonan wrote some wonderful words on the matter back in August of 2013 (unfortunately now behind the WSJ paywall). She writes, “Privacy is connected to personhood. It has to do with intimate things—the innards of your head andcheart, the workings of your mind—and the boundary between those things and the world outside. A loss of the expectation of privacy in communications is a loss of something personal and intimate, and it will have broader implications.” She continues, making the suggestion that when the Fourth Amendment’s balance of power shifts away from the individual to the State, the First Amendment becomes far less important. We self-censor. We fear.

TN-PJ_Forced_Exposure.jpgUnfortunately, this is now all too true. During the summer of 2013, Pamela Jones of Groklaw, shuttered her blog. Her work from 2003 to 2013 was invaluable to the Linux community, providing comfort in the wake of a very scary copyright infringement lawsuit brought by SCO against IBM that threatened the very existence of Linux. As a paralegal, she carefully documented all the details of the case, providing daily insight to the technical community, making the legal world bearable. She was there to say, do not panic, the assertions are only assertions, there will be an end to it all. Pamela Jones, not her real name, was an anonymous blogger. Her work existed in a mine field. What we learned from the SCO case is anyone can bring suit, and it is costly to defend. Had she ever been outed, unmerited defamation and slander suits would surely follow. So, farewell Pamela Jones. We miss your thoughts and your right to free speech.

TN-PJ_Forced_Exposure.jpg What makes the shuttering of Groklaw a terrific story is its relationship to another shuttered site, Lavabit. Lavabit provided secure e-mail services. Edward Snowden was alleged to be a customer. The founder and CEO, Ladar Levison, received a national security letter, not just asking for one SSL private key, but all keys. Lavabit’s foundation is based on trust. Once broken, there is nothing left. Realizing this, Ladar Levison literally pulled the plug. He took everything down, and eventually turned over encryption keys (albeit in a 4pt font on printed paper). Ironically, the NSA may have already had the keys to Lavabit’s kingdom, if Lavabit was using rev 1.0.x of OpenSSL. During this time, OpenSSL was (unknowingly to some) suffering from heartbleed, allowing for remote decryption. Farewell Ladar Levison.

TN_CC_Tracking_Bloomberg.jpg
It does seem bleak. While Edward Snowden was talking about government surveillance at SXSW 2014, tech companies were introducing all of their wearable smart devices. There is no better example of this contradiction. We want privacy, but are perfectly willing to ignore license agreements and contracts in order to run our gadgets. With the Internet of Things moving into high gear, options for surveillance will blossom horrifically. Keep in mind, too that surveillance is not limited to communications. If you use plastic to purchase anything, data brokers purchase the right to analyze your spending patterns down to the item level, and sell that information. This scenario is by far the most frightening, because it zeroes in on the core of economics. The Summer of 2013 may be remembered as the good old days.

Is there anything that can be done? Absolutely, but it is up to the individual to care, to cherish the freedoms not yet lost. Now that chess table is set, we can see what it takes to play. This game will be difficult, but it is a game that can be played. To start, every person who cares about free thought and speech should take a position that this inevitability is not certain, and that there are countermeasures, that good people do win. Technology meets the needs of those who desire it, and it also is cyclical.

Peggy Noonan: What We Lose if We Give Up Privacy – August 16th, 2013
Pamela Jones Farewell Notice – August 20th, 2013
Lavabit Farewell Notice – August 8th, 2013
Ladar Levison Battle with FBI (NPR) – October 3rd, 2013
On the Media, New Security Standard for Journalists – August 16th, 2013

Goodbye Freecode

Posted on by June 25, 2014

Sadly, on June 18th, 2014, FreeCode.com (fromerly Freshmeat.net) became static. This web site has been one of the most useful for system administrators. It aggregated software projects and provided a list of subsequent updates. Think of it a searchable RSS feed for all free and open source software on the Internet.

I consider it a site that helped system administrators today (and yesterday) craft some of our modern platforms that we enjoy. Thank you Freshmeat.net for all of these years!!!

The Magic of Rsync

Posted on by January 27, 2014

Without question, rsync is one of a handful tools required for any systems administrator to keep their heads above water. It hooks in wonderfully with other tools to allow synchronization of files across a wide variety of media.

COW, unionfs, aufs – they all rely on a read-only source, with any changes getting written to a separate area. From the perspective if the user, both areas appear to be a single source, all writable. For an operating system, a builder can create a master copy, allowing different types of hardware adjust where necessary: one copy, infinite combinations.

While rsync does not provide real-time access as described above, it does allow for a system administrator to look at two given trees and see what has changed over time.

For a basic example, look at the following:

example $ mkdir foo
example $ touch foo/{1,2,3,4,5}
example $ ls foo/
1  2  3  4  5

There are five files in $PWD/foo, 1, 2, 3, 4, 5. Now, using rsync to back them up:

example $ rsync -av foo/ backup/
sending incremental file list
./
1
2
3
4
5

sent 273 bytes  received 110 bytes  766.00 bytes/sec
total size is 0  speedup is 0.00
example $ ls backup/
1  2  3  4  5

Here, a perfect copy of the files 1, 2, 3, 4, and 5 are made from the source (foo) to the destination (backup).

For the next example, $PWD/foo gets new data, and only the delta should be placed in a new directory:

example $ touch foo/{a,b,c,d,e}
example $ ls foo
1  2  3  4  5  a  b  c	d  e
example $ rsync -av --compare-dest=$PWD/backup foo/ delta/
sending incremental file list
./
a
b
c
d
e

sent 323 bytes  received 110 bytes  866.00 bytes/sec
total size is 0  speedup is 0.00
example $ ls foo
1  2  3  4  5  a  b  c	d  e
example $ ls backup
1  2  3  4  5
example $ ls delta
a  b  c  d  e

Nice! If I overlay delta/ on top of backup/ then it will equal foo/. This is a pretty basic example, but it is apparent that if this is combined with cron, some pretty fancy things can happen.

For example, if you needed to know any file changes (and what they were) from any point in the past, this process would allow it.