Some notes about testing network block devices

I tested another random thing: network boot options short of iscsi. The first thing I tried was mounting the root filesystem from the startup environment over network block device. That was extremely simple and worked fine. There is a likely shutdown glitch, but if the root filesystem is synced the effect should be relatively minor. I am using xfs and it recovers quickly from that kind of incomplete shutdown relatively easily; well, at least it did in my limited tests. Another interesting idea for testing is to check if it is somehow possible to mount the root filesystem using ssh+nbd. That introduces some extra complications for the root filesystem, but it could definitely be adapted to encrypt traffic for another mount point, such as /home. I guess this may become practical for those who have fiber optic connections. The technique I tried is similar to sshfs, but uses only the the default tools.

Here is a rough idea of how to do this. First, login at the computer hosting the image that is to be mounted somewhere in the filesystem. Start the network block device accessible only locally. Add more local security here, if necessary.

ssh -K douglas.mayne@somecomputer
losetup -f somefile.img
nbd-server  127.0.0.1@2000 -C /root/non /dev/loop1

Then setup the connection back at the computer that would like to use this network disk:

modprobe nbd
ssh -K douglas.mayne@somecomputer -n -L 2500:localhost:2000
nbd-client localhost 2500 /dev/nbd0
mount /dev/nbd0 /home

For the encrypted traffic for the root filesystem, I am wondering if a strategy based on pivot_root might work.

On Friday, I tested this a bit more. I missed the obvious solution for encrypting the root filesystem using network block device. The obvious answer is to use device mapper on top of a network block device. I tested this and it works in the standard way.

dm-live issue and fix. Also debugging with kexec

I noticed that successful bootup to a root filesystem connected via USB was broken on certain hardware. I initially assumed that the failure might be associated with a kernel bug, or perhaps to do with specific motherboard chipsets, or a flaw in my understanding of udev. I was stumped and invested some time to figure out what was wrong. This screenshot shows a typical bootup failure. Googling the failure shows others are experiencing the same troubles, perhaps for similar reasons. I didn't see any obvious solutions. For the last while I have mostly been working around the problem by using a root filesystem connected directly over SATA.

I stumbled on the solution by chance:

Include the ehci-pci module among those that are loaded at boot. That ensures the best performance for the external drive, be it a flash disc or a magnetic disc.

Because booting to a root filesystem on USB is the heart of a live USB install, it's good to know that it was a case of PEBKAC and not something more endemic.

With that fix out of the way, there are some other changes to my dm-live startup environment including following along with the stable kernel releases in the 3.10.x series. Currently, I am testing with kernel 3.10.51.

By the way, in the debug phase of this problem, I was pointed to kdump. I played around with the kexec utility to see some of the basics of kernel debugging and how it is designed to work. I found right away that the basic kernel configuration that I have been using, which is extremely similar to the basic default Slackware kernel, is configured to not generate symbols, enable boot-time reservation of memory using the parameter crashkernel=, or be arbitrarily relocatable. To use kdump there are a few changes, including adding the full compiler symbols the compiled objects. In general, I agree with Slackware's decision not to include the symbols because they increase the size of the resulting compile by a factor of 5 or 6 times. In the end, throughout my attempt to debug the above problem, I wasn't able to generate the right kind of crash, i.e. one that actually writes its crash data. I was pretty sure I was doing it right because kexec -l and kexec -e were working as expected. The panic code when loaded with kexec -p just wasn't tripped for whatever reason. It was too hard of crash, I guess.

The following table details some of the hardware I have used in the PC era. It begins with the venerable Compaq luggable that launched the "clone" wars. I've seen a lot of hardware along the way. In the early era these brands come to mind: Everex, ALR, Northstar, Gateway 2000. Each row in the table shows what were compelling upgrade points along the path. At each step, the blazingly fast new designs left previous generations in the dust. Somehow, what had been a pleasure to use now seemed to be a chore, or too slow to bootup. What had been fast, was now painfully slow. I hope I can add benchmarks to put a mathematical value on each row. Perhaps, by comparing the next row to the previous, or some other baseline.

The genesis for making this table was XP's expiration. That didn't turn out to be as big of a deal as I thought it would. The heartbeep SSL bug made bigger news. XPs overall worldwide usage is estimated as low as 10% currently. The same estimates peg all Linux at 5%. Of course, I use Slackware Linux as my daily OS. XPs expiration forced many to upgrade to marginally better hardware, at least, for those using the Windows platform. I rolled out just over a dozen machines that use i5 CPUs. They have 16G RAM typically. That is a factor of 8 times more over the standard amount that I had incorporated at the previous level (i.e. at Core 2 E6600)

As I was finishing the rollout of the latest generation of PCs with the evolutionary step in operating systems, I had two main thoughts. First, would this be the last hurrah for desktop PCs? Will everyone demand tablets? Will Android and iOS eclipse the Microsoft juggernaut that lasted for a generation? My guess is that the days of PCs as we have known them are limited. Second, I thought how the Windows interface has gotten progressively worse. These are subjective opinions, I know. But I find that Windows 8 is not an evolutionary step that is better than Windows 7. Likewise, Windows 7's interface was worse than XP. I may be an old fogey, but give me consistency for the best productivity. These upgrades have pulled the rug out from under users for no go reason. Again, just my opinion.

8088 clocked @ 4.77 MHz	typ. less than 640k	8-bit compromise of 16-bit 8086; 8087 math coprocessor optional. Used in the Compaq "Luggable".
80286	typ. 1MB, OS limited use beyond 640k	IBM "AT." These machines typically had sockets for 1 MB RAM.
80386 DX @up to 20 MHz	1MB designs still prevalent	included virtual 8086 mode; first to use 32-bit mode; still required a coprocessor for fast math functions; a very important chip. Motherboards could support 1+4MB RAM on proprietary buses.
80486 DX	4 to 8 MB RAM	upgrade to 80386 included on board math coprocessor. Rolled out in Gateway 2000 desktops.
Pentium	First machine outfitted with 32 MB RAM	OS: Windows NT 3.51
dual Pentium Pro @200 MHz	32 to 512 MB	dual CPUs in the "686" era kicked the Pentium's ass. OS: Windows NT 4.0
Celeron w/ 128k L1 cache @800 MHz	128 - 512 MB	budget chip in the "686" line disabled support for multi-CPU in hardware. Motherboard chipset support for SDRAM. These boards were plagued by a bad capacitor problem that caused premature failure.
Celeron single CPU upgrade, w/ 256k cache @1300 Mhz	512 - 1.5 GB	Motherboard offered supported for SDRAM clocks and ecc memory
Pentium 4	Ran hot compared to predecessors
dual Pentium III each w/ 512k cache @up to 1400 MHz	1 - 2 GB	dual chip configuration was good at keeping the machine responsive to the user. Path to memory was showing its age when compared to P4 designs.
Pentium M @up to 1870 MHz	1 - 2 GB	First single core chip to do better than P4. Dell D610 used this chip.
Core Duo	2 - 4 GB	Dual core CPU with good mobile potential. Dell D620 used the T2400.
Core 2	2 - 8 GB	64-bit CPUs spurred the move to 64-bit OSs. E6600, E6750, etc.
i3	4 - 8 GB	22nm architecture saves power on mobile platform. i3-3227U has 4 cores @1900 MHz
i5	16 - 32 GB	i5-3570k, i5-4670k. XP's premature expiration spurred a hardware upgrade to Windows 7, 64 bit at many offices committed to using the Microsoft platform.