Locking myself out

I've become quite adept with Android ROMs of late. In the last month I've re-flashed my HTC Legend at least eight times, one of them a desperate measure to get back in after idiotic inattentiveness.
The after-market firmware scene is amazingly vibrant, and I'm typing this post on Cyanogenmod's CM7.2, which brings Gingerbread (Android 2.3) to my two-year old device, something the manufacturer has no intentions of doing. Just as in my previous post about digital legacy, the open-source community is doing what vendors simply won't, and surprisingly well. I'm so pleased I've even offered to upgrade a (decidedly untechnical) friend's Legend.
Doing this requires bypassing the vendor's OS locks - "rooting" - and optionally the firmware locks. By default, open ROMs leave these locks off to preserve your Newfoundland freedom. There's a problem there.
Yes, I want to continue hacking away, but I'm not comfortable leaving root accessible for my friend's device. A diagnostic mechanism called USB Debugging doesn't reset itself on reboots rendering the screen lock rather useless. The Cyanogenmod authors have stated root access will be off by default from version 9 (Android 4.0) onwards, but the Tom I tried on my device (so very close to useable) had no such measure.
Should a rooted and reflashed device be misplaced or stolen, the new owner could quite easily get in. Remote wipe and location apps can be disabled (Link2SD makes this trivial) and data compromised. As a network administrator I wouldn't let one of these devices anywhere near my messaging system, since the policy enforcement engines rely in unprivileged users to function.
I could of course re-applt the vendor recovery software to prevent the OS image being altered, and re-enable firmware controls to stop this being undermines, but this is all moot once the OS can do as it pleases.
Open-source software thrives on freedom, and I love that attitude, but ensuring a well-controlled network often works against that when viewed from certain angles. So, what path?
Security systems relying on obscurity of design have repeatedly been subverted, but how do you convince open-sourcers to build effective blockouts into their project?

Perhaps by helping them realise the real loss their moms and kids might experience when their unrelentingly open device is abused?

Blizzard, a Few Words on API Implementation

Over on battle.net, Blizzard is defending their action of banning users for using Wine to run copies of their hugely popular Sword & Sorcery game Diablo III. In Blizzard’s words:

Account Action: Account Closure
Offense: Unapproved Third Party Software
A third party program is any file or program that is used in addition to the game to gain an unfair advantage. These programs may increase movement speed or teleport heroes from one place to another beyond what is allowed by game design. It also includes any programs that obtain information from the game that is not normally available to the regular player or that transmit or modify any of the game files.

Now I happen to know a fair bit about the internals of Linux and Windows, and can say one thing for certain: Wine is not the problem.
As mentioned on Ubuntu Vibes, the Wine developers are keen to distance themselves from the issue, and fair to them it is not their fight. But Blizzard definitely has some explaining to do.
I have been very much on the side of those calling foul of Blizzard's method of providing the game state, essentially running a single-player game on their servers. While Diablo II saved locally and ran game logic entirely independent of servers, a single-player game in Diablo III is essentially a single user multiplayer instance – all state is held on their servers. This has advantages, like quickly and more reliably allowing me to invite internet users to join my game, but that’s about the only benefit I can see.
It would appear that Blizzard are trying to clarify that they don’t ban based on OS (or API implementation), and its a subtle point. Whether cheating is actually happening in this particular community as they allege is not quite evident, but since Blizzard don’t reveal the exploit a user is supposed to be using there is no recourse for refund or rectification. I’ll leave that point alone, save to say that I would prefer other users don't cheat, but that it’s irrelevant if I’m in a single-player game. Unfortunately, the way the game is designed there is no such thing.
There’s been ranting about other problems, but one I’m most interested in is legacy. The reason almost any software has a minimum specification for operating system is a guarantee that they APIs they need will be present. Wine is an undertaking to replicate the APIs in Windows for use on other platforms, prominently Linux. Backwards compatibility is the bane of Microsoft’s existence, but there’s no option there really.
The big issue here is the digital legacy. I used to play with my father’s toys as a kid, and he with mine – that is until I started gaming. I have a partner who went barking mad as the release for Diablo III approached simply because she was excited to relive the enormous amount of time she wasted in college playing Diablo II. The problem is even graver than if Steam went dark for good where i simply couldn’t install my games. If Blizzard pulls the plug I cant launch games I’ve already installed, ever! And frankly, twenty years from now I just might like a shot of nostalgia.
But the hardware/operating system of the future might not support it you say? I’d say you’re dead wrong! Steam released the classic X-COM games packaged running in DOSBox, showing that there are solutions to anciently legacy games on modern platforms, and open-source no less.
So just what is Blizzard’s problem? The fact that Wine presents a functional implementation of the Win32 API is of little relevance. there’s no clear statement that non-Microsoft/Apple platforms are enough to get you kicked, far from it they leave it up to users to employ best-efforts. Blizzard may very well claim that they only support native implementations, but WOW64 is really an implementation of the Win32 APIs, when the underlying system is in fact 64-bit.
So, we have both closed-source and open-source solutions to the problem of running legacy code, both apparently providing a reasonably good experience (in fairness I think the Windows implementation is excellent, no idea on quality using Wine). Twenty years from now I have a fairly good feeling there will be legacy emulators (Windows 7-in-DOSBox, Wine for Linux 7 or something as-yet undreamed of), but I am almost certain Blizzard won’t be keeping up a service for Diablo III, even if I just want to show my kids how their dad used to kill demons fell more in love with their mom.

Are YOUR hard drives noisy enough?

A conversation with a colleague presented an interesting question: Which hard drives are the quietest?

Now there are excellent resources online like Silent PC Review and QuietPC for finding the right components to build Home Theatre PCs. But this got me thinking, are my hard drives too quiet?

A lot of parameters are accessible on the firmware of modern hard drives, including readahead optimisations, cache policies, as well as monitoring options for errors, failures, temperature and even how many times the drive has been switched on. Linux distributions provide the smartctl utility for retrieving the monitoring variables, and the hdparm tool for setting all kinds of parameters, one of which is quite interesting...

The "-M" option sets the Automatic Acoustic Management level, which in most disks is one of OFF, QUIET and FAST, corresponding to integer values of 0, 128 and 254 respectively. Now the system I'm typing this on is my laptop, and I don't want it clunking away while I work (well, apart from the keypresses required to type this article). So, I set it to QUIET (128).

But, I have a server with four disks, two 750GB and two 1.5TB, running my server applications (file shares, telephony, media streaming etc), and crucially my virtualised lab which tend to be disk-heavy, especiall at high concurrency when each OS instance thinks it has exclusive control over volumes and optimises access accordingly. This server used to be located in my office alongside my desk, and yes it got a bit clattery. Now, it's in another room tucked under a cupboard, very headless. What do I care about noise?

Now, I could do a big benchamrking experiment, but this should give a reasonable first glance. Yes, the system is doing other things, but since the cpu sits at 97% of the time in a low-power state, the load isn't that high anyway and is probably negligible to the result.

Doing a simple read of one gigabyte from the RAID-5 array, at different offsets to remove cache interference, shows a remarkable difference. Reading a 1GB data segment from the logical drive (under the filesystem) increases throughput from 147MB/s to 220MB/s when the acoustic mode is set to FAST.

On the filesystem itself (ext4, defaults), extracting the latest Linux kernel source from a tar.bz2 file found on kernel.org more than halves the duration, from 3:03 to 1:30. Reading the resulting directory tree with `ls -lR` improves from 10s to 5s.

In the end, it probably makes little difference to my day-to-day tasks, but optimisation is central to any real techie's heart.

How to set or measure these features in Windows? No idea.

The Bolt-on Operating System

For years I’ve wondered why on earth so much is crammed into the Windows base image. Sure, they’ve got a decade worth of hardware to support, and since vendors don’t create standardised, reusable code like Linux, this is a significant factor in bloat.

But one of the longest running gripes I have with Microsoft’s OS offerings is that all manners of features are included that I don’t care about – and some that I am super-passionate about are just plain gone! The base installation for almost any major distribution of Linux will include a lot of productivity tools, but leave out some others. It comes down to personal choice, but since all the software is free and available to install from the Internet, this is no inconvenience at all - assuming good Internet connectivity.

That last point is actually quite big, since the distribution may contain an impressive array of software, in the end I’m probably going to want something that didn’t ship on the CD. Can’t have it both ways I guess.

But obviously, the last thing Microsoft would want is to not install something, then have you go back to the installation disc for the features you’re enabling. Of course, they STILL haven’t figured out that Unknown Device from Unknown Manufacturer isn’t a helpful message.

One of my recurring gripes (that is before I started blogging, so I can’t really prove it) is that my Windows Server has a GUI. Seriously, I don’t want a GUI. I want my apps installed on my server, and the management interface installed on a workstation somewhere else. Or the way Linux handles it, X as a process, in my own privilege space that can be launched just for me, and perhaps VNC or X-like somewhere else. The problem of course is that almost every Windows application requires a GUI to install. Sure, MSIs offer silent installs, but so often these line-of-business apps don’t have neat MSI or respect the silent option.

There are two things I’m getting at here;

First, even though the product is named “Windows”, and it’s grown out of a desktop graphical OS that pretty much reinvented the way we deal with computers (though much cred to Apple), I want a server. No people standing in front of it, so no pretty colours required. All the prettiness should be produced by the apps and displayed as IP packets. And on the same note, I see a bunch of sound drivers hanging around.

Well, Windows Server 2008 has the Core option you tell me. It’s a good step in the right direction, but I was genuinely disappointed to see the command prompt surrounded by a window. MMC is still there. I had actually expected to see a text console only, no graphics. Quite simply, it’s a waste of resources. Any app, service or component worth its’ salt is manageable remotely, from simple DHCP up to complex SQL Server clusters. It’s also a danger, as I recall at least one major outage at a previous company thanks to faulty graphics card drivers from <server vendor name censored>. And the GDI component, dating back to Windows 95, handles page and print rendering. Exploits in the 16/32 bit era and in the 32-bit era come to mind. That last one spans products over three release generations, all for a component that doesn’t belong there in the first place.

If I want graphics processing, print rendering etc, then let me add it on later, the way I would Ghostscript on Linux to make a PDF. That brings me on to my second point, and the trust of my argument: components that don’t belong.

The Sasser worm devastated computer estates around the world, by exploiting code hooks in the LSASS.exe process that handles security arbitration between requesting apps/users and security providers. Read closely into the articles, and you’ll notice that it is specifically a problem with code for dcpromo, the process that turns Windows Server into a Domain Controller. The applicable hotfixes patch the code on Windows Server.

On XP, the hotfix removes the code.

Just what was it doing there in the first place? I know the server and desktop products share a codebase, but this irks me. I’ve personally implemented (though not used) a hack to enable RAID on Windows XP that officially doesn’t support it, since the raid driver and all the GUI code is present but disabled. I suspect the same is true of the EFS code in all Home versions of Vista (since you can read EFS-encrypted files from Windows XP upgrades just fine).

Windows Server 2008 requires you to specifically add features to your server before activating them, like AD Directory Service (AD DS), and installations of SQL Server and Exchange (at least) check for updates to the installer before running, getting them closer to the model of Linux distributions – adding a feature from the online repository ALWAYS adds the latest version.

In the end, the development models of these two are very different, so I’m keen to see what further advances can be made on both sides. As always, security and functionality butt heads, somehow I end up with the headache…

Virtualisation of basic services

So, virtualisation is everywhere. It departments are running more and more of their workloads on virtual hardware, and in the data centre that’s a big thing. It’s an accepted fact that most servers in an organisation run at near idle all day long, and even core systems like directory and file servers barely crack a sweat serving their users. For small and medium-sized entities (yes, they have datacentres, just not raised floor, air-conditioned ones like the big guys) this is a problem.

I was asked a while ago why I enabled file compression on my laptop, and seriously considered it on file servers. A few years later another query was why I enabled full-disk encryption on the same piece of kit. Doesn’t it affect the responsiveness?

The simple answer is yes. And no.

CPUs, memory and disks today are so, so fast that there’s not much they can’t do for an individual user. when asked what laptop a person should buy, I recommend the pink one if it’s a girl asking, and jet-black for the guys. Seriously, laptops today can wipe the floor of a desktop from five years ago, and last I checked we’re still doing the same things as back then: browsing the web, writing emails, laughing at cats. If you’re a developer, gamer, or serious showoff, you probably have a dedicated desktop for the hardcore tasks, and the point remains: Moore’s law has seen to that.

Of course, anything you put in the path of data from the disk to CPU/memory is going to slow things down, but this comes back to the more important point: Do I notice?

When I’m running a compile job, chances are the disk’s head movements trying to find each source file and library is going to be the first bottleneck. Then, if it’s a biggie like mplayer, the CPU is going to be loaded for a while doing the compile. Encryption? Oh, I didn’t even notice it’s on.

This brings me to my point, with uber-powerful, multi-core servers cheap as chips (e.g. HP ML110 G5, Dual-core 2.8Ghz Core 2, 1G RAM, 250G SATA for 350 ex tax, HP.nl website 2009-12-07), do I really need to treat these things like fragile porcelain ware?

So, given a department of 100 users in a remote office, where I need to provide local AD, file services, updates, VPN to head office, backups, what’s to stop me buying a cheap server, loading it up with RAM and disks and virtualising all functions in one (or a few), neat little grey cubes with a reputable brand name on the front?

Now here, I’m thinking of Linux as the hypervisor, but that’s my flavour of choice. Something as simple as AD is a good case, since it’s by definition a replicated, resilient database so failures are not a big impact (authenticate somewhere else), and doesn’t do much in the day. It’s also advised to segregate roles on Windows servers, and for resilience perhaps have a second box if you’re paranoid . That’s one, maybe two little cubes that I can’t share for other functions. And it’s going to consume a few gigabytes of disk space, and even less RAM. A whole cube, just for you? Share!

Linux virtualisation has come a long way, and I consider it to be solid, fast and these days wonderfully easy to administer remotely. KVM, as I use, is just another OS process, no funky drivers or installations and since it runs on a full (and standard) instance of Linux, all the management tools for networking, storage, logging and troubleshooting are available.

One simple example: since all traffic traverses the host’s virtual network switch, all traffic can be inspected by a packet sniffer as powerful as Wireshark. No drivers in Windows, no strange binaries. If everything’s virtualised on one box, ALL the traffic is visible for troubleshooting in one spot. Neat!

Another, consider three servers, each requiring 100G storage. That’s probably six disks (minimum) if you’re doing things right (RAID-1). One one hypervisor host, that’s four 100GB disks (RAID-5), only two if you go the 500G route (RAID-1, plus a spare I suppose). Now I’m also a fan of software RAID. My case above stands, there are other factors more likely to impact performance, no least the client’s ability to pull data fast enough from a file server.

So, Now I’ve one box with my local services running in VMs. Full-disk encryption? Well, Windows 2008 has BitLocker, with all the TPM and USB key requirements. How about logging in to the host remotely via SSH, one of the most bulletproof protocols for remote management, and enable the encryption remotely. With a script. Not one byte in plaintext, not even the partition table. Break that, evil hardware theif!

Snapshotting? check
SNMP/performance monitoring and alerting? check
Hardware support? check
Virtualising legacy systems? check (especially MS-DOS on pesky 64-bit Windows)

Worth a think

The case for open-source virtualisation

I’ve been quite a keen follower of virtualisation since the early days of availability on commodity hardware. Obviously, big rigs like IBM, HP etc have been doing hardware partitioning since time immemorial, but I’m interested in the stuff that lets me run an OS concurrently with my main OS on a desktop.

To my mind, one of the big advantages of Windows (and to a lesser extent Linux) is the homogeneity of the OS on desktops and server. That is, if I write an application, website, database etc on my Windows PC, compile and run it, it should deploy to my Windows Server without any changes. This is in stark contrast to the development model for older big iron systems, where the development happened on the system itself, probably on a dedicated development partition, but quite certainly not on my Windows (OS/2, DOS etc) workstation without cross-compilation.

Nowadays, it’s taken for granted, but that’s always been in the back of my mind. With virtualisation, I can now deploy a test partition on a development server for Windows. It’s not just for development, but for testing too, and in my current path to get my Microsoft certifications up-to-date, it’s a godsend. I’ve got a fairly sizeable server that does almost all of my home tasks; file shares, email relay and filtering, VNC, photo sharing website, music streaming, proxying, VPN, and I’m working on VoIP. The thing is, it’s Linux.

I run Fedora. At one point I banished Microsoft from my home, just to see if it was possible. I wanted to accomplish as much of my home automation and services using open-source software as possible, and put Fedora on my main server (it’s actually been there for ages), Fedora on my power laptop and Ubuntu on my lighter laptop. and hey-ho, it works!

Well, actually, my mileage varied. Ubuntu is just great, it works well with laptop hardware (especially the Intel graphics), wireless and sound. Fedora also just works, that is until I tried to virtualise on the laptop.

I had read about Xen some years back, and how it offered paravirtualisation on Linux. I tried to get that working, but once I figured out it interrupts communications with my Radeon chip (they claimed it wouldn’t) I dropped it. Enter KVM, stage right!

I’ve been a keen follower of AMD’s hardware virtualisation (Pacifica) since the inception. I’ve actually been a very big fan of AMD’s for some time, but that’s for another post. VMware and other players have been doing hardware virtualisation for some time, but the Intel architecture just doesn’t play well with others, meaning the host OS and hypervisor needed to do a lot more work than ideally required to keep up the illusion for the guest. VMware’s intellectual property in the regard is substantial, and for years were the cut-and-dried leader in the field.

Qemu is a mature hardware emulator I’ve been using for years that runs all kinds of CPUs on multiple host OSs, but still runs in user space (i.e. no kernel privileges, and a lot of context switching for privileged ops). As soon as the kernel module is loaded (kqemu), things perform rather well indeed, but still noticeably slower than bare metal.

Hardware virtualisation (HVM) in Intel chips changed that, since now a lot of the grunt work like intercepting privileged interrupts can be caught by the CPU and sent off to the Hypervisor efficiently. In no time at all, the existing Qemu binaries were extended to include KVM, Kernel-based Virtual Machines. I’ve been a convert ever since.

By using a well-established virtualisation platform like Qemu, with excellent hardware support, KVM runs a lot of guests!

So why am I raving about this? Well, I often get asked why a Microsoft techie runs Linux. As I have previously stated, I tinker, and Linux offers me that chance. I get to play with raid in granular detail. Layered on that is LVM, which virtualises storage. You don’t have to get that complex, but it’s the closest I can come to simulating a SAN in my own home, with resilient, abstracted hardware hidden from the VM.

I make and break bridges and virtual networks on the fly. I’ve got three gigabit Ethernet ports, and segregate them by function – to the point where ALL VM traffic is on a dedicated port, so that if a VM talks to the host it pops out through the physical switch and back in on the front interface.

Here’s the key thing: While all of this is probable available from other vendors, and some parts may even be free, I’m in control. Sure documentation can be sketchy, and it requires a lot more basic knowledge of networking, storage and hardware architectures than other solutions, I’m hardly take my car to a garage to be serviced if I wanted to be a mechanic, I’d get dirty. Break things definitely, but that’s a great learning tool.

My power laptop used to break whenever I entered standby, and I mean trash the root file system (ext3 to boot, supposedly bullet-proof) so I filed a bug. As it turns out, it only happened when I was running HVM, so I applied a patch published 30 days before to my kernel. Voila!

Now this is a big thing! Without being a paid customer of Redhat’s, I got a problem resolved quickly and comfortably. I must stress here that Fedora is BETA software, permanently, and I expect it to break, but frankly for my needs it’s just fine, and when I can get things like this resolved, even better.

Sure, it’s painful, but it’s all under my control. I’ve scripted Windows 2003 deployments with snapshots that boot a new instance, sysprepped and ready to go, in 7 minutes. IPTables lets me simulate complex networks, firewalls and even lossy links.

I remain firmly impressed!

So why am I so keen on Linux?

I've been trying to dissect my career, certifications and interests, and how the three don't really match up.

You see, I'm a Microsoft certified Professional (and now Technology Specialist, soon to be IT Professional, exams, exams, exams). I have been since 1997, when I first sat the Windows 95 MCP exam. I didn't really know what I was doing it for, except that employers wanted to see that sort of thing. I'd been using '95 since a few months after the release, when I went out and bought a copy.

Since then, my career has moved from technical support on desktops, to the point now where I'm quite comfortable giving advice to large organisations on how best to spend a million or so dollars (Euros, etc). This is not small-time stuff, I've advised corporations and governments on how to do mergers and streamlining, and how technology can support their immediate and long-term goals. I've solved interesting problems fraught with politics (but I'm nowhere near arrogant enough to deal with that effortlessly), and sometimes I get to have some fun.

It struck me on an assignment in June 2008 that I had zero allegiance. I'd been sent off to see how best a pharmaceutical company could make use of it's new acquisition, a prominent retail brand in it's sector. The big guy had Microsoft Exchange, the little guy had Notes. they kinda knew where they wanted to be after the merger, that it didn't make sense to keep two teams doing the same role on different platforms. Makes sense.

As I dug deeper, I noticed that the e-mail and collaboration part of the story was not all that important. As is the norm for Notes, a little ecosystem of apps had been developed. However, whereas most organisations tend to have a cluster of disparate Notes apps, the small guy had a plan, a comprehensive one too that made an application platform out of Notes, and a rich set of interconnected apps delivering business intelligence, dashboarding, some data mining. Wow.

So in the end, all my credentials (and by then I'd achieved five Microsoft qualifications) meant nothing, since it made no sense to throw notes out, or even relegate it to an application platform and move all e-mail to Exchange. So my recommendation was to ditch Exchange, at the very least for user e-mail.

That's kind of my point. I'm not a standard bearer, but I am a flag waver. My days of technology for tech's sake are behind me. Don't get me wrong, tech is cool and I'm always drooling over the latest gadget or software release, but when it comes to my work it has to mean something, be pointed in a particular direction, support a bigger plan.

So frankly, I'd be negligent if I just supported Microsoft's plans. Or any other platform for that matter. It disappoints me to see MS and FOSS guys going head-to-head. I enjoy using Linux for the same reason people tinker with classic cars. Sure, the new model from the dealer is likely to be more fuel-efficient, safer, better riding. But I like to get my hands dirty, and you can't do that with most modern cars.

I don't know if I've made a point here, but maybe I don't need to.