Dawson College: What Island Are You On?

I've been viewing the growing story about Ahmed Al-Khabaz, a Computer Science student at Dawson College in Montreal, Canada, who was expelled for running a security scan against their public web presence to discover if a flaw he found was resolved. I stumbled on a subsequent interview with prominent IT Security professional Chris Wysopal. I hadn't heard of him, but when I saw he was previously associated with l0pht Heavy Industries my eyes snapped open.

This guy has credentials, and I don't know of an IT professional active around 1997 to 2003 who hadn't heard of, or actually used, l0pthcrack, often to solve real-world problems. First and foremost a password auditing tool it can be used maliciously, but the so can a toaster oven. It is a piece of code art: Necessary, useful and (at the time) industry-shaking.

White Hat hacking is a tricky business. Even I've done it, against a bank no less, fully in the knowledge that I was doing something the system owners would be very unhappy about. In some cases it can get you arrested. I was was pleased with the results when my concerns were taken seriously and fixed fairly quickly. I've worked in financial services companies and know their software release process is iceberg slow so this was very reassuring. There's one thing Mr Al-Khabaz and I both know that drives thousands around the world to the same end: I'm at risk.

Dawson College is hand-wringing and special pleading: "the law ... forbids us from discussing your personal student files" is in this case weak. I am pretty sure the former student would agree to a waiver of his right to privacy to clear the air, but I have seen no mention of an offer. Fourteen out the fifteen professors convened voted for his expulsion, for doing what some professionals get paid extremely well to do (even I've been offered this job): Evaluate the security of publicly-accessible websites. I would like someone better informed than me to comment on what the implications would be for the institution if it was discovered a breach because of this flaw caused losses thanks to the personal information disclosed.

I can appreciate that the college does in fact have to abide by law, and is unwilling to get into a mudslinging match in the public forum. They have rules for ethical behaviour that may have been violated (I haven't seen them). But beyond those considerations, every one of the fourteen professors needs to answer one simple question:

Why, if these actions are so outrageous of a Computer Science graduate that it demonstrates
"behavior that is unacceptable in a computing professional" has the company whose software flaws he exposed taken it upon themselve to pay for his further education?

Academia is often seen as disconnected from reality; some lines of research beggar belief, and the same could be said of Computer Science. I've met a few graduates who arrive in the IT industry ill-prepared, full of theory of operation and design but unable to command a command-line. No matter what their actual instruction is, a critical point they need to learn is that the Internet is a hostile place. It is also a collaborative place, where FOSS abounds and Creative Commons is richly rewarding. Poking around is the norm, and if this college is telling their students that they are to accept their instruction blindly without considering real-world implications, or use those skills to explore, then they don't deserve to be associated with the term Higher Education.

They may perhaps be able to educate Code Monkeys, but thinking professionals able to design and protect systems that impact their lives? Not really.

How Important is MariaDB? Let's test the fork with butter.

MySQL has interested me for quite a long time. I first came across it in 2000 when trying to find a better way to analyse the contents of a 20,000 user Active Directory and needed more relational DB-stuff than Microsoft Access could deliver and cheaper than SQL Server (wow MSDE was terrible). I was deeply impressed (though probably because I was easily impressed back then) with the performance and cross-platform support, and ever since it's been around my life.

I currently use it for my XBMC and Logitech Media Server (SqueezeBox) media databases, as the back-end for my Gallery3 site, and other ad-hoc databases whenever I need to crunch data. Before my 64-bit processor created a new ISA that ensured a reasonably complete instruction set, it was a favourite of mine for optimising binary compiles over the stock i386 build supplied by most distros, but more for interest's sake than actually squeezing performance for any measurable benefit.

MySQL AB was of course the owner of the copyrights and code and opted for a relatively unique license, both proprietary and open. As the owners of the code, they could choose to do this, but anyone trying to make a buck out of the code was obliged to release their modifications. Now that Oracle (through their acquisition of Sun, who acquired MySQL AB) have that right, the open-source community is in a bit of a fluster. Can we trust a corporate giant with custody of the code that runs a significant fraction of the Internet's websites? The answer is slowly coming down on the side of "no".

Oracle (and others, and unsurprisingly) is being guarded about bugs and fixes. Stories of vendors forcing customers into NDAs before even admitting bugs exist, hiding bugs from other customers, and silently including fixes are common. It's face-saving. Andy Grove's "Only the Paranoid Survive" starts off with how Intel hoped to keep their Pentium FPU bug quiet while they implemented a workaround simply smacks of arrogance. While it doesn't yet seem Oracle are trying to hide any actual code and still supply source, MySQL has historically had test cases for bugs published alongside them to protect against regression and anyone can run the suite on their installation to verify code quality. Not only are they apparently now keeping some cases secret, they are also not clearly marking which code updates fix bugs they are refusing to publish.

This is not how open-source works, but I don't agree with the prevailing rationale. RedHat came into the firing line for being less than open they handled a code signing infrastructure breach, but in that instance I support the way they behaved as it was not their source they concealed, rather their own systems and controls that were embarrassingly compromised. They have shareholders, and revealing too much would have cost them. Oracle too have value invested in their products and would like to keep flaws hidden. This is not nefarious, it's capitalism.

MySQL as a product is different, no matter who owns it. It is very closely tied to the spirit of the open-source movement, being both highly regarded for performance and features, and for the competition it gives proprietary offerings. For Oracle to claim that ground back is entirely within their right, but the edge is gone. The most ardent supporters and influencers of purchasing are not happy and a slow exodus may be starting.

So Fedora and Wikipedia are both contemplating pulling out. The MariaDB fork has all the features and more, is fully open in the original spirit of the project, and is attracting attention including mine. I have no idea how easy it will be to do the fabled "drop-in replacement" every source claims is possible but I feel ethically compelled to leave MySQL in the dust. I have a server that runs my digital life and it is a conscious choice to run on open software only and it has not been easy, but as an experiment and learning tool it is invaluable.

The great thing about open-source is anybody can fork. I can clone a source and apply my changes as I like, but the moment I try to give it to anyone else (especially selling the result) I have to disclose my whole body of work. This can lead to some confusion as the early days of Linux showed, but in the end the market weeds out the under-performers and delivers better products through sheer market forces. MariaDB seems to be that winner.

I do know one thing: testing the transition is going to be a breeze: After switching from Fedora to Gentoo four months ago, I rolled the root over to BTRFS (once kernel 3.6 gave me the necessary confidence). Add a distinct IP to the NIC, snapshot, chroot, and I've got a clone of my server ready to go in about two seconds without that system-level virtualisation stuff and hideously slow LVM2 snapshots.

Rollback to base for a fresh attempt? Yep, two seconds.

Do the ends still matter?

One of the biggest conflicts of interest in my role as an infrastructure architect (I'm responsible for servers, applications, security, storage etc) is that I have a heavily vested interest in the service they provide. This may sound like a no-brainer, but often I need to find ways to get my traffic to users at the expense of someone else. I want preferential treatment if conflicts with other traffic.
Of course, it's not quite as simple as that. Pragmatically I realise I am competing with others on limited resources, whether it's rack space, hypervisor RAM or network bandwidth. One of the less-known principles underlying the Internet ethos is the End-to-end Principle. Put simply, it's this:
Complexity in any network should be implemented in the end points - the network stacks of communicating nodes and in applications - and not in the network itself. Since standards change any benefit of implementing too much intelligence in the network is quickly undermined with the need to continually match those changes in end-points, as well as legacy issues.
Essentially, the network should be as dumb as possible. Packet comes in, packet goes out, wait for next packet. This is reasonable in principle but different traffic flows require different treatments - VoIP and video streaming protocols prefer low latency and are almost always jitter sensitive, while file transfers can tolerate enormous latencies as long as they are accompanied by high bandwidth. If both types occupy the same link the risk is that the insensitive consumption of one protocol can impact the requirements of another, so QoS arrives.
This is already a necessary evil (though some may object to my use of that label in this context) since we now need to build knowledge of services into the network layer. Large IP transport installations (Internet backbones) handling gigabits of traffic per second literally don't have the time to implement protocols like this as processing adds both cost and latency. This is different from implementing different virtual circuits for different traffic types, in that it forms logic links and is a very common practice with ISPs in the final mile. As far as intelligence on network devices go, this is very low - again packets come in, figure out where packets go out.
A very interesting turn is in the development of new forms of network acceleration. Routers have long been capable of doing in-line compression of data to reduce consumption of a specific link, but this is point-to-point. If an application protocol can truly benefit from compression, this really should be done at the application protocol level (basically above Layer 3) so routers and switches can shuffle packets. A possible side-effect of link compression is to mask real versus usable bandwidth from applications (some portions of a stream may be highly compressible, others not) hindering flow-control algorithms built into TCP.
TCP sessions can be rather chatty, and some applications implement redundant techniques inside their own specification, so some network acceleration can initially make sense. Essentially, new sessions follow a well-known pattern of window sizes and other parameters, so network accelerators intercept packets and simulate the repetitive parts on each end of a link, reducing session setup time. This sounds simple, but now we enter dangerous waters. Next comes protocol caching; I request a file from a file server across the WAN and the contents are cached on my local acceleration appliance so that the user next to me can get a cached copy when her request is made. Again, sounds simple, but to prevent interception and modification the protocol implements signing, so re-generated content requires a re-generated signature. SSL acceleration similarly can be implemented using reverse proxies holding a copy of the private key for the service to extract the plaintext and look for compression and caching opportunities. I've been involved in design and deployment of many reverse proxy and SSL acceleration projects but these were explicitly part of the service.
To accelerate generic payloads the appliance needs to get heavily involved in the infrastructure, either by hosting all SSL keys and being able to masquerade as endpoints. This is where it gets very complex and risky, not only in the dissemination of privileged access (private keys and domain credentials being the most highly-prized items in any network), but also a continual catch-up game to implement these techniques on newer protocols as they are developed. There are also more intrusive techniques such as automatic downscaling of images when using mobile data to browse the web that are subtle but insidious.
Net Neutrality is the overall drive in this direction and correlates with the End-to-end Principle, treating network devices as simplistic and all traffic as equal.Smarter protocols such as (distributed) BranchCache and BitTorrent (yes, BitTorrent is a case for Net Neutrality) reduce redundancy over constrained links and better content intelligence (I explicitly convert all images in my documents to 8-bit PNGs before embedding) are far better strategies. Content distribution networks are an active participant and used by some of the largest providers (both content generators and ISPs) to reduce long-haul bandwidth and improve user responsiveness. HTTP compression is a rigorously defined standard but very sparsely used even on static pages where on-the-fly zipping is unnecessary. I could go on...
When looking into how best to transport content I prefer to let the network do what it does best and work my requirements into it than engineer a network to suit my needs unless absolutely necessary. Where my application is burdensome, I would prefer to engineer the application than throw bandwidth at the problem.
Technology marches, and bandwidth - while logically finite - seems to be keeping up rather well.

When is security "Over the Line"?

I've been trying out various security apps for Android, including location and theft tracking, secure e-mail and other device control techniques. I came across Google Apps Device Policy in the Play Store (among others). There's a handy little device management panel. I found the features a bit sparse - I could only see options to remotely wipe the device, locate it and make it ring nice and loud (I have a fairly discreet ringtone), though even this came in handy this very morning (housewarming, cocktails, phone-based DJ system, details are fuzzy towards the end of the night, great party).
A bit more digging and I found the full-blown control panel in my Apps admin page, and there all the meat is laid out - requiring the device management policies be enabled before e-mail can be subscribed to, forcing password policies and the like.
I did notice one comment that stood out in the review page on Play Store entitled "Over the Line". It's a bit of a rant, with the poster complaining about the power the device policy gives his IT department. Some education is certainly required here.
We're living in a very close approximation of Arthur C. Clarke's World of Tomorrow, but even he would probably have been surprised by the sheer power and connectedness of the devices we carry in our pockets and purses. The old adage of our phones outpowering the lunar module is somewhat behind us; they're performing megabytes of 256-bit encryption per second and rendering 3D worlds at a rate that would make older Silicon Graphics workstations blink. Yet we treat them with no more concern than a wristwatch, merely a tool for consuming information and perhaps flinging a bird from a catapult.
E-mail is sensitive. I still don't think we've fully understood the ramifications of transferring our brick-and-mortar existence fronted by a postbox to simply occupying a few square millimeters of hard drive space. Pinging an e-mail address to verify my identity when recovering a password (on services as sensitive as PayPal no less) have deep flaws, most based on the assumption that users understand account security and host the address at a secure provider. Corporate e-mail is generally even more precious, and teams of people in IT departments around the world spend their entire day protecting digital assets from the wide world of internets.
So how do casual devices interact with secure systems? A lot of banks have pared-down versions of their mobile banking websites available as apps with little risk of being vectors for attack - sensitive information is not stored on the device (and my previous post describes how easily and frequently that device's security can be engineered for flimsiness), and practices such as limiting the potential harm through daily limits.
E-mail clients tend to be more of a problem, with contents stored for long periods of time, users expecting to be able to retrieve arbitrary content outside of the retention window, creating and sending new messages without explicit authentication, and others. This is something of a nightmare scenario for organisations, large and small.
There are two outcomes from this. First, the powers provided by these policies are only invoked as necessary. I can imagine a very real case of damage claim should the department decide to wipe your device without notifying you up-front, regardless of reason. Yes, there is a lot of power there, but it only really makes sense if the company is facing imminent loss, either through the user misplacing the device or having it stolen, or worse yet if the user has malicious intent. The warning screen is there to explicitly inform you what you're signing up for and gaining your acquiescence. I remember no such introduction when handed my first Blackberry, but it doesn't grant the IT administrators any less power to remotely wipe it (including my own data that happened to be stored) when necessary.
The second though is that the device becomes a very real part of that network. Companies have policies for all kinds of things, the sense of which are not immediately apparent - sign-in at the front desk is ostensibly to ensure you are accounted for in the event of fire evacuation where security is not primary, something I didn't realise for some years when I first started working. A friend of mine with a very corporate job had a clause in her contract that she was permitted to dye her hair, but only to the range of natural hair colours. I can certainly imagine reputational harm if she arrived at a customer properly suited and booted as required, but sporting electric-blue locks. While there is no accounting for taste in corporate contracts, I would imagine this seems entirely reasonable to the vast majority of people. I suspect those who find such a policy repulsive probably wouldn't want to work there for other reasons.
A compromised device closely coupled with a corporate e-mail system can cause very real harm. Address book caching, calendar searches and assuming the identity of the original owner are powerful tools for an intruder. There is an option here.
While a company can require you wear their uniform for a job, they're far less likely to require you sew a logo onto your own clothes. You can choose to, but I doubt they would be happy if you did this in an inappropriate position as it would cause them reputational harm, and e-mail is similarly interesting.
If you truly require mobile e-mail access, perhaps your company should be paying for that service instead of free-riding your mobile contract and making you nervous about your device integrity? I know my Android device allow my to remove the enforcement agent myself, which very quickly removes the corporate e-mail from the phone as a consequence. You're in control, which fits since you're paying for it.
And if you're the only one who thinks you need it, why do you get to decide how much risk the company should accept? I know my organisation has absolutely no policy for remote wipe even though I accepted exactly that power when hooking my device to the corporate e-mail system, but it still jumped out at me.
With great power comes great responsibility.
It's not a wristwatch.

Locking myself out

I've become quite adept with Android ROMs of late. In the last month I've re-flashed my HTC Legend at least eight times, one of them a desperate measure to get back in after idiotic inattentiveness.
The after-market firmware scene is amazingly vibrant, and I'm typing this post on Cyanogenmod's CM7.2, which brings Gingerbread (Android 2.3) to my two-year old device, something the manufacturer has no intentions of doing. Just as in my previous post about digital legacy, the open-source community is doing what vendors simply won't, and surprisingly well. I'm so pleased I've even offered to upgrade a (decidedly untechnical) friend's Legend.
Doing this requires bypassing the vendor's OS locks - "rooting" - and optionally the firmware locks. By default, open ROMs leave these locks off to preserve your Newfoundland freedom. There's a problem there.
Yes, I want to continue hacking away, but I'm not comfortable leaving root accessible for my friend's device. A diagnostic mechanism called USB Debugging doesn't reset itself on reboots rendering the screen lock rather useless. The Cyanogenmod authors have stated root access will be off by default from version 9 (Android 4.0) onwards, but the Tom I tried on my device (so very close to useable) had no such measure.
Should a rooted and reflashed device be misplaced or stolen, the new owner could quite easily get in. Remote wipe and location apps can be disabled (Link2SD makes this trivial) and data compromised. As a network administrator I wouldn't let one of these devices anywhere near my messaging system, since the policy enforcement engines rely in unprivileged users to function.
I could of course re-applt the vendor recovery software to prevent the OS image being altered, and re-enable firmware controls to stop this being undermines, but this is all moot once the OS can do as it pleases.
Open-source software thrives on freedom, and I love that attitude, but ensuring a well-controlled network often works against that when viewed from certain angles. So, what path?
Security systems relying on obscurity of design have repeatedly been subverted, but how do you convince open-sourcers to build effective blockouts into their project?

Perhaps by helping them realise the real loss their moms and kids might experience when their unrelentingly open device is abused?

Blizzard, a Few Words on API Implementation

Over on battle.net, Blizzard is defending their action of banning users for using Wine to run copies of their hugely popular Sword & Sorcery game Diablo III. In Blizzard’s words:

Account Action: Account Closure
Offense: Unapproved Third Party Software
A third party program is any file or program that is used in addition to the game to gain an unfair advantage. These programs may increase movement speed or teleport heroes from one place to another beyond what is allowed by game design. It also includes any programs that obtain information from the game that is not normally available to the regular player or that transmit or modify any of the game files.

Now I happen to know a fair bit about the internals of Linux and Windows, and can say one thing for certain: Wine is not the problem.
As mentioned on Ubuntu Vibes, the Wine developers are keen to distance themselves from the issue, and fair to them it is not their fight. But Blizzard definitely has some explaining to do.
I have been very much on the side of those calling foul of Blizzard's method of providing the game state, essentially running a single-player game on their servers. While Diablo II saved locally and ran game logic entirely independent of servers, a single-player game in Diablo III is essentially a single user multiplayer instance – all state is held on their servers. This has advantages, like quickly and more reliably allowing me to invite internet users to join my game, but that’s about the only benefit I can see.
It would appear that Blizzard are trying to clarify that they don’t ban based on OS (or API implementation), and its a subtle point. Whether cheating is actually happening in this particular community as they allege is not quite evident, but since Blizzard don’t reveal the exploit a user is supposed to be using there is no recourse for refund or rectification. I’ll leave that point alone, save to say that I would prefer other users don't cheat, but that it’s irrelevant if I’m in a single-player game. Unfortunately, the way the game is designed there is no such thing.
There’s been ranting about other problems, but one I’m most interested in is legacy. The reason almost any software has a minimum specification for operating system is a guarantee that they APIs they need will be present. Wine is an undertaking to replicate the APIs in Windows for use on other platforms, prominently Linux. Backwards compatibility is the bane of Microsoft’s existence, but there’s no option there really.
The big issue here is the digital legacy. I used to play with my father’s toys as a kid, and he with mine – that is until I started gaming. I have a partner who went barking mad as the release for Diablo III approached simply because she was excited to relive the enormous amount of time she wasted in college playing Diablo II. The problem is even graver than if Steam went dark for good where i simply couldn’t install my games. If Blizzard pulls the plug I cant launch games I’ve already installed, ever! And frankly, twenty years from now I just might like a shot of nostalgia.
But the hardware/operating system of the future might not support it you say? I’d say you’re dead wrong! Steam released the classic X-COM games packaged running in DOSBox, showing that there are solutions to anciently legacy games on modern platforms, and open-source no less.
So just what is Blizzard’s problem? The fact that Wine presents a functional implementation of the Win32 API is of little relevance. there’s no clear statement that non-Microsoft/Apple platforms are enough to get you kicked, far from it they leave it up to users to employ best-efforts. Blizzard may very well claim that they only support native implementations, but WOW64 is really an implementation of the Win32 APIs, when the underlying system is in fact 64-bit.
So, we have both closed-source and open-source solutions to the problem of running legacy code, both apparently providing a reasonably good experience (in fairness I think the Windows implementation is excellent, no idea on quality using Wine). Twenty years from now I have a fairly good feeling there will be legacy emulators (Windows 7-in-DOSBox, Wine for Linux 7 or something as-yet undreamed of), but I am almost certain Blizzard won’t be keeping up a service for Diablo III, even if I just want to show my kids how their dad used to kill demons fell more in love with their mom.

Zero-effort Theft Tracking

This little nugget (nhinkle on SuperUser blog) popped up on Slashdot recently, describing just how Windows 7 (and I presume Windows Vista) knows whether your network connection is an unrestricted Internet conenction, has no connectivity beyond the local LAN or are behind a hotspot authorisation page. Of course, the state of your network connection may be more complex than that (proxy server requiring authorisation, badly configured routing, or wrongly reporting a working connection that drops your VPN traffic), but it's mostly quite good.

Essentially, the first thing the Network Location Awareness service does is check if it can retrieve a url specified in the registry, and if the contents of that file match a set string. If successful, NLA assumes you have full connectivity. If it fails, a DNS lookup similarly needs to match a specified IP, in which case it assumes you need to authenticate (e.g. you're on a hotspot).

The privacy concern is that your system is checking-in with a server controlled by Microsoft every time it attaches to a network, and periodically after that. Obviously, nobody knows how or if Microsoft is using any logs of these check-ins, but it opens an intriguing possibility now that instructions in the article reveal how to customise this behaviour. Essentially, you can instruct NLA to check-in with any URL you like.

As long as the file is retrieved with a 200 OK result and the string matches (apparently it's even sensitive to an extra CR/LF, so make sure it's generated correctly), everything is hunky-dory. But here's another possibility...

The URL retrieved doesn't have to be a simple text file. The URL could actually query a script (PHP, ASP(X) etc), as long as it returns a specified result. Using Group Policy, an administrator can update the URL to include a client-specific parameter (say an asset code, MAC address, serial number, username etc) in the form of http://some.server.com/NLACheckin.php?Serial=1234. The server script can then note the source IP/subnet of the request in a database, and presto! Real-time location tracking. Group Policies and/or client-side scripts make this rather trivial.

The great part of this is that it doesn't require any specific check-in, theft tracking or other location software to be installed, and the service cannot be disabled by an ordinary user. It doesn't look like there's any action that can be launched on a specific response in the client end (say shuitting down the device), but it offers an extremely simple way to find stolen computers. It's no defense against a wipe-and-reinstall, but for the gain it's well worth the effort.