WYCRMS Part 3. Console Applications, Java, Batch Files and Other Red Herrings

In 1997, a HP 9000 engineer wouldn't blink telling about a server that had been running continuously for over five years.I found this remarkable at the time, and couldn't imagine a Windows server lasting that long. I have moved on, and frankly expect my Windows servers to survive that long today. Very few share this position, and I'm trying to find out why it's so lonely on this side of the fence.

3. Console Applications, Java, Batch Files and Other Red Herrings

Not to start this off on a downer, but I need to let you know I'll be insulting a few types of people in this post. I also need to make one things extra-clear: I hate Java.

The idea is great: write code once and run it on any platform without recompilation. Apart from the hideously long time it took for Java to come to 64-bit Linux, support is pretty good too. Sun (then Oracle) have been responsive in fixing bugs, but being a platform it is somewhat more difficult to roll out updates so huge amounts of obsolete JRE deployments are available for nefarious types and buggy software to run amuck. The reason I hate it is twofold: it allows developers to be lazy about memory management and rely on automatic garbage collection, and the fact that almost every application I've come across (except Apache Tomcat-based WARs) explicitly constrain support to certain platforms. This is not what I was promised.

When someone talks about "Windows running out of handles", "memory leaks", "stale buffers" or any number of technical-sounding pseudo-buzzphrases they almost always are trying to describe a software malfunction that appears as a Windows failure, or are simply too lazy to investigate and realise it almost invariably is caused by lazy programming. Java does this, but I don't blame Java, I blame Java programmers. The opinion is rife that Windows get less stable the longer Java applications run and that reboots are a Good Thing™. If someone genuinely believes that server stability can be impacted by poor software, but not report it to the vendor, I will inform that person he/she is lazy.

As I mentioned in Part 1, Windows engineers seem to scale their experience of Windows at home to their professional roles, and I've seen developers do the same. Windows doesn't do pipes very well, or they are language- or IDE-specific Outputting to the Event Log is slightly arcane and in fact requires compilation of a DLL to make most output meaningful. It's rarely used outside Microsoft themselves. So developers rely on consoles for display of meaningful output.

These consoles then become part of the deployment practice, perhaps wrapped in a batch file. If your program relies on a console window (and therefore a logged-in, interactive user session) or worse requires me to edit a batch file to apply configuration changes (as opposed to, say, a settings file parsed by said batch file), your software is nowhere near mature enough to be deployed on a system I would expect people to depend on. As a programmer, I question your maturity too.

It's people and organisations like that who typically have one response to issues that crop up: install the latest Service Pack, maybe the latest Windows Updates too (that fixes everything, right?), and if all else fails upgrade to the Latest Version of our software - don't worry that it's got a slew of new features that likely have bugs too, they'll be fixed in the next Latest Version. Rinse, repeat.

As a Windows Engineer, your job is to defend the platform from all attackers. That's not just bad folks out there trying to steal credit card numbers and use you as a spam bot, it's also bad-faith actors trying to deflect the blame from their own inadequacy. It's application owners prepared to throw your platform under the bus to hide their poor procurement and evaluation standards. It's users who saw a benefit in a reboot once and think it's a panacea.

It is in everyone's interest to call people out when they fail to deal with this stuff properly, or you'll quickly find yourself supporting a collection of workarounds instead of a server platform.

Previous: Part 2. Windows Just Isn't That Stable

Next: Part 4. Windows Updates and File Locking

WYCRMS Part 1: But I Have to Reboot My Own Windows System All the Time!

In 1997, a HP 9000 engineer wouldn't blink telling about a server that had been running continuously for over five years.I found this remarkable at the time, and couldn't imagine a Windows server lasting that long. I have moved on, and frankly expect my Windows servers to survive that long today. Very few share this position, and I'm trying to find out why it's so lonely on this side of the fence.

1. But I Have to Reboot My Own Windows System all The Time!

I've mentioned before how Windows makes you lazy. One of the great things about Microsoft Windows as a platform is that software developed on a $500 workstation can be installed on a $50,000 server and probably work without problems. Of course, getting your home-brew software to scale is a different matter, but you get the idea: One platform, different size.

Almost every Windows engineer cuts their teeth on Windows at home, and this informs their experience and expectations of the platform. Like everyone I get tired of the bogging down after a few days/weeks/months uptime and reboot just to clear things up, but that's my fault and not Windows.

I'm lazy.

Typically, I'm running browsers, office suites, anti-virus, any number of games, and install new stuff roughly once a fortnight. Flash, Java and Windows Update are constantly pestering me to reboot after updates. I've even been the one to reinstall completely after a year to see the wonder of a zippy start-up and responsive GUI, only to have it slowly crawl as I add functionality (including those games). Happily, my Windows 7 installation has lasted two years by now with no significant falloff in responsiveness, so that's getting much better, and I only power down/reboot of my own volition when I'm fitting lights and need mains power off - even then it's more likely to be a hibernate.

Servers are not workstations. Any good enterprise has controls for how changes are made to IT systems, and even simple patching requires testing and approved windows to take the system down and update it. In my experience a server will undergo a major overhaul at most twice in its' operational lifetime, and organisations with exceptional controls have zero - new version? New server!

A good server (and I think of Windows Server 2003+ as good servers) will run for decades given quality power and no moving parts. Of course hardware fails, but Microsoft have put in man-decades to get Windows to handle routine changes without downtime. I remember Windows NT 4.0 needing a reboot for an additional IP address. Modern versions of Windows can hot-plug an entire NIC (physically) without a blink, though admittedly I've never actually encountered anyone who uses the facility.

If an engineer merely mentions that, in their experience, Windows needs rebooting I question their experience. I mean it: I question their experience!

Windows is solid, and I can recall only one confirmed bug where Windows will fail (actually, begin to fail, an outage is not a certainy) for the simple factor of running continuously for a given time. When someone speaks of a memory leak that has caused Windows to run out of (insert wooly term here), again I question their experience and the quality of the software/vendor driver code. I've stopped blaming Microsoft.

When I run my applications on Windows Server and, more importantly, when I am paying someone to manage those systems for me, I expect them to have faith in their products and promise me server availability. Rebooting breaks availability.

Previous: Why You Can't Reboot my Server

Next: Part 2. Windows Just Isn't That Stable

Streetview and WiFi - Courts Need Some Education

I'm hanging my head in my palm in a manner not unlike the Jean-Luc Picard meme today, after reading a decision by a court in the United States. EFF has a great article summarising the effects, but I'd like to expand and go into the cause too.

So Google drove around with an antenna on their Street View cars for a few years, sniffing for wireless networks. This is very useful if you'd like to know your location but can't get a GPS signal, especially on devices with lower quality antennas or a location with poor sky visibility. As long as you have a data connection, you send the local Access Points off to Google's servers, and they will look up the location and feed it back to you. Simple, right?

Well, sniffing for access points (AP) is ridiculously simple. Your phone does that every time you look at a list of networks in your location, by looking for a special network frame called a beacon, which a regular access point send out roughly ten times a second. It's crucial to how WiFi works. It uses the same frame type as normal traffic, so even without a beacon you can still see at least the presence of traffic, and the MAC address associated with the AP. If the network is unencrypted, your WiFi network card automatically accepts those traffic frames too, then has it discarded by your kernel because it is not destined for your computer/smartphone/tablet etc ("your device").

I've done sniffing myself, in a practise known as "war driving". It sounds ominous, but it's also a very interesting excercise, for which I purchased a specific Atheros WiFi NIC, thanks to their products having excellent Linux support. Hook up a GPS receiver, and just go places. The software figures out where you are, looks at the list of APs nearby and pins them to a map. The problem here is that simply enabling your card to listen for APs does cause your system to store those traffic frames, since they are usefuol for determining the IP range in use on that network. Note I'm not trying to use those networks, I'm just interested in seeing how they're being used.

Now comes the interesting part: The court decided these signals being intercepted were not "radio communications" (despite being carried by photons) for the purposes of a legal interpretation, not "readily accessible to the general public" without "sophisticated hardware and software", and finally "most of the general public lacks the expertise to intercept and decode payload data transmitted over a Wi-Fi network".

On each point:

• It's radio. Learn physics. The lower court's opinion that the law covered "predominantly auditory broadcast" is an inference form the court; 18 USC § 2511 mentions audio only for satellite transmission, and only then to describe an audio channel used as a carrier for digital communications therein. If I speak ones and zeros into my walkie-talkie, does this suddenly become digital communications? Your meaning is divorced from reality, and I think it suits an agenda instead of fact.
• If I walk down a street, glancing into shops as I go, and see a person in one shop/office/etc handing a big photo with a red X through it to another person, with person in photo turning up dead and either of the two persons implicated at trial, I can't be prohibited for testifying because I got information not "readily accessible to the general public"(i.e. they were not standing on a street in plain view); a police officer seeing the same thing under the same circumstances is not prohibited from using the information for lack of a warrant or probable cause - it still doesn't affect the fact that photons bounced off the photo and got interpreted by "specialised hardware " (eyes) and "specialised software" (brain). Nobody's privacy is invaded, but information was blasted into the street nonetheless.
• WiFi NIC cost: $10 (shipping costs vary), down to as low as one. These devices are sophisticated, don't get me wrong, but then most computers are mind-numbingly powerful today. Linux kernel cost: nothing (download costs may vary, but is unlikely to make you hit your ISP's download cap),  easily installed in an array of distributions. Laptop cost: Variable, but if you've got one lying around you already have your solution.
• Interception is easy stuff. Decoding is easy stuff, and your device "decodes" it as part of its' primary function.
• A further point the court asserted is that regular (AM/FM) radio communications can be received miles away, versus WiFi that "fail to travel far beyond the walls" of a location. Again, physics. Oh, and define "far" - my balcony is less than 15 meters from my AP and my signal comes and goes, while the street is 30 metres away (in the opposite direction) and I still get an association (but not much throughput) fairly reliably. I despise vagueness in court proceedings.

The fact is, sophisticated software is required to not receive the payload. If somebody decides to configure and use an unencrypted access point and I happen to walk past with my phone doing the searching (I left the WiFi on when I left my home), or researching which models of router or service provider is prevalent on the street, or finally if the NSA, FBI, federal or local officials are parked in a van across the road, simply turning on the function puts the traffic in RAM. Even if it destroyed a microsecond later, under this ruling I (and they) have broken the law. The bar for warrant-required searches just shot up.

I'm used to seeing courts being out of touch with reality, especially in computing cases, but this is beyond unreasonable. I have no doubt Google had zero intention of capturing user detail such as e-mails, usernames or passwords (why would they, they run an e-mail service?), and are now being prosecuted for users' inability to secure their own networks.

But mostly: Physics!

When is security "Over the Line"?

I've been trying out various security apps for Android, including location and theft tracking, secure e-mail and other device control techniques. I came across Google Apps Device Policy in the Play Store (among others). There's a handy little device management panel. I found the features a bit sparse - I could only see options to remotely wipe the device, locate it and make it ring nice and loud (I have a fairly discreet ringtone), though even this came in handy this very morning (housewarming, cocktails, phone-based DJ system, details are fuzzy towards the end of the night, great party).
A bit more digging and I found the full-blown control panel in my Apps admin page, and there all the meat is laid out - requiring the device management policies be enabled before e-mail can be subscribed to, forcing password policies and the like.
I did notice one comment that stood out in the review page on Play Store entitled "Over the Line". It's a bit of a rant, with the poster complaining about the power the device policy gives his IT department. Some education is certainly required here.
We're living in a very close approximation of Arthur C. Clarke's World of Tomorrow, but even he would probably have been surprised by the sheer power and connectedness of the devices we carry in our pockets and purses. The old adage of our phones outpowering the lunar module is somewhat behind us; they're performing megabytes of 256-bit encryption per second and rendering 3D worlds at a rate that would make older Silicon Graphics workstations blink. Yet we treat them with no more concern than a wristwatch, merely a tool for consuming information and perhaps flinging a bird from a catapult.
E-mail is sensitive. I still don't think we've fully understood the ramifications of transferring our brick-and-mortar existence fronted by a postbox to simply occupying a few square millimeters of hard drive space. Pinging an e-mail address to verify my identity when recovering a password (on services as sensitive as PayPal no less) have deep flaws, most based on the assumption that users understand account security and host the address at a secure provider. Corporate e-mail is generally even more precious, and teams of people in IT departments around the world spend their entire day protecting digital assets from the wide world of internets.
So how do casual devices interact with secure systems? A lot of banks have pared-down versions of their mobile banking websites available as apps with little risk of being vectors for attack - sensitive information is not stored on the device (and my previous post describes how easily and frequently that device's security can be engineered for flimsiness), and practices such as limiting the potential harm through daily limits.
E-mail clients tend to be more of a problem, with contents stored for long periods of time, users expecting to be able to retrieve arbitrary content outside of the retention window, creating and sending new messages without explicit authentication, and others. This is something of a nightmare scenario for organisations, large and small.
There are two outcomes from this. First, the powers provided by these policies are only invoked as necessary. I can imagine a very real case of damage claim should the department decide to wipe your device without notifying you up-front, regardless of reason. Yes, there is a lot of power there, but it only really makes sense if the company is facing imminent loss, either through the user misplacing the device or having it stolen, or worse yet if the user has malicious intent. The warning screen is there to explicitly inform you what you're signing up for and gaining your acquiescence. I remember no such introduction when handed my first Blackberry, but it doesn't grant the IT administrators any less power to remotely wipe it (including my own data that happened to be stored) when necessary.
The second though is that the device becomes a very real part of that network. Companies have policies for all kinds of things, the sense of which are not immediately apparent - sign-in at the front desk is ostensibly to ensure you are accounted for in the event of fire evacuation where security is not primary, something I didn't realise for some years when I first started working. A friend of mine with a very corporate job had a clause in her contract that she was permitted to dye her hair, but only to the range of natural hair colours. I can certainly imagine reputational harm if she arrived at a customer properly suited and booted as required, but sporting electric-blue locks. While there is no accounting for taste in corporate contracts, I would imagine this seems entirely reasonable to the vast majority of people. I suspect those who find such a policy repulsive probably wouldn't want to work there for other reasons.
A compromised device closely coupled with a corporate e-mail system can cause very real harm. Address book caching, calendar searches and assuming the identity of the original owner are powerful tools for an intruder. There is an option here.
While a company can require you wear their uniform for a job, they're far less likely to require you sew a logo onto your own clothes. You can choose to, but I doubt they would be happy if you did this in an inappropriate position as it would cause them reputational harm, and e-mail is similarly interesting.
If you truly require mobile e-mail access, perhaps your company should be paying for that service instead of free-riding your mobile contract and making you nervous about your device integrity? I know my Android device allow my to remove the enforcement agent myself, which very quickly removes the corporate e-mail from the phone as a consequence. You're in control, which fits since you're paying for it.
And if you're the only one who thinks you need it, why do you get to decide how much risk the company should accept? I know my organisation has absolutely no policy for remote wipe even though I accepted exactly that power when hooking my device to the corporate e-mail system, but it still jumped out at me.
With great power comes great responsibility.
It's not a wristwatch.

Locking myself out

I've become quite adept with Android ROMs of late. In the last month I've re-flashed my HTC Legend at least eight times, one of them a desperate measure to get back in after idiotic inattentiveness.
The after-market firmware scene is amazingly vibrant, and I'm typing this post on Cyanogenmod's CM7.2, which brings Gingerbread (Android 2.3) to my two-year old device, something the manufacturer has no intentions of doing. Just as in my previous post about digital legacy, the open-source community is doing what vendors simply won't, and surprisingly well. I'm so pleased I've even offered to upgrade a (decidedly untechnical) friend's Legend.
Doing this requires bypassing the vendor's OS locks - "rooting" - and optionally the firmware locks. By default, open ROMs leave these locks off to preserve your Newfoundland freedom. There's a problem there.
Yes, I want to continue hacking away, but I'm not comfortable leaving root accessible for my friend's device. A diagnostic mechanism called USB Debugging doesn't reset itself on reboots rendering the screen lock rather useless. The Cyanogenmod authors have stated root access will be off by default from version 9 (Android 4.0) onwards, but the Tom I tried on my device (so very close to useable) had no such measure.
Should a rooted and reflashed device be misplaced or stolen, the new owner could quite easily get in. Remote wipe and location apps can be disabled (Link2SD makes this trivial) and data compromised. As a network administrator I wouldn't let one of these devices anywhere near my messaging system, since the policy enforcement engines rely in unprivileged users to function.
I could of course re-applt the vendor recovery software to prevent the OS image being altered, and re-enable firmware controls to stop this being undermines, but this is all moot once the OS can do as it pleases.
Open-source software thrives on freedom, and I love that attitude, but ensuring a well-controlled network often works against that when viewed from certain angles. So, what path?
Security systems relying on obscurity of design have repeatedly been subverted, but how do you convince open-sourcers to build effective blockouts into their project?

Perhaps by helping them realise the real loss their moms and kids might experience when their unrelentingly open device is abused?

What is a Read Receipt?

I saw a posting on GMail's messaging forums today about their apparent lack of the facility to have a Read Receipt generated. I felt compelled to reply, and I thought I'd post it here too as it's something not quite as obvious as it might first seem. Google is moving more and more into the enterprise, and this is a standard feature that is just plain missing, so obviously it's noticed. To me, it's more of a social than technical challenge...

A Read Receipt is quite the misnomer: the best you can say is that the message was shown on a screen. Which screen though? GMail is accessible through  the webmail, POP3, IMAP, mobile devices (GMail native client, Exchange Sync), mobile web... what exactly are you trying to prove happened?

I've found Read Receipts to be near useless, while Delivery Receipts are generally poorly implemented and probably achieve the same thing: some systems will say that they've taken receipt of the message into the system, but what you actually want to know is that it has successfully arrived in the user's mailbox. Signing for a courier delivery is exactly that, but you'll never know if the recipient ever opened the package.

If you're so keen to know whether the person opened the mail, ask them to confirm receipt manually. Having a computer tell you that you read something is at best trivial and at worst misleading. For a one-line e-mail,this facility may be useful and mostly true, you're likely to have absorbed it. A 1000-word essay, what you really want to know is not just that it was displayed on some screen (and reading that much on a mobile is pointless), but rather that the end-user absorbed everything. Only the user can attest to that.

On principle, keep read receipts out of GMail (or other providers, even Exchange!), and these problems go away. Just ask for a reply that your message is received, understood and/or acted upon. Much more useful.