WYCRMS Part 3. Console Applications, Java, Batch Files and Other Red Herrings

In 1997, a HP 9000 engineer wouldn't blink telling about a server that had been running continuously for over five years.I found this remarkable at the time, and couldn't imagine a Windows server lasting that long. I have moved on, and frankly expect my Windows servers to survive that long today. Very few share this position, and I'm trying to find out why it's so lonely on this side of the fence.

3. Console Applications, Java, Batch Files and Other Red Herrings

Not to start this off on a downer, but I need to let you know I'll be insulting a few types of people in this post. I also need to make one things extra-clear: I hate Java.

The idea is great: write code once and run it on any platform without recompilation. Apart from the hideously long time it took for Java to come to 64-bit Linux, support is pretty good too. Sun (then Oracle) have been responsive in fixing bugs, but being a platform it is somewhat more difficult to roll out updates so huge amounts of obsolete JRE deployments are available for nefarious types and buggy software to run amuck. The reason I hate it is twofold: it allows developers to be lazy about memory management and rely on automatic garbage collection, and the fact that almost every application I've come across (except Apache Tomcat-based WARs) explicitly constrain support to certain platforms. This is not what I was promised.

When someone talks about "Windows running out of handles", "memory leaks", "stale buffers" or any number of technical-sounding pseudo-buzzphrases they almost always are trying to describe a software malfunction that appears as a Windows failure, or are simply too lazy to investigate and realise it almost invariably is caused by lazy programming. Java does this, but I don't blame Java, I blame Java programmers. The opinion is rife that Windows get less stable the longer Java applications run and that reboots are a Good Thing™. If someone genuinely believes that server stability can be impacted by poor software, but not report it to the vendor, I will inform that person he/she is lazy.

As I mentioned in Part 1, Windows engineers seem to scale their experience of Windows at home to their professional roles, and I've seen developers do the same. Windows doesn't do pipes very well, or they are language- or IDE-specific Outputting to the Event Log is slightly arcane and in fact requires compilation of a DLL to make most output meaningful. It's rarely used outside Microsoft themselves. So developers rely on consoles for display of meaningful output.

These consoles then become part of the deployment practice, perhaps wrapped in a batch file. If your program relies on a console window (and therefore a logged-in, interactive user session) or worse requires me to edit a batch file to apply configuration changes (as opposed to, say, a settings file parsed by said batch file), your software is nowhere near mature enough to be deployed on a system I would expect people to depend on. As a programmer, I question your maturity too.

It's people and organisations like that who typically have one response to issues that crop up: install the latest Service Pack, maybe the latest Windows Updates too (that fixes everything, right?), and if all else fails upgrade to the Latest Version of our software - don't worry that it's got a slew of new features that likely have bugs too, they'll be fixed in the next Latest Version. Rinse, repeat.

As a Windows Engineer, your job is to defend the platform from all attackers. That's not just bad folks out there trying to steal credit card numbers and use you as a spam bot, it's also bad-faith actors trying to deflect the blame from their own inadequacy. It's application owners prepared to throw your platform under the bus to hide their poor procurement and evaluation standards. It's users who saw a benefit in a reboot once and think it's a panacea.

It is in everyone's interest to call people out when they fail to deal with this stuff properly, or you'll quickly find yourself supporting a collection of workarounds instead of a server platform.

Previous: Part 2. Windows Just Isn't That Stable

Next: Part 4. Windows Updates and File Locking

WYCRMS Part 2. Windows Just Isn't That Stable

In 1997, a HP 9000 engineer wouldn't blink telling about a server that had been running continuously for over five years.I found this remarkable at the time, and couldn't imagine a Windows server lasting that long. I have moved on, and frankly expect my Windows servers to survive that long today. Very few share this position, and I'm trying to find out why it's so lonely on this side of the fence.

2. Windows Just Isn't That Stable

Ah BlueScreen of Death, how I've missed you. Actually, I haven't, since finding out what caused them was a nightmare, and recovering without a remote console solution is not conducive to a predictable social life (or sleep schedule). That said, they were so common we even had joke screen savers mimicking them for our own geekish amusement. Since Microsoft acquired Sysinternals they're even available to download directly from Microsoft. Imagine your in-car entertainment system being configured to show you fake warnings of a failed brake line, or a cracked cylinder head. "Would you like the free video package of Ford vehicles endangering passengers' lives with your new Focus sir?". IT people are weird.

I've analysed my Windows 7 x64 installation, and in the last three years I've had six bluescreens. Once was my graphics card (pretty unique), all the others were my Bluetooth headphones putting my cheapo-Bluetooth dongle in a spin. I blame the dongle, not Windows.

OK, that's not fair to the dongle maker: I blame Windows, but only the Bluetooth stack since it's never been something I expect Windows to do well - multiple dongle-headphone combinations have yet to produce a pleasant experience (three dongles, two headphone models). The network card, storage stack, print drivers, memory management, process scheduler (NUMA-aware these days apparently): These all work so well I haven't notice them doing their job, and I am very familiar with what a complex job they have.

I expect roughly once a month to see a BSoD on public transport, or at stations, or many airports, or billboards. The layout of the BSoD has changed over the years, with each version of Windows getting a little tweak so that you can spot the version even if the error itself is gibberish, and I conclude from viewing these blue non-advertisements: These systems tend to be A) old, B) written in languages and coding styles that aren't that good, and C) interface with devices with terrible drivers.

This is not typical of modern Windows servers.

I would never dream of subjecting a server to the amount of change my hard-working personal workstation endures. AMD updates my video drivers multiple times a year, I attach and detach USB/phone/iSCSI devices more often than I refill my car's tank, and run code from pretty much anywhere as long as it promises me utility or entertainment. A server is different, running things I trust to go on processing without attendance, cleaning up after itself, and basically staying up. If I do make changes, it's controlled, tested and left the hell alone.

Windows Server is solid, and every iteration gets more solid. It's expanding to 64-bit spaces, handling multipath-iSCSI with ease, more cores than I have fingers in byzantine NUMA layouts, hosting server instances in their own right with Hyper-V and pushing gigabytes around through network cards and storage interfaces, crunching data and most importantly providing services.

Yet the very people who spend time and money proving they are skilled in designing and administering these systems so that they can adorn their signatures and office receptions with impressive Microsoft-approved decals are the first to tell you not to trust a given server (without even knowing the workload or configuration) to remain available. They express surprise and concern on viewing a server continuously running for over a year.

I'm surprised and, yes, concerned that they react this way. Isn't this what your sales folks promised me in the first place?

Previous: Part 1: But I Have to Reboot My Own Windows System All the Time!

Next: Part 3. Console Applications, Java, Batch Files and Other Red Herrings

WYCRMS Part 1: But I Have to Reboot My Own Windows System All the Time!

In 1997, a HP 9000 engineer wouldn't blink telling about a server that had been running continuously for over five years.I found this remarkable at the time, and couldn't imagine a Windows server lasting that long. I have moved on, and frankly expect my Windows servers to survive that long today. Very few share this position, and I'm trying to find out why it's so lonely on this side of the fence.

1. But I Have to Reboot My Own Windows System all The Time!

I've mentioned before how Windows makes you lazy. One of the great things about Microsoft Windows as a platform is that software developed on a $500 workstation can be installed on a $50,000 server and probably work without problems. Of course, getting your home-brew software to scale is a different matter, but you get the idea: One platform, different size.

Almost every Windows engineer cuts their teeth on Windows at home, and this informs their experience and expectations of the platform. Like everyone I get tired of the bogging down after a few days/weeks/months uptime and reboot just to clear things up, but that's my fault and not Windows.

I'm lazy.

Typically, I'm running browsers, office suites, anti-virus, any number of games, and install new stuff roughly once a fortnight. Flash, Java and Windows Update are constantly pestering me to reboot after updates. I've even been the one to reinstall completely after a year to see the wonder of a zippy start-up and responsive GUI, only to have it slowly crawl as I add functionality (including those games). Happily, my Windows 7 installation has lasted two years by now with no significant falloff in responsiveness, so that's getting much better, and I only power down/reboot of my own volition when I'm fitting lights and need mains power off - even then it's more likely to be a hibernate.

Servers are not workstations. Any good enterprise has controls for how changes are made to IT systems, and even simple patching requires testing and approved windows to take the system down and update it. In my experience a server will undergo a major overhaul at most twice in its' operational lifetime, and organisations with exceptional controls have zero - new version? New server!

A good server (and I think of Windows Server 2003+ as good servers) will run for decades given quality power and no moving parts. Of course hardware fails, but Microsoft have put in man-decades to get Windows to handle routine changes without downtime. I remember Windows NT 4.0 needing a reboot for an additional IP address. Modern versions of Windows can hot-plug an entire NIC (physically) without a blink, though admittedly I've never actually encountered anyone who uses the facility.

If an engineer merely mentions that, in their experience, Windows needs rebooting I question their experience. I mean it: I question their experience!

Windows is solid, and I can recall only one confirmed bug where Windows will fail (actually, begin to fail, an outage is not a certainy) for the simple factor of running continuously for a given time. When someone speaks of a memory leak that has caused Windows to run out of (insert wooly term here), again I question their experience and the quality of the software/vendor driver code. I've stopped blaming Microsoft.

When I run my applications on Windows Server and, more importantly, when I am paying someone to manage those systems for me, I expect them to have faith in their products and promise me server availability. Rebooting breaks availability.

Previous: Why You Can't Reboot my Server

Next: Part 2. Windows Just Isn't That Stable

Zero-effort Theft Tracking

This little nugget (nhinkle on SuperUser blog) popped up on Slashdot recently, describing just how Windows 7 (and I presume Windows Vista) knows whether your network connection is an unrestricted Internet conenction, has no connectivity beyond the local LAN or are behind a hotspot authorisation page. Of course, the state of your network connection may be more complex than that (proxy server requiring authorisation, badly configured routing, or wrongly reporting a working connection that drops your VPN traffic), but it's mostly quite good.

Essentially, the first thing the Network Location Awareness service does is check if it can retrieve a url specified in the registry, and if the contents of that file match a set string. If successful, NLA assumes you have full connectivity. If it fails, a DNS lookup similarly needs to match a specified IP, in which case it assumes you need to authenticate (e.g. you're on a hotspot).

The privacy concern is that your system is checking-in with a server controlled by Microsoft every time it attaches to a network, and periodically after that. Obviously, nobody knows how or if Microsoft is using any logs of these check-ins, but it opens an intriguing possibility now that instructions in the article reveal how to customise this behaviour. Essentially, you can instruct NLA to check-in with any URL you like.

As long as the file is retrieved with a 200 OK result and the string matches (apparently it's even sensitive to an extra CR/LF, so make sure it's generated correctly), everything is hunky-dory. But here's another possibility...

The URL retrieved doesn't have to be a simple text file. The URL could actually query a script (PHP, ASP(X) etc), as long as it returns a specified result. Using Group Policy, an administrator can update the URL to include a client-specific parameter (say an asset code, MAC address, serial number, username etc) in the form of http://some.server.com/NLACheckin.php?Serial=1234. The server script can then note the source IP/subnet of the request in a database, and presto! Real-time location tracking. Group Policies and/or client-side scripts make this rather trivial.

The great part of this is that it doesn't require any specific check-in, theft tracking or other location software to be installed, and the service cannot be disabled by an ordinary user. It doesn't look like there's any action that can be launched on a specific response in the client end (say shuitting down the device), but it offers an extremely simple way to find stolen computers. It's no defense against a wipe-and-reinstall, but for the gain it's well worth the effort.

 

Are YOUR hard drives noisy enough?

A conversation with a colleague presented an interesting question: Which hard drives are the quietest?

Now there are excellent resources online like Silent PC Review and QuietPC for finding the right components to build Home Theatre PCs. But this got me thinking, are my hard drives too quiet?

A lot of parameters are accessible on the firmware of modern hard drives, including readahead optimisations, cache policies, as well as monitoring options for errors, failures, temperature and even how many times the drive has been switched on. Linux distributions provide the smartctl utility for retrieving the monitoring variables, and the hdparm tool for setting all kinds of parameters, one of which is quite interesting...

The "-M" option sets the Automatic Acoustic Management level, which in most disks is one of OFF, QUIET and FAST, corresponding to integer values of 0, 128 and 254 respectively. Now the system I'm typing this on is my laptop, and I don't want it clunking away while I work (well, apart from the keypresses required to type this article). So, I set it to QUIET (128).

But, I have a server with four disks, two 750GB and two 1.5TB, running my server applications (file shares, telephony, media streaming etc), and crucially my virtualised lab which tend to be disk-heavy, especiall at high concurrency when each OS instance thinks it has exclusive control over volumes and optimises access accordingly. This server used to be located in my office alongside my desk, and yes it got a bit clattery. Now, it's in another room tucked under a cupboard, very headless. What do I care about noise?

Now, I could do a big benchamrking experiment, but this should give a reasonable first glance. Yes, the system is doing other things, but since the cpu sits at 97% of the time in a low-power state, the load isn't that high anyway and is probably negligible to the result.

Doing a simple read of one gigabyte from the RAID-5 array, at different offsets to remove cache interference, shows a remarkable difference. Reading a 1GB data segment from the logical drive (under the filesystem) increases throughput from 147MB/s to 220MB/s when the acoustic mode is set to FAST.

On the filesystem itself (ext4, defaults), extracting the latest Linux kernel source from a tar.bz2 file found on kernel.org more than halves the duration, from 3:03 to 1:30. Reading the resulting directory tree with `ls -lR` improves from 10s to 5s.

In the end, it probably makes little difference to my day-to-day tasks, but optimisation is central to any real techie's heart.

How to set or measure these features in Windows? No idea.

Getting IP Right in Windows: 4. Disable NetBIOS

Networking in Windows is deceptively easy. The level of development Microsoft has achieved to make it so is quite considerable, and I contrast it here with the amount of tweaking required to get Unix services off the ground.

That said, a well-implemented IP structure is the cornerstone of any enterprise (or even serious home) office deployment. I’ve composed a series of five articles on topics you should be really getting right! There are certainly more, but these stick out in my mind.

4. Disable NetBIOS over TCP/IP (NBT)

The first network I ever configured around 1996 used the NetBIOS Extended User Interface (NetBEUI) protocol, and worked fantastically on a Windows 3.11 or 95 computer with 4MB RAM, happily fetching my files on my LAN and helping me (virtually) shoot my friends. Locating the file server (or peer) was accomplished using broadcasts, routing wasn’t an option and I had absolutely no need to talk to anything but other Windows devices, which was fine.

These days, I expect to be able to retrieve 4MB per second on my LAN, probably more, my computer regularly sends packets destined for a server thousands of miles away running who-knows-what, and modern network topologies would have baffled me back then. Microsoft has gone a long way to make sure every product of theirs, and supporting services for applications, are fully transitioned to TCP/IP, and yet NetBIOS is still in there, broadcasting the names of my computer, domain and the servers back at the office to all and sundry, just in case.

Turn it off!

There is a minor security concern that these broadcasts advertise to everyone on whatever LAN you’re plugged into where you work, what version of Windows you’re running etc, and there’s even been some mutterings of an exploit or two, but the threat is not significant.

NetBIOS advertises hostname of a service, be it a file share, chat endpoint or workgroup in a 16-byte field, with the last being reserved for the node type (e.g. 00 for Workstation, 03 for Messages, 20 for a File Server etc). From this, we’ve inherited the hideous 15-character limitation on hostnames and domains. Now I’m not advocating long hostnames as a rule, your naming system should be concise and accurate, but just as 8.3 filenames giving way to 255 characters in Windows 95 freed us from ever-more cryptic shorthand, this is a system that is long past the shelf date.

The short hostnames are a bother, but the biggest evil of NetBIOS (specifically NetBIOS over TCP/IP, or NBT) is to hide mistakes. If your DNS is improperly functioning, a NetBIOS Name Service (NBNS) broadcast or Windows Internet Name Service (WINS) query picks up the slack by asking everyone on the network in the hope that the right node will respond, or forcing you to rely on the WINS service, which is steadily being obsoleted by the folks at Microsoft.

Do yourself a favour, disable NetBIOS over TCP/IP (NBT) on every interface of systems in your lab and home from the word go. If you’re doing labs for training, make this part of the base install, or include it in your domain policy. Of course, for your company network run this through your testing process first. You may spend some time fixing the problems that crop up, but like me you’ll be quite surprised just how much you were depending on it in the first place.

Previous: 3. IPv6 is Coming
Next: 5. NAT is not a Firewall

The Bolt-on Operating System

For years I’ve wondered why on earth so much is crammed into the Windows base image. Sure, they’ve got a decade worth of hardware to support, and since vendors don’t create standardised, reusable code like Linux, this is a significant factor in bloat.

But one of the longest running gripes I have with Microsoft’s OS offerings is that all manners of features are included that I don’t care about – and some that I am super-passionate about are just plain gone! The base installation for almost any major distribution of Linux will include a lot of productivity tools, but leave out some others. It comes down to personal choice, but since all the software is free and available to install from the Internet, this is no inconvenience at all - assuming good Internet connectivity.

That last point is actually quite big, since the distribution may contain an impressive array of software, in the end I’m probably going to want something that didn’t ship on the CD. Can’t have it both ways I guess.

But obviously, the last thing Microsoft would want is to not install something, then have you go back to the installation disc for the features you’re enabling. Of course, they STILL haven’t figured out that Unknown Device from Unknown Manufacturer isn’t a helpful message.

One of my recurring gripes (that is before I started blogging, so I can’t really prove it) is that my Windows Server has a GUI. Seriously, I don’t want a GUI. I want my apps installed on my server, and the management interface installed on a workstation somewhere else. Or the way Linux handles it, X as a process, in my own privilege space that can be launched just for me, and perhaps VNC or X-like somewhere else. The problem of course is that almost every Windows application requires a GUI to install. Sure, MSIs offer silent installs, but so often these line-of-business apps don’t have neat MSI or respect the silent option.

There are two things I’m getting at here;

First, even though the product is named “Windows”, and it’s grown out of a desktop graphical OS that pretty much reinvented the way we deal with computers (though much cred to Apple), I want a server. No people standing in front of it, so no pretty colours required. All the prettiness should be produced by the apps and displayed as IP packets. And on the same note, I see a bunch of sound drivers hanging around.

Well, Windows Server 2008 has the Core option you tell me. It’s a good step in the right direction, but I was genuinely disappointed to see the command prompt surrounded by a window. MMC is still there. I had actually expected to see a text console only, no graphics. Quite simply, it’s a waste of resources. Any app, service or component worth its’ salt is manageable remotely, from simple DHCP up to complex SQL Server clusters. It’s also a danger, as I recall at least one major outage at a previous company thanks to faulty graphics card drivers from <server vendor name censored>. And the GDI component, dating back to Windows 95, handles page and print rendering. Exploits in the 16/32 bit era and in the 32-bit era come to mind. That last one spans products over three release generations, all for a component that doesn’t belong there in the first place.

If I want graphics processing, print rendering etc, then let me add it on later, the way I would Ghostscript on Linux to make a PDF. That brings me on to my second point, and the trust of my argument: components that don’t belong.

The Sasser worm devastated computer estates around the world, by exploiting code hooks in the LSASS.exe process that handles security arbitration between requesting apps/users and security providers. Read closely into the articles, and you’ll notice that it is specifically a problem with code for dcpromo, the process that turns Windows Server into a Domain Controller. The applicable hotfixes patch the code on Windows Server.

On XP, the hotfix removes the code.

Just what was it doing there in the first place? I know the server and desktop products share a codebase, but this irks me. I’ve personally implemented (though not used) a hack to enable RAID on Windows XP that officially doesn’t support it, since the raid driver and all the GUI code is present but disabled. I suspect the same is true of the EFS code in all Home versions of Vista (since you can read EFS-encrypted files from Windows XP upgrades just fine).

Windows Server 2008 requires you to specifically add features to your server before activating them, like AD Directory Service (AD DS), and installations of SQL Server and Exchange (at least) check for updates to the installer before running, getting them closer to the model of Linux distributions – adding a feature from the online repository ALWAYS adds the latest version.

In the end, the development models of these two are very different, so I’m keen to see what further advances can be made on both sides. As always, security and functionality butt heads, somehow I end up with the headache…

The case for open-source virtualisation

I’ve been quite a keen follower of virtualisation since the early days of availability on commodity hardware. Obviously, big rigs like IBM, HP etc have been doing hardware partitioning since time immemorial, but I’m interested in the stuff that lets me run an OS concurrently with my main OS on a desktop.

To my mind, one of the big advantages of Windows (and to a lesser extent Linux) is the homogeneity of the OS on desktops and server. That is, if I write an application, website, database etc on my Windows PC, compile and run it, it should deploy to my Windows Server without any changes. This is in stark contrast to the development model for older big iron systems, where the development happened on the system itself, probably on a dedicated development partition, but quite certainly not on my Windows (OS/2, DOS etc) workstation without cross-compilation.

Nowadays, it’s taken for granted, but that’s always been in the back of my mind. With virtualisation, I can now deploy a test partition on a development server for Windows. It’s not just for development, but for testing too, and in my current path to get my Microsoft certifications up-to-date, it’s a godsend. I’ve got a fairly sizeable server that does almost all of my home tasks; file shares, email relay and filtering, VNC, photo sharing website, music streaming, proxying, VPN, and I’m working on VoIP. The thing is, it’s Linux.

I run Fedora. At one point I banished Microsoft from my home, just to see if it was possible. I wanted to accomplish as much of my home automation and services using open-source software as possible, and put Fedora on my main server (it’s actually been there for ages), Fedora on my power laptop and Ubuntu on my lighter laptop. and hey-ho, it works!

Well, actually, my mileage varied. Ubuntu is just great, it works well with laptop hardware (especially the Intel graphics), wireless and sound. Fedora also just works, that is until I tried to virtualise on the laptop.

I had read about Xen some years back, and how it offered paravirtualisation on Linux. I tried to get that working, but once I figured out it interrupts communications with my Radeon chip (they claimed it wouldn’t) I dropped it. Enter KVM, stage right!

I’ve been a keen follower of AMD’s hardware virtualisation (Pacifica) since the inception. I’ve actually been a very big fan of AMD’s for some time, but that’s for another post. VMware and other players have been doing hardware virtualisation for some time, but the Intel architecture just doesn’t play well with others, meaning the host OS and hypervisor needed to do a lot more work than ideally required to keep up the illusion for the guest. VMware’s intellectual property in the regard is substantial, and for years were the cut-and-dried leader in the field.

Qemu is a mature hardware emulator I’ve been using for years that runs all kinds of CPUs on multiple host OSs, but still runs in user space (i.e. no kernel privileges, and a lot of context switching for privileged ops). As soon as the kernel module is loaded (kqemu), things perform rather well indeed, but still noticeably slower than bare metal.

Hardware virtualisation (HVM) in Intel chips changed that, since now a lot of the grunt work like intercepting privileged interrupts can be caught by the CPU and sent off to the Hypervisor efficiently. In no time at all, the existing Qemu binaries were extended to include KVM, Kernel-based Virtual Machines. I’ve been a convert ever since.

By using a well-established virtualisation platform like Qemu, with excellent hardware support, KVM runs a lot of guests!

So why am I raving about this? Well, I often get asked why a Microsoft techie runs Linux. As I have previously stated, I tinker, and Linux offers me that chance. I get to play with raid in granular detail. Layered on that is LVM, which virtualises storage. You don’t have to get that complex, but it’s the closest I can come to simulating a SAN in my own home, with resilient, abstracted hardware hidden from the VM.

I make and break bridges and virtual networks on the fly. I’ve got three gigabit Ethernet ports, and segregate them by function – to the point where ALL VM traffic is on a dedicated port, so that if a VM talks to the host it pops out through the physical switch and back in on the front interface.

Here’s the key thing: While all of this is probable available from other vendors, and some parts may even be free, I’m in control. Sure documentation can be sketchy, and it requires a lot more basic knowledge of networking, storage and hardware architectures than other solutions, I’m hardly take my car to a garage to be serviced if I wanted to be a mechanic, I’d get dirty. Break things definitely, but that’s a great learning tool.

My power laptop used to break whenever I entered standby, and I mean trash the root file system (ext3 to boot, supposedly bullet-proof) so I filed a bug. As it turns out, it only happened when I was running HVM, so I applied a patch published 30 days before to my kernel. Voila!

Now this is a big thing! Without being a paid customer of Redhat’s, I got a problem resolved quickly and comfortably. I must stress here that Fedora is BETA software, permanently, and I expect it to break, but frankly for my needs it’s just fine, and when I can get things like this resolved, even better.

Sure, it’s painful, but it’s all under my control. I’ve scripted Windows 2003 deployments with snapshots that boot a new instance, sysprepped and ready to go, in 7 minutes. IPTables lets me simulate complex networks, firewalls and even lossy links.

I remain firmly impressed!