WYCRMS Part 2. Windows Just Isn't That Stable

In 1997, a HP 9000 engineer wouldn't blink telling about a server that had been running continuously for over five years.I found this remarkable at the time, and couldn't imagine a Windows server lasting that long. I have moved on, and frankly expect my Windows servers to survive that long today. Very few share this position, and I'm trying to find out why it's so lonely on this side of the fence.

2. Windows Just Isn't That Stable

Ah BlueScreen of Death, how I've missed you. Actually, I haven't, since finding out what caused them was a nightmare, and recovering without a remote console solution is not conducive to a predictable social life (or sleep schedule). That said, they were so common we even had joke screen savers mimicking them for our own geekish amusement. Since Microsoft acquired Sysinternals they're even available to download directly from Microsoft. Imagine your in-car entertainment system being configured to show you fake warnings of a failed brake line, or a cracked cylinder head. "Would you like the free video package of Ford vehicles endangering passengers' lives with your new Focus sir?". IT people are weird.

I've analysed my Windows 7 x64 installation, and in the last three years I've had six bluescreens. Once was my graphics card (pretty unique), all the others were my Bluetooth headphones putting my cheapo-Bluetooth dongle in a spin. I blame the dongle, not Windows.

OK, that's not fair to the dongle maker: I blame Windows, but only the Bluetooth stack since it's never been something I expect Windows to do well - multiple dongle-headphone combinations have yet to produce a pleasant experience (three dongles, two headphone models). The network card, storage stack, print drivers, memory management, process scheduler (NUMA-aware these days apparently): These all work so well I haven't notice them doing their job, and I am very familiar with what a complex job they have.

I expect roughly once a month to see a BSoD on public transport, or at stations, or many airports, or billboards. The layout of the BSoD has changed over the years, with each version of Windows getting a little tweak so that you can spot the version even if the error itself is gibberish, and I conclude from viewing these blue non-advertisements: These systems tend to be A) old, B) written in languages and coding styles that aren't that good, and C) interface with devices with terrible drivers.

This is not typical of modern Windows servers.

I would never dream of subjecting a server to the amount of change my hard-working personal workstation endures. AMD updates my video drivers multiple times a year, I attach and detach USB/phone/iSCSI devices more often than I refill my car's tank, and run code from pretty much anywhere as long as it promises me utility or entertainment. A server is different, running things I trust to go on processing without attendance, cleaning up after itself, and basically staying up. If I do make changes, it's controlled, tested and left the hell alone.

Windows Server is solid, and every iteration gets more solid. It's expanding to 64-bit spaces, handling multipath-iSCSI with ease, more cores than I have fingers in byzantine NUMA layouts, hosting server instances in their own right with Hyper-V and pushing gigabytes around through network cards and storage interfaces, crunching data and most importantly providing services.

Yet the very people who spend time and money proving they are skilled in designing and administering these systems so that they can adorn their signatures and office receptions with impressive Microsoft-approved decals are the first to tell you not to trust a given server (without even knowing the workload or configuration) to remain available. They express surprise and concern on viewing a server continuously running for over a year.

I'm surprised and, yes, concerned that they react this way. Isn't this what your sales folks promised me in the first place?

Previous: Part 1: But I Have to Reboot My Own Windows System All the Time!

Next: Part 3. Console Applications, Java, Batch Files and Other Red Herrings

WYCRMS Part 1: But I Have to Reboot My Own Windows System All the Time!

In 1997, a HP 9000 engineer wouldn't blink telling about a server that had been running continuously for over five years.I found this remarkable at the time, and couldn't imagine a Windows server lasting that long. I have moved on, and frankly expect my Windows servers to survive that long today. Very few share this position, and I'm trying to find out why it's so lonely on this side of the fence.

1. But I Have to Reboot My Own Windows System all The Time!

I've mentioned before how Windows makes you lazy. One of the great things about Microsoft Windows as a platform is that software developed on a $500 workstation can be installed on a $50,000 server and probably work without problems. Of course, getting your home-brew software to scale is a different matter, but you get the idea: One platform, different size.

Almost every Windows engineer cuts their teeth on Windows at home, and this informs their experience and expectations of the platform. Like everyone I get tired of the bogging down after a few days/weeks/months uptime and reboot just to clear things up, but that's my fault and not Windows.

I'm lazy.

Typically, I'm running browsers, office suites, anti-virus, any number of games, and install new stuff roughly once a fortnight. Flash, Java and Windows Update are constantly pestering me to reboot after updates. I've even been the one to reinstall completely after a year to see the wonder of a zippy start-up and responsive GUI, only to have it slowly crawl as I add functionality (including those games). Happily, my Windows 7 installation has lasted two years by now with no significant falloff in responsiveness, so that's getting much better, and I only power down/reboot of my own volition when I'm fitting lights and need mains power off - even then it's more likely to be a hibernate.

Servers are not workstations. Any good enterprise has controls for how changes are made to IT systems, and even simple patching requires testing and approved windows to take the system down and update it. In my experience a server will undergo a major overhaul at most twice in its' operational lifetime, and organisations with exceptional controls have zero - new version? New server!

A good server (and I think of Windows Server 2003+ as good servers) will run for decades given quality power and no moving parts. Of course hardware fails, but Microsoft have put in man-decades to get Windows to handle routine changes without downtime. I remember Windows NT 4.0 needing a reboot for an additional IP address. Modern versions of Windows can hot-plug an entire NIC (physically) without a blink, though admittedly I've never actually encountered anyone who uses the facility.

If an engineer merely mentions that, in their experience, Windows needs rebooting I question their experience. I mean it: I question their experience!

Windows is solid, and I can recall only one confirmed bug where Windows will fail (actually, begin to fail, an outage is not a certainy) for the simple factor of running continuously for a given time. When someone speaks of a memory leak that has caused Windows to run out of (insert wooly term here), again I question their experience and the quality of the software/vendor driver code. I've stopped blaming Microsoft.

When I run my applications on Windows Server and, more importantly, when I am paying someone to manage those systems for me, I expect them to have faith in their products and promise me server availability. Rebooting breaks availability.

Previous: Why You Can't Reboot my Server

Next: Part 2. Windows Just Isn't That Stable

Dawson College: What Island Are You On?

I've been viewing the growing story about Ahmed Al-Khabaz, a Computer Science student at Dawson College in Montreal, Canada, who was expelled for running a security scan against their public web presence to discover if a flaw he found was resolved. I stumbled on a subsequent interview with prominent IT Security professional Chris Wysopal. I hadn't heard of him, but when I saw he was previously associated with l0pht Heavy Industries my eyes snapped open.

This guy has credentials, and I don't know of an IT professional active around 1997 to 2003 who hadn't heard of, or actually used, l0pthcrack, often to solve real-world problems. First and foremost a password auditing tool it can be used maliciously, but the so can a toaster oven. It is a piece of code art: Necessary, useful and (at the time) industry-shaking.

White Hat hacking is a tricky business. Even I've done it, against a bank no less, fully in the knowledge that I was doing something the system owners would be very unhappy about. In some cases it can get you arrested. I was was pleased with the results when my concerns were taken seriously and fixed fairly quickly. I've worked in financial services companies and know their software release process is iceberg slow so this was very reassuring. There's one thing Mr Al-Khabaz and I both know that drives thousands around the world to the same end: I'm at risk.

Dawson College is hand-wringing and special pleading: "the law ... forbids us from discussing your personal student files" is in this case weak. I am pretty sure the former student would agree to a waiver of his right to privacy to clear the air, but I have seen no mention of an offer. Fourteen out the fifteen professors convened voted for his expulsion, for doing what some professionals get paid extremely well to do (even I've been offered this job): Evaluate the security of publicly-accessible websites. I would like someone better informed than me to comment on what the implications would be for the institution if it was discovered a breach because of this flaw caused losses thanks to the personal information disclosed.

I can appreciate that the college does in fact have to abide by law, and is unwilling to get into a mudslinging match in the public forum. They have rules for ethical behaviour that may have been violated (I haven't seen them). But beyond those considerations, every one of the fourteen professors needs to answer one simple question:

Why, if these actions are so outrageous of a Computer Science graduate that it demonstrates
"behavior that is unacceptable in a computing professional" has the company whose software flaws he exposed taken it upon themselve to pay for his further education?

Academia is often seen as disconnected from reality; some lines of research beggar belief, and the same could be said of Computer Science. I've met a few graduates who arrive in the IT industry ill-prepared, full of theory of operation and design but unable to command a command-line. No matter what their actual instruction is, a critical point they need to learn is that the Internet is a hostile place. It is also a collaborative place, where FOSS abounds and Creative Commons is richly rewarding. Poking around is the norm, and if this college is telling their students that they are to accept their instruction blindly without considering real-world implications, or use those skills to explore, then they don't deserve to be associated with the term Higher Education.

They may perhaps be able to educate Code Monkeys, but thinking professionals able to design and protect systems that impact their lives? Not really.

How Important is MariaDB? Let's test the fork with butter.

MySQL has interested me for quite a long time. I first came across it in 2000 when trying to find a better way to analyse the contents of a 20,000 user Active Directory and needed more relational DB-stuff than Microsoft Access could deliver and cheaper than SQL Server (wow MSDE was terrible). I was deeply impressed (though probably because I was easily impressed back then) with the performance and cross-platform support, and ever since it's been around my life.

I currently use it for my XBMC and Logitech Media Server (SqueezeBox) media databases, as the back-end for my Gallery3 site, and other ad-hoc databases whenever I need to crunch data. Before my 64-bit processor created a new ISA that ensured a reasonably complete instruction set, it was a favourite of mine for optimising binary compiles over the stock i386 build supplied by most distros, but more for interest's sake than actually squeezing performance for any measurable benefit.

MySQL AB was of course the owner of the copyrights and code and opted for a relatively unique license, both proprietary and open. As the owners of the code, they could choose to do this, but anyone trying to make a buck out of the code was obliged to release their modifications. Now that Oracle (through their acquisition of Sun, who acquired MySQL AB) have that right, the open-source community is in a bit of a fluster. Can we trust a corporate giant with custody of the code that runs a significant fraction of the Internet's websites? The answer is slowly coming down on the side of "no".

Oracle (and others, and unsurprisingly) is being guarded about bugs and fixes. Stories of vendors forcing customers into NDAs before even admitting bugs exist, hiding bugs from other customers, and silently including fixes are common. It's face-saving. Andy Grove's "Only the Paranoid Survive" starts off with how Intel hoped to keep their Pentium FPU bug quiet while they implemented a workaround simply smacks of arrogance. While it doesn't yet seem Oracle are trying to hide any actual code and still supply source, MySQL has historically had test cases for bugs published alongside them to protect against regression and anyone can run the suite on their installation to verify code quality. Not only are they apparently now keeping some cases secret, they are also not clearly marking which code updates fix bugs they are refusing to publish.

This is not how open-source works, but I don't agree with the prevailing rationale. RedHat came into the firing line for being less than open they handled a code signing infrastructure breach, but in that instance I support the way they behaved as it was not their source they concealed, rather their own systems and controls that were embarrassingly compromised. They have shareholders, and revealing too much would have cost them. Oracle too have value invested in their products and would like to keep flaws hidden. This is not nefarious, it's capitalism.

MySQL as a product is different, no matter who owns it. It is very closely tied to the spirit of the open-source movement, being both highly regarded for performance and features, and for the competition it gives proprietary offerings. For Oracle to claim that ground back is entirely within their right, but the edge is gone. The most ardent supporters and influencers of purchasing are not happy and a slow exodus may be starting.

So Fedora and Wikipedia are both contemplating pulling out. The MariaDB fork has all the features and more, is fully open in the original spirit of the project, and is attracting attention including mine. I have no idea how easy it will be to do the fabled "drop-in replacement" every source claims is possible but I feel ethically compelled to leave MySQL in the dust. I have a server that runs my digital life and it is a conscious choice to run on open software only and it has not been easy, but as an experiment and learning tool it is invaluable.

The great thing about open-source is anybody can fork. I can clone a source and apply my changes as I like, but the moment I try to give it to anyone else (especially selling the result) I have to disclose my whole body of work. This can lead to some confusion as the early days of Linux showed, but in the end the market weeds out the under-performers and delivers better products through sheer market forces. MariaDB seems to be that winner.

I do know one thing: testing the transition is going to be a breeze: After switching from Fedora to Gentoo four months ago, I rolled the root over to BTRFS (once kernel 3.6 gave me the necessary confidence). Add a distinct IP to the NIC, snapshot, chroot, and I've got a clone of my server ready to go in about two seconds without that system-level virtualisation stuff and hideously slow LVM2 snapshots.

Rollback to base for a fresh attempt? Yep, two seconds.

Do the ends still matter?

One of the biggest conflicts of interest in my role as an infrastructure architect (I'm responsible for servers, applications, security, storage etc) is that I have a heavily vested interest in the service they provide. This may sound like a no-brainer, but often I need to find ways to get my traffic to users at the expense of someone else. I want preferential treatment if conflicts with other traffic.
Of course, it's not quite as simple as that. Pragmatically I realise I am competing with others on limited resources, whether it's rack space, hypervisor RAM or network bandwidth. One of the less-known principles underlying the Internet ethos is the End-to-end Principle. Put simply, it's this:
Complexity in any network should be implemented in the end points - the network stacks of communicating nodes and in applications - and not in the network itself. Since standards change any benefit of implementing too much intelligence in the network is quickly undermined with the need to continually match those changes in end-points, as well as legacy issues.
Essentially, the network should be as dumb as possible. Packet comes in, packet goes out, wait for next packet. This is reasonable in principle but different traffic flows require different treatments - VoIP and video streaming protocols prefer low latency and are almost always jitter sensitive, while file transfers can tolerate enormous latencies as long as they are accompanied by high bandwidth. If both types occupy the same link the risk is that the insensitive consumption of one protocol can impact the requirements of another, so QoS arrives.
This is already a necessary evil (though some may object to my use of that label in this context) since we now need to build knowledge of services into the network layer. Large IP transport installations (Internet backbones) handling gigabits of traffic per second literally don't have the time to implement protocols like this as processing adds both cost and latency. This is different from implementing different virtual circuits for different traffic types, in that it forms logic links and is a very common practice with ISPs in the final mile. As far as intelligence on network devices go, this is very low - again packets come in, figure out where packets go out.
A very interesting turn is in the development of new forms of network acceleration. Routers have long been capable of doing in-line compression of data to reduce consumption of a specific link, but this is point-to-point. If an application protocol can truly benefit from compression, this really should be done at the application protocol level (basically above Layer 3) so routers and switches can shuffle packets. A possible side-effect of link compression is to mask real versus usable bandwidth from applications (some portions of a stream may be highly compressible, others not) hindering flow-control algorithms built into TCP.
TCP sessions can be rather chatty, and some applications implement redundant techniques inside their own specification, so some network acceleration can initially make sense. Essentially, new sessions follow a well-known pattern of window sizes and other parameters, so network accelerators intercept packets and simulate the repetitive parts on each end of a link, reducing session setup time. This sounds simple, but now we enter dangerous waters. Next comes protocol caching; I request a file from a file server across the WAN and the contents are cached on my local acceleration appliance so that the user next to me can get a cached copy when her request is made. Again, sounds simple, but to prevent interception and modification the protocol implements signing, so re-generated content requires a re-generated signature. SSL acceleration similarly can be implemented using reverse proxies holding a copy of the private key for the service to extract the plaintext and look for compression and caching opportunities. I've been involved in design and deployment of many reverse proxy and SSL acceleration projects but these were explicitly part of the service.
To accelerate generic payloads the appliance needs to get heavily involved in the infrastructure, either by hosting all SSL keys and being able to masquerade as endpoints. This is where it gets very complex and risky, not only in the dissemination of privileged access (private keys and domain credentials being the most highly-prized items in any network), but also a continual catch-up game to implement these techniques on newer protocols as they are developed. There are also more intrusive techniques such as automatic downscaling of images when using mobile data to browse the web that are subtle but insidious.
Net Neutrality is the overall drive in this direction and correlates with the End-to-end Principle, treating network devices as simplistic and all traffic as equal.Smarter protocols such as (distributed) BranchCache and BitTorrent (yes, BitTorrent is a case for Net Neutrality) reduce redundancy over constrained links and better content intelligence (I explicitly convert all images in my documents to 8-bit PNGs before embedding) are far better strategies. Content distribution networks are an active participant and used by some of the largest providers (both content generators and ISPs) to reduce long-haul bandwidth and improve user responsiveness. HTTP compression is a rigorously defined standard but very sparsely used even on static pages where on-the-fly zipping is unnecessary. I could go on...
When looking into how best to transport content I prefer to let the network do what it does best and work my requirements into it than engineer a network to suit my needs unless absolutely necessary. Where my application is burdensome, I would prefer to engineer the application than throw bandwidth at the problem.
Technology marches, and bandwidth - while logically finite - seems to be keeping up rather well.

What is a Read Receipt?

I saw a posting on GMail's messaging forums today about their apparent lack of the facility to have a Read Receipt generated. I felt compelled to reply, and I thought I'd post it here too as it's something not quite as obvious as it might first seem. Google is moving more and more into the enterprise, and this is a standard feature that is just plain missing, so obviously it's noticed. To me, it's more of a social than technical challenge...

A Read Receipt is quite the misnomer: the best you can say is that the message was shown on a screen. Which screen though? GMail is accessible through  the webmail, POP3, IMAP, mobile devices (GMail native client, Exchange Sync), mobile web... what exactly are you trying to prove happened?

I've found Read Receipts to be near useless, while Delivery Receipts are generally poorly implemented and probably achieve the same thing: some systems will say that they've taken receipt of the message into the system, but what you actually want to know is that it has successfully arrived in the user's mailbox. Signing for a courier delivery is exactly that, but you'll never know if the recipient ever opened the package.

If you're so keen to know whether the person opened the mail, ask them to confirm receipt manually. Having a computer tell you that you read something is at best trivial and at worst misleading. For a one-line e-mail,this facility may be useful and mostly true, you're likely to have absorbed it. A 1000-word essay, what you really want to know is not just that it was displayed on some screen (and reading that much on a mobile is pointless), but rather that the end-user absorbed everything. Only the user can attest to that.

On principle, keep read receipts out of GMail (or other providers, even Exchange!), and these problems go away. Just ask for a reply that your message is received, understood and/or acted upon. Much more useful.

I'm very disappointed to report that it works

A friend of mine was recently reviewing a friend's CV, and noted he had an ethical hacking qualification. I found this rather amusing, since I know there are a gazillion unqualified, uncertified hackers (and simply curious people) who I'd much, much rather be on some kind of radar. It reminds me of the DRM/DMCA debate: it's the ones not following the rules that tend to get the benefit, or at least know enough about the rules to not care about enforcement.

I have been having a series of discussions with a few security specialists in the last two weeks, and they've put a few seeds in my brain. I've reviewed a few articles about application vulnerabilities, and with more and more of the world moving into "the cloud" (btw, I hate that phrase) we're handing over more control to this nebulous entity. Google are definitely at the front, at least as far as end-user experience goes. Heck, I'm hosting this very blog on Google's servers, and neither know nor care where they are.

I also recently acquired an Android phone, and allowed Google's hooks into my life to sink just that bit deeper with integrated messaging, contacts, calendaring, apps I really don't need, Facebook on-the-go, Twitter... it's all cloud! I left my laptop at a friend's house recently, and realised (to my own shock) that frankly, I can live without it for a day or two, such is the functionality in this great new device.

So while all the focus is off in the cumulo-nimbus, I'm still dealing with daily life that's hosted and automated on some providers that are definitely well-defined. My bank is one of them, and as an extra I do some share dealing with their attached brokerage. Side note: three years ago I had spare cash and thought "what's safer than banks?".

Today, I placed an order to sell a few shares and received an order number. I recalled a conversation with one of these specialists about session identifiers where we discussed collision avoidance and non-sequentialness as two good markers for session tracking. On a whim, I took the URL generated by the transaction ID to view details and incremented the trade identifier by one digit.

Lo and behold, I got the details of someone else's trade. One thousand shares of an oil company, concluded around the same time as mine. Alarmed, I did it again, this time decrementing (I'd hit an upper bound), and found an incomplete trade. This is an order, as yet unfulfilled, awaiting the conditions set by the initiator. Now granted, i couldn't see the identity of the trader (in either case), so perhaps on the surface not such a big deal. But, if you know dealing, complete trades are not so significant as they are done and dusted, while incomplete trades show intention. Script this query for current and future IDs, and you could get a feel for investor sentiment that gives you an advantage.

I've been using this particular trading system for years, and regardless of the losses I've made (seriously, I have no aptitude for this) thought of the security measures as fairly robust: SSL encryption, separate login and dealing password both never revealed in full, limits on trading volume by account type and history, approved browser versions only. How easily we are placated.

Handing over so much of our personal info into the (at least free) cloud scares me, though I'm conscious of the fact that free products are, in the sage words of my father, worth what you paid for them. Paid services may not fare much better; by abstracting services into this fog, we run the risk of losing touch with how services are delivered, how we control them, and what we stand to lose if it all goes wrong.

But most of all, they're still built and run to the same rules as traditional systems, no matter how abstractly they're presented. The same DBMSs, the same web servers and runtimes, the same developers and critically the same developer mentalities.

A sobering lesson indeed.

Oh, and yes I raised this with the brokerage concerned. Does that get me the ethical badge?

Multihomed DNS and how Windows makes you lazy

I was called in to solve an interesting problem today. There's a good principle in effective IT security to segregate different services into LANs appropriate to their function. If there is more than one function, then more than one NIC (perhaps a VLAN ID) is required, with a unique IP address.

Windows is a great tool, a platform of continual development (and yes, that means it gets better, so don't think I've always found it to be great) over the last two decades that now runs a fair chunk of global business, and in some demanding environments. One of the simple beauties of the platform is the unified codebase, libraries and APIs from the smallest XP Embedded right up to Windows Server Datacentre Edition: The machine I'm developing on is almost identical, apart from scale, to the machine I'm likely to deploy on.

Yes, there are other differences, but I find most of them to be paid features like clustering and more speed (I can't do it captain). The hardware abstraction layer and other consistencies like the IP stack, filesystems and memory management are wonderful tools for developers. Unfortunately, so many admins cut their teeth on Windows desktop editions, or at least smaller servers under their absolute control, that they struggle to make the transition to enterprise administration.

My previous rant about NetBIOS is a case in point. With all this abstraction, details like network interfaces and network service location are so well hidden, they're essentially invisible. Ever try to catch the Invisible Man to ask him what he's doing?

The problem I had to solve today was around multihomed servers. Windows IT admins tend to be lazy, and NetBIOS broadcasts are only one factor where we rely on the wizardry of the OS to figure out what we're trying to do and make it happen. Dynamic DNS registration removes some of the tedium and mistakes from the process of getting systems deployed, but blindly assuming it knows what you want is just wrong.

The convergence of an Active Directory domain and the DNS namespace is a nifty feat, but in multihomed systems it's a nightmare. If all interfaces are routable and reachable, then this is slightly moot, but put up a firewall or routing restriction in the way and intermittent problems (the worst kind) crop up, and troubleshooting without a solid foundation in networking is tough. DHCP, DDNS, NetBIOS, even APIPA, all seek to hide the complexity from Windows admins, and they end up woefully underskilled in the cornerstone that makes their network tick.

The problem in this instance is that the FQDN of the server is comprised of the hostname and the AD DNS Name. No problem for a typical, single-NIC server. Unfortunately, this is an abstraction when it comes to multiple NICs: just how is DNS supposed to know what your topology is when giving you an answer.

Trying to convince a religiously Microsoft admin to use a subzone to specify the interface is absorbed with something approaching heresy. Do a traceroute (or tracert for Windows guys) to any internet address and you'll see FQDNs of routers, with the hostname portion wildly different along the way, including multiple digit groups. Most of these are Internet routers, and the DNS entries correspond to the interface rather than the router itself.

Of course, Windows has a mild cow if you try to refer to it by anthing other than the system name as the first part of the FQDN, and always expects all interfaces to be present in the machine's dns suffix. The best solution...

Change the way you think about finding servers. When you're connecting, you're probably interested in a particular interface anyway. Some services may not even be listening on particular interfaces. Getting your brain tuned to how your network is built, using that to figure out how your systems are connected, and habitually spelling out exactly which way you want to connect by an explicit FQDN can only do good.

Of course, some applications take it as read that a server is reachable by the short FQDN. Sometimes system admins can be even more hardcoded. Both are very, very difficult to change.

Utility as a Learning Tool

IT, as with other high-skilled vocations, requires a constant cycle of learning and certification if you are to attain, retain and prove your skills. Each practitioner I know has specific competencies or bias towards particular product sets, architectures and vendors, so naturally we keep up with the latest developments and new releases.

One of the recurring problem engineers have is getting to grips with the ins and outs of a new product. Supporting the infrastructure for large organisations requires a lot of time to explore the feature sets (especially as they contrast with the vendor’s stated features), their utility, recoverability, capacity etc etc. This is quite entertaining in itself; incremental product releases generally tend to build on the feature sets of earlier versions, and if the vendor is any good they’ll provide good documentation and training for the upgrade path.

The one aspect I have problems with is a brand new product set from a particular vendor. Headline products servicing databases, messaging and operating systems are very infrequently created from scratch, but the myriad supporting products and protocols are under constant evolution. Quite often they are aimed at improving a small part of a whole, and the learning path can be intriguing at best.

I enjoy getting to grips with new products, but one can only go so far without a goal. My biggest problem with PowerShell was always that it was a reinvention of what most administrators were doing just fine with other technologies. Granted it’s streamlined and feature-rich, but as a fairly hefty departure from Windows command-line scripts or even Windows Scripting Host, the clear need to adopt it wasn’t ever really there while the learning curve was rather steep. This is a big problem; I knew I needed to learn it, but without a problem to solve the effort required doesn’t seem to match the reward.

I’m not the sort of person who will gladly sit through pages of manuals or RFCs to understand a new product or protocol. I’m much more hands-on; my personal systems include a myriad of products that I never imagined I’d come to rely on, they were mostly simply trying to understand how they work. Now that I do depend on them, I have bumped into each of the nasty bugs and side-effects they present, as well as discovering both features that are not in the headline literature and uses that even I hadn’t anticipated when I set out. If you build it, they will come.

And this is the focus of my argument. Pure learning, whether theoretical or practical, has no use in and of itself. Only when technologies are applied do they have value. Storage Area Networks (at least before iSCSI), systems monitoring, ERP applications and even the larger database configurations are beyond the need of the average technical user, and require hardware that should exist outside of a raised-floor, fluorescent data centre. Yet almost every technical enthusiast and support professional I know has some form of lab at home to explore these technologies and the products that offer them.

In my own explorations of technology, I have become decidedly indifferent to the specific products I am evaluating, since they come and go. I am vastly more interested in what’s going on under the hood, since these implementations are much more stable across product versions than the latest trend in user interfaces that mark the biggest visible change in product releases. Using open-source software I’ve been able to emulate and get to grips with almost all of the concepts used in large infrastructure installations, from SANs to firewalling to virtualisation to build and deployment to robust databases, and all in a single server. If my employer found it expedient to spring for a lab to learn about the proprietary equivalents, it would cost in the region of thousands to tens of thousands of dollars, and still I’d only learn more about how to perform specific tasks rather than understand the deeper concepts. Of course, YMMV.

So how do you keep up-to-date with current products, that come and go, while still learning skills you can use professionally? Well obviously the specifics of any implementation is important, and getting to know the interfaces, procedures and maintenance of the products is critical if you’re in the support role. But branching out into other vendors can bring a much deeper understanding of the underlying principles and methods than just installing the latest shiny package. If you are fortunate enough to have an employer that has a good lab, schedule a few hours a week for tinkering, and write the results of your evaluation out.

The modern workplace (especially in IT) is less about protecting your job by hiding what you know than in previous decades, and if you can demonstrate your ability to command a new approach, sharing your experiences can only do you good. I’ve found open IT departments and companies that constantly, and critically, evaluate themselves and the ecosystem they work in are definitely more productive and rewarding places to work.

Getting IP Right in Windows: 5. NAT is not a Firewall

Networking in Windows is deceptively easy. The level of development Microsoft has achieved to make it so is quite considerable, and I contrast it here with the amount of tweaking required to get Unix services off the ground.

That said, a well-implemented IP structure is the cornerstone of any enterprise (or even serious home) office deployment. I’ve composed a series of five articles on topics you should be really getting right! There are certainly more, but these stick out in my mind.

5. NAT is not a Firewall

Here’s the part where I put on my flame-resistant suit. I know this is divisive, so let it be known this part is entirely my opinion :)

NAT was devised as a mechanism for hosts on networks with incompatible routing structures (either overlapping network numbers or RPIPA addresses seeking Internet connectivity) to have their addresses transformed into something more palatable. This happens every day in millions of home and corporate routers and firewalls, allowing millions more computers to consume Internet services without consuming the Internet’s most precious resource – global IP addresses.

Since these private networks use IP space that cannot be Internet routed, they are translated on the fly to, typically, one address which is what the destination sees as the source, while the router/firewall maintains a mapping of who asked for what from where, so that replies make it back to the requestor. If a packet arrives that has no apparent previous relationship to an internal host, it is dropped. In this way, NAT is an implied firewall, dropping unsolicited packets from the nasty Internet. Of course, if we need, say, HTTP or VoIP to be let in, we poke some holes and make exceptions.

Precisely because this is an implicit form of security, it is dangerous. Security is all about paying attention, making sure we understand how a threat can enter a network, how the people are affected (or risks themselves), what systems are vulnerable and how to defend against them etc. Defense in Depth, an NSA-derived concept, is all about layering security at different points in the network to increase the overall robustness.

Yet so often, NAT is simply assumed to be a line of defense. True, unsolicited traffic is bounced, but this causes problems for traffic like FTP (unless the firewall has application-layer awareness) and VoIP, whose Session Initiation Protocol has a rough time of NAT. Why then is the security only played out one way?

A commonly portrayed threat is of a trojan application or other type of malware being installed on your computer, scanning for personal data like credit cards and bank statements then uploading them to the nefarious source. NAT, in assuming that your network is the safe place and the Internet bad, gladly allows the outbound traffic through without question, and bang goes your credit rating.

IPv6 makes the need for NAT moot, since the address space and allocation policy should allow everyone to hold their own huge chunk of the address space with Internet-valid addresses. I haven’t yet seen a convincing argument why NAT should live on in an IPv6 world.

While NAT does indeed provide a great amount of protection, blindly approving that it makes you safer is missing the point. IP is a versatile protocol suite, and the fact that NAT is so readily implemented proves it, but without a little attention, you’re letting your router vendor dictate how your network is protected.

Recent versions of Windows include a host-based firewall, allowing each device to control what traffic is allowed to arrive at the network interfaces, and even what traffic is allowed out. Get to know the workings of the firewall and how to define the rules that are appropriate for your environment, including specific applications and how they communicate. Unfortunately, a lot of the protocols used on Windows tend to negotiate dynamic ports for communication, but since the firewall is also application-aware (specific executables are allowed to communicate instead of simply this or that port), it is a fairly easy task to secure your Windows hosts from a lot of the prevalent threats.

Enterprises know this and carefully craft the types of traffic that are allowed in and out of the network, with a little thought your networks can be secure, responsive and available.

Previous: 4. Disable NetBIOS

Getting IP Right in Windows: 4. Disable NetBIOS

Networking in Windows is deceptively easy. The level of development Microsoft has achieved to make it so is quite considerable, and I contrast it here with the amount of tweaking required to get Unix services off the ground.

That said, a well-implemented IP structure is the cornerstone of any enterprise (or even serious home) office deployment. I’ve composed a series of five articles on topics you should be really getting right! There are certainly more, but these stick out in my mind.

4. Disable NetBIOS over TCP/IP (NBT)

The first network I ever configured around 1996 used the NetBIOS Extended User Interface (NetBEUI) protocol, and worked fantastically on a Windows 3.11 or 95 computer with 4MB RAM, happily fetching my files on my LAN and helping me (virtually) shoot my friends. Locating the file server (or peer) was accomplished using broadcasts, routing wasn’t an option and I had absolutely no need to talk to anything but other Windows devices, which was fine.

These days, I expect to be able to retrieve 4MB per second on my LAN, probably more, my computer regularly sends packets destined for a server thousands of miles away running who-knows-what, and modern network topologies would have baffled me back then. Microsoft has gone a long way to make sure every product of theirs, and supporting services for applications, are fully transitioned to TCP/IP, and yet NetBIOS is still in there, broadcasting the names of my computer, domain and the servers back at the office to all and sundry, just in case.

Turn it off!

There is a minor security concern that these broadcasts advertise to everyone on whatever LAN you’re plugged into where you work, what version of Windows you’re running etc, and there’s even been some mutterings of an exploit or two, but the threat is not significant.

NetBIOS advertises hostname of a service, be it a file share, chat endpoint or workgroup in a 16-byte field, with the last being reserved for the node type (e.g. 00 for Workstation, 03 for Messages, 20 for a File Server etc). From this, we’ve inherited the hideous 15-character limitation on hostnames and domains. Now I’m not advocating long hostnames as a rule, your naming system should be concise and accurate, but just as 8.3 filenames giving way to 255 characters in Windows 95 freed us from ever-more cryptic shorthand, this is a system that is long past the shelf date.

The short hostnames are a bother, but the biggest evil of NetBIOS (specifically NetBIOS over TCP/IP, or NBT) is to hide mistakes. If your DNS is improperly functioning, a NetBIOS Name Service (NBNS) broadcast or Windows Internet Name Service (WINS) query picks up the slack by asking everyone on the network in the hope that the right node will respond, or forcing you to rely on the WINS service, which is steadily being obsoleted by the folks at Microsoft.

Do yourself a favour, disable NetBIOS over TCP/IP (NBT) on every interface of systems in your lab and home from the word go. If you’re doing labs for training, make this part of the base install, or include it in your domain policy. Of course, for your company network run this through your testing process first. You may spend some time fixing the problems that crop up, but like me you’ll be quite surprised just how much you were depending on it in the first place.

Previous: 3. IPv6 is Coming
Next: 5. NAT is not a Firewall

Getting IP Right in Windows: 3. IPv6 is coming

Networking in Windows is deceptively easy. The level of development Microsoft has achieved to make it so is quite considerable, and I contrast it here with the amount of tweaking required to get Unix services off the ground.

That said, a well-implemented IP structure is the cornerstone of any enterprise (or even serious home) office deployment. I’ve composed a series of five articles on topics you should be really getting right! There are certainly more, but these stick out in my mind.

3. IPv6 is coming

If you haven’t already started looking at IPv6, you should. Even though there are billions of valid IPv4 addresses, a lot are wasted by the way they’re carved up so there won’t be enough to go around. The predictions of doom get revised by the week, but at the very least the protocols themselves are long overdue for a makeover, and you should get ready sooner than later/

IPv6 includes some considerable improvements, the most obvious and famous is the gargantuan address size, so big we have to dump it down to images like addressing every grain of sand of every beach on the planet.

The big benefit here is that address spaces virtually as large as the entire IPv4 space can be assigned to single countries, and over-provisioning of the space is a key factor in deciding how to carve it up. Internet routers have a lot of work deciding which of the myriad paths is right for traffic, and by dividing the space into these huge units, the routing tables can become much, much smaller, allowing the Internet to continue it’s amazing rate of expansion.

But the address space is only one of the improvements. Considerable work has been done to ensure IPv6 networks just work. One of these innovations is the creation of link-local addresses, a form of DHCP, and Router Solicitation. The task of configuring your devices has been moved from your centralised or distributed DHCP server to the devices that know your network the best: your routers.

IPv4 evolved from the first networks mostly when 256kbps was FAST! The protocols have been extended and augmented with things like Quality of Service, IPSec and all kinds of other solutions for secure (and plain) tunnelling. This has resulted in a confusing array of features and incompatibilities.

IPv6 includes a lot of these as standard (IPSec is now mandatory), and improves on others. QoS is vitally important for letting your routers know that your VoIP conversation is much more important than downloading your iTunes purchase, and IPv6 handles these decisions much more intelligently and consistently. Each part of the data packet (IP header, IP payload, TCP/UDP payload, and frequently the application itself) is also checksummed to detect errors, and each layer adds its own checksum, so IPv6 assumes these problems will be detected higher up in the protocol stack and does away with its own layer, further increasing speed.

You should even be able to request addresses for your entire organisation that are all internet-valid, doing away with RPIPA-type addressing (as I mentioned in my previous post here). How organisations deal with the change is still to be seen, but I sincerely hope NAT dies the death it deserves. More on this in my later article, NAT is not a Firewall.

Not all ISPs route or offer the protocol yet, nor do most Internet services, so don’t expect your Internet connection to be switched over any time soon. Versions of Windows from Vista and Server 2003 onwards (XP/2000 has limited support) now including IPv6 out-the-box running gladly alongside the IPv4 stack, you’re free to experiment and explore.

These are challenges you’ll be facing before long, so getting to grips now is well worth the effort.

Previous: 2. Subnets and Private IP space
Next: 4. Disable NetBIOS

Getting IP Right in Windows: 2. Subnets and Private IP space

Networking in Windows is deceptively easy. The level of development Microsoft has achieved to make it so is quite considerable, and I contrast it here with the amount of tweaking required to get Unix services off the ground.

That said, a well-implemented IP structure is the cornerstone of any enterprise (or even serious home) office deployment. I’ve composed a series of five articles on topics you should be really getting right! There are certainly more, but these stick out in my mind.

2. Subnets and Private IP space

The IP address space is global, centrally controlled and handed out to the bigger ISPs and national controlling bodies in big chunks, and broken into progressively smaller groups to hand out to their customers.

Knowing how to carve the IP address space into networks, subnets, supernets, using broadcast domains and multicasts can be readily understood with a little thought, is certainly very logical, and a bit of planning up front can save you a lot of headache down the line. A great graphical explanation can be found here, and a famous graphical map of world’s IP space is here.

If you’re not responsible for the network design at your organisation, have a chat with the guy who is to understand the principles, how it is expected to evolve over time and what you can expect as it does.

IPX was all the vogue in Novell’s heyday, and it is still a neat protocol, able to route across a vast network, but without a central registry for network numbers it was easy to find conflicts, and certainly inappropriate for Internet-type deployments, hence the rise of IP in the corporate space.

I worked at a 2,500+ user company that had a randomly chosen Class-A IP network to deploy internally, at the time I discovered these numbers had officially been assigned to the People’s Republic of China. Thankfully, none of the business interest lay there, but getting these overlapping networks to communicate had there been a need would have been a tedious task.

A special class of IP address are the Reserved Private IP Addresses (RPIPA), and if an Internet router sees traffic from or to these, it is simply discarded, so you can be certain you won’t conflict with someone on the Internet and deploy them as you like!

192.168.0.0/16 is probably the most well-known, and the two most common (192.168.0.0/24 and 192.168.1.0/24) are used almost universally in the default configuration of home routers. Since home networks rarely get integrated with others, this tends to work just fine.

Unfortunately, this also means that a lot of inexperienced network engineers use these as a default. In fact windows XP Internet Connection Sharing (ICS) requires that the internal interface receive the 192.168.0.1/24 address, no other will do, and they’re so common that very little attention is required to remember them.

But that’s the problem. This can lead to big headaches when two sites with these common subnets do want to communicate, from simple VPN access to your company network to handling a merger and linking it up to your network. There are two other ranges to choose from (172.16.0.0/12 and 10.0.0/8) offering literally hundreds of thousands of the good-old 24-bit subnets, so get creative.

A well-implemented DNS (as I wrote about earlier) will mask your numbering for day-to-day tasks, DHCP keeps track of the pool of assigned addresses, and if you're managing a larger network with different WAN links and routers, you should be documenting it in a coherent design anyway, so there isn’t much of a reason not to.

Previous: 1. Understand DNS
Next: 3. IPv6 is Coming

Putting an MCITP in its place

I have noticed that the new raft of credentials from Microsoft don’t necessarily make sense to folks, especially those that are already familiar with the old set of MCSE-type credentials. I mentioned to some friends that I've got the “new MCSE”, a lot of them got it, but it dawned on me that this is, in fact, a field of some confusion. A quick search on Google came up with one fault, and that is how this new credential relates to the ones we (certainly I) already know.

The point of any credential, be it Cisco, VMware, Microsoft or embroidery is to show to an external party that you are qualified in a particular field of endeavour. This was quite plain with Microsoft’s old regime, the Microsoft Certified Professional (MCP), and the Microsoft Certified Systems Engineer (MCSE), as well as the Microsoft Certified Database Administrator (MCDBA). However, as Microsoft branch out into new fields and offer solid, integrated products in fields not entirely related to Windows Server or SQL Server, the approach of cobbling together a new acronym for a new product or role is unwieldy – imagine the Microsoft Certified System Centre Engineer – MCSCE??

So, what is the transition?

MCP –> MCTS

The first Microsoft certification I got in 1997 was an MCP: Windows 95. This showed that, according to Microsoft, I was competent in installing, administering and troubleshooting Windows 95. I remember just how proud I was that day.

The problem though is that the term, “Certified Professional”, encompasses both the specific credential and the entire field of Microsoft certified persons, so is not entirely appropriate. “Technology Specialist” on the other hand, clearly shows what the candidate is trying to demonstrate, that he knows his stuff on a particular product. This bit is key, a specialist in SQL Server configuration is not necessarily a specialist in database development or administration, and in larger organisations the roles are very clearly separate. The MCTS credential clearly segregates say, an application server specialist who can administer web applications, from the server network specialist who will hook it up to the various internal and external parties accessing it.

MCDST/MCSA/MCSE/MCDBA –> MCITP

A big failing of the old MCSE credential was the elective system. While Microsoft may introduce the idea in the future, I sincerely hope not as it adds doubt and confusion to the mix.

I hold an MCSE on Windows NT 4 (incorporating the MCP on Windows 95 I mentioned above). It included two “elective” exams from a list of many more, specifically TCP/IP Networking and Exchange Server 5.5. This means I need to explain to anyone asking just what kind of MCSE I’ve achieved. This was partially remedied in the 2003 track to include an MCSE: Messaging credential, but no such moniker exists for a SQL Server specialist.

The phrase “Systems Engineer” was especially limiting, since it implies an ability to design and implement server infrastructure centred on Windows Server. While that is indeed my own focus, it is of little use to someone specialising in monitoring and management systems, or even the venerable desktop support guru. While the DBA and the Desktop Support guy had their own acronym (MCDBA and MCDST respectively), I certainly don’t want to have to memorise the ever-growing list as a hiring or support manager.

By asserting that someone is an IT Professional in a named field, it indicates a proficiency in a technology set rather than one product. It also narrows the competency; while an Enterprise Administrator demonstrates competency in designing and implementing infrastructure from SANs and Terminal Services down to the desktop, the Server Administrator credential is more focused on those with competency in Windows Server itself.

These credentials are not easy to come by, and are especially hard if the individual has no relevant experience in the real world.

While the plethora of MCITP credentials may seem like a dilution of the fairly focused MCSE, it offers the opportunity for many more product specialist to demonstrate their competency in their field, with a credential on a par with the more established Systems Engineer we’ve come to know.

MCM/MCA

Now we get to the good stuff. The Microsoft Certified Master and Architect credentials are not for the faint of heart or newbies. The intensive certifications are for those with five or more years experience leading complex design, implementation and migration projects and a demonstrated history as a technology leader and expert. Standing up in front of a panel of recognised experts purporting to know your stuff is a daunting proposition, probably even for a few of the members of the very panel you’d be standing before.

For anyone claiming to be hot stuff on the range of Microsoft products, services and solutions, this is where you should be aiming. If you’re already that good, convincing your company to stump up for the three weeks training in Redmond for the MCM should be no effort, and I look forward to getting to that level myself.

Virtualisation of basic services

So, virtualisation is everywhere. It departments are running more and more of their workloads on virtual hardware, and in the data centre that’s a big thing. It’s an accepted fact that most servers in an organisation run at near idle all day long, and even core systems like directory and file servers barely crack a sweat serving their users. For small and medium-sized entities (yes, they have datacentres, just not raised floor, air-conditioned ones like the big guys) this is a problem.

I was asked a while ago why I enabled file compression on my laptop, and seriously considered it on file servers. A few years later another query was why I enabled full-disk encryption on the same piece of kit. Doesn’t it affect the responsiveness?

The simple answer is yes. And no.

CPUs, memory and disks today are so, so fast that there’s not much they can’t do for an individual user. when asked what laptop a person should buy, I recommend the pink one if it’s a girl asking, and jet-black for the guys. Seriously, laptops today can wipe the floor of a desktop from five years ago, and last I checked we’re still doing the same things as back then: browsing the web, writing emails, laughing at cats. If you’re a developer, gamer, or serious showoff, you probably have a dedicated desktop for the hardcore tasks, and the point remains: Moore’s law has seen to that.

Of course, anything you put in the path of data from the disk to CPU/memory is going to slow things down, but this comes back to the more important point: Do I notice?

When I’m running a compile job, chances are the disk’s head movements trying to find each source file and library is going to be the first bottleneck. Then, if it’s a biggie like mplayer, the CPU is going to be loaded for a while doing the compile. Encryption? Oh, I didn’t even notice it’s on.

This brings me to my point, with uber-powerful, multi-core servers cheap as chips (e.g. HP ML110 G5, Dual-core 2.8Ghz Core 2, 1G RAM, 250G SATA for 350 ex tax, HP.nl website 2009-12-07), do I really need to treat these things like fragile porcelain ware?

So, given a department of 100 users in a remote office, where I need to provide local AD, file services, updates, VPN to head office, backups, what’s to stop me buying a cheap server, loading it up with RAM and disks and virtualising all functions in one (or a few), neat little grey cubes with a reputable brand name on the front?

Now here, I’m thinking of Linux as the hypervisor, but that’s my flavour of choice. Something as simple as AD is a good case, since it’s by definition a replicated, resilient database so failures are not a big impact (authenticate somewhere else), and doesn’t do much in the day. It’s also advised to segregate roles on Windows servers, and for resilience perhaps have a second box if you’re paranoid . That’s one, maybe two little cubes that I can’t share for other functions. And it’s going to consume a few gigabytes of disk space, and even less RAM. A whole cube, just for you? Share!

Linux virtualisation has come a long way, and I consider it to be solid, fast and these days wonderfully easy to administer remotely. KVM, as I use, is just another OS process, no funky drivers or installations and since it runs on a full (and standard) instance of Linux, all the management tools for networking, storage, logging and troubleshooting are available.

One simple example: since all traffic traverses the host’s virtual network switch, all traffic can be inspected by a packet sniffer as powerful as Wireshark. No drivers in Windows, no strange binaries. If everything’s virtualised on one box, ALL the traffic is visible for troubleshooting in one spot. Neat!

Another, consider three servers, each requiring 100G storage. That’s probably six disks (minimum) if you’re doing things right (RAID-1). One one hypervisor host, that’s four 100GB disks (RAID-5), only two if you go the 500G route (RAID-1, plus a spare I suppose). Now I’m also a fan of software RAID. My case above stands, there are other factors more likely to impact performance, no least the client’s ability to pull data fast enough from a file server.

So, Now I’ve one box with my local services running in VMs. Full-disk encryption? Well, Windows 2008 has BitLocker, with all the TPM and USB key requirements. How about logging in to the host remotely via SSH, one of the most bulletproof protocols for remote management, and enable the encryption remotely. With a script. Not one byte in plaintext, not even the partition table. Break that, evil hardware theif!

Snapshotting? check
SNMP/performance monitoring and alerting? check
Hardware support? check
Virtualising legacy systems? check (especially MS-DOS on pesky 64-bit Windows)

Worth a think