WYCRMS Part 7. I Don't Think You Understand What A Server Is

In 1997, a HP 9000 engineer wouldn't blink telling about a server that had been running continuously for over five years.I found this remarkable at the time, and couldn't imagine a Windows server lasting that long. I have moved on, and frankly expect my Windows servers to survive that long today. Very few share this position, and I'm trying to find out why it's so lonely on this side of the fence.

7. I Don't Think You Understand What A Server Is

It's taken me a long time to find the right words to explain this title, because it's a bit contentious. This is a longer article, but I hope it's of value and I hope expresses my sense of pride and love of computing.

I've explained in previous posts that Windows Server is accessible to those new to IT engineering because it is a simple learning path from the system most of us have in our homes, schools, libraries and elsewhere. Moving to depending on a server to provide service takes different people different amounts of time to get right, and some organisations are more tolerant of slip-ups than others.

There is something that starts to take over. It's not obvious, and since it applies to something commonplace and often the subject of passion for those true geeks, it is also often unrecognised in oneself.

Fear

In no other industry would a consumer or customer tolerate advice such as rebuilding a product, service or transaction because of a minor fault. No mechanic would expect to be paid if they told their customers that the brakes may fail after an hour on the freeway, advising the driver to just come to a complete stop and then move off again, yet this is what we in the (Windows) IT Service industry resort to all-too-otfen. It's a mild paranoia that all things Microsoft (and others, as I will show) are prone to misbehave, and I've been told this explicitly, in writing, more than once from multiple providers after I've requested a feature or function be deployed. Providing vendor documentation in favour of the product's capabilities doesn't sway these types, as they've experienced their own horror stories of staying up until 2am while functionality is restored. I know, I've been in that trench.

But imagine for a moment if you'd gone to a garage affiliated directly with your car's manufacturer. A Ford mechanic providing me the aforementioned brake-fault workaround would be held to account if he also displayed his certification of training or affiliation with Ford Motor. I could challenge him and go to his management to insist that real advice and a fix is provided for my product. If he could show you that his advice was sound my beef would be with Ford selling me an underperforming product. I'd have recourse.

IT engineers seem to think they don't, either in being the place to receive this criticism (hey, they didn't write the software they just run it), or in being able to back up their fears with proof that vendor products in fact don't behave as they would hope. Yet the offices of IT service providers proudly display partner certifications, while engineers with IT credentials flowing of their resumes continue to fear the products that are their livelihood and the foundation for the logos they too show off with some pride. Doublethink indeed.

I am very active in reporting bugs in open-source and even paid-for products, because I expect that the product is only good as those who help make it better. I've already mentioned that failure to even start reporting faults to vendors is negligent, and how engineers should have more pride in their platforms and confidently defend it from detractors with authoritative sources. What I haven't spoken about is the relationship those engineers have with the platform itself.

There is a widely held belief in ICT over a decade old that is demonstrably false: Ethernet autonegotiation is quirky, and potentially dangerous. There was a time this was true, but not for at least five years since 100Base-TX lost it's leading position in datacenter, server and finally desktop connectivity method. Implementing Fast Ethernet (as 100Mb Ethernet was known) needs some knowledge of standards. When the Institute of Electrical and Electronics Engineers (IEEE) published their 802.3u standard for Fast Ethernet, I recall an interview with one of the panel members who stated that it is technically possible to run 100Mbit over stacheldracht (German for barbed wire) and you may have success. He made it clear though that your experience cannot be guaranteed as it is not 802.3u-compliant.

That's the crux: when a vendor states something as a standard, part of a reference architecture, or included in their documentation, they're making a promise.

The section of the new standard dealing with how the two nodes selected the operating speed and duplex setting was, unfortunately, not precise enough and open to some interpretation. Cisco and a few other vendors chose one interpretation while everyone else chose another, and the resulting duplex mismatch is notoriously hard to diagnose, occurring as it does only at moderate load and a ping test over an idle cable will likely succeed. It's insidious, and resulted in the universal abandonment of Autonegotiation in implementations (especially datacenters and core networks).

The problem is, Autonegotiation is not only working well in Gigabit Ethernet (over twisted-pair copper, or Cat5e/Cat6 cabling), it is mandatory. Even network professionals, burnt previously in the 90s and later with Fast Ethernet, advise against turning on a feature that is explicitly required to be a truly standards-compliant implementation, with all the promises attached. A prime reason is that the applicable line is buried deep inside section 28 of the IEEE-802 standard, as amended for Gigabit. It's dry reading...

Gigabit Ethernet was a big jump forward that started to seriously tax memory buses and CPUs like no other iteration before, and includes a highly valuable feature known as a Pause Frame to stop transfers flooding receive buffers and being dropped. This facility is only used if the opposite end cooperates, and the only mechanism to advertise this is autonegotiation.

I've seen an implementation of Microsoft Exchange 2010 come to its knees for lack of Pause Frames, and it is again an insidious failure since packets are only dropped under load, and ping tests and even high-load throughput tests succeed. It is the clinging to an old wisdom without knowing the cause, and then failing to keep up with developments that has caused this issue. Not running with Autonegotiation means you aren't running a standards-compliant Gigabit Ethernet network, and all promises are void.

Not following vendor advice is a bad idea. If the vendor promises a feature that you feel is not ready for primetime then by all means hold off. But if I expect something to work that a vendor promises will work, I don't expect to be told war stories of how this breaks - especially when I last saw that issue, myself, over 12 years ago. It's old thinking, stuck in past fears, and it's stopping you from unleashing your platform's potential. Windows Server especially has become a solid, dependable and performant platform, yet doubts linger and fears cling to dark corners, an uneasiness that is sometimes not even apparent to those harbouring it.

I enjoy reading on the history of computing, and contemplating how modern computers implement both Harvard and von-Neumann architectures depending on how closely you're looking. It's esoteric to speak of privilege rings or context switches, but knowing these things has been of immense help to round out my understanding of computing and gain trust in the models deployed. But the biggest thing I would like to see engineers embrace is this:

The Turing Machine

It's a simplistic representation of any computer, from your old calculator wristwatch to supercomputing clusters: A processor reads instructions from a sequence, implements those instructions with some data, and stores the result somewhere before moving to the next instruction. The next instruction may be a pointer to a different instruction, but all of computing boils down to this concept. there may be more than one processor, and there may be complex layouts of memory, but at its most basic form every computer works this way, and building on your model of a system's internals starts here.

It is deterministic, in that the state after an instruction is performed can be predicted from the initial state. In principal all of computing conforms with this principle, and any unexpected behaviour simply means the initial state was not well-understood enough. It is this mountain that engineers need to climb to truly excel at their profession, and I've met some expert climbers in my time. They have no fear of digging down to each root cause, and unearthing an even deeper root.

Rebooting is not the answer. It indicates a lack of knowledge on cause of faults. It is a sign of an unwillingness to investigate further. Worst, it is a misunderstanding of what your server is, what is meant to do, and the longer you allow that mentality to perpetuate the worse off you will be.

Old tales have value, but they are no substitute for knowledge and verifiable fact. If those facts contradict your experience, investigate, shout at vendors, check your implementation.

But most of all, be proud of your platform, because as obscure as It appears ot be, it is genuinely not that hard if you are willing to do better.

Previous: Part 6. It's OK, the Resilient Partner Can Take Over

Putting an MCITP in its place

I have noticed that the new raft of credentials from Microsoft don’t necessarily make sense to folks, especially those that are already familiar with the old set of MCSE-type credentials. I mentioned to some friends that I've got the “new MCSE”, a lot of them got it, but it dawned on me that this is, in fact, a field of some confusion. A quick search on Google came up with one fault, and that is how this new credential relates to the ones we (certainly I) already know.

The point of any credential, be it Cisco, VMware, Microsoft or embroidery is to show to an external party that you are qualified in a particular field of endeavour. This was quite plain with Microsoft’s old regime, the Microsoft Certified Professional (MCP), and the Microsoft Certified Systems Engineer (MCSE), as well as the Microsoft Certified Database Administrator (MCDBA). However, as Microsoft branch out into new fields and offer solid, integrated products in fields not entirely related to Windows Server or SQL Server, the approach of cobbling together a new acronym for a new product or role is unwieldy – imagine the Microsoft Certified System Centre Engineer – MCSCE??

So, what is the transition?

MCP –> MCTS

The first Microsoft certification I got in 1997 was an MCP: Windows 95. This showed that, according to Microsoft, I was competent in installing, administering and troubleshooting Windows 95. I remember just how proud I was that day.

The problem though is that the term, “Certified Professional”, encompasses both the specific credential and the entire field of Microsoft certified persons, so is not entirely appropriate. “Technology Specialist” on the other hand, clearly shows what the candidate is trying to demonstrate, that he knows his stuff on a particular product. This bit is key, a specialist in SQL Server configuration is not necessarily a specialist in database development or administration, and in larger organisations the roles are very clearly separate. The MCTS credential clearly segregates say, an application server specialist who can administer web applications, from the server network specialist who will hook it up to the various internal and external parties accessing it.

MCDST/MCSA/MCSE/MCDBA –> MCITP

A big failing of the old MCSE credential was the elective system. While Microsoft may introduce the idea in the future, I sincerely hope not as it adds doubt and confusion to the mix.

I hold an MCSE on Windows NT 4 (incorporating the MCP on Windows 95 I mentioned above). It included two “elective” exams from a list of many more, specifically TCP/IP Networking and Exchange Server 5.5. This means I need to explain to anyone asking just what kind of MCSE I’ve achieved. This was partially remedied in the 2003 track to include an MCSE: Messaging credential, but no such moniker exists for a SQL Server specialist.

The phrase “Systems Engineer” was especially limiting, since it implies an ability to design and implement server infrastructure centred on Windows Server. While that is indeed my own focus, it is of little use to someone specialising in monitoring and management systems, or even the venerable desktop support guru. While the DBA and the Desktop Support guy had their own acronym (MCDBA and MCDST respectively), I certainly don’t want to have to memorise the ever-growing list as a hiring or support manager.

By asserting that someone is an IT Professional in a named field, it indicates a proficiency in a technology set rather than one product. It also narrows the competency; while an Enterprise Administrator demonstrates competency in designing and implementing infrastructure from SANs and Terminal Services down to the desktop, the Server Administrator credential is more focused on those with competency in Windows Server itself.

These credentials are not easy to come by, and are especially hard if the individual has no relevant experience in the real world.

While the plethora of MCITP credentials may seem like a dilution of the fairly focused MCSE, it offers the opportunity for many more product specialist to demonstrate their competency in their field, with a credential on a par with the more established Systems Engineer we’ve come to know.

MCM/MCA

Now we get to the good stuff. The Microsoft Certified Master and Architect credentials are not for the faint of heart or newbies. The intensive certifications are for those with five or more years experience leading complex design, implementation and migration projects and a demonstrated history as a technology leader and expert. Standing up in front of a panel of recognised experts purporting to know your stuff is a daunting proposition, probably even for a few of the members of the very panel you’d be standing before.

For anyone claiming to be hot stuff on the range of Microsoft products, services and solutions, this is where you should be aiming. If you’re already that good, convincing your company to stump up for the three weeks training in Redmond for the MCM should be no effort, and I look forward to getting to that level myself.

So why on earth does certification cost so much?

I'm in a foul mood.

Here's what I wrote to my local Sun Education Centre:

I would like to sit the Certified MySQL Associate exam.

On the MySQL website I am informed that the cost of sitting an exam is around $200 plus any local taxes etc (http://www.mysql.com/certification/candguide.html), which should be around €135. However, booking your exam costs €280 incl VAT, DOUBLE the price in the US.

I am not being sent on a course by my company. I have not had learning materials bought for me, nor is anyone other than me intending to pay for the cost of the exam, I am doing this out of my own pocket. I am intending to take this exam to demonstrate my PERSONAL skill at MySQL, and my willingness to use it. I am certified by Microsoft since 1999, and so tend to labelled a Microsoft guy, when in fact I have skills across the IT landscape. Getting a MySQL Associate certification is one
of my ways of doing that.

I'm disappointed the exam is set at such a high price. I took a Microsoft exam this morning, and I paid €119,00 incl VAT for the privelege. It also included use of a virtual lab hosted by Prometric, when the CMA exam is purely multiple-choice. The exam was three hours long, instead of the 60 minutes of the CMA exam.

This means that somehow the CMA exam is around seven times as expensive per hour!

I WANT to demonstrate my passion for platforms (and companies) other than Microsoft, I don't understand why I would need to pay an exorbitant rate to promote your product. Please re-read the last sentence; your company benefits here, for which you expect to be paid a high fee.

If your standard response is something like "the prices are set", then please make sure you also add that your lack of desire to have me certified and promote your products is as clearly stated. Seriously, this is an ENTRY LEVEL exam, 60 minutes, multiple choice. How this can be priced the same as your more advanced products is beyond me.

I wonder what kind of reply I'll get?