In 1997, a HP 9000 engineer wouldn't blink telling about a server that had been running continuously for over five years.I found this remarkable at the time, and couldn't imagine a Windows server lasting that long. I have moved on, and frankly expect my Windows servers to survive that long today. Very few share this position, and I'm trying to find out why it's so lonely on this side of the fence.
1. But I Have to Reboot My Own Windows System all The Time!
I've mentioned before how Windows makes you lazy. One of the great things about Microsoft Windows as a platform is that software developed on a $500 workstation can be installed on a $50,000 server and probably work without problems. Of course, getting your home-brew software to scale is a different matter, but you get the idea: One platform, different size.
Almost every Windows engineer cuts their teeth on Windows at home, and this informs their experience and expectations of the platform. Like everyone I get tired of the bogging down after a few days/weeks/months uptime and reboot just to clear things up, but that's my fault and not Windows.
I'm lazy.
Typically, I'm running browsers, office suites, anti-virus, any number of games, and install new stuff roughly once a fortnight. Flash, Java and Windows Update are constantly pestering me to reboot after updates. I've even been the one to reinstall completely after a year to see the wonder of a zippy start-up and responsive GUI, only to have it slowly crawl as I add functionality (including those games). Happily, my Windows 7 installation has lasted two years by now with no significant falloff in responsiveness, so that's getting much better, and I only power down/reboot of my own volition when I'm fitting lights and need mains power off - even then it's more likely to be a hibernate.
Servers are not workstations. Any good enterprise has controls for how changes are made to IT systems, and even simple patching requires testing and approved windows to take the system down and update it. In my experience a server will undergo a major overhaul at most twice in its' operational lifetime, and organisations with exceptional controls have zero - new version? New server!
A good server (and I think of Windows Server 2003+ as good servers) will run for decades given quality power and no moving parts. Of course hardware fails, but Microsoft have put in man-decades to get Windows to handle routine changes without downtime. I remember Windows NT 4.0 needing a reboot for an additional IP address. Modern versions of Windows can hot-plug an entire NIC (physically) without a blink, though admittedly I've never actually encountered anyone who uses the facility.
If an engineer merely mentions that, in their experience, Windows needs rebooting I question their experience. I mean it: I question their experience!
Windows is solid, and I can recall only one confirmed bug where Windows will fail (actually, begin to fail, an outage is not a certainy) for the simple factor of running continuously for a given time. When someone speaks of a memory leak that has caused Windows to run out of (insert wooly term here), again I question their experience and the quality of the software/vendor driver code. I've stopped blaming Microsoft.
When I run my applications on Windows Server and, more importantly, when I am paying someone to manage those systems for me, I expect them to have faith in their products and promise me server availability. Rebooting breaks availability.
Previous: Why You Can't Reboot my Server
Monday, 7 October 2013
Why You Can't Reboot My Server
When I was an on-site server engineer in 1997, I stood next to a HP
9000 engineer waiting for a SCSI hard drive at our parts depot, and we
got chatting about his next work order: He was off to install a tape drive. I
asked him what the new hard drive had to do with it, and he mentioned
that the server in question had been running continuously for over seven years, and at
least one drive was likely to get stuck and refuse to spin again once
he turned the frame back on.
I found this remarkable at the time, and couldn't imagine a Windows server lasting that long. I have moved on, and frankly expect my Windows servers to survive that long today. Very few share this position, and I'm trying to find out why it's so lonely on this side of the fence.
In this series of posts, I'll be looking at the most common complaints from Windows engineers and administrators they feel are adequate to justify rebooting servers, either as (or instead of) a diagnostic step, on a schedule that can best be described as arbitrary, or even artificially to apply fixes for problems the system doesn't have.
In this series:
Part 1: But I Have to Reboot My Own Windows System All the Time!
Part 2. Windows Just Isn't That Stable
Part 3. Console Applications, Java, Batch Files and Other Red Herrings
Part 4. Windows Updates and File Locking
Part 5. Nobody Ever Runs a Server That Long
Part 6. It's OK, the Resilient Partner Can Take Over
Part 7. I Don't Think You Understand What A Server Is
I found this remarkable at the time, and couldn't imagine a Windows server lasting that long. I have moved on, and frankly expect my Windows servers to survive that long today. Very few share this position, and I'm trying to find out why it's so lonely on this side of the fence.
In this series of posts, I'll be looking at the most common complaints from Windows engineers and administrators they feel are adequate to justify rebooting servers, either as (or instead of) a diagnostic step, on a schedule that can best be described as arbitrary, or even artificially to apply fixes for problems the system doesn't have.
In this series:
Part 1: But I Have to Reboot My Own Windows System All the Time!
Part 2. Windows Just Isn't That Stable
Part 3. Console Applications, Java, Batch Files and Other Red Herrings
Part 4. Windows Updates and File Locking
Part 5. Nobody Ever Runs a Server That Long
Part 6. It's OK, the Resilient Partner Can Take Over
Part 7. I Don't Think You Understand What A Server Is
How to Make Your Customers Feel Like Meat in a Tube
Few things annoy me more than web-based forms for initiating customer contact. My experience of them ranges from poor to dismal, and even when I point out that I expect companies to fail in their response I am rarely surprised by brilliance (or even adequacy).
The first problem with these forms is actually the result: Your enquiry ends up not as an e-mail for a person, but a record in a database. Some forms are worse than others in betraying this, but if you even have to select your company size or decision-making company role you can be sure you're being slotted into a Customer Spamming Service machine.
From there, around four out of five responses make no reference to your original query. Unlike e-mail, where you can save your initial contact in your Sent folder, and typically hiting Reply generates a new mail on top of your original one, the first response you receive almost always has no history, so you're left scratching your head wondering if you really forgot to mention your product's model number, even when you remember having to look up the unicode for the unnecessarily accented é in the product name.Whether a human typed out your reply, selected a form response or some machine logic matched your keywords to information already available in the FAQ, I will offer odds, without knowing who the company is, that the original question is not included for reference.
It's a pain to fill in forms like these repeatedly for each individual question, so you might be tempted to put more than one question in your query. Beware traveller - the company will choose which answer most closely matches their prepared form responses and send that to you, regardless of the amount of prominence you try to give to the one you really need answered first.
Errors on the form? How about not residing in the US so skipping the "state" field, only to be told the field is mandatory. OK, I live in Wyoming, Netherlands. Ah, the form now tells me having a state filled in outside the US in an invalid choice? Check the dropdown - yep, only US states available and no way to not pick one. Don't bother complaining about the logic in your actual request - you see, the people choosing stock responses to send that don't adequately deal with your query, they're in no way connected with the end of the sausage maker that ruins your customer contact experience from the start. They just turn the handle.
While sending an enquiry to a prominent software vendor, I happened to have NoScript turned on and found the form broken beyond use. This is simply not justifiable. Oh well, I'll enable the site for JS, but lo! The form fails to complete again. This time it is because a piece of code from a marketing firm has not arrived. So prominent, it even has the name market in its' name - answering my question vs completing my digital profile for a third party: Which do you think they care about most?
All this from an IT Security company, that sells products to control mobile phone policies to stop users from doing things like installing untrusted software that sends their data to unknown parties, without telling you.
Why am I running a marketing company's JavaScript to collect my personal information to initiate an evaluation of your products?
I am just meat in a sausage to you people, aren't I?
The first problem with these forms is actually the result: Your enquiry ends up not as an e-mail for a person, but a record in a database. Some forms are worse than others in betraying this, but if you even have to select your company size or decision-making company role you can be sure you're being slotted into a Customer Spamming Service machine.
From there, around four out of five responses make no reference to your original query. Unlike e-mail, where you can save your initial contact in your Sent folder, and typically hiting Reply generates a new mail on top of your original one, the first response you receive almost always has no history, so you're left scratching your head wondering if you really forgot to mention your product's model number, even when you remember having to look up the unicode for the unnecessarily accented é in the product name.Whether a human typed out your reply, selected a form response or some machine logic matched your keywords to information already available in the FAQ, I will offer odds, without knowing who the company is, that the original question is not included for reference.
It's a pain to fill in forms like these repeatedly for each individual question, so you might be tempted to put more than one question in your query. Beware traveller - the company will choose which answer most closely matches their prepared form responses and send that to you, regardless of the amount of prominence you try to give to the one you really need answered first.
Errors on the form? How about not residing in the US so skipping the "state" field, only to be told the field is mandatory. OK, I live in Wyoming, Netherlands. Ah, the form now tells me having a state filled in outside the US in an invalid choice? Check the dropdown - yep, only US states available and no way to not pick one. Don't bother complaining about the logic in your actual request - you see, the people choosing stock responses to send that don't adequately deal with your query, they're in no way connected with the end of the sausage maker that ruins your customer contact experience from the start. They just turn the handle.
While sending an enquiry to a prominent software vendor, I happened to have NoScript turned on and found the form broken beyond use. This is simply not justifiable. Oh well, I'll enable the site for JS, but lo! The form fails to complete again. This time it is because a piece of code from a marketing firm has not arrived. So prominent, it even has the name market in its' name - answering my question vs completing my digital profile for a third party: Which do you think they care about most?
All this from an IT Security company, that sells products to control mobile phone policies to stop users from doing things like installing untrusted software that sends their data to unknown parties, without telling you.
Why am I running a marketing company's JavaScript to collect my personal information to initiate an evaluation of your products?
I am just meat in a sausage to you people, aren't I?
Wednesday, 18 September 2013
Streetview and WiFi - Courts Need Some Education
I'm hanging my head in my palm in a manner not unlike the Jean-Luc Picard meme today, after reading a decision by a court in the United States. EFF has a great article summarising the effects, but I'd like to expand and go into the cause too.
So Google drove around with an antenna on their Street View cars for a few years, sniffing for wireless networks. This is very useful if you'd like to know your location but can't get a GPS signal, especially on devices with lower quality antennas or a location with poor sky visibility. As long as you have a data connection, you send the local Access Points off to Google's servers, and they will look up the location and feed it back to you. Simple, right?
Well, sniffing for access points (AP) is ridiculously simple. Your phone does that every time you look at a list of networks in your location, by looking for a special network frame called a beacon, which a regular access point send out roughly ten times a second. It's crucial to how WiFi works. It uses the same frame type as normal traffic, so even without a beacon you can still see at least the presence of traffic, and the MAC address associated with the AP. If the network is unencrypted, your WiFi network card automatically accepts those traffic frames too, then has it discarded by your kernel because it is not destined for your computer/smartphone/tablet etc ("your device").
I've done sniffing myself, in a practise known as "war driving". It sounds ominous, but it's also a very interesting excercise, for which I purchased a specific Atheros WiFi NIC, thanks to their products having excellent Linux support. Hook up a GPS receiver, and just go places. The software figures out where you are, looks at the list of APs nearby and pins them to a map. The problem here is that simply enabling your card to listen for APs does cause your system to store those traffic frames, since they are usefuol for determining the IP range in use on that network. Note I'm not trying to use those networks, I'm just interested in seeing how they're being used.
Now comes the interesting part: The court decided these signals being intercepted were not "radio communications" (despite being carried by photons) for the purposes of a legal interpretation, not "readily accessible to the general public" without "sophisticated hardware and software", and finally "most of the general public lacks the expertise to intercept and decode payload data transmitted over a Wi-Fi network".
On each point:
I'm used to seeing courts being out of touch with reality, especially in computing cases, but this is beyond unreasonable. I have no doubt Google had zero intention of capturing user detail such as e-mails, usernames or passwords (why would they, they run an e-mail service?), and are now being prosecuted for users' inability to secure their own networks.
But mostly: Physics!
So Google drove around with an antenna on their Street View cars for a few years, sniffing for wireless networks. This is very useful if you'd like to know your location but can't get a GPS signal, especially on devices with lower quality antennas or a location with poor sky visibility. As long as you have a data connection, you send the local Access Points off to Google's servers, and they will look up the location and feed it back to you. Simple, right?
Well, sniffing for access points (AP) is ridiculously simple. Your phone does that every time you look at a list of networks in your location, by looking for a special network frame called a beacon, which a regular access point send out roughly ten times a second. It's crucial to how WiFi works. It uses the same frame type as normal traffic, so even without a beacon you can still see at least the presence of traffic, and the MAC address associated with the AP. If the network is unencrypted, your WiFi network card automatically accepts those traffic frames too, then has it discarded by your kernel because it is not destined for your computer/smartphone/tablet etc ("your device").
I've done sniffing myself, in a practise known as "war driving". It sounds ominous, but it's also a very interesting excercise, for which I purchased a specific Atheros WiFi NIC, thanks to their products having excellent Linux support. Hook up a GPS receiver, and just go places. The software figures out where you are, looks at the list of APs nearby and pins them to a map. The problem here is that simply enabling your card to listen for APs does cause your system to store those traffic frames, since they are usefuol for determining the IP range in use on that network. Note I'm not trying to use those networks, I'm just interested in seeing how they're being used.
Now comes the interesting part: The court decided these signals being intercepted were not "radio communications" (despite being carried by photons) for the purposes of a legal interpretation, not "readily accessible to the general public" without "sophisticated hardware and software", and finally "most of the general public lacks the expertise to intercept and decode payload data transmitted over a Wi-Fi network".
On each point:
- It's radio. Learn physics. The lower court's opinion that the law covered "predominantly auditory broadcast" is an inference form the court; 18 USC § 2511 mentions audio only for satellite transmission, and only then to describe an audio channel used as a carrier for digital communications therein. If I speak ones and zeros into my walkie-talkie, does this suddenly become digital communications? Your meaning is divorced from reality, and I think it suits an agenda instead of fact.
- If I walk down a street, glancing into shops as I go, and see a person in one shop/office/etc handing a big photo with a red X through it to another person, with person in photo turning up dead and either of the two persons implicated at trial, I can't be prohibited for testifying because I got information not "readily accessible to the general public"(i.e. they were not standing on a street in plain view); a police officer seeing the same thing under the same circumstances is not prohibited from using the information for lack of a warrant or probable cause - it still doesn't affect the fact that photons bounced off the photo and got interpreted by "specialised hardware " (eyes) and "specialised software" (brain). Nobody's privacy is invaded, but information was blasted into the street nonetheless.
- WiFi NIC cost: $10 (shipping costs vary), down to as low as one. These devices are sophisticated, don't get me wrong, but then most computers are mind-numbingly powerful today. Linux kernel cost: nothing (download costs may vary, but is unlikely to make you hit your ISP's download cap), easily installed in an array of distributions. Laptop cost: Variable, but if you've got one lying around you already have your solution.
- Interception is easy stuff. Decoding is easy stuff, and your device "decodes" it as part of its' primary function.
- A further point the court asserted is that regular (AM/FM) radio communications can be received miles away, versus WiFi that "fail to travel far beyond the walls" of a location. Again, physics. Oh, and define "far" - my balcony is less than 15 meters from my AP and my signal comes and goes, while the street is 30 metres away (in the opposite direction) and I still get an association (but not much throughput) fairly reliably. I despise vagueness in court proceedings.
I'm used to seeing courts being out of touch with reality, especially in computing cases, but this is beyond unreasonable. I have no doubt Google had zero intention of capturing user detail such as e-mails, usernames or passwords (why would they, they run an e-mail service?), and are now being prosecuted for users' inability to secure their own networks.
But mostly: Physics!
Thursday, 24 January 2013
Dawson College: What Island Are You On?
I've been viewing the growing story about Ahmed Al-Khabaz, a Computer Science student at Dawson College in Montreal, Canada, who was expelled for running a security scan against their public web presence to discover if a flaw he found was resolved. I stumbled on a subsequent interview with prominent IT Security professional Chris Wysopal. I hadn't heard of him, but when I saw he was previously associated with l0pht Heavy Industries my eyes snapped open.
This guy has credentials, and I don't know of an IT professional active around 1997 to 2003 who hadn't heard of, or actually used, l0pthcrack, often to solve real-world problems. First and foremost a password auditing tool it can be used maliciously, but the so can a toaster oven. It is a piece of code art: Necessary, useful and (at the time) industry-shaking.
White Hat hacking is a tricky business. Even I've done it, against a bank no less, fully in the knowledge that I was doing something the system owners would be very unhappy about. In some cases it can get you arrested. I was was pleased with the results when my concerns were taken seriously and fixed fairly quickly. I've worked in financial services companies and know their software release process is iceberg slow so this was very reassuring. There's one thing Mr Al-Khabaz and I both know that drives thousands around the world to the same end: I'm at risk.
Dawson College is hand-wringing and special pleading: "
"behavior that is unacceptable in a computing professional" has the company whose software flaws he exposed taken it upon themselve to pay for his further education?
Academia is often seen as disconnected from reality; some lines of research beggar belief, and the same could be said of Computer Science. I've met a few graduates who arrive in the IT industry ill-prepared, full of theory of operation and design but unable to command a command-line. No matter what their actual instruction is, a critical point they need to learn is that the Internet is a hostile place. It is also a collaborative place, where FOSS abounds and Creative Commons is richly rewarding. Poking around is the norm, and if this college is telling their students that they are to accept their instruction blindly without considering real-world implications, or use those skills to explore, then they don't deserve to be associated with the term Higher Education.
They may perhaps be able to educate Code Monkeys, but thinking professionals able to design and protect systems that impact their lives? Not really.
This guy has credentials, and I don't know of an IT professional active around 1997 to 2003 who hadn't heard of, or actually used, l0pthcrack, often to solve real-world problems. First and foremost a password auditing tool it can be used maliciously, but the so can a toaster oven. It is a piece of code art: Necessary, useful and (at the time) industry-shaking.
White Hat hacking is a tricky business. Even I've done it, against a bank no less, fully in the knowledge that I was doing something the system owners would be very unhappy about. In some cases it can get you arrested. I was was pleased with the results when my concerns were taken seriously and fixed fairly quickly. I've worked in financial services companies and know their software release process is iceberg slow so this was very reassuring. There's one thing Mr Al-Khabaz and I both know that drives thousands around the world to the same end: I'm at risk.
Dawson College is hand-wringing and special pleading: "
"behavior that is unacceptable in a computing professional" has the company whose software flaws he exposed taken it upon themselve to pay for his further education?
Academia is often seen as disconnected from reality; some lines of research beggar belief, and the same could be said of Computer Science. I've met a few graduates who arrive in the IT industry ill-prepared, full of theory of operation and design but unable to command a command-line. No matter what their actual instruction is, a critical point they need to learn is that the Internet is a hostile place. It is also a collaborative place, where FOSS abounds and Creative Commons is richly rewarding. Poking around is the norm, and if this college is telling their students that they are to accept their instruction blindly without considering real-world implications, or use those skills to explore, then they don't deserve to be associated with the term Higher Education.
They may perhaps be able to educate Code Monkeys, but thinking professionals able to design and protect systems that impact their lives? Not really.
Wednesday, 23 January 2013
How Important is MariaDB? Let's test the fork with butter.
MySQL has interested me for quite a long time. I first came across it in 2000 when trying to find a better way to analyse the contents of a 20,000 user Active Directory and needed more relational DB-stuff than Microsoft Access could deliver and cheaper than SQL Server (wow MSDE was terrible). I was deeply impressed (though probably because I was easily impressed back then) with the performance and cross-platform support, and ever since it's been around my life.
I currently use it for my XBMC and Logitech Media Server (SqueezeBox) media databases, as the back-end for my Gallery3 site, and other ad-hoc databases whenever I need to crunch data. Before my 64-bit processor created a new ISA that ensured a reasonably complete instruction set, it was a favourite of mine for optimising binary compiles over the stock i386 build supplied by most distros, but more for interest's sake than actually squeezing performance for any measurable benefit.
MySQL AB was of course the owner of the copyrights and code and opted for a relatively unique license, both proprietary and open. As the owners of the code, they could choose to do this, but anyone trying to make a buck out of the code was obliged to release their modifications. Now that Oracle (through their acquisition of Sun, who acquired MySQL AB) have that right, the open-source community is in a bit of a fluster. Can we trust a corporate giant with custody of the code that runs a significant fraction of the Internet's websites? The answer is slowly coming down on the side of "no".
Oracle (and others, and unsurprisingly) is being guarded about bugs and fixes. Stories of vendors forcing customers into NDAs before even admitting bugs exist, hiding bugs from other customers, and silently including fixes are common. It's face-saving. Andy Grove's "Only the Paranoid Survive" starts off with how Intel hoped to keep their Pentium FPU bug quiet while they implemented a workaround simply smacks of arrogance. While it doesn't yet seem Oracle are trying to hide any actual code and still supply source, MySQL has historically had test cases for bugs published alongside them to protect against regression and anyone can run the suite on their installation to verify code quality. Not only are they apparently now keeping some cases secret, they are also not clearly marking which code updates fix bugs they are refusing to publish.
This is not how open-source works, but I don't agree with the prevailing rationale. RedHat came into the firing line for being less than open they handled a code signing infrastructure breach, but in that instance I support the way they behaved as it was not their source they concealed, rather their own systems and controls that were embarrassingly compromised. They have shareholders, and revealing too much would have cost them. Oracle too have value invested in their products and would like to keep flaws hidden. This is not nefarious, it's capitalism.
MySQL as a product is different, no matter who owns it. It is very closely tied to the spirit of the open-source movement, being both highly regarded for performance and features, and for the competition it gives proprietary offerings. For Oracle to claim that ground back is entirely within their right, but the edge is gone. The most ardent supporters and influencers of purchasing are not happy and a slow exodus may be starting.
So Fedora and Wikipedia are both contemplating pulling out. The MariaDB fork has all the features and more, is fully open in the original spirit of the project, and is attracting attention including mine. I have no idea how easy it will be to do the fabled "drop-in replacement" every source claims is possible but I feel ethically compelled to leave MySQL in the dust. I have a server that runs my digital life and it is a conscious choice to run on open software only and it has not been easy, but as an experiment and learning tool it is invaluable.
The great thing about open-source is anybody can fork. I can clone a source and apply my changes as I like, but the moment I try to give it to anyone else (especially selling the result) I have to disclose my whole body of work. This can lead to some confusion as the early days of Linux showed, but in the end the market weeds out the under-performers and delivers better products through sheer market forces. MariaDB seems to be that winner.
I do know one thing: testing the transition is going to be a breeze: After switching from Fedora to Gentoo four months ago, I rolled the root over to BTRFS (once kernel 3.6 gave me the necessary confidence). Add a distinct IP to the NIC, snapshot, chroot, and I've got a clone of my server ready to go in about two seconds without that system-level virtualisation stuff and hideously slow LVM2 snapshots.
Rollback to base for a fresh attempt? Yep, two seconds.
I currently use it for my XBMC and Logitech Media Server (SqueezeBox) media databases, as the back-end for my Gallery3 site, and other ad-hoc databases whenever I need to crunch data. Before my 64-bit processor created a new ISA that ensured a reasonably complete instruction set, it was a favourite of mine for optimising binary compiles over the stock i386 build supplied by most distros, but more for interest's sake than actually squeezing performance for any measurable benefit.
MySQL AB was of course the owner of the copyrights and code and opted for a relatively unique license, both proprietary and open. As the owners of the code, they could choose to do this, but anyone trying to make a buck out of the code was obliged to release their modifications. Now that Oracle (through their acquisition of Sun, who acquired MySQL AB) have that right, the open-source community is in a bit of a fluster. Can we trust a corporate giant with custody of the code that runs a significant fraction of the Internet's websites? The answer is slowly coming down on the side of "no".
Oracle (and others, and unsurprisingly) is being guarded about bugs and fixes. Stories of vendors forcing customers into NDAs before even admitting bugs exist, hiding bugs from other customers, and silently including fixes are common. It's face-saving. Andy Grove's "Only the Paranoid Survive" starts off with how Intel hoped to keep their Pentium FPU bug quiet while they implemented a workaround simply smacks of arrogance. While it doesn't yet seem Oracle are trying to hide any actual code and still supply source, MySQL has historically had test cases for bugs published alongside them to protect against regression and anyone can run the suite on their installation to verify code quality. Not only are they apparently now keeping some cases secret, they are also not clearly marking which code updates fix bugs they are refusing to publish.
This is not how open-source works, but I don't agree with the prevailing rationale. RedHat came into the firing line for being less than open they handled a code signing infrastructure breach, but in that instance I support the way they behaved as it was not their source they concealed, rather their own systems and controls that were embarrassingly compromised. They have shareholders, and revealing too much would have cost them. Oracle too have value invested in their products and would like to keep flaws hidden. This is not nefarious, it's capitalism.
MySQL as a product is different, no matter who owns it. It is very closely tied to the spirit of the open-source movement, being both highly regarded for performance and features, and for the competition it gives proprietary offerings. For Oracle to claim that ground back is entirely within their right, but the edge is gone. The most ardent supporters and influencers of purchasing are not happy and a slow exodus may be starting.
So Fedora and Wikipedia are both contemplating pulling out. The MariaDB fork has all the features and more, is fully open in the original spirit of the project, and is attracting attention including mine. I have no idea how easy it will be to do the fabled "drop-in replacement" every source claims is possible but I feel ethically compelled to leave MySQL in the dust. I have a server that runs my digital life and it is a conscious choice to run on open software only and it has not been easy, but as an experiment and learning tool it is invaluable.
The great thing about open-source is anybody can fork. I can clone a source and apply my changes as I like, but the moment I try to give it to anyone else (especially selling the result) I have to disclose my whole body of work. This can lead to some confusion as the early days of Linux showed, but in the end the market weeds out the under-performers and delivers better products through sheer market forces. MariaDB seems to be that winner.
I do know one thing: testing the transition is going to be a breeze: After switching from Fedora to Gentoo four months ago, I rolled the root over to BTRFS (once kernel 3.6 gave me the necessary confidence). Add a distinct IP to the NIC, snapshot, chroot, and I've got a clone of my server ready to go in about two seconds without that system-level virtualisation stuff and hideously slow LVM2 snapshots.
Rollback to base for a fresh attempt? Yep, two seconds.
Thursday, 19 July 2012
Do the ends still matter?
One of the biggest conflicts of interest in my role as an infrastructure architect (I'm responsible for servers, applications, security, storage etc) is that I have a heavily vested interest in the service they provide. This may sound like a no-brainer, but often I need to find ways to get my traffic to users at the expense of someone else. I want preferential treatment if conflicts with other traffic.
Of course, it's not quite as simple as that. Pragmatically I realise I am competing with others on limited resources, whether it's rack space, hypervisor RAM or network bandwidth. One of the less-known principles underlying the Internet ethos is the End-to-end Principle. Put simply, it's this:
Complexity in any network should be implemented in the end points - the network stacks of communicating nodes and in applications - and not in the network itself. Since standards change any benefit of implementing too much intelligence in the network is quickly undermined with the need to continually match those changes in end-points, as well as legacy issues.
Essentially, the network should be as dumb as possible. Packet comes in, packet goes out, wait for next packet. This is reasonable in principle but different traffic flows require different treatments - VoIP and video streaming protocols prefer low latency and are almost always jitter sensitive, while file transfers can tolerate enormous latencies as long as they are accompanied by high bandwidth. If both types occupy the same link the risk is that the insensitive consumption of one protocol can impact the requirements of another, so QoS arrives.
This is already a necessary evil (though some may object to my use of that label in this context) since we now need to build knowledge of services into the network layer. Large IP transport installations (Internet backbones) handling gigabits of traffic per second literally don't have the time to implement protocols like this as processing adds both cost and latency. This is different from implementing different virtual circuits for different traffic types, in that it forms logic links and is a very common practice with ISPs in the final mile. As far as intelligence on network devices go, this is very low - again packets come in, figure out where packets go out.
A very interesting turn is in the development of new forms of network acceleration. Routers have long been capable of doing in-line compression of data to reduce consumption of a specific link, but this is point-to-point. If an application protocol can truly benefit from compression, this really should be done at the application protocol level (basically above Layer 3) so routers and switches can shuffle packets. A possible side-effect of link compression is to mask real versus usable bandwidth from applications (some portions of a stream may be highly compressible, others not) hindering flow-control algorithms built into TCP.
TCP sessions can be rather chatty, and some applications implement redundant techniques inside their own specification, so some network acceleration can initially make sense. Essentially, new sessions follow a well-known pattern of window sizes and other parameters, so network accelerators intercept packets and simulate the repetitive parts on each end of a link, reducing session setup time. This sounds simple, but now we enter dangerous waters. Next comes protocol caching; I request a file from a file server across the WAN and the contents are cached on my local acceleration appliance so that the user next to me can get a cached copy when her request is made. Again, sounds simple, but to prevent interception and modification the protocol implements signing, so re-generated content requires a re-generated signature. SSL acceleration similarly can be implemented using reverse proxies holding a copy of the private key for the service to extract the plaintext and look for compression and caching opportunities. I've been involved in design and deployment of many reverse proxy and SSL acceleration projects but these were explicitly part of the service.
To accelerate generic payloads the appliance needs to get heavily involved in the infrastructure, either by hosting all SSL keys and being able to masquerade as endpoints. This is where it gets very complex and risky, not only in the dissemination of privileged access (private keys and domain credentials being the most highly-prized items in any network), but also a continual catch-up game to implement these techniques on newer protocols as they are developed. There are also more intrusive techniques such as automatic downscaling of images when using mobile data to browse the web that are subtle but insidious.
Net Neutrality is the overall drive in this direction and correlates with the End-to-end Principle, treating network devices as simplistic and all traffic as equal.Smarter protocols such as (distributed) BranchCache and BitTorrent (yes, BitTorrent is a case for Net Neutrality) reduce redundancy over constrained links and better content intelligence (I explicitly convert all images in my documents to 8-bit PNGs before embedding) are far better strategies. Content distribution networks are an active participant and used by some of the largest providers (both content generators and ISPs) to reduce long-haul bandwidth and improve user responsiveness. HTTP compression is a rigorously defined standard but very sparsely used even on static pages where on-the-fly zipping is unnecessary. I could go on...
When looking into how best to transport content I prefer to let the network do what it does best and work my requirements into it than engineer a network to suit my needs unless absolutely necessary. Where my application is burdensome, I would prefer to engineer the application than throw bandwidth at the problem.
Technology marches, and bandwidth - while logically finite - seems to be keeping up rather well.
Of course, it's not quite as simple as that. Pragmatically I realise I am competing with others on limited resources, whether it's rack space, hypervisor RAM or network bandwidth. One of the less-known principles underlying the Internet ethos is the End-to-end Principle. Put simply, it's this:
Complexity in any network should be implemented in the end points - the network stacks of communicating nodes and in applications - and not in the network itself. Since standards change any benefit of implementing too much intelligence in the network is quickly undermined with the need to continually match those changes in end-points, as well as legacy issues.
Essentially, the network should be as dumb as possible. Packet comes in, packet goes out, wait for next packet. This is reasonable in principle but different traffic flows require different treatments - VoIP and video streaming protocols prefer low latency and are almost always jitter sensitive, while file transfers can tolerate enormous latencies as long as they are accompanied by high bandwidth. If both types occupy the same link the risk is that the insensitive consumption of one protocol can impact the requirements of another, so QoS arrives.
This is already a necessary evil (though some may object to my use of that label in this context) since we now need to build knowledge of services into the network layer. Large IP transport installations (Internet backbones) handling gigabits of traffic per second literally don't have the time to implement protocols like this as processing adds both cost and latency. This is different from implementing different virtual circuits for different traffic types, in that it forms logic links and is a very common practice with ISPs in the final mile. As far as intelligence on network devices go, this is very low - again packets come in, figure out where packets go out.
A very interesting turn is in the development of new forms of network acceleration. Routers have long been capable of doing in-line compression of data to reduce consumption of a specific link, but this is point-to-point. If an application protocol can truly benefit from compression, this really should be done at the application protocol level (basically above Layer 3) so routers and switches can shuffle packets. A possible side-effect of link compression is to mask real versus usable bandwidth from applications (some portions of a stream may be highly compressible, others not) hindering flow-control algorithms built into TCP.
TCP sessions can be rather chatty, and some applications implement redundant techniques inside their own specification, so some network acceleration can initially make sense. Essentially, new sessions follow a well-known pattern of window sizes and other parameters, so network accelerators intercept packets and simulate the repetitive parts on each end of a link, reducing session setup time. This sounds simple, but now we enter dangerous waters. Next comes protocol caching; I request a file from a file server across the WAN and the contents are cached on my local acceleration appliance so that the user next to me can get a cached copy when her request is made. Again, sounds simple, but to prevent interception and modification the protocol implements signing, so re-generated content requires a re-generated signature. SSL acceleration similarly can be implemented using reverse proxies holding a copy of the private key for the service to extract the plaintext and look for compression and caching opportunities. I've been involved in design and deployment of many reverse proxy and SSL acceleration projects but these were explicitly part of the service.
To accelerate generic payloads the appliance needs to get heavily involved in the infrastructure, either by hosting all SSL keys and being able to masquerade as endpoints. This is where it gets very complex and risky, not only in the dissemination of privileged access (private keys and domain credentials being the most highly-prized items in any network), but also a continual catch-up game to implement these techniques on newer protocols as they are developed. There are also more intrusive techniques such as automatic downscaling of images when using mobile data to browse the web that are subtle but insidious.
Net Neutrality is the overall drive in this direction and correlates with the End-to-end Principle, treating network devices as simplistic and all traffic as equal.Smarter protocols such as (distributed) BranchCache and BitTorrent (yes, BitTorrent is a case for Net Neutrality) reduce redundancy over constrained links and better content intelligence (I explicitly convert all images in my documents to 8-bit PNGs before embedding) are far better strategies. Content distribution networks are an active participant and used by some of the largest providers (both content generators and ISPs) to reduce long-haul bandwidth and improve user responsiveness. HTTP compression is a rigorously defined standard but very sparsely used even on static pages where on-the-fly zipping is unnecessary. I could go on...
When looking into how best to transport content I prefer to let the network do what it does best and work my requirements into it than engineer a network to suit my needs unless absolutely necessary. Where my application is burdensome, I would prefer to engineer the application than throw bandwidth at the problem.
Technology marches, and bandwidth - while logically finite - seems to be keeping up rather well.
Subscribe to:
Posts (Atom)