Don't Just Tell Me "It's Encrypted"

"It's Encrypted"

These two words are a trigger. They seem innocuous enough, but the devil is in the details, and the details of cryptography are notoriously byzantine. It's tempting to allow ourselves to be soothed by these words, but they fall well short of an assurance.

In 2002, Bill Gates sent a memo inside Microsoft introducing his vision for Trustworthy Computing, a form of which we now know as Trusted Computing (these are distinct). In addition to Security and Privacy, the Availability goal in that memo has since been likened to a light switch. You don't think about whether the power will arrive when you flip that switch, you simply trust that it will. It's hard to see the forest for the trees sometimes, but when you consider the immense complexity and level of engineering in the products we depend on fifteen years later (tablets, laptops, mobile phones, streaming music systems to name a few), it is a remarkable achievement.

The light switch analogy is instructive. Power generation and distribution is a gargantuan field, and consumes spectacular amounts of money to provide you with the simplicity and affordability of that light switch. Similarly, achieving the level of stability and performance in our computing devices has consumed millions of productivity hours and billions of dollars. Privacy, the often forgotten cousin of Security and Availability in the original memo, is expected to varying degrees by different audiences but nonetheless depended on for our daily lives.

"It's Encrypted" sounds like the light switch - problem solved, it's encrypted, I can rest. It is a rare day indeed where that claim is picked apart that I do not find (sometimes significant) issues with the claim. If not done correctly, badly implemented security (but encryption especially) can be worse than no security.

Cryptography is an esoteric field. As I write this, Mozilla is due to start warning users of websites that continue to use certificates signed with the SHA-1 algorithm. This is likely to be Greek to the average user, but to security professionals this is a reasonable step in the march forward, and if naming and shaming is required to prod those last holdouts into the future, then at least users are informed that these providers are not up to snuff. This stuff is ubiquitous, and few users realise that their $150 smartphone is loaded with some of the most advanced cryptography available to the average Joe. They depend on it often more than they know, so interventions can be useful.

The problems are not in the ciphers, and handshakes & protocols are continually improved to counter new threats while enabling new functionality. The biggest headache (and the most frequent downfall of "It's Encrypted" claims) is key management. As Bruce Schneier reminds us in Applied Cryptography: "In the real world, key management is the hardest part of cryptography."

Private keys in PKI are often treated with trivial regard. The most recognisable for administrators would be the PFX, or .P12 file. Incorrectly referred to as a "Certificate file" (a wildly inappropriate shorthand), these few kilobytes are dealt with as arbitrary packages that a system uses to function, as opposed to critical pins in the security apparatus. This type of file (PKCS#12) as designed, is intended for key archival, yet is often sent via e-mail, stored on accessible drives to ease administration and troubleshooting.

On the other hand, Pre-Shared Keys are frequently recorded when they are not strictly required - if a system has issues, generate a new pair, then discard any record, though only a storage corruption would necessitate this, in which case you have bigger problems. TLS1.2 recommends this with their Ephemeral Keys, but IPSec VPN administrators have yet to learn the lesson and these are still recorded as if they will be required in the future, undermining the confidentiality and integrity of the link.

I worked on a proposal for a customer who demanded full encryption of all storage volumes in a cloud hosting provider, and quickly determined the theory and practise of these cryptosystems prohibited a robust solution. Of course, it could be fudged (and eventually was over my objections), but it took me a while to realise that the actual integrity of the system was not the point. It was merely to be able to make the claim "It's Encrypted".

The eventual implementation made use of "self-decryption". This is not a thing. By all means, system rely on obfuscation to hide key material and defeat trivial intrusions, but if a system is aware of how to decrypt its' own resources, an attacker will be able to achieve the same goal. The Trusted Computing Modules located on modern mobile devices, with the built-in Hardware Security Module, provides a slick solution to this question in performing both assurance that the system is trustworthy, and providing the eventual decryption keys once this assurance is performed. A virtual environment has not such opportunity.

Another issue in the eventual implementation was the use of a central key repository, using COTS. Since the designers have no real training or experience in cryptography or security principles, they made the fatal mistake of trusting the systems they were trying to protect too much. Every server is hosted at a provider, and on startup will request access to a database of keys over CIFS. Authentication is handled by means of the computer AD account. Since the single database is decrypted in its entirety by any participating server, every server has access to the keys for every other. This is disastrously insecure; the cloud provider has administrative access to these systems, and it is against their view that the cryptosystem is supposed to protect; ISVs and integrators are frequently given administrative credentials to install and test their products, again allowing them to retrieve as many keys as they please should a nefarious actor be present.

But hey, "It's Encrypted".

Do the ends still matter?

One of the biggest conflicts of interest in my role as an infrastructure architect (I'm responsible for servers, applications, security, storage etc) is that I have a heavily vested interest in the service they provide. This may sound like a no-brainer, but often I need to find ways to get my traffic to users at the expense of someone else. I want preferential treatment if conflicts with other traffic.
Of course, it's not quite as simple as that. Pragmatically I realise I am competing with others on limited resources, whether it's rack space, hypervisor RAM or network bandwidth. One of the less-known principles underlying the Internet ethos is the End-to-end Principle. Put simply, it's this:
Complexity in any network should be implemented in the end points - the network stacks of communicating nodes and in applications - and not in the network itself. Since standards change any benefit of implementing too much intelligence in the network is quickly undermined with the need to continually match those changes in end-points, as well as legacy issues.
Essentially, the network should be as dumb as possible. Packet comes in, packet goes out, wait for next packet. This is reasonable in principle but different traffic flows require different treatments - VoIP and video streaming protocols prefer low latency and are almost always jitter sensitive, while file transfers can tolerate enormous latencies as long as they are accompanied by high bandwidth. If both types occupy the same link the risk is that the insensitive consumption of one protocol can impact the requirements of another, so QoS arrives.
This is already a necessary evil (though some may object to my use of that label in this context) since we now need to build knowledge of services into the network layer. Large IP transport installations (Internet backbones) handling gigabits of traffic per second literally don't have the time to implement protocols like this as processing adds both cost and latency. This is different from implementing different virtual circuits for different traffic types, in that it forms logic links and is a very common practice with ISPs in the final mile. As far as intelligence on network devices go, this is very low - again packets come in, figure out where packets go out.
A very interesting turn is in the development of new forms of network acceleration. Routers have long been capable of doing in-line compression of data to reduce consumption of a specific link, but this is point-to-point. If an application protocol can truly benefit from compression, this really should be done at the application protocol level (basically above Layer 3) so routers and switches can shuffle packets. A possible side-effect of link compression is to mask real versus usable bandwidth from applications (some portions of a stream may be highly compressible, others not) hindering flow-control algorithms built into TCP.
TCP sessions can be rather chatty, and some applications implement redundant techniques inside their own specification, so some network acceleration can initially make sense. Essentially, new sessions follow a well-known pattern of window sizes and other parameters, so network accelerators intercept packets and simulate the repetitive parts on each end of a link, reducing session setup time. This sounds simple, but now we enter dangerous waters. Next comes protocol caching; I request a file from a file server across the WAN and the contents are cached on my local acceleration appliance so that the user next to me can get a cached copy when her request is made. Again, sounds simple, but to prevent interception and modification the protocol implements signing, so re-generated content requires a re-generated signature. SSL acceleration similarly can be implemented using reverse proxies holding a copy of the private key for the service to extract the plaintext and look for compression and caching opportunities. I've been involved in design and deployment of many reverse proxy and SSL acceleration projects but these were explicitly part of the service.
To accelerate generic payloads the appliance needs to get heavily involved in the infrastructure, either by hosting all SSL keys and being able to masquerade as endpoints. This is where it gets very complex and risky, not only in the dissemination of privileged access (private keys and domain credentials being the most highly-prized items in any network), but also a continual catch-up game to implement these techniques on newer protocols as they are developed. There are also more intrusive techniques such as automatic downscaling of images when using mobile data to browse the web that are subtle but insidious.
Net Neutrality is the overall drive in this direction and correlates with the End-to-end Principle, treating network devices as simplistic and all traffic as equal.Smarter protocols such as (distributed) BranchCache and BitTorrent (yes, BitTorrent is a case for Net Neutrality) reduce redundancy over constrained links and better content intelligence (I explicitly convert all images in my documents to 8-bit PNGs before embedding) are far better strategies. Content distribution networks are an active participant and used by some of the largest providers (both content generators and ISPs) to reduce long-haul bandwidth and improve user responsiveness. HTTP compression is a rigorously defined standard but very sparsely used even on static pages where on-the-fly zipping is unnecessary. I could go on...
When looking into how best to transport content I prefer to let the network do what it does best and work my requirements into it than engineer a network to suit my needs unless absolutely necessary. Where my application is burdensome, I would prefer to engineer the application than throw bandwidth at the problem.
Technology marches, and bandwidth - while logically finite - seems to be keeping up rather well.

Security by Default in the Defined Domain

Some simple security concepts are tougher for engineers to grasp than they should be. A pervasive view I've found is that firewalls, security policies, LAN partitioning and excess of routing tend to just get in the way of making systems work.

Now certainly, large enterprises and especially security sensitive ones requiring secure access to and transmission of data such as financial services, R&D, law enforcement and others do require a higher level of attention to detail when designing and operating their systems. Most engineers cut their teeth building systems in the privacy of their homes or labs, neither of which can match the scale and complexity of real-world systems spanning continents, regulatory domains and most importantly untrusted links. When learning and experimenting, the engineer tends to be in control of everything.

Secure authentication is enabled by default in Windows, with Kerberos providing one of the most resilient systems available, and AD integration makes expansion and administration a breeze. But most of the remaining protocols don't pay too much attention, starting with the most pervasive - CIFS (Windows File Sharing). Even RDP starts off with secure authentication then moves on to a plaintext stream for keyboard inputs and screen updates.

I'm a big fan of IPSec, since it allows for transparent encryption and authentication without having to do pretty much anything to your applications or network layout. The problem comes in getting all your nodes to play nice, since it does require some configuration. Windows domain policies and domain membership vastly simplifies this - if your system is a domain member, enabling opportunistic IPSec encryption is a breeze.

This is the "Defined Domain", a set of systems over which you have complete control. A visitor to your network would not have these policies defined, so their device would not be able to participate, but placing them in a guest LAN and tunneling the connection at the router over a secure connection in Tunnel mode can help solve that problem, but it can quickly spiral out from there - how do you get simpler devices and protocols like SNMP and even PING to cooperate?

The Defined Domain needs to incorporate not just the systems you're protecting, but also how and why. Secure protocols aren't that hard to come by these days (e.g. SNMPv3 incorporates some measure of encryption, SSL is available in all major web platforms), but still requires some configuration and attention. Applications need to be designed with at least some measure of awareness of the security domain if they are to be trusted, and administrators and designers most of all need to keep this in their head when going about their work.

The biggest threat to any security domain is and will always be the human factor. By incorporating security devices in an easy-to-use package, and often transparent, it can make the average engineer immune to the considerations required to make systems truly secure. Yet again, ease induces laziness, and the worst kind of security is the kind you can't (or won't) verify. It took about two hours of use before a colleague of mine noticed his newly-secured RDP session (certificate-based authentication) placed a new padlock symbol on the control bar of his Terminal Services Client. Until then, the idea that the connection was relatively more secure was fairly unverifiable, and eventually irrelevant.

Attention is required, and a lot of education. No one security solution is a panacea (everyone kept screaming certificates at me, as a one-size fits all solution, no no no), and this will always be the problem.

At least in a defined domain, it's more manageable, more approachable. And then along come the users...

Virtualisation of basic services

So, virtualisation is everywhere. It departments are running more and more of their workloads on virtual hardware, and in the data centre that’s a big thing. It’s an accepted fact that most servers in an organisation run at near idle all day long, and even core systems like directory and file servers barely crack a sweat serving their users. For small and medium-sized entities (yes, they have datacentres, just not raised floor, air-conditioned ones like the big guys) this is a problem.

I was asked a while ago why I enabled file compression on my laptop, and seriously considered it on file servers. A few years later another query was why I enabled full-disk encryption on the same piece of kit. Doesn’t it affect the responsiveness?

The simple answer is yes. And no.

CPUs, memory and disks today are so, so fast that there’s not much they can’t do for an individual user. when asked what laptop a person should buy, I recommend the pink one if it’s a girl asking, and jet-black for the guys. Seriously, laptops today can wipe the floor of a desktop from five years ago, and last I checked we’re still doing the same things as back then: browsing the web, writing emails, laughing at cats. If you’re a developer, gamer, or serious showoff, you probably have a dedicated desktop for the hardcore tasks, and the point remains: Moore’s law has seen to that.

Of course, anything you put in the path of data from the disk to CPU/memory is going to slow things down, but this comes back to the more important point: Do I notice?

When I’m running a compile job, chances are the disk’s head movements trying to find each source file and library is going to be the first bottleneck. Then, if it’s a biggie like mplayer, the CPU is going to be loaded for a while doing the compile. Encryption? Oh, I didn’t even notice it’s on.

This brings me to my point, with uber-powerful, multi-core servers cheap as chips (e.g. HP ML110 G5, Dual-core 2.8Ghz Core 2, 1G RAM, 250G SATA for 350 ex tax, HP.nl website 2009-12-07), do I really need to treat these things like fragile porcelain ware?

So, given a department of 100 users in a remote office, where I need to provide local AD, file services, updates, VPN to head office, backups, what’s to stop me buying a cheap server, loading it up with RAM and disks and virtualising all functions in one (or a few), neat little grey cubes with a reputable brand name on the front?

Now here, I’m thinking of Linux as the hypervisor, but that’s my flavour of choice. Something as simple as AD is a good case, since it’s by definition a replicated, resilient database so failures are not a big impact (authenticate somewhere else), and doesn’t do much in the day. It’s also advised to segregate roles on Windows servers, and for resilience perhaps have a second box if you’re paranoid . That’s one, maybe two little cubes that I can’t share for other functions. And it’s going to consume a few gigabytes of disk space, and even less RAM. A whole cube, just for you? Share!

Linux virtualisation has come a long way, and I consider it to be solid, fast and these days wonderfully easy to administer remotely. KVM, as I use, is just another OS process, no funky drivers or installations and since it runs on a full (and standard) instance of Linux, all the management tools for networking, storage, logging and troubleshooting are available.

One simple example: since all traffic traverses the host’s virtual network switch, all traffic can be inspected by a packet sniffer as powerful as Wireshark. No drivers in Windows, no strange binaries. If everything’s virtualised on one box, ALL the traffic is visible for troubleshooting in one spot. Neat!

Another, consider three servers, each requiring 100G storage. That’s probably six disks (minimum) if you’re doing things right (RAID-1). One one hypervisor host, that’s four 100GB disks (RAID-5), only two if you go the 500G route (RAID-1, plus a spare I suppose). Now I’m also a fan of software RAID. My case above stands, there are other factors more likely to impact performance, no least the client’s ability to pull data fast enough from a file server.

So, Now I’ve one box with my local services running in VMs. Full-disk encryption? Well, Windows 2008 has BitLocker, with all the TPM and USB key requirements. How about logging in to the host remotely via SSH, one of the most bulletproof protocols for remote management, and enable the encryption remotely. With a script. Not one byte in plaintext, not even the partition table. Break that, evil hardware theif!

Snapshotting? check
SNMP/performance monitoring and alerting? check
Hardware support? check
Virtualising legacy systems? check (especially MS-DOS on pesky 64-bit Windows)

Worth a think