You Win Again

I haven’t done this much for a long while, but there used to be a fairly typical failure mode where the database becomes unable to keep pace with combined write traffic. What happens while they deal with the backlog is that all the write connections end up held open, until you simply run out of available connections.

The real tragedy is that this is almost always non-linear. Once you reach a tipping point in aggregate load, the efficiency of each write operation begins to decline. On old mechanical hard drives this was due to head thrash. On modern SSDs, especially on database servers, it has to do with write amplification. Databases tend to like to scatter small writes all over the disk, often as small as 512 bytes per write. But SSDs can’t actually perform such a small overwrite. SSDs optimized for cost/storage ratios often 32 KB sector sizes a few years back. More expensive SSDs suitable for database use might try to limit that to 4 KB.

The problem is that to overwrite a 512 byte sector, the SSD has to lift the underlying sector into memory (4 KB or 32 KB read), erase the underlying sector (a 32 KB block erase), merge the new data into the old data in memory, then write the whole thing back again (4 KB or 32 KB). Erase used to be darn slow on SSDs, so what you do is keep a large list of preerased sectors, and then you juggle locations with internal mapping tables (also needed to implement wear-leveling). How much the internal SSD sets aside for the erase list depends on how much is stored on the SSD drive and what kind of load the SSD is optimized for.

If aggregate load exhausts the internal SSD preerase list, all hell fails to defrost in an infinity of Sundays. In the bad old days, block erase could involve ms latencies. (Every actual storage chip on the SSD drive can erase in parallel, but that still isn’t much. And probably these large modern chips support multiple concurrent erases, but that probably still isn’t much relative to advertised performance and capacity.)

If that’s not the problem, then turning small 512 byte writes into a 4 KB read followed by a 4 KB write is often the problem on database servers (or if you’ve bought exactly the wrong SSD for the application, 32 KB reads followed by 32 KB writes).

Read connections really shouldn’t be your problem, because in the limit with infinite memory, all disk access is write access (everything else is cached in memory all the time). Any competent web host simply throws more memory at a database server where the read traffic is a major issue in intermittent service failure. The memory is almost always cheaper than the technical competence to pursue any other solution.

Back in 2017, Intel unveiled 3D XPoint (sold as Optane) and promised the moon in all three spumoni flavours. It was supposed to be better than cherry, pistachio, and chocolate combined.
What a dog it turned out to be, instead.

The concept is good, but it was expensive, and it had a terrible problem with durability. It was only good for about 30,000 drive writes before the bits got wobbly and every time wrote a sector you had to cross your fingers and count to a ten-month PTO for an 39-year-old free agent. The problem is that this technology was ideally suited for applications with insanely high write volumes. Poof! Your über expensive drive was toast within the first year.

What makes Optane great is that it has essentially no write amplification and no need to maintain lists of preerased blocks, either.

What makes Optane not so great is that—last I heard—Intel is only able to fabricate this with a small number of layers (four layers in the 2nd generation), whereas ordinary SSDs are now made with a stupid number of layers. I see 176 layer drives now announced. You have severe alignment problems building up a 176-layer Jenga tower and that comes at the expense of the bit patterns having thunder thighs, but thunder thighs alone does not divide 176 layers down to 4 layers.

And there’s your cost ratio in a nutshell.

Standard SSD: 4 KB minimum write block, 4 KB or 32 KB minimum erase block, up to 176 capacity loving layers.

Optane: Write any size anywhere at any time with next to zero latency. Trunk space of a red Ducati — credit card and a spare pair of bikini briefs is pretty much all a man needs to experience the true Ducati lifestyle.

Given some of the performance advantages, Optane is often still a win.

Here’s where it depends on the application. Some database application have a tiny percentage of live data (data that might grow or change) and a large amount of static data (400 immutable lamentations per game thread over an entire decade of darkness).

For that you’ll need a giant fat disk, and you can’t afford the Ducati.

If your live data is only the discussion threads where users are still allowed to post and update, you’d need less than 1% as much space. Now you can afford the Ducati. You’ll buy a cheap SSD for the static data from yesteryear.

But not all web forum software is set up to partition static data onto a different database than live data.

There are some advanced file systems that play around with putting an Optane cache drive in front of a giant SSD backing layer in a mostly database transparent way. These is vastly tricky to get right for all the possible combinations and permutations of loads, and so you tend to see this kind of architecture on proprietary database servers with a known application-specific load profile.

What happens with your average cost-driven web host is that they make a guess about how much load an SSD disk system can handle, based on collective averages of the working set to cold storage set for the aggregate client.

When they guess wrong, write performance goes to hell in a handbasket, all the connections are held open waiting for pending writes to clear, and no connections remain available for any other purpose, including simple read requests to draw the “hello world” front page.

If you happen to have a very large static storage requirement and also a very large burst-activity write requirement (the no-win corner of the non-volatile storage options), they classify you as a bad customer, and then drool you along with substandard service until you pack up your bag and go somewhere else.

I’m so far out of this field at this point, I’m waving my arms around on the general principles of yesteryear, but if the architecture was up to me, all the user comment data would go to a different database server than everything else the site depends upon (which would not reside on a busy database server, but one overbuilt for never falling down).

If it was further up to me, all the user comments from the past two weeks (hot material) would go to a server backed by tiny and expensive DucatiOptane enterprise disk drives that simply never slow down and all the rest would rotate into cold storage backed by regular SSD (but still not on the primary site database server).

If your forum software is not architected to permit this, good luck with that.

There are some crazy cases where you’re actually running into network connectivity problems, but that doesn’t happen in a competent shop. You have to be an idiot to get your fabric that wrong. Not that idiocy never happens …

You can also run out of file descriptors at the OS level and many other gruesome technical things, but again, those are problems that mainly betray substandard competence in your service provider. Once every so often a bad Linux kernel security patch can exacerbate a problem that wasn’t previously visible. I tend not to score that case against the service provider in charge, but rather under “shit happens”.

The one case where the service provider needs to make some hard compromises ahead of the fact—to remain in any way cost competitive—is aggregate load factor on the database server disk subsystems.

And so that’s always the first stone to turn if problems occur at a competent web host.

Probably mid-tier hosts don’t even offer dedicated Optane database servers for very small databases with insanely high write concurrent write loads. That’s a whole other thing to manage. And then the customer is confused by the price model. You want HOW MUCH? for 2 GB of writable database store on the bulletproof Optane database server?

Does Lowetide generate 2 GB of user comments in two weeks? Not even if Taylor Hall was traded from the Oilers to Seattle in the middle of rare Oiler appearance in the playoff’s third round (though it might be a close run thing).

What the service provider will typically do is stall the matter by offering to migrate your database to a different database server with different load patterns. It might fix the problem, or it might not. Days soon turn into weeks on this program, with their technicians never doing more than 10 minutes of ticket resolution on your behalf in any 24-hour period. The problem is that most shops don’t have the detailed disk telemetry to act as competent match-makers. You can’t just use simple statistics, because some interactions have tipping points and other forms of extreme non-linearity. So you have to try it out to find out, and that can take far more time than any sane person wishes to endure.

There are also differences from one piece of forum software to another in how many database connections it needs to open per user interaction. I rarely spent much time under the hood of actual forum code, so this I can’t speak to this even in archaic terms.

Cue all the kids to say “grandpa, get off my lawn”. Yeah, I’m totally out of the loop these days. I haven’t been able to quote Optane latency statistics off by heart for a good 24-months now. I was mainly following this scene because of the looming competition from Nantero with their carbon nanotube technology, which is purported not to have a wear limit you can reasonable exceed whatsoever (trillions of overwrites); but the density—so far—is that much worse than Optane.

Nantero was kind of fun to follow. They pop out of stealth to announce $20 or $30 million in additional funding, followed by a glowing technology press release, a new appointment to the board … and then back to stealth mode again. If this kind of shop has a “jobs” page you head there. If not hiring, not firing (on all cylinders), as a general rule of thumb. Which the collapse of the fabrication supply chain since 2019, it might not in this case foretell a failing technology, they just might emerge out of stealth mode like a chameleon on methadone, yet again.

Since COVID, there’s no point checking in more than about once per year. Automakers these days can’t get themselves a hardened HC12 from yesteryear for love nor money. Only the brontosaurs of the world are getting any chip love at all right now.

What I’ve been following more closely for the duration is Terrestrial Energy.
Terrestrial Energy Appoints the Rt. Hon. Stephen Harper, Canada’s 22nd Prime Minister, to Advisory Board — 11 February 2020

If this technology flies and one of these doesn’t end up in Fort Mac, I’ll pretty much eat my shirt (prudently seasoned with highly iodized sea salt).

The process heat is ideal for cracking heavy bitumen, and then the electricity is gravy on top. Canada is among the leaders of the charge on generation IV reactor technology because our nuclear regulatory framework supports innovation, whereas the American framework (at least until recently) was a total killjoy.

Key number: Seven years. The old technology briefs—apparently too revelatory—used to specify retiring each core after seven years. This is because you can’t validate materials to handle radioactive molten salt for forty years in less than … forty years. There’s no viable template for accelerated aging studies against this perfect glowing chemical storm of ultimate sandpaper. There’s much to like about the concept, if the steel pipe this toxic slurry flows through is not made out of your skin.

The fuel cycle is easy, but the materials engineering challenge would make Fritz Haber weep—he’s the guy who invented mustard gas, and was never known for being much of a weepy fellow.