The Line That Never Widened
Copyright© 2026 by Alexander Winston
Chapter 12: The Service That Could Have Only One Owner
June 1999 through April 2000
Two Compaq servers stood side by side in the clustering lab, both powered, both showing green status lights, both able to reach the shared storage cabinet between them. A hand-lettered card taped to the first read MAY OWN THE POST OFFICE. A card taped to the second read MUST NOT OWN THE POST OFFICE.
An engineer reached behind the rack and pulled the private heartbeat cable.
Nothing on either machine changed. Each server stayed healthy. Each simply lost sight of the other. The shared GroupWise test post office sitting on the storage cabinet between them hadn’t changed either — only the question of who was allowed to open it, which had just become dangerous in a way no green light on either chassis would ever show. A careless failover routine would let the second machine take over on the reasoning that the first had gone silent. The first machine, still running, still healthy, had received no evidence at all that its authority had ended.
Both machines kept their green lights on the whole time. That was the part Sarah kept coming back to afterward — nothing on either chassis ever suggested a problem existed at all.
The clustering lead, a broad-shouldered engineer named Marcus Webb who’d spent two years building toward exactly this demonstration, turned to Sarah. “Which one’s dead?”
“Neither,” Sarah said, and understood, before he’d finished nodding slowly back at her, that this was the entire, uncomfortable problem she’d just walked into.
Marcus didn’t wait for anyone to ask what his team had been doing. He’d brought a list, and he read from it the way a man reads something he’s proud of and slightly tired of having to prove: node-membership and heartbeat prototypes running for months; a shared-storage arbitration test that had already survived two rounds of deliberate sabotage; early resource start, stop, and monitoring scripts; NSS pool movement demonstrated between UnixWare nodes; a manual GroupWise restart performed using the exact recovery boundary Diane’s team had documented; early integration with Compaq’s own hardware-management interfaces; and a set of fault cases his people had reproduced independently, without so much as reading the GroupWise certification report until afterward, just to see whether they’d find the same gaps on their own.
“None of it came out of anyone else’s budget,” Marcus said. “Sarah and Eric gave us common UnixWare contracts and a certification bar. GroupWise, print, and NSS gave us their own service requirements when we asked. Nobody funded this but clustering.” He said it without heat, the way a fact gets stated rather than defended, and Sarah recognized the cadence — she’d heard nearly the same sentence from Diane less than a year earlier.
GroupWise had kept shipping. Print services kept shipping. NSS’s own roadmap hadn’t paused to wait for this meeting, and neither had the still-unnamed management-products team quietly reading the same interface documents from two floors down. Nobody had been waiting on anybody. Marcus said it a second way, plainer than the first, once he saw Sarah wasn’t going to argue with the list: “I didn’t come here today to ask permission to keep doing what we’ve been doing. I came to ask which parts of it stop being mine to decide alone.”
What had risen past Marcus’s own authority were four questions he named in order, the way Diane had once named three: whether automatic failover could forcibly remove or restart a server that might still be running; what hardware behavior Novell would require before certifying a cluster at all; which Novell services and applications could carry a public high-availability claim; and whether clustering itself belonged inside the common Novell Services license or needed a separate one of its own.
Gordon answered the last question first, since it was the only one that was actually his to answer. “Clustering serves several products at once. It’s common infrastructure, the same as the directory or the storage engine. It goes in the Novell Services package.” He looked at Diane, who’d come to sit in on the meeting uninvited and unsurprised. “GroupWise stays exactly where it is. Being certified to run on a cluster is a new way to deploy it. It isn’t a new owner for it.”
The architecture review that followed spent most of its energy tearing down an idea everyone in the building had started reaching for out of convenience: that clustering meant failing over a server. Eric wouldn’t let the phrase stand unchallenged. Customers didn’t own servers for their own sake, he said; they owned services, and a service had to be free to move while the physical machines under it stayed replaceable.
What actually moved, the team decided, was a resource — a bundle holding everything a service needed to keep being itself regardless of which node was running it: a virtual name and network address a client could always find; the NSS pool and volume carrying its writable state; whatever NCP file-service identity or access point it required; the application’s own processes; a documented sequence for loading, unloading, monitoring, and recovering it; and an ordered list of which nodes were allowed to run it and in what preference. A client talked to the virtual name, never to a physical machine, and when ownership changed hands the name and address simply moved with it.
Someone sketched the old model on the whiteboard first, purely so the room could watch it get crossed out: a server as a fixed point, applications nailed to it, a customer’s mental map matching the asset tag on the front of the case. The new sketch replaced the fixed point with a moving label, floating above whichever two or three boxes happened to be qualified to carry it that week. Eric liked the second drawing enough that he had someone photograph the whiteboard before it got erased.
eDirectory’s part in this was, once again, exactly as large as it had ever been and no larger. It stored the cluster, node, resource, and virtual-service objects, and it governed who was allowed to configure or move any of them — but it could not become the thing membership depended on at the moment a machine actually failed. A directory replication delay on the worst possible afternoon couldn’t be allowed to leave a cluster unable to decide, right then, which of its own nodes were still permitted to act. eDirectory stayed distributed across its own replicas the same way it always had; it never became a shared database that one cluster node mounted at a time the way NSS did.
A newer engineer on Eric’s own team asked the question everyone else had apparently decided not to ask out loud: if eDirectory already tracked which nodes belonged to which cluster, why not simply let it decide who was alive right now, too, and save building a second membership system from scratch? Eric didn’t dismiss it. “Because the directory can be unreachable for reasons that have nothing to do with whether a node is dead — a switch, a cable, a bad afternoon on a WAN link three buildings away,” he said. “The moment we let a remote lookup gate a local ownership decision, we’ve built a system that can be paralyzed by exactly the kind of failure it exists to survive.”
Different resources needed different manners entirely. An NSS file resource mounted its storage and then opened for business. A GroupWise resource had to mount its volume, confirm the application’s own state, and only then start the MTA or POA in the order Diane’s team had already fixed. A print resource could restore its spool and queue state, but the outcome-unknown category for a job already sent to a physical printer stayed exactly as unresolved as it had always been — clustering gave the service back its queue, not a printer’s memory. A generic UnixWare process, meanwhile, got no certification of any kind until somebody who actually understood it wrote down how to start it, stop it, watch it, and bring it back. “The cluster manager places resources,” Eric said, closing the review. “It doesn’t understand what any of them mean. That’s still the application’s job.”
The failure lab was where the engineering argument actually got settled, and Marcus’s team had built it specifically to force the argument into the open. Early prototypes had used missing heartbeats alone to decide a node was gone, and the disconnected-cable demonstration existed to show precisely why that had never been good enough. Both machines stayed capable of running. Each lost sight of the other. Both could still reach the shared storage cabinet through separate physical paths, which meant both, left alone, could talk themselves into believing they were the only survivor.
Marcus’s people gave the condition its plain name — split brain — and refused to soften it: two machines still alive, two incompatible pictures of who belonged to the cluster, and one piece of shared state that could not survive being owned twice. The fix required more than a faster heartbeat. It required a second, independent heartbeat path so one bad cable couldn’t manufacture a partition by itself; a quorum requirement before any shared resource was allowed to load; a protected arbitration area where membership and ownership claims actually got recorded rather than assumed; a fencing mechanism that could cut a doubtful node off from shared storage before anyone else was allowed to write to it; and, underneath all of it, a rule that uncertainty stopped a resource rather than letting two copies run side by side.
In the two-node test that followed, the machine that still answered didn’t take the post office simply because its partner had stopped responding. It had to win the arbitration outright and confirm the other node had actually been shut out of the storage path before it was allowed to touch anything — retaining authorized membership, not merely outlasting a silence. The excluded machine, for its part, had to surrender cleanly — stop its own cluster services, release storage access, or be reset through a certified out-of-band path — before the resource could move anywhere at all. When the cluster couldn’t settle on one authorized membership, the design simply gave up on availability rather than risk the alternative: two idle servers were an inconvenience; two servers writing the same GroupWise post office was a catastrophe with a much longer cleanup.
Denise and Gordon had to sign off on what that actually meant in practice — that automatic software could, under published rules and audited configuration, forcibly cut an uncertain machine off from the shared storage it could otherwise still reach, up to and including a certified reset or power-down, before another node was allowed to assume ownership. A machine that could no longer talk to its partner but could still write to a shared volume was the entire danger; isolating it on the network alone would have protected nothing. Neither of them signed it lightly; Denise asked, twice, exactly which log would prove after the fact that a fencing action had been authorized rather than merely convenient, and didn’t accept an answer until Marcus’s team could point to the specific record. It was one of Marcus’s own engineers, watching the arbitration succeed on the fortieth run in a row, who said the line that ended up on the whiteboard afterward in someone else’s handwriting: silence is not proof of death.
Proving the theory meant watching it survive an actual migration, and GroupWise supplied the test case because Diane’s team already had opinions about every step of it. A planned move went in a fixed order: stop new client connections and drain outstanding work; let GroupWise flush its own transactions and queues on its own terms; stop the agents cleanly; dismount the NSS volume; release the virtual identity and the storage ownership together; establish ownership on the destination node; mount and verify NSS there; start the agents back up in the sequence they required; publish the service identity on the new machine; and only then, after watching it hold steady for a defined interval, call the resource online.
An unplanned failure didn’t get to skip to the happy ending. It used GroupWise’s own recovery rules from wherever the crash had actually left things, not a fiction where the graceful steps had somehow already happened. Marcus’s team was careful to keep three different failures from blurring into one: a failed agent process could often be restarted right where it stood, if the node and its storage were still sound; a failed node needed a membership change, fencing, a storage handoff, and full application recovery somewhere else; and a failed storage path, if Compaq’s own redundancy could absorb it, might not need to move the application at all. Print recovery kept its own earlier honesty intact — clustering could bring the service and its spool state back, but it still couldn’t tell anyone whether a given page had physically finished printing before the failure. A lost eDirectory replica triggered ordinary directory replication and repair, never a shared-storage resource move, because it had never been that kind of problem. Oracle and the rest of Novell’s third-party neighbors stayed outside the claim entirely, pending their own vendors’ recovery contracts and their own certification — nobody was going to advertise a promise on someone else’s behalf.
To read the complete story you need to be logged in:
Log In or
Register for a Free account
(Why register?)
* Allows you 3 stories to read in 24 hours.