A Random Story

So randomness and probabilities are inevitable parts of any discussion of distributed consensus. Just how comfortable are we with the fact that our daily choices must be guided, at least in part, by the laws of probability?

When we get behind the wheel of an automobile or strap ourselves into an airplane seat, we are literally trusting our lives to the laws of probability. Since we can't know everything about the pilot of the airplane or the other drivers on the road with us, we wrap ourselves every day in the comfortable knowledge that, with high probability, we will survive to see another day.

But when it comes to computer programs and programmers, we are much more demanding. We expect computers to be deterministic. Period. They perform exactly the same instructions over and over and any deviation from that determinism we call a "bug" or a "fault." We demand that our family photos be kept safe even in the eventuality of multiple, byzantine faults. When you check yourself into a hospital, the chances that an accident or mistake will lead to the loss of your life are quite a bit higher than the chances that an accident or mistake will result in the loss of that kitty cat picture on Facebook. (Small caveat: I totally made that up.)

The fact is, we accept that the universe is stochastic but demand that computers be deterministic. Here's a true story about how I once forced a customer of mine to face that bias and, I hope, start to change his attitude a bit.

While I was at Caringo (they probably still do this) we would periodically conduct training classes for customers of our products, which includes CAStor, a scalable, distributed object storage solution. I would typically join the students and instructors for lunch one day to answer any technical questions they had about the products.  During one such training class, I learned from the instructors that one of the students was being vocally skeptical about one of the foundational assumptions we had made in the product.

CAStor can store billions, even trillions of distinct data objects, each of which has a universally unique identifier (UUID) associated with it.  Other similar products on the market generate these UUIDs in a deterministic fashion that necessarily involves a central naming authority - an expensive and fragile solution IMHO. CAStor's innovation in this area was to instead use a large truly random number for these UUIDs, removing the requirement for a central authority and significantly simplifying the generation and management overhead.

Using this mechanism, the chances that two distinct objects will be assigned exactly the same UUID are very, very, very (to the 42nd power) small. But it is possible and, of course, it would be a bad thing if it happened, which is exactly the objection being raised by our skeptical student.  I mean, we're talking about computers here, and he expected a deterministic guarantee that such a collision is not possible.

So I expected a question along these lines during our meet-the-architect lunch. But it didn't come.  At the end of the lunch, after we'd all finished our deli sandwiches, I decided to bring up the topic myself. I went into the kitchen of our corporate offices and retrieved a wine glass that had been sitting on the top shelf collecting dust. Back in the training room, I tapped the glass with my pen to get everyone's attention and, without saying a word, placed the glass into my empty lunch sack, twisted it shut, and put it on the conference table. Then I picked up a heavy metal power strip and, again without a word to the class, pounded the sack with the power strip until the glass was thoroughly shattered.

Everyone moved away from me there in the training room. The instructors, whom I hadn't warned about this, considered calling building security.

After a brief pause for effect, I picked up the lunch sack and began shaking it vigorously. Then I asked, "Who here believes the glass will spontaneously reassemble itself if I keep shaking the bag?"

No one answered. So I upped the ante.

"What if I continue shaking it like this all day?  All year?"  Nobody answered.

"What if I continue shaking the sack for the rest of my life? Who believes the glass shards will accidentally find their way back in the exact same configuration they started in to reform the wine glass?"

Finally, the troublemaker student answered. He timidly said, "Well, it could happen."

"Exactly," I said.  "It could happen. And if any of you believe it actually will, I recommend you should not buy our product." And I left the room.

I'm happy to report that our "troublemaker" student became a big believer in our product and is still a customer.