Randomness shouldn’t work. But often it does. That is the paradox of randomness.
Entropy happens because the vast majority of possibilities within chaos are dead ends. We define rationality as choosing optimal solutions from the raw ore of all these possibilities, discarding the rest. Usually, this is the best approach. But not always.
Sometimes we simply don’t know what to do, or even how to figure out what to do. And sometimes we get stuck. We have been doing what worked, then for some unknown reason, it stopped working. The sensible thing is to try something else. But what else?
Science has the problem of how to come up with hypotheses. Lots of advice about creativity seems to involve random association.
The bottom line: it’s all well and good to say you should choose the optimal, but how do you do that? It’s not practical to consider mindfully each and every possibility. That approach doesn’t scale. So we fall into ruts, into biases, into habits of thought that may themelves be arbitrary.
Deliberate randomness compensates for that. I put a (pseudo-) random selector in my custom note management scripts to prevent me forgetting about all the thoughts and data that I’m not deliberately focussed on. I feel like this is helping me. Certainly I’m less anxious about losing track of something important by failing to recognize its importance or by losing it in the shuffle.
Evolution uses randomness, and so we call it mindless. But what is mind, anyway? I say mind is as mind does. Don’t call something stupid if it works. And don’t call it inefficient unless you’ve got a better way. Sometimes randomness is the most efficient approach, because sometmes it’s the only one that works at all. If God plays dice with the universe, He might know something about dice, or about universes, that you don’t.
When we fully understand a problem, a little logic offers the solution, but such a problem is so trivial it barely counts as a problem. When we half understand a problem we can apply a heuristic, and that usually works. Bounded rationality has its uses. But when we don’t understand a problem at all, and no available heuristic works, then randomness is the only solution. At this juncture, a deliberate randomness is as rational as it gets.
There’s a mystical dimension to this: we don’t actually know what randomness is. That may paradoxically be the best definition of randomness: the unknown and unknowable cause. The human mind is a pattern recognizing machine, and what we call randomness is simply our failure to discern a pattern. When we can’t fathom how something works, we dismissively call it mere chance – thus salving our egos. But really, we don’t know if chance even exists. It could be an illusion of our limited perception. There are statistical tests for randomness, but they all boil down to failing to detect any hint of non-randomness. True non-determinacy can’t be proven, only disproven.
We know from computer science that a deterministic but exremely nonlinear algorithm can be virtually indistinguishable from what we call true randomness. This is the pseudorandom number generator. Why can’t there be some sort of Supreme Being lurking behind the quantum white noise of the universe? Or at least a hidden order? Perhaps the universe is encrypted. Can this hidden order act through the seeming randomness of a divination method, or a dart thrown at a dartboard? When you’re at the end of your rope, you may as well surrender to the luminous Void. Let go and let God.
Just hope it doesn’t turn out to be Azathoth.
(No, I’m not endorsing organized religion or any specific superstition. Not endorsing nihilism either. There is a humility and openness to a certain sort of agnosticism, which I find preferable to every developed dogma I’ve yet encountered.)