Is randomness real or illusion?

 Let’s look at quantum mechanics (QM). I watched a YouTube video on Closer To Truth with Fred Alan Wolf, a theoretical physicist, whom I admit I’d never heard of. It’s worth watching the first 7 mins before he goes off on a speculative tangent that maybe dreams are a more fundamental level of reality, citing Australian Aboriginal ‘dreamtime’ mythology, of which I have some familiarity, though no scholarship.
 
In the first 7 mins he describes QM: its conceptual frustrations juxtaposed with its phenomenal successes. He gives a good synopsis, explaining how it describes a world we don’t actually experience, yet apparently underpins (my term, not his) the one we do. In particular, he explains:
 
There is a simple operation that takes you out of that space into (hits the table with his hand) this space. And that operation is simply multiplying what that stuff - that funny stuff – is, by itself (waves his hands in circles) in a time-reverse manner, called psi star psi (Ψ*Ψ) in the language of quantum physics.
 
What he’s describing is called the Born rule, which gives probabilities of finding that ‘stuff’ in the real world. By ‘real world’ I mean the one we are all familiar with and that he can hit his hand with. Ψ (pronounced sy) is of course the wave function in Schrodinger’s eponymous equation, and Schrodinger himself wrote a paper (in 1941) demonstrating that Born’s rule effectively multiplies the wave function by itself running backwards in time.
 
Now, some physicists argue that Ψ is just a convenient mathematical fiction and Carlo Rovelli went so far as to argue that it has led us astray (in one of his popular books). Personally, I think it describes the future, which explains why we never see it, or as soon as we try to, it disappears, and if we’re lucky, we get a particle or some other interaction, like a dot on a screen, all of which exist in our past. Note that everything we observe, including our own reflection in a mirror, exists in the past.
 
Wolf then goes on to speculate that the infinite possibilities we use for our calculations are perhaps the true reality. In his own words: What I’m interested in are the things we can’t see… And he makes an interesting point that most people don’t know: that if we don’t take into account the things we can’t see, ‘we get the wrong answers’.
 
And this is where it gets interesting, because he’s alluding to Feynman’s sum-over-histories methodology, which takes into account all the infinite paths that the particle (as wave function) can take. In fact, the more paths that are allowed for, the more accurate the calculation. Wolf doesn’t mention Feynman, but I’m sure that’s what he’s referring to.
 
Feynman’s key insight into QM was that it obeys the least-action principle, which is mathematically expressed as a Lagrangian. It’s the ‘least-action principle’ that determines where light goes through a change in medium (like glass), obeying Fermat’s law where it takes the path of ‘least time’. It also determines the path a ball follows if you throw it into the air by following the path of ‘maximum relativistic time’. I elaborate on this in another post.
 
There is something teleological about this principle, as if the ball, particle, light, ‘knows’ where it has to go. Freeman Dyson, who was a close collaborator with Feynman, argued that QM cannot describe the past, but only the future, and that only classical physics describes the past. So these infinitude of paths that are part of the calculation to determine the probability of where it will actually be ‘observed’ make more sense to me if they exist in the future. I don’t think we need a ‘dream state’ unless that’s a euphemism for the future.
 
Like Dyson, I don’t think we need consciousness to make a quantum phenomenon become real, but it does provide the reference point. In his own words:
 
We do not need a human observer to make quantum mechanics work. All we need is a point of reference, to separate past from future, to separate what has happened from what may happen, to separate facts from probabilities.
 
The thing about consciousness is that it exists in a ‘constant present’, as pointed out by Schrodinger himself (when he wasn’t talking about QM), so it logically correlates with 'a point of reference, to separate past from future', that Dyson refers to.
 
Schrodinger coined a term, ‘statistico-deterministic’, to describe quantum phenomena, because, at a statistical level, it can be very predictable, otherwise we wouldn’t be able to call it ‘successful’. He gives the example of radioactive decay (exploited in his eponymous cat thought experiment) whereby we can’t determine the decay of a single isotope, yet we can statistically determine the half-life of astronomical numbers of atoms very accurately, as everyone knows.
 
I contend that real randomness, that we all observe and are familiar with, is caused by chaos, but even this is a contentious idea. I like to give the example of tossing a coin, but a lot of physicists will tell you that tossing a coin is not random. In fact, I recently had a lengthy, but respectful, discourse with Mark John Fernee (physicist at Qld Uni) on Quora on this very topic. When I raised the specific issue of whether tossing a coin is ‘random’, he effectively argued that there are no random phenomena in physics. To quote him out of context:
 
Probability theory is built from statistical sampling. There is no assumed underlying physics.
 
The underlying physics can be deterministic, while a statistical distribution of events can indicate random behaviour. This is the assumption that is applied to every coin toss. Because this is just an assumption, you can cheat the system by using specific conditions that ensure deterministic outcomes.
 
What I am saying is that randomness is a statistical characterisation of outcomes that does not include any physical mechanism. As such, it is not a fundamental property of nature. (Emphasis in original)
 
I get the impression from what I’ve read that mathematicians have a different take on chaos to physicists, because they point out that you need to calculate initial conditions to infinite decimal places to achieve a 100% predicted outcome. Physicist, Paul Davies, provided a worked example in his 1988 book, The Cosmic Blueprint. I quoted Davies to Fernee during our ‘written’ conversation:
 
It is actually possible to prove that the activity of the jumping particle is every bit as random as tossing a coin.
 
The ‘jumping particle’ Davies referred to was an algorithm using clock arithmetic, that when graphed produced chaotic results, and he demonstrated that it would take a calculation to infinity to get it ‘exactly right’. Fernee was dismissive of this and gave it as an example of a popular science book leading laypeople (like myself) astray, which I thought was a bit harsh, as Davies actually goes into the mathematics in some detail, and I possibly misled Fernee by quoting just one sentence.
 
Just to be clear, Fernee doesn’t disagree that chaotic phenomena are impossible to predict; just that they are fully deterministic and, in his words, only ‘indicate random behaviour’.
 
Sabine Hossenfelder, who argues very strongly for superdeterminism, has a video demonstrating how predicting chaotic phenomena (like the weather) has a horizon (my term, not hers) of predictability that can never be exceeded, even in principle (10 days in the case of the weather).
 
So Fernee and Hossenfelder distinguish between what we ‘cannot know’ and what physically transpires. But my point is that chaotic phenomena, if rerun, will always produce a different result – it’s built into the mathematics underlying the activity – and includes significant life-changing phenomena like evolutionary biology and the orbits of the planets, as well as weather and earthquakes. Even the creation of the moon is believed to be a consequence of a chaotic event, without which life on Earth would never have evolved.
 
Note that both QM and chaos have mathematical underpinnings, and whilst most see that as modelling or a very convenient method of making predictions, I see it as more fundamental. I contend that mathematics transcends the Universe, yet it’s also a code that allows us to plumb Nature’s deepest secrets and fathom the dynamics of the Universe on all scales.

 

Follow-up (30 Mar 2024)

Following my discourse with Fernee, I reread Davies’ book, The Cosmic Blueprint (for the third time since I bought it in the late 80s), or at least the part that was relevant. I really did Davies a disservice by just quoting one sentence out of context. In fact, Davies goes to a lot of trouble to try and define what randomness means. He also acknowledges that, despite being totally unpredictable, chaotic phenomena are still ‘deterministic’ – it’s just the initial conditions that are unattainable (mathematically as well as physically). That is why, when you rerun a chaotic event, you get a different result, despite being so-called ‘deterministic’.
 
As well as the mathematical example I gave above, Davies discusses in detail 2 physical systems that are chaotic – the population of certain species of animals and the forcing of a pendulum (where a constant force is applied to a pendulum at a different frequency to its natural frequency). Marcus du Sautoy in his book, What We Cannot Know, interviews ex-pat Australian, Robert May (now a Member of the House of Lords), who did pioneering work on chaos theory in animal populations.
 
Davies quotes Ilya Prigogine concerning ‘…the conviction that the future is determined by the present…  We may perhaps even call it the founding myth of classical science.’
 
He also quotes Joseph Ford: ‘…the fact that determinism actually reigns only over a quite finite domain; outside this small haven of order lies a largely uncharted, vast wasteland of chaos where determinism has faded into an ephemeral memory of existence theorems and only randomness survives.’
 
And then Davies himself:
 
But in reality, our universe is not a linear Newtonian mechanical system; it is a chaotic system… No finite intelligence, however powerful, could anticipate what new forms or systems may come to exist in the future, The universe is in some sense open; it cannot be known what new levels of variety or complexity may be in store.
 
In light of these comments from last century, and considering that under Newton and Pascale, everyone thought that given enough information, the entire universe’s future could be foreseen, I see ‘strong determinism’ (as opposed to weak determinism) as a scientific ‘fashion’ that’s come back into favour. By ‘weak determinism’, I mean that all physical phenomena have a causal relationship; it’s just impossible to predict beyond a horizon, which is dependent on the nature of the phenomenon (whether it be the weather or the planets). Therefore, I think randomness is built into the Universe, and its principal mechanism is chaos, not quantum.