Paul P. Mealing

Check out my book, ELVENE. Available as e-book and as paperback (print on demand, POD). 2 Reviews: here. Also this promotional Q&A on-line.

Sunday, 28 March 2010

Karl Popper’s criterion

Over the last week, I’ve been involved in an argument with another blogger, Justin Martyr, after Larry Niven linked us both to one of his posts. I challenged Justin (on his own blog) over his comments on ID (Intelligent Design), contending that his version was effectively a ‘God-of-the-gaps’ argument. Don’t read the thread – it becomes tiresome.

Justin tended to take the argument in all sorts of directions, and I tended to follow, but it ultimately became focused on Popper’s criterion of falsifiability for a scientific theory. First of all, notice that I use the word, falsifiability (not even in the dictionary) whereas Justin used the word, falsification. It’s a subtle difference but it highlights a difference in interpretation. It also highlighted to me that some people don’t understand what Popper’s criterion really means or why it’s so significant in scientific epistemology.

I know that, for some of you who read this blog, this will be boring, but, for others, it may be enlightening. Popper originally proposed his criterion to eliminate pseudo-scientific theories (he was targeting Freud at the time) whereby the theory is always true for all answers and all circumstances, no matter what the evidence. The best contemporary example is creationism and ID, because God can explain everything no matter what it entails. There is no test or experiment or observation one can do that will eliminate God as a hypothesis. On the other hand, there are lots of tests and observations (that have been done) that could eliminate evolutionary theory.

As an aside, bringing God into science stops science, which is an argument I once had with William Lane Craig and posted as The God hypothesis (Dec.08).

When scientists and philosophers first cited Popper’s criterion as a reason for rejecting creationism as ‘science’, many creationists (like Duane T. Gish, for example) claimed that evolution can’t be a valid scientific theory either, as no one has ever observed evolution taking place: it’s pure conjecture. So this was the first hurdle of misunderstanding. Firstly, evolutionary theory can generate hypotheses that can be tested. If the hypotheses aren’t falsifiable, then Gish would have had a case. The point is that all the discoveries that have been made, since Darwin and Wallace postulated their theory of natural selection, have only confirmed the theory.

Now, this is where some people, like Justin, for example, think Popper’s specific criterion of ‘falsification’ should really be ‘verification’. They would argue that all scientific theories are verified not falsified, so Popper’s criterion has it backwards. But the truth is you can’t have one without the other. The important point is that the evidence is not neutral. In the case of evolution, the same palaeontological and genetic evidence that has proved evolutionary theory correct, could have just as readily proven it wrong. Which is what you would expect, if the theory was wrong.

Justin made a big deal about me using the word testable (for a theory) in lieu of the word, falsification, as if they referred to different criteria. But a test is not a test if it can’t be failed. So Popper was saying that a theory has to be put at risk to be a valid theory. If you can’t, in principle, prove the theory wrong, then it has no validity in science.

Another example of a theory that can’t be tested is string theory, but for different reasons. String theory is not considered pseudo-science because it has a very sound mathematical basis, but it has effectively been stagnant for the last 20 years, despite some of the best brains in the world working on it. In principle, it does meet Popper’s criterion, because it makes specific predictions, but in practice those predictions are beyond our current technological abilities to either confirm or reject.

As I’ve said in previous posts, science is a dialectic between theory and experimentation or observation. String theory is an example, where half the dialectic is missing (refer my post on Layers of nature, May.09) This means science is epistemologically dynamic and leads to another misinterpretation of Popper’s criterion. In effect, any theory is contingent on being proved incorrect, and we find that, after years of confirmation, some theories are proved incorrect depending on circumstances. The best known example would be Newton’s theories of mechanics and gravity being overtaken by Einstein’s special and general theories of relativity. Actually, Einstein didn’t prove Newton’s theories wrong so much as demonstrate their epistemological limitation. In fact, if Einstein’s equations couldn’t be reduced to Newton’s equations (by eliminating the speed of light, c, as a factor) then he would have had to reject them.

Thomas Kuhn had a philosophical position that science proceeds by revolutions, and Einstein’s theories are often cited as an example of Kuhn’s thesis in action. Some science philosophers (Steve Fuller) have argued that Kuhn’s and Popper’s positions are at odds, but I disagree. Both Newton’s and Einstein’s theories fulfill Popper’s criterion of falsifiability, and have been verified by empirical evidence. It’s just that Einstein’s theories take over from Newton’s when certain parameters become dominant. We also have quantum mechanics, which effectively puts them both in the shade, but no one uses a quantum mechanical equation, or even a relativistic one, when a Newtonian one will suffice.

Kuhn effectively said that scientific revolutions come about when the evidence for a theory becomes inexplicable to the extent that a new theory is required. This is part of the dialectic that I referred to, but the theory part of the dialectic always has to make predictions that the evidence part can verify or reject.

Justin also got caught up in believing that the methodology determines whether a theory is falsifiable or not, claiming that some analyses, like Bayesian probabilities for example, are impossible to falsify. I’m not overly familiar with Bayesian probabilities but I know that they are a reiterative process, whereby a result is fed back into the equation which hones the result. Justin was probably under the impression that this homing into a more accurate result made it an unfalsifiable technique. But, actually, it’s all dependent on the input data. Bruce Bueno de Mequita, whom New Scientist claim is the most successful predictor in the world, uses Bayesian techniques along with game theory to make predictions. But a prediction is falsifiable by definition, otherwise it’s not a prediction. It’s the evidence that determines if the prediction is true or false, not the method one uses to make the prediction.

In summary: a theory makes predictions, which could be right or wrong. It’s the evidence that should decide whether the theory is right or wrong; not the method by which one makes the prediction (a mathematical formula, for example); nor the method by which one gains the evidence (the experimental design). And it’s the right or wrong part that defines falsifiability as the criterion.

To give Justin due credit, he allowed me the last word on his blog.

Footnote: for a more esoteric discussion on Steve Fuller’s book, Kuhn vs. Popper: The Struggle for the Soul of Science, in a political context, I suggest the following. My discussion is far more prosaic and pragmatic in approach, not to mention, un-academic.

Addendum: (29 March 2010) Please read April's comment below, who points out the errors in this post concerning Popper's own point of view.

Addendum 2: This is one post where the dialogue in the comments (below) is probably more informative than the post, owing to contributors knowing more about Popper than I do, which I readily acknowledge.

Addendum 3: (18 Feb. 2012) Here is an excellent biography of Popper in Philosophy Now, with particular emphasis on his contribution to the philosophy of science.


larryniven said...

Yeah, this whole conversation would be much more pleasant if there was a wider understanding of the various evolutionary predictions that have failed. It's not like there aren't any - certain epigenetic theories, for instance, are unsupportable - but even people like me don't really have a good enough grasp of them to bring them up in conversation.

It would be nice, of course, if Justin would be less inane, but we both know that ain't gonna happen. What might help instead is the ability to classify evolution as a group of accurate predictive theories, but, again, that is rather hard to do without knowing the history of it in some detail. I keep trying to find a casual source of information like this - blogs, for instance - but without any luck. I guess the eventual answer is to just go hit the library one day, but it hasn't reached that point yet.

April Sage said...

I want to begin by saying that I enjoy your blog in general, and even this one, though I cannot agree with what you've said. While I appreciate your line of thought, and agree with your use of the word falsifiability, it is rather painfully obvious that you have not read Popper. That is, that you have not read what Popper had to say about this particular issue. It wasn't just Freud that Popper was contending with. Indeed, having read Popper himself, I know that the very issue with which Popper was dealing was Darwin's evolutionary theory.

Popper called Darwin's theory "a metaphysical research program." This is because, in spite of your claims to the contrary, Karl Popper contended that there is no way to disprove Darwin's theory of Natural Selection as the force behind evolution. That is why we have such contradictory claims as that camoflage has been selected for its survival value, and also that flamboyant coloration (as in the male peacock's tale) has been selected for survival value. This is a clear case of whatever is has survival value. This is what Popper was talking about.

Thanks for the lively blog.

Paul P. Mealing said...

Hi April,

You're obviously better read on Popper than I am, and, yes, I was unaware that he was targeting Darwinian evolutionary theory.

He also targeted another contemporary of Freud's, but I've forgotten who it was.

I always appreciate people pointing out my ignorance on a topic.

Regards, Paul.

larryniven said...

Do note, though, that that particular objection of Popper's was limited to the time of his making it:

"One might say that it 'almost predicts' a great variety of forms of life. In other fields, its predictive or explanatory power is still more disappointing.

...though Darwin's theory of evolution does not have sufficient power to explain the terrestrial evolution of a great variety of forms of life, it certainly suggests it."

If he were still alive today and able to observe the computer simulations of evolution, he may well have recanted: those are capable of making specific predictions and are therefore falsifiable in a strict sense. I also feel like he probably sold Darwin short a little - it's not like you can't read predictions into On The Origin Of Species, even if you have to squint a little every now and then. In particular, "the more diversified the descendants from any one species become in structure, constitution, and habits, by so much will they be better enabled to seize on many and widely diversified places in the polity of nature, and so be enabled to increase in numbers" is a pretty solidly testable idea.

So, yes, the natural selection part in and of itself is strictly a consequence of mathematics and therefore not testable. But "Darwin's theory" broadly speaking is very much testable.

Paul P. Mealing said...

Hi Larry,

Darwin’s and Wallace’s theory was really natural selection and nothing else (which is all Popper would have known about). I saw a really good BBC documentary on Darwin, Huxley, Lyell and Wallace only last year. Since then we’ve discovered genetics and DNA, not to mention a lot of fossils, all of which support evolutionary theory. As I pointed out in my last post on Speciation, despite everything we know, there’s still a lot to discover and explain.

The fact is that we actually observe natural selection amongst viruses, as well as both flora and fauna in the wild – it’s something that can be seen happening in situ – so I consider it a natural law; but it’s only one part of the whole puzzle.

April was quite correct to pick me up on my ignorance of Popper. It doesn’t change my argument; but, in effect, it’s my version of falsifiability and not his.

Regards, Paul.

April Sage said...

Thanks so much for taking my comment in such good spirit. You're right that all Popper would have known about (and commented upon) is Natural Selection, which does have its limitations. Genetics has taken us into a whole new realm of scientific fact. I don't believe that Popper denied Evolution as a natural occurance. I think he was hoping for a theory that would explain evolution better than Darwin's rather vague one (after all Darwin had to deal with the limitations of the time in which he lived), which we are finding in the new genetic research.

I take it as an optimistic sign that Creationists are on the wane, and that ID is on the rise. After all, that means that fewer people are denying that evolution takes place. They are finding a way, within their religious faith, to accept the scientific fact of evolution. That is an enormous step in the direction of unanimity on this issue.

Paul P. Mealing said...

Hi April,

Yes, I see ID as a retreat in a God-of-the-gaps sort of way, which is the argument I took up to Justin, and started this whole debate.

I wrote about this over 2 years ago if you're interested, Is evolution fact? Is creationism myth? It covers much more of the science than the epistemology, but also juxtaposes religion from my point of view.

I argue that evolution is 'design' but it's nature that does the designing, not God. One can use 'artificial selection' in industrial design very successfully, whereby you randomly change parameters and match the successful ones. Over a series of 'generations' you can get a very successful outcome with no mathematics required. Natural selection is an excellent methodology for refining a design to a very exact purpose.

I should be embarrassed by your revelation, but, believe me, I would prefer to be told of my ignorance than continue to remain ignorant.

Regards, Paul.

Timmo said...

I think there is considerable confusion here about what Popper and Kuhn's criteria for demarcating science from pseudo-science really are. For Popper, a theory is scientific if it requires you to stick your neck out: there is some possible observation or set of observations that could refute or falsify that theory. Popper worried that Freudian psychology, the Marxist theory of history, and Darwin's theory of evolution were unscientific because they seem so elastic and are able to be deformed to meet experience come what may.

And Popper is correct that one cannot state in advance definite conditions under which we would take the theory of evolution to be falsified. But, this is a failure of Popper's demarcation criterion; the "dialectic" as you put it between theories and experimental observations is more dynamic than this. By Popper's criterion, very little of modern science survives. Newton's laws of motion are not directly compared with experience. There's no way to do to this. One has to make all kinds of auxiliary assumptions about the thing you are looking at, the instruments you are probing it with, and so on. The recorded observations themselves are not neutral facts, but are already partially interpreted and theory-laden.

This is where Kuhn comes in. For Kuhn, a genuine science involves a tradition of puzzle-solving. There is a framework of accepted methodology, fundamental principles, and canonical examples of solved problems -- a paradigm -- that is not challenged or open to falsification, except when it massively fails the scientific community. In the normal course of science (so, not revolutions), research is an activity of puzzle-solving within the framework. It is the individual scientists who are tested, not the paradigm itself.

From a Kuhnian point of view, then, ID and creationism are pseudo-scientific because they don't form or play a role in a puzzle-solving tradition. They generate no problems to solve and no criteria or principles to engage in study with.


Paul P. Mealing said...

Hi Timmo,

I always appreciate your comments. You invariably manage to tell me something I didn’t know.

In the case of Darwin’s theory, I believe that even in its early days it could have been falsified through fossil evidence – not necessarily contemporary fossil evidence but future fossil evidence – I think that would have been obvious. In fact, Darwin seemed to be aware himself, that the ultimate success of his theory was completely dependent on fossil evidence.

Your point about Newton’s laws is an interesting one. But his ‘mechanics’, as they are expressed in his calculus, certainly could be tested. Anything that can be expressed in a mathematical form can usually be tested. Like Einstein’s theory, for example, assuming one has the technology to perform the test, which is why string theory is not really a theory, or, at best, a theory in limbo.

Regards, Paul.

Timmo said...

Hi Paul,

I think we can press the point a little and look at some historical examples. Copernican astronomy did not emerge unchallenged and faced difficult attacks. According to the heliocentric theory, the Earth orbited the Sun, moving against a fixed background of stars. It is a matter of common experience that if you move around a room, the fixed background of the furniture and walls appears to move. Similarly, critics argued that if the Earth really were moving we would see the fixed background of stars moving in a way that we simply don’t observe. One possible response would have been, “Ah, so we have predictions which are in conflict with experience; the heliocentric theory is false.” Instead, astronomers held fast to the Copernican hypothesis, and the motionless background became a puzzle. Instead, it became something for the Copernican astronomers to intellectually digest. The puzzle was solved by finding evidence that the stars were much further away than was previously believed.

Another example. Newton developed laws of motion and a theory of gravitation that formed the basic framework of classical physics. Astronomers made careful observations of the motion of Uranus and found that it did not obey Newtonian rules. One possible response was, “Ah, so we have predictions were are in conflict with experience; the Newtonian theory of gravitation is falsified.” Instead, astronomers held fast to the Newtonian theory, and the puzzle generated by the orbit of Uranus was resolved by postulating the existence of Neptune. Afterwards, it was hunted for and observed.

I don’t know the early history of the theory of evolution, which is why I haven’t spoken directly to it. But, I would not be surprised if there were fossils that didn’t fit – and they would not have been taken as falsifying the general principles, but as posing a research problem for future investigators to resolve. There were challenges like this: Lord Kelvin calculated the lifetime of the Sun assuming that gravitational collapse was its primary source of energy (the dominant theory at the time) and it was nowhere near the millions of years required for new Darwinian view.

To first approximation, then, we should distinguish between basic framework of ideas, principles, methodologies, and canonical examples of solved puzzles that comprise the paradigms in which scientists work from the particular models or puzzle-solutions that are devised to explain this or that phenomenon. The parts of the paradigm are held more or less rigidly (they are sacrosanct) and are not open to direct falsification and are not even capable of being directly compared with experience. What can be refuted or confirmed are proposed solutions to particular research puzzles. Falsifiability is not, contra Popper, the hallmark of science.

There's more that can be added, but the word-count of this comment is growing quickly! >:-O

Paul P. Mealing said...

Hi Timmo,

I take your point. I’m aware of all those examples, but I had never thought about them in that context before.

It’s true that if we get a false result, we don’t necessarily throw the theory away – I’ve always accepted that. And I can see that, in principle, people would argue that there is a conflict between Popper and Kuhn, because Kuhn argues that results get harder and harder to accommodate before we have a paradigm-shift, which means we’re getting a lot of false results. That is why I contend that science is a dialectic: theories do change, and usually because evidence doesn’t fit, or we learn something new that changes our assumptions (as per 2 of the examples you gave).

In my own mind, I’ve always seen them as compatible, because I see Popper’s falsifiability as a selection criterion at the beginning, and Kuhn’s paradigm-model as part of the ongoing process – a logical consequence of the dialectic. In physics, it’s very rare not to come across a theory that doesn’t have limitations of application. I guess that’s why I’m sceptical about a TOE (Theory-of-everything). The most universal equation (if not a theory) that I can think of is Einstein’s E= mc2.

As far as the theory of evolution goes, it’s here to stay, but there are still a lot of unanswered questions. There are still some Kuhnian revolutions to occur, but I don’t expect the theory to be ever rejected outright.

There is also the correspondence principle (I think that’s what it’s called) that any new theory has to explain what any previous theory explained.

Regards, Paul.

Timmo said...

Hi Paul,

It is important not to confuse Popper’s views on demarcation with the banal comment that theories ought to be empirically adequate and are problematic if they are not. The conflict is much deeper than your remarks indicate.

In his historical survey of science, Kuhn found that, any moment, there are anomalies -- puzzles which have stubbornly resisted solution in terms of the paradigm even by the most skilled scientists. Sometimes they are eventually resolved; others times, along with many other anomalies, they cause the members of the scientific community to lose confidence in the paradigm and initiate a crisis (which may or may lead to a revolution). What the existence of anomalies shows it that Popper’s naïve falsification criterion does not correctly demarcate scientific inquiry. Scientists very rationally continue to work within a paradigm even though there are outstanding problems that seem like they cannot be solved. One never specifies in advance what conditions would refute or falsify elements of the paradigm al la Popper. This, to first-approximation, there exists a two-tier feature which is totally incompatible with Popper’s views.

You might have Bohr’s correspondence principle in mind. This states that in the limit of very high quantum numbers, one ought to numerically recover classical results. This is actually contentious and it’s best to bracket technical questions like that. When it comes to scientific revolutions, it is not the case that new paradigms have to include everything the previous theory explained as a special case. There are examples of “Kuhn-loss,” cases in which a phenomenon went from explained to unexplained as the result of a scientific revolution.

Indeed, Kuhn’s work begins to take shape as a criticism of Popper, and Popper reviled Kuhn’s ideas. In the form in which they formulated them, they are not at all compatible. Sophisticated methodologists like Lakatos have devised ways of synthesizing strands of Popper with strands of Kuhn. But Popper’s demarcation criterion itself doesn’t survive sustained historical inquiry (and theory-ladeness, holism, etc), I think.

Paul P. Mealing said...

Hi Timmo,

I thought Bohr's principle dealt with 'complementarity' whereby a quantum phenomenon was either wave or particle but never both simultaneously. 'Correspondence principle' is the wrong term for what I'm describing, I know, but I don't know another corresponding term (pun unintended).

Yes, I can see my view is very simplistic compared to yours, but, fundamentally, a theory has to make predictions and it has to have explanatory value. I've always thought that Popper's criterion, in principle, should eliminate psuedo-theories where the evidence is always neutral, for which God is the perfect example.

In my last 2 posts, I've touched on the subject of free will. Free will is rejected by science because it has no explanatory value to science.

Back in Sep.07, I quoted Gregory Graffin and William Provine in American Scientist who wrote: "it adds nothing to the science of human behaviour."

And they're right: free will, almost by definition, is totally unpredictable. If a theory can't make testable predictions it's useless. This may be a 'looser' interpretation of Popper, but I think it's one that is used even if it's not formulated as such.

I have to admit I wasn't aware of any 'Kuhn-loss' cases, but what you say makes sense. You may ditch something that was explicable previously if the new theory or paradigm has greater overall explainability.

It's 20 years since I read Kuhn's famous book, and unfortunately I don't have a copy.

Regards, Paul.

Timmo said...

Hi Paul,

Of course, the solution to your conundrum about Bohr is that he has many principles! If you are interested, there is a very clear presentation of his interpretation of the quantum theory here:

Again, I would urge that “if a theory can’t make a testable prediction it’s useless” is not even a loose interpretation of Popper: it’s a banal, trivial remark that everyone agrees with (or should, anyway) instead of an interpretation of Popper or anyone else. (It would be boring if that were all Popper had to say!) But perhaps we are at the ends of our ropes on this, so I shouldn’t harangue you about it! :-P

It’s not clear to me that free will, or I might add creativity, “adds nothing to the science of human behavior.” In fact, I would argue that it is one of the great mysteries or important phenomena for the brain sciences to try to investigate. As Chomsky points out, we are all constantly linguistically creative: we produce novel utterances not under the control of external circumstances which are in some sense appropriate to circumstances. Similar remarks go for making choices: we can make new choices which are not pre-programmed or controlled by external circumstances which are not random but appropriate to situations in which we find ourselves. That’s a real phenomenon, and we can raise, if not answer, scientific questions about what enters into it.

I am not myself a cognitive scientist or a professional philosopher so I can’t speak with any authority about these questions. But, it is worthwhile being skeptical when someone dismisses things like this offhand. It’s certainly untrue that free will is by definition “totally unpredictable.” Why accept that assumption?

Paul P. Mealing said...

Hi Timmo,

I think your criticisms are all valid. I acknowledge that you have a deeper understanding of this than I have. My remark is trivial – sometimes I admit that all I do is state the obvious. I generally don’t resort to Popper concerning arguments over religion and science. When it comes to ID, it should be enough to say that God is a science-stopper, and that’s usually my starting point.

And you are right to point out that there are subtleties here that I obviously haven’t grasped. I acknowledge that.

It’s certainly untrue that free will is by definition “totally unpredictable.” Why accept that assumption?

Philosophically, there has been a view that free will and determinism are incompatible and determinism is generally considered totally predictable. Although, having just said that, Dennett once pointed out that determinism and predictability are not the same thing, if I remember him correctly. And, after reading Schrodinger, and his ideas on ‘statistico-determinism’, I would tend to agree. You can make predictions on a large scale for phenomena, which, on a unitary scale, are completely indeterministic (but this has nothing to do with free will).

Anyway, right or wrong, that’s where I was coming from. I would like you to elaborate on why ‘it’s certainly untrue’. Personally, I don’t see a conflict between free will and fate (as I like to call it). But free will is unpredictable in the sense that it assumes spontaneity. I was alluding that it was unpredictable in a scientific sense as well, but, if that’s your point of contention, you may be right.

To clarify my position on Graffin’s and Provine’s comment, I actually challenge them in my original post. I say they’re right in as much as the scientific position contends that human behaviour can only be determined by our genes or our environment. As far as science is concerned, free will doesn’t enter the picture. I actually disagree with their position, and I’ve stated the argument better elsewhere. In my last comment, I was trying to make the comment relevant to our discussion, and I probably should have left it alone.

I agree pretty much with everything you say about free will in your second-last paragraph, and I think it’s well stated.

Regards, Paul.

Paul P. Mealing said...

Hi Timmo again,

You're making me wonder if there's any merit in my post at all. I don't mind being wrong if I learn something new from it.

You're effectively saying that Popper's entire methodology is flawed because scientists will continue to work with a theory, even when, Khun-like, it gives false results. So you're saying that falsifiability is not a valid criterion to determine if a theory is scientific or not. And that my interpretation, whilst correct, is simplistic and naive, and really just empericism.

Does Popper's falsifiability criterion have any merit at all then?

Regards, Paul.

Timmo said...

Hi Paul,

Does Popper's falsifiability criterion have any merit at all then?

Of course, opinions about this differ. Feyerabend advances epistemological anarchism, arguing that anything goes: there are no general methodological principles that always govern scientific practice. For various reasons, I am sympathetic to Lakatos’ “sophisticated falsificationism” ( According to Lakatos, the falsification criterion should not be applied to particular theories, but entire research programs. He says that there is a hard core of principles which are held constant and a protective belt of auxiliary hypotheses that can be changed. We look favorably upon a research program when workers within that program are making novel predictions which are then confirmed.

When it comes to ID, it should be enough to say that God is a science-stopper, and that’s usually my starting point.

I suppose this depends on your purpose. I don’t think that being a ‘science-stopper’ is much of an objection to anything. Human cognitive faculties have limitations. There may be domains about which we can raise questions, but are unable to answer them simply because we are biologically barred from having the categories or conceptual resources required. Colin McGinn, in fact, has argued that consciousness, while a completely natural phenomenon, is permanently mysterious to us because we lack the resources to articulate how consciousness can emerge from brain processes. That’s a science-stopper, but it doesn’t show it is false. ID proponents can accept your charge and say: we have pushed scientific investigation that far and there is nothing further possible by us.

It may be better to patiently wade through their material and show point by point how it is wrong. This also has the political benefit of not giving the impression of dismissing them out of hand, which will always prevent them from coming to understanding.

Philosophically, there has been a view that free will and determinism are incompatible and determinism is generally considered totally predictable.

Yes, there is a long history of seeing free will and determinism as incompatible and trying to determine which horn of the dilemma we should grasp. Still, that’s a philosophical thesis, not a given. There is also a history of compatibilism ( I suspect that the metaphysical demands of libertarianism are more stringent than is required for what we ordinarily take to be freedom of the will or what freedom is required for moral responsibility.

Cheers! :)