So lately I’ve been reading a bit of Overcoming Bias and Less Wrong. While the posts on these sites are always interesting they frequently, especially at Less Wrong, seem to promote a sort of cult of rationality. Of course I too value reaching the right conclusions instead of the wrong ones and am broadly sympathetic with the goal of ameliorating the negative effects of psychological shortcuts that interfere with our utility but alot of the content on these sites seems to go much further than this. For example consider these posts by Robin Hanson and Eliezer Yudkowsky. Underlying these remarks seems to be the assumption that there is some kind of objective standard of (perfect?) rationality to which we could aspire that would somehow capture our intuitive notion of rationality as distinct from merely being lucky. While I rarely see rationality so earnestly venerated as it is on these two sites the same assumption pervades much of analytic philosophy and many puzzles and papers simply take it for granted that there is some well defined notion of rational thinking/belief(I’ll leave act rationality out of this for the moment). However, despite being an extremely useful tool in describing common situations and deciscions it’s long been my view that, when considered in full generality, rationality isn’t even a coherent concept.

To explain what I mean we first need to go back to Quine’s seminal paper Two Dogma’s of Empiricism. Even though I think Carnap clearly had the better of the argument about analyticity¹ I want to give Quine credit for pointing out to me the way in which the concepts we use depend on our background assumptions (how we model/conceptualize things) for their very coherence. Just as the concept of an equivalence class stops making sense once you start talking about non-transitive relations so too do many of our scientific and everyday notions cease to be well defined when we no longer accept the assumptions they were defined with respect to.

Stealing an example from Quine consider the Newtonian concept of kinetic energy. A good Newtonian physicist would have said that the kinetic energy is defined to be .5mv² where m is the object’s mass and v it’s velocity. However, kinetic energy is obviously also intended to be in some sense a measure of the work it would take to stop that object. Since these two notions coincide on the Newtonian picture there isn’t any problem. So long as we believe (or even use as an approximation) Newtonian physics there isn’t any question as to which is the right definition of kinetic energy. We are simultaneously committed to the concept capturing both notions. What Quine observed is that once we abandon the Newtonian conceptual framework there isn’t really any objective fact about which commitments we should honor and which we should discard. If scientists had responded to special relativity by using kinetic energy to describe the Newtonian formula and started theorizing about the conservation of Eisensteinian smenergy we couldn’t really accuse them of having made a mistake. Just as there is no right way to extend the notion of an equivalence class to non-transitive relations there often isn’t any right way to extend our scientific or everyday concepts outside of the frameworks they were conceived in.

The upshot of all this, in my view, is that our most useful concepts often presuppose certain assumptions. When these assumptions no longer hold the concepts themselves may cease to be coherent. So keeping this in mind let’s take a look at the assumptions that give rise to our concept of rationality.

Without going into too much detail I think it’s fairly safe to assume that a major (primary?) grip on (belief) rationality comes by way of postulating that people hold various beliefs where we take those beliefs to behave in some loose way like propositions. In other words we gloss over complexities like the effect of context and social situation on the views people express and simply pretend they either do or don’t believe some claim. Of course you can embelish this view a great deal and allow people to believe things in varying degrees or even take them to merely have some transitive implication relation. However, we can only stretch these concepts so far before they become unwieldy and useless, something we all implicitly recognize when we hesitate to attribute beliefs to ethnic groups, countries, or our computers.

So what? It’s hardly news that some aspects of people’s behavior won’t be well described by idealizing them as having something like beliefs. However, the point I want to press home is that rationality isn’t a property that big fleshy globs of atoms have. Rationality, is a concept grasped in terms of a certain kind of idealization about human behavior. It’s a useful concept and useful idealization but it’s still a type error to think of it as a property that applies to actual physical beings. We frequently forget this because in most contexts there is an obvious “right” way to idealize someone as an agent with certain beliefs so we talk about people having irrational beliefs and find it useful. However, it’s important to remember this shorthand only makes sense as long as this kind of idealization makes for a decent model of human behavior. Just like it’s simply confused to talk about the Newtonian kinetic energy of a particle traveling at .999c there are situations in which idealizing people as having something like belief is such a bad way to model their behavior that talking about rationality is similarly confused.

But the situation for a viewpoint independent concept of rationality applicable to real people only gets worse once you realize just how sensitive the ascription of rationality is to the way we choose to idealize the situation. Choosing to idealize a split brain patient as a single agent will yield very different judgments about his degree of rationality than idealizing his actions as the result of two seperate agents with distinct beliefs. It’s not that one of these idealizations is wrong and the other right (what could that even mean?) but just that in certain contexts one will be more useful than the other. And it’s not just split brain patients, Frued and many others have often taught that people were better modeled as the result of several competing agents or personalities. To really drive home the dependence of ascriptions of rationality on your choice of model just try to work out how you could make a principled application of the concept to a network of partial autonomous, partially integrated AIs.

My point is that it’s not just that we can’t ever be fully rational. It’s that the very notion as applied to living breathing people isn’t even coherent. Rationality is a concept that lives in an abstract idealize realm populate by agents possesing something like beliefs. It’s only applicable to real creatures insofar as this kind of idealization is useful and people largely agree on how it should be done. Step beyond that and it just doesn’t make sense anyone. I also think this realization can help alleviate some of the confusion over various paradoxes like the surprise quiz but that’s another post.