The capability of computers and our ability to program them seems to be increasing exponentially. Even if we hit a brick wall in terms of increased miniaturization and frequency our CS knowledge seems sure to continue building on itself. It stands to reason that within the next century we will have the ability to build computers, or at least augment our own brains, to create entities smarter than ourselves (whether or not you think they will have experiences). But if our creations are smarter than us then, barring any limit imposed by fundamental physics, one would think they could improve on our design and design another generation that was even smarter. These machines (or augmented humans) would soon reach transcendent levels of intelligence and change our society beyond recognition.

At least this is (more or less) the notion of the Singularity as popularized by Vernor Vinge and Ray Kurzweil. For more details I recommend reading Vinge himself or checking out one Kurzweil’s many interviews and talks (audio) as well as his webpage. These are certainly two very smart individuals who have the rare ability to look beyond the specifics and take a fairly clear headed look at how technology will transform society. But smart doesn’t mean infallible and predicting the future is a notoriously difficult business.

While I used to find the arguments for the singularity convincing I’m now much more skeptical. In particular it seems the argument for the singularity rests on a misconception of intelligence. I mean it seems obvious to us that if someone was significantly smarter than us they would be significantly better at designing intelligent computers or human augmentation. But that’s because we both assume that intelligence is some kind of fully general ability to solve problems and conflate intelligence with technical skill and achievement. After all we rarely see people’s raw IQ scores so we tend to simply call people intelligent if they are especially capable in technical fields or other academic endeavors. However, while intelligence is certainly helpful much of what makes for a good scientist or engineer is their store of accumulated experience, both personal and distilled into formal education.

While it does seem that people’s ability at a wide range of reasoning tasks is substantially correlated this doesn’t mean talking about intelligence makes sense for anyone but biologically natural humans. It seems quite plausible that there is no such thing as general reasoning ability. Rather there are only heuristics applicable to certain types of problems, e.g., ability to do mental rotations, solve crosswords, recognize objects etc.. Yet if so there is no reason to believe that there is any good heuristic for designing good heuristics, in fact it seems downright unlikely. Thus just because we were able to find a collection of heuristics that give rise to something better at math and play chess than us doesn’t mean we should expect it to have a substantially easier time discovering better heuristics for the next generation. Sure, we will probably be able to create beings who can remember more numbers, do CAD drawings in their heads and so forth but the singularity requires an exponential (or at least super-linear) increase in capability over time so mere elimination of minor inefficiencies we have at AI design isn’t sufficient.

Even in mathematics people primarily reason inductively. We don’t blindly search for a formal proof, rather, we try the same techniques we’ve seen work in ’similar’ problems in the past and attempt minor modifications. In other words what makes someone a good mathematician is largely their mental collection of heuristics they use to approach problems. While continued miniaturization of computer chips might enable AI to reduce the time it takes to do mathematics pure increases in computational speed a may already be near the physically practical limit (though going 3D and using light should eventually give a few more orders of magnitude) and certainly this effect wouldn’t be sufficient to create the singularity. Thus it seems the singularity requires a sequence of exponentially increase sequence of better and better heuristics to guess the true theory based on limited data. In other words a more effective form of scientific induction.

In other words people currently use some heuristic to guess at a rule underlying a set of observations. We make some finite number of observations about disease occurring near wells near sick families and hypothesize that disease can be spread through the water. We observe some examples of current generated by metal exposed to various frequencies of light and hypothesize that light must come in quantized units. The singularity seems to require that not only is there a heuristic that lets us make equally effective guesses at the true theory based on less information but that there is an exponentially increasing sequence of such heuristics. Moreover, it would be necessary that each heuristic can discover the next in roughly the same amount of time despite the substantially greater performance each subsequent heuristic requires. Frankly, I find this somewhat implausible.