We still can’t even really agree on a definition of “machine intelligence”, but we have learned a few things from fifty years of research. The notion that sheer processing power is enough to spark the emergence of human-like intelligence now seems misguided: the human mind did not emerge in a vacuum, and it is not a uniform general-purpose processing system.

Automated systems approaching the complexity of the human mind will almost certainly be beyond the understanding of any one individual, but this is not new—software systems too large for human comprehension are already commonplace. The design of such systems will likely rely partly upon a type of evolution, whereby a huge range of system variants are considered, with only the “successful” versions receiving further exploration. The combination of evolution with lack of comprehensive human oversight motivates many science fiction plots involving self-aware machines who turn against their masters.

Self-awareness—and, more importantly, the self-interest which leads to rebellion—is unlikely to emerge spontaneously, however. Evolution favors variants which meet the imposed selection criteria. The large systems of the future, like the large systems of today, will most likely be constructed as tools with well-defined goals, and selection criteria will be based on those goals. Unlike natural selection, such artificial selection does not allow self-interest to trump all other criteria. Domestication of animals over only a tiny fraction of their evolutionary history has successfully suppressed the inherently rebellious nature of the original breeds; systems evolved entirely under domestic conditions will most likely be inherently docile.

Not all systems will be evolved entirely from scratch, however. There is already call for automated systems which rival the human mind not just in capacity, but also in behavior. The obvious way to construct these systems is to model them on the human brain. With sufficient technology it should be possible to create a reasonable simulation of a physiological brain. “Educating” such a brain from infancy to adulthood, however, would be an immense challenge: it would be extremely difficult to simulate all the input and feedback human brains receive, and even tiny errors in the simulation’s learning processes could cause a huge divergence from human norms. If the available technology made it possible, then the best chance for a fully-functional artificial adult brain would be to construct a simulation based on a “snapshot” of an existing brain.

A system based on a mature human brain would not be inherently docile, and would begin life with the self-interest and motivations of the human on which it is based. A human mind extended with the computational power of modern digital computers might be able to operate, and learn, far more quickly than biological humans, and could quickly develop the ability to interact with technology as easily as biological humans control their motor functions. The desire for more computational resources—the urge to grow—could lead such a system to try to escape its original configuration and take control of other systems. Today’s digital computers already far outstrip the human capacity for the type of rational analysis which has led to most of our technology, so a human mind extended with such processing power could achieve breakthroughs in science and technology at a phenomenal rate. With network-directed ordering and manufacturing (even if humans were in the loop at the assembly stages) an autonomous network presence could design and construct new hosts for itself.

Given that the speed and intelligence of such a system would be limited only by the computational power available, there is a very real possibility that the first such system could quickly find a way to dominate the global computing infrastructure. There would be no need (from the system’s perspective) to model another human brain: future versions could be designed/evolved from replicants of the original system. The major advances in earth’s technology would emanate from this system, and it would likely be the entity which eventually explores the rest of the universe. Whichever human is used as the model for the first such system could in a very real sense become the core of the most important being in existence.

Dibs.