A common and frequently correct view of biological evolutionary change is that once an adaptive trait has arisen, it is honed to ever greater states of refinement, like the ideal match between the shape of a hummingbird's beak and its flower. This concept fits seamlessly into our engineering modes of thinking: you first design something, and then you carefully improve it, adding additional constraints on its operation with perfection as the goal.
But evolution does not always follow such a pattern. In some cases, selection backs off, in which case the trait accumulates mutations over time since there's no selection to weed the variants out. This could, of course, have disastrous consequences, but sometimes, as constraints relax, creative alternatives arise. Such is the story of some finches under study by Kazuo Okanoya, a story used by Terry Deacon to suggest possible parallels with the evolution of human language — a core feature of human evolution writ large.
Okanoya's story begins with the white-rumped munia (Lonchura striata), a finch that lives in the wild throughout Asia. Male munias sing a highly stereotyped song: motif A is always followed by B which is followed by C. Such song constraints are thought to be crucial to attracting females. Indeed, under theories of sexual selection, first articulated by Charles Darwin, the females of many species exert "mate choice," interpreting the adept performance of a ritualized behavior or the generation of an impressive visual display (think peacock tails) as indicating that she is in the presence of a "high-quality male." Accuracy makes the heart grow fonder.
About 250 years ago, a few munias were brought to Japan and subjected to a completely different mode of selection: birds with particularly colorful plumage were mated with one another to generate ever more interesting plumage, and the progeny were sold to bird fanciers. Some 1000 generations of such selective breeding has generated the present-day Bengalese finch, (Lonchura striata var. domesticus). What attracted Okanoya's attention was that the domesticated Bengalese sing a highly unconstrained song: A might be followed by B, or by a repeat of A, or by something else altogether. Moreover, the domesticated male birds, as chicks, are highly adept at learning the song of another male in their enclosure; indeed, they have no problem learning the munia's song if that's what they are exposed to. By contrast, wild munia chicks show no ability to learn songs from a Bengalese.
Meanwhile, neurophysiologists have been comparing the brains and vocalizations of bird species that are capable of learning songs with those that cannot. It turns out that brains capable of producing only an innate song have very simple neural pathways: the primary forebrain motor center, called the robust nucleus of arcopallium (RA), connects to midbrain vocal outputs which in turn project to brainstem motor nuclei. By contrast, in brains capable of learning songs, the RA receives input from numerous additional forebrain regions, including those involved in learning and social experience. Control over song generation has become less constrained, more distributed, and more flexible.
So what does this have to do with humans? When compared with our primate relatives, whose communication system is restricted to a highly stereotypic repertoire of hoots and calls, humans have very few prespecified vocalizations, extant examples being laughter and sobbing. Moreover, these remaining innate vocalizations are generated by restricted neuronal pathways, whereas language is generated by a highly distributed system involving numerous regions of the human brain, a system so complex that we remain very far from being able to explain how language works. The finch story offer a scenario — speculative to be sure, but intriguing — for what might have happened during the evolution of human language competence.
A salient feature of language is that while language competency is inherited, the languages themselves are transmitted via culture. Also transmitted via culture are understandings, such as technological ways of doing things, that are framed as language-based explanations. Hence one would expect a robust co-evolutionary trajectory between language competency and culture: proto-humans capable of the first, and presumably rudimentary, versions of protolanguage would have better access to cultural understandings, and cultural understandings, conveyed in protolanguages that children's brains could readily learn, were more likely to be transmitted, thereby conferring the benefits accrued.
Hence proto-humans indubitably engaged in, and continue to engage in, what is called "niche construction", creating cultural niches that provide understandings key to survival, and undergoing evolutionary changes that optimize their ability to flourish in such niches. Selection pressures that operated to sustain instincts important for survival in prior niches would be expected to relax as humans became increasingly dependent on their self-created cultural niches, while any innovations that facilitated cultural adaptation — in this case, innovations in language competency — would be expected to spread.
So, Deacon suggests, one way to think about human evolution is that we are basically self-domesticated apes. Just as domestication relaxed selection for stereotypic songs in the finches — mate choice was supplanted by choices made by the aesthetic sensibilities of bird breeders and their customers — so might our cultural domestication have relaxed selection on many of our primate behavioral traits, allowing old pathways to degenerate and reconfigure. Given the highly indeterminate way that mammalian brains develop — they basically construct themselves "bottom up," with one set of neuronal interactions setting the stage for the next round of interactions — degraded pathways would tend to seek out and find new opportunities for synaptic hookups. Such inherited de-differentiations of brain pathways might have contributed to the functional complexity that characterizes human language. And, as exemplified by the finches, such de-differentiations can occur in very rapid timeframes.
Such a scenario in no way suggests that classic natural selection did not play important roles in sculpting our modern language capabilities — doubtless it did. But it lifts up the important role that can also be played by various forms of drift. Another example, found in all organisms, is the path taken by gene duplications: one copy typically continues to cover the original function while the second accumulates mutations and, on occasion, generates proteins with novel shapes that participate in novel functions.
As Deacon puts it: "Perhaps our great leap forward required first taking a few steps back."