Translating a bird’s brain activity into song

It is possible to re-create a bird’s song by reading only its brain activity, shows a first proof-of-concept study from the University of California San Diego. The researchers were able to reproduce the songbird’s complex vocalizations down to the pitch, volume and timbre of the original. Published in Current Biology, the study lays the foundation for building vocal prostheses for individuals who have lost the ability to speak. “The current state of the art in communication prosthetics is implantable devices that allow you to generate textual output, writing up to 20 words per minute,” said senior author Timothy Gentner, a professor of psychology and neurobiology at UC San Diego. “Now imagine a vocal prosthesis that enables you to communicate naturally with speech, saying out loud what you’re thinking nearly as you’re thinking it. That is our ultimate goal, and it is the next frontier in functional recovery.” The approach that Gentner and colleagues are using involves songbirds such as the zebra finch.  The team implanted silicon electrodes in male adult zebra finches and monitored the birds’ neural activity while they sang. Specifically, they recorded the electrical activity of multiple populations of neurons in the sensorimotor part of the brain that ultimately controls the muscles responsible for singing.

The researchers fed the neural recordings into machine learning algorithms. The idea was that these algorithms would be able to make computer-generated copies of actual zebra finch songs just based on the birds’ neural activity. But translating patterns of neural activity into patterns of sounds is no easy task. The team used simple representations of the birds’ vocalization patterns. These are essentially mathematical equations modeling the physical changes—that is, changes in pressure and tension—that happen in the finches’ vocal organ, called a syrinx, when they sing. The researchers then trained their algorithms to map neural activity directly to these representations. This approach, the researchers said, is more efficient than having to map neural activity to the actual songs themselves. “With our collaboration,” said Gentner, “we are leveraging 40 years of research in birds to build a speech prosthesis for humans—a device that would not simply convert a person’s brain signals into a rudimentary set of whole words but give them the ability to make any sound, and so any word, they can imagine, freeing them to communicate whatever they wish.”

Source: University of California San Diego news release