May 13 2021
Communicating Through Handwriting with Thought
We have another incremental advance with brain-machine interface technology, and one with practical applications. A recent study (by Krishna Shenoy, a Howard Hughes Medical Institute investigator at Stanford University and colleagues) demonstrated communication with thought alone at a rate of 15 words (90 characters) per minute, which is the fastest to date. This is also about as fast as the average person texts on their phone, and so is a reasonably practical speed for routine communication.
I have been following this technology here for year. The idea is to connect electrodes to the brain, either on the scalp, on the brain surface, inside blood vessels close to the brain, or even deep inside the brain, in order to read electrical signals generated by brain activity. Computer software then reads these signals and learns to interpret them. The subject also undergoes a training period where they learn to control their thoughts in such a way as to control something connected to their brain’s output. This could mean moving a cursor on a computer screen, or controlling a robotic arm.
Researchers are still working out the basics of this technology, both hardware and software, but are making good steady progress. There doesn’t appear to be any inherent biological limitation here, only a technological limitation, so progress should be, and has been, steady.
The researchers did something very clever. The goal is to facilitate communication with those who are paralyzed to the point that they cannot communicate physically. One method is to control a cursor and move it to letters on a screen in order to type out words. This works, but is slow. Ideally, we would simply read words directly from the language cortex – the subject thinks words and they appear on a screen or are spoken by a synthesizer. This, however, is extremely difficult because the language cortex does not have any obvious physical organization that relates to words or their meaning. Further, this would take a high level of discrimination, meaning that it would requires lots of small electrodes in contact with the brain.
Therefore, they did not look at the language cortex. Instead they had the subject imagine they were writing words. They implanted electrodes in the motor cortex, which is activated when you imagine movement, even if you are paralyzed and cannot actually move. The subject has a spinal cord injury, so their brain is intact but they have partial paralysis. The computer then has to interpret the motor cortex activity and decide which of 24 letters the subject is imagining writing. That creates a much more finite space, rather than having to interpret thousands of words. The motor cortex also has what is called somatotopic mapping (the motor homunculus) so we know what each part of the cortex does. Further, the hand has a lot of cortical representation – it’s controlled by a large part of the motor cortex. All of this likely contributed to the success of this approach.
The subject therefore imagined himself writing words with his hand and the computer software, using a neural network (a type of computer learning) deciphers the motor cortex activity.
The main limitation here is that this required invasive surgery to implant electrodes on the brain. That is not that big a deal, but it is more invasive than scalp surface electrodes, which likely would not have worked. I would be interested to see this study replicated with endovascular electrodes – putting them in the veins in the brain. Perhaps getting under the skull and near the cortex is enough.
But in order for this technology to continue to progress is seems likely that we need more advanced electrodes. Perhaps the best bet so far is the microwire approach. Safety and longevity are also key. The electrodes have to flex with the pulsations of the brain so that they do not move in relation to brain tissue. This would also reduce scar tissue formation and increase the longevity of the wires.
But even at this current state of technology, this system would allow, for example, someone who is locked in to communicate fairly quickly. That would be a life-changer for someone in that condition. Being able to read the motor cortex is also an obvious way to control robotic limbs. Using the method of imagining a specific movement and then connecting that to a particular outcome, this could allow for a wide range of control through a computer interface. It’s feasible that with current technology a system could be developed so that a person suffering from locked in syndrome, quadraplegia, or paralysis from ALS could control a motorized wheelchair equipped with a robotic arm, and also connected to a voice synthesizer that they can use to communicate at slow but natural speeds. The computer could also be connected wirelessly to their entire home, that they can therefore control. They could have a computer screen mounted in front of them, and they could control a virtual mouse giving them access to the world through the internet.
Again, the primary limiting factor is the longevity of implanted electrodes. That’s why I would love to see this study done with the stentrodes, or microwires, or flexible electrodes. Once we have safely implantable electrodes that are stable for decades rather than years, there is no reason not to do this.