May 21 2012

Mental Control of a Robotic Arm

Another step has been taken in the research to develop practical brain-machine interfaces (BMI). The basic concept, as I have discussed previously, is to read electrical activity from the brain, then use a computer to interpret that electrical activity and use it to control something, which can be a cursor on a computer screen, a robotic arm, or any other electrical device.

In order for this technology to be useful it has to be possible for the person whose mental activity is being monitored to learn how to control the cursor or robot with their thoughts. Previous research shows that brain plasticity allows for such BMI prosthetics to feel natural – in other words, it is possible to learn how to control external devices through a BMI just as if they were a part of the body.

There are two basic ways to read electrical activity, through implanted sensors or through scalp sensors. The implanted sensors are much better because they are in direct contact with the brain, but then there needs to be some way for the sensor to communicate outside the brain. This is done currently with wires. This, of course, is an invasive procedure and can have complications. The scalp sensors are much safer and easier to apply, but the resolution is much lower as the electrical activity of the brain is attenuated by the skull and scalp – so it’s like looking through a thick pane of foggy glass. For this reason it seems that the future of BMI will be implantable sensors.

A new paper published in Nature reports on two patients who are paralyzed due to brain injury who have had sensors implanted in their brain. One patient, who is a paralyzed in all four limbs due to stroke, had a 96 channel sensor implanted in her motor cortex five years ago. This demonstrates the long term viability and safety of such implants. Further, she has had quadraparesis for 15 year. This means there was ten years between her injury and the implanting of the sensors. This is not a small point – it means that the motor cortex remains viable years after it has been disconnected from the muscles it controls. It is important to note that the subjects did not have injury to their motor cortex itself, but to structures farther downstream – either the brainstem or spinal cord.

One subject, shown in the video, was able to control a robotic arm with enough control to reach out and pick up a cup, bring it to her mouth, and then drink through a straw. The level of control is quite impressive, although still much slower and more clumsy than normal limb movement. The article notes that she learned to do this without explicit training.

I have been fascinated with this line of research for years and as each new study comes out the future of BMI looks brighter and brighter. Every critical step in the process has been demonstrated: sensors can function for years in the brain, they can read brain activity, relevant brain areas remain active even years after injury, the brain’s plasticity can adapt to the sensors which can even feel “natural”, and subjects implanted with such devices can learn to control them well enough to perform specific functional tasks, like eating, drinking, or communicating via a computer. I can’t think of any significant component to the BMI system that has not already been demonstrated in humans.  The BMI paradigm works.

Researchers will still do basic research on BMI to gain more knowledge and familiarity with the details of how such systems can work, but essentially at this point the technology just needs to mature. One obvious improvement will be larger sensors that are able to read a larger portion of brain activity with more sensors. Computer hardware to interpret these signals does not seem to be a limiting factor in any way, but of course continued computer advances can only help. Software development also seems to be advanced enough to read and interpret the signals, but this too will likely continue to incrementally advance. Once you can control a computer, the whole world of information and communication is available. And robotics itself continues to incrementally advance, but is already at the point that useful robotic arms are available.

In addition to controlling a computer screen and controlling a robotic arm, a third possible application of BMI technology is to control the person’s own limbs. This has already been accomplished also – another team used a BMI on monkeys whose spinal cords were temporarily paralyzed. The monkeys were able to control their own arm and hand through a BMI that read their motor cortex activity and then bypassed the spinal cord to directly stimulate the muscles of the arm. The monkeys were able to perform tasks like picking up  a ball. Similar research has yet to be done in humans, but it should work just as well.

It may seem like this would be the ultimate goal of BMI technology – to restore control of one’s own body by bypassing an injury – there is a significant limitation to this approach. Muscles that have lost their nerve supply also lose their function. With an upper motor neuron lesion (brain or spinal cord above the nerves cell to the relevant muscles) muscles will become spastic with involuntary contractions. For lower motor neuron lesions (spinal cord region that supplies the muscles or the peripheral nerves that connect to the muscles) the muscle will become severely atrophied. Supplying electrical stimulation to the muscles does not reverse or prevent these effects, especially the atrophy. Muscles need specific trophic factors from the nerve endings to stay alive and healthy, electrical stimulation is not enough. There would be no point, therefore, in controlling either a spastic or extremely atrophied limb.

These are non-trivial issues that have been researched for years, without a good solution. Controlling a robotic limb, therefore, seems like a more viable option, at least for the foreseeable future. The robot limbs, however, could be designed like an exoskeleton to wear over a subject’s own limbs. This could theoretically enable them to walk, and also could keep their limbs active (a form of physical therapy) to help prevent contractures of the joints.

We are already at the point where BMI technology can be useful to a paralyzed individual. For those who are essentially “locked in” the ability to communicate or manipulate their environment at all is extremely useful. I would like to see this technology go beyond the research phase and be developed for use in  such patients. Meanwhile it will be interesting to see this technology continue to develop.

24 responses so far

24 thoughts on “Mental Control of a Robotic Arm”

  1. mlegower says:

    Crowdsourced proofreading to the rescue!
    para 3: “so it’s like looking through a thick pain of foggy glass” pain = pane.

  2. mlegower says:

    Also, para 4 — “who is a paralyzed”
    para 5 — “bring in to her mouth” & “The articles notes”
    para 8 — “to directly stimulated the muscles of the arm”
    para 9 — “There would be no point, therefore, in control either a spastic or extremely atrophied limb”

    Realize I’m not trying to be pedantic or annoying. I am sure these are just typos. And feel free to tell me that you don’t care and I will shut up in the future. My only thought was that for pieces likely to be referenced by cranks and such (not so much this particular article), it would reduce the potential for ad hominem type arguments.

  3. ccbowers says:

    I am amazed at what people have already done with BMI, but reading this article made me wonder: In order for limbs to feel “natural” there needs to be feedback from the limb as well for propioception. Is this way beyond what is currently possible or is it right around the corner?

    Just curious- I know little about this technology, but without feedback from that limb it seems that the person would have to look at the limb with much greater visual focus, which would be limiting in many ways in terms of performance and how “natural” it would feel.

  4. Eric Thomson says:

    ccbowers, you are talking about the cutting edge, which is known as BMBI (brain-machine-brain interface). Recently someone in our lab published a study in monkeys using BMBI. Article link: Active tactile exploration using a brain–machine–brain interface. The monkey controlled a virtual avatar with brain signals, and the virtual limb sent simulated sensory feedback to the brain to help it perform the task.

    Such feedback from prosthetic limbs should improve accuracy, reaction time, and the “feeling of ownership” of the limb, just as you suggested. My boss wrote a book looking at these issues which is pretty good, if I do say so myself (Beyond Boundaries). It is a semi-popular-level treatment of research in perception, motor control, and neuroprosthetics research.

    I am working on similar stuff, about to be sent out wish I could talk about it. Soon. Literally taking break from working on paper right now.

  5. elmer mccurdy says:

    “Muscles need specific trophic factors from the nerve endings to stay alive and healthy.”

    Can you explain this? I’ve been curious for a while about why you can’t strengthen muscles with electrical stimulation. Is it understood?

  6. Bronze Dog says:

    For lower motor neuron lesions (spinal cord region that supplies the muscles or the peripheral nerves that connect to the muscles) the muscle will become severely atrophied. Supplying electrical stimulation to the muscles does not reverse or prevent these effects, especially the atrophy. Muscles need specific trophic factors from the nerve endings to stay alive and healthy, electrical stimulation is not enough. There would be no point, therefore, in control either a spastic or extremely atrophied limb.

    I did not know that. For a while, I thought it’d be possible to electrically stimulate muscles in some manner that’d ‘exercise’ them, though I recognized it likely would have to be in some precise manner instead of the twitchy gimmick way.

    Of course, another question that comes to mind: Can the gap in the nervous system be bridged? It probably wouldn’t be good for someone who’s been paralyzed long enough for muscle atrophy, but it might be a better option if doctors could use it shortly after the injury.

  7. There has been a lot of research into using electrical stimulation to exercise muscles or stave off atrophy. It just doesn’t work. The muscle cells eventually die from lack of the trophic hormones from the nerve cells. There is no known way around this yet.

  8. Eric Thomson says:

    Steven do you have any references you can recommend?

    I thought stimulation of denervated muscle did help in many instances. For instance:

  9. Matt Lloyd says:

    Thanks for the great article Steve,

    I studied this a bit in my undergrad at SFU under Andy Hoffer (google him for some interesting research). Bronze Dog, practically one of the problems with bridging the gap is that because the sympathetic nervous system (fight/flight) (SNA) has been cut off at the level of spinal cord injury or brain lesion, stimulation of muscles or skin below the injury can cause something called Autonomic Dysreflexia (AD) which essentially causes an enormous spike in blood pressure that can rise as high as 300/150 mmHg (normal is 120/80). It’s hypothesized that this contributes to hemorrhagic stroke in these patients. I’m not sure how prevalent this is in brain injury but it is fairly common in spinal cord injury.

    Steve, AD and orthostatic intolerance in patients with a cervical spinal cord injury (and also possibly a lower level brain injury?) could pose significant problems to exoskeleton-type technology. I wonder if there’s been any research into dealing with these issues? Maybe Eric Thompson could shine a light on this issue.

  10. ccbowers says:

    Eric Thomson-

    Thanks for the information. Very exciting stuff. Its great to be able to find out information like this so quickly (thanks internet).

  11. Eric,

    Thanks for the links. I was not aware of some of this newer research.

    Having read through it, and searching around for other articles, it seems that older methods of muscle stimulation were of no benefit, but this newer technique of high frequency stimulation is of some benefit. Here’s another – -seems to be the longest follow up to date.

    That study shows that after 1 year of daily stimulation (FES) 20% of the subjects were able to stand.

    While this is more than prior research has indicated, we need to keep this in perspective also. One review concluded:

    “Nonetheless, the stimulated-denervated muscles appear to remain abnormal in terms of excitability, force-generation capacity, fatigue resistance, isometric contractile speed, and fiber-type composition.”


    “If stimulation is discontinued, the muscles revert to their former wasted condition. It is therefore inherent in a treatment of this kind that it calls for an ongoing commitment from the patient. ”

    So – good news is that this type of stimulation can significantly increase muscle bulk and function. Bad news is that the results are still significantly limited, and it takes a huge commitment – daily stimulation with the benefits rapidly going away if the regimen is stopped.

    One study suggests that the stimulation partly works by increasing the release of neurotrophic factors. But, seems to still be a lot of questions about exactly how it is working, and what is causing the muscle atrophy in the first place or limiting nerve regeneration.

  12. Eric Thomson says:

    Thanks Steven for the summary. I wonder if coupling stimulation with the release of neurotrophic factors would improve things.

    Also, your point is well taken overall: in many cases a prosthetic limb, or even exoskeleton, is the way to go.

  13. Eric Thomson says:

    That is, I wonder if we could have, at the site of stimulation, a cannula that released a cocktail that included neurotrophic factors. Seems easy enough to try!

  14. Studies of exogenous trophic factors are a bit mixed but underwhelming. Some speculate it’s not the only limiting factor. Schwann cell degeneration is another issue, but there are probably others. Bottom line – we need more basic science to get a fuller picture.

    But – one possibility is to engineer stem cells to release trophic factors and also act as support cells to help keep the muscle cells alive and kicking.

    Also – the exoskeleton idea can be combined with functional electrical stimulation – it provides stimulation daily to keep the muscle bulk up, and also provide extra strength for functionality.

  15. SARA says:

    Once the tech is developed enough, why would we wait long enough for the muscle to atrophy before implanting?

    I’m not sure how long it takes for atrophy to set in, but if it’s more than 3 or 4 months, surely we could implant and all that would be needed is training and rehab.

    In fact, in the case of an obvious permanent injury, it could be implanted much much sooner, right?

    If not, I suppose we would start to seriously look at amputation for the useless living limb and replace it with a useful robotic limb.

  16. naveed.ejaz says:

    Thanks for the short review on the recent BMI studies Steven. One aspect of BMIs that you overlooked was that the limiting factor in invasive studies comes from neural tissue forming around the implanted electrodes which over a period of time degrades the signals recorded from the individual neurons. This both reduces the spatial resolution as well as the quality of the recordings for motor control. I don’t have references off the top of my head but I believe that currently electrodes are able to provide stable recordings over a time period of 5-10 years.

  17. BillyJoe7 says:

    SARA: “I’m not sure how long it takes for atrophy to set in, but if it’s more than 3 or 4 months…”

    My uneducated answer: about 7 days.
    When training for a race, it doesn’t matter what I do in the last week before the race (the two months before that is of course vitally important).

  18. Captain Quirk says:

    This is really cool. Is there anything robots can’t do? *Homer Simpson-esque awe*

  19. Jared Olsen says:

    Very cool. More evidence of the coming robotic revolution…

  20. Mantiki says:

    So when the command to move the robotic arm is issued, is it freewill or an illusion of freewill?

  21. etatro says:

    I wonder whether the visual feedback of seeing the robotic arm move serves as feedback to strengthen the connection. Along with the visual feedback that the robotic arm had moved, comes the sense of satisfaction and reward (via dopamine in the caudate / putamen), in order to strengthen the networks that lead to the initial executive function – decision/desire to move the arm; so that eventually, the basal ganglia will create a shortcut sufficient for the robotic arm-moving to bypass the frontal cortex the way it can with our own arm. Our brain gets feedback from our own arms through various nerve-muscle connections, but this processed through the spinal cord and medulla. We only really know that our arms are in the correct position when we see it in the correct position, feel a landmark. It is possible to have sufficient experience to remember (our caudate remembers, and we might not be conscious of it) what the result of the movement would be, we also get feedback in the form of muscle strain & tension; however the feedback from the result of a motion (seeing your arm in the right position, feeling the stair step on your foot, feeling the lightswitch on your fingers), are quite powerful.

    I would think that the visual stimulus and reward for having achieved the goal would be powerful enough to strengthen the brain-machine connection to make it easier for the user over time.

    I suppose an experiment would be to blindfold the user, instruct her to move the robotic arm, not inform her of the results; and determine whether a user with our without the visual feedback performed better at the tasks over time. The predicted results seem obvious.

  22. In 2003 I was struck by lighting. The current traveled from the top left side of my body, including my left arm, down to my right leg. I have experienced numbness on my right foot ever since (among other symptoms) but more interestingly, my left triceps, which contains a large area of superficial scarring from the (2nd degree) burn, does not respond to growth from resistance training as it used to.
    I’ve searched high and low for an explanation to this but the best answer I’ve got so far is that it must be due to some scar tissue in the muscle.

    I’m wondering if this is in any way related to what you wrote above, Steve, regarding the release of trophic hormones from the nerve cells. Is there a test a specialist could do that would show this?

  23. I love the whole field of BMIs and studied them during my Neuroscience Masters.

    This tech is moving so fast, pretty soon we’ll really be able to help quadriplegics and people with locked in syndrome.

    Resolution is still a major problem with scalp sensors and finding the right materials which will not cause scarring and deterioration of signal over time is a major problem with implanted sensors.

    However, one day we may be able to feed back to the somatosensory cortex.

    Then we’ll really have taken the next step!

    London Skeptic

  24. elmer mccurdy says:

    Anyway, I’m still a bit curious. Which trophic factors from nerves affect muscle growth? I’m afraid Wikipedia isn’t very helpful on this.

Leave a Reply