Alisa Brownlee, ATP, CAPS blog offers recent articles and web information on ALS, assistive technology--augmentative alternative communication (AAC), computer access, and other electronic devices that can impact and improve the quality of life for people with ALS.
Email--abrownlee@alsa-national.org.
Any views or opinions presented on this blog are solely those of the author and do not necessarily represent those of the ALS Association.
The finalists have been selected in the Invent 2015 competition, which aims to turn the best innovations into businesses
By Margaret Canning – 01 June 2015
Wearable technology enabling sufferers of 'locked-in' syndrome to communicate by translating their brainwaves is among the finalists in a competition to find our best inventions.
Dr Damien Coyle from NeuroConcise has been working on his innovation for 15 years - and now it's one of 12 to make it through to the final of the Invent 2015, a programme run by the Northern Ireland Science Park.
The Londonderry man said: "NeuroConcise will benefit people who have no means of communication whatsoever.
"A lot of them have been in a so-called "locked-in" state or in some cases, people don't even know if they are aware of their surroundings or not. Our technology will first of all provide clarity around their awareness and secondly, offer them a movement-free communication channel."
The technology can non-invasively measure and translate brainwaves into control signals, allowing people to communicate and interact without moving.
Dr Coyle had to pitch his technology to a team of entrepreneurs to be in with a chance of winning a position on the NI Tech Mission to California, a share of the £33,000 prize fund, as well as mentoring, networking and business development from the NI Science Park's NISP Connect.
The final of the programme will take place in October - but last Thursday night, 24 hopefuls had to present their concept, before facing questions from the judges.
Other finalists who emerged from Thursday night's event include Chirag Gujral of Queen's University, who has discovered a method of reducing the effects of eye diseases, and brothers from the north coast who have used their love of surfing to set up their own surfboard company Skunkworks.
The animal world is also represented by EquiNutritive Equi-Liquid Gold - an anti-inflammatory supplement for horses which helps their general well-being.
Alexandra Frazer from Equi-liquid Gold said: "I'm looking forward to what's to come over the next few months in the lead-up to the final.
"I definitely want to tap into the export markets and expand with other product ranges and become a leader in the equine supplement industry. It's a long road ahead but this competition is definitely an amazing step up the ladder for new technologies."
Organiser Peter Edgar said: "QuickPitch is the semi-final stage of Invent, which is all about telling the story of a breakthrough idea which has the potential to change an industry or solve a global problem which can make people's lives better. It's fast and exciting."
Julie Ann O'Hare, director of business banking at Bank of Ireland UK, which sponsors Invent 2015, said: "I congratulate all 24 finalists on their excellent pitches and encourage them to stick with it."
Sesame Enable co-founder and CEO Oded Ben Dov (right) with a user of the company's smartphone. Basti Hansen
In some quarters, the growth of the smartphone market may finally be slowing, but it's just getting started elsewhere.
Sesame Enable, a company that modifies smartphones so that people with disabilities can control a cursor with only slight head movements, just shipped its first 50 phones, said Oded Ben Dov, the Israel-based company's co-founder and CEO.
That means people with spinal cord injuries, cerebral palsy, multiple sclerosis, Lou Gehrig's disease, muscular dystrophy, severe arthritis and other conditions can now make unassisted phone calls, check email, play Angry Birds and just about anything else a typical smartphone addict does daily.
All they need to do is say "Open Sesame" to turn the phone on, and then look at the phone's screen so its camera can capture slight left and right head movements.
About 2 million people in the U.S. could benefit from such a device, Ben Dov said. The market is indeed large. All told, about 6 million Americans live with some form of paralysis, according to Peter Wilderotter, president and CEO of the Christopher & Dana Reeve Foundation.
"Smarter devices that can be activated by voice, vision or slight movement have the potential of transforming many lives, increasing independence, access and communication," he said in an email.
Basti Hansen
The foundation has already been in touch with Sesame Enable and has been keeping tabs on other companies investing in related technologies, such as Nokia (NOK), Samsung, Apple (AAPL) and Microsoft (MSFT).
"Other tech trends that will benefit individuals living with disabilities include the connected home -- or smart home. As an emerging innovation, there are a number of players at the table from Verizon (VZ) to AT&T (T), Samsung, GE (GE) and even Google (GOOG)," Wilderotter said.
Sesame Enable's other co-founder, Giora Livne, was also its first customer. An ex-navy officer and electrical power engineer, Livne is now a quadriplegic as a result of a fall off a ladder. The co-founders started the work with their own money, and received $215,000 from an Israeli Office of the Chief Scientist matching grant and an angel investor.
From there, they launched a $30,000 Indiegogo crowdfunding campaign, which turned into gravy once they won the $1 million Verizon's Powerful Answers Award and then $100,000 from Michael Bloomberg's Genesis Prize.
They're not courting the venture capital market at this point, in part so they aren't pressured to focus on developing the technology for games or another larger market.
"I really think our focus on the disability market is also the right thing to do morally, and also from the business angle were building on a very solid need," Livne said.
With the unexpected Verizon grant in hand, they were able to give away 23 of the first phones. "We decided to donate all funds raised on Indiegogo -- past, present and future -- toward devices that will go to people in need," Ben Dov says in his updated video on the crowdfunding site. People who want to buy a smartphone directly can also purchase one on the site for $700.
Basti Hansen
Currently the phone is only available as a Google (GOOG) Nexus 5, but the company is working on other models, including tablets, Ben Dov said. The Sesame Enable technology is something of a cross between an app and an operating system, and can be used with almost any app in the Google Play store. They're working on integrating with Google Now personal assistant and a voice command that will allow users to dictate emails rather than tap out text on a keyboard with the head-controlled cursor.
Now that Sesame Enable has gone from futuristic prototype to Version. 1.0, users are offering feedback and are anxious for more.
"What really surprised me, the questions were already into the experience: 'How do you do this, how do you change that?'" Ben Dov said. "We got the 'Oh my god, this is amazing!' six months ago when we had the crowdfunding campaign. Someone had said he's been waiting for this technology for the past 15 years. 'You have no idea what you've done. It's like you've opened the Matrix.' But we're past that excitement, and now it's: Let's get down to business."
In the field of assistive technology, scientists, engineers and entrepreneurs are also making life easier for people with disabilities, chipping away at truly big problems at steady pace. Their successes are likely to accelerate.
“I think we’re in the middle of a revolution in technology for people with impairments,” says David Reinkensmeyer, a biorobotics researcher at the University of California, Irvine.
“This field, from when I started 15 years ago until now, has really seen an attraction of younger people wanting to get involved,” says Alex Mihailidis, a University of Toronto professor who holds a chair in rehabilitation technology.
“People are seeing this as a challenging and exciting application.”
Richard Lautens
Wire inputs used in a video game that helps Parkinson's patients retain their ability to speak clearly. Users attach sensors to their mouths and then control a dragon that lights trees on fire.
Yet these researchers are often confronted with hurdles that designers of mainstream tech — whether silly or genuinely useful — are not: regulatory barriers, marketplace fragmentation, and even our own unconscious biases.
Growing up in Alberta, Gary Kurek was a high school science-fair phenom, ranking provincially, nationally and internationally. In his senior year, after watching his grandmother struggle with the effects of cancer, he set out to build a better wheelchair.
His novel design piqued the interest of Silicon Valley investor Peter Thiel, who chose Kurek for a prestigious $100,000 Thiel Fellowship for entrepreneurs under 20. Among other mobility devices, Kurek built a wheelchair that could climb stairs.
But he eventually dropped the project, in large part due to regulatory struggles.
“We got really bogged down . . . It was going to take a ton of money and a ton of time,” he says. The market for his product wasn’t big enough to justify the investment. Though he would like to come back to mobility products, Kurek is now working on automated manufacturing.
Because many assistive technologies fall under the banner of medical devices, they must be approved by oversight bodies such as Health Canada or the U.S. Food and Drug Administration. Those in the field acknowledge that safety is paramount, but say the process can sometimes stifle innovation: it is lengthy, costly, and “still quite prohibitive to a lot of startups,” says Mihailidis. “That’s one of the big next hurdles we need to tackle as a field.”
Compounding that problem, the market for assistive technologies is highly fragmented, even though it is large and growing rapidly with the aging populations of most developed countries.
“For sure there’s a big, big need ... but the market is pretty heterogeneous, in the sense that there are a vast numbers of impairments that cause disabilities to arise,” says Melanie Baljko, a York University professor who specializes in assistive technology.
“Everyone experiences their impairment differently.”
Even reaching those consumers is a challenge. Many people who need these products don’t buy them directly: insurance companies, caregivers and clinicians all act as mediators.
At the same time, other technological advances have begun to carve new pathways forward. The advent of 3D printers, Baljko and Mihailidis both note, has fuelled a thriving DIY culture where designs for items like modified light switches and spoons are shared openly online, circumventing the need for a traditional marketplace and allowing for low-cost customization.
“This is really putting the power of technology development into the hands of people who know best,” Baljko says.
Other general advances have spurred assistive tech too: the price of sensors is dropping, and machine-learning algorithms are only getting smarter. Smartphones and tablets have created a platform for the proliferation of interactive games that help people with challenges.
Shifting social attitudes have also helped. “It’s becoming increasingly socially acceptable to have technology on your body — Apple Watch for example,” says Reinkensmeyer. “We’re all just using technology to enhance our abilities.”
But attitude is the most invisible barrier for assistive tech, he adds. “You know what holds us back? The attitude the people can’t: ‘You can’t have a spinal cord injury and play basketball, or ski . . . ’ How long did that societal attitude hold back the development of technologies?” he asks. “We don’t even know what our assumptions are.”
For Kate Allen, writing about assistive technologies hits home
Peter Allen chats with Yana Yunusova, a U of T professor of speech language pathology who is beta-testing a video game that helps Parkinson's patients.
“Kate I hope that your gum problem has lessened and then you I know a virtuous Flossie — try this again — and that you are a virtuous Flossie to Flossie newsflash to the Flossie Turley — what I’m trying to say is if you have seen the error of your ways and now floss your teeth regularly — my God!, It said what I wanted to thank you for his time to say goodbye.”
In case you couldn’t tell, he’s trying to urge me to floss better (I’m 30, for the record). But the voice-recognition software he uses — designed to circumvent his shaky hands — clearly had other ideas.
My English-professor father sent the garbled email for hilarity’s sake, as he often does. I forwarded to my brother for laughs and forgot about it. But a few weeks later when I was aflutter over a new app that offers on-demand lunch delivery, I stopped and thought: this is ridiculous. Why are so many people trying to make it easier for me to buy a sandwich, when it was already really, really easy for me to buy a sandwich? How many people are trying to “disrupt” the crappy technology available to my Dad, who actually needs it?
Richard Lautens / Toronto Star Order this photo
Peter Allen, who has Parkinson's disease, poses with his daughter, Kate, for a selfie with sensor wires hanging out of his mouth. Researchers at the Toronto Rehab Institute are beta-testing a video game that helps Parkinson's patients retain their ability to speak clearly.
In the course of trying to answer that question, I stumbled across a fascinating new study led by scientists at the Toronto Rehabilitation Institute. The research team, led by University of Toronto speech-language pathology professor Yana Yunusova, is testing whether using video games to visualize speech therapy exercises will be more effective at helping Parkinson’s patients improve their characteristically slurred speech. Users repeat tongue-twisters with electromagnetic sensors in their mouths, and the better they do, the more fire a virtual dragon spits out, burning down a stand of trees.
My Dad was game. We showed up the TRI on a Tuesday morning, and the team carefully wired him up. “Super Sue sat sewing,” he said over and over again.
He found the process frustrating — “They have a strange idea of what a game is, I must say,” he wrote me later — but recognized the importance of what the scientists were trying to accomplish, and will return to be a full participant in the trial.
As I discovered, it’s just a lot harder to bring assistive technology to market than it is to build an Uber for sandwiches or an app that only says “Yo.” But it’s not for lack of a community of people who care deeply, and who are working extraordinarily hard.
Four Novel Assistive Technologies
AT-Knee
Patrick Brown/Panos
The AT-Knee offers a unique balance between stability, durability and motion to mimic the movement of a natural knee.
Engineered by researchers in the Bloorview Research Institute at Toronto’s Holland Bloorview rehabilitation hospital and backed by Grand Challenges Canada, AT-Knee’s designers spent years studying how to make it as biomechanically efficient as possible, with a novel locking mechanism that mimics the stability of real knees.
While the trend in prosthetics has been toward high-tech robotics and motorization, LegWorks’ chief technology officer Jan Andrysek notes that the bulk of lower-limb amputees live in countries and communities where such expensive technology is out of reach.
“We came up with very simple concept that provided the function that we need,” said Andrysek. After rounds of clinical testing in Chile, Tanzania, Burma, Canada, and elsewhere, LegWorks is aiming for widespread adoption: “we’re trying to be everywhere.”
Swizzle Shave
University of Toronto
A team of University of Toronto engineering undergraduates designed this razor to accommodate the jerky movements of Huntington’s disease patients. It is a finalist in the 2015 Innovative Designs for Accessibility, a competition for Ontario university students. Michelle Samfira, a co-creator, called the team’s work “really rewarding.”
Music Glove
Courtesy of Flint Rehabilitation Devices
The sensor-filled glove is designed to by worn by stroke patients and played with a game that works a lot like Guitar Hero. The system coaxes stroke survivors to use the kind of hand movements that will help them regain dexterity. It was created by Nizan Friedman, a former graduate student in the lab of University of California, Irvine’s David Reinkensmeyer.
Brian
University of Toronto
Professor Goldie Nejat, who directs the Autonomous Systems and Biomechatronics Laboratory at U of T’s Mechanical and Industrial Engineering Department, designed Brian, a socially assistive robot. Using machine learning, Brian can talk and joke, and is designed to help cognitively impaired seniors accomplish tasks like finishing meals or taking medication on time.
The Centers for Medicare and Medicaid Services (CMS) has issued a number of different policy changes regarding coverage for speech generating devices (SGDs) that have created some confusion and raised significant concerns within the ALS community. The ALS Association’s Public Policy Department has worked on these issues since they first arose and continues to advocate to ensure that people with ALS have access to communications devices that are so vital to living with this disease. The issues are summarized below and include what action The Association is taking and what people with ALS can do to help.
Capped Rental:
Beginning on April 1, 2014, Medicare is changing how it pays for SGDs, switching to a system called “capped rental.” Since 2001, people with ALS always have had the option of renting SGDs; however the overwhelming majority purchases them. Beginning on April 1, people no longer will have that option. Instead, they will be required to rent the device over a 13 month period. During the capped rental period, people with ALS will be contacted each month by the manufacturer to ask whether the SGD will be needed during the next month. As long as the answer is yes, the patient can keep the device. After the 13th month, these monthly questions about further use will stop and the patient will own the device. The payment system change does not apply to anyone on Medicare who currently owns an SGD. Capped rental also does not affect which devices Medicare will cover, or the evaluation and documentation required to support Medicare coverage.
The ALS Association has been actively engaged on this issue since CMS first proposed the change in the summer of 2013. We have submitted formal comments to CMS and have partnered with other organizations, those in industry and with Members of Congress who share our concerns to urge CMS not to make this change. Our concerns are outlined in the comments to CMS, here http://bit.ly/1h2e2X8. They include:
Access: If people have an extended hospital stay or are in hospice while they are in the rental period, Medicare will not cover the rental fees. Instead, the device could be returned to the manufacturer while the patient would have to obtain a new one from the hospital or hospice or pay the entire monthly rental fee out-of-pocket.
Cost: People who rent SGDs for the full 13 month rental period will pay 5% more out of pocket than if they had purchased the device up front. Most recently, several Members of Congress sent a letter (http://bit.ly/1lX98PA) to CMS urging them to: 1) delay the implementation date; 2) reevaluate the data on which their decision was based (CMS relied on 1987 claims data); and 3) meet with stakeholders. A number of meetings have taken place with CMS and a number of scenarios are possible. Those include:
Full implementation on April 1
Delay implementation
Grandfather certain DME introduced to the market after 1987
Grandfather all SGDs since the devices are not intended for short-term use
The ALS Association continues to partner with other organizations to oppose the switch to capped rental and we will continue to strongly communicate that message to CMS. As the regulatory process moves forward, we will provide additional information and will alert the ALS community if grassroots action is needed. If CMS does not act to address our concerns, The Association will pursue other options, potentially including legislation, to ensure people with ALS have access to SGDs that play a critical role in their lives.
In the meantime, if you or someone you know experiences any difficulty accessing SGDs or other Durable Medical Equipment, such as power wheelchairs or accessories, please contact your local ALS Association Chapter immediately. Chapter contact information is available here: http://www.alsa.org/community/. If you do not have a local Chapter, please contact The ALS Association’s Public Policy Department at advocacy@alsa-national.org. The Association will actively provide assistance to anyone experiencing difficulties accessing these devices and also will share these difficulties with CMS officials and Members of Congress so that they fully understand how policy changes impact people with ALS and why they must change.
Dedicated Devices
On February 27, 2014, CMS issued a guideline titled a "coverage reminder" that addresses the types of SGDs and the features of the devices that Medicare will cover. This guideline raises several questions about the features of currently available SGDs, and about the temporary "locking" or "dedication" practice that has been in place since 2001 for computer-based devices. Under current practice, non-medical applications such as email and word processing software, are “locked” on computer-based devices because Medicare will not cover those applications. However, people with ALS subsequently may have the manufacturer “unlock” these additional features by paying a fee, which again is not covered by Medicare.
Both the wording of the document and the manner in which it was issued (coverage reminder as opposed to amending coverage policy) make it unclear exactly what implications it has for SGD coverage now and in the future. There is some speculation that this guideline would end coverage for SGDs or disallow coverage for devices that include non-medical applications such as word processing software, regardless of whether those applications are “locked.” At this time it is just speculation and it is not clear exactly how the guideline impacts current coverage policy. However, it is clear that this guideline does not end Medicare coverage of SGDs and it does not end coverage for computer-based devices, which have been available to Medicare recipients since May 2001. It also does not change the evaluation or documentation required to support Medicare coverage.
As with “capped rental,” The Association is working with other organizations and with industry partners who share our concerns about the potential implications of the guideline. Together we will be working with CMS to clarify the meaning of the guideline and to ensure that people with ALS will continue to have access to SGDs, including computer-based devices. We also will keep the ALS community updated as the regulatory process moves forward and will alert the community if action is necessary.
In the meantime, if you or someone you know experiences any difficulty accessing SGDs, including computer-based devices, please contact your local ALS Association Chapter immediately. Chapter contact information is available here: http://www.alsa.org/community/. If you do not have a local Chapter, please contact The ALS Association’s Public Policy Department at advocacy@alsa-national.org. The Association will actively provide assistance to anyone experiencing difficulties accessing these devices and also will share these difficulties with CMS officials and Members of Congress so that they fully understand how policy changes impact people with ALS and why they must change.
If you have any questions about these issues or would like additional information, please contact the Public Policy Department at advocacy@alsa-national.org.
Without doubts Google Glass has the unique potential to form the basis for a new generation of portable brain-computer interfaces. Now Neurogadget has the honour to introduce to you one of the first Google Glass Explorers who’s been using Glass in brain-computer interface research.
Adriane Randolph, executive director of Kennesaw State University’s BrainLab, together with her team, has developed a working prototype that takes input from an evoked brain response to trigger the four basic interface commands for Google Glass: swipe left, swipe right, swipe down, and tap to select.
While this isn’t the first time we hear about Google Glass being used for BCI purposes, there are significant differences between BrainLab’s work and other similar projects, namely This Place’s MindRDR application, which uses NeuroSky’s MindWave to let users take and share photos on Facebook, just by thinking.
According to Adriane Randolph, “both the MindRDR app and our system currently use a separate bioamplification system to capture and read brainwaves and transmit feedback to an application on Google Glass. Where the MindRDR appears to be using a continuous brainwave such as alpha according to the placement of the sensor and description, we are using an evoked response called the P300. With this ‘aha’ response, we are instead able to overlay several different commands to control Glass. Thus, a user will be able to control more than taking a picture but instead access all of the functionality of Glass.”
P300 The user is presented with a string of characters from which he/she must select and attend to one. The characters flash in a randomized pattern. When the character the user desires flashes, he/she elicits a neural response approximately 300 milliseconds later, called a P300. This response is noted by the computer and a selection made.
In other words, “while the MindRDR allows the user to take pictures while thinking, the BrainLab has been developing with Glass to completely control the user experience of Glass with the user’s brain. That would be the main difference besides the BrainLab’s project being a long-term research-based project”, adds Josh Pate, BrainLab associate.
Desktop to Mobile
Last summer, Randolph was selected to pilot the wearable technology device, Google Glass. She had big plans for her new accessory beyond its everyday use for checking email, taking photos and surfing the web. She intended to expand her BCI research to a mobile platform.
Within a few months, another key member of her research team was outfitted with Google Glass and their study took a new turn. The wireless platform opened new possibilities in working with those with limited physical capabilities.
Instead of nodding, swiping or talking to give commands to Google Glass, the research team developed a method for controlling a mobile device using only brain waves.
“We believe this is the first working prototype designed for the Google Glass platform. We know that selection-type commands exist using neural input, but we had to figure out how to use that in Google Glass in a way that benefits our research,” Randolph said. “We chose evoked responses which are like an ‘aha’ response that we record as surface EEGs as input signals.”
Randolph also told Neurogadget that “BrainLab shares This Place’s excitement that Google Glass holds tremendous possibilities for people living locked-in to their bodies, but who are otherwise cognitively intact. We also recognize Google’s technically accurate statement that “Google Glass cannot read your mind” from the perspective that Glass is not doing the actual EEG-recording and filtering needed to interpret brainwaves. However, as a small computer, Glass is taking the results of this separate processing and using it as input to control embedded apps. The real distinction is in how seamlessly the brainwave processing and feedback to an interface can be implemented. Certainly, in a similar vein of deflecting from Glass’ capabilities of facial recognition, it may not wish to stir up another hornet’s nest by extolling mind-reading capabilities.”
Beside being an enthusiastic Google Glass Explorer, Adriane Randolph has been researching brain-computer interfaces for twelve years and received PhD in Computer Information Systems from Georgia State University. She has directed the KSU BrainLab since its founding in 2007 with hopes to improve the quality of life for people with severe motor disabilities.
More info: http://coles.kennesaw.edu/brainlab
Kotobuki Hayashi, 56, reads out random lines of text at a recording studio in Tokyo's Shinjuku Ward on May 16. | MIZUHO AOKI
Online:
“I bake sweet-chestnut bread,” a volunteer says into a microphone.
“I no longer understand what’s going on,” she carefully reads out next.
The volunteer, Kotobuki Hayashi, 56, is reading short lines of text popping up on a computer screen in front of her. The phrases have been taken randomly from newspapers and books. Studio staff check for any misreads.
In an hourlong session, Hayashi gets through about 150 phrases that will be used to create synthesized voices for people with amyotrophic lateral sclerosis, also known as ALS or Lou Gehrig’s disease, who can no longer speak.
“I thought it would be great if I could help those people simply by recording my voice,” Hayashi said. “Also, it’s exciting to imagine that fragments of my speech will be used to reconstruct voices.”
Hayashi is one of around 200 volunteers who have participated in a so-called voice bank project that kicked off in November last year. The goal is to reconstruct the voices of ALS sufferers by creating synthetic ones using an archive of other people’s voices.
The technology was developed by a team of researchers led by Junichi Yamagishi, an associate professor at the National Institute of Informatics who specializes in speech synthesis.
ALS is a progressive neurological disease that attacks the nervous system and paralyzes muscles. As it develops, patients lose the ability to speak, and in some cases can lose their voices within six months of diagnosis, experts say. One ALS sufferer widely known worldwide is physicist Stephen Hawking.
According to the Japan Intractable Diseases Information Center, there are some 9,000 people with ALS in Japan. Many communicate by typing into a personal computer or tablet PC by using whatever muscles they still have, and having a synthesized voice read it out loud. But the voice sounds impersonal and robotic.
“Patients very much needed to communicate in their own natural voice. But no such system existed that could provide a personalized synthesized voice for them,” Yamagishi said in a recent interview in Tokyo with The Japan Times.
Yamagishi and his team set out to re-create their own voices, initially in trials at Britain’s Edinburgh University in 2011.
The project has been running for three years and has seen about 600 volunteers take part, with 10 patients using the software. It is considered to be in its evaluation phase.
In Japan, the project is still in its initial phase and Yamagishi needs to collect as many voices as possible. The recording is done at rented studios in Tokyo, Osaka and Nagoya.
He anticipates it will take one or two years to develop a synthesizer but thinks it could help people with ALS and possibly other disorders.
Yamagishi’s system analyzes the recordings, processing them by using statistical models of the components of speech, and produces a basic voice model for each age group, sex and dialect. This model then serves as the framework for synthesizing the patient’s voice.
“It’s like transplanting part of the volunteers’ voices,” he said. “We find donors whose background matches the patients’ voices, such as in terms of age and home town. We then transplant elements of the donors’ voices, such as the speed with which they move their tongues,” to reconstruct the patients’ voices.
Some companies in Japan already conduct personalized voice synthesis by cutting and pasting recordings of the patients’ own voices. However, this requires hours of recording and is physically impossible for some ALS patients or for those who are already mute.
Yamagishi’s technology requires a 5-minute recording of a patient’s voice. Even if some of the words cannot be pronounced, the system can draw on examples from volunteers to guess at the patient’s original pronunciation.
It also helps to hear the voices of the patients’ siblings, Yamagishi said, since close relatives often have a similar accent or tone.
“There is no particular cure for ALS and it’s really hard for their families to see (their loved ones) develop the illness,” Yamagishi said. It’s important for families to have something to help improve the quality of the patients’ lives even a little, and the voice bank may be one of those things, he said.
“Now we are conducting a large-scale demonstration experiment . . . I want many volunteers from all the regions across Japan,” he said.
By Erica Klarreich Communications of the ACM, Vol. 57 No. 3, Pages 12-14 10.1145/2567649
A patient wears a cap studded with electrodes during a demonstration of a noninvasive brain-machine interface by the Swiss Federal Institute of Technology of Lausanne in January 2013.
Credit: Fabrice Coffrini / AFP / Getty Images
Mind reading has traditionally been the domain of mystics and science fiction writers. Increasingly, however, it is becoming the province of serious science.
A new study from the laboratory of Marcel van Gerven of Radboud University Nijmegen in the Netherlands demonstrates it is possible to figure out what people are looking at by scanning their brains. When volunteers looked at handwritten letters, a computer model was able to produce fuzzy images of the letters they were seeing, based only on the volunteers' brain activity.
The new work—which builds on an earlier mathematical model by Bertrand Thirion of the Institute for Research in Computer Science and Control in Gif-sur-Yvette, France—establishes a simple, elegant brain-decoding algorithm, says Jack Gallant, a neuroscientist at the University of California, Berkeley. Such decoding algorithms eventually could be used to create more sophisticated brain-machine interfaces, he says, to allow neurologically impaired people to manipulate computers and machinery with their thoughts.
As technology improves, Gallant predicts, it eventually will be possible to use this type of algorithm to decode thoughts and visualizations, and perhaps even dreams. "I believe that eventually, there will be something like a radar gun that you can just point at someone's head to decode their mental state," he says. "We have the mathematical framework we need, for the most part, so the only major limitation is how well we can measure brain activity."
In the new study, slated to appear in an upcoming issue of the journal Neuroimage, volunteers looked at handwritten copies of the letters B, R, A, I, N, and S, while a functional magnetic resonance imaging (fMRI) machine measured the responses of their primary visual cortex (V1), the brain region that does the initial, low-level processing of visual information. The research team then used this fMRI data to train a Bayesian computer model to read the volunteers' minds when they were presented with new instances of the six letters.
"It's a very elegant study," says Thomas Naselaris, a neuroscientist at The Medical University of South Carolina in Charleston.
According to Bayes' Law, to reconstruct the handwritten image most likely to have produced a particular pattern of brain activity, it is necessary to know two things about each candidate image: the "forward model," the probability that the candidate image would produce that particular brain pattern; and the "prior," the probability of that particular image cropping up in a collection of handwritten letters. Whichever candidate image maximizes the product of these two probabilities is the most likely image for the person to have seen.
To create the forward model, the research team showed volunteers hundreds of different handwritten images of the six letters while measuring their brain activity, then used machine-learning techniques to model the most likely brain patterns that any new image would produce. To construct the prior, the team again set machine learning algorithms to work on 700 additional copies of each letter, to produce a model of the most likely arrangements of pixels when people write a letter by hand. Both models used simple linear Gaussian probability distributions, making brain decoding into a straightforward calculation, van Gerven says.
"We've shown that simple mathematical models can get good reconstructions," he says.
The research team also experimented with limiting the model's prior knowledge of the world of handwritten letters. If the model's prior information consisted only of images of the letters R, A, I, N, and S, for example, it could still produce decent reconstructions of the letter B, though not as good as when the prior included images of all six letters. The results, van Gerven says, demonstrate the decoding algorithm's ability to generalize—to reconstruct types of letters it has never "seen" before.
The human brain is, of course, the master of this kind of generalization, and this ability goes much farther than simple reconstruction of unfamiliar images. "The visual system can do something no robot can do," Naselaris says. "It can walk into a room filled with things it has never seen before and identify each thing and understand the meaning of it all."
While van Gerven's paper deals only with reconstructing the image a person has seen, other researchers have taken first steps toward deciphering the meanings a brain attaches to visual stimuli. For example, Gallant's group (including Naselaris, formerly a postdoc at Berkeley) has combined data from V1 and higher-order visual processing regions to reconstruct both the image a person has seen and the brain's interpretation of the objects in the image. More recently, in partially unpublished work, the team has done the same thing for movies, instead of still images.
"We are starting to build a repertoire of models that can predict what is going on in higher levels of the vision hierarchy, where object recognition is taking place," Naselaris says.
Other researchers are working on reading a brain's thoughts as it responds to verbal stimuli. For example, in 2010, the laboratory of Tom Mitchell at Carnegie Mellon University in Pittsburgh developed a model that could reconstruct which noun a person was reading. Van Gerven's lab is currently working on decoding the concepts volunteers consider as they listen to a children's story while inside an fMRI scanner.
Most mind-reading research to date has focused on reconstructing the external stimuli creating a particular pattern of brain activity. A natural question is whether brain-decoding algorithms can make the leap to reconstructing a person's private thoughts and visualizations, in the absence of any specific stimulus.
The answer depends on the extent to which, for example, the brain processes mental images and real images in the same way. "The hypothesis is that perception and imagery activate the same brain regions in similar ways," van Gerven says. "There have been hints that this is largely the case, but we are not there yet."
If, Naselaris says, "highly visual processes get evoked when you are just reasoning through something—planning your day, say—then it should be possible to develop sensitive probes of internal thoughts and do something very much like mind-reading just from knowing how V1 works," he says. "But that is a big 'if.' "
Even if mind-reading turns out not to be as simple as decoding V1, Naselaris predicts that as neuroscientists develop forward models of the brain's higher-level processing regions, the decoding models will almost certainly provide a portal into people's thoughts. "I don't think there is anything that futuristic about the idea that in five to 20 years, we will be able to make pictures of what people are thinking about, or transcribe words that people are saying to themselves," he says.
What may prove more difficult, Gallant says, is digging up a person's distant memories or unconscious associations. "If I ask you the name of your first-grade teacher, you can probably remember it, but we do not understand how that is being stored or represented," he says. "For the immediate future, we will only be able to decode the active stuff you're thinking about right now."
Dream decoding is likely to prove another major challenge, Naselaris says. "There is so much we don't understand about sleep," he says. "Decoding dreams is way out in the future; that's my guess."
Part of the problem is that with dreams, "you never have ground truth," Gallant says. When it comes to building a model of waking thoughts or visions, it is always possible to ask the person what he or she is thinking, or to directly control the stimuli the person's brain is receiving, but dreams have no reality check.
The main option available to researchers, therefore, is to build models for reconstructing movies, and then treat a dream as if it were a movie playing in the person's mind. "That is not a valid model, but we use it anyway," Gallant says. "It is not going to be very accurate, but since we have no accuracy now, having lousy accuracy is better than nothing."
In May 2013, a team led by Yukiyasu Kamitani of ATR Computational Neuroscience Laboratories in Kyoto, Japan, published a study in which they used fMRI data to reconstruct the categories of visual objects people experienced during hypnagogic dreams, the ones that occur as a person drifts into sleep. "They are not real dreams, but it is a proof of concept that it should be possible to decode dreams," Gallant says.
The dystopian future Gallant pictures, in which we could read each other's private thoughts using something like a radar gun, is not going to happen any time soon. For now, the best tool researchers have at their disposal, fMRI, is at best a blunt instrument; instead of measuring neuronal responses directly, it can only detect blood flow in the brain—which Gallant calls "the echoes of neural activity." The resulting reconstructions are vague shadows of the original stimuli.
What is more, fMRI-based mind reading is expensive, low-resolution, and the opposite of portable. It is also easily thwarted. "If I did not want my mind read, I could prevent it," Naselaris says. "It is easy to generate noisy signals in an MRI; you can just move your head, blink, think about other things, or go to sleep."
These limitations also make fMRI an ineffective tool for most kinds of brain-machine interfaces. It is conceivable fMRI could eventually be used to allow doctors to read the thoughts of patients who are not able to speak, Gallant says, but most applications of brain-machine interfaces require a much more portable technology than fMRI.
"The hypothesis is that perception and imagery activate the same brain regions in similar ways," van Gervan says.
However, given the extraordinary pace at which technology moves, some more effective tool will replace fMRI before too long, Gallant predicts. When that happens, the brain decoding algorithms developed by Thirion, van Gerven, and others should plug right into the new technology, Gallant says. "The math is pretty much the same framework, no matter how we measure brain activity," he says.
Despite the potential benefit to patients who need brain-machine interfaces, Gallant is concerned by the thought of a portable mind-reading technology. "It is pretty scary, but it is going to happen," he says. "We need to come up with privacy guidelines now, before it comes online."
Horikawa, T., Tamaki, M., Miyawaki, Y., Kamitani, Y. Neural Decoding of Visual Imagery During Sleep, Science Vol. 340 No., 6132, 639–642, 3 May 2013. Kay, K. N., Naselaris, T., Prenger, R. J., Gallant, J. L. Identifying Natural Images from Human Brain Activity, Nature 452, 352–355, March 20, 2008. http://www.ncbi.nlm.nih.gov/pubmed/18322462 Mitchell, T., Shinkareva, S., Carlson, A., Chang, K-M., Malave, V., Mason, R., Just, M. Predicting Human Brain Activity Associated with the Meanings of Nouns, Science Vol. 320 No. 5880, 1191–1195, 30 May 2008. http://www.sciencemag.org/content/320/5880/1191 Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., Gallant, J. L. Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies, Current Biology Vol. 21 Issue 19, 1641–1646, 11 October 2011. http://www.sciencedirect.com/science/article/pii/S0960982211009377 Schoenmakers, S., Barth, M., Heskes, T. van Gerven, M. Linear Reconstruction of Perceived Images from Human Brain Activity, Neuroimage 83, 951-61, December 2013. http://www.ncbi.nlm.nih.gov/pubmed/23886984 Thirion, B., Duchesnay, E., Hubbard, E., Dubois, J., Poline, J. B., Lebihan, D., Dehaene, S. Inverse Retinotopy: Inferring the Visual Content of Images from Brain Activation Patterns, Neuroimage 33, 1104–1116, December 2006. http://www.ncbi.nlm.nih.gov/pubmed/17029988
Figure. A patient wears a cap studded with electrodes during a demonstration of a noninvasive brain-machine interface by the Swiss Federal Institute of Technology of Lausanne in January 2013.
.
