As the Covid pandemic continues to ravage nations and we see spikes in behavioral health needs worldwide, it would be phenomenal if call centers and providers alike could use voice analysis technology to track the disaster’s social impact. Sound like a futuristic reality? Not in Japan.
Shinichi Tokuno, a professor at the University of Tokyo, in collaboration with colleagues and the tech company PST Inc., has developed a smartphone health app that analyzes vocal cord vibration. It’s called Mind Monitoring Systems. MIMOSYS for short.
The MIMOSYS team has been using the app to analyze the Covid disaster’s mental health effects on its app users in Japan. They’ve identified changes in people’s stress levels in response to Covid-related events such as the first reported Covid case in Japan, when the government declared a state of emergency—and when it was later extended and lifted, and the announcement of school closures.
Publicly available in Japan through Google Play and the iPhone app store, MIMOSYS analyzes the user’s voice during calls, storing each analysis in a cloud system. The data is anonymous, and while subjects agree to have their data used, they won’t know if and when it is. Because of anonymity, there’s no way to tailor the data by specific groups, such as healthcare workers, those infected by the virus, or the elderly. “What it illustrates,” says Tokuno, “is how Covid events are affecting the general Japanese population.”
Tracking how people in Japan are responding to the pandemic is dire, he says. Suicide is the leading cause of death for Japanese citizens ages 15 to 29.
Japan has fared far better with Covid than the United States and Europe, but numbers spiked in early 2021, with 7,537 new cases confirmed on January 7, a record high. Prime Minister Yoshihide Suga responded by declaring a state of emergency in Tokyo and three neighboring prefectures. The government lifted the last state of emergency, which was nationwide, at the end of May.
“It’s not just the virus itself that has troubled people,” says Tokuno, “but also the restrictions that have come with it, especially voluntary self-isolation.”
Throughout the pandemic, experts around the world have tried to predict what’s next in behavioral health. Typically, after a disaster, there’s a delay, with psychiatric and substance use healthcare demands sometimes increasing a year or more later. But the Covid pandemic, says Tokuno, is a disaster that has been ongoing for over a year, making predictions without data, well, unpredictable.
What the MIMOSYS Smart Phone App Revealed: Covid’s Social Impact in Japan
In April, Japan’s suicide rate fell by 20%, but by October, more people died of suicide than Covid, with women affected the most. There were 2,153 suicides in Japan in October, while the virus’ death toll at that point was 2,087. Though women make up a smaller proportion of suicides, those who died by suicide in Japan increased by 88.6% compared to the same month the previous year. It’s a concerning trend that began in June—after the nationwide state of emergency started on April 16, 2020—and hasn’t yet let up.
Yasuyuki Shimizu, representative director at the Japan Suicide Countermeasures Promotion Center, told the Japan Times that Covid-related challenges in Japan have disproportionately affected women for many reasons. He said it’s not unusual for women to be the family breadwinner, but they’re often in nonpermanent positions, making them more vulnerable to layoffs and financial hardship. He suspects domestic violence has also played a role. Tokuno agrees, pointing out one more stressor: while women are increasingly employed in Japan, their caretaking and other household responsibilities haven’t diminished. The government closed schools in March and again in April.
The MIMOSYS app detects increases in people’s stress by analyzing voice fluctuations and could help to detect shifts in behavioral health needs after a Covid-related event such as infection spikes, widespread job losses, changes in physical distancing regulations, school closures, or even after a celebrity dies by suicide, which Shimizu told the Japan Times might have contributed to suicide increases in the summer and fall, with the deaths of actors Yuko Takeuchi and Haruma Miura. Tokuno says that people’s voices vary, going up and down a great deal when they’re doing well. The inverse is true when they experience stress. “The more stressed a person is, the less energetic their voice is.”
If widely used, Tokuno says the app has the potential to detect levels of distress, not only of individuals but across communities. “It could help governments and the behavioral health field gauge the distress levels of an entire society,” he says. The data could predict behavioral health needs and encourage increases in government funding for resources.
Tokuno and his colleagues analyzed 8,669 calls made by 90 users in Japan between December 29, 2019 and June 6, 2020. Covid events that coincided with temporary and moderate increases in stress include when Japan reported its first case of infection, the World Health Organization (WHO) declared a public health emergency of international concern, and people disembarked the Diamond Princess cruise ship and went into quarantine in Yokohama. Declines sped up, notes Tokuno, when people began to self-isolate. “That’s what most affected app users,” he says.
Japan didn’t have a mandatory lockdown, but when the government declared a state of emergency in April, it strongly requested people stay-at-home and that non-essential businesses close. (A month later, Shinzo Abe, then Prime Minister, stated in a parliamentary committee meeting in mid-June that if the situation worsened, the government would look into punishable stay-at-home measures. Such measures didn’t transpire.) The decision to abide was entirely each person’s choice, and there were no penalties or fines for refusing. However, that’s when suicides began to double.
Tokuno and his colleagues also noticed that positive occurrences, like holidays, helped buffer the adverse effects of negative Covid-related events except for self-isolation. For example, a holiday took place right before Tokyo Gov. Yuriko Koike announced the possibility of a lockdown. “It,” says Tokuno, “appears to have mitigated people’s anxieties.” However, the impact of voluntary self-isolation was too great, and people’s stress levels continued to rise, even though a holiday preceded it as well.
Another noteworthy finding is that having a stay-at-home end date provided relief for MIMOSYS app users. When the government extended the state of emergency, it did so with an exact deadline for lifting the restriction. “This appears to have made it easier for people to endure the extension by knowing when activities could resume,” says Tokuno.
Tokuno notes that stay-at-home orders had more of a negative impact on MIMOSYS app users that fear of the virus itself. “This may be because Japan hasn’t experienced the skyrocketing and sustained rates faced by other nations.” He says that even as Japan experienced a second Covid wave and the number of infections were on the rise, it didn’t decrease people’s mood because stay-at-home orders didn’t accompany it. That has recently changed, as stay-at-home orders resumed on January 7.
The App Counters Stigma and Self-Reporting Bias
How a voice sounds—its tone, and whether it’s strained or hoarse—is a biomarker we’re all aware of. After all, it’s not uncommon to ask someone we care about, “Are you okay? You don’t sound so good,” when detecting the person may have a cold or had a rough day. What is novel, though, is using data analysis of voice pathophysiology to track health status. Not only does it allow people to monitor their emotions over time, giving them a window into their own mental health, Tokuno says it also helps counter self-reporting bias and stigma.
“There’s often a disconnect,” he says, ‘between what a person says they’re feeling versus how they actually feel.” It’s partly because of mental health stigma, but also that people can’t always identify how they’re doing. In fact, the app might reveal shifts in a person’s distress level before they’ve even realized it themselves.
“It allows users to catch changes or decreases in mood before they might realize them otherwise, letting them know maybe it’s time to go for a walk or reach out to your mental health provider,” Tokuno says.
In some communities, like in the military, people rarely share when they’re feeling low. “They view it as weakness,” says Tokuno. So when a superior asks, “How are you doing?” a soldier will probably answer, “I’m okay, or I’m good,” even if that’s far from the truth.
Tokuno knows this firsthand. He’s spent much of his career working with the military, both as a hospital physician and researcher in computer simulation and catastrophe medical care. He also lost a dear colleague, a physician and friend in the military who died of suicide after returning home from a hospital reconstruction mission in Iraq. The personal loss fueled his interest in voice-based monitoring, particularly for populations where “it’s not okay to say you’re not okay.”
“I needed to find another way to understand soldiers’ mental health,” Tokuno says. “And that’s when I began my research.”
Tokuno started by looking at depression and post-traumatic stress disorder (PTSD), tracking the voice analyses of 1,000 soldiers. He compared the soldiers’ questionnaire responses with the voice snippets and discovered a vital discrepancy.
“In the questionnaire, they said they’re completely okay, but their voices said otherwise, which revealed dangerous levels of distress,” says Tokuno. “They have no way to say they’re not well.”
Tokuno says Japan’s military has a robust psychiatric safety net, but identifying soldiers in need has been incredibly challenging. “Soldiers in crisis don’t want to say they need help, so they don’t get help, and their risk of suicide increases.” He developed the app with his colleagues and PST Inc., hoping to find a way to identify soldiers and others at risk that didn’t depend on self-identification.
“I thought, if we can do this, maybe we can save lives.”
Tokuno highlights that reporting discrepancies aren’t just an issue in the military. In fact, in the general population, people might answer a self-assessment questionnaire in a way that under or overestimates how they’re doing. “Either consciously or not,” says Tokuno. The MIMOSYS app bypasses the subjectivity of self-reporting bias and stigma and gets to what’s really going on.
“That’s what we need right now for individuals and communities at large,” he says.
