Technology Will Help Us Learn About Emotions for Personal Well-Being

Computers that analyze people’s feelings and stress? Those days are already here. Dr. Rosalind Picard, founder and director of the Affective Computing research group at the MIT Media Lab, introduced the area of research called “affective computing” approximately 20 years ago and is considered a pioneer in this field. Her research promotes collaboration between medical institutions, corporations, universities, and players from many other fields to achieve the ultimate goal of using AI to make human lives better. One such example is the joint research her team is conducting with NTT DATA on “emotion navigation.” This project is designed to analyze the emotions and stress levels of drivers while they are behind the wheel to improve driving safety and overall performance, even helping them arrive at their destination with better well-being. We spoke to Dr. Picard to learn more about affective computing and its applications.

Computing that lends a hand to human emotion

--Please tell us about your research on “affective computing.”

Originally, affective computing was an area of research created to give technology the skills of emotional intelligence. For example, a computer can be taught how to differentiate if the person it is interacting with is bored, agitated, or relaxed. The goal is to create technology that shows people respect, such as by not continuing to do things that cause people to become frustrated or annoyed. Some people say that AI will take over human work someday, but my interest lies in creating the type of AI that respects and expands people’s capabilities, not AI that replaces people. I research how to make technology that will work with humans, show respect for human values and emotions, and positively influence our behavior.

--How is technology used to analyze emotions?

In order to show respect for human feelings, we first must consider “What is a person comfortable to have sensed?” Given that answer, then we use technologies that are capable of reading and evaluating emotions and stress levels based on what sensors people are comfortable with. Perhaps I might be comfortable having the computer process a video input of how I am sitting now, how I am holding my smartphone, the way that I am driving, how I talk, or my facial expressions. Information about our mental state can be measured using video, audio, physiology, or other sensors. The sensor data are then processed using AI, and the results are used to adjust the environment. For example, if I am driving and it senses I am increasingly stressed, it might change the voice of the driving assistant to sound calmer.

Dr. Picard’s prototype, “mood mirror.” The mirror detects the expressions of the persons standing in front of it and overlaps a corresponding icon onto each person’s face. This prototype is designed to show people how the technology is likely to interpret the emotional messages we send with our faces, even when it might not correspond accurately to our true internal feelings.

--The type of sensor and its precision must be important for gathering accurate measurements and data. What kind of sensors do you use now?

Yes, precision and accuracy are very helpful in sensing; however, sometimes we can also get accurate results by smartly combining various low-quality sensors. There are many types of sensors we can combine. For instance, when surveying the driving environment, we may attach sensors to auto parts like the steering wheel, door handle, pedals, and gearshift to detect a driver’s actions. Is he gripping the steering wheel tightly, making sudden movements? A seatbelt sensor may detect heart rate and pulse. Additionally, on the phone, we have motion sensors, video sensors, and sensors to gauge the weather since we know that the weather can have a significant impact on human emotion and stress.

But, we need to exercise caution. There is no magic sensor that will accurately convey how someone is feeling. We need to combine AI learning with lots of information from multiple channels gathered over time to even make a guess at feelings. Even if the person in front of you appears to be laughing or sad, you do not know what is going on under the surface. I think it is probably good that computers are not perfect at this since there are things people want to keep to themselves. However, sharing data from different channels on a daily basis can help elevate AI and machine learning into more reliable tools, and they can at least try to learn to infer what another person might infer about you–even though other people also don’t know what you are truly feeling inside.

Can we forecast tomorrow’s stress, and contribute to preventing depression?

--Would it be possible to analyze daily emotion and stress to forecast tomorrow’s conditions?

I never thought that it would be possible, but several years ago I was pleasantly surprised when I learned it actually works. This finding is thanks to my talented graduate students. The study results are limited right now to college students in the New England area, but we verified that forecasting with a high degree of precision is possible. By comparing data from the previous day, and learning from lots of students with similar lifestyles, we can forecast stress, health condition, and the mood of a student for the following day with accuracy around 80% (compared to random, which in this case is close to 50%). We are in the midst of joint research with multiple companies now, including Japanese companies, using data they are gathering from their employees to learn if this can be extended to non-students.

Inside Dr. Picard’s research lab

--What are the determining factors in forecasting stress and emotions?

It is a complex mixture of factors, and what works best can vary with each person; however, one of the most impactful factors we see is sleep regularity. For most of the students in our study, stress can be reduced by going to sleep and waking up at about the same time every day – making sleep timing more regular. Another big factor in our data is social interaction – positive communication with others at night tends to lower the stress experienced during the following day. The difference was apparent in the physiological features during sleep. Stress could be forecasted by taking a closer look at physiology factors, particularly the electrical measure from the surface of the skin known as the “electrodermal response.” The differences are apparent in the data gathered while sleeping versus while working during the day.

--What will these forecasts make possible?

People will have more control over their personal condition. It may not be possible for you to control physiological characteristics like your body temperature or how much you sweat, but our new affective technologies help you see how to lower your stress by adjusting your sleep timing, or other behaviors that are easy to control. However, the forecast and control should only be accessible and shared by the individual. The important thing is for each person to be able to privately access their own personal data and use it to improve their well-being. Privacy must be respected – if employee data were given directly to an employee’s boss, it could actually hurt that employee’s health and performance.

--Each year, NTT DATA publishes the NTT DATA Technology Foresight, which forecasts trends for the coming three to ten years. One of the trends, Digital Life Science is, “How advanced technology such as digital intervention in gene and brain science will create well-being in the future.” In recent years, depression has become an increasingly grave social issue, but affective computing could truly help people feel better.

Exactly! Today, depression and suicide have become global issues. The World Health Organization (WHO) has stated that by 2030 the number of people affected is expected to exceed that of cancer and stroke. This is a tragic issue, one that the entire society needs to face and work on measures to prevent. Of course, the problem is complex and situations affecting psychological and social factors cannot be resolved in one sweep.

Nevertheless, we are also learning that improving sleep and social relationships can help alleviate depression. Personal religious views, spirituality, and faith also have a significant impact on one’s well-being. A majority of the contributing factors can be changed, and a lot can be done to prevent depression by looking into how these factors interact. We could use sensing data and send, for example, helpful information soon after retirement when many people tend to experience depression, or in stressful situations such as when experiencing financial difficulties. We teach children how to play sports. Likewise, we can teach the children of the future how to protect their inner selves, their “hearts.”

From medical institutions to corporations – applying research to society

--Applying research findings to society is essential in inducing change. Do you have any examples of partnerships you can share?

We are currently partnering with many medical institutions. We are working together with Harvard Medical School doctor – experts on suicide, the depression clinic, psychiatry experts, and many neurologists to explore which parts of the brain are involved in different psychological conditions. In other areas, we are working with children’s medical institutions such as the Boston Children’s Hospital, Brigham and Women’s Hospital, Emory Healthcare Children’s Specialty Services, and New York University. These partners provide daily physical data so we can gain a better understanding of what is happening deep in the brain. We are also working with surgeons to develop a system that enables sensors attached to the skin of a patient undergoing surgery to provide information on the changes in the patient’s psychological condition and emotions.

--I saw a TED Talk you once gave where you touched upon Sudden Unexpected Death in EPilepsy (SUDEP) prevention. Can you tell us about that?

Thank you for sharing information about this – you can help save lives by talking about this. There is still very little known about SUDEP, but it refers to the sudden unexpected death of a person with epilepsy. I never thought I would be working on seizure data, but coincidentally our team discovered the correlation during an experiment in which a participant was wearing a sweatband sensor on his wrist to measure skin conductance, which is often a good indicator of stress.

The experiment attempted to deepen our understanding of the psychological states of children who could not articulate their feelings well. One day, I was looking at sensor data measured from a child’s wrists. I noticed how the measurement of one wrist was abnormally higher than the other. At first, I thought it was a sensor malfunction, but then I learned that the child had an epileptic seizure. Taking a closer look, and running larger studies at Boston Children’s Hospital, we discovered that the wrist signals were pointing toward abnormal electrical activity inside the brain. The abnormal electrical signals triggered changes in the body’s nervous system that could be measured at the wrists. Further, we learned that the type of seizure that triggered the largest skin conductance changes can cause apnea – where the patient stops breathing after the seizure appears to have ended. In some cases, it has been shown that simply stimulating the patient during the apnea can restart their breathing. For example, sometimes just turning them on their side can be life-saving (especially if their face is in a pillow and they are too exhausted to turn their head after the seizure).

The scientific literature shows that people are less likely to die or be injured after a seizure if another person is there. Thus, it is very important that a person not be left alone during the minutes following a grand mal seizure. We created an AI algorithm to run directly on the wristband, to automatically detect grand mal seizures and send the GPS location and an emergency alert out to a patient’s caregiver list when a grand mal seizure was detected. Getting a caregiver there quickly may provide life-saving aid. We thus began developing a product, “Embrace,” which has now become the first-ever FDA- and CE-approved smartwatch, used in Neurology for monitoring grand mal seizures in adults and children age six and higher. I have heard of cases where it has actually helped save lives.

--That is amazing. You are now engaged in joint research with NTT DATA and others in what is called “Emotion Navigation” where you research psychological and physical conditions while someone is driving. Can you tell me more about that?

We have high hopes for this project. The aim is to analyze the cause and impact of stress on the daily lives of drivers so that they can be steered toward greater comfort while commuting to and from work or school. Sometimes a person who has driven home may take their stress from driving out on their families. I believe that alleviating stress while driving can immensely impact the driver and those around them. Our research can be put to excellent use when autonomous and semi-autonomous driving is introduced in the near future.

Physiological sensors and cameras attached to NTT DATA’s driving simulator monitor emotion and stress experienced by the driver under various conditions and environments. According to the condition, the system provides appropriate intervention to improve driver safety and comfort. Know-how accumulated through this research will be applied to building an automotive data analysis platform, and to exploring and innovating technologies that can be adopted by the automotive sector.
Hyundai and Daimler are also taking part in the project.
Emotion Navigation:https://enavigation.media.mit.edu/

--Is there merit in collaborating with businesses on your research?

Yes! Through the initiatives of companies, the findings from our lab can be shared with a larger audience and brought into practical life. And only then can a real future start to take shape. It is extremely exciting to be working with a company like NTT DATA. Going forward, sensors can be embedded in mobile and wearable devices to prevent depression or to help maintain one’s health.

Meanwhile, there is an even greater need for integrity and moral values when handling personal data. Nowadays, the business market is shaped by personal data such as the information available on social media. In an age where a person’s identity can be detected through data, advertisers send you information without regard for your psychological health or mood. Whenever we ask people for their data, it’s important that we first make sure they fully understand and approve of what will be done with their data. People’s data should not be used when they do not understand or approve of the use.

--How should personal data be protected in the future? The EU has enacted laws¹ like the GDPR, but it is also true that the world is overflowing with data on a level that cannot be controlled by a single user.

That is a very important issue. I work with data every day, but even I do not have the time to manage all of my personal data. So, then what? There is a need for such services to be provided by a reliable third party. Just like we trust some banks with our money, we need choices of different platforms that we can trust with our data.

¹The “General Data Protection Regulation” was enacted in the EU to protect personal data. The law stipulates users’ rights in relation to personal data acquired within the EU and is enforced on companies even outside the EU.

--More specifically, what kind of platform is needed?

In the case of medical data, the platform should propose ways to manage and operate data for multiple hospitals and insurance companies. Advertising agencies are not the only companies that want to get their hands on personal data and everyone should be aware of that, understand the value of their personal data, and use it wisely. One of the companies I co-founded, Empatica, offers a system that shares data for medical research purposes if the participants give their prior approval to do so. Physiological data gathered day to day is a precious source of information for pharmaceutical companies to improve their products and services. What if individuals could benefit directly from providing their data? If we can create such an environment, we can create greater value for both individuals and companies.

Interdisciplinary fields supporting ”crazy research!”

--Your work that spans across multiple disciplines is truly amazing, but is that partly due to the environment at the MIT Media Lab?

Yes, I think this work would not have happened had I remained in a traditional electrical engineering department. To start with, I would not have had the courage to write papers about emotion. I started this research more than 20 years ago and was repeatedly told how absurd it was. But the Media Lab director said to me, “We do crazy things! And we *want* to do crazy things.” I said, “Well, this is crazy!”

--So, there was an environment that encouraged craziness.

The Media Lab encourages taking risks and doing things that others might be embarrassed or scared to do. And, there is clearly an interdisciplinary environment here. I am an electrical engineer, but I work on a project that develops sensors, recognizes AI patterns, and dives into several medical fields – including neurology and psychiatry. The ability to share our respective expertise lets us come together to do exceptional work. And we interact a lot with the outside world, letting it influence our research, instead of just listening to other research colleagues. The needs of the real world get us thinking about what we should be doing next.

We should not limit the possibilities of our interest and work for reasons like, “Because I’m a researcher” or “Because I work at XX.” The Media Lab taught me how to step outside such boxes and be curious about everything, regardless of how crazy it seems or if anyone understands.

--Moving on from the interdisciplinary environment I’d like to ask you about diversity – gender, race, culture. As a female researcher, what are your thoughts?

I have often been the only woman in a room full of men. I understand that my presence can make others uncomfortable: Humans, by nature, feel more comfortable with people from a similar background who have the same outlook. But if you really want to solve an important problem, you need different perspectives and different methods. The more different you are, the more help you might bring.

In the business world, you must consider all sorts of cases from the initial phases of product development. Businesses grow when there is input from different perspectives. Diversity at a company is important for a successful business and not just for the sake of ethics.

--Thank you. Lastly, tell us about your future vision for affective computing.

I no longer think affective computing should be just about making more intelligent machines. Respecting people’s emotions while improving people’s lives and their health is a higher bar than AI has aimed for, a greater goal for researchers. To reach this higher goal, we must constantly think about and listen to what it is that people truly need. I would never say that technology is the solution to everything. But emotion and health have not received the understanding and attention that they deserve, so I do believe that there are a huge number of new ways that technology could contribute to aid humankind in this respect. To help each person improve their well-being and to give insight into what helps them feel better, feel happier. That is the kind of future I envision.