THERMALWEAR
Exploring Wearable On-chest Thermal Displays to Augment Voice Messages with Affect
Affective computing | Augmented communication | Thermal display | Wearables | Empirical study
Master thesis 2019.08
Cooperation: DIS group of CWI | Emerging Material Lab of IDE, TU Delft
Supervisor group: Dr. Abdallah El Ali | Dr. Hartcher-O'Brien, J. | Prof. dr. ir. Jansen, K.M.B. | Prof. Pablo Cesar
My work Literature review. Fabrication. Electronics prototyping. Experiment design and execution. Team communication arrangement
INTRODUCTION
ThermalWear is a a collaborative research project between Centrum Wiskunde & Informatica (CWI) and TU Delft. The project investigates wearable, on-chest thermal displays and how they influence voice processing. My work contributes a better understanding of how thermal displays can augment voice perception, which can enhance voice assistants and support individuals with emotional prosody impairments. I tested in a controlled study (N=12) the effects of fabric, thermal intensity, and direction of change. Thereafter, I synthesized 12 neutrally-spoken voice messages, validated (N=7) them, then tested (N=12) if thermal stimuli can augment their perception with affect.
A long paper was developed out of this work, which has been published as a full paper at CHI 2020.
link to the CHI paper: https://dl.acm.org/doi/abs/10.1145/3313831.3376682 link to the thesis: https://repository.tudelft.nl/islandora/object/uuid:393d2261-8f01-406f-acd3-9bb813dc430c?collection=education
Research process
BACKGROUND
Emotional prosody impaired
Emotion coordinates human behaviour and psychological states. It is an important clue in social interaction that helps people to infer others’ conditions. However, emotional prosody production can be situationally or medically impaired. Typical cases are like remote communication and autism.
Human’s daily communication usually consists of two main parts: verbal communication and nonverbal communication. Between the two, verbal communication is usually for information exchange. Nonverbal communication is deeply connected to communicating thoughts and feelings.
In remote communication, nonverbal information is usually not conveyed (text message) or partly conveyed (voice call). For people suffering autism, it’s hard for them to infer others’ emotional states through nonverbal clues. In both cases, the lack of emotional clue might cause misunderstandings and weak social connections.
Neutral expression with affect
Through interviewing about the emotion perception issues when communicating remotely, the neutral expression with affective information is the reason for confusing emotion understanding in most cases. The neutral expression doesn’t match the emotional content, which brings difficulties to perceive others’ emotional states.
RESEARCH SCOPE
PHYSICAL EMOTION PERCEPTION
According to embodying emotion theory, human experience emotion physically. Research has revealed that both the generation and the perception of bodily responses are essential sources of variability in emotion experience and intuition. Therefore, the physical stimulus might enrich the stimulus presentation as an extra dimension.
Assumption: Physical stimuli as a channel to communicate emotion in neutral voice message
In my assumption, the external stimulus will be added as an extra channel. When appraising the emotional states, people get not only linguistic elements but also the physical stimulus related to emotional experience. The expectation is that the physical stimulus will contribute to emotion perception and augment the emotional experience from a neutral voice message.
RESEARCH QUESTION
Before formulating the research question, what modality to study is the first thing to settle. Thermal stimulus has a deep connection with emotion experience and addresses the privacy issue at the same time. Therefore, this project focuses on thermal, which is explored to a limited extent as a potentially interesting means of communication and interaction.
Given the promise of providing feedback at any time or place, it is important to carefully consider offering the stimulus with wearables and what the interaction with fabrics are.
Thus, the research question can be formulated as the follows:
PRE - STUDY
The pre-study helps to determine some factors for the main experiments. The elements explored during the study are:
body location to perceive the thermal stimulus.
the construction and material of the fabric as the contact medium.
parameters of the thermal stimulus including the baseline temperature, the extent of change (EOC) and the rate of change (ROC).
Body location
The choice is made based on two aspects: 1) affective communication. 2) wearability.
Body map of emotion
Nummenmaa et al. propose a bodily map of emotions that is stated to be culturally universal and applicable across child development. In this model, discernible sensation patterns associated with different emotions have a strong correspondence with large changes in physiological functions. Importantly, most emotions were associated with sensations of elevated activity in the central, upper chest area (which likely correspond to changes in heart rate, breathing, and other vitals), or the sensations with two changing directions in limbs. Moreover, prior work has also shown that the central (upper) chest and limbs are both on-body site where the skin is thermally sensitive. This leads to the upper chest and limb area as promising sites for investigation.
Wearability guidelines
To design our ThermalWear prototype, we drew on Zeagler’s wearability guidelines. The choices for on-body location came down to a balance between the desired use of the wearable device and the affordances different parts of the body offer. Among the 13 wearability body maps, three were relevant: Motion Impedance, Garment Manufacturing, and Social Acceptability. The maps can be found here: http://wcc.gatech.edu/content/wearable-technology-affordances-body-maps. To factor in these considerations, I overlay the three maps to find common acceptable areas, which leads to a new body map. From the new map, the potential areas are head, forearm, upper arm, upper chest, the front of thigh, the front of shin and instep.
Combining the location with the potential options from the emotion body map, the body location for experiment is the upper chest.
Fabric as contact medium
Since the stimulus is offered for communication, the fabric used should：
have a high thermal conductivity without delaying the real-time conversation.
The feeling of touching the fabric apart from thermal sensation should be relatively neutral without much effect on emotional feelings.
The construction of the fabric should hold a stable structure for the electronics.
Woven or knitted
I tested the woven and knitted fabric to see which one is better to hold a stable structure for embedded electronics. The result indicates that woven fabric enables a more stable structure. Moreover, research has shown that knitted fabrics have lower values of thermal conductivity in comparison with woven fabrics because they are generally thicker than that of woven fabrics. Therefore, considering the stability and thermal conductivity, woven fabrics are the better choice.
Fibre composition
Given that the fibre composition does not strongly affect thermal conductivity, we ran early tests on material choice between cotton, wool, silk and polyester. For these tests, we applied 4V across fabrics, providing either warm or cold stimulation. Based on this early feedback, we found that silk provides the least irritation, is less sensationally stimulating than polyester, and allows heat to spread evenly (in contrast with cotton). Given this, woven silk with elastane is chosen, which is a lightweight silk that includes Lycra in the weft.
Parameters of thermal stimulus
Human skin sits at a neutral temperature of 30-36°C. I chose 32°C for the touching area between skin and the Peltier element as this is within the so-called neutral zone of thermal sensation. Based on previous work, I consider 6°C as high intensity. I furthermore test 3°C as low intensity, given this was found to be perceivable within brief reasonable detection times, and also to ensure participant comfort.
PROTOTYPING
During the experiments, the participant should experience thermal stimulus with four conditions on the upper chest through the fabric and naked skin and give their feedbacks. Besides, for the second experiment, the participant will listen to voice messages at the same time. A system was built to experiment. The system consists of two shirts (male version, female version) with electronic components, an Android application to perform experiment and collect data, and a monitoring window on the laptop.
Thermoelectric modules
As in prior work, we used Peltier elements (TEC1-127062 thermoelectric modules) as primary thermal actuators, given their capability of providing fast changes in temperature for creating cool or warm stimuli.
Clothes
Hardware components
Components in experiment one
Modules were controlled using a Proportional, Integral, Derivative (PID) controller to ensure accurate temperature control. Each module for this first experiment was driven through a motor driver (L298N Dual H Bridge DC stepper motor3) that was controlled by a custom built ESP8266 ESP-12 microcontroller that is Arduino compatible.
Illustration of the system
Components in experiment two
hardware components for Experiment 2 are shown in Figure 1b. It consists of our custom ESP8266 ESP-12 microcontroller, a 3.3v regulator (so it can be powered with a standard 5V-16V power supply), aMAX31850 thermocouple to digital converter, a K-type thermocouple, a DC motor driver (DCMotor Control Shield BTN8982TA), and a Peltier element attached to a heat sink, that connects to a laptop through a Serial (RS-232) to USB cablewith 115200 baud rate. The board communicates to the Android device via a local HTTP server enabled through a REST API. A thermocouple records the real-time temperature (100ms intervals) of the touching area and sends it to the board. The motor driver enables the change of output value in terms of voltage and direction to offer different thermal stimuli.
Illustration of the system
EXPERIMENT ONE
Research question : What’s the acceptable and distinguishable thermal stimulus in the context of voice message
Participants: 12 (6 male, 6 female)
Independent variables: changing direction of thermal stimulus, extent of change (EOC), contact medium (silk/naked skin)
Dependent variables: subjective intensity, subjective comfort, time to detect the stimulus
Conditions: 4(32°C ± 3°C，± 6°C) * 2(silk, naked skin)
Room temperature: 23°C
Process
Participants in the experiment
Experiment 1 Peltier element on fabric temperature plot
The experiment is completed on the Android application. They first received a calibration session to got familiar with the thermal stimuli and application by experiencing the coolest and warmest stimuli.
Afterward, the main part of the experiment started. Participants experienced 16 trials including four thermal conditions. Eight of them were experienced through naked skin, the other eight were experienced through a piece of silk. Each stimulus lasted for seven seconds. There was a two-minute break between the two parts. Each condition was presented twice. The sequence of the stimulus was randomised. The first condition was counterbalanced across participants. Before every stimulus, there was a ten-second duration to adapt the temperature of the touching area to the baseline of 32°C. In the period of the seven-second stimulus, the participant would press 'I feel it' button once they felt the stimulus and the application would record the time used to detect the stimulus. After experiencing each stimulus, the participant gave feedback in terms of intensity and comfort on the phone application.
Throughout the process, they were encouraged to think aloud about their feelings in terms of thermal sensation and comfort. After going through all the trials, a short semi-structured interview was conducted. They were asked to recall the stimuli experienced and talk about the difference they felt between different conditions. They were also encouraged to give open feedback.
Application for experiment one
Result
Subjective intensity
People can differentiate the high intensity from low intensity.
Low intensities are difficult to detect.
Adding a piece of silk weakens the perceived thermal intensity.
Subjective comfort
People don’t feel warm and cold are different in terms of comfort.
The different intensities don’t have much effect on comfort .
Adding a piece of silk doesn’t have much effect on comfort.
Time to detect stimuli
Adding a piece of silk always delays the thermal sensation.
Warm low intensity: 2.3s
Warm high intensity: 5.4s
Cold low intensity: 0.6s
Cold high intensity: 3.0s
THERMAL STIMULI DECISION
Based on the results from experiment one, I make the decision of using thermal stimuli as follows:
Warm stimulus: 38°C Cold stimulus: 28°C
While higher thermal intensity made a few participants remark on the intense feeling, the comfort ratings indicate overall that this comfort is not different across contact medium. Also, since the thermal stimuli should be detectable for most people within an acceptable range, the low intensities tested are out of options. For the warm stimulus, high intensity (38°C) is used for the second experiment. For the cold one, the intensity is decreased from minus six (26°C) to minus four (28°C) considering the complaints of discomfort from participants.
Without fabric
For the second experiment, people will experience the thermal stimulus in the context of conversation. Since the voice goes very fast, the thermal stimulus should be perceived quickly as well. Therefore, when appraising the stimuli to interpret emotion, people are more likely to relate the two clues together. Adding a piece of silk weakens the subjective intensity of the temperature ratings, prolong detection times, and does not significantly increase perceived comfort. Therefore, the thermal stimulus will be added on people’s skin without fabric.
EXPERIMENT TWO
Research question : How thermal stimuli affect emotion perception from neutrally spoken voice messages?
Participants: 12 (6 male, 6 female)
Independent variables: changing direction of thermal stimulus, the content of voice messages
Dependent variables: valence, arousal
Conditions: 3(baseline:32°C, warm:38°C, warm:28°C) * 2(positive content, negative content)
Room temperature: 23°C
Voice stimuli
Generating
The voice stimuli used for experiment two are from the EU emotion stimulus database, in which each stimulus is described in terms of the emotion expressed, the emotional intensity, the valence, and arousal on 5-point scales. I first narrowed down the choice to audio recordings spoken in UK English. Then, voice stimuli were selected from the database with the following considerations:
The contents of the messages should somehow reveal the valence. Thus, even spoken in a neutral tone, people can still indicate if the speaker is pleasant or not about what he or she says.
Another consideration is to keep the speakers' genders balanced as much as possible and therefore to get rid of the effect brought by gender difference.
Considering the criteria, the scope was narrowed down to 12 voice messages (six positive and six negative). To generate the neutrally spoken voice messages for the experiment, Google Text to Speech, an AI voice generator based onWaveNet, a deep learning based generative model for raw audio. All the messages generated are between 1s to 3s.
Annotation study
To validate that the neutrally spoken messages are perceived more neutral than the normally spoken ones, the 12 messages were rated by seven participants through an online questionnaire. Participants listened to a voice message and gave their ratings in terms of valence and arousal on Self-Assessment Manikin (SAM) technique. From the result, the generated messages are perceived as more neutral compared to the normally spoken ones.
The comparison between the ratings of original voice messages and the generated ones
By looking at the individual mean ratings, I decided to exclude messages 2,6,9 as the arousal ratings were either too high or too low. To ensure a balanced design, I furthermore excluded message 12, as the mean valence was decreasing, and going further away from neutral. Thus, a voice list with eight messages are formulated.
Voice stimuli for experiment two
Process
Participants in experiment
Experiment 2 Peltier element on fabric temperature plot
The process of the introduction and calibration phase is the same as the first experiment.
For the main tasks of the experiment, participants experienced 24 trials (four positive messages, four negative messages, each paired with three thermal conditions including warm, cold and baseline). There were two-minute breaks after trial eight and sixteen to cool the Peltier element down. The sequence of the stimulus was randomised. The first condition was counterbalanced across participants. Before every stimulus, there was a ten-second duration to adapt to the temperature of the touching area to the baseline of 32°C. In the period of the four-second stimulus, the participant would listen to a voice message accompanied with a thermal stimulus which last for four seconds. After experiencing each stimulus, the participant gave feedback in terms of valence and arousal on the 9-point likert scales on the phone application. Throughout the process, they were encouraged to think aloud about their feelings in terms of thermal sensation and the emotion perceived from the voice.
After going through all the trials, a short semi-structured interview was conducted. They were asked to recall the stimuli experienced and talk about the differences they felt between different conditions. They were also encouraged to give open feedback.
Application for experiment two
Result
Valence
Warm stimulus increases the valence for positive messages.
Cold stimulus decreases the valence for both positive and negative messages.
Arousal
Warm stimulus increases the arousal for positive messages and increases the arousal for negative ones slightly.
Cold stimulus increases the arousal for negative messages.
Subjective feedback
Most participants (9/11) stated that for positive messages, warm stimuli make the message even more positive. P3 stated that: "If it’s warm, I can sense the positive emotion is sincere."
For cool stimuli, almost all participants (9/11) found that cool stimuli lowered the valence of the positive message (P1: "I feel it’s not that positive anymore.") and made it less sincere (P8: "He is not speaking sincerely.").
For negative messages, almost half of participants (5/11) did not believe warm stimuli made the message worse.
With cool stimuli, around half (6/11) found it exacerbated the negativity (P2: "It makes it more angry or more severe.").
Few participants (3/11) mentioned that the rate of change was more important than the temperature in interpreting speaker arousal.
Other participants (2/11) stated that when the message content itself lends no clues about valence, they turn to the thermal stimuli for interpretation.
