Emotion-Tagged Streaming Data Architecture for Human-Centric Analytics

Abstract

The increasing number of data streams available in real time, such as those produced by social media, wearable physiological data, surveillance cameras, and conversational interfaces, has opened an unprecedented opportunity to monitor the content of affective states on a large scale. To enable human-centric analytics at scale, this paper introduces a novel Emotion-Tagged Streaming Data (ETSD) architecture that combines distributed stream processing frameworks with multimodal emotion recognition pipelines. The architecture is inspired by discrete categorical models of affect such as those proposed by Ekman [4] and the continuous Valence-Arousal-Dominance (VAD) model that is rooted in Russell's circumplex model [13] to label each streaming event with a structured emotion label before it is persisted and analyzed downstream. A survey of recent face detection deep learning classifiers [8] and speech [5] classifiers as well as physiological signal [3,9] classifiers and text [14] classifiers is integrated in the proposed pipeline to inform the selection of the modality. According to the experimental benchmarks, the proposed architecture can reach an overall emotion classification accuracy of 91.3%, an end-to-end emotion tagging latency of 22 milliseconds at a throughput of 900,000 events per second with an accuracy gain of 19.9 percentage points compared to AVEC 2018 baseline [12]. Ethical considerations, consent, algorithmic bias, risk of surveillance are systematically listed [16].