The Artificial Intelligence Based Real-Time Emotion Detection Through Voice Using Machine Learning and Deep Learning

Abstract

Speech Emotion Recognition (SER) tries to work out a speaker's emotional state straight from their voice, and it now sits as a core problem spanning affective computing, human–computer interaction, and intelligent healthcare alike. Most published systems, however, are trained and tested on a single, curated, studio-quality corpus, so they end up overfitted, weak on speaker variability, and less accurate than the reported numbers suggest once they meet new voices or noisier environments. This paper presents a real-time SER framework built around two main ideas: combining four different English-language emotional-speech databases into one training set, and fusing cepstral, spectral, tonal, and prosodic features together with their time derivatives into a single representation. We merge RAVDESS, TESS, SAVEE, and CREMA-D into a shared seven-class label space, giving 12,162 utterances from 121 speakers, and then augment this training set at the waveform level. Each speech segment is turned into a fixed-length 284-dimensional feature vector made up of MFCCs (plus their first- and second-order deltas), chroma, mel-band energy, spectral contrast, zero-crossing rate, root-mean-square energy, and spectral shape descriptors. These vectors go into a small deep neural network with batch normalization and dropout, trained with a balanced focal loss to counter the skew in class labels. On a held-out balanced test set, the system reaches 76.7% accuracy and a 77.2% macro-averaged F1 score across the merged four-database problem, with per-class AUROC between 0.94 and 0.99, all while staying fast enough for interactive, real-world use.