From unaltered raw waveform to emotion: Synergizing convolutional and gated recurrent networks for holistic speech emotion analysis

Recognizing emotions from speech remains a challenging task due to the variability of vocal expression and the information loss associated with conventional feature extraction methods. This study introduces CNN-n-GRU, an end-to-end deep learning architecture developed to perform speech emotion recognition directly from raw waveform data. The model combines an n-layer convolutional neural network for extracting hierarchical local acoustic features with an n-layer gated recurrent unit designed to model long-term temporal dependencies in speech. The convolutional component acts as a feature extractor, generating progressively abstract representations of the input signal. These features are then passed to the gated recurrent layers, which capture sequential information and selectively retain emotionally relevant cues. By eliminating the need for handcrafted features or spectrogram transformations, the model preserves narrow-band emotional information and can efficiently handle speech signals of varying durations without explicit segmentation. The design draws conceptual motivation from auditory perception: the initial convolutional layers emulate Cochlear frequency selectivity, while the recurrent layers mirror the brain’s ability to focus on salient acoustic patterns over time. The proposed architecture is evaluated on three benchmark datasets. On TESS, the model achieves 99.2% accuracy and 99.0% F1-score, on the IEMOCAP dataset, it reaches 81.3% accuracy and 80.9% F1-score, and on the RAVDESS dataset, it attains an accuracy of 86.6% and an F1-score of 86.7%. These results represent improvements over state-of-the-art methods, with statistical significance and additional analysis supporting the consistency and robustness of the performance. Although effective in controlled data sets, the generalizability of the model to spontaneous emotional speech and speaker variability remains an open area for exploration. Future work will focus on expanding its applicability to more diverse, real-world acoustic environments. The source code of this work is available at: https://github.com/alaaNfissi/CNN-n-GRU-for-speech-emotion-recognition.