AI and Empathetic Companionship

1. Context

Humans broadcast emotion constantly through implicit signals — facial expression, posture, and especially voice. When we speak, pitch, tone, rate, pauses, and dynamic range all carry emotional meaning, layered on top of the semantic content of what is said. Yet on digital platforms, these signals mostly go uncaptured, so the information that could help people understand their own emotional state is lost.

Maslo's mission depends on reading that signal well. The company already ran content-based mood insight; the question this study set out to answer was whether the audio itself — how something is said, not just what — could produce a reliable, complementary mood signal.

Predicting mood from behavioral signals can enable artificially intelligent beings to make better empathy-led decisions.

— Maslo, Catching Feels

2. Problem & Constraints

Research question: Can we estimate a person's emotional state from short voice recordings accurately enough to supplement Maslo's existing mood-prediction pipeline — and where does that estimate break down?

• No labeled Maslo audio at scale. The study had to rely on open, acted emotion datasets rather than in-the-wild user recordings.
• Ground truth is genuinely hard. The 'true' emotion label is contested — self-reported mood versus third-party perception — which bounds how good any model can honestly claim to be.
• Class imbalance. Limited samples for some emotions forced a reduction to four labels: angry, happy, sad, neutral.
• Ethical framing. Consistent with Maslo's stance, the output was scoped as a supplementary insight for human-validated use, never an autonomous diagnosis.

3. Method Rationale

• Combine four datasets. RAVDESS, CREMA-D, TESS, and SAVEE were merged into one 12,162-recording corpus (57 female, 64 male actors) to broaden coverage of voices and emotions.
• Prevent data leakage by actor. Actors in the training set never appeared in the test set — a critical, often-skipped safeguard that keeps the model from memorizing voices instead of emotions.
• Random Forest over deep nets. For a first, interpretable pass, Random Forests give feature-importance insight the team could reason about, rather than an opaque score.
• Test a gender-specific hypothesis explicitly. Exploratory analysis suggested performance varied by gender, so three models were trained and compared head-to-head.

4. Process

Feature engineering

For each recording, 44 audio features were extracted with librosa — including Mel-frequency cepstral coefficients (MFCCs), root-mean-square loudness, and polynomial coefficients fitted to spectrogram columns. Because speech is time-series data, features were computed across overlapping time windows, producing a 3-D matrix (7,472 recordings × 44 features × w windows). Summary statistics (min, max, variance, median) per feature collapsed this into a 7,472 × 176 feature matrix for training.

Model design

Three Random Forest classifiers were trained and compared using Area Under the ROC Curve (AUC) via cross-validation:

• A gender-nonspecific model trained on all recordings.
• A female-specific model trained on female recordings only.
• A male-specific model trained on male recordings only.

5. Findings

Gender-specific models outperformed the general model despite training on less data — the central result.

• Precision gains too. Precision improved by up to 20% for predicting 'happy' in females under the gender-specific model.
• Which features mattered. Loudness (RMS), polynomial spectrogram coefficients, and MFCCs carried the highest importance; lower MFCCs mattered more for male voices, higher MFCCs for female voices.
• Interpretation. Because male and female voices occupy different frequency ranges, splitting the data lets each model learn range-appropriate weights and avoids internal gender-based stratification inside the forest — so it prioritizes emotion-specific detail.

6. Synthesis: Interrogating the Method

The study's most valuable contribution is its explicit accounting of assumptions and limitations — the discipline that separates a real result from a demo:

• Can a person feel only one emotion at once? The model forces a single label, but real feelings mix (sad/happy, sad/angry). Proposed fix: use the model's per-label probabilities alongside content-based insight.
• Is this how people actually talk? Training clips were words or sentences; Maslo recordings run up to 60 seconds, which shifts summary statistics. Proposed fix: random sub-sampling with a mode vote.
• A more valid validation. Some sentences overlapped between train and test; the team flagged the need for larger datasets to filter on both actor and statement.
• Gender is not binary. The 'gender' model was reframed honestly as predicting higher- vs. lower-frequency voices rather than gender itself, with further study called for.

7. Outcome

The study produced a concrete, deployable pipeline proposal for Maslo products:

• Use reported gender, or predict a voice-frequency class, for each recording.
• Run the matching gender-specific model and the gender-nonspecific model.
• Aggregate: prioritize the gender-specific predictions where they were shown to help, otherwise fall back to the general model.

The pipeline was designed to supplement — not replace — existing content-based mood insight, and to surface edge cases where audio-based and content-based signals disagree. The code was open-sourced (HeyMaslo/AllTheFeels).

8. Reflection

Catching Feels models the kind of research that earns trust: a measurable result, a production plan, and an unusually candid limitations section that resists overclaiming. The natural next steps are the harder ones — moving from acted clips to real user audio, testing multi-label emotion, and exploring sequence models (e.g., RNNs) that use combined content and audio features. For any team building affective AI, the transferable lesson is that stating what a model cannot yet do is part of the deliverable, not an afterthought.

Note on sensitivity: Emotion and mood inference is a sensitive area. Consistent with Maslo's approach, this work was framed as supplementary insight requiring human validation, never autonomous diagnosis or clinical prediction.