Towards Detecting Neural Audio Codec Synthesized Heart Sounds

Abstract

In this paper, we introduce Synthetic Heart Sound Detection (SHAC), a task aimed at identifying phonocardiograms (PCGs) synthesized using neural audio codecs (NACs). To facilitate research in this direction, we release CARDIOFAKE, the first benchmark dataset for SHAC containing both real and codec-synthesized PCGs.

We benchmark spectral representations (MFCC, LFCC) and self-supervised learning (SSL) representations (e.g., WavLM) for the task. Furthermore, we propose GROOT, a fusion framework that integrates spectral and SSL features for leveraging their complementary behavior.

Experiments show that GROOT, combining MFCC and WavLM, achieves state-of-the-art performance, outperforming individual representations and competitive baselines.

Key Contributions

• 1
  We coin the novel task of Synthetic Heart Sound Detection (SHAC) as a new research direction for secure biometric authentication.
• 2
  We release CARDIOFAKE, the first benchmark dataset for SHAC, containing both real and codec-synthesized heart sounds, and conduct a comprehensive evaluation of spectral and SSL-based representations.
• 3
  We propose GROOT, a novel framework that integrates spectral and SSL representations via a grammian optimal transport (Gram-OT) mechanism, achieving SOTA performance and outperforming individual representations and competitive baselines.

Method

GROOT (Fusion via GRammian Optimal TranspOrT) fuses spectral and SSL representations for SHAC. Spectral features (MFCC, LFCC) are highly sensitive to NAC-induced distortions at the acoustic level, while SSL representations (Wav2vec2, UniSpeech-SAT, WavLM) capture broader temporal structure and variability in heart sounds. GROOT aligns these complementary representations using a novel grammian optimal transport mechanism that compares representations through their gram matrices rather than raw features.

Figure: The GROOT framework. Two representation branches (R1, R2) are each processed by a 1D-CNN and max-pooling, flattened, and aligned via the Gram-OT fusion module before classification.

Gram-OT Alignment

The fused representations F1 and F2 are passed through FCNs with a dense layer of 80 neurons each, concatenated, and processed by a final FCN (120 and 30 neurons) with a sigmoid output layer for binary classification.

Results

Table 1: Individual representations for SHAC

CNN-based downstream models consistently outperform their FCN counterparts. Among individual representations, WavLM with CNN is the strongest performer, and SSL features consistently outperform spectral features (MFCC, LFCC) in both Seen and Unseen conditions.

Accuracy (ACC, higher is better) and Equal Error Rate (EER, lower is better) in %, for Seen and Unseen conditions.

Table 2: Fusion of representations

GROOT consistently achieves the best overall performance across both Seen and Unseen conditions. Heterogeneous fusion of spectral and SSL representations outperforms homogeneous fusion, and the best performance is achieved by fusing MFCC and WavLM through GROOT.

Accuracy (ACC, higher is better) and Equal Error Rate (EER, lower is better) in %. OT: Optimal Transport baseline (same architecture as GROOT, without Gram-OT).

Comparison to SOTA Audio Deepfake Baselines

GROOT (MFCC + WavLM) is compared against AASIST and MiO, strong general audio deepfake detection baselines, trained under the same configuration.

Accuracy (ACC, higher is better) and Equal Error Rate (EER, lower is better) in %.

Dataset and Codecs

CARDIOFAKE pairs authentic phonocardiograms with codec-resynthesized counterparts produced by 7 neural audio codec families, with dedicated seen and unseen evaluation protocols.

Audio Samples

Sample rows below are loaded from Audio/manifest.js. Each row pairs a real heart sound recording with its codec-generated counterpart.

Citation

If you use the CARDIOFAKE dataset or GROOT, please cite the paper as follows:

Update page numbers in the citation once the proceedings version is finalized.