OfflineDiarizerManager fails to indentify speakers

[ashayk]

Hi,

I'm seeking some help, likely with configuration. OfflineDiarizerManager is failing to correctly identify different speakers. I'm not sure what do do. I attach a small clip from a podcast. Regardless of the settings I've tried, OfflineDiarizerManager continues to return 30+ segments with only one speaker. Curiously, the last speaker is marked as "S2". I have also performed diarization on an extended version (+ 1 hour) and still with nearly the same results, a failure to correctly identify speakers.

Sortformer does a nearly perfect job, but is impossibly slow to load and process (especially on iOS).

I'm quite stuck and hoping for some help. Thanks.

Matan_Trim_1080.m4a.zip

[Alex-Wengg]

@ashayk could you provide the ground truth for that audio track

[ashayk]

Too big to attach. Please find the full track here.

https://www.dropbox.com/scl/fi/1i976x800ctsfwp2tznus/Matan_Full_1080.m4a.zip?rlkey=i6wyxao7cchbj1i34pjln6mbj&dl=0

It's extracted from this: https://www.youtube.com/watch?v=T3TeKmX13iM

[Alex-Wengg]

You could try lowering clustering.threshold from the default 0.6 to 0.4, or use config.withSpeakers(exactly: N) if you know the speaker count. this "everything is one speaker" behavior is typically caused by the threshold being too conservative.

[ashayk]

Thanks, I've tried. It didn't make any difference. Do you get the same results on that audio with Offline VBX?

[ethanwangakl]

The same problem here. I tried OfflineDiarizerManager and it never worked. So I use DiarizerManager instead now. The DiarizerManager doesn't do a very good job though. Some of my users complain the accuracy.

The Sortformer can do a much better job, but unfortunately it supports only 4 speakers and no embeddings.

FluidAudio is really a great project. Very keen to see improvments for the diarization.

[thechatk]

vDSP_mtrans arguments are swapped in OfflineEmbeddingExtractor.swift

The write loop populates rowMajorPointer with stride frameIndex * speakerCount, producing a [frameCount][speakerCount] layout. Per Apple's vDSP.h, vDSP_mtrans(A, IA, C, IC, M, N) treats A as [N][M]. So N=frameCount, M=speakerCount.

The call passes them backwards:

vDSP_mtrans(
    rowMajorPointer, 1,
    transposedPointer, 1,
    vDSP_Length(frameCount),    // M — wrong, should be speakerCount
    vDSP_Length(speakerCount))  // N — wrong, should be frameCount

This tells vDSP the input is [speakerCount][frameCount] when it's [frameCount][speakerCount]. The output is garbled, each speaker's mask contains data mixed from all speakers.

Effect: Every speaker gets essentially the same mask. All embeddings come out nearly identical. On short files the speakers collapse into one.

Fix: Swap the last two arguments:

vDSP_mtrans(
    rowMajorPointer, 1,
    transposedPointer, 1,
    vDSP_Length(speakerCount),  // M = columns in source
    vDSP_Length(frameCount))    // N = rows in source

[ashayk]

Oh great, have you tested this?

[thechatk]

The vDSP fix is necessary but not sufficient. It helps with longer files, but the pipeline still struggles with shorter ones.

Fix 2: Activity filter

The Swift extractor only skips speakers with zero active frames (baseSum <= 0), which lets through a lot of junk embeddings from overlapping windows. Pyannote's pipeline drops speakers with fewer than 20% clean frames per chunk in filter_embeddings() (clustering.py:77-125). I'm seeing ~1400 embeddings where pyannote produces ~630 on the same file.

After cleanSum is computed (OfflineEmbeddingExtractor.swift:448):

let minActiveRatio: Float = 0.2
if cleanSum < Float(frameCount) * minActiveRatio {
    emptyMaskCount += 1
    continue
}

Fix 3: Binary masks

The segmentation processor computes per-speaker activations by marginalizing softmax probabilities across powerset classes (matrixVectorMultiply), which gives continuous [0, 1] weights. Pyannote uses Powerset.to_multilabel(soft=False), which is argmax on the powerset logits, producing binary {0, 1} masks.

In OfflineSegmentationProcessor.swift, replace the matrixVectorMultiply block (lines 404-419):

let paddedProbabilityBuffer: [Float]
if probabilityBuffer.count < powerset.count {
    paddedProbabilityBuffer =
        probabilityBuffer
        + [Float](
            repeating: 0,
            count: powerset.count - probabilityBuffer.count
        )
} else {
    paddedProbabilityBuffer = Array(probabilityBuffer.prefix(powerset.count))
}

let speakerActivations = VDSPOperations.matrixVectorMultiply(
    matrix: speakerToClassMapping,
    vector: paddedProbabilityBuffer
).map { min(max($0, 0), 1) }

with:

var speakerActivations = [Float](repeating: 0, count: speakerCount)
for speaker in winningSpeakers {
    speakerActivations[speaker] = 1.0
}

winningSpeakers and bestIndex are already computed a few lines above, so this is a net reduction in code.

Test result on a 467s 3-speaker file: 97% F1 vs PyTorch pyannote. All while maintaining x120RTF

[ashayk]

Fantastic! Very grateful for you looking into this. Yup, making these changes has created massive improvement and Offline diarization seems to be working as expected in several different scenarios. Have you forked the project? This should be a pr. If you identify any further improvements, please share them.

[ashayk]

Thanks. The next issue seems to be getting different results on iOS and MacOS (i86). Only difference is that MacOS requires threshold = .8, as opposed to the default .6 which works fine on iOS. Github editor sucks to can't past the rest of the 2nd (iOS) column.

Edit, so far I've been able to tell the calculation of timedEmbeddings in OfflineDiarizationManger produces different numbers of embeddings between iOS and MacOS:

let (timedEmbeddings, embeddingTime) = embeddingResult
    logger.debug("Embedding extraction produced \(timedEmbeddings.count) vectors in \(embeddingTime)s (async)")

<style> </style>

MacOS | iOS

S1: 0:00.00 - 0:02.00	S2: 0:00.95 - 0:02.00
S1: 0:02.76 - 0:03.78	S1: 0:02.73 - 0:03.76
S1: 0:04.00 - 0:05.11	S1: 0:04.00 - 0:05.02
S1: 0:06.84 - 0:08.69	S1: 0:06.85 - 0:08.67
S2: 0:09.96 - 0:13.70	S2: 0:09.96 - 0:13.63
S2: 0:15.89 - 0:22.00	S2: 0:14.83 - 0:16.67
S1: 0:22.78 - 0:34.05	S2: 0:16.68 - 0:21.51
S2: 0:34.83 - 0:40.15	S1: 0:21.68 - 0:25.31
S1: 0:40.79 - 0:43.46	S2: 0:25.31 - 0:26.34
S1: 0:45.78 - 0:49.01	S1: 0:27.13 - 0:30.18
S1: 0:52.00 - 1:00.00	S2: 0:30.18 - 0:31.68
S2: 1:00.00 - 1:07.99	S1: 0:31.68 - 0:32.86
S2: 1:09.06 - 1:12.83	S2: 0:33.41 - 0:40.15
S1: 1:14.00 - 1:17.58	S1: 0:40.81 - 0:43.46
S1: 1:18.21 - 1:23.75	S2: 0:44.83 - 0:46.06
S1: 1:23.76 - 1:42.68	S1: 0:46.06 - 0:48.98
S2: 1:47.67 - 1:52.03	S2: 0:49.59 - 0:53.76
S1: 1:52.03 - 1:56.57	S1: 0:55.07 - 0:57.75
S1: 2:00.00 - 2:04.04	S1: 0:59.25 - 1:00.66
S1: 2:04.32 - 2:05.60	S2: 1:01.32 - 1:07.72
S2: 2:07.09 - 2:08.23	S1: 1:07.72 - 1:09.13
S2: 2:08.25 - 2:09.54	S2: 1:09.13 - 1:12.68

[ashayk]

Bump. Still struggling with the diversion between iOS and MacOS results as per the above. When can we expect the pull request to be merged?