Systematic issues with multiple speakers in realistic sound conditions (And suggested solution)
First of all great work on Edda V0.2! It would easily be my preffered choice for danish transcription were it not for a systematic flaw I found while testing it.
I wanted to see how Edda did on my own testset vs 3 other refference models including Whisper V3. My testset focuses exclusively on spontanious speech - that is speech that is not read out loud. It covers a mixture of improvised dictation and actual conversations between 2 speakers. In total the testset is around 1.5 hours of speech.
It turns out Edda was by far the best at transcribing improvised dictation but struggled so much with multiple speaker conversations that it ended up performing worse than even Whisper 3 which was the worst of my original refference models. The main issue seemed to be the timeslots where the speaker was changing, leading to significant pieces of audio not being transcribed at all. Interestingly the issue disappeared when testing on podcast audio leading me to believe the issue is related to the change in direction from speaker to microphone. Here are the total averages for context.
Model WER
Best model 16.7%
Whisper V3 21.1%
Edda 22.6%
I would gues that some of the issues are related to Edda being trained so heavily on read out loud speach. In addition I'm fairly confident that you could achieve better results on spontanious speach by using more data augmentation on your training data, simulating more realistic sound conditions of multiple speakers. This combined with a lower total percentage read-out-loud speach in your training-set should decrease the tendency of the model to "forget" the robustness of the original whisper model during fine tuning.
Keep up the great work!
Ahh thanks for letting us know to see @ChristofferSA !
Just in case you haven't seen these two:
- https://huggingface.co/danish-foundation-models/edda-v0.2-duo
- https://huggingface.co/danish-foundation-models/edda-v0.2-large
they are larger but trained using the same recipe. I assume you are probably correct though that it is indeed the recipe that is the issue. I am also slightly worried it might partly be the way that the spont. speech from coral is annotated. I found some similar concerns here when there is overlapping speakers only one is annotated: https://github.com/Rye-A1/danish-asr-leaderboard/issues/114
So at least two possible explanations here (yours is testable though with just a different data distribution).
If you have the data we would be very happy to add it to our test suite then we can consider it for the next release (but totally fair if it is private). Do you have an example of which models perform well on the benchmark? (then I will look into some data distributions).
I made and labeled the dataset as part of my last job, so I'm not sure if I'm allowed to share it - at least I would have to ask my former employer. I'd be happy to test any future models on it, though.
A year ago, when I made the dataset, the best model I tested on it was Sonix, which got a WER of 16.7%. I just wanted to check if any better models had come along since then, which is how I stumbled upon Edda.
However, since writing my original post, I've found that Soniox (not the same as Sonix, despite the similar names) completely beats every other model on the dataset by a large margin, with a WER of 9.6%. It seems to be especially robust to poor sound conditions and is also surprisingly good at medical terms and even beats Edda on spontaneous speech with a single spearker.
Model WER
Soniox 9.6%
Sonix 16.7%
Whisper 21.1%
Edda 22.6%
Yeah the Danish ASR leaderboard doesn’t really cover the breadth of use. But good to hear with the dataset
We should add sonix to the Danish ASR just to see how it performs in those cases
I have added issue on that in Danish ASR benchmark
The edda-v0.2 model have not really been trained on data with poor noise conditions, so it doesn't surprise me it doesn't do well, but is definitely on the todo list for next release.
Yeah might be good to do some evals on noisy data as well - augmented if nothing else
Yeah the Danish ASR leaderboard doesn’t really cover the breadth of use. But good to hear with the dataset
We should add sonix to the Danish ASR just to see how it performs in those cases
I have added issue on that in Danish ASR benchmark
Just to be sure you didn't miss it; Sonix and Soniox are two seperate models with similar names. Soniox and not Sonix was the best one in my test by far.
@ChristofferSA can you say which exactly model have you run
I was running Soniox stt-async-v5 and it seems to perform ok, but not the best on Danish ASR benchmark actually
I used the same as you "Soniox stt-async-v5".
I imagen the Danish ASR benchmark just differs alot from mine. My dataset has a wide range of different sound conditions exclusively created to mimic that of real world conversations where people change directions, move around or mumble on occation. In addition it has an overrepresentation of medical terms though most likely not enough to account for more than 1-2% of the total WER. In general WER for all models on the benchmark tends to be alot higher compared to other benchmarks, even though I use quite a bit of text normalisation on both label and model transcription to mainly meassure errors that actually matter.
As mentioned the WER for single speaker transcription for Edda on my benchmark is still the second best after Soniox.
@ChristofferSA Thanks you so much for detailed feedback, it really helps us to build better models and help community establish more robust benchmark
We have added several issues to Danish ASR benchmark to include more data with real-world multi turn conversations
Happy to help :)
How come you don't have seperate seperate benchmarks for dictation and conversational speech?
I think CoRal conversation task on the Danish ASR benchmark can be counted as conversational speech, but yeah ultimately I think we just currently don't have enough open data to produce as diverse benchmark as we would have liked
That being said I am not an expert on the ASR and benchmark creator @emilstabil might know better here
To clarify, the Danish ASR benchmark is not ours, but it does cover both read-aloud and conversations (though dictation is quite different that read-aloud)