Zach Anderson
Nov 07, 2024 15:59
An in depth comparability of Common-2 and OpenAI’s Whisper fashions reveals Common-2’s superior efficiency in accuracy, correct noun detection, and lowered hallucination charges.
In a complete evaluation of main Speech-to-Textual content fashions, AssemblyAI’s Common-2 has emerged as a high performer when in comparison with OpenAI’s Whisper variants, in accordance with a latest report by AssemblyAI. The analysis centered on real-world use circumstances, assessing fashions on duties important for creating correct transcripts, akin to correct noun recognition, alphanumeric transcription, and textual content formatting.
Mannequin Comparability
The evaluation in contrast Common-2 and its predecessor Common-1 with OpenAI’s Whisper large-v3 and Whisper turbo fashions. Every mannequin was evaluated primarily based on parameters like Phrase Error Charge (WER), Correct Noun Error Charge (PNER), and different metrics vital for Speech-to-Textual content duties.
Efficiency Metrics
Common-2 achieved the bottom Phrase Error Charge (WER) at 6.68%, marking a 3% enchancment over Common-1. Whisper fashions, whereas aggressive, had barely greater error charges, with large-v3 recording a WER of seven.88% and turbo at 7.75%.
In correct noun recognition, Common-2 demonstrated superior accuracy with a 13.87% PNER, outperforming each Whisper large-v3 and turbo. This mannequin additionally excelled in textual content formatting, reaching a U-WER of 10.04%, which signifies higher dealing with of punctuation and capitalization.
Alphanumeric and Hallucination Charges
Whisper large-v3 confirmed power in alphanumeric transcription with the bottom error charge of three.84%, barely forward of Common-2’s 4.00%. Nonetheless, Common-2’s lowered hallucination charges had been a major benefit, with a 30% discount in comparison with Whisper fashions, making it extra dependable for real-world purposes.
Conclusion
Common-2’s developments over Common-1 are evident, with enhancements in accuracy, correct noun dealing with, and formatting. Regardless of Whisper’s strengths in sure areas, its susceptibility to hallucinations poses challenges for constant efficiency.
For additional insights and detailed metrics, the total analysis is accessible by AssemblyAI’s official report.
Picture supply: Shutterstock
