Name: FA-Bench leaderboard
Creator: Olewave

ASR with timestamps, word timestamp F1 at 20 ms

Rank	Range	System	Access	Mean F1	95% interval	TIMIT clean	TIMIT noisy	Buckeye clean	Buckeye noisy	WER TIMIT	WER Buckeye	Price, $ per audio hour
1	1	Google Chirp 2	Commercial	0.486	0.475–0.498	0.488	0.497	0.492	0.469	4.2	13.1	$0.96
2	2–3	CrisperWhisper ⚠	Open	0.455	0.442–0.466	0.484	0.472	0.463	0.399	3.6	11.5	–
3	2–3	Google Chirp 3 ‡	Commercial	0.452	0.442–0.462	0.460	0.438	0.479	0.431	5.2	13.5	$0.96
4	4	Qwen3-ASR-Flash	Commercial	0.363	0.353–0.374	0.398	0.355	0.376	0.322	1.8	10.1	$0.13
5	5	Mistral Voxtral Mini	Commercial	0.308	0.298–0.319	0.315	0.294	0.325	0.299	2.4	13.4	$0.18
6	6–7	ElevenLabs Scribe v2	Commercial	0.281	0.275–0.288	0.304	0.297	0.267	0.256	2.1	10.7	$0.22
7	6–7	Amazon Transcribe	Commercial	0.273	0.263–0.284	0.259	0.264	0.289	0.281	2.7	10.3	$0.36
8	8–9	AssemblyAI Universal-3.5 Pro	Commercial	0.213	0.204–0.222	0.237	0.218	0.211	0.185	2.0	9.0	$0.21
9	8–9	TorchAudio (ASR)	Open	0.209	0.203–0.216	0.254	0.213	0.211	0.159	10.8	27.8	–
10	10–11	Deepgram Nova-3 ‡	Commercial	0.170	0.161–0.178	0.191	0.185	0.155	0.148	2.4	10.8	$0.26
11	10–11	Parakeet-TDT	Open	0.164	0.155–0.172	0.172	0.174	0.156	0.153	2.4	11.0	–
12	12	IBM Watson STT en-US	Commercial	0.149	0.142–0.157	0.135	0.132	0.176	0.155	5.1	12.4	$1.27
13	13–14	Azure AI Speech	Commercial	0.100	0.094–0.107	0.117	0.112	0.089	0.083	3.4	11.5	$1.00
14	13–14	Speechmatics Enhanced	Commercial	0.097	0.091–0.103	0.110	0.088	0.107	0.083	2.5	11.4	$0.75
15	15–16	Whisper large-v3	Open	0.049	0.046–0.052	0.029	0.029	0.067	0.069	2.9	13.7	–
16	15–16	Whisper-timestamped	Open	0.049	0.046–0.052	0.021	0.022	0.075	0.076	2.5	13.4	–
–	–	Olewave Olign v1	Commercial	0.681	0.673–0.690	0.786	0.695	0.660	0.585	2.2	10.2	$0.18

⚠ CrisperWhisper's authors used TIMIT to build its word timing and don't say which TIMIT files. FA-Bench tests on TIMIT's test files, so if the authors used those too, its TIMIT scores come from audio it has already seen.

‡ Chirp 3 ends the last word of 15% of clean Buckeye test utterances more than 0.3 s after the audio does.

‡ Deepgram ends the last word of 24% of clean Buckeye test utterances more than 0.3 s after the audio does.

Every row is one system as shipped. A cascade of a recognizer and a forced aligner is not ranked here. It shows on the forced alignment boards under Recognized words, and in the F1 Drop of its aligner's row. Olign's price is $0.18 an hour, which is its launch promotion price. Its list price is $0.21.

Google Chirp 2 is chirp_2, Speech-to-Text API v2. Google Chirp 3 is chirp_3, Speech-to-Text API v2. Qwen3-ASR-Flash is qwen3-asr-flash-filetrans, DashScope API v1. Mistral Voxtral Mini is voxtral-mini-2602, API v1. ElevenLabs Scribe v2 is scribe_v2, API v1. Amazon Transcribe is en-US, batch API 2017-10-26. AssemblyAI Universal-3.5 Pro is universal-3-5-pro, API v2. Deepgram Nova-3 is nova-3, build 2025-07-31.0, API v1. IBM Watson STT en-US is en-US large speech model, API v1. Azure AI Speech is en-US, short audio REST API v1. Speechmatics Enhanced is enhanced, builds 16.0.0 and 16.1.0, API v2. Olewave Olign v1 is olign-1.0, Olign API v1.

Olign no longer takes part in FA-Bench. Its last results, from October 2026, are listed unranked at the bottom of the table.

Download the CSV.

Forced alignment, word timestamp F1 at 20 ms

Rank	Range	System	Access	Mean F1	95% interval	F1 Drop	TIMIT clean	TIMIT noisy	Buckeye clean	Buckeye noisy	MAE TIMIT	MAE Buckeye	Price, $ per audio hour
1	1	MFA 3.4	Open	0.658	0.650–0.667	0.030	0.685	0.649	0.706	0.594	21.8	21.3	–
2	2	PocketSphinx	Open	0.648	0.639–0.658	0.033	0.645	0.594	0.728	0.627	30.5	22.1	–
3	3	NeuFA ⚠	Open	0.599	0.589–0.607	–	0.600	0.493	0.712	0.589	56.9	33.9	–
4	4	Charsiu	Open	0.563	0.553–0.573	0.024	0.597	0.532	0.627	0.497	28.1	28.1	–
5	5	MAPS ⚠	Open	0.547	0.537–0.556	0.024	0.712	0.412	0.642	0.421	25.5	37.9	–
6	6	CrisperWhisper ⚠	Open	0.473	0.461–0.485	0.017	0.489	0.480	0.491	0.434	34.1	43.1	–
7	7	Meta UnitY2	Open	0.439	0.428–0.449	0.022	0.380	0.393	0.495	0.486	52.2	40.6	–
8	8	Qwen3-FA	Open	0.402	0.391–0.413	0.013	0.416	0.386	0.426	0.380	32.2	33.8	–
9	9	Meta MMS-FA	Open	0.345	0.338–0.352	0.014	0.353	0.350	0.344	0.332	31.1	37.1	–
10	10	ElevenLabs-FA	Commercial	0.295	0.289–0.302	0.015	0.299	0.300	0.293	0.288	31.9	35.5	$0.22
11	11–13	BFA	Open	0.272	0.264–0.280	0.009	0.276	0.255	0.285	0.272	51.0	57.2	–
12	11–13	WhisperX	Open	0.271	0.264–0.277	0.014	0.280	0.267	0.281	0.254	37.2	41.7	–
13	12–13	TorchAudio	Open	0.265	0.258–0.272	0.013	0.273	0.255	0.275	0.256	39.8	42.4	–
14	14	stable-ts	Open	0.251	0.243–0.260	0.015	0.197	0.210	0.301	0.297	96.8	71.2	–
15	15	NVIDIA NeMo-FA †	Open	0.155	0.149–0.163	0.008	0.157	0.158	0.151	0.156	79.2	91.4	–
–	–	Olewave Olign v1	Commercial	0.735	0.726–0.743	–	0.805	0.710	0.755	0.668	18.2	22.4	$0.18

F1 Drop is how far an aligner's mean F1 falls when it is handed the words Qwen3-ASR recognized in place of the reference transcript. MFA 3.4, for example, scores 0.658 on the reference transcript and 0.628 on recognized words, an F1 Drop of 0.030.

⚠ NeuFA was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.

⚠ MAPS was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.

⚠ CrisperWhisper's authors used TIMIT to build its word timing and don't say which TIMIT files. FA-Bench tests on TIMIT's test files, so if the authors used those too, its TIMIT scores come from audio it has already seen.

† NeMo-FA here is its 80 ms model. The 40 ms model's mean F1 is 0.199.

The Transcript switch shows each aligner handed the words Qwen3-ASR recognized in place of the reference transcript. Those words have a WER of 2.0% on TIMIT core test and 10.2% on Buckeye test on clean audio. NeuFA was not run on recognized words. MFA 3.4 on Parakeet-TDT's words scores 0.625, and WhisperX on Whisper large-v3's words scores 0.234. Olign's price is $0.18 an hour, which is its launch promotion price. Its list price is $0.21.

NeuFA is trained with its open-source code on GitHub, not released. ElevenLabs-FA is forced-alignment, API v1. Olewave Olign v1 is olign-1.0, Olign API v1.

Olign no longer takes part in FA-Bench. Its last results, from October 2026, are listed unranked at the bottom of the table.

Download the CSV.

Forced alignment, phone boundary F1 at 20 ms

Rank	Range	System	Access	Mean F1	95% interval	F1 Drop	TIMIT clean	TIMIT noisy	Buckeye clean	Buckeye noisy	PER TIMIT	PER Buckeye	Price, $ per audio hour
1	1	MFA 3.4	Open	0.436	0.431–0.443	0.008	0.393	0.372	0.518	0.463	34.8	26.1	–
2	2	PocketSphinx	Open	0.402	0.396–0.408	0.007	0.338	0.326	0.494	0.450	33.9	26.0	–
3	3	MAPS ⚠	Open	0.383	0.377–0.389	0.005	0.454	0.319	0.445	0.313	30.7	34.9	–
4	4	Charsiu	Open	0.372	0.367–0.378	0.003	0.360	0.347	0.424	0.359	30.3	31.7	–
5	5	NeuFA ⚠	Open	0.366	0.360–0.371	–	0.326	0.277	0.460	0.399	36.0	28.0	–
6	6	TorchAudio	Open	0.235	0.230–0.240	0.005	0.208	0.206	0.266	0.261	35.9	28.5	–
7	7	BFA	Open	0.217	0.212–0.221	0.004	0.183	0.178	0.255	0.251	36.2	32.1	–
–	–	Olewave Olign v1	Commercial	0.452	0.446–0.459	–	0.410	0.393	0.523	0.483	35.4	28.2	$0.18

F1 Drop is how far an aligner's mean F1 falls when it is handed the words Qwen3-ASR recognized in place of the reference transcript. MFA 3.4, for example, scores 0.436 on the reference transcript and 0.428 on recognized words, an F1 Drop of 0.008.

⚠ MAPS was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.

⚠ NeuFA was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.

FALCON is left off this board because it is handed the reference phones, which no other aligner here gets. Its mean F1 is 0.560. The Transcript switch shows each aligner handed the words Qwen3-ASR recognized in place of the reference transcript. Those words have a WER of 2.0% on TIMIT core test and 10.2% on Buckeye test on clean audio. NeuFA was not run on recognized words. MFA 3.4 on Parakeet-TDT's words scores 0.429.

NeuFA is trained with its open-source code on GitHub, not released. Olewave Olign v1 is olign-1.0, Olign API v1.

Olign no longer takes part in FA-Bench. Its last results, from October 2026, are listed unranked at the bottom of the table.

Download the CSV.

Forced alignment with recognized words, word timestamp F1 at 20 ms

Rank	Range	System	Access	Mean F1	95% interval	F1 Drop	TIMIT clean	TIMIT noisy	Buckeye clean	Buckeye noisy	MAE TIMIT	MAE Buckeye	Price, $ per audio hour
1	1	MFA 3.4	Open	0.628	0.619–0.637	0.030	0.677	0.642	0.649	0.545	21.8	22.7	–
2	2	PocketSphinx	Open	0.615	0.605–0.624	0.033	0.632	0.585	0.669	0.573	30.6	22.6	–
3	3	Charsiu	Open	0.539	0.529–0.550	0.024	0.587	0.522	0.586	0.463	28.2	26.1	–
4	4	MAPS ⚠	Open	0.523	0.514–0.532	0.024	0.702	0.407	0.595	0.387	25.5	39.8	–
5	5	CrisperWhisper ⚠	Open	0.456	0.443–0.468	0.017	0.485	0.472	0.464	0.403	33.8	39.2	–
6	6	Meta UnitY2	Open	0.417	0.406–0.428	0.022	0.375	0.385	0.461	0.447	52.4	39.8	–
7	7	Qwen3-FA	Open	0.389	0.377–0.399	0.013	0.408	0.383	0.405	0.358	32.6	30.6	–
8	8	Meta MMS-FA	Open	0.331	0.324–0.337	0.014	0.349	0.346	0.323	0.305	31.1	34.3	–
9	9	ElevenLabs-FA	Commercial	0.280	0.274–0.287	0.015	0.295	0.293	0.271	0.261	31.9	34.0	$0.22
10	10–11	BFA	Open	0.263	0.255–0.271	0.009	0.272	0.251	0.283	0.247	51.0	48.6	–
11	10–12	WhisperX	Open	0.257	0.250–0.263	0.014	0.276	0.263	0.259	0.230	37.3	39.7	–
12	11–12	TorchAudio	Open	0.252	0.245–0.260	0.013	0.269	0.252	0.254	0.234	39.9	39.8	–
13	13	stable-ts	Open	0.236	0.228–0.244	0.015	0.193	0.204	0.278	0.270	97.8	70.8	–
14	14	NVIDIA NeMo-FA †	Open	0.147	0.140–0.154	0.008	0.154	0.154	0.139	0.140	79.3	88.8	–
–	–	Olewave Olign 0.9	Commercial	0.704	0.694–0.712	0.031	0.791	0.697	0.703	0.623	18.3	20.8	–

F1 Drop is how far an aligner's mean F1 falls when it is handed the words Qwen3-ASR recognized in place of the reference transcript. MFA 3.4, for example, scores 0.658 on the reference transcript and 0.628 on recognized words, an F1 Drop of 0.030.

⚠ MAPS was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.

⚠ CrisperWhisper's authors used TIMIT to build its word timing and don't say which TIMIT files. FA-Bench tests on TIMIT's test files, so if the authors used those too, its TIMIT scores come from audio it has already seen.

† NeMo-FA here is its 80 ms model. The 40 ms model's mean F1 is 0.189.

Qwen3-ASR's words have a WER of 2.0% on TIMIT core test and 10.2% on Buckeye test on clean audio. NeuFA was not run on recognized words. Olign v1 was not run on recognized words, so the unranked Olign row here is Olign 0.9.

ElevenLabs-FA is forced-alignment, API v1. Olewave Olign 0.9 is olign-0.9.

Olign no longer takes part in FA-Bench. Its last results, from October 2026, are listed unranked at the bottom of the table.

Download the CSV.

Forced alignment with recognized words, phone boundary F1 at 20 ms

Rank	Range	System	Access	Mean F1	95% interval	F1 Drop	TIMIT clean	TIMIT noisy	Buckeye clean	Buckeye noisy	PER TIMIT	PER Buckeye	Price, $ per audio hour
1	1	MFA 3.4	Open	0.428	0.423–0.434	0.008	0.394	0.372	0.502	0.446	34.8	28.2	–
2	2	PocketSphinx	Open	0.395	0.389–0.401	0.007	0.337	0.325	0.482	0.436	34.0	27.2	–
3	3	MAPS ⚠	Open	0.378	0.372–0.384	0.005	0.455	0.319	0.435	0.303	30.7	35.8	–
4	4	Charsiu	Open	0.369	0.363–0.375	0.003	0.361	0.347	0.417	0.352	30.3	32.7	–
5	5	TorchAudio	Open	0.230	0.226–0.235	0.005	0.207	0.205	0.260	0.250	35.9	30.0	–
6	6	BFA	Open	0.213	0.208–0.217	0.004	0.183	0.177	0.253	0.238	36.2	33.6	–
–	–	Olewave Olign 0.9	Commercial	0.447	0.440–0.453	0.005	0.410	0.393	0.513	0.470	35.4	29.2	–

F1 Drop is how far an aligner's mean F1 falls when it is handed the words Qwen3-ASR recognized in place of the reference transcript. MFA 3.4, for example, scores 0.436 on the reference transcript and 0.428 on recognized words, an F1 Drop of 0.008.

⚠ MAPS was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.

Qwen3-ASR's words have a WER of 2.0% on TIMIT core test and 10.2% on Buckeye test on clean audio. NeuFA was not run on recognized words. Olign v1 was not run on recognized words, so the unranked Olign row here is Olign 0.9.

Olewave Olign 0.9 is olign-0.9.

Olign no longer takes part in FA-Bench. Its last results, from October 2026, are listed unranked at the bottom of the table.

Download the CSV.