Forced alignment and ASR timestamp accuracy

FA-Bench Leaderboard

Name: FA-Bench leaderboard
Creator: Olewave

How closely word and phone timestamps land on the boundaries human linguists placed. FA-Bench ranks 28 ASR pipelines that hear only the audio and 15 forced aligners handed the reference transcript.

Records 202610 · Updated October 13, 2026 · github.com/olewave/fa-bench · Paper on arXiv

ASR with timestamps, word boundary F1 at 20 ms

#	System	Access	Mean F1	TIMIT clean	TIMIT noisy	Buckeye clean	Buckeye noisy	WER TIMIT	WER Buckeye
1	Olewave Olign 1.0 olign-1.0, Olign API v1	Commercial	0.636	0.756	0.650	0.605	0.534	2.2	10.2
2	Qwen3 → MFA 3.4	Open	0.585	0.628	0.608	0.599	0.504	2.0	10.4
3	Qwen3 → PocketSphinx	Open	0.573	0.565	0.533	0.640	0.552	2.0	10.2
4	Qwen3 → MAPS ⚠	Open	0.506	0.698	0.394	0.551	0.383	2.0	10.2
5	Qwen3 → Charsiu	Open	0.493	0.524	0.472	0.543	0.433	2.4	11.4
6	Google Chirp 2 chirp_2, Speech-to-Text API v2	Commercial	0.436	0.443	0.446	0.442	0.414	4.2	13.1
7	Qwen3 → CrisperWhisper ⚠	Open	0.414	0.427	0.417	0.434	0.377	2.7	10.6
8	CrisperWhisper ⚠	Open	0.410	0.425	0.413	0.430	0.371	3.6	11.5
9	Qwen3 → Meta UnitY2	Open	0.375	0.336	0.335	0.422	0.407	2.0	10.2
10	Qwen3 → Qwen3-FA	Open	0.370	0.385	0.367	0.386	0.341	2.0	10.2
11	Amazon Transcribe en-US, batch API 2017-10-26	Commercial	0.269	0.253	0.262	0.285	0.275	2.7	10.3
12	Qwen3 → BFA	Open	0.255	0.265	0.237	0.275	0.242	2.0	10.4
13	Qwen3 → Meta MMS-FA	Open	0.227	0.271	0.258	0.196	0.184	2.0	10.2
14	Qwen3 → stable-ts	Open	0.217	0.172	0.189	0.257	0.251	2.0	10.2
15	AssemblyAI Universal-3.5 Pro universal-3-5-pro, API v2	Commercial	0.210	0.236	0.217	0.204	0.184	3.7	12.2
16	Deepgram Nova-3 ‡ nova-3, build 2025-07-31.0, API v1	Commercial	0.197	0.208	0.198	0.197	0.184	2.4	10.8
17	Parakeet-TDT	Open	0.184	0.180	0.180	0.192	0.185	2.4	11.0
18	Qwen3 → NVIDIA NeMo-FA †	Open	0.178	0.179	0.175	0.181	0.176	2.0	10.3
19	Qwen3 → TorchAudio	Open	0.172	0.205	0.187	0.154	0.141	2.0	10.2
20	IBM Watson STT en-US en-US large speech model, API v1	Commercial	0.157	0.147	0.141	0.181	0.159	5.1	12.4
21	Qwen3 → WhisperX	Open	0.152	0.175	0.164	0.138	0.130	2.0	10.2
22	Whisper → WhisperX	Open	0.145	0.178	0.163	0.125	0.116	2.9	14.2
23	Speechmatics Enhanced enhanced, builds 16.0.0 and 16.1.0, API v2	Commercial	0.143	0.155	0.143	0.144	0.130	2.5	11.4
24	Whisper-timestamped	Open	0.140	0.113	0.108	0.175	0.164	2.5	13.4
25	Azure AI Speech en-US, short audio REST API v1	Commercial	0.138	0.139	0.138	0.142	0.133	3.4	11.5
26	ElevenLabs Scribe v2 scribe_v2, API v1	Commercial	0.137	0.166	0.163	0.109	0.108	2.0	10.6
27	Whisper large-v3	Open	0.118	0.104	0.098	0.138	0.132	2.9	13.7
28	TorchAudio (ASR)	Open	0.106	0.166	0.124	0.083	0.053	10.8	27.8

⚠ MAPS was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.

⚠ CrisperWhisper's authors used TIMIT to build its word timing and don't say which TIMIT files. FA-Bench tests on TIMIT's test files, so if the authors used those too, its TIMIT scores come from audio it has already seen.

‡ Deepgram ends the last word of 24% of clean Buckeye test utterances more than 0.3 s after the audio does.

† NeMo-FA here is its 80 ms model. The 40 ms model's mean F1 is 0.175.

Download the CSV.

Forced alignment, word boundary F1 at 20 ms

#	System	Access	Mean F1	TIMIT clean	TIMIT noisy	Buckeye clean	Buckeye noisy	MAE TIMIT	MAE Buckeye
1	Olewave Olign 1.0 olign-1.0, Olign API v1	Commercial	0.714	0.782	0.669	0.745	0.659	18.2	22.4
2	MFA 3.4	Open	0.633	0.644	0.624	0.682	0.583	21.8	21.3
3	PocketSphinx	Open	0.626	0.584	0.550	0.727	0.642	30.5	22.1
4	NeuFA ⚠ trained with its open-source code on GitHub, not released	Open	0.568	0.523	0.446	0.700	0.604	56.9	33.9
5	MAPS ⚠	Open	0.545	0.715	0.403	0.620	0.444	25.5	37.9
6	Charsiu	Open	0.532	0.540	0.487	0.607	0.493	28.1	28.1
7	CrisperWhisper ⚠	Open	0.449	0.437	0.430	0.487	0.442	34.1	43.1
8	Meta UnitY2	Open	0.409	0.343	0.346	0.475	0.471	52.2	40.6
9	Qwen3-FA	Open	0.400	0.399	0.377	0.432	0.391	32.2	33.8
10	BFA	Open	0.276	0.272	0.244	0.301	0.289	51.0	57.2
11	Meta MMS-FA	Open	0.246	0.275	0.264	0.225	0.221	31.1	37.1
12	stable-ts	Open	0.240	0.177	0.195	0.294	0.295	96.8	71.2
13	NVIDIA NeMo-FA †	Open	0.199	0.186	0.183	0.212	0.216	79.2	91.4
14	TorchAudio	Open	0.189	0.210	0.192	0.180	0.175	39.8	42.4
15	WhisperX	Open	0.170	0.180	0.170	0.165	0.165	37.2	41.7

⚠ NeuFA was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.

⚠ MAPS was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.

† NeMo-FA here is its 80 ms model. The 40 ms model's mean F1 is 0.196.

ElevenLabs Forced Alignment joins this board once our free ElevenLabs credits replenish.

Download the CSV.

ASR with timestamps, phone boundary F1 at 20 ms

#	System	Access	Mean F1	TIMIT clean	TIMIT noisy	Buckeye clean	Buckeye noisy	PER TIMIT	PER Buckeye
1	Olewave Olign 1.0 olign-1.0, Olign API v1	Commercial	0.438	0.410	0.392	0.495	0.454	35.4	28.5
2	Qwen3 → MFA 3.4	Open	0.428	0.394	0.372	0.502	0.446	34.8	28.2
3	Qwen3 → PocketSphinx	Open	0.395	0.337	0.325	0.482	0.436	34.0	27.2
4	Qwen3 → MAPS ⚠	Open	0.378	0.455	0.319	0.435	0.303	30.7	35.8
5	Qwen3 → Charsiu	Open	0.369	0.361	0.347	0.417	0.352	30.3	32.7
6	Qwen3 → TorchAudio	Open	0.230	0.207	0.205	0.260	0.250	35.9	30.0
7	Qwen3 → BFA	Open	0.213	0.183	0.177	0.253	0.238	36.2	33.6

⚠ MAPS was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.

Download the CSV.

Forced alignment, phone boundary F1 at 20 ms

#	System	Access	Mean F1	TIMIT clean	TIMIT noisy	Buckeye clean	Buckeye noisy	PER TIMIT	PER Buckeye
1	FALCON ⚠	Open	0.560	0.673	0.519	0.585	0.464	0.0	0.7
2	Olewave Olign 1.0 olign-1.0, Olign API v1	Commercial	0.452	0.410	0.393	0.523	0.483	35.4	28.2
3	MFA 3.4	Open	0.436	0.393	0.372	0.518	0.463	34.8	26.1
4	PocketSphinx	Open	0.402	0.338	0.326	0.494	0.450	33.9	26.0
5	MAPS ⚠	Open	0.383	0.454	0.319	0.445	0.313	30.7	34.9
6	Charsiu	Open	0.372	0.360	0.347	0.424	0.359	30.3	31.7
7	NeuFA ⚠ trained with its open-source code on GitHub, not released	Open	0.366	0.326	0.277	0.460	0.399	36.0	28.0
8	TorchAudio	Open	0.235	0.208	0.206	0.266	0.261	35.9	28.5
9	BFA	Open	0.217	0.183	0.178	0.255	0.251	36.2	32.1

⚠ FALCON is handed the reference phones in place of the words. It was also trained on its own speaker split of Buckeye, which puts at least 8 of the 16 speakers in FA-Bench's Buckeye dev and test sets in its training data.

⚠ MAPS was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.

⚠ NeuFA was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.

Download the CSV.

ASR with timestamps and forced alignment

The first switch picks the task. ASR with timestamps hears only the audio and returns its own words, each with a start and an end time. A subtitle has to know when each word starts and ends, and most speech recognition APIs return word times without saying how accurate those times are. Forced alignment is handed the reference transcript, so it can make no recognition error and its boundary error is timing error alone.

The FA-Bench paper names two ways to get word times from audio alone. A one-step system is an ASR with timestamps. The ASR times its own words. A two-step system aligns after ASR. An ASR decodes the utterance, and an aligner then receives those decoded words in place of the reference transcript and times them.

On the ASR boards, → denotes ASR then aligner. Qwen3 → MFA 3.4 means Qwen3-ASR decodes the words and MFA 3.4 aligns them.

Words and phones

The second switch picks the boundaries. The word boards score the boundaries between words, and the phone boards score the boundaries between phones, on the same four test cells. Each system aligns in its own phone set, so FA-Bench maps its phones and the reference phones into TIMIT-39, the conventional reduction for these corpora.

The ASR phone board holds only the systems that return phone timestamps from audio alone. Their PER, the phone error rate, carries the recognizer's word errors too. On the forced alignment phone board, each aligner is handed the reference transcript, and PER is the phone error rate of the phones it returns against the annotators' phones.

How the score works

Each bar is the boundary F1 at 20 ms. A boundary is a hit when the words or phones on both sides of it match the reference and its time falls within 20 ms of where a human annotator placed it. A misheard word costs a system the boundaries beside it, so on the ASR boards the score charges recognition errors and timing errors together.

The default view averages four test cells. They are TIMIT core test and Buckeye test, each on clean audio and on the mean of four degraded versions with reverb, noise, music or babble added. Choose a single cell from the menu to rank the systems by one corpus or one condition.

Choose Open models to compare only the pipelines you can run on your own hardware. A two-step pipeline counts as commercial when either step calls a paid API.

Each bar carries the colour and logo of the organization behind it. For a two-step pipeline that is the maker of the aligner, the step that sets the times.

API versions

Each commercial API was called with the model below. Deepgram and Speechmatics name a build in every response, and the other APIs name none. FA-Bench called the eight other APIs between September 17 and September 26, 2026, and Olign 1.0 in October 2026.

API	Model called	API version
Olewave Olign 1.0	olign-1.0	v1
Google Chirp 2	chirp_2	Speech-to-Text v2
Amazon Transcribe	en-US	Batch, 2017-10-26
AssemblyAI Universal-3.5 Pro	universal-3-5-pro	v2
Deepgram Nova-3	nova-3, build 2025-07-31.0	v1
IBM Watson STT en-US	en-US large speech model	v1
Speechmatics Enhanced	enhanced, builds 16.0.0 and 16.1.0	Batch v2
Azure AI Speech	en-US	Short audio REST v1
ElevenLabs Scribe v2	scribe_v2	v1

FA-Bench and Artificial Analysis

Artificial Analysis also ranks speech to text APIs, by its AA-WER v2 score. The table sets the two benchmarks side by side, using what each one publishes about itself.

	FA-Bench	Artificial Analysis
What it scores	Word timing and recognition, as F1 at 20 ms	Word error rate
Test audio	TIMIT and Buckeye, research corpora anyone can license	50% private AA-AgentTalk, 50% public VoxPopuli and Earnings22
Word timestamps	Scored against human annotators	Not scored
Scoring code	Public on GitHub	Not published
Text normalizer	Public on GitHub	Not published
Rerun by others	Yes, with TIMIT and Buckeye	No

Artificial Analysis details come from its methodology page and its AA-WER v2 announcement, read on October 11, 2026.

Reproduce every number

The numbers come from FA-Bench's records, snapshot 202610. The boards keep one pipeline per aligner, and NeMo-FA appears with its 80 ms model only. The records hold all 34 ASR pipelines on words and all 12 on phones, and all 18 aligners on words and all 11 on phones. Download the CSV under each board, or clone the repository and rescore every system yourself.