FA-Bench Leaderboard · github.com/olewave/fa-bench · Paper on arXiv

Updated on October 28, 2026 with the October results.

We released FA-Bench on August 17, 2026. It measures how closely word and phone timestamps land on the boundaries human linguists placed.

FA-Bench scores speech recognition systems that return their own words with times. It also scores forced aligners handed a transcript. Every system runs through the same code on the same test sets, so you can compare any two numbers directly. The FA-Bench Leaderboard ranks 17 ASR systems and 15 forced aligners.

Why a benchmark

Forced aligner results are hard to reproduce. Each paper picks its own phoneme set and test split, and it scores with its own metric. So a number in one paper rarely measures the same thing as a number in the next. A speech researcher can't run a clean ablation study, and a voice AI engineer can't tell which system to deploy.

FA-Bench publishes the whole evaluation as code, from how labels are folded to how each boundary is scored. You run the same pipeline we do.

What FA-Bench tests

FA-Bench scores two corpora whose boundaries human linguists corrected by hand. TIMIT core test holds 192 read sentences from 24 speakers. Buckeye test holds 4,513 segments of conversation from 8 speakers. FA-Bench adds no annotation of its own. It keeps the boundary times exactly as the corpora give them and folds the phone labels into the shared TIMIT-39 set.

Each test set runs clean and in four degraded copies, built with the Kaldi augmentation recipes from fixed seeds. The noise copy, for example, plays MUSAN noise clips at 15, 10, 5 and 0 dB SNR. A system's headline score is the mean of four test cells. They are TIMIT core test and Buckeye test, each clean and noisy.

Two tasks

ASR with timestamps hears only the audio and returns its own words, each with a start and an end time. A misheard word costs a system both of its times. So the score charges recognition errors and timing errors together.

Forced alignment is handed the reference transcript, so its timing is all that is scored. FA-Bench also hands each aligner the words Qwen3-ASR recognized. F1 Drop is how far its score falls when it gets those words. MFA 3.4, for example, scores 0.658 on the reference transcript and 0.628 on recognized words, an F1 Drop of 0.030.

How FA-Bench scores

Word timestamp F1 at 20 ms ranks the word boards. Every reference word owns a start time and an end time. A time is a hit when the system produced the same word and placed that time within 20 ms of the annotator's.

Phone boundary F1 at 20 ms ranks the phone boards. A phone boundary is a hit when the phones on both sides of it match the reference and its time falls within 20 ms of the annotator's.

Every score comes with a 95% interval from 2,000 bootstrap draws of the test utterances. Each system also gets the range of ranks it takes across those draws. Two systems whose ranges overlap are tied on these test sets. Google Chirp 3 and CrisperWhisper, for example, both have a range of 2 to 3.

The FA-Bench overview explains every number on the leaderboard, with a worked figure.

The October 2026 results

These are word timestamp F1 at 20 ms, the mean of the four test cells, from the leaderboard's October 2026 results.

On ASR with timestamps, Google Chirp 2 ranks first at 0.486, with a 95% interval of 0.475 to 0.498. CrisperWhisper, at 0.455, and Google Chirp 3, at 0.452, share second and third.

On forced alignment with the reference transcript, MFA 3.4 ranks first at 0.658 and PocketSphinx second at 0.648. Handed the words Qwen3-ASR recognized, MFA 3.4 still ranks first, at 0.628. On phones, scored by phone boundary F1 at 20 ms, MFA 3.4 ranks first at 0.436.

Olewave makes both FA-Bench and Olign. Olign no longer takes part in FA-Bench, so no board ranks it, and the leaderboard lists its last results unranked. Every system on the boards is scored by the same public code against the same human annotations.

Run it yourself

You can check the whole measurement chain before you touch any licensed audio. The self-test builds a synthetic corpus and runs every step on it. Each gate compares a result against an answer fixed in advance, so a broken chain fails its gate.

git clone https://github.com/olewave/fa-bench && cd fa-bench
uv venv --python 3.12 .venv && . .venv/bin/activate
uv pip install -e ".[test]"
fabench selftest
fabench gates

To score real systems, get TIMIT from the LDC and Buckeye through OSU registration, under their own licences. Then run fabench init, and it asks where each corpus is. Every board on the leaderboard offers its numbers as a CSV, so you can check yours against ours.

FA-Bench is released under the Apache License 2.0 from version 2.0.0. Anyone can use, change and share the code, commercial use included. The name FA-Bench stays with Olewave, and results you produce with the code can be described as scored with the FA-Bench scorer, version 2.0.