Whisper

Whisper is OpenAI’s open, general purpose speech recognition model family. It is a Transformer sequence-to-sequence model trained to predict text directly from audio: the input is a log-mel spectrogram, the output a sequence of tokens. Because the tasks are expressed with special tokens, one model covers multilingual recognition, translation into English, language identification and voice activity detection.

The family was trained on 680,000 hours of audio and matching transcripts from the internet. About 17 percent of that is non-English audio with transcripts in the same language, covering 98 languages in total. Larger models are more accurate and slower, so measure on your own audio.

Models and hardware needs

Six sizes exist, and the smaller ones also come in an English-only .en variant:

• tiny (39 M parameters), base (74 M), small (244 M) and medium (769 M): the official table lists roughly 1 GB of VRAM for the first two, 2 GB for small and 5 GB for medium.
• large (1550 M): about 10 GB of VRAM, multilingual variant only.
• turbo (809 M): a version of large-v3 optimized for inference speed, at roughly 6 GB. turbo was not trained for translation, so use tiny, base, small, medium or large for translation into English.

Those figures describe English speech on an A100; the real requirement depends on the language and hardware.

Language detection and output

• The model detects the spoken language on its own; detect_language returns probabilities, and setting the language explicitly gives a shorter, more predictable run.
• The transcript is a list of segments with start and end times. transcribe reads the whole file and predicts autoregressively over sliding 30 second windows.
• Word level timestamps can be requested (word_timestamps=True in faster-whisper) and are approximate.
• Speaker diarisation is not part of the model; the model card treats it as a separate capability.

Running on CPU and GPU

The reference implementation runs on PyTorch and starts on CPU, only slowly. faster-whisper runs the same weights on the CTranslate2 inference engine, which per its own benchmark is up to four times faster while using less memory, and adds 8-bit quantization on CPU and GPU. Code and weights are MIT licensed.

• Transcribing 13 minutes of audio with large-v2 on GPU: the reference takes 2m23s and 4708 MB, faster-whisper 1m03s and 4525 MB, or 16s and 4500 MB in int8 with a batch of 8.
• On CPU with the small model: the reference takes 6m58s and 2335 MB, faster-whisper in int8 takes 1m42s and 1477 MB.
• faster-whisper decodes with PyAV, so no system ffmpeg is needed; GPU execution requires cuBLAS and cuDNN for CUDA 12.
• Silero VAD is built in for silence filtering, dropping gaps longer than two seconds.

On our GPU machines LM Studio and gpt-oss compete for the same VRAM, so transcription does not run alongside language model inference, or the service gets its own Docker container.

Limitations

• The models were trained with weak supervision on noisy data, so they can produce text that was never spoken (hallucination). This is more common on silence, noise and low resource languages.
• The sequence-to-sequence layout is prone to repetitive output, which beam search and temperature scheduling reduce but do not remove.
• Accuracy is uneven across languages: the model card reports strong results in about ten languages and weaker results where training data is scarce.
• The models were not built for real time transcription, and the model card places classification outside the intended use.

Further reading

• Whisper repository
• Model card
• faster-whisper

At CyberElectro transcription runs on our own hosts: a Whisper based transcription app processes the audio, so the recording never leaves the machine.