whisper

Whisper is OpenAI's automatic speech recognition (ASR) library that transcribes and translates audio into text using transformer-based neural networks. It processes audio files by converting them to mel spectrograms, encoding them through a neural network, and decoding the output into timestamped text with word-level alignment capabilities.

Key Features

Audio-to-Mel Spectrogram Processing
Transformer-based Speech Recognition
Word-Level Timing Alignment
Multilingual Tokenization
Text Normalization
Model Management

Entry Points

Start here when working with this codebase

Library API

whisper/__init__.py

Model discovery, downloading, validation, and initialization - exposes load_model(), available_models(), and core transcription functions

Start here when adding new model variants, changing model loading behavior, or modifying the public API surface

CLI Interface

whisper/__main__.py

Bootstrap entry point that delegates to CLI handler in transcribe.py

Start here when adding new CLI commands or modifying how the CLI is invoked

Transcription Pipeline

whisper/transcribe.py

Main transcription orchestration, CLI argument parsing, and result formatting

Start here when modifying transcription behavior, adding CLI arguments, or changing output formats

Test Suite

tests/conftest.py

Pytest fixtures and test configuration shared across all test modules

Start here when adding new test fixtures or modifying test setup

How to Make Changes

Common modification patterns for this codebase

Add a new audio preprocessing feature

whisper/audio.py

Add new preprocessing function following the existing pattern of load_audio() and log_mel_spectrogram()

Pattern: See how pad_or_trim() handles tensor operations with configurable parameters

whisper/transcribe.py

Integrate the new preprocessing step into the transcription pipeline

Pattern: Follow how log_mel_spectrogram is called and its output passed to the model

whisper/__init__.py

Export the new function if it should be part of the public API

Pattern: See how load_model and transcribe are exposed in __all__

Conventions to follow:

Use torch tensors for audio data, numpy for intermediate processingSupport both file paths and numpy arrays as inputInclude SAMPLE_RATE constant (16000) for audio operations

Add a new decoding strategy or sampling option

whisper/decoding.py

Add new sampling/decoding logic to the DecodingOptions dataclass and implement in the decoding pipeline

Pattern: See how temperature, beam_size, and best_of parameters are defined and used in decode()

whisper/transcribe.py

Add CLI argument for the new option and pass it to DecodingOptions

Pattern: Follow how --temperature and --beam_size arguments are parsed and forwarded

whisper/model.py

Modify model inference if the strategy requires changes to forward pass

Pattern: See how Whisper.decode() interfaces with the decoding module

Conventions to follow:

Use dataclasses for configuration options (see DecodingOptions)Support both greedy and beam search decoding pathsReturn DecodingResult with all metadata (tokens, text, avg_logprob, etc.)

Add a new text normalizer

whisper/normalizers/basic.py

Create normalizer class or add methods to BasicTextNormalizer

Pattern: Follow BasicTextNormalizer's __call__ pattern that takes text and returns normalized text

whisper/normalizers/english.py

If language-specific, add to EnglishTextNormalizer or create new language file

Pattern: See how EnglishTextNormalizer extends BasicTextNormalizer and adds number/contraction handling

whisper/normalizers/__init__.py

Export the new normalizer class from the package

Pattern: Follow existing imports: from .basic import BasicTextNormalizer

Conventions to follow:

Normalizers are callable classes with __call__(self, text: str) -> strUse Unicode normalization (NFKC) as first stepLanguage-specific normalizers inherit from BasicTextNormalizer

Add a new output format for transcription results

whisper/utils.py

Create new Writer subclass for the output format

Pattern: Follow WriteTXT, WriteVTT, WriteSRT patterns - each has write_result() method

whisper/transcribe.py

Pattern: See how --output_format argument maps to writer classes via get_writer()

Conventions to follow:

Writers inherit from ResultWriter base classImplement write_result(result, file) methodHandle word-level timestamps if available in result

Add a new model architecture variant

whisper/model.py

Define new ModelDimensions configuration and modify Whisper class if architecture differs

Pattern: See how ModelDimensions dataclass defines n_mels, n_audio_ctx, n_vocab, etc.

whisper/__init__.py

Add model to _MODELS dict with download URL and expected hash

Pattern: Follow existing entries like 'tiny', 'base', 'small' with their URLs and SHA256 hashes

whisper/tokenizer.py

Update tokenizer if model uses different vocabulary or special tokens

Pattern: See how get_tokenizer() handles multilingual vs English-only models

Conventions to follow:

Model names follow pattern: size or size.en (e.g., 'small', 'small.en')Use ModelDimensions dataclass for all architecture parametersCheckpoint files are .pt format with state_dict and dims keys

Add GPU-optimized operation with Triton

whisper/triton_ops.py

Add new Triton kernel with @triton.jit decorator

Pattern: Follow dtw_kernel and median_filter_kernel patterns for block-based GPU computation

whisper/timing.py

Integrate the optimized operation into timing alignment pipeline

Pattern: See how dtw_cuda() is called with fallback to CPU implementation

Conventions to follow:

Provide CPU fallback for systems without TritonUse triton.jit decorator with autotune configsHandle tensor device placement (cuda vs cpu) explicitly

Add word-level timing feature

whisper/timing.py

Modify alignment logic using cross-attention weights and DTW

Pattern: See how find_alignment() extracts word boundaries from token-level alignments

whisper/tokenizer.py

Add token manipulation helpers if needed for word boundary detection

Pattern: See how Tokenizer.decode_with_timestamps() handles timestamp tokens

whisper/transcribe.py

Enable feature via CLI flag and include in result dict

Pattern: Follow --word_timestamps argument and how it triggers add_word_timestamps()

Conventions to follow:

Word timestamps stored as list of dicts with 'word', 'start', 'end' keysUse cross-attention weights from decoder for alignmentDTW aligns mel frames to token positions

Coding Conventions

Standards and patterns used in this codebase

Type Hints

Use comprehensive type annotations with Optional, Union, List, Dict, and Tuple from typing module for all function signatures and class attributes

def detect_language(
    model: "Whisper", mel: Tensor, tokenizer: Tokenizer = None
) -> Tuple[Tensor, List[dict]]:

whisper/decoding.pywhisper/transcribe.pywhisper/tokenizer.py

Dataclasses

Use @dataclass decorator for configuration and result containers, with frozen=True for immutable data structures

@dataclass(frozen=True)
class DecodingOptions:
    task: str = "transcribe"
    language: Optional[str] = None
    temperature: float = 0.0

whisper/decoding.py

Module Organization

Place imports at top organized by: standard library, third-party packages, then local imports, with lazy imports inside functions for optional heavy dependencies

import torch
import numpy as np

from .audio import SAMPLE_RATE
from .utils import exact_div

whisper/__init__.pywhisper/transcribe.py

Constants

Define module-level constants in UPPER_SNAKE_CASE at the top of files after imports

SAMPLE_RATE = 16000
N_FFT = 400
HOP_LENGTH = 160
CHUNK_LENGTH = 30

whisper/audio.pywhisper/model.py

Error Handling

Use assertions for internal invariants and raise specific exceptions (ValueError, RuntimeError) with descriptive messages for user-facing errors

if audio.shape[-1] > N_SAMPLES:
    raise ValueError("audio is longer than 30s")
assert x.shape[1:] == self.positional_embedding.shape, "incorrect audio shape"

whisper/audio.pywhisper/model.pywhisper/decoding.py

Docstrings

Use concise docstrings for public functions describing parameters and return values, with inline comments for complex logic

def load_audio(file: str, sr: int = SAMPLE_RATE):
    """
    Open an audio file and read as mono waveform, resampling as necessary
    """

whisper/audio.pywhisper/transcribe.py

Device Handling

Accept device parameter as string or torch.device, defaulting to CUDA if available, and use consistent device placement patterns

device: str = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

whisper/__init__.pywhisper/transcribe.py

Tensor Operations

Use torch.no_grad() context manager for inference, and prefer in-place operations with underscore suffix where appropriate

with torch.no_grad():
    mel = log_mel_spectrogram(audio, model.dims.n_mels)
    logits[logits == -np.inf] = -1e8

whisper/decoding.pywhisper/transcribe.py

CLI Design

Use argparse with add_argument for CLI, organizing arguments into logical groups with clear help text and sensible defaults

parser.add_argument(
    "--model",
    default="turbo",
    choices=available_models(),
    help="name of the Whisper model to use",
)

whisper/transcribe.py

Testing

Use pytest fixtures in conftest.py for shared test resources, with descriptive test function names prefixed with test_

@pytest.fixture
def model():
    return whisper.load_model("tiny")

def test_transcribe(model):

tests/conftest.py