Deep-unlearning's smol-audio: Colab notebooks for fine-tuning audio models

Deep-unlearning has released smol-audio, an open collection of Jupyter notebooks for hands-on work with modern audio models directly in Google Colab. The project is built as a set of reproducible recipes for people who don't want to read abstract surveys, but to quickly fine-tune ASR, run audio captioning and pick apart multimodal pipelines.

How smol-audio is built

The core idea is simple: instead of yet another all-purpose framework, the team built a flat repository of independent notebooks, each solving one specific task. Every scenario is built on the Hugging Face stack — transformers, datasets, peft and accelerate — and is designed to run without any local GPU setup. Open Colab, connect a runtime and you get a working starting point, not a pile of scattered snippets from issue trackers. That makes the project look more like an engineering cookbook than a showcase demo.

One important detail is transparency. smol-audio doesn't hide the training loop and data preparation behind convenient wrappers, so engineers can see how batches are built, where the loss is computed and what exactly changes during fine-tuning. For beginners it is learning material; for experienced teams it is a convenient base to adapt to their own dataset. According to the authors, most recipes fit within the 16 GB of Colab memory, meaning they don't require expensive infrastructure even at the first step.

Which models are included

For now the collection mainly covers fine-tuning of ASR models, but it is not limited to speech recognition. The repository and the accompanying overview include scenarios for several architectures that differ noticeably in design and training requirements. That is useful: instead of a generic "just make it work", the user gets working templates for a specific class of model. This format lowers the barrier to entry when you need to quickly test a hypothesis on your own audio set.

• Whisper — adaptation to a new language or a narrow domain.
• Parakeet from NVIDIA — fine-tuning of a CTC model, including a LoRA variant.
• Voxtral from Mistral — ASR tuning with prompt masking for an LLM architecture.
• Granite Speech from IBM — an example of language fine-tuning on the Italian YODAS-Granary corpus.
• Audio Flamingo 3 and PE-AV — audio captioning, zero-shot video classification and audio-to-text retrieval.

The differences between these models are not cosmetic. Whisper is a sequence-to-sequence system that generates a transcript token by token. Parakeet relies on a CTC approach, which is usually lighter and faster at inference but requires different logic for aligning audio frames with text. Voxtral is closer to LLM-based speech understanding, so prompt masking is critical there: the loss must be computed on the transcript, not on the text prompts themselves. The overview also separately notes a scenario for Dia-1.6B, aimed at dialogue TTS.

Why engineers should care

smol-audio's strongest point is not a list of big names but the time it saves on routine engineering. When a team picks up a new audio model, weeks often go not to research but to basics: assembling the dataset correctly, not mixing up preprocessing, choosing a fine-tuning mode and not running out of GPU memory. Here the authors show both full fine-tuning and a lighter LoRA option, which matters especially for large audio and multimodal models.

This shows in the more complex scenarios too. For Audio Flamingo 3, the project demonstrates how to fine-tune the model for audio captioning, which is useful for accessibility, content indexing and media library search. For Meta PE-AV* it demonstrates multimodal inference with a shared embedding space for audio, video and text: this approach enables zero-shot video classification and cross-modal search without separate task-specific fine-tuning. In other words, smol-audio is useful not only for ASR but for the broader voice AI and multimodal ecosystem.

What this means

smol-audio turns work with audio AI from a set of scattered experiments into a clear practical cookbook. If the trend toward voice assistants, multimodal models and local adaptation to languages continues, such repositories may become basic infrastructure for ML teams: not a replacement for research, but a short path from idea to a first working prototype.

*Meta is designated an extremist organization and banned in Russia.