speech AI / ASR and TTS
NVIDIA NeMo Speech is a scalable open-source framework for developing and deploying speech AI models, primarily automatic speech recognition (ASR) and text-to-speech (TTS), with pretrained checkpoints and training/inference tooling.
Same problem, same approach. Swapping one for another is a config change, not a rewrite.
choose when you need a deployment-focused ASR toolkit with built-in serving and MIT-licensed stack, less GPU-heavy than NeMo.
Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
choose for a modular PyTorch toolkit with strong recipes and no NVIDIA vendor lock-in.
A PyTorch-based Speech Toolkit
choose for broad end-to-end speech task coverage and reproducible PyTorch recipes.
End-to-End Speech Processing Toolkit
choose for efficient production ASR with streaming and unified decoding, lighter than NeMo.
Production First and Production Ready End-to-End Speech Recognition Toolkit
Solves the same problem with a different architecture or at a different layer. Expect to rewrite the integration.
choose when you are already in the PaddlePaddle ecosystem and want integrated ASR/TTS/speaker tooling.
Easy-to-use Speech Toolkit including Self-Supervised Learning model, SOTA/Streaming ASR with punctuation, Streaming TTS with text frontend, Speaker Verification System, End-to-End Speech Translation and Keyword Spotting. Won NAACL2022 Best Demo Award.
choose for very low-latency ASR/TTS on CPU/edge for voice agents, not a training framework.
Very low latency speech to text, intent recognition, and text to speech, for building voice agents and interfaces
choose when building Swift/CoreML Apple apps needing on-device speech models.
Frontier CoreML audio models in your apps — text-to-speech, speech-to-text, voice activity detection, and speaker diarization. In Swift, powered by SOTA open source.
choose for Apple Silicon apps needing MLX/CoreML ASR, TTS, VAD, and diarization in Swift.
AI speech toolkit for Apple Silicon — ASR, TTS, speech-to-speech, VAD, and diarization powered by MLX and CoreML
choose when you need a TensorFlow-native seq2seq speech model framework with Google-scale recipes.
Lingvo
An earlier or more limited way to do the job. Still the right call when you need something small, proven, or CPU-only.
choose for lightweight multilingual ASR in five Asian languages with emotion/event tags, no full toolkit needed.
Open-source SenseVoiceSmall model for Mandarin, Cantonese, English, Japanese, and Korean ASR, language ID, emotion recognition, and audio event detection.
choose for high-performance GPU inference of Whisper models in C++ without Python or a full toolkit.
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
choose for lightweight ggml-based inference across many ASR model families on CPU/edge.
ggml speech-to-text inference for 16+ model families
choose for a tiny all-in-one ASR/TTS/voice-conversion model when compute is very limited.
[AutoArk] GPA (General Purpose Audio) can do ASR, TTS and voice conversion with one tiny model!
choose for state-of-the-art multilingual TTS and voice cloning if you don't need ASR.
VoxCPM2: Tokenizer-Free TTS for Multilingual Speech Generation, Creative Voice Design, and True-to-Life Cloning
choose for speech-to-speech/text translation models, not general ASR/TTS development.
Foundational Models for State-of-the-Art Speech and Text Translation
choose for long-form streaming multi-speaker TTS without needing ASR or a full toolkit.
Long-form streaming TTS system for multi-speaker dialogue generation
choose for lightweight offline ASR on mobile/embedded devices with simple APIs and no GPU training.
Offline speech recognition API for Android, iOS, Raspberry Pi and servers with Python, Java, C# and Node
choose for a standalone custom-trainable STT toolkit if you accept stale maintenance (2.5 years since last push).
🐸STT - The deep learning toolkit for Speech-to-Text. Training and deploying STT models has never been so easy.
Does the same job, but ships as an app. Useful to a person, not swappable into a codebase.
choose when you want a ready Python realtime STT library with VAD/wake-word handling instead of building a model pipeline.
A robust, efficient, low-latency speech-to-text library with advanced voice activity detection, wake word activation and instant transcription.
choose when you need a ready Linux note-taking app with offline STT/TTS, not a development framework.
Speech Note Linux app. Note taking, reading and translating with offline Speech to Text, Text to Speech and Machine translation.
choose for a quick Whisper-based ASR+diarization pipeline without retraining or framework setup.
Automatic Speech Recognition with Speaker Diarization based on OpenAI Whisper
choose when you want to dub videos via a GUI, not build speech models.
Synchronized Translation for Videos. Video dubbing
choose when you need a ready Whisper REST API service rather than a full speech framework.
OpenAI Whisper ASR Webservice API
choose when you need a local real-time STT server with OpenAI/Deepgram-compatible APIs.
Real-time, local speech-to-text with streaming ASR, speaker diarization, translation, and OpenAI/Deepgram-compatible APIs.
choose for word-level timestamps and diarization on Whisper, not a general speech framework.
WhisperX: Automatic Speech Recognition with Word-level Timestamps (& Diarization)
These projects were analysed and named Speech among their alternatives. The relationship is not symmetric — how Speech rates them is a separate judgement, made when Speech is analysed in its own right.
calls Speech “Speech AI toolkit (ASR/TTS) in the NeMo ecosystem; choose for integration with other NVIDIA NeMo features.”
Easy-to-use Speech Toolkit including Self-Supervised Learning model, SOTA/Streaming ASR with punctuation, Streaming TTS with text frontend, Speaker Verification System, End-to-End Speech Translation and Keyword Spotting. Won NAACL2022 Best Demo Award.
calls Speech “Choose when you want NVIDIA's PyTorch speech toolkit with tight integration across ASR, TTS, and the NeMo ecosystem.”
A PyTorch-based Speech Toolkit
calls Speech “Choose when you need a production-grade speech AI framework integrated with NeMo for large-scale training and deployment.”
End-to-End Speech Processing Toolkit
calls Speech “Choose when you need a scalable framework for training and deploying custom ASR/TTS models at scale.”
Robust Speech Recognition via Large-Scale Weak Supervision
calls Speech “use it for a full training/inference framework to customize ASR/TTS models, heavier than a pretrained model”
Open-Source Frontier Voice AI
calls Speech “choose when you need a full speech AI framework with custom training and deployment options.”
Instant voice cloning by MIT and MyShell. Audio foundation model.
calls Speech “choose for a comprehensive speech AI framework instead of a single TTS model.”
An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
calls Speech “choose when you need NVIDIA's NeMo ecosystem with multi-modal speech+LLM training and deployment.”
Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
calls Speech “choose for a comprehensive framework for speech/LLM/multimodal AI instead of standalone example scripts.”
State-of-the-Art Deep Learning scripts organized by models - easy to train and deploy with reproducible accuracy and performance on enterprise-grade infrastructure.
calls Speech “Use when you need a full generative-AI framework with trainable speech models and LLM integration, at the cost of heavy GPU/server requirements.”
Speech-to-text, text-to-speech, speaker diarization, speech enhancement, source separation, and VAD using next-gen Kaldi with onnxruntime without Internet connection. Support embedded systems, Android, iOS, HarmonyOS, Raspberry Pi, RISC-V, RK NPU, Axera NPU, Ascend NPU, x86_64 servers, websocket server/client, support 12 programming languages
calls Speech “choose when you need a scalable generative AI framework with ASR/TTS and LLM integration for researchers and cloud deployments.”
Very low latency speech to text, intent recognition, and text to speech, for building voice agents and interfaces
calls Speech “Choose for a research-grade speech framework with many models and training support; it is far heavier than a lightweight realtime library.”
A robust, efficient, low-latency speech-to-text library with advanced voice activity detection, wake word activation and instant transcription.
calls Speech “Use for a production-grade speech framework with TTS/ASR, if you need a full toolkit.”
High-Quality Voice Cloning TTS for 600+ Languages
calls Speech “an enterprise speech framework with ASR/TTS models; choose if you are already in the NeMo ecosystem.”
Open-source SenseVoiceSmall model for Mandarin, Cantonese, English, Japanese, and Korean ASR, language ID, emotion recognition, and audio event detection.