By

Asif Razzaq

June 6, 2026

NVIDIA’s Nemotron Speech team has released Nemotron 3.5 ASR. It is a 600M-parameter streaming Automatic Speech Recognition (ASR) model. A single checkpoint transcribes 40 language-locales in real time. Punctuation and capitalization are built in natively. The model ships as open weights on Hugging Face. The license is OpenMDW-1.1. The architecture is a Cache-Aware FastConformer-RNNT.

What is Nemotron 3.5 ASR

Nemotron 3.5 ASR extends nvidia/nemotron-speech-streaming-en-0.6b to many languages. It adds prompt-based language-ID conditioning to the base model. That lets one 600M-parameter checkpoint cover 40 language-locales. No per-language model or model-swapping is required.

The model targets two workloads. The first is low-latency streaming for live audio. The second is high-throughput batch transcription. Output is production-ready text with proper casing and punctuation. No separate punctuation-restoration step is needed.

Image source: https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b

How Cache-Aware FastConformer-RNNT Works

The model has two main pieces. The first is a Cache-Aware FastConformer encoder with 24 layers. FastConformer is an efficient evolution of the Conformer architecture. It uses linearly scalable attention. The second piece is an RNNT (Recurrent Neural Network Transducer) decoder. RNNT emits text frame by frame as audio streams in.

The “cache-aware” design is the efficiency lever. Buffered streaming re-processes overlapping audio windows at every step. That repeats the same work and adds delay. This model caches encoder self-attention and convolution activations instead. It reuses those cached states as new audio arrives. So each audio frame is processed exactly once, with no overlap. Compute and end-to-end latency both drop, without an accuracy penalty.

The Latency Knob: att_context_size

One inference setting controls the latency-accuracy tradeoff. It is the attention context size, att_context_size. Smaller context emits text sooner but sees less future audio. Larger context raises accuracy at higher latency.

The same checkpoint covers the full range. Settings map to chunk sizes of 80ms, 160ms, 320ms, 560ms, and 1.12s. For example, [56,0] gives an 80ms ultra-low-latency mode. The [56,13] setting gives 1.12s for highest accuracy. Teams pick the operating point at inference time, with no retraining.

Language Detection and Coverage

The 40 language-locales include English, Spanish, German, and French variants. They also cover Arabic, Japanese, Korean, Mandarin, Hindi, and Thai. Several other European and Nordic languages are included too.

Language conditioning works two ways. Setting target_lang to a known locale usually gives the best accuracy. Setting target_lang=auto lets the model detect the language itself. In auto mode, it emits a language tag after terminal punctuation. One deployment can then transcribe mixed-language traffic. No separate language-ID component is required.

Comparison

Fine-Tuning Results

Because the weights are open, teams can fine-tune for a language, domain, or accent. NVIDIA published a worked example on Greek and Bulgarian. It fine-tuned the base checkpoint with the same Cache-Aware FastConformer-RNNT recipe. Each clip carried a target_lang tag for language conditioning. Training data came from public corpora, including Granary, Common Voice, and FLEURS.

Results were measured as WER on held-out FLEURS, at the 80ms setting. Greek WER fell from 35 to 24, a 32% relative improvement. Bulgarian fell from 22 to 15, a 31% relative improvement. These are raw WER percentages at the lowest-latency streaming mode. NVIDIA notes that evaluating at deployment latency, on held-out data, gives honest numbers.

Strengths and Considerations

Strengths:

• One 600M-parameter checkpoint covers 40 language-locales, cutting deployment sprawl.
• Cache-aware streaming processes each frame once, reported at 17x buffered concurrency on an H100.
• att_context_size tunes latency from 80ms to 1.12s at inference, with no retraining.
• Punctuation, capitalization, and auto language tagging are built in.
• Open weights enabled a 31–32% relative WER drop on Greek and Bulgarian after fine-tuning.

Considerations:

• The model handles English, but NVIDIA recommends its dedicated English model for English-only use.
• The 80ms mode trades some accuracy for the lowest latency.
• Japanese and Korean use CER, so cross-language error comparisons need care.
• Throughput figures are measured on H100, so results on other GPUs will differ.
• The production NIM with gRPC streaming is announced, but not yet released.

Key Takeaways

• NVIDIA’s Nemotron 3.5 ASR is an open-weights (OpenMDW-1.1), 600M-parameter streaming model transcribing 40 language-locales from one checkpoint.
• Its Cache-Aware FastConformer-RNNT design processes each audio frame once, reported at 17x the concurrent streams of buffered approaches on an H100.
• Latency is configurable from 80ms to 1.12s at inference via att_context_size, with no retraining.
• A short fine-tune cut FLEURS WER 32% on Greek (35?24) and 31% on Bulgarian (22?15), at the 80ms setting.
• It is self-hostable and streaming-native, unlike closed APIs (Deepgram, AssemblyAI, ElevenLabs) or offline Whisper.

Marktechpost’s Visual Explainer

NEMOTRON 3.5 ASR1 / 10

NVIDIA · STREAMING SPEECH AI · OPEN WEIGHTS

Nemotron 3.5 ASR

A 600M-parameter cache-aware streaming model that transcribes 40 language-locales in real time, from a single checkpoint.

600M parameters40 language-locales80ms–1.12s latencyOpenMDW-1.1

01 — WHAT IT IS

One model, 40 language-locales

• Extends nvidia/nemotron-speech-streaming-en-0.6b with prompt-based language-ID conditioning.
• A single 600M-parameter checkpoint covers 40 language-locales. No model-swapping.
• Punctuation and capitalization are built in. No separate post-processing step.
• Targets two workloads: low-latency streaming and high-throughput batch.
• NVIDIA still recommends its English-only model for English-only use.

02 — ARCHITECTURE

Cache-Aware FastConformer-RNNT

• A 24-layer FastConformer encoder paired with an RNNT decoder.
• Buffered streaming re-processes overlapping audio windows at every step.
• This model caches encoder self-attention and convolution states, then reuses them.
• Each audio frame is processed exactly once, with no overlap.
• Compute and end-to-end latency drop, with no accuracy penalty.

03 — THE LATENCY KNOB

One setting tunes latency vs. accuracy

Same checkpoint, chosen at inference time. No retraining required.

04 — LANGUAGES & DETECTION

Coverage and automatic language ID

• 40 language-locales, including English, Spanish, German, and French variants.
• Also covers Arabic, Japanese, Korean, Mandarin, Hindi, and Thai.
• Set target_lang to a known locale for the best accuracy.
• Set target_lang=auto to let the model detect the language.
• In auto mode, it emits a language tag after terminal punctuation.
• One deployment handles mixed-language traffic, with no separate language-ID component.

05 — THROUGHPUT

Half the size, more concurrent streams

• NVIDIA compares it against Parakeet RNNT 1.1B multilingual, which uses buffered streaming.
• Nemotron 3.5 ASR is roughly half the size: 0.6B versus 1.1B.
• The team reports 17x the concurrent streams of buffered approaches, on the same H100.
• Avoiding redundant recomputation lowers the cost per stream in production.

The 17x figure is from the release announcement; the model card states the qualitative claim directly.

06 — FINE-TUNING RESULTS

A short fine-tune lifts weaker languages

Raw WER (%) on held-out FLEURS at the 80ms setting. Data: Granary, Common Voice, FLEURS.

07 — AVAILABILITY & ACCESS

Open weights, plus a hosted path

• Weights on Hugging Face under the OpenMDW-1.1 license.
• Runtime is NeMo 26.06 or newer. Input must be mono-channel.
• Hosted on DeepInfra, which adds word boosting for domain vocabulary.
• NVIDIA says a NIM release is planned for later in the month, with gRPC streaming.
• Stated GPU support: Ampere, Hopper, Blackwell, Lovelace, Turing, Volta, and Jetson.

08 — HOW IT COMPARES

Where it sits in the landscape

Latency and WER are not directly comparable across vendors; this compares structure, not a ranking.

09 — KEY TAKEAWAYS

The short version

• An open-weights 600M streaming model transcribing 40 language-locales from one checkpoint.
• Cache-aware design processes each frame once; reported 17x buffered concurrency on an H100.
• Latency configurable from 80ms to 1.12s at inference, with no retraining.
• A short fine-tune cut FLEURS WER 32% (Greek) and 31% (Bulgarian).
• Self-hostable and streaming-native, unlike closed APIs or offline Whisper.

? Prev

Next ?

Curated for AI engineers byMarktechpost— practitioner-first coverage of AI & ML.

Check out the Model weights. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us