Meta open-sources Muse Spark Open: an 816B-parameter MoE with a 1M-token context

Meta has released Muse Spark Open, an open-weight multimodal model with a mixture-of-experts architecture. It has about 816B parameters in total, activates roughly 42B on each token, and has a context window of up to a million tokens. It takes text, images, audio and video as input and outputs text. According to Hugging Face's description, it is the strongest open model from the US to date. Meta trained it primarily for agentic coding: reading a codebase, writing and debugging code, calling tools, planning, acting, observing results and correcting itself on long tasks.

Muse Spark Open is the flagship of the Muse lineup. In August, Meta released Muse Glimmer, a small dense model for local use. The two share a tokenizer and vocabulary, and the new model's chat template extends the Glimmer template to cover the new modalities. In practice, you can feed a screenshot, a voice note or a screen recording to the same model that is editing your code at that moment.

Where the 816 billion are hidden

The language part consists of 62 layers with a hidden dimension of 8192. Attention is grouped-query: 128 query heads and 16 key-value heads of 64 dimensions each. The model has an output gate, output normalization at the end of each block, and QK-norm for queries and keys. Instead of a plain sum, the block combines its input and output through a learnable residual gate. Glimmer has a single dense feed-forward network in each layer; Muse Spark Open replaces it with an MoE block of 256 experts, and the router picks 8 per token. Each expert projects 8192 to 4096 and back to 8192, and normalizes its own output. Experts account for about 97% of all parameters.

This is the main MoE trade-off: all 256 experts must sit in memory, so deployment needs the resources of an 816B model, but each token is computed at ~42B, so speed is noticeably higher. The input embeddings and the output layer are separate matrices of about 1.66B parameters each.

Eyes, ears and a million tokens

Images and video frames are turned into tokens by a perception encoder with 1.8B parameters: 50 layers, dimension 1536, 16 attention heads, 14×14 patches and 4096-dimensional features. Every fourth layer and the last one use full attention; the rest use sparse attention. Positions are encoded with learnable absolute embeddings and 2D RoPE. Muse Glimmer 30B uses the same encoder.

Audio goes through a separate tower that requires a mono signal at 24 kHz. One audio token covers 80 ms, which is 12.5 tokens per second, so ten seconds of recording produce 125 tokens. The encoder is a convolutional pyramid with patch embedders of sizes 16, 6, 5 and 4. Video is sampled at 2 frames per second by default, up to 96 frames, and can now be processed together with its audio, which Glimmer could not do. Frames are interleaved with text timestamps such as "Time: 3.5s". A higher FPS catches events between frames more precisely, but memory use and prefill latency grow.

The context is 1 048 576 tokens, and images, video and audio share that window with text. Thanks to GQA, the cache stays moderate: each token adds 2 × 16 × 64 × 62 = 126 976 values, which is about 250 KB in BF16 and roughly 266 GB for a full million tokens.

Key specifications

• Size: ~816B parameters in total, ~42B active per token
• Routing: 256 experts per MoE layer, 8 per token
• Input: text, images, audio, video; output: text
• Context: up to 1M tokens
• Layers: 62, hidden dimension 8192, 128 query heads and 16 KV heads
• Vision encoder: 1.8B parameters, 50 layers
• Audio: 12.5 tokens per second, 24 kHz mono
• Checkpoints: BF16 (~1.5 TB, safetensors) and NVFP4 (4-bit, for NVIDIA Blackwell, reportedly with negligible accuracy loss)
• Repositories: Muse-Spark-Open-816B-A42B and Muse-Spark-Open-816B-A42B-NVFP4

How to run it and what to know

transformers includes the MuseSparkForConditionalGeneration and MuseSparkProcessor classes. The processor combines the tokenizer, the image and video processors, the audio extractor and the chat template. Parallelism is set through DistributedConfig: tp_size shards the dense layers, and ep_size distributes experts across devices. In transformers, the BF16 version runs on four nodes with 8 H200 GPUs each using tensor parallelism TP=32, while NVFP4 runs on a single B200 node. For production, Hugging Face has prepared recipes for llama.cpp, vLLM and SGLang. Tool calling is supported, including multimodal tool calling.

There is one catch with reasoning: it cannot be turned off. The levels low, medium, high and xhigh correspond to the values 16, 64, 256 and 512, and the default is xhigh. For simple tasks, set low, or tokens will be wasted. The numeric value can also be set with a "Reasoning strength" line in the system prompt. For teams building coding agents, the main question is no longer whether the weights are available, but how much they can spend on hardware and on memory for the KV-cache at long contexts.