Exo

EXO v1.0.71 Release Notes

This is a small patch release, with better defaults for sampling and bug fixes for M5 series Macs and RDMA, along with support for the new Kimi K2.6.

Models

• Add Kimi K2.6, including support for multimodality (#1952)
• Add model cards for Qwen3.6 27B, GLM 5.1 and Kimi K2.6 quantisations (#1970)

Performance

• Add recommended sampling defaults (from model providers) to improve model output quality, including min_p and top_k (#1947)
• Prevent JACCL all_sum from corrupting output (#1952)

Quality of Life

• Add a Pi integration tab! (#1925)
• Handle multiple tool calls correctly in the Claude and Responses APIs (#1952)
• Add a friendly context message for bug reporting (#1959)

Bug Fixes

• Fix vision models on M5 series Macs by fixing addmm on NAX (#1927)
• Fix prefix cache for Qwen3.6 (#1952)
• Prevent complete downloads from losing their Completed status (#1956, #1958)
• Populate the image cache from events to prevent crashes (#1961)
• Fix DeepSeekV32 parsing and tensor parallelism for Gemma 4 models with effective parameters (#1967)

New Contributors

And a thank you to @nhwaani who made their first contribution here!

Full Changelog: https://github.com/exo-explore/exo/compare/v1.0.70...v1.0.71

<div align="center">

<picture>

<source media="(prefers-color-scheme: light)" srcset="/docs/imgs/exo-logo-black-bg.jpg">

<img alt="exo logo" src="/docs/imgs/exo-logo-transparent.png" width="50%" height="50%">

</picture>

exo: Run frontier AI locally. Maintained by exo labs.

<p align="center">

<a href="https://discord.gg/TJ4P57arEm" target="_blank" rel="noopener noreferrer"><img src="https://img.shields.io/badge/Discord-Join%20Server-5865F2?logo=discord&logoColor=white" alt="Discord"></a>

<a href="https://x.com/exolabs" target="_blank" rel="noopener noreferrer"><img src="https://img.shields.io/twitter/follow/exolabs?style=social" alt="X"></a>

<a href="https://www.apache.org/licenses/LICENSE-2.0.html" target="_blank" rel="noopener noreferrer"><img src="https://img.shields.io/badge/License-Apache2.0-blue.svg" alt="License: Apache-2.0"></a>

</p>

</div>

---

exo connects all your devices into an AI cluster. Not only does exo enable running models larger than would fit on a single device, but with day-0 support for RDMA over Thunderbolt, makes models run faster as you add more devices.

Features

• Automatic Device Discovery: Devices running exo automatically discover each other - no manual configuration.
• RDMA over Thunderbolt: exo ships with day-0 support for RDMA over Thunderbolt 5, enabling 99% reduction in latency between devices.
• Topology-Aware Auto Parallel: exo figures out the best way to split your model across all available devices based on a realtime view of your device topology. It takes into account device resources and network latency/bandwidth between each link.
• Tensor Parallelism: exo supports sharding models, for up to 1.8x speedup on 2 devices and 3.2x speedup on 4 devices.
• MLX Support: exo uses MLX as an inference backend and MLX distributed for distributed communication.
• Multiple API Compatibility: Compatible with OpenAI Chat Completions API, Claude Messages API, OpenAI Responses API, and Ollama API - use your existing tools and clients.
• Custom Model Support: Load custom models from HuggingFace hub to expand the range of available models.

Dashboard

exo includes a built-in dashboard for managing your cluster and chatting with models.

<p align="center">

<img src="docs/imgs/dashboard-cluster-view.png" alt="exo dashboard - cluster view showing 4 x M3 Ultra Mac Studio with DeepSeek v3.1 and Kimi-K2-Thinking loaded" width="80%" />

</p>

<p align="center"><em>4 × 512GB M3 Ultra Mac Studio running DeepSeek v3.1 (8-bit) and Kimi-K2-Thinking (4-bit)</em></p>

Benchmarks

<details>

<summary>Qwen3-235B (8-bit) on 4 × M3 Ultra Mac Studio with Tensor Parallel RDMA</summary>

<img src="docs/benchmarks/jeffgeerling/mac-studio-cluster-ai-full-1-qwen3-235b.jpeg" alt="Benchmark - Qwen3-235B (8-bit) on 4 × M3 Ultra Mac Studio with Tensor Parallel RDMA" width="80%" />

<p>

<strong>Source:</strong> <a href="https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5">Jeff Geerling: 15 TB VRAM on Mac Studio – RDMA over Thunderbolt 5</a>

</p>

</details>

<details>

<summary>DeepSeek v3.1 671B (8-bit) on 4 × M3 Ultra Mac Studio with Tensor Parallel RDMA</summary>

<img src="docs/benchmarks/jeffgeerling/mac-studio-cluster-ai-full-2-deepseek-3.1-671b.jpeg" alt="Benchmark - DeepSeek v3.1 671B (8-bit) on 4 × M3 Ultra Mac Studio with Tensor Parallel RDMA" width="80%" />

<p>

<strong>Source:</strong> <a href="https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5">Jeff Geerling: 15 TB VRAM on Mac Studio – RDMA over Thunderbolt 5</a>

</p>

</details>

<details>

<summary>Kimi K2 Thinking (native 4-bit) on 4 × M3 Ultra Mac Studio with Tensor Parallel RDMA</summary>

<img src="docs/benchmarks/jeffgeerling/mac-studio-cluster-ai-full-3-kimi-k2-thinking.jpeg" alt="Benchmark - Kimi K2 Thinking (native 4-bit) on 4 × M3 Ultra Mac Studio with Tensor Parallel RDMA" width="80%" />

<p>

<strong>Source:</strong> <a href="https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5">Jeff Geerling: 15 TB VRAM on Mac Studio – RDMA over Thunderbolt 5</a>

</p>

</details>

---

Quick Start

Devices running exo automatically discover each other, without needing any manual configuration. Each device provides an API and a dashboard for interacting with your cluster (runs at http://localhost:52415).

There are two ways to run exo:

Run from Source (macOS)

If you have Nix installed, you can skip most of the steps below and run exo directly:

nix run .#exo

Note: To accept the Cachix binary cache (and avoid the Xcode Metal ToolChain), add to /etc/nix/nix.conf:

trusted-users = root    (or your username)
experimental-features = nix-command flakes

Then restart the Nix daemon: sudo launchctl kickstart -k system/org.nixos.nix-daemon

Prerequisites:

• Xcode (provides the Metal ToolChain required for MLX compilation)
• brew (for simple package management on macOS)

  /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

• uv (for Python dependency management)
• node (for building the dashboard)

  brew install uv node

• rust (to build Rust bindings, nightly for now)

  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
  rustup toolchain install nightly

• macmon (for hardware monitoring on Apple Silicon)

Install the pinned fork revision used by this repo instead of Homebrew macmon.

Homebrew macmon 0.6.1 still crashes on Apple M5.

  cargo install --git https://github.com/vladkens/macmon \
    --rev a1cd06b6cc0d5e61db24fd8832e74cd992097a7d \
    macmon \
    --force

Clone the repo, build the dashboard, and run exo:

# Clone exo
git clone https://github.com/exo-explore/exo

# Build dashboard
cd exo/dashboard && npm install && npm run build && cd ..

# Run exo
uv run exo

This starts the exo dashboard and API at http://localhost:52415/

Please view the section on RDMA to enable this feature on MacOS >=26.2!

Run from Source (Linux)

Prerequisites:

• uv (for Python dependency management)
• node (for building the dashboard) - version 18 or higher
• rust (to build Rust bindings, nightly for now)

Installation methods:

Option 1: Using system package manager (Ubuntu/Debian example):

# Install Node.js and npm
sudo apt update
sudo apt install nodejs npm

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install Rust (using rustup)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup toolchain install nightly

Option 2: Using Homebrew on Linux (if preferred):

# Install Homebrew on Linux
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install dependencies
brew install uv node

# Install Rust (using rustup)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup toolchain install nightly

Note: The macmon package is macOS-only and not required for Linux.

Clone the repo, build the dashboard, and run exo:

# Clone exo
git clone https://github.com/exo-explore/exo

# Build dashboard
cd exo/dashboard && npm install && npm run build && cd ..

# Run exo
uv run exo

This starts the exo dashboard and API at http://localhost:52415/

Important note for Linux users: Currently, exo runs on CPU on Linux. GPU support for Linux platforms is under development. If you'd like to see support for your specific Linux hardware, please search for existing feature requests or create a new one.

Configuration Options:

• --no-worker: Run exo without the worker component. Useful for coordinator-only nodes that handle networking and orchestration but don't execute inference tasks. This is helpful for machines without sufficient GPU resources but with good network connectivity.

  uv run exo --no-worker

• --legacy-daemon: Run exo as a legacy SysV-style background daemon using double-fork daemonization. This is intended for legacy init scripts; systemd and launchd should run exo in the foreground without this flag.

  uv run exo --legacy-daemon

File Locations (Linux):

exo follows the XDG Base Directory Specification on Linux:

• Configuration files: ~/.config/exo/ (or $XDG_CONFIG_HOME/exo/)
• Data files: ~/.local/share/exo/ (or $XDG_DATA_HOME/exo/)
• Cache files: ~/.cache/exo/ (or $XDG_CACHE_HOME/exo/)
• Log files: ~/.cache/exo/exo_log/ (with automatic log rotation)
• Custom model cards: ~/.local/share/exo/custom_model_cards/

You can override these locations by setting the corresponding XDG environment variables.

macOS App

exo ships a macOS app that runs in the background on your Mac.

<img src="docs/imgs/macos-app-one-macbook.png" alt="exo macOS App - running on a MacBook" width="35%" />

The macOS app requires macOS Tahoe 26.2 or later.

Download the latest build here: EXO-latest.dmg.

You can also install the latest build with Homebrew:

brew install --cask exo

The app will ask for permission to modify system settings and install a new Network profile. Improvements to this are being worked on.

Custom Namespace for Cluster Isolation:

The macOS app includes a custom namespace feature that allows you to isolate your exo cluster from others on the same network. This is configured through the EXO_LIBP2P_NAMESPACE setting:

• Use cases:
• Running multiple separate exo clusters on the same network
• Isolating development/testing clusters from production clusters
• Preventing accidental cluster joining

• Configuration: Access this setting in the app's Advanced settings (or set the EXO_LIBP2P_NAMESPACE environment variable when running from source)

The namespace is logged on startup for debugging purposes.

Uninstalling the macOS App

The recommended way to uninstall is through the app itself: click the menu bar icon → Advanced → Uninstall. This cleanly removes all system components.

If you've already deleted the app, you can run the standalone uninstaller script:

sudo ./app/EXO/uninstall-exo.sh

This removes:

• Network setup LaunchDaemon
• Network configuration script
• Log files
• The "exo" network location

Note: You'll need to manually remove EXO from Login Items in System Settings → General → Login Items.

---

Enabling RDMA on macOS

RDMA is a new capability added to macOS 26.2. It works on any Mac with Thunderbolt 5 (M4 Pro Mac Mini, M4 Max Mac Studio, M4 Max MacBook Pro, M3 Ultra Mac Studio).

Please refer to the caveats for immediate troubleshooting.

To enable RDMA on macOS, follow these steps:

1. Shut down your Mac.
2. Hold down the power button for 10 seconds until the boot menu appears.
3. Select "Options" to enter Recovery mode.
4. When the Recovery UI appears, open the Terminal from the Utilities menu.
5. In the Terminal, type:

   rdma_ctl enable

and press Enter.

1. Reboot your Mac.

After that, RDMA will be enabled in macOS and exo will take care of the rest.

Important Caveats

1. Devices that wish to be part of an RDMA cluster must be connected to all other devices in the cluster.
2. The cables must support TB5.
3. On a Mac Studio, you cannot use the Thunderbolt 5 port next to the Ethernet port.
4. If running from source, please use the script found at tmp/set_rdma_network_config.sh, which will disable Thunderbolt Bridge and set dhcp on each RDMA port.
5. RDMA ports may be unable to discover each other on different versions of MacOS. Please ensure that OS versions match exactly (even beta version numbers) on all devices.

---

Environment Variables

exo supports several environment variables for configuration:

| Variable | Description | Default |

|----------|-------------|---------|

| EXO_DEFAULT_MODELS_DIR | Default directory for model downloads and caches. Always first in the writable dirs list. | ~/.local/share/exo/models (Linux) or ~/.exo/models (macOS) |

| EXO_MODELS_DIRS | Colon-separated additional writable directories for model downloads. Checked in order after the default; first with enough free space is used. | None |

| EXO_MODELS_READ_ONLY_DIRS | Colon-separated read-only directories to search for pre-downloaded models (e.g., NFS mounts, shared storage). Models here cannot be deleted. | None |

| EXO_OFFLINE | Run without internet connection (uses only local models) | false |

| EXO_ENABLE_IMAGE_MODELS | Enable image model support | false |

| EXO_LIBP2P_NAMESPACE | Custom namespace for cluster isolation | None |

| EXO_FAST_SYNCH | Control MLX_METAL_FAST_SYNCH behavior (for JACCL backend) | Auto |

| EXO_TRACING_ENABLED | Enable distributed tracing for performance analysis | false |

Example usage:

# Use pre-downloaded models from NFS mount (read-only)
EXO_MODELS_READ_ONLY_DIRS=/mnt/nfs/models:/opt/ai-models uv run exo

# Download models to an external SSD (falls back to default dir if full)
EXO_MODELS_DIRS=/Volumes/ExternalSSD/exo-models uv run exo

# Run in offline mode
EXO_OFFLINE=true uv run exo

# Enable ima