Distributed Compilation Guide

Distributed Compilation

When parallelism on a single machine isn't enough. Scale your C/C++ builds across multiple machines using ccache, sccache, distcc, or icecream.

Which tool should I use?

Single developer, local machine: → ccache
CI farm with shared S3 / GCS bucket: → sccache
Dedicated LAN build servers (simple setup): → distcc + ccache
LAN cluster, want automatic load balancing: → icecream
Cross-compiling (e.g., ARM on x86 cluster): → icecream (has native cross-compile support)
Unreal Engine build farm: → UBT + FastBuild or sccache

ccache

Recommended first step

ccache caches the results of compilation. On the next build—even after make clean—unchanged files compile in milliseconds. Ideal for local development and CI pipelines that reuse the cache between runs.

Expected speedup: 5–20× on incremental CI builds
Setup complexity: Low
Best for: Any
Installation
# macOS
brew install ccache

# Ubuntu / Debian
sudo apt-get install ccache
Configuration
# In your CMakeLists.txt
find_program(CCACHE_FOUND ccache)
if(CCACHE_FOUND)
  set(CMAKE_C_COMPILER_LAUNCHER ccache)
  set(CMAKE_CXX_COMPILER_LAUNCHER ccache)
endif()

# In your Makefile
CC  := ccache gcc
CXX := ccache g++

sccache

Best for CI farms

sccache is a ccache-like tool from Mozilla that supports cloud storage backends (S3, GCS, Azure Blob) for sharing the cache across multiple CI machines, drastically reducing cold-start build times.

Expected speedup: 10–50× on multi-agent CI
Setup complexity: Medium
Best for: 5+ engineers / CI farm
Installation
# cargo
cargo install sccache

# Or download a pre-built binary from:
# https://github.com/mozilla/sccache/releases
Configuration
# In your CMakeLists.txt
find_program(SCCACHE_FOUND sccache)
if(SCCACHE_FOUND)
  set(CMAKE_C_COMPILER_LAUNCHER sccache)
  set(CMAKE_CXX_COMPILER_LAUNCHER sccache)
endif()

# Point to an S3 bucket for shared cache:
export SCCACHE_BUCKET=my-build-cache-bucket
export SCCACHE_REGION=us-east-1

distcc

On-premise / LAN

distcc distributes the preprocessing and compilation of C/C++ files across a cluster of machines. Each worker needs the same compiler version. Works best with dedicated build servers on a fast LAN.

Expected speedup: 4–15× depending on cluster size
Setup complexity: High
Best for: 10+ engineers or CI farm
Installation
# On every machine (controller + workers):
sudo apt-get install distcc

# Start the daemon on each worker:
distccd --daemon --allow 192.168.0.0/24
Configuration
# Controller: set hosts list
export DISTCC_HOSTS="localhost/4 build1.local/8 build2.local/8"

# Makefile
CC  := distcc gcc
CXX := distcc g++

# CMakeLists.txt
set(CMAKE_C_COMPILER_LAUNCHER distcc)
set(CMAKE_CXX_COMPILER_LAUNCHER distcc)

# Combine with ccache for maximum speed:
set(CMAKE_CXX_COMPILER_LAUNCHER ccache;distcc)

icecream (icecc)

Modern distcc alternative

icecream (icecc) is a more modern alternative to distcc. It uses a scheduler daemon to automatically route jobs to the least-loaded machine, supports cross-compilation, and requires much less manual configuration than distcc.

Expected speedup: 5–20× on 5+ node cluster
Setup complexity: Medium
Best for: 10+ engineers
Installation
# Ubuntu / Debian:
sudo apt-get install icecc icemon

# Start the scheduler (once, on any machine):
sudo systemctl start iceccd

# Start the daemon on each worker:
sudo systemctl start iceccd
Configuration
# CMakeLists.txt
find_program(ICECC_FOUND icecc)
if(ICECC_FOUND)
  set(CMAKE_C_COMPILER_LAUNCHER icecc)
  set(CMAKE_CXX_COMPILER_LAUNCHER icecc)
endif()

# Monitor the cluster graphically:
icemon

Upload your build log to see the impact

After setting up distributed compilation, upload a new build log to MakeOptimize to compare your before/after build times and confirm the regression baseline has been reset.

Analyze Build Log