One Source for Two CUDAs: crucible on Pascal (CUDA 12.9) and Ada (CUDA 13), Compiled in Docker on an ARM Mac

Abstract

saskatchewan, the fleet's census machine, runs Copal (Alpine, musl), and its NVIDIA card has never been used. The card turned out to be an RTX 4060 Ti 16 GB (10de:2805, Ada Lovelace, compute 8.9), two generations past the fleet's Pascal cards. Paul asked whether, on Ubuntu 26.04, the fleet's CUDA program could run "without much translation", and whether one source could serve both kinds of card, "only differing by a small DEFINE statement or like that. Some machine specific aware."

It can, and it needs no DEFINE at all:

• The source: crucible uses 26 CUDA runtime and Thrust names. CUDA 13 removed exactly one of them: the field cudaDeviceProp::clockRate, read once, for a line of print. It is replaced by cudaDeviceGetAttribute(cudaDevAttrClockRate), which exists in CUDA 12 and 13 alike.
• The Makefile: it now reads the card's compute capability from nvidia-smi and builds for it. It refuses, with a sentence, the one pairing that cannot work: CUDA 13 on a card older than Turing.
• tools/cuda-node.sh: it now holds Ubuntu 24.04 and 26.04 in one table, and reads the card's generation from its PCI id before any driver is installed.

With no CUDA 13 machine in the fleet, the CUDA 13 build was proved in NVIDIA's own linux/arm64 development container on this Mac (Apple silicon, no NVIDIA GPU). Device code is compiled for the architecture you name, not the one you run on, so a Mac can compile for an Ada card:

• the old source failed there, with the one predicted error, at line 451;
• the new source built for sm_89;
• the Makefile's guard refused sm_61 under CUDA 13.

On calber, the new source built and ran on CUDA 12.9 as before.

I. Objective

1. Find what in crucible depends on the CUDA version and the card.
2. Make one source and one Makefile build correctly on every card in the fleet, with nothing set by hand.
3. Prepare cuda-node.sh for Ubuntu 26.04 on saskatchewan.
4. Prove the CUDA 13 build without a CUDA 13 machine.

II. Materials

III. What crucible asks of CUDA

Every CUDA and Thrust name in the source, counted with grep -o:

So two things tie crucible to a CUDA and a card: one field, and one line of the Makefile. Neither is in the simulation, the net or the training.

IV. The hybrid: the machine says, the source does not branch

Why there is no DEFINE. A #if CUDART_VERSION >= 13000 around the print would have worked. But the replacement call exists in both toolkits, so a branch would only keep the removed field alive on 12.9, for nothing. The one thing that truly differs by machine is the architecture to compile for. That belongs to the build, and the card can be asked for it.

The source (src/crucible.cu, at the device line):

// The clock as an attribute, not cudaDeviceProp::clockRate: CUDA 13
// removed the field, and the attribute is in 12 and 13 alike, so one
// source builds on the Pascal cards (12.9) and the Ada one (13).
int khz = 0;
CK(cudaDeviceGetAttribute(&khz, cudaDevAttrClockRate, 0));

The Makefile, in three parts:

make which prints what this machine would build: card sm_61, CUDA 12 (/usr/local/cuda/bin/nvcc), H=32, MINB=2.

Why not nvcc -arch=native? It also builds for the card present, and cuda-node.sh's check program uses it. But it says nothing about what it chose, and it cannot be checked against the toolkit before nvcc runs. The explicit CAP can be printed, overridden and guarded.

The node script (tools/cuda-node.sh). What differs between the two releases is one table, and nothing else in the script branches on the release:

The card is read before there is a driver. NVIDIA numbers its chips in order, and Turing begins at 10de:1e00 (TU102). A device id below that is Maxwell, Pascal or Volta, with compute under 7.5. On 26.04 such a card stops the run before anything is installed: "Ubuntu 26.04 has only CUDA 13, which does not build for this card: install Ubuntu 24.04". On 24.04 it is held to driver 580 and CUDA 12.

V. Results

VI. Discussion

Compiling for a GPU does not need a GPU. nvcc turns device code into code for the architecture it is given (sm_89 here). The host CPU only decides what the host half of the binary runs on. So NVIDIA's arm64 development image, on a Mac with no NVIDIA hardware at all, is a full CUDA 13 compiler for an Ada card. The container's warning, "The NVIDIA Driver was not detected", is true and does not matter for a build. This turned the question "will it compile on 13?" from install day's problem into a minute's work today, and it showed that the predicted single error was the only one. Its limits:

• It proves the build, not the run. The kernels' behaviour on Ada, driver 595 and CUDA 13.4 is for the first segment on saskatchewan.
• The versions differ. The container was CUDA 13.0.1 and saskatchewan will get 13.4. The removal of clockRate was in 13.0, so a later 13 removes nothing crucible needs unless its notes say so.
• colima shares only the home folder. The first attempt mounted the session's scratch folder under /private/tmp and the container saw nothing ("No such file or directory"). The copy went under ~/.cache.

Ask the card what it is, not the machine its name. The Makefile asks the driver for the compute capability, and the node script reads the PCI id. Neither names a machine. A card added next year, or saskatchewan's if it moves, needs no edit. The fleet-wide effect comes free, because the planner already sends the Makefile with every segment.

The one pairing that cannot work stops with a sentence. CUDA 13 on Pascal fails inside nvcc with an architecture error that names neither the card nor the fix. The guard says both, once in the Makefile and once, earlier, in the node script. That is the same lesson as the first CUDA report: say what is happening.

What the Ada card brings. 16 GB, and more shared memory and registers per SM than Pascal. A wider net (make H=48) fits as it does on Pascal. Whether anything larger is worth it is a question for crucible's own numbers.

VII. Install day, saskatchewan

1. Install Ubuntu 26.04 Server with "third-party" ticked, from Copal's boot menu or the ISO.
2. From the Mac: scp tools/cuda-node.sh user@saskatchewan: && ssh -t user@saskatchewan sudo bash cuda-node.sh. It reads the card, keeps or installs 595-server, installs CUDA 13.4, then compiles and runs its check program. It says if it needs a reboot.
3. Add saskatchewan-cuda to the queue's hosts (caps cuda) beside saskatchewan (caps cpu), as calber and boburner have.
4. The first crucible segment builds itself (card sm_89, CUDA 13). Its log is the first run of crucible on Ada.

While it is down, saskatchewan's CPU lane, the largest in the fleet, waits: the keeper marks it unreachable, and its running job goes back on the queue.

References

1. NVIDIA, "CUDA Features Archive", CUDA 13.0: deprecated fields removed from cudaDeviceProp, clockRate → cudaDeviceGetAttribute(cudaDevAttrClockRate). https://docs.nvidia.com/cuda/cuda-features-archive/index.html
2. NVIDIA, "CUDA Toolkit 13.0 Release Notes". https://docs.nvidia.com/cuda/archive/13.0.0/cuda-toolkit-release-notes/
3. Eigen issue 3000, "CUDA 13 compatibility": clockRate and computeMode breaking a build. https://gitlab.com/libeigen/eigen/-/issues/3000
4. orrery, docs/cuda-node-lab-report.md: the 24.04, 580, CUDA 12.9 choice for Pascal.