Backend Dispatch#

The glass::nvidia::gemm / gemv / row_strided_* / gemm_batched_1d primary templates auto-dispatch at compile time: for shapes where pure-SIMT wins they fall through to ::glass::*; for shapes where the vendor library wins they route to cuBLASDx via the DEFINE_NVIDIA_* macros.

This means glass::nvidia::gemm<float, 6, 6, 6>(...) “just works” without any DEFINE macro — small shapes route to SIMT automatically. Larger shapes such as glass::nvidia::gemm<float, 32, 32, 32>(...) still require a DEFINE_NVIDIA_GEMM(32, 32, 32) in scope (placed inside namespace glass { namespace nvidia { namespace block {), but produce a clean compile-time message when it is missing.

Note

This cuBLASDx-vs-SIMT decision is one of three distinct dispatch layers. glass::suggested_backend<> (Tuning for Your Hardware) is a host-side ladder that advises launch-level packing — which tier (thread / warp / block / nvidia) to launch, i.e. the shape of <<<grid, block>>>. Newest, and different from both: glass::dispatch_body() (glass-dispatch.cuh) picks the in-block body behind the bare glass::op / glass::nvidia::op face, under a fixed block-scope calling contract — the launch does not change. The measured in-block body sweep (bench/tune.py --legs body, Phase 2, 2026-07-30) moved the winning cells to a warp- or thread-body executed inside the block — an attested, receipt-gated retune, regenerated per arch like the ladder. A moved cell matches the block body to tolerance, not bit-exactly; ops with no moved cell remain exactly glass::block::. Pin glass::block:: where determinism is load-bearing; see Namespaces, suffixes, and flags and Tuning for Your Hardware.

The dispatch flow#

           caller writes:  glass::nvidia::gemm<float, M, N, K>(...)
                                  │
                                  ▼
                should_use_cublasdx<float, M, N, K, SMS>()
                                  │
           ┌──────────────────────┴──────────────────────┐
           ▼                                             ▼
         false                                          true
           │                                             │
────────── ▼ ──────────              ────────── ▼ ──────────
SIMT fallback:                       Need a DEFINE_NVIDIA_GEMM*
::glass::gemm<T,M,N,K>(...)          to specialize for cuBLASDx;
(no DEFINE needed; no smem)          else: static_assert error.

The decision is made at compile time by should_use_cublasdx*<T,M,N,K,SM>() (see src/nvidia/query.cuh / query_simt.cuh), which consults — in order:

A per-build local override table (when GLASS_TUNING_TABLE_LOCAL is defined).
The shipped global table (src/nvidia/tuning_table.cuh).
A fallback static heuristic.

The shipped table currently covers sm_120 (Blackwell-class) for square shapes from 3×3×3 up to 64×64×64; regenerate for another arch with bench/autotune.py (a new arch gets its own entries without touching the existing ones).

The size heuristic#

When no tuning-table entry matches, each API falls back to a per-API heuristic that reflects its arithmetic intensity:

Template	Default heuristic
`cublasdx_wins<M, N, K, SM>` (gemm)	`max(M,N,K) >= 16 AND min(M,N,K) >= 4`
`cublasdx_wins_gemv<M, N, SM>`	`max(M,N) >= 32`
`cublasdx_wins_batched<M, N, K, BATCH, SM>`	`BATCH >= 8 AND max(M,N,K) >= 8`
`cublasdx_wins_gemm_strided<...>`	delegates to `cublasdx_wins<>`
`cublasdx_wins_gemv_strided<...>`	delegates to `cublasdx_wins_gemv<>`

Restricted to float: the heuristic returns SIMT for non-float types.

Inspecting the decision#

The print_dispatch* host helpers (__host__ __device__, so you can also drop one into a kernel for runtime diagnostics) report the chosen path:

glass::nvidia::print_dispatch<float, 4, 4, 4>();
// → glass::nvidia::gemm<T,4,4,4,SM=860>: SIMT fallback

glass::nvidia::print_dispatch<float, 32, 32, 32>();
// → glass::nvidia::gemm<T,32,32,32,SM=860>: cuBLASDx (needs DEFINE_NVIDIA_GEMM*)

glass::nvidia::print_dispatch_gemv<float, 64, 64>();
// → glass::nvidia::gemv<T,64,64,SM=860>: cuBLASDx (needs DEFINE_NVIDIA_GEMV*)

Overriding the dispatch#

Goal	How
Force cuBLASDx for a shape the heuristic puts in SIMT	Add `DEFINE_NVIDIA_GEMM(M,N,K)` in your `.cu` file — the explicit specialization always overrides the primary template.
Force SIMT for a shape the heuristic puts in cuBLASDx	Call `::glass::gemm<T,M,N,K>(...)` directly (skip the `nvidia::` path).
Per-host tuning without editing source	Run `python bench/autotune.py` to generate `bench/tuning/<hostname>.cuh`, then compile with `-DGLASS_TUNING_TABLE_LOCAL='"bench/tuning/<hostname>.cuh"'`.
Different SM in-tree (for a PR)	`python bench/autotune.py --in-tree` rewrites the marker-delimited specializations inside `src/nvidia/tuning_table.cuh`.

The shipped values are sensible defaults, but small-GEMM performance is highly SM-dependent. See Tuning for Your Hardware for the full per-host autotuning workflow that measures both paths side by side on your hardware so you can override these defaults.