Development and validation#

./tools/install.sh --test --dev creates the project-local .venv, installs the pinned grid-rbd wheel with the [test] + [dev] extras (pinocchio, mujoco, scipy, gymnasium, pytest, pytest-gpu-proof) and regenerates the vendored GRiD headers. That .venv is the only Python for this repo: receipts, tests and timing all run from it. Never import models, environments or generated files from a sibling checkout.

Installing from a wheel: register_robot and the per-host cache#

pip install gato-mpc (or pip install . from a checkout) ships GATO’s native sources under gato/_native and no compiled module. gato.register_robot(name, urdf, N=[...]) runs codegen into the cache (GATO_CACHE_DIR, default the platform user cache, e.g. ~/.cache/gato): robots/<codegen_key>/<name>/ holds grid.cuh + limits.cuh, manifest.json maps names to entries. The first BSQP(..., plant_type=name) for a horizon compiles bsqpN{N}_{name} into modules/<name>/<module_key>/ (CMake on the bundled tree with GATO_PLANT_ROOT pointing at the cache, GATO_MODULE_OUTPUT_DIR at the entry; build tree removed on success; build.log and meta.json kept) and logs the one-time cost. The module key hashes the robot’s codegen key, N, variant, the native source identity (tools/native_identity.py, the same content hash every module embeds) and the toolchain (CUDA arch from nvidia-smi, nvcc, host compiler, CMake, Python ABI tag, pybind11) — any change is a cache miss, never a stale binary; register_robot(..., build=True) compiles up front. The vendored robots are pre-registered from the bundled headers (_cache.resolve). Package modules in python/gato/ (the receipt profile) win over cache modules with the same name; gato.available() lists both. test/test_register_robot.py is the gate (its slow tier compiles into a temporary cache).

Releasing to PyPI#

.github/workflows/publish.yml is the only upload path (PyPI trusted publishing, OIDC; no token is stored anywhere). It builds the universal wheel and sdist, installs the wheel outside the checkout and runs tools/check_installed_package.py without a GPU, then uploads to TestPyPI and, for a v* tag, to PyPI — each upload behind a protected environment whose reviewer approves by hand. One-time set-up, by a maintainer:

  1. On pypi.org and test.pypi.org, add a pending trusted publisher for the project gato-mpc: owner A2R-Lab, repository GATO, workflow publish.yml, environment pypi (resp. testpypi). The project is created by its first upload.

  2. In the GitHub repository settings create environments testpypi and pypi, each with a required reviewer; restrict pypi to tags v*.

Per release: the signed receipt must be valid at the release commit; bump version in pyproject.toml and python/gato/__init__.py, date the CHANGELOG entry, merge to main. Rehearse with workflow_dispatch (approve testpypi), then on a GPU box, download the exact candidate with python -m pip download --no-deps --index-url https://test.pypi.org/simple/ "gato-mpc==<version>". Install that downloaded wheel into a clean venv using the normal PyPI index for its dependencies, then run python -I tools/check_installed_package.py --work /tmp/gato-smoke --gpu. Do not use an unpinned multi-index install for this check: it can select a different release instead of the candidate just uploaded. Release: push the tag v<version> (the workflow refuses a tag that does not match pyproject.toml), approve testpypi, re-check, approve pypi; write the GitHub release from CHANGELOG.md and attach SHA256SUMS.txt from the build artifact. Wheels never contain compiled solver modules.

Controller replay before release#

examples/benchmarks/controller_replay.py defaults to correctness-only mode: all five arms at N64, batches 1 and 8, limits off/default ADMM, NumPy/CUDA PyTorch/CUDA JAX. The inputs are a small sinusoidal joint-motion replay with future FK references, not a simulated plant. Full plans, iteration counts and row telemetry must match across array paths; the warm-start error scalar is compared within floating-point reduction tolerance. No task-success or physical-feasibility claim follows from replay parity.

.venv/bin/python examples/benchmarks/controller_replay.py --mode correctness \
  --out examples/benchmarks/night_logs/replay-check/results.jsonl
bash examples/benchmarks/run_controller_replay.sh --dry-run
# Only during an explicitly assigned quiet CPU/GPU window:
GATO_QUIET_WINDOW=1 bash examples/benchmarks/run_controller_replay.sh

The timing launcher verifies the signed receipt and native identities, takes the cooperative /tmp/a2rlab-timing.lock, and writes into a unique run directory. It never builds modules, installs calibration or edits tracked results. Each of three repeats rotates the array-path order, discards 20 warmup calls and samples 200 calls per cell (180 rows total). External host wall through device completion and native solve duration have separate raw samples. Input creation, upload, controller construction/reset and output parity downloads are excluded; this is not the sensor-to-actuator latency or an asynchronous-throughputput test.

The October 20 correctness fence plus replay took about 70 seconds per collection on the lab RTX 5090. Reserve ten minutes for two collections and preflight on that box; other hardware may take longer. Results and boundaries include host tails and do not establish a device-path speedup. A STOP file in the announced run directory stops at the next three-path group. Partial results remain diagnostic; restart into a fresh directory and retain the old one. Do not combine repeats across revisions or report an incomplete inventory as a completed batch. Never kill another agent’s process. Publication requires reviewing the completed inventory, matching input/source hashes, parity, raw samples and recorded box state.

GRiD and GLASS are developed in their own repositories; GATO consumes both through the released grid-rbd wheel, pinned to an exact version in pyproject.toml and python/gato/_grid.py (which also pins the GLASS commit the wheel must bundle — gato._grid.require() refuses any other install, at codegen, configure and test time). The wheel supplies the code generator, URDFParser, the global glass:: headers and grid_collision_geometry.cuh; CMake asks python/gato/_grid.py for the include directories. After a grid-rbd bump run .venv/bin/python tools/regen_grid.py and pytest test/test_codegen.py (the vendored gato/dynamics/<robot>/grid.cuh must regenerate byte-identically — never edit it by hand). A dependency update requires rebuilding the receipt profile (every module embeds a content identity of the dependency headers it was built from) and passing the existing numerical goldens. Previously collected timings remain evidence for their recorded pins, not the new checkout.

Build#

Modules are compiled per (plant, knot_points[, variant]) into python/gato/bsqpN{N}_{plant}[_fc|_eh].so; the batch size is a runtime argument. tools/build.sh configures and builds incrementally in build/:

MODULES="indy7:8,16;go2:16:fc" JOBS=1 ./tools/build.sh   # explicit plant:N[:variant] list
./tools/build.sh --profile receipt                       # exactly test/receipt_modules.txt
./tools/build.sh --variant fc                            # fc (or eh) variants of PLANT x KNOTS
./tools/build.sh --clean                                 # wipe build/ and reconfigure

PLANT, KNOTS, ARCH and JOBS (default 4) are environment overrides. ARCH is the CUDA architecture (default native); set it explicitly, e.g. ARCH=120 on the lab box, whenever native detection could pick another target (worktree builds) — the symptom of a wrong architecture is frozen solves, not a build error. Every translation unit pulls the whole generated grid.cuh, so one go2 compile peaks near 7 GB of RAM; gato.build(...) reconfigures build/ for its own module request, so re-run your usual configure afterwards (gato.register_robot / example 05 build in the per-host cache and leave build/ alone).

On the shared lab box every build or test command runs in a memory-capped scope with one compiler job, and the GPU is left to whoever is timing:

JOBS=1 systemd-run --user --scope -p MemoryMax=36G -p MemorySwapMax=0 --same-dir --quiet \
  ./tools/build.sh --profile receipt
systemd-run --user --scope -p MemoryMax=36G -p MemorySwapMax=0 --same-dir --quiet \
  .venv/bin/python -m pytest test/ -q

Correctness runs may share the GPU; timing may not (below). Do not kill processes you did not start.

Tests#

test/ is the pytest suite; markers select the tier:

pytest -m "not gpu"           # host-only: packaging, math, codegen determinism
pytest -m "gpu and not slow"  # smoke solves, bit-determinism, shapes, controller, constraint gates
pytest                        # + slow: codegen diff for every robot, gato.build dogfood, kernel harnesses

  • test/test_parity_golden.py is the bit-parity gate for kernel changes: for every module in test/receipt_modules.txt it solves fixed problems on the pcg and bdsv paths (plus the constrained cases: limit ADMM + EE row, collision AL/ADMM, masked CONTACT_POS AL/ADMM) and compares xu, merits and iteration counts bitwise against test/golden/*.npz. A golden that moves means something changed, never how much (the solves are deliberately unconverged). Attribute the drift with the referee gates (test_dynamics_fingerprint.py, test_f_ext.py, test_exact_hessian.py, test_rowgroups.py) and compute-sanitizer --tool memcheck on the cheapest module first; only then GATO_GOLDEN_REBASELINE=1 pytest test/test_parity_golden.py -k <case> and say why in the commit message.

  • test/test_kernel_gates.py builds and runs the five standalone harnesses in test/cuda/ with nvcc -DNDEBUG -arch=native (slow). Shared-memory changes also get compute-sanitizer --tool racecheck build/bsqp (the C++ demo, built by default).

  • A missing dependency is a broken environment, never a skip: test/expected_skips.txt is empty and CI fails on any host-tier skip.

Signed receipt#

GPU CI is a signed pytest-gpu-proof receipt (gpu-proof.json at the repo root, schema 3, plugin 0.4.1) produced on the lab GPU and verified CPU-only by the verify-gpu-proof workflow against the signer’s GitHub keys. The receipt attests the full suite on exactly the test/receipt_modules.txt module set; the fingerprint covers native sources and bindings, Python, tests, examples, build tools/configuration, CI and dependency gitlinks (the exact scope is in pyproject.toml [tool.gpu_proof] and test/gpu-proof-policy.yaml). Changes in that scope require a fresh receipt before an authorized push. A missing receipt fails CI; test/expected_tests.txt enforces the complete collected suite.

  1. Commit the source (and any grid-rbd pin bump); the tree must be clean.

  2. Configure and build the receipt profile: ./tools/build.sh --profile receipt (capped, as above). Each module embeds a content identity of its actual native inputs. The signer refuses missing or stale/unidentified binaries; reconfigure after native edits so CMake updates the identity and rebuilds affected modules.

  3. PYTHON=.venv/bin/python ./test/run_gpu_proof.sh on the GPU box. It refuses a dirty tree, test-selection arguments, PYTEST_ADDOPTS, a wrong plugin version and a Python without the [test] deps. It installs nothing, runs the whole suite and writes gpu-proof.json.

  4. Verify locally the way CI does:

    .venv/bin/gpu-proof verify --receipt gpu-proof.json --repo . \
  --policy test/gpu-proof-policy.yaml --expected-skips test/expected_skips.txt --require-gpu

  5. git add gpu-proof.json and commit it on its own (“attest …”), one SHA per receipt. Push only when authorized, then check the workflow run.

Never edit a signed receipt; a dirty-tree local pass is not a receipt.

Documentation site#

The reference tier of the public site (https://a2r-lab.org/GATO/docs/) is built from this repository’s markdown by docs/site/build.py (links rewritten and checked, a receipt page from gpu-proof.json, the API reference from the docstrings) and deployed by .github/workflows/docs.yml on pushes to main; the project page stays on the gh-pages branch (website.md). pip install -e ".[docs]" then python docs/site/build.py builds it locally into docs/site/_build/html; the host-tier test test/test_docs_site.py runs the same build with warnings as errors, so a dangling link or an invalid docstring fails the suite. Edit the markdown in place; docs/site/ holds only the configuration, the landing and the API page.

Timing#

Timing needs an otherwise idle machine and an explicitly declared quiet window; nothing in the test suite or the build measures performance. The coordinator entry point is examples/benchmarks/run_timing_handoff.sh, which refuses to run without GATO_QUIET_WINDOW=1, a clean tree, a committed receipt and the cooperative /tmp/a2rlab-timing.lock:

examples/benchmarks/run_timing_handoff.sh --dry-run                         # prints the plan, touches nothing
GATO_QUIET_WINDOW=1 examples/benchmarks/run_timing_handoff.sh --suite checkpoint
GATO_QUIET_WINDOW=1 examples/benchmarks/run_timing_handoff.sh --suite all   # checkpoint, compile, calibrate

--suite is one of checkpoint (three iiwa14 N64 B1/8/128 repeats plus ten B128 Fig-7 scenarios), seed-ab, boundaries, compile, calibrate or all. The boundaries suite pairs MPCController.step wall time with internal solver time on Indy7 and iiwa14, N64, B1/8/128, three repeats. It excludes sensing, reference preparation, simulation and actuator I/O. all selects checkpoint, compile and calibrate; it does not include boundaries or seed-ab. Logs land in examples/benchmarks/night_logs/handoff_*/ with source, dependency, receipt and module hashes recorded before and after. The narrower run_merge_checkpoint.sh takes the same --dry-run / GATO_QUIET_WINDOW=1 gate. Do not run heavy CPU work or compiles while a timing leg is in flight, and do not quote numbers from a contended box. The paper-figure scripts (examples/paper-figures/) consume the resulting CSVs; see figure protocols.