Chan’s Research Note
    • 한국어
  • Home
  • About
  • Blog
  • Publications
  • Activities
    QuantGate: A Deployment Review for Quantized Robot Policies

    Nebius × NVIDIA Global AI Hackathon · Physical AI track · Submitted 30 October 2026

    Chanseok Kang

    Personal project

    Code (Apache 2.0) Video (3 min) Dashboard Datasets (Hugging Face) Devpost write-up

    TL;DR

    One robot policy can ship as many different builds. QuantGate compares the FP32, FP16 and 24 INT8 builds of one ACT checkpoint by cost measured on the Jetson and behavior measured in 745 sealed real-arm trials, then says which build to validate next and whether any may be deployed. In this study FP16 was the cheapest build, the INT8 offline ranking did not predict the arm (73/75 vs 73/75), and no build is approved yet: the evidence is not strong enough, and the review names the test that could change that.

    Same robot, same task: one run succeeds, one fails. Move the can 3 cm and it misses (overhead camera, 4×).

    Same robot, same task: one run succeeds, one fails. Move the can 3 cm and it misses (overhead camera, 4×).

    An SO-101 arm, driven by a LeRobot ACT policy on a Jetson Orin Nano Super, picks up a can and stands it on a coaster. Benchmarks and open-loop metrics looked fine for every build; neither said whether the arm would still put the can on the coaster, or how sure we could be.

    11.6 ms · 82 mJ FP16 per inference on the Jetson (trtexec, 15 W), faster and cheaper than every INT8 build (16.8–17.1 ms, 101 mJ); FP32 53.4 ms, 403 mJ

    73/75 vs 73/75 real-arm successes of the offline audit’s top-ranked INT8 builds and of randomly picked ones

    745 real-arm trials, all under sealed, interleaved schedules; logs hashed before unsealing, primary analyses pre-registered

    Overview

    Problem

    Shipping a policy to an edge device means choosing a build: FP32, FP16, or one of many INT8 calibrations. Each is a different artifact. Speed benchmarks and offline scores are cheap, but they do not say whether the arm still completes the task, and small real-arm samples are easy to overread.

    Approach

    A deployment review per build: cost measured on the target device, an offline audit frozen before any real trial, and sealed, interleaved real-arm trials under rules fixed in advance. Code writes every number and verdict (GO, HOLD or BLOCK); AI models judge frames and explain, but never decide.

    Outcome

    FP16 is the next build to validate. No precision effect was shown; deployment conditions (grip, start position, a knocked camera) moved success instead. A position weakness was fixed with 100 new demonstrations and two fine-tunes, each re-tested sealed at conditions never in training.

    The video

    Three minutes, about 101 s of it on the real arm: what the review tells a developer, how it was built, and the failures it found.

    How the review works

    Measure Cost on the device

    26 TensorRT engines of one ACT checkpoint (51.6M parameters), built on the Jetson with a frozen timing cache and timed with trtexec and tegrastats at 15 W.

    →

    Screen Offline audit, frozen

    2,000 held-out states scored per INT8 cell on Nebius Jobs (CPU fan-out). The ranking was frozen before any real trial, so the arm could test it.

    →

    Test Sealed real-arm trials

    Interleaved schedules; the operator saw trial numbers only. Outcomes from an automatic judge and a blind re-judge of the saved video by code; logs hashed before unsealing.

    →

    Decide Code writes the verdict

    Non-inferiority against FP32 with Newcombe intervals, a placement tolerance and an on-device preflight. GO, HOLD or BLOCK, and the next test’s trials per arm.

    The system: the bench (SO-101 arms, cameras, the Jetson with its TensorRT engine, trial runner and preflight), Nebius AI Cloud (training, the audit fan-out, Cosmos-Reason2 and Nemotron), the repository where code decides, and what is public.

    The system: the bench (SO-101 arms, cameras, the Jetson with its TensorRT engine, trial runner and preflight), Nebius AI Cloud (training, the audit fan-out, Cosmos-Reason2 and Nemotron), the repository where code decides, and what is public.

    Which model does what

    Part Model or tool Its job What it is not allowed to do
    Robot policy ACT (51.6M), trained on Nebius Jobs (L40S) picks up the can —
    Candidates TensorRT FP32 / FP16 / INT8 engines the builds under review —
    Judge Cosmos-Reason2-2B on a Nebius AI Endpoint says whether the can ended standing on the coaster decide a verdict on its own; validated on held-out recordings first
    Writer Nemotron 3 Super on Nebius Token Factory explains the review in words write any number, verdict or approval word (code rejects them)
    Verdicts plain code every number, interval and GO / HOLD / BLOCK —

    Results

    Cost is not approval

    Real-arm success (Wilson 95%) against GPU time per chunk on the Jetson. Cheaper is to the left; the intervals overlap.

    Real-arm success (Wilson 95%) against GPU time per chunk on the Jetson. Cheaper is to the left; the intervals overlap.

    FP16 is the fastest and the cheapest build on this device, not INT8, a result only on-device measurement showed. Its success against FP32 is still undecided: with 25 trials per arm the success interval is about ±28 pp wide, so every comparison is INSUFFICIENT_EVIDENCE and deployment stays HOLD. The pre-registered ceiling rule keeps two arms that are both at or above 95% undecided at any sample size; under a revised, pre-registered rule, about 120 trials per arm give 80% power for the success criterion.

    The offline audit did not predict the arm

    The audit’s top-ranked INT8 cells went 73/75 on the arm; randomly picked cells also went 73/75. The offline rank did not track placement error either (Spearman 0.02–0.08). A deliberately broken, black-frame-calibrated build scored badly offline yet went 11/12 on the arm; an input check catches it before the arm.

    Deployment conditions moved success, not precision

    No precision effect was shown. Removing the gripper’s anti-slip pads dropped success to 8/24; moving the start 3 cm gave 0/3. INT8 seemed to make the gripper chatter twice as much; a pre-registered test found 1.3×, not significant (p = 0.12). Grip, start position, can size and camera mounting are recorded as deployment conditions, so a failure is not blamed on the build by default.

    Found, fixed with data, re-tested sealed

    Position fine-tune against the old policy, interleaved, 12 vs 12 per held-out mark (24 vs 24 at the training mark), Wilson 95%.

    Position fine-tune against the old policy, interleaved, 12 vs 12 per held-out mark (24 vs 24 at the training mark), Wilson 95%.

    50 new demonstrations at −4…+4 cm and a 25-minute fine-tune on Nebius Jobs, compared under a rule fixed in advance at positions never in training: 12/12 vs 3/12 at +3 cm (+75 pp, 95% CI [+38, +91]) and 10/12 vs 2/12 at −3 cm. At the training mark it is still undecided (22/24 vs 23/24, with a pre-registered replication), so the review does not approve it there.

    How far the fix reaches, and the second fix

    Where the fine-tune works on the table: success out of 12 per start mark for the fine-tuned and old policies, and fine-tune v2 at +6 cm.

    Where the fine-tune works on the table: success out of 12 per start mark for the fine-tuned and old policies, and fine-tune v2 at +6 cm.

    A new pre-registration mapped the reach of the first fine-tune: 8/12 vs 0/12 at −6 cm, beyond its demonstrations, but 0/12 vs 0/12 at +6 cm and 0/12 vs 3/12 with a smaller yellow can at +3 cm. Fifty more demonstrations (the yellow can, starts at +5 and +7 cm) and a second fine-tune closed both gaps: 12/12 vs 7/12 with the yellow can and 12/12 vs 2/12 at +6 cm, again at conditions never in training, with no drop at the trained spot (12/12 vs 12/12).

    The AI judge is a part to validate

    With a one-word prompt Cosmos-Reason2 caught 1 of 21 failures. A validated prompt caught 20/21 on held-out recordings, then held up live on 96 new trials: 22/24 failures caught, 2/72 false alarms. It still misses some: a can left standing at the start looks like a success to it. A 2 Hz shadow supervisor did not detect failures and is reported as such.

    The model writes words, code writes facts

    The Nemotron report went through three versions. In the last, code writes every verdict, number and status; the model writes only why the review looks as it does and the open questions, and a validator rejects any digit, verdict label, approval word or known misreading. In a six-model comparison on Token Factory, Nemotron 3 Super was the only variant with zero unmatched numbers and a complete format in every repeat.

    The approval rule

    1. Rules are fixed before outcomes are read: pre-registrations and post-lock hashes of the sealed logs (LOCK-1 to LOCK-4); every amendment and error is logged.
    2. Each candidate is compared with FP32 from interleaved trials in the same session.
    3. The success criterion is met when the Newcombe 95% lower bound of \(p_{\text{cand}} - p_{\text{FP32}}\) is above −10 pp.
    4. A final GO also needs placement error within its 5 px tolerance, the ceiling rule not holding the comparison undecided, and the on-device preflight passing (engine hash, normalization manifest, the runner’s 200 ms chunk deadline).
    5. Otherwise the review says HOLD (not enough evidence) or BLOCK (a check failed), and states how many trials per arm the next test needs. Passing the gate means “worth validating next”, not “approved”.

    What the review tells a developer

    Question Answer in this study
    Which build do I validate first? FP16: 11.6 ms, 82 mJ per inference
    Which do I distrust? the black-frame calibration (input check); any build that fails the preflight
    Can I approve deployment? Not yet: INSUFFICIENT_EVIDENCE, HOLD
    What would settle it? a revised, pre-registered ceiling rule and about 120 trials per arm
    Bring your own results quantgate.py judge your_results.csv applies the same rules

    Failures we publish

    A knocked wrist camera

    Success at the trained spot suddenly fell. Every result stayed sealed while we ruled out the software, found stray markers and a wrist camera knocked off-axis, and fixed the mount. A pre-registered health check then read back to nominal. Six trials run in the bad scene were discarded unopened.

    The scene mattered

    In the fine-tune v2 session the first fine-tune also did better than before (7/12 and 2/12 against 0/12), which fits the knocked camera having caused part of the earlier gap. A comparison across sessions cannot settle it, so the page does not claim it.

    The simulator cannot predict the arm yet

    An Isaac Sim twin of the task agreed with 10 of 14 real runs only because it predicts failure every time. It stays exploratory until it is calibrated against the arm.

    Takeaways and limits

    Takeaways

    • Measure cost on the target device; the precision name does not tell you the cost.
    • An offline ranking is a hypothesis for the arm to test, not a substitute for it.
    • “No failure observed” is not safety evidence. A review should say how far its evidence goes.
    • An AI judge or writer is a component to validate on held-out data, not an oracle.

    Limits

    • One arm, one task, one base policy; 12–25 trials per arm, so intervals are wide.
    • Operator blinding was partial in the fine-tune blocks, where the two policies move visibly differently.
    • Sessions were not compared under control, and the review has not yet been tried by outside teams.

    Provenance

    Numbers are frozen at LOCK-4 (23 October 2026) and come from committed artifacts; quantgate.py verify | review rebuilds the review with no GPU, robot or API key. Training data are public as three LeRobot v3 datasets (120 + 50 + 50 demonstrations, CC BY 4.0). The serverless Nemotron model used for the report text was retired by Token Factory on 9 November 2026; the committed reports are unchanged. Code was written with Claude Code; the study design, the bench and every trial are mine.

    Citation

    @misc{kang2026quantgate,
      title        = {QuantGate: A Deployment Review for Quantized Robot Policies},
      author       = {Kang, Chanseok},
      howpublished = {Nebius $\times$ NVIDIA Global AI Hackathon, Physical AI track},
      year         = {2026},
      url          = {https://github.com/goodboychan/nebius-physical-ai}
    }