NeurIPS 2026 Workshop on On-Device Intelligence: Foundation Models under Real-World Constraints (ODI) · Poster · ICC Sydney, 12 December 2026
LG Electronics
TL;DR
An “INT8” label does not name one controller. With the same policy, the same Hailo-8 toolchain path and the same 1,024 paired starts, changing only which recorded states set the quantizer ranges gives 96.6% versus 21.4% terminal success. A compiled artifact should be qualified by paired task outcomes for its exact deployment instance, not by its checkpoint, its precision label or a local error statistic.
96.6% vs 21.4% terminal success of two Hailo-8 builds of the same policy on the same 1,024 starts; only the calibration rows differ
≈30 cm best-to-worst return spread 14.93 [14.32, 15.55] as mean goal distance over the episode (Edge TPU: 0.49)
7/8 independently trained policies in which the same calibration change reproduces the gap; policy-mean contrast 8.60 [4.42, 12.51]
Overview
Problem
Post-training quantization fixes 8-bit numerical ranges from a few calibration states. On the audited Hailo path the compiler used its default of 64 rows and reported the count but not which rows; a CPU-only build host silently reduced optimization to level 0. Neither fact was in the build record.
Approach
A deployment instance is the exact executable bound to its runtime, device, runner and I/O processing. We qualify it with a provenance record from source to decision and with paired complete-task audits: FP32 and the instance run from identical starts, with the physical accelerator in every control step.
Outcome
Eight calibration rules on one policy give eight controllers whose task outcomes differ by up to 96.6% versus 21.4% success on Hailo-8; the effect recurs prospectively in 7/8 independently trained policies and survives compiler levels 1 and 3. Fixed-state error proxies do not predict it.
What the build record leaves out
The audit: vary only the calibration rows
One SAC FetchReach policy, one ONNX export, one toolchain–device path, 1,024 paired reset seeds, and the physical accelerator in every control step: the simulator advances only after the device returns an action.
→
Eight rules fixed before any outcome was read: even/odd episodes, episode head/tail, phase early/middle/late, fixed random.
→
Paired return (the whole route; FP32 mean -0.663) and terminal success (the endpoint) against FP32 from the same starts. 0.5 return ≈ 1 cm mean goal distance.
Results
Same policy, same path, eight controllers
Eight artifacts per path, 1,024 starts each; within-path heterogeneity p < 0.001 (Holm) on both paths. Hailo-8 range 14.93 [14.32, 15.55]; Edge TPU 0.49 [0.46, 0.52]. On Edge TPU the two worst-return rules keep the highest success rates: return and success diverge, so either alone hides part of the behavior. Within-path effects, not a vendor comparison.
It recurs across independently trained policies
Prospective replication: three rules and 256 paired starts for each of 8 FetchReach SAC seeds; the Hailo early-minus-middle contrast was the sole confirmatory test. Middle underperforms early in 7/8 policies; policy-mean contrast 8.60 [4.42, 12.51] (≈17 cm), sign-flip p = 0.0156; pooled success 45.9% vs 99.0%.
Forcing the compiler harder does not remove it
| Hailo DFC build of the eight-rule family | Family range | 95% interval | Status |
|---|---|---|---|
| Level 0 (auto), 64 rows, stride 4 | 14.93 | [14.32, 15.55] | locked |
| Level 1 (forced), 64 rows, stride 4 | 14.57 | [13.92, 15.21] | informed |
| Level 0 (forced), all 256 rows | 14.97 | [14.35, 15.58] | informed |
| Level 3 AdaRound, all 256 rows | 14.42 | [13.83, 15.04] | informed |
Forcing level 1 changes every artifact’s action field, yet the best-to-worst range stays. Neither the level-0 fallback nor AdaRound clamping alone explains the spread; the “informed” rows were designed after the first result.
The record needs row count, identity and order
Supplying the same 256 rows in reverse order changed which 64 rows set the ranges: terminal success 69.9% → 97.1% and return +1.23 [1.07, 1.39] on 512 paired starts, while the full-256 build matched an explicit first-64 control on every action and outcome. The toolchain-reported count identifies none of this.
What the failing artifacts change
| Calibration rule | Hailo output range | FP32 outside (%) | At bound (%) | Hailo success | Edge TPU success |
|---|---|---|---|---|---|
| Episode head | [-1.01, 1.00] | 1.2 | 0.4 | 92.7 | 91.7 |
| Episode tail | [-1.01, 1.00] | 0.0 | 1.2 | 96.4 | 96.6 |
| Even episodes | [-1.01, 1.00] | 0.0 | 1.6 | 96.6 | 96.7 |
| Odd episodes | [-1.01, 1.00] | 1.2 | 1.2 | 94.6 | 91.6 |
| Phase early | [-1.01, 1.00] | 0.0 | 1.6 | 95.7 | 96.4 |
| Phase middle | [-0.66, 0.20] | 24.2 | 21.1 | 21.4 | 97.9 |
| Phase late | [-0.66, 0.20] | 24.2 | 20.7 | 22.0 | 97.9 |
| Fixed random | [-1.01, 1.00] | 0.0 | 1.2 | 92.4 | 96.8 |
Post-outcome inspection on a fixed 800-state early-phase panel. Only the two rules with no early-phase rows calibrate the output (tanh) layer to a restricted range; a quarter of early FP32 outputs then fall outside it and Hailo-8 success collapses, while Edge TPU under the same rules stays near 98%. A contributor, not an isolated cause.
Local proxies do not qualify a deployment instance
5 outcome-blind artifact pairs matched within 10% in mean action-error norm and RMSE still differ by 0.08–0.17 in paired return and by up to 5.0 points in terminal success. A closed-loop amplification diagnostic (CLAE) ranked alerts better than action magnitude in selected windows but missed its pre-specified criterion on fresh data.
What a failing artifact does to the reach
Device-in-the-loop follow-up run on this paper’s pipeline: policy seed 202609201, Hailo-8 level-0 build calibrated on 256 phase middle rows, 128 new paired starts, FP32 arm and device arm from identical reset states. The FP32 policy reaches the goal in 128/128 starts, the artifact in 46/128; mean return -1.06 (FP32) versus -11.07 (device). This run illustrates the mechanism; the numbers elsewhere on this page come from the paper’s own cohorts.
All 128 starts, step by step
Left: gripper position relative to the goal (cm), seen from above; the green disc is the 5 cm success radius and 20 device paths leave the ±40 cm frame. Right: distance to the goal over the 50-step episode. Thin lines are single starts, thick lines medians. Press play to replay the episode; the checkbox keeps only the failing starts.
Qualify by paired outcomes, under the application’s contract
- The task owner fixes the contract before building: a tolerance τ on paired return loss and a success floor ρ.
- Construction fields are recorded at build time; execution, audit and decision fields during qualification.
- FP32 and the instance run from start blocks not used to choose the artifact, with the physical accelerator in every control step.
- With return-loss bounds [LR, UR] and success bounds [LS, US]: pass if UR ≤ τ and LS ≥ ρ; fail if LR > τ or US < ρ; otherwise indeterminate. Families need a disjoint start block or simultaneous bounds before shipping.
- Changes to construction or execution conditions require renewed qualification; contract changes require reassessing the decision against the available evidence.
Record per deployment instance
| Category | Fields |
|---|---|
| Source | policy, ONNX, script hashes |
| Calibration | pool, rule, seed, order hashes; supplied, reported and inferred counts |
| Build | compiler, flags, requested and effective level, host, executable hash |
| Execution | runtime, device, runner, I/O processing, action-dump hash |
| Audit | task version, seeds, horizon, endpoint |
| Decision | tolerance, floor, interval, evidence status |
Each rectangle is the product of the 95% intervals for paired return loss and deployment success of one artifact; the contract’s tolerance and floor cut the plane into pass, fail and indeterminate regions. A retrospective illustration, not a prospective shipping decision.
Takeaways and limits
Takeaways
- Two executables can share checkpoint, path and INT8 label and implement different controllers.
- Calibration rows are provenance: count, identity and order all changed outcomes.
- Fixed-state error matched; task outcomes did not. Audit the complete task, paired, on the device.
Limits
- The study shows that this can happen, not how often: three tasks, eight policy seeds, two accelerators.
- No GPU-host builds, real-time deadlines, physical robots or foundation-model controllers.
- The qualification rule is illustrated retrospectively, not validated end to end.
Provenance
Every number on this page is generated from the committed audit outputs by the scripts that build the poster; the poster manifest lists each source file and its hash. Hailo artifacts were built on a CPU-only host with DFC 3.33.1; Edge TPU artifacts with TFLite full-integer PTQ. Scripts, build configurations and per-run records will be released with the paper.
Citation
@inproceedings{kang2026pairedtaskaudits,
title = {Same Policy, Different Artifacts: Paired Task Audits for On-Device Neural Control},
author = {Kang, Chanseok},
booktitle = {NeurIPS 2026 Workshop on On-Device Intelligence: Foundation Models under Real-World Constraints},
year = {2026},
url = {https://openreview.net/forum?id=gvQyNdXzRc}
}
