Results · final board, 2026-10-30
216 runs of record. Losses included.
72 routes, three live runs each at the sensor's real rate: 216 valid runs of record, scored on what xclean published live: latency over every frame, accuracy (ATE) at keyframe level on its map stream (LIVE-MAP). The rules were written down before the runs; the amendments made during the board are listed below.
Disclosures
Read this first
- Most of our pre-registered claims did not pass. On the 38 unseen routes, 1 of 12 claims passed. The full list.
- The engine was frozen with a failed kill test.
- Runs of record, not every attempt.
- A first launch was set aside.
- The IMU was often not in use. In the final board it stayed in use to the end on only 50 of 216 runs (mostly EuRoC and 4Seasons); on MADMAX, Rosario and our own OAK-D-Lite recordings xclean's consistency check dropped or refused it, and KITTI has no IMU. Per dataset.
- The protocol was amended during the board. No accuracy number of any board run was read before an amendment, but validity outcomes (void and incomplete runs, feeder timing) were read and drove 8.36 and 8.40-8.44. They are listed below.
- Robustness is not perfect. No run of record was lost on the 38 held-out routes, but on 2 of them a replay attempt crashed or hung, and the unsupported-frames check passed on 9, failed on 4 and is pending on 25 (all MADMAX, whose ground truth has known faults).
- Some wins are against a failed baseline. How forfeits are counted.
- Our runs were live, on a loaded desktop. The other systems ran in other configurations, named beside each comparison.
Latency
Host-side, from the arrival of a frame to xclean's loop-corrected pose for that frame, on one desktop (AMD Ryzen 9 9950X, RTX 5090, Windows 11).
The pre-registered real-time bars, which also judge dropped frames and frame gaps per route, passed on 28 of 38 unseen routes. Per-route latency is in the table.
Robustness
Accuracy: we publish our losses
Trajectory error (ATE, metres) per route. Against offline ORB-SLAM3 on the 38 unseen routes: 10 wins, 2 ties, 26 losses, every route in the table below. We do not claim to be more accurate than either system.
- xclean
- live, causal, paced at the sensor's rate, on the desktop with its normal background load; 3 runs per route.
- ORB-SLAM3
- V1.0 (UZ-SLAMLab/ORB_SLAM3, commit 4452a3c), built on Windows with portability-only patches (headless, MSVC); tracking, mapping, loop-closing and optimiser code unmodified. Offline, untuned: one generic configuration per sensor type; 5 runs per route; its end-of-run trajectory.
- OKVIS2-X
- ethz-mrl/OKVIS2-X, commit 38043e4, built on Windows with Windows-port patches (build system, integer width, compiler shims); per our build notes, no patch changes its estimator, front end or dataset logic. Causal, unpaced; 1 run per route.
vs ORB-SLAM3
vs OKVIS2-X
A ratio band: within 5 % of theirs is a tie. The OKVIS2-X comparison uses frame-level ATE (each pose held until the next frame), as pre-registered; the table's xclean ATE is keyframe-level.
vs ORB-SLAM3
Includes the routes we designed on.
vs OKVIS2-X
Includes the routes we designed on. Frame-level ATE, as pre-registered.
ORB-SLAM3 verdict rule (the board's own): keyframe-level ATE on the time window both systems cover, after alignment; our 3 runs against its 5. If the two ranges overlap, the route is a tie. The headline is the 38 unseen routes; the full-board tallies are reported, never claimed.
Lost runs and forfeits (the board's rule, applied to both sides). A run is lost as defined below for xclean; an ORB-SLAM3 run is lost when its final map is split into more than one map. A lost run leaves its system's ATE range. A system that lost most of its runs forfeits the route to one that did not. A diverged run (a wildly wrong scale) is not "lost": its ATE on the common window decides.
IMU use
xclean uses the IMU while a built-in consistency check (W1) accepts it, and otherwise continues as stereo visual odometry with loop closure. Over the 216 valid runs of record:
| Dataset | IMU in use to the end | Dropped during the run | Refused at start | No IMU |
|---|
Source: data/board.json on this site (runs[].imu of the final board).
Both baselines are visual-inertial systems. ORB-SLAM3 ran stereo-inertial where the dataset has an IMU, except on rosario_01, 02, 03 and 06, where its IMU start-up crashed on every run and it ran stereo-only, and on 1 of its 5 madmax_B0 runs; on KITTI, which has no IMU, it ran stereo. OKVIS2-X ran visual-inertial, with the IMU off on KITTI. Our accuracy comparisons are therefore mostly against visual-inertial configurations, while xclean ran without its IMU on most runs.
Pre-registered claims
Before the final runs we wrote down what we would claim, and the rule for each, in a pre-registration document (its fingerprint below). A claim passes only if every route of the table is counted and passes. On the 38 unseen routes:
| Claim | Result | Measured |
|---|
The pre-registration, and its amendments
The pre-registration document is published with the code release.
Every route
| Route · set | Dataset | xclean ATE, m | ORB-SLAM3 ATE, m | vs ORB-SLAM3 | vs OKVIS2-X | Latency, ms | Lost | Flags |
|---|---|---|---|---|---|---|---|---|
| Loading the board… | ||||||||
ATE: median of the runs; below it the range (lowest – highest) when the runs differ, and the number of runs used. OKVIS2-X ratio: our frame-level ATE divided by OKVIS2-X's (both shown under the ratio); above 1.05 means OKVIS2-X was more accurate. The OKVIS2-X comparison uses frame-level ATE (each pose held until the next frame), as pre-registered; the table's xclean ATE is keyframe-level. Latency: median of the runs' p50, with their p95 below. Flags: crash/hang a replay attempt crashed or hung; unsupported failed the unsupported-frames check; pending that check awaits a decision because the route's ground truth has known faults.
Definitions
- Seen routes
- The other 29 routes with ground truth: design routes, used while building xclean, and routes seen once by one of the earlier pre-registered kill tests (K12-e, the IMU-safety test; K13, the loop-closure test) or by the last one (K14), each run once before the board.
- Unseen (held-out) route
- A route held out from xclean's development: 22 "unsealed" and 16 "sealed" (held out under an earlier charter). None was used to build xclean; 10 of the 22 unsealed routes were run in the set-aside first launch (see Disclosures), whose results were never read. Our previous system had run on 36 of the 38; claim C5 compares against it. About these routes we looked only at file metadata before the final runs (paths, frame and IMU counts, durations, flag codes); no ground-truth trajectory and no xclean result was read. Two accidental exposures to other systems' numbers are disclosed in the pre-registration: the paper 1 v3 board record (another system's and ORB-SLAM3's ATE ranges) of the sealed route 4s_20201222_115, and an older system's usability values on v2_01_easy; no rule depends on them. Seen routes were used while building xclean or by earlier kill tests.
- Lost run
- A run in which more than 10 % of the frames have no published map-stream pose, or a state other than tracking, relocalised or initialising, or are degraded; or which ends in an unrecovered loss (pre-registration 2.6). The board's accuracy rule uses the same definition through its export: such a run is marked as more than 10 % coasted.
- Continuity
- xclean publishes two streams: the odometry stream (LIVE-ODOM), which loop closure never corrects, and the map stream (LIVE-MAP), which carries the loop corrections. Claim C2 passes on a route when, in every run, every jump of the map stream is flagged or also appears in the odometry stream; it passed on 37 of 38 unseen routes. The odometry stream itself had no jump at all in every run on 6 of the 38.
- Unsupported frames
- Frames with no published pose, a state other than tracking, or a two-second motion that disagrees with ground truth, as a share of the run. The bar is 2 % per route.
- ATE
- Absolute trajectory error in metres, at keyframe level on xclean's live map stream (LIVE-MAP at the keyframe stamps), on the time window both systems cover, after rigid alignment to ground truth. This is the ATE in the table and in the ORB-SLAM3 comparison. The OKVIS2-X comparison (claim C6) uses frame-level ATE instead, as pre-registered: each system's per-frame poses, each held until the next frame, scored by the same live scorer.
- Latency
- Recorder receipt of the loop-corrected pose minus the release of its frame, over every frame; a dropped frame counts as infinitely late.
Source
Every number on this page is read from the final board file.