Problem Statement

Let $s_t:=(p_t,v_t)$, $a_t$, and $o_t$ denote a robot’s state, action, and observation, respectively. We consider stochastic systems evolving according to the unknown dynamics $s_{t+1}\sim f(s_t,a_t)$. The available approximate dynamics model $\tilde f$ has arbitrary fidelity, with uncertainty arising from both external disturbances and model mismatch. In our experiments, $\tilde f$ is a linear-Gaussian model.

Given access to an exchangeable dataset of robot transitions $D_{\mathrm{cal}}:=\{(s_t,a_t,o_t,s_{t+1})_i\}_{i=1}^{n}$, collected in environments that are different from but visually similar to the deployment environment, our goal is to construct a state-action-observation-dependent and volume-efficient prediction region $\hat{\mathcal C}\subseteq \mathcal S$ that is guaranteed to contain the unknown future system state $s_{t+1}$ with at least a user-selected likelihood $1-\alpha \in(0,1)$, i.e., satisfy

$$\mathbb P\!\left(s_{t+1} \in \hat{\mathcal C}\right) \ge 1-\alpha.$$

OCULAR calibrates the approximate dynamics model $\tilde f$ to construct the prediction set $\hat{\mathcal C}$ without requiring data from the test-time environment.

Method: OCULAR

OCULAR performs local conformal calibration using robot-frame perception, body velocity, and action information. Given raw calibration samples $X_i^{\mathrm{raw}}:=\left(s_t,a_t,o_t\right)_i$, it maps each observation to a planar semantic footprint $\mathrm{Project}(o_t)$ and encodes that footprint with a CAE. The processing step constructs $X_i:=\mathrm{Process}(X_i^{\mathrm{raw}})=\left(v_t,a_t,\mathrm{Encode}(\mathrm{Project}(o_t))\right)_i$, and a Regression Decision Tree partitions this processed input space using fitted scores. SplitCP is then performed per Decision Tree leaf, resulting in an input-dependent score threshold $\hat q_k\in \mathbb R$, which acts as a local probabilistic upper bound on motion uncertainty. The calibration data is split into two disjoint subsets $D_{\mathrm{cal}} = D_{\mathrm{cal}}^{\mathrm{part}} \sqcup D_{\mathrm{cal}}^{\mathrm{CP}}$.

Experiments

We validate OCULAR on a double-integrator in Isaac Sim using a floating camera providing depth and semantic segmentation images. We evaluate across three snowy T-junction environments. The white lower-friction regions are out-of-distribution (OOD) relative to the approximate linear-Gaussian model $\tilde{f}$, which captures the robot dynamics over the asphalt road (ID). In all experiments, OCULAR uses no data from the test map (e.g., for icyMain evaluation, we use system transition data collected in icyMiddle and icySide).

Using Monte Carlo propagation, we estimate the likelihood of the prediction regions $\hat{\mathcal C}\subseteq \mathcal S$ containing the future unknown state $s_{t+1}$. This containment likelihood (i.e., coverage) is reported separately for the nominal road regions (ID relative to $\tilde f$) and the icy regions (OOD relative to $\tilde f$). Prediction region volume-efficiency is reported as a ratio relative to a linear-Gaussian oracle with access to the unknown ground-truth dynamics.

Test-case results across three Isaac Sim roads.

red means coverage below 90%. Prediction region volume ratio is reported relative to an oracle using the minimum scaling factor needed to achieve 90% coverage. Each map has 4464 test transitions.

OCULAR calibrates the approximate dynamics on both nominal and icy conditions without sacrificing volume efficiency relative to baselines using environment-specific data.

We also conduct motion planning experiments to demonstrate the utility of our method for probabilistically safe MPC under model mismatch and external perturbations. For all methods the objective function includes a distance-to-subgoal term, a collision penalty, and a penalty for transitions with high estimated uncertainty.

We observe that using the uncalibrated dynamics directly leads to gaining significant momentum over ice and hence collisions. SplitCP can produce overconfident (i.e., collisions) or overconservative (i.e., timeouts) behavior depending on the calibration data distribution. LUCCa is a baseline requiring robot location data in an inertial frame and hence needs transitions collected in the test environment. OCULAR moves more slowly over ice and faster over the nominal road, being both safe and efficient. By using perception information, it does not require data from the test environment. Below we visualize rollouts from all compared methods at once.

These results indicate that OCULAR can generalize to new unseen environments and achieve both adequate planning performance and safety by leveraging perception information obtained in other environments. Below are numerical results.

Planning results across three Isaac Sim roads (30 trials each).

Success = reaching all subgoals without collisions.

BibTeX (cite this!)

@misc{marques2026localconformalcalibrationdynamics,
      title={Local Conformal Calibration of Dynamics Uncertainty from Semantic Images},
      author={Luís Marques and Dmitry Berenson},
      year={2026},
      eprint={2605.13028},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2605.13028},
}