ROAMing: Real-Time Occlusion-Aware 4D Surfel Mapping via Neural Scene Flow

David Recasens^*, David Tingdahl^*, Remo Steiner, Vikram Ramasamy, Alexander Millane, Clemens Volk, Tuur Stuyck, Soha Pouya

NVIDIA. Zurich and Santa Clara.

^*Equal contribution

10th Conference on Robot Learning (CoRL 2026), Austin, TX

Paper arXiv (coming soon) Code (coming soon) Dataset (coming soon)

ROAMing overview: posed RGB-D frames pass through neural scene flow and surfel fusion to produce a surfel map; static fusion leaves a stale ghost arm while ROAMing keeps a coherent map with 3D point tracks.

ROAMing: warp before fusion for online dynamic mapping. Left: Neural scene flow propagates the map from posed RGB-D frames before depth fusion. Right: Static fusion leaves stale arm geometry; ROAMing maintains a coherent map. Colored curves show 3D point tracks in an ArenaPickPlace test scene.

88.3%

completion of currently hidden surfaces on moving objects (DynaGSLAM: 60.7%)

56 ms

per frame for tracking + fusion in the fast configuration (+14 ms RAFT flow)

0 masks

no object segmentation, no future frames: updates from frames t−1 and t only

38 scenes

ArenaPickPlace: ray-traced robot interactions with complete surface trajectories

Abstract

Robots need maps that remain consistent through motion and occlusion. We present ROAMing, a causal RGB-D mapper that maintains an adaptive surfel map without object segmentation. A local-relative transformer refines lifted optical flow and propagates motion to non-visible neighbors; depth fusion adds newly observed surfaces and removes stale geometry. We introduce ArenaPickPlace, a synthetic robot-interaction dataset with complete surface trajectories. Our fast configuration achieves 88.3% completion of currently hidden surfaces on objects that move during the sequence. Evaluation includes zero-shot Sim4D reconstruction and real-world PW-DROID tracking. ROAMing trades some tracking accuracy relative to non-causal trackers for immediate updates. With precomputed optical flow, fast tracking and fusion take 56 ms per ArenaPickPlace frame.

Method

At each time step ROAMing receives a posed RGB-D frame and updates its surfel map in two stages. It first warps every existing surfel, visible or hidden, with a predicted 3D motion, and only then fuses the new depth image. Because the map is moved before association, moving objects do not corrupt data association, and surfaces that pass behind an occluder keep travelling with their neighbours instead of being frozen or discarded.

Motion refinement module: source points, visibility mask and visible coarse 3D flow are embedded into tokens, then refined through source attention, a warp by the current flow, warped attention, a fusion MLP and a motion head, looped three times, to produce a refined 3D flow.

Motion refinement before fusion. Source attention uses a fixed neighborhood graph; warped attention recomputes neighbors as the motion estimate changes. Three refinement iterations predict a residual for every surfel, including those with no visible correspondence.

Geometric prior. Forward optical flow between frames t−1 and t (RAFT in the fast configuration, WAFT in the accurate one) is lifted to 3D with depth and the camera poses. This supplies motion wherever a valid correspondence exists; hidden surfels and rejected correspondences receive a zero prior, so the prior alone can never move hidden geometry.

Visibility-aware refinement. A local-relative transformer, in the spirit of Point Transformer and RAFT’s iterative refinement, predicts a residual displacement for every surfel. Tokens encode the prior, its magnitude, visibility and k-nearest-neighbour distance; attention encodes relative offsets rather than absolute positions. Each of three iterations attends over a fixed source graph (k = 16), warps the centres by the current estimate, rebuilds a warped graph (k = 32) and attends again. A hidden surfel remains a token even with a zero prior, so motion propagates through partially observed local structure without any segmentation. Motion below 2 mm is zeroed to suppress drift on static geometry.

Surfel fusion. Established projective surfel fusion is applied to the propagated map: an information filter combines depth-consistent associations, uncovered measurements spawn new surfels, free-space contradictions prune stale ones, and occluded or out-of-frustum surfels persist.

ArenaPickPlace dataset

Existing 4D reconstruction datasets do not label ground-truth surface points per time step as static or dynamic and visible or hidden, and 3D tracking benchmarks only provide trajectories for pixel-aligned points visible in the first frame. ArenaPickPlace fills that gap: 38 tabletop manipulation scenes, each with 10 episode initializations and 10 cameras, captured as 200-frame RGB-D sequences rendered with ray tracing in Isaac Lab Arena. Every sequence ships with depth, optical flow, meshes, 6-DoF poses, and complete surface point and normal trajectories labelled static/dynamic and visible/hidden, including points that become occluded or leave the camera view. We train on all episodes and views of 17 scenes and test on the first episode and camera of the remaining scenes.

Rows of ArenaPickPlace samples showing RGB, depth, optical flow, visibility labels and motion labels, with novel-view renderings that reveal surfaces hidden from the input camera.

ArenaPickPlace sequences and annotations. Columns show RGB, depth, optical flow, visibility, and motion labels. The final two columns use novel views to reveal hidden surfaces and distinguish the robot from other moving objects.

Dynamic reconstruction

We compare with 4DTAM and DynaGSLAM. Accuracy (map → ground truth) and completion (ground truth → map) count the fraction of points within 3 cm of the other set, scoring only ever-observed geometry. Dynamic labels objects that move at any scored frame; hidden includes occluded, back-facing and out-of-view points. The methods differ most on what they do with geometry that stops being observed, so the dynamic-hidden column is where ROAMing’s warp-before-fuse design shows: with 31k versus 30k primitives, fast ROAMing improves dynamic-hidden completion over DynaGSLAM from 60.7% to 88.3%, and tracks six times more dynamic-hidden primitives (6.1% vs 1.0% of the map).

Method	Total	Static		Dynamic
Method	Total	Visible	Hidden	Visible	Hidden
DynaGSLAM (30k pts)	96.4 / 60.7	83.1 / 56.8	80.7 / 57.5	98.8 / 99.9	86.9 / 60.7
4DTAM (12k pts)	81.4 / 82.9	89.6 / 86.6	75.6 / 77.9	74.7 / 66.2	56.1 / 55.7
ROAMing, accurate (102k pts)	97.4 / 99.1	97.5 / 99.6	95.5 / 98.9	99.4 / 100.0	89.7 / 90.2
ROAMing, fast (31k pts)	96.9 / 98.7	97.6 / 99.3	95.5 / 98.9	98.1 / 100.0	86.0 / 88.3

Accuracy / completion at 3 cm (%, higher is better) on ArenaPickPlace. DynaGSLAM uses oracle masks. Mean map size per frame in parentheses.

Novel-view renderings of a tabletop scene: input RGB, then DynaGSLAM, 4DTAM and ROAMing reconstructions with points colored by nearest-surface error from 0 to 5 cm.

Dynamic-hidden reconstruction on ArenaPickPlace from a novel view. Target points show nearest-surface error (0–5 cm); geometry visible to the input camera supplies ground-truth RGB context.

Two more ArenaPickPlace scenes, Black items in bin and Clutter pumpkin, comparing DynaGSLAM, 4DTAM and ROAMing on hidden geometry.

Additional dynamic-hidden examples. The novel views show how much previously observed moving geometry each method retains once it is hidden from the input camera; colour encodes residual reconstruction error.

Zero-shot transfer to Sim4D

Sim4D contains animated non-rigid meshes rendered over randomized textures, well outside the training distribution in geometry, motion and appearance. Applied zero-shot and measured by held-out-view depth L1 where both rendered and ground-truth depth are valid, accurate ROAMing and 4DTAM have similar mean error (1.77 vs 1.79 cm) across eight sequences; the fast configuration reaches 2.08 cm, and DynaGSLAM 7.59 cm.

Sim4D sequences Curtain, Mercedes and Wave toy: RGB, ground-truth depth, then absolute depth-error maps for DynaGSLAM, 4DTAM and ROAMing on a 0 to 5 cm scale.

Sim4D qualitative comparison at held-out viewpoints. After RGB and ground-truth depth, the method columns show absolute depth error on a shared 0–5 cm scale (blue is lower).

Causal 3D tracking

The tracking evaluation follows first-frame query identities through the sequence, without fusion, spawning or pruning. Track4World processes the whole video and uses ground-truth scale alignment; TAPIP3D uses 16-frame windows with stride 8, so it sees the future. ROAMing updates from frames t−1 and t only and trades a few millimetres of error for immediate, causal estimates. On the real-world PW-DROID clips we restore the robot-arm annotations that the released pseudo ground truth omits.

Fixed-query EPE3D (cm, ↓)	ArenaPickPlace	PW-DROID
Track4World (whole video, scaled)	13.61	12.21
TAPIP3D (16-frame window)	0.53	0.35
ROAMing, accurate (causal)	0.99	0.54
ROAMing, fast (causal)	1.56	0.72

Runtime	Arena ms / pair	Arena GiB	DROID ms / pair	DROID GiB
Track4World	661^†	35.6^†	146	26.3
TAPIP3D	11,594^†	42.6^†	2,924	11.1
4DTAM	228	8.6	—	—
DynaGSLAM	149	1.1	—	—
ROAMing, accurate	186	4.8	254	10.4
ROAMing, fast	56	3.1	58	1.9

RTX 5090; ROAMing numbers include tracking and fusion. Optical flow adds 14 / 10 ms (RAFT, fast) and 312 / 60 ms (WAFT, accurate) on Arena / PW-DROID. ^† L40 GPU for methods exceeding the 5090’s 31.4 GiB. Clip inference is amortized.

Nine panels: first and last RGB frames with a bottle that is initially occluded, the last frame in 3D, then 3D point tracks for ground truth, Track4World, TAPIP3D, ROAMing geometric prior only, ROAMing with refinement, and ROAMing with fusion.

Persistent 3D tracking through occlusion. (a)–(b) First and last RGB frames of an ArenaPickPlace test clip; the zoomed regions show the initially occluded bottle. (c) Back-projected frame 97 highlights the disoccluded bottle. (d)–(i) 3D tracks: ground truth, two fixed-query baselines, and our geometric, refined and full tracking-and-fusion variants. Only the full mapper adds tracks on the newly revealed bottle.

What each component contributes

Three renderings of the same scene with 3D point tracks: geometric prior only leaves hidden points behind, learned refinement carries them along, and fusion adds tracks on the disoccluded bottle.

Component behavior. Geometric flow leaves hidden points behind (a); refinement propagates them (b); fusion adds tracks on the disoccluded bottle (c).

The geometric prior can only move surfels that are visible in both frames, so points that pass behind the arm are left frozen in place. The refiner gives every surfel a token and lets hidden ones borrow motion from their visible neighbours, which carries them along the arm’s trajectory. Fusion then corrects accumulated drift against new depth, removes surfels that free-space observations contradict, and begins tracking surfaces the moment they are revealed. Trackers that follow only first-frame queries never pick up the bottle at all.

Limitations

Reconstruction is evaluated only in simulation; training and the main benchmark use fixed cameras, leaving camera motion untested. The system depends on external poses and optical flow, and cannot infer arbitrary motion of fully hidden components disconnected from visible local structure. Real-world PW-DROID results evaluate 12-frame tracking clips, not the long-term accuracy and completeness of persistent maps.

BibTeX

@inproceedings{recasens2026roaming,
  title     = {ROAMing: Real-Time Occlusion-Aware 4D Surfel Mapping via Neural Scene Flow},
  author    = {Recasens, David and Tingdahl, David and Steiner, Remo and Ramasamy, Vikram and Millane, Alexander and Volk, Clemens and Stuyck, Tuur and Pouya, Soha},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}