First in both evaluated outputs · View results

Geometric and Semantic Couplingfor Interaction Understanding in 3D Scenes

A door and its handle tell a shared story. We use their physical relationship to understand what moves, how it moves, and where to interact.

Hanyang Kong¹ Xingyi Yang^2,✉

¹ National University of Singapore² The Hong Kong Polytechnic University

Corresponding author: Xingyi Yang

Paper Code Hugging Face

Watch the 73-second overview

One scene. Connected predictions.Expand figure

Handles guide motion. Parts refine handles. Two directed uses of the same physical relationship, each applied once.

Handles → part motion

+27.25pp

Handle-guided hinge placement

Parts → extra handles

+5.01pp

Complementary handle proposals

Part classes → handle labels

+0.98pp

Part-only motion-class correction

Articulate3D validation · percentage-point changes in motion AP or handle AP, as labelled. Different mechanisms, not additive gains.

The idea in 73 seconds

See the connections
in motion.

Start with a closed door. See how a handle reveals its hinge, and how part context helps find and interpret handles.

Method illustrations, validation results, and selected predictions. Animated motion visualizes estimates, not observed physical movement. Captions are included in the video.

Download video

The task

Beyond recognizing
the cabinet.

A static 3D scan should tell us how a scene can be interacted with—not only what objects it contains.

01
What can move?
Locate individual movable parts, such as doors and drawers.
02
How does it move?
Infer rotation or translation, its axis, and the hinge location for a rotating part.
03
Where do we interact?
Find the small operating regions we call handles, and associate their motion class.

The difficulty: broad surfaces and tiny handles need different prediction support. A closed door also hides its hinge side; a handle alone does not reveal whether its part rotates or slides.

How it works

Learn the pieces.
Connect the evidence.

Three independent predictors see the same RGB point cloud. A fixed geometric decoder and a contextual label rule connect their outputs.

Part predictor

Groups broad surfaces and predicts rotation or translation. These are the final part instances, augmented with geometric motion.

Dense handle predictor

Labels individual points and groups them into small handle instances. Their locations guide hinges; their detections stay in the final output.

Joint part-handle predictor

Uses its own parent query features to propose associated handles. Its parts remain internal; only the additional handles enter the final output.

A physical clue. No motion-regression training.

A handle helps place the hinge.

A door handle is usually away from its hinge. We fit a box to the reliable part surface, construct candidate hinge lines, and let a nearby handle select the opposite side. Rotational axes use an upright prior; translating parts use the fitted surface’s thin direction. These explicit assumptions also define the decoder’s limits.

Training-free
motion decoding

Across real scenes

The same scene.
A richer interaction map.

Compare movable parts and motion against REACT3D on all 42 validation scenes, then inspect the predictions from the same viewpoint.

Same evaluated output. Different inputs.

AP at IoU 0.5 (%) · higher is better

Swipe the table to compare all metrics

Results over all 42 validation scenes. Motion AP additionally requires correct axes and rotational hinge locations.
Method	Motion AP	Part AP	Rotation	Translation
REACT3D	8.09	14.46	12.29	3.89
Segment–Snap ours	40.98	47.93	56.37	25.59

Segment–Snap receives an RGB point cloud and is trained on Articulate3D. REACT3D receives RGB-D keyframes, camera poses and a mesh, without training on Articulate3D. The output and evaluator are shared; the input and training conditions are not.

Motion AP checks the mask, axis and rotational hinge location. Rotation and translation are its per-class scores. The 32.89-point gap has a paired 95% scene interval of [18.49, 47.29]. This measures scene-sampling uncertainty, not retraining variability.

Timing and evaluation scope

A complete articulated-part inference pass takes a median 33.78 s for Segment–Snap and 3638.1 s for REACT3D. Our timed path includes superpoint extraction, the part and dense-handle networks, and motion decoding; it does not run the third predictor for extra handles. REACT3D starts from extracted keyframes, poses and mesh and includes its 13 stages.

All 42 scenes are retained, including three with verified empty REACT3D predictions. Twenty scenes also appear in REACT3D's authors' development lists. Our model and hyperparameter selection consulted validation data, so these are development-set comparisons. Timing is not a claim of lower GPU memory use.

Look inside the predictions

Four views. One shared scene.

Choose a scene. Compare the RGB scan and annotations with each method’s parts and motion axes. Select any image to enlarge it.

Read the predictions: saturated part colors mean a correct mask and motion; pale colors mean a matched mask with incorrect motion; gray means an unmatched prediction. A missing annotated color reveals a missed part.

Purple axes: rotation Teal axes: translation

Scene 01

31 annotated movable parts.

Both methods miss annotated parts. Here, more of our displayed detections recover the correct mask and motion.

RGB point cloud of validation scene 01 in the same view used for the annotations and both predictions. — **RGB input**The shared scene point cloud.

Annotated movable parts and rotation or translation axes for validation scene 01. — **Ground truth**Annotated parts and their motion axes.

Segment-Snap part masks and motion axes in validation scene 01; saturated colors mark predictions that pass all motion checks. — **Segment–Snap**10 matched of 15 displayed predictions.

REACT3D part masks and motion axes in validation scene 01; gray marks unmatched detections and pale colors mark motion failures. — **REACT3D**1 matched of 5 predictions.

Scene 02

13 annotated movable parts.

The paired views expose differences in part extent and motion parameters, not just the number of detected objects.

RGB point cloud of validation scene 02 in the same view used for the annotations and both predictions. — **RGB input**The shared scene point cloud.

Annotated movable parts and rotation or translation axes for validation scene 02. — **Ground truth**Annotated parts and their motion axes.

Segment-Snap part masks and motion axes in validation scene 02; saturated colors mark predictions that pass all motion checks. — **Segment–Snap**10 matched of 10 displayed predictions.

REACT3D part masks and motion axes in validation scene 02; gray marks unmatched detections and pale colors mark motion failures. — **REACT3D**2 matched of 6 predictions.

Scene 03

9 annotated movable parts.

A visible prediction is not necessarily a correct interaction: it must pass both the part-mask and motion checks.

RGB point cloud of validation scene 03 in the same view used for the annotations and both predictions. — **RGB input**The shared scene point cloud.

Annotated movable parts and rotation or translation axes for validation scene 03. — **Ground truth**Annotated parts and their motion axes.

Segment-Snap part masks and motion axes in validation scene 03; saturated colors mark predictions that pass all motion checks. — **Segment–Snap**7 matched of 9 displayed predictions.

REACT3D part masks and motion axes in validation scene 03; gray marks unmatched detections and pale colors mark motion failures. — **REACT3D**0 matched of 7 predictions.

Scene 04

11 annotated movable parts.

Our predictions recover more valid part-motion instances in this example, but some annotated parts remain missing.

RGB point cloud of validation scene 04 in the same view used for the annotations and both predictions. — **RGB input**The shared scene point cloud.

Annotated movable parts and rotation or translation axes for validation scene 04. — **Ground truth**Annotated parts and their motion axes.

Segment-Snap part masks and motion axes in validation scene 04; saturated colors mark predictions that pass all motion checks. — **Segment–Snap**7 matched of 8 displayed predictions.

REACT3D part masks and motion axes in validation scene 04; gray marks unmatched detections and pale colors mark motion failures. — **REACT3D**0 matched of 3 predictions.

These four favorable examples follow a fixed ranking by the difference in displayed matched instances, among scenes with at least five annotated parts. Our display threshold is confidence ≥ 0.50; every REACT3D prediction is shown. The plot below reports all scenes, not just these examples.

See the complete pictureAll 42 validation scenes

Validation

Small connections.
Measurable gains.

Controlled comparisons on 42 public validation scenes. All three perception models are trained on the 195 training scenes.

Handles → motionMotion AP₅₀^AO

Same part masks.
Better hinge placement.

Only the origin rule changes. Masks, classes, confidence scores and axes stay fixed.

Centroid origin13.74
Handle-guided origin40.98

+27.25percentage points from the handle cue

Parts → handlesHandle AP₅₀

More handles.
Better motion labels.

Add part-associated candidates, then use context to refine only the added handles’ classes.

Dense detections24.63
+ part-associated proposals29.65
Union + part-only labels30.63
Union + full context30.99

Reading the results. Motion AP requires a correct mask, axis and rotational origin; handle AP evaluates instance masks and motion classes. Scores are percentages. Gains use unrounded scores, so rounded endpoints may not subtract exactly.

Part-only and full-context labels are alternative corrections of the same union. Full context includes dense-label fallback and adds 1.34 pp, not 0.98 + 1.34 pp. Proposal gains show complementarity, not an isolated causal effect of parent conditioning.

What do the stability controls show?

Appending children improves handle AP by 3.94–5.01 pp across three child-training draws. The reference proposal gain has a paired 95% scene interval of [+1.47, +8.56] pp.

Full contextual correction is positive in all three draws but varies from 0.19 to 1.34 pp. Scene intervals use fixed predictions and do not measure retraining variability. Validation also informed model selection; these are development-set comparisons.

Read the controls in the paper

See the difference

What does coupling change?

Compare hinge placement, handle recovery and motion-class correction on selected validation examples. The last tab shows the method’s limits. Light gray points are scene context; each plot includes a legend.

Can a handle resolve the hinge side?

Switch between Centroid and Handle-guided to compare two origin estimates for the same predicted movable part. This isolates how the handle cue changes hinge placement while the part mask stays fixed.

Green predicted door surface with its centroid far from the black annotated hinge; origin error is 0.42 metres. — Centroid origin · 0.42 m error

The same predicted door surface with a handle-guided origin close to the annotated hinge; origin error is 0.05 metres. — Handle-guided origin · 0.05 m error

Change the origin. Keep the part.

The baseline puts the origin at the part’s center. Our geometric decoder uses a predicted handle to choose the opposite attachment side. The plot shows the resulting origin estimate; the handle itself is not drawn.

How to read the plot

Green points — predicted movable part
Black line — annotated hinge
Colored dot — predicted origin
Colored segment — perpendicular origin error

0.42mCentroid

0.05mHandle-guided

Origin error in this example. Red/green and FAIL/PASS indicate whether the error is below 0.25 m; they do not evaluate the full detection.

View before View after

The part mask and specimen are fixed, but the source renderer uses separate projection directions: compare the origin and error, not pixel alignment. This is the median centroid-error example among cases improved by handle guidance.

Can parts help us find and interpret handles?

The two examples illustrate different benefits: an extra part-associated proposal recovers a handle location, while a containing part’s motion class corrects the meaning of an already predicted handle. They are separate examples, not a before/after pair.

Green additional handle proposal overlapping black annotated handle points in a location missed by the dense source. — Additional proposals
Find a missing handle.

**Black** — annotated handle
**Green** — extra predicted handle

The joint part-handle predictor adds a candidate that overlaps the annotation at IoU > 0.5. The dense source alone has no matching mask here. This illustrates why combining the two proposal sources increases coverage.

Black predicted handle points beside a translucent green containing-part prediction; part context corrects the handle label from translation to rotation. — Contextual labels
Correct what the handle controls.

**Black** — predicted handle from the joint branch
**Green** — predicted containing part

The part’s rotation class changes this handle’s label from **translation to rotation**. The black handle points and their confidence stay unchanged: this is a semantic correction, not a better mask.

The two panels use different legends: black is the annotation in the recovery example, but the prediction in the context example. These selected localization and relabeling cases are not per-example AP measurements.

What can these connections not repair?

Handle guidance still depends on the motion decoder’s geometric assumptions. Part-based label correction changes a handle’s class, not its shape. These examples separate those two limits.

Green predicted part with a nonvertical black annotated hinge and a red predicted origin whose perpendicular error is 0.85 metres. — Geometric assumption
Not every hinge is upright.

**Green** — predicted movable part
**Black line** — annotated hinge
**Red dot and segment** — origin and error

This nonvertical hinge falls outside the decoder’s upright-axis assumption. The handle-guided origin still has 0.85 m error. A useful handle cue cannot correct the wrong axis model.

Translucent red dense-handle prediction extending beyond black annotated handle points. — Perception limit
A correct class cannot fix a mask.

**Black** — annotated handle
**Red** — overextended dense prediction

The predicted handle covers too much of the surrounding surface. Relabeling cannot remove those extra points, and our contextual correction leaves dense detections untouched.

The model assumes approximately planar supports and upright rotational axes. The report includes the full failure census and additional paired examples.

These are selected validation examples explaining behavior, not estimates of how frequently each outcome occurs. Aggregate performance is reported in the quantitative results above.

Research resources

Take a closer look.

The paper, code, and model checkpoints.

Read the paperarXiv:2609.25247 · Method and results Code repositorySegment-Snap on GitHub Model checkpointsSegment-Snap on Hugging Face

Cite this work.

If this work is useful to your research, please cite our arXiv preprint.

Download BibTeX

BibTeX · arXiv

@article{kong2026segmentsnap,
  title   = {Geometric and Semantic Coupling for
             Interaction Understanding in 3D Scenes},
  author  = {Kong, Hanyang and Yang, Xingyi},
  journal = {arXiv preprint arXiv:2609.25247},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.25247}
}

Geometric and Semantic Couplingfor Interaction Understanding in 3D Scenes

See the connectionsin motion.

Beyond recognizingthe cabinet.

What can move?

How does it move?

Where do we interact?

Learn the pieces.Connect the evidence.