Paper 
Code 
Hugging Face Part predictor
Groups broad surfaces and predicts rotation or translation. These are the final part instances, augmented with geometric motion.
1st placeArticulate3D Challenge
First in both evaluated outputs · View results
Geometric and Semantic Couplingfor Interaction Understanding in 3D Scenes
A door and its handle tell a shared story. We use their physical relationship to understand what moves, how it moves, and where to interact.
1 National University of Singapore2 The Hong Kong Polytechnic University
Corresponding author: Xingyi Yang
One scene. Connected predictions.Expand figure
Handles → part motion
+27.25pp
Handle-guided hinge placement
Parts → extra handles
+5.01pp
Complementary handle proposals
Part classes → handle labels
+0.98pp
Part-only motion-class correction
Articulate3D validation · percentage-point changes in motion AP or handle AP, as labelled. Different mechanisms, not additive gains.
The task
Beyond recognizing
the cabinet.
A static 3D scan should tell us how a scene can be interacted with—not only what objects it contains.
- 01
What can move?
Locate individual movable parts, such as doors and drawers.
- 02
How does it move?
Infer rotation or translation, its axis, and the hinge location for a rotating part.
- 03
Where do we interact?
Find the small operating regions we call handles, and associate their motion class.
The difficulty: broad surfaces and tiny handles need different prediction support. A closed door also hides its hinge side; a handle alone does not reveal whether its part rotates or slides.
How it works
Learn the pieces.
Connect the evidence.
Three independent predictors see the same RGB point cloud. A fixed geometric decoder and a contextual label rule connect their outputs.
Dense handle predictor
Labels individual points and groups them into small handle instances. Their locations guide hinges; their detections stay in the final output.
Joint part-handle predictor
Uses its own parent query features to propose associated handles. Its parts remain internal; only the additional handles enter the final output.
A physical clue. No motion-regression training.
A handle helps place the hinge.
A door handle is usually away from its hinge. We fit a box to the reliable part surface, construct candidate hinge lines, and let a nearby handle select the opposite side. Rotational axes use an upright prior; translating parts use the fitted surface’s thin direction. These explicit assumptions also define the decoder’s limits.
motion decoding
Validation
Small connections.
Measurable gains.
Controlled comparisons on 42 public validation scenes. All three perception models are trained on the 195 training scenes.
Handles → motionMotion AP50AO
Same part masks.
Better hinge placement.
Only the origin rule changes. Masks, classes, confidence scores and axes stay fixed.
+27.25percentage points from the handle cue
Parts → handlesHandle AP50
More handles.
Better motion labels.
Add part-associated candidates, then use context to refine only the added handles’ classes.
Reading the results. Motion AP requires a correct mask, axis and rotational origin; handle AP evaluates instance masks and motion classes. Scores are percentages. Gains use unrounded scores, so rounded endpoints may not subtract exactly.
Part-only and full-context labels are alternative corrections of the same union. Full context includes dense-label fallback and adds 1.34 pp, not 0.98 + 1.34 pp. Proposal gains show complementarity, not an isolated causal effect of parent conditioning.
What do the stability controls show?
Appending children improves handle AP by 3.94–5.01 pp across three child-training draws. The reference proposal gain has a paired 95% scene interval of [+1.47, +8.56] pp.
Full contextual correction is positive in all three draws but varies from 0.19 to 1.34 pp. Scene intervals use fixed predictions and do not measure retraining variability. Validation also informed model selection; these are development-set comparisons.
Read the controls in the reportSee the difference
What does coupling change?
Compare hinge placement, handle recovery and motion-class correction on selected validation examples. The last tab shows the method’s limits. Light gray points are scene context; each plot includes a legend.
Can a handle resolve the hinge side?
Switch between Centroid and Handle-guided to compare two origin estimates for the same predicted movable part. This isolates how the handle cue changes hinge placement while the part mask stays fixed.
Compare the origin rules.Same predicted part.
Change the origin. Keep the part.
The baseline puts the origin at the part’s center. Our geometric decoder uses a predicted handle to choose the opposite attachment side. The plot shows the resulting origin estimate; the handle itself is not drawn.
How to read the plot
- Green points — predicted movable part
- Black line — annotated hinge
- Colored dot — predicted origin
- Colored segment — perpendicular origin error
0.42mCentroid
0.05mHandle-guided
Origin error in this example. Red/green and FAIL/PASS indicate whether the error is below 0.25 m; they do not evaluate the full detection.
The part mask and specimen are fixed, but the source renderer uses separate projection directions: compare the origin and error, not pixel alignment. This is the median centroid-error example among cases improved by handle guidance.
Can parts help us find and interpret handles?
The two examples illustrate different benefits: an extra part-associated proposal recovers a handle location, while a containing part’s motion class corrects the meaning of an already predicted handle. They are separate examples, not a before/after pair.
Find a missing handle.
- Black — annotated handle
- Green — extra predicted handle
The joint part-handle predictor adds a candidate that overlaps the annotation at IoU > 0.5. The dense source alone has no matching mask here. This illustrates why combining the two proposal sources increases coverage.
Correct what the handle controls.
- Black — predicted handle from the joint branch
- Green — predicted containing part
The part’s rotation class changes this handle’s label from translation to rotation. The black handle points and their confidence stay unchanged: this is a semantic correction, not a better mask.
The two panels use different legends: black is the annotation in the recovery example, but the prediction in the context example. These selected localization and relabeling cases are not per-example AP measurements.
What can these connections not repair?
Handle guidance still depends on the motion decoder’s geometric assumptions. Part-based label correction changes a handle’s class, not its shape. These examples separate those two limits.
Not every hinge is upright.
- Green — predicted movable part
- Black line — annotated hinge
- Red dot and segment — origin and error
This nonvertical hinge falls outside the decoder’s upright-axis assumption. The handle-guided origin still has 0.85 m error. A useful handle cue cannot correct the wrong axis model.
A correct class cannot fix a mask.
- Black — annotated handle
- Red — overextended dense prediction
The predicted handle covers too much of the surrounding surface. Relabeling cannot remove those extra points, and our contextual correction leaves dense detections untouched.
The model assumes approximately planar supports and upright rotational axes. The report includes the full failure census and additional paired examples.
These are selected validation examples explaining behavior, not estimates of how frequently each outcome occurs. Aggregate performance is reported in the quantitative results above.
Research resources
Take a closer look.
The paper, code, and model checkpoints.
Read the paperMethod, experiments, and discussion
Code repositorySegment-Snap on GitHub
Model checkpointsSegment-Snap on Hugging Face
Cite this work.
If this work is useful to your research, please cite the technical report.
BibTeX · technical report
@techreport{kong2026segmentsnap,
title = {Geometric and Semantic Coupling for
Interaction Understanding in 3D Scenes},
author = {Kong, Hanyang and Yang, Xingyi},
year = {2026},
institution = {National University of Singapore
and The Hong Kong Polytechnic University},
url = {https://hyokong.github.io/segment-snap-page/}
}