InstanceSplat

InstanceSplat

Instance-Aware Feed-Forward 3D Gaussian Splatting for Scene Understanding

Minchao Jiang1,2 Xiaoxuan Ma3 Shunyu Jia4 Haoru Wang5 Zhang Liang4 Wentao Zhu2

1 Shanghai Jiao Tong University 2 Eastern Institute of Technology, Ningbo 3 Carnegie Mellon University
4 Xidian University 5 Peking University

arXiv Code

TL;DR We propose an instance-aware feed-forward 3DGS method.

InstanceSplat takes pose-free multi-view images and produces an instance-aware 3D Gaussian representation for reconstruction, instance segmentation, and open-vocabulary scene understanding.

Given pose-free multi-view images (a), InstanceSplat jointly predicts explicit 3D Gaussians with compact instance and open-vocabulary semantic features (b). Shared 3D geometry provides cross-view consistency, while instance cues enhance boundary-aware 3DGS reconstruction and semantic coherence. The resulting representation supports novel-view synthesis (NVS), instance segmentation, and open-vocabulary scene understanding (c).

Abstract

Feed-forward 3D Gaussian Splatting (3DGS) enables efficient and generalizable 3D reconstruction, but current feed-forward 3DGS methods for scene understanding remain largely category-oriented. In contrast, instance-aware 3DGS methods typically rely on per-scene optimization and often decouple reconstruction from instance and semantic learning, limiting reciprocal interactions among them. We present InstanceSplat, a unified feed-forward 3DGS framework for generalizable 3D reconstruction and instance-aware scene understanding from pose-free multi-view images. In a single forward pass, InstanceSplat constructs an instance-aware Gaussian representation that jointly encodes appearance, geometry, instance identity, and language-aligned semantics. Shared 3D Gaussians ground instance identities across views, producing renderable and cross-view-consistent instance features. To allow reconstruction and scene understanding to benefit from each other, we further design an instance-centric learning strategy that connects reconstruction, instance learning, and semantic learning through shared instance structure. Specifically, instance cues guide reconstruction, language-aligned semantics strengthen the discrimination of confusing same-category instances, and instance regions aggregate semantic evidence into coherent object-level predictions.

Method

Overview of the InstanceSplat architecture, including the feed-forward backbone, 3D-consistent instance grounding, and instance-centric coupling.
Given pose-free multi-view images, the feed-forward backbone constructs an instance-aware Gaussian representation. 3D-consistent grounding and instance-centric coupling then connect reconstruction with instance and semantic learning.
01

Instance-Aware Gaussian Representation

Each Gaussian carries appearance, geometry, an 8-dimensional instance embedding, and a compact language-aligned semantic feature. Confidence-aware voxelization merges redundant primitives while preserving this shared structure.

02

3D-Consistent Instance Grounding

Differentiable feature rendering supervises shared Gaussians in image space. Prototype-based contrastive learning promotes compact instances, separates different objects, and aligns identities across views.

03

Instance-Centric Coupling

Instance boundaries guide reconstruction, semantic similarity emphasizes confusing same-category objects, and predicted instance regions aggregate semantic evidence into coherent object-level labels.

Experiment

Novel-View Rendering

Swipe left or right to compare scenes.

1 / 5

ScanNet++ scene 0a76e06478

RGB Rendered appearance from novel camera views.
Depth Predicted geometry rendered along the same view path.
Instance features PCA visualization of the rendered instance embedding.

ScanNet++ scene 0b0bb941b7

RGB Rendered appearance from novel camera views.
Depth Predicted geometry rendered along the same view path.
Instance features PCA visualization of the rendered instance embedding.

ScanNet++ scene 0d2ee665be

RGB Rendered appearance from novel camera views.
Depth Predicted geometry rendered along the same view path.
Instance features PCA visualization of the rendered instance embedding.

ScanNet++ scene 0e28968c1f

RGB Rendered appearance from novel camera views.
Depth Predicted geometry rendered along the same view path.
Instance features PCA visualization of the rendered instance embedding.

ScanNet++ scene 0f0191b10b

RGB Rendered appearance from novel camera views.
Depth Predicted geometry rendered along the same view path.
Instance features PCA visualization of the rendered instance embedding.

Interactive Instance Removal

Click an instance to hide or restore its full Gaussian group, and drag the scene to orbit.

Loading full Gaussian scene...

Drag to orbit · scroll to zoom · click an instance button to hide or restore that complete Gaussian group.

Open-Vocabulary Instance-Level 3D Localization

Text queries match semantic prototypes to clustered 3D Gaussian groups, which are highlighted in red.

Open-vocabulary instance-level 3D localization results. Text queries identify complete Gaussian groups highlighted in red across several reconstructed indoor scenes.
Open-vocabulary instance-level 3D localization on ScanNet. Text queries retrieve the corresponding 3D Gaussian groups, highlighted in red.

Reference

Citation

@misc{jiang2026instancesplat,
  title  = {InstanceSplat: Instance-Aware Feed-Forward 3D Gaussian
            Splatting for Scene Understanding},
  author = {Minchao Jiang and Xiaoxuan Ma and Shunyu Jia and Haoru Wang
            and Zhang Liang and Wentao Zhu},
  journal = {arXiv preprint arXiv:2608.07144},
  year   = {2026}
}