Learning Sparse Visual Representations via Spatial-Semantic Factorization

Abstract

Self-supervised learning (SSL) faces a tension between semantic understanding and image reconstruction. High-level semantic SSL methods such as DINO encourage transformation-invariant global representations for augmentation alignment, while reconstruction-oriented methods such as MAE preserve dense feature grids for spatial grounding but often yield weaker semantic abstractions. We introduce STELLAR, a framework that alleviates this tension by factorizing visual features into a low-rank product of semantic concepts and their spatial distributions. This disentanglement allows us to perform augmentation alignment on semantic tokens while maintaining spatial localization through a localization matrix for reconstruction. We demonstrate that as few as 16 sparse tokens under this factorized form can simultaneously support high-quality reconstruction (2.60 FID) and strong semantic transfer (79.10% ImageNet accuracy). Our results highlight STELLAR as a versatile sparse representation that bridges discriminative and generative vision by separating semantic identity from spatial geometry. Code is available at https://github.com/microsoft/STELLAR.