Sifei Liu

I am a Principal Research Scientist and Tech Lead at NVIDIA Research, where I work with the LPR team led by Jan Kautz. My current research focuses on embodied foundational models, efficient transformer architectures, and spatial reasoning.

Before joining NVIDIA, I earned my Ph.D. at the VLLAB at UC Merced, advised by Ming-Hsuan Yang. I have been fortunate to receive the Baidu Graduate Fellowship, the NVIDIA Pioneering Research Award, and the Rising Star EECS recognition.

CV Scholar GitHub NVIDIA Twitter

Current Focus

Building grounded multimodal systems that can reason, scale, and act in open-world settings.

Embodied foundation models Efficient transformers Spatial reasoning

Research Themes

What I work on

I work on making multimodal systems more grounded, more efficient, and more capable in open-world environments.

Embodied VLM

Perception for agents and robotics

Building multimodal systems that can perceive, reason, and act in 3D environments for navigation and embodied decision making.

Efficient Models

Transformer efficiency at high resolution

Designing token-efficient architectures and attention mechanisms that preserve detail without paying the full compute cost.

Spatial Intelligence

Grounded multimodal understanding

Connecting images, language, and geometry so models can reason about structure, localization, and relationships across views.

News

Jan 2026
We released SR-3D, a 3D-aware region-prompted VLM for grounded spatial reasoning across views and scenes.
Oct 2025
Our ICCV 2025 paper Describe Anything introduces DAM for detailed localized image and video captioning.
Oct 2025
Our ICCV 2025 paper Token-Efficient VLM presents an efficient VLM for high-resolution visual understanding.
Mar 2025
SpatialRGPT was demoed at GTC 2025 as a part of Agentic AI for Physical Operations!
Feb 2025
We release the GSPN, a fast vision attention module that accelerates Stable Diffusion inference 84x. Stay tuned for more details!
Feb 2025
5 papers was accepted to CVPR 2025! Stay tuned for more updates!
Jan 2025
We released the NaVILA, a navigation agent that can navigate in a 3D environment with a language instruction.
Dec 2024
We presented CosAE at NeurIPS 2024! Stay tuned for code release.
Oct 2024
We released the SpatialRGPT code, datasets, and models! Welcome to try demos!

Selected Publications

ICLR 2026

3D Aware Region Prompted Vision Language Model

An-Chieh Cheng, Yang Fu, Yukang Chen, Zhijian Liu, Xiaolong Li, Subhashree Radhakrishnan, Song Han, Yao Lu, Jan Kautz, Pavlo Molchanov, Hongxu (Danny) Yin, Xiaolong Wang, Sifei Liu

SR-3D unifies single-view 2D and multi-view 3D representations for flexible region prompting and grounded spatial reasoning.

Project Page PDF arXiv Code

ICCV 2025

Describe Anything: Detailed Localized Image and Video Captioning

Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, Yin Cui

DAM generates detailed localized captions for user-specified regions in images and videos, preserving both local detail and global context.

Project Page PDF arXiv Code

ICCV 2025

Token-Efficient VLM: High-Resolution Image Understanding via Dynamic Region Proposal

Yitong Jiang, Jingwei Gu, Tianfan Xue, Ka Chun Cheung, Pavlo Molchanov, Hongxu (Danny) Yin, Sifei Liu

TEVA improves high-resolution image understanding by dynamically selecting detail-rich regions while keeping token usage efficient.

Project Page Code

CVPR 2025

Parallel Sequence Modeling via Generalized Spatial Propagation Network

Hongjun Wang, Wonmin Byeon, Jiarui Xu, Jingwei Gu, , Xiaolong Wang, Kai Han, Jan Kautz, Sifei Liu

GSPN is a fast vision attention module that accelerates generic vision foundation models for high-resolution input images.

Project Page PDF arXiv Code

arxiv 2025

NaVILA: Legged Robot Vision-Language-Action Model for Navigation

An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu (Danny) Yin, Sifei Liu, Xiaolong Wang

NaVILA is a two-level framework that combines VLAs with locomotion skills for navigation. It generates high-level language-based commands, while a real-time locomotion policy ensures obstacle avoidance.

Project Page PDF arXiv Code

CVPR 2025

NVILA: Efficient Frontier Visual Language Models

Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang, Yukang Chen, Cheng-Yu Hsieh, De-An Huang, An-Chieh Cheng, Vishwesh Nath, Jinyi Hu, Sifei Liu, Ranjay Krishna, Daguang Xu, Xiaolong Wang, Pavlo Molchanov, Jan Kautz, Hongxu Yin, Song Han, Yao Lu

Efficient frontier VLM models with efficient training and inference.

Project Page PDF arXiv Code

ICLR 2025

No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images

Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Marc Pollefeys, Ming-Hsuan Yang, Songyou Peng

No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images

Project Page PDF arXiv Code

NeurIPS 2024

SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models

An-Chieh Cheng, Hongxu (Danny) Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, Sifei Liu

SpatialRGPT is a grounded spatial reasoning model that can reason about spatial relationships in images.

Project Page PDF arXiv Code

CVPR 2025

COLMAP-Free 3D Gaussian Splatting

Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A. Efros, Xiaolong Wang

3D Gaussian Splatting without COLMAP computation.

Project Page PDF arXiv Code

arXiv 2023

TUVF: Learning Generalizable Texture UV Radiance Fields

An-Chieh Cheng, Xueting Li, Sifei Liu, Xiaolong Wang

The paper introduces TUVF, a method for learning generalizable texture UV radiance fields.

Project Page PDF arXiv Code

CVPR 2023

Open-vocabulary panoptic segmentation with text-to-image diffusion models

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, Shalini De Mello

We present ODISE: Open-vocabulary DIffusion-based panoptic SEgmentation, which unifies pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation.

Project Page PDF arXiv Code