Haoru Xue

I am a first-year PhD student at Berkeley AI Research (BAIR), UC Berkeley.

Previously I was a MS Robotics student at CMU LeCAR lab and DRIVE lab, working with Professor Guanya Shi and John M. Dolan. In 2023 I was a visiting student with Professor Francesco Borrelli at MPC lab, UC Berkeley.

I’m interested in humanoid ๐Ÿฆพ and quadruped ๐Ÿฆฟ robots that can do agile and dexterous loco-manipulation. I also want to strategically interface them with foundation models for robotics to perform long-horizon tasks ๐Ÿ”ญ.

From 2021 to 2024 I led AI Racing Tech ๐ŸŽ๏ธ, a multi-million $ autonomous racing research project in UCB, CMU and UCSD across 5 labs. I deployed real-world robot learning on F1-level self-driving race cars at 160 MPH. We are the top U.S. team in the Indy Autonomous Challenge. Learn more in my playground.

๐Ÿ”ฅ Latest

[10/2024, UCB] ๐ŸŽ‰ Thrilled to announce AnyCar. Follow the release on X!
[09/2024, UCB] ๐ŸŽ‰ Thrilled to announce DIAL-MPC โ˜Ž๏ธ. Follow the release on X!
[08/2024, UCB] ๐ŸŽ“ I joined UC Berkeley EECS PhD program (AI Robotics).
[05/2024, ICRA] (presentation | slides) Oral on “Learning MPC with Error Dynamics Regression for Autonomous Racing“.
[04/2024, CMU] (presentation | slides) Master thesis talk on “Optimal Control and Robot Learning on Agile Safety-Critical Systems“.
[04/2024, Stanford] Talk on “Towards Foundational Control Models for Agile Locomotion” at Stanford Intelligent Systems Laboratory (SISL).
[01/2024, UCB] Talk on “Safe Online Adaptation for Robots with Extreme Dynamics” at DARPA ANSR program.
[10/2023, UCB] Talk on “Learning MPC with Error Dynamics Regression for Autonomous Racing” at Model Predictive Control (MPC) Lab.

๐Ÿ”ฌ Research

Ultimately, I want to build generalist robots that will fundamentally change the human society. Recently, these are my research interests:

  • Scale up inference-time compute with sampling and diffusion to better generalize in unseen tasks.
  • The interface between high-level LLM/VLM planning and low-level RL policy is crutial for enabling both long-horizon and dexterous/agile loco-manipulation.
  • Demonstrate agility and adaptability for whole-body control.

Full-Order Sampling-Based MPC for Torque-Level Locomotion Control via Diffusion-Style Annealing
Haoru Xue*, Chaoyi Pan*, Zeji Yi, Guannan Qu, Guanya Shi

Under Review | website | arxiv | code | social media

DIAL-MPC โ˜Ž๏ธ : diffusion + sampling-based MPC can achieve adaptive and training-free whole-body control on legged systems using RL-style rewards. Deployed in real on agile quadrupedal torque control.

AnyCar to Anywhere: Learning Universal Dynamics Model for Agile and Adaptive Mobility
Wenli Xiao*, Haoru Xue*, Tony Tao, Dvij Kalaria, John M. Dolan, Guanya Shi

Under Review | website | arxiv | code | social media | IEEE Spectrum

AnyCar ๐ŸŽ๏ธ ๐Ÿš— ๐Ÿš™ ๐Ÿ›ป ๐Ÿšš: a generalist dynamics model ๐ŸŒŽ built with transformer + large-scale sim pre-training + small-scale real fine-tuning. Achieves agile and adaptive control on a family of wheeled embodiments (few or zero shot) and outperforms specialist policies.

Learning Model Predictive Control with Error Dynamics Regression for Autonomous Racing
Haoru Xue*, Edward L. Zhu, John M. Dolan, Francesco Borrelli

ICRA 2024 | website | arxiv | video | code | social media | talk

Use MPC + robot learning to explore optimal policy and dynamics model online safely. Perform Sim2Real and learn the handling limit of extreme driving like a professional race car driver. Deployed on a full-size race car!

WROOM: An Autonomous Driving Approach for Off-Road Navigation
Dvij Kalaria*, Shreya Sharma, Sarthak Bhagat, Haoru Xue, John M. Dolan

ICRA 2024 Workshop | arxiv | website | code

WROOM brings a gym environment for training off-road driving RL policy. We use PPO + CBF to train an end-to-end agent to safely navigate in the real world.

Segment Anything Model for Road Network Graph Extraction
Congrui Hetang*, Haoru Xue, Cindy Le, Tianwei Yue, Wenping Wang, Yihui He

CVPR 2024 Workshop | arxiv | code

We propose a SAM-based pipeline for large-scale road network extraction from aerial image. Comparable accuracy with SOTA is achieved while being 40 times faster.

Best Paper (2nd Workshop on Scene Graphs and Graph Representation Learning)

Spline-Based Minimum-Curvature Trajectory Optimization for Autonomous Racing
Haoru Xue*, Tianwei Yue, John M. Dolan

2023 | arxiv | code

๐ŸŽ“ Education

UC Berkeley EECS
PhD (AI Robotics) 2024

CMU Robotics Institute
MS Robotics 2022 – 2024

Prof. Guanya Shi, John Dolan

UC San Diego
Electrical Eng. 2018 – 2021

๐Ÿ“„ Experience

LeCAR Lab, CMU Robotics Institute
Visiting Researcher
May – Aug. 2024

Advisor: Prof. Guanya Shi

MPC Lab, UC Berkeley
Visiting Researcher
Apr. – Aug. 2023

Advisor: Prof. Francesco Borrelli

AI Racing Tech
Lead Graduate Researcher
Dec. 2019 – May 2024

Advisor: Prof. Shankar Sastry, John Dolan; Dr. Allen Yang, Jack Silberman

Autoware Foundation
Software Engineer
Dec. 2021 – Aug. 2022

๐Ÿ† Awards & Honors