Hi, I'm Peter Karkus.

Senior Research Scientist @ NVIDIA Research

pkarkus@nvidia.com | Google Scholar | LinkedIn

Previously:

PhD Candidate, Research Assistant @ National University of Singapore
Research Intern @ DeepMind
Research Intern @ Google Brain
Visiting Researcher @ MIT CSAIL
Fulbright Visiting Researcher @ CMU Robotics Institute
Intern @ NTT Basic Research Lab

My research interests include autonomous driving and robotics, with a focus on end-to-end vision-language-action and world-action models, closed-loop simulation, and RL post-training. At NVIDIA, I work on Alpamayo, an open platform for reasoning-based vehicle autonomy, focusing on closed-loop evaluation and training of driving models. I am a core contributor to Alpamayo models, AlpaSim, and AlpaGym. My earlier work introduced differentiable algorithm networks for robot planning, navigation, mapping, and localization.

Selected papers

Beyond Behavior Cloning in Autonomous Driving: A Survey of Closed-Loop Training Techniques [pdf] P. Karkus*, M. Igl*, Y. Chen, K. Chitta, J. Packer, B. Douillard, R. Tian, A. Naumann, G. Garcia-Cobo, S. Tan, A. Degirmenci, A. Popov, N. Smolyanskiy, U. Muller, B. Ivanovic, M. Pavone. *equal contribution In submission, 2026
Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models [pdf] [website] Z. Zhang, P. Karkus, M. Igl, W. Ding, Y. Chen, B. Ivanovic, M. Pavone. Conference on Computer Vision and Pattern Recognition (CVPR), 2025 Oral presentation (top 3.3%).
Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail [pdf] [code] NVIDIA: Y. Wang, W. Luo, J. Bai, Y. Cao, T. Che, K. Chen, Y. Chen, J. Diamond, Y. Ding, W. Ding, L. Feng, G. Heinrich, J. Huang, P. Karkus, B. Li, et al. arXiv preprint arXiv:2511.00088, 2025
DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles [pdf] [website] P. Karkus, B. Ivanovic, S. Mannor, M. Pavone. Conference on Robot Learning (CoRL), 2022
Differentiable Algorithm Networks for Composable Robot Learning [pdf] [talk] P. Karkus, X. Ma, D. Hsu, L.P. Kaelbling, W.S. Lee, T. Lozano-Pérez. Robotics: Science and Systems (RSS), 2019 Best paper nomination (top 5 papers).

All publications

Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation [pdf] [website] W. Gu, A. Gupta, A. Sah, R. Varghese, L.S. Vanam, P. Adireddi, P. Karkus, N. Gopalan. Conference on Robot Learning (CoRL), 2026
Learning Multiple Initial Solutions to Optimization Problems [pdf] [website] E. Sharony, H. Yang, T. Che, M. Pavone, S. Mannor, P. Karkus. International Conference on Robotics and Automation (ICRA), 2026
RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies [pdf] G. Garcia-Cobo*, M. Igl*, P. Karkus*, Z. Zhang*, M. Watson, Y. Chen, B. Ivanovic, M. Pavone. *equal contribution Conference on Computer Vision and Pattern Recognition (CVPR) Findings, 2026
Beyond Behavior Cloning in Autonomous Driving: A Survey of Closed-Loop Training Techniques [pdf] P. Karkus*, M. Igl*, Y. Chen, K. Chitta, J. Packer, B. Douillard, R. Tian, A. Naumann, G. Garcia-Cobo, S. Tan, A. Degirmenci, A. Popov, N. Smolyanskiy, U. Muller, B. Ivanovic, M. Pavone. *equal contribution In submission, 2026
Planning-Aligned Token Compression for Long-Context Autonomous Driving [pdf] Z. Liang, Y. Chen, Y. You, P. Karkus, W. Ding, B. Li, A. Popov, Y. Wang, M. Igl, Y. Li, D. Xu, N. Smolyanskiy, B. Ivanovic, P. Luo, M. Pavone. IEEE Robotics and Automation Letters (RA-L) 11(10), 2026
Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives [pdf] [website] L. Wang, M.-A. Lavoie, S. Papais, B. Nisar, Y. Chen, W. Ding, B. Ivanovic, H. Shao, A. Abuduweili, E. Cook, Y. Zhou, P. Karkus, J. Li, C. Liu, M. Pavone, S.L. Waslander. Foundations and Trends in Robotics 13(1–2), 2026
OPTED: On-Policy Fine-Tuning for End-to-End Driving Using a Render-Free Teacher [pdf] [website] D. Da Col, M. Igl, P. Karkus, K. Chitta, B. Ivanovic, M. Pavone, K. Schindler, C. Sakaridis. arXiv preprint arXiv:2609.20756, 2026
Vision-Language-Action Autonomous Driving Agent with Language-Based Memory [pdf] K. Yan, X. Chen, Y. Cao, A. Naumann, P. Karkus, Y. Wang, J. Packer, A. Schwing, Y. Wang, B. Ivanovic, W. Luo, M. Pavone. arXiv preprint arXiv:2609.38641, 2026
CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight [pdf] [website] C. Peng, W. Ding, R. Tian, Z. Zhou, J. Packer, M. Igl, P. Karkus, Y. Wang, M. Tomizuka, B. Ivanovic, M. Pavone, Y. Chen. arXiv preprint arXiv:2610.00859, 2026
Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models [pdf] [website] Z. Zhang, P. Karkus, M. Igl, W. Ding, Y. Chen, B. Ivanovic, M. Pavone. Conference on Computer Vision and Pattern Recognition (CVPR), 2025 Oral presentation (top 3.3%).
STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes [pdf] [website] J. Yang, J. Huang, B. Ivanovic, Y. Chen, Y. Wang, B. Li, Y. You, A. Sharma, M. Igl, P. Karkus, D. Xu, Y. Wang, M. Pavone. International Conference on Learning Representations (ICLR), 2025
LoRD: Adapting Differentiable Driving Policies to Distribution Shifts [pdf] [code] C. Diehl, P. Karkus, S. Veer, M. Pavone, T. Bertram. International Conference on Robotics and Automation (ICRA), 2025
Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail [pdf] [code] NVIDIA: Y. Wang, W. Luo, J. Bai, Y. Cao, T. Che, K. Chen, Y. Chen, J. Diamond, Y. Ding, W. Ding, L. Feng, G. Heinrich, J. Huang, P. Karkus, B. Li, et al. arXiv preprint arXiv:2511.00088, 2025
DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features [pdf] [website] L. Wang, S.W. Kim, J. Yang, C. Yu, B. Ivanovic, S.L. Waslander, Y. Wang, S. Fidler, M. Pavone, P. Karkus. Advances in Neural Information Processing Systems (NeurIPS), 2024
DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Driving [pdf] [code] Z. Huang, P. Karkus, B. Ivanovic, Y. Chen, M. Pavone, C. Lv. International Conference on Robotics and Automation (ICRA), 2024
Partial-View Object View Synthesis via Filtering Inversion [pdf] [website] F.-Y. Sun, J. Tremblay, V. Blukis, K. Lin, D. Xu, B. Ivanovic, P. Karkus, S. Birchfield, D. Fox, R. Zhang, Y. Li, J. Wu, M. Pavone, N. Haber. International Conference on 3D Vision (3DV), 2024
System-Level Analysis of Module Uncertainty Quantification in the Autonomy Pipeline [pdf] S. Deglurkar, H. Shen, A. Muthali, M. Pavone, D.D. Margineantu, P. Karkus, B. Ivanovic, C.J. Tomlin. IEEE Conference on Decision and Control (CDC), 2024
Interactive Joint Planning for Autonomous Vehicles [pdf] Y. Chen, S. Veer, P. Karkus, M. Pavone. IEEE Robotics and Automation Letters (RA-L) 9(2), 2024
Tree-Structured Policy Planning with Learned Behavior Models [pdf] Y. Chen, P. Karkus, B. Ivanovic, X. Weng, M. Pavone. International Conference on Robotics and Automation (ICRA), 2023
Planning with Occluded Traffic Agents Using Bi-Level Variational Occlusion Models [pdf] F. Christianos, P. Karkus, B. Ivanovic, S.V. Albrecht, M. Pavone. International Conference on Robotics and Automation (ICRA), 2023
Receding Horizon Planning with Rule Hierarchies for Autonomous Vehicles [pdf] [code] S. Veer, K. Leung, R.K. Cosner, Y. Chen, P. Karkus, M. Pavone. International Conference on Robotics and Automation (ICRA), 2023
DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles [pdf] [website] P. Karkus, B. Ivanovic, S. Mannor, M. Pavone. Conference on Robot Learning (CoRL), 2022
Differentiable SLAM-Net: Learning Particle SLAM for Visual Navigation [pdf] [website] P. Karkus, S. Cai, D. Hsu. Conference on Computer Vision and Pattern Recognition (CVPR), 2021
Differentiable Mapping Networks: Learning Structured Map Representations for Sparse Visual Localization [pdf] [website] [talk] P. Karkus, A. Angelova, V. Vanhoucke, R. Jonschkowski. International Conference on Robotics and Automation (ICRA), 2020
Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations [pdf] [website] [code] X. Ma, P. Karkus, D. Hsu, W.S. Lee, N. Ye. International Conference on Learning Representations (ICLR), 2020
Particle Filter Recurrent Neural Networks [pdf] [code] X. Ma*, P. Karkus*, D. Hsu, W.S. Lee. *equal contribution AAAI Conference on Artificial Intelligence (AAAI), 2020
Beyond Tabula-Rasa: A Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban [pdf] [website] P. Karkus, M. Mirza, A. Guez, A. Jaegle, T. Lillicrap, L. Buesing, N. Heess, T. Weber. arXiv preprint arXiv:2010.01298, 2020
Physically Embedded Planning Problems: New Challenges for Reinforcement Learning [pdf] [code] M. Mirza, A. Jaegle, J.J. Hunt, A. Guez, S. Tunyasuvunakool, A. Muldal, T. Weber, P. Karkus, S. Racanière, L. Buesing, T. Lillicrap, N. Heess. arXiv preprint arXiv:2009.05524, 2020
Differentiable Algorithm Networks for Composable Robot Learning [pdf] [talk] P. Karkus, X. Ma, D. Hsu, L.P. Kaelbling, W.S. Lee, T. Lozano-Pérez. Robotics: Science and Systems (RSS), 2019 Best paper nomination (top 5 papers).
Factored Contextual Policy Search with Bayesian Optimization [pdf] R. Pinsler*, P. Karkus*, A. Kupcsik, D. Hsu, W.S. Lee. *equal contribution International Conference on Robotics and Automation (ICRA), 2019
Particle Filter Networks with Application to Visual Localization [pdf] [code] P. Karkus, D. Hsu, W.S. Lee. Conference on Robot Learning (CoRL), 2018
QMDP-Net: Deep Learning for Planning under Partial Observability [pdf] [code] P. Karkus, D. Hsu, W.S. Lee. Advances in Neural Information Processing Systems (NeurIPS), 2017
Generation of Entangled Photons Using an Arrayed Waveguide Grating [pdf] N. Matsuda, H. Nishi, P. Karkus, T. Tsuchizawa, K. Yamada, W.J. Munro, K. Shimizu, H. Takesue. Journal of Optics 19(12), 2017
Recursive Constraint Manifold Subsearch for Multirobot Path Planning with Cooperative Tasks [pdf] P. Karkus, G. Wagner, H. Choset. Annual Symposium on Combinatorial Search (SOCS), 2016
On-Chip Generation and Demultiplexing of Quantum Correlated Photons Using Silicon-Silica Monolithic Waveguide Platform [pdf] N. Matsuda, P. Karkus, H. Nishi, T. Tsuchizawa, W.J. Munro, H. Takesue, K. Yamada. European Conference on Optical Communication (ECOC), 2014
On-Chip Generation and Demultiplexing of Quantum Correlated Photons Using a Silicon-Silica Monolithic Photonic Integration Platform [pdf] N. Matsuda, P. Karkus, H. Nishi, T. Tsuchizawa, W.J. Munro, H. Takesue, K. Yamada. Optics Express 22(19), 2014
Designing the Hardware and Software System of a RoboCup Robot Soccer Player P. Karkus, D. Kiss. MicroCAD International Scientific Conference G/2 (MicroCAD), 2013
The RobonAUT Robot Construction Contest in 2013 T.G. Kiss, L.C. Burkodi, P. Karkus, D. Varga, I. Vajk. Advances in Control Education (ACE), 2013
Team Organization and Participation in the Eurobot 2012 Contest L.C. Burkodi, T.G. Kiss, P. Karkus, D. Varga, I. Vajk. Advances in Control Education (ACE), 2013