📄️ Chapter 22: Introduction to VLA Models
This chapter introduces Vision-Language-Action (VLA) models, explaining what they are, contrasting traditional robotics with VLA-powered systems, and highlighting their ability to enable general-purpose robots. It also traces the historical evolution of VLAs, covering RT-1, RT-2, and beyond, and discusses open-source initiatives like OpenVLA alongside the current state-of-the-art models.
📄️ Chapter 23: VLA Architecture and Design
This chapter focuses on VLA architecture and design, detailing multi-modal transformer architectures, visual encoders (such as DINOv2, SigLIP, CLIP), language models for robotics (like LLaMA and Phi), action decoders, end-to-end learning pipelines, and design choices between single-model and dual-system approaches.
📄️ Chapter 24: Training VLA Models
This chapter covers training VLA models, including the selection of training datasets and benchmarks, data collection from real robots and simulations, imitation learning and behavior cloning, fine-tuning of pretrained models, multi-task learning across diverse demonstrations, and computational scaling requirements.
📄️ Chapter 25: Deploying VLA Models on Humanoid Robots
This chapter examines deploying VLA models on humanoid robots, discussing OpenVLA-7B for real-world robotics, domain-specific fine-tuning, quantization and model compression, edge deployment on devices like NVIDIA Jetson, latency optimization for real-time control, and strategies for handling failures and fallback behaviors.
📄️ Chapter 26: Vision and Language Understanding in Robotics
This chapter delves into vision and language understanding in robotics, including scene understanding from RGB images, semantic segmentation for object recognition, 3D spatial reasoning, interpreting natural language instructions, compositional understanding of complex tasks, and grounding language in the physical world.
📄️ Quickstart: Getting Started with Vision-Language-Action (VLA) Systems
This quickstart guide provides a brief introduction to understanding and conceptually experimenting with Vision-Language-Action (VLA) systems in robotics. Due to the rapid evolution and diverse implementations of VLA models, specific installation and execution steps can vary greatly. This guide focuses on the general approach and directs you to where to find detailed, up-to-date instructions.