From Physical World to Autonomous Machine

Every autonomous vehicle, from a campus shuttle to the cargo tug at Harbor Yard, runs the same pipeline. A physical place exists. Someone captures it. That captured data becomes a digital representation. The representation is tagged with meaning. Rules are added. A planner reads the result. The vehicle moves.

That is seven stages. The rest of this book unpacks them one at a time. This chapter is the view from altitude, the one you come back to when you lose track of where a particular format or tool fits.

The seven stages

Reality. The physical world, as it is. Harbor Yard on a Tuesday morning. The asphalt, the docks, the faded stop line, the container that was moved yesterday and is now blocking what used to be a clear path. Reality is the ground truth and it does not care about your map.

Capture. Turning reality into data. A LiDAR scanner mounted on a vehicle or carried by hand. A camera. A drone. A surveyor’s total station. Each capture method records a slice of reality, from one vantage point, at one moment. The output is raw: a point cloud, a set of images, a GPS trace.

Representation. Structuring the captured data into a digital model. A point cloud becomes registered and cleaned. A vector layer is traced. An OpenStreetMap extract provides the road skeleton. This is where raw sensor data becomes something a computer can work with.

Semantics. Adding meaning to geometry. This cluster of points is a building. This line is a lane boundary. This polygon is a crosswalk. Semantics turn a 3D model into a map a robot can reason about, rather than just render.

Rules. Encoding what is allowed and what is not. The speed limit in this zone. The right-of-way at this intersection. The access restriction on this gate. Rules are the difference between a map that describes a place and a map that governs behavior within it.

Planning. Reading the map to produce a path. The planner asks: given where I am, where I need to go, and what the map says about this place, what is the safest, most efficient sequence of motions? Planning is where the map proves its worth.

Motion. The vehicle executes. Motors, steering, braking. The map’s final test is whether the robot moves correctly through the real world, safely, on the first attempt.

Why the order matters

The stages flow from Reality to Motion, in order. Each stage consumes the output of the one before it. You cannot add rules to a map you have not represented. You cannot represent what you have not captured.

But the relationship goes the other way too. Decisions made at the planning stage reveal gaps in the rules. A rule that seemed clear on paper turns out to be ambiguous when a planner tries to apply it at 10 Hz. Validation, the subject of Part VI, is the discipline of running the pipeline backwards: start with motion, find where it breaks, and trace the failure upstream to its source.

Harbor Yard through the pipeline

Let us run Harbor Yard through all seven stages, quickly.

The yard exists (Reality). A mobile scanner drives the loop, capturing a point cloud of the asphalt, the docks, the parked containers, and the gate (Capture). The raw point cloud is cleaned, registered to GNSS coordinates, and a vector layer is traced on top of it: the lane centerlines, the dock faces, the loop geometry (Representation). Each traced element is tagged: this is a loading lane, this is a one-way segment, this is a keep-clear zone (Semantics). The speed limit is set to 5 mph, the one-way direction is encoded, and the gate is marked as requiring authorization (Rules). Autoware’s planner reads the resulting Lanelet2 map and produces a route from Dock A to the yard exit (Planning). The cargo tug drives it (Motion).

Seven stages, end to end. That is the whole game.

Where each Part fits

Stage	Handbook Part
Reality	Part I: The Big Picture
Capture	Part III: Scanning Your Site
Representation	Part IV: Describing the World
Semantics	Part IV: Describing the World
Rules	Part V: Rules of the Road
Planning	Part VII: From Map to Autonomy
Motion	Part VII: From Map to Autonomy

Part II, Coordinate Systems, sits underneath the whole pipeline: every stage needs a shared address system. The formats themselves live in Parts IV and V, where geometry becomes a map a robot can read. Part VI, Building HD Maps With Veer Studio, is the practical application of stages 2 through 5. Part VII, From Map to Autonomy, is where that finished map meets the robot: Autoware loads it, localizes and perceives against it, plans through it, and when the robot misbehaves, you trace the failure back into the map.

What comes next

Part I is complete. You now know what an HD map is (Chapter 1), the four questions every robot answers (Chapter 2), why multiple representations coexist (Chapter 3), and the pipeline that connects physical reality to machine motion (this chapter). That is the conceptual frame for everything that follows.

Part II begins the foundations: coordinates. Because before anything else, we need to agree on what “where” means.