Robotics Journey
A roadmap into robotics simulation and sim-to-real, in order: every study session and the deliverable that closes each stage. The red line marks today and shows where the plan expects progress to be. This is a plan, not a course: it comes with no learning materials of its own. I find the materials for every topic myself, in open and paid courses, and I share whatever I can on my social media along the way.
- Simulation engineers build the virtual worlds robots are developed and tested in: physics models, simulated sensors, and the tooling that runs them at scale. Sim-to-real is the part that checks those worlds against reality and says how far to trust them. Sensor virtualization makes simulated data reach other software as if it came from a real device.
- Learn to model how robotic systems behave, check simulations against real recorded data, and build software tools to test them. The focus is simulation engineering and sim-to-real, not all of robotics, and everything runs without physical hardware.
- About 20 hours a week for six months: 24 weeks of study and 2 weeks of reserve, roughly 520 hours. If time runs short, the scope shrinks; the plan does not grow. Every date on this page is an estimate.
Roadmap
Dates are estimates at 20 hours a week (currently ending around May 18, 2027) and may be adjusted as the work progresses. A stage ends when its deliverable meets its criteria, not on a date.
Simulation Foundations
Every robot simulator, however large, is a loop: take the state, apply the physics for one small timestep, repeat. This stage builds that loop by hand for a two-wheeled robot, so later tools are not black boxes. C++ and Python are refreshed selectively, not learned from zero.
- Vectors, the dot product, 2×2 and 3×3 matrices, radians, rotations, local and world coordinate frames, composing transforms.
- Position, velocity, acceleration; differential-drive kinematics.
- Euler and Runge–Kutta 4 integration, timestep, numerical error and stability.
- C++20 with CMake and tests for the model; Python with NumPy and Matplotlib for analysis.
Not in this stage: Manipulator dynamics, contact solvers, optimal control theory, a full linear algebra course, 3D quaternions (they come in Stage 2).
CMake project and compiler warnings
Set up a C++20 project that builds with one CMake command and treats compiler warnings as signals, not noise.
C++ structs, values and RAII
Refresh value types, references, and RAII: the parts of C++ a simulator leans on every day.
Vec2 operations and numerical checks
Vectors and the dot product in code, with tests that check results within a numerical tolerance.
Pose2, radians and rotation matrices
A 2D pose as position plus heading: radians, 2×2 rotation matrices, and moving between local and world frames.
Differential drive: forward motion
Kinematics of a two-wheeled robot: how equal wheel speeds turn into straight-line motion.
Wheel-speed kinematics and turns
Different wheel speeds give an angular velocity; the robot follows an arc around a turning centre.
Euler integration and timestep
Step the motion forward in time with Euler integration and see how the timestep size creates error.
RK4 comparison on curved paths
Runge–Kutta 4 against Euler on curved paths: accuracy, cost, and stability for the same timestep.
CSV output and Python plots
The C++ simulator writes CSV; NumPy and Matplotlib turn it into trajectory and error plots.
Reproducible run, tests and stage review
Same parameters, same result: tests, a clean rebuild, and a review against the stage criteria.
2D differential-drive simulator
Wheel speeds, wheel radius, wheelbase, timestep, and duration in; a CSV of time, x, y, heading and plots out. The model is C++, the analysis Python.
- To do: Clean build with one CMake command
- To do: Drives straight with equal wheel speeds and turns with different ones
- To do: Coordinate transforms pass simple checks
- To do: Repeated runs with the same parameters give the same result
- To do: Plots and an explanation of how the timestep affects the error
Robotics Simulation
The everyday toolset of robotics simulation: describe a robot as a model with frames, joints, mass, and contacts, put it in a physics engine, drive it through ROS 2, and give it simulated sensors. The result is a digital robot prototype whose behavior can be recorded, changed, and explained.
- 3D rotations and rigid transforms (SO(3), SE(3)), quaternions, frame trees, forward kinematics, the pinhole camera model.
- Mass and inertia, friction, restitution, contact stability, generalized coordinates.
- Simulated camera, IMU, and LiDAR, with configurable noise.
- ROS 2 (topics, QoS, tf2, rosbag2), Gazebo and SDF, MuJoCo and MJCF, URDF/Xacro, one small CAD-to-simulation exercise, Rerun and Foxglove for visual debugging.
Not in this stage: A separate CAD course, deep SLAM and navigation, reinforcement learning.
SO(3), SE(3) and Rodrigues formula
Rotations and rigid-body transforms in 3D: the groups SO(3) and SE(3), and the axis–angle Rodrigues formula.
Quaternion rotations and frame inverses
Quaternions as a compact, stable way to store rotations; composing and inverting frame transforms.
TF tree and forward kinematics
A robot as a tree of coordinate frames; forward kinematics chains transforms from the base to each part.
3D frame visualization in Rerun
Draw frames and trajectories in Rerun so 3D mistakes become visible instead of hiding in numbers.
ROS 2 topics, QoS and tf2 diagnostics
How ROS 2 nodes talk: topics, quality-of-service settings, and diagnosing the tf2 frame tree.
URDF/Xacro robot model and joint frames
Describe a robot in URDF/Xacro: links, joints, and the frames between them.
Build123d CAD-as-code: one robot part
Model one robot part as code with Build123d, so geometry is versioned and reproducible.
Trimesh mass, volume and inertia check
Compute volume, mass, and the inertia tensor from a mesh with Trimesh, and check them against the model.
Collision geometry and model validation
Simplified collision shapes versus visual meshes, and checks that the model loads and behaves sensibly.
CAD-to-URDF reproducible pipeline
One script from CAD part to URDF, so the simulated robot can be rebuilt from source.
Gazebo SDF world and contact parameters
Build a Gazebo world in SDF and look at the contact parameters that decide how bodies touch.
Gazebo friction and restitution experiment
Change friction and bounciness (restitution) and measure how the motion changes.
ros_gz_bridge and cmd_vel control
Connect ROS 2 to Gazebo with ros_gz_bridge and drive the robot with velocity commands.
ros2_control architecture: guided exploration
A guided look at ros2_control: how controllers, hardware interfaces, and the simulator fit together.
Contact stability and repeatable baseline
Find contact settings that do not jitter or explode, and save them as a repeatable baseline.
ROS 2–Gazebo bridge
Carry sensor and control topics both ways between ROS 2 and Gazebo, and check what arrives.
Simulation clock and reset
Simulated time versus wall-clock time, and resetting a scenario to a known starting state.
rosbag2 record and playback
Record topics with rosbag2 and play them back: the basis of every later experiment.
Timestamp and frame consistency
Check that every message has a sensible timestamp and frame, before trusting any data.
Reproducible dataset run
Record the same scenario twice and confirm the two runs give the same data.
Virtual camera setup
Add a simulated camera to the robot and publish its images.
Camera intrinsics and image topics
The pinhole camera model: focal length, principal point, and how they appear in ROS image topics.
Camera frame transformations
Project 3D points into the image and move them between the camera, robot, and world frames.
Image recording and visualization
Record camera streams and inspect them frame by frame.
Foxglove sensor stream debugging
Use Foxglove to inspect live and recorded sensor streams side by side.
IMU orientation and acceleration
A simulated IMU: what orientation, angular velocity, and linear acceleration really measure, gravity included.
LiDAR scan geometry
How a LiDAR scan is built from rays: angles, ranges, and the geometry of a scan line.
GPU LiDAR and raycasting concepts
How GPU raycasting simulates LiDAR, and what it simplifies compared with a real sensor.
Noise configuration and data checks
Add sensor noise in the simulator and check the output statistics match the configuration.
Multisensor scenario
Run camera, IMU, and LiDAR together in one recorded scenario.
MuJoCo MJCF and generalized coordinates
Describe a model in MuJoCo’s MJCF and see how it represents motion with generalized coordinates.
Basic actuators and motion
Drive joints with MuJoCo actuators and watch the resulting motion.
MuJoCo contact solref/solimp experiment
Tune MuJoCo’s soft-contact parameters (solref, solimp) and measure their effect.
Compare equivalent scenarios with Gazebo
Run the same scenario in MuJoCo and Gazebo and explain where and why they differ.
Stage 2 demonstration
Demonstrate the digital robot prototype against the stage criteria.
Digital robot prototype
A simulated robot with control, a camera, an IMU, and a LiDAR, used as a test fixture for physics and sensors rather than a hardware design.
- To do: Controlled robot with a camera, an IMU, and a LiDAR
- To do: TF tree, URDF/Xacro, mass and inertia, and contact parameters checked
- To do: A reproducible experiment, run and recorded
- To do: Change physical parameters and explain how the system behaves
Sim-to-Real Engineering
A simulation is only useful if you know how far to trust it. Sim-to-real is the work of measuring the reality gap: compare the model with real recorded data, estimate its parameters, model sensor noise and delays, and state where the simulation holds and where it does not. Public datasets stand in for hardware; this is not presented as testing on a physical robot.
- System identification with least squares and optimization, frame and time alignment, one narrow extended Kalman filter (EKF) and one ICP point-cloud experiment.
- Error distributions, train and held-out validation splits, absolute trajectory error (ATE), uncertainty, robustness across random seeds.
- Camera intrinsics and extrinsics, IMU white noise and bias random walk, Allan variance, dropout, outliers, latency, jitter.
- Calibration, sensitivity analysis, domain randomization, controlled experiment matrices.
Not in this stage: A full state-estimation or SLAM course.
Choose a public robot/sensor dataset
Pick a public dataset of real robot or sensor recordings to stand in for hardware.
Inspect dataset units and metadata
Read the dataset carefully: units, frames, timestamps, and sensor setup.
ATE and sensor fidelity metrics
Absolute trajectory error (ATE) and sensor metrics: how “close to reality” becomes a number.
Build a simulated baseline for comparison
Recreate the dataset’s scenario in simulation as the baseline to compare against.
Document mismatches and assumptions
Write down every known difference between the simulation and the recording.
Parameter estimation concept
System identification: estimating model parameters from recorded data with least squares.
Fit wheel radius from recorded motion
Estimate the effective wheel radius from recorded motion with an optimizer.
Fit wheelbase or turning parameter
Estimate the wheelbase, the parameter that decides how sharply the robot turns.
Train/validation split by runs
Fit on some recordings and check on others, so the model is not just memorizing.
Parameter fit report
Report the fitted parameters, their error, and how much to trust them.
Friction and slip sensitivity
How much results change when friction and wheel slip change.
Mass/inertia sensitivity
How much results change when mass and inertia are off.
Parameter sweep runner
A small runner that tries many parameter combinations and collects the results.
Compare error distributions
Compare whole error distributions, not single numbers.
Identify unobservable parameters
Find parameters the data cannot pin down, and stop pretending to know them.
Camera intrinsic calibration concepts
How intrinsic camera calibration works, and what a calibrated model gets right.
Extrinsics and coordinate frames
Extrinsics: where each sensor sits relative to the robot, and how errors there spread.
IMU white noise, random walk and Allan variance
IMU error models: white noise, bias random walk, and reading them from an Allan variance plot.
Timestamp offsets
Estimate and correct time offsets between sensors.
Calibration report and limitations
Report calibration results and what they cannot guarantee.
Gaussian noise and bias
Model sensor noise as Gaussian noise plus bias, and check the model against data.
Allan deviation on recorded IMU data
Compute the Allan deviation on real IMU recordings and compare with the simulated IMU.
Sensor dropout and outliers
Simulate missing samples and outliers, the failures real sensors actually have.
Latency and jitter
Model sensor latency and timing jitter, and measure their effect.
Noise model validation against recorded data
Check the noise models against recorded data, not just against intuition.
Select plausible parameter ranges
Choose realistic parameter ranges for domain randomization from the data.
Randomize friction and mass
Domain randomization: vary friction and mass across runs.
Randomize lighting and camera noise
Vary lighting and camera noise across runs.
Run a controlled experiment matrix
Run a planned matrix of experiments, changing one factor at a time.
Quantify robustness across seeds
Repeat runs with different random seeds and measure the spread.
Separate baseline and validation scenes
Keep tuning scenes and test scenes apart.
Create deterministic scenario definitions
Define scenarios in files so every run is exactly repeatable.
ATE trajectory evaluation and tolerances
Evaluate trajectories with ATE and set the tolerances that count as a pass.
Investigate the worst-case mismatch
Dig into the run where simulation and reality disagree most.
Write an evidence-based validity claim
State where the simulation is valid, with the evidence for each claim.
Compare trajectories on held-out data
Compare simulated and recorded trajectories on data never used for tuning.
Align coordinate frames and time
Align frames and timestamps before comparing anything.
EKF with robot_localization: limited experiment
One narrow experiment with an extended Kalman filter (robot_localization), not a full estimation course.
Open3D ICP: limited point cloud experiment
One narrow point-cloud alignment experiment with ICP in Open3D.
Error breakdown visualization
Plots that show where the error comes from.
Rebuild the experiment from scratch
Rebuild the whole experiment from a clean checkout to prove it is reproducible.
Tune against training recordings
Tune the model on the training recordings only.
Evaluate on an unseen recording
Evaluate the tuned model on a recording it has never seen.
Reproducibility and uncertainty report
Report how reproducible the results are and how uncertain.
Stage 3 review and limitations
Review the stage and write down its limits honestly.
Held-out sim-to-real evaluation
The simulation is tuned on some real recordings and judged on others it has never seen. A dataset comparison does not replace a test on a physical robot.
- To do: Model compared with real recorded data, with timestamps and frames aligned
- To do: Parameters adjusted and the improvement measured (ATE or a sensor metric)
- To do: Effect of noise and latency estimated
- To do: Documented where the simulation is valid, where it is not, and what shows it
Simulation Infrastructure & Validation
Simulation at scale is software engineering: run scenarios headless and in batches, replay recordings, compare against a baseline, and catch regressions in CI. Sensor virtualization makes simulated data look like a real device to other programs. The stage produces two tools: a Simulation Reliability Toolkit and a Virtual Sensor Bridge.
- Headless and batch runs, scenario configuration, seeds, run metadata, recording and replay, CI with GitHub Actions, profiling.
- A simulated Gazebo camera delivered through ROS 2 or as a Linux V4L2 device, with frame rate, format, timestamps, latency, and dropped frames measured.
- Baseline metrics and thresholds, regression tests, noisy and failure scenarios, machine-readable reports.
Not in this stage: Other sensor types in the bridge unless a scenario needs them. Everything runs without physical hardware.
Gazebo headless execution
Run Gazebo without a screen, as a test machine or CI would.
Command-line scenario configuration
Choose scenarios and parameters from the command line.
Batch runner with seeds
Run many scenarios in a batch, each with a recorded random seed.
Run isolation and output folders
Keep every run isolated, with its own output folder.
Baseline batch execution
Run the full batch once to produce the reference baseline.
Standardize run metadata
Record what produced each result: version, parameters, seed, and time.
rosbag2 replay harness
Replay recordings automatically into the system under test.
Compare recorded and simulated messages
Compare recorded and simulated messages field by field.
Detect missing or late messages
Detect missing or late messages automatically.
Replay reproducibility report
Report how repeatable replays are.
Select a camera interface and consumer
Choose how the virtual camera is delivered and which program consumes it.
Gazebo camera ROS image stream
Publish the Gazebo camera as a ROS 2 image stream.
Bridge images to a consumer via ROS 2
Deliver those images to a consumer program through ROS 2.
Validate encoding and frame rate
Check the image encoding and the frame rate the consumer actually gets.
Virtual Sensor Bridge: measurable camera demo
A camera demo with measured numbers, not just a picture on screen.
Choose a Linux V4L2 option if the environment supports it
Decide whether Linux V4L2 can present the virtual camera as a normal webcam here.
Implement or configure a V4L2 loopback bridge
Set up v4l2loopback so any program can read the simulated camera as a webcam.
Bridge latency, dropped frames and timestamp fidelity
Measure the bridge: latency, dropped frames, and whether timestamps survive.
Fallback ROS 2 consumer compatibility
Keep a pure ROS 2 path working for where V4L2 is not available.
Bridge interface and troubleshooting docs
Document the bridge interface and how to troubleshoot it.
Virtual Sensor Bridge
A simulated camera presented to other programs through a standard interface, as if it were a real device.
- To do: A virtual camera stream from Gazebo reaches a consumer through ROS 2 or Linux V4L2
- To do: Frame rate, format, timestamps, latency, and dropped frames measured
Define baseline metrics and thresholds
Decide which metrics matter and what change counts as a regression.
Scenario regression test harness
Turn scenarios into automated regression tests.
Test noisy and failure scenarios
Test noisy and failing scenarios too, not just the happy path.
Produce a machine-readable report
Produce a report that other tools and CI can read.
CI integration
Run the tests in GitHub Actions on every change.
Unify runner, replay and bridge
Join the runner, replay, and bridge into one tool.
Config schema and validation
A configuration schema that rejects bad input early.
Profiling CPU and memory
Profile CPU and memory and fix the worst hotspots.
Documentation and one-command demo
Documentation and a demo that runs with one command.
End-to-end acceptance test
One test that exercises the whole toolkit end to end.
Simulation Reliability Toolkit
A tool that runs simulation scenarios, collects results, compares them with a baseline, and flags regressions.
- To do: An experiment reproduces with one command
- To do: Results are compared with a baseline using stated criteria, and a report is produced
- To do: Regressions are detected automatically
Buffer & Final Review
Two weeks kept free on purpose. They absorb whatever ran late, then close the plan: a clean install on a fresh machine, a recorded demo, a published technical README, and a final audit of every deliverable against this roadmap.
Revisit the weakest prerequisite
Go back to the topic that held things up the most.
Fix flaky Gazebo or ROS configuration
Fix configuration that fails only sometimes.
Re-run a failed sensor experiment
Re-run a sensor experiment that did not work the first time.
Strengthen tests and diagnostics
Add the tests and diagnostics that were missing.
Verify the reproducible demo
Check the demo still reproduces from scratch.
Improve the held-out dataset evaluation
Improve the evaluation on held-out data.
Resolve remaining bridge issues
Close the remaining Virtual Sensor Bridge issues.
Independent clean install test
Install everything on a clean machine, following only the README.
Record a demo and publish the technical README
Record a demo and publish the technical README.
Final audit against the roadmap
Check every deliverable against this roadmap.