Robotics and Robot Vision
Introduction
What Question Does It Ask?
Robotics studies how to design and control machines that sense and move in the physical world. The course combines perception, localization, planning and control under noisy measurements and real-time constraints; robot vision is the eye of this loop.
At a glance
sense-plan-act loop
sensor measurements
motor commands, path
task success, safety
high: transformations, control
physical experiments and simulation
Introduction
A Short History
Unimate, the first industrial robot, starts work at a General Motors plant.
The Shakey project starts at SRI: the first mobile robot to sense and plan; A* is developed for it.
Brooks proposes the subsumption architecture: direct sense-act behaviors instead of planning.
The Sojourner rover explores the surface of Mars on the Mars Pathfinder mission.
The Roomba vacuum goes on sale; mobile robots enter homes.
Stanford's Stanley completes and wins the DARPA Grand Challenge desert race with no driver.
The Course
Core Concepts
The number of axes along which a robot can move independently.
World, robot, camera; transformation matrices between them.
Finding the end-effector position from joint angles.
Finding the joint angles for a desired end-effector position.
Feedback that corrects error by proportional, integral and derivative terms.
Position from noisy measurements with Kalman and particle filters.
Building a map while locating yourself in it.
A*, RRT: a collision-free path to the goal.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Position and orientation: rotation matrices, frames
- Forward kinematics and Denavit-Hartenberg notation
- Inverse kinematics and the Jacobian
- Robot dynamics: forces, torques, equations of motion
- Feedback control: PID and stability
- Trajectory generation and manipulator control
- Mobile robot kinematics and odometry
- Sensors: cameras, lidar, IMU, encoders
- Probabilistic state estimation: the Kalman filter
- Particle filters and Monte Carlo localization
- Mapping and SLAM
- Motion planning: A*, probabilistic roadmaps, RRT
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
A Robot Arm Picks Up a Part
The tip of a two-joint arm on an assembly line must reach a part on the table, but motors take joint angles, not tip positions. Click a target: inverse kinematics computes the angles and the arm turns there.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Linear Algebra | coordinate transforms, kinematics |
| Physics (mechanics) | force, torque, velocity and acceleration: the language of robot dynamics |
| Microprocessors | reading sensors, driving motors and real-time software |
| Programming with C | embedded control software is mostly written in C/C++ |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you move a 2D point into another frame with a rotation matrix? | Linear Algebra |
| Can you explain velocity, acceleration and torque with an example? | Mechanics from physics: Newton's laws and rotational motion |
| Can you explain what the mean and variance of a Gaussian mean? | Probability and Statistics |
| Can you write a class in C++ or Python and process sensor data in a loop? | An intro C course, then a short introduction to classes in Python |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| builds on | This course builds on Linear Algebra | coordinate transforms, kinematics |
| shares methods | Shares methods with Reinforcement Learning | control, learning motor skills |
| shares methods | Shares methods with Fuzzy Logic and Fuzzy Systems | fuzzy controllers |
| shares methods | Shares methods with Multi-Agent Systems | swarm robotics, coordination |
| shares methods | Shares methods with Computer Vision | robot vision, SLAM |
| shares methods | Shares methods with Embedded AI | real-time on-device inference |
Differences
Where It Differs (1/2)
| Criterion | Robotics | Computer Vision | Artificial Intelligence |
|---|---|---|---|
| Core question | How does a machine sense the physical world, know where it is and move? | How does a computer understand what is in an image or a video? | How does a machine act intelligently by searching, reasoning and planning? |
| Approach | sense-plan-act loop | extracting meaning from images | search, logic, probability and learning combined |
| Input | sensor measurements | image or video | problem definition: states, actions, goal |
| Output | motor commands, path | label, box, depth, 3D structure | action sequence, plan or decision |
| Success measure | task success, safety | accuracy, mAP, IoU | reaching the goal, cost, rationality |
| Math load | high: transformations, control | high: geometry, linear algebra | medium: logic, probability, graphs |
| Data needs | physical experiments and simulation | many labeled images | little; often a model and knowledge suffice |
- Computer Vision: Computer vision understands what is in a scene; robot vision turns the image into position and motion decisions.
- Artificial Intelligence: AI plans over abstract states; robotics carries out the plan in a continuous, noisy, physical world.
Differences
Where It Differs (2/2)
| Criterion | Robotics | Reinforcement Learning | Embedded AI |
|---|---|---|---|
| Core question | How does a machine sense the physical world, know where it is and move? | How can good decisions be learned by trial and error from a reward signal alone? | How do AI models run on small, low-power devices? |
| Approach | sense-plan-act loop | trial and error with rewards | resource-constrained inference |
| Input | sensor measurements | environment: state, reward | trained model, device constraints |
| Output | motor commands, path | policy: state → action | small, fast model |
| Success measure | task success, safety | total expected reward | latency, energy, memory |
| Math load | high: transformations, control | high: probability, dynamic programming | medium |
| Data needs | physical experiments and simulation | generated through interaction | training data already exists |
- Reinforcement Learning: Reinforcement learning learns control by trial and error; robotics mostly designs it from models and control theory.
- Embedded AI: Embedded AI fits the model onto a small device; robotics ties that device into a loop with sensors and motors.
Practice
Applications
Welding, painting, assembly lines.
Moving shelves and picking orders.
Perception, localization and planning.
Making the surgeon's motion more precise.
Mars rovers, autonomous driving at a distance.
Robot vacuums and lawn mowers.
Practice
Risks and Ethics
Robot arms exert large forces; speed and force limits, emergency stops and safety zones are part of collaborative robot design.
A dirty lidar, a camera blind spot or a spoofed GPS signal can lead a robot confidently to the wrong place.
In crashes such as the 2018 Uber test vehicle that killed a pedestrian in Arizona, blame is split between maker, operator and driver.
Camera-equipped mobile robots and drones open private spaces to monitoring; autonomous weapons are a separate international debate.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Robotics is only mechanics and electronics. | Today's hard problems are mostly in software: perception, localization and planning. |
| A good plan gets the robot to the goal. | Wheels slip and sensors are noisy; a plan holds only with continuous feedback. |
| What works in simulation works in reality. | The sim-to-real gap is a research topic of its own in robotics. |
| Robot vision is the same as computer vision. | Robot vision stresses depth, pose and real time; its output is a motion. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Siegwart, Nourbakhsh and Scaramuzza, Introduction to Autonomous Mobile Robots, 2nd ed., MIT Press, 2011.
- Thrun, Burgard and Fox, Probabilistic Robotics, MIT Press, 2005.
- Lynch and Park, Modern Robotics: Mechanics, Planning, and Control, Cambridge University Press, 2017.
- Corke, Robotics, Vision and Control, 2nd ed., Springer, 2017.