Hands-On Vision and Behavior for Self-Driving Cars

Hands-On Vision and Behavior for Self-Driving Cars

by Luca Venturi, Krishtof Korda
Released October 2020
Publisher(s): Packt Publishing
ISBN: 9781800203587

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

A practical guide to learning visual perception for self-driving cars for computer vision and autonomous system engineers

Key Features

  • Explore the building blocks of the visual perception system in self-driving cars
  • Identify objects and lanes to define the boundary of driving surfaces using open-source tools like OpenCV and Python
  • Improve the object detection and classification capabilities of systems with the help of neural networks

Book Description

The visual perception capabilities of a self-driving car are powered by computer vision. The work relating to self-driving cars can be broadly classified into three components - robotics, computer vision, and machine learning. This book provides existing computer vision engineers and developers with the unique opportunity to be associated with this booming field.

You will learn about computer vision, deep learning, and depth perception applied to driverless cars. The book provides a structured and thorough introduction, as making a real self-driving car is a huge cross-functional effort. As you progress, you will cover relevant cases with working code, before going on to understand how to use OpenCV, TensorFlow and Keras to analyze video streaming from car cameras. Later, you will learn how to interpret and make the most of lidars (light detection and ranging) to identify obstacles and localize your position. You'll even be able to tackle core challenges in self-driving cars such as finding lanes, detecting pedestrian and crossing lights, performing semantic segmentation, and writing a PID controller.

By the end of this book, you'll be equipped with the skills you need to write code for a self-driving car running in a driverless car simulator, and be able to tackle various challenges faced by autonomous car engineers.

What you will learn

  • Understand how to perform camera calibration
  • Become well-versed with how lane detection works in self-driving cars using OpenCV
  • Explore behavioral cloning by self-driving in a video-game simulator
  • Get to grips with using lidars
  • Discover how to configure the controls for autonomous vehicles
  • Use object detection and semantic segmentation to locate lanes, cars, and pedestrians
  • Write a PID controller to control a self-driving car running in a simulator

Who this book is for

This book is for software engineers who are interested in learning about technologies that drive the autonomous car revolution. Although basic knowledge of computer vision and Python programming is required, prior knowledge of advanced deep learning and how to use sensors (lidar) is not needed.

Table of contents

  1. Hands-On Vision and Behavior for Self-Driving Cars
  2. Why subscribe?
  3. Contributors
  4. About the authors
  5. About the reviewer
  6. Packt is searching for authors like you
  7. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Code in Action
    6. Download the color images
    7. Conventions used
    8. Get in touch
    9. Reviews
  8. Section 1: OpenCV and Sensors and Signals
  9. Chapter 1: OpenCV Basics and Camera Calibration
    1. Technical requirements
    2. Introduction to OpenCV and NumPy
      1. OpenCV and NumPy
      2. Image size
      3. Grayscale images
      4. RGB images
    3. Working with image files
    4. Working with video files
      1. Working with webcams
    5. Manipulating images
      1. Flipping an image
      2. Blurring an image
      3. Changing contrast, brightness, and gamma
      4. Drawing rectangles and text
    6. Pedestrian detection using HOG
      1. Sliding window
      2. Using HOG with OpenCV
      3. Introduction to the camera
      4. Camera terminology
      5. The components of a camera
      6. Considerations for choosing a camera
      7. Strengths and weaknesses of cameras
    7. Camera calibration with OpenCV
      1. Distortion detection
      2. Calibration
    8. Summary
    9. Questions
  10. Chapter 2: Understanding and Working with Signals
    1. Technical requirements
    2. Understanding signal types
    3. Analog versus digital
    4. Serial versus parallel
      1. Universal Asynchronous Receive and Transmit (UART)
      2. Differential versus single-ended
      3. I2C
      4. SPI
    5. Framed-based serial protocols
      1. Understanding CAN
      2. Ethernet and internet protocols
      3. Understanding UDP
      4. Understanding TCP
    6. Summary
    7. Questions
    8. Further reading
      1. Open source protocol tools
  11. Chapter 3: Lane Detection
    1. Technical requirements
    2. How to perform thresholding
      1. How thresholding works on different color spaces
      2. RGB/BGR
      3. HLS
      4. HSV
      5. LAB
      6. YCbCr
      7. Our choice
    3. Perspective correction
    4. Edge detection
      1. Interpolated threshold
      2. Combined threshold
    5. Finding the lanes using histograms
    6. The sliding window algorithm
      1. Initialization
      2. Coordinates of the sliding windows
      3. Polynomial fitting
    7. Enhancing a video
      1. Partial histogram
    8. Rolling average
    9. Summary
    10. Questions
  12. Section 2: Improving How the Self-Driving Car Works with Deep Learning and Neural Networks
  13. Chapter 4: Deep Learning with Neural Networks
    1. Technical requirements
    2. Understanding machine learning and neural networks
      1. Neural networks
      2. Neurons
      3. Parameters
      4. The success of deep learning
    3. Learning about convolutional neural networks
      1. Convolutions
      2. Why are convolutions so great?
    4. Getting started with Keras and TensorFlow
      1. Requirements
    5. Detecting MNIST handwritten digits
      1. What did we just load?
      2. Training samples and labels
      3. One-hot encoding
      4. Training and testing datasets
    6. Defining the model of the neural network
      1. LeNet
      2. The code
      3. The architecture
      4. Training a neural network
      5. CIFAR-10
    7. Summary
    8. Questions
    9. Further reading
  14. Chapter 5: Deep Learning Workflow
    1. Technical requirements
    2. Obtaining the dataset
      1. Datasets in the Keras module
      2. Existing datasets
      3. Your custom dataset
    3. Understanding the three datasets
      1. Splitting the dataset
    4. Understanding classifiers
      1. Creating a real-world dataset
      2. Data augmentation
    5. The model
      1. Tuning convolutional layers
      2. Tuning MaxPooling
      3. Tuning the dense layer
      4. How to train the network
      5. Random initialization
      6. Overfitting and underfitting
    6. Visualizing the activations
    7. Inference
    8. Retraining
    9. Summary
    10. Questions
  15. Chapter 6: Improving Your Neural Network
    1. Technical requirements
    2. A bigger model
      1. The starting point
      2. Improving the speed
      3. Increasing the depth
    3. A more efficient network
    4. Building a smarter network with batch normalization
      1. Choosing the right batch size
    5. Early stopping
    6. Improving the dataset with data augmentation
    7. Improving the validation accuracy with dropout
      1. Applying the model to MNIST
      2. Now it's your turn!
    8. Summary
    9. Questions
  16. Chapter 7: Detecting Pedestrians and Traffic Lights
    1. Technical requirements
    2. Detecting pedestrians, vehicles, and traffic lights with SSD
      1. Collecting some images with Carla
      2. Understanding SSD
      3. Discovering the TensorFlow detection model zoo
      4. Downloading and loading SSD
      5. Running SSD
      6. Annotating the image
    3. Detecting the color of a traffic light
      1. Creating a traffic light dataset
      2. Understanding transfer learning
      3. Getting to know ImageNet
      4. Discovering AlexNet
      5. Using Inception for image classification
      6. Using Inception for transfer learning
      7. Feeding our dataset to Inception
      8. Performance with transfer learning
      9. Improving transfer learning
    4. Recognizing traffic lights and their colors
    5. Summary
    6. Questions
    7. Further reading
  17. Chapter 8: Behavioral Cloning
    1. Technical requirements
    2. Teaching a neural network how to drive with behavioral cloning
    3. Introducing DAVE-2
      1. Getting to know manual_control.py
      2. Recording one video stream
      3. Modeling the neural network
      4. Training a neural network for regression
      5. Visualizing the saliency maps
    4. Integrating the neural network with Carla
    5. Self-driving!
      1. Training bigger datasets using generators
      2. Augmenting data the hard way
    6. Summary
    7. Questions
    8. Further reading
  18. Chapter 9: Semantic Segmentation
    1. Technical requirements
    2. Introducing semantic segmentation
      1. Defining our goal
      2. Collecting the dataset
      3. Modifying synchronous_mode.py
    3. Understanding DenseNet for classification
      1. DenseNet from a bird's-eye view
      2. Understanding the dense blocks
    4. Segmenting images with CNN
    5. Adapting DenseNet for semantic segmentation
    6. Coding the blocks of FC-DenseNet
      1. Putting all the pieces together
      2. Feeding the network
      3. Running the neural network
      4. Improving bad semantic segmentation
    7. Summary
    8. Questions
    9. Further reading
  19. Section 3: Mapping and Controls
  20. Chapter 10: Steering, Throttle, and Brake Control
    1. Technical requirements
    2. Why do you need controls?
      1. What is a controller?
    3. Types of controllers
      1. PID
      2. MPC
    4. Implementing PID in CARLA
      1. Installing CARLA
      2. Cloning Packt-Town04-PID.py
      3. Walking through your Packt-Town04-PID.py control script
      4. PIDLongitudinalController
      5. PIDLateralController
      6. Running the script
    5. An example MPC in C plus plus
    6. Summary
    7. Questions
    8. Further reading
  21. Chapter 11: Mapping Our Environments
    1. Technical requirements
    2. Why you need maps and localization
      1. Maps
      2. Localization
    3. Types of mapping and localization
      1. Simultaneous localization and mapping (SLAM)
    4. Open source mapping tools
    5. SLAM with an Ouster lidar and Google Cartographer
      1. Ouster sensor
      2. The repo
      3. Getting started with cartographer_ros
      4. Cartographer_ros configuration
      5. Docker image
    6. Summary
    7. Questions
    8. Further reading
  22. Assessments
    1. Chapter 1
    2. Chapter 2
    3. Chapter 3
    4. Chapter 4
    5. Chapter 5
    6. Chapter 6
    7. Chapter 7
    8. Chapter 8
    9. Chapter 9
    10. Chapter 10
    11. Chapter 11
  23. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think

Product information

  • Title: Hands-On Vision and Behavior for Self-Driving Cars
  • Author(s): Luca Venturi, Krishtof Korda
  • Release date: October 2020
  • Publisher(s): Packt Publishing
  • ISBN: 9781800203587