By: Antonio Zugaldia
TL;DR: I built a small Robocar that can use either Computer Vision or Machine Learning to drive autonomously. It uses Android, off-the-shelf hardware components, and the code is available on GitHub. Come and join us for racing @Locate May 30–31; Donkeycar racing team signups just opened.
Earlier this year Android launched Android Things, a flavor of Android that runs on Raspberry Pi and similar IoT devices; so I did the reasonable thing to do — I built an autonomous RC car from scratch.
The hardware: building your Robocar
For the hardware, I just bought all off-the-shelf components. I had a feeling that there would be a few crashes before my neural networks were fully tuned, so I wanted to be able to replace broken pieces easily. The shopping list is short (a little over $100):
- A Raspberry Pi (about $35).
- A motor controller (about $22) — turns out that you can’t connect motors directly to a Raspberry Pi (early lesson)
- A portable charger to power the Pi and the motors (about $30).
- And, finally, a frame. This Junior Runt Rover is a good starting point ($28) but you could recycle an existing frame from another toy.
The car can be controlled using a companion app (also built with Android) that uses a local server to communicate with the car. However, I decided to use a retro Nintendo controller (this adds an extra $35 to your budget). Putting it all together is documented; a couple of things to note:
- Soldering time: I haven’t been able to find a Raspberry Pi-compatible motor controller that comes pre-soldered. You’ll need to dust off your soldering station or find a friend with one. It isn’t difficult, but it requires some practice considering the small size of the pins.
- There’s life beyond the Raspberry Pi: Android Things is an abstraction layer over IoT devices and while Raspberry Pi is the most popular one for hobbyists, other boards will work out of the box without having to change a single line of your code. My favorite so far is the NXP Pico. The advantage of this board is that it comes with fast onboard storage (so you don’t need to buy an additional SDCard), and it contains a SoM (System-on-Module). This is the kind of module that real production devices would use if you wanted to productize your hardware project.
Software for your autonomous Robocar
This is where things got a little bit out of control. Udacity has a self-driving car “nanodegree” that teaches you how autonomous cars work. They cover things like Computer Vision and Deep Learning for recognizing lanes and pedestrians, all from a very practical angle where you get to code on simulators (they use Unity).
One of the first things you learn during the class is that there are two different approaches that autonomous cars use to recognize objects (often together): Computer Vision and Machine Learning. Regardless of the approach, you’re gonna require a camera capable of taking pictures of what’s in front of the car. That’s the easy part. Nowadays, you can get an 8 megapixels camera for about $30 that connects directly to a Raspberry Pi and the other boards we mentioned before.
Computer Vision for Robocars
The CV approach basically consists of applying image filters and transformations to pictures taken from the car and extracting information about lanes (the camera needs to be positioned fairly high to have a good angle). To make this analysis performant and have the car respond in “real time” to the information captured by the camera, I used an approach similar to the one we use with our Maps SDK: do the heavy lifting on C++ (with OpenCV) and expose the methods needed on the Java side so that Android can make decisions on whether to turn or continue going straight.
Once this was done, one possible pipeline goes like this. Let’s start with this image, part of the project test suite:
- Transform the image so that it looks like the photo was taken from above (if you’re familiar with OpenCV, this is the combination of applying getPerspectiveTransform followed by warpPerspective). This removes any distracting objects surrounding the picture and lets you focus on the lanes:
2. Do some color thresholding to separate the lanes from everything else in the picture. Depending on the color of your lanes (bright yellow tape in my case) it’s useful to change the color space from BGR (OpenCV’s default) to HSV (Hue, Saturation, Value). This color space fits better with the way human vision perceives colors and makes filtering a specific color easier using hue and saturation ranges:
3. Finally, we need to compute a histogram to find the most likely location for the lane (represented by the vertical line below):
Putting it all together, you can instruct the car to center on the lane before advancing:
Next is adding Lidar support 😃 . If you’re curious to see how this looks with pictures from a real car, check out this video. It’s one of the exercises you have to make during the Udacity training.
Machine Learning for Robocars
Another way to tackle this problem is using Machine Learning (aka Deep Learning Convolutional Neural Networks) to teach the car how to drive autonomously. You use real-world data to train a model to classify traffic signs or to make the car make driving decisions using behavioral cloning. You need to take thousands of pictures, and for each picture, you need to encode information that you want the car to learn (e.g. the speed of each wheel).
The problem is that designing a Convolutional Neural Network from scratch isn’t trivial and goes beyond what I wanted to accomplish with this project. So what I did instead was to take advantage of NVIDIA’s public research in the field (see this paper) and implement one of their network architectures. This one in particular consists of 9 layers, including a normalization layer, 5 convolutional layers, and 3 fully connected layers:
With tools like Keras, this is something that can be implemented in just a few lines of code.
Note that this is all implemented in Python instead of Java for two reasons: On one hand, Python is probably the de-facto language for ML these days. It has great libraries and documentation, and it seems to be the language of choice for frameworks like TensorFlow. On the other hand, training a model is something that you wouldn’t want to do on a Raspberry PI anyway, it’s slow and taxing on such modest hardware. Instead, you’d want to use a powerful laptop or a server in the cloud.
The Robocar Racing Community
The best part is being a part of a group of enthusiasts building their own small, autonomous cars (“robocars”). This is a meetup that Chris Anderson started in Oakland, CA that quickly expanded to other locations. You might know him from his time at Wired magazine or for starting DIY Drones. He likes to say that his passion is to put “the letters DIY in front of a big established industry and see what happens.”
I am involved with the DC chapter, and in less than a year we’ve organized a dozen events, and we now have more than 200 people registered. This chart below, comparing CV vs ML, comes from a recent Robocar meetup. As you can see, both are doing pretty well, and while CV obtained better results faster, looks like the ML approach is quickly catching up:
Enter your racing team @ Locate
If you’re interested in learning more about autonomous cars, or their smaller siblings the Robocars, join us at Locate.
- Donkeycar racing team signups are open.
- If you’re familiar with Java/Android, follow the instructions on how to build your own Android Robocar. If you prefer Python, the Donkey Car project is the best place to start. Its code and build are more advanced (it comes with 3D models that you could print), but definitely worth the investment.
- Check out the next DIY Robocar meetup in SF, in DC, and other locations. Even better, start your own if one isn’t available in your area.
- Ping me (@zugaldia) if you have any questions.
Build your own (small) Autonomous Robocar was originally published in Points of interest on Medium, where people are continuing the conversation by highlighting and responding to this story.