Using Machine Learning

Here are some instructions for using the Camera Car ML program, and using the Machine Learning feature to customize the image recognitition for your environment and experiment with your own models. Due to the way this kind of image recognition works, you will need to "re-train" the model to get it to work well for you. See this video for an example of the model in action.

Doing image recognition from a moving vehicle is tricky because the background of the image is always changing, and the image recognition takes the entire image into account. The software recognizer (a neural network) does not know what it is supposed to be looking at, so it processes the entire image, including the walls and furniture in the background. All aspects of the image are considered, including colors, shapes, patterns, etc. (Image recognition on a model where the phone doesn't move is a lot easier if you want to try that idea somewhere).

1. Load the Camera Car ML program on your phone, then tap on the Machine Learning icon on the right (below the Remote Control and Monitor icons).

2. Tap the icon on the camera preview image to grant camera access to the software, and then make sure the back camera of the phone is selected (looking away from you, not at you).

3. Tap the Library button, then select the Driving Directions model, then tap Edit.

4. You will see four "classes" named Go, Stop, Turn, and None. You can tap each to see a set of training images that I took for these actions. The Go and Stop classes have 8-10 images of the front and back of the hanging sign, held fairly close to the camera so that it fills most of the frame. This makes these reliable to recognise since they block out most of the background. You can try using these two classes as-is, but they will work better if you delete my images and retake your own in the room where you plan to use the robot. 

5. The None class has 30 background images of the room where I was using the robot, all taken at different angles (since the robot can move around). Since the image recognizer is always trying to recognize something and will just take the closest match when unsure, this class is necessary to keep it from wildly guessing when there is no sign in front of it. Unless your room looks a lot like mine, (very unlikely), you will need to delete my images and take your own and re-train the model. The phone doesn't have to be on the robot, but try to use the perspective that the camera will have on the robot, and take 30 or more images, turning slightly with each one, to get the room from as many angles as possible, the more images the better. Real autonomous driving software (e.g. Tesla Autopilot) uses millions of images taken from many thousands of cars on as many roads as possible to try to recognize things like stop signs.

6. The Turn class has images of the side edge of the hanging sign. Since this makes the image contain mostly background, it is hard for the software to distinguish this from the background (None) images, so I took more pictures than Go and Stop here. Also in the program code you will see that I required the confidence to be much higher than default (> 95%) to distinguish Turn vs. None.

7. After taking all of your images, press the Train button to re-train the model, then you can press Done and then the Run button to run the program and give it a try. Be prepared for a few tries at this and to experiment a bit to see what works better for you.