Integrating ML & AI with LEGO Robotics
Software
Python/MicroPython
JavaScript
LabVIEW
ThingWorx Composer & Analytics
Teachable Machine/Tensor Flow
MQTT & HTML
Google Colab
Hardware
LEGO SPIKE Prime Kit
ESP8266
For the summer of 2021, I worked as an undergraduate research intern at Tufts CEEO FETLAB under my robotics professor, Chris Rogers, and focused on exploring various machine learning methods and using them to train LEGO's SPIKE Prime Robots. PTC funded my research and gave me access to their industrial IoT platform, ThingWorx, which contains a powerful Analytics extension that can be utilized for machine learning and training.
My main achievements include:
Designed LEGO robotic systems integrated with IoT, AI, and data analytics
Trained systems using JavaScript, Python, and advanced ML algorithms using PTC’s industrial IoT platform ThingWorx and TensorFlow
Expanded SPIKE Prime robot functionalities by developing WiFi communication interfaces with REST API and MQTT
Below are examples showing the training of a SPIKE puppy robot to sit/stand using 3 levels of Machine Learning training methods.
Please visit my Github for a more extensive overview!
Training Locally
The simplest method of training is locally on the SPIKE Puppy. My partner and I created a set of lesson plans that trained the puppy through coding Python Machine Learning Programs (KNN, linear regression, and other basic Machine Learning methods) directly onto the brick, or brain, of the LEGO robot. The puppy was trained to sit/stand, recognize food via color, fetch, and more.
The goal is for students, especially younger ones, to easily access and understand Machine Learning concepts through completing these lesson plans.
My SPIKE Puppy (ESP822 WiFi Dongle not pictured)
Local Training Cycle
Training with Teachable Machine
The second level of Machine Learning training was through Google’s Teachable Machine, which uses neural networks in TensorFlow.
Teachable Machine Training Cycle
A training model was created on Teachable Machine given various training inputs (image stills, audio clips, etc.). Then, the predictive model was coded into an HTML page, which took input from sensors on a computer or whatever device the HTML page was loaded on. The resulting prediction (eg. “sit” or “stand”) was funneled to AirTable. Finally, the SPIKE would make API calls to AirTable via the ESP8266, receiving the information and behaving accordingly.
Training with ThingWorx Analytics
The final level of Machine Learning training was also an exploration into the capabilities and applications of PTC's IoT platform ThingWorx, specifically its Analytics Builder and server. ThingWorx played an important role in my overall goal of this summer: explore machine learning methods integrable with Lego SPIKE Prime robotics to create an educational AI experience for students.
First, a simple ball launcher was built and trained using linear regression to better understand how to integrate ThingWorx with SPIKE Prime. The video for this is at the very top of this page.
ThingWorx Training Cycle
Then, more complex methods of training (eg. neural networks, gradient boosts, random forests, etc.) were explored using the SPIKE Prime puppy system, with an MQTT protocol acting as the main communication method between SPIKE and ThingWorx Analytics.
Here I have trained the puppy to sit and stand using ThingWorx Analytics. Raw data was sent to ThingWorx through an MQTT broker. A predictive model was generated using API calls from the ESP8266 (calls various Analytics services), and the resultant model was called in real-time to show the results of the training. This entire process is completely automated.
