A small course robot that acts on what its own camera sees: an on-board detector spots a plate, a bucket and an exit arrow, and a state machine decides the next move. Built and debugged; the full mission was not completed.
In KTH’s embedded machine-learning course we connected a Google Coral camera board, which runs an object detector on the board itself, to an XRP teaching robot, so that what the camera sees decides what the robot does next. The course mission sets the order: follow a line, find a plate and wait on it, find a bucket and pick it up, then find the exit arrow and return to the line. I worked on the training data, the mission logic and the link between the two boards. I can show the data and the controller. I cannot show a complete run: the link between the boards never became reliable in both directions.
WORK
Training data · embedded inference · robot state machine
PLATFORM
Coral Dev Board Micro + XRP robot
OUTCOME
Built · mission not completed
A handheld look at the camera assembly, wiring and robot during bench development. Original recording, 00:00–00:11.
01 / The view from the robot
Training data from the robot’s own camera.
The training images come from the robot’s own camera, in the room where it ran: reflections on the floor, objects at changing distances, the same object from different sides. Those conditions are part of what the detector had to learn.
The detector has four labels: arrow, picked-up bucket, bucket and plate. The tiles below are samples from that dataset as captured, without detection boxes or scores, because I have no measured detector output to show.
Training sample from the robot’s camera, 320 × 320.
What the data shows, and what it doesn’t+
What I have is the camera dataset, the exported models and the robot-control code. That shows the data and implementation work. It does not show detector accuracy on unseen images, real-time latency, or a completed mission.
Source: original dataset samples; project README and robot_control modules.
The camera board and mount on a bench fixture, beside handwritten object labels.
02 / The mission logic
A detection is useful when it changes a decision.
Our controller waits for fresh detections, steers by an object’s position and apparent width as it approaches, and combines what the camera sees with the line sensors, the ultrasonic wall reading and the wheel encoders. I wanted each of those responsibilities visible rather than buried in one loop, so the mission is a state machine with twenty named states. Some fixes came straight from the robot’s logs: the width at which it should stop at the plate had been set to an unreachable 2.99999, and we recalibrated it to 0.70 of the image width from logged detections.
The course mission supplies the order. Our implementation supplies the transitions, the motor actions and the fallback behaviour. The sequence below explains that logic; it is not a recording of a successful run.
01 · PLATE
Find the plate and wait
02 · BUCKET
Explore, centre and approach
03 · PICKUP
Lower, advance and grab
04 · EXIT
Search for the arrow and return to the line
A specific integration gap+
In the pickup handler, LABEL_PICKED is checked and a success or warning message is printed, but both branches then move on to BACK_FROM_BUCKET, so a failed pickup does not stop the mission. I have left it as the code shows it. A dependable sequence needs a confirmed, or explicitly handled, pickup, and we did not get there.
Source: state_machine.py, _step_pickup and _step_back_from_bucket; MF2143 assignment.
The physical test course, with floor tape, cardboard boundaries and the robot.
03 / A fallback input
Another way to give a command.
I also tried keyword spotting, recognising a few spoken command words, as a fallback way to steer the robot. The recording shows that test at the bench, with an on-screen maze giving feedback.
It is a separate experiment from the camera-driven mission, and it does not show recognition accuracy. The original sound is in the player.
Keyword-spotting fallback test for the robot, with an on-screen maze for feedback. Original recording, 00:06–00:36, with sound.
04 / Between two boards
Where most of the debugging went.
Most of my debugging went into the link between the two boards. The robot’s USB port was damaged, so I flashed its controller over SWD through a second Pico set up as a debug probe, and that became our recovery route. I moved the serial link off the default pins, which sit inside the robot’s encoder circuitry and picked up noise, to a second UART. We had already ruled out a wireless link: its round trips of 100–500 ms and its dropouts while driving were too slow for control, where the cable answers in about a millisecond.
One direction of the link worked; the other arrived garbled. My best explanation is the Coral’s serial clock. It is derived from a 24 MHz RC oscillator, accurate to about 1–2 %, and at 115,200 baud a 2 % error drifts by about 1.7 µs per byte, a fifth of a bit: enough to break framing after a few characters. Dropping to 9,600 baud helped without curing it. The measurement that would settle it, the actual clock frequency on a logic analyser, is still to be made. The link never became reliable in both directions.
CORAL
Camera input and object detections
SERIAL LINK
Message format, timing and reception
XRP
State transitions, drive and pickup control
What I can show+
I can show the camera dataset, the controller, photographs of the prototype, bench footage and the separate keyword test. I cannot show reliable two-way communication, a confirmed pickup or a complete autonomous mission, and I have no measured detector accuracy or inference latency.
