Vision and Multimodal Four-Axis Robot

00 / Context & objective

A desktop robot that recognizes objects, responds to commands and uses human feedback for cat-like trial and correction.

01My contributions

• Trained and deployed a YOLOv8 detector for three tabletop object classes and organized image capture and target localization.
• Used OpenCV for camera calibration, distortion correction and homography mapping to the work plane.
• Integrated Vosk Chinese speech, keyboard input and MediaPipe hand detection into trial, correction and reward interactions.
• Connected perception, coordinate conversion and interaction decisions to the team's robot execution interface for picking and placement.

02System flow

1. 01Camera images
2. 02Three-class detection
3. 03Calibration and plane mapping
4. 04Commands and feedback decisions
5. 05Robot motion

03Recorded results

• The historical training CSV records a best validation mAP@0.5 of 86.21%, a detection metric.
• Stored camera reprojection RMS is 0.704 px; the eight-point homography fit has an RMS residual of 7.145 mm.
• The physical project was completed and handed in; the public archive retains application code, calibration files and training records.

04Validation scope & limitations

Read results alongside their measurement targets and conditions. Historical records, host checks, and device validation are distinguished.

• The robot was built as a team project. My work covers perception, interaction and integration, rather than sole ownership of mechanics, electronics and drivers.
• Detection and calibration metrics are not grasp success rates; no fixed-protocol physical repeatability study is available.
• Archive checks were offline only, with no new training, inference or robot motion. Weights, datasets and vendor DLLs are excluded.

05Public materials