Overview
ProjecTo is a desktop-scale visual projection robot assistant that bridges the gap between traditional PCs and the physical desktop environment. By fusing projection display, computer vision, and a servo-controlled robotic arm into a movable spatial UI, it enables gesture commands, handwriting recognition, object annotation, and reminder input — all without touching a screen.
Role
Hardware Engineer · Software Architect · Interaction Designer
Year
2025
Disciplines
MACHINE VISIONROBOTICSPYTHONOPENCVMEDIAPIPEYOLOV8PYQT6HARDWAREUI PROJECTIONUCL
BACKGROUND
Your desk is invisible to your computer
Traditional PCs can't perceive or assist with the physical objects on your desk. Every sticky note, sketch, or reference book exists in a blind spot. ProjecTo makes the desktop a first-class interaction surface — projecting UI onto it and reading it back with computer vision.
HARDWARE
SO-ARM101 robotic arm, repurposed
The SO-ARM101 mechanical arm was structurally modified to carry a projector module and camera. Sensor and projection module selection, mechanical adaptation, and spatial calibration were all done independently — from CAD to physical assembly.
SOFTWARE
Python full-stack pipeline
A Python pipeline integrates OpenCV image capture, MediaPipe gesture recognition, YOLOv8n object detection, and PyQt6 for the projected UI. An alternating inference and frame management strategy achieves real-time interactive response on low-cost hardware.
INTERACTION
Gesture, handwriting, annotation
Users interact through mid-air gestures, handwriting on any surface, and tapping projected UI elements. The system recognizes objects on the desk and allows users to annotate or attach reminders to physical items — bridging digital and physical workflows.