Automation

Hands Action Demo

Hands Action Demo is a real-time computer vision pipeline that detects hand gestures — thumbs up, peace sign, open palm, fist, swipes, and more — and maps each one directly to a desktop action: opening Chrome, switching tabs, taking a screenshot, or playing a system sound, all triggered purely by hand movement.

2 weeksSource

The Problem

Most desktop automation still assumes a keyboard and mouse, even for simple, repeatable actions like switching tabs or taking a screenshot. There was no lightweight way to trigger common OS-level actions purely from hand gestures.

The Solution

Built a pipeline that captures webcam frames, runs MediaPipe Hands with TensorFlow Lite for fast landmark detection, classifies the pose against a defined gesture set, and dispatches the matching action through PyAutoGUI, winsound, and OS-specific subprocess calls.

Architecture

Webcam capture loop driving real-time frame processing

MediaPipe Hands + TensorFlow Lite for landmark extraction

Custom rule-based gesture classification

Action dispatcher mapping gestures to PyAutoGUI / winsound / subprocess calls

JSON config for user-specific paths (Chrome, project directories)

Technologies Used

PythonMediaPipeTensorFlow LitePyAutoGUI

Challenges

1Classifying gestures reliably in real time without a heavyweight custom-trained model
2Keeping false-positive triggers low enough that the system feels controllable rather than twitchy
3Mapping detected gestures cleanly onto cross-platform OS-level actions

Engineering Trade-offs

1Chose rule-based classification on top of MediaPipe's landmarks over training a custom model — fast to build and easy to extend with new gestures, at the cost of being less robust to lighting, hand size, or camera angle than a trained classifier
2Separated gesture detection from action execution into distinct modules — new gestures can be added without touching automation code, at the cost of an extra layer of indirection for a project this size
3Built for a single-user, single-camera setup rather than generalizing early — kept the scope realistic for a demo project, at the cost of the system not yet handling multiple hands or users robustly

Lessons Learned

1Real-time computer vision projects live or die on classification stability, not raw model accuracy
2Designing for extensibility — clear separation between detection and action — makes a small project genuinely reusable