Building Without Buttons - Computer Vision Controllers
Building Without Buttons: Computer Vision Controllers with MediaPipe and Python
Recently, I’ve been exploring how to bridge the gap between physical human movement and digital software execution. Stepping away from traditional hardware inputs, I wanted to see how far I could push real-time computer vision using OpenCV and the modern MediaPipe Tasks API.
What started as a simple curiosity grew into multiple projects. Over a week, I went from capturing raw joint coordinates to manipulating the Windows core audio system, and finally, emulating a literal physical Xbox 360 controller to drive cars in video games.
Here is the complete, behind-the-scenes breakdown of what I built, the engineering challenges I faced, and the core architectural lessons I learned along the way.
1. The Foundation: Low-Latency Modular Hand Tracking
My first step was creating a optimized tracking script. Rather than using legacy versions of MediaPipe, I built this using the modern MediaPipe Tasks Vision API.
1
2
3
4
5
6
7
8
9
# Setting up the modern MediaPipe Task Configuration
model_path = "hand_landmarker.task"
base_options = python.BaseOptions(model_asset_path=model_path)
options = vision.HandLandmarkerOptions(
base_options=base_options,
running_mode=vision.RunningMode.VIDEO, # Highly optimized for live feeds
num_hands=2
)
The Technical Hurdles & Architecture
When configuring the RunningMode.VIDEO pipeline, MediaPipe expects a continuous stream of frames rather than isolated independent images. This requires a strict architecture:
- The Timestamp Engine: You cannot just pass an image; you must pass an explicitly calculated timestamp in milliseconds for every single frame (
int(time.time() * 1000)). If your loop drops frames or the timestamps aren’t strictly increasing, the underlying model’s tracking continuity shatters. - The Coordinate Translation: MediaPipe returns joint coordinates (Landmarks) as normalized floating-point values between $0.0$ and $1.0$ relative to the frame’s boundaries. To draw a pixel-perfect circle over a specific knuckle, you have to manually map those boundaries back into your camera’s resolution:
1
2
3
4
5
6
7
# Converting normalized coordinates to pixel values
for lm_id, lm in enumerate(hand_landmarks):
cx, cy = int(lm.x * w), int(lm.y * h)
if lm_id == 4: # Tip of the thumb
print(f"Thumb Tip ID {lm_id}: X={cx}, Y={cy}")
cv2.circle(img, (cx, cy), 6, (255, 0, 255), cv2.FILLED)
The MediaPipe Hand Landmark Skeleton map breaking down the 21 tracking joints.
To make the application feel natural, you must flip the incoming frame horizontally (cv2.flip(img, 1)). Webcams act as a camera lens, not a mirror; without the horizontal inversion, moving your hand to your right looks like a leftward movement on screen, creating an immediate cognitive disconnect.
2. Hardware Interaction via Euclidean Distance
Once I could isolate individual joint coordinates on the fly, the next step was applying that mathematical data to a real operating system task. I decided to turn my hand into a gesture-controlled volume knob.
By utilizing the tracking logic from project one, I filtered out every joint except two: Landmark 4 (Thumb Tip) and Landmark 8 (Index Finger Tip).
1
2
3
if lm_id == 4: thumb_xy = (cx, cy)
if lm_id == 8: index_xy = (cx, cy)
When both coordinates are active, the script calculates the straight-line distance between them. This is achieved using the Euclidean distance formula via Python’s built-in math.hypot():
Interfacing with the Windows Core Audio API
To change the system volume, I used PyCaw (Python Core Audio Windows Library) to hook directly into the system’s endpoints.
The immediate challenge was a massive data mismatch. The distance between my fingers in a webcam feed varies dynamically based on how close I am standing to the lens (typically moving between 25 pixels to 120 pixels). However, the Windows Core Audio endpoint doesn’t accept percentages, it strictly accepts a non-linear decibel range (such as $-65.25 \text{ dB}$ for mute up to $0.0 \text{ dB}$ for max volume).
To bridge this, I used NumPy’s interpolation function to cleanly map the dynamic pixel distances to the exact boundaries required by the computer hardware:
1
2
3
4
5
6
7
# Interpolating pixel distance across two vastly different data scales
volume = np.interp(distance, [25, 120], [min_volume, max_volume])
volume_bar = np.interp(distance, [25, 120], [400, 150]) # For the visual UI HUD
volume_per = np.interp(distance, [25, 120], [0, 100])
volume_manager.SetMasterVolumeLevel(volume, None)
Real-time volume control mapping finger distance to native system decibels.
3. Emulating an Xbox 360 Virtual Steering Wheel
With system-level single-hand tracking working flawlessly, I wanted to see if I could push this system to handle high-stakes, multi-input real-time environments. I set out to build a fully functional virtual steering wheel that could inject inputs directly into modern PC racing video games.
To do this, I integrated vgamepad, a brilliant library that emulates a virtual bus driver, convincing Windows that a literal physical Xbox 360 controller has been plugged into a USB port.
The architecture for this final project required handling two completely separate inputs simultaneously:
Mechanics Breakdown
The Steering Angle (Trigonometry)
The application requires two hands to be actively tracked on screen. The script reads both hand centers, sorts them by their horizontal positions to isolate the Left Hand from the Right Hand, and calculates the rise-over-run slope between them.
Using math.atan2(dy, dx), the script extracts the exact arc tangent angle of the imaginary line connecting your hands, converting the output from radians into standard degrees:
1
2
3
4
5
6
7
8
# Calculating the rotation angle of the hands
dy = right_hand[1] - left_hand[1]
dx = right_hand[0] - left_hand[0]
steering_angle = math.degrees(math.atan2(dy, dx))
# Clamp angle to reasonable driving limits (-45 to 45 degrees)
steering_angle = max(min(steering_angle, 45), -45)
We then interpolate this degree value into the native boundaries of an Xbox analog joystick, which expects a signed 16-bit integer ranging precisely from $-32768$ to $32767$.
The Gas Pedal / Throttle (Dynamic Extensibility)
Steering means nothing if you can’t move forward. I wanted an elegant way to accelerate without needing a keyboard. I isolated the Right Hand and tracked the absolute distance between the Thumb Tip (ID 4) and the Index Knuckle Base (ID 5).
When my right thumb is tucked against my palm, the value falls around 30 pixels (representing 0% throttle). When I stretch my thumb completely outwards, the distance jumps to 85 pixels or more. This stretch distance is mapped natively onto the Xbox’s Right Trigger hardware state ($0$ to $255$).
1
2
3
4
5
6
7
8
# Map thumb stretch data to the Xbox Trigger range
gas_value = int(np.interp(thumb_stretch, [30, 85], [0, 255]))
# Push the translated values directly to the virtual gamepad hardware
gamepad.left_joystick(x_value=joystick_value, y_value=0)
gamepad.right_trigger(value=gas_value)
gamepad.update()
The complete tracking HUD displaying calculated steering angles and thumb-stretch throttle percentages.
The Engineering Paradox: Flipped Handedness
During development, I hit an incredibly frustrating logic error. When tracking the throttle, my code would occasionally completely stop registering inputs.
I discovered an amazing side effect of mirror-image video processing. Because the script uses cv2.flip(img, 1) to keep the perspective natural for the user, MediaPipe’s underlying machine learning model receives an inverted frame. Consequently, the AI consistently identifies your physical Right Hand as "Left" in its data structure.
Overcoming this required leaning into the paradox: checking if the underlying handedness label read "Left", knowing that it meant the user was utilizing their physical right hand for throttle management.
Now that the control pipeline is fully established, the next logical step is taking it a layer deeper. Maybe I’ll explore reinforcement learning models or train a custom machine learning agent to see how they work.