Drunky
A two-armed robot bartender that takes a drink order and builds the cocktail end-to-end.
Why We Built This
Honestly, it started because we all just like making cocktails. Bimanual manipulation needed a real task to test itself against, and this felt like a genuinely fun one to build around.
Digging in, we realized most bartending robots have a blind spot: move a bottle, and they have no idea where it went. Most systems on the market work off fixed positions, calibrated once and expected to stay that way.
Could we build something that actually saw the bottle, instead of just remembering where it used to be?
Using YOLO-based detection, the answer was yes — the robot could find and grab the right bottle no matter where it sat on the shelf.
System Architecture
bartender_ui (tkinter)
- • Menu + camera health
- • Owns the 2 wrist YOLOs
- • Sequences the two arms
Find + align the bottle
- • Overhead YOLO + depth → 3D pose
- • Wrist YOLO + visual-servo loop
- • Aligns in image space
MoveIt 2 + trajectories
- • Per-arm collision-aware plans
- • ROS 2 action servers
- • Recorded pour / toss replay
Bimanual SO-101
- • 2× 5-DoF arms + grippers
- • 1× overhead RealSense D435
- • 2× wrist USB cameras
Button press → detect & localize → plan & approach → grab, pour, toss → poured drink
Depth trick: sample the nearest 10% of depth pixels inside each YOLO box, so the table doesn't drag the reported depth back.
The Pipeline
The full loop: an order placed on the bartender UI dispatches to the arm, which executes five phases in sequence — Find → Go to → Align → Grab → Pour — to complete the drink.
Find
Overhead YOLO + depth pose
Go to
MoveIt plan to a standoff pose
Align
Wrist-cam visual servo (closed loop)
Grab
Lower, level, open, enter, close, lift
Pour
Recorded joint-space trajectory
Toss
Recorded toss + release into glass
One arm takes about 1 min 7 s for 6 phases — sequential arms make a full two-ingredient cocktail in about 2 min 15 s.
Challenges
An unstable LeRobot + Rosetta integration drove the pivot to a modular classical pipeline
V1 — End-to-end imitation learning
- • Teleop demos → trained an ACT imitation policy (LeRobot)
- • A real proof of concept — it did grab and pour
- • But it ran through Rosetta, which applied a constant offset to the SO-101 shoulder joint
- • That made the LeRobot + Rosetta integration unstable and unpredictable
- • A black box on top of it — hard to inspect a bad run
V2 — Modular classical + ML pipeline
- • Every phase is explicit, testable, tunable
- • Perception decoupled from control
- • Failures attributable to a specific stage
- • Enables the failure-mode analysis in Evaluation & Results
- • Traded some autonomy for safety + repeatability
The learned policy remains in the repo history as a working proof of concept.
Engineering challenges overcome
5-DoF grasp geometry
Level the wrist to the ground (not the forearm) via TF pitch; approach along the gripper axis so it doesn't slide and tip the bottle; lift in joint space (cartesian lifts loop).
Wrist-camera plumbing
Two identical USB cams collide on serial — pin each by physical USB port (/dev/v4l/by-path). MJPEG segfaulted, so we stream raw YUYV → RGB.
Detection robustness
COCO labels bottles as 'bottle' OR 'vase' by material — accept both, lock onto the largest (closest) box; back off the standoff for a cleaner view.
Gravity sag + resilience
Centering loop holds non-pan joints fixed and pre-compensates droop per arm; pour / toss retries a failed pose once and continues.
Safe two-arm coordination
Simultaneous planning through the shared workspace fails and is unsafe — we run the arms sequentially: collision-free and reliable.
Evaluation & Results
total trials
shelf positions
trials per cell
overall success rate
Top shelf vs. bottom shelf
Right arm (alcohols) vs. left arm (mixers)
Best liquid — vodka
Worst liquid — coke
one arm / single liquid, best case (std ≈ 8 s)
full two-ingredient cocktail (12 phases)
Occlusion and camera distance drive the gap — lower shelf cells sit farther from the overhead camera and are partly blocked by the shelf above.
Where it fails
Share of 179 failures across all 480 trials.
- Detection / alignment loss45%
- Grasp miss / bottle tipped30%
- Pour / toss execution miss12%
- Camera dropout8%
- Other (planning / timeout)5%
Mitigations: per-shelf lower/forward offsets, retry-then-continue on pour and toss, and a GUI Refresh control that recovers camera dropouts mid-service.
What's Next
Close the occlusion gap. Top shelf hits 71.7% success vs. 53.8% on the bottom, and most failures trace back to occluded, distant camera views. A second viewpoint or active sensing is the clear next step.
Cut grasp misses. 30% of failures are grasp misses or tipped bottles, the second-largest failure category. Worth digging into gripper geometry and approach angle.
Speed up the cycle. 2:15 for a full cocktail, with arms running sequentially for safety. Safer simultaneous coordination could cut that time meaningfully.
Revisit imitation learning. The V1 ACT policy was shelved over a LeRobot + Rosetta integration bug, not a failure of the approach itself. Worth another look, especially on the hardest occluded cases.
Automate camera recovery. Camera dropout causes 8% of failures and currently needs a manual refresh. An automatic reconnect would remove that step.
