Motion input · August 2026

Steering a fractal with an iPhone, a Watch, and AirPods

An app that zooms into a fractal forever, steered without touching the screen. Phone tilt pans, head pitch zooms, wrist tilt pans hands-free, and a head flick or crown wiggle changes the palette. All three devices at once.

Demo recorded August 2026. The control layout has moved on; the rules below are current.

Where this came from

My day work is Run Fun, a running-gait app: Apple IMUs and on-device pose estimation streamed live into an app while someone is actually running. Gait is where the hard parts of body-motion input show up — drift, calibration, axis conventions, keeping a Watch alive while the wrist swings — and where getting them wrong is expensive. The fractal is the testbed: same layer, none of the domain risk.

Who controls what

Everything that steers continuously rides a gravity-referenced axis. Drifting or bursty inputs (head yaw, crown wiggle) only fire discrete events.

The rule that made it work

Tilt is measured against gravity, so it can't drift. Head yaw has nothing to measure against, so it does. That decided the whole control layout.

Illustrative, not a measurement plot. Pitch and roll are corrected against gravity on every sample, so their error can't accumulate. Yaw has nothing absolute to correct against, so it integrates — which is why yaw gets to fire events and never gets to steer.

I had yaw panning the view. Fine for a minute, then the centre wandered until I was cranking my neck sideways to hold still. Filtering doesn't fix an integrating error, it delays it. The rule that did: drifting axes fire events, never steer. Yaw flips the palette past a threshold, where a drifted centre is harmless.

Corollary, learned twice: one gesture, one continuous job. Wrist tilt driving zoom and pan together felt like the app was fighting me. Then the Watch crown — turning it set zoom rate while wiggling the same crown changed the palette, so every deliberate turn risked firing a colour change. When a control has two jobs the bug isn't "sometimes wrong", it's "the app feels possessed."

Two numbers worth knowing first

Tilt carries ~2.3 bits/second against a mouse's 4–5. Coarse continuous steering only — never put menu targets on it.
Competence varies ~10× between people. Whatever you tune to your own hands is unusable for someone. Ship a sensitivity control.

The walls I hit

Each cost real time and none are documented loudly. Naming them so you can budget for them; the fixes are in my implementation.

AirPods head tracking can silently give you nothing. Connected, worn, API reporting success — and zero motion data, no error. There's an undocumented precondition. Until you find it, all you get is silence.
watchOS owns certain wrist motions. A wrist-flick gesture worked three times, then the system gesture it collided with threw my app back to the clock face. No API disables that. Check what the OS already claims before designing a Watch gesture.
Keeping a Watch app alive while the wrist moves is its own project. The obvious API isn't enough. The working setup needs two session types together, started in a specific order.
Half my axis directions were backwards. Correct on paper, wrong on a real wrist. Head tracking has a convention — look left means pan left, as with FPV drone pilots and assistive head-mice — and it is not what joystick habits suggest.
Never solved: detecting a physical tap on the Watch case from accelerometer data. Two approaches, zero detections on hardware. If you've done it, I'd like to hear how.

Motion-only is an exclusion

What I got most wrong was quieter than any bug above: tilt was the only way to steer. WCAG 2.5.4 wants anything motion-operable also operable by a normal control. ADA §309.4 says operable parts must not require "twisting of the wrist" — which is what a crown is. And in a 2023 study of 25 people with upper-body motor impairments, 51% of gestures were made with hands resting on an armrest, a posture that cannot produce a tilt excursion at all.

So it grew a second way in: drag to steer, visible controls, a sensitivity slider, posture presets, Reduce Motion support. None of it replaced the motion layer — it made the motion layer optional, which is what makes it shippable.

Where this goes

The reusable part is small: a control law shaping tilt into velocity, pan speed scaled by zoom depth, the drift rule, and a non-motion path beside it. The same layer could steer a camera in a simulator, an exhibit in a museum, or a cursor for someone whose hands don't cooperate.

Building motion controls on Apple hardware and could use help? [email protected]