Timeline
Use casesGuidesSign inGet the app
Timeline for iPhone
Use casesGuidesAboutPrivacyTermsImpressumSupportYour timeline
  1. Guides
  2. Agents

Proprioception for LLMs

We gave LLMs a brain, eyes, hands and memory, then left them disembodied. What changes when an agent knows where you physically are.

Marcel MaraisUpdated 22 Nov 20267 min read

On this page

  1. Hands without posture
  2. Same event, four physical states
  3. Nervous systems compress
  4. Phantom limbs and noisy nerves
  5. What stays in the skull
Timeline for iPhone

Give your agents your timeline and arrival events.

Download TimelineFree on iPhone · No account
On this page
  1. Hands without posture
  2. Same event, four physical states
  3. Nervous systems compress
  4. Phantom limbs and noisy nerves
  5. What stays in the skull

Over the past two years we have built most of a body for LLMs. Reasoning models gave them a better brain. Vision and computer use gave them eyes. Tool calling, shells and MCP gave them hands. Realtime speech gave them ears and a voice, and markdown files and retrieval gave them a passable memory.

The one sense we skipped is proprioception: the background awareness that lets you touch your nose with your eyes closed. Even when you are not looking at your arm, your brain knows where the elbow is bent, whether your hand is moving, and how hard you are gripping a mug. You never have to think about it until it disappears.

BrainReasoningEyesVisionVoiceSpeechMemoryFiles, RAGHandsCodex, Amp, Pi, MCPBrainReasoningEyesVisionVoiceSpeechMemoryFiles, RAGHandsCodex, Amp,Pi, MCP
Proprioception (missing)
  • where you are
  • whether you’re moving
  • what hour it is

In 1985, the neurologist Oliver Sacks wrote about a 27-year-old computer programmer named Christina in The Man Who Mistook His Wife for a Hat ("The Disembodied Lady"). After a sudden bout of polyneuritis, Christina lost her proprioception from the neck down. Her muscles still worked. Her vision and sense of touch were intact. Yet she dropped whatever she held the instant she looked away, and she could only sit up or walk by staring directly at her own limbs and steering them on purpose with her eyes. Every ordinary movement took conscious, exhausting supervision.

If you spend your day running agents in harnesses like Codex, Amp, Pi, or T3 Code, that description should feel uncomfortably familiar.

Hands without posture

Every motor nerve in a modern agent works fine. Codex can refactor a package, Amp can open a pull request, Pi can run a shell workflow, and T3 Code can juggle five threads at once. Wire up Gmail, Slack, GitHub and Calendar and your agent can read almost everything you do on a screen.

Yet the moment an agent tries to act on its own outside a prompt you just typed, it drops the fork. Most personal agents still behave like smart interns you have to babysit, because digital context only tells an agent what is sitting in your inbox or repository. It has no idea what your body is doing: whether you are at a desk with a terminal open, standing on a crowded U-Bahn with one thumb free, sitting at dinner, or winding down at 23:00.

So you end up compensating the same way Christina had to with her eyes. Half the prompts you send to an assistant start by manually narrating your physical state ("I'm on a train, keep this to two bullets", "stepping out for lunch for an hour", "at dinner, only bother me if billing breaks"). The second you stop feeding it that state by hand, it either sits idle or interrupts you at the worst possible moment.

A few months ago I started wiring my phone's stops and transitions into my own agents to see what happens when they get a rough sense of physical state. (I build Timeline, an iPhone location history app with an MCP server and arrival webhooks, though OwnTracks, Home Assistant, or a Shortcuts script can supply the same signal.)

Same event, four physical states

Reminders are the least interesting part of location-aware AI. Old IFTTT recipes and "remind me when I arrive" geofences never stuck because they treated every place like a dumb light switch.

Physical state is mostly a routing and restraint layer. Where you are, how long you have been there, and what hour it is are the strongest signals for how much attention you can spare right now. Take a single production alert for a billing webhook spike:

  • At the office (10:30, stationary 1h 42m): open a T3 Code thread with the stack trace, run Codex over the last two deploys, and drop a two-line Slack DM while you are sitting at a keyboard.
  • On the U-Bahn (18:15, in transit): reception is patchy and you have one thumb free, so send a one-line heads-up to your phone and tag the secondary on-call engineer on Slack before the alert sits untouched for twenty minutes.
  • At a restaurant (20:10, stationary 48m): routine notifications are muted during dinner, so an ordinary Slack message goes unread. If error rates cross 5%, break glass with a single phone call instead of posting a thread.
  • At home (22:45, stationary 1h 15m): pull the failing commit in Amp and prepare a revert PR without merging it, then send one lock-screen summary with a confirm button.

Now swap that production alert for a flaky CI test on an open PR. At the office, the agent re-runs the job and attaches the trace to your active Pi session. On the train home, it re-runs once in the background and keeps your phone silent. At dinner or at 22:45 on the sofa, it holds everything until morning. Once an agent can read your place, dwell time and local clock, the "If I'm at X, do Y" rule handles itself, and you stop typing out the same situational guardrails over and over.

Nervous systems compress

Human proprioception does not dump raw firing rates from thousands of muscle spindles straight into your conscious mind. Your spinal cord and cerebellum compress that flood into a low-bandwidth awareness of posture and movement.

The naive way to give an LLM location context does the exact opposite. Piping raw GPS coordinates into a cloud service every thirty seconds kills an iPhone battery, fills context windows with jittery latitude and longitude rows around the same café table, and feels creepy to run.

Software proprioception works much better as a small discrete state machine. An agent almost never needs to know you are standing at 52.526418, 13.401827. It needs four small fields: whether you are stopped or moving, a plain name for the place (or travel mode like walking, transit, or driving), how long you have been there, and the local hour.

To feel natural, you need both pull and push. Pull is an MCP query when you ask a question ("which restaurants in Mitte have I actually gone back to more than once?"). Push is a transition event (arrived or left) delivered over a webhook or MCP Events when your state flips, so an agent can wake up, check what changed, and decide whether anything is worth doing.

Phantom limbs and noisy nerves

Real sensory organs are noisy, and phones are no exception. Once you live with physical state wired into an agent loop, you run straight into the software version of phantom-limb bugs.

Confirmation delay. Battery-friendly visit detection on iPhone (CLVisit) deliberately waits several minutes before declaring that you stopped somewhere, because otherwise every red light and bus stop looks like a visit. That delay makes phone visits useless for unlocking a front door or flipping hallway lights as you step inside. They only fit work that tolerates a five-minute grace window: triaging an alert on your commute, surfacing an errand note once you are in the store, or starting background repo prep on the walk to lunch.

Indoor drift. GPS signals bounce off brick and concrete. Sit in a café next door to a gym or hardware store, and your phone can briefly place you on the wrong side of the wall. If an agent treats a single noisy reading as certainty, it starts firing gym routines while you are drinking an espresso.

Missing data is not stillness. Phones run out of battery, ride through underground tunnels, or get throttled by iOS in the background. A gap in callbacks proves only that the server heard nothing; it is never proof that you stayed put.

Because the nerves are imperfect, the agent has to treat physical context as probabilistic and pair it with two boring safety rails. First, silence by default: if place, dwell time and local hour do not line up cleanly with a real routine, emit nothing and leave the lock screen alone. Second, reversibility: background actions while your phone sits in your pocket should be cheap to undo (drafting a reply, rebasing a feature branch, flipping a Slack status, queuing a note), while sending email, merging to main, or spending money still waits for a human thumb.

What stays in the skull

The privacy boundary follows the same nervous-system rule: raw sensory processing belongs on the device, inside the skull. Raw GPS breadcrumbs, motion samples and camera roll photos never need to leave your phone.

When you do opt into connecting an agent, only the compressed posture layer should cross the wire: resolved stops, travel modes, notes you wrote, and explicit arrival or departure events. Every event that wakes an agent up should sit in a plain delivery log so you can inspect the exact payload that triggered a run at 18:15 and revoke access in one click.

Adding a brain, eyes and hands made LLMs capable inside a terminal or chat tab. Giving them a basic sense of where you physically are is what stops them from feeling like brilliant interns locked in a windowless room. If you want to see concrete routines wired up with MCP and arrival webhooks, there are interactive examples on use cases and a field report from living with one in Grok Bot symbiosis.

Timeline for iPhone

Give your agents your timeline and arrival events.

Free on iPhone · No account
Download Timeline

More guides

  • Grok Bot symbiosis

    Connecting my phone’s arrivals and departures to Grok Bot, and getting it to stay quiet.

    7 min
  • Ask ChatGPT or Claude about your location history

    What works and what breaks when you upload a Google export or connect via MCP.

    7 min
  • MCP Events: who writes the first prompt now?

    What SEP-3415 specifies, what ChatGPT and Vercel shipped, and the security shift.

    7 min