homunculus

One command line. Any body that describes itself.

An agent drives a robot with a handful of commands: describe, observe, goal, stop. Those commands never change. What changes is the robot's definition: which joints it has and in what units, which goals it accepts, which cameras are its own.

Below, four bodies run the same commands: three published robot models in MuJoCo, and one that isn't MuJoCo at all. Every command, output, and image on this page was recorded from the real CLI. The only edit is shortening long scratch-folder paths.

What the agent saw: images robot observe saved from each robot's own camera. Simulated physics (MuJoCo 4.0.1), not hardware.

One contract, any body

The agent talks to the robot CLI. The CLI talks to a robot service, which enforces the same rules for every body: one controller at a time, goals checked against declared limits, success decided by measurement. Under the service, a backend speaks to one particular body.

agent  (Claude, a script, a person)
robot describe · observe · goal · stop
robot CLI + service: one contract
Backend: describe() · apply() · stop() · advance() · capture()

Defining a body properly

A body is defined by what its backend declares and does. Declare it honestly and every command works. Leave something out and the CLI refuses that thing by name, instead of guessing.

describe()
declare

What the body is

  • Joints, with units and ranges
  • Goals it accepts (profiles)
  • Its own cameras
  • Spectator cameras, for people
  • Clocks and frames
  • Limits and recovery ops
advance()
act + measure

What the body does

  • Take a goal: apply()
  • Send it commands each tick
  • Report measured positions
  • Report where readings came from
  • Stop and hold: stop()
capture()
see

What the body sees

  • Image bytes, as encoded
  • Lens: focal and centre, px
  • Camera pose in a frame
  • Capture time on its clock
  • Converted once, at the source

For a MuJoCo model, all of this comes from four arguments: the model file and which of its joints and cameras belong to the robot. For anything else (a real arm's driver, another simulator, a recording), you write the backend. examples/lift.py is a complete one in 148 lines.

the definitions used on this page

Same commands, four bodies

Each cell is one command run against one body, colored by what the CLI reported. Nothing is faked to look uniform. A body that lacks a capability refuses the command and names what it does offer.

command outcomes

succeededrefused before movingfailed by measurementnot run

Run it yourself, step by step

The recorded sessions, rendered. Pick a body, then a step. The note above each output says what to look for.

What each body declared

Every joint each body declared to robot describe, as the range a goal may use. The dot is where the body holds at start. The CLI checks every goal against these ranges before anything moves: lift=1.5 was refused with limit_exceeded.

declared joint ranges

Is the camera information right?

An image is only useful to an agent if the CLI says correctly where it was taken from. For each saved observation, a separate MuJoCo process rebuilt the scene at the observed joint angles and took each object's true centre. That point was projected using only the lens and pose from the CLI's JSON. A segmentation render from the model's own camera then shows whether the pixel is on that object.

projected object centres

lands on the objectmisses

Every image, from any source, is the same small record. MuJoCo, the G1 task simulator, or a real camera driver each convert to it once, inside capture().

  • focal_px, principal_px: an undistorted pinhole lens, in pixels from the top-left.
  • camera_pose: where the camera was, in a named frame. Optical axes: x right, y down, z forward.
  • capture_time_ns: on the same clock as the joint readings.
  • file: the saved image the agent opens.

Not shown here

Robots and camera: MuJoCo Menagerie commit 370c439 (Franka Emika Panda, RobotStudio SO-101, Universal Robots UR5e, Intel RealSense D435i), each under its own license there. Objects: Google Scanned Objects for MuJoCo commit 6ff8d27. The meshes and textures are by Google under CC BY 4.0 (Downie et al., ICRA 2022), and the MJCF is by Kevin Zakka under MIT. The scan in the folder Great_Dinos_Triceratops_Toy is an Android figure; it is called "figure" here. Design after the Terminal-Bench 4.0 post.