homunculus
One command line. Any body that describes itself.
An agent drives a robot with a handful of commands: describe, observe, goal, stop. Those commands never change. What changes is the robot's definition: which joints it has and in what units, which goals it accepts, which cameras are its own.
Below, four bodies run the same commands: three published robot models in MuJoCo, and one that isn't MuJoCo at all. Every command, output, and image on this page was recorded from the real CLI. The only edit is shortening long scratch-folder paths.
What the agent saw: images robot observe saved from each robot's own camera. Simulated physics (MuJoCo 4.0.1), not hardware.
One contract, any body
The agent talks to the robot CLI. The CLI talks to a robot service, which enforces the same rules for every body: one controller at a time, goals checked against declared limits, success decided by measurement. Under the service, a backend speaks to one particular body.
Defining a body properly
A body is defined by what its backend declares and does. Declare it honestly and every command works. Leave something out and the CLI refuses that thing by name, instead of guessing.
What the body is
- Joints, with units and ranges
- Goals it accepts (profiles)
- Its own cameras
- Spectator cameras, for people
- Clocks and frames
- Limits and recovery ops
What the body does
- Take a goal: apply()
- Send it commands each tick
- Report measured positions
- Report where readings came from
- Stop and hold: stop()
What the body sees
- Image bytes, as encoded
- Lens: focal and centre, px
- Camera pose in a frame
- Capture time on its clock
- Converted once, at the source
For a MuJoCo model, all of this comes from four arguments: the model file and which of its joints and cameras belong to the robot. For anything else (a real arm's driver, another simulator, a recording), you write the backend. examples/lift.py is a complete one in 148 lines.
the definitions used on this page
Same commands, four bodies
Each cell is one command run against one body, colored by what the CLI reported. Nothing is faked to look uniform. A body that lacks a capability refuses the command and names what it does offer.
command outcomes
Run it yourself, step by step
The recorded sessions, rendered. Pick a body, then a step. The note above each output says what to look for.
What each body declared
Every joint each body declared to robot describe, as the range a goal may use. The dot is where the body holds at start. The CLI checks every goal against these ranges before anything moves: lift=1.5 was refused with limit_exceeded.
declared joint ranges
Is the camera information right?
An image is only useful to an agent if the CLI says correctly where it was taken from. For each saved observation, a separate MuJoCo process rebuilt the scene at the observed joint angles and took each object's true centre. That point was projected using only the lens and pose from the CLI's JSON. A segmentation render from the model's own camera then shows whether the pixel is on that object.
projected object centres
Every image, from any source, is the same small record. MuJoCo, the G1 task simulator, or a real camera driver each convert to it once, inside capture().
focal_px,principal_px: an undistorted pinhole lens, in pixels from the top-left.camera_pose: where the camera was, in a named frame. Optical axes: x right, y down, z forward.capture_time_ns: on the same clock as the joint readings.file: the saved image the agent opens.
Not shown here
- No physical robot. Every result here is simulation, and the CLI labels its readings
simulation_ground_truth. A real robot's backend would reportphysics: real. - MuJoCo cameras send colour only. Depth comes from the G1 task backend.
- Goals are joint angles. "Move the gripper to this pixel" waits for tool-pose goals.
- A custom MuJoCo model can only use cameras its own files define. MJCF can't add a camera inside a robot defined in another file, which is why the UR5e has a fixed RealSense rather than a wrist camera.
- The SO-101 and UR5e scene files (desk, bench, objects, camera pole) were written for this page and are not in homunculus. The Panda scene and
examples/lift.pyship with it.
Robots and camera: MuJoCo Menagerie commit 370c439 (Franka Emika Panda, RobotStudio SO-101, Universal Robots UR5e, Intel RealSense D435i), each under its own license there. Objects: Google Scanned Objects for MuJoCo commit 6ff8d27. The meshes and textures are by Google under CC BY 4.0 (Downie et al., ICRA 2022), and the MJCF is by Kevin Zakka under MIT. The scan in the folder Great_Dinos_Triceratops_Toy is an Android figure; it is called "figure" here. Design after the Terminal-Bench 4.0 post.