SeeBFM is BFM-Zero trained further with a height scan of the ground around the robot. A reward, a motion or a typed sentence becomes one prompt, and the policy reads the ground to carry it out.
With its scan made to read level ground, it crosses 4%.
Trained the same way without the scan, it keeps 76% of its backward speed there, and falls.
A student turns the two cameras' depth into the scan, and a sentence encoder turns a command into a prompt.
Pick a prompt and a model to see them on four grounds at once. The bar under each clip shows how all its attempts on 0.10–0.15 m steps ended.
Each clip names its prompt. The numbers under it count all attempts of that kind.
Pick a model, a prompt and a terrain, and replay any of its attempts in 3D. It holds 31,528 attempts from simulation: SeeBFM with 22 reward prompts and 121 typed sentences, and the three other models with the six locomotion prompts.
A 598-point height scan enters the policy and the networks that score it, each through a layer that starts at zero, so training begins exactly at BFM-Zero.
The network that turns a reward into a prompt still reads only the body, so one reward gives one prompt on every terrain. Subtracting that network's average from each prompt keeps every walking prompt going the way it names.
A student learns to predict the scan from a head and a body camera, first by matching the scan and then by matching the policy's actions.
A sentence encoder and a small network, trained on the descriptions that come with the motion data, map a typed command to a prompt.
The scan's points on the stairs ahead, which the policy reads, and the head camera's view, which the depth student uses.
How the six locomotion prompts do on 0.10–0.15 m steps. On flat ground SeeBFM also tracks motion better than BFM-Zero, with an error of 0.68 against 0.93.
The network that turns a reward into a prompt gives the same prompt on every terrain if it never sees the ground. Giving it the scan made terrain tracking worse, with an error of 1.31 against 1.14.
How it worksWithout that, every prompt leans toward one shared behavior. On flat ground the sideways prompt then moves at only 0.05 to 0.36 m/s in its direction, and at 0.73 to 0.88 m/s once the average is removed.
How it worksBackward and sideways walking nearly stop on stairs and boxes, and the robot learns this along with the scan. Adding 30 h of that motion on stairs and boxes to the data left both below a third of their flat-ground speed there, so the cause is still open.
Speed kept on reliefTyped wordings that make the robot walk on flat ground cross stairs, boxes, slopes and obstacles in 86 to 97% of attempts, against 93% for the reward prompt. One prompt per command cannot hold a jump.
Typed commandsA real robot has no height scan, so a student learns to predict it from a head camera and a body camera, and the policy reads the prediction instead. It crosses stairs nearly as often as with the perfect scan but falls more, mostly on ground the cameras had not yet seen.
