Real robots have no “pause button.” Mainstream vision-language-action models often generate a future sequence of actions in one pass. The larger the model, the longer the inference time, but a robot cannot stop every few hundred milliseconds to wait for the next action segment. This is especially true in highly dynamic tasks such as throwing, where a single pause can cause accumulated velocity to drop and the entire task to fail.
As a result, real deployments more commonly use asynchronous inference: the robot continues executing its current actions while the model computes the next segment in the background—the hand performs A while the model is already calculating B. But this creates a difficult problem for online reinforcement learning. When a model generates an action sequence in one pass, often only part of it enters the physical world, so the actions the model “planned” and the actions the robot “actually performed” are no longer consistent.
Recently, the foundation model team at Stardust Intelligence released SmoothRL, an online reinforcement learning framework that can execute asynchronously. It divides an action sequence by execution status into three zones: the committed zone, the execution zone, and the discard zone. The core idea is direct: whatever the robot actually executes is what should be used for reinforcement learning, and gradients should flow only through the portion that will actually be executed.
It also follows a second principle: reinforcement learning in deployment. Training rollouts directly run asynchronous inference, and a replay buffer records trajectories with real temporal relationships, so training and deployment follow the same timing cadence. In implementation, it uses π0.5, fine-tuned per task, as the base policy, with a lightweight TD3-style actor-critic predicting residual corrections in the original action space. Stardust S1 executes actions at 30Hz and receives a new inference result every 200 milliseconds.
Tests were run on the Stardust S1 tendon-driven body and three real-machine tasks: dynamic throwing success rate rose from 39% with the base policy to 94%; pen-capping, which allows about 5 mm of pose error, rose from 8% to 83%; and package opening, which requires inserting into an approximately 2 mm-wide lid seam, rose from 30% to 90%. In one real autonomous throw, end-effector acceleration RMS dropped 52% and jerk dropped 47%, making the motion noticeably smoother.
The authors also set clear boundaries: reinforcement