MemoAct: Atkinson–Shiffrin-Inspired Memory-Augmented Visuomotor Policy for Robotic Manipulation

Memory-augmented robotic policies are essential in handling memory-dependent tasks. However, existing approaches typically rely on simple observation window extensions, struggling to simultaneously achieve precise task state tracking and robust long-horizon retention. To overcome these challenges, inspired by the Atkinson–Shiffrin memory model, we propose MemoAct, a hierarchical memory-based policy that leverages distinct memory tiers to tackle specific bottlenecks. Specifically, lossless short-term memory ensures precise task state tracking, while compressed long-term memory enables robust long-horizon retention. To enrich the evaluation landscape, we construct MemoryRTBench based on RoboTwin 2.0, specifically tailored to assess policy capabilities in task state tracking and long-horizon retention. Extensive experiments across simulated and real-world scenarios demonstrate that MemoAct achieves superior performance compared to both existing Markovian baselines and history-aware policies.

(a) An example of a memory-dependent task. (b) Policies lacking historical awareness fail under identical observations, while existing representative memory mechanisms suffer from limited long-horizon retention and poor task state tracking. (c) Inspired by the Atkinson–Shiffrin memory model, we propose MemoAct, which simultaneously enables precise task state tracking and robust long-horizon retention. (d) Results on MemoryRTBench, RMBench and real-world experiments demonstrate that MemoAct significantly outperforms baseline algorithms.

The sensory distillation module first encodes RGB images and proprioceptive states into high-fidelity features, termed sensory memory. This memory serves as a query to retrieve relevant historical context from the long short-term memory bank, which is processed by a temporal transformer encoder. Subsequently, a gating network adaptively fuses the retrieved history with the current sensory memory to produce a condition token. Guided by this token, the action decoder iteratively denoises random noise into history-aware action trajectories. Finally, the consolidation module updates the memory bank after each forward pass. For a detailed introduction to the memory consolidation module, please refer to our paper.

To comprehensively evaluate task state tracking and long-horizon retention, we introduce MemoryRTBench, which consists of four simulation tasks. To ensure a thorough assessment, we also benchmark MemoAct on the language-independent tasks of RMBench, followed by validation on two real-world tasks. These tasks are characterized by multiple subtasks and frequent occurrences of perceptually identical observations across different states, compelling the policy to rely on historical context for disambiguation. Moreover, their extended execution horizons require recalling the initial states of objects, posing a strict demand for robust long-term memory.