MemoAct: Atkinson–Shiffrin-Inspired Hierarchical Memory-Augmented Policy for Robotic Manipulation

Memory-augmented robotic policies are essential in handling memory-dependent tasks. However, existing approaches typically rely on simply extending the observation window, struggling to simultaneously achieve precise task-state tracking and robust long-horizon retention. To overcome these challenges, inspired by the Atkinson–Shiffrin memory model, we propose MemoAct, a hierarchical memory- augmented policy that leverages distinct memory tiers to tackle specific bottlenecks. Specifically, sensory memory filters immediate perceptual inputs, lossless short-term memory supports precise task-state tracking, and compressed long-term memory facilitates robust long-horizon retention. To enrich the evaluation landscape, we construct MemoryRTBench based on RoboTwin 2.0, comprising 6 manipulation tasks that systematically evaluate policy memory capabilities across three dimensions: sequential, spatial, and episodic memory. Extensive experiments across simulated and real-world scenarios demonstrate that MemoAct achieves superior performance compared to both existing Markovian baselines and history-aware policies.

(a) An example of a memory-dependent task. (b) Policies lacking historical awareness fail under identical observations, while existing representative memory mechanisms suffer from limited long-horizon retention and poor task-state tracking. (c) Inspired by the Atkinson–Shiffrin memory model, we propose MemoAct, which simultaneously enables precise task-state tracking and robust long-horizon retention. (d) Results on MemoryRTBench, RMBench, and real-world experiments demonstrate that MemoAct significantly outperforms baseline algorithms.

First, the sensory distillation module encodes RGB images and proprioceptive states into high-fidelity sensory memory. Subsequently, the distilled sensory memory queries relevant historical context from the long short-term memory bank. Next, a gating network adaptively fuses the retrieved history with the current sensory memory to produce a condition embedding, which guides the action decoder to iteratively denoise noisy action trajectories into history-aware action trajectories. Finally, the long short-term memory consolidation module updates the memory bank after each forward pass. Please refer to our paper for details.

To evaluate MemoAct, we establish MemoryRTBench with 6 manipulation tasks assessing sequential, spatial, and episodic memory. These three memory dimensions evaluate the ability to execute sub-tasks in the prescribed order, recall the initial scene state after intermediate operations, and repeat a specified task the required number of times, respectively.

As shown in Tables I, II, and III, MemoAct achieves the best overall performance across simulation and real-world evaluations. It obtains average success rates of 94.5%, 49.1%, and 77.5% on MemoryRTBench, RMBench, and real-world tasks, outperforming the strongest baseline MVMP by 19.1%, 20.6%, and 11.25%, respectively. As shown in Table IV, MemoAct demonstrates certain generalization ability, and its memory module remains effective under distribution shifts. As shown in Table VI, our memory module can be seamlessly integrated into existing policies to improve history-aware decision making.

As shown in the figure above, the long-term and short-term memory capacities control long-horizon retention and task-state tracking, respectively.

We conduct an efficiency evaluation using 50 CTBO expert demonstrations with batch size 32. All measurements are conducted on a single NVIDIA RTX 4090 GPU. We measure the parameter count, training time, inference latency, and GPU memory usage for MemoAct and all compared baselines. The results are summarized above.