Memory Building
RT‑Cache builds a unified vector memory from diverse image–action trajectories. The memory stores universal vision features across different morphologies and tasks.
Memory Building Demo
To be updated soon
To be updated soon
Real robots must repeat the same behavior in new environments with very little new data, yet modern controllers either incur heavy per‑step inference or require deployment‑time fine‑tuning.
We present RT‑Cache, a training‑free retrieval‑as‑control pipeline that caches diverse image–action trajectories in a unified vector memory and, at test time, embeds the current frame to retrieve and replay multi‑step snippets, replacing per‑step model calls. A hierarchical search keeps lookups sub‑second at million scale, shifting cost from compute to storage and enabling real‑time control on modest GPUs.
Across real‑robot tasks and large open logs, RT‑Cache achieves higher success and lower completion time than strong retrieval baselines (≈2× success, ~30% faster), and a single‑episode anchoring study shows immediate adaptation to a contact‑rich task without fine‑tuning.
To be updated soon
RT‑Cache builds a unified vector memory from diverse image–action trajectories. The memory stores universal vision features across different morphologies and tasks.
To be updated soon
Our hierarchical search enables sub‑second lookup at million scale through centroid filtering and local indexing for efficient multi‑step replay.
To be updated soon
RT‑Cache replays coherent N‑step action snippets instead of per‑timestep copying. Use the slider to see how different horizon lengths (N=1,3,5) affect the replay behavior.
Query Frame
Retrieved Snippet
Using RT‑Cache, robots can execute retrieved action snippets in real‑time, achieving higher success rates with faster completion times.
To be updated soon
RT‑Cache builds upon recent advances in retrieval‑based robot learning and memory‑augmented control.
VINN pioneered visual imitation through nearest neighbor retrieval for robot manipulation tasks.
Behavior Retrieval and other works explore offline policy retrieval but lack test‑time adaptation capabilities.
Recent transformer‑based policies like RT‑1 and RT‑X require heavy per‑step inference.
Our hierarchical retrieval approach enables million‑scale memory search while maintaining sub‑second lookup times for real‑time control.
@article{rtcache2025,
title={RT-Cache: Training-Free Retrieval for Real-Time Manipulation},
author={Kwon, Owen and George, Abraham and Bartsch, Alison and Farimani, Amir Barati},
journal={arXiv preprint arXiv:2501.01234},
year={2025}
}