Memory-V2V: Augmenting Video-to-Video Diffusion Models with Memory

Method

Memory-V2V is an efficient finetuning framework that equips video-to-video foundation models with explicit memory. For multi-turn novel view synthesis, previously generated videos and their camera poses are stored in an external cache, and relevant past results are retrieved by computing VideoFOV overlap with the current target trajectory. To control the large number of conditioning tokens, we apply priority-based dynamic tokenization using different kernel sizes and further compresses low-importance frames with an adaptive, learnable convolutional compressor inside the model, enabling compact yet informative memory tokens that preserve cross-consistency across sequential edits.

For long video editing, the input is divided into multiple segments, and each new segment is generated sequentially by retrieving previously produced segments based on source-video similarity. Since dedicated long-video editing datasets are scarce, the target video can be extended using a generative model, and the extended sequence is used as memory during training. In this way, Memory-V2V can easily and reliably incorporate memory by adopting retrieval strategies tailored to each video-to-video task.

Proof-of-Concept Experiments: Ideal Context Encoder

We investigate which type of memory encoder best preserves and transfers information by conditioning the model on the states from CUT3R[5], LVSM[6], and a VAE. While CUT3R and LVSM fail to provide sufficiently informative guidance to the video diffusion model, the VAE state effectively conveys the necessary information, leading to consistent generation of same region in different views.