基于时序增强的光场视频编码算法
Temporal-Enhancement-Based Light-Field Video Coding Algorithm
-
摘要: 光场视频能够同时记录场景的时空与角度信息,为沉浸式体验提供了可能,但其海量数据给存储和传输带来了严峻挑战。为解决此问题,基于稀疏采样的重建方法应运而生。它通过仅编码传输少量关键视图,在解码端利用深度学习网络恢复完整视点,成为实现低码率光场视频压缩的有效途径。然而,现有的大多数稀疏重建方法在设计上存在一个核心局限:由于它们通常将视频序列视为独立的图像帧集合,采用逐帧重建的策略,虽然充分挖掘了单帧内的空间-角度相关性,却完全忽略了视频序列中蕴含的宝贵时序信息,导致重建时无法利用相邻帧信息来修复重建受限区域,使得性能出现瓶颈。其次,在大视差场景下,由于视角遮挡频繁,单帧的稀疏视图信息不足以推断被遮挡区域的内容,而这些缺失的纹理细节往往在相邻帧的对应位置是可见的。针对上述问题,本文提出了一种融合时序信息的光场视频稀疏编码与密集重建框架。该框架的核心在于设计了一个基于时序增强的光场视频重建网络。不同于传统的独立帧处理,该网络创新性地引入了时序增强模块,通过利用可变形卷积技术,自适应地对齐并融合相邻帧的特征。这种机制能够从参考帧中有效检索纹理细节,对当前帧因遮挡而缺失的信息进行补偿,从而在重建过程中强制施加时序一致性约束。实验结果表明,本文提出的框架在不显著增加码率开销的前提下,实现了更优的率失真性能,有效验证了所提时序增强策略的优越性。Abstract: A light field (LF) video captures both the temporal evolution and angular information of a scene, enabling advanced immersive applications such as free-viewpoint videos and virtual reality. However, the extremely high dimensionality of LF video data results in massive storage and transmission costs, which severely limit its practical deployment. To address this challenge, sparse-sampling-based reconstruction methods have emerged as effective solutions for low-bitrate LF video compression. By encoding and transmitting only a small number of key views, these methods reconstruct the full LF video at the decoder using deep-learning models, thereby significantly reducing the required bitrate. However, despite their success, most existing sparse-LF-reconstruction methods suffer from a fundamental limitation in that they treat LF videos as collections of independent frames and adopt a frame-by-frame reconstruction strategy. While such approaches effectively exploit the spatial-angular correlations within a single LF frame, they completely overlook the rich temporal information inherently present in LF video sequences. As a result, these methods fail to leverage information from adjacent frames to assist the reconstruction of regions that are poorly constrained by sparse angular sampling in the frame under consideration, leading to a clear performance bottleneck. This limitation becomes particularly severe in scenes with large disparities and frequent occlusions. In such cases, the sparse views within a single frame are often insufficient to infer the content of occluded regions, causing missing structures or texture degradation in the reconstructed LF views. Nevertheless, because of scene or camera motion, these occluded regions are often visible in neighboring frames at the corresponding spatial locations. Ignoring this temporal redundancy prevented the existing methods from fully exploiting the available information for high-quality LF video reconstruction. To overcome these challenges, this study proposes a sparse coding and dense reconstruction framework for LF videos incorporating temporal information, which explicitly incorporates temporal cues into the sparse-reconstruction process. The core component of the proposed framework is an LF video reconstruction network based on temporal enhancement, which is designed to move beyond the traditional independent-frame processing. The proposed network introduces a dedicated temporal-enhancement module that facilitates effective feature interactions across adjacent frames. Specifically, the temporal-enhancement module employs deformable convolutions to adaptively align features extracted from neighboring reference frames with those of the current target frame. This adaptive alignment mechanism allows the network to handle complex motions and large disparity variations, enabling it to selectively retrieve complementary texture details from temporally adjacent frames. By fusing these aligned features, the proposed network compensates for the missing information in the current frame due to sparse angular sampling or occlusions while implicitly enforcing temporal consistency during reconstruction. Extensive experiments conducted on representative LF video datasets demonstrate that the proposed framework consistently outperforms the state-of-the-art sparse-LF-reconstruction methods under various sampling configurations. Notably, the proposed method achieves superior rate-distortion performances without introducing significant additional bitrate overheads. Qualitative results further demonstrate that the proposed temporal-enhancement strategy effectively restores occluded regions and preserves fine texture details, particularly in challenging large-disparity scenes. These results validate the integration of temporal information as an effective and practical solution for improving sparse LF video reconstruction quality.
下载: