DynIBaR: Neural Dynamic Image-Based Rendering
CVPR• 2023
Abstract
We address the problem of synthesizing novel views from a monocular video
depicting a complex dynamic scene. State-of-the-art methods based on temporally
varying Neural Radiance Fields (aka dynamic NeRFs) have shown impressive
results on this task. However, for long videos with complex object motions and
uncontrolled camera trajectories, these methods can produce blurry or
inaccurate renderings, hampering their use in real-world applications. Instead
of encoding the entire dynamic scene within the weights of MLPs, we present a
new approach that addresses these limitations by adopting a volumetric
image-based rendering framework that synthesizes new viewpoints by aggregating
features from nearby views in a scene-motion-aware manner. Our system retains
the advantages of prior methods in its ability to model complex scenes and
view-dependent effects, but also enables synthesizing photo-realistic novel
views from long videos featuring complex scene dynamics with unconstrained
camera trajectories. We demonstrate significant improvements over
state-of-the-art methods on dynamic scene datasets, and also apply our approach
to in-the-wild videos with challenging camera and object motion, where prior
methods fail to produce high-quality renderings. Our project webpage is at
dynibar.github.io.