Video Synthesis of Personalized Moderations

Abstract

This thesis is concerned with novel view and video synthesis: For a given set of calibrated images or videos, we would like to render the captured scene from novel camera angles, for example to render natural scene transitions between two camera angles of that scene.

The use of video data imposes significant cost on each observed view, since we need one camera per view, which is not the case for static scenes, as different camera angles may be captured by the same camera over time. Furthermore cameras must be synchronized and calibrated, which also increases the setup time and complexity for each additional camera. Therefore, we set as an additional constraint that the novel views must be rendered with the help of only a small number of captured camera angles.

We employ a novel view synthesis algorithm based on Neural Radiance Fields (or NeRFs for short), which was introduced by Mildenhall et al. [MST+20]. To explore the goals and constraints above we perform a number of experiments:

1. We analyse the effect of various design decisions on the performance of NeRFs when only few captured views are available.
2. We develop a series of architectural changes to NeRFs that enable video synthesis, i.e. novel view synthesis for dynamic scenes.
3. We try to alleviate the issues with visual quality and geometric consistency that arise when only very few camera angles are available to supervise NeRFs. To this end we develop a series of alternative loss functions which may be used to optimize NeRFs in such cases.
   

Additionally, we discuss the following issues: We analyze a particular type of local minimum and we alleviate some of the computational constraints that complicate video synthesis with a sampling technique.