Real-Time All-in-Focus Video-Based Rendering Using a Network Camera Array

Yuichi Taguchi Keita Takahashi Takeshi Naemura

The University of Tokyo


Our camera array system	Output synthesized images and depth maps, calculated at various viewpoints using the same input frames.

Abstract

We present a real-time video-based rendering system using a network camera array. Our system consists of 64 commodity network cameras that are connected to a single PC through a Gigabit Ethernet. To render a high-quality novel view, we estimate a view-dependent per-pixel depth map in real-time by using a layered representation. The rendering algorithm is fully implemented on a GPU, which allows our system to efficiently use CPU and GPU independently and in parallel. Using QVGA input video resolution, our system renders a free-viewpoint video at up to 30 fps depending on the output video resolution and the number of depth layers. Experimental results show high-quality images synthesized from various scenes.

Videos

Output free-viewpoint videos

Meeting room (4.4 MB)
Soccer (4.5 MB)
Juggling (9.7 MB) Special Thanks: Hideki Hashimoto, Fumiya Ichino, and Takumi Kida

Comparison of depth maps with or without temporal depth smoothing

Meeting room (6.8 MB)
Soccer (5.9 MB)
Juggling (12 MB)

Comparison of free-viewpoint videos with or without temporal depth smoothing

The videos rendered at a fixed viewpoint show the effect of the temporal depth smoothing more clearly, while the videos rendered at a moving viewpoint confirm that the temporal smoothing can be used even when the rendering viewpoint smoothly moves.

Meeting room (fixed viewpoint) (6.6 MB) / Meeting room (moving viewpoint) (12 MB)
Soccer (fixed viewpoint) (8.1 MB) / Soccer (moving viewpoint) (8.2 MB)
Juggling (fixed viewpoint) (16 MB) / Juggling (moving viewpoint) (18 MB)

Papers

Yuichi Taguchi, Keita Takahashi, and Takeshi Naemura: ``Design and Implementation of a Real-Time Video-Based Rendering System Using a Network Camera Array,'' IEICE Trans. Information and Systems, vol. E92.D, no. 7, pp. 1442-1452, July 2009.
Yuichi Taguchi, Keita Takahashi, and Takeshi Naemura: ``Real-Time All-in-Focus Video-Based Rendering Using a Network Camera Array,'' Proc. 3DTV-Conference 2008, pp. 241--244, Istanbul, Turkey, May 2008. [ IEEE Xplore ]