Algorithm and Architecture Design of Virtual View Synthesis in High Definition Free-Viewpoint TV System

• Main Authors: Pin-Chih Lin, 林品志
• Other Authors: Liang-Gee Chen
• Format: Others
• Language: zh-TW
• Published: 2010
• Online Access: http://ndltd.ncl.edu.tw/handle/79824704496364088621

碩士 === 國立臺灣大學 === 電子工程學研究所 === 98 === Multi-view video and its applications are the epochal impacts to the history of TV display system for bringing the viewers a three-dimensional and real perceptual experience by transmitting different video sequences simultaneously on the display. By special multi-view displays, different views are projected to different eyes of viewers. As the display technology growing, more and more related applications, like 3D-TV and free-viewpoint TV (FTV) are closer to be realized. Further, the requirement of high quality video is emerged in these years. Especially for the reality achievement of FTV system, displayed images are demanded to be synchronized to the viewpoint change of viewers with any possible locations and viewing angles. Multi-view video format is not enough to support free viewpoint sequences since its samples of spatial dimension is finite. For this purpose, the virtual view synthesis algorithm is developed for rendering images seen from any virtual viewpoints by the finite source of images seen from some fixed viewpoints only. In so doing, FTV system is therefore able to be fulfilled and the virtual reality is established for viewers in the future. In this thesis, Single Iteration View Synthesis (SIVS) algorithm is firstly proposed. To support the smooth and free view-point switching in realizing the virtual view synthesis technique for FTV, both matrix-based depth image based rendering and the complex virtual view interpolation schemes are required. In order to reduce the high computation complexity and avoid the iterative processing scheduling in conventional view interpolation flows, a single-iterative view synthesis algorithm is proposed. By the usage of the proposed algorithm, the redundant warping operations are reduced by 86\%. In addition, based on the proposed artifact detecting and removing algorithm, artifacts due to imperfect depth maps can be detected and eliminated at the same time. Therefore, no additional post-processing or iteration is required and the single iteration processing is achieved. Second, another serious problem arose from free viewpoint supporting is the complicated epipolar geometry constraint for non-restricted camera setting. By the concept of rectification and un-rectification flow, the hardware design of free viewpoint virtual view synthesis is therefore feasible. The convenience of supporting variant types of sloped block loading is also bringed on and the core engine of parallel line-based warping becomes easier to develop. Furthermore, Line drift compensation and deviation detection are implemented to reduce the line-shaped holes in order to resolve the artifact of synthesized image caused by rectification and un-rectification flow. 87.8\% of artifact can be recovered by those two compensation schemes and only 0.54\% of image pixels become un-warped by the proposed virtual view synthesis hardware design in experimental analysis. A low-area architecture is also proposed by employing homographic transform concept and the linear-interpolated approximation algorithm, the large area requirement due to the synthesis parameters is resolved. In addition, redundant information for fraction bits of parameters is further reduced by precision fitting analysis. 95.9\% of area for matrix parameter rendering stage and 69.5\% for vector transform stage are reduced with only 0.0059 dB overhead of PSNR performance. More details are introduced of hardware architecture in the section of hardware designing, such like pipelining scheme and parallel mode extension so as to increase the hardware performance and accelerate the processing throughput. In order to resolve the data bandwidth problem, input and output buffering techniques are proposed and adapted to rectification and un-rectification flow. Based on the proposed algorithm and architecture, a "High Definition FTV System Virtual View Synthesis Engine" with the specification of Quad-HD 4096x2160 sequence with 24 fps for 9 simultaneous different viewpoints and no restriction about camera arrangement and rotation. And it accompanies with the chip design "FTV-system setup box" which is introduced in the end of this thesis.