JP6501348B2 - Free-viewpoint image generation device, method and program - Google Patents

Description

  The present invention relates to a free viewpoint image generating apparatus, method, and program, and in particular, a free viewpoint image at an arbitrary virtual viewpoint from camera images of objects captured by a plurality of cameras arranged at intervals. The present invention relates to a viewpoint image generation device, method, and program.

  Interpolation technology and model-based rendering (MBR) are known as a method of generating a free viewpoint image at an arbitrary virtual viewpoint from a camera image acquired by capturing an object at a plurality of different viewpoints. It is done.

  Patent Literature 1-3 and Non-Patent Literature 1 disclose a method of free viewpoint image generation by an interpolation technique. In free viewpoint image generation by the interpolation technique, a free viewpoint image in a virtual viewpoint is generated using linear interpolation from two camera images acquired by capturing an object from different viewpoints.

  Patent Document 4 and Non-Patent Documents 2 and 3 disclose methods of free viewpoint image generation by model-based rendering. In the generation of free viewpoint images by model-based rendering, geometric 3D models of objects using polygons and voxels from many camera images acquired by shooting the objects from many viewpoints (in the following A dimension “is described as“ D ”), and the 3D model is subjected to projective transformation corresponding to the virtual viewpoint to generate a free viewpoint image in the virtual viewpoint.

JP, 2010-282295, A Unexamined-Japanese-Patent No. 2006-65862 JP 2004-297734 A JP 2012-128884 A

Seitz, S. M., & Dyer, C. R. (1996). "View Morphing", In Siggraph 1996 Conference Proceedings, Annual Conference Series, pp. 21-30. Shum, Heung-yeung, and Sing Bing Kang. "A survey of image-based rendering techniques". In Videometrics, SPIE. 1999. Chai, Jin-Xiang, et al. "Plenoptic sampling". Proceedings of the 27th annual conference on computer graphics and interactive techniques, ACM Press / Addison-Wesley Publishing Co., pp. 307-318 2000.

  However, in free viewpoint image generation by the interpolation technique, since the free viewpoint image in the virtual viewpoint is generated by linear interpolation from two camera images acquired by capturing the same object from different viewpoints, the camera arrangement There is a problem that the image quality of the generated free viewpoint image is degraded when the interval of positions becomes wide. That is, there is a problem in that it is difficult to secure sufficient image quality when a free viewpoint image in a virtual viewpoint is generated by an interpolation technique from camera images of objects captured by cameras arranged at sparse intervals. In addition, there is a problem that when the virtual viewpoint moves, the free viewpoint image does not change smoothly in response to the movement.

  On the other hand, in free-viewpoint image generation by model-based rendering, a large number of camera images are required to generate a 3D model, and free-view images of objects taken by cameras arranged at sparse intervals can be used in a virtual viewpoint Not suitable for generating viewpoint images. In addition, since a large number of camera images are handled to generate a 3D model, there is a problem that computational cost increases.

  An object of the present invention is to provide an arbitrary virtual viewpoint from camera images of objects taken by a plurality of cameras arranged at intervals, without requiring a large number of camera images as in free-viewpoint image generation by model-based rendering It is an object of the present invention to provide a free viewpoint image generation apparatus, method and program capable of generating a free viewpoint image smoothly.

In order to solve the above problems, the present invention relates to a free viewpoint image generating device that generates a free viewpoint image at an arbitrary virtual viewpoint from camera images of objects captured by a plurality of cameras fixedly arranged at intervals. Object two-dimensional information extraction means for extracting object two-dimensional information from each camera image, and calibration of the object two-dimensional information extracted by the object two-dimensional information extraction means in advance with parameters of each camera and each point of the three-dimensional coordinate system and a object 3-dimensional information extracting means for extracting the object 3-dimensional information using the parameters of each camera obtained from the relationship between each point in the two-dimensional coordinate system, a plurality Feature points of the object region of the reference image as the reference image, Extract feature points matching the feature points of the reference image in the object area of each camera image other than the image, and generate the three-dimensional point group of the object from the positions of these feature points and the parameters of each camera Object three-dimensional information extraction means for generating a three-dimensional outline and three-dimensional skeleton of an object from the three-dimensional point group as the object three-dimensional information, and a three-dimensional object from the object three-dimensional information extracted by the object three-dimensional information extraction means estimating the parameters relating to the parameters and modifications relating to the size of each part of, according to the parameter, and adjust the size of preset adjustable 3D object template model, the three-dimensional object template model adjustment means for deforming the three-dimensional object Conditioned by preparative template model adjustment means, with a projective transformation means for generating a free-viewpoint images in the virtual viewpoint by performing projective transformation corresponding to an arbitrary virtual viewpoint with respect to the deformed three-dimensional object template model The point has the first feature.

  In addition, in the adjustable 3D object template model, there is a second feature in that the surface, internal skeleton and nodes of the object are defined, and parameters related to the size of each part of the object and parameters related to deformation are defined. .

  An object is a person, and the parameter regarding the size of each part of the object has a third feature in that it includes the height, fat percentage and waist-hip ratio of the human body.

  An object is a person, and the parameter relating to the deformation of the object has a fourth feature in that it includes the positions of 18 nodes between parts of the human body.

  Further, the fifth aspect of the invention is that the parameters of each camera include internal parameters and external parameters estimated by detecting corners of respective squares in a chessboard pattern image acquired by photographing the chessboard with each camera. There is a feature.

  A sixth feature is that the object 2D information extraction means extracts an approximate object area from each camera image by the background subtraction method and then extracts an accurate object area.

Further, the 3D object template model adjusting means estimates the height of the object from the object 3D information extracted by the object 3D information extracting means, and estimates the fat ratio and the waist-hip ratio from around each part in the 3D shape. There is a seventh feature in that they adjust the value of the parameter related to the size defined by the adjustable 3D object template model.

Further, the 3D object template model adjusting means detects 18 nodes between parts of the human body from the object 3D information extracted by the object 3D information extracting means, detects the positions thereof, and thereby the adjustment is possible. An eighth feature lies in adjusting values of parameters related to deformation defined in the 3D object template model.

Moreover, the projective transformation means, specify any of the virtual viewpoint, the 3D and 3D object template model adjusted by the object template model adjustment means in terms of projective transformation on the camera image when viewed from the virtual viewpoint of the 9 There is a feature.

A tenth feature is that the image forming apparatus further comprises coloring means for coloring the free viewpoint image generated by the projection conversion means.

An eleventh feature is that a background image combining unit is further provided which combines the background image with the free viewpoint image generated by the projective transformation unit to generate a composite image.

In addition, the free viewpoint image generating device having the eleventh feature is provided, and further, a free viewpoint image is generated by an interpolation technique using two camera images acquired by cameras on both sides of an arbitrary virtual viewpoint. comprising an evaluation means for evaluating a free-viewpoint image generating apparatus, the image quality of the free viewpoint free viewpoint image generated by the image generation device and the first 11 free-view image created by the free viewpoint image generating device having the features of, A twelfth feature lies in that a free viewpoint image of the higher evaluation by the evaluation means is output.

A thirteenth feature of the present invention is that the evaluation means analyzes the power spectrum of the free viewpoint image to evaluate the image quality of each free viewpoint image.

  The present invention can be realized not only as a free-viewpoint image generation device but also as a free-viewpoint image generation method in which processing in each means is taken as a step, and also as a program that causes a computer to function as each means.

  According to the present invention, it is possible to generate a free viewpoint image at an arbitrary virtual viewpoint with high image quality even from camera images of objects taken by cameras arranged at sparse intervals, and even when the virtual viewpoint moves, Accordingly, it is possible to generate a free viewpoint image which changes smoothly. Furthermore, processing costs can be reduced because a large number of camera images are not required.

It is a functional block diagram showing a 1st embodiment of a free viewpoint image generation device concerning the present invention. It is a figure which shows the example of the chessboard pattern image utilized for estimation of the parameter of a camera. It is a figure which shows the mode of coordinate transformation of the world coordinate (3D coordinate) system, a camera coordinate (2D coordinate) system, and an image coordinate (2D coordinate) system in estimation of the parameter of a camera. It is explanatory drawing of the setting of an adjustable 3D object template model. It is an explanatory view of processing in object 2D information extraction means. It is an explanatory view of processing in object 3D information generation means. It is explanatory drawing of the process in a projective transformation means. It is a figure which shows the relationship between the 3D position of the adjusted 3D object template model, and the projection point in a projection surface. It is explanatory drawing of the process by a coloring means. It is a functional block diagram showing a 2nd embodiment of the free viewpoint picture generation device concerning the present invention. It is explanatory drawing of free viewpoint image generation by the interpolation technique.

  Hereinafter, the present invention will be described with reference to the drawings.

  The present invention is a free viewpoint image in an arbitrary virtual viewpoint from a plurality of camera images acquired by capturing an object with a plurality of fixed cameras (hereinafter, simply referred to as "cameras") arranged at intervals. In particular, it is suitable for generating a free viewpoint image with a person (player) in a sport video image as an object. However, it is also possible to use other images or other objects, for example, animals as objects, in addition to the images of sports videos and people.

  FIG. 1 is a functional block diagram showing a first embodiment of a free viewpoint image generation apparatus according to the present invention.

  The free viewpoint image generation device of this embodiment includes an object 2D information extraction unit 11, an object 3D information generation unit 12, a 3D object template model adjustment unit 13, a projection conversion unit 14, a coloring unit 15, and a background synthesis unit 16. Each means may be configured by software in one or more processors.

  The object 2D information extraction means 11 takes as input a plurality of camera images (original camera images) obtained by actually photographing an object in a field such as a field with a plurality of cameras arranged at intervals. The 2D position, outline and texture of the object in the camera image are extracted as object 2D information. The number of camera images is not as large as that of free viewpoint image generation by model-based rendering, and may be two to three, for example.

  The object 3D information generation unit 12 generates object 3D information using the object 2D information extracted by the object 2D information extraction unit 11 and the parameters of each camera that acquired the original camera image as input.

  The 3D object template model adjusting means 13 receives the adjustable 3D object template model set in advance offline and the object 3D information generated by the object 3D information generating means 12 as input, and the adjustable 3D object template model is an object 3D information Adjust according to to generate an adjusted 3D object template model.

  The projective transformation unit 14 performs projective transformation corresponding to an arbitrarily specified virtual viewpoint on the adjusted 3D object template model generated by the 3D object template model adjustment unit 14 to obtain an object 2D in the virtual viewpoint. Generate an image.

  The coloring unit 15 colors the object 2D image generated by the projective transformation unit 14.

  The background combining means 16 combines an object 2D image colored by the coloring means 15 with a 2D background image to generate one 2D image.

  In the following, assuming that the object is a person, processing in each means will be described in detail.

The object 3D information generating means 12 receives as input the parameters of each camera estimated offline in advance, and the 3D object template model adjusting means 13 accepts as input the adjustable 3D object template model set in advance offline. , To explain them.
(Camera parameters)

  The parameters of each camera are necessary for the object 3D information generating means 12 to generate the object 3D information, and calibration is performed in a state where each camera is fixedly arranged at the site where the original camera image is to be acquired. Presumed. For example, in the case of sports video, calibration is performed in a state where a camera is fixedly arranged at a site where a person (player) is actually photographed.

  For example, a chessboard pattern can be used for camera calibration. In the following, this case will be described as an example.

  FIG. 2 shows an example of a chessboard pattern image used for camera calibration. The procedure of estimating camera parameters in this case is, for example, as shown in (1) to (5) below.

  (1) Photograph a chess board with each camera in advance, and prepare a chess board pattern image as shown in FIG. 2 (1) as an original copy of the chess board pattern image. The original copy of the chessboard pattern image can be obtained by photographing the chessboard with the camera facing directly.

  (2) A chessboard pattern image is acquired by photographing a chessboard placed in front of the camera. The chessboard pattern image acquired by this differs from the original of the chessboard pattern image of said (1) by the orientation of the camera with respect to a chessboard. FIG. 2 (2) shows an example of a chessboard pattern image acquired by a camera.

  (3) A 3D coordinate is set using the original copy of the chessboard pattern image prepared in advance in (1) above, and this is set as a world coordinate system. For example, a rectangle (black portion) of the original (FIG. 2 (1)) of the chessboard pattern image prepared in advance in (1) above is detected, and a rectangle in which the number of diagonally adjacent rectangles is only one is detected Do. Then, the corner that is adjacent to the other rectangle is detected as the first point, and all the corners inside the frame are detected. In the original chessboard pattern image shown in FIG. 2 (1), the upper left corner is detected as the first point (A), and there are 7 * 10 (7 rows and 10 columns) in total of the corners inside the frame. 70 corners from the first point (A) to the last point are detected in one chessboard pattern image. "*" Means multiplication (the same applies hereinafter). Then, the first point (A) is set as the origin of the 3D coordinate (world coordinate) system, and values calculated according to the size and number of pattern squares (rectangles) are substituted for the X and Y values of the other corners. Do. Also, 0 or 1 is substituted for all the Z values.

  (4) In the same manner as in (3) above, the corner inside the frame is detected also for the chessboard pattern image (FIG. 2 (2)) acquired in (2) above, and the first chessboard pattern image is detected first Detect 70 corners from point (A) to the last point. The position of the corner detected by this is set as the position (u, v) of the image coordinate (2D coordinate) system. FIG. 2 (2) also illustrates how corners are detected from the first point (A) to the last point. These corners can be detected, for example, using the OpenCV function cvFindChessboardCorners () disclosed at http://opencv.jp/sample/camera_calibration.html.

  (5) The camera parameters are estimated using the positions of the corners detected in (3) and (4) above. The camera parameters include internal parameters and external parameters. There is a relationship of equation (1) between camera parameters and each point in the world coordinate (3D coordinate) system and each point in the image coordinate (2D coordinate) system.

  Here, (X, Y, Z) is the position of a point (corner) in the world coordinate (3D coordinate) system, and (u, v) is the position of a point (corner) in the image coordinate (2D coordinate) system . Further, s is a scaling factor, fx, fy are camera focal distances, and (cx, cy) are camera reference points, and are usually selected as image center points. The scaling factor s is known, and the camera focal lengths fx and fy may be expressed in pixel units.

  In equation (1), the matrix including fx, fy, cx, cy, 1 is the matrix [A] of the internal parameters of the camera, and the matrix including r11 to r33 and t1 to t3 is simultaneous rotation and translation of the camera. It is a transformation matrix [R | t], which represents rotation / translation with respect to the world coordinate system, and is called a matrix of camera external parameters. The matrix [R] of only the r11 to r33 part is a transformation matrix of camera rotation, and the matrix [t] of only the t1 to t3 part is a translation transformation matrix of the camera. Note that since the internal parameter [A] of the camera does not depend on the view, once it is estimated, it can be used repeatedly as long as the focal length is fixed.

  If z ≠ 0, equation (1) is equivalent to equation (2). Here, S = 1. FIG. 3 shows a state of coordinate conversion between a world coordinate (3D coordinate) system, a camera coordinate (2D coordinate) system, and an image coordinate (2D coordinate) system by equation (2). Here, P (X, Y, Z) is a 3D position in the world coordinate system, and P '(x, y, z) is a 3D position in the camera coordinate system. Also, p ′ (x, y) is the 2D position of the projection point in the projection plane (the same plane as the image) coordinate system with the camera optical axis z as the origin, and p (u, v) is the image It is a 2D position in the coordinate system.

  In the estimation of camera parameters, first, initial values are set to internal parameters and external parameters. Next, the position (X, Y, Z) of the 3D coordinate system of each corner set in (3) above corresponds to the position (u, v) of the 2D coordinate system of the corner detected in (4) above The parameters (internal parameters and external parameters) are estimated by substituting them into the equation (1) so as to minimize the sum of back projection errors. For this estimation, for example, a closed-form solution disclosed in Zhang Transactions on Pattern Analysis and Machine Intelligence, 2000, 22 (11): 1330-1334. Zhang Z. "A flexible new technique for camera calibration". You can use the method. Here, the sum of the back projection errors is calculated using the camera parameters and the position when the position (X, Y, Z) of the 3D coordinate system of each corner is projected onto the 2D coordinate system. It means that the sum of squares of the difference (distance) with the position (u, v) of the 2D coordinate system of each corner of the actual image is obtained.

If estimation of camera parameters is performed not only from one chessboard pattern image but also for each of several separate chessboard pattern images, and the results are averaged, etc. The effects of noise can be reduced.
(Setting of adjustable 3D object template model)

  As an adjustable 3D object template model required by the 3D object template model adjusting means 13, for example, a 3D human body template model (original model) as shown in FIG. 4 (1) is set in advance offline. Such a 3D human body template model can be created, for example, by the method disclosed in Tony Mullen (2007) (English). Introducing character animation with Blender. Indianapolis, Ind: Wiley Pub. ISBN 9780470102602. OCLC 76794771. However, it is not limited to the method.

  In the 3D human body template model, the surface, internal skeleton and nodes of the object are defined in 3D, and parameters related to the size of each part and parameters related to deformation are defined. The size of each part of the human body can be adjusted by changing the value of the parameter related to the size of each part, and the posture of the human body can be changed by changing the value of the parameter related to deformation. If the internal skeleton and the nodes are not defined in the 3D human body template model, they may be defined by the same process as the object 3D information generating means 12 described later. Further, parameters related to the size of each part and parameters related to deformation may be defined by the same processing as the 3D object template model adjustment means 13 described later.

  Parameters relating to size include, for example, height, fat ratio, and hip-waist ratio. Figure 4 (2) is based on the original model (Original model: Height: 170 cm, Fat: 10%, Hip-Waist Ratio; 0.9), the Fat is changed to 30%, 45%, the Hip-Waist Ratio is 1.249, The 3D human body template model when changing to 1.049 and changing Height to 160 cm and 178 cm is shown.

  For example, as shown in FIG. 4 (3), the parameters relating to deformation include the positions of 18 nodes between parts of the human body. FIG. 4 (4) shows an example of a 3D human body template model deformed by changing the position of each node of the original model.

The adjustable 3D human body template model (original model) is preferably created in the natural posture of the human body. If the original model is not created in the natural posture of the human body, the position of the nodal point may be matched with the position of the nodal point of the human body in the natural posture in the same manner as the deformation of the 3D template model described later.
(Object 2D information extraction means 11)

  Next, processing in the object 2D information extraction unit 11 will be described. The object 2D information extraction means 11 extracts the 2D position, outline and texture of a person as object 2D information from each camera image (original camera image) of a person photographed by a plurality of cameras arranged at intervals. .

For that purpose, first, the background is separated from each original camera image acquired by a plurality of cameras to extract an approximate person area. For example, a background subtraction method can be used for this extraction. Then, the human region is accurately extracted, and its outline and texture are detected. The extraction of the accurate person area is disclosed in, for example, Xu C, Prince J L. "Snakes, shapes, and gradient vector flow". IEEE Transactions on Image Processing, 1998, 7 (3): 359-369. It is possible to use a segmentation method based on shape deformation called the gradient vector flow (GVF) snake. In this method, first, an edge is detected from an original image, and a normalized GVF image is created from the edge. Next, a scalar field image is generated from the normalized GVF image. Furthermore, in the image of the scalar field, the background is separated to extract an approximate object area, and a plurality of seed points (generally, the centers of local area units) are selected from the inside of the approximate object area, Region expansion and region merging are performed based on the image and seed point of to extract an accurate object region.
FIG. 5 (1) shows an example of one original camera image, FIG. 5 (2) shows an approximate person area with a separated background, and FIG. 5 (3) shows an accurately extracted person. Indicates an area.
(Object 3D information generation means 12)

  Next, the process of the object 3D information generation unit 12 will be described. The object 3D information generation unit 12 generates a 3D mesh of a person, extracts a 3D skeleton from the 3D mesh, and generates a node from the 3D skeleton as 3D information of the person. For that purpose, first, a 3D point group constituting a person area is created. Then, a 3D mesh of a person is generated, and further, a 3D skeleton is extracted to generate nodes.

  The procedure of object 3D information generation is, for example, as described in (1) to (5) below.

  (1) An arbitrary one of a plurality of original camera images is selected as a reference image.

  (2) A feature point is extracted from the human region of the reference image selected in (1) above. Since the person area has already been extracted by the object 2D information extraction means 11, it can be used. For extraction of feature points, the Harris corner detection method disclosed in, for example, Derpanis K G. "The harris corner detector [J]". York University, 2004. can be used.

  (3) A feature corresponding to a feature point extracted in the reference image by, for example, the least squares matching (LSM) method, for a person area of a camera image other than the reference image selected in (1) above. Extract points. The least squares matching method is disclosed, for example, in Gruen A. “Least squares matching: a fundamental measurement algorithm”. Close Range Photogrammetry and Machine Vision, 1996: 217-255.

  (4) A 3D point group of a person is created by combining the position of the feature point extracted in the above (2) and (3) and the parameters of each camera. For this creation, for example, the Automatic 3D model acquisition and generation disclosed in Fitzgibbon A, Zisserman A. "Automatic 3D model acquisition and generation of new images from video sequences", European Signal Processing Conference. 1998: 311-326. The method can be used.

(5) A 3D mesh of the person is generated from the 3D point group of the person created in (4) above, a 3D skeleton of the person is generated from the 3D mesh, and nodes are generated from the 3D skeleton. For generating a 3D mesh, for example, Greedy Projection Triangulation (GPT) disclosed in A free library called Point Cloud Library (PCL), "http://pointclouds.org/documentation/tutorials/greedy_projection.php". You can use the method. Also, for the generation of 3D skeletons, see: Reniers D, van Wijk JJ, Telea A. “Computing multiscale curve and surface skeletons of genus 0 shapes using a global importance measure”. IEEE Transactions on Visualization and Computer Graphics, 2008, 14 (2 (2 ): A global importance measure-based surface skeleton extraction method disclosed in 355-368 can be used.
(3D object template model adjustment means 13)

  Next, processing in the 3D object template model adjustment means 13 will be described. The 3D object template model adjusting means 13 sets parameters relating to the size and deformation of the 3D object template model, and adjusts the adjustable 3D object template model (original model) according to these parameters to generate an adjusted 3D object template model .

  The procedure for adjusting the size of the 3D object template model is, for example, as described in (1) to (4) below, and according to this procedure, set parameters related to the size of each part of the 3D object template model and adjust according to the parameters Adjust the size of each part of the possible 3D object template model (original model).

  (1) The height of the person is estimated from the 3D information of the person generated by the object 3D information generation means 12. For this estimation, for example, Gocha TP, Vercellotti G, McCormick LE, et al. "Formulae for Estimating Skeletal Height in Modern South-East Asians". Journal of forensic sciences, 2013, 58 (5): 1279-1283. The method disclosed as A formulae for Estimating Skeletal Height can be used.

  (2) Each part (neck, waist, hip) of the human body is selected from the 3D mesh generated by the object 3D information generation means 12 by user instruction or using image processing technology, and the 3D shape of each part is extracted Estimate the perimeter of The value of the perimeter of each part can be estimated from the relationship with the value of height.

  (3) The fat ratio of the person and the waist-hip ratio are calculated from the height estimated in the above (1) and the peripheral length of each part estimated in the above (3). For example, the fat percentage can be calculated by (163.205 * Log 10 (around waist + hip around-neck) -97.684 * Log 10 (height) -78.387) for women, and for men (86.01 * Log 10 (around waist- Neck circumference)-70.041 * Log 10 (height) + 36. 76) It can calculate. Also, the waist / hip ratio can be calculated by (around waist / around hip). Other general purpose calculation methods may be used.

  (4) Based on the adjustable 3D object template model (original model), the size of each part is calculated according to the height estimated in the above (1) and the fat ratio and waist-hip ratio calculated in the above (3) An adjustment is performed to generate a 3D human body template model (adjusted 3D human body template model) adapted to the person.

  The procedure of deformation of the 3D object template model is, for example, as in the following (1) to (4), parameters according to the deformation of the 3D object template model are set according to the procedure, and the adjustable 3D object template model Transform the (original model).

  (1) FIG. 6 (1) is a 3D skeleton of the original model, which is the same as FIG. 4 (3). On the other hand, the 3D skeleton of the person generated by the object 3D information generating apparatus 12 is, for example, as shown in FIG. 6 (2). Therefore, first, from the 3D skeleton of FIG. 6 (2), five end (end) nodes (Head, HandR, HandL, FootR, FootL), one neck node (Neck) and one pelvic node (Pelvis) To detect In order to detect these seven nodes, for example, all voxels on the 3D skeleton are checked. FIG. 6 (3) shows the state of this check. For example, in FIG. 6 (3), the nearest 26 voxels (voxels numbered 1 to 26) of black filled voxels are searched, and only one voxel on the 3D skeleton is present therein. In this case, black voxels are determined to be end nodes. In a similar manner, if there are four voxels on the 3D skeleton among the nearest 26 voxels, it is determined that the voxel is a cervical node, and if there are three voxels, the voxel is a pelvic node It is determined that

(2) A deformation matrix is estimated which associates each node (FIG. 6 (1)) of the adjustable 3D object template model (original model) with each of the seven nodes detected in (1) above. For example, in order to minimize E _F in equation (3), a deformation matrix that corresponds between seven contact points is estimated. Equation (3) is a function representing the distance between the position of each node of the adjustable 3D object template model (original model) and the position of each node detected in (1) above. For this calculation, for example, a method called Iterative gradient descent disclosed in Press WH, Flannery BP, Teukolsky SA, and Vetterling WT, "Numerical Recipes in C, The art of scientific computing", Cambridge University Press, 1988. Available.

  Here, m is the number of nodes (= 7), Pi is the position of the node i of the 3D skeleton of the object 3D information, Pi ′ is the position of the node i of the adjustable 3D object template model (original model) .

  (3) After changing the positions of 18 nodes of the adjustable 3D object template model (original model) according to the 3D deformation matrix for the 7 nodes estimated in (2) above, 11 other than 7 are obtained. Estimate a 3D transformation matrix that matches the nodes better. In this estimation, a method called Pinocchio disclosed in Illya Baran and Jovan Popovic. Automatic rigging and animation of 3D characters. In Proc. Of the ACM SIGGRAPH, 2007. can be used.

  (4) The positions of all the 18 nodes of the adjustable 3D object template model (original model) are changed using the 3D deformation matrix for the 7 nodes and the 3D deformation matrix for the other 11 nodes. FIG. 6 (4) is an example of the 3D object template model deformed by this.

According to the above processing by the 3D object template model adjusting means 13, parameters relating to the size and deformation of a person (player) are calculated from a camera image of a sports video, for example, and adjustable 3D object template model (original model) according to these parameters By adjusting the size of and modifying it, an adjusted 3D object template model adapted to the player can be generated.
(Projective transformation means 14)

  Next, processing in the projective transformation means 14 will be described. The projective transformation unit 14 projectively transforms the adjusted 3D object template model generated by the 3D object template model adjustment unit 13 into an object 2D image in a specified virtual viewpoint. The virtual viewpoint can be specified arbitrarily.

  FIG. 7 is an explanatory diagram of projective transformation of the adjusted 3D object template model. Here, the adjusted 3D object template model is simplified and illustrated with a cylinder and a cube. Note that XYZ is a world coordinate system, xyz is a viewpoint coordinate system, and uvn is a coordinate system that defines a projection plane.

  The projective transformation means 14 projectively transforms the adjusted 3D object template model in the world coordinate system to the projection point of the coordinate system defining the projection plane, as shown in FIG. The origin o of the viewpoint coordinate system is set to a virtual viewpoint. The z-axis of the viewpoint coordinate system passes through the origin of the uvn coordinate system and the center of the adjusted 3D object template model, and is orthogonal to the uv plane.

  As shown in FIG. 8, since the point P (Px, Py) of the adjusted 3D object template model and the projection point (x, y) on the projection plane have the relationship of equation (4), the projection point (x, y) y) can be calculated by equation (5).

Here, Px, Py, Pz are x, y, z coordinate values of the point P of the adjusted 3D object template model, fx, fy, f are camera focal lengths, and w, h are the width of the image, respectively. It is the vertical width of the image. Note that Px, Py, Pz, f, w, and h are represented in units of millimeters, and fx and fy are respectively represented in units of pixel horizontal width and units of pixel vertical width.
(Coloring means 15)

  Next, the process by the coloring means 16 will be described. Since the object 2D image generated by the projective transformation means 14 is generated based on the adjustable object template model, it is different from the real color even if it is not colored or colored. Therefore, the coloring unit 16 colors the object 2D image generated by the projection conversion unit 14.

  FIG. 9 is an explanatory view of the coloring in the coloring means 15. As shown in FIG. 9, when an adjusted 3D object template model is generated based on N camera images acquired by capturing objects with N cameras, an adjusted 3D object is obtained from each camera image. The color information (R, G, B) of the point corresponding to the point P of the template model is acquired. The color information (R, G, B) may be obtained by acquiring the color information of the texture of the point corresponding to the point P among the 2D information extracted by the object 2D information extraction unit 11. Let these color information (R, G, B) be C1, ..., CN, and let the angle between the vector from the virtual viewpoint to the point P and the vector from the camera to the point P be n (n = 1, ..., N The weight (wi) for the color of each camera image is calculated by equation (6), and the color of point P at the virtual viewpoint is determined by equation (7).

  In the equation (6), when the value in the parenthesis is only cos θ, the weight (wi) for the color takes a negative value, and hence cos θ + 1. Note that α is a value determined empirically, and may be, for example, α = 5.

According to the equations (6) and (7), each of the adjusted 3D object template models is increased by increasing the weight of the color of the image of the camera closer to the virtual viewpoint and decreasing the color of the image of the far camera. Since the part is colored, it is possible to color the color close to reality. Note that θ n (n = 1,..., N) may be determined by the fixed position of the camera and the rough position of the object. Alternatively, only images of a predetermined number of cameras may be used from a direction closer to the virtual viewpoint.
(Background combining means 16)

  Next, the process of the background synthesizing unit 16 will be described. The background combining means 16 combines the object 2D image colored by the coloring means 15 with the 2D background image to generate one 2D image. For the 2D background image, only the background is photographed with a plurality of cameras fixedly arranged at the site, and a 3D background model is created in advance using a plurality of background images acquired by this, and it is used as a projection plane It can be generated by projective transformation.

  As the 3D background model, for example, using the method of IBR (Image-based rendering) disclosed in Non-Patent Document 2, the same coordinate system (world coordinate system in FIG. 6) as the adjusted 3D object template model Create a model Background models and 2D background images do not need to be created with high accuracy, so they can be created from several camera images.

  Then, this 3D background model is projectively converted to the projection plane of FIG. 7 by equation (5) to generate a 2D background image at a virtual viewpoint. Then, the generated 2D background image is combined with the object 2D image colored by the coloring unit 15. In this composition, an object 2D image may be embedded in the 2D background image. The 2D background image can be colored in the same manner as the processing by the coloring unit 15. However, if the background image is already colored, it is not necessary.

  FIG. 10 is a block diagram showing a second embodiment of the free viewpoint image generation device according to the present invention. In the free viewpoint image generation device of this embodiment, in addition to the generation of the free viewpoint image in the virtual viewpoint as in the first embodiment, the free viewpoint image in the same virtual viewpoint is generated by the interpolation technique. Then, the image quality of the free viewpoint images of both parties is evaluated, and the free viewpoint image of the higher evaluation is output.

  In FIG. 10, the first free viewpoint image is a free viewpoint image generated by adjusting the adjustable object template model as described above. In addition, a second free viewpoint image with the same virtual viewpoint is generated by interpolation. In the interpolation technique, two adjacent camera images are selected from a plurality of original camera images and input to the interpolation unit 17 to generate a second free-view image. The image quality evaluation / image selection means 18 evaluates the image quality of the first and second free viewpoint images, and selects and outputs the free viewpoint image of the higher evaluation.

FIGS. 11 (1) to 11 (3) are explanatory diagrams of free viewpoint image generation by the interpolation technique.
In free viewpoint image generation by the interpolation technique, among camera images obtained by capturing an object X with a plurality of cameras fixedly arranged at intervals, two cameras 1 and 2 located on both sides of the virtual viewpoint Pc Camera images A and B are selected. Then, a free viewpoint image at a designated virtual viewpoint Pc is generated from the two camera images A and B by interpolation.

  Specifically, as disclosed in Non-Patent Document 1, first, as shown in FIGS. 11 (1) and (2), parallelization processing of camera images A and B of two viewpoints Pa and Pb is performed. Do. In this parallelization process, the camera image A at the viewpoint Pa and the original image B at the viewpoint Pb are parallel, that is, the light of the camera 1 at the viewpoint Pa and the light of the camera 2 at the viewpoint Pb. The camera images A and B are rotated to be camera images A 'and B' so that the axis 2 is parallel. Next, as shown in FIG. 11C, an intermediate image C ′ is generated by performing linear interpolation between the camera images A ′ and B ′ after the parallelization processing. Thereafter, as shown in FIG. 11 (3), the intermediate image C ′ at the virtual viewpoint Pc is generated by back projecting the intermediate image C ′ onto the space before the parallelization processing.

  The image quality evaluation / image selection means 18 is disclosed in, for example, NB Nill and BH Bouzas, "Objective Image Quality Measure Derived from Digital Image Power Spectra", Optical Engineering, vol. 31, pp. 813-825, 1992. As described above, the image quality IQM (image quality measurement) is evaluated by analyzing the power spectrum of the free viewpoint image according to the equation (8), and the free viewpoint image with the higher evaluation is output.

Where M2 is the size of the image, S (θ1) is the orientation image scale parameter, W (() is the modified Wiener noise filter, A2 (Tρ) is the MTF of the human visual system (T: fixed) And P (ρ, θ) are power spectra of a luminance normalized image, and ρ and θ are spatial frequencies in polar coordinates.

  As mentioned above, although embodiment was described, this invention is not limited to the said embodiment, The thing variously deformed within the range of the technical thought is included. For example, in the above embodiment, an object 2D image generated by projective transformation is colored, and a 2D background image is further synthesized. However, the aspect of the object in the virtual viewpoint is a 2D image colored even in a noncolored 2D image. However, it is also effective to output an image without performing such processing because it can be sufficiently grasped.

  When a plurality of objects are included in one image, processing may be performed on each object while specifying the objects by texture or the like.

  Further, the present invention can be realized not only as a free viewpoint image generation apparatus but also as a free viewpoint image generation method in which processing in each means is taken as a step, and a program for free viewpoint picture generation which causes a computer to function as each means. Can also be realized. The free viewpoint image generation method may include the step of sequentially executing the processing in each means, and the free viewpoint image generation program may be any method as long as the computer functions as each means.

  11 object 2D information extraction means 12 object 3D information generation means 13 3D object template model adjustment means 14 projection conversion means 15 coloring means 16 Background synthesis means, 17 ... interpolation means, 18 ... image quality evaluation / image selection means

Claims (17)

A free-viewpoint image generation apparatus that generates a free-viewpoint image at an arbitrary virtual viewpoint from camera images of objects captured by a plurality of cameras fixedly arranged at an interval.
Object two-dimensional information extraction means for extracting object two-dimensional information from each camera image;
Perform calibration in advance with the object two-dimensional information extracted by the object two-dimensional information extraction means, and obtain it from the relationship between the parameters of each camera and each point in the three-dimensional coordinate system and each point in the two-dimensional coordinate system Object three-dimensional information extracting means for extracting object three-dimensional information using the parameters of each of the selected cameras, wherein one camera image of a plurality of camera images is used as a reference image and the feature points of the object region of the reference image Are extracted, and feature points matching the feature points of the reference image in the object regions of camera images other than the reference image are extracted, and the positions of these feature points and the parameters of the respective cameras Generate a group, and from the three-dimensional point group, the three-dimensional contour and three-dimensional skeleton of the object And object three-dimensional information extracting means for generating a three-dimensional information,
Estimating the parameters relating to the parameters and modifications relating to the size of each part of the three-dimensional object from the object 3-dimensional information extracted by the object 3-dimensional information extracting means, in accordance with the parameters, the preset adjustable 3D object template model 3D object template model adjusting means for adjusting size and deforming ,
A projective transformation means for generating a free viewpoint image in the virtual viewpoint by applying a projective transformation corresponding to an arbitrary virtual viewpoint to the transformed three-dimensional object template model adjusted and deformed by the three-dimensional object template model adjusting means An apparatus for generating a free viewpoint image, comprising: In the adjustable 3D object template model,
The free viewpoint image generating device according to claim 1, wherein the surface, the internal skeleton and the nodes of the object are defined, and the parameters related to the size of each part of the object and the parameters related to the deformation are defined. The object is a person,
Parameters related to the size of each part of the object are
The apparatus for generating a free viewpoint image according to claim 2, wherein the height, fat percentage and waist-hip ratio of a human body are included. The object is a person,
Parameters related to the deformation of the object are
4. A free viewpoint image generating apparatus according to claim 2, wherein the position of 18 nodes between each part of the human body is included. The parameters of each camera are
The internal parameter and the external parameter estimated by detecting the corner of each square in the chessboard pattern image acquired by photographing a chessboard with each camera are included, any one of 1 to 4 characterized by the above-mentioned. Free viewpoint image generation device described in the above. The object two-dimensional information extraction unit
6. The free viewpoint image generating apparatus according to claim 1, wherein an approximate object area is extracted from each camera image by a background subtraction method and then an accurate object area is extracted. The three-dimensional object template model adjustment means
The height of the object is estimated from the object three-dimensional information extracted by the object three-dimensional information extraction means, and the fat ratio and the waist-hip ratio are estimated from around each part in the three-dimensional shape. 4. The free viewpoint image generation device according to claim 3, wherein the value of the parameter related to the size defined in the dimensional object template model is adjusted. The three-dimensional object template model adjustment means
From the object three-dimensional information extracted by the object three-dimensional information extraction means, 18 nodes between each part of the human body are detected and their positions are detected, thereby defining the adjustable three-dimensional object template model 5. The free viewpoint image generating device according to claim 4, wherein values of parameters related to the deformation are adjusted. The projection conversion means
An optional virtual viewpoint, either a three-dimensional object template model adjusted by the three-dimensional object template model adjustment means according to claim 1 to 8, characterized in that projective transformation on the camera image when viewed from the virtual viewpoint The free viewpoint image generation device according to any one of the above. Furthermore, free viewpoint image generating apparatus according to any one of claims 1 to 9, characterized in that it comprises a coloring unit for performing coloring on the free viewpoint image generated by the projective transformation means. 11. The free viewpoint image generating apparatus according to claim 10 , further comprising background image combining means for combining the background image with the free viewpoint image generated by the projective transformation means to generate a composite image. An apparatus for generating a free viewpoint image according to claim 11;
Furthermore, a free viewpoint image generation device that generates a free viewpoint image by interpolation using two camera images acquired by cameras on both sides of an arbitrary virtual viewpoint.
An evaluation unit that evaluates the image quality of the free viewpoint image generated by the free viewpoint image generation apparatus and the free viewpoint image generated by the free viewpoint image generation apparatus according to claim 11 ;
A free-viewpoint image generating apparatus characterized by outputting a free-viewpoint image having a higher evaluation by the evaluation means. The evaluation means is
The apparatus according to claim 12 , wherein the power spectrum of the free viewpoint image is analyzed to evaluate the image quality of each free viewpoint image. A free-viewpoint image generation method for generating a free-viewpoint image at an arbitrary virtual viewpoint from camera images of objects captured by a plurality of cameras fixedly arranged at intervals.
A first step of extracting object two-dimensional information from each camera image;
The object two-dimensional information extracted in the first step is calibrated in advance from the parameters of each camera and each point of the three-dimensional coordinate system and each point of the two-dimensional coordinate system by performing calibration A second step of extracting object three-dimensional information by using parameters of each camera, extracting a feature point of an object region of the reference image using one camera image of a plurality of camera images as a reference image; Feature points in the object region of each camera image other than the reference image are extracted, and a three-dimensional point group of the object is generated from the positions of these feature points and the parameters of each camera A second method of generating a three-dimensional contour and a three-dimensional skeleton of an object from the three-dimensional point group as the three-dimensional object information; And the step,
From the object three-dimensional information extracted in the second step, a parameter relating to the size of each part of the three-dimensional object and a parameter relating to deformation are estimated, and the size of the adjustable three-dimensional object template model set in advance is calculated according to the parameters. A third step of adjusting and deforming ,
Including a fourth step of generating a free viewpoint image of the virtual viewpoint by performing projective transformation corresponding to an arbitrary virtual viewpoint on the three-dimensional object template model adjusted and deformed in the third step; A free viewpoint image generation method characterized in that. Further, a fifth step of coloring the free viewpoint image generated in the fourth step;
The method according to claim 14 , further comprising a sixth step of combining the background image with the free viewpoint image colored in the fifth step to generate a composite image. A program that causes a computer to function as a free viewpoint image generation device that generates a free viewpoint image at an arbitrary virtual viewpoint from camera images of objects captured by a plurality of cameras fixedly arranged at intervals. ,
Object two-dimensional information extraction means for extracting object two-dimensional information from each camera image;
Perform calibration in advance with the object two-dimensional information extracted by the object two-dimensional information extraction means, and obtain it from the relationship between the parameters of each camera and each point in the three-dimensional coordinate system and each point in the two-dimensional coordinate system Object three-dimensional information extracting means for extracting object three-dimensional information using the parameters of each of the selected cameras, wherein one camera image of a plurality of camera images is used as a reference image and the feature points of the object region of the reference image Are extracted, and feature points matching the feature points of the reference image in the object regions of camera images other than the reference image are extracted, and the positions of these feature points and the parameters of the respective cameras Generate a group, and from the three-dimensional point group, the three-dimensional contour and three-dimensional skeleton of the object And object three-dimensional information extracting means for generating a three-dimensional information,
Estimating the parameters relating to the parameters and modifications relating to the size of each part of the three-dimensional object from the object 3-dimensional information extracted by the object 3-dimensional information extracting means, in accordance with the parameters, the preset adjustable 3D object template model 3D object template model adjusting means for adjusting size and deforming ,
A projective transformation means for generating a free viewpoint image in the virtual viewpoint by applying a projective transformation corresponding to an arbitrary virtual viewpoint to the transformed three-dimensional object template model adjusted and deformed by the three-dimensional object template model adjusting means A program to function as Computer,
And coloring means for coloring the composite image generated by the projective transformation means.
The program according to claim 16 , wherein the program functions as background image combining means for combining the background image with the free viewpoint image colored by the coloring means to generate a combined image.