JP2011170487A - Virtual point of view image generation method and program based on geometric information of camera placement in large space - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To compose a human region with a high degree of accuracy by estimating a projection matrix of each camera based on the information of a target field and roughly calculating human position information. <P>SOLUTION: The image generation method divides a plurality of viewpoint images into one or a plurality of moving regions and a background region. Then, the method forms a model coordination space based on geometric information in the plurality of viewpoint images, calculates a coordinate of the moving region at each viewpoint image, calculates, by using the coordinate, a coordinate of the moving region in the model coordination space, and generates an image of the moving region at a virtual point of a view based on the coordinate information. Then, the method generates an image of the background region at the virtual point of the view from the image of the background region. Then, the method composes the generated image of the moving region and the generated image of the background region to generate an image at the virtual point of view. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a method and a program for generating a virtual viewpoint image between cameras. More specifically, the present invention relates to a method and a program for generating an image at a virtual viewpoint based on a multi-viewpoint image obtained by capturing an event performed in a large space.

  A model-based method, which is a typical synthesis method for generating virtual viewpoint image generation, is performed by creating a three-dimensional voxel model. In this method, it is necessary to perform strong calibration of the camera in advance to estimate the projection relationship between the Euclidean coordinate system in the space and the camera coordinate system. However, it is extremely difficult to accurately perform strong camera calibration in a large space.

  Therefore, in Non-Patent Document 1, as a method for generating a virtual viewpoint image in a large-scale space, a method for synthesizing an image in which the viewpoint is moved to the middle of a real camera using only the projective geometric relationship established between the cameras. Has proposed. Here, the image is divided into three areas: a moving area whose position and shape change with time, a far-field area far from the camera, and the other field areas, and generates a composite image for each of them. A virtual viewpoint image is generated by superimposing the images.

  The moving area is processed by being divided into a plurality of moving areas. First, background difference / binarization is performed on images obtained from two adjacent viewpoints to create two silhouette images from which all moving regions are extracted. Each image is labeled and divided into moving regions. Each divided silhouette image is associated between viewpoints, and an epipolar line is projected to each corresponding silhouette to calculate a corresponding point. Morphing is performed based on the obtained corresponding point information to synthesize an image at a virtual viewpoint.

  In the field area, a plane approximation is performed to obtain corresponding points as in the movement area, and a composite image of the virtual viewpoint is generated by morphing. The distant view areas are connected by image mosaicing, and the viewpoint is interpolated by cutting out the intermediate area from the panoramic image.

Naho Inamoto, Hideo Saito, “Viewpoint Interpolation Method for Free Viewpoint Video Generation in Soccer Scene” TVRSJ vol. 7, no. 4, 2002

  However, in the composition of moving regions, it is important to associate people between viewpoints. However, in the conventional technology, the accuracy is increased due to the effect of occlusion where people overlap each other and the noise in the silhouette image that occurs during background subtraction. There was a problem that would be greatly reduced.

  Therefore, the present invention estimates the projection matrix of each camera based on the geometric information of the target field and roughly calculates the position information of the person, thereby realizing a virtual viewpoint image that realizes the synthesis of the person region with high accuracy. An object is to provide a generation method and a program.

  In order to achieve the above object, a method for generating an image at a virtual viewpoint according to the present invention is a method for generating an image at a virtual viewpoint from a plurality of viewpoint images, wherein one or a plurality of the plurality of viewpoint images are selected. Dividing the movement area and the background area of the image, a step of constructing a model coordinate space based on the geometric information in the plurality of viewpoint images, and calculating coordinates of the movement area for each viewpoint image, Calculating the coordinates of the moving area in the previous model coordinate space from the image, generating an image of the moving area at the virtual viewpoint based on the coordinate information, and from the image of the background area, Generating an image; and synthesizing the generated image of the moving area and the generated image of the background area.

  Further, the step of generating an image of the moving region includes, for each viewpoint image, a sub-step of calculating a plane projection matrix established between the viewpoint image and the model coordinate space, and the viewpoint image by the plane projection matrix. A sub-step of projecting the coordinates of the moving region onto the floor plane in the model coordinate space, a sub-step of calculating coordinates in the model coordinate space of the moving region from the projected coordinates, and the model at a virtual viewpoint A sub-step of calculating a spatial projection matrix established between the coordinate space and generating an image of a moving region at a virtual viewpoint based on the coordinates in the model coordinate space based on the spatial projection matrix. preferable.

  In calculating the coordinates of the moving area in the model coordinate space, it is also preferable to calculate the center of gravity within the same group by grouping the coordinates based on the distance between the coordinates.

  Further, the sub-step of generating an image of the moving area at the virtual viewpoint sets a rectangular area of an appropriate size for each moving area in the model coordinate space, and the corresponding area at the virtual viewpoint of the rectangular area and It is also preferable to calculate the corresponding area at the real viewpoint and extract the texture of the corresponding area at the virtual viewpoint from the corresponding area at the real viewpoint close to the virtual viewpoint.

  Further, the calculation of the corresponding area at the virtual viewpoint and the corresponding area at the real viewpoint calculates a central projection matrix established between the virtual viewpoint and the real viewpoint with the model coordinate space, and the central projection matrix It is also preferable to determine an image region by projecting a rectangular region set for each moving region in the model space onto each virtual image and a real viewpoint.

  Further, the method of extracting the texture of the corresponding area at the virtual viewpoint from the corresponding area at the real viewpoint calculates a planar projection matrix between the corresponding area at the virtual viewpoint and the corresponding area at the real viewpoint, It is also preferable to project the texture of the corresponding area at 1 to the corresponding area at the virtual viewpoint.

  Further, the method of extracting the texture of the corresponding area at the virtual viewpoint from the corresponding area at the real viewpoint has a function of determining whether or not occlusion occurs in the corresponding area at the actual viewpoint. In such a case, it is also preferable to synthesize based on the corresponding area from another real viewpoint.

  Further, the geometric information is the length of the line segment and the corresponding point in the viewpoint image, and the step of constructing the coordinate space determines the origin and coordinates the corresponding point based on the length of the line segment. Is also preferred.

  Further, the geometric information is parallel lines in the viewpoint image, and the step of constructing the coordinate space rotates each viewpoint image so that the parallel lines are in the same direction, and on the parallel lines of the viewpoint image It is also preferable that the points are grouped based on the distance, and the coordinates are assigned to the corresponding points with the center of gravity in the same group as the corresponding point.

  The planar projection matrix is preferably calculated from the corresponding points.

  In order to achieve the above object, a program according to the present invention divides a computer for generating an image at a virtual viewpoint from a plurality of viewpoint images, and divides the plurality of viewpoint images into one or a plurality of moving areas and a background area. Means for constructing a model coordinate space based on geometric information in the plurality of viewpoint images, calculating coordinates of the moving region for each viewpoint image, and moving the movement in the previous model coordinate space from the coordinates Means for calculating the coordinates of the area and generating an image of the moving area at the virtual viewpoint based on the coordinate information; means for generating an image of the background area at the virtual viewpoint from the image of the background area; It functions as a means for synthesizing the generated moving area image and the generated background area image to generate an image at a virtual viewpoint.

  According to the present invention, it is possible to obtain a robust result with respect to the influence of occlusion, which has been a problem in the conventional method, and the influence of noise or the like in a silhouette image generated during background subtraction, and to generate a highly accurate virtual viewpoint image Became possible.

3 is a flowchart illustrating a method for constructing a virtual viewpoint image according to the present invention. It is a flowchart which shows the process which determines the position of the corresponding point by this invention. An example of vanishing points V ₁ and V ₂ calculation from the vertical and horizontal lines of the futsal court will be shown. The image which converted the image of the futsal court so that the field plane may be in front is shown. The example which mapped the corresponding point on the straight line in the image of a futsal court to a plane is shown. The example of calculation of the player position in the plane coordinate in a futsal court is shown. The example of the mapping result for all the viewpoints in the plane coordinate in a futsal court is shown. The rectangular area projected on the viewpoint image and the background image is shown. The example of the camera image of the viewpoint 1 in the futsal court, the viewpoint 2, and the virtual viewpoint image produced | generated from this camera image is shown.

  The best mode for carrying out the present invention will be described in detail below with reference to the drawings. FIG. 1 is a flowchart illustrating a method for constructing a virtual viewpoint image according to the present invention. Hereinafter, description will be given based on this flowchart.

  The present invention is a composition method that uses only image information and does not require strong calibration of the camera. Based on the geometric information of the target field, a coordinate space is constructed in the target field, and a projection matrix of each camera image is obtained. The position information of the moving area is estimated, the moving area image is generated at the position of the moving area by the projection matrix at the virtual viewpoint, and the image of the virtual viewpoint is generated with high accuracy.

  Step 1: A plurality of viewpoint images and background images of the target field are acquired by a plurality of cameras. The background image is an image in which the moving area is not shown. Strong calibration of this camera is not necessary. Further, as geometric information of the target field, the length of the line segment in the target field and the corresponding point are measured in advance. At least, four or more plane coordinates and two or more spatial coordinates are measured for each viewpoint image. Further, as the geometric information of the target field, it is possible to photograph so that two or more sets of parallel lines are included in the viewpoint image. A parallel line is a line of a competition court, for example. In this case, the corresponding points are obtained from the parallel lines.

  Step 2: The viewpoint image of the target field is divided into a movement area and a background area. The moving region is a region whose position and shape change with time, and corresponds to a person region when the target field is a futsal court.

  Step 3: The positions of corresponding points between viewpoint images are determined, a model coordinate space is constructed, and a planar projection matrix H, focal length f, and center projection matrix P are calculated for each viewpoint image.

  In the present embodiment, since the corresponding points are not measured in advance, the corresponding points are obtained from the parallel lines in the viewpoint image. FIG. 2 is a flowchart showing a process for determining the position of the corresponding point according to the present invention. Hereinafter, description will be given based on this flowchart.

Step 21: Calculate the focal length from the vanishing point. The vanishing points V ₁ (u ₁ , v ₁ ), V ₂ (u ₂ , v ₂ ) are obtained from the intersection coordinates of two parallel straight lines in each camera viewpoint image,
u ₁ u ₂ + v ₁ v ₂ + f ² = 0
From the above, the focal length f is obtained. FIG. 3 shows an example of vanishing points V ₁ and V ₂ calculation from vertical and horizontal lines of futsal court.

  Step 22: Convert the image to the field front view point. A parallel line in a reference direction is determined, and the azimuth angle θ is 0 ° and the elevation angle φ is −90 ° with respect to the vanishing point so that the coat line is in front of the image. Translate to come in. FIG. 4 shows an image obtained by converting the futsal court image so that the field plane is in front.

により求まる。なお、ｕ_∞、ｖ_∞は、画像中心の座標を表している。例えば、１９２０×１０８０の画像の場合、ｕ_∞＝９６０，ｖ_∞＝５４０となる。 The azimuth angle θ and elevation angle φ are determined from the vanishing point V ₁ (u ₁ , v ₁ ) serving as a reference and the focal length f.

It is obtained by. Note that u _∞ and v _∞ represent the coordinates of the image center. For example, in the case of a 1920 × 1080 image, u _∞ = 960 and v _∞ = 540.

と表される。なお、Ａは視点での内部パラメータ行列である。 The transformation matrix H _F that makes the field plane front is:

It is expressed. A is an internal parameter matrix at the viewpoint.

Step 23: The corresponding points on the straight line are grouped based on the distance, and the position of the corresponding point is determined. Corresponding points are taken on a straight line for each viewpoint image and mapped to a plane. The mapped corresponding points are grouped according to the Euclidean distance, and the points in the group are combined into one point. Starting from a small threshold value, the points within the threshold value are combined into one, but may not be combined into one. In that case, the threshold value is gradually increased, the corresponding points are combined into one, and the position of the corresponding point is determined.

  FIG. 5 shows an example in which corresponding points on a straight line in the futsal court image are mapped to a plane. In this example, seven corresponding points are taken on three straight lines. Corresponding points are summarized by gradually increasing the threshold values as in FIGS. 5a, 5b, 5c, and 5d. In FIG. 5d, the position of the corresponding point on the straight line is determined.

  As described above, the position of the corresponding point is determined for each viewpoint image, and using this corresponding point, a coordinate space is constructed for each viewpoint image, and the planar projection matrix H, focal length f, and central space projection matrix P are obtained. It becomes possible to ask.

The planar projection matrix H is calculated from the corresponding points on four or more planes of the viewpoint image, for example, the coordinates of four coat lines. The focal length f is obtained from four or more corresponding points of the viewpoint image, for example, vanishing points V ₁ (u ₁ , v ₁ ), V ₂ (u ₂ , v ₂ ) from the vertex coordinates of the coat line,
u ₁ u ₂ + v ₁ v ₂ + f ² = 0
Obtained from The central projection matrix P is calculated from the corresponding points on four or more planes of the viewpoint image and the positions and heights of the corresponding points on two or more spaces.

  Step 4: The coordinates on the floor plane in the model coordinates of each moving region are calculated from the plane projection matrix H. A viewpoint image and a background image are prepared, a difference is obtained between each color component and hue, and binarized to detect a moving area (player position). Each moving area is labeled, and the step point of each moving area is detected. The foot point of each moving area is mapped to the floor plane by the plane projection matrix H.

  FIG. 6 shows an example of calculating the player position in the plane coordinates in the futsal court. 6a is a viewpoint image, FIG. 6b is a binarized image, FIG. 6c is a diagram in which player positions are labeled from L0 to L8, and FIG. 6d is an image in which foot points are mapped to a plane, The foot point is indicated by +.

Step 5: The moving region position on the model coordinates is determined using the mapping results for all viewpoints. The mapping result of the foot position from each viewpoint is
・ In the overlapping area, only the position of the foremost player can be mapped. ・ The mapping of the same player is slightly deviated with the accuracy of the H matrix. It is out. FIG. 7 shows an example of mapping results for all viewpoints in the plane coordinates in the futsal court. Here, FIG. 7a is an image in which all viewpoints are mapped. For example, the position indicated by the arrow 1 is a position where the same player is displaced, but it can be seen that it is slightly displaced.

The moving region position is determined by grouping based on the Euclidean distance.
Euclidean 2 between all points for _n points P ₁ (x ₁ , y ₁ , 0),..., P _n (x _n , y _n , 0) mapped on the model coordinates Riding distance d _ij ² = (x _i −x _j ) ² + (y _i −y _j ) ² (i, j = 1, 2,..., N)
Seeking
When d _ij <threshold, it is determined that the point P _i (x _i , y _i , 0) and the point P _j (x _j , y _j , 0) belong to the same group. vice versa,
When d _ij ≧ threshold, it is determined that the point P _i (x _i , y _i , 0) and the point P _j (x _j , y _j , 0) do not belong to the same group.

  Next, when the number of elements in the group is 1, it is determined that it is dust and the point is deleted. When the number of elements is 2 or more, the center of gravity of the point in the group is obtained, and that point is set as the player's position. FIG. 7B is a diagram in which the moving region position is determined by grouping. It can be seen that the points indicated by arrows 1 are grouped together.

  Step 6: A rectangular area is set at the moving area position on the model coordinates, and projected onto each viewpoint image. A player area is assumed centering on the coordinates of the player position on the floor plane coordinates. The player area is assumed to be rectangular, and a vertical direction of 2 m (20 px) and a width of 1 m (10 px) are assigned, and the direction is the front direction of each camera viewpoint. The distance between the camera position and each moving area is obtained to obtain depth information. Next, the rectangular region is projected onto the viewpoint image and the background image using the central projection matrix P. A player area (moving area) is included in the projected rectangular area. FIG. 8 shows a rectangular area projected on the viewpoint image and the background image.

  Step 7: Cut out the texture of the rectangular area. A portion included in the rectangular area projected to each viewpoint is cut out from the viewpoint image. All parts except the player area are removed by background difference. Collects textures for all viewpoints as a billboard along with viewpoint direction information.

Step 8: Determine the overlap of the rectangular areas and perform an occlusion process. The Euclidean distance between each player position on the field plane is obtained with respect to the camera position. Based on this Euclidean distance, the overlapping order is obtained. Overlap determination is performed for all pairs of rectangular regions in each viewpoint image. If it overlaps with the other rectangular area and is in front, the following processing is performed on the extracted texture area.
-When the occlusion is not caused by the adjacent viewpoint: The background area of the adjacent viewpoint is projected to the occluding viewpoint. The texture of the person area is extracted using the projected background area.
-When the occlusion has occurred in the adjacent viewpoint: In this case, the extracted texture is used as it is.

と表せる。ここで、ｆ_Ａ１，ｆ_Ａ２は、視点１、２における焦点距離ｆであり、ｘ_Ａ１，ｘ_Ａ２は、画像の縦の長さであり、ｙ_Ａ１，ｙ_Ａ２は、画像の横の長さである。 Step 9: 2-viewpoint and the camera parameter matrix, using the interpolation ratio alpha, and calculates the spatial projection matrix P _C of the virtual viewpoint c. Space projection matrix P _C of the virtual viewpoint c is determined as follows.
· View 1, the internal parameter matrix _A 1, _{A 2} of the viewpoint 2 and calculates an internal parameter matrix _{A c} of the virtual viewpoint c. The internal parameter matrices A ₁ and A ₂ are

It can be expressed. Here, f _A1 and f _A2 are the focal lengths f at the viewpoints 1 and 2, x _A1 and x _A2 are the vertical lengths of the images, and y _A1 and y _A2 are the horizontal lengths of the images. It is.

と表せる。
・視点１、視点２の回転行列Ｒ_１，Ｒ_２から仮想視点ｃでの回転行列Ｒ_ｃを求める。回転行列Ｒ_１，Ｒ_２に対して、方位角θ、仰角φ、チルト角τに分解する（ただし、本実施形態では水平線に対して傾いていることは想定しないのでτ＝０とする）。回転行列Ｒ_１，Ｒ_２の方位角θ_１，θ_２、仰角φ_１，φ_２、すると、仮想視点ｃの方位角θ_ｃ、φ_ｃは、
θ_ｃ＝（１−α）θ_１＋αθ_２
φ_ｃ＝（１−α）φ_１＋αφ_２
と表せる。このθ_ｃ、φ_ｃとを組み合わせて、Ｒ_ｃが求まる。
・視点１、視点２の並進ベクトルＴ_１，Ｔ_２から仮想視点ｃでの並進ベクトルＴ_ｃを求める。視点１、視点２、仮想視点ｃのモデル座標上での位置を、それぞれ

とすると、

であるため、仮想視点ｃのモデル座標上での位置は、

と表せる。並進ベクトルＴ_ｃは、

より求めることができる。
・空間射影行列Ｐ_ｃは、Ａ_ｃ、Ｒ_ｃ、Ｔ_ｃから、
Ｐ_ｃ＝Ａ_ｃ［Ｒ_ｃ｜Ｔ_ｃ］
で求められる。ここで、［Ｒ_ｃ｜Ｔ_ｃ］は、３×３の行列Ｒ_ｃと１×３のベクトルＴ_ｃを、あわせた３×４の行列である。Ａ_ｃとこの行列の積により、空間射影行列Ｐ_ｃが求まる。 The center of the image is unchanged, and the focal length at the virtual viewpoint c can be expressed as follows using the focal lengths at the viewpoints 1 and 2 and the interpolation ratio α.
f _Ac = (1-α) f _A1 + αf _A2
Using this f _Ac , the internal parameter matrix _Ac at the virtual viewpoint _c is

It can be expressed.
· View 1 obtains a rotation matrix _{R c} of the virtual viewpoint c from the rotation matrix _R 1, _{R 2} viewpoints 2. The rotation matrices R ₁ and R ₂ are decomposed into an azimuth angle θ, an elevation angle φ, and a tilt angle τ (however, in the present embodiment, τ = 0 is assumed since it is not assumed to be inclined with respect to the horizontal line). The azimuth angles θ ₁ , θ _{2 of} the rotation matrices R ₁ , R ₂ , the elevation angles φ ₁ , φ ₂ , and the azimuth angles θ _c , φ _c of the virtual viewpoint _c are
θ _c = (1−α) θ ₁ + αθ ₂
φ _c = (1−α) φ ₁ + αφ ₂
It can be expressed. R _c is obtained by combining θ _c and φ _c .
· View 1 obtains a translation vector _{T c} of the virtual viewpoint c from translation vector _T 1, _{T 2} of view 2. The positions of viewpoint 1, viewpoint 2, and virtual viewpoint c on the model coordinates are

Then,

Therefore, the position of the virtual viewpoint c on the model coordinates is

It can be expressed. The translation vector T _c is

It can be obtained more.
The spatial projection matrix P _c is obtained from A _c , R _c , and T _c
P _c = A _c [R _c | T _c ]
Is required. Here, [R _c | T _c ] is a 3 × 4 matrix including a 3 × 3 matrix R _c and a 1 × 3 vector T _c . A spatial projection matrix P _c is obtained by the product of A _{c and} this matrix.

Step 10: A rectangular area is set at the moving area position on the model coordinates and projected onto the virtual viewpoint. A player area is assumed centering on the coordinates of the player position on the floor plane coordinates. The player area is assumed to be rectangular, and a vertical direction of 2 m (20 px) and a width of 1 m (10 px) are assigned, and the direction is the front direction of each camera viewpoint. The distance between the camera position and each moving area is obtained to obtain depth information. Next, the rectangular region is projected onto the virtual viewpoint using the spatial projection matrix _Pc .

  Step 11: A billboard is selected, texture mapping is performed, and an interpolated image is synthesized. In each rectangular area, the billboard with the direction closest to the virtual viewpoint direction is selected. However, if the selected billboard is occluded, the closest billboard that is not occluded is selected. The texture of the selected billboard is mapped according to the size of the rectangular area of the virtual viewpoint, and the interpolated image is synthesized.

  Step 12: Generate an image of the background area at the virtual viewpoint from the image of the background area. The image of the background area at the virtual viewpoint is generated by an existing method. For example, the background area can be divided into a distant view area and a field area, and can be created by the method described in Non-Patent Document 1.

  Step 13: The generated moving area image and the generated background area image are combined to generate a virtual viewpoint image. This synthesis is also performed by existing methods.

  Next, the effect of the present invention is shown by an actually generated image at a virtual viewpoint. FIG. 9 shows an example of camera images of viewpoints 1 and 2 in futsal court and a virtual viewpoint image generated from the camera images. 9a is an example of a camera image at viewpoint 1, FIG. 9b is an example of a camera image at viewpoint 2, and FIG. 9c is an example of a virtual viewpoint image generated by the prior art from FIGS. 9a and 9b. FIG. 9d is an example of a virtual viewpoint image generated by the method of the present invention from FIGS. 9a and 9b. FIGS. 9c and 9d are images of a virtual viewpoint that is intermediate between viewpoint 1 and viewpoint 2 (α = 0.5). Each original image is a 1920 × 1080 full HD image, but a part is extracted for comparison.

  Comparing FIG. 9c and FIG. 9d, in FIG. 9c, the correspondence between the images cannot be taken, so the human image may be removed and may not be shown, but in FIG. 9d, it is not deleted. Further, comparing human images, it can be seen that the image of FIG. 9d is clearer and the accuracy of the image is higher.

  The above-described embodiments are all illustrative of the present invention and are not intended to limit the present invention, and the present invention can be implemented in various other variations and modifications. Therefore, the scope of the present invention is defined only by the claims and their equivalents.

Claims (11)

A method for generating an image at a virtual viewpoint from a plurality of viewpoint images,
Dividing the plurality of viewpoint images into one or more moving regions and a background region;
Building a model coordinate space based on geometric information in the plurality of viewpoint images;
Calculating the coordinates of the moving area for each viewpoint image, calculating the coordinates of the moving area in the previous model coordinate space from the coordinates, and generating an image of the moving area at the virtual viewpoint based on the coordinate information; ,
Generating a background region image at a virtual viewpoint from the background region image;
Combining the generated image of the moving area and the generated image of the background area;
A method for generating an image at a virtual viewpoint characterized by including: The step of generating an image of the moving area includes:
Substep for calculating a plane projection matrix established between the viewpoint image and the model coordinate space for each viewpoint image;
A sub-step of projecting the coordinates of the moving region to the floor plane in the model coordinate space for each viewpoint image by the plane projection matrix;
A sub-step of calculating coordinates in the model coordinate space of the moving region from the projected coordinates;
A sub-step of calculating a spatial projection matrix established between the virtual coordinate point and the model coordinate space, and generating an image of a moving region at the virtual viewpoint based on the coordinates in the model coordinate space based on the spatial projection matrix When,
The method of generating an image at a virtual viewpoint according to claim 1, comprising:   3. The virtual viewpoint according to claim 2, wherein the calculation of the coordinates of the moving area in the model coordinate space is performed by grouping the coordinates based on a distance between the coordinates and calculating a center of gravity within the same group. How to generate an image with   The sub-step of generating an image of the moving area at the virtual viewpoint sets a rectangular area of an appropriate size for each moving area in the model coordinate space, and the corresponding area and the actual viewpoint at the virtual viewpoint of the rectangular area 4. The image at the virtual viewpoint according to claim 2, wherein the corresponding area at the virtual viewpoint is calculated, and the texture of the corresponding area at the virtual viewpoint is extracted from the corresponding area at the real viewpoint close to the virtual viewpoint. how to.   The calculation of the corresponding region at the virtual viewpoint and the corresponding region at the real viewpoint is performed by calculating a central projection matrix established between the virtual viewpoint and the real viewpoint with the model coordinate space, and using the central projection matrix, the model 5. The method for generating an image at a virtual viewpoint according to claim 4, wherein the image area is determined by projecting a rectangular area set for each moving area in the space onto each virtual image and the actual viewpoint.   The method of extracting the texture of the corresponding area at the virtual viewpoint from the corresponding area at the real viewpoint is to calculate a planar projection matrix between the corresponding area at the virtual viewpoint and the corresponding area at the real viewpoint, and 6. The method for generating an image at a virtual viewpoint according to claim 4, wherein the texture of the corresponding area is projected onto the corresponding area at the virtual viewpoint.   The method of extracting the texture of the corresponding area at the virtual viewpoint from the corresponding area at the real viewpoint has a function of determining whether or not occlusion has occurred in the corresponding area at the real viewpoint. The method for generating an image at a virtual viewpoint according to any one of claims 4 to 6, wherein the image is synthesized based on a corresponding region at another real viewpoint. The geometric information is a length of a line segment in the viewpoint image and a corresponding point,
The virtual viewpoint according to any one of claims 1 to 7, wherein the step of constructing the coordinate space determines an origin and assigns coordinates to corresponding points based on a length of the line segment. How to generate images. The geometric information is parallel lines in the viewpoint image,
In the step of constructing the coordinate space, each viewpoint image is rotated so that the parallel lines are in the same direction, and points on the parallel lines of the viewpoint image are grouped based on the distance, and the center of gravity in the same group is obtained. The method for generating an image at a virtual viewpoint according to claim 1, wherein coordinates are assigned to the corresponding points as corresponding points.   The method for generating an image at a virtual viewpoint according to claim 8 or 9, wherein the planar projection matrix is calculated from the corresponding points. A computer for generating an image at a virtual viewpoint from a plurality of viewpoint images,
Means for dividing the plurality of viewpoint images into one or more moving regions and a background region;
Means for constructing a model coordinate space based on geometric information in the plurality of viewpoint images;
Means for calculating coordinates of the moving area for each viewpoint image, calculating coordinates of the moving area in the previous model coordinate space from the coordinates, and generating an image of the moving area at the virtual viewpoint based on the coordinate information; ,
Means for generating an image of a background area at a virtual viewpoint from the image of the background area;
Means for synthesizing the generated image of the moving region and the image of the generated background region;
And a program for generating an image at a virtual viewpoint.

Images