JP2000137815A - New viewpoint image generating method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable easy application to a three-dimensional body which has a complicated surface shape and a complicated texture and further a high-speed process by covering an object body in a photographed image with a triangular area and finding a projection conversion matrix from a three-dimensional space to an image of a new viewpoint. SOLUTION: The projection conversion matrix from the three-dimensional space where the object body 22 is arranged to photographed images 25a to 25c is found by using a calibration box 21. In the box 21, quadrangular or triangular marks 23 are drawn on the floor surface and wall surfaces. The projection conversion matrix includes data, such as the positions, attitudes, and field angles of cameras 24a to 24c, regarding the cameras. All the photographed images 25 are related by specifying reference points at parts showing the shape of the object body 22 and the reference points are used to cover the whole image of the object body 22 with triangular areas. Then the projection conversion matrix to the image of the new viewpoint is found to generate a new-viewpoint image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention belongs to the technical field of generating an image of a target object in a three-dimensional space. More specifically, the present invention relates to a technique for generating an image viewed from another viewpoint using a plurality of captured images.

[0002]

2. Description of the Related Art In recent years, image processing of an object in a three-dimensional space has been rapidly developed due to an increase in a processing speed of a computer and an increase in a memory capacity. For example, as described in Japanese Patent Application Laid-Open No. 10-27268, the technique is used for generating a change in a three-dimensional image as the viewpoint of an object or landscape in a three-dimensional space changes.
In such a case, a simple and theoretical method is to record the three-dimensional coordinates of all points on the surface of the three-dimensional object and information on the points in a memory, and calculate and display them when the viewpoint has changed. It is considered to be a clear method. But,
At the current stage, it is impractical due to the relationship between the memory capacity and the processing time. On the other hand, a two-dimensional image can be processed by the current technology, but it is not possible to generate an image in which the shape or the like of an object changes with respect to a change in viewpoint from only the two-dimensional image.

[0003] Therefore, conventionally, a large number of two-dimensional images are photographed, as in a certain kind of virtual reality, and when the viewpoint changes, the photographed image is switched and shown according to the change in the viewpoint. Techniques for experiencing three-dimensional space have also been developed. However, this method has a drawback that it is necessary to prepare a large number of captured images in advance, and the prepared image restricts a change in viewpoint.
In addition, transmission of an image requires a lot of time, which is a problem.

Another conventional method is, for example,
The image drawing apparatus disclosed in Japanese Patent Laid-Open Publication No. Hei 9-212689 focuses on a three-dimensional image, that is, an individual object constituting an image of a three-dimensional object in a three-dimensional space, and performs basic three-dimensional image processing. A method is used in which the positional relationship and the three-dimensional surface pattern are stored by means of characters and the like, and an actual three-dimensional shape is generated from a basic shape described by characters and the like.

However, such a method is applied to the generation of an image of a three-dimensional object having a complicated shape or an object having a complicated texture because the method expresses and records the images and concepts of human beings. It is difficult to do.

[0006]

SUMMARY OF THE INVENTION The present invention has been made under the above-described background, and can express distance information of a three-dimensional image by objective means, and can express a complicated surface shape or a complicated texture. It can be easily applied to three-dimensional objects with
It is an object of the present invention to provide a method and an apparatus for generating a three-dimensional image, which can perform high-speed processing only by constructing a simple three-dimensional model as compared with a conventional method. The present applicant has filed a patent application (1998 No. 227069) to solve the above problems.
Have already proposed several solutions. The present invention provides additional, improved, and new solutions to the above solutions.

[0007]

The present invention employs the following means to solve the above-mentioned problems. That is, claim 1
The invention described in the above is a method of generating an image of a target object viewed from a new viewpoint based on images of the target object in a three-dimensional space taken from a plurality of different viewpoints, wherein the target object is arranged in a 3D space. A first step of obtaining a projection transformation matrix from a dimensional space to a captured image, designating a point in a portion representing the shape or the like of the captured target object, and specifying the same point of the target object on the plurality of captured images A second step of associating the points of each captured image with each other, and specifying three points among the associated points to form a triangular area, and defining the target object of the captured image as the triangle. A third step of covering an area, a fourth step of obtaining a projection transformation matrix from the three-dimensional space to an image of a new viewpoint, and an image of a new viewpoint based on the data obtained in the first to fourth steps. Generating method including generating a fifth image Oite, the second step is characterized in that searching by using the correlation points epipolar line, automatically extracting corresponding points. The invention according to claim 1 is characterized in that, if a reference point is specified for the first captured image, the corresponding reference point is automatically given to the remaining other captured images.

According to a second aspect of the present invention, in the image generation method described above, the fifth step is to select the photographed image and to apply a new viewpoint to the texture of the selected photographed image by inverse warping using homography conversion. The method is characterized by including a step of warping a texture in the image.

A third aspect of the present invention is characterized in that, in the second aspect of the present invention, the homography conversion obtains a conversion matrix using a Gentech method. One of the features of the present invention is that a gentech method that can obtain an accurate and stable solution is adopted.

According to a fourth aspect of the present invention, in the above-described image generation method, the fifth step performs sorting on each of the triangular regions so that an invisible region does not appear on the new viewpoint image plane. And One feature of the present invention is that an invisible image is made invisible.

A fifth aspect of the present invention is characterized in that, in the fourth aspect of the invention, the sorting uses a bounding box so that an invisible area does not appear. One feature of the present invention is that the sorting is simplified by using a bounding box.

According to a sixth aspect of the present invention, in the fourth or fifth aspect of the invention, back face culling is performed before or after the sorting step. One feature of the present invention is that no warping is performed when the surface of the triangular area cannot be seen from the viewpoint.

According to a seventh aspect of the present invention, in the above-described image generating method, the first step includes a step of correcting a projection transformation matrix by a bundle adjustment method when the number of the photographed images is three or more. I have. One feature of the present invention is that the conversion between all captured images is adjusted to match.

[0014]

FIG. 1 is an overall flowchart of an algorithm according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an arrangement relationship between the calibration box and the captured image. FIG. 3 is a diagram showing an example of an epi curve. FIG. 4 is a diagram showing an automatic search method for corresponding points using correlation. FIG.
FIG. 4 is a diagram showing an example in which an object of a captured image is covered by a triangular region. FIG. 6 is a flowchart showing the details of step S50. 7 to 9 are a flowchart and an explanatory diagram illustrating a sorting method using a bounding box. FIG. 10 is a diagram illustrating back face culling. 11 and 12 are diagrams showing specific examples of the back face or the ring. FIG. 13 is a diagram for explaining warping by homographic conversion. 14 to 16 are diagrams illustrating the bundle adjustment. Hereinafter, embodiments of the present invention will be described with reference to the drawings.

The overall procedure of the method according to the present embodiment will be described with reference to FIG. First, as shown in FIG. 2, the calibration box 21 is used in a first step of obtaining a projection transformation matrix from the three-dimensional space in which the target object 22 is arranged to the photographed image 25 (25a to 25c), as shown in FIG. I do. That is, in the calibration box 21 placed in the three-dimensional space, a plurality of marks 22 such as a square or a triangle are drawn on the floor surface and the wall surface. The coordinates of the vertices of the mark, such as a quadrangle, have known three-dimensional coordinates (X, Y, Z) defined in a three-dimensional space. Each photographed image 25a-2
5c, the mark 23 of the calibration box 21 is photographed together with the target object 22.

Next, a method of obtaining a projection transformation matrix from a three-dimensional space to each photographed image will be described. First, photographed image 2
A projection transformation matrix is obtained for 5a. The three-dimensional coordinates (X, Y, Z) of the designated landmark point 13 and the two-dimensional coordinates (x, y) of the corresponding landmark point of the captured image 25a are read. The three-dimensional coordinates of the specified landmark points (at least 6 or more),
When the two-dimensional coordinates are read, a projective transformation matrix is obtained from these coordinate data by solving six simultaneous equations (or, if there are six or more landmark points, by an operation using the least squares method). This calculation method is publicly known, and is determined by using a conventional technique.

The projection transformation matrix includes all data relating to the camera, such as the position, attitude, and angle of view of the camera 24a. The obtained projective transformation matrix is recorded in the memory. For the other remaining captured images 25b and 25c, a projection transformation matrix is similarly obtained and recorded. The more the number of designated mark points, the more accurate the projection transformation matrix is obtained. Further, in order to generate an image viewed from an arbitrary viewpoint by calculation, it is not sufficient to use only a pair of images, and it is desirable to prepare captured images of the target object viewed from various angles. In this case, in order to make each captured image consistent, it is necessary to adjust the projection transformation matrix by bundle adjustment. The method of adjusting the bundle will be described later.

In step S20 (second step), a point (hereinafter, referred to as a reference point) is designated in a portion representing the shape of the target object for all the captured images. By associating a reference point with each photographed image, each photographed image is associated. By this association, a relationship between an image viewed from the new viewpoint and an arbitrary captured image is determined, and pixel data of the captured image can be used to generate a new viewpoint image. Note that the reference point on each captured image points to the same point on the three-dimensional object. The reference point of the captured image can be automatically specified for the second and subsequent captured images using the correlation as described below.

First, as shown in FIG. 3, an arbitrary one of the photographed images (hereinafter, referred to as a first photographed image) is displayed on a display, and a reference point Pi of a portion representing the shape of the target object is indicated by a mouse (light). (A pen or the like may be used.)
The reference point Pi to be specified is preferably a point at which an image can be easily specified, such as an edge and a corner portion of a target object, a noticeable texture or a cut thereof, a vertex or a concave portion of a projection of the target object, or the like. The reference point is a base point for dividing a captured image of the target object in a third step described later. First
The reference point of the image is specified by the operator using the above items as a guide.

Next, a second photographed image is selected and displayed (hereinafter, referred to as a second photographed image). The display is preferably displayed together with the first image. In addition, when extracting automatically, display is not necessarily required. The corresponding point Qi of the second image is obtained from the reference point Pi of the first captured image (see FIG. 3). The reference point Pi and the reference point Qi point to the same point on the target object. Therefore, the reference point Qi is the epipolar Ei of the reference point Pi.
It is known to be on. The epi curve Ei is the reference point P
This is a linear relationship derived from a constraint condition (epipolar constraint) for i and Qi being the same point of the target object. Therefore, the reference point Qi must be on the epipolar line Ei. Reference point Q
In order to determine i, as shown in FIG. 4, data (pixel data) in the vicinity of the reference point Pi and points Qi (1), Qi (2),. The correlation with the data in the neighboring area of n) is calculated, and the point at which the correlation becomes maximum is determined as a reference point Qi.

Next, a reference point can be determined for a third captured image (hereinafter, referred to as a third image) as follows. First, the coordinates (X, Y, Z) of the corresponding point (point on the target object) in the three-dimensional space are calculated by using the projection transformation matrices of the first and second captured images and the coordinate data of the reference point obtained above. ). The reference point of the third captured image is obtained by applying the projection transformation matrix of the third captured image to the coordinates of the point on the target object. Hereinafter, similarly, the reference point can be determined by calculation for the fourth and subsequent captured images. Repeat the above procedure,
Determine all reference points.

In step S30 (third step), the image of the target object in each captured image is covered with a triangular area. That is, the entire image of the target object is covered with the triangular area using the reference points obtained in step S20. This triangular area is also associated between the captured images. In each captured image, the target object is regarded as a combination of a plurality of triangular regions connected to each other. That is, each triangular area is regarded as a plane passing through its vertex, and this plane is given a corresponding texture of the captured image. Each vertex is given the coordinates of a point in the corresponding three-dimensional space, and the curved surface of the target object in the three-dimensional space is approximated by a polygonal plane connecting a plurality of plane triangles, and the texture is represented in the captured image. Assume that you have a texture that is The above-described covering of the triangular region is performed for all the photographed images, and each triangular region is associated with each reference image along with the reference point, and the data is recorded in the memory.

FIG. 5 shows an example in which a target object represented by a photographed image is divided and covered by a triangular area. 3
It is preferable that the designation of the division of the polygonal region is such that the surface shape of the target object is accurately approximated and the number of the triangular regions is small. Therefore, a plane portion is designated by a large triangle, and a curved surface portion is designated by a small triangle. Triangles are specified so as not to overlap to facilitate later processing. Also,
If the specified triangular area does not accurately reproduce the surface shape of the target object, for example, if the triangular area is too large or incorrectly specified, a reference point can be added or changed. It is. FIG. 5 is an example performed on an image of a can.

In step S40 (fourth step), the angle or position at the new viewpoint is determined, and a projection transformation matrix for the image at the new viewpoint is determined. First, the rotation angle or position is determined. This may be input by numerical data from the keyboard 6 or may be given by the dragged distance of the mouse 7 (the distance moved while pressing the click).

The rotation angle is based on a reference camera angle, for example, the camera angle of the first captured image. That is, both the horizontal angle and the vertical angle of the straight line connecting the viewpoint of the first camera and the origin of the three-dimensional space are selected as a reference and set to zero. The rotation angle is measured from the fixed reference angle, and when stopped, the rotation angle is recorded in the memory, and when dragged again, the rotation angle corresponding to the drag distance is added. From the obtained angle, a corresponding rotation matrix is obtained by calculation, and the rotation matrix is applied to a projection transformation matrix for a captured image to obtain a projection transformation matrix of the new viewpoint image.

Step S50 is a rendering step (fifth step) of generating a new viewpoint image. FIG. 6 shows the details of this step. In step S51, two captured images are selected to determine a reference point of the new viewpoint image. Therefore, the camera optical axis of the new viewpoint is obtained. The camera optical axis refers to a straight line that passes through the optical center (a point corresponding to the center point of the lens) with a straight line perpendicular to the imaging plane of the new viewpoint image and the focal plane of the camera. Calculate the distance between the viewpoint of each captured image and the calculated camera optical axis,
The two photographed images having the shortest distance are selected.

In step S52, corresponding reference points and corresponding triangle vertex data are obtained from the data of the two selected photographed images. For example, there are the following methods for calculating the corresponding reference point. That is, there is a method of using a projective transformation matrix for two captured images and corresponding reference point coordinates. There is also a method in which two epipolar lines are generated from data of two captured images, and the epipolar lines are obtained from their intersection. Further, there is a method using a trifocal tensor. In the present embodiment, the first method is adopted, but the case where another method is used also belongs to the scope of the present invention.

In the first method, three-dimensional coordinates of a corresponding point on a target object in a three-dimensional space are obtained from two projective transformation matrices and corresponding reference point coordinates. This is a method of obtaining a reference point of a new viewpoint image by applying a matrix. Note that the corresponding points on the target object in the three-dimensional space may be obtained in advance, recorded, and used.

Next, in step S53, the obtained reference point corresponds to the triangle vertex data of one selected photographed image (a photographed image having a minimum distance from the optical axis is selected). A triangular area of the new viewpoint image is obtained. The obtained triangular area and the new reference point data are assigned the same index and recorded together with the new triangular vertex data in a memory.

In step S54, sorting for hidden surface processing is performed. In this sorting, the case where the triangular regions intersect as shown in FIG. 7A or 7B does not occur due to the method of forming the triangular regions in the present invention. FIG.
FIG. 7A shows a case where three triangles are in a front-rear relationship with each other, and FIG. 7B shows a case where two triangles are partially in a front-rear relationship. In order to perform sorting, it is necessary to check the context of two triangular regions, and a bounding box (hereinafter referred to as BB) is used as a means for examining the context for sorting.

BB indicates the minimum value of the x coordinate and the minimum value of the y coordinate of the vertex of the triangle as (x, y) as shown in FIG.
Point S to be coordinated, maximum value of x-coordinate of triangle vertex, y
It is defined as a rectangle having a point T at which the maximum value of the coordinates is (x, y) coordinates as a corner point. Similarly, the range of the z coordinate (z minimum value, z maximum value) determined by the minimum value and the maximum value of the z coordinate of the vertex of the triangle is defined as the z range of the triangle. The z coordinate here uses the distance between the optical center and each vertex in the three-dimensional space.

First, any two triangles are selected from among the triangles in the new viewpoint image, and two triangles for hidden surface processing are selected.
FIG. 8 shows a procedure for examining the context of a square. In FIG. 8, in step S1, BB is obtained for two triangles. A step S2 checks whether or not the two BBs overlap. If they overlap, step S3
To check the z range for two triangles. If the BBs do not overlap (step S4), the drawing order is independent, and either one may be drawn first.

In step S5, it is checked whether or not the z ranges overlap. If they do not overlap, processing A is performed in step 6. Processing A is performed without further consideration.
The order is set so that the triangular area having the larger coordinates is drawn first. When the z ranges overlap, the number of vertices shared among the two triangular vertices in the (x, y) plane is checked. The case where there is no shared vertex is the case of (A) to (C) shown in FIG.

In the process B, it is checked whether or not one triangle is inside the other triangle, and if it is inside, it corresponds to the case of FIG. In this case, two triangular z
The coordinates are checked before and after, and the rear triangle is drawn first (or omitted). If it is not inside, it is further checked whether there is any intersection of the line segments on both sides of the triangle. The case where there is no intersecting line segment corresponds to the diagram (B), and the drawing order is independent. The case where there is an intersecting line segment corresponds to FIG. In this case, the anteroposterior relationship of the intersection is examined, and the triangle behind is drawn first.

If one vertex is shared, step 12
The process C is performed. In the process C, it is checked whether or not the line segments intersect. If the line segments do not intersect, this corresponds to FIG. 9D, and the drawing order is independent. In the case of crossing, it corresponds to the diagram (E). The front-back relationship of the crossing point is checked, and the triangle behind is drawn first. 2
If the vertices are shared, it is checked whether they are on the same side of the common line segment. If they are not on the same side, this corresponds to FIG. 11F, and the drawing order is independent. If they are on the same side, check the anteroposterior relationship of the intersections of the intersecting line segments, and draw the rear one first. According to the above-described procedure, the order of drawing two triangles is given. This is performed for all combinations of triangular regions in the new viewpoint image.

In step S55, back face culling is performed. Back face culling refers to omitting texture warping of the triangular area when the triangular area represents the back side when viewed from the focal point F of the camera (see FIG. 10). In other words, the operation is to increase the processing speed without warping the pixel data in a portion that does not represent the surface when viewed from the camera focus at the new viewpoint. As shown in FIG. 11, consider a case where a new viewpoint image is generated from a captured image A for a convex target object. The heart 17 is drawn on the surface ABCD, the diamond 18 is drawn on the surface CDHG, and the spade 19 is drawn on the surface ADHE of the cubic target object ABCDEFGH. First, in the case without back face culling, first, the farthest triangular area EGH is warped as shown in FIG.
A part 19a of the spade 19 and a part 18a of the diamond 18 are warped. In addition, warping is performed on the triangular area EFG. Similarly, this is also performed for a triangular area of the plane ABFE and a triangular area (not shown) of the plane BCGF. After that, the front AEHD, front C
Warping is performed on a triangular area (not shown) specified in the DHG and the plane ABCD, and the heart 17, the diamond 18, and the spade 19 are overwritten as shown in FIG. That is, warping is performed in the order of arrows. On the other hand, when a new viewpoint image is generated by back face culling, the heart 17, the diamond 18 and the spade 19 are directly warped as shown in FIG.
The procedure is only an arrow. Therefore, in this case, the same new viewpoint image is obtained as a result. That is, the back face culling has the effect of shortening the processing time.

When the target object is a concave body, for example, a glass or a dish, the above is different. Cup 1 as shown in FIG.
When the inner surface 17 of 6 appears in the new viewpoint image, the inner surface pattern or color is drawn unless back face culling is performed,
When the back face culling is performed, this portion 17 is not warped, so that it is represented as black (not a pattern and no pixel data). Therefore, what looks like an inner surface like a glass better matches the actual image without backface culling.

The determination as to whether each triangular region in the new viewpoint image is the front surface or the back surface is performed as follows. FIG.
, The triangle 10 is a triangle ABC on the target object in the three-dimensional space. The coordinates of the vertices A, B, and C are known, and the numbers of the vertices 13, 1 and
If it is assumed that the triangle 10 is a plane, the normal vector n of an arbitrary vertex A (or B, C) can be easily obtained. The normal vector n is a triangle 10
The direction and the direction are the same as the normal vector at an arbitrary point in. The focus F of the camera is set, the image of the point A on the image 12 is set as a point a, and the vector from the point F to the point a is set as u.
The narrow angle θ between the vector n and the vector u is −90 degrees <θ <
Whether the angle is 90 degrees or θ is 90 degrees <θ <180 degrees, that is, whether the surface of the triangle 10 is visible from the focal point F is determined by calculating the inner product W = un of the vectors u and n. Good. It can be immediately determined whether or not the surface of the inner product W of the triangle 10 appears in the image 12. If the inner product W is positive, -90
When the degree <θ <90 degrees, the surface of the triangle 10 is not visible, and back face culling is performed. If inner product W is negative, triangle 10
Are visible from the focal point F. The triangle 10 is a plane 3
Since it is assumed to be rectangular, it does not happen that a part of the surface is visible and the other part of the surface is hidden behind.

In step S56, reverse warping is performed. In the backward warping, as shown in FIG. 13, a corresponding point m ′ is obtained from the area of the corresponding triangle 16 of the captured image selected at the point m of the triangle 14 on the new viewpoint image, and the corresponding point m ′ is obtained. Is a process for giving the pixel data of the point m as the pixel data of the point m. Therefore, here, the projective transformation H for transforming the point m of the triangle 14 of the new viewpoint image into the triangle point m ′ of the designated captured image (hereinafter referred to as a source image).
Is a problem.

The projective transformation H includes a method using an affine transformation and a homography transformation using an epicurve. The affine transformation is an excellent approximation method in which a transformation matrix can be easily obtained, but is not an accurate transformation. Homography conversion using an epicurve has a problem that numerical instability occurs when an epicurve is near a straight line passing through two vertices of the same triangle in a case where the triangle in question is elongated or the like. Therefore, when accurate conversion is required, the method proposed below is adopted.

The proposed new system (hereinafter referred to as Gentech system) will be described below. The Gentech method is a method for obtaining a transformation matrix H (3 × 3 matrix) satisfying the following conditions and conditions. Condition: mi ′ = H · mi (i = 0, 1, 2) Condition: (mi ′) ^t · F · mi = 0 (i = 0, 1, 2) The condition is the vertex of the triangle 14 of the new viewpoint image. mi (i = 0,
1, 2) are vertices mi '(i =
0, 1, 2), and the condition is an epipole constraint condition. F is one elementary matrix from a new viewpoint image to a source image. Note that “ ^t ” means transposition. The epipole constraint is transformed like the condition from the result of substituting the condition into the condition. Therefore, this problem is equivalent to obtaining a transformation matrix H that satisfies the condition. The condition is as follows: Condition: ^Ht · F + ^Ft · H = 0. That is, the condition means that the matrix (H ^t · F) is a distortion target matrix.

The conditions are modified based on the conditions. Condition: A · h = 0, where h is a vertical vector of (1 × 9)
Torr, a vector in which the rows of the matrix H are arranged vertically, and h = ｛H
₀₀, H₀₁, H₀₂, H_Ten,. . . H_{twenty two}｝^t It is. Matrix A
Is a (6 × 9) matrix whose coefficients are taken from the fundamental matrix F
You. When the singular value decomposition of the matrix A, A = U · S · V^t When
Can be expressed. Here, U is a (6 × 6) orthogonal matrix, and S is (6 × 6)
9) Matrix, V is a (9 × 9) orthogonal matrix. In addition, S
Last term S of diagonal matrix₆₆Is S₆₆ = 0 and the condition
The solution h has four degrees of freedom. Therefore, the solution h is
Hector h_{0 ,}h_{1 ,}h_{Two ,}h_ThreeCan be expressed as a linear combination of
That is, h = c_{0 ・}h₀+ C₁・ H₁+ C_{Two ・}h₂₊c_{Three ・}h_Three= C
c. Where c = ｛c₀, C₁, C_Two, C_Three｝^t,
And c₀~ C_ThreeIs an unknown real coefficient. The matrix C is (4
x4) matrix, C = ｛h_{0 ,}h_{1 ,}h_{Two ,}h_Three It becomes｝.

The condition may be rewritten as the form of a vector cross product, that is, condition '. Conditions _{': m i' x H ·} m i = 0, by substituting the coordinates of (i = 0, 1, 2) conditions 'in _{m i, m i' (i} = 0,1,2), and rearranging The condition is obtained. Conditions: B · h = 0 where, B is (6x9) in the matrix, its elements m _i and m _i '
(I = 0, 1, 2) coordinate values x _i , y _i and x _i ′, y _i ′
(I = 0, 1, 2), and h is the solution of the condition. Therefore, the target homography H can be obtained by solving the simultaneous equations B, C, c = 0 for c.

The simultaneous equations BCCC = 0 are the equations
And the unknown c is ｛c₀, C₁ , C_Two, C_Three４ four
Therefore, it is uniquely obtained using the least squares method. that's all
, The matrix H was obtained, so that it was suitable for the new viewpoint image 14.
A corresponding point m 'of the point m arranged as appropriate is obtained. Therefore,
By adding the pixel data of the point m 'to the point m, 3
The inverse warping of the squares 14, 16 is performed. The above steps
Is performed for all the triangular regions of the new viewpoint image. This
Gentech's method uses a least-squares method to find a solution
Therefore, it does not become unstable. Therefore, Gentech
Using a homography transform that uses the equation
Constant reverse warping can be performed.

Next, the bundle adjustment method will be described.
If an image viewed from a wide range of new viewpoints is generated from a pair of captured images, an accurate new viewpoint image cannot be generated. Therefore, it is desirable to prepare photographed images of the target object viewed from various angles. For example, as shown in FIG. 14, it is desirable to arrange cameras at regular intervals and to use the captured images. FIG. 14 shows a case where six cameras are arranged at equal intervals. In this case,
For example, in order to generate a new viewpoint image viewed from an angle between the cameras 3 and 4, it is conceivable to use an image captured by the camera 3 or 4 or both captured images. In addition, camera 4
In order to generate a new viewpoint image as viewed from an angle between 5 and 5, it is convenient to use an image captured by the camera 4 or 5.

However, when a plurality of cameras are arranged, if the geometries of the plurality of cameras do not match, the new viewpoint image differs depending on the selected photographed image, which is not preferable. That is, as shown in FIG.
Assuming that a point on the three-dimensional object is M, the points of the images captured by the cameras 1 to 5 are m _{1 to} m ₅ . In this case the optical center of the camera 1 to 5 (point corresponding to the pinhole of the pinhole camera) and a C ₁ -C _5, half lines (m ₁ C ₁₎ ~ half line (m
₅ C ₅ ) must meet at one point M. If they do not intersect at one point M, m _{1 to} m ₅ cannot be regarded as an accurate image of the point M. Therefore, in this case, the parameters of the projective transformation matrix are modified by bundle adjustment.

There are two methods for bundle adjustment. That is, all projective transformation matrices P _i (i = 1, 2,... M) and all reference points M _j (j = 1, 2) on the three-dimensional target object
2. . . N), an explicit bundle adjustment that explicitly minimizes the evaluation J, and a projective transformation matrix P _i (i = 1,
2. . . There is an implicit bundle adjustment that minimizes the evaluation J for M). As the evaluation J, for example, the total sum J (or the square root of the evaluation J) of the square of the standardized deviation distance is adopted as the evaluation. J = {'{(Cx _ij ) ² + (Cy _ij ) ² }, where Cx _ij = m _ij ¹ / m _ij ³ − (P _i · M _j ) ¹ / (P
_i · M _j ) ³ , Cy _ij = m _ij ² / m _ij ³ − (P _i · M _j ) ² /
(P _i · M _j ) ³ . Note that the first Σ is i = 1,
2,. . Represents the sum up to M, the second Σ ′ is j = 1,
2,. . . Represents the sum up to N. Also, m _ij and (P _i ·
M _j ) The number at the right shoulder of indicates the element number.

[0048] The explicit bundle adjustment calculates the evaluation J as P _i (i =
1,2. . . M) and M _j (j = 1, 2,... N) to minimize the optimized P _i ^* (i =
1,2. . . M) and M _j ^* (j = 1, 2,... N) are determined simultaneously. On the other hand, the implicit bundle adjustment minimizes the evaluation J with respect to P _i (i = 1, 2,... M), and then evaluates using the minimized P _i ^* (i = 1, 2,. the J say that minimizes the M _j, optimized P _i by repeating the procedure ^* (i = 1,2 ... M) and M _j ^* _(j =
1,2. . . N).

Although the two solutions should theoretically agree with each other, the intermediate calculation process is different. As a calculation method for minimizing, a minimization method similar to Newton's minimization method is known, and this known technique is used. In the present invention, either of them may be used, and an appropriate one is selected according to the problem.

The embodiments and examples of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to this example, and the design may be changed without departing from the scope of the present invention. The present invention is included in the present invention. For example, the projective transformation in warping is not limited to the Gentech method described above, but may be a homographic transformation using an epi-curve or a case using an affine transformation is within the scope of the present invention. The scope of the present invention is as described in each claim.

[0051]

As described above, according to the configuration of the present invention, according to the first aspect of the present invention, if the corresponding point (or reference point) in each photographed image is determined by one photographed image, the other points can be obtained. Since the photographed image is automatically determined, the operation is facilitated. According to the second and third aspects of the present invention, there is an effect that accurate warping can be performed, and furthermore, the third aspect of the present invention has an effect that calculation for obtaining a transformation matrix becomes stable.

According to the fourth and fifth aspects of the present invention, an image of an invisible shape or the like does not appear in the new viewpoint image. According to the sixth aspect of the present invention, since the back surface of the object in the new viewpoint image is displayed separately from the image of the front surface, there is an effect that the generated image is easy to see. According to the sixth aspect of the present invention, by performing back face culling, the warping of the portion is omitted, so that the processing speed is increased. According to the seventh aspect of the present invention, when a large number of photographed images are used, the coincidence between the photographed images is obtained. Therefore, there is an effect that the difference between the new viewpoint images hardly occurs depending on the selected image in the warping.

[Brief description of the drawings]

FIG. 1 shows a flowchart of an algorithm according to an embodiment of the present invention.

FIG. 2 is a diagram showing a relationship between a target object and an image.

FIG. 3 is a diagram showing an example of an epi curve.

FIG. 4 is a diagram showing an automatic search method of a corresponding point using a correlation. .

FIG. 5 is a diagram illustrating an example in which an object of a captured image is covered by a triangular region.

FIG. 6 shows a flowchart of an algorithm for generating a new viewpoint image.

FIGS. 7A and 7B show intersecting triangles,
(C) is an explanatory view of a boundary box.

FIG. 8 is a flowchart showing a sorting algorithm.

FIGS. 9A to 9G are diagrams showing a relationship between two triangular regions.

FIG. 10 is a diagram for explaining back face culling, and is a diagram showing a principle of obtaining a back face.

FIG. 11 is a diagram showing a specific example of back face culling.

FIG. 12 is a diagram showing a specific example of back face culling.

FIG. 13 is a diagram illustrating reverse warping.

FIG. 14 is a diagram showing an example in which many cameras are arranged.

FIG. 15 is a diagram illustrating that bundle adjustment is necessary when a large number of cameras are arranged.

FIG. 16 shows a sign relationship between an object in a three-dimensional space and a point of a captured image.

[Explanation of symbols]

BB. . . Bounding box Ei. . . Epicurve H. . . Projective transformation matrix between images M. . . A point (vector) on an object in three-dimensional space m. . . M (point) on the captured image of P . . Projection transformation matrix from 3D space to image

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Philip Boromier 5-19-9 Hiroo, Shibuya-ku, Tokyo Hiroo ON Building Inside Gen-Tech Co., Ltd. (72) Inventor Ban Tenrin 5-19-9 Hiroo, Shibuya-ku, Tokyo Hiroo ON Building Gen-Tech Co., Ltd. F-term (reference) 5B050 BA04 BA07 BA09 DA07 EA05 EA11 EA26 5B057 BA02 CA13 CB13 CD20 5B080 BA07 GA21

Claims (7)

[Claims] 1. A method for generating an image of a target object as viewed from a new viewpoint based on images of the target object taken from a plurality of different viewpoints in a three-dimensional space, the method comprising the steps of: A first step of obtaining a projection transformation matrix from space to a captured image, and specifying a point in a portion representing the shape or the like of the captured target object, and specifying the same point of the target object on the plurality of captured images A second step of associating the points of each captured image with each other, and specifying three points among the associated points to form a triangular area, and defining the target object of the captured image as the triangular area. A third step of covering with
Image generation including a fourth step of obtaining a projection transformation matrix from a dimensional space to an image of a new viewpoint, and a fifth step of generating an image of a new viewpoint based on the data obtained in the first to fourth steps In the method, the second step is to search for a point on the epipolar line using the correlation, and to automatically extract a corresponding point, wherein a new viewpoint image is generated. 2. A method for generating an image of a target object as viewed from a new viewpoint based on images of the target object taken from a plurality of different viewpoints in a three-dimensional space, the method comprising the steps of: A first step of obtaining a projection transformation matrix from space to a captured image, and specifying a point in a portion representing the shape or the like of the captured target object, and specifying the same point of the target object on the plurality of captured images A second step of associating the points of each captured image with each other, and specifying three points among the associated points to form a triangular area, and defining the target object of the captured image as the triangular area. A third step of covering with
Image generation including a fourth step of obtaining a projection transformation matrix from a dimensional space to an image of a new viewpoint, and a fifth step of generating an image of a new viewpoint based on the data obtained in the first to fourth steps In the method, the fifth step includes a step of selecting the photographed image, and warping the texture of the selected photographed image to a new viewpoint image by inverse warping using a homography transform. Viewpoint image generation method. 3. The new viewpoint image generation method according to claim 2, wherein the homography conversion obtains a conversion matrix using a Gentech method. 4. A method for generating an image of a target object viewed from a new viewpoint based on images of the target object taken from a plurality of different viewpoints in a three-dimensional space, the method comprising: A first step of obtaining a projection transformation matrix from space to a captured image, and specifying a point in a portion representing the shape or the like of the captured target object, and specifying the same point of the target object on the plurality of captured images A second step of associating the points of each captured image with each other, and specifying three points among the associated points to form a triangular area, and defining the target object of the captured image as the triangular area. A third step of covering with
Image generation including a fourth step of obtaining a projection transformation matrix from a dimensional space to an image of a new viewpoint, and a fifth step of generating an image of a new viewpoint based on the data obtained in the first to fourth steps In the method, the fifth step may include performing sorting on each of the triangular regions so that invisible regions do not appear on the new viewpoint image plane. 5. The new viewpoint image generation method according to claim 4, wherein the sorting uses a bounding box so that an invisible area does not appear. 6. The new viewpoint image generating method according to claim 4, wherein back face culling is performed before or after the sorting step. 7. A method for generating an image of a target object as viewed from a new viewpoint based on images of the target object taken from a plurality of different viewpoints in a three-dimensional space, the method comprising the steps of: A first step of obtaining a projection transformation matrix from space to a captured image, and specifying a point in a portion representing the shape or the like of the captured target object, and specifying the same point of the target object on the plurality of captured images A second step of associating the points of each captured image with each other, and specifying three points among the associated points to form a triangular area, and defining the target object of the captured image as the triangular area. A third step of covering with
Image generation including a fourth step of obtaining a projection transformation matrix from a dimensional space to an image of a new viewpoint, and a fifth step of generating an image of a new viewpoint based on the data obtained in the first to fourth steps In the method, the first step includes a step of correcting a projection transformation matrix by a bundle adjustment method when the number of the captured images is three or more.