JPH11351843A - Apparatus and method of generating three-dimensional picture - Google Patents

Abstract

PROBLEM TO BE SOLVED: To construct three dimensional picture data of an object, using a plurality of recorded pictures by a digital camera, etc., to compose pictures as seen from arbitrary points of view and direction other than at recording. SOLUTION: From a plurality of recorded pictures composed of digitized pixels seen from a plurality of points of view 1, 2,..., i, the position information of an object shown by these pixels is obtd. to construct three dimensional picture data. A plurality of pictures G1, G2,..., Gi composed of pixels holder color data are generated by recording an object from desired no. of points of view V1, V2,..., Vi wherein the view point position and azimuth of records are selected so that the part of the object to be represented as a composite picture is included in any one of the pictures and an identical part of the object to be faithfully represented is included in two or more as many as possible pictures among these pictures.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001] The present invention relates to a means and a method for constructing three-dimensional shape and color data of a target object using an image constituted by pixels holding color information. The present invention also relates to means and a method for synthesizing an image of a target object when observed from an arbitrary viewpoint and azimuth from the data.

[Prior Art] Conventionally, a so-called polygon technique has been widely used as an effective means for constructing and displaying a three-dimensional object as numerical data. To use this, it is necessary to obtain the coordinates of a large number of vertices of a polygon representing an object by some method, and since it is generally not easy, various measures have been taken. Further, since these vertices are approximated by a simple plane or the like, the image to be expressed tends to be different from the actual one unless a large number of polygon vertices are generated. Therefore, it is not easy to generally express the three-dimensional aspect of a real object as an image.

[0003] The present invention is to easily construct a three-dimensional state of a target object as numerical data by using a plurality of recorded images obtained by a digital camera or the like, and to further construct an arbitrary three-dimensional state. Is intended to synthesize and display an image of the target object viewed from the position of.

[Means for Solving the Problems] In the present invention, the target object is first viewed from a plurality of arbitrary viewpoints (V1, V1 in FIG. 1).
2, V3,. . . Vi), a plurality of images (G1, G2 in FIG. 1) composed of pixels holding color data.
2,. . . Gi). At this time, the viewpoint position and orientation of the recording are selected so that the portion of the target object to be expressed as a composite image is included in any of the images, and the target object to be faithfully represented in the three-dimensional shape in these images is selected. Are included in as many images as possible.

[0005] To make the description easier to understand, first, two images G1 and G2 will be described with reference to FIG. In this case, it is assumed that one point p of the target object O is recorded as a pixel p1 in the image G1 and as a pixel p2 in the image G2. Here, the coordinates of the viewpoints (center points of the projection of the image) V1 and V2 of the camera at the time of recording, and the two axes orthogonal to each other, which serve as a reference for the principal axis of the imaging lens of the camera and the position in the image plane orthogonal to this. If the coordinates of three directions of the reference axis (hereinafter referred to as the camera direction or the recording direction) are known, the directions of the straight lines p1-V1 and p2-V2 with respect to the camera direction (ie, V
Since the orientations of 1-p and V2-p) are determined from the position coordinates of the pixels p1 and p2 in the image, the imaging distance of the camera (the focal length of the imaging lens for a target object sufficiently far from the camera) and If the size of the pixel is known, the three-dimensional position coordinates of p can be calculated from the principle of triangulation for the triangle p-V1-V2-p in FIG.

As described above, the conditions that can be used to determine the three-dimensional position coordinates consisting of three unknown values of p are 2
There are four conditions for satisfying four values in total, two for each position coordinate value of the pixels p1 and p2 in the two-dimensional image. That is, since the number of conditional expressions is excessive compared with the number of unknowns, it cannot always be strictly satisfied. Therefore, in practice, it is necessary to determine the position coordinates of p by, for example, the least square method so that the error of the conditional expression is minimized.

At the same time, even if the viewpoint and orientation of the camera, which have been known so far, furthermore, the imaging distance of the camera for each image, or the distortion coefficient of the imaging lens inherent in the imaging lens, etc., is unknown. If a sufficient combination of pixels (hereinafter referred to as a corresponding pixel set) corresponding between the images as representing the same point of the target object is obtained, p
Means that these new unknowns as well as the position coordinates of are obtained at the same time. This can be said not only between two images but also between a large number of images as long as the number of independent conditional expressions more than necessary to determine the unknown is obtained.

For example, in the case of two images, the viewpoint, azimuth, and imaging distance of the camera at the time of image recording are also unknown, and if the minimum necessary independent corresponding pixel sets are N sets, the number of unknowns becomes Since the viewpoint and the direction of the camera are relative, if the viewpoint of one camera is set as the origin of the orthogonal coordinate system and the coordinate axis is set to the direction of the camera, the viewpoint of the other camera becomes 3 and the direction of the camera , 3 for the imaging distance, 2 for each image, 2 × 1 = 2, and 3 for the position coordinates of the point in the target object related to the corresponding pixel set, 3 × N × 3, for a total of 3N + 8. Since the number is 4 for each corresponding pixel set, a total of N × 4, 3N
From + 8 = 4N, N = 8. When nine or more independent corresponding pixel sets are obtained and the conditional expressions become excessive, these unknowns are obtained by the least squares method. That is, the above unknowns may be selected so that the value obtained by adding the square of the error of the conditional expression for all conditional expressions is minimized. In addition,
The coefficients of the imaging distortion of the imaging lens (a plurality of coefficients up to the required order) and the like are also obtained as unknowns if a sufficient number of independent corresponding pixel sets are obtained.

[0009] With respect to the pixels associated as a corresponding pixel set among these images, the three-dimensional position of the point of the target object represented by the pixel is determined. If the target object is assumed to consist of a smooth surface, by assuming that the surface between these fixed points is a plane or a suitable curved surface, the position between the fixed points is not known yet. The three-dimensional position of the point can also be approximately calculated.
Therefore, for all the pixels of these images, the information on the three-dimensional position and the color of the point of the target object to which the pixel relates is determined.

By the way, how close the three-dimensional information of the target object obtained by the above method is to reality depends on how many corresponding pixel sets can be extracted. However, the correspondence between pixels between images is not always sufficiently obtained simply by searching for characteristic points in the image, regardless of whether the image is obtained by analyzing the image using a computer program or by visual observation. .

In the present invention, in order to improve such a problem,
A characteristic line (straight line or curve) existing in the image is used. For example, patterns and wrinkles in clothes worn by humans, folds on the surface of boxes and the like, shadows caused by lighting conditions, contour lines of the target object resulting from the positional relationship between the camera's viewpoint and the target object, etc. There are many in the image and between the images as representing the same part of the object,
For example, there are many things that can be easily associated with each other by visual inspection (out of these, the outline is slightly different, but this will be described separately later).

However, in most of these lines, although it is obvious that the lines correspond between the images, it is not clear which part (pixel) of the line corresponds to which part (pixel) of the corresponding line. I cannot judge it immediately. According to the present invention, the correspondence between the pixels belonging to the line can be obtained from the correspondence between the lines in the following manner, and therefore, the three-dimensional positions are determined as a new corresponding pixel set. This contributes to increasing the number of points of the target object to be determined.

First, as described above, characteristic points in an image are examined to extract as many corresponding pixel sets as possible, and three-dimensional position coordinates relating to the corresponding pixel sets are obtained, and at the same time, these images are obtained. Find the coordinates of the viewpoint and azimuth at the time of recording. If the image is already known in some way or the like, the view direction can be omitted. Once the viewpoint orientation of the image recording (including the numerical values necessary for determining the position coordinates of the point of the target object such as the imaging distance) is determined, as shown in FIG. Can be determined.

That is, as shown in FIG. 2, pixels included in one of the two lines s3 of the image G3 and s4 of the image G4, which correspond to each other, are represented by ps3 and ps3, respectively. Assuming that a point is po, a plane L including ps3, viewpoints V3 and Po of the image G3, and viewpoint V4 of the image G4 is considered. At this time, the pixel ps4 in s4 corresponding to po is naturally included in the plane L. Since po originally corresponds to ps3, the pixel ps4 at the intersection of s4 and L is ps3.
It can be seen that the pixel corresponds to. Since such a correspondence relationship of pixels can be applied to any pixel on the line in principle, it is possible to obtain as many corresponding pixel sets as necessary from these lines.

In order for the corresponding pixel to be determined with high accuracy in this way, it is necessary that the plane L intersects the lines s3 and s4 at a sufficient angle. It is also desirable that s3 or s4 does not cross L twice or more. When crossing two or more times, it is necessary to determine the corresponding pixel by finely dividing the line so as to make one crossing, or to specify the corresponding pixel using other surrounding image information.

A feature of this method is that unlike the conventional case where three-dimensional position information relating to pixels is obtained from analysis of a correlation function between images, it is not necessary that the viewpoints of recording both images are close to each other. In the image correlation method, if the viewpoints are too far apart, the composition of the image is too different, making it difficult to obtain an effective correlation function. However, here, the viewpoints of both images may be far apart.

Among the characteristic lines of these images, the contour lines have different properties. If the contour line matches another structural feature line of the target object, such as a fold on the surface, it can be handled in the same way as described above. Apart from that, a contour generated from the positional relationship between the surface curvature and the viewpoint from which it is viewed The line does not have a corresponding structural or pattern line of the target object. Therefore, it is difficult to obtain a clear line corresponding to such a contour line in another image.

However, considering the composition of the image determined from the positional relationship of the viewpoints at the time of image recording, it is not enough to estimate the approximate position of the point sequence on the target object forming the outline in another image. There are many things you can do. In such a case, a line is artificially set at the approximate position, and in the present invention, this is used as a special corresponding line approximately corresponding to the outline as follows. Obtaining the three-dimensional position information of the surface is very effective because the shape of the surface of the target object is clearly captured when viewed from the side.

The special correspondence line s6 (FIG. 3) corresponding to such a contour line s5 (FIG. 3) shows an approximate correspondence and is not accurate. Therefore, using only the corresponding pixel sets obtained from other points, the points of the target object
First, dimensional position information is obtained. Next, consider an approximated surface of the target object surface that smoothly connects these three-dimensional positions,
Furthermore, in the vicinity of the line so6 on the target object corresponding to the special corresponding line s6 (for example, within a range of a predetermined distance error), the recording of the image G5 (FIG. 3) to which the contour line s5 belongs is performed. When viewed from the time point V5 (FIG. 3), the curved surface is conditioned so as to form the contour line s5.

That is, in FIG. 3, a straight line connecting the pixel ps5 belonging to the contour line s5 and the viewpoint V5 and the curved surface correspond to a line so6 obtained by projecting the corresponding line s6 on the curved surface with respect to the viewpoint V6.
Condition that the surface touches in the vicinity of. Contact point p at this time
The position of o5 is not necessarily on so6, and the above-mentioned conditions are best adapted including the condition that the curved surface passes through a point group of the target object in the vicinity of which the three-dimensional position has already been determined. By optimizing the curved surface, for example, it is determined by expressing the curved surface with a polynomial of a three-dimensional coordinate variable of a predetermined order and optimizing the coefficient of each term by the least square method. In this way, by arbitrarily selecting the pixel ps5 on the contour line s5, the corresponding po
5 is determined as the three-dimensional position of the corresponding point on the target object, and the pixel p near s6 corresponding to po5
s6p is obtained as a pixel corresponding to ps5.

Of course, the accuracy is inferior, but the above method is simplified, and the line s6, which means an approximate correspondence, is treated as a definite position correspondence line in the same way as the corresponding line set up to now. Is also good.

From the above, many points of the target object whose three-dimensional position coordinates are known can be obtained. These information are
The coordinates of the vertices of the three-dimensional polygon model can be used for modeling and displaying the target object according to the related art.

More generally, the recorded image is divided into a plurality of areas in advance according to the shape of the target object.
From the information of these three-dimensional position coordinates, for each region, the target object is approximated as a plane or a curved surface by, for example, a polynomial of a coordinate variable in a rectangular coordinate system, and the information is held as a coefficient of the polynomial, and the information is held. Can be used for display. In this case, a three-dimensional model can be formed for each recorded image.
Of course, one image does not necessarily record all the sides of the target object. For this reason, when it is intended to synthesize an image of the target object from a place different from the viewpoint and azimuth at the time of recording the image, not only the three-dimensional position information of the pixels but also the color information is obtained from the plurality of recorded images. There is a need to gather.

By the way, since each image has different conditions at the time of recording, it is usual that the same point of the target object is recorded with mutually different color information. Therefore, it is desirable to adjust the color information in the above-described image synthesis. As a method, in the present invention, for each of the above-described divided regions of the image, each pixel is divided between the corresponding portion of the target object and the divided region of another image having a portion having the same color as that of the target object. Compare the colors and adjust them so they appear in the same color. For this purpose, first, a plurality of regions whose colors are to be mutually adjusted are selected, and a pixel or a pixel group to be matched in color is selected from each region. Next, a reference color is determined from the color information of these pixels by an averaging operation or the like, and an adjustment coefficient of, for example, an RGB value of each region is calculated in accordance with the reference color. Then, the pixels in each area are corrected with this coefficient. The corrected color information may be held as a new color information of each pixel in the form of a corrected image or the like and used for image synthesis, or may be held in the form of an adjustment coefficient and used for image synthesis. At times, the color may be corrected.

[Embodiment of the Invention] It is most direct and easy to use a digital camera using a semiconductor image sensor as an image generating means. Since the image output is color information already digitized in pixel units, this is input to a computer, and the above processing is executed by a computer program. Various modifications are conceivable in the actual process, and one example will be described below.

First, for each input image, a boundary line is appropriately set for each surface constituting the object in the image in preparation for approximating the target object with a surface (generally, a curved surface). Divide into regions. This is performed by displaying the image on a display device of a computer and designating a main point of the boundary line with a pointing device. At that time, the boundary line holds information on the positional relationship between the adjacent surfaces (or regions) (whether they are continuous or whether one surface hides the other). .

Next, by comparing the images, it is visually determined that the same characteristic point of the target object is recorded in two or more images, and these points are designated by a pointing device. Are stored in the table as the corresponding pixel set. If the same point on the target object is 3
When it corresponds to more than one pixel, it is preferable that some pixels are overlapped every two pixels and held as a corresponding pixel set. This work may be automatically performed by analysis using a correlation function between images.

Next, a comparison is made between the images to visually check that two or more characteristic lines of the target object are recorded in two or more images, and their principal points are designated by a pointing device. Then, those pixel coordinates are stored in the table as a corresponding line set. If the same line of the target object corresponds to three or more image lines, two lines at a time
Some lines may be overlapped and held as a corresponding line set. This operation may be automatically performed by analyzing the contrast in the images, calculating the correlation function between the images, and the like.
At this time, if the corresponding line set is a special one related to the above-mentioned contour line, information is added so that the line can be identified.

Next, using the corresponding pixel set obtained above, the three-dimensional position coordinates of the point of the target object relating to the pixel, the coordinates of the position and orientation of the camera when these images were recorded, and the image formation The distance and the like are obtained by the method described above (if any of these values is known, enter it in advance).

Next, using the camera position and azimuth coordinates obtained above, the image forming distance, and the like, a new corresponding line set other than the special corresponding line set relating to the contour line is newly obtained by the method described above. A corresponding pixel set is generated, and the three-dimensional position coordinates of the point of the target object relating thereto are obtained.

Next, as described above, the three-dimensional position coordinates of the pixels belonging to the special corresponding line set relating to the contour are calculated as follows.
The distance between the points of the target object whose position is known so far is obtained by a method of approximating with a curved surface. In this example, this curved surface is generated for each region of the divided image.

Here, the curved surface generated for each divided area is
It is not limited to the case where it is determined only from the corresponding point set and the corresponding line set belonging to the area. Since each region is in contact with another region at its boundary line, it is 3
Dimensional positional relationship (mutual concealment relationship or mutual continuity relationship)
Consider. That is, the curved surface is optimized according to the boundary conditions that depend on the state of the curved surface in the surrounding region, in addition to the three-dimensional position information obtained from the corresponding point set and the corresponding line set belonging to the region. In this optimization, each curved surface is approximated by a polynomial of a coordinate variable in a rectangular coordinate system or its modification.

The optimization of the curved surface is performed on the curved surface in all the areas of all the recorded images by the sequential method as follows. First, an approximate expression is determined using the planes of all the regions as appropriate planes. Next, the above-described optimization is performed for each curved surface in each region. When this optimization has been completed for all the regions, the above-described optimization is repeated for each curved surface using the respective approximate expressions obtained so far and for each curved surface of each region from the beginning. Then, the optimization is repeated until all the curved surfaces are within the predetermined error range. Note that some of these curved surfaces can be approximated by limiting them to planes to simplify the calculation.

In this way, three-dimensional position information of many points of the target object is obtained, and a three-dimensional model approximated by a curved surface is generated for each recording image. As already mentioned,
The result can be used as it is as the vertex data of the conventional three-dimensional polygon model. In this case, in addition to the three-dimensional position information of the point of the target object obtained from the corresponding point set and the corresponding line set, three-dimensional information of points between these points is obtained from the surface approximation.
In addition to the dimensional position information, more precise data can be obtained.

Here, a three-dimensional image is easily synthesized by directly using the generated curved surface model. For this purpose, a computer, a computer program for performing the following processing, and a display device for displaying a result are used. That is,
An appropriate one of the generated three-dimensional models for each recorded image is selected, and an image of the target object at an arbitrary viewpoint and azimuth is synthesized by ray tracing. This is performed by obtaining each pixel of the composite image from the model by emitting a corresponding ray from the viewpoint. If that fails, the process is repeated until pixels are found in the three-dimensional model of another recorded image or until all the three-dimensional models have been examined. As described above, the color information of the pixel is adjusted for each region, that is, for each curved surface. The composite image thus obtained is displayed on a display device.

[0036]

According to the present invention, an extremely large number of target objects can be obtained from a plurality of images recorded by a digital camera or the like.
It is possible to acquire dimensional position information, and to generate a precise three-dimensional model of the object, and thereby to synthesize an image at an arbitrary viewpoint and azimuth. Since this method can be equally applied to all target objects included in a recorded image, it is possible to collectively generate a three-dimensional model of an object or a landscape over a wide range only by recording a relatively small number of images. .

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a relationship between a target object, a viewpoint, an orientation, and the like when generating a plurality of recorded images thereof.

FIG. 2 is a diagram illustrating a correspondence between one feature line on a target object, a corresponding line set in an image thereof, and pixels included therein.

FIG. 3 is a diagram illustrating pixel correspondence in a special correspondence line set generated from a contour of a target object.

[Explanation of symbols]

O Target object O5, O6 Image of target object G1, G2, G3,. . . , Gi a plurality of images V1, V2, V3,. . . , Vi View points for recording a plurality of images p Pixels in the image corresponding to points p1, p2 p on the target object so So feature lines s3, s4 on the target object Targets belonging to the corresponding line set line po so corresponding to so Pixels in the image corresponding to the points ps3, ps4 po on the object (pixels of the corresponding pixel set) Ls3, V3, po, V4 A plane passing through V4, a special correspondence line set generated from the contour of the target object Line. s5 is a contour line of the target object, s6 is a line sos6 corresponding to the approximate line so6 s6 A line on the target object corresponding to s6 ps5 A pixel D5 belonging to a pixel po5 ps5, V5 belonging to s5 and a point Dps5, V5, Cross section of the target object by a plane passing through V6 ps6 Intersection of s6 with a plane passing through ps5, V5, V6 po6 Point on the target object corresponding to ps6 so6p In the image corresponding to line s6p so6p on the target object corresponding to s5 Line

Claims (5)

[Claims] 1. A plurality of recorded images of a target object at a plurality of viewpoints and orientations generated by an image generating means for generating an image in which pixels composed of digitalized color information of each point of the target object are formed as constituent units. A plurality of pixels that record the same point of the target object are calculated or visually extracted, acquired and held as a corresponding pixel set, and a plurality of the corresponding pixel sets are acquired, and based on these corresponding pixel sets, these recordings are performed. While acquiring information on the three-dimensional position coordinates of the point of the target object corresponding to each of a part or all of the pixels of the image, the coordinates of the viewpoint and azimuth of the imaging at the time of recording each image, or each of the images An apparatus comprising means for enabling acquisition of information regarding the imaging distance of an imaging lens when recording, or the coefficient of imaging distortion of the imaging lens, etc. The information acquisition method called. 2. A plurality of recorded images of a target object at a plurality of viewpoints and azimuths generated by an image generating means for generating an image in which pixels composed of digital value color information of each point of the target object are formed as constituent units. A plurality of pixels that record the same point of the target object are calculated or visually extracted, acquired and held as a corresponding pixel set, and a plurality of the corresponding pixel sets are acquired, and based on these corresponding pixel sets, these recordings are performed. Acquires information on the three-dimensional position coordinates of the point of the target object corresponding to each of some pixels of the image, and simultaneously records the coordinates of the viewpoint and azimuth of the imaging when each of the images is recorded, or records each of the images The imaging distance of the imaging lens at the time of acquisition, or the coefficient of imaging distortion of the imaging lens, or the like, or the viewpoint of imaging when each of these images is recorded. The coordinates of the azimuth, or the imaging distance of the imaging lens when each image is recorded, or all or part of the imaging distortion coefficient of the imaging lens, etc., is obtained in advance by other measuring means, etc. In the plurality of images of the target object, a plurality of curvilinear or linear pixel groups that record the same portion of the target object are calculated or visually extracted, acquired and held as a corresponding line set, and Or using a plurality of these and the viewpoint and the azimuth information, for any pixel in one of the corresponding line sets,
Pixels in other lines in the same corresponding line set corresponding to representing the same point of the target object are obtained by calculation, and these mutually corresponding pixels are held as a new corresponding pixel set, and A necessary and possible number of new corresponding pixel sets are acquired and held for each corresponding line set, and information on the three-dimensional position coordinates of points of the target object corresponding to those pixels obtained from these new corresponding pixel sets is obtained. An apparatus having means for obtaining a more precise three-dimensional shape of the target object, together with information on three-dimensional position coordinates of a point of the target object relating to pixels acquired and held as a corresponding pixel set from the beginning, and A method for obtaining information on three-dimensional position coordinates of a point of a target object corresponding to a pixel in a corresponding line set. 3. An apparatus and method according to claim 2, wherein all or a part of the corresponding line set consisting of the contour line of the target object and a line substantially corresponding thereto is included in the corresponding line set. Regardless of the description, when the surface approximating the target object is viewed from the viewpoint of recording of the image to which the contour of the target object in the corresponding line set belongs, the surface roughly corresponds to the contour in the corresponding line set. In the vicinity of a line on the target object corresponding to the line to be formed, by conditioning a surface approximating the target object so as to form the contour, the three-dimensional position of a point of the target object corresponding to a pixel of the contour is determined. Acquiring and obtaining pixels in the vicinity of a line approximately corresponding to the contour line corresponding to the point of the target object, a more accurate 3
Apparatus and method for obtaining dimensional shape information. 4. The apparatus or method according to claim 1, wherein the plurality of recorded images are divided into a plurality of regions in advance according to the shape of the target object, and the target object Curved shape (including plane)
And a boundary condition that defines a three-dimensional positional relationship between portions of the target object corresponding to portions of the region that are in contact with each other at the boundary is given to the boundary of the division, Three-dimensional position information of a point of the target object obtained by the apparatus or the method of claim 2 or 3;
In the case of claim 3, the condition relating to the contour line formation;
An apparatus and method for optimizing an approximated curved surface in the region in combination with the boundary condition. 5. Using the information or the like obtained by the apparatus or method according to claim 1, 2, 3, or 4, and using the information or the like obtained by recording the target object, any information other than the viewpoint and azimuth when recording the target object. In an apparatus and a method for synthesizing an image of the target object viewed from the viewpoint and the azimuth, a plurality of recorded images used for acquiring the information are divided into a plurality of regions in advance according to the shape of the target object. The surface of the target object is approximated by a plane or a curved surface using the information for each of the regions, and another region having the same portion of the corresponding portion on the target object or a portion having the same color as the corresponding portion on the target object is provided for each of the regions. By comparing the color information of each pixel with the region in one or more images, the color of the pixels belonging to each region is adjusted, and the pixel information of the plurality of images is combined. Apparatus and methods for its synthesis and wherein to synthesize a single image by using Te.