JP2001324310A - Method, device and system for preparing three- dimensional data - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method, a device and a system for preparing three- dimensional data, capable of providing accurate three-dimensional data from a plurality of arbitrarily photographed images. SOLUTION: A plurality of static images of a space with an unknown space structure are captured by a computer 14 from a plurality of cameras 10 and 12. The positions of observation points to the static images are computed from the relative positions of not more than two pixels, or not more than eight pixels, having only a known correspondence with three or four pixels having known coordinates in the plurality of images. The positional relations of the plurality of images relative to the observation points are found in the same coordinate system to prepare three-dimensional data on a photographed object. Pictures comprising the images are put in the same coordinate system and the plurality of images photographed from different positions are each divided into a plurality of image areas, thus determining one-to-one correspondences between the image areas in each of the plurality of images.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a modeling method and apparatus for creating three-dimensional data of a photographing object from a plurality of images of the same photographing object.

[0002]

2. Description of the Related Art In the prior art, there has been no method for inexpensively creating three-dimensional data to be photographed with high accuracy. The inventor of the present invention disclosed a stereoscopic data creation method and apparatus disclosed in Japanese Patent Application Laid-Open No. 2000-76453 (hereinafter simply referred to as “stereoscopic data creation method”) to convert stereoscopic data of an imaging target space whose spatial structure is unknown. We have devised a device to create and a device to convert the appearance of any object without a design document into three-dimensional data. This makes it possible to create high-precision three-dimensional data required by the above-mentioned problem with an inexpensive device. However, even in this method, when irradiating the slit light source to the object to be photographed, the object is slid vertically and horizontally. In addition, when creating stereoscopic data from an arbitrarily photographed image, it is necessary to use a stereoscopic data model separately prepared in advance for each part.

[0003] Further, a method of approximately creating three-dimensional data from one or a plurality of still images obtained by photographing an indoor or outdoor space using an ordinary camera by an ordinary person has been devised and put into practical use in several ways. However, three-dimensional data could not be created with high accuracy.

Japanese Patent Application Laid-Open No. H11-96374 discloses a method in which the same object is photographed by a plurality of cameras to create three-dimensional data. However, it is not possible to determine the position of the viewpoint from an image taken arbitrarily, and if the position of the screen and viewpoint is determined from the image of the interior or external space, the error must be large, It is difficult in terms of accuracy to perform collation in pixel units and create three-dimensional data in the target space.

[0005] Particularly, in an image photographed in an indoor space or outdoors, a photographing object having a fine pattern and unevenness as exemplified in item 0022 is highly likely to be projected, and a large number of objects whose photographing space is discontinuous are high. It is likely that a large number of pixels having exactly the same color data are likely to be generated, and the method of simply voting in the voxel space poses a risk that the estimation conditions are not effective. In addition, there are many objects to be photographed having fine patterns, such as trees, patterned sofas, and wallpaper, and it is often difficult to collate the correspondence in pixel units due to the limited number of pixels. . On the other hand, unlike a human face or the like, since it is composed of pixels of various color data, it is often easy to associate one part in image region units. Therefore, when converting three-dimensional data into indoor and outdoor spaces, a system based on the above characteristics is desired.

[0006]

SUMMARY OF THE INVENTION It is an object of the present invention to reinforce the above-mentioned "three-dimensional data creation method" and to provide the following three-dimensional data creation method, apparatus and system. 1) Three-dimensional data creation method and apparatus for creating three-dimensional data of an object to be photographed simply by running one vertical (or horizontal) slit light source in the normal direction. 3D data creation method and apparatus for automatically or semi-automatically creating 3D data of a photographing target without hitting 3) Matching a plurality of points from a plurality of images arbitrarily photographed in the same photographing space to check the viewpoint and screen of the camera 3D data creation method and apparatus for determining 3D position and creating 3D data in the target space 4) From the images taken from a plurality of points, the position information of the shooting points is simultaneously obtained to determine the viewpoint of the camera and the position of the screen. ,
3D data creation method and apparatus for creating 3D data of the target space 5) By analyzing the spatial structure and creating 3D data on the server side from a plurality of image data input on the terminal side, simple on the terminal side Data creation system using communication network that enables simulation to be performed

[0007]

According to a first aspect of the present invention, there is provided a method for producing three-dimensional data, wherein slit light beams projected on the surface of an object to be measured are simultaneously transmitted from photographing means located at a plurality of viewpoints having different positions. While measuring, the vanishing point is measured in the same space to determine the vanishing point, the viewpoint of the photographing means is obtained, the basic composition derived from the plurality of photographing means is overlapped and collated, and the slit projected on the measurement object A method of identifying the position of each point on the surface to be measured from light rays and creating three-dimensional data of the object that was the target of imaging, and pixels to be corresponded in a plurality of images measured from a plurality of imaging means, The correspondence between pixels is obtained on condition that the pixels are located on the same plane as the viewpoints of the plurality of images, and stereoscopic data is created based on the correspondence. Further, the stereoscopic data creating method according to claim 2 of the present invention measures slit light rays projected on the surface of an object to be measured simultaneously from photographing means located at a plurality of viewpoints having different positions, and simultaneously measures the slit light rays in the same space. Measure the vanishing point in the reference frame to determine the viewpoint of the photographing means, superimpose and compare the basic compositions derived from the plurality of photographing means, arrange the viewpoint and the screen in the same coordinate system, or The viewpoint and the screen are arranged in the same coordinate system based on the positional relationship of the corresponding pixels of five or more points whose upper coordinates are unknown, and the position of each point on the surface of the measurement target is specified from the slit light beam projected on the measurement target and photographed. Is a method of creating three-dimensional data of a target object, which is formed by intersecting a plane or a curved surface of each screen created by connecting each pixel and a viewpoint in a plurality of images measured from a plurality of photographing means. Tertiary It is to create a three-dimensional data based on the straight or curved in space.

According to a third aspect of the present invention, there is provided a method for determining a correspondence between image regions, wherein a plurality of still images in a space having an unknown spatial structure are captured, and the still images are captured based on a positional relationship of straight lines in the captured still images. A method of determining a position of a viewpoint with respect to an image, and determining a one-to-one correspondence between image regions in three-dimensional data creation for creating three-dimensional data of a spatial structure in a still image based on information of a still image and a viewpoint,
Each screen composed of images is placed in the same coordinate system, a plurality of still images photographed from different positions are divided into a plurality of image regions, and a one-to-one correspondence between the image regions in the plurality of images is determined. .

According to a fourth aspect of the present invention, there is provided a method for determining a correspondence between image regions, wherein a plurality of still images in a space having an unknown spatial structure are fetched, and three or four points whose coordinates in the fetched still images are known. The position of the viewpoint with respect to the still image is calculated from the positional relationship of two or less pixels whose correspondence is known only with the pixel of 7 or less, or the pixel whose correspondence is known only with seven or less, and the relative positions of the plurality of still images and the viewpoint are determined. A method of obtaining a relationship in the same coordinate system, where each screen composed of images is placed in the same coordinate system, a plurality of still images photographed from different positions are divided into a plurality of image regions, and the The one-to-one correspondence between the image areas is determined.

According to a fifth aspect of the present invention, there is provided a three-dimensional data creating method using the image area correspondence relationship determining method according to the third or fourth aspect, wherein each of the two screens has a one-to-one correspondence. The corresponding pixels in the region are located on the same plane as the viewpoint of each of the two screens, and at two points located on the same plane, the left and right positional relationship on one screen is also on the other screen. The correspondence in pixel units is determined on the condition that they have the same left-right relationship or the same point, and three-dimensional data is created based on the correspondence.

According to a sixth aspect of the present invention, there is provided a three-dimensional data creating method using the image area correspondence determining method according to the fourth aspect of the present invention. The corresponding pixels on the boundary line of the corresponding image area are located on the same plane as the viewpoints of the plurality of screens, and at two points located on the same plane, the left and right positional relationship on one screen is On the other hand, on the condition that the left-right relationship is also the same on the screen, the correspondence in pixel units is specified, and a part of the three-dimensional data of the shooting target is created.

According to a third aspect of the present invention, there is provided the three-dimensional data creating method according to the fifth or sixth aspect, wherein the position information of the photographing point obtained at the time of photographing and the map information of the external space are obtained. A plurality of images obtained by photographing the same photographing target space are selected on the basis of the above, the approximate coordinates or basic composition of three or more points in the photographing target external space are obtained, and the positions of a plurality of screens and viewpoints are set in the same coordinate system. Things.

According to a third aspect of the present invention, there is provided the three-dimensional data creating method according to the fifth or sixth aspect, wherein the viewpoint of each of the two screens arranged in an arbitrary coordinate system is different from the viewpoint of the three-dimensional data creating method. For pixels located on the same plane and arranged on a straight line on each of the two screens, the saturation order of the color data, the maximum value of the brightness, the minimum value, or the On the condition that they match on the screen, the correspondence in pixel units is performed.

According to a ninth aspect of the present invention, there is provided the three-dimensional data creation method according to the fifth or sixth aspect, wherein three or more screens arranged in the same coordinate system are used to generate the pixel data. When performing the association, the pixels associated with any two screens and the pixels associated with the screens other than the two screens correspond to the viewpoints of the screens other than the two screens, the viewpoints of each of the two screens, and the correspondence. By utilizing the fact that the pixels are on the same plane as the attached pixels, the positions of the pixels on the screen other than the two pixels are obtained, and the quality of the association of the associated pixels is determined from the color data.

According to a tenth aspect of the present invention, there is provided a three-dimensional data creating apparatus, comprising: a plurality of photographing means; a slit light source for projecting a slit light beam on a surface of an object to be measured; Calculation means for simultaneously measuring the projected slit light rays from the imaging means located at a plurality of different viewpoints, and measuring the vanishing point in the same space as a reference frame to obtain the viewpoint of the imaging means, Generate and generate three-dimensional data of an object to be photographed by superimposing and collating basic compositions derived from a plurality of photographing means, specifying the position of each point on the surface of the measurement object from the slit light beam projected on the measurement object Means for generating a pixel phase on the condition that pixels to be corresponded in a plurality of images measured from the plurality of photographing means are located on the same plane as the viewpoints of the plurality of images. Seeking correspondence, it is to create a three-dimensional data based on this. Further, the three-dimensional data creating apparatus according to claim 11 of the present invention includes a plurality of photographing means, a slit light source for projecting a slit light beam on the surface of the measurement target object, and a slit light source projecting on the surface of the measurement target object. Calculating means for simultaneously measuring slit light rays from photographing means located at a plurality of viewpoints at different positions, measuring a vanishing point in a same frame as a reference frame, and obtaining a viewpoint of the photographing means; The viewpoint and the screen are arranged in the same coordinate system by superimposing and comparing the basic composition derived from the means, or the viewpoint and the screen are arranged in the same coordinate system based on the positional relationship of five or more corresponding pixels whose coordinates on the object to be photographed are unknown. And generating means for specifying the position of each point on the surface of the measurement target from the slit light beam projected on the measurement target and creating three-dimensional data of the object to be photographed, the generation means comprising a plurality of Is it a shooting method? Is to create a three-dimensional data based on the straight or curved in a three-dimensional space formed by the intersection of a plane or curved surface of each screen to be created by connecting the respective pixel and the viewpoint of the measured plurality of images.

According to a twelfth aspect of the present invention, there is provided a three-dimensional data creating apparatus, wherein a plurality of photographing means, and a plurality of still images in a space having an unknown spatial structure are captured from the plurality of photographing means, and the captured still image Calculating means for calculating the position of the viewpoint with respect to the still image based on the positional relationship of the straight line, and a pair of image regions in the three-dimensional data creation for creating three-dimensional data of the spatial structure in the still image based on information on the still image and the viewpoint Generating means for determining three-dimensional correspondence and creating three-dimensional data of the object to be photographed, wherein the generating means places each screen composed of images in the same coordinate system, and a plurality of images photographed from different positions. A still image is divided into a plurality of image areas, a one-to-one correspondence between image areas in the plurality of images is determined, and three-dimensional data is created based on the correspondence.

According to a thirteenth aspect of the present invention, there is provided a three-dimensional data creating apparatus, wherein a plurality of photographing means, and a plurality of still images in a space having an unknown spatial structure are captured from the plurality of photographing means, and the captured still image Calculating means for determining the position of the viewpoint with respect to the still image from the positional relationship of two or less pixels whose correspondence is known only with three or four pixels whose coordinates are known, or the seven or less pixels whose only correspondence is known And generating means for obtaining the relative positional relationship between the plurality of still images and the viewpoint in the same coordinate system, and generating three-dimensional data of the object to be photographed. Are placed in the same coordinate system, a plurality of still images photographed from different positions are divided into a plurality of image regions, a one-to-one correspondence between the image regions in the plurality of images is determined, and a three-dimensional It is intended to create an over data.

According to a fourteenth aspect of the present invention, in the three-dimensional data creating apparatus according to the twelfth or thirteenth aspect, the generating means further includes a one-to-one correspondence between each two screens. That the corresponding pixels in the image area are located on the same plane as the viewpoints of the two screens, and that at two points located on the same plane, the left and right positional relationship on one screen is the other The correspondence is obtained on a pixel-by-pixel basis under the condition that the left and right relationships are the same or the same point on the screen, and stereoscopic data is created based on this.

According to a fifteenth aspect of the present invention, in the three-dimensional data producing apparatus according to the thirteenth aspect, the generating means further comprises a one-to-one image area corresponding to each of the two screens. That the corresponding pixels on the boundary line of the image area corresponding one-to-one are located on the same plane as the viewpoints of the plurality of screens, and at two points located on the same plane, the right and left On the other hand, the positional relationship is to specify a corresponding relationship in pixel units on the condition that the same right and left relationship is also present on the other screen, and to create a part of the stereoscopic data of the photographing target.

According to a sixteenth aspect of the present invention, there is provided the three-dimensional data creating apparatus according to the fourteenth or fifteenth aspect, wherein the generating means further comprises: Based on the location information of the point and the map information of the external space, a plurality of images obtained by shooting the same shooting target space are selected, and three or more approximate coordinates or a basic composition in the shooting target external space are obtained, and a plurality of screens are obtained. The position of the viewpoint is placed in the same coordinate system.

According to a seventeenth aspect of the present invention, in the three-dimensional data creating apparatus according to the fifteenth or sixteenth aspect, the generating means further comprises: For pixels arranged on a straight line on each of the two screens, which are located on the same plane as the viewpoint of the two screens, have the saturation, the maximum value, the minimum value, or a unique value of the color data of the color data On the condition that the order in which the pixels are arranged is the same in each screen, the correspondence in pixel units is performed.

The three-dimensional data creation device according to the eighteenth aspect of the present invention is the three-dimensional data creation device according to the fifteenth or sixteenth aspect, wherein the generating means further includes three or more three-dimensional data arranged on the same coordinate system. In the case where the pixels are associated using the screens, the pixels associated with any two screens and the pixels associated with the screens other than the two screens are different from the viewpoints of the screens other than the two screens, each of the two screens. Is used to determine the positions of the pixels on the screen other than the two by utilizing the respective viewpoints and being on the same plane as the associated pixels, and determining whether the associated pixels are good or bad based on the color data. Things.

A three-dimensional data creation system according to a nineteenth aspect of the present invention is connected to at least one terminal and the terminal via a communication network, and at least one of the first, fourth, fifth, sixth, seventh, and eighth aspects. Or a server for performing stereoscopic data creation processing using the stereoscopic data creation method according to 9 above, wherein the terminal is configured to input a plurality of still image data in a space whose spatial structure is unknown, and that the input spatial structure is Means for transmitting a plurality of still image data in an unknown space to a server via a communication network, receiving data and information transmitted from the server, and displaying or outputting the received data, the server comprising: Means for receiving a plurality of still image data in a space whose space structure is unknown transmitted from the terminal, and from the received plurality of still image data, three-dimensional data of a shooting target space and Means for creating at least one of a plurality of screens composed of a plurality of still images placed in one coordinate system and position information of a viewpoint, and means for transmitting the created three-dimensional data and position information to a terminal and displaying or outputting the information from the terminal And

A three-dimensional data creation system according to a twentieth aspect of the present invention is connected to at least one terminal and the terminal via a communication network, and at least one of the first, fourth, fifth, sixth, seventh, and eighth aspects. Or a server for performing stereoscopic data creation processing using the stereoscopic data creation method according to 9 above, wherein the terminal is configured to input a plurality of still image data in a space whose spatial structure is unknown, and that the input spatial structure is Means for transmitting a plurality of still image data in an unknown space to a server via a communication network, receiving data and information transmitted from the server, and displaying or outputting the received data, the server comprising: Means for receiving a plurality of still image data in a space whose space structure is unknown transmitted from the terminal, and from the received plurality of still image data, three-dimensional data of a shooting target space and Means for creating position information of a plurality of screens and viewpoints composed of a plurality of still images placed in one coordinate system, and means for creating image data based on the created stereoscopic data of the shooting target space and the position information of the viewpoint. Means for transmitting the created image data to the terminal and displaying or outputting the data from the terminal.

A three-dimensional data creation system according to a twenty-first aspect of the present invention is connected to at least one terminal via a communication network, and at least one of the first, fourth, fifth, sixth, seventh, and eighth aspects. Or a server for performing stereoscopic data creation processing using the stereoscopic data creation method described in 9, wherein the terminal inputs a plurality of still image data in a space whose spatial structure is unknown and position information of a viewpoint, Means for transmitting a plurality of still image data and the position information of the viewpoint in a space whose unknown space structure is unknown to a server via a communication network, receiving the data and information transmitted from the server, and displaying the received data or Means for outputting, and the server receives a plurality of still image data in a space whose space structure is unknown and position information of the viewpoint transmitted from the terminal, Means for creating three-dimensional data of the imaging target space from the number of still image data, means for creating image data based on the created three-dimensional data of the imaging target space, and viewpoint information transmitted from the terminal; Means for transmitting the generated image data to the terminal and displaying or outputting the image data from the terminal.

[0026]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 A method and an apparatus for creating three-dimensional data of an object to be photographed by simply running one vertical (or horizontal) slit light source in the normal direction will be described. A plurality of screens of the same object photographed from different positions and viewpoints of the photographed images are arranged in the same coordinate system. Regarding this method, a “3D data creation method” and a plurality of methods are described in the present invention. An arbitrary point on the object appears on each screen at the intersection of the line connecting the point on the object to be photographed and the viewpoint at which the screen was photographed. In other words, the actual three-dimensional coordinates of one point appearing on one screen are
It is on the extension of the straight line connecting the point on the screen and the viewpoint, and similarly on the extension of the straight line connecting the point on the other screen and the viewpoint. Therefore, an intersection in a three-dimensional space on an extension of a straight line connecting two or more screens and a viewpoint of the screen is a three-dimensional coordinate of a point to be obtained.

When an arbitrary line (a straight line or a curve in a three-dimensional space) on the same object is considered in the same manner, a plane (a set of straight lines connecting each point on the line appearing on one screen and the viewpoint) is used. On a plane or a curved surface), and at the same time on a surface (a flat or curved surface) composed of a set of straight lines connecting each point on a line appearing on another screen and the viewpoint. Therefore, it can be said that the coordinates of the line on the object in the three-dimensional space are a straight line or a curve formed by the intersection of the two surfaces.

One slit light source illuminating the object to be photographed appears as a continuous or discontinuous straight line or curve in the space to be photographed. In other words, it can be said that the surface of the imaging target irradiated with the light irradiated on the imaging target is on a plane or a curved surface including all the sets of the straight lines connecting the pixels constituting the lines projected on the respective screens and the viewpoints. In the following, a hypothetical line or curve connecting continuous pixels is assumed,
A method using a curved surface connecting each point on the polygonal line or curve and the viewpoint will be described.

The coordinates of an object point to be photographed by an arbitrary point on a line appearing on one screen are formed by connecting the viewpoint with the line on the other screen, which is an extension of a straight line connecting the point on the screen and the viewpoint. Is the intersection with the surface When the same operation is repeated for each pixel on one line on the screen, coordinates in a three-dimensional space can be obtained by the number of points. Similarly, the same operation is performed for each point on the line on the other screen to obtain coordinates in a three-dimensional space.

These two sets of coordinates in the three-dimensional space are both on a straight line or a curve in the three-dimensional space obtained by the intersection of a plane or a curved surface connecting the line on each image and the viewpoint. Accordingly, by joining the two points in the three-dimensional space obtained by the previous operation and connecting them to one another to form a single line, the coordinates of the surface of the imaging target hit by the desired slit light source can be obtained. . When all the lines projected by one screen are not projected by the other screen, the start and end of both planes do not necessarily match. In other words, where one plane and the other plane do not intersect, it is a part that has been projected on only one camera, so that three-dimensional data cannot be naturally obtained.

This operation is performed in the vertical (or horizontal) direction.
By running the slit light source in the normal direction and obtaining a large number of three-dimensional coordinates on the subject from multiple image sets,
It is possible to obtain three-dimensional data of the entire surface of the imaging target.
When the slit light source passes through the viewpoints Sa and Sb and is placed on the Z axis in FIG. 3, the images projected on both the screens A and B are both on the plane Y = αX. The plane connecting the image and the viewpoint always coincides with the plane connecting the image of the screen B and the viewpoint and does not intersect in the three-dimensional space.

Therefore, the above-mentioned analysis becomes impossible. Even if the slit light source is not placed on the Z axis,
When illuminating horizontally in a direction parallel to the axis, three-dimensional data can theoretically be obtained by crossing the planes, but the measurement error increases due to the limited number of pixels. Therefore,
It is desirable that the slit light source emits light in the vertical direction perpendicular to the Z axis.

In connecting each pixel on the screen with a line, a quadratic curve, a cubic curve, an arc, or the like passing through each pixel can be approximately applied to create a curve that is considered to have higher accuracy. For example, repeating a process of calculating a cubic curve passing through four adjacent points using four adjacent points, setting a curve between two intermediate points, and calculating a curve between adjacent pixels similarly by shifting one point. A method is conceivable. In such a case, assuming that there are infinitely many pixels on the curve thus created, by following the above method,
Obtain three-dimensional data of a location on the object where the light hits. By using such a method, it is possible to reduce errors caused by pixel roughness.

In the "3D data creation method", when creating 3D data of an arbitrary target object, a slit light source is placed between two cameras, and light is run from the slit light source in the vertical and horizontal directions. Then, time-series data (spatial code) of light that does not hit each pixel unit on the screen is collated to identify the same point. If the concept of collating lines on the screen is used, the collation can be performed only in one of the vertical and horizontal directions of the slit light source.

The above-described concept of collating lines on the screen can be grasped in a similar manner even if the lines form a closed ring when no light is irradiated. The line that forms the closed ring on the target object is displayed as a closed ring on the image of the captured object even if it is deformed by the angle at which the image is taken, as long as the entire line is imaged. If the closed ring on the image is divided into two curves on the left and right at the point where the α value in FIG. 3 is the maximum and the point where the α value is the minimum, the object is a ring existing on the surface of the object. As long as the images are not transparent, even if the images are taken from different angles, the right side is an image of the same part on both the A and B screens, and the left side is the same part on the left side. Therefore,
If the coordinates of a point indicating the closed ring on the actual object are obtained from the collation for each of the right and left lines, three-dimensional space data of the entire closed ring including the left and right sides can be obtained. In the above description, ordinary light is used as the slit light source, but any light source, electromagnetic wave, or the like may be used as long as it is suitable for measurement such as infrared light.

Embodiment 2 In an actual photographed image of a digital photograph, the color information of each pixel can take various values of 16.770,000 colors. When the same object is photographed from different angles by a plurality of cameras, if the color information on the image of each corresponding point on the object is the same, and if each pixel takes a unique value, Since it is possible to easily check which pixel on one image corresponds to which pixel on the other image, three-dimensional data of the object can be easily created. It should be noted that, in a plurality of screens, pixels that project the same point on the surface of the imaging target are described as corresponding in the present invention.

However, it is generally considered that such a state does not often occur. One is that the number of pixels is finite, so that the surface area of the imaging target represented by one pixel on one camera image is shifted from the surface area of the imaging target represented by the other corresponding pixel, , A situation occurs in which the color data of the two pixels that should correspond to each other is different. Also, in actual shooting, the shooting target naturally reflects light, but this reflectance differs depending on the shooting direction, and differs even on the images of cameras with different angles, even though the same place is shot. It may be recorded as color information. Further, there is a possibility that a difference in color data due to a habit of each camera and dust of a lens may occur.

A system for semi-automatically creating three-dimensional data of an object to be photographed from an image of the same object without illuminating a light source or the like will be described. It is theoretically impossible to always uniquely create three-dimensional data of a shooting target from a plurality of normal photographic images without illuminating a light source or the like. Here are some examples that are not possible: When shooting an object with a spherical shape, a circle is shown on both the A and B screens, but even when the object is a flat ellipse, the same circular image is displayed on both the A and B screens. In some cases, the same result can be obtained when the flat ellipse is slightly expanded or depressed. In this way, there are many shapes that can visually evoke the illusion, and theoretically A, B
It can be easily proved that it is impossible to uniquely determine the three-dimensional shape of the photographing target from only the information of the images on both screens.

Also, due to differences in pixel roughness, light reflectivity, and errors in the positions of the screens and viewpoints, the color data of the pixels on both the A and B screens representing the same object differ, making it difficult to make one-to-one correspondence. In many cases. Subtle habits when sensing color data for each camera can also be a problem. When creating three-dimensional data of an object having almost the same color or a fine pattern of the same color, a subtle change in color in a plurality of images is collated, and each pixel is represented by a correspondence relationship between pixels in both images. The three-dimensional coordinates of the point will be obtained. However, in the process of collation, the subtle changes captured on multiple screens show many fine irregularities on the surface of the subject, compared to the roughness of the pixels, the reflectance of light, and the density of the pixels. There are cases where the same color is displayed as a different color in both images, or there is a color that can be seen only from one side, and it is not possible to find an exact correspondence, or the degree of color change Is extremely small,
It is assumed that accurate matching in pixel units is impossible.
It is also conceivable that an image that makes an illusion of the same on both the A and B screens is captured while capturing completely different parts. In such a case, it is theoretically impossible to find the one-to-one correspondence in the pixel unit.

When projecting an external space or an indoor space, finely textured walls, sofas with finely woven patterns, etc.
Although it seems that there are many such photographing objects, in the present invention, the constraints when matching the pixels and obtaining the correspondence are adjusted, and the difference in the color data when the correspondence is made on a pixel basis is reduced as a whole. A method of creating three-dimensional data of a shooting target from two or three images while reducing a measurement error by obtaining such a correspondence will be described.

In the first stage, screens A, B, and A, in which the same object space or object is photographed from different angles.
The viewpoint (C) and the basic composition of the target space are determined, and the screens A, B, and (C) are placed in the space of the same coordinate system (FIG. 3).
Here, what is referred to as the basic composition is the coordinates of the reference frame that joins the two rectangular frames at a certain angle as referred to in the "stereoscopic data creation method", the mutual wall to guide the vanishing point, the ceiling, Points and lines that determine the positional relationship between the imaging target, the image, and the viewpoint, such as a boundary line with a floor or the like, or three points whose coordinates are determined, or eight points whose correspondences are known. As shown in FIG. 3, a straight line passing through the viewpoints Sa and Sb of the screens A and B is set as a Z axis, an X axis is set in the direction of the imaging target space, and a Y axis is set perpendicularly to both axes. The coordinates of each pixel on the B screen are converted to this newly set coordinate system (hereinafter simply referred to as the same coordinate system). Here, the coordinates of Sa and Sb are (0, 0, Zsa) and (0, 0, Zsb), respectively. In setting the coordinate system, the position of the origin at which Z = 0 is arbitrary.

A plane Y = αX (α) passing through the Z axis of the same coordinate system
Is arbitrarily set), and a line where the plane Y = αX intersects the screens A and B, lineA and lineB, respectively, is obtained. The three-dimensional coordinates of the point on the object to be photographed represented by the pixel on lineA on the screen A are on an extension line passing through the viewpoint Sa and the pixel.
That is, it can be said that the plane Y = αX. The position of the pixel on the screen B at which a point on the same object is photographed must also be on this plane Y = αX. In other words, when comparing each pixel or each area of the screens A and B, it can be said that a pixel on the screen B corresponding to an arbitrary point on the line A is on the line B and cannot be any other pixel. .

The problem of obtaining three-dimensional data of the object space or object from screens A and B in which the same object space or object is imaged from different angles is as follows.
The correspondence between each pixel on the screen A and each pixel on the screen B is obtained by collating points on the lines lineA and lineB where the plane Y = αX and the screens A and B intersect. , An intersection on the extension line connecting each viewpoint and the pixel is determined, and the intersection is used as space coordinates of the object, and the set is used as the stereoscopic data of the imaging space or the imaging object.

As described above, each pixel on lineA and lin
It is theoretically impossible to always uniquely correspond each pixel on eB. However, if a certain constraint element is added using the similarity of the color data of a pixel and the relationship with surrounding pixels, li
It is possible to construct a method for associating points (pixels) on neA with points (pixels) on lineB at a sufficiently practical level, and image areas with exactly the same color data are usually manufactured as industrial products. Therefore, the position can be estimated by a plane or a simple curved surface.

In the second stage, both the A and B images are divided into a plurality of image areas (hereinafter simply referred to as areas) depending on the difference in color data. The color data of each pixel is composed of saturation and lightness. The difference between the color data of the two pixels is based on an appropriate hue space to create and give an appropriate formula indicating the distance or other degree of difference. For example, the degree of difference in color data = (the amount of difference in saturation between two pixels) +
(The amount of difference in lightness between two pixels) or an expression in which saturation is emphasized and only the saturation portion of the above expression is increased by a factor of 5.

As described above, pixels on a plurality of screens representing the same point do not always have completely the same color data, so that a certain difference must be allowed. If the numerical value calculated by the formula arbitrarily determined as described above is within a predetermined arbitrarily determined range, this is regarded as the same color data as an allowable range. Hereinafter, the description “same (same) color data” means that the color data is within the allowable range, and is used separately from “completely identical color data”.

As a method of dividing into regions, a portion where the color data changes by a certain degree or more is checked, and this is divided into regions by using this as a boundary line. Even for the same color, it is expected that there will be some change in the color data depending on the viewing angle, so the degree of this change is set to some extent in this second stage. In this case, if one part has a pattern, it is divided for each pattern. However, it is expected that it will be difficult to make one-to-one correspondence on both A and B screens due to pixel roughness, such as when the pattern is fine, so the number of divided areas is less than a certain number (usually In the case where only the number of pixels is within the same range, and there are minute areas of the same color data in the vicinity thereof, they are not considered to be a single area, but are combined with the area between them. Area. That is, it is assumed that this is a part of the pattern. A unique number is assigned to each area set in this way.

For all the pixels n (Xn, Yn, Zn), the α value of the plane Y = αX where they are located = Yn / Xn is given. Each pixel n is stored in the pixel file of the computer in a form having the following data. For each pixel n of the A screen,
B that may correspond one-to-one with the area to which the pixel belongs
The number of the area of the screen is assigned to each pixel n of the B screen, the number of the area of the A screen which may correspond one-to-one with the area to which the pixel belongs. (At this stage, it is still blank.) Data format of pixel n: (pixel number, screen number to which it belongs, X
Coordinates, Y coordinates, Z coordinates, area number, determination of whether or not the area is on a boundary line, α value of coordinates, color data (saturation, lightness) The other screen where the area to which the pixel belongs may correspond one-to-one. (A plurality of area numbers above, and a plurality of pixel numbers on the other screen that the pixel may correspond to) The image division method conforms to the data format described in this section in addition to the method described in the second embodiment. If you set the data format to
Any method may be used.

In the third stage, the one-to-one correspondence between the areas set on the screens A and B is obtained. As long as the same area of the same object to be photographed can be simultaneously viewed from both the viewpoints Sa and Sb, the same area is displayed on either of the screens A and B as one of the areas. At this time, the criteria for judging the same are as follows.

B) Among the pixels belonging to both regions, there are pixels in the range of the common α value. B) Pixels of the same color data exist. C) When the pixel n on the screen A is located outside the plane passing from the viewpoint Sa in parallel with the plane of the screen B, as shown in FIG. And a line drawn from the viewpoint Sb in parallel with the line connecting the viewpoint Sa and the pixel n, and must lie inside the intersection of lineB. Therefore, the existence area on the B screen of the imaging target part represented by a part on the A screen is calculated as the possible area in the B image for each of the innermost pixels in each α value of the part on the A screen. Screen area. The above three points are necessary conditions.

Furthermore, if the following conditions are satisfied, there is a high possibility that one-to-one correspondence is established. 1) The maximum value of the α value of the pixel in each area is the same. (It is within the error range.) 2) The minimum value of the α value of the pixel in each area is the same. (It is in the error range.) 3) The ratio of the range of the common α value is large. Visually explaining these conditions 1) to 3), Y = α
As α of X is gradually reduced from infinity to negative infinity, the plane Y = αX rotates about the Z-axis and gradually descends from above the object to be photographed. When the plane Y = αX touches a part of the object,
It is a point that can be seen from both Sa and viewpoint Sb at the same time, and usually there is only one point. Therefore, the maximum value and the minimum value of the α value of the same region on both the A and B screens coincide unless the image is the shadow of another region if the region is the same. If this is only one point, the pixels on both screens correspond one-to-one. If there is more than one, they represent planes located on the plane Y = αX, and the upper part of the object has a shape cut off by the plane.
The same applies to a case where there are a plurality of minimum value points.

When the target portion has a shape having a protruding portion, the α value at one point which is the local maximum value (minimum value) is the same in both the A and B images. In addition, candidates for one-to-one correspondence of parts can be further narrowed down. However, if the part with the maximum (minimum) α value of the target part cannot be seen from any viewpoint as a shadow of another part,
Since the maximum value and the minimum value do not match, it cannot be determined that the images do not represent the same part just because the maximum value and the minimum value do not match. 4) The range and distribution state of the color data of the pixels included in the one-to-one corresponding area substantially match. In the part determined to have the pattern, the combination of the range of the color data and the state of the distribution substantially match. An expression for evaluating this state is arbitrarily created. 5) Each area of the A and B screens is adjacent to the other one-to-one image area in the same direction (left and right) in both the A and B screens. 6) Each area of both A and B screens is in the same direction (left and right) in both A and B screens with respect to the other one-to-one image areas.

If another part is located behind or in front of a certain part, the corresponding image area may be reversed on both the A and B screens. Therefore, just because the left and right are reversed does not necessarily mean that they are not the same part. However, if it is located at the back, there is a possibility that the portion projected to both the A and B screens is small. Satisfies the requirements of a) to c), 1),
Since a plurality of one-to-one correspondences satisfying the condition of 2) can be found, the degree of satisfying the conditions of 1) to 6) is evaluated based on them, by an appropriate method, and the one-to-one correspondence is evaluated in descending order of evaluation. With this one-to-one correspondence as a given condition, the evaluation is performed again to find the next set of one-to-one correspondence areas, and by repeating this, the one-to-one correspondence with each image area of both A and B screens Relationships can be defined. Furthermore, a method may be used in which a plurality of evaluation methods are prepared and a combination in which the correspondence of the largest number of regions in the one-to-one correspondence determined by each of the evaluation methods can be determined.

When images taken from three or more locations are used, the following relationship is established for each of three screens A, B, and C placed in the same space. The viewpoints of the three screens are set to Sa, Sb, and Sc, respectively. Pixels on the screen B corresponding to an arbitrary pixel na (assuming α value = αa) on the screen A are the viewpoints Sa and S
In the same coordinate system (X, Y, Z) using the straight line connecting b as the Z axis, the pixels on the same straight line of Y = αc * X, and the corresponding pixels on the screen C are the viewpoints Sa and Sc In the same coordinate system (X ', Y', Z ') with the straight line connecting
It is on the straight line * X '. The pixels of the screen B on the straight line where Y = αc * X and the pixels of the screen C on the straight line where Y ′ = αb * X ′ have a line connecting the viewpoints Sb and Sc as the Z ″ axis. In the same coordinate system (X ", Y", Z "), they must exist on the same plane where Y" = αa * X ".

The same can be said for the one-to-one correspondence of the image area which is a set of pixels.
Is satisfied in each two of the three images, it can be said that this one-to-one correspondence has been almost certainly confirmed.

Based on the above conditions, each area on both the A and B screens has a one-to-one correspondence. It is highly probable that a part that is not one-to-one correspondence is a part that is displayed only in one image or a part that has a plurality of candidates that correspond one-to-one and is not fully squeezed. Display and confirm the correspondence between each area of both A and B screens on the monitor, and if it can be visually judged that there is another area that has one-to-one correspondence, enter it and enter section 0049 The correction is performed by additionally performing the processes of the items 0055 to 0055.

In the fourth stage, a correspondence relationship between pixels included in a set of areas corresponding one-to-one on both the A and B screens is obtained. This process proceeds from a region where the left and right positional relationship matches many other regions. After creating three-dimensional data with priority given to this, if matching is performed on a pixel-by-pixel basis in an area where the left and right are reversed, a corresponding combination of pixels located in an invisible area that is a shadow of another part will be obtained. Excluded and can be calculated.

In a special case in which pixels of exactly the same color data form a continuous area, there is no way to exactly correspond those pixels. In this case, the area is set as a separate area, and three-dimensional data is derived from pixels on the boundary line of the relevant area in both the A and B screens in accordance with the term 0035. A curved surface created by connecting the straight lines is set as target three-dimensional data of the area.

In the fifth stage, the correspondence of each pixel is determined under the following conditions. In the following description, the α-axis is taken in the vertical direction of the screen, and β moves to the right on the same α-value line.
Assume an anomalous coordinate system with a β axis with a higher value. The color data of the one-to-one corresponding pixels must be substantially the same. As described above, it is expected that the color data of the corresponding pixel is slightly different due to the density of the pixel and the like, so that a difference within a certain range is regarded as the same color data. The alpha value of both pixels must be the same, thus
The α value of the B screen pixel corresponding to the A screen pixel must be within an error range caused by the α value of the A screen pixel and the pixel density. Note that a range of an error larger than the following equation may be allowed. If the pixel α value of the A screen is tan θ1, the α value of the pixel immediately above the pixel is tan θ2, and the α value of the pixel immediately below the pixel is tan θ0, tan [(θ1 + θ2) / 2] ≧ (pixel of the B screen α value) ≧ tan [(θ1
+ Θ0) / 2] The same applies to pixels on screen A corresponding to pixels on screen B.

When the pixel n on the screen A is located outside the plane passing from the viewpoint Sa in parallel with the plane of the screen B, FIG.
As shown in the figure, the corresponding pixel of the B image exists on line B having the same α value and inside the intersection of the line B and the line drawn from the viewpoint Sb in parallel with the line connecting the viewpoint Sa and the pixel n. Have to do. The same applies to the pixels on the screen A. Multiple pixels on one screen may correspond to a single pixel on the other screen, but one of the multiple pixels may also simultaneously correspond to another pixel on the other screen. I can't get it.

On the same plane Y = αX, for a pair of corresponding pixels nak and nbp and naj and nbq representing two arbitrary points simultaneously viewed from both viewpoints, if β value of nak <β value of naj For example, β value of nbp ≦ β value of nbq, β value of nbp <β of nbq
If it is a value, the relationship of β value of nak ≦ β value of naj is established. Any pixel naj on one screen and adjacent pixel n immediately above it
In the pixels nbp, nbq, and nbr on the other screen to which ak and the immediately adjacent pixels nal correspond, respectively, the pixel nbp is on a continuous plane or curved surface, so that the pixel na as shown in FIG.
At least k corresponds to either a pixel in the range of a rectangle having the same α value with the pixel nbq corresponding to the pixel nnal and the pixel nbr corresponding to the pixel nal as a diagonal, or a pixel located on both sides thereof. Based on these six constraints, the correspondence between pixels on both the A and B screens is determined. (FIG. 4)

At the stage where the pixel comparison has been completed for all the one-to-one correspondence regions, the condition that the pixels in the region which did not correspond one-to-one, the remaining regions are grouped and the correspondence in the invisible region is impossible. Addition from 0059 to 0
The process in section 061 is performed again to confirm the possibility of the one-to-one correspondence.

If it is assumed that the color data of the pixels to be handled is different from each other due to limitations on the precision of the image, etc., strictly speaking, the method of obtaining the one-to-one correspondence by combining the above six elements is univocal. It is not fixed. Therefore, by adding some conditions, an optimal combination (correspondence relationship in pixel units) in which the number of corresponding pixels is the largest and the total amount of difference in color data of the corresponding pixels is the smallest is obtained.
How to use these two evaluations is arbitrary. Since the number of pixels included in each region is finite, all combinations of pixels satisfying the above conditions are calculated using a high-speed computer, and the combination in which the number of corresponding pixels is the largest, color data of the corresponding pixels is calculated. It is possible to determine the combination or the like that minimizes the sum of the differences. However, since this requires a large amount of calculation, a simpler method is described in the embodiment.

In the sixth stage, three-dimensional data to be photographed is obtained. Once a set of corresponding pixels is determined in pixel units on both A and B screens, the intersection of the extension of a straight line connecting each corresponding pixel and the viewpoint is the target solid represented by the corresponding one-to-one set of pixels. Coordinates in space. Since the pixels are finite, it is highly likely that these two straight lines pass each other delicately.However, the closest point on the straight line extended from the viewpoint Sa is obtained as a solid point, and this solid point is used as the A screen. The color data of the corresponding pixel is given. Similarly, a solid point is obtained from a pixel on the B screen. The method to find the closest point is "3D data creation method"
As described in, an arbitrary point on each straight line is represented by an expression having one variable, and the distance between the two points is obtained by an expression having the two variables. The closest approach point can be obtained under the condition that both the values of the above equations become 0. The set of points in the three-dimensional space obtained in this way is estimated to be the stereoscopic data of the photographed target.

As another method, an arbitrary pixel nak on one screen A corresponds to a pixel nbp on the other screen B and four pixels nbq1, nbq2, nbq3, and nbq4 adjacent thereto on the screen.
On the same line as the α value of pixel nak, these four pixels and pixel nbp
From the color data, the position closest to the color data of pixel nak is obtained by proportional proportional division, and that point is set as the coordinates of the corresponding virtual pixel on screen B, and the straight line and the pixel connecting the virtual pixel and viewpoint Sb
If a method is used in which a point that intersects on an extension of a straight line connecting the nak and the viewpoint Sa is used as the coordinates of the three-dimensional data and the color data of the pixel nak is used, more accurate three-dimensional data can be created. The same operation is performed for an arbitrary pixel nbp on the screen B. As described above, three-dimensional data of a shooting target is obtained from two photographs of the same target. The method of determining the position on the same α-value line by proportional division using color data is nak, n
An appropriate arbitrary expression based on the color data of bq1 to bq4 is created and used. Apart from the above method, a three-dimensional shape may be generated and texture mapping may be performed based on a known porting process as disclosed in Japanese Patent Application Laid-Open No. H11-96374.

When it is determined from the obtained three-dimensional data that the boundary line of the same part can be seen in both the A and B images, that is, an obstacle that blocks the line of sight between the boundary line and the viewpoints Sa and Sb. When it is determined that there is no three-dimensional data, the three-dimensional data is created by the same method as that for obtaining three-dimensional data using one slit light source described above, and the three-dimensional data obtained here is obtained. , And more accurate three-dimensional data can be obtained. Further, the three-dimensional data can be corrected using the three-dimensional data obtained thereby. That is, by giving this as a given value and performing the process of the embodiment again, three-dimensional data with higher accuracy is created.

When there are three or more images of the same subject, the operations of the above-mentioned items 0049 to 0061 are performed in parallel for each two images, and the calculation results are appropriately compared in the middle stage. Thus, the accuracy of the system can be further improved. When the above operation is performed in parallel for each set of two screens when there are three screens, the α value of one point on the screen of the surface to be captured becomes Is the same as Therefore, when setting a one-to-one correspondence in a region unit or setting a correspondence relationship in a pixel unit, the same α is set.
The values of lineA and lineB in FIG. 3 as well as lineC similarly differ for each combination of screens. Therefore, since the regions and pixels to be compared are different, the results calculated on the respective screens are compared and examined, and except for the correspondences that match, it is possible to create more accurate three-dimensional data.

As described above, in both the A and B screens, there is a possibility that a deviation due to an error in setting the viewpoint and the screen, and a measurement error of color data due to a camera habit or an illumination angle may occur. . When each screen is divided into a large number of regions and collated, the maximum and minimum parts of the α value are collated, but usually there are many sets of regions where this α value should match, so the screen Can be corrected. When a statistically significant number of pixels having the same α value is obtained and a tendency common to the pixel set is observed, that is, in each case, α If it can be determined at a statistically significant level that the value is low in common with α of the other screen, or the saturation and brightness of the color data are low in common, only the average value of the screen By increasing the height or correcting the saturation and brightness of each pixel on the screen, more accurate matching between pixels can be performed.

If the continuity of the area is guaranteed, for example, when the process is repeated based on the input from the operator or the result obtained once, it is considered that there is a one-to-one correspondence.
By comparing (after correcting chromatic aberration) the pixels at both ends on the same Y = α X-ray of both the A and B screens, the same (Y = α)
It is also possible to correct the hue of the pixel (on the X-ray) and to determine a subtle change in the hue more accurately.

Embodiment 3 Next, a system will be described in which a plurality of points within a range of 3 to 8 are collated from a plurality of images arbitrarily photographed in the same photographing space to determine the viewpoint of the camera, and create stereoscopic data of the target space. In the “3D data creation method”, the coordinates of a part of the target space from the still image and the positional relationship between the image and the viewpoint are placed in the same coordinate system, and as a method for determining the relative relationship, the wall surface, floor surface, or reference Method using a frame or the like, which constitutes a parallelogram 4
A method using point coordinates and a method using three points whose coordinates in a three-dimensional space are known are shown. Using these methods, a fine or approximate basic composition of a plurality of screens can be determined and placed in the same coordinate system. In this case, for example, the basic composition of one image uses a wall surface, a floor, etc.
Using the coordinates determined there, the basic composition may be obtained from the coordinates of three points in the other image. Further, a rectangle such as a window frame may be commonly used.

When the coordinates of the object point represented by the three pixels on one screen are known, the coordinates are represented by m (Xm, Ym,
Hm) Set m = 1,2,3. Given the size of the screen on which this is projected, the variables that determine the position of the screen and the viewpoint are the coordinates of the center of the screen (Xo, Yo, Ho), the screen tilt angles θx, θy, θh, and the focal length L. One. Coordinates (Xm ', Ym', Hm ') of the three points (Vm, Zm) m = 1, 2, 3 on the screen on which these three points are projected, and coordinates of the viewpoint in the same coordinate system (Xs, Ys, Hs) is a function (Fxm, Fym, Fhm) of these seven variables, m = 1, 2,
3 and (Fxs, Fys, Fhs). On the other hand, an expression indicating that each pixel is located at the projected position of the imaging target point on this screen is expressed by the following three sets of three simultaneous equations.

[0072]

(Equation 1)

The three simultaneous equations can be rewritten into the following three sets of two simultaneous equations, and a total of six equations are obtained. (Xm-Fxs) * (Fym-Fys) = (Ym-Fys) * (F
xm-Fxs) (Xm-Fxs) * (Fhm-Fhs) = (Hm-Fhs) * (F
xm-Fxs) A solution cannot be obtained with only 6 equations for 7 variables, and another condition needs to be added.

In the "stereoscopic data creation method", in the case of an aerial photograph or the like, a method of using three points whose three-dimensional coordinates are known in an image and placing the three points, the image plane, and the viewpoint in the same coordinate system. Has invented. Hereafter, it is quoted and the explanation is supplemented. "Even when the height difference of the ground surface is large and a parallelogram point cannot be extracted as a plane, three points with known positions and elevations on the ground surface are known, and if that point can be identified on the photograph, By converting the image to the ground surface screen (with the same coordinate system as the above, and the coordinates of each pixel in the same coordinate system) according to the method, and comparing the ground surface screen from multiple viewpoints (with the same coordinate system), Can be determined accurately.

In this case, the line of sight of the image is naturally inclined downward, but the left and right sides of the image at that time must be kept horizontal. That is, it is desirable that the camera has a function of taking pictures while keeping both ends of the camera horizontal or dropping a horizontal line in an image. In the latter case, the calculation is performed after rotating the image horizontally about the center of the image. ... The center point of the image is set to (V0, Z0). The line running up and down from the center of the image is assumed to be V = 0, and the projected line on the ground surface is defined as Y on the ground surface.
X = 0 is given as an axis, and the X and Y coordinates of the ground surface are converted. Hereinafter, point D in FIG. 4 is ignored. Note: Here, "keeping the image horizontal" means that, from the viewpoint, the Z axis of the image and the X axis of the ground surface appear completely coincident, and the V
The axis is located parallel to the plane of the ground surface where H = 0.

The coordinates of the three points on the resulting image are represented by A (Va,
Za), B (Vb, Zb), C (Vc, Zc), and the coordinates of the ground surface are A '(Xa, Ya, Ha), B' (Xb, Yb, Hb), C '(Xc, Y
c, Hc) and the coordinates of the viewpoint are (Xs, Ys, Hs). ...
In the following, in the case of obtaining from three points, (FIG. 6) (FIG. 4 in the present patent application)
Xs, Zs, θ, Zq, Hs, and Ys are obtained by the following equations. Note: In FIG. 6 of “3D data creation method”, φ is indicated, but this is a calibration omission that should be θ as shown in FIG. Note that if a set of three points is selected such that any of the lines connecting the two points A, B, and C is positioned parallel to the screen, the following equation cannot be solved, and care must be taken.

Xs = 0, fa = Xa / Va, fb = Xb / Vb, fc = Xc / Vc,
F = Ya * (fb-fc) + Yb * (fc-fa) + Yc * (fa-fb), Zs = (Y
a * (fb * Zb-fc * Zc) + Yb * (fc * Zc-fa * Za) + Yc * (fa * Za-fb * Zb))
/ Fsinθ = F / [fa * Za * (fb-fc) + fb * Zb * (fc-fa) + fc * Zc * (fa
-fb)] G = Ha * (fb-fc) + Hb * (fc-fa) + Hc * (fa-fb), and Zq = (Ha * (fb * Zb-fc * Zc) + Hb * ( fc * Zc-fa * Za) + Hc * (fa * Za-fb
* Zb)) / G cosθ = G / [fa * Za * (fb-fc) + fb * Zb * (fc-fa) + fc * Zc * (fa-f
b)] Hs = Xa * (Zq-Za) * cosθ / Va + Ha, Ys = Ya-Xa * (Za-Zs) * sinθ /
Va "

In order to place the coordinates of the image and the coordinates of the ground surface in the same coordinate system, apart from the three points in the image, the V axis of the image and the X axis of the ground surface are located on the same plane, and the V axis is H = 0 must be placed parallel to the plane. Here, the cause of the error of the initial setting occurs. If the screen is not kept completely horizontal in the sense described in the previous section, the square of sinθ found in the previous section and cosθ
Do not add up to 1. As described above, even in the “three-dimensional data creation method”, an operation such as adjusting the center line of the screen to the corresponding position of the imaging target is added, and the solutions of the items 0071 to 0073 are obtained by adding some conditions such as the focal length.

However, in actual application, the focal length of the camera to be used is measured in advance, the center line of the screen is overlapped with the map, and the coordinates of the approximate image center line from the photographing target building are grasped. With the method, additional conditions can be added with relatively high accuracy, and an approximate solution close to the desired solution can be obtained. Such work and 0074 to 0077
By using a simple equation such as a term, it is possible to expect an effect such as obtaining a relatively accurate approximate solution and utilizing it by itself, and obtaining an initial value as described in the embodiment to shorten the calculation process. .

If two images sharing the three pixels whose coordinates of the photographing target point are known and the two points that are determined only to capture the same target point are used, the two screens and the viewpoint are the same. Can be placed in a coordinate system. The number of variables in the two screens and the position setting is 7
, One screen center coordinates (XOa, YOa, HOa), screen tilt angles θxa, θya, θha, focal length La, and the other screen center coordinates (XOb, YOb, HOb), screen tilt angle θxb,
θyb, θhb, focal length Lb, 14 in total. Three points (Vmp, Zmp) m = 1 on the screen for displaying three points of each image,
2, 3, p = a or b in the same coordinate system (Xm
p ′, Ymp ′, Hmp ′) and coordinates (Xsp, Ysp, Hsp) of the viewpoint in the same coordinate system are functions (Fxmp, Fymp, Fhmp) of these seven variables, m = 1, 2, 3, a or b, and (Fxsp, Fysp, Fhsp) p = a or b.

On the other hand, the conditional expression for obtaining the variable is 007
As with items 1 to 0073, there are 6 expressions each, for a total of 12 expressions. Further, another two sets of pixels that capture the same shooting target point in the two screens are on the same plane as the two viewpoints (Fxsp, Fysp, Fhsp) a or b. Holds.

[0082]

(Equation 2)

Solving the above-mentioned quaternary four simultaneous linear equations gives d1,
Two new conditional expressions composed of only 14 variables excluding d2, d3, and d4 are generated. Therefore, only the correspondence between the three pixels at which the coordinates of the shooting target point are known and the correspondence is known.
If two images sharing points are used, a total of 12 + 2 = 14 conditional expressions can be obtained for 14 variables. The solution exists because it is based on images obtained by capturing the same five coordinates. (There is a possibility that only an approximate solution is obtained based on an error caused by a finite number of pixels.)

As long as the correspondence between the eight pixels is determined on two screens obtained by shooting the same object, the two screens can be arranged in the same coordinate system even if the coordinates are unknown. Given the size of both screens and the position of one of the screens in the two images, the variables are the focal length La of that screen and the coordinates of the center of the other screen (XOb, YOb, HOb). ), The screen tilt angles θxb, θyb, θhb, and focal length Lb.
Three points (Vmp, Zmp) on the screen that show three points of each image m
= 1 to 8, p = a or b in the same coordinate system (Xm
p ′, Ymp ′, Hmp ′) and the coordinates (Xsp, Ysp, Hsp) of the viewpoint in the same coordinate system are functions (Fxmp, Fymp, Fhmp) of these eight variables, m = 1 to 8, p = A or b, and (Fxsp, Fysp, Fhsp) p = a or b.
The two corresponding pixel pairs are located on the same plane as the two viewpoints. Therefore, the following simultaneous equations hold at eight points m = 1 to 8.

[0085]

(Equation 3)

As in the items 0080 to 0083, this 4
The simultaneous equations are one function Fxma, Fyma, Fhma, Fx
mb, Fymb, Fhmb, Fxsa, Fysa, Fhsa, Fxsb, Fys
The relational expressions (conditional expressions) of b and Fhsb are summarized. Eight such conditional expressions are obtained, and the solution of the above eight variables is obtained by solving the eight simultaneous equations. As a result of obtaining these eight variables, it is possible to determine the arrangement of screens, that is, the relative positional relationship between two screens and two viewpoints. In the case of the same camera and a camera photographed with the same focal length, La = Lb and seven variables, so seven corresponding pixels on the screen may be specified. If the focal lengths of both cameras are known, six corresponding sets of pixels are identified.

When three images of three points whose coordinates are known are used, another set of pixels which can be determined to correspond to the three images in common is set, and a total of four pixels are set. The position of the entire screen and the viewpoint is determined by the same coordinate system using pixels. The number of variables that determine the position of the screen and viewpoint in the same coordinate system differs depending on various conditions, such as the number of images, whether the coordinates of the shooting target are known or unknown, whether the focal length is known, and whether the same camera is used. Come. Under any condition, a variable solution can be obtained by giving the same number of independent conditional expressions as the number of variables to be obtained.

The following is an example of the number of corresponding pixel sets required for each condition such as the number of images, whether the coordinates of the object to be photographed are known or unknown, whether the focal length is known, and whether the same camera is used. Is shown. When the coordinates of the object to be photographed are unknown and the focal length may be different for each image. When two images are used, the required number of sets of corresponding pixels is seven. When three or four images are used, it is supported. The required number of pairs of pixels to be used is five. When the number of images to be used is five or more, the required number of pairs of pixels is four. The image to be used when the coordinates of the shooting target are unknown and the focal length is the same for all images. When there are two images, the required number of corresponding pixels is six. When three or more images are used, the required number of corresponding pixels is four. When the coordinates of the imaging target are known, the required number of corresponding pixels The number is four sets. However, if the focal length is known, three sets.

By providing a set of pixels satisfying this condition, giving a relatively accurate initial setting, and providing a calculation process for gradually increasing the accuracy of the approximate solution, an approximate solution having the same level of accuracy as the pixel density can be obtained. Can be obtained. Regarding the accuracy issues of the camera itself, that is, the issues such as the misalignment of the installation angle of the lens and the image receiving surface, and the distortion of the image, five or more points whose coordinates are known from the points determined in advance in the industrial test are used. You can shoot and get the correction parameters. If more than eight pixels correspond to each other, or if a boundary line intersects a corner of a building, for example, a coordinate is obtained as an intersection of a straight line obtained by the least square method from the pixels constituting the boundary line. , It is also possible to reduce the error. This makes it possible to set a plurality of screens in the same coordinate system and to set the positions of the screens necessary for creating three-dimensional data by defining correspondences in pixel units as shown in the section 0042 and thereafter. In addition, pixels in which the line connecting the two shooting target points is considered not to be parallel to the line connecting the shooting points are selected so that the set of a plurality of pixels is not located on the same plane as the two viewpoints.

In each of the A and B screens, a set of corresponding pixels is obtained, and eight variables are obtained from the relative positional relationship of the coordinates. In all eight simultaneous equations, the corresponding pixels have the same plane Y = Αm * X (m = 1 to 8), but does not define the distance between the viewpoints Sa and Sb, so that a single solution is not determined. As a result, the viewpoint S
In the same coordinate system where the straight line passing through a and Sb is the Z axis, 8
The screen center point (Xa0, Y
a0, ZaO) (Xb0, Yb0, Zb0) and both viewpoints can also be solved by moving them in parallel in the Z-axis direction. The three-dimensional data created based on the combination of the different solutions is similar in the shape of the three-dimensional data, except for the relative size of the three-dimensional data.

Therefore, when creating stereoscopic data, the distance between the viewpoints Sa and Sb may be arbitrary. When combining with other three-dimensional data, for example, at any two points whose distances on the three-dimensional data are known, the viewpoint Sa is set at the same ratio as the ratio of the distance obtained from the coordinates of the two points and the distance based on the desired scale. ,
The distance of Sb (difference in Z coordinate) may be reduced or enlarged, the Z coordinate on the screen may be corrected, and the size of the three-dimensional data may be determined.

Assume that the size and position of one screen A and the size of the other screen B are given. Set the focal length of screen A to the variable La
Is given, the viewpoint Sa is determined. Here, the coordinates of the viewpoint Sa of the screen A are (0, 0, 0), and the coordinates of the center point of the screen A are (L
a, 0, 0), if the vertical and horizontal coordinate axes on the screen are placed parallel to the Y axis and the Z axis, the coordinates of each point on the screen A are determined.
(La, Yan, Zan) n = 1-8.

When the distance between the two viewpoints is an arbitrary constant LAB, and the values of the variable Xsb and the variable Ysb are respectively given as the X coordinate and the Y coordinate of the viewpoint Sb, the relative positional relationship between both the A and B screens and the object to be shot is obtained. (The positional relationship between the left and right) is known, so the Z of the viewpoint Sb is
The coordinates Zsb are also obtained. Zsb = √ (LAB ** 2-Xsb ** 2-Ysb ** 2) Given the variable Lb as the focal length of the screen B and the values of the variables Xb0 and Yb0 as the coordinates of the center point of the screen B, , B, since the relative positional relationship between the two screens is known, the coordinates Zb0
Is found. Zb0 = Zsb-√ [Lb ** 2- (Xb0-Xsb) ** 2- (Yb0-Ysb) ** 2)] When the screen tilt angle θb0 is given, the size of the other screen B is given. Since the screen is perpendicular to the line connecting the viewpoint and the center point of the screen, the coordinates of all pixels on both the A and B screens are determined in the same coordinate system. That is, the variables La, Xsb, Ysb, Lb,
The total coordinates of both the A and B screens can be represented by a total of seven, Xb0, Yb0, and θb0.

The coordinates of the corresponding pixel and viewpoint of both screens A and B are determined by a function of seven variables, that is, the coordinates (Fxan, Fxan,
Fyan, Fzan), coordinates of the pixel Pbn of the screen B (Fxbn, Fybn, Fz
bn), coordinates of viewpoint Sa of screen A (0, 0, 0), viewpoint S of screen B
The coordinates of b (Xsb, Ysb, Fzsb), Fzsb = Zsb.

Since the extended lines of the lines connecting the corresponding pixels on both the A and B screens and the viewpoints Sa and Sb intersect at the photographing target point, the following equation is established. γFxan = Xsb + ε * (Fxbn−Xsb) γFyan = Ysb + ε * (Fybn−Ysb) γFzan = Fzsb + ε * (Fzbn−Fzsb) where γ and ε are unknown constants. Thus, a relational expression of seven variables is obtained. Fxbn * (Ysb * Fzan−Fzsb * Fyan) + Fybn * (Fzsb * Fxan−Xsb * F
zan) + Fzbn * (Xsb * Fyan−Ysb * Fxan) = 0 Place the viewpoint Sa at the origin (0,0,0) and set the center point of the screen A to (L
a, 0, 0), you can put Fxan = La, Fyan = yan, Fzan = zan. Here, La is the focal length of the screen A, and yan and zan are the coordinates of the vertical axis and the horizontal axis of the pixel n on the screen A, respectively.

When both sides of the above equation are divided by Fzsb, La * (Fybn-Fzbn * Ysb / Fzsb) + yan * (Fzbn * Xsb / Fzsb-Fxbn) + zan * (Fxbn * Ysb-Fybn * Xsb) = 0 La * [(Fybn-Ysb)-(Fzbn-Fzsb) * Ysb / Fzsb)] + yan * [(Fzbn-Fzsb) * Xsb / Fzsb- (Fxbn-Xs b)] + zan * [(Fxbn-Xsb) * Ysb- (Fybn-Ysb) * Xsb)] = 0 The above formulas are variables La, Ysb / Fzsb, Xsb / Fzsb, (Xsb-Xb0),
(Ysb-Yb0), (Fzsb-Zb0), φ, a relational expression of a total of seven variables, independent of seven sets of yan and zan (n = 1 to 7). Here, (Xb0, Yb0, ZbO) is the center point of the screen B, and φ is the inclination of the screen B. If these seven variables are obtained and Fzsb is arbitrarily determined, seven variables La, Ysb, Xsb, Xb0, Yb0, Zb0, and φ are obtained.

In items 0084 to 1986, the corresponding 2
Fxsa = Fysa = F also in the quaternary simultaneous equation reflecting that the pixel, the viewpoint Sa, and the viewpoint Sb are located on the same plane.
If hsa = 0, that is, if the viewpoint Sa is placed at the origin, d4 = 0, and the same equations as the above relational equations are derived from the remaining three simultaneous equations. Therefore, the corresponding pixels and the viewpoints Sa,
It is confirmed that the condition that the extension line of the line connecting Sb intersects at the imaging target point is equivalent to that the corresponding two pixels and the four points of the viewpoint Sa and the viewpoint Sb are located on the same plane.

In addition, the corresponding pixels and viewpoints of both A and B screens
The condition that the extension line of the line connecting Sa and Sb intersects at the shooting target point needs to specify the corresponding pixel on both the A and B screens, but the corresponding two pixels and the viewpoint Sa and the viewpoint Sb are four points. Are located on the same plane, it is not always necessary to specify a pixel in advance.
As shown in the paragraph 44, the maximum value and the minimum value of the α value of the image of the same part, or the maximum value and the minimum value
It is also possible to replace it with the condition that they match on both the A and B screens.

Further, when the corresponding pixel candidates are narrowed down to pixels on the same α-value line, or when three or more images are used,
On the condition that they are located on the same plane, it is also possible to confirm the quality of the correspondence between the pixels using any two pixels using another image. Therefore, the corresponding two pixels and the viewpoint S
a, the condition that the four points of the viewpoint Sb are located on the same plane includes the condition that the extension line of the line connecting the corresponding pixel of both the A and B screens and the viewpoint Sa, Sb intersects at the photographing target point. It can be said that the condition is more useful.

Note that it is desirable to avoid selection of pixels that may cause the independence of the seven variables, such as designating seven pixels for which zan has the same value. Since the above equation is an independent equation for each corresponding pixel set, the above equation is obtained for each of seven different sets of corresponding pixels that are not located on the same α-value line, and solving these simultaneous equations , 7 variables are obtained. In practice, a small amount of variation is given to each variable as in the 0147 to 0148 terms, and a more accurate approximate solution is obtained. Alternatively, the values of the above formulas are calculated, and a more accurate approximate solution is obtained so that the value of each formula approaches 0.

Although the constant LAB of the distance between the two focal points actually photographed and the distance between the two viewpoints is different,
Even if the constant LAB is arbitrarily determined, no problem occurs because the shapes of the imaging targets to be obtained are similar. If coordinates are converted so that a straight line connecting the viewpoints Sa and Sb is set as a new Z axis, and a condition is provided so that the α values of the corresponding pixels on both the A and B screens match, seven independent variables for each corresponding pixel Is obtained. If we give seven equations that show the relationship of seven independent variables, we get
Find the solution of the variable. Therefore, the corresponding 7 in both A and B screens
The relative positional relationship between the A and B screens can be obtained in the same manner as in the items 0084 to 008 using the pixels.

Similarly, if the same camera is used at the same focal length, then La = Lb, and similarly using six pixels,
The relative positional relationship between the A and B screens is obtained. If the focal length of the two screens is known, A, B using five pixels
The relative positional relationship between the two screens can be obtained. The position of the screen and the viewpoint is translated in parallel so that the generated stereoscopic data has the desired scale of the imaging target, and the distance between any two points of the stereoscopic data and the distance between the corresponding two points of the imaging target is calculated. LAB
change. When the corresponding seven pixels cannot be specified in both the A and B screens, the pixels having the maximum value or the minimum value of the α value of the part viewed from both the two viewpoints Sa and Sb are determined for some of the seven pixels. You may make it correspond.

Here, the above-mentioned processing will be described in detail using mathematical expressions. γFxan = Xsb + ε * (Fxbn−Xsb) γFyan = Ysb + ε * (Fybn−Ysb) γFzan = Fzsb + ε * (Fzbn−Fzsb) γ and ε are unknown constants

Fxan * [Ysb + ε * (Fybn−Ysb)] = Fy
an * [Xsb + ε * (Fxbn−Xsb)] Fxan * [Fzsb + ε * (Fzbn−Fzsb)] = Fzan * [Xsb +
ε * (Fxbn-Xsb)]

[Fxan * Ysb-Fyan * Xsb] = ε * [Fyan * (F
xbnXsb)-Fxan * (Fybn-Ysb)] [Fxan * Fzsb -Fzan * Xsb] = ε * [Fzan * (Fxbn-Xsb)-
Fxan * (Fzbn-Fzsb)]

[Fxan * Ysb-Fyan * Xsb] * [Fzan * (Fx
bn−Xsb) −Fxan * (Fzbn−Fzsb)] = [Fxan * Fzsb−Fza
n * Xsb] * [Fyan * (Fxbn−Xsb) −Fxan * (Fybn−Ys
b)]

(Fxbn-Xsb) * [Fzan * (Fxan * Ysb-Fy
an * Xsb)-Fyan * (Fxan * Fzsb -Fzan * Xsb)]-Fxan
* (Fzbn−Fzsb) * [Fxan * Ysb-Fyan * Xsb] + Fxan
* (Fybn−Ysb) * [Fxan * Fzsb-Fzan * Xsb] = 0

(Fxbn-Xsb) * [Fzan * Fxan * Ysb-Fzan * Fyan *
Xsb-Fyan * Fxan * Fzsb + Fyan * Fzan * Xsb] + Fxan * [(Fybn-Ysb)
* (Fxan * Fzsb-Fzan * Xsb) + (Fzbn-Fzsb) * (Fyan * Xsb-Fxan * Y
sb)] = 0

(Fxbn-Xsb) * [Fzan * Fxan * Ysb-Fyan * Fxan *
Fzsb] + Fxan * [(Fybn-Ysb) * (Fxan * Fzsb-Fzan * Xsb) + (Fzbn-
Fzsb) * (Fyan * Xsb-Fxan * Ysb)] = 0

Fxan * (Fxbn-Xsb) * (Fzan * Ysb-Fyan * Fzsb)
+ Fxan * [(Fybn−Ysb) * (Fxan * Fzsb -Fzan * Xsb) + (Fzbn−F
zsb) * (Fyan * Xsb-Fxan * Ysb)] = 0 Fxan ≠ 0.

Therefore, (Fxbn−Xsb) * (Ysb * Fzan−Fzsb * Fyan) + (Fybn−Ysb) * (F
zsb * Fxan−Xsb * Fzan) + (Fzbn−Fzsb) * (Xsb * Fyan−Ysb * F
xan) = 0

-Xsb * (Ysb * Fzan-Fzsb * Fyan) -Ysb * (Fzsb
* Fxan−Xsb * Fzan) −Fzsb * (Xsb * Fyan−Ysb * Fxan) = 0

Therefore, Fxbn * (Ysb * Fzan−Fzsb * Fyan) + Fybn * (Fzsb * Fxan−Xsb * F
zan) + Fzbn * (Xsb * Fyan−Ysb * Fxan) = 0

Fxan * (Fybn * Fzsb-Fzbn * Ysb) + Fyan * (F
zbn * Xsb-Fxbn * Fzsb) + Fzan * (Fxbn * Ysb- Fybn * Xsb)
= 0

Fxan = Fxa1 = La (n = 2,3,...), Fya
n = yan, Fzan = zan (since the position of screen A is given)

La * (Fybn * Fzsb-Fzbn * Ysb) + yan * (Fzbn * Xsb-Fxbn * Fzsb) + zan * (Fxbn * Ysb-Fybn * Xsb) = 0 La * [(Fybn-Ysb) * Fzsb -(Fzbn-Fzsb) * Ysb] + yan * [(Fzbn-Fzsb) * Xsb- (Fxbn-Xsb) * Fzs b] + zan * [(Fxbn-Xsb) * Ysb- (Fybn-Ysb) * Xsb] = 0 ・ ・ ・ ・

Xsb * [yan * (Fzbn-Fzsb) -zan * (Fybn-Ysb)] + Ysb * [zan * (Fxbn-Xsb) -La * (Fzbn-Fzsb)] + Fzsb * [La * (Fybn -Ysb) -yan * (Fxbn-Xsb)] = 0 (Xsb / Fzsb) * [yan * (Fzbn-Fzsb) -zan * (Fybn-Ysb)] + (Ysb / Fzsb) * [zan * (Fxbn- Xsb) -L a * (Fzbn-Fzsb)] + [La * (Fybn-Ysb) -yan * (Fxbn-Xsb)] = 0 ・ ・ ・ ・

By solving two simultaneous linear equations using (Xsb / Fzsb) and (Ysb / Fzsb) as variables, a five-element five-system equation can be obtained.

The equation is expressed as [(Fxbn-Xsb) * Ysb- (Fybn-Ys
b) * Xsb] and the like, it is confirmed that each pixel set n is independent. In the plane AX + BY + CZ = 1, (XS, Y
The condition that the point of closest approach of (S, ZS) is (X0, Y0, Z0) is obtained.

Since the plane does not pass through the origin (0, 0, 0),... = 1. Also, the plane AX + BY + CZ = 1
And a straight line connecting (XS, YS, ZS) and the closest point (X0, Y0, Z0) is orthogonal.

In the plane AX + BY + CZ = 1, S = (X-XS) **
If 2+ (Y-YS) ** 2+ (Z-ZS) ** 2, the point where S is minimum becomes the closest approach point, which coincides with (X0, Y0, Z0). S = (X-XS) ** 2 + (Y-YS) ** 2 + [(1-AX-BY) / C-ZS] ** 2 dS / dX = 2 (X-XS) -2A [( 1-AX-BY) / C-ZS] / C = 0 → (X-XS) * (C ** 2) -A [(1-AX-BY) -ZS * C] = 0 (A ** 2 + C ** 2) X -XS * (C ** 2) + AB * Y + AC * ZS -A = 0 ・ ・ ・ dS / dY = 2 (Y-YS) -2B [(1-AX-BY ) / C-ZS] / C = 0 → (Y-YS) * (C ** 2) -B [(1-AX-BY) -ZS * C] = 0 (B ** 2 + C ** 2 ) Y -YS * (C ** 2) + AB * X + BC * ZS -B = 0 ・ ・ ・

* (A ** 2 + C ** 2)-* AB (A ** 2 + C ** 2) AB * X -XS * (C ** 2) AB + (AABB) * Y + AABC * ZS -AAB = 0 (A ** 2 + C ** 2) AB * X -YS * (C ** 2) * (A ** 2 + C ** 2) + (A ** 2 + C * * 2) * (B ** 2 + C ** 2) * Y + (A * * 2 + C ** 2) BC * ZS -B (A ** 2 + C ** 2) = 0 XS * ( C ** 2) AB -YS * (C ** 2) * (A ** 2 + C ** 2) + (C ** 2) (A ** 2 + B ** 2 + C ** 2) * Y + BC (C ** 2) * ZS -B (C ** 2) = 0 C ** 2 ≠ 0 → XS * AB -YS * (A ** 2 + C ** 2) + (A * * 2 + B ** 2 + C ** 2) * Y + BC * ZS -B = 0 XS * AB -YS * (A ** 2 + B ** 2 + C ** 2-B ** 2) + (A ** 2 + B ** 2 + C ** 2) * Y + BC * ZS -B = 0 (A ** 2 + B ** 2 + C ** 2) * (Y-YS) + B (A * XS + B * YS + C * ZS -1) = 0 (Y-YS) = -B (A * XS + B * YS + C * ZS -1) / (A ** 2 + B * * 2 + C ** 2) Y = YS -B (A * XS + B * YS + C * ZS -1) / (A ** 2 + B ** 2 + C ** 2) = Y0

'(A ** 2 + C ** 2) * (X-XS) + (A ** 2) XS + AB * (Y-YS) + AB * YS + AC * ZS -A = 0 ( A ** 2 + C ** 2) * (X-XS) + AB * (Y-YS) + A (A * XS + B * YS + C * ZS -1) = 0 (A ** 2 + C ** 2) * (X-XS) -AB * B (A * XS + B * YS + C * ZS -1) / (A ** 2 + B ** 2 + C ** 2) + A (A * XS + B * YS + C * ZS-1) = 0 (A ** 2 + C ** 2) * (X-XS) + (A * XS + B * YS + C * ZS -1) * A * [-(B ** 2) / (A ** 2 + B ** 2 + C ** 2) +1] = 0 (A ** 2 + C ** 2) * (X-XS) + ( A * XS + B * YS + C * ZS-1) * A * [(A ** 2 + C ** 2) / (A ** 2 + B ** 2 + C ** 2)] = 0 ( A ** 2 + C ** 2) ≠ 0 (X-XS) + (A * XS + B * YS + C * ZS -1) * A * / (A ** 2 + B ** 2 + C * * 2) = 0 (X-XS) = -A (A * XS + B * YS + C * ZS -1) / (A ** 2 + B ** 2 + C ** 2) X = XS -A (A * XS + B * YS + C * ZS -1) / (A ** 2 + B ** 2 + C ** 2) = X0

From AX + BY + CZ = 1, CZ = 1-AX-BY = 1-A [XS-A (A * XS + B * YS + C * ZS-1) / (A ** 2 + B ** 2 + C ** 2)] -B [YS-B (A * XS + B * YS + C * ZS-1) / (A ** 2 + B ** 2 + C ** 2)] = 1-(A * XS + B * YS) + (A * XS + B * YS + C * ZS-1) * (A ** 2 + B ** 2) / (A ** 2 + B ** 2 + C ** 2) = 1-(A * XS + B * YS) + (A * XS + B * YS + C * ZS-1) * [1-(C ** 2) / (A ** 2 + B ** 2 + C ** 2)] = 1 + (C * ZS-1)-(A * XS + B * YS + C * ZS-1) * (C ** 2) / (A ** 2 + B ** 2 + C ** 2)] = C * ZS-(A * XS + B * YS + C * ZS-1) * (C ** 2) / (A ** 2 + B ** 2 + C ** 2) Z = ZS -C (A * XS + B * YS + C * ZS-1) / (A ** 2 + B ** 2 + C ** 2) = Z0

That is, (X-XS) =-A (A * XS + B * YS + C * ZS-1) / (A ** 2 + B ** 2 + C ** 2), → X0 = XS -A (A * XS + B * YS + C * ZS-1) / (A ** 2 + B ** 2 + C ** 2) (Y-YS) =-B (A * XS + B * YS + C * ZS-1) / (A ** 2 + B ** 2 + C ** 2), → Y0 = YS-B (A * XS + B * YS + C * ZS-1) / (A * * 2 + B ** 2 + C ** 2) (Z-ZS) =-C (A * XS + B * YS + C * ZS-1) / (A ** 2 + B ** 2 + C * * 2), → Z0 = ZS-C (A * XS + B * YS + C * ZS-1) / (A ** 2 + B ** 2 + C ** 2)

(A ** 2 + B ** 2 + C ** 2) = A (A * XS + B * YS + C * ZS-1) / (XS-X0) = B (A * XS + B * YS + C * ZS-1) / (YS-Y0) = C (A * XS + B * YS + C * ZS-1) / (ZS-Z0) A / (XS-X0) = B / (YS -Y0) = C / (ZS-Z0) → C = A * (ZS-Z0) / (XS-X0), B = A * (YS-Y0) / (XS-X0)

Since (XO, YO, ZO) is on the plane AX + BY + CZ = 1, AX0 + A * (YS-Y0) / (XS-X0) * Y0 + A * (ZS-Z0) / (XS-X0) * Z0 = 1 A * [X0 + (YS-Y0) / (XS-X0) * Y0 + (ZS-Z0) / (XS-X0) * Z0] = 1 A * [(XS- X0) * X0 + (YS-Y0) * Y0 + (ZS-Z0) * Z0] = (XS-X0) A = (XS-X0) / [(XS-X0) * X0 + (YS-Y0) * Y0 + (ZS-Z0) * Z0] Similarly, B = (YS-Y0) / [(XS-X0) * X0 + (YS-Y0) * Y0 + (ZS-Z0) * Z0] C = (ZS -Z0) / [(XS-X0) * X0 + (YS-Y0) * Y0 + (ZS-Z0) * Z0]

Therefore, AX + BY + CZ = 1 is (XS-X0) * X + (YS-
Y0) * Y + (ZS-Z0) * Z = (XS-X0) * X0 + (YS-Y0) * Y0 + (ZS-Z0) * Z0.

(Cutting line at plane Y = YO) (YS-Y0) * Y = (YS-Y0) * Y0 (XS-X0) * X + (ZS-Z0) * Z = (XS-X0) * X0 + (ZS-Z0) * Z0 (ZS-Z0) * Z = (ZS-Z0) * Z0-(XS-X0) * (X-X0) Z = Z0-(X-X0) * (XS-X0 ) / (ZS-Z0)

Cutting line [X, Y0, Z0- (X-X0) * (XS-X0) / (ZS
-Z0)] (Line in plane Y = YO, orthogonal to the above cutting line) Slope (ZS-ZO) * X-(XS-X0) * Z and passes through (X0, Y0, Z0). (ZS-ZO) * X-(XS-X0) * Z = (ZS-ZO) * X0-(XS-X0) * Z0 Plane (ZS-ZO) * X-(XS-X0) * Z = (ZS -ZO) * X0-(XS-X0) * Z0
And the plane (XS-X0) * X + (YS-Y0) * Y + (ZS-Z0) * Z = (XS-X0) * X0 +
Intersection line of (YS-Y0) * Y0 + (ZS-Z0) * Z0 (ZS-ZO) * (X-X0)-(XS-X0) * (Z-Z0) = 0 → (Z-Z0) = (X-X0) *
(ZS-Z0) / (XS-X0) (XS-X0) * (X-X0) + (YS-Y0) * (Y-Y0) + (ZS
-Z0) * (Z-Z0) = 0 (XS-X0) * (X-X0) + (YS-Y0) * (Y-Y0) + (X-X0) * (ZS-ZO) ** 2 /
(XS-X0) = 0-(YS-Y0) * (XS-X0) * (Y-Y0) = (X-X0) * [(XS-X0) ** 2 + (ZS-Z
O) ** 2] (Y-Y0) =-(X-X0) * [(XS-X0) ** 2 + (ZS-ZO) ** 2] / [(XS-X0) *
(YS-Y0)] Y = Y0-(X-X0) * [(XS-X0) ** 2 + (ZS-ZO) ** 2] / [(XS-X0) * (Y
S-Y0)]

Assuming that the distance between the center of the screen and the coordinates (V, W) is R, as the coordinates (V, W) on the screen and the rotation angle φ between the line Y = Y0 on the screen and the V axis, V = Rcosθ, W = Rsinθ
Can be expressed as Where θ is the angle between the line connecting the coordinates and the origin and the V axis. V '= Rcos (θ + φ) = Rcosθcosφ-Rsinθsinφ = Vcosφ-Ws
inφ, W '= Rsin (θ + φ) = Rsinθcosφ-Rcosθsinφ = Wcosφ + Vs
inφ (V ', W') coordinate axes (V ', 0): [X, Y0, Z0- (X-X0) * (XS-X0) / (ZS-Z0)] (0, W'): { X, Y0-(X-X0) * [(XS-X0) ** 2 + (ZS-ZO) ** 2] /
[(XS-X0) * (YS-Y0)], Z0 + (X-X0) * (ZS-Z0) / (XS-X0)}.

First, the three-dimensional coordinates of (V ', 0) and (0, W') are obtained. V '** 2 = (X-X0) ** 2+ (Y0-Y0) ** 2+ [Z0-Z0 + (X-X0) * (XS-X0) / (ZS-Z0)] ** 2 = (X-X0) ** 2 * {1 + [(XS-X0) / (ZS-Z0)] ** 2} = (X-X0) ** 2 * [(ZS-Z0) ** 2 + ( XS-X0) ** 2] / [(ZS-Z0) ** 2]

If the screen B is on the right side, (ZS-Z0)> 0 from the positional relationship between the photographing object and the screen. If (X-X0)> 0, V '> 0. V '= (X-X0) / (ZS-Z0) * SQRT [(ZS-Z0) ** 2 + (XS-X0) ** 2] Note that [(ZS-Z0) ** 2 + (XS- X0) ** 2] = [Lb ** 2- (YS-Y0) ** 2] (X-X0) = V '* (ZS-Z0) / SQRT [Lb ** 2- (YS-Y0) * * 2] X = X0 + V '* (ZS-Z0) / SQRT [Lb ** 2- (YS-Y0) ** 2] Z = Z0-V' * (ZS-Z0) * (XS-X0) / {(ZS-Z0) * SQRT [Lb ** 2- (YS-Y0) ** 2]} = Z0-V '* (XS-X0) / SQRT [Lb ** 2- (YS-Y0) * * 2] The three-dimensional coordinates of (V ', 0) are {X0 + V' * (ZS-Z0) / SQRT [Lb ** 2
-(YS-Y0) ** 2], Y0, Z0-V '* (XS-X0) / SQRT [Lb ** 2- (YS-Y0)
** 2]}

W '** 2 = (X-X0) ** 2+ {YO-Y0 + (X-X0) * [(XS-X0) ** 2+ (ZS-ZO) ** 2] / [( XS-X0) * (YS-Y0)]} ** 2+ [Z0 + (X-X0) * (ZS-Z0) / (XS-X0) -Z0] ** 2 = (X-X0) ** 2 + {(X-X0) * [(XS-X0) ** 2 + (ZS-ZO) ** 2] / [(XS-X0) * (YS-Y0)]} ** 2 + [(X- X0) * (ZS-Z0) / (XS-X0)] ** 2 = (X-X0) ** 2 * {1 + {[(XS-X0) ** 2 + (ZS-ZO) ** 2 ] / [(XS-X0) * (YS-Y0)]} ** 2 + [(Z S-Z0) / (XS-X0)] ** 2} = {(X-X0) ** 2 / [ (XS-X0) * (YS-Y0)] ** 2} * {[(XS-X0) * (YS-Y0)] ** 2 + [(XS-X0) ** 2+ (ZS-ZO) ** 2] ** 2 + [(ZS-Z0) * (YS-Y0)] ** 2} = {(X-X0) ** 2 / [(XS-X0) * (YS-Y0)] * * 2} * [(XS-X0) ** 2 + (ZS-ZO) ** 2] * [(XS-X0) ** 2 + (YS-YO) ** 2 + (ZS-ZO) ** 2] = {(X-X0) ** 2 / [(XS-X0) * (YS-Y0)] ** 2} * [Lb ** 2- (YS-YO) ** 2] * Lb ** Two

From the positional relationship between the screen and the object to be photographed, (XS-X0)
<0. If (YS-Y0) <0, the line of sight is upward, if W '> 0,
(X-X0) <0 W '=-{(X-X0) / [(XS-X0) * (YS-Y0)]} * Lb * SQRT [Lb ** 2- (YS-YO) ** 2 ] (X-X0) =-W '* (XS-X0) * (YS-Y0) / {Lb * SQRT [Lb ** 2- (YS-YO) ** 2]} X = X0 -W' * (XS-X0) * (YS-Y0) / {Lb * SQRT [Lb ** 2- (YS-YO) ** 2]} Y = Y0-(X-X0) * [(XS-X0) ** 2 + (ZS-ZO) ** 2] / [(XS-X0) * (YS-Y0)] = Y0 + W '* (XS-X0) * (YS-Y0) / (Lb * SQRT [Lb * * 2- (YS-YO) ** 2]} * [Lb ** 2- (YS-YO) ** 2] / [(XS-X0) * (YS-Y0)] = Y0 + W '* SQRT [Lb ** 2- (YS-YO) ** 2]} / Lb

Z = Z0 + (X−X0) * (ZS−Z0) / (XS−X0) = Z0−W ′ * (XS−X0) * (YS−Y0) / (Lb * SQRT [Lb ** 2- (YS-YO) ** 2]} * (ZS-Z0) / (XS-X0) = Z0 -W '* (YS-Y0) / {Lb * SQRT [Lb ** 2- (YS-YO ) ** 2]} * (ZS-Z0) = Z0 -W '* (YS-Y0) * (ZS-Z0) / {Lb * SQRT [Lb ** 2- (YS-YO) ** 2]}

The three-dimensional coordinates of (V ', 0): (X0 + V' * (Z
S-Z0) / SQRT [Lb ** 2- (YS-Y0) ** 2], Y0, Z0-V '* (XS-X0) / SQ
RT [Lb ** 2- (YS-Y0) ** 2]) Three-dimensional coordinates of (0, W '): (X0-W' * (XS-X0) * (YS-Y0) / {Lb * S
QRT [Lb ** 2- (YS-YO) ** 2]}, Y0 + W '* SQR
T [Lb ** 2- (YS-YO) ** 2] / Lb, Z0-W '* (YS-Y0) * (ZS-Z0) / {Lb *
SQRT [Lb ** 2- (YS-YO) ** 2]}) Intermediate between the three-dimensional coordinates (X ', Y', Z ') of (V', W ') and (X0, Y0, Z0) The coordinates are the three-dimensional coordinates of (V ', 0) and the three
Matches the middle coordinates of the dimensional coordinates.

Therefore, X '= {X0 + [V' * (ZS-Z0) * Lb-W '* (XS-X0) * (YS-Y0)] / {Lb * SQRT [Lb ** 2- ( YS-YO) ** 2], Y '= Y0 + W' * SQRT [Lb ** 2- (YS-YO) ** 2] / Lb, Z '= Z0- [V' * (XS-X0) * Lb + W '* (YS-Y0) * (ZS-Z0)] / {Lb * SQRT [Lb ** 2- (YS-YO) ** 2]} (X'-XS) = (X0-XS ) + [V '* (ZS-Z0) * Lb -W' * (XS-X0) * (YS-Y0)] / {Lb * SQRT [Lb ** 2- (YS-YO) ** 2], (Y'-YS) = (Y0-YS) + W '* SQRT [Lb ** 2- (YS-YO) ** 2] / Lb, (Z'-ZS) = (Z0-ZS)-[V '* (XS-X0) * Lb + W' * (YS-Y0) * (ZS-Z0)] / {Lb * SQRT [Lb ** 2- (YS-YO) ** 2]}

(Independence of Equation 7) La * [(Fybn-Ysb) * Fzsb- (Fzbn-Fzsb) * Ysb] + yan * [(Fzbn-Fzsb) * Xsb- (Fxbn-Xsb) * Fz sb] + zan * [(Fxbn-Xsb) * Ysb- (Fybn-Ysb) * Xsb] = 0 ... La * [(Fybn-Ysb)-(Fzbn-Fzsb) * Ysb / Fzsb] + yan * [( Fzbn-Fzsb) * Xsb / Fzsb- (Fxbn-Xs b)] + zan * [(Fxbn-Xsb) * Ysb / Fzsb- (Fybn-Ysb) * Xsb / Fzsb] = 0 ...

The three-dimensional coordinates of each pixel (Vn ', Wn') are represented by (Fxb
n, Fybn, Fzbn), Fxbn = X0 + [V '* (ZS-Z0) * Lb-W' * (XS-X0) * (YS-Y0)] / {Lb *
SQRT [Lb ** 2- (YS-YO) ** 2], Fybn = Y0 + W '* SQRT [Lb ** 2- (YS-YO) ** 2] / Lb, Fzbn = Z0- [V' * (XS-X0) * Lb + W '* (YS-Y0) * (ZS-Z0)] / {Lb *
SQRT [Lb ** 2- (YS-YO) ** 2] where XS = Xsb, YS = Ysb, ZS = Fzsb.

For example, (Fybn-Ysb)-(Fzbn-Fzsb) * Ysb / Fzs
b = [-(Ysb-YO) + Wn '* SQRT [Lb ** 2- (Ysb-YO) ** 2] / Lb] + {(Fz
(sb-ZO) + [Vn '* (Xsb-X0) * Lb + Wn' * (Ysb-Y0) * (Fzsb-Z0)] / {L
b * SQRT [Lb ** 2- (Ysb-YO) ** 2]} * Ysb / Fzsb The variables in the above equation are (Xsb-X0), (Ysb-Y0), (Fzsb-Z
0), φ, and Ysb / Fzsb.

[(Fzbn-Fzsb) * Xsb / Fzsb- (Fxbn-Xsb)], [(F
xbn-Xsb) * Ysb / Fzsb- (Fybn-Ysb) * Xsb / Fzsb], the expression is La, (Xsb-X0), (Ysb-Y0), (Fzsb-Z0),
φ, Xsb / Fzsb, and Ysb / Fzsb. In the expression represented by these seven variables, yan, za
Since n, Vn ', and Wn' are independent, it can be confirmed that the formula is independent for each pixel in principle.

In the case where a plurality of screens in which the same object is photographed from different positions and the viewpoints in which the images are photographed are arranged in the same coordinate system in the item 0026, when seven or eight points on the photographing object are arranged.
Using the points, the positions of both the A and B screens and the two viewpoints can be determined. Scan the slit light source in the vertical and horizontal directions, identify the pixels located at the boundary where the slit light source is irradiated, and identify the pixels where the vertical and horizontal boundaries intersect, Select 6 to 8 pixels, A, B
The positions of both screens and the viewpoint may be obtained. Further, in the case where a shooting target having a complicated pattern is converted into three-dimensional data, a large number of points are specified on both the A and B screens in this manner.
In the same manner as the singular point of, the association of pixels on the same α-value line sandwiched between the identified and associated pixels may be made easier.

Instead of the slit light source, a spot light beam having an angular edge may be used. The coordinates of the center of a single or a plurality of pixels that have sensed a spot light beam are made to correspond on both the A and B screens, and the relative positional relationship between the A and B screens is calculated using such corresponding 5 to 8 points. You may ask.

By specifying a part that can be viewed simultaneously from both the Sa and Sb viewpoints without specifying a pixel, the specification can be replaced with specifying a pixel. That is, since the maximum value and the minimum value of the α value of the image of the same region, or the maximum value of the convex portion and the α value of the minimum value match in both the A and B screens, such an α value is the maximum value,
The coordinates of yan, zan, Fxbn, Fybn, Fzbn, etc. are calculated from the coordinates of the seven sets of pixels on both the A and B screens that are the minimum value, the maximum value, the minimum value, etc., and the seven variables are calculated using the expression of the 0096 term. You may ask. By doing so, both the A and B screens can be placed in the same coordinate system only by specifying the same part.

Further, if four or more parts are identified by the distribution of color data and the like, and if only one set of pixels having the same α value such as the maximum value or the minimum value of seven or more is found,
It can be considered that this is a point that can be seen from both Sa and Sb viewpoints at the same time, and the coordinates of the screen can be automatically determined. If the image centers of the screens A and B to be used are unknown, the number of variables is increased by two for each screen, that is, a total of four. Therefore, in that case, 11 sets of corresponding pixels are determined, and 11 variables are obtained under the condition that the α value matches each corresponding set of pixels.

One of the methods for finding a solution is to solve the above simultaneous equations. Although it is possible to do so, since the solution method is very complicated, usually, an approximate solution (initial value) is obtained, and three cases are obtained in which the approximate solution of each variable is adjusted by a certain small amount of variation. , 3 combinations of each variable, that is, if it is 8 variables, 3 8 −1 = 6,
We can simulate 560 ways and gradually increase the accuracy of the approximate solution.

In this case, the expressions of the items 0074 to 0077 are effective means for obtaining an approximate solution. In a building or the like, many points whose coordinates can be assumed are found, and even if the accuracy is lacking, the coordinates of three points sufficient as a means for obtaining an approximate solution can be obtained. Also, the positional relationship between the three coordinates and the center line of the image can be roughly determined. If an appropriate approximate solution is obtained,
According to the above method, it is possible to determine an accurate positional relationship with a plurality of screens and viewpoints in the same coordinate system with a smaller amount of calculation.

By using these methods, the positional relationship between a plurality of screens can be obtained, and they can be placed in the same coordinate system. In this case, for example, the basic composition of one image uses a wall surface, a floor, and the like, and the other image uses the 3
Different methods may be combined, such as using the coordinates of points or using a rectangle such as a window frame in common. It is difficult to find three points with accurate coordinates in an actual cityscape photograph. However, they use building drawings and maps,
Approximate coordinates can be obtained by floor height, column spacing, and the like. The basic composition can be determined by the “three-dimensional data creation method”, and the coordinates can be estimated from this. Even from the coordinates thus obtained, an approximate solution with sufficient accuracy can be obtained for setting the initial values of the terms 0147 to 0148.

[0150] From a plurality of images placed in the same coordinate system, three-dimensional data in the space to be imaged can be created based on the three-dimensional data creation method described in the section 0042 and thereafter. In addition, for parts where three-dimensional data with sufficient accuracy could not be created due to the roughness of the pixels in the image, etc., use an appropriate three-dimensional data model as described in “3D data creation method” to refine the data. Can be corrected to three-dimensional data determined to be. By giving the color data where the line connecting the points of the completed three-dimensional data and the viewpoint intersects the screen to each point of the three-dimensional data in the three-dimensional space, the three-dimensional data having the color data is completed. (Texture Mapping) The three-dimensional data constituted by the set of points can be converted into three-dimensional polygon data by selecting an appropriate method such as taking an intermediate point between three adjacent points.

When the one-to-one correspondence between the regions 0049 to 0053 in the second embodiment has been completed, the approximate position of the corresponding pixel set is designated by an input operation, and the protruding tip having the same color data. Are automatically selected to correspond to six to eight pairs of pixels, and the approximate coordinates of the imaging target point represented by any three of them are given.
By the method described in the section, the two screens can be arranged in the same coordinate system. When an image is captured from a camera, a system for automatically creating three-dimensional data of a shooting target by such a simple operation can be created.

[0152]

[Embodiment 1] An embodiment in which three-dimensional data is created using the slit light source 11 will be described. FIG. 1 is a block diagram showing the configuration of this embodiment. In the figure, reference numeral 10 denotes a camera A as a photographing means, 11 denotes a slit light source, 12 denotes a camera B as a photographing means, 13 denotes an image storage area, 14 denotes a computer, 15 denotes a monitor, 16 denotes an operator, 17 denotes map data, Reference numeral 18 denotes a frame serving as a reference, and 19 denotes a position measuring means G.
The computer 14 has a calculating unit and a generating unit, and the computer 14 performs a process of creating three-dimensional data.

Next, the operation of this embodiment will be described. First, as described in “3D data creation method”, a basic composition is determined from both the A and B screens captured by the camera A10 and the camera B12 with the frame 18 serving as a reference, and the basic composition is calculated. Are placed in the same coordinate system with the straight line passing through the viewpoint Sa and the viewpoint Sb as the Z axis. A, B
Each pixel na, nb of both screens has the following data format and is stored in the image storage area 13. Pixel na (b) = (screen No., three-dimensional coordinates Xna, Yna, Zna, α value of coordinates, time when the pixel starts sensing light, time when sensing ends) Note that α value is α = Yna / Give as Xna.

Irradiation is performed while the slit light source 11 is placed in the vertical direction and runs from left to right (or vice versa). Camera A1
0, the camera B12 simultaneously captures the images at short intervals, and as a result, data unique to each pixel in the image storage area 13 includes:
The time when the light starts to be sensed and the time when it ends are recorded.

In general, when identifying pixels corresponding to light at the same time and making them correspond to each other, pixels located at a boundary between a portion irradiated by the slit light source and a portion not irradiated (the pixel in FIG. 7)
tv). Although such a known method may be adopted, one of the technical problems in that case is that the virtual pixel Pv, which is the original boundary line, and the pixel pv, which is related to light, are at most the distance between adjacent pixels. An error of Δlp (the distance between the point pv and the point pv + 1) occurs. Assuming that the pixels have an infinite density on the same α-value line, the β value of the virtual pixel Pva (b), which started to sense light at time tv, starts to sense light at the actual finite number of pixels. Pva (b) = pva (b) where pva (b) is the pixel
+ Δlpa (b).

Note that Δlpa (b) represents an error in the screen A (B) within one pixel interval. In FIG. 7, Pva (b), pva (b), Pu
a (b) and pua (b) are simply described as Pv, pv, Pu, and pu. Similarly, the β value of the virtual pixel Pua (b) that finishes sensing light at time tu is Pua (b), with the pixel actually starting to sense light as pua (b).
= Pua (b) -Δlpa '(b).

If the slit light source is extremely thin, the time tv
tu, the virtual pixels Pv and Pu are as close as possible. This is a virtual pixel
Pua2 and Pub2. If the rotation speed of the slit light source is constant from time tv to time tu, the time (t
Approximately the positions of the virtual pixels Pua2 and Pub2 on both the A and B screens in v + tu) / 2, Pu2 = (Pva + Pua) / 2, Pub2 = (Pvb + Pub) / 2
Can be given as Pua2 = (Pva + Pua) / 2 = (pva + Δlpa) / 2 + (pua−Δlpa ′) / 2 = (pva + pua) / 2 + (Δlpa−Δlpa ′) / 2 Pub2 = (Pvb + Pub) / 2 = (pvb + Δlpb) / 2 + (pub) −Δlpb ′) / 2 = (pvb + pub) / 2 + (Δlpb−Δlpb ′) / 2

Naturally, the slit light source and the A and B screens are arranged at different angles and positions, respectively, and this causes an error. However, if the rotation speed of the slit light source is sufficiently slow as compared with the photographing interval. For example, on the same α value line, the number of pixels (for example, about 1 to 3) that can be recognized as the same shooting target (the same sensing start time and the same sensing end time) on both screens A and B can be narrowed down, The error is considered to be sufficiently smaller than the pixel interval. (Δlpa−Δlp
Since the standard deviations of a ′) / 2 and (Δlpb−Δlpb ′) / 2 are smaller than Δlpa and Δlpb, the accuracy is more accurate than simply setting the position at time tv to the pixel Pv of the boundary line of the slit light source. Can be expected to improve.

On the same α-value line, a plurality of pixels start sensing light at the same time tv and finish sensing light at the same time tu.
(pu... pv), the coordinates of the virtual pixel at time (tv + tu) / 2 are set to the coordinates ((X
pv + Xpu) / 2, (Ypv + Ypu) / 2, (Zpv + Zpu) / 2)
If there is a single pixel that starts sensing light at the same time tv and finishes sensing light at the same time tu, the coordinates of the pixel itself are used. When creating the 3D data, A, B with different α values
Since the three-dimensional data is not obtained by associating the pixels of both screens, it is not necessary to match the standards for capturing the time at different α values.

Although it is not necessary to obtain high precision in the rotation speed of the slit light source for obtaining the coordinates of the virtual pixel at the time (tv + tu) / 2, the correspondence between the pixels at the same shooting time in both the A and B screens is not required. Therefore, when it is necessary to correct the coordinates of one of the screens in accordance with the photographing time of one of the screens, the accuracy required to prevent the occurrence of an error is required. The same applies to a known method using the boundary of the slit light source as coordinates. If the rotation speed of the slit light source is high and it is determined that the error is greater in the above method, the coordinates of the pixel pv of the boundary line of the slit light source at time tv, and light adjacent to the same α value line A well-known method of giving an intermediate value of the coordinates of the pixel pv + 1 immediately before the start of sensing as the coordinates of the time tv may be used.

If the photographing times of the two cameras are not completely synchronized, the slit light source rotates by the difference between the times, and the same point cannot be identified by photographing at the same time as it is. If it is possible to precisely measure the shooting time of each of the A and B images, the shooting time (tv + tu) / 2 of one screen is given, and the shooting time (tv '+ tu') / 2 of the other screen is given. It is necessary to estimate the coordinates of the virtual pixel at the shooting time (tv + tu) / 2 from the coordinates of the virtual pixel in.

The computer 14 determines a pixel which senses the slit light source 11 irradiated at an arbitrary time t by the following method. Pixels that are between the time at which the time t began to be sensed and the time at which the sense ends are extracted, and the time when the sensing was started for each column on the same α value line in order from the upper row of the screen from among the pixels was detected. The pixel p whose median value at the end time is closest to the time t is selected. If there are a plurality of such pixels, an intermediate value of the coordinates of the pixels at both ends is given as the coordinates of the virtual pixel p.

Therefore, for each pixel selected, a virtual pixel r representing an image at time t is assumed using a pixel q having a value close to its α value and having a median value close to its median value. The coordinates are determined by the following equation. If time t = median time of the (virtual) pixel p, the pixel r is the pixel p itself; otherwise, the coordinates of the virtual pixel r are obtained based on the following equation. (Time t−median time of pixel p) / (median time of pixel q−median time of pixel p) = (X coordinate of pixel r−X coordinate of pixel p) / (X coordinate of pixel q− X coordinate of pixel p) = (Y coordinate of pixel r−Y coordinate of pixel p) / (Y coordinate of pixel q−Y coordinate of pixel p) = (Z coordinate of pixel r−Z coordinate of pixel p) / ( (Z coordinate of pixel q−Z coordinate of pixel p) At this time, if the median value of the pixel p is later than t, it is more preferable to select the pixel q from earlier than t and from earlier the pixel q from later than t. Seems desirable. If there is no such point, the pixel p itself is set as the pixel r.

A group of a plurality of pixels r hypothetically obtained in this way is created on both the A and B screens. At this stage, the line on the screen is represented by a set of points, and the following operation is performed to make this a continuous line. On each screen, use four consecutive points and pass through those four points.
Find the next curve. Since the calculation process becomes complicated if the three-dimensional coordinates of each point are used as they are, the screen is once converted to two-dimensional, and a three-dimensional curve is obtained as coordinates in a two-dimensional space. A line between two intermediate points in the obtained three-dimensional curve is defined as the three-dimensional curve. By repeating this process one point at a time, the entire curve is obtained. Since this calculation cannot be performed only at both ends of the line, a cubic curve adjacent to the line is used. In this way, on both screens A and B, an image obtained by sensing the slit light source at time t is obtained as a curve.

For an arbitrary α value, the plane Y = αX and A, B
The intersection between the curve of the 0164 item and the two lines extending from the viewpoint intersect each other. Finding this intersection,
This is the position where the slit light source irradiates the object to be photographed. The set of the three-dimensional coordinates of this location is the three-dimensional data of the line projected and projected by the slit light source on the object. Therefore, from the maximum to the minimum of the α value of each screen,
The plane Y = αX is obtained with the α value at regular intervals, whereby a large number of three-dimensional coordinates of the position irradiated on the object can be obtained. The set of the three-dimensional coordinates becomes the three-dimensional data of the photographing target object determined at the time t.

By changing the variable time t at a short time interval such as a photographing time interval and repeating the processes from 0162 to 0165, three-dimensional data of the entire photographing object can be obtained.

A method for producing three-dimensional data of a boundary of a ring or a region on an object to be photographed is described in the second embodiment of the present invention.
The pixels on the boundary line are determined by the methods described in the paragraphs 0048 to 0049 and 0049 to 0053, and the method is combined with the method described in the above item 0164. In this case, there is no slit light source, but the lines on both A and B screens are specified,
In each screen, a three-dimensional space formed by intersecting in a three-dimensional space a curved surface formed by connecting surfaces formed by extension lines connecting adjacent pixels to the viewpoints This is possible by setting the upper polygonal line as the coordinates of the ring on the photographing object. Therefore, the details will be omitted because they are repeated. In addition, an appropriate straight line or curve may be applied between pixels on both boundary lines of the adjacent region using a linear or nonlinear square method, and this may be used as the boundary line.

Embodiment 2 FIG. An embodiment of an apparatus for automatically or semi-automatically creating three-dimensional data of an imaging target from an image of the same imaging target without illuminating a light source or the like will be described.
FIG. 2 is a diagram showing a flow of analysis in this embodiment. In this embodiment, the cameras A10 and B1 of FIG.
2 is replaced with the image A and the image B, and the slit light source 11, the reference frame 18, the map data 17, and the GPS 19 are unnecessary in this embodiment.

The image A10 and the image B12 are stored in the image storage area 13
To save. Each image is divided into a plurality of regions on the basis of a change of the color data of adjacent pixels at a certain level or more. The area-based number to which the pixel at the upper left corner of the screen belongs is set to 1, and the determination operation is repeated from the adjacent pixels. When the color data of the pixel to be determined is different from the color data of one surrounding pixel for which the determination operation has already been completed by a certain level or more, the two pixels are on the boundary of the area.

If the color data of the pixel to be determined is the same as one of the surrounding pixels (within a certain difference range), the region number to which the pixel to be determined belongs is the same as the region number of the pixel of the same color data. Do the same. If there is no such pixel around, a new area-specific number is given to the pixel to be determined.
In this way, all pixels are assigned to each area.

The image dividing method may be any method, such as the existing method, in addition to the method described in the second embodiment, as long as it conforms to the data format described in this section. At this time,
The number of pixels of the one area is equal to or less than a certain number (for example, 10 or less), and such an area having a certain scale or less continuously exists in a certain narrow area (for example, about 1,000 pixels) or more. At this time, the entire area is defined as one area having a pattern, and the area numbers are unified.

All pixels na and nb of both the A and B screens have the following data format in the image storage area 13. Pixel na = (pixel No., screen No., three-dimensional coordinates Xna, Yna, Zn
a, area number, determination of whether or not it is on the boundary line, α value of coordinates,
Pixel color data (saturation, brightness), candidates P1, P2, ..., candidates Q1, Q2, Q3 ...)

.. Are the numbers of a single or a plurality of areas in the area on the B screen where the area to which the pixel belongs may correspond one-to-one. Hereinafter, it is simply referred to as a candidate P. Candidates Q1, Q2, ... are areas on screen B,
This is the number of a single pixel or a plurality of pixels to which the pixel may correspond. Hereinafter, it is simply referred to as candidate Q. α value is α = Y
Give as na / Xna. In addition, since data that may have a one-to-one correspondence has not been determined at this stage, this data is blank. The same applies to the pixels on the B screen.

A pixel that captures a portion that can be viewed simultaneously from two viewpoints should correspond to at least one pixel on the other screen. When the difference between the saturation and the brightness of the color data is equal to or less than a certain value, it is treated as the same color data. Hereinafter, completely the same color data will be referred to as completely the same. The criterion for judging whether the α values of the pixels are the same is as described in the paragraphs 0059 to 0061.

The maximum and minimum of the α value of each part on both the A and B screens are obtained, and for each part on the A screen, the area on the B screen having an α value overlapping the maximum and minimum range of the α value is obtained. All the numbers are obtained, and among these areas, the numbers of all the areas that match the distribution of the color data of the pixels existing in the area on the screen A based on a predetermined standard are set as candidates P.

When the pixel n on the screen A is located outside the plane passing from the viewpoint Sa in parallel with the plane of the screen B, the corresponding pixels on the screen B have the same α value as shown in FIG. li
The viewpoint Sb on neB and parallel to the line connecting the viewpoint Sa and the pixel n
Must be inside the intersection of the line drawn from and lineB. Therefore, the existence area on the B screen of the imaging target portion represented by a certain area on the A screen is calculated as the possible area on the B screen for each pixel located at the innermost side in each α value of the area on the A screen. The combined image area is obtained. Therefore, candidates that do not satisfy this condition are excluded.

The one-to-one correspondence for each region is determined based on whether or not the conditions of items 0049 to 0053 1) to 6) are satisfied. When the maximum value and the minimum value of the α value coincide with each other among the candidates P and the candidates satisfying the condition of 4) are narrowed down to one, it is determined that the set of the areas corresponds one-to-one, and Exclude candidates. In this case, in each region, the ratio of pixels having the same color data among pixels on the other screen exceeds a certain value (for example, 50%), and the condition of 4) is satisfied. The error range in which the color data is determined to be the same is as described in the paragraphs 0045 to 0048. These one-to-one areas are excluded from other area candidates P.

At this stage, a one-to-one correspondence has been set for a portion having only one candidate, and a one-to-one correspondence relationship between the other portions is estimated based on the one-to-one correspondence. 004
Assuming that a set of regions satisfying five or more (four or more if none) conditions among the conditions of 1) to 6) of items 9 to 0053 are in one-to-one correspondence, the other candidates of candidate P are exclude. In addition, a region that has a one-to-one correspondence is excluded from candidates P that have a one-to-one correspondence with other regions. If a plurality of candidates remain, a candidate having a high degree of matching of the condition of 4) is selected. Here, based on the one-to-one correspondence area, the same operation is repeated for another area. This is repeated until a new one-to-one correspondence area cannot be found.

When the one-to-one correspondence is not completed,
The operation is repeated in the same manner, assuming that the matching condition of 4) is loose and that a set of regions satisfying three or more conditions has a one-to-one correspondence. If the one-to-one correspondence has not been established up to this point, it is determined that the area is displayed on only one screen.

If the initially set part division is appropriate,
Usually, at this stage, it is considered that one-to-one correspondence of most parts is established. For confirmation, the one-to-one correspondence of each part on both the A and B screens is displayed on the monitor 15 and the operator 16
Confirms this, and if an image area that should correspond one-to-one is left or an incorrect one-to-one correspondence, it is complemented by correcting it by an input operation.

Next, for each one-to-one image area,
The correspondence in pixel units is determined. Embodiment 2 of the invention
Under the conditions as described in above, it is possible to obtain a combination of all the pixels and obtain a combination in which the number of pixels on the screen having the smaller number of corresponding pixels is the largest and the total amount of the difference amount of the color data is the smallest. , Is most desirable for obtaining three-dimensional data, but because of the huge amount of calculation,
An example of a method for obtaining an optimal solution or a solution close thereto with a smaller amount of calculation will be described.

When the pixels na and nb correspond to each other, the constraints 0059 to 0061 are satisfied. Among these constraint conditions, is a criterion that can be determined for each corresponding pixel set, but is a criterion that cannot be determined unless pixel sets are compared with each other. Therefore, first, all the pixels on the other screen that satisfy the constraint conditions are placed as candidates Q for each pixel. The obtained single or plural candidates (pixels on the other screen) are rearranged as Q1, Q2, Q3,... In order from the closest color data. In the case where three images are used, it is possible to confirm the possibility of establishing a correspondence between pixels based on the conditions of the items 0054 to 0055, and candidates that do not satisfy this condition are excluded.

In the image areas corresponding to the screens A and B on a one-to-one basis, two sets of pixels having the closest color data among the pixels on the same α-line, and the situation where they are adjacent to each other in the two screens correspond to the original situation. A precondition is given that the pixel is most likely to occur in the pixel to be processed. This seems to be a reasonable prerequisite if the hue shifts caused by the habits and lighting of the two cameras are properly corrected. In particular, it seems to apply well to major parts such as the center of the screen. The method for correcting the hue for each area is also described from the items 0068 to 00.
This is described in section 69. Therefore, a discriminating operation is performed so as to obtain the maximum number of pixel sets that simultaneously satisfy such conditions and the conditions of 0059 to 0061.

A large number of sets whose elements are empty are prepared. Included as elements of the set are sets of corresponding pixels that are simultaneously adjacent in both the A and B screens, indicating that they are in an adjacent relationship in the three-dimensional space. Hereinafter, being around means eight pixels surrounding a certain pixel. In the combination of each pixel and its candidate Q1, the candidate Q1 of the pixel m around a certain pixel n
Is the candidate Q1 of the pixel n itself or in the vicinity thereof and satisfies the conditions of the items 0059 to 0061, it is determined that there is a high possibility that the pixel Q is in an adjacent relationship, and the set of the pixels is included in the same set W1. include. When the candidates Q1 are separated from each other, another set W2 is prepared, and a set of the pixel m and the candidate Q1 is included in the set.

In each of the screens A and B, if there is no discrimination error in the color data, every pixel usually has at least two or more adjacent pixels. Therefore, many combinations of pixels having such an adjacent relationship often exist. This operation is sequentially repeated for all pixels. If there is a high possibility that the pixel is adjacent to one pixel of one of the sets Wu of the set generated up to that point, the pixel is included in the set Wu. And include it in it. When there is an adjacent relationship with the pixels of a plurality of sets including Wu, all of the plurality of sets are combined into one set Wu '. this
This is performed for the pixels on both the A and B screens. Duplicate sets of identical pixels are eliminated.

In the many sets Wu 'generated in this way, if there is a set of pixels that do not satisfy the conditions of items 0059 to 0061, a new set (plurality) is set for each of the sets of pixels that do not satisfy the conditions. Wr is generated, and a set of pixels not inconsistent with the conditions of 0059 to 0061 are copied from the set Wu 'and added to each Wr.
Each time a set of pixels is added to Wr, 00
The determination on the condition of 61 is repeated.

Among the many sets generated in this manner, the sets are sequentially integrated from the set having the largest number of pixels on the premise that the conditions of 0059 to 0061 are satisfied within the range of the same α value. To generate a new set Wu ". The generated set Wu" is a set of sets of pixels having a correspondence relationship between the A and B screens. A set having only one pixel is handled similarly.

The pixels in the set Wu "generated in this manner are deleted from the candidates after Q2. Pixels which have become the plurality of pixel candidates Q1 are included in the set Wu" on the other screen. Are excluded from the candidate Q of all the pixels that are not.
A set of these pixels and the candidate Q1 is considered to be a set of pixels representing the stereoscopic data of the imaging target, but some holes are still open.

Next, using the set of pixels having the corresponding relationship thus obtained as a condition, pixels having no corresponding relationship are associated with each other. For each pixel that has no correspondence,
The candidate Q is narrowed down based on the given condition and the constraint conditions from 0059 to 0061. The α value is divided for each pixel, and the association is optimized for each single pixel or for each pixel that does not have a correspondence divided by the same α value. If both sides of each pixel that are not associated are associated, the solution Qn is determined to be one. When a plurality of lines are arranged on the same α-value line, all combinations satisfying the constraint conditions of the items 0059 to 0061 are generated, and the combination with which the largest number of pixels are associated is selected. Also, apart from this method, Q2 is replaced with Q1, and
Items 3 to 0188 may be repeated.

Based on the combination of the corresponding pixels, 0
The three-dimensional data is obtained according to the processes of the items 064 to 0065, and texture mapping is performed.

In the three-dimensional data created in this way, when the resolution is not sufficient because the number of pixels is finite, for example, a small projected unclear character on a signboard or an originally flat surface In the case where a supposed surface has fine irregularities, for example, a three-dimensional data model or image prepared separately is used to appropriately correct the three-dimensional data to create more accurate three-dimensional data.

When it is clear that the portion surrounded by the boundary line, such as a rectangular parallelepiped or a cylinder, is formed of a flat surface or a simple curved surface, as shown in item 0035, one-to-one. It is also possible to create three-dimensional boundary data of the photographing target from the corresponding boundary line, apply an appropriate plane or simple curved surface by a linear or nonlinear square method, perform texture mapping, and use this as three-dimensional data. In an external space such as a building or a signboard, there are many parts composed of a flat surface or a simple curved surface. In such a case, the correction method described in this section is rather appropriate.

When the peripheral pixels among the uncorrelated pixels have pixels corresponding to the other screen, and the color data is completely different, this means that the folding pattern is located at a position that can be seen only from one viewpoint. Alternatively, it is highly possible that the concave portion is a fine concave portion. At the stage when the three-dimensional data is created, for pixels having color data that can only be found on one screen,
On the extension line connecting the coordinates of the pixel and the viewpoint, the shallowest position among the positions invisible from the other line of sight may be given as three-dimensional data, and the three-dimensional data may be corrected.

Embodiment 3 FIG. An embodiment will be described in which the positional relationship between the screen and the viewpoint is determined by associating a plurality of points in a plurality of images obtained by shooting the same shooting target. In this embodiment, the cameras A10 and B12 in FIG. 1 are replaced with images A and B obtained by photographing the same photographing target from different viewpoints with one camera. The slit light source 11, the map data 17, the reference frame 18, and the GPS 19 are unnecessary in this case.

The two images are read into the image storage area 13. The operator 16 designates three points (pixels) to be associated with the two images on the monitor 15 using a mouse or the like, and the two screens are displayed in the same coordinate system by the method described in paragraphs 0196 to 0200 or another method. Put on 3
The coordinates of the point and the initial values of 7 variables (or 8 to 14 variables) are obtained. When the one-to-one correspondence between the image areas is determined by the process of the paragraphs 0175 to 0180,
Based on the coincidence of the maximum value and the minimum value of the α value, five sets (or eight sets) of pixels in which the correspondence between the two screens is determined are automatically determined.

A process for automatically setting the initial value will be described. The image is taken such that the center line on the side of the screen is substantially horizontal, that is, substantially horizontal on the plane H = 0. When creating 3D data from aerial photographs, etc., it is as follows. Of the three points, two points near the H coordinate, two pairs are P1
(X1, Y1, H1), P2 (X2, Y2, H2), P2 and P3 (X3, Y3, H
3) as pixels n1 (V1, Z1) and n2 (V2, Z1)
2) In n2 and n3 (V3, Z3), the point at which the line connecting n1 and n2 intersects the Z axis is n4 (0, Z4), and the point at which the line connecting n2 and n3 intersects the Z axis is n5 ( 0, Z5).

[0197]

(Equation 4)

The point P6 becomes a point P6 '(0, 0, H6).
The points P1, P2, and P3 are translated without changing the H coordinate. Further, with the H axis as an axis, points P4 and P5 are point P4 '(0, Y4', H
4), point P5 '(0, Y5', H5), and points P1 ', P2', P3 '
Are rotated so that the left and right coincide with each other, and the point P1 '(X1', Y1 ', H
1 '), point P2' (X2 ', Y2', H2 '), point P3' (X3 ', Y3', H3 ')
Get.

In the case of photographing an external space as described later in the fifth embodiment, the coordinates of the three known points and the viewpoint are represented by the Y-axis, the line connecting the point P6 and the viewpoint, and the ground surface is represented by the plane H = Convert to a coordinate system with 0. By the above method, the coordinate system is converted such that the Y axis and the Z axis are located on substantially the same plane and the V axis is substantially horizontal to the plane H = 0 when viewed from the viewpoint.
(Xs, Ys, Zs) was calculated by the method described in Item 077, and
Derive initial values of variables.

When the indoor space is photographed, the basic composition is obtained based on the “stereoscopic data creation method” to determine the position of the screen and the viewpoint, or any three points on the basic composition created from one image. Are determined, and the positions of the screen and the viewpoint are determined in the same manner as in the external space. It is also possible to record approximate shooting points and directions and estimate and determine the initial values of seven variables from them. Alternatively, the operator 16 may correct the position on the screen with the mouse or the like on the monitor 15 and set the initial value.

First, by using two images which show a total of five points, three pixels showing three shooting target points whose coordinates are known and two pixels judged to correspond on both the A and B screens, An embodiment for obtaining the positional relationship between the coordinates, the screen, and the viewpoint will be described. Using three points whose coordinates on the screen are known, the V axis and X
Assuming that the axes are parallel and the Y axis and the Z axis are located on the same plane, the coordinates of the three points are transformed, and
The coordinates of the shooting target point and the position (coordinates) of the viewpoint on the image are obtained by the equation of item 7, and are obtained by using the same coordinate system (X, Y,
In H), an approximate value of 14 variables is obtained, and this is used as an initial value.

The number of variables is 14 as shown in items 0080 to 0083.
Twelve conditional expressions are obtained by arranging the point to be photographed, the viewpoint, and the pixel on the same line, and two conditional expressions are obtained in the other two sets of pixels corresponding to the A and B2 screens. The solution is sought.

The target for evaluating the validity of the approximate solution is | Ym−Ymp ”for pixels for which the coordinates of the shooting target point are known.
| / (Distance between viewpoint p and shooting target point m) (where m =
1, 2, 3, p = a or b, Ymp "defines only the correspondence relationship between X = Xm and H = Hm (Y coordinate in m = 1, 2, 3) on the extension of the line connecting the viewpoint and the pixel mp. For each pixel set, | αam−θbm |, m = 4, 5 when the α value of the pixel ma on the A screen is tan θam and the α value of the pixel mb on the B screen is tanθbm for each pixel set .

Therefore, the evaluation formula for confirming the validity of the approximate solution in these 14 conditional expressions is MAX [| θam−θbm |, m
= 4,5, | Ym-Ymp "| / (distance between viewpoint p and shooting target point m) m = 1, 2, 3, p = a or b].
If the photographing target point is photographed, 16 conditional expressions are obtained and a solution is obtained.

Although the coordinates of the object to be photographed are unknown, A, B2
An embodiment will be described in which the relative positional relationship of the screen is obtained from eight sets of pixels that have been confirmed to correspond on the screen. Using any three points on the screen, the coordinates of the shooting target point to be projected are estimated by an arbitrary method, and an initial value is obtained in the same manner as in the previous section.

As shown in items 0084 to 1986, the number of variables is eight (7 if taken with the same camera and the same focus).
), And the corresponding pixels 8 on the two screens A and B (7)
Since a set of 8 (7) conditional expressions is obtained and they are independent of each other, a solution is obtained. So, 8
(7) An evaluation expression for confirming the validity of the approximate solution in the conditional expression is as follows.
nθam, when the α value of the pixel mb of the screen B is tanθbm,
MAX [| θam−θbm |, m = 1 to 8]. Note that nine or more sets of corresponding pixels may be obtained to reduce errors caused by pixel density. (M> 8)

A highly accurate approximate solution is obtained based on the evaluation formula as in the preceding two terms. If the value of the evaluation formula of the approximate solution of the N-ary simultaneous equations is set to another dimension, it is considered that the periphery of the solution forms a smooth curved surface in the N + 1-dimensional space if the evaluation formula is appropriate. If the value of the evaluation formula is 0 for the solution,
In other places, if the evaluation formula is set so as to take a positive value exceeding 0, if the first approximate solution is set in the region between the local maximum values around the solution, a slight variation is given to each variable, and An approximate solution closer to can be obtained. If the surface is a smooth surface within the range of the local maximum value, an approximate solution approaching zero can always be found in the vicinity.

In this case, it is not necessary to combine all three cases for each of the N variables by adjusting the minute fluctuation amount. For each variable, perform 3 cases of (3 × N-1) simulations in which the amount of minute fluctuation is adjusted only for each variable, and correct the values of only the variables in the case where the value of the evaluation formula approaches 0 most. Or change all variables for which the value of the evaluation expression has approached 0, or select the one in which the value of the evaluation expression has approached 0. If an approximate solution approaching 0 cannot be found, the amount of small fluctuation is further reduced, and (3 × N−1) simulations are repeated. A more desirable approximate solution can be obtained by gradually reducing the minute fluctuation amount. The calculation ends when the minute variation falls within the error range caused by the pixel roughness.

Only when the value of the evaluation expression is higher than the pixel density despite the fact that the minute fluctuation amount falls within the error range caused by the pixel roughness, the previous simulation process is set to “(3 × N− "1) ways" is read as "(maximum 3N-1) ways" and the processing is performed again. If a solution approaching 0 is found during the simulation, a maximum of (3N-1) simulations is performed again as a new approximate solution.

In this calculation process, the expressions of the items 0074 to 0077 are effective for setting an initial value inside the local maximum value from the viewpoint of the solution. Other methods having the same effect may be used. Although the value of this evaluation formula gradually approaches 0 by the calculation process of this section, it may be displayed and confirmed on the monitor 15 that the α value referred to in the section 0042 gradually approaches.

In the method of identifying the several pixels and obtaining the positions of the screen and the viewpoint, the measurement error is limited by the pixel density. However, in the method using the reference frame,
Since the vanishing point is obtained from a plurality of pixels based on the least square method, higher accuracy can be expected. The latter method is suitable when a slit light source is used. Note that a triangular pyramid shape having four or more points at which the relative three-dimensional coordinates of the intersections of the two lines are known may be used as a reference frame.

In an aerial photograph or the like, since the number of pixels is finite, it may be difficult to obtain detailed parts of the created three-dimensional data, for example, fine three-dimensional data of a small house or the like. In such a case, the use of the separately created three-dimensional data of the relevant part makes it possible to synthesize precise three-dimensional data of the entire ground surface.

Embodiment 4 FIG. An example in which three-dimensional data of a street is created will be described. Buildings, signboards, and other parts of the city where you want to create 3D data are all captured,
The cityscape is photographed from many points. At this time, for the part to be converted into three-dimensional data, the shooting location and the number of shots are determined so that shooting is performed from three points at different positions, at least two points.

The photographing posture is such that the camera is horizontally horizontal (although the elevation angle is arbitrary). The photographing point data is recorded using a GPS or a map, and the photographing time data is also recorded. At the same time, shooting time data is recorded in the image data. The map data 17 is stored in the form of electronic map data of a region where a photograph is to be taken.

[0215] When all the photographing of the street is completed and the image data, photographing point data and photographing time data are stored in the storage area 13, the computer 14 determines the one-to-one correspondence between the photographing point data and the image data based on the photographing time data. Then, the one-to-one correspondence between the photographing point and the image is added to the map data 17 and stored. It should be noted that the image data and the photographing point data may be integrated and stored in the storage area 13 and the map data 17 by radio or the like every time a photograph is taken.

When the operator 16 selects a plurality of images of the same object to be converted into three-dimensional data on the monitor 15,
Map data around the point where the image was captured is called from the map data 17 and displayed on the monitor 15. The operator 16 specifies (inputs) three points for which coordinates are to be obtained for each of the one image and the map using a mouse or the like. Thus, the X and Y coordinates of the three points and the viewpoint are obtained.

Further, the operator 16 sets the approximate H of the three points.
Enter the coordinates and the height of the photographer. The three points for obtaining the coordinates are selected from the corners of a building or the like, and the H coordinates are input with a rough guide, for example, the height of the first floor is 5 m and the second floor is 4 m or more in the case of a building. The H coordinate of the viewpoint may be +1.5 m on the ground. When the above operation is performed, the computer 16 creates three-dimensional data of the imaging target according to the second and third embodiments. Note that regardless of the second and third embodiments, any of the three-dimensional data creation methods described in the present invention, or other methods such as the “three-dimensional data creation method” and the sixth embodiment may be used.

The computer 14 combines all of the three-dimensional data created by the processes of the second and third embodiments from the plurality of images of the same object to be photographed, and completes the three-dimensional data of the entire cityscape. In synthesizing 3D data,
Whether to match the size based on the coordinates of the map data 17,
Alternatively, matching is performed with the same length of the same portion of the overlapping portion of the adjacent three-dimensional data.

[0219] In the external space, there are many moving objects such as people and cars, and when photographing the same subject from different viewpoints at different times, there are a plurality of image areas that are displayed only on one screen. There is. In this case, the shadowed part of the person, the car, or the like is also displayed only on the other screen. In such a case, an image area showing a moving object such as a person or a car is designated on a monitor by an interactive device, and a shadowed portion of three or more images is commonly used. Create 3D data from a set of screens to be projected.

[0220] A portion that reflects light, such as a glass window, is perceived as different objects to be photographed from images photographed from different viewpoints.
Therefore, for a reflection surface such as a glass surface, the operator 16
Needs to be input as a reflection surface for the corresponding image area on the monitor. (A symbol that is a reflection surface is given to each pixel of the image area and each point of the three-dimensional data.) In this case, a three-dimensional shape such as a plane is given, and the position is determined from the boundary line of the area. When an image is created from three-dimensional data, an image is obtained using color data of three-dimensional data corresponding to a line of sight reflected on the reflection surface from a viewpoint as color data of each reflection surface.

Embodiment 5 FIG. The terminal transmits a plurality of image data input to the server via the communication network to the server side, creates three-dimensional data on the server side, or obtains position information of a screen and a viewpoint in which a plurality of screens are placed in the same coordinate system. An embodiment will be described in which analysis is performed and the data is transmitted back to the terminal via the communication network.

A user inputs a plurality of images of the same object to be photographed indoors or outdoors from different points in the room. The terminal transmits this to the server via the communication network. As a communication network in this case, any network such as digital broadcasting and data communication of a portable terminal may be used other than the Internet.

Based on the received image data, the server creates three-dimensional data of the space to be photographed by the process of the second, third, or sixth embodiment or the “three-dimensional data creation method” or other methods. Then, the position information data of the screen and the viewpoint in which the screen is placed in the same coordinate system is generated. The server sends the created three-dimensional data or the generated position information data back to the terminal via the communication network. The terminal displays, on the monitor, a simulation image created based on an arbitrary viewpoint and line of sight input by the user from the received stereoscopic data, or creates stereoscopic data from the received position information data.

In order to perform the simulation, the user needs to obtain information on the viewpoint and gaze direction that he / she wants to see,
At the time of transmitting the image data of item 0221, the information regarding the part to be replaced and the part to be replaced may be transmitted to the server via the communication network at the same time. In this case, the server creates an image from the three-dimensional data based on the information on the designated viewpoint and line-of-sight direction, and sends the image to the terminal. Alternatively, three-dimensional data and image data obtained by replacing the three-dimensional data of only the part to be replaced with a three-dimensional data model of a designated product or the like are transmitted back to the terminal.

Embodiment 6 FIG. One straight line or curve on the surface of the imaging target (hereinafter, simply referred to as a line of the imaging target) is displayed as one straight line or curve on both the A and B screens.
The constraint conditions corresponding to the pixels on both the A and B screens are 00
6 as described in paragraphs 59 to 0061
There are conditions. Furthermore, the following hypothesis is given.

"If the line to be photographed is traced sequentially from the end, the change in the saturation and brightness of this line is reflected in the lines projected on both the A and B screens. The fine area constituting the line to be photographed The maximum value and the minimum value should be captured as the maximum value and the minimum value of the saturation and lightness in the color data of the pixels on the straight line or the curve of both the A and B screens.
Therefore, it is possible to associate the pixels having the maximum value or the minimum value of the saturation or brightness on the line in both the A and B screens with the color data. Similarly, a portion where the color data changes abruptly can be associated with the pixels adjacent to the boundary. (A pixel in such a state is hereinafter simply referred to as a singular point.)

Here, it is assumed that the definition of the singular point satisfies one of the following conditions. For any three pixels lined up with Pj-1, Pj, and Pj + 1 on the same α-value line, the saturation (or lightness) value of pixel Pj is Pj-1, Pj
+1 If the saturation (or brightness) of both pixels is higher or lower, this pixel Pj has a maximum or minimum value
(Singular point). For any three pixels lined up with Pj-1, Pj, and Pj + 1 on the same α-value line, the saturation (or lightness) value of pixel Pj is Pj-1, Pj
When the saturation (or brightness) of any one of the pixels is different from the saturation (or brightness) by a certain reference or more, this pixel Pj is set as a singular point. (See Fig. 8)

Pixels at both ends on the same α-value line are also singular points. Note that the above-mentioned line does not need to be a line that can be visually recognized on the screen, and may be a line arbitrarily drawn on the surface of the imaging target. Generally, it is not possible to specify a line corresponding to an arbitrary line on the screen A on a one-to-one basis on the screen B. However, considering lineA and lineB on the same plane Y = αX, pixels on lineB correspond to pixels on lineA. The same plane Y =
Pixels on lineA and lineB on αX can be made to correspond using the above-mentioned singularity. This makes it possible to perform association with a limited number of pixels, and it is possible to greatly reduce the amount of calculation.

Although the above hypothesis is a very natural hypothesis, if the surface of the object to be photographed has an extremely fine pattern compared to the pixel density, each pixel can project the object to be photographed so precisely. Therefore, the singularity may not be completely corresponded. Also in this case, a hypothesis is set that “the combination with the largest number of associated pixels is the association of the pixel to be obtained”. Furthermore, after repeatedly performing the above-mentioned association for each α value, the consistency of the correspondence relationship in the vertical line segment on the screen is checked, and a correction is made for a portion that does not satisfy the constraint condition, thereby providing a precise correspondence. Can be attached. This method will also be described later.

The following describes the image areas Ta and Tb corresponding one-to-one.
Is described. Note that a set WW of a set of pixels associated with each other
A set of pixels Wajl, Wblu (u = 1, 2,...) such as lv is a set in which elements (a set of pixels or pixels) have an order relationship. a and b represent a set of pixels on each of the screens A and B. As data unique to each pixel, there are indices fs and fm (integers from 0 to 4; initial values are all 0) indicating a singular point.

When the α value of the same α-value line is changed so as to change by one pixel on the Z-axis side of the image area Tb, the area between two adjacent same α-value lines is changed. Means that pixels are arranged one by one on the Z-axis side, but two pixels vertically (within a range of approximate β value) as it goes farther from the Z-axis (the larger the value of the X coordinate). More and more places are lined up.
In the image areas Ta and Tb of both the A and B screens, the right and left sides generally correspond to each other, so that one pixel on the Z-axis side of the screen B corresponds to two pixels at the back of the screen A. However, in many cases, two pixels at the back of the screen B correspond to one pixel on the Z-axis side of the screen A.

Therefore, hereinafter, "located on the same α value line" is redefined as follows. With respect to an arbitrary pixel Pb on one screen, a pixel located on the same α-value line on the other screen is a pixel having an α value different from the α value of the pixel Pb within a certain range. When there are a plurality of pixels within the range of the same α value and within the range of the β value within a certain range, a pixel closer to the α value of the pixel Pb is determined as a pixel located on the same α value line. "

[0233] Pixels on the same α-ray value in the image area Tb of the screen B are extracted as P1 to Pn in a row from the left.
These pixels are placed in a set Wbl (1 is a suffix for each α value). In any three pixels arranged in order of Pj-1, Pj, and Pj + 1,
If the saturation value of pixel Pj is lower or higher than the saturation of both Pj-1 and Pj + 1 pixels, or if it differs from Pj-1 or Pj + 1 by a certain value or more, use this as a singular point and minimize it. For values
fs = 1, fs = 2 for a local maximum value, and fs = 3 for a difference of a certain value or more. (If two or more conditions are satisfied, fs =
3 is assumed. Similarly, when the value of the brightness of the pixel Pj is lower or higher than the brightness of both the pixels Pj-1, Pj + 1, or Pj-1,
If it differs from any one of Pj + 1 by a certain value or more, this is set as a singularity. If the minimum value is fm = 1, if the maximum value is fm =
2. If it differs by a certain value or more, fm = 3. (If two or more conditions are satisfied, fm = 3.) By this operation, the pixels are narrowed down so that the association can be performed only with the singular point.

The following operation is performed for each singular point Pj. Pixels having the same α value as each singular point Pj are extracted from the image area Ta, and set to p1 to pm in order from the left, and these pixels are placed in a set Wajl. The operation of item 0233 is performed on each pixel of the set Wajl to give an fs value and an fm value. Of the pixels in each set Wajl,
Satisfy the constraints of 0059 to 0061,
In addition, all the pixels pk having the same fs value or the same fm value as the singular point Pj are extracted and placed in the candidate Q of the singular point Pj. If there is a concern about the problems described in the paragraphs after Sections 0225 to 0229, measures are taken only in Section 0234 by increasing the allowable range of the same color data.

A set of singularities Wblu (u = 1,2,...) Is obtained for all combinations of singularities, including a case where each singularity is included and a case where each singularity is not included, for all singularities belonging to the set Wbl.
.. M) are generated. Even if a set is generated for all combinations in this way, the number of the set is usually not so large because the target is limited to a singular point.

In each set Wblu, all combinations (including combinations corresponding to a plurality of candidates) of each singular point Pj belonging to the set Wblu and each pixel pk serving as its candidate Q are generated, and 0059 to 0061 of the combinations are generated. Set W of sets of pixels that may only correspond to those that satisfy the term constraint
It is generated as Wlv (v = 1, 2,..., M ′). Further, the set WW
In lv (v = 1, 2,..., m ′), the same color data is not associated between the singular points pa and pb and qa and qb associated on both screens A and B. Exclude sets that have both singularities.

In items 0235 to 0236, all possible combinations of the singular points of both the A and B screens on the same α-value line are generated as a set WWlv of sets of pixels. A single or a plurality of pixels (hereinafter simply referred to as a simple increase / decrease pixel group) in which the saturation and lightness of the color data are simply increased or decreased between the singular points are defined as one unit.
In both the A and B screens, the correspondence is made in simple increment / decrement pixel group units on the same α value line. The constraint conditions for the simple increase / decrease pixel group to correspond are as follows.

The corresponding singular point pa on both the A and B screens
And qa, and simple pixel groups sandwiched between pb and qb. The corresponding simple pixel group has pixels of the same color data. On the same α-value line, both the saturation and the brightness of the color data are simply reduced or simply increased to the right.

For example, in FIG. 10, even when a singular point appears only on one screen and it is impossible to associate it with a singular point on the other screen, it is associated with both A and B screens on both sides of the screen. Since a simple pixel group can be associated between singular points, there is only a limited combination of associations.

For example, in FIG. 11, even if there is a portion that can be seen only on one screen A and there is a singular point on screen A that does not correspond to the singular point on screen B, It is possible to associate a limited number of combinations in units of the sandwiched simple pixel group. Each set WWlv (v
= 1,2,..., M ′), all combinations of simple increase / decrease pixel groups satisfying the above constraint conditions are set WWwl
Generated as v (v = 1, 2,... m ").

If the association is made in the unit of the simple increase / decrease pixel group, the association of each pixel in each set WWwlv becomes 0
This can be performed uniquely according to paragraphs 059 to 0061. In this association, it is considered appropriate to perform the association using the set of pixels that are the closest to the color data, or to perform the association using the difference between the saturation and the lightness, or to use either or a combination. A method may be used.

In the correspondence of each pixel in the unit of simple increase / decrease pixel group, only a part of one simple increase / decrease pixel group corresponds to the simple increase / decrease pixel group of the other screen, or all pixels correspond to only one pixel of the other screen. It may be possible to respond.
This is automatically determined by the distribution of the color data of the simple increase / decrease pixel group to be associated. Note that portions where pixels of exactly the same color data continue are associated as separate areas in advance in section 0058.

A set WWwlv of each singular point and a simple increase / decrease pixel group
For each of the combinations of the corresponding pixels on the same α-value line of both the A and B screens
Generate as WWlv (v = 1, 2,... M "). Set WWWlv
(v = 1, 2,... m ") is the region Tb extracted in the 0233 term
Are all combinations of pixels on the same α-value line and pixels that may correspond to the area Ta. In order to reduce the amount of calculation, the set WWWlv whose corresponding number of pixels is less than a certain number may be excluded at this stage.

The processes from the items 0233 to 0243 are performed on the entire image area Tb by changing the α value of the same α value line so as to change by one pixel on the Z-axis side of the image area Tb, and the set WWWlv ( l = 1,2, ... L, v = 1,2, ...
..M "). Further, as described in item 0230, since there is a possibility that there are pixels that have not yet been associated in the region Tb, they are aligned on the same α-value line or For each pixel of, the process of the terms 0233 to 0243 is executed, and the set WWWlv (l = L + 1, L + 2,... L ′)
Generate As described above, all combinations that may associate all the pixels in the region Tb with the pixels in the region Ta are set WW
Wlv (l = 1, 2,... L ').

Next, a combination in which the number of corresponding pixels is the largest among those satisfying the constraint conditions of 0059 to 0061 is extracted so that the created three-dimensional data becomes a plane or a smooth curved surface. Within the area Tb, a set WWWlv (l = h, h + 1, v = 1, 2,...) On any two adjacent identical α value lines within a range where the area Ta and the α value overlap. m "), and extracts all the combinations into a set WTvv '(v = 1, 2,...).
n, v ′ = 1, 2,... n ′). At this time, in order to reduce the amount of calculation, a place where the number of generated sets is small may be selected.

For each set WTvv ′, a set WW of sets of pixels on the same α-value line adjacent to the two same α-value lines
Wlv ", those which satisfy the constraints of 0059 to 0061 are selected, and if they are single, the set WTv
v ′ together with the set WTvv′1, and if there is more than one set WTvv ′
Sets WTv obtained by adding each set WWWlv "to the
Generate v′g (g = 1, 2,...).

Incidentally, there is a case where an uncorrelated pixel is left even in the area Ta in some cases.
When it is not possible to determine whether or not the constraint conditions of items 9 to 0061 are satisfied, the pixels of the uncorrelated area Ta are made to correspond to appropriate pixels of the area Tb.
It is determined whether or not it is possible to satisfy the constraint condition of item 0061. If possible, set WT including the new association for the unassociated pixel
vv'g. Exclude if not possible. (Do not sort.)

Hereinafter, the same operation is performed for successive sets WWWlv on the same α-value line, and the pixels satisfying the constraints 0059 to 0061 for all the pixels of the region Tb (all of the set WWWlv). In the above operation, if there is a possibility that a set WWWlv that matches when the constraints of items 0059 to 0061 are strictly applied may not be found, an arbitrary combination is generated. From 0059 to 00
You may relax the constraint condition of 61 items. It is also assumed that a set WWWlv to be matched cannot be found because a corresponding pixel is not found due to a setting error between the screen and the viewpoint,
In such a case, the set WWWlv having the largest number of pixels satisfying the constraint conditions of the items 0059 to 0061 may be selected.

Of the sets WTvv'g (g = 1, 2,...) Generated in the terms 0245 to 0248, the set having the largest number of pixels is determined by the combination of the corresponding pixels in the regions Ta and Tb. I do. If the three-dimensional data created by the operation after item 0250 has a shape different from the actual shooting target,
The operator 16 selects another appropriate set WTvv′g from the shape of the three-dimensional data displayed on the monitor 15.

[0250] As shown in items 0230 to 0232, there is a possibility that a pixel that has not yet been associated is left in the area Ta. Given a correspondence relationship of the set WTvv'g selected in the item 0249, the region Tb is set so that the pixels in the region Ta that are not associated with each other satisfy the constraint conditions from the items 0059 to 0061. Are made to correspond to the pixels in.

When there are a plurality of candidates, they are associated with the pixel having the closest color data. Alternatively, they may be associated with each other on the same α-value line in proportion to the difference in saturation or brightness.
Further, when there is a pixel that does not satisfy all the constraints from the items 0059 to 0061 in the entire set of corresponding pixels (usually not), as long as the correspondence between the surrounding pixels is not broken, The correspondence is finely modified so that the constraint condition can be satisfied more.

Although a large number of sets are generated by the processes from the items 0233 to 0252, in the first stage, the pixels are associated only with the singular points, and the pixels other than the singular points are associated with each other. , The simple increase / decrease pixel group is associated between the singular points sandwiching the .alpha., And is uniquely performed on a pixel-by-pixel basis.
059
The amount of computation is considerably reduced. Further, the generated set basically satisfies the constraints from the items 0059 to 0061, and three-dimensional data having a smooth curved surface is generated. Furthermore, even in the event that stereoscopic data is created based on an image that causes a visual illusion, the operator 16 can select an appropriate set WTvv'g on the monitor 15, which can be dealt with. is there.

[0253] Three images A, B,
If C is used, the accuracy can be further improved. As described above, the corresponding pixel sets pa and pb and qa in both the A and B screens
qb is the same α value line lineA-b (viewpoints Sa, Sb and pixels pa, pb, q
The intersection of plane A and plane A passing through a and qb. The same applies hereinafter. ) And lin
Located on eB-a. Pixel pc on screen C corresponding to pixels pa and pb is located at the intersection of lineC-a and lineC-b. Pixel qa,
The pixel qc on the screen C corresponding to qb is located at the intersection of lineC-a, lineC-b ', and lineC-a' and lineC-b 'having different α values. If the pixels pb and qb have the same color data, it is not possible to determine whether the pixel corresponding to the pixel pa is the pixel pb or qb with only two screens,
If pixel qb is selected as the corresponding pixel, the corresponding pixel on screen C is lineC-a corresponding to pixel pa and pixel qb
Is the pixel located at the intersection of lineCb ′ corresponding to. (Position of □ in FIG. 9) The pixel at this intersection is the pixel pc
Is a pixel that is completely unrelated to the pixel pa.
Otherwise, it turns out that the selection of the pixel qb was incorrect.

In this way, by using three or more images, it is possible to prevent the selection of the corresponding pixel from the pixels on the same α-value line by mistake. In the case of determining the correspondence only of the singular point, such an error can be particularly avoided, and accurate correspondence can be expected. Further, if the pixels pa and pb take values such as the maximum value on the line A-c and the line B-c, the pixel pc on the screen C
Has a high possibility of taking a value such as a maximum value, and a clear distinction can be made.

When three screens in the same coordinate system are used, when selecting the candidate Q of the 0233 item in each α value, pixels having fs> 1 to fm> 1 have the same color on the same α value line. Even for a singular point that is data, the consistency is confirmed as described above, and erroneous candidates are excluded. Thereby, the candidate Q in the item 0234 can be further narrowed down.

In both A and B screens, it is assumed that the color data of the pixels to be handled in both A and B screens are slightly different due to lighting and other conditions. In such a case, the color data is corrected. At the stage where the singular point of item 0234 is associated, at the singular point having only one pixel as the candidate Q of the entire image area, the color data of the two associated singular points is compared, and the color data of each corresponding pixel is compared. If the average value and the standard deviation of the set (saturation of screen A-saturation of screen B) and (brightness of screen A-brightness of screen B) have a statistically significant tendency, the difference is determined. The color data of the pixels on one screen may be corrected by the average value of. This makes it possible to more accurately associate pixels other than the singular points.

When it is confirmed that the front or a part of the pixel located at the boundary between the regions Ta and Tb corresponds to the operation up to the item 0256, the boundary is used to perform the processing according to the first embodiment. It is also possible to create three-dimensional data of a boundary line.

[0258] A method for efficiently creating fine stereoscopic data with a smaller amount of calculation from the associated pixels will be described.
A known method is generally used to find an intersection on an extended line connecting a corresponding pixel and a viewpoint by the principle of triangulation. However, in this method, since the pixel density is finite, the two lines rarely intersect strictly, and even if the closest approach point is obtained, the amount of calculation increases.

A plane connecting an arbitrary straight line in which pixels are arranged vertically on the screen A to the viewpoint Sa, and each extension connecting a pixel on the screen B corresponding to each pixel arranged in a straight line on the screen A and the viewpoint Sb. A plurality of points at which the lines intersect are used as the coordinates of the three-dimensional data to be captured, and color data of pixels of the B screen corresponding to each of the three-dimensional points is given. Perform the same operation again with the screens A and B reversed. In this way, three-dimensional points having color data and coordinates by the total number of corresponding pixels on both the A and B screens are formed. With such a calculation method, the calculation amount is considerably reduced.

[0260]

According to the method of 1) using only one slit light source, it is possible to create high-precision stereoscopic data of a photographing object with a simpler apparatus. As a result, the prototype model is photographed to create an industrial mold, the three-dimensional data of the product is created, and an image of the product viewed from an arbitrary angle specified by the user using the Internet is created. Many types of 3D data that are presented or distributed to the user via the Internet, and the application placed on the user's terminal allows the product to be displayed as a moving image and used as a means for promoting the product By preparing, the combination can be simulated and provided for learning, games, and various other uses. Clothes and the like can be modeled or mannequins to create three-dimensional data and appeal three-dimensionally.

By generating stereoscopic data from a plurality of images of the same photographing target space in 2), stereoscopic data with considerably higher precision, which does not reach 1), can be obtained. In the present invention, since three-dimensional data is converted into image area units, it is possible to easily perform a simulation in which a part is replaced with another part, thereby achieving a significant cost reduction as compared with the conventional CG production. It is characterized by high reality because it uses a real image.

[0262] A virtual reality of walking around in a virtual space to be photographed, which is converted into three-dimensional data, is realized. You can also build an Internet shopping mall where you can walk through the shopping streets in the actual town, go into the actual stores, shop, and view the products from any angle. In the case of research, etc., it is possible to convert the shooting target space from the image taken by the camera into three-dimensional data and view the room as virtual reality from various angles later, so that the surveyed site can be verified with more reality. Will be possible. It can also be used as a presentation to those who did not go to the site.

The interior of the user's personal home is changed
In the case of renovation, it is possible to simulate the case where some parts such as furniture and wallpaper are replaced with other products, and you can realize in advance what kind of atmosphere will be achieved if you purchase and distribute products. This makes it possible to visually grasp it, and it is an effective means for promoting sales of building materials such as furniture, curtains, and wallpaper. Such simulations can be provided on the Internet or the like, and CD-ROM, DVD-ROM
For example, simulation software may be distributed, or simulation may be performed at a terminal in a store. You can also take pictures of your home from the outside of the building and perform simulations such as renovation.

3) 4) A plurality of images obtained by photographing a space to be photographed, such as a large-scale building or a cityscape, by a system for creating three-dimensional data of the object space from a plurality of images arbitrarily photographed in the same photographing space. Thus, the stereoscopic data of the photographing target can be created. You can create highly realistic guidance systems, computer game stages, and backgrounds that actually walk around the city. It can provide tourist information in virtual reality in European and American cities and travel experiences as entertainment. The historical cityscape can be saved as highly realistic 3D data. It is also possible to virtually restore the cityscape from past landscape photos.

A simulation system for city planning can be constructed. It can also be used to investigate illegal buildings. Recently, many buildings have been converted into 3D data at the time of design due to the spread of CAD, and if local governments acquire and store such 3D data at the time of confirmation application, increasingly sophisticated city data will be constructed. Angles where there is no foothold to shoot, places with high danger, etc.
This makes it possible to create images with a high degree of reality, even from angles that would otherwise be difficult to shoot, and this 3D data material can be integrated into scenes such as movies, used for virtual experience learning, etc. Is assumed.

In converting the aerial photograph into three-dimensional data on the ground surface, the operation conventionally using a high-priced system can now be performed with an inexpensive system. Models can also be made. Also, when conducting a site survey or a building survey, three-dimensional data of the entire object to be photographed can be obtained, so that it can be used for simple surveying means or a survey for examining the inclination of a building.

In the above description, the camera is assumed to be a digital camera. However, a film camera and a scanner may be combined, or a video camera or any other camera may be used. Instead of the GPS, a PHS, a car navigation, or any other means for obtaining location information may be used. P
Using a communication system such as an HS, the captured image data and the imaging location data may be wirelessly transmitted, and the imaging time data may not be necessary.

The confirmation and other operations performed by the operator may not be necessary. For example, the elevation angle of the camera to be photographed is automatically recorded, the position of the coordinates used for setting the initial value of the variable is automatically selected at the end of the image division, one-to-one correspondence, and the coordinates of the height are used as the map data. It may be automatically calculated from the photographing point data and the elevation angle of the camera. When the image division and the one-to-one correspondence are completed, the position of the screen and the viewpoint may be automatically obtained from the positional relationship of the boundary line of the same area, and the initial value may be obtained therefrom.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration of each element in Examples 1, 2, 3, and 4.

FIG. 2 is a diagram showing a flow of analysis in a second embodiment.

FIG. 3 is the same α described in the paragraphs 0042 to 0043.
FIG. 9 is a diagram illustrating a positional relationship between pixels on the other screen to which adjacent pixels on the value line correspond.

FIG. 4 is a diagram showing a concept of obtaining a position on a screen of a virtual pixel described in the items 0059 to 0061.

FIG. 5 is a diagram illustrating a concept of obtaining a position on a screen of a virtual pixel described in the items 0064 to 0065.

FIG. 6 is a diagram showing the positional relationship between three points whose coordinates are known, a screen, and a viewpoint described in the items 0074 to 0077.

FIG. 7 is a diagram showing a positional relationship between a time at which light starts to be sensed and a pixel, a time at which sensing is completed, a pixel, and a virtual pixel in the items from 0155 to 0160.

FIG. 8 shows the saturation,
FIG. 7 is a diagram illustrating three pixels arranged on the same α-value line for comparing brightness.

FIG. 9 is a diagram showing a method of checking the right or wrong of the correspondence relationship of the pixels associated with any two images using the remaining images when three or more images are used in the items 0253 to 0256. It is.

FIG. 10 is a diagram illustrating that, in the sections from 0237 to 0240, a simple pixel group can be associated even when a singular point that appears only on one screen exists.

FIG. 11 is a diagram illustrating that, in the sections from 0237 to 0240, a simple pixel group can be associated even when there is a portion that can be seen only on one screen.

[Explanation of symbols]

Nak, nbp, and the like displayed in the figure are all pixel names.

Claims (21)

[Claims] 1. A slit light beam projected on the surface of an object to be measured is simultaneously measured from photographing means located at a plurality of viewpoints having different positions, and a vanishing point is measured in a same frame as a reference frame. Obtain the viewpoint of the photographing means, superimpose and compare the basic composition derived from the plurality of photographing means, specify the position of each point on the surface of the measurement target from the slit light beam projected on the measurement target, and A method for creating three-dimensional data of an object that has become, provided that pixels to be corresponded in a plurality of images measured from the plurality of photographing means are located on the same plane as a viewpoint of the plurality of images. A three-dimensional data creation method, wherein a correspondence between pixels is obtained, and three-dimensional data is created based on the relationship. 2. A slit light beam projected on the surface of an object to be measured is simultaneously measured from photographing means located at a plurality of viewpoints at different positions, and a vanishing point is measured in a reference frame in the same space. To determine the viewpoint of the photographing means, superimpose and compare the basic compositions derived from the plurality of photographing means, arrange the viewpoint and the screen in the same coordinate system, or handle five or more points whose coordinates on the photographing target are unknown. The viewpoint and the screen are arranged in the same coordinate system based on the positional relationship of the pixels to be measured, and the position of each point on the measurement target surface is specified from the slit light beam projected on the measurement target to create stereoscopic data of the object to be captured A straight line or curve in a three-dimensional space formed by the intersection of a plane or a curved surface for each screen created by connecting each pixel and a viewpoint in a plurality of images measured from the plurality of photographing means. On the basis of the Volumetric data creation method characterized by creating a body data. 3. A still image in a space whose spatial structure is unknown is captured, a viewpoint position with respect to the still image is determined based on a positional relationship between straight lines in the captured still image, and the information of the still image and the viewpoint is obtained. Is a method of determining a one-to-one correspondence between image areas in three-dimensional data creation for creating three-dimensional data of a spatial structure in the still image based on the An image region correspondence determination method, comprising dividing a plurality of still images taken from a position into a plurality of image regions, and determining a one-to-one correspondence between the image regions in the plurality of images. 4. A plurality of still images in a space whose spatial structure is unknown, and two or less pixels whose coordinates in the captured still image are known only in correspondence with three or four known pixels, or A method of determining a position of a viewpoint with respect to the still image from a positional relationship of 7 or less pixels for which only the correspondence is known, and determining a relative positional relationship between the plurality of still images and the viewpoint in the same coordinate system; Each screen composed of images is placed in the same coordinate system, a plurality of still images taken from different positions are divided into a plurality of image regions, and a one-to-one correspondence between the image regions in the plurality of images is determined. Image area correspondence determination method. 5. The method according to claim 3, wherein pixels corresponding to each of the two screens correspond to one-to-one image areas of the two screens. The two points located on the same plane with the viewpoint, and the right and left positional relationship on one screen at the two points located on the same plane must be the same left / right relationship or the same point on the other screen A three-dimensional data creation method, wherein a correspondence relationship in pixel units is obtained as a condition, and three-dimensional data is created based on the relationship. 6. A method according to claim 4, wherein the corresponding pixels on the boundary between the one-to-one corresponding image areas in the one-to-one corresponding image areas of the two screens are determined. On the condition that they are located on the same plane as the viewpoints of a plurality of screens, and that the right and left positional relationship on one screen is the same on the other screen at two points located on the same plane. A three-dimensional data creation method characterized by identifying a correspondence relationship in pixel units and creating a part of three-dimensional data of a shooting target. 7. The method for creating three-dimensional data according to claim 5, wherein a plurality of images obtained by photographing the same photographing target space based on positional information of a photographing point obtained at the time of photographing and map information of the external space are obtained. A stereoscopic data generating method for selecting three or more approximate coordinates or a basic composition in the external space to be photographed, and setting the positions of the plurality of screens and the viewpoint in the same coordinate system; 8. The three-dimensional data creating method according to claim 5, wherein each of the two screens is located on the same plane as a viewpoint of each of the two screens arranged in an arbitrary same coordinate system. For the pixels arranged on the upper straight line, the saturation of color data, the maximum value of lightness,
A three-dimensional data creation method, wherein pixel-by-pixel correspondence is performed on the condition that the order in which pixels having a local minimum value or a unique value are aligned in each screen. 9. The three-dimensional data creation method according to claim 5, wherein when two or more screens arranged on the same coordinate system are used to associate pixels, any two of the screens are displayed. The associated pixels and the pixels associated with the screen other than the two screens are on the same plane as the viewpoints of the screens other than the two screens, the respective viewpoints of the two screens, and the associated pixels. A three-dimensional data creation method, wherein the position of a pixel on the screen other than the two pixels is determined using the color data, and the quality of the association of the associated pixel is determined from the color data. 10. A plurality of photographing means, a slit light source for projecting a slit light beam on a surface of an object to be measured, and a plurality of viewpoints having different positions of the slit light beam projected on the surface of the object to be measured. And simultaneously calculating the vanishing point in the same space and measuring the vanishing point in the same space to obtain the viewpoint of the photographing means, and superimposing the basic composition derived from the plurality of photographing means. Generating means for identifying the position of each point on the measurement target surface from the slit light beam projected on the measurement target and creating three-dimensional data of the object to be photographed, the generation means, Pixels to be corresponded in a plurality of images measured from a plurality of photographing means are determined on the condition that they are located on the same plane as the viewpoints of the plurality of images. Stereoscopic data creating apparatus characterized by creating a three-dimensional data Zui. 11. A plurality of photographing means, a slit light source for projecting a slit light beam on a surface of an object to be measured, and a plurality of viewpoints having different positions of the slit light beam projected on the surface of the object to be measured. And simultaneously calculating the vanishing point in the same space and measuring the vanishing point in the same space to obtain the viewpoint of the photographing means, and superimposing the basic composition derived from the plurality of photographing means. The viewpoint and the screen are arranged in the same coordinate system, or the viewpoint and the screen are arranged in the same coordinate system based on the positional relationship of five or more corresponding pixels whose coordinates on the object to be photographed are unknown, and projected onto the measurement object. Generating means for identifying the position of each point on the surface to be measured from the slit light beam and creating three-dimensional data of the object to be photographed, the generating means being measured from the plurality of photographing means Duplicate Creating three-dimensional data based on a straight line or a curve in a three-dimensional space formed by the intersection of a plane or a curved surface for each screen created by connecting each pixel in the image and the viewpoint. apparatus. 12. A plurality of photographing means, and a plurality of still images in a space having an unknown spatial structure are fetched from the plurality of photographing means, and a viewpoint to the still image is determined based on a positional relationship of straight lines in the fetched still images. Calculating means for calculating the position of the image, and determining a one-to-one correspondence between image regions in three-dimensional data creation for creating three-dimensional data of a spatial structure in the still image based on the still image and viewpoint information. Generating means for generating three-dimensional data of the object that has become, wherein the generating means divides a plurality of still images taken from different positions into a plurality of image areas by placing each screen composed of images in the same coordinate system. A three-dimensional data creating apparatus that determines one-to-one correspondence between image areas in the plurality of images, and creates three-dimensional data based on the one-to-one correspondence. 13. A plurality of photographing means, and a plurality of still images in a space having an unknown spatial structure are fetched from the plurality of photographing means, and the fetched still images correspond to three or four pixels whose coordinates are known. Calculating means for determining the position of the viewpoint with respect to the still image from the positional relationship between two or less pixels whose relationship is known only, or seven or less pixels whose only correspondence is known; Generating means for obtaining the positional relationship in the same coordinate system and creating three-dimensional data of the object to be photographed, wherein the generating means places each screen composed of images in the same coordinate system, and The method is characterized in that a plurality of photographed still images are divided into a plurality of image regions, a one-to-one correspondence between the image regions in the plurality of images is determined, and three-dimensional data is created based on the correspondence. 3D data creation device. 14. The generation unit further includes: a pixel to be corresponded in an image region corresponding to each of the two screens in a one-to-one correspondence, being located on the same plane as a viewpoint of each of the two screens; At the two points located on the same plane, the left and right positional relationship on one screen is the same left and right relationship on the other screen or the same relationship is determined on a pixel-by-pixel basis, 14. The three-dimensional data creating apparatus according to claim 12, wherein three-dimensional data is created based on the three-dimensional data. 15. The image forming apparatus according to claim 1, further comprising: a corresponding pixel on a boundary between the image areas corresponding to each other in the one-to-one correspondence of the two screens is the same as the viewpoint of the plurality of screens. On the condition that the two points located on the same plane and the two points located on the same plane have the same left and right positional relationship on one screen, the corresponding relationship on a pixel basis must be the same. 14. The three-dimensional data creating apparatus according to claim 13, wherein the three-dimensional data of a part to be photographed is identified and created. 16. The image processing apparatus according to claim 1, wherein the generation unit further selects a plurality of images obtained by shooting the same shooting target space based on position information of a shooting point obtained by the position measurement unit at the time of shooting and map information of the external space. 16. The three-dimensional object according to claim 14, wherein three or more approximate coordinates or a basic composition in the external space to be photographed are obtained, and the positions of the plurality of screens and the viewpoint are set in the same coordinate system. Data creation device. 17. The apparatus according to claim 17, wherein the generating unit is further arranged on a straight line on each of the two screens, which is located on the same plane as a viewpoint of each of the two screens arranged on an arbitrary coordinate system. For pixels, the saturation of color data, the maximum value of brightness, the minimum value, or the order in which pixels having unique values are lined up is matched on each screen on a condition that pixel-by-pixel correspondence is performed. The three-dimensional data creation device according to claim 15 or 16, which performs the three-dimensional data creation. 18. The image processing apparatus according to claim 18, further comprising: when three or more screens arranged in the same coordinate system are used for associating pixels with each other, the pixels associated with any two screens and the two or more screens. Pixels associated with screens other than the two, using the viewpoints of the screens other than the two, the viewpoint of each of the two and the corresponding pixels on the same plane, using the two 17. The three-dimensional data creating apparatus according to claim 15, wherein a position of a pixel on the screen other than the image data is obtained, and the quality of the association of the associated pixel is determined based on the color data. 19. Three-dimensional data creation using at least one terminal and the three-dimensional data creation method according to at least one of claims 1, 4, 5, 6, 7, 8, and 9 connected to the terminal via a communication network. A server for performing processing, wherein the terminal is configured to input a plurality of still image data in a space whose space structure is unknown, and the plurality of still image data in the space whose space structure is unknown is transmitted to the communication network. And means for receiving data and information transmitted from the server, and displaying or outputting the received data, wherein the server has a spatial structure transmitted from the terminal. Means for receiving a plurality of still image data in an unknown space; and a plurality of three-dimensional data in an imaging target space and the plurality of still images placed in the same coordinate system from the plurality of received still image data. Send means for creating at least one of position information of a plurality of screens and viewpoint including images, stereoscopic data and position information created in the terminal,
Means for displaying or outputting from the terminal. 20. At least one terminal and connected to the terminal via a communication network, and the three-dimensional data creation method according to at least one of claims 1, 4, 5, 6, 7, 8 and 9. A server for performing processing, wherein the terminal is configured to input a plurality of still image data in a space whose space structure is unknown, and the plurality of still image data in the space whose space structure is unknown is transmitted to the communication network. And means for receiving data and information transmitted from the server, and displaying or outputting the received data, wherein the server has a spatial structure transmitted from the terminal. Means for receiving a plurality of still image data in an unknown space; and a plurality of three-dimensional data in an imaging target space and the plurality of still images placed in the same coordinate system from the plurality of received still image data. Means for creating positional information of a plurality of screens and viewpoints composed of images; means for creating image data based on the created stereoscopic data of the imaging target space and the positional information of the viewpoint; and sending the created image data to the terminal. Means for transmitting and displaying or outputting from the terminal. 21. At least one terminal and connected to the terminal via a communication network, and the three-dimensional data creation method according to at least one of claims 1, 4, 5, 6, 7, 8, and 9. Means for inputting a plurality of still image data in a space whose spatial structure is unknown and position information of a viewpoint, and a plurality of still images in a space whose input spatial structure is unknown. Means for transmitting data and position information of the viewpoint to a server via the communication network; receiving data and information transmitted from the server; and displaying or outputting the received data. Means for receiving a plurality of still image data in a space whose spatial structure is unknown and position information of a viewpoint transmitted from the terminal; Means for creating three-dimensional data of the elephant space; means for creating image data based on the created three-dimensional data of the imaging target space and the position information of the viewpoint transmitted from the terminal; and sending the created image data to the terminal. Means for transmitting and displaying or outputting from the terminal.