JP4511147B2 - 3D shape generator - Google Patents

Description

The present invention relates to a three-dimensional shape generation apparatus that obtains a three-dimensional shape of an object based on a moving image or a continuous still image obtained by a camera and generates a three-dimensional map.
In particular, the present invention is capable of generating a three-dimensional shape that can easily and accurately determine a three-dimensional shape of an object without requiring expensive equipment or enormous effort and cost. Relates to the device.

In a car navigation apparatus or the like, a 3D map generation technique for displaying an object three-dimensionally is used. In a 3D map, for example, three-dimensional 3D information including the height direction is displayed for an object such as a road or a building, so that the object can be easily and accurately grasped. There will be no useful map information.
Conventional three-dimensional map generation technology includes surveying the ground surface from an aircraft with a laser, etc., acquiring three-dimensional data on the ground surface, measuring the detailed parts in combination with three-dimensional data based on parallax photographed from the ground, and then 3DCG (computer graphics) is generated, and further, the texture of an object photographed from the ground is cut out and pasted on 3DCG (see Patent Documents 1-3).

JP 2000-066583 A JP 2002-098538 A JP 2003-114614 A

The conventional three-dimensional map technology as described above has a problem that an expensive device is required and a three-dimensional map cannot be created without enormous effort and cost.
Here, even if the video is shot with a normal inexpensive camera, the video contains 3D information, and it is possible to extract 3D information necessary for generating a 3D map from the video. Also, it was possible to extract three-dimensional information from parallax photographed by two cameras. Therefore, it is theoretically possible to shoot an object while moving a normal camera mounted on a vehicle or the like, extract 3D information therefrom, and generate a 3D map.

However, simply taking a picture with a normal camera moves the camera to shoot, so it is subject to disturbances such as shaking, the coordinates are shaken, rotated, and stable images cannot be acquired and extracted as positions. The original information cannot be extracted with high accuracy.
In particular, since the camera position and the viewpoint angle change every moment as the vehicle or the like equipped with the camera moves, it is necessary to accurately measure the three-dimensional position. In addition, if the movement of the camera in the three-axis directions cannot be extracted accurately, a highly accurate three-dimensional map cannot be generated.
In addition, three-dimensional extraction from parallax photographed by two cameras is limited in accuracy, and some technique and means for reducing the error are required.

Therefore, as a result of earnest research, the inventor of the present application automatically detects a sufficient number of feature points from a plurality of frame images of a moving image, further automatically tracks feature points between frames, and overlaps many feature points. By calculating and obtaining the camera position and rotation angle with high accuracy, it was conceived that even an inexpensive camera can extract highly accurate 3D data and generate a highly accurate 3D map.
That is, the present invention has been proposed in order to solve the problems of the prior art, and does not require expensive equipment, and is an inexpensive camera and, in principle, a single camera for a vehicle. Take a picture of the surroundings by loading it, or take a picture of the surroundings with a person holding the camera, analyze the video, extract 3D information from it, and generate a 3D shape using image processing technology An object of the present invention is to provide a three-dimensional shape generation apparatus capable of generating a three-dimensional map.

To achieve the above object, the three-dimensional shape generation apparatus of the present invention includes an image acquisition unit that acquires an image from the camera to obtain a video or continuous still images provided to the moving body acquired by the image acquisition unit An image recording unit for temporarily recording an image, a three-dimensional position of a point or a small area corresponding to each frame from the image data read from the image recording unit, a three-dimensional position of the camera, and three axes of the camera A vector detection unit that calculates a camera vector by performing statistical processing so that the error distribution is minimized, and a three-dimensional rotation of the camera determined by the vector detection unit and a three-axis rotation Based on a camera vector additional image recording unit that records all frame images to which camera vector data is added, and each frame image and the camera vector obtained for each frame, a plurality of frames are recorded. Using a large number of parallaxes obtained from a frame image or an average value of a number of optical flows obtained from a plurality of frame images, an overlap calculation is performed for each point on the minute surface of the object in the image, and the three-dimensional shape of the object is obtained. The multiple parallax calculation unit to be generated and the three-dimensional shape generation unit that continuously generates the three-dimensional shape of the object in the multiple parallax calculation unit to form a three-dimensional shape.

Further, the three-dimensional shape generation apparatus of the present invention includes an image acquisition unit that acquires an image from the camera to obtain a video or continuous still images provided to the moving body is temporarily recorded to the image acquired by the image acquisition unit From the image recording unit and the image data read out from the image recording unit, a three-dimensional position of a point or small area corresponding to each frame, a three-dimensional position of the camera, and a three-axis rotation of the camera are divided into a plurality of frames. And a vector detection unit that obtains a camera vector by performing statistical processing so that the error distribution is minimized, and a correction signal for correcting the original image is generated from the camera vector obtained by the vector detection unit. A correction signal generation unit, a normalized image conversion unit that records a normalized image corrected by the correction signal, a three-dimensional position of the camera in each frame, and a plurality of normalized images corrected by the correction signal. Based on a plurality of parallaxes obtained from a plurality of frame images, or an average value of a plurality of optical flows obtained from a plurality of frame images, and performing an overlap calculation on each point of the minute surface of the object in the image. And a three-dimensional shape generation unit that generates a three-dimensional shape.

Further, the three-dimensional shape generation apparatus of the present invention, the vector detecting section, the the image data read from the image recording unit, it should become a feature point in an image or a small area image is automatically extracted automatically A feature point extraction unit to be designated, a feature point correspondence processing unit for obtaining a correspondence relationship between the frames of the automatically designated feature points, and a three-dimensional position coordinate and a camera position of the feature point for which the correspondence relationship has been obtained. A feature point / camera position calculation unit to be obtained by calculation and a plurality of three-dimensional positions of the feature points are calculated, statistical processing is performed so that the distribution of the position of each feature point and the camera position is minimized, and a feature having a large error An error minimizing unit that detects a point, deletes it, performs recalculation based on another feature point, and minimizes the entire error is provided.

Further, the three-dimensional shape generation apparatus of the present invention, the vector detecting section, the image to the feature points are characterized in an image automatically detects obtains the corresponding point of the feature point in each frame image, the camera vector Focusing on the n-th and n + m-th frame images Fn and Fn + m used for the calculation, the unit calculation is repeated by setting n and m to predetermined values, and the distance from the camera to the feature point in the image The feature points are classified into a plurality of stages, and the frame interval m is set so as to increase as the distance from the camera to the feature points increases. N is continuously advanced with the progress of the image, the calculation is continuously advanced, and the same feature point is repeatedly calculated several times at each stage of n and m. Position error is Integration by scale adjustment to minimize, perform distance calculation, delete feature points with large error distribution, and recalculate to improve the accuracy of calculation at each feature point and camera position It is configured.

Further, the three-dimensional shape generation apparatus of the present invention, the three-dimensional shape generation section, based on the camera vector already obtained, by repeating the operation while moving the small region in the frame image, the image of all Multiple parallax three-dimensional shape generation unit that extracts a large number of parallaxes for a small area and generates a three-dimensional shape with less error by overlapping calculation based on the parallax, and a three-dimensional shape generated by the multiple parallax three-dimensional shape generation unit And a multi-parallax three-dimensional shape output unit that outputs a three-dimensional map by integrating them in the same coordinate system as the camera.

Further, the three-dimensional shape generation apparatus of the present invention, the three-dimensional shape generation unit, an image acquired by the image acquisition unit, a planar converted to images from the camera viewpoint to be perpendicular to the plane of the desired object A specified direction plane conversion processing unit, a target plane optical flow extraction specifying unit that calculates an optical flow of a video in a certain area by using the plane-converted video and extracts an optical flow of a desired target plane, and the target plane A non-objective surface exclusion unit that excludes optical flows other than the above, a target plane is extracted as a plane having a shape, and a plane shape and coordinate acquisition unit that gives three-dimensional coordinates to the plane, and calculation while moving the certain area Is repeated, and for all desired objects in the region, a three-dimensional shape generating unit for generating a three-dimensional shape based on the plane configuration, and those plane configurations A planar reconfigurable three-dimensional shape output unit for outputting the dimension shape as a three-dimensional map, it is constituted with a.

Further, the three-dimensional shape generation apparatus of the present invention is detected, the vector detecting unit detects a vector of the moving camera, the camera separately from the movement vector of the moving object moving, in the same manner as the camera vector Configured to detect the vector of the moving object from the moving camera and detect the posture of the moving object with respect to the static coordinate form as a three-dimensional vector by adding the vector of the moving vector of the camera and the moving vector of the moving object. It is.

Further, the three-dimensional shape generation apparatus of the present invention is composed of a plurality of cameras provided in the same moving body separately from or serving as the camera provided in the moving body, and overlapped visual fields set so that the visual fields overlap. A unit of time obtained by calculation from a camera, a field-of-view overlap image recording unit that records a plurality of images at the same time acquired by the field-of-view overlap camera, and a plurality of images at the same time that overlap the field of view. A continuous three-dimensional image generation unit that generates a three-dimensional image having a unique coordinate system, and a camera vector of the moving camera obtained from the vector detection unit by using the three-dimensional image obtained by the continuous three-dimensional image generation unit. And a coordinate integration unit that converts and integrates the coordinate system into a common coordinate system with the moving camera system, and an integrated 3D shape generation unit that generates an object and a surrounding 3D shape in the integrated coordinate system, E was there as a constituent.

Furthermore, the three-dimensional shape generation apparatus of the present invention comprises a plurality of cameras as a camera provided in the movable body, a short-range partial detailed image acquiring unit for mainly capturing a portion that requires a short distance of detailed data, A parallel image recording unit that records a plurality of images at the same time set in such a manner that the fields of view are overlapped, acquired in a plurality of cameras, in a form that can be recalled in association with each other, and an image with overlapping fields of view A short-distance partial three-dimensional shape generation unit that detects parallax and generates a detailed three-dimensional shape of the short-distance portion, and detects parallax from the overlapping images of the field of view, and other than the moving body including the camera Another moving object detection unit that detects a moving object, another moving object exclusion unit that excludes the parallax data of the other moving object from the parallax data of the images with overlapping fields of view, and the three-dimensional shape generation unit, , The short distance part three Coordinate calibration integration unit that calibrates the scale with the measured length by parallax from the three-dimensional shape of the target object that overlaps the original shape generation unit, and integrates each coordinate system, and integration that records the integrated data The coordinate recording unit and a general image display unit that displays the integrated data are provided.

The 3D map generation apparatus of the present invention includes the 3D shape generation apparatus according to the present invention, and is configured to generate a 3D map by continuously generating a wide range of 3D shapes of objects. It is.

According to the three-dimensional shape generation apparatus of the present invention as described above, a sufficient number of feature points are automatically detected from a plurality of frame images of a moving image, and a large number of feature points are automatically tracked between frames. It is possible to calculate the camera position and the rotation angle with high accuracy by performing overlap calculation on the feature points. Therefore, even if it is a normal inexpensive camera, a person takes a picture while moving with the camera, or takes a surrounding image from a moving vehicle carrying a camera, etc. Can be sought.
Then, using the 3D camera position as a reference, the plane part in the camera image is extracted from multiple parallax, optical flow, etc., the plane is extracted with texture, and the 3D shape of the plane is reconstructed. A three-dimensional map can be generated by expressing a three-dimensional shape and continuously executing the three-dimensional shape.

This makes it possible to generate a three-dimensional map at low cost and high accuracy without requiring expensive equipment as in the prior art and enormous effort and expense.
In the present invention, not only images obtained by moving cameras but also information from parallax from two cameras installed in parallel can be used in combination.
For example, not only by images acquired by moving cameras, but also by combining 3D images acquired from cameras installed in parallel with the present invention, temporally continuous 3D acquired by a fixed camera Images can be three-dimensionally integrated by the camera vector according to the present invention and displayed in a single coordinate system.

Hereinafter, a preferred embodiment of a three-dimensional shape generation apparatus according to the present invention will be described with reference to the drawings.
Here, the following three-dimensional shape generation apparatus of the present invention is realized by processing, means, and functions executed by a computer in accordance with instructions of a program (software). The program sends commands to each component of the computer, and performs predetermined processing and functions as shown below, such as automatic extraction of feature points, automatic tracking of extracted feature points, calculation of three-dimensional coordinates of feature points, camera Perform vector operations. Thus, each process and means in the three-dimensional shape generation apparatus of the present invention are realized by specific means in which a program and a computer cooperate.
Note that all or part of the program is provided by, for example, a magnetic disk, optical disk, semiconductor memory, or any other computer-readable recording medium, and the program read from the recording medium is installed in the computer and executed. The The program can also be loaded and executed directly on a computer through a communication line without using a recording medium.

[First embodiment]
First, with reference to FIGS. 1-12, 1st embodiment of the three-dimensional shape production | generation apparatus which concerns on this invention is described.
FIG. 1 is a block diagram showing a schematic configuration of a three-dimensional shape generation apparatus according to the first embodiment of the present invention.
The three-dimensional shape generation apparatus 10 according to the present embodiment shown in the figure includes, for example, a camera mounted on a moving vehicle (in principle, one unit) to generate three-dimensional information about a desired object. The change in coordinates due to the movement is accurately extracted.
In order to accurately extract the camera coordinates, a technique for detecting video feature points in an image and tracking the movement thereof is used. By automating the detection of feature points and automating tracking, manual work can be greatly omitted.

Here, in order to obtain the camera position from the feature points in the image, for example, the feature points are tracked in the image so that there are 6 to 7 or more feature points at the same time. Then, the epipolar geometry is used for these feature points, and the camera position can be obtained by calculation. However, with about 6 to 7 feature points, the accuracy of the obtained camera position is insufficient.
Therefore, in the present embodiment, the number of feature points to be extracted and tracked is sufficiently increased, and multiple parallaxes are acquired by using a sufficient number of frames, so that excess feature points and the number of frames are obtained. . Using multiple parallaxes with surplus feature points and surplus number of frames, statistical processing is performed, repeated calculation is repeated, error distribution of camera position is obtained, and statistical processing is used to obtain highly accurate camera position as 3D coordinates Try to ask. In this way, the camera position of each frame can be obtained with high accuracy. If the camera position is obtained with high accuracy, then the three-dimensional coordinates for all the pixels in the image can be obtained by a technique for obtaining the three-dimensional coordinates from the parallax.

Specifically, as illustrated in FIG. 1, the three-dimensional shape generation apparatus 10 includes an image acquisition unit 11, an image recording unit 12, a vector detection unit 13, a camera vector addition image recording unit 14, and multiple parallaxes. A calculation unit 15 and a three-dimensional shape generation unit 16 are provided.
The image acquisition unit 11 acquires, for example, a moving image or a continuous still image captured by a camera loaded on a moving body such as a vehicle or a camera that a person moves with a hand. The moving body (moving object) provided with the camera includes a moving person, an automobile, a ship, an aircraft, a robot, a moving machine, and the like.
The image recording unit 12 temporarily records the image acquired by the image acquisition unit 11.
From the image data read out from the image recording unit 12, the camera vector detection unit 13 has a point corresponding to each frame or a three-dimensional position of a small region corresponding to each frame, and a three-dimensional position of the camera and the camera. The three-axis rotation and the like are overlapped in a plurality of frames, and statistical processing is performed so as to minimize the error distribution to obtain with high accuracy.
The vector detection unit 13 will be described in detail later with reference to FIG.

The camera vector additional image recording unit 14 records all frame images to which the data of the camera three-dimensional position and the three-axis rotation camera vector obtained by the vector detection unit 13 are added.
The multiple parallax calculation unit 15 has a sufficiently large number of parallaxes obtained from a plurality of frame images or a sufficiently large number of frames obtained from a plurality of frame images from a camera vector obtained in units of each frame and each frame image. Using the average value of the optical flow, overlap calculation is performed for each point on the minute surface of the object in the image to generate a three-dimensional shape of the object.
The three-dimensional shape generation unit 16 continuously repeats the generation of the shape of the object performed by the multiple parallax calculation unit 15 to form a three-dimensional shape.

In the three-dimensional shape generation apparatus 10 configured as described above, an image is acquired by an in-vehicle camera or the like, and a camera vector is obtained with high accuracy by using a sufficiently large number of points that have a correspondence relationship between frames. In principle, if there are 6 to 7 feature points, the 3D coordinates can be obtained, but in this embodiment, for example, by using a sufficiently large number of points such as about 100 points, the distribution of the solution is obtained. Each vector is obtained from the distribution by statistical processing, and a camera vector is obtained as a result.
From the three-dimensional position of the camera thus obtained and the three-axis rotation of the camera, it is added as data to each frame image, and a plurality of parallaxes obtained from a plurality of frame images, that is, multiple parallaxes, have already been acquired. The three-dimensional shape of the object can be obtained by calculation from the three-dimensional position of the camera. In this way, the three-dimensional shape of the object can be acquired.

According to the three-dimensional shape generation apparatus 10 as described above, not only the in-vehicle camera but also, for example, a person can freely hold the camera in his / her hand to photograph the object, and calculate the camera vector from the image after photographing. And the three-dimensional shape of the photographed object can be obtained from the camera vector.
Then, by repeating the above processing, a wide range of three-dimensional shapes, that is, three-dimensional maps are generated.
FIG. 2 shows a display example of a three-dimensional shape (three-dimensional map) generated in the present embodiment.

Next, details of the vector detection unit 13 of the three-dimensional shape generation apparatus 10 shown in FIG. 1 will be described.
FIG. 3 is a block diagram showing a schematic configuration of the vector detection unit 20 (vector detection unit 13 shown in FIG. 1) according to the first embodiment of the present invention.
As shown in the figure, the vector detection unit 20 includes a feature point extraction unit 21, a feature point correspondence processing unit 22, a feature point / camera position calculation unit 23, and an error minimization unit 24.

The feature point extraction unit 21 automatically extracts and automatically designates a small region image to be a feature point in the recorded image based on the image data read from the image recording unit 12 of the three-dimensional shape generation apparatus 10. .
The feature point correspondence processing unit 22 automatically obtains the correspondence relationship by automatically tracking the feature points automatically extracted in each frame image between the frames.
The feature point / camera position computing unit 23 obtains the three-dimensional position coordinates of the feature point for which the correspondence relationship is obtained, and automatically obtains a camera vector corresponding to each frame image from the three-dimensional position of the feature point.
The error minimizing unit 24 performs statistical processing so that the distribution of the position of each feature point and the camera position is minimized by calculating a plurality of feature points, detects a feature point having a larger error, and deletes it. Then, by recalculation, the overall error is minimized. Thereby, the camera position direction subjected to the error minimization process statistically processed so as to minimize the distribution of the solutions of the plurality of camera vectors is automatically determined.

There are several methods for detecting a camera vector from feature points of a plurality of images (moving images or continuous still images), but the vector detection unit 20 of the present embodiment shown in FIG. By automatically extracting a number of feature points and automatically tracking them, a three-dimensional vector and a three-axis rotation vector of the camera are obtained by epipolar geometry.
By taking a sufficient number of feature points, camera vector information is duplicated, and an error can be minimized from the duplicated information to obtain a more accurate camera vector.

Here, the camera vector refers to a vector of degrees of freedom possessed by the camera.
In general, a stationary three-dimensional object has six degrees of freedom of position coordinates (X, Y, Z) and rotation angles (Φx, Φy, Φz) of the respective coordinate axes. Therefore, the camera vector refers to a vector of six degrees of freedom of the camera position coordinates (X, Y, Z) and the rotation angles (Φx, Φy, Φz) of the respective coordinate axes. When the camera moves, the direction of movement also enters the degree of freedom, which can be derived by differentiation from the above six degrees of freedom.
As described above, the detection of the camera vector by the vector detection unit 20 of the present embodiment is such that the camera takes six degrees of freedom for each frame and determines six different degrees of freedom for each frame. is there.

Hereinafter, a specific camera vector detection method in the vector detection unit 20 will be described with reference to FIG.
First, a feature point extraction unit 21 automatically extracts a point or a small area image to be a feature point from a frame image appropriately sampled, and a feature point correspondence processing unit 22 extracts feature points between a plurality of frame images. The correspondence is automatically obtained. Specifically, more than a sufficient number of feature points that are used as a reference for detecting a camera vector are obtained. Examples of feature points between images and their corresponding relationships are shown in FIGS. In the figure, “+” is a feature point that is automatically extracted, and the correspondence is automatically tracked between a plurality of frame images (see correspondence points 1 to 4 shown in FIG. 6).
Here, for feature point extraction, as shown in FIG. 7, it is desirable to specify and extract a sufficiently large number of feature points in each image (see circles in FIG. 7). For example, about 100 feature points are extracted. Extract points.

Subsequently, the feature point / camera position calculation unit 23 calculates the three-dimensional coordinates of the extracted feature points, and calculates the camera vector based on the three-dimensional coordinates. Specifically, the feature point / camera position calculation unit 23 includes a sufficient number of feature positions existing between successive frames, a position vector between moving cameras, a three-axis rotation vector of the camera, and each camera. In this embodiment, for example, the camera motion is solved by solving the epipolar equation from the epipolar geometry of the 360-degree all-round image. (Camera position and camera rotation) are calculated.

Images 1 and 2 shown in FIG. 6 are images obtained by performing Mercator expansion of 360-degree all-round images. When latitude φ and light θ are assumed, points on image 1 are (θ1, φ1) and points on image 2 are ( θ2, φ2). The spatial coordinates of each camera are z1 = (cos φ1 cos θ1, cos φ1 sin θ1, sin φ1), z2 = (cos φ2 cos θ2, cos φ2 sin θ2, sin φ2). When the camera movement vector is t and the camera rotation matrix is R, z1 ^T [t] × Rz2 = 0 is the epipolar equation.
By providing a sufficient number of feature points, t and R can be calculated as a solution by the method of least squares by linear algebra calculation. This calculation is applied to a plurality of corresponding frames.
In FIG. 6, in order to make the processing in the vector detection unit 20 easier to understand, a 360-degree spherical image obtained by combining images taken by one or a plurality of cameras is developed by Mercator projection, which is a map projection. However, the actual vector detection unit 20 does not necessarily have to be a developed image by the Mercator projection.

Next, the error minimizing unit 24 calculates a plurality of vectors based on each feature point according to a plurality of calculation equations based on the plurality of camera positions and the number of feature points corresponding to each frame, Statistical processing is performed so that the distribution of the position of each feature point and the camera position is minimized to obtain a final vector. For example, for multiple camera positions, camera rotations, and multiple feature points, the least square method is estimated using the Levenberg-Marquardt method, and the errors are converged to obtain the camera position, camera rotation matrix, and feature point coordinates. .
Further, feature points having a large error distribution are deleted, and recalculation is performed based on other feature points, thereby improving the accuracy of computation at each feature point and camera position.
In this way, the position of the feature point and the camera vector can be obtained with high accuracy.

8 to 10 show examples of three-dimensional coordinates of feature points and camera vectors obtained by the vector detection unit 20. 8 to 10 are explanatory diagrams illustrating the vector detection method of the present embodiment, and are diagrams illustrating the relative positional relationship between the camera and the object obtained from a plurality of frame images acquired by the moving camera. .
FIG. 8 shows the three-dimensional coordinates of the feature points 1 to 4 shown in the images 1 and 2 in FIG. 6 and the camera vector that moves between the images 1 and 2.
9 and 10 show a sufficiently large number of feature points, the positions of the feature points obtained from the frame image, and the position of the moving camera. In the figure, a circle mark that continues in a straight line at the center of the graph is the camera position, and a circle mark located around the circle indicates the position and height of the feature point.

Here, in the calculation in the vector detection unit, as shown in FIG. 11, in order to obtain more accurate feature points and camera position three-dimensional information at a high speed, a plurality of feature points are selected according to the distance from the camera to the feature points. Is set to repeat multiple operations.
Specifically, the vector detection unit automatically detects feature points that are characteristic in video in the image, and obtains corresponding points of the feature points in each frame image. Focusing on the n + m-th two frame images Fn and Fn + m, unit calculation is performed, and unit calculation with n and m appropriately set is repeated.
m is the frame interval, and the feature points are classified into a plurality of stages according to the distance from the camera to the feature point in the image. The distance from the camera to the feature point is set so that m becomes larger. It is set so that m is smaller as the distance to is shorter. This is because the change in position between images is less as the distance from the camera to the feature point is longer.

Then, while sufficiently overlapping the classification of the feature points by the m value, a plurality of stages of m are set, and as n progresses continuously with the progress of the image, the calculation proceeds continuously. Then, the overlap calculation is performed a plurality of times for the same feature point in each step of n and m.
In this way, by performing unit calculation focusing on the frame images Fn and Fn + m, a precise camera vector is calculated over a long time between each frame sampled every m frames (frames are dropped). However, in m frames (minimum unit frames) between the frame images Fn and Fn + m, a simple calculation that can be performed in a short time can be performed.

If there is no error in the precision camera vector calculation for every m frames, both ends of the camera vector of the m frames overlap with the Fn and Fn + m camera vectors that have been subjected to the high precision calculation. Accordingly, m minimum unit frames between Fn and Fn + m are obtained by a simple calculation, and both ends of the camera vector of the m minimum unit frames obtained by the simple calculation are Fn and Fn + m obtained by high precision calculation. The scale adjustment of m consecutive camera vectors can be made to match the camera vectors.
Accordingly, it is possible to speed up the arithmetic processing by combining simple arithmetic operations while obtaining a highly accurate camera vector having no error.

Here, there are various simple calculation methods depending on the accuracy. For example, when (1) many feature points of 100 or more are used in high-precision calculation, the minimum number of simple calculation is about ten. (2) Even if the number of the same feature points is the same as the feature points and camera positions, innumerable triangles are established there, and equations for that number are established. By reducing the number of equations, it can be simplified.
In this way, integration is performed by adjusting the scale so that the error of each feature point and camera position is minimized, distance calculation is performed, and feature points with large error distribution are deleted, and other features are added as necessary. By recalculating the points, the calculation accuracy at each feature point and camera position can be improved.

And the camera vector calculated | required in this way can be displayed in the produced | generated three-dimensional map.
For example, as shown in FIG. 12, the image from the in-vehicle camera is developed in a plane, the corresponding points on the target plane in each frame image are automatically searched, and the corresponding points are combined to match the target plane. A combined image is generated and displayed in the same coordinate system. Then, the camera position and the camera direction can be detected one after another in the common coordinate system, and the position, direction, and locus can be plotted.

As described above, according to the 3D shape generation apparatus according to the present embodiment, a sufficient number of feature points are automatically detected from a plurality of frame images of a moving image, and feature points are automatically tracked between frames. Thus, it is possible to calculate the camera position and the rotation angle with high accuracy by performing overlap calculation on a large number of feature points. Therefore, even if it is a normal inexpensive camera, a person takes a picture while moving with the camera, or takes a surrounding image from a moving vehicle carrying a camera, etc. Can be sought.
Then, using the 3D camera position as a reference, the plane part in the camera image is extracted from multiple parallax, optical flow, etc., the plane is extracted with texture, and the 3D shape of the plane is reconstructed. A three-dimensional map can be generated by expressing a three-dimensional shape and continuously executing the three-dimensional shape.

This makes it possible to generate a three-dimensional map at low cost and high accuracy without requiring expensive equipment as in the prior art and enormous effort and expense.
Note that the 3D shape generation apparatus according to this embodiment can use not only images obtained by a moving camera but also information from parallax from two cameras installed in parallel. For example, by combining this embodiment with a 3D image acquired from a camera installed in parallel as well as a video acquired with a moving camera, a temporally continuous tertiary acquired with a fixed camera The original image can be three-dimensionally integrated by the camera vector according to the present embodiment, and can be integrated and displayed in one coordinate system.

[Second Embodiment]
Next, a second embodiment of the three-dimensional shape generation apparatus according to the present invention will be described with reference to FIG.
FIG. 13 is a block diagram showing a schematic configuration of a three-dimensional shape generation apparatus according to the second embodiment of the present invention.
The three-dimensional shape generation apparatus 10 of this embodiment shown in the figure omits the camera vector additional image recording unit 14 and the multiple parallax calculation unit 15 in the three-dimensional shape generation apparatus 10 of the first embodiment shown in FIG. The correction signal generation unit 17 and the normalized image conversion unit 18 are provided.
The image acquisition unit 11, the image recording unit 12, and the vector detection unit 13 are the same as those in the first embodiment.

The correction signal generation unit 17 generates a correction signal for correcting a shake or the like of the original image from the result obtained by the vector detection unit 13.
The normalized image conversion unit 18 converts and records a normalized image free from shaking and the like by the correction signal generated by the correction signal generation unit 17.
Then, the three-dimensional shape generation unit 16 has a sufficiently large number of parallaxes obtained from a plurality of frame images, or a plurality of plural images obtained from a three-dimensional position of the camera in each frame and a plurality of normalized images corrected for shaking. Using the average value of a sufficiently large number of optical flows obtained from the frame image, overlap calculation is performed for each point on the minute surface of the object in the image,
Generate a three-dimensional shape of the object.
As described above, in the three-dimensional shape generation apparatus 10 shown in FIG. 2, the three-dimensional shape is generated after correcting the shake of the image based on the camera vector obtained by the vector detection unit 13. A more accurate three-dimensional map can be generated.

[Third embodiment]
Next, a third embodiment of the three-dimensional shape generation apparatus according to the present invention will be described with reference to FIG.
FIG. 14 is a block diagram showing a schematic configuration of a three-dimensional shape generation apparatus according to the third embodiment of the present invention.
The present embodiment shown in the figure is another embodiment of the three-dimensional shape generation apparatus 10 shown in FIG. 1, and employs a parallax method as the three-dimensional shape generation unit 16.
Specifically, the three-dimensional shape generation unit 16 of the present embodiment includes a multi-parallax three-dimensional shape generation unit 16-1 and a multi-parallax three-dimensional shape output unit 16-2.

The multi-parallax three-dimensional shape generation unit 16-1 repeats the calculation based on the already obtained camera vector and the image information obtained while moving the small area, and has a sufficiently large number for all the small areas. The parallax is extracted, and a three-dimensional shape with less error is generated by the overlap calculation.
The multi-parallax three-dimensional shape output unit 16-2 integrates the three-dimensional shape obtained by the multi-parallax three-dimensional shape generation unit 16-1 into the same coordinate system as the camera, and outputs a three-dimensional map. .
The target 3D shape can be acquired if the camera position and rotation are determined in an image captured by a human hand or an image acquired by capturing from a vehicle carrying a camera.
Therefore, if the parallax of an object common to a plurality of frames is obtained as in the three-dimensional shape generation unit 16 of the present embodiment, multiple parallaxes equal to or greater than the number of frames can be obtained.
As a result, it is possible to perform an overlap calculation, to obtain a three-dimensional shape without an error of the object, and to generate a more accurate three-dimensional map.

[Fourth embodiment]
Next, a fourth embodiment of the three-dimensional shape generation apparatus according to the present invention will be described with reference to FIGS.
FIG. 15 is a block diagram showing a schematic configuration of a three-dimensional shape generation apparatus according to the fourth embodiment of the present invention.
The present embodiment shown in the figure is another embodiment of the three-dimensional shape generation apparatus 10 shown in FIG. 1 and adopts an optical flow (Opt. F.) system as the three-dimensional shape generation unit 16.
Specifically, the three-dimensional shape generation unit 16 of the present embodiment includes a designated direction plane conversion processing unit 16a, an optical flow calculation unit 16b, a target plane optical flow extraction designation unit 16c, and a non-purpose surface exclusion unit 16d. A plane shape and plane coordinate acquisition unit 16e, a 3D shape generation unit 16f, and a plane reconstruction 3D shape output unit 16g.

The designated direction plane conversion processing unit 16a converts the image acquired by the image acquisition unit 11 (see FIG. 1) into an image from a camera viewpoint perpendicular to the surface of a certain object.
The optical flow calculation unit 16b calculates an optical flow in a certain area by calculation.
The target plane optical flow extraction designation unit 16c extracts an optical flow of a desired target plane based on the optical flow obtained in a certain region.
The non-target surface exclusion unit 16d excludes an optical flow other than the target plane from which the optical flow is extracted.
The plane shape and coordinate acquisition unit 16e extracts the target plane as a plane having a shape and gives three-dimensional coordinates to the plane.
The three-dimensional shape generation unit 16f repeats the calculation while moving a certain region, extracts all the regions, and generates a three-dimensional shape having a planar configuration.
The plane reconstructed 3D map output unit 16g outputs a 3D map having a plane configuration generated by the 3D shape generation unit 16f.

FIG. 16 is an image diagram of a plane of the object extracted by the optical flow in the three-dimensional generation unit 16 of the present embodiment. As shown in the figure, if the vertical distance of the plane to which each object belongs from the standard position of the camera is D, all planes can be separated and extracted as a plurality of parallel plane groups. At this time, as with the roadside tree shown in the figure, for a curved object, even for a single object, even if it is a curved object having points or faces that do not ride on one plane, Is treated as a collection of a plurality of planes, and given a distance from the reference plane (street tree plane (1) in the figure), it can be grasped as information on one object belonging to that plane.
In this manner, by detecting the camera position and direction, the position of the object can be specified from the camera positions of a plurality of images, and a three-dimensional map can be reconstructed from the planar development image. Therefore, it is possible to automatically generate a three-dimensional map for a region in the traveled range simply by photographing with an in-vehicle camera or the like.

As described above, the three-dimensional shape generation unit 16 of the present embodiment employs an optical flow method as a method of generating a three-dimensional map that does not depend on map generation based on parallax.
By performing plane conversion on the video acquired by the camera, it is possible to calculate a plane optical flow from the plane-converted image. From the rule that planes parallel to the moving direction of the camera have the same optical flow, only pixels on the same plane can be extracted.
Thereby, plane conversion processing can be performed from the camera position obtained with high accuracy by the three-dimensional shape generation apparatus 10 of the present invention to obtain an accurate plane, that is, a target plane having the same optical flow can be defined. Then, it becomes possible to paste the texture of the plane on the definition plane.
Furthermore, by repeating the same method, defining planes one after another, and reconstructing planes with textures attached three-dimensionally, it is possible to generate a three-dimensional map with a considerably small amount of data. Become.

[Fifth embodiment]
Next, a fifth embodiment of the three-dimensional shape generation apparatus according to the present invention will be described with reference to FIG.
FIG. 17 is an explanatory diagram conceptually showing a moving object vector detection method in the three-dimensional shape generation apparatus according to the fifth embodiment of the present invention.
As shown in the figure, in this embodiment, the movement vector of the moving object is detected by the same method as the vector detection in the three-dimensional shape generation apparatus 10 shown in FIG.

Specifically, the three-dimensional shape generation apparatus 10 of the present embodiment detects a moving camera vector as in the above-described embodiments.
Next, apart from the camera, the moving object from the moving camera is detected by detecting the moving vector of the moving moving object (for example, another traveling vehicle) in the same way as detecting the moving vector of the camera. The vector of is detected.
Then, the posture of the moving object with respect to the stationary coordinate form is detected as a three-dimensional vector by adding the vector of the moving vector of the camera and the moving vector of the moving object.

Since the motion is relative, the same method can be applied regardless of whether the camera moves or the object moves.By adding the camera vector and the apparent vector of the moving object viewed from the camera, The relative value of the movement vector can be detected.
Thereby, for example, a known length in a moving object can be found and converted to an absolute length. In addition, the absolute length is obtained from the road surface if the absolute length is measured by parallax from a plurality of cameras, or if the vehicle is in contact with the road surface, for example, from a part touching an object of known length. Thus, the absolute length of the moving object can be acquired.
In this way, in the present embodiment, the absolute length of the moving object can be obtained.

[Sixth embodiment]
Next, a sixth embodiment of the three-dimensional shape generation apparatus according to the present invention will be described with reference to FIGS.
FIG. 18 is a block diagram showing a schematic configuration of a three-dimensional shape generation apparatus according to the sixth embodiment of the present invention.
FIG. 19 is a perspective view schematically showing a specific example in a case where a field-of-view overlap camera and a wide-angle camera provided in the three-dimensional shape generation apparatus according to the present embodiment are fixedly provided.

The present embodiment shown in these drawings is another embodiment of the three-dimensional shape generation apparatus 10 shown in FIG. 1, and acquires a plurality of images with overlapping fields of view, 3D shapes can be integrated.
Conventionally, means for obtaining a temporally continuous three-dimensional image from overlapping images obtained by a plurality of cameras is known. However, since the three-dimensional images obtained from time to time are independent coordinates, it is not possible to relate images with different frames as they are.
Therefore, in the present embodiment, a camera vector is obtained by a feature point tracking method using continuous images in the time axis direction, and a temporally continuous camera vector is obtained. It is possible to generate a three-dimensional whole image by uniformly expressing the common coordinates.

Specifically, the three-dimensional shape generation apparatus 30 of the present embodiment includes a field-of-view overlap camera unit 31, a field-of-view overlap image recording unit 32, a continuous three-dimensional image generation unit 33, and a moving camera coordinate system image conversion unit 34. The coordinate calibration integration unit 35 and the integrated three-dimensional shape generation unit 36 are provided.
The field-of-view overlap camera unit 31 is a camera that is stacked on the same moving object separately from or in addition to the camera loaded on the moving object, and is configured with a plurality of cameras so that the fields of view overlap.
The field-of-view overlap image recording unit 32 records the images of the same time acquired from the field-of-view overlap camera unit in association with each other.
The continuous three-dimensional image generation unit 33 generates a three-dimensional image composed of a unique coordinate system at a unit time, which is obtained by calculation from a plurality of images at the same time with overlapping visual fields.
Note that the field-of-view overlap camera unit 31, the field-of-view overlap image recording unit 32, and the continuous three-dimensional image generation unit 33 are field-of-view overlap as in a stereo vision camera (camera 1 shown in FIGS. 19A and 19B). It can be configured by a camera that can acquire an image.

The moving camera coordinate system image conversion unit 34 and the coordinate integration unit 35 use the three-dimensional image obtained by the continuous three-dimensional image generation unit 33 as the camera vector of the moving camera obtained from the vector extraction unit 13 (see FIG. 1). Combine, convert to a common coordinate system with the moving camera system, and integrate.
Here, in the present embodiment, the camera from which the movement vector is extracted by the vector extraction unit 13 is, for example, a wide-angle camera 2 (FIGS. 19A and 19B) that can capture a wide range of images such as a 360-degree all-round image. The camera 2) shown in FIG.
The integrated three-dimensional shape generation unit 36 is a coordinate system integrated by the coordinate integration unit 35 and generates a target object and a surrounding three-dimensional shape.
With the configuration as described above, it is possible to integrate three-dimensional images of each time unit acquired from overlapping cameras and combine them in a single coordinate system to display a three-dimensional whole image.

As a camera for acquiring a short-distance three-dimensional image with overlapping fields of view, for example, “DIGICROPS” manufactured by PGR of Canada can be used (see camera 1 shown in FIGS. 19A and 19B). A three-dimensional image at a short distance is acquired by such a known field-of-view overlap camera.
Further, as a camera for obtaining a camera vector, a camera capable of shooting a wide-range video such as a 360-degree all-round video, for example, “Lady Bug” manufactured by PGR Canada (FIG. 19A, ( It is also possible to use a camera with a wide-angle lens or fisheye lens, a moving camera, a fixed camera, a camera with a plurality of cameras fixed, a camera that can rotate around 360 degrees, and the like.

And by fixing the positional relationship between such a wide-angle camera and a field-of-view overlap camera, the camera vector obtained by a wide-angle camera such as a 360-degree camera is the same as the camera vector of the field-of-view overlap camera, Reflecting the original image, the three-dimensional image can be integrated into a unified image. In the example shown in FIGS. 19A and 19B, the camera 1 with overlapping fields of view and the camera 2 for 360 degrees are fixed to the fixed base 3.
Thus, in this embodiment, even a moving object can be integrated into the same coordinates.
Each unique coordinate system is integrated by sharing a common feature point depending on the camera vector, but in this case, not only the feature point but also the entire image is regarded as a feature point and the overall shape matches. It is also possible to integrate.

Hereinafter, as a more specific example of the three-dimensional shape apparatus 30 according to the present embodiment, a combination of a camera capable of continuously obtaining stereo vision and a camera capable of capturing all-around images, An embodiment for obtaining the original shape will be described.
First, as shown in FIGS. 19 (a) and 19 (b), the stereo vision camera 1 is integrated with the omnidirectional camera 2 and fixed by the fixed base 3, and the vehicle roof or the like so that the mutual positional relationship does not change. To load.
In order to extract a camera vector, it is desirable that the field of view is wide. Therefore, the camera 2 that can shoot all around 360 degrees is selected.
The stereo vision camera 1 uses “Degiclops” manufactured by PGR, Canada, and the all-around camera 2 uses “LadyBug” manufactured by Canada PGR.

As shown in FIGS. 19A and 19B, the stereo vision camera 1 and the omnidirectional camera 2 are fixed on the fixed base 3 and attached to the roof of the vehicle. The three-dimensional positional relationship between the camera mounting position and the vehicle shape needs to be measured in advance, but once measured, the vehicle position can be determined by measuring the camera position unless the camera is moved.
Moreover, since the positional relationship between the stereo vision camera 1 and the omnidirectional camera 2 is fixed, each camera is integrated with the vehicle, rotates in the same way, travels, and receives a shake.
In FIG. 19A, only one stereo vision camera 1 is mounted, but in FIG. 19B, a plurality of stereo vision cameras 1 are mounted. Thus, by providing a plurality of stereo vision cameras 1, a wide range of three-dimensional shapes can be acquired simultaneously, so that the efficiency of the image processing of the present invention can be further improved.

In the camera mounted on the vehicle in this way, first, in the stereo vision camera 1, a continuous 3D image is obtained by the continuous 3D image generation unit 33 (see FIG. 18). However, the positional relationship of the images between the frames has no relationship and cannot be combined as they are.
It can be considered that the images can be combined if they are matched three-dimensionally. However, if they are used alone, errors gradually accumulate, and images are not normally combined as the frame advances.

Therefore, an image is acquired by the omnidirectional camera 2 which is another camera, and the camera vector is extracted by the vector extraction unit 13 (see FIG. 18), so that the posture of the camera is fed back based on the camera vector. Correctly align the posture and reflect it in the image.
Moreover, since the moving distance of the camera can also be obtained from the camera vector, the camera position and rotation of each frame can be obtained by three-dimensional calculation from the camera vector data, and thus the obtained three-axis rotation amount and three-dimensional position are obtained. By reflecting the image of each frame from each stereo vision camera, the stereo image of each frame can be expressed by common coordinates in the moving camera coordinate system image conversion unit 34 (see FIG. 18). Since they are a common coordinate system, they can be synthesized directly by the coordinate integration unit 35 (see FIG. 18).

As described above, it is meaningful to perform three-dimensional matching as a fine adjustment of the binding site after synthesizing images in a common coordinate system.
By doing in this way, the integrated three-dimensional shape generation unit 36 (see FIG. 18) can continuously acquire a three-dimensional shape, and can also make it a three-dimensional map.
In this embodiment, the camera that obtains stereo vision and the camera that obtains the camera vector are separated, but it is naturally possible to use a common camera.

[Seventh embodiment]
Next, a seventh embodiment of the three-dimensional shape generation apparatus according to the present invention will be described with reference to FIG.
FIG. 20 is a block diagram showing a schematic configuration of a three-dimensional shape generation apparatus according to the seventh embodiment of the present invention.
The present embodiment shown in the figure is another embodiment of the three-dimensional shape generation apparatus 10 shown in FIG. 1, and means for calibrating a three-dimensional map by detecting unevenness of a road surface on which a vehicle equipped with a camera passes. It has.
Specifically, the three-dimensional shape generation apparatus 40 according to the present embodiment includes a short-distance partial detail image acquisition unit 41, a parallel image recording unit 42, a short-distance partial three-dimensional measurement unit 43, and other moving object detection. A unit 44, another moving object exclusion unit 45, a coordinate coupling unit 46, a coupled coordinate recording unit 47, and an overall image display unit 48.

The short-distance portion detailed image acquisition unit 41 is a unit that mainly mounts a plurality of cameras as cameras loaded on a moving object and shoots mainly a portion that requires detailed data of a short distance.
The parallel image recording unit 42 records the images at the same time set so that the fields of view by the moving object loading camera partially overlap in a form that can be recalled.
The short-distance partial three-dimensional shape generation unit 43 detects parallax from images with overlapping fields of view, and generates a detailed three-dimensional shape of the short-distance portion.
The other moving object detection unit 44 detects parallax from images with overlapping fields of view, and detects other moving objects other than the moving object loaded with the camera.

The other moving object exclusion unit 45 excludes the parallax data of other moving objects from the parallax data of images with overlapping fields of view.
The coordinate combining unit 46 calculates the measured length by parallax from the three-dimensional shape of the overlapping object in the three-dimensional shape generation unit 16 and the short-distance partial three-dimensional shape generation unit 43 shown in the above-described embodiments. Calibrate the scale and integrate each coordinate system.
The integrated coordinate recording unit 47 records the integrated data.
The comprehensive image display unit 48 displays the data integrated by the integrated coordinate recording unit 47.

When it is intended to accurately measure the unevenness of the road surface on which the vehicle equipped with the camera passes, the road surface unevenness can be detected by photographing the road surface with two cameras and detecting parallax. However, by itself, the accuracy of short distance can be obtained due to the limitation of the baseline between cameras, but the accuracy of long distance cannot be obtained.
Therefore, in this embodiment, only in the ultra-short distance measurement, the near-distance 3D shape generation is accurately performed by the image processing based on the parallax by the two cameras, and integrated with the overall coordinates, so that the 3D shape and the 3D map are obtained. Is generated. Accordingly, it is possible to improve the calculation accuracy by detecting a moving body other than the hourly vehicle and excluding the moving body from the image.

In addition, the three-dimensional shape generation apparatus 40 of this embodiment can be implemented in combination with the three-dimensional shape generation apparatus 30 (see FIG. 18) shown in the sixth embodiment described above.
That is, the three-dimensional shape generation apparatus 40 according to this embodiment is not limited to the basic embodiment using a single camera, but includes a plurality of cameras (including a stereo vision camera, a wide-angle camera, etc.) as shown in the sixth embodiment. It can also be implemented as a system using parallax due to the above.

[Eighth embodiment]
Furthermore, an eighth embodiment of the three-dimensional shape generation apparatus according to the present invention will be described with reference to FIG.
FIG. 21 is a block diagram showing a schematic configuration of the three-dimensional shape generation apparatus according to the eighth embodiment of the present invention.
This embodiment shown in the figure is configured to be implemented by appropriately combining the three-dimensional shape generation apparatuses according to the first to seventh embodiments described above.

Specifically, as illustrated in FIG. 21, the three-dimensional shape generation apparatus 100 according to the present embodiment includes a video acquisition unit 101, a vector detection unit 102, an overlap calculation unit 103, a correction signal generation unit 104, Normalized image conversion unit 105, designated direction plane conversion processing unit 106, Opt. F. The calculation unit 107 and the target plane Opt. F. The extraction designation unit 108, the plane shape and coordinate calculation unit 109, the non-surface processing unit 110, the integrated output unit 111, and the plane reconstruction three-dimensional map output unit 112 can be arbitrarily combined and implemented. It has become.
The specific contents of each component are as shown in the above-described embodiments.
According to the 3D shape generation apparatus 100 of this embodiment as described above, a 3D map generation apparatus that can generate a 3D map by continuously generating a 3D shape of an object over a wide range is provided. Can be realized. In the present invention, a three-dimensional map can be obtained if the three-dimensional shape generation is not performed on a limited object but is executed continuously over a wide range.

  The three-dimensional shape generation apparatus of the present invention has been described with reference to the preferred embodiment, but the three-dimensional shape generation apparatus according to the present invention is not limited to the above-described embodiment, and the scope of the present invention. Needless to say, various modifications can be made.

  The present invention can be used as, for example, a three-dimensional shape generation device suitable for generating a three-dimensional map provided in a car navigation device.

It is a block diagram which shows schematic structure of the three-dimensional shape production | generation apparatus which concerns on 1st embodiment of this invention. It is explanatory drawing which shows the example of a display of the three-dimensional shape (three-dimensional map) produced | generated with the three-dimensional shape production | generation apparatus which concerns on 1st embodiment of this invention. It is a block diagram which shows schematic structure of the vector detection part which concerns on 1st embodiment of this invention. It is explanatory drawing which shows the specific detection method of the camera vector in the vector detection part which concerns on 1st embodiment of this invention. It is explanatory drawing which shows the specific detection method of the camera vector in the vector detection part which concerns on 1st embodiment of this invention. It is explanatory drawing which shows the specific detection method of the camera vector in the vector detection part which concerns on 1st embodiment of this invention. It is explanatory drawing which shows the designation | designated aspect of the desirable feature point in the detection method of the camera vector by the vector detection part which concerns on 1st embodiment of this invention. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the vector detection part which concerns on 1st embodiment of this invention, and a camera vector. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the vector detection part which concerns on 1st embodiment of this invention, and a camera vector. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the vector detection part which concerns on 1st embodiment of this invention, and a camera vector. In the vector detection part which concerns on 1st embodiment of this invention, it is explanatory drawing which shows the case where a some feature point is set according to the distance of a feature point from a camera, and a some calculation is repeated. It is a figure at the time of displaying the locus | trajectory of the camera vector calculated | required in the vector detection part which concerns on 1st embodiment of this invention in the produced | generated three-dimensional map. It is a block diagram which shows schematic structure of the three-dimensional shape production | generation apparatus which concerns on 2nd embodiment of this invention. It is a block diagram which shows schematic structure of the three-dimensional shape production | generation apparatus which concerns on 3rd embodiment of this invention. It is a block diagram which shows schematic structure of the three-dimensional shape production | generation apparatus which concerns on 4th embodiment of this invention. It is an image figure of the plane of the target object extracted by optical flow in the three-dimensional generation part of a 4th embodiment of the present invention. It is explanatory drawing which shows notionally the vector detection method of the moving object in the three-dimensional shape production | generation apparatus which concerns on 5th embodiment of this invention. It is a block diagram which shows schematic structure of the three-dimensional shape production | generation apparatus which concerns on 6th embodiment of this invention. It is a perspective view which shows typically the specific example at the time of fixing and providing the visual field overlap camera and wide-angle camera with which the three-dimensional shape production | generation apparatus which concerns on 6th embodiment of this invention is equipped. It is a block diagram which shows schematic structure of the three-dimensional shape production | generation apparatus which concerns on 7th embodiment of this invention. It is a block diagram which shows schematic structure of the three-dimensional shape production | generation apparatus which concerns on 8th embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 3D shape production | generation apparatus 11 Image acquisition part 12 Image recording part 13 Vector detection part 14 Camera vector addition image recording part 15 Multiple parallax calculation part 16 3D shape generation part 17 Correction signal generation part 18 Normalization image conversion part 20 Vector Detection unit 21 Feature point extraction unit 22 Feature point correspondence processing unit 23 Feature point / camera position calculation unit 24 Error minimization unit

Claims (9)

An image acquisition unit that acquires a video from a camera that obtains a moving image or a continuous still image provided in the moving body;
An image recording unit for temporarily recording the image acquired by the image acquisition unit;
From the image data read out from the image recording unit, a three-dimensional position of a point or a small area corresponding to each frame, a three-dimensional position of the camera and a three-axis rotation of the camera are overlapped in a plurality of frames, A vector detection unit that calculates a camera vector by performing statistical processing so that an error distribution is minimized;
A camera vector-added image recording unit that records all frame images to which the data of the camera three-dimensional position and the three-axis rotation camera vector obtained by the vector detection unit are added;
Based on each frame image and the camera vector obtained for each frame, using a large number of parallaxes obtained from a plurality of frame images or an average value of a plurality of optical flows obtained from a plurality of frame images, A multiple parallax calculation unit that performs overlap calculation for each point of the micro surface and generates a three-dimensional shape of the object;
A three-dimensional shape generation unit that continuously generates the three-dimensional shape of the object in the multiple parallax calculation unit to form a three-dimensional shape , and
The vector detection unit
Automatically detect feature points with image features in the image, find corresponding points in each frame image,
Focusing on the n-th and n + m-th frame images Fn and Fn + m used for camera vector calculation, unit calculation is performed, n and m are set to predetermined values, and unit calculation is repeated.
The feature points are classified into multiple stages according to the distance from the camera to the feature point in the image, and the frame interval m is set so as to increase as the distance from the camera to the feature point increases.
Set multiple levels of m while overlapping multiple classification of feature points by m,
As the image progresses, n continuously advances, the calculation is continuously advanced, and at each stage of n and m, the same feature point is repeatedly calculated several times, and the error of each feature point and camera position is calculated. Integrate by adjusting the scale to the minimum, perform distance calculation,
By deleting feature points with large error distribution and recalculation,
Three-dimensional shape generation apparatus characterized by resulting in higher accuracy of calculation at each feature point and the camera position. An image acquisition unit that acquires a video from a camera that obtains a moving image or a continuous still image provided in the moving body;
An image recording unit for temporarily recording the image acquired by the image acquisition unit;
From the image data read out from the image recording unit, a three-dimensional position of a point or a small area corresponding to each frame, a three-dimensional position of the camera and a three-axis rotation of the camera are overlapped in a plurality of frames, A vector detection unit that calculates a camera vector by performing statistical processing so that an error distribution is minimized;
A correction signal generation unit for generating a correction signal for correcting the original image from the camera vector obtained by the vector detection unit;
A normalized image conversion unit that records the normalized image corrected by the correction signal;
Based on the three-dimensional position of the camera in each frame and a plurality of normalized images corrected by the correction signal, a number of parallaxes obtained from a plurality of frame images or a number of optical flows obtained from a plurality of frame images A three-dimensional shape generation unit that performs an overlap calculation on each minute surface of the target object in the image using the average value and generates a three-dimensional shape of the target object ,
The vector detection unit
Automatically detect feature points with image features in the image, find corresponding points in each frame image,
Focusing on the n-th and n + m-th frame images Fn and Fn + m used for camera vector calculation, unit calculation is performed, n and m are set to predetermined values, and unit calculation is repeated.
The feature points are classified into multiple stages according to the distance from the camera to the feature point in the image, and the frame interval m is set so as to increase as the distance from the camera to the feature point increases.
Set multiple levels of m while overlapping multiple classification of feature points by m,
As the image progresses, n continuously advances, the calculation is continuously advanced, and at each stage of n and m, the same feature point is repeatedly calculated several times, and the error of each feature point and camera position is calculated. Integrate by adjusting the scale to the minimum, perform distance calculation,
By deleting feature points with large error distribution and recalculation,
Three-dimensional shape generation apparatus characterized by resulting in higher accuracy of calculation at each feature point and the camera position. The vector detection unit
A feature point extraction unit that automatically extracts and designates a point or a small area image that should be a feature point in the image, based on the image data read from the image recording unit;
A feature point correspondence processing unit for obtaining a correspondence relationship between the frames of the automatically designated feature points;
A feature point / camera position calculation unit for calculating a three-dimensional position coordinate and a camera position of the feature point for which the correspondence relationship is obtained;
A plurality of three-dimensional positions of the feature points are calculated, statistical processing is performed so that the distribution of the position of each feature point and the camera position is minimized, the feature point having a large error is detected and deleted, An error minimizing unit that performs recalculation based on feature points and minimizes the overall error;
A three-dimensional shape generation apparatus according to claim 1 or 2. The three-dimensional shape generation unit
Based on the already obtained camera vector,
Multiple parallax 3D that repeats the operation while moving the small area in the frame image, extracts a large number of parallaxes for all the small areas in the image, and generates a three-dimensional shape with less error by overlapping calculation by the parallax A shape generation unit;
A multi-parallax three-dimensional shape output unit that outputs the three-dimensional map by integrating the three-dimensional shape generated by the multi-parallax three-dimensional shape generation unit into the same coordinate system as the camera;
A three-dimensional shape generation apparatus according to any one of claims 1 to 3. The three-dimensional shape generation unit
A designated direction plane conversion processing unit that plane-converts the video acquired by the image acquisition unit into a video from a camera viewpoint perpendicular to the surface of a desired object;
A target plane optical flow extraction specifying unit that calculates an optical flow of a desired target plane by calculating the optical flow of a video in a certain area from the plane-converted video,
A non-purpose surface excluding portion that excludes an optical flow other than the target surface;
A plane shape and coordinate acquisition unit that extracts a target plane as a plane having a shape and gives three-dimensional coordinates to the plane;
A plane configuration three-dimensional shape generation unit that generates a three-dimensional shape based on a plane configuration for all desired objects in the region by repeating the calculation while moving the certain region,
A plane reconstructed three-dimensional shape output unit that outputs a three-dimensional shape according to those planar configurations as a three-dimensional map;
A three-dimensional shape generation apparatus according to any one of claims 1 to 4. The vector detection unit
In addition to detecting the vector of the moving camera, the vector of the moving object from the moving camera is detected by detecting the moving vector of the moving moving object separately from the camera by the same method as the camera vector. The three-dimensional shape generation apparatus according to any one of claims 1 to 5, wherein a posture of a moving object with respect to a stationary coordinate form is detected as a three-dimensional vector by adding a vector of the movement vector and the movement vector of the moving object. In addition to or in addition to the camera provided in the moving body, a plurality of cameras provided in the same moving body, and a field-of-view overlap camera unit set so that the fields of view overlap,
A field-of-view overlap image recording unit that records a plurality of images of the same time acquired by the field-of-view overlap camera unit in association with each other;
A continuous three-dimensional image generation unit that generates a three-dimensional image consisting of a unique coordinate system at a unit time, which is obtained by calculation from a plurality of images at the same time with overlapping fields of view;
Coordinate integration that combines the three-dimensional image obtained by the continuous three-dimensional image generation unit with the camera vector of the moving camera obtained from the vector detection unit, and converts and integrates it into a common coordinate system with the moving camera system And
An integrated 3D shape generation unit that generates an object and surrounding 3D shape in an integrated coordinate system;
A three-dimensional shape generation apparatus according to claim 1, comprising: A short-distance portion detailed image acquisition unit that includes a plurality of cameras as cameras provided in the moving body, and mainly shoots a portion that requires short-distance detailed data;
A parallel image recording unit that records a plurality of images acquired by the plurality of cameras at the same time set so that the fields of view partially overlap in a callable form,
Detecting a parallax from the overlapping images of the visual field and generating a detailed three-dimensional shape of the short-distance portion;
Other moving object detection units for detecting parallax from images with overlapping fields of view and detecting other moving objects other than the moving object provided with the camera;
Another moving object exclusion unit that excludes the parallax data of the other moving object from the parallax data of the images with overlapping fields of view;
Coordinates that integrate the respective coordinate systems by calibrating the scale with the measured length based on the parallax from the three-dimensional shape of the object in which the three-dimensional shape generating unit and the short-distance partial three-dimensional shape generating unit overlap The calibration integration unit,
An integrated coordinate recording unit for recording the integrated data;
A comprehensive image display unit for displaying the integrated data;
The three-dimensional shape generation apparatus according to claim 1, comprising: A three-dimensional map comprising the three-dimensional shape generation device according to any one of claims 1 to 8, wherein a three-dimensional map is generated by continuously generating a wide range of three-dimensional shapes of an object. Map generator.