WO2005040721A1

WO2005040721A1 - 3d automatic measuring apparatus

Info

Publication number: WO2005040721A1
Application number: PCT/JP2004/015766
Authority: WO
Inventors: Waro Iwane
Original assignee: Waro Iwane
Priority date: 2003-10-29
Filing date: 2004-10-19
Publication date: 2005-05-06
Also published as: JPWO2005040721A1; JP4545093B2

Abstract

A three-dimensional measurement is performed, with a high degree of precision, for any object in a moving image by analyzing the moving image captured by a 360-degree all-round camera. There are included round image capturing part (101) for capturing a moving image including desired points to be measured and reference points whose coordinates have already been known; an image record part (102) for recording captured image; a characteristic point extraction part (103) for extracting image characteristic points in the image; a measured point determination part (104) for automatically extracting measured points in the image; a reference point determination part (105) for automatically extracting reference points in the image; an associated point trace part (106) for associating the measured points, reference points and characteristic points within each frame image; a vector calculation part (107) for calculating three-dimensional relative coordinates of the associated measured points, reference points and characteristic points; an error minimization part (108) for repeating the foregoing calculation to perform statistic processings of the three-dimensional relative coordinates; an absolute coordinate acquisition part (109) for converting, by use of the known coordinates of the reference points, the three-dimensional relative coordinates to absolute coordinate system; a measured data record part (110) for recording ultimate coordinates; and a display part (111) for displaying recorded measured data.

Description

Specification

3D automatic surveying equipment

Technical field

The present invention relates to a surveying device that measures a size of a desired object, a distance between the objects, and the like based on image data captured by a camera.

In particular, the present invention analyzes a moving image captured from a 360-degree omnidirectional camera ('This enables highly accurate three-dimensional measurement of any object in an image, By arbitrarily designating the two points, the start point and the end point, freely across multiple frame images taken by the camera, the three-dimensional distance between the specified two points is measured, and more than two points are determined. The present invention relates to a 3D automatic surveying device capable of three-dimensionally measuring the area and volume of a desired object or the like by designating it.

Background art

[0002] Generally, an image surveying technique for analyzing an image photographed by a plurality of cameras to measure a size of an object, a distance between the objects, and the like is known.

This type of image surveying is, for example, a technique based on a stereo method of measuring a distance from an image with parallax obtained by two cameras installed in parallel, and is used as a simple surveying technique (Patent Literature 1-2.).

In this type of image surveying, an attempt has been made to generate a map from the obtained measurement data, and the field of application is expanding.

[0003] Patent Document 1: JP 08-278126 A

Patent Document 2: JP-A-2000-283753

Disclosure of the invention

Problems to be solved by the invention

[0004] However, in the stereo method using two cameras, there is a restriction on the distance between the cameras to be installed, so that it is impossible to measure a long distance such as measuring a distance between distant objects.

In addition, short distance measurement where the accuracy of measurement is poor because of surveying using the parallax of two cameras Other than that, it has become practical as a precise surveying technology. /.

If a high-precision survey is to be performed using the conventional stereo method, the two cameras must have extremely high-precision consistency, and the distance and angle between the cameras must be adjusted with high precision. There is a risk that errors may easily occur due to such factors, and it has been difficult to actually commercialize the system.

[0005] Accordingly, as a result of earnest research, the present inventor has extracted a sufficient number of feature points from a plurality of frame images of a ning image obtained from a 360-degree omnidirectional camera, thereby obtaining desired feature points. 3D coordinates that indicate the relative position between the camera position and the rotation angle can be determined with high precision. By converting the 3D relative coordinates into absolute coordinates, a single 360-degree omnidirectional camera can move and shake We came to realize that high-precision surveying, which is not affected by such factors, can be realized.

[0006] In other words, the present invention has been proposed to solve the problems of the above-described conventional technology, and requires a plurality of cameras by analyzing a moving image acquired from a 360-degree omnidirectional camera. High-precision three-dimensional measurement of any object in the image without being affected by camera vibration or shaking, etc., and between any two points specified over multiple frame images 3D distance measurement is possible without being restricted by distance.Furthermore, by specifying two or more points, any area or volume in the image can be measured in three dimensions. The purpose is to provide a 3D automatic surveying device that can do this. .

Means for solving the problem

[0007] In order to achieve the above object, the 3D automatic surveying device of the present invention uses a moving 360-degree omnidirectional camera to provide a moving or continuous image including a desired measurement point and a predetermined reference point whose three-dimensional absolute coordinates are known. An omnidirectional image capturing unit that captures still images, an image recording unit that records images captured by the omnidirectional image capturing unit, and images recorded in the image recording unit that have image characteristics other than measurement points. A feature point extraction unit that extracts a portion as a feature point, a measurement point identification unit that automatically extracts measurement points in the image recorded in the image recording unit, and a reference point in the image recorded in the image recording unit And a corresponding point tracking unit that tracks measurement points, reference points, and feature points in each frame image and associates them. A vector calculation unit for calculating the three-dimensional relative coordinates of the measurement points, reference points, and feature points associated with the corresponding point tracking unit and, if necessary, the camera vector indicating the position and rotation of the camera; It repeats the operation in the vector operation unit, repeats the overlap operation so as to minimize the error of the obtained three-dimensional relative coordinates, and performs an error minimization processing unit that performs statistical processing, and the vector operation from the known three-dimensional absolute coordinates of the reference point The absolute coordinate acquisition unit converts the three-dimensional relative coordinates obtained by the unit into the absolute coordinate system and attaches the three-dimensional absolute coordinates to the measurement points, reference points, and feature points. It is configured to include a measurement data recording unit that records the final absolute coordinates given thereto, and a display unit that displays the measurement data recorded in the measurement data recording unit. ,

[0008] Further, the 3D automatic surveying apparatus of the present invention provides a moving 360-degree omni-directional camera that captures a moving image or a continuous still image including a desired measurement point and a predetermined reference point whose three-dimensional absolute coordinates are known. A surrounding image capturing unit, an image recording unit that records images captured by the omnidirectional image capturing unit, and a feature that extracts a portion having a visual feature from the image recorded in the image recording unit as a feature point. A point extraction unit, a reference point identification unit that automatically extracts reference points in the plane image recorded in the image recording unit, and a corresponding point tracking unit that tracks and associates the reference points and feature points in each frame image. And a vector operation unit for calculating the three-dimensional relative coordinates of the camera vector indicating the position and rotation of the camera from the reference point and the feature point associated with the corresponding point tracking unit, and an operation in the vector operation unit. Repeatedly required An error minimization processing unit that repeats overlapping operations to minimize errors in three-dimensional relative coordinates and performs statistical processing, and the cubic of the camera calculated by the vector operation unit from the known three-dimensional absolute coordinates of the reference point. An absolute coordinate acquisition unit that converts the original relative coordinates into an absolute coordinate system and assigns three-dimensional absolute coordinates, and a measurement point identification unit that automatically extracts measurement points in the image recorded in the image recording unit And a measurement point tracking unit that tracks and associates the measurement points extracted by the measurement point identification unit in each frame image, and calculates the measurement values of the measurement points from the camera vector obtained by the vector calculation unit. It has a configuration including a measurement point measurement calculation unit, a measurement data recording unit that records the absolute coordinates of the measurement point, and a display unit that displays the measurement data recorded in the measurement data recording unit.

[0009] Further, the 3D automatic surveying device of the present invention is configured such that the reference point is a reference point having a known three-dimensional absolute coordinate. Together with the reference point whose absolute three-dimensional coordinates are known, or a length reference point whose length is known, the vector operation unit calculates the distance between the two length reference points by calculation, and calculates the minimum error. The conversion processing unit is configured to repeat the overlap operation and perform statistical processing so that the distance between the two length reference points obtained by the calculation by the vector calculation unit matches the known length of the length reference point. There is. .

[0010] Further, the 3D automatic surveying device of the present invention includes a measurement point designation working unit for designating an arbitrary measurement point in an image recorded in the image recording unit, and an image recorded in the image recording unit. Within the image, a reference point designating unit for designating an arbitrary reference point is provided, and an image 內 preparation working unit is provided. The measurement point and the reference point are designated and extracted.

[0011] Further, in the 3D automatic surveying device of the present invention, the belt calculation unit may include any two frame images Fn and Fn used for three-dimensional relative coordinate calculation of a measurement point, a reference point, a feature point, or a camera vector. The unit operation for finding the desired three-dimensional relative coordinates is repeated with + m (m = frame interval) as the unit image, and the error minimization processing unit has the same characteristics as η progressing continuously as the image progresses. The configuration is such that the scale is adjusted and integrated so that the error of each three-dimensional relative coordinate obtained by calculating a point a plurality of times is minimized, and the final three-dimensional relative coordinate is determined.

[0012] Further, in the 3D automatic surveying device of the present invention, the vector calculation unit sets the frame interval m as the distance of the camera power increases as the distance between the camera power measurement point, the reference point, and the feature point increases. The configuration is such that the unit calculation is performed by setting to be large.

[0013] In the 3D automatic survey device of the present invention, the vector calculation unit deletes feature points having a large error distribution of the obtained three-dimensional relative coordinates, and based on other feature points as necessary. The recalculation is performed to improve the accuracy of the measurement point calculation.

[0014] Further, the 3D automatic surveying device of the present invention starts at an arbitrary measurement point or an arbitrary feature point in an arbitrary image recorded in the image recording unit, and sets the starting point in the image or another different image. A three-dimensional distance between any start point and end point specified within the same image or between different images based on the absolute coordinates recorded in the measurement data recording part, specifying any measurement point or any feature point as the end point Is provided with a distance calculation unit that obtains by calculation. [0015] Further, the 3D automatic surveying apparatus of the present invention specifies a plurality of points within an arbitrary same image or between different plane images recorded in the image recording unit, and specifies a plurality of points between the start point and the end point obtained by the distance calculation unit. The configuration is such that a plurality of three-dimensional distance measurements are combined, and an area / volume calculator is provided for calculating the area or volume of a desired object within the same image or between different images by calculation. .

[0016] Further, the 3D automatic surveying device of the present invention includes a traveling direction control unit that fixes or controls an image obtained by the omnidirectional image photographing unit in the traveling direction by using a turtle vector obtained by the vector operation unit, An image vertical plane development unit that spreads the image stabilized in the traveling direction by the traveling direction control unit on a vertical plane, and a road surface that generates a basic shape model of the road surface with undefined parameters of the road surface shape The basic model generation unit, the road surface three-dimensional measurement unit that measures the three-dimensional coordinates of the road surface from the image of the road surface developed in a vertical plane by the image vertical plane development unit, and the road surface three-dimensional measurement unit A road transparent CG generation unit that obtains the parameters of the road surface shape from the road surface measurement data and generates a transparent CG of the road surface, and a transparent CG generated by the road transparent CG generation unit and the traveling direction Stable in the traveling direction by the control unit The synthesized road surface image is synthesized, and the road surface texture is added and averaged to the synthesized road surface plane dry section that develops the image parallel to the road surface, and the image developed by the synthesized road surface plane expansion section. A texture averaging unit that reduces noise in the image, and, if necessary, block the road surface and flexibly combine the characteristic parts of the road surface without changing the texture order, and then texture the output. Road surface texture to be sent to the averaging unit-The object area that largely cuts out the area of road surface figures such as road markings and obstacles from the image reduced in noise by the texture averaging unit. Section, a road marking recognition and coordinate acquisition section for recognizing a target object from the object area cut out by the object area cutout section and obtaining its coordinates, and an eye for which the coordinates are obtained. 3D map of the road surface by reconstructing the output of the measurement data recording unit that obtained the absolute coordinates by inputting each point of the polygon constituting the object And a three-dimensional map generation device having a three-dimensional map generation unit for generating a map.

The invention's effect According to the present invention as described above, by using a 360-degree omnidirectional image, a sufficient number of feature points are extracted from a plurality of frame images of a moving image to include a desired measurement point. Three-dimensional relative coordinates indicating the relative positions of many feature points can be obtained with high accuracy. Note that it is possible to treat a normal image as a part of a 360-degree omnidirectional image in the same way as a 360-degree omnidirectional image. However, since the accuracy is lower than that of the 360 ° omnidirectional image, it is preferable to use the 360 ° omnidirectional image as much as possible.

Then, the obtained three-dimensional relative coordinates can be converted into an absolute coordinate system based on the known three-dimensional absolute coordinates of the reference point obtained in advance by surveying or the like.

If it is not necessary to obtain absolute coordinates, the absolute coordinates such as latitude / longitude, etc., may be determined in advance based on a known length in surveying or by placing an object with a known length around the measurement point. Even if the coordinate values cannot be obtained, the scale can be corrected and the measurement results can be obtained. Accordingly, in the present invention, in principle, one .360-degree omni-directional camera captures an image with a camera arbitrarily moving in free space, and designates a desired survey point in the image. Alternatively, by taking images of survey points marked in advance and analyzing them, extremely accurate 3D surveying can be performed.

As described above, by analyzing a moving image obtained by one 360-degree omnidirectional camera, three-dimensional absolute coordinates of a desired object or the like can be obtained, and a large number of feature points can be obtained. By extracting and generating three-dimensional information, errors can be minimized as much as possible.In images that do not require multiple cameras and are not affected by camera shake or shaking, etc. High-precision three-dimensional measurement can be performed for any given object. That is, in the present invention, the same measurement point is determined by analyzing a moving image composed of a large number of frame images including a desired measurement point by moving one camera, not by parallax between the two cameras. A large number of included frame images can be used, and calculations with high accuracy can be performed with sufficient information.

Further, according to the present invention, in which a highly accurate absolute coordinate can be obtained at a desired measurement point, a three-dimensional distance specifying an arbitrary start point and an end point can be measured. High-precision three-dimensional distance measurement is possible without being restricted by distance, even between any two points specified by using multiple frame images. Furthermore, by specifying three or more points, it becomes possible to three-dimensionally measure the area and volume of an arbitrary object or region in an image or over a plurality of images. Brief Description of Drawings

FIG. 1 is a block diagram showing a schematic configuration of a 3D automatic surveying device according to a first embodiment of the present invention. ·

FIG. 2 is a block diagram showing a schematic configuration of a 3D automatic surveying device according to another embodiment of the first embodiment of the present invention.

FIG. 3 is an explanatory view showing a specific camera vector detection method in the 3D automatic surveying device according to one embodiment of the present invention.

FIG. 4 is an explanatory view showing a specific camera vector detection method in the 3D automatic surveying device according to one embodiment of the present invention.

FIG. 5 is an explanatory diagram showing a specific method for detecting a camera vector in the 3D automatic surveying device according to one embodiment of the present invention.

FIG. 6 is an explanatory diagram showing a desirable feature point designation mode in a camera vector detection method by the 3D automatic surveying device according to one embodiment of the present invention.

FIG. 7 is a graph showing an example of three-dimensional coordinates of a feature point and a camera vector obtained by the 3D automatic surveying device according to one embodiment of the present invention.

FIG. 8 is a graph showing an example of three-dimensional coordinates of a feature point and a camera vector obtained by the 3D automatic surveying device according to one embodiment of the present invention.

FIG. 9 is a graph showing an example of three-dimensional coordinates of a feature point and a camera vector obtained by the 3D automatic surveying device according to one embodiment of the present invention.

FIG. 10 is an explanatory diagram showing a case where a plurality of feature points are set according to the distance between the camera and the feature points and a plurality of calculations are repeatedly performed in the 3D automatic surveying device according to one embodiment of the present invention. .

[FIG. 11] A 3D automatic surveying device according to the first embodiment of the present invention shown in FIG. 1 or FIG. FIG. 4 is a block diagram showing a schematic configuration in a case where an arbitrary measurement point is measured based on a camera vector already obtained.

FIG. 12 is a block diagram showing a schematic configuration of a 3D automatic surveying device according to a second embodiment of the present invention. It is. '

13 is a flowchart showing a procedure of a surveying process in the 3D automatic surveying device shown in FIG. 12.

FIG. 14 is an explanatory diagram showing a procedure for measuring a three-dimensional distance between arbitrary two points in the 3D automatic surveying device shown in FIG. 12. .

FIG. 15 is an explanatory diagram showing a procedure for calculating the area or volume of an arbitrary region in the 3D automatic surveying device shown in FIG. 12.

FIG. 16 is an explanatory diagram showing an example of an image in which an arbitrary point for obtaining a three-dimensional distance is designated in the same image by the 3D automatic surveying device shown in FIG. 12.

FIG. 17 is a block diagram showing a schematic configuration of a 3D automatic surveying device according to a third embodiment of the present invention.

FIG. 18 is a block diagram showing a schematic configuration of a 3D automatic surveying device according to a fourth embodiment of the present invention.

FIG. 19 is a diagram showing an example of a three-dimensional map generated by the 3D automatic surveying device according to the fourth embodiment of the present invention, where (a) is a sectional view of a road represented by the three-dimensional map, (B) is a projection of the road shown in (a) taken as an example of the three-dimensional map of the road, and (c) is a projection of the three-dimensional map shown in (b). FIG. 2 is a view showing an operator part used for this.

FIG. 20 is a three-dimensional view of the road shown in FIG. 19, in which an operator part (CG part) of a road sign is combined.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, a preferred embodiment of the 3D automatic surveying device according to the present invention will be described with reference to the drawings.

Here, the 3D. Automatic surveying device of the present invention described below is realized by processing, means, and functions executed by a computer according to instructions of a program (software). The program sends commands to each component of the computer, and performs predetermined processing and functions as described below, for example, automatic extraction of feature points, automatic tracking of extracted feature points, calculation of three-dimensional coordinates of feature points, The calculation of the camera vector is performed. As described above, the 3D automatic surveying device and the plan Each process and means in the image stabilizing device is realized by specific means in which a program and a computer cooperate.

Note that all or a part of the program is provided, for example, by a magnetic disk, an optical disk, a semiconductor memory, or any other computer-readable recording medium, and the read program is installed in the computer. Executed. Also, programs are loaded directly into a computer via a communication line and executed without using a recording medium.

First, an embodiment of a 3D automatic surveying device according to the present invention will be described with reference to FIG. 1 and FIG. '

FIGS. 1 and 2 are block diagrams each showing a schematic configuration of a 3D automatic surveying device according to an embodiment of the present invention.

The 3D automatic surveying device according to the embodiment shown in FIG. 1 includes a field preparation work unit 10 for performing preparation work such as designation of survey points in advance and a 3D automatic surveying device 100 for performing surveying processing within a captured camera image. Have.

In addition, the 3D automatic surveying apparatus of the embodiment shown in FIG. 2 is a means for performing preparation work such as designation of survey points in advance, instead of the on-site preparation work section 10 shown in FIG. It has.

The on-site preparation work section 10 is a means for performing on-site preparation work prior to the measurement work, and includes a on-site measurement point designation work section 11 and a on-site reference point designation work section 12, as shown in FIG. Prior to the measurement work, the on-site measurement point designation work unit 11 designates all desired measurement points. The designation of the measurement point can be performed, for example, by adding a mark indicating the measurement point at the site, placing an object indicating the measurement point, or the like. With this designation, the 3D automatic surveying device 100 described later can extract and specify measurement points in the captured camera image.

The local reference point designation work unit 12 designates a point to be a predetermined reference point prior to the measurement work.

Here, the reference point is, as described later, a reference point when converting the three-dimensional relative coordinates into the absolute coordinates, and is known in advance by any method. ) Is total The point to be measured (coordinate reference point>.

In addition, the reference point can include a reference point having a known length (length reference point) together with a reference point having a known three-dimensional absolute coordinate or in place of a reference point having a known three-dimensional absolute coordinate. . The length reference point is a reference point that consists of two or more points and treats the distance between the two points as a known one.For example, the interval between the length reference points is set to 1 meter It can be obtained by setting up a large number of 1-meter sticks and so on. Then, shooting is performed so that at least one length reference point overlaps each image. By providing such a length reference point, a scale calibration can be performed for each image based on the known length of the length reference point, as described later, and the accuracy can be greatly improved. .

Here, the length reference point can be considered to be the same as setting a plurality of coordinate reference points. However, setting a large number of length reference points that are “lengths” means “points”. This is more effective than setting many coordinate reference points. That is, coordinate reference points can be converted to absolute coordinates if only two points are set in the entire measurement range.Coordinate reference points are not necessarily observed from all images, and multiple coordinate reference points are set. Providing more than one length reference point is more advantageous in terms of cost and labor.

Therefore, for example, in the entire measurement range, only two coordinate reference points are used, and a number of rods of a predetermined length (for example, 1 meter) indicating the length reference are simply and randomly set in the measurement range. Such an automatic survey can be performed, and the labor and cost of the measurement work can be greatly reduced. '

The measurement of the three-dimensional coordinates and the length of the reference point (the coordinate reference point or the length reference point) may be performed by any method, for example, by a conventionally known measurement method such as a trigonometric method. Absolute coordinates and length can be obtained.

The local reference point designating unit 12 marks the reference point with a reference point that can be clearly distinguished from the measurement point, or places an object or bar indicating the reference point to specify the reference point. Do. With this designation, in the 3D automatic surveying device 100 to be described later, a predetermined reference point can be extracted and specified in the captured force image.

[0027] Specifically, the preparation work in the on-site preparation work unit 10 is a work in which all the target measurement points to be measured are marked at the surveying site so that they can be recognized. This mark etc. In the processing of the 3D automatic surveying device # 00-, it is for automatically extracting by image recognition on an image. Therefore, in order to enable automatic extraction by image recognition, simple figures and the like are good, and features that do not confuse with other measurement points, reference points, other figures, etc. are given. X may be used, or a colored stake may be hit at the measurement point. .

Similarly, for a predetermined reference point, a mark or the like is attached in the case of a coordinate reference point, and a bar or the like is provided in the case of a length reference point.

It is desirable that the reference point and the measurement point be different marks, and that a plurality of reference points be individually marked differently. As a result, the measurement point and the reference point can be clearly distinguished, and a plurality of reference points can also be distinguished from each other, so that the start point and the end point can be easily specified.

By specifying two or more reference points whose absolute coordinates are known, three-dimensional relative coordinates can be converted into absolute coordinates as described later.

Here, the work of designating the measurement point and the reference point can be automated by a machine or the like, or can be performed manually by an operator. In addition, for example, when the number of reference points is small, the on-site work is omitted, and the reference points are detected directly from the image and marked on the image.

FIG. 2 shows a 3D automatic surveying apparatus including an in-image preparation work unit 20 for performing a preparation operation for designating a measurement point and a reference point in an image.

As shown in the figure, the in-image preparation work unit 20 omits the on-site preparation work at the site by the ground preparation work unit 10 shown in FIG. It is a means for designating a desired measurement point and a reference point within the device, and includes an in-image measurement point designation work unit 21 and an in-image reference point work unit 22.

[0030] The in-image measurement point designation work unit 21 designates a desired measurement point in the video imaged by the surrounding image imaging unit 101 of the 3D automatic surveying device 100 described later.

Similarly, the in-image reference point designating unit 22 designates a predetermined reference point whose absolute coordinates are known in advance in the video captured by the surrounding image capturing unit 101. -The 3D automatic surveying device described later is specified by the in-image preparation work unit 20. The target measurement point is specified and specified in the image at the 100 measurement point specifying unit 104, and similarly, the target reference point is specified and specified in the image at the reference point specification unit 105. Will be done. '

Specifically, in the in-image preparation work section 20, for example, an operation is performed in which an operator marks desired measurement points and reference points in the image.

By providing the in-image preparation work section 20 in this way, the on-site preparation work can be omitted as much as possible. As a result, the 3D automatic surveying device of the present embodiment can be positioned as an in-image 3D measuring device, and all the on-site preparation work for designating a measurement target in the on-site measurement preparation work is omitted. Outside work can be performed only by the surrounding image capturing unit 101.

[0032] Then, through the above-described preparation work for designating measurement points and reference points in the on-site preparation work unit 10 or the in-image preparation work unit 20, specific survey processing is performed in the 3D automatic surveying apparatus 100.

In the 3D automatic surveying device 100, video feature points including measurement points and reference points are automatically extracted from images taken by a 360-degree omnidirectional camera, and the feature points are automatically tracked between frame images. Thus, the camera vector can be obtained first.

Then, if the camera vector is obtained by automatic extraction and tracking of feature points, calibration can be performed based on, for example, an object having a known length in the image, and the absolute length can be obtained. Since the camera height at the time of shooting can also be a reference for the absolute length, it is desirable to keep the camera height at the time of shooting constant.

If the camera vector is obtained, the three-dimensional coordinates of an arbitrary point can be obtained from the camera coordinates. Further, if the three-dimensional coordinates of an arbitrary point are obtained, the three-dimensional distance between two points, or the area or volume can be easily obtained.

[0033] Specifically, as shown in FIG. 1 or 2, the 3D automatic surveying apparatus 100 includes an omnidirectional image capturing unit 101, an image recording unit 102, a feature point extracting unit 103, a measurement point specifying unit 104, A reference point specifying unit 105, a corresponding point tracking unit 106, a vector calculation unit 107, an error minimization processing unit 108, an absolute coordinate acquisition unit 109, a measurement data recording unit 110, and a measurement data display unit 111 are provided.

[0034] The omnidirectional image photographing unit 101 receives all 360-degree omnidirectional cameras, such as a vehicle-mounted camera. All measurement points and base-ground-points are photographed as moving images or continuous still images.

In the photographing by the omnidirectional image photographing unit 101, for example, one camera is mounted on a vehicle or the like, and a desired survey area is photographed by using the movement of the vehicle. The image captured by the surrounding image capturing unit 101 is subjected to the image analysis according to the present invention, so that the in-image survey is performed at a desired measurement point. .

When photographing by the surrounding image photographing unit 101, the driving range of the vehicle

By increasing ¾, it is possible to secure a long line and a baseline.

It is also possible to measure by using the distance between frames for long distance, medium distance and short distance.

The image recording unit 102 records an image photographed by the surrounding image photographing unit 101.

The feature point extracting unit 103 extracts a portion having a visual feature other than the designated measurement point and reference point in the image recorded in the image recording unit 102 as a feature point.

The video feature point extraction by the feature point extraction unit 103 enables a required number of feature points to be automatically extracted from an image by an image processing technique. 'For example, if a “corner” portion in an image is designated as a feature point and only the “corner” portion is selectively extracted by image recognition, it becomes a feature point.

The measurement point specifying unit 104 automatically extracts measurement points from the image recorded in the image recording unit 102.

The reference point specifying unit 105 automatically extracts reference points (coordinate reference points and Ζ or length reference points) from the image recorded in the image recording unit 102.

As described above, the extraction of the measurement points and the reference points in the measurement point specifying unit 104 and the reference point specifying unit 105 can be performed by using the marks attached to the actual measurement points and characteristic points by Marks and the like added to the image by the preparation work unit 20 are automatically performed by image recognition.

[0037] Corresponding point tracking section 106 tracks measurement points, reference points, and feature points in each frame image and associates them.

The vector calculation unit 107 calculates the three-dimensional coordinates of the measurement point, the reference point, the feature point, and, if necessary, the camera coordinates and the rotation (camera vector). The error minimizing unit 8 repeats the operation in the vector operation unit 107, thereby repeating the overlap operation so as to minimize the error of the obtained three-dimensional relative coordinates, performs statistical processing, and increases the accuracy of the operation.

The absolute coordinate acquisition unit 109 converts the obtained three-dimensional relative coordinates into the absolute coordinate system from the known coordinates of the reference point, and converts the three-dimensional relative coordinates into all of the measurement points, the reference points, the feature points, or necessary predetermined points. Give absolute coordinates.

[0038] When absolute coordinates such as latitude and longitude are not required, the length can be calibrated for each image using the length reference point indicating the length standard, and the scale can be adjusted. Can be obtained.

In this case, the vector calculation unit 107 obtains three-dimensional coordinates at both ends of the length reference point, and calculates the distance between the two length reference points from the obtained three-dimensional coordinates. Then, the error minimization processing unit 108 repeats the overlap calculation so that the distance between the two length reference points obtained by the calculation by the vector calculation unit 107 matches the known length of the length reference point. Statistical processing.

Of course, the coordinate reference point and the length reference point can be used simultaneously, in which case the accuracy can be further improved.

[0039] The measurement data recording unit 110 calculates and records the final coordinates of the measurement point.

Then, the measurement data display section 111 displays the measurement data.

Here, the measurement data recorded in the measurement data recording unit 110 and displayed on the measurement data display unit 111 is three-dimensional coordinate information of a measurement point, a reference point, and a feature point. For example, it may be a “table” of numerical values indicating three-dimensional coordinates, or a “point” indicating the position of a measurement point on a map. Numerical values indicating three-dimensional coordinates can be indicated by, for example, values of XYZ coordinates or values of latitude, longitude, and altitude.

[0040] The 3D automatic surveying apparatus 100 having the above-described configuration reads marks and the like indicating measurement points and feature points in a captured image, and uses the epi-polar geometry together with other visual feature points. The three-dimensional position can be calculated.

Although the result can be obtained even if the calculation is performed only for the measurement point and the reference point, the accuracy is further improved by using the feature points in the image other than the measurement point. Feature points are automatically extracted from the image. Will be issued.

In addition, it is not always necessary to obtain the camera position, but by obtaining the camera position first, the calculation is simplified and the calculation becomes easy with respect to the increase of the measurement points and the reference points.

Hereinafter, with reference to FIG. 3 and subsequent figures, the process of extracting feature points in the 3D automatic surveying apparatus 100 and the process of calculating the three-dimensional relative coordinates of the feature points and camera positions based on the extracted feature points will be described in more detail. explain. ..

There are several methods for detecting a three-dimensional relative coordinate between a feature point and a camera vector in a plurality of images (moving images or continuous still images), but the 3D automatic surveying apparatus 100 according to the present embodiment employs several methods. By extracting a sufficiently large number of feature points in an image and automatically tracking them, the three-dimensional relative coordinates of the feature points and camera position and the camera's three-axis rotation are obtained by epipolar geometry. It is like that.

By taking a sufficient number of feature points, camera vector information is duplicated, and an error can be minimized from the duplicated information, and a more accurate camera vector can be obtained.

Here, the camera vector refers to a vector of the degree of freedom of the camera.

In general, a stationary three-dimensional object has six degrees of freedom: position coordinates (X, Y, Z) and rotation angles (Φχ, Φ _Υί Φζ) of each coordinate axis. Therefore, the camera vector refers to a vector with six degrees of freedom of the camera position coordinates (X, Υ, Ζ) and the rotation angles (Φ 回転, Φγ, Φζ) of the respective coordinate axes. In addition, when the camera moves, the force in which the moving direction is included in the degree of freedom. This can be derived by differentiating the above six degrees of freedom force.

As described above, in the detection of the camera vector by the 3D automatic survey device 100 of the present embodiment, the camera takes six degrees of freedom values for each frame and determines six different degrees of freedom for each frame. It is to be.

Hereinafter, a specific method of detecting a camera vector in the 3D automatic surveying apparatus 100 will be described with reference to FIG.

First, the feature point extraction unit 103 automatically extracts a point or a small area image to be a feature point from a properly sampled frame image. The measurement point and the feature point indicated by the mark or the like designated in the image are automatically extracted by the measurement point specifying unit 104 and the reference point specifying unit 105. Extracted The corresponding points of the feature points, the measurement points, and the reference points are automatically determined by a corresponding point tracking unit 106 between a plurality of frame images.

More specifically, the feature points that are sufficiently necessary and serve as a reference for detecting the camera vector are obtained. Also, a desired number of measurement points are specified, and at least two reference points whose absolute coordinates are known are specified. Figures 3 to 5 show examples of correspondences between feature points (or measurement points and reference points) for which correspondence is required between images. In the figure, “+” is the automatically extracted feature point, and the correspondence is automatically tracked between a plurality of frame images (corresponding points shown in FIG. 5; see! To 4). Here, as shown in FIG. 6, it is desirable to extract and extract a sufficiently large number of feature points in each image (refer to the mark “〇” in FIG. 6). About 100 feature points are extracted.

In the vector calculator 107, the three-dimensional relative coordinates of the extracted feature point, measurement point, and reference point are calculated, and the camera vector is calculated based on the three-dimensional relative coordinates. More required. Specifically, the vector calculation unit 107 calculates the positions of a sufficient number of features existing between consecutive frames, the position vectors between moving cameras, the three-axis rotation vector of the cameras, the camera positions and the feature points. Continuously calculate the relative value of each type of three-dimensional vector, such as the vector connecting the (measurement point, reference point), etc.

In this embodiment, a 360-degree omnidirectional image is used in principle as a camera image, and camera motion (camera position and camera rotation) is calculated by solving an epipolar equation from the epipolar geometry of the 360-degree omnidirectional image. ing.

[0045] The 360-degree omnidirectional image is, for example, a panoramic image, an omnidirectional image, or a 360-degree omnidirectional image captured by a camera with a wide-angle lens or a fisheye lens, a plurality of cameras, a rotating camera, or the like. Since a wider range is shown than the video taken by the camera, it is preferable because a high-precision camera vector calculation can be calculated easily and quickly. It should be noted that even though it is a 360-degree full-circle image, a part of the 360-degree entire circumference, which is not necessarily only an image including the entire 4π space, can be treated as a video for camera vector calculation. In this sense, the video taken by a normal camera can be regarded as a part of the 360-degree omnidirectional video, and the excellent effect as in this embodiment is small, but there is essentially no difference. It can be handled in the same way as the 360-degree omnidirectional image (4% image) of the present invention. [0046] Images 1 and 2 shown in Fig. 5 are-, 360 degree omnidirectional images are Mercator-expanded, and if latitude Φ and longitude Θ, points on image 1 are (0 1, φ 1), The point on image 2 is (0 2, φ 2). And the spatial coordinates of each camera are zl = (cos φ lcos' e1, cos φ lsin Θ 1, βϊη 1), z2 = (cos φ 2cos Θ 2, cos φ 2sin Θ 2, sin φ 2) It is. Assuming that the camera movement vector is t and the camera rotation matrix is R. zl ^T [t] XRz2 = 0 is the epipolar equation.

By giving a sufficient number of feature points, t and R can be calculated as solutions by the method of least squares by linear algebra. This operation is applied to a plurality of corresponding frames to perform the operation.

(2) Here, as described above, a 360-degree full-circumference image is used as an image to be used for calculating the camera stickiness.

Any image can be used in principle for the camera vector calculation, but a wide-angle image such as the 360-degree omnidirectional image shown in Fig. 5 makes it easier to select many feature points and enables long tracking. Become. Therefore, in the present embodiment, the 360-degree omnidirectional image is used for the camera vector calculation, whereby the tracking distance of the feature points can be lengthened, a sufficient number of feature points can be selected, and the distance can be increased. It will be possible to select characteristic points that are convenient for distance, medium distance and short distance. In addition, when correcting the rotation vector, the calculation processing can be easily performed by adding the polar rotation conversion processing. From these facts, more accurate calculation results can be obtained.

In addition, Fig. 5 shows the spherical image around 360 degrees, which is composed of images taken by one camera (or multiple cameras), in a map format in order to make it easier to understand the processing in the 3D automatic surveying device 100. In the actual 3D automatic surveying apparatus 100, it is not always necessary to use an image developed by the Mercator projection.

[0048] Next, the error minimization processing unit 108 uses the plurality of camera positions corresponding to each frame and the number of feature points to generate a plurality of vectors based on each feature point by using a plurality of operation equations. Then, statistical processing is performed so that the distribution of each feature point (measurement point, reference point) and the camera position is minimized, and the final vector is obtained. For example, for camera positions, camera rotations, and multiple feature points for multiple frames, Levenb erg-Mar quai-dt

The optimal solution of the least squares method is estimated by the method, and the errors are converged to obtain the three-dimensional relative coordinates of the camera position, camera rotation matrix, feature points, measurement points, and reference points. '

Further, feature points having a large coordinate error distribution are deleted, and recalculation is performed based on other feature points, thereby improving the accuracy of the calculation at each feature point and camera position. In this manner, the tertiary relative coordinates indicating the positions of the feature point, the measurement point, the reference point, and the camera vector can be obtained with high accuracy.

FIGS. 7 to 9 show examples of three-dimensional coordinates of feature points (measurement points and reference points) obtained by the 3D automatic surveying device 100 and camera vectors. 7 to 9 are explanatory diagrams showing the vector detection method of the present embodiment, and show the relative positional relationship between the camera and the object obtained from a plurality of frame images obtained by the moving camera. FIG.

In FIG. 7, the three-dimensional coordinates of the feature points 1 to 4 shown in images 1 and 2 of FIG. 5 and the camera vector moving between image 1 and image 2 are shown.

8 and 9 show a sufficiently large number of feature points, the positions of the feature points obtained from the frame images, and the positions of the moving cameras. In the same figure, a mark continuous in a straight line at the center of the graph indicates the camera position, and a mark located around the mark indicates the position and height of the feature point.

Here, the calculation in the 3D automatic surveying apparatus 100 is performed as shown in FIG. 10 in order to obtain three-dimensional information of feature points, measurement points, reference points, and camera positions with higher accuracy. Multiple feature points are set according to the distance between feature points ('measurement points, reference points) from force, and multiple calculations are repeated.

Specifically, the 3D automatic surveying apparatus 100 automatically detects feature points having image characteristics in an image, and calculates feature points, measurement points, Focusing on the _n- th and _n + _m- th frame images Fn and Fn + m used for the reference point or camera vector calculation, the unit calculation is performed, and the unit calculation in which n and m are appropriately set is repeated. m is the frame interval, and the feature points are classified into multiple stages according to the distance to the feature points (measurement points, reference points) in the camera image, and the distance from the camera to the feature points (measurement points, reference points) is long. The distance from the camera to the feature point (measurement point, reference point) is set so that Set so that m becomes smaller as the distance becomes closer. The reason for this is that the farther the distance from the camera to the feature point, the smaller the change in position between images.

[0051] Then, while the classification of the feature points by the m-values is sufficiently overlapped, m is set in a plurality of stages, and as n progresses continuously as the image progresses, the operation is continuously performed. Proceed to Then, in the progress of n and each stage of m, the duplicate operation is performed a plurality of times for the same feature point.

In this way, by performing the unit calculation focusing on the frame images Fn and Fn + m, it takes a long time between each frame sampled every m frames (frames are dropped between frames). A precise camera vector is calculated, and m frames (minimum unit frames) between the frame images Fn and Fn + m can be simplified calculations that can be performed in a short time.

If there is no error in the precision camera vector calculation for every m frames, both ends of the camera vector of the m frames will be overlapped with the Fn and Fn + m camera vectors that have been subjected to the high precision calculation. Therefore, the m minimum unit frames between Fn and Fn + m were obtained by simple calculation, and both ends of the camera vector of the m minimum unit frames obtained by simple calculation were obtained by high precision calculation. The scale of m continuous camera vectors can be adjusted to match the camera vectors of Fn and Fn + m.

In this way, as n progresses continuously as the image progresses, the error of the three-dimensional relative coordinates of each feature point, measurement point, reference point, and camera vector obtained by calculating the same feature point multiple times is obtained. Can be scaled and merged to minimize, and the final three-dimensional relative coordinates can be determined.

Thus, the calculation processing can be sped up by combining the simple calculation while obtaining the high-precision three-dimensional relative coordinates without any error.

[0053] There are various methods for simple calculation according to the accuracy. For example, in (1) high-precision calculation, when many feature points of 100 or more are used, simple calculation is used. A method using a minimum of about 10 feature points, or (2) Even if the number of feature points is the same, if the feature points and the camera position are considered equally, innumerable triangles will be established there, and only the number of triangles will be established. Since the following equation is satisfied, a simple operation can be performed by reducing the number of equations.

In this way, the scale is adjusted so that the error between each feature point and camera position is minimized. By performing the distance calculation and deleting the feature points with a large error distribution, and recalculating the other feature points as necessary, the accuracy of the calculation at each feature point and camera position Can be raised.

When the three-dimensional position relative coordinates of each point are obtained as described above, the absolute coordinate obtaining unit 109 replaces the three-dimensional relative coordinates with a known reference point whose absolute coordinates are measured in advance. Are given, the three-dimensional relative coordinates are converted into an absolute coordinate system, and absolute coordinates are given to all points (or required predetermined points) of the measurement points, the reference points, and the feature points. As a result, the final absolute coordinates of the desired measurement point and the arbitrarily designated point in the feature point are obtained, the measurement data is recorded in the measurement data recording unit 110, and the measurement data display unit It will be displayed and output via 111.

In the above description, the measurement point, the reference point, the feature point, the camera coordinates and the rotation (camera vector) are simultaneously determined by the vector calculation unit 107. Once the camera vector is determined, a new For the measurement points, feature points, and any specified points in the feature points, two images, that is, vertices whose bases are the two camera positions, are obtained from the already obtained camera vector without recalculation together with the camera vector. It can be easily calculated as one point.

That is, for an arbitrary measurement point, its three-dimensional absolute coordinates can be obtained by calculation based on the already obtained camera vector.

In this case, as shown in FIG. 11, the 3D automatic surveying apparatus 100 determines that the measurement point specifying unit 104 determines a desired measurement point in a 360-degree omnidirectional image in which a camera vector is obtained for a predetermined reference point. Specified · Automatically extracted or manually 'specified' extracted and extracted measurement points Force Measurement point tracking unit 104a tracks and associates each frame image. The tracking of the measurement points in the measurement point tracking unit 104a is performed in the same manner as the corresponding point tracking in the corresponding point tracking unit 106 described above.

Then, based on the measurement points that are tracked and associated in each frame image and specified, the measurement point measurement calculation unit 104b uses the two images, that is, two images, based on the camera vectors already obtained. The three-dimensional absolute coordinates can be easily and quickly obtained by calculating one vertex having the bottom of each camera position. Even in this case, since the accuracy of the camera vector does not change, the accuracy of a new measurement point, a feature point, and any designated point does not change. However, if a camera vector is calculated again and recalculated, the accuracy generally improves.

The measurement point, the reference point, and the feature point are names that are distinguished in work processing, and are essentially equivalent points in coordinate calculation. There is no. Therefore, in the present invention, not only points, places, and regions designated as measurement points in advance, but also any points (designated points) designated thereafter, the three-dimensional position or the three-dimensional position between any two points. It is possible to measure a distance, an area, and a volume (see a second embodiment described later).

In other words, by setting the measurement points at the site from the beginning, it is possible to extract the measurement points from the video and perform the measurement operation later. However, as long as the measurement point can be specified manually or automatically in the video after shooting, measurement calculation can be performed using that point as the measurement point.

As described above, according to the 3D automatic surveying apparatus according to the present embodiment, a plurality of frame image powers of a moving image obtained by a 360-degree omnidirectional camera are extracted with a sufficient number of feature points.

As a result, three-dimensional relative coordinates indicating the relative positions of a large number of feature points including a desired measurement point can be obtained with high accuracy. Then, the obtained three-dimensional relative coordinates can be converted into an absolute coordinate system based on the known three-dimensional absolute coordinates with respect to a reference point previously obtained by surveying or the like.

If it is not necessary to obtain absolute coordinates, the absolute coordinates such as latitude / longitude, etc., may be determined in advance based on a known length in surveying or by placing an object with a known length around the measurement point. Even if the coordinate values cannot be obtained, the correctness of the scale and the measurement result can be obtained. Accordingly, in the present embodiment, in principle, one camera captures an image with a camera arbitrarily moving in free space, and specifies a desired survey point in the image, or By capturing images of survey points with landmarks and analyzing them, extremely accurate 3D surveys can be performed.

As described above, by analyzing a moving image obtained by one camera, it is possible to obtain three-dimensional absolute coordinates of a desired object or the like. By generating dimensional information, it is possible to minimize the error as much as possible.It is not necessary to use multiple cameras, and any image in the image that is not affected by camera vibration or shaking. High-precision three-dimensional measurement of an object can be performed.

That is, in the present embodiment, the same measurement point is analyzed by analyzing a moving image composed of a large number of frame images including a desired measurement point by moving one camera rather than by parallax between the two cameras. Many frame images can be used, and calculations with high accuracy can be performed with sufficient information.

[Second Embodiment]

Next, a second embodiment of the 3D automatic surveying device of the present invention will be described with reference to FIGS.

FIG. 12 is a block diagram showing a schematic configuration of the 3D automatic surveying device according to the second embodiment of the present invention.

The 3D automatic surveying device shown in the same drawing is a modified embodiment of the above-described first embodiment, and has the same configuration as the 3D automatic surveying device (see FIGS. 1 and 2) shown in the first embodiment, and further has a distance. The operation unit 112 and the area / volume operation unit 113 are further added.

Therefore, other components are the same as those in the first embodiment, and the same components are denoted by the same reference numerals, and detailed description is omitted.

[0060] Specifically, the 3D automatic surveying apparatus 100 of the present embodiment shown in Fig. 12 performs three-dimensional measurement between desired two points based on absolute coordinate data of measurement points, reference points, feature points, or camera vectors. A distance calculation unit 112 for obtaining a distance, and an area / volume calculation unit 113 for obtaining an area or volume of a desired region based on a plurality of distances between two points obtained by the distance calculation unit 112 are provided. The distance calculation unit 112 starts from an arbitrary measurement point or an arbitrary feature point in an arbitrary image recorded in the image recording unit 101, and obtains an arbitrary measurement point or an arbitrary characteristic in the image or another different image. A point is designated as an end point, and a three-dimensional distance between arbitrary start points and end points designated in the same image or between different images is calculated based on the absolute coordinates recorded in the measurement data recording unit 110.

The area / volume calculation unit 113 designates a plurality of points within the same image or between different images recorded in the image recording unit 101, and specifies a plurality of points between the start point and end point obtained by the distance calculation unit 112. The three-dimensional distance measurement is combined with a number of neurons, and the area or volume of a desired object within the same image or between different images is obtained by calculation.

With reference to FIG. 13 to FIG. 16, a specific method of three-dimensional distance measurement and area · ¼: product operation will be described.

As shown in FIG. 13, in the present embodiment, as the pre-processing, as in the case of the first embodiment described above, the camera positions and rotations (camera vectors) of all the captured frame images are determined (

5001), the camera vector obtained once is stored as data and tabulated (

5002). In this way, the three-dimensional coordinate calculation can be simplified and speeded up by using the already obtained camera vector. This processing is performed by the vector calculation unit 107 and the error minimization processing unit 108, as in the first embodiment.

Next, when the three-dimensional distance measurement between the two is performed, first, the image recorded in the image recording unit 102 is displayed on a display or the like, and the displayed arbitrary image (the arbitrary frame Fn shown in FIG. 14). In (), an arbitrary measurement point or an arbitrary point is designated as a start point (S003). Next, an arbitrary measurement point or an arbitrary point in the image or another different image (arbitrary frame Fn + m shown in FIG. 14) is designated as an end point.

The designation of the start point and the end point can be performed, for example, with a mouse or the like.

[0062] When an arbitrary start point and end point are designated, the points are regarded as measurement points, and automatic tracking is performed between frame images (S005 to S006). The automatic tracking of the corresponding points is performed by the corresponding tracking unit 106 as in the first embodiment.

For the start point and end point that correspond between the frame images, three-dimensional coordinate calculation is performed on each image using the camera vector already obtained in S001 to S002 using the known camera vector, and the final absolute coordinates are obtained Is done. Similar to the first embodiment, the coordinate calculation processing of the corresponding point is performed by the vector calculation unit 107, the error minimization processing unit 108, and the absolute coordinate acquisition unit 109, and the data is stored in the measurement data recording unit 110.

[0063] Then, by reading the absolute coordinate data of the known start point and end point, the three-dimensional distance force S between the two points of the start point and end point is calculated (S007). This three-dimensional distance calculation is performed in the distance calculation unit 112.

Here, the distance calculation between the two points is performed assuming that the absolute coordinates of the start point and the end point are known. The calculation, that is, the three-dimensional measurement can be performed not only when the start point and the end point are both in the same image but also when the start point and the end point are in different images (see FIG. 14). The obtained three-dimensional distance between the two points can be displayed and output as needed, for example, via the measurement data display unit 111 (S008).

Further, by repeating the three-dimensional distance measurement between the above-mentioned start point and end point (S009), the area or volume is calculated based on a desired region or the like by combining the obtained three-dimensional distances. (S010).

That is, as shown in Fig. 15, by specifying a plurality of measurement points or arbitrary specified points, the three-dimensional coordinates and three-dimensional distance of each point are obtained, and by calculating them, the area or volume can be calculated. You can ask. This area or volume calculation is performed in the area / volume calculation unit 113.

As a result, the area or volume in the three-dimensional coordinate system where the object in the image exists is three-dimensionally measured by calculation, and the result can be displayed and output as necessary (S011).

FIG. 16 shows an example of an image in which an arbitrary point for obtaining a three-dimensional distance is designated in the same image by the 3D automatic surveying device of the present embodiment.

FIG. 16 (a) is an arbitrary image (camera vector image) for which a camera vector has been obtained, and an arbitrary point can be designated in such an arbitrary image using a mouse or the like. Specifically, as shown in FIG. 16 (b), any two points for obtaining the three-dimensional distance can be specified, and the specified two points are connected by a straight line.

Then, the three-dimensional distance between the two designated points is obtained by the above-described calculation, and the result is output and displayed in a predetermined format.

As described above, the 3D automatic surveying apparatus 100 of the present embodiment utilizes the fact that high-precision absolute coordinates can be obtained for an arbitrary measurement point, and obtains a three-dimensional distance specifying a desired start point and end point. Measurement can be performed.

This makes it possible to perform high-precision three-dimensional distance measurement without any distance restrictions, even between any two points specified across multiple frame images that are not only between two points in the same image. Become. Furthermore, by designating three or more points, it is possible to three-dimensionally measure the area or volume of any object or region within an image or across multiple images.

[Third Embodiment]

Next, a third embodiment of the 3D automatic surveying device of the present invention will be described with reference to FIG.

FIG. 17 is a block diagram showing a schematic configuration of the 3D automatic surveying device according to the third embodiment of the present invention.

The 3D automatic surveying device shown in the figure is a modified embodiment of the above-described first embodiment, and the 3D automatic surveying device shown in the first embodiment (see FIGS. 1, 2 and 11) has a plano unevenness measuring device. This is the addition of the quantity device 200.

Specifically, as shown in FIG. 17, the plane unevenness surveying apparatus 200 of the present embodiment includes a plane detailed image acquisition unit 201, a parallel image recording unit 202, a plane unevenness three-dimensional measurement unit 203, and a coordinate integration unit. 204, an integrated measurement coordinate recording unit 205, and a total measurement data display unit 206.

[0068] The plane detailed image acquisition unit 201 mounts a plurality of cameras on a vehicle or the like, and captures an uneven plane portion such as a road existing along a traveling path or the like.

The parallel image recording unit 202 records a plurality of images having parallax captured by the plane detailed image acquisition unit 201.

The plane unevenness three-dimensional measuring unit 203 three-dimensionally measures the unevenness of the plane from the image having the parallax recorded in the parallel image recording unit 202.

The coordinate integration unit reads out the absolute value three-dimensional data of the plane portion three-dimensionally measured by the three-dimensional unevenness three-dimensional measuring unit 203 from the measurement data recording unit 109 (see the first embodiment) of the 3D automatic surveying device 100. Integrate the three-dimensional data with the coordinates of the plane unevenness. 'The coordinate-integrated data is recorded in the integrated measurement coordinate recording unit 205 and, if necessary, displayed and output via the integrated measurement data display unit 205.

[0069] According to the planar unevenness surveying apparatus 200 of the present embodiment as described above, the 3D automatic surveying apparatus 100 shown in the above-described first embodiment is basically used, and the unevenness around the road and the road is further improved. It is possible to realize a 3D automatic surveying device capable of performing three-dimensional measurement on. To measure the unevenness of the road surface on which a vehicle equipped with an in-vehicle camera runs, the first embodiment As in the case of the state, it is necessary to acquire the three-dimensional coordinates of the entire surroundings, measure the road surface as a surface, and measure the unevenness.

In order to measure a road surface as a surface, for example, a mark such as a marker is put on the road surface in detail and the three-dimensional coordinates are obtained from the image of the road surface in the same manner as in the case of the measurement points in the first embodiment described above. I can do it. On a road surface with a marker or the like, three-dimensional relative coordinates are obtained from an image and given known absolute coordinates, as in the case of the measurement points in the first embodiment. The position coordinates are obtained.

[0070] However, in the case of a road surface, unlike a building or the like, it is often not possible to attach a marker due to obstacles to traffic, and even if a marker is attached, unevenness cannot be measured. It is generally difficult to apply a marker over the entire road surface to the extent possible.

Therefore, actually, it is necessary to perform three-dimensional measurement of unevenness based on an image of a road surface without a mark or the like.

Here, it is possible to catch a pattern formed by projecting light on the road surface as a mark. Therefore, in this embodiment, when it is not possible to attach a marker or the like to the road surface, as a means for measuring the unevenness of the road surface with high accuracy, the road surface is photographed with a plurality of (for example, two) cameras, and the parallax is detected. To detect the unevenness of the road surface.

More specifically, the plane detailed image acquisition unit 201 acquires images without time lag by synchronously photographing a desired road surface with a plurality of cameras installed in parallel, and acquires images without time lag. To record. -Then, the three-dimensional coordinates are acquired from the parallax of the recorded video, and the three-dimensional measurement of the road surface is performed in the three-dimensional unevenness measuring unit 203.

The irregularities obtained here are only relative values and do not have absolute coordinates, so they are still incomplete as distortions in the entire scale. Therefore, the entire scale is detected by the 3D automatic surveying apparatus 100 of the first embodiment, and only the unevenness of the road surface at a short distance is measured by the parallax in the present embodiment. Then, these are coordinate-integrated in the coordinate integrating unit 204. As a result, the unevenness of the road surface can be accurately described.

[0072] In the preparation work for measuring the unevenness of the road surface in the present embodiment, marking a marker on the road surface as described above is preferable as a method of measuring three-dimensional data with high accuracy. Les ,. For example, when measuring unevenness on a flat surface where the texture is difficult to find feature points, such as unevenness on the road surface, adding a marker enables accurate three-dimensional imaging even with the continuous image power of one camera. Measurement can be performed (see the first embodiment).

The measurement of unevenness by parallax alone can achieve near-range accuracy but not long-range accuracy due to the limitation of the baseline between cameras.

Therefore, in the present embodiment, the road surface unevenness is determined by image processing using the two-camera parallax by using the flatness unevenness measuring device 200 only for the ultra-short distance measurement for measuring the unevenness of the road surface. Distance.Medium distance 'Long distance absolute coordinate measurement is performed by a marker method using one camera using the 3D automatic surveying device 100 shown in the first embodiment, and integrating these coordinates, the unevenness of the road surface Three-dimensional measurement is possible for road surface irregularities of all sizes, such as deformation, deflection, and distortion.

In addition, the deflection of the road surface due to heavy objects can be measured by comparing the measurement twice with and without load.

[Fourth Embodiment]

Further, a fourth embodiment of the 3D automatic surveying device of the present invention will be described with reference to FIG.

FIG. 18 is a block diagram showing a schematic configuration of the 3D automatic surveying device according to the fourth embodiment of the present invention.

The 3D automatic surveying device shown in the figure is a modified embodiment of the first embodiment described above. The 3D automatic surveying device shown in the first embodiment (see FIGS. 1, 2 and 11) has a tertiary road surface. This is the one in which the former map creator 300 is added.

The road surface three-dimensional map creating device 300 of the present embodiment automatically extracts a road marking portion from the 360-degree omnidirectional image obtained by the 3D automatic surveying device 100 and performs 3D surveying of the road surface to obtain a desired road surface. It is possible to create a three-dimensional map of.

Specifically, as shown in FIG. 18, the road surface three-dimensional map creating device 300 includes an image stabilizing unit 301, a traveling direction control unit 302, an image vertical plane developing unit 303, and a basic road surface shape model. Generation unit 304, Road surface 3D measurement unit 305, Road surface parameter determination unit 306, Road transparent CG generation unit 307, Synthetic road surface plane development unit 308, Road surface vector extraction unit 309, Road surface technology It includes a flexible flexible connection unit 310; a texture averaging unit 311; a target area cutout unit 312; a road marking recognition and coordinate acquisition unit 313; and a three-dimensional map generation unit 314.

[0076] The image stability unit 301 is captured by the omnidirectional image capturing unit 101 based on the camera vector with the error minimized obtained by the error minimizing unit 108 of the 3D automatic surveying apparatus 100. The rotation of the entire surrounding image is corrected, and the shaking is corrected to stabilize the image.

The traveling direction control unit 302 fixes the traveling direction of the image subjected to the stabilization processing by the image stabilizing unit 301 to a target direction, or controls movement of the image in the target direction.

The image vertical plane development unit 303 develops the image whose traveling direction is controlled by the traveling direction control unit 302 on a vertical plane (antarctic plane development). That is, in order to generate a three-dimensional road surface model, processing is performed using an image developed on a vertical plane.

In a 360-degree omnidirectional image, all directions are equal and there is no optical axis. All directions are the optical axis. Therefore, in order to display a 360-degree omnidirectional image as a plane developed image that has a perspective taken with a normal lens, set a virtual optical axis, determine the plane to be developed, and image that has a perspective on that plane Needs to be converted to;

Therefore, in order to process structures that are built on the basis of the vertical direction or road surfaces and road markings that have a slight gradient even when the vertical direction is the reference, a linear plane that is perpendicular to the coordinate axis is used. It is advantageous to simplify the work by developing the image on a plane that becomes a linear scale on the road surface including the slope of the road surface, and the ability to develop the image so that it becomes a scale. In general, vertical plane development is advantageous for obtaining three-dimensional coordinates of roads, and road plane development is advantageous for processing road surfaces and road markings.

Therefore, in the present embodiment, the image is developed in the vertical plane and processed in the image vertical plane development unit 303.

Note that this vertical plane development processing is not necessarily required for generating a ^ -dimensional map, but since the operation can be simplified, in the present embodiment, the processing is provided with the image vertical plane development unit 303. .

[0077] The basic road shape model generation unit 304 generates a basic road surface model in which the parameters of the road surface shape are undecided. The road surface three-dimensional measurement unit # 0'5 measures the three-dimensional coordinates of the road surface from the image of the road surface developed on the vertical plane by the image vertical plane development unit. Specifically, several places on the road screen of each frame image are divided into large blocks, and three-dimensional measurement is performed by correlation. Here, since the correlation is obtained over a large area, the accuracy can be improved.

The road surface parameter determination unit 306 determines each parameter of the road surface model and automatically determines the three-dimensional shape of the road surface. ..

[0078] The road transparent CG generation unit 307 acquires each parameter of the shape of the road surface from the road surface measurement data obtained by the road surface three-dimensional measurement unit, and generates a transparent CG of the road surface. . In other words, the road transparent CG generation unit 307 generates a transparent CG of the road surface using the road surface model whose shape is determined by fixing the nomometer.

The road plane development unit 308 combines the transparent CG generated by the road transparent CG generation unit 307 with the road surface image stabilized in the traveling direction by the traveling direction control unit 302, and develops the image parallel to the road surface. I do. That is, the road surface plane developing unit 308 slightly modifies the developed surface of the image obtained by the processing up to the previous process and develops the image on the road parallel plane. Since the road is not always a horizontal plane, a plane close to a linear plane (a plane with the same length in the same image has the same length) is selected for correlation processing in the next process.

[0079] The road surface vector extraction unit 309 extracts only the road surface vector and deletes the others by selecting a vector in the developed image that has been developed in a plane. The road surface can be extracted by selecting the stop vector (smallest movement vector), and the moving object with the movement vector is also deleted. .

The road surface texture flexible connection unit 310 acquires the road surface texture using the rubber strap theory of the road surface, and connects the road surface using the rubber strap theory for later processing. That is, the road surface texture flexible coupling unit 310 blocks the road surface as necessary, flexibly combines characteristic portions of the road surface without changing the texture order, and outputs the output to the next stage of the texture. Sent to averaging unit 311.

[0080] The texture averaging unit 311 extracts the road surface on the road surface of the transparent CG, performs averaging on the transparent CG, and eliminates noise. Since the distance between the camera and the road surface is constant, superposition in a stationary coordinate system is possible, and averaging is possible. The target area cutout unit 322 largely cuts out an outline of an area such as a road surface figure such as a road sign or an obstacle from the image in which the noise is reduced by the texture-averaging unit 311. For example, only an arbitrary road sign is extracted from the road surface histogram on the transparent CG. Here, the purpose is to cut out a large area, and it may be incomplete.

[0081] The road sign recognition and coordinate acquisition unit 313 recognizes a target object from the object region cut out by the object region cutout unit 312 and obtains its coordinates. For example, the extracted road marking part is recognized by PRM and the coordinates are determined. That is, PRM processing is performed on the region cut by the target region cutout unit 312. Since the three-dimensional shape of the road surface is fixed, determining the coordinates two-dimensionally improves the spirit.

Where PRM is Parts

It is an abbreviation of Reconstruction Method (3D space recognition method), and is a technology for recognizing an object that the applicant has already applied for a patent for (see International Application No. PCT / JP01 / 05387). Specifically, the PRM technology prepares in advance all the shapes and attributes of an object to be expected as parts (operator parts), and compares those parts with actual shot images to match. This is a technology for selecting a part to be recognized and recognizing the target. The `` parts '' of the objects required for automatic guidance driving and automatic driving of the vehicle include lanes, white lines, yellow lines, pedestrian crossings, speed signs as road signs, guidance signs, etc. Since these are standard types, their recognition can be easily performed using PRM technology. Also, when an object is searched for in a video (CV image) for which a camera vector has been obtained, the predicted three-dimensional space in which the object exists can be limited to a narrow range, and recognition efficiency can be improved. It becomes possible.

Then, the output of the figure whose coordinates have been determined by the road sign recognition and coordinate acquisition unit 313 is regarded as a measurement point and sent to the measurement point identification unit 104 of the 3D automatic surveying device (see FIG. 18). The 3D automatic surveying apparatus 100 acquires the absolute coordinates of the road surface figure such as the road sign and the obstacles on the road surface by the above-described processing, and outputs the acquired absolute coordinates from the measurement data recording unit 110. You. As a result, all the figures are reconstructed by the absolute coordinates and sent to the three-dimensional map generator 314 in the next step.

The three-dimensional map generation unit 314 reconstructs the output (absolute coordinates) from the measurement data recording unit, retrieves and rearranges the output as a three-dimensional figure having a predetermined specification, and outputs a three-dimensional map of a desired road surface. Generate the original map.

FIG. 19 shows an example of a case where a three-dimensional map is generated based on an image converted so as to be equivalent to an image taken on a road by aerodynamics! The road image shown in the figure is a 360-degree omnidirectional image (CV image) calculated by the camera vector using the 3D automatic surveying device 100, and is a road surface observed from several meters above ground, which is not a complete plan view. .

When generating a three-dimensional map of a road, the shape near the road surface is important, and high measurement accuracy is required. In general, since it is known in advance that the road structure has a structure as shown in the cross-sectional view of FIG. 19 (a), three-dimensional measurement can be performed by predicting the shape. '

[0084] Also, by making use of the features of the 360-degree omnidirectional image and setting the road surface display so that the viewpoint direction is directly below the road surface, matching and gripping in a wide area becomes possible. Specifically, matching and gripping in an area of about 15 * 15 pixels was usually limited in arbitrary directions, but in the direct display, the viewpoint and road surface are almost right angles, and the image between frames changes shape Since the image is moved without performing, the image distortion due to each frame can be ignored. As a result, matching and gripping (M & G) can be performed over a wide area of, for example, 50 * 50 pixels or more, and matching and gripping can be performed even on a road surface with few features, thereby improving measurement accuracy.

Furthermore, since the road markings (center line, shoulder line, etc.) are drawn on the pavement surface according to the determined standard, the pattern is displayed by the PRM operator (PRM operator).

It is possible to detect its three-dimensional position by preparing it in advance as an operator part and comparing the image with the prepared operator part.

[0085] Specifically, as a road surface operator, there is a pattern as shown in Fig. 19 (c). Note that many other patterns (not shown) are assumed as operator parts.For example, force on a three-dimensional map is not necessary, so it is not necessary to measure the entire road surface. Since the figure only needs to be completed, the degree shown in Fig. 19 is sufficient. In addition, by preparing a 3D PRM operator part (PRM 3D Operator) and performing three-dimensional matching, it is possible to accurately reproduce, for example, steps on the curb of a road. [0086] Fig. 20 shows a three-dimensional map of the road shown in Fig. 19 in a stereoscopic view.

As shown in the figure, in the image of the pavement road, the PRM operator is more effective in recognizing the three-dimensional road sign than the road surface display such as the center line shown in FIG. In other words, regarding the recognition of road signs, as shown in Fig. 20 (a), assuming an expected road sign space on the CV image, the type, position, shape and shape of the target road sign in the limited space The coordinates can be recognized.

For CV video, the expected road sign space can be combined and arranged on the actual image as CG, and the target road sign can be searched only within the limited range.

In addition, since the shape and size of the road sign are usually determined, the three-dimensional operator of each road sign prepared in advance is used as a part (see Fig. 20 (b)), and the three-dimensional It is possible to search for a sign of a predetermined size and find it. Then, the type, position, coordinates, and shape of the found sign are recognized.

[0087] As described above, a CV video image can be treated in the same way as an object having three-dimensional coordinates, and is extremely advantageous for searching. For a road sign, such as a road sign, for which the shape of the object to be searched has already been determined, the apparent size at its three-dimensional position can be calculated, so it is advantageous to use the PRM operator. By preparing various kinds of markers, it is possible to recognize the target sign by searching for a matching part from the prepared sign parts.

As described above, according to the 3D automatic surveying device 100 of the present embodiment, the 3D automatic surveying device 100 shown in the first embodiment described above is provided by including the road surface three-dimensional map creating device 300. In addition to the basics, a high-precision three-dimensional map of an arbitrary road surface can be generated.

[0088] As described above, the preferred embodiment of the 3D automatic surveying device of the present invention has been described. However, the 3D automatic surveying device according to the present invention is not limited to only the above-described embodiment, but the scope of the present invention. It goes without saying that various modifications can be made.

For example, the 3D automatic surveying device 100 shown in the above embodiment, the preparation work units 10 and 20, the distance calculating unit 112, the area 'volume calculating unit 113, the planar unevenness measuring device 200, and the Original map creator 300 is implemented in any combination However, the present invention is not limited to only the combinations shown in the above-described embodiments. Some of them may be omitted as appropriate, or all the devices may be provided at the same time. Industrial applicability

INDUSTRIAL APPLICABILITY The present invention can be used, for example, as an image surveying apparatus for obtaining the position, distance, and area of a desired measurement point based on a moving image video captured by a vehicle-mounted camera.

Claims

[1] a moving 36 ° omnidirectional camera, an omnidirectional image capturing section for capturing a moving image or a continuous still image including a predetermined measurement point and a predetermined reference point whose three-dimensional absolute coordinates are known, and the omnidirectional image An image recording unit that records an image photographed by the photographing unit;

A feature point extracting unit that extracts a certain part as a feature point from the image recorded in the image recording unit,

A measurement point specifying unit for automatically extracting the measurement points in an image recorded in the image recording unit;

A reference point specifying unit that automatically extracts the reference point in an image recorded in the image recording unit;

A corresponding point tracking unit for tracking and associating the measurement point, the reference point, and the feature point in each frame image,

A vector for calculating the three-dimensional relative coordinates based on the measurement point, the reference point, and the feature point associated with the corresponding point tracking unit and, if necessary, the camera vector indicating the position and rotation of the camera. An operation unit;

An error minimization processing unit that repeats the operation in the vector operation unit, repeats the overlap operation so as to minimize the error of the obtained three-dimensional relative coordinates, and performs statistical processing, and from the known three-dimensional absolute coordinates of the reference point, An absolute coordinate acquisition unit that converts the three-dimensional relative coordinates obtained by the vector calculation unit into an absolute coordinate system and assigns a three-dimensional absolute coordinate to the measurement point, the reference point, and the feature point;

A measurement data recording unit that records final absolute coordinates assigned to the measurement points, the reference points, and the feature points;

A display unit for displaying the measurement data recorded in the measurement data recording unit; and a 3D automatic surveying device.

[2] a 360-degree omnidirectional camera that moves, captures a moving image or a continuous still image including a desired measurement point and a predetermined reference point whose three-dimensional absolute coordinates are known, An image recording unit that records an image photographed by the image photographing unit;

In the image 內 recorded in the image recording unit, a portion having a visual feature is defined as a feature point. A feature point extraction unit for extracting

A reference point specifying unit that automatically extracts the reference point in the plane image recorded in the image recording unit;

A corresponding point tracking unit for tracking and correlating the reference point and the feature point in each frame image, and a camera vector indicating the position and rotation of the camera, the reference point and the feature point correlated by the corresponding point tracking unit. An error minimization processing unit for repeating a calculation in the vector operation unit for calculating three-dimensional relative coordinates by the calculation and the vector operation unit, repeating an overlap operation so as to minimize an error in the obtained three-dimensional relative coordinates, and performing statistical processing An absolute coordinate acquisition unit that converts the three-dimensional relative coordinates of the camera determined by the vector calculation unit from the known three-dimensional absolute coordinates of the reference point into an absolute coordinate system, and assigns three-dimensional absolute coordinates,

A measurement point identification unit that automatically extracts the measurement point in the image recorded in the image recording unit;

A measurement point tracking unit that tracks and associates the measurement points extracted by the measurement point identification unit in each frame image,

A measurement point measurement / calculation unit for calculating a measurement value of the measurement point associated with the measurement point tracking unit from the camera level determined by the vector calculation unit;

A measurement data recording unit that records the absolute coordinates of the measurement point;

[3] The reference point includes a length reference point having a known length, together with a reference point having known three-dimensional absolute coordinates, or in place of a reference point having known three-dimensional absolute coordinates,

The vector calculation unit calculates the distance between the two length reference points by calculation, and the error minimization processing unit calculates the distance between the two length reference points obtained by the calculation in the vector calculation unit. The 3D automatic surveying device according to claim 1 or 2, wherein the overlap calculation is repeated and statistical processing is performed so that the distance matches the known length of the length reference point.

[4] In the image recorded in the image recording unit, within the image to designate an arbitrary measurement point Measurement point designation work unit, and to designate an arbitrary reference point in the image recorded in the image recording unit An in-image preparation operation unit having an in-image reference point designation operation unit. The in-image preparation operation unit designates an arbitrary measurement point and a reference point in the measurement point specifying unit and the reference point specifying unit. 3. The method according to claim 1, wherein

[5] The vector operation unit includes:

Using the arbitrary two frame images Fn and Fn + m (m = frame interval) used for the three-dimensional relative coordinate calculation of the measurement point, reference point, feature point or camera vector as a unit image, obtain desired three-dimensional relative coordinates Repeat the unit operation,

The error minimization processing unit,

As n progresses continuously as the image progresses, scale adjustment is performed so that the error of each three-dimensional relative coordinate obtained by calculating multiple times for the same feature point is minimized and integrated. The 3D according to any one of claims 1 to 4, wherein a final three-dimensional relative coordinate is determined.

[6] The vector operation unit includes:

The 3D according to claim 5, wherein the frame interval m is set according to the distance from the camera to the measurement point, the reference point, and the feature point such that the greater the distance from the camera, the greater the m. Automatic surveying equipment.

[7] The vector calculation unit:

A feature point having a large error distribution of the obtained three-dimensional relative coordinates is deleted, and if necessary, re-calculation is performed based on another feature point, and the accuracy of measurement point calculation is improved. The 3D automatic surveying device according to any one of 1 to 6.

C8] An arbitrary measurement point or an arbitrary feature point in an arbitrary image recorded in the image recording unit is set as a start point, and an arbitrary measurement point or an arbitrary feature point in the image or another different image is set as an end point. A distance calculating unit for calculating a three-dimensional distance between arbitrary start points and end points specified within the same image or between different images based on the absolute coordinates recorded in the measurement data recording unit. Item 3D automatic surveying device according to any one of Items 1 to 7

[9] A combination of a plurality of three-dimensional distance measurement between a start point and an end point obtained by the distance calculation unit by designating a plurality of points in an arbitrary same image or between different images recorded in the image recording unit. 9. The 3D automatic surveying device according to claim 8, further comprising an area / volume calculator for calculating an area or volume of a desired object in the same image or between different images by calculation.

[10] A traveling direction control unit that fixes or controls the image obtained by the omnidirectional image photographing unit in the traveling direction with the camera vector obtained by the vector calculation unit,

An image vertical plane development unit for developing an image stabilized in the traveling direction by the traveling direction control unit on a vertical plane;

A road surface basic form model generation unit for generating a road surface basic form model in which each parameter of the road surface shape is undetermined;

A road surface three-dimensional measurement unit that measures three-dimensional coordinates of the road surface, the image force of the road surface developed on the vertical surface by the image vertical surface development unit,

A road transparent CG generation unit that acquires parameters of the shape of the road surface from the road surface measurement data obtained by the road surface three-dimensional measurement unit and generates a transparent CG of the road surface; A synthetic road surface plane development unit that combines the transparent CG generated by the unit with the road surface image stabilized in the traveling direction by the traveling direction control unit and develops the image parallel to the road surface;

A texture averaging unit that adds and averages road surface textures to the image developed by the composite road surface plane development unit to reduce noise in the image;

If necessary, the road surface is blocked, characteristic portions of the road surface are flexibly combined without changing the order of the textures, and the output is sent to the texture averaging unit.

From the image noise reduced by the texture averaging unit, an object region cutout unit that largely cuts an outline of a region such as a road surface figure or an obstacle such as a road sign, and an object cut out by the object region cutout unit A road marking recognition and coordinate acquisition unit for recognizing a target object from the area and acquiring its coordinates, Each point of the polygon constituting the target object from which the coordinates have been acquired is input to the measurement point identification unit for obtaining absolute coordinates using measurement points as measurement points, and the output from the measurement data recording unit for which absolute coordinates have been acquired is output. A three-dimensional map generator for reconstructing a three-dimensional map of the road surface;

The 3D automatic surveying device according to any one of claims 1 to 9, further comprising a three-dimensional map generation device having: