JPH08159762A - Method and apparatus for extracting three-dimensional data and stereo image forming apparatus - Google Patents

Abstract

PURPOSE: To produce DEM data from video images. CONSTITUTION: An object region is picked up by a video means from the sky (S1). At this time, the position of a camera is measured by a differential GPS. The camera is mounted on a vibration isolator. The inclination of the camera is precisely measured based on the output of a gyroscope and the output of a magnetic bearing sensor. An external locating element is accurately determined by the matching of the fields, which are overlapped by 60% in the video image (S2). The leading line, the central line and the final line in each field are extracted and individually composited, and the continuous mosaic image, which becomes the front visual image, the visual image in direct underside and the rear visual image is formed (S3). The longitudinal parallax is removed from the continuous mosaic image (S4). The parallax difference is computed based on the front visual image and the rear visual image (or visual image in direct underside) (S5), and the altitude is computed based on the parallax difference (S6).

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for extracting three-dimensional data and a stereo image forming apparatus, and more specifically, a method and apparatus for extracting three-dimensional data from a video image and a stereo image from the video image. The present invention relates to a stereo image forming apparatus that forms a sheet.

[0002]

2. Description of the Related Art Conventionally, aerial survey technology using aerial photographs has been used to create a three-dimensional topographic map. But,
The aerial survey technology is to take a stereo image of the ground while flying a helicopter or a light aircraft over the field, and analyze and process the stereo image obtained with it, and it takes a lot of money and time just to obtain a stereo image. The analysis also requires a lot of labor and cost. When three-dimensional measurement is performed by stereo matching using an aerial photograph taken at low altitude, a matching error occurs due to the effect of occlusion. This is because the two images forming the stereo image are viewed from different line-of-sight directions, and the difference in the images due to the difference in the viewing direction makes perfect matching impossible. The conventional example tries to eliminate such an influence by using multiple ground control points.
Automation is not possible with this.

[0003]

On the other hand, the technique of creating a topographic map using video images is easy to automate. However, in the conventional technique, in order to extract the three-dimensional data of the photograph, as in the case of the aerial photogrammetry, a few anti-aircraft markers (marks whose clear three-dimensional coordinates are known) are required in the image, which is necessary. Even if a large number of anti-aircraft signs are secured, the error is inaccurate in meters and it is not practical.

A three-dimensional topographic map is useful for the management of roads, rivers, railways, etc., new route plans for it, and surveys on the development status of cities, etc., and a system for obtaining three-dimensional topographic data quickly, inexpensively and easily. Is desired. If three-dimensional topographic data is obtained, a bird's-eye view can be easily created (displayed or output to a printer), and various simulations can be performed. In addition, 3 by video image processing
If the dimensional data can be obtained, it will be easy to extract only the changed portion, and thus it will be easy to survey the development situation of cities and the like.

According to the present invention, three-dimensional data is automatically extracted.
It is an object to present a method and an apparatus for extracting dimension data.

Another object of the present invention is to provide a three-dimensional data extracting method and device for extracting three-dimensional data from a video image.

Another object of the present invention is to provide a stereo image forming apparatus for forming a stereo image (two images suitable for stereo matching) from a video image.

[0008]

According to the present invention, image data of a predetermined line of a video image is extracted to form a continuous mosaic image with different parallax. After removing the vertical parallax from those continuous mosaic images, the parallax is calculated by stereo matching. The height is calculated from the obtained parallax.

Preferably, at least three or more screens that overlap at a predetermined ratio are extracted from successive screens of photographed images, and the screens are matched at the overlapping parts to determine the external orientation element. Then, according to the external orientation element determined by this orientation calculation, the external orientation element is interpolated into each line of the continuous mosaic image, and each line of the continuous mosaic image is converted into a projection image to a predetermined altitude according to the external orientation element of the line. Convert.

[0010]

The above processes can be automated on a computer. Therefore, the entire process of obtaining the stereo image required for stereo matching from the video imaged and calculating the height is automatically performed on the computer. It becomes possible to execute, and it is possible to quickly obtain three-dimensional data such as a desired area.

Since it is a video image, it is possible to obtain abundant image information necessary for determining the external orientation element, and the accuracy of the external orientation element is improved. As a result, the height data that is finally obtained is also highly accurate. In addition, even in the matching operation of stereo images, by temporarily creating an intermediate image of the stereo images and searching for corresponding points in a chained manner, the corresponding points between stereo images can be more accurate than in the case of stereo photographs. You will be able to better determine and completely solve the occlusion problem.

[0012]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a schematic configuration block diagram of an aerial measurement system according to an embodiment of the present invention, FIG. 2 is a schematic configuration block diagram of a ground measurement system, and FIG. 3 is a schematic configuration block diagram of a ground analysis system. .

The aerial measurement system shown in FIG. 1 will be described.
The aerial measurement system shown in FIG. 1 is mounted on a helicopter in this embodiment. In this embodiment, a high-definition camera 10 is mounted on a high-precision image stabilization device (anti-vibration device) 12, and the high-definition video signal output is output from the high-definition video tape recorder 1.
Record on video tape by 4. The camera 10
Is generally downward, and is installed so that the image immediately below moves in a direction perpendicular to the scanning line. The output video signal of the camera 10 is also applied to the high quality monitor 16. As a result, the subject and shooting condition of the camera 10 can be visually confirmed.

The high-precision anti-vibration stabilizer 12 prevents the vibration from the machine body from affecting the camera 10. This allows
You will be able to record video without blur. That is, the high-precision anti-vibration stabilizer 12 combines the gyro and the gimbal servo so that the 10 optical axes of the camera can provide an inertial space against the fluctuations of the roll axis, the pitch axis, and the yaw axis around the machine body. It has a space stabilization function that always faces in a certain direction.

Reference numeral 18 collects and records measurement data, controls the image stabilization device 12 via the three-axis control device 20, controls the camera 10 via the camera control device 22,
It is a personal computer that controls the VTR 14 via the VTR controller 24. The three-axis control device 20 can arbitrarily set the target orientation of the image stabilization device 12, the camera control device 22 controls the focus, zoom, aperture value, color balance, etc. of the camera 10, and the VTR control device 24 controls the VTR 14 The recording start, the recording end, and the pause of the computer 18 are controlled, and the time code recorded together with the output video signal of the camera 10 is acquired.
Transfer to. This time code is used to synchronize the video information recorded on the VTR 14 with other measurement data when analyzed by the ground analysis system.

The ground altitude sensor 26 detects the ground altitude, and the magnetic azimuth sensor 28 detects the magnetic azimuth. Since the azimuth drift causes a slow directional movement even by the high-precision image stabilization device 12, it is necessary to correct the orientation of the camera 10 by the magnetic azimuth sensor 28. The outputs of the sensors 26, 28 are applied to the computer 18 as digital data. In addition, 3-axis gyro data (roll angle, pitch angle and yaw angle) indicating the 3-axis direction of the camera 10 is input to the computer 18 from the image stabilization device 12, and the zoom value indicating the zoom value is input from the camera 10.
Data to enter.

Reference numeral 30 is a GPS (Global Positioning System) receiving antenna, and 32 is a GPS receiving apparatus for collecting current position coordinates (latitude, longitude and altitude) from the received signal of the GPS antenna 30. GP output from the GPS receiver 32
The S positioning data is applied to the computer 18 for recording and also to the navigation system 34 for navigation. The navigation system 34 three-dimensionally graphically displays the current position with respect to the set survey line on the screen of the monitor 38 according to the navigation data (measurement line data) recorded in the floppy 36 in advance. As a result, it is possible to perform imaging along a desired survey line in an area where there is no target on the ground or an area that is difficult to understand (for example, a mountainous area or a sea area).

As a method of improving the measurement accuracy of GPS, a differential GPS (D-GPS) method is known in which the GPS is measured even at a reference point whose coordinates are known and the GPS positioning data is corrected by the measurement error. . In this embodiment, this differential GPS system is adopted, the coordinates of a reference station whose coordinates are already known are simultaneously measured by GPS, and the measurement error data is transmitted to the helicopter as GPS correction data by wireless communication. The communication device 40 receives the GPS correction data from the reference station and transfers it to the computer 18.

The computer 18 inputs flight data (ground altitude data, magnetic direction data, zoom data, 3
Along with the axis gyro data), GPS correction data, and the time code from the VTR controller 24, the data is recorded on the floppy 42. The computer 18 can also display each input data on the monitor 44 as needed, and an operator can input various instructions to the computer 18 from the keyboard 46.

In the present embodiment, as shown in FIG. 2, the reference station also stores the measured GPS correction data in a floppy disk in case the communication device 40 fails to communicate with the reference station. That is, the GPS receiving device 50 is
The current position of the GPS antenna 52 is calculated from the output of the PS antenna 52, and the GPS positioning data is output to the computer 54. Accurate coordinates (reference position data) of the GPS antenna 52 are measured in advance, and the data are input or set in the computer 54. Computer 54
Calculates the error between the GPS positioning data from the GPS receiver 50 and the reference position data, and records it in the floppy 56 as GPS correction data. Of course, the measurement time information is also recorded at the same time. GPS positioning data and its error (ie,
The GPS correction data) is displayed on the screen of the monitor 58 as necessary. The operator can input various commands to the computer 54 using the keyboard 60. The computer 54 also sends GPS correction data to the computer 1 of the aerial measurement system shown in FIG. 1 via the communication device 62.
Send to 8).

Each data measured by the aerial measurement system shown in FIG. 1 (and the ground measurement system shown in FIG. 2 if necessary) is analyzed by the ground analysis system shown in FIG. 3 to calculate three-dimensional data. That is, high-quality VTR7
0 reproduces the video tape recorded by the aerial measurement system shown in FIG. 1, the video image signal is stored in the frame buffer 74, and the reproduced time code is engineered.
Apply to workstation 76. The video data temporarily stored in the frame buffer 74 is applied to the monitor 78 and displayed as a video. It goes without saying that the reproduced time code may be displayed on the monitor 78 at the same time.

The personal computer 80 collects flight data and GPS correction data simultaneously collected by the aerial measurement system shown in FIG. 1 (in the case of communication failure, GPS correction data measured by the ground measurement system shown in FIG. 2). Is read out, the GPS positioning data is corrected with the GPS correction data, the three-axis gyro data is corrected with the magnetic direction data, and the other measurement data and the time code recorded together are transferred to the workstation 76. The workstation 76 controls the VTR 70 according to the time code supplied from the computer 80, and causes the VTR 70 to reproduce an image having the same time code. As a result, the workstation 76 can associate the shooting condition and shooting position with the shot image at that time, and generates three-dimensional data by the calculation described in detail below.

FIG. 4 shows a flow from measurement to three-dimensional data extraction in this embodiment. First, each device shown in FIG. 1 is mounted on an aircraft, the target area is photographed while flying over the target area at a constant altitude and at a constant speed, and flight information is recorded (S1). At this time, the object to be photographed basically moves in the direction perpendicular to the scanning line of the camera 10.
The video image captured by the camera 10 is recorded on the video tape by the VTR 14. At the same time, the information on the accurate position (latitude, longitude, height) and orientation of the camera 10 is displayed by the VTR1.
It is recorded in the floppy 42 together with the time code from 4. The time code is used to synchronize the position and orientation of the camera with the reproduced video during ground analysis.

The position of the camera 10 is basically known from the GPS positioning data output from the GPS receiver 32,
In order to improve the accuracy, differential processing is performed using GPS correction data from the reference station. The differential processing may be performed on the aircraft, but in consideration of poor communication of GPS correction data, the output of the GPS receiver 32 (GP
It is preferable that the S positioning data) and the GPS correction data are separately recorded in the floppy 42 and differentially processed at the time of analysis on the ground. When there is a communication failure of the GPS correction data, the GPS positioning data is differentially processed by the GPS correction data recorded and stored by the ground measurement system shown in FIG.

Regarding the orientation of the camera 10, a value obtained by correcting the output of the gyro sensor of the image stabilization device 12 with the output of the magnetic direction sensor 28 is recorded in the floppy 42. Specifically, the tilts (pitch, roll, and yaw) of the three axes are recorded on a floppy disk. Of course, for simplification, or when the performance of the image stabilization device 12 is good, or when it may be simplified, the inclination of the camera 10 may be constant.

By the way, in the case of a high-definition camera using a 2 million pixel CCD image sensor with a photographing altitude of 1,000 feet, the focal length is 8.5 mm.
The shooting range is 339 m, the resolution is 17.7 cm, and the shooting range is 2 when the focal length is 102.0 mm.
8 m, resolution is 1.5 cm.

The recorded information (video and flight information) is reproduced and analyzed by the ground analysis system shown in FIG. As described above, the workstation 76 uses the information from the computer 80 at the time of shooting (camera position and orientation,
In addition, the VTR 70 is controlled according to the time code) to reproduce the video of the same time code. The reproduced video signal is digitized and stored in the frame buffer 74. In this way, the workstation 76
Can obtain video data and camera position and tilt data at the time of shooting, orientation calculation (S2), continuous mosaic image creation (S3), vertical parallax elimination (S4), parallax difference calculation (S5) and DEM. DEM data is output through each process of creation.

The orientation calculation (S2) will be described. In photogrammetry, when performing three-dimensional measurement from two stereo images,
An exact orientation factor for each image is required. The orientation elements include
There are an external orientation element composed of the position of the camera at the time of photographing and an inclination of three axes, and an internal orientation element composed of a camera principal point position displacement amount, a lens distortion coefficient, a film flatness and the like. As for the internal orientation element, it is possible to measure in advance what may have individual differences for each camera.

The external orientation element can be obtained by the following general photogrammetric method. That is, a scene (field image in this embodiment) in which the overlap rate becomes 60% (or almost 60%) from the flight information simultaneously recorded at the time of video recording, specifically, the position and inclination of the camera 10, Extract from the captured video. For example, in the example shown in FIG. 5, scene # 2 overlaps scene # 1 by 60%, and scene # 3
Overlaps scene # 2 by 60%. Scene # 3 also overlaps scene # 1 by 20%. The field number of the extracted scene is necessary for the interpolation processing of the external orientation element in the vertical parallax removal processing described later, so the external orientation element of each extracted scene is paired with an auxiliary storage device (not shown) (for example, Hard disk device).

However, since the overlapping portions of the three scenes extracted according to the flight information include the influence of the sensor error and the height of the terrain, it is rare that they are perfectly matched. Therefore, in practice, the overlapping portion is determined as follows.
That is, as shown in FIG. 6, a plurality of matching regions are set in the overlapping portions of each scene, and common points are set in each matching region of each scene by a matching method such as a residual sequential test method or a cross correlation coefficient method. To calculate. This calculation can of course be automated and this common point is called the pass point.

A pair of two images forming a three-dimensional image is called a model, and obtaining the relationship between relative position and inclination from the coordinate values of the path points of the overlapping parts is called mutual orientation. The model has a common path
It is called connection orientation to combine points to convert them into a unified course coordinate system. The relationship between the mutual orientation and the connection orientation is shown in FIG.

By repeatedly performing the above-described mutual orientation and connection orientation on a series of photographed images, each model can be converted into a unified course coordinate system. The flight information (external orientation element) recorded at the time of video recording contains measurement error or fluctuation depending on the measurement system, but by such orientation calculation (mutual orientation and connection orientation), the value of the external orientation element is highly accurate. Can be confirmed.

From the above, the orientation calculation (S2) is completed,
Next, a continuous mosaic image is created (S3). The relationship between the image pickup surface (CCD surface) of the camera and the terrestrial image pickup range is as shown in FIG. As is well known, a video image consists of 30 frames per second, and one frame consists of two fields, an odd field and an even field. The odd field and the even field are arranged in a staggered manner so that the scanning lines do not overlap, and are alternately displayed every 1/60 seconds.

That is, the video image photographed by the camera 10 is composed of scenes (fields) whose photographing positions are changed every 1/60 seconds. In this embodiment, as shown in FIG. Each line data of the line, the center line and the last line is extracted. An image formed from the data of the first line of each field is called a forward-view image, an image formed from the data of the center line of each field is called a direct-down view image, and an image formed from the data of the last line of each field is called a rear-view image. . If the moving speed at the time of shooting is constant and the tilt of the camera is also constant, stereoscopic viewing can be performed using these front-view image, direct-down view image, and rear-view image. In order to emphasize the height, that is, to increase the resolution, a lens with a short focal length may be used.

However, in video shooting with an aircraft, there are fluctuation factors such as changes in flight speed, deviation in flight course, changes in flight altitude, and changes in three axes (pitch, roll, and yaw). It is necessary to remove parallax (S4).

In the vertical parallax elimination processing (S4), first, based on the external orientation elements of each image having the overlapping ratio of 60% obtained in the orientation calculation (S2), continuous mosaic images (forward-view image, direct-down image and The value of the external orientation element corresponding to each line data of the backward view image) is interpolated. For example, as shown in FIG. 10, the external orientation elements at the three photographing points P, Q, and R, that is, the position and the tilt of the camera are (Xp, Yp, Zp,
ωp, φp, κp), (Xq, Yq, Zq, ωq, φq, κq)
And (Xr, Yr, Zr, ωr, φr, κr), on the continuous mosaic image, the lines of these external orientation elements are located on the lines corresponding to the direct-view images at the respective photographing points P, Q, and R. Values are assigned, and interpolated values are assigned to the other lines. In this way, the external orientation elements (X _i , Y _i , Z) are added to each line of the continuous mosaic image as shown in FIG.
_i , ω _i , φ _i , κ _i ) are assigned.

In this embodiment, since the calculation software for photogrammetry is diverted, the coordinate system of the video image is converted into the coordinate system of a single photogram of ordinary photogrammetry as shown in FIG. Immediately

Then,

As shown in Equation 1, the image coordinate system (u, v) is transformed into the photographic coordinate system (x, y).

[0040]

X = (u−u ₀ ) × x _C y = (v−v ₀ ) × y _C where u ₀ and v ₀ are the central image coordinate values of the video image, x
_C and y _C are the lengths per pixel on the CCD image plane in the x and y directions.

Then, the photographic coordinate system is converted into the ground coordinate system. Photographic coordinate system (x, y) and ground coordinate system (X, Y, Z)
There is a geometrical relation shown in FIG. 13 between and, and the conversion formula is represented by the following formula.

[0042]

[Equation 2]

Here, f is the screen distance, (Xo, Yo,
Zo) is the ground seat with the projection center O of the photograph

It is a mark.

Equation 2 is called a collinear conditional expression. 9 coefficients a ₁₁ ~
a ₃₃ is obtained by the equation 3 from the inclination (ω, φ, κ) of the photographing axis of the camera. [omega], [phi], and [kappa] represent rotation angles around the X-axis, Y-axis, and Z-axis, respectively, and the clockwise direction is positive in the positive direction of each axis.

[0045]

(Equation 3)

Therefore,

[0047]

[Equation 4]

[0048]

The inverse transformation formula of [Equation 2] is as follows. That is,

[0049]

(Equation 5)

[0050]

2 and

The unknown variables in Equation (5) are the ground coordinates (Xo, Yo, Zo) of the projection center of the photograph and the tilt (ω, φ,
κ). These six unknown variables are external orientation elements, and the photographic coordinate system can be converted to the ground coordinate system for each line by the previously determined external orientation element for each line.

A continuous mosaic image converted into the ground coordinate system in this way

By applying the equation (5), an image with vertical parallax removed is created as shown in FIG. 14 (S4). That is, using the external orientation elements previously determined for each line, the coefficients a _{11 to} a ₃₃ are obtained,
Continuous mosaic

The front-view image, nadir-view image, and rear-view image of the image

[Equation 5] is applied to create an image projected at an altitude of 0 m. At this time, the size of one pixel of the image to be created is calculated from the photographing altitude, the camera focal length, the CCD length of one pixel of the video image, and the like, and is set to an optimum value.

Even if the size of one pixel of the output image is set to an optimum value, there may be some missing pixels due to changes in flight speed and shooting direction during video shooting. This is interpolated or interpolated from surrounding pixels by filtering. For the interpolation method, the nearest neighbor interpolation method in which the value of the observation point closest to the point to be interpolated is diverted as it is, and the co-linearity is used as the value to obtain the average value of the four observation points around the point to be interpolated There are an interpolation method and a cubic convolution interpolation method that performs a cubic convolution process on the values of 16 observation points around the point to be interpolated.

The nearest neighbor interpolation method produces a position error of up to ½ pixel, but has the advantage of not destroying the original image data, and the algorithm is simple. The bilinear interpolation method has a drawback that the original data is destroyed, but has an advantage that the smoothing effect can be obtained by averaging. The cubic convolution interpolation method has a drawback that original data is destroyed, but has an advantage that image smoothing and sharpening can be realized at the same time. Depending on the purpose and seeing the final result,
It suffices to select which interpolation method is applied in whole or in part.

The parallax is calculated from the image from which the vertical parallax is removed (S5). An automatic stereo matching method is used to calculate the parallax. In the present embodiment, as shown in FIG. 15, after the vertical parallax removed images are rearranged so that the shooting direction of the vertical parallax removed images becomes the direction orthogonal to the horizontal scanning line, an image for stereo matching, That is, a matching image is created. As a result, not only the following processing can be speeded up, but also the memory capacity required for the processing can be saved.

A template image (for example, N × N pixels) whose height is to be calculated is set on the forward-view image of the matching image obtained by the above processing, and an image approximate to this template image is set as another image. Within the predetermined search range on the matching image, that is, the backward-viewing image or the down-looking image. In the present embodiment, since the vertical parallax is removed in advance, the search range is limited to the direction orthogonal to the horizontal scanning line. After matching, the difference between the positions, that is, the parallax is calculated.

The mosaic image formed from the data of the adjacent lines of the field of the video image is very close to each other, but the farther the extracted lines are, the more the obtained mosaic images have different gaze directions. Even when optimally matched, the matching error becomes large. Therefore, in the present embodiment, even when stereo matching is performed between the front-view image and the rear-view image, a mosaic image is formed from intermediate lines (of course, vertical parallax is removed), and corresponding points are sequentially determined. I tried to explore. As a result, the influence of occlusion can be eliminated and the corresponding points can be detected with high accuracy.

An example of a quadrangular truncated pyramid-shaped object as shown in FIG. 17 will be described. FIG. 18 shows the correspondence between the extracted line in the shooting screen and the mosaic image (vertical parallax removed) obtained by synthesizing each line when the object is video-shot. FIG. 18 (1) shows that when extracting the first line and the last line, (2) when extracting the inner line, (3) when extracting the inner line, and (4) This is the case when the adjacent lines in the approximate center are extracted. Thus, in the present embodiment, the original forward-looking image F1
In addition to the rear-view image R1 and the intermediate front-view image F2.
There are F3, F4 and rear view images R2, R3, R4. Instead of the intermediate front-view image F4 and rear-view image R4, a down-view image may be adopted.

Needless to say, the intermediate images F2, F
The number of F3, F4, R2, R3, and R4 may be set smaller or larger according to need. An appropriate number may be set according to the result of the trial matching operation.

When searching a point corresponding to the point A of the front-view image F1 from the rear-view image R1, first, a corresponding point of the point A is searched on the intermediate front-view image F2. Since the lines of sight of the images F1 and F2 are slightly deviated from each other, the corresponding points can be searched with high accuracy even by automatic stereo matching. The template including the corresponding points searched for in the image F2 sets an image, and the corresponding points are searched for in the image F3. Then, the image F4, the image R4, the image R3, the image R2, and the image R are sequentially displayed.
Performs a stereo matching operation with 1. In this way,
The corresponding point of the point A on the image F1 can be detected with high accuracy on the image R1. When the image F1 and the image R1 are directly stereo-matched, a large error cannot be avoided depending on the object due to the influence of occlusion. However, in the present embodiment, the sequential search is performed in this manner to completely eliminate the influence of occlusion. Can be removed.

Here, although it has been stated that an intermediate image is created in advance, while performing the matching operation for calculating the parallax, the intermediate image is sequentially intermediated by the binary search method until the matching error becomes equal to or less than the allowable value. Alternatively, a corresponding image may be formed to search for the corresponding point. For example, if there is a matching error in the backward-looking image that is equal to or greater than the allowable value, a mosaic image extracted from an intermediate line is formed (of course, vertical parallax is also removed), and the mosaic image is displayed on the mosaic image. Perform a matching operation. If a corresponding point is found with a certain accuracy or more, the corresponding point is searched for by the matching process on the backward-view image using the found corresponding point as a guide. If the intermediate mosaic image has a matching error equal to or larger than the allowable value, a mosaic image extracted from an intermediate line is formed and a matching operation is performed on the mosaic image. By such an iterative process, the corresponding points are finally detected between the front-view image and the rear-view image. By doing so, the influence of occlusion can be completely eliminated and the corresponding points can be detected with high accuracy.

In any case, the intermediate image only needs to form the portion necessary for the matching calculation. Then, the matching calculation time can be shortened and the memory capacity for storing intermediate image data can be reduced.

Next, a DEM (Digital Elev)
The height can be calculated from the parallax according to the following formula in order to create the ation model). That is,

[0064]

Where h is the height (m) to be obtained, d is the parallax (m), B is the base length (m), and H is the photographing altitude (m). FIG.
Indicates the relationship between parallax and height.

When the length of the image plane of the CCD image pickup device of the camera 10 is c1 (mm), the length of the CCD image plane per pixel is dc (mm), and the focal length is f (mm). ,
When obtaining the height from the parallax between the front-view image and the rear-view image,

[0066]

## EQU00007 ## h = d.times.f / (c1-dc), and when the height is obtained from the parallax difference between the front-view image or the rear-view image and the direct-view image,

[0067]

Equation 8 h = d × 2 × f / (c1-dc)

The height accuracy is determined by the B / H ratio, but can be freely set by changing the focal length of the camera 10 at the time of photographing.

When the height data is obtained, a median filter or the like is used to detect data having irregularities more than the surrounding data, and the data is corrected with the surrounding data.

Through the above processing, height data can be calculated for a desired area in the object to be photographed, and three-dimensional data can be obtained.

In the present embodiment, the installation work of the anti-aircraft sign and the surveying work for measuring the position thereof are not necessary, so that the survey can be conducted quickly in any area. In addition, 3D topographic maps can be created for dangerous areas where people cannot enter (landslides, debris flows, volcanic eruption areas, etc.). Also, since the orientation calculation is automated, the analysis process can be advanced quickly. The height accuracy can be freely set by changing the focal length of the camera 10.

The present invention is not limited to the creation of a three-dimensional topographical map, and can be used for a disaster prevention information collection system, railway and road route planning, and sea / river erosion surveys.

Orientation calculation (S2) and parallax calculation (S5)
The matching method used in 1. will be briefly described. As described above, conventionally used matching methods include the residual sequential test method and the cross-correlation coefficient method.

First, the residual sequential test method will be described. FIG.
As shown in, the template image of N × N pixels is searched for in the search range (M−N +
1)

² Move up,

It is considered that the superposition is achieved at the position where the residual of (Equation 9) is minimized.

[0076]

[Equation 9]

However, (a, b) indicates the upper left position of the template image in the input image, and I _{(a, b)} (m,
n) is a partial image of the input image, T (m, n) is a template image, and R (a, b) is a residual between the partial image of the input image and the template image.

If the overlay is deviated, the residual error increases rapidly when the pixels are sequentially added. Therefore, if the residual exceeds a certain threshold during the addition, it is determined that the superposition is not good, the addition is aborted, and the process proceeds to the next (a, b). This is the residual sequential test method (s
equality1 similarity dete
motion algorithm, and SSDA
Abbreviated as law. ). The SSDA method was proposed by Barnea et al., But the problem was how to give a threshold value. In this regard, Onoe et al. Have proposed a threshold automatic determination method.

The automatic threshold determination method proposed by Onoue et al.
For the time being, the minimum value of the past residual is adopted as the threshold. At the beginning, the threshold value is added to the end without any threshold value, and the resulting residual is used as the first threshold value. After that, until the end, each time the sum is added without exceeding the threshold value, the residual is set as a new threshold value. This method guarantees that the true minimum is always reached.

The SSDA method consists of only addition, and in many cases is terminated halfway, so the calculation time can be greatly shortened. Onoe et al. Applied it to cloud movement tracking and obtained extremely good results with the same accuracy and a reduction in processing time of one digit or more compared to the method using the cross-correlation coefficient.

The cross-correlation coefficient method will be described. In the cross-correlation coefficient method,

The upper left position (a, b) of the template image in the input image that maximizes [Formula 10] is obtained.

[0082]

[Equation 10]

With this method, there is no termination of calculation as in the SSDA method, and therefore the calculation time cannot be shortened so much.
However, in the field of photogrammetry, the cross-correlation coefficient method is a convenient method for searching the corresponding points in the left and right images when measuring the elevation of grid points from a stereo aerial photograph digital image and drawing contour lines. So it is often used.

According to the present embodiment, a three-dimensional topographic map or the like can be created from images recorded by a high-resolution video camera, instead of the aerial photographs that have been conventionally well used, so that it can be used in a very wide range of applications. . For example, it can be used for various management of roads, rivers, railways, etc., and if only the changed portion is extracted by image processing, it can be used for surveying the development status of cities and the like.
It can also be used to create a three-dimensional bird's-eye view and simulate the completion status of a new route plan.

Although an example of photographing the ground from a helicopter has been described, it goes without saying that the present invention can also be applied to the case of photographing from other airplanes, surveillance satellites, and automobiles traveling on the ground in addition to the helicopter.

Further, in the above embodiment, the front-view image is formed from the frontmost line of the field image and the rear-view image is formed from the last line. It goes without saying that the line may be two or more lines before.

[0087]

As can be easily understood from the above description, according to the present invention, basic data can be collected by video shooting while moving, and height data can be obtained by image processing thereof. Since all or most of the post-imaging analysis processing can be automated, desired three-dimensional data can be quickly obtained. Also, three-dimensional data can be obtained at a lower cost than before.

[Brief description of drawings]

FIG. 1 is a schematic configuration block diagram of an aerial measurement system according to an embodiment of the present invention.

FIG. 2 is a schematic configuration block diagram of the ground measurement system of the present embodiment.

FIG. 3 is a schematic block diagram of the ground analysis system of the present embodiment.

FIG. 4 is a flow chart from measurement to three-dimensional data extraction in this embodiment.

FIG. 5 is an explanatory diagram of a 60% overlapping scene for orientation calculation.

FIG. 6 is an explanatory diagram of mutual orientation and pass points.

FIG. 7 is an explanatory diagram of mutual orientation and connection orientation.

FIG. 8 is a diagram showing a relationship between an image pickup surface of a camera and a ground shooting range.

FIG. 9 is a conceptual diagram of creating a continuous mosaic image.

FIG. 10 is a relationship diagram between a continuous mosaic image and an external orientation element.

FIG. 11 is a diagram illustrating interpolation of external orientation elements.

FIG. 12 is a diagram illustrating an image coordinate system and a photographic coordinate system of an imaging surface.

FIG. 13 is a relationship diagram of a photographic coordinate system and a ground coordinate system.

FIG. 14 is an example of images before and after vertical parallax removal.

FIG. 15 is an explanatory diagram of matching images for calculating parallax.

[Fig. 16] Fig. 16 is an explanatory diagram of stereo matching for calculating parallax.

FIG. 17 is a quadrangular truncated pyramid-shaped object model as a measurement target.

[Fig. 18] Fig. 18 is a diagram showing the correspondence between the extracted lines in the shooting screen and the mosaic image (vertical parallax removed) obtained by combining the lines when the object shown in Fig. 17 is video-shot.

FIG. 19 is an explanatory diagram of a relationship between parallax and height.

FIG. 20 is a basic diagram of matching between an input image and a template image.

[Explanation of symbols]

 10: High-definition camera 12: High-precision anti-vibration stabilizer (anti-vibration device) 14: High-definition video tape recorder 16: High-definition monitor 18: Personal computer 20: 3-axis control device 22: Camera control device 24: VTR control device 26: Earth altitude sensor 28: Magnetic direction sensor 30: GPS antenna 32: GPS receiver 34: Navigation system 36: Floppy 38: Monitor 40: Communication device 42: Floppy 44: Monitor 46: Keyboard 50: GPS reception Device 52: GPS antenna 54: Computer 56: Floppy 58: Monitor 60: Keyboard 62: Communication device 70: High quality VTR 74: Frame buffer 76: Engineering workstation 78: Monitor 80: Personal computer F1: Front view Image F2, F3, F4: Intermediate forward-view image R1: Rear-view image R2, R3, R4: Intermediate backward-view image

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Office reference number FI technical display location G06T 17/00 7/00

Claims (25)

[Claims] 1. A basic information collecting step of shooting a moving object of three-dimensional data while moving, recording a video signal of the moving object, and recording shooting information including a position and a tilt of a camera to shoot, and collecting the basic information. A three-dimensional data extracting method comprising a three-dimensional data generating step of generating three-dimensional data of the extraction target from the image and the photographing information collected in the step, wherein the three-dimensional data generating step is a continuous screen of photographed images. A continuous mosaic image generating step of extracting image data of a predetermined line on a predetermined screen of the image and generating at least two continuous mosaic images of a forward-view image, a down-view image and a backward-view image; In the vertical parallax removal step that removes vertical parallax from the continuous mosaic image generated in From the parallax calculation step that calculates the parallax for the predetermined position in the continuous mosaic image from which the vertical parallax has been removed, and the height calculation step that calculates the height of the predetermined position from the parallax calculated in the parallax calculation step. And a three-dimensional data extraction method. 2. The three-dimensional data extracting method according to claim 1, further comprising a step of calculating orientations for determining the external orientation elements by mutual orientations and connection orientations from consecutive screens of photographed images. 3. The orientation calculation step extracts two screens that overlap each other by a predetermined ratio from consecutive screens of captured images, and connects the mutual orientation step for mutual orientation and the model mutually oriented by the mutual orientation step. The three-dimensional data extraction method according to claim 2, further comprising a connection orientation step. 4. The predetermined ratio is 60%.
The three-dimensional data extraction method described in 1. 5. The vertical parallax removing step interpolates an external orientation element into each line of the continuous mosaic image generated in the continuous mosaic image generating step according to the external orientation element determined in the orientation calculating step. 3. An orientation element interpolation step, and a projection step of converting each line of the continuous mosaic image generated in the continuous mosaic image generation step into a projection image at a predetermined altitude according to the external orientation element of the line. The three-dimensional data extraction method according to any one of 4 above. 6. The parallax calculation step comprises an intermediate image forming step of forming one or more intermediate images between the continuous mosaic images, and the continuous mosaic image through the one or more intermediate images. 6. The method according to claim 1, further comprising: a corresponding point detecting step of detecting a corresponding point in step 1; and a calculation step of calculating a parallax of the corresponding point according to a detection result of the corresponding point detecting step. Dimensional data extraction method. 7. The intermediate image forming step according to claim 6, wherein the intermediate line image data is formed by extracting image data of an intermediate line on a predetermined screen among successive screens of the captured video. Dimensional data extraction method. 8. The three-dimensional data extraction method according to claim 1, further comprising GPS receiving means as means for detecting the position of the camera. 9. The three-dimensional data extraction method according to claim 8, further comprising a correction unit that differentially corrects the output of the GPS receiving unit. 10. The three-dimensional data extraction method according to claim 1, wherein the camera is image-stabilized by image stabilization means. 11. A video camera for photographing a target for extracting three-dimensional data, a positioning means for measuring the position of the video camera, and a recording for recording a video image taken by the video camera and a measurement value of the positioning means. Means and carrying means for carrying the video camera and the positioning system,
Playback means for playing back video information and position information recorded in the recording means, and image data of a predetermined line in a predetermined screen among continuous screens of captured video are extracted to obtain a front-view image, a down-view image and a rear-view image. With the continuous mosaic image generation unit that generates at least two continuous mosaic images of the visual image, the vertical parallax removal unit that removes the vertical parallax of the continuous mosaic image generated by the continuous mosaic image generation unit, and the vertical parallax removal unit. A parallax calculation unit that calculates a parallax in a continuous mosaic image from which vertical parallax has been removed, and a height calculation unit that calculates the height of the predetermined position from the parallax calculated by the parallax calculation unit. A characteristic three-dimensional data extraction device. 12. Further, from a continuous screen of captured images,
The three-dimensional data extraction device according to claim 11, further comprising orientation calculating means for determining an external orientation element by mutual orientation and connection orientation. 13. The orientation calculation means extracts two screens that overlap each other at a predetermined ratio from consecutive screens of photographed images, and connects the mutual orientation step of mutual orientation and the model that is mutually oriented by the mutual orientation step. The three-dimensional data extraction device according to claim 12, further comprising a connection orientation step. 14. The three-dimensional data extraction device according to claim 13, wherein the predetermined ratio is 60%. 15. The vertical parallax removing means interpolates an external orientation element into each line of the continuous mosaic image generated by the continuous mosaic image generating means according to the external orientation element determined by the orientation calculating means. Orientation element interpolation means,
The projection means for converting each line of the continuous mosaic image generated by the continuous mosaic image generating means into a projection image at a predetermined altitude according to an external orientation element of the line, according to any one of claims 12 to 14. The three-dimensional data extraction device described. 16. The parallax calculation means is an intermediate image forming means for forming at least one intermediate image between the continuous mosaic images, and the continuous mosaic image is through the at least one intermediate image. 12. A corresponding point detecting means for detecting a corresponding point in claim 1, and a calculating means for calculating a parallax difference of the corresponding point according to a detection result of the corresponding point detecting means.
The three-dimensional data extraction device according to any one of 1 to 15. 17. The method according to claim 16, wherein the intermediate image forming means extracts the image data of an intermediate line on a predetermined screen among the continuous screens of the captured video to form the intermediate image. Dimensional data extraction device. 18. The three-dimensional data extraction device according to claim 11, wherein the positioning unit is a GPS receiving unit. 19. The three-dimensional data extraction device according to claim 18, further comprising a correction means for differentially correcting the output of the GPS receiving means. 20. The three-dimensional data extraction device according to claim 11, wherein the camera is mounted on the transport means via a vibration isolation means. 21. The transport means is an aircraft.
The three-dimensional data extraction device according to any one of 1 to 20. 22. The three-dimensional structure according to claim 11, further comprising azimuth detecting means for detecting the azimuth of the camera, and the output of the azimuth detecting means is also recorded in the recording means. Data extraction device. 23. The three-dimensional data extracting apparatus according to claim 11, wherein the camera is installed so that the moving direction of the carrying means is a direction perpendicular to the scanning line direction of the camera. . 24. An extracting means for extracting line image data of two or more different predetermined line positions on a screen of a video imaged by video, and a synthesizing means for synthesizing line image data of the same line position. And stereo image forming apparatus. 25. The stereo image forming apparatus according to claim 24, further comprising a vertical parallax removing unit that removes the vertical parallax of the image combined by the combining unit by an external orientation element for each original line image data. .