JPH0816930B2 - Method to obtain three-dimensional numerical data from stereo image data - Google Patents

Description

TECHNICAL FIELD The present invention relates to three-dimensional measurement based on image data obtained from an artificial satellite, an aircraft, or the like, and creates a terrain-corrected image using this data. The present invention also relates to a three-dimensional image processing technique for creating an image (bird's-eye view image) whose viewpoint is freely changed as needed.
Suitable for use in three-dimensional image processing technology,
The present invention relates to a method for obtaining three-dimensional numerical image data from image data.

2. Description of the Related Art A method for obtaining three-dimensional numerical image data from this type of image data is a digital terrain model (Digital Terrain Mode).
l), which is known as a method for obtaining a so-called DTM, and currently, there are the following methods for creating a DTM.

A. Method using topographic maps At present, there are the following two methods for acquiring DTM using existing maps (topographic maps such as 1/50000 and 1/25000).

(1) Manual mode This method is to draw the mesh on the topographic map and read the elevation of the mesh points from the contour.

(2) Scanner mode This method is a method of creating a DTM by reading a topographic map with a drum scanner, performing raster vector conversion, thinning processing, elevation attribute addition processing, and meshing processing.

B. Method using an analysis plotter (1) Mesh mode This method is a method of creating a DTM while stepping a female mark with a specified step size.

(2) Contour mode This is a method of digitizing random elevation values while tracing the equal elevation values (contours) as in the case of normal plotting work.

C. Method of performing fully automatic measurement using a general-purpose computer This method uses AD conversion data of aerial photographs as input data and automatically performs left-right image matching by the area correlation method using a general-purpose computer to obtain DTM. Is the way.

Problems to be Solved by the Invention Among the various DTM creating methods described above, the method using a topographic map is inferior to the method of directly obtaining from a photograph because the base data is the topographic map. Moreover, since it is not done automatically, it is a difficult work and takes time. In addition, the DTM acquired by the mesh mode method using the analysis plotter
When the contour is created from the contour, the phenomenon that the contour undulates occurs depending on the measurement order and the skill level of the operator, and the final product is not obtained unless the smoothing process is performed several times. Furthermore, in the contour mode by the analysis plotter, the contour accuracy is better than that in the mesh mode, but the amount of data is enormous (data is acquired in the equidistant, isochronous mode) and the mesh is generated from random data. The need arises to convert to data (DTM).

In addition, the method of performing full-automatic measurement using a general-purpose computer, in that the DTM is automatically acquired, the work load of the operator is significantly reduced compared to the conventional method described above. Since the orientation calculation needs to be performed separately, and because calculation is performed using a large amount of image data,
A general-purpose computer requires a considerable amount of calculation time, and the DTM data obtained by batch processing has some errors due to mismatching of corresponding points, making it very difficult to correct them. .

An object of the present invention is to provide a method for obtaining three-dimensional numerical image data from image data, which can solve the above-mentioned conventional problems.

Means for Solving the Problem A method for obtaining three-dimensional data from stereo image data according to the present invention uses a computer with AD conversion data corresponding to the image data as input data to obtain left and right images by batch processing in units of one image data. Matching is automatically performed to find the corresponding points on the left and right images, and the correlation coefficient indicating the quality of the matching, and the corresponding points at which the matching is not good are output on the left and right graphic displays by the residual sum of squares. For matching points where matching abnormalities have occurred, the operator specifies the corresponding points that appear to be correct with the cursor, and causes the computer to perform image matching on the left and right again. Characterized by obtaining three-dimensional data of each desired point of image data To.

EXAMPLES Next, the present invention will be described in more detail with reference to the accompanying drawings with reference to the examples of the present invention.

FIG. 1 shows a flowchart of a semi-automatic three-dimensional measurement system using image data as an embodiment of the present invention. The case of obtaining three-dimensional numerical image data from image data of an aerial photograph according to the present invention will be described. First, as shown in step 1 of the flowchart of FIG. 1, the aerial photograph to be analyzed can be directly handled by a computer. Therefore, data is converted from analog to digital. In addition, if the data to be analyzed is digital data such as artificial satellite data, format conversion is performed to a format that can be handled by this system. Next, as shown in step 2, the coordinates of a photograph index, a pass point, a reference point, and the like, which serve as a reference for performing orientation calculation by a computer, are measured. In the coordinate measurement method, the operator moves the cursor to the measurement point while observing the measurement target image output on the CRT to measure the coordinates. Also, in the case of pass point measurement, CR
Output on T and measure using two cursors. Then, as shown in step 3, orientation calculation is performed.

Here, the orientation calculation includes the internal orientation for obtaining the focal length of the camera and the distortion of the lens of the camera, the position (X, Y, Z) of the shooting point (camera), and the inclination (κ, ω, φ) of the camera (sensor). )
It includes an external orientation that determines the positional relationship between the left and right photographs, and a ground orientation (absolute orientation) that determines the positional relationship between the model formed by the left and right photographs and the ground. The method is carried out using a usual analytical photogrammetry method. When the analysis target is not an aerial photograph but artificial satellite data such as spot data, the projection method of the image is not a central projection but a projection method like a slit camera, so the orientation element is calculated as a function of time. .

The image data usually has a huge amount of data, and it is impossible to perform matching all at once due to restrictions on memory and time. In addition, it is desirable not to carry out batch processing for the reason of preventing straying when searching for corresponding points. Therefore, as shown in step 4, the analysis area is divided into a plurality of units (this is called a patch) to be processed at one time, and the matching processing described later can be performed independently for each patch.

Next, as shown in step 5, the image is projected onto the XY plane of the model coordinate system to correct the deviation prior to obtaining the corresponding points of the left and right images. In this way, since the corresponding points of the left and right images always have the same Y coordinate, it is only necessary to search the corresponding points in the primary direction of the X axis, and the matching process becomes easy.

In the matching using the area correlation method, which will be described later, the high frequency part of the image is converted into noise due to the projection distortion, and the correlation is lowered, so that the accurate matching cannot be performed. As described above, the image is passed through a Laplacian-Gauss filter (L / G filter) which is a narrow band filter in advance, and the effect of projection distortion is suppressed to improve the convergence of matching.
Specifically, three types of L / G filters having different frequency characteristics are applied to the image after the deviation correction, and three types of images changing from a low frequency to a high frequency are created. Then, when performing matching, first, rough matching is performed using low-frequency images, and matching is performed using images with gradually higher frequencies, and the positions of corresponding points are converged while considering the results.

Next, as shown in step 7, the filtered image at each stage of shifting from low frequency to high frequency is sampled by 4, 2, 1 pixel by pixel to create a compressed patch image. Subsequent matching is performed using these reduced patch images. Then, as shown in step 8, grid points are arranged at 8-pixel intervals on the left image of the reduction patch image at each stage. In the computer matching described below,
The corresponding points in the right image corresponding to the grid points in the left image are searched. The arrangement of grid points on the original image is shown in FIG.

Next, as shown in step 9, in the first-stage matching, it is assumed that the flat terrain has an average altitude H, and grid points are arranged on the image to perform matching. In the next stage, the left and right images are resampled so as to eliminate the projection distortion in the X axis direction from the X parallax obtained at each grid point.

Here, the most important problem when obtaining three-dimensional information from a stereo image using a computer is how to accurately find corresponding points in the left and right images. The generally known methods for obtaining the corresponding points of the left and right images (stereo matching method) can be roughly classified into the following three types.

A. Area Correlation of Density Distribution Compare the similarity of density distribution in finite parts of left and right images.
Normally, the correlation between the left and right images is calculated to find the maximum point.

B. Feature matching Extract features from the left and right images and symbolize them.
Map it.

C. Divided images Matching matching images Divide an image into similar parts according to some characteristics, and associate each divided image. As the characteristic indicating the similarity, a uniform texture portion, an edge contour, and a color image having the same color portion are used.

Of the above-mentioned three types of methods, the area correlation method in section A. has been used for matching various patterns for the longest time, not limited to stereo. Specifically, a square correlation window is taken on the left image, and the inside of the search window defined on the right image is moved using this as a template to search for the corresponding point with the best degree of coincidence. Therefore, the images to be matched need to be rich in texture. In photogrammetry,
This method has been used exclusively because the image to be handled is often an aerial photographic image with a relatively large amount of texture, and the area correlation is a method that is easy to handle from the viewpoint of practical use.

With respect to this area correlation, a relatively large amount of research has been conducted to improve the sensitivity of peaks and to make it possible to perform stable matching even in an image with a low signal-noise ratio by devising a scale of the correlation.

When the image function and the noise correlation function are known, the adaptive filter approach can be used.

When n (x) is white noise and f (x) is a signal of the left image, it is assumed that there is a relationship of g (x) = f (x) + n (x). At this time, an appropriate linear filter h
Output when (x) is passed through g (x) In order to maximize the signal-to-noise ratio of x at x = a, h (x) = f (ax) may be set. This is the normal cross correlation Means that gives the maximum signal to noise ratio. This is the reason why mutual relations are often used as matching criteria, and the area correlation method is also used in the embodiment of the present invention.

In step 10, the computer searches for left and right corresponding points by means of the cross-correlation coefficient. As shown in FIG. 3, this is because the grid points on the patch whose resampling has been finished in step 9 are completed. This is done by setting a correlation window around. This correlation window is moved within the search window provided on the right patch (X direction) to search for the corresponding point that gives the maximum correlation. A cross-correlation coefficient is used as a matching criterion.

The size of the correlation window is 16 * 16 pixels, and the density values of the pixels included in the correlation window and the search window on the left and right patches are a (i, j) and b (i, j) (i, j = 1, respectively). , 2, 〜, 16) The search width was ± 16 pixels in the first stage and ± 3 pixels in the second and third stages.

If a mismatch occurs in this stereo matching stage, it will adversely affect the matching in the next stage, and the matching will stray. So step 11
As shown in (1), the X-parallax is filtered by a median filter having a filter characteristic that removes sudden mismatching and keeps the edge of the sudden change in specific height.

In this way, all stereo matching is automatically performed using a computer, but the results may not be all correct. Therefore, in the present system, as shown in step 12, these correction processes are output on a graphic display (corresponding points of a stereo image are output on the image), and the operator corrects them while monitoring them.

In this system, the operation method of such stereo matching is performed in the following three modes.

A. Mode 1 Batch processing mode (flat ground adaptation) A stereo display is calculated by batch processing in model units, and a correlation display showing the quality of the matching, and a graphic display of only the areas where the matching is bad due to the residual sum of squares The corresponding points are output to the above, the operator checks this, and for the corresponding points where the matching abnormality occurs, the operator specifies the approximate position with the cursor, calculates the stereo matching again and calculates the corresponding points. fix.

B. Mode 2 ... Single point processing mode (sudden ground adaptation) Stereo matching calculation is performed for each single point, the corresponding points are output on the graphic display, and the operator checks this and a matching error occurs. For the corresponding points, the operator specifies the approximate position with a cursor, and again performs stereo matching calculation and recalculates the corresponding points.

C. Mode 3 Area processing mode (adapted to a combined area of flat land and steep land) In advance, image processing (density slicing, maximum likelihood method, etc.) of flat land and steep land (shaded area) or by operator Divide, and first perform batch processing mode based on this division result,
Next, the shadow portion is processed in the single point processing mode.

More specifically, in this point, the patch image is subjected to deviation correction in step 5, filtering is performed in step 6, and left and right corresponding points are searched in step 10.
At this time, as shown in FIG. 4, left and right deviation corrected images are displayed on the display device. Then, the position of the searched corresponding point is blinked (blink) on the display device. If the process is a check process for only abnormal points as in batch processing mode, "OK" and "NO" are displayed only when the corresponding point search result is determined to be abnormal.
The operator specifies whether the position of the corresponding point is correct or not. If "OK" is specified, the process proceeds to the next grid point. If the processing is all-point check processing, as in the single-point processing mode, when the corresponding points are searched, "OK" and "NO" are displayed. The operator specifies whether the position of the corresponding point is correct or not. If you specify "OK", you will proceed to the next grid point. When the corresponding points of all grid points are found,
The process moves to the corresponding point check process in step 12. Then, when all patches are completed, the process proceeds to the DTM output process of step 14.

In the corresponding point check processing in step 12, if the matching is partially unsuccessful, processing is performed according to the following principle. First, if there are almost no left and right image patterns and it is determined that stereoscopic vision is impossible and measurement cannot be performed, measurement is not performed. Although there are left and right image patterns, if there is a place where the measurement point is judged to be bad (halation, shadow, etc. are seen in only one side of the photo) and there is a place nearby where it is easy to handle, Specify the measurement point on the left side with the mouse and the corresponding point zone on the right side. That is, from step 13 to step
Return to 10 and perform the matching calculation processing of the designated point again on the computer side, display the result on the monitor and check again. When the operator also determines that the corresponding points displayed at the first time are too large because the parallax is too large, the operator determines that there is no corresponding point on the left and right images, and the operator determines in step 6 Return to the low frequency L / G filtering from the first step. Also,
If the matching process is not successful locally (regionally), it is processed according to the following principle. Since it is judged that mismatching occurs continuously because the matching search area is defective, point the start point of mismatching and the mismatching area with the mouse, and specify the corresponding point zone of the start point on the right image. Then, the matching process is performed again from the start point.

More specifically, when "NO" is designated by the correctness of the corresponding point position, the correct corresponding point position on the right image is designated by the mouse as shown in the lower part of FIG. After specifying the position, specify "OK" with the mouse. When the positions of all corresponding points in the patch have been obtained in this way, the grid points and their corresponding points are displayed on the display device, as shown in FIG. If it is OK as it is, specify "OK" with the mouse. If "OK" is specified, the process proceeds to the next patch. If you want to make corrections, click "NO" with the mouse.
Is specified. If "NO" is specified, specify how to make corrections with the mouse. If it is “redo all points”, search for corresponding points in the patch is restarted from the beginning. If "Redo one by one", add one point at a time.

In the case of "redoing one point at a time" processing, as shown in FIG. 6, the position of the grid point on the left image is first designated with the mouse for the point to be corrected. Next, the position of the corresponding point on the right image is designated with the mouse. To make corrections to other points, specify the "next point" with the mouse. When the correction is over, specify "End" with the mouse. The process returns to the corresponding point check process again.

In the case of the single point mode, as shown in FIG. 7, the deviation correction processing is performed on the patch image, and the left and right deviation correction images are output on the display device. First, specify the position on the left image with the mouse for the point you want to obtain the altitude,
Next, specify the position on the right image with the mouse. After specifying the position, specify the next process with the mouse. That is,
If "Next point" is specified, the next point is input. If "End" is specified, the next patch is entered. Thus, when all the patches have been completed, the process proceeds to step 14 DT.
Proceed to the output of M.

In the creation of the DTM in step 14, after the matching is completed, the ground elevation is calculated by the computer using the X parallax obtained for each grid point. In FIG. 8, the ground coordinates of the point P are represented by the photographic coordinates after the deviation correction as shown in the following formula.

However, B: shooting baseline length S: scale factor C: screen distance d _ij (i = 1, 2, 3: j = 1, 2, 3): rotation matrix of the entire model From the above equation, the ground coordinates of each grid point ( X _GP , Y _GP , Z _GP ), and create DTM. This set of ground points does not form a square grid in the ground coordinate system, although it is somewhat regular. Therefore, for the convenience of later use, these random
Interpolate the mesh into a square mesh. The DTM measured in this way is recorded on a disk and made into a file.

The squared mesh (DT
M) can be used to create a contour map (contour map) or a three-dimensional bird's-eye view, and a square map and image data can be used to create a photographic map or a three-dimensional bird's-eye view image.

EFFECTS OF THE INVENTION According to the method of the present invention, three-dimensional information can be automatically obtained from image data by stereo matching by a computer, so that three-dimensional graphic processing such as a photographic map and a bird's-eye view image can be efficiently performed. The problem that the computer has gone astray for a long time when the mismatch is corrected in the past can be solved because the operator can correct the mismatch semi-automatically.

[Brief description of drawings]

FIG. 1 is a diagram showing a flow chart of a semi-automatic three-dimensional measurement system using image data as one embodiment of the present invention, and FIG. 2 is a diagram showing an arrangement example of grid points on an original image. The figures are diagrams for explaining the relationship between a correlation window and a search window used in stereo matching, and FIGS. 4 to 7 are for explaining an example of a procedure for searching and checking corresponding points in stereo matching. Illustration of
FIG. 8 is a diagram for explaining an equation for obtaining DTM. 1 ... A / D conversion, 2 ... Coordinate measurement, 3 ... Orientation calculation, 4
…… Patch division, 5 …… Displacement correction, 6 …… L / G filtering, 7 …… Sampling, 8 …… Layout point arrangement, 9
...... Resampling of left and right images, 10 ・ ・ ・ Search for left and right corresponding points, 11 …… Median filtering of X parallax, 12 ……
Check corresponding points, create 14 …… DTM.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Toshio Ikeda 948-9, Atsugi-shi, Kanagawa Aina Green Heights No. 203 (56) References JP-A-61-209315 (JP, A)

Claims (1)

[Claims] 1. A method for obtaining three-dimensional data from stereo image data, in which AD conversion data corresponding to image data is used as input data by a computer to automatically perform left-right image matching by batch processing in units of one image data. Then, the corresponding points of the left and right images are obtained, and the correlation coefficient indicating the quality of the matching and the corresponding points of the poor matching by the residual sum of squares are output on the left and right graphic displays. As for the corresponding points, the operator specifies the corresponding points that are considered to be correct by using the cursor and causes the computer to perform the left and right image matching again, thereby correcting the matching abnormality by the computer and semi-automatically obtaining the desired image data. A method characterized by obtaining three-dimensional data of each point.