KR20110089299A - Stereo matching process system, stereo matching process method, and recording medium - Google Patents

Abstract

In the stereo matching processing system, the auxiliary line A2 on the left image and the auxiliary line A1 on the right image are inputted in correspondence with each other by an operator, and the auxiliary line B2 on the left image and the beam on the right image are input. When the shipbuilding B1 is input in correspondence, the coordinates a2 of the intersection of the scan line and the auxiliary line A2 on the left image and the coordinates of the intersection of the scan line and the auxiliary line A1 on the right image ( correspond to a1) on the search plane. Further, the coordinates b2 of the intersections of the scan lines and the auxiliary lines B2 on the left image and the coordinates b1 of the intersections of the scan lines and the auxiliary lines B1 on the right image are matched on the search plane. Thereby, the stereo matching processing system 1 can improve the misalignment on a search plane, and can exactly match the same position between a left image and a right image.

Description

Stereo matching processing system, stereo matching processing method, and recording medium {STEREO MATCHING PROCESS SYSTEM, STEREO MATCHING PROCESS METHOD, AND RECORDING MEDIUM}

The present invention relates to a stereo matching processing system, a stereo matching processing method, and a computer-readable recording medium on which a program is recorded. The present invention relates to a stereo matching processing system, a stereo matching processing method, capable of accurately matching the same position among a plurality of images, And a recording medium.

In the method for automatically generating three-dimensional data, a method for generating three-dimensional data (DSM (Digital Surface Model) data) representing the terrain by stereo matching is widely performed on the basis of images obtained from satellites, aircraft, and the like. A method of correcting the point where no correspondence is taken through an operator has also been proposed.

Here, the stereo matching process refers to two images taken from different viewpoints, a so-called stereo image, to obtain corresponding points in each image photographing the same point, and to the subject by the principle of triangulation using the parallax. It is to find the depth or shape.

Various methods have already been proposed for this stereo matching process. For example, a commonly used area correlation method is to set a correlation window in a left image as a template, move the search window in the right image to calculate a cross correlation coefficient with the template, and consider this as a high degree of agreement. The method of obtaining a corresponding point by searching for this is disclosed (for example, refer patent document 1).

In the above method, in order to reduce the throughput, by limiting the moving range of the search window to the epipolar line direction in the image, the position shift amount in the x direction of the point in the corresponding right image, i.e., parallax, with respect to each point in the left image Can be obtained. Here, the epipolar line is a straight line which can be drawn as an existence range of a point corresponding to the point in the other image with respect to a point in one image in the stereo image (for example, refer to Non-Patent Document 1).

Usually, the epipolar line direction is different from the scanning line direction of the image, but by performing coordinate transformation, the epipolar line direction can be rearranged by matching the epipolar line direction to the scanning line direction of the image. The method of this coordinate conversion is described in said nonpatent literature 1.

In the rearranged stereo images, since the movement range of the search window of the corresponding point can be limited on the scanning line, the parallax is obtained as the difference of the x coordinate between the corresponding points in the left and right images.

Patent Document 1: Japanese Patent Application Laid-open No. Hei 8-16930

Non-Patent Document 1: "Image Analysis Handbook" (Supervised by Mikagi Takagi, Haruhi Shimoda, Tokyo University Press, January 1991, pages 597-599)

However, in the three-dimensional data generation method by the stereo matching, an area without a texture or an area where a correspondence by a correlation coefficient cannot be obtained includes a wrong height significantly different from the surroundings in the image of the three-dimensional data. A lot of points are included. Particularly, since concealment occurs around a building, etc., there are many points where no correspondence is taken, and a remarkably high value may be exhibited or the building may be largely missing.

Therefore, in the three-dimensional data generation method by the related stereo matching, the error by the mismatch of a corresponding point generate | occur | produces, and high precision three-dimensional information is not obtained, so it is applicable to a complex image with many buildings, such as an urban part. There was the problem that it was difficult.

This invention is made | formed in order to solve the said subject, and an object of this invention is to provide the stereo matching processing system, the stereo matching processing method, and the recording medium which can exactly match the area | region which shows the same position among a some image. .

In order to achieve the above object, in the stereo matching processing system according to the first aspect of the present invention, in a plurality of images obtained by photographing the same object from different directions, regions having maximum correlation coefficients on the same scan line are located at the same position. And a line segment discriminating unit for discriminating whether or not a corresponding line segment is drawn by representing the same position in each of the plurality of images, wherein the matching unit is formed by the line segment discriminating unit. When it is determined that the line segment is drawn, the intersections of the scan line and the line segment are matched with each other by indicating the same position, not the regions where the correlation coefficient is maximum on the same scan line.

In addition, the stereo matching processing method according to the second aspect of the present invention corresponds to a plurality of images obtained by photographing the same object from different directions, in which regions having maximum correlation coefficients on the same scan line represent the same position. And a line segment discrimination step for discriminating whether or not a corresponding line segment is drawn by representing the same position in each of the plurality of images. The correspondence step includes the line segment discriminating step. When it is determined that the image is drawn, the intersections between the scanning line and the line segment are matched with each other by indicating the same positions, not the regions having the maximum correlation coefficient on the same scanning line.

In addition, the computer-readable recording medium which records the program which concerns on the 3rd viewpoint of this invention is used for the computer which reads the area | region which the correlation coefficient becomes largest on the same scanning line in the several image obtained by image | photographing the same object from a different direction. A computer-readable recording medium having recorded thereon a program for executing a matching sequence to be associated with the same position and a line segment discrimination procedure for discriminating whether or not a corresponding line segment is drawn by representing the same position in each of the plurality of images. For example, when it is determined that the line segment is drawn according to the line segment discrimination procedure, the matching procedure is not the regions where the correlation coefficient is maximum on the same scan line, but the intersection points of the scan line and the line segment at the same position. Corresponding by representing It characterized.

According to the present invention, it is possible to provide a stereo matching processing system, a stereo matching processing method, and a computer-readable recording medium having recorded thereon a program capable of accurately matching the same position between a plurality of images.

1 is a block diagram illustrating a configuration example of a stereo matching processing system.
2 is a diagram illustrating a display example of an auxiliary line input screen;
3A is a diagram illustrating a display example of stereo matching results.
3B is a diagram illustrating a display example of stereo matching results.
4 is a diagram for explaining a DP matching process.
5 illustrates a search plane.
6 is a diagram for explaining stereo matching processing.
7 is a flowchart showing an example of auxiliary line input processing;
8 is a flowchart showing an example of stereo matching processing.
9A illustrates a search plane.
9B illustrates a search plane.
Fig. 10 is a diagram for explaining improvement of correspondence using auxiliary lines.
11 is a diagram illustrating a search plane in a modification.
FIG. 12 is a diagram for explaining improvement of correspondence using auxiliary lines in a modification. FIG.
13 is a diagram showing a search plane in a modification.

Next, the best form for implementing this invention is demonstrated with reference to drawings. 1 is a block diagram illustrating a configuration example of a stereo matching processing system according to an embodiment of the present invention. The stereo matching processing system 1 is composed of, for example, a general-purpose computer, and as shown in FIG. 1, the display unit 10, the auxiliary line input unit 11, the mutual expression unit 12, and stereo The matching part 13 and the ortho process and the earth expression part 14 are provided.

The display unit 10 is composed of, for example, a liquid crystal display (LCD) or the like. 2 is a diagram illustrating a display example of an auxiliary line input screen. 3A and 3B are diagrams showing display examples of stereo matching results. The display unit 10 is an auxiliary line input screen shown in FIG. 2 including images of two aerial photographs (hereinafter referred to as left and right images) obtained by photographing the same object from different directions, and FIGS. 3A and FIG. The result of stereo matching between the left image and the right image shown in 3b is displayed.

The auxiliary line input unit 11 is configured by, for example, a keyboard, a mouse, or the like, and is used when the operator draws auxiliary lines on the left image and the right image on the auxiliary line input screen displayed on the display unit 10. Here, the auxiliary line is a line segment for associating the point which becomes the same position between a left image and a right image by operation of an operator.

The mutual expression unit 12 is realized by, for example, a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), a hard disk drive, or the like, and expresses camera parameters at the time of shooting. ), Parallelizing processing of reprojecting the left image and the right image onto a common parallel surface as necessary. Here, a facial expression means obtaining a predetermined value with respect to the object of evaluation.

Specifically, in the facial expression processing, the mutual expression unit 12 reads the coordinate values of the object that is commonly drawn out in the left image and the right image, and uses the two coordinate values read out. The camera parameters at the time of photography, such as a camera rotation angle, between an image and a right image are expressed. Thereby, the mutual expression part 12 can express a camera parameter at the time of photography which is difficult to grasp by the influence of the attitude | position change of an aircraft, etc., even when an aerial photograph is normally taken from the direction close to a perpendicular | vertical.

Thereafter, the mutual expression unit 12 performs parallelization processing so that the left image and the right image are arranged on the common parallel plane so that the epipolar line connecting the epipoles on the left image and the right image matches one of the plurality of scanning lines. Reproject.

The stereo matching unit 13 is realized by, for example, a CPU, a ROM, a RAM, a hard disk drive, or the like. The RAM includes an auxiliary line input flag indicating that the auxiliary line is input to each of the left image and the right image, and an intersection coordinate buffer for storing coordinates of the intersection between the auxiliary line and the scan line.

The stereo matching unit 13 performs stereo matching processing, specifically, DP (Dynamic Programming) matching processing, on the left image and the right image (parallelized image pair) to which the parallel expression processing is performed in the mutual expression unit 12. By doing so, positional shift (parallax) between a left image and a right image is measured, and parallax data is generated.

4 is a diagram for explaining a DP matching process. Specifically, the stereo matching unit 13 is shown in FIG. 4 when the auxiliary line is not input to the left image and the right image by the operation of the auxiliary line input unit 11 by the operator in the DP matching process. As described above, a correlation between the left image and the right image on the same scan line is taken, and the lattice region having the maximum correlation coefficient is searched for.

5 is a diagram illustrating a search plane. The stereo matching unit 13 associates the coordinates of the centers of the lattice regions with the maximum correlation coefficient on the search plane shown in FIG. 5. In this search plane, the horizontal axis x1 represents the x coordinate in the right image, and the vertical axis x2 represents the x coordinate in the left image.

Then, the stereo matching section 13 performs such matching for each scan line to generate parallax data, and display the image as shown in FIG. 3A on the display section 10 as a stereo matching result.

On the other hand, when the auxiliary line is input to each of the left image and the right image by the operation of the auxiliary line input unit 11 by the operator, the coordinates of the intersection point of the scan line and the auxiliary line are searched on the search plane at the location where the auxiliary line is input. We correspond to. On the other hand, in the place where an auxiliary line is not input, the coordinates of the center position of the grating | lattice area in which a correlation coefficient becomes the maximum correspond to each other on a search plane.

6 is a diagram for explaining a stereo matching process. As shown in FIG. 6, the stereo matching unit 13 performs such matching for each scan line to generate parallax data corrected by the auxiliary lines, and an image as shown in FIG. 3B. Is displayed on the display unit 10 as a stereo matching result.

Thereafter, the stereo matching unit 13 combines the generated parallax data with the camera parameters expressed by the mutual expression unit 12 to determine the position in the three-dimensional coordinate system corresponding to each pixel by the principle of triangulation. By calculating and extracting DSM (Digital Surface Model) data including elevation data indicating the height of the surface layer of the object, the depth and shape to the object are obtained.

The ortho processing and site expression unit 14 is realized by, for example, a CPU, a ROM, a RAM, a hard disk drive, or the like, and ortho which orthogonally converts an aerial photograph image and DSM data using DSM data. The ortho image and the ortho DSM data are processed by using the image of the aerial photograph subjected to the processing or the ortho processing and the DSM data, and the earth expression processing for obtaining the exact coordinates of the object, specifically, the longitude and latitude of the object. Create

Here, the ortho image contains color data, latitude data and longitude data indicating latitude and longitude determined by the expression of the earth. In addition, the ortho DSM data includes elevation data indicating the height of the surface layer of the object, latitude data, and longitude data. By obtaining such latitude data and longitude data, the position of the same object can be correlated with each other between images of aerial photographs photographed at different timings.

Next, a process performed by the stereo matching processing system having the above configuration will be described with reference to the drawings.

In the stereo matching processing system, the auxiliary line input processing and the stereo matching processing are periodically executed. The auxiliary line input process and the stereo matching process are executed at an arbitrary timing, for example, when there is an instruction from the operator, when there is a predetermined image, or when a predetermined time is reached.

7 is a flowchart showing details of auxiliary line input processing. In the auxiliary line input processing, the stereo matching unit 13 is displayed on the left image and the right image by the operation of the auxiliary line input unit 11 by the operator on the auxiliary line input screen displayed on the display unit 10. It is judged whether or not shipbuilding is input (step S11). At this time, if the auxiliary line is not input (step S11; NO), the stereo matching unit 13 ends the auxiliary line input processing as it is.

In contrast, when it is determined in step S11 that the auxiliary line is input (step S11; YES), the stereo matching unit 13 coordinates the intersection of the scan line and the auxiliary line with respect to the left image and the right image, respectively. Is obtained and stored in the intersection coordinate buffer installed in the RAM (step S12).

Then, the stereo matching unit 13 sets the auxiliary line input flag provided in the RAM on (step S13), and then ends the auxiliary line input processing.

8 is a flowchart showing details of stereo matching processing. In the stereo matching process, the mutual expression unit 12 performs an expression process to express the camera parameters at the time of photographing (step S21), and to perform parallelization processing so that the epipolar line is divided into a plurality of scan lines. The left image and the right image are reprojected on a common parallel plane so as to match one (step S22).

Next, the stereo matching unit 13 checks whether the auxiliary line input flag provided in the RAM is on, and determines whether or not the auxiliary line is input to each of the left image and the right image (step S23). .

Here, in the case where it is determined in step S23 that the auxiliary line is not input (step S23; NO), the stereo matching unit 13 generates parallax data without correction by the auxiliary line (step S24). .

In the process of step S24, the stereo matching part 13 searches the grid area | region where the correlation coefficient becomes the maximum by taking the correlation of the left image and the right image on the same scanning line. Then, the stereo matching unit 13 associates the coordinates of the centers of the lattice regions with the maximum correlation coefficient on the search plane. The stereo matching section 13 performs such matching for each scan line, thereby generating parallax data without correction by the auxiliary line.

In contrast, when it is determined in step S23 that the auxiliary line is input (step S23; YES), the stereo matching unit 13 generates parallax data with correction by the auxiliary line (step S25).

In the process of step S25, the stereo matching part 13 matches the coordinate of the intersection of the scanning line and auxiliary line stored in the intersection coordinate buffer provided in RAM, on the search plane at the location into which the auxiliary line was input. In the places where the auxiliary line is not input, the coordinates of the center positions of the lattice regions where the correlation coefficient is maximum are associated with each other on the search plane. The stereo matching section 13 performs such matching for each scan line, thereby generating parallax data with correction by the auxiliary line.

And the stereo matching part 13 displays the image based on the parallax data produced | generated by the process of step S24 or step S25 on the display part 10 as a stereo matching result (step S26), and the parallax data of step S21 is carried out. The DSM data including elevation data indicating the height of the surface layer of the object is extracted using the camera parameter expressed in the processing (step S27).

Subsequently, the ortho process and mount facial expression unit 14 performs orthogonal processing using the DSM data extracted in the process of step S27 to orthogonally convert the image of the aerial photograph and the DSM data (step S28).

Then, the ortho processing and site expression unit 14 performs orthogonal processing using the DSM data subjected to the ortho processing in step S28, so that the ortho image and elevation data including elevation data indicating the height of the surface layer of the object are processed. DSM data is generated (step S29).

Subsequently, an operation of the stereo matching processing system that executes the above processing will be described with reference to specific examples.

First, when the auxiliary line is not input, since the auxiliary line input flag is determined to be off in the process of step S23 shown in FIG. 8, the center coordinates of the grid area where the correlation coefficient is maximum in the process of step S24 are compared. Is matched on the search plane.

9A and 9B are diagrams illustrating a search plane. 10 is a figure for demonstrating the improvement of the matching using an auxiliary line. In the process of step S24, when incorrect correspondence is made on the search plane as shown in Fig. 9A, as shown in Fig. 10 by the operation of the auxiliary line input unit 11 by the operator, the beam on the left image is shown. The ship A2 and the auxiliary line A1 on the right image are input in correspondence with each other, and the auxiliary line B2 on the left image and the auxiliary line B1 on the right image are correspondingly input.

Then, in the process of step S12 shown in FIG. 7, x coordinate a1, a2, b1, and b2 of the intersection of a scanning line and auxiliary line A1, A2, B1, and B2 are calculated | required, respectively, and of step S13 The auxiliary line input flag is set on in the processing.

Since the auxiliary line input flag is determined to be on at this time in the processing of step S23 shown in FIG. 8, in the processing of step S25, as shown in FIG. 9B, the scanning line and the auxiliary line A2 on the left image are shown. The coordinate a1 of the intersection of the coordinate a2 of the intersection and the scanning line on the right image, and the coordinate line a1 of the auxiliary line A1, the scanning line on the left image and the scanning line on the coordinate b2 of the intersection of the auxiliary line B2 and the right image And the coordinates b1 of the intersection points of the auxiliary lines B1 are respectively associated on the search plane.

Thereby, the stereo matching processing system 1 can improve the misalignment on a search plane, and can exactly match the same position between a left image and a right image.

As described above, according to the stereo matching processing system 1 according to the present embodiment, since the mismatching on the search plane can be improved by correcting by the auxiliary line, the stereo matching results as a result of the stereo matching. The parallax data can be corrected. Then, the stereo matching processing system 1 can obtain elevation data that accurately represents the height of the surface layer of the object by extracting the DSM data using the corrected parallax data.

In addition, this invention is not limited to the said embodiment, A various deformation | transformation and application are possible. Hereinafter, the modified aspect of the said embodiment applicable to this invention is demonstrated.

In the said embodiment, in the process of step S25 shown in FIG. 8, the coordinate a1 of the intersection of the scanning line and auxiliary line A2 on a left image, and the coordinate a1 of the intersection of the scanning line and auxiliary line A1 on a right image are In addition, the coordinate b2 of the intersection of the scanning line and the auxiliary line B2 on the left image and the coordinate b1 of the intersection of the scanning line and the auxiliary line B1 on the right image are respectively corresponded on the search plane, and between them, The center coordinates of the lattice region where the correlation coefficient is maximum have been described as being associated on the search plane. However, the present invention is not limited to this. 11 is a diagram illustrating a search plane in a modification. As shown in Fig. 11, the coordinates a1 and a2 on the search plane and the points on the line segments connecting the coordinates b1 and b2 may all correspond to each other.

Then, whether or not to input an auxiliary line (forced auxiliary line) for performing such a matching may be determined by an operation by the auxiliary line input unit 11 by the operator.

In the above embodiment, the auxiliary line is drawn perpendicular to the scan line as an example, but the present invention is not limited thereto. FIG. 12 is a diagram for explaining improvement of the correspondence using the auxiliary line in the modification. FIG. 13 is a figure which shows the search plane in a modification. As shown in FIG. 12, the auxiliary line may be drawn horizontally on the scan line. When the auxiliary line is drawn horizontally on the scanning line, as shown in FIG. 13, points (equal points) that divide the auxiliary line (line segment) into n (n is a natural number) in each of the left image and the right image, What is necessary is just to correspond on a test plane from the start of a line segment to the end point of a line segment in order.

In the said embodiment, the mutual expression part 12 reads the coordinate value of the target object common to the left image and the right image in an expression process, and uses the two coordinate values read, and the left image and the right image It demonstrated as what expresses a camera parameter at the time of imaging | photography, such as a camera rotation angle, with an image. However, the present invention is not limited to this, but the method of expressing a camera parameter at the time of shooting is arbitrary, for example, the camera parameter at the time of photography is used using the value computed by a figure drawing program. You may look.

In the above embodiment, the ortho processing and site expression unit 14 has been described as performing an earth expression for obtaining the longitude and latitude of the object using the image of the aerial photograph subjected to the ortho processing and the DSM data. However, the present invention is not limited to this, and the technique for performing the earth look is arbitrary, for example, a plurality of images on the image of the aerial photograph in which the longitude, latitude, and elevation value (image coordinates) are detected in advance. The conversion equation from the image coordinates of the point to the ground coordinates (longitude, latitude, and elevation) of the ground surface may be obtained.

In addition, aerial triangulation data in which longitude, latitude, and elevation have been measured may be used by taking an aerial photograph and taking aerial photographs. In this way, it is possible to obtain ground coordinates in arbitrary coordinates on an image. do. Here, the anti-airmark means that when the aerial photograph is taken by various sensors from the aircraft, the shape can be clearly recognized on the image of the aerial photograph and the image coordinates can be measured. Therefore, at the point where the anti-airmark is set, accurate three-dimensional coordinates are displayed.

In the above embodiment, the ortho image has been described as including color data, latitude data, and longitude data, and ortho DSM data includes elevation data, latitude data, and longitude data. However, the present invention is not limited to this, and the ortho image and the ortho DSM data may include coordinate value data expressed in another coordinate system instead of latitude data and longitude data, and other criteria instead of the elevation data. Height data indicating relative heights from may be included.

In addition, about this application, the priority based on Japanese Patent Application No. 2008-300103 is claimed, and it uses the specification of a Japanese Patent Application No. 2008-300103, a claim, and the whole drawing as a reference in this specification. .

1: stereo matching processing system
10: display unit
11: auxiliary line input unit
12: mutual expression
13: stereo matching unit
14: ortho treatment, earth expression part

Claims (6)

In a plurality of images obtained by photographing the same object from different directions, regions where the correlation coefficient is maximum on the same scan line coinciding with the epipolar line to which the epipoles on the plurality of images are connected indicate the same position. Matching counterparts,
A line segment discriminating unit for discriminating whether or not the corresponding line segment is drawn by indicating the same position in each of the plurality of images.
And
When the said matching part judges that the said line segment is drawn by the said line segment discrimination part, it is assumed that the intersection points of the scan line and the line segment are the same position, not the regions where the correlation coefficient is maximum on the same scan line. Counterpart
Stereo matching processing system, characterized in that. The method of claim 1,
When the plurality of line segments are drawn on each of the plurality of images, the correspondence unit is associated with the line segments connecting the intersection points of the scan line and the line segments to represent the same position. . The method of claim 2,
And the correspondence unit is associated with each of the plurality of images when the line segment parallel to the scan line is drawn so as to indicate the same positions between the start points and the end points of the line segments. The method of claim 3,
When the said line correspondence part is drawn with the said line segment horizontal to the said scan line in each of the said some image | video, the said correspondence | matching part makes a correspondence by making the same position in order from said viewpoint, respectively, equally dividing the said line segment, It is characterized by the above-mentioned. Stereo matching processing system. Matching step of associating a plurality of images obtained by photographing the same object from different directions, in which regions having maximum correlation coefficients represent the same position on the same scan line that matches the epipolar line to which the epipoles on the plurality of images are connected. and,
Line segment determination step of judging whether the corresponding line segment is drawn by showing the same position in each of the plurality of images.
Including,
In the matching step, when it is determined that the line segment is drawn by the line segment discriminating step, it is assumed that the intersection points between the scan line and the line segment are the same position, not the regions where the correlation coefficient is maximum on the same scan line. Countering
Stereo matching processing method, characterized in that. In a plurality of images obtained by photographing the same object from a different direction, correspond to regions where the correlation coefficients are the same on the same scan line that coincides with the epipolar line to which the epipoles on the plurality of images are connected. A computer-readable recording medium having recorded thereon a program for executing a matching procedure and a line segment discrimination procedure for determining whether corresponding line segments are drawn by indicating the same position in each of the plurality of images,
The matching procedure indicates that when the line segment is drawn according to the line segment discrimination procedure, not the regions where the correlation coefficient is maximum on the same scan line but the intersection points of the scan line and the line segment are the same. Countering
A computer-readable recording medium having recorded thereon a program.

Images