JP2010128575A - Stereo image processing device, stereo image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To correct vertical deviation in stereo pair images. <P>SOLUTION: A stereo image processing device specifies a line pair having a high correlation between one of stereo pair images and the other image for an image region having a prescribed width in parallel in a scanning line direction (X direction) in the stereo pair images that are prescribed regions photographed from two different positions and have a scanning line direction and an epipolar line direction matched with each other, obtains deviation (vertical parallax) in a vertical direction (Y direction) between one of the images and the other image based on each position of the specified line pair, and corrects the stereo image so as to eliminate the obtained deviation. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a stereo image processing apparatus, a stereo image processing method, and a program.

  A method of generating three-dimensional data (DEM (Digital Elevation Map) data) indicating terrain by stereo matching based on an image obtained from an artificial satellite or an aircraft is widely used.

  Here, stereo matching processing refers to the principle of triangulation using the parallax obtained from two images taken from different viewpoints, that is, so-called stereo images, by finding corresponding points in each image capturing the same point. To obtain the depth and shape to the target.

  Various methods have already been proposed for this stereo matching process. For example, Patent Document 1 discloses a method using an area correlation method that is generally widely used. This area correlation method sets a correlation window in the left image as a template, moves the search window in the right image to calculate the cross-correlation coefficient with the template, This is a method of obtaining corresponding points by searching.

  In the above method, in order to reduce the amount of processing, the movement range of the search window is limited to the direction of the epipolar line in the image, thereby shifting the position of the corresponding point in the right image for each point in the left image. Can be obtained, that is, parallax. Here, the epipolar line is a straight line that can be drawn as an existing range of a point corresponding to the point in one image in a stereo image. The epipolar line is described in, for example, “Image Analysis Handbook” (supervised by Mikio Takagi and Yoshihisa Shimoda, published by the University of Tokyo Press, January 1991, pages 597-599).

  Usually, the epipolar line direction is different from the scanning line direction of the stereo image. However, by performing coordinate conversion, the epipolar line direction can be matched with the scanning line direction of the stereo image, and rearrangement can be performed. This coordinate transformation method is described in the above “Image Analysis Handbook”.

  In the stereo image that has been rearranged as described above, the moving range of the search window for the corresponding points can be limited to the scanning line (epipolar line), so the y coordinate between the corresponding points in the left and right images is one. In addition, the parallax can be expressed only by the difference in x-coordinates between corresponding points in the left and right images.

Japanese Patent Laid-Open No. 3-167678

  However, due to camera shake at the time of shooting a stereo image, even if the process of coordinate conversion is performed to match the epipolar line direction with the scan line direction of the image, the corresponding points in the left and right images of the stereo image There is a possibility that a shift occurs in the vertical direction.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a stereo image processing apparatus capable of correcting a vertical shift of a stereo pair image.

A stereo image processing apparatus according to a first aspect of the present invention provides:
Stereo pair image data acquisition means for acquiring image data of a stereo pair image in which a predetermined area is photographed from two different positions, and the scanning line direction and the epipolar line direction substantially coincide with each other;
Row specifying means for specifying a pair of rows having a high correlation between one image of the stereo pair image and the other image with respect to an image region having a predetermined width parallel to the scanning line direction of the stereo pair image;
A shift obtaining means for obtaining a shift between one image of the stereo image and the other image based on each position of the pair of rows specified by the row specifying means;
Image correcting means for correcting the stereo image so as to eliminate the deviation calculated by the deviation determining means;
It is characterized by providing.

A stereo image processing method according to a second aspect of the present invention includes:
A stereo pair image data acquisition step of acquiring image data of a stereo pair image obtained by photographing a predetermined region from two different positions;
A row specifying step for specifying a pair of rows having a high correlation between one image and the other image of the stereo pair image for an image region having a predetermined width parallel to the scanning line direction of the stereo pair image;
A shift obtaining step for obtaining a shift between one image of the stereo image and the other image based on each position of the pair of rows specified in the row specifying step;
An image correction step of correcting the stereo image so that the shift calculated in the shift determination step is eliminated;
It is characterized by providing.

The program according to the second aspect of the present invention is:
Computer
Stereo pair image data acquisition means for acquiring image data of a stereo pair image in which a predetermined area is photographed from two different positions, and the scanning line direction and the epipolar line direction substantially coincide with each other;
Row specifying means for specifying a pair of rows having a high correlation between one image of the stereo pair image and the other image with respect to an image region having a predetermined width parallel to the scanning line direction of the stereo pair image;
A shift obtaining means for obtaining a shift between one image of the stereo image and the other image based on each position of the pair of rows specified by the row specifying means;
Image correcting means for correcting the stereo image so as to eliminate the deviation calculated by the deviation determining means;
It is made to function as.

  According to the present invention, it is possible to correct a vertical shift of a stereo pair image.

The stereo image processing apparatus 1 according to the embodiment of the present invention will be described below. The stereo image processing device 1 is a device that corrects a shift in the position of the left and right images with respect to an image (stereo pair image) composed of a left image and a right image obtained by photographing the same region from different positions on the left and right.
Note that this stereo pair image is obtained by, for example, the epipolar line direction according to the method described in “Image Analysis Handbook” (supervised by Mikio Takagi / Yoshihisa Shimoda, published by the University of Tokyo, January 1991, pages 597-599). Are rearranged (parallelized) so that the scanning directions of the images coincide with each other. That is, the y coordinates of corresponding points in the right image and the left image are matched to some extent.

  As shown in FIG. 1, the stereo image processing apparatus 1 includes a control unit 11, a RAM 12, a ROM 13, and a storage unit 14.

The control unit 11 includes a CPU (Central Processing Unit) and the like, and executes predetermined processing based on a program stored in the ROM 13 or the storage unit 14 to control the entire stereo image processing apparatus 1.
For example, the control unit 11 determines the scan line (row) having the highest correlation between the right image and the left image of the stereo pair image, and shifts to the vertical position (Y coordinate) of the scan line determined to have the high correlation (vertical). When there is a parallax, vertical parallax correction processing is performed to correct this. Details of the vertical parallax correction processing will be described later.

  A RAM (Random Access Memory) 12 is a volatile memory that functions as a work memory for storing a program read by the control unit 11 to execute a predetermined process and data necessary for the control unit 11 to execute the program. It is memory.

  A ROM (Read Only Memory) 13 is a non-volatile memory in which a program for the control unit 11 to execute predetermined processing, fixed data, and the like are stored in advance. The control unit 11 reads out a program or the like from the ROM 13 as necessary, develops it in the RAM 12, and executes predetermined processing based on the program or the like.

  The storage unit 14 includes a storage device such as a hard disk drive, and stores various information. For example, the storage unit 14 stores a stereo pair image that is a target of vertical parallax correction processing.

In addition, the storage unit 14 stores various parameters used in vertical parallax correction processing, which will be described later, as illustrated in FIG.
The counter i is a counter used in the vertical parallax correction process.
In y1, y2, and y3, the value of the Y coordinate of the row for comparing the correlation between the right image and the left image in the vertical parallax correction processing is set.

  Note that the stereo image processing apparatus 1 can be configured using a computer or the like that has been widely used.

  Next, the operation of the vertical parallax correction process will be described.

  First, the control unit 11 selects one unprocessed stereo pair image, that is, one pair of a right image and a left image as shown in FIG. 4 from the storage unit 14 (step S101).

  Subsequently, the control unit 11 initializes the value of the counter i to 1 (step S102).

  Subsequently, the control unit 11 selects a set (row) of pixels whose Y coordinate value is yi from the left image (step S103). This row may be a set of pixels having a height of several pixels in the Y direction.

  Subsequently, the control unit 11 selects a total of three rows separated by ± 1 pixel from y1 and y1 from the right image (step S103). That is, in this process, three rows having Yi-1, yi, and yi + 1 from the bottom are selected from the right image (step S104).

  Subsequently, as shown in FIG. 5, the control unit 11 obtains a correlation between the yi row of the left image selected in step S102 and the yi-1, yi, and yi + 1 rows of the right image selected in step S103. The right image row having the highest correlation with the yi row of the left image is specified (step S105).

Specifically, the processing in step S105 may be performed as follows.
First, using the method described in JP-A-4-299474, the control unit 11 obtains each pixel of the right image corresponding to each pixel constituting the yi row of the left image. Subsequently, the control unit 11 may identify one of the yi−1, yi, and yi + 1 rows that most closely matches the position of each pixel of the obtained right image as the row of the right image having the highest correlation.
Note that the processing in step S105 is not limited to the above example, and other matching methods may be used as long as the row with the highest correlation can be identified.

Subsequently, the control unit 11 stores, in the storage unit 14, the amount of shift between the positions of the left and right image rows identified as having a high correlation in step S105 (step S106).
For example, if the line yi of the right image is identified as the line having the highest correlation with the line yi of the left image in step S105, it is stored that there is no line shift (amount of deviation: 0). If the yi + 1 row of the right image is identified as the row having the highest correlation with the yi row of the left image in step S105, the right image is 1 upward (Y axis positive direction) with respect to the left image. It is stored that there is a pixel shift.

  Subsequently, the control unit 11 determines whether or not the value of the counter i is 3 (step S107). When it is determined that the value of the counter i is not 3 (step S107; No), the control unit 11 increments the counter i by 1 (step S108) and repeats the processes of steps S103 to S107. That is, the vertical shift between the right image and the left image is calculated and stored at another yi position.

  When it is determined that the value of the counter i is 3 (step S107; Yes), the control unit 11 calculates the average value of the deviation amounts stored in step S106 (that is, the deviation amounts at y1, y2, and y3). Obtained (step S109). When this process ends, the control unit 11 deletes (resets) the information indicating the amount of deviation stored in step S106 because it will not be used thereafter.

  Subsequently, the control unit 11 corrects the stereo pair images (the right image and the left image) based on the average value of the shift amounts obtained in step S109 (step S110). For example, if the average value of the shift amounts obtained in step S109 indicates that the right image is shifted by 2 pixels in the upward direction, the entire right image is moved downward by 2 pixels (right image) The overall y coordinate value is set to -2. Conversely, the entire left image may be moved upward by two pixels (the y coordinate value of the entire left image is incremented by +2).

  Subsequently, the control unit 11 determines whether or not there is an unprocessed (unselected) stereo pair image (step S111).

When it is determined that there is an unprocessed (unselected) stereo image (step S111; Yes), the control unit 11 moves the process to step S101, and the right image and the left image are shifted from the unprocessed stereo image. The process of correcting (Steps S101 to S110) is repeated.
When it is determined that there is no unprocessed (unselected) stereo image (step S111; No), the control unit 11 ends the vertical parallax correction process.

  As described above, in the present embodiment, in the parallel stereo pair images, the correlation is examined in the vicinity of the position (yi) corresponding to both images, and the line shift is obtained. Then, the vertical position of the stereo pair image is corrected so that the obtained line shift is eliminated. Accordingly, it is possible to correct the vertical shift of the stereo pair image.

The present invention can be variously applied and modified.
For example, in the present embodiment, in step S106, the line with the highest correlation is identified from the three lines yi-1, yi, and yi + 1 in the right image with respect to the line yi in the left image. However, the present invention is not limited to this. For example, as shown in FIG. 6, the row with the highest correlation may be specified from the five rows yi−2, yi−1, yi, yi + 1, and yi + 2 in the right image. In short, the row of the right image for specifying the highly correlated row with respect to the row of yi of the left image may be any plurality of rows near yi and yi.

  Further, in the present embodiment, the amount of misalignment between the left and right images is obtained and recorded in three rows y1, y2, and y3 (step S106), and the average value is obtained (step S109). However, the number of lines to be recorded for obtaining the amount of deviation may be three or less or more.

  In the present embodiment, the stereo pair image is described as being composed of a left image and a right image taken from the right and left. However, the present invention is not limited to this, and the present invention is applied to a stereo pair image obtained by photographing the same region from different positions. Is applicable.

  The above-described hardware configuration and flowchart are examples, and can be arbitrarily changed and modified.

  The central part for performing the processing of the stereo image processing apparatus 1 including the control unit 11, the RAM 12, the ROM 13, the storage unit 14, and the like can be realized using a normal computer system without using a dedicated system. is there. For example, a computer program for executing the above operation is stored and distributed in a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.), and the computer program is installed in the computer. Thus, the stereo image processing apparatus 1 that executes the above-described processing may be configured. In addition, the stereo image processing apparatus 1 may be configured by storing the computer program in a storage device included in a server device on a communication network such as the Internet and downloading it by a normal computer system.

  Further, when the functions of the stereo image processing apparatus 1 are realized by sharing of an OS (operating system) and an application program, or by cooperation between the OS and the application program, only the application program portion is stored in a recording medium or a storage device. It may be stored.

  It is also possible to superimpose a computer program on a carrier wave and distribute it via a communication network. For example, the computer program may be posted on a bulletin board (BBS, Bulletin Board System) on a communication network, and the computer program distributed via the network. The computer program may be started and executed in the same manner as other application programs under the control of the OS, so that the above-described processing may be executed.

It is a block diagram which shows the internal structure of a stereo image processing apparatus. It is the figure which showed the example of the parameter stored in the memory | storage part. It is a flowchart which shows an example of operation | movement of a vertical parallax correction process. It is the figure which showed the example of the stereo pair image. It is a figure for demonstrating the process which specifies the pair of a line with high correlation from the left image and right image of a stereo pair image. It is a figure for demonstrating the process which specifies the pair of a line with high correlation from the left image and right image of a stereo pair image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Stereo image processing apparatus 11 Control part 12 RAM
13 ROM
1 storage unit

Claims (7)

Stereo pair image data acquisition means for acquiring image data of a stereo pair image in which a predetermined area is photographed from two different positions, and the scanning line direction and the epipolar line direction substantially coincide with each other;
Row specifying means for specifying a pair of rows having a high correlation between one image of the stereo pair image and the other image with respect to an image region having a predetermined width parallel to the scanning line direction of the stereo pair image;
A shift obtaining means for obtaining a shift between one image of the stereo image and the other image based on each position of the pair of rows specified by the row specifying means;
Image correcting means for correcting the stereo image so that the deviation calculated by the deviation determining means is eliminated;
A stereo image processing apparatus comprising: The row specifying means has the highest correlation with the one row among a plurality of rows of the other image of the stereo pair image corresponding to a peripheral position of one row of the one image of the stereo pair image. Identify a row as a highly correlated pair with the one row,
The stereo image processing apparatus according to claim 1. The row specifying means specifies a pair of rows having a high correlation for each image region for a plurality of image regions having a predetermined width parallel to the scanning line direction of the stereo pair image,
The deviation obtaining means obtains a deviation based on the position of the row pair for each pair of rows identified by the row identifying means, and calculates an average value of the obtained plural deviations from one image of the stereo image and the other As a gap between the images of
The stereo image processing apparatus according to claim 1 or 2, wherein A stereo pair image data acquisition step of acquiring image data of a stereo pair image obtained by photographing a predetermined region from two different positions;
A row specifying step for specifying a pair of rows having a high correlation between one image and the other image of the stereo pair image for an image region having a predetermined width parallel to the scanning line direction of the stereo pair image;
A shift obtaining step for obtaining a shift between one image of the stereo image and the other image based on each position of the pair of rows specified in the row specifying step;
An image correction step of correcting the stereo image so that the shift calculated in the shift determination step is eliminated;
A stereo image processing method comprising: The row specifying step has the highest correlation with the one row among a plurality of rows of the other image of the stereo pair image corresponding to a peripheral position of one row of the one image of the stereo pair image. Identify a row as a highly correlated pair with the one row,
The stereo image processing method according to claim 4. In the row specifying step, for a plurality of image regions having a predetermined width parallel to the scanning line direction of the stereo pair image, a pair of rows having a high correlation is specified for each image region,
The deviation obtaining step obtains a deviation based on the position of the pair of rows for each pair of rows identified in the row identifying step, and calculates an average value of the obtained plural deviations from one image of the stereo image and the other As a gap between the images of
The stereo image processing method according to claim 4 or 5, wherein Computer
Stereo pair image data acquisition means for acquiring image data of a stereo pair image in which a predetermined area is photographed from two different positions, and the scanning line direction and the epipolar line direction substantially coincide with each other;
Row specifying means for specifying a pair of rows having a high correlation between one image of the stereo pair image and the other image with respect to an image region having a predetermined width parallel to the scanning line direction of the stereo pair image;
A shift obtaining means for obtaining a shift between one image of the stereo image and the other image based on each position of the pair of rows specified by the row specifying means;
Image correcting means for correcting the stereo image so as to eliminate the deviation calculated by the deviation determining means;
A program characterized by functioning as