JP5338724B2 - Image processing apparatus and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing program for executing corresponding point search processing for deriving coordinates of corresponding points in a second image corresponding to a point of interest specified in a first image of a stereo image. It is.

  Corresponding point search processing using a stereo image composed of a plurality of images obtained by viewing an object from different viewpoints is an important image processing technique in various fields such as three-dimensional image measurement and computer vision. The corresponding point search process based on the stereo image derives the coordinates of the corresponding point in the second image corresponding to the point of interest specified in the first image based on the first image and the second image constituting the stereo image. It is processing to do. The corresponding point is a point in the second image where an image having a high correlation with the image of the target point in the first image exists. Recently, in the corresponding point search process, it is required to derive the coordinates of the corresponding point with high accuracy.

  Conventionally, there is a method of phase difference analysis (Phase-difference Spectrum Analysis) as a method of high-accuracy corresponding point search processing. Hereinafter, this method is referred to as a PSA method. The PSA method is shown in, for example, Non-Patent Document 1, Non-Patent Document 2, Patent Document 1, and Patent Document 2.

  Hereinafter, the PSA method will be described with reference to FIGS. In the PSA method, the coordinates (dx0, dy0) of the deviation of the coordinates (x02, y02) of the corresponding point in the second image with respect to the coordinates (x01, y01) of the target point in the first image are derived by the procedure shown below. It is a technique. Hereinafter, the coordinates of the deviation are referred to as corresponding point deviation coordinates. Note that (dx0, dy0) = (x02-x01, y02-y01).

  In the PSA method, a first phase image and a second phase image, which are phase images in the spatial frequency domain corresponding to the first image and the second image, are calculated. These phase images are obtained by performing a two-dimensional discrete Fourier transform on each of the first image and the second image.

  Next, for each frequency component of both phase images, a phase difference Δθ (u, v) corresponding to the frequency component (u, v) between the first phase image and the second phase image is calculated. U and v are coordinate values in the frequency space.

  In the PSA method, a voting space 100 indicating a possible range of the corresponding point deviation coordinates (dx0, dy0) is set. The voting space 100 is discretized by being divided into a plurality of unit regions at a predetermined spatial period. The resolution (accuracy) for deriving the corresponding point deviation coordinates (dx0, dy0) is determined according to the spatial period.

  FIG. 17 is a schematic diagram illustrating an example of the voting space 100. In the example shown in FIG. 17, the voting space 100 is discretized into 10 sections in each of the ΔX axis direction and the ΔY axis direction. A plurality of unit areas that are segmented (discretized) in a predetermined spatial period in the voting space 100 are referred to as voting areas 20. Setting the voting space 100 means that the image processing apparatus secures a data area corresponding to the array of voting areas 20 in the memory. That is, each voting area 20 corresponds to each element data in the data area secured in the memory. The size of the voting space 100 and the spatial period of the voting area 20 are determined in advance according to the application target.

  In the PSA method, the frequency component (u, v), the phase difference Δθ (u, v) based on the frequency component, and the coordinate (dx, dy) in the voting space are changed into a predetermined expression as variables. , v) and a phase difference Δθ (u, v) calculated based on the frequency component thereof, a straight line (primary expression) in the voting space 100 is derived. The linear expression thus derived is generally called a voting function. The frequency component (u, v) and the phase difference Δθ (u, v) corresponding to the frequency component (u, v) are changed to the frequency component (u, v), the phase difference Δθ (u, v) and the coordinates (dx, A straight line in the voting space 100 obtained by applying to a predetermined expression having dy) as a variable (parameter) is a candidate for the corresponding point deviation coordinate (dx0, dy0) determined according to the frequency component (u, v). Represents the position. Hereinafter, this straight line is referred to as a candidate straight line.

  FIG. 18 is a schematic diagram illustrating an example of the voting space 100 including one candidate straight line 32 corresponding to a certain frequency component (u, v). By applying the frequency component (u, v) and the phase difference Δθ (u, v) to the voting function, a linear expression representing the candidate straight line 32 that is a straight line in the coordinate system of the voting space 100 is obtained. In the PSA method, the image processing apparatus specifies the voting area 20 that overlaps with the candidate straight line 32 plotted in the voting space 100. Hereinafter, the voting area 20 specified in this way is referred to as a specific voting area 23. In FIG. 18, the specific voting area 23 is shaded with sand.

  Further, in the PSA method, for each frequency component (u, v), the derivation of the candidate straight line 32 and the specification of the specific voting area 23 overlapping the candidate straight line 32 are sequentially repeated. Further, every time the specific voting area 23 is specified, a process of sequentially adding a predetermined voting value to the specific voting area 23 is executed.

  FIG. 19 is a schematic diagram illustrating an example of a state of the voting space 100 including a plurality of candidate straight lines 32 corresponding to a plurality of frequency components (u, v). In FIG. 19, the specific voting area 23 that overlaps with only one candidate straight line 32 is shaded with sand, and the specific voting area 23 that overlaps with a plurality of candidate straight lines 32 is shaded or shaded. ing. The state shown in FIG. 19 is obtained by repeating the derivation of the voting line 32 and the specification of the specific voting region 23 for each of the plurality of frequency components (u, v).

  In the PSA method, for all frequency components (u, v), when the specification of the specific voting area 23 and the integration of the voting values are completed, the coordinates of the peak positions of the integrated values of the voting values are the corresponding point deviation coordinates. Identified as (dx0, dy0). Further, the coordinates (x02, y02) of the corresponding point in the second image are calculated based on the coordinates (x01, y01) of the attention point and the corresponding point deviation coordinates (dx0, dy0) in the first image.

  The coordinate of the peak position of the integrated value of the vote value is, for example, the coordinate of the specific vote area 23 (voting area 20) where the integrated value of the vote value is the maximum, or the specific vote area 23 using the integrated value of the vote value as a weighting factor. The coordinates of the center of gravity position of the set. FIG. 19 shows a state in which the specific voting area 23 having the maximum integrated voting value is specified as the peak position 230 (black portion) of the integrated voting value.

  On the other hand, epipolar geometry is known as a law that holds in stereo images. The details of the epipolar geometry are shown in Non-Patent Document 3, for example. According to the epipolar geometry, when one attention point is designated in the first image of the stereo image, the position of the corresponding point corresponding to the attention point of the first image in the second image of the stereo image is referred to as an epipolar line. Is limited to a straight line. The position of the epipolar line in the second image can be derived experimentally, and can also be derived by theoretical calculation based on the characteristics of two cameras that capture a stereo image and their positional relationship.

JP 2001-175864 A JP 2003-099787 A

Isamu KOUZAKI, Tomonori KANEKO, Minoru ITO, "Precise and reliable image shift detection by a new phase-difference spectrum analysis", IEICE TRANSACTIONS on Information and Systems Academic Journal 2004, E87D / 1 58-65 Isamu Kozuki and 4 others, “Highly accurate image shift detection method from phase difference”, Image Recognition and Understanding Symposium (MIRU 2002), published July 2002, II-379-II-384 Tsuyoshi Tsuyoshi and Saburo Tsuji, “Three-dimensional Vision”, published April 20, 1998 Kyoritsu Publishing Co., Ltd., P.61-77

  However, in the conventional PSA method, it is necessary to determine whether or not to add up vote values for each combination of all voting areas 20 and all frequency components (u, v) in the voting space 100. The computation load of voting processing is high. Also, by reducing the number of sections of the voting area 20 in the voting space 100, the calculation load of the voting process is reduced, but the calculation accuracy of the coordinates of the corresponding points is deteriorated. Therefore, the conventional PSA method has a problem that it is difficult to achieve both high accuracy and high speed calculation of the coordinates of corresponding points.

  Further, simply using the epipolar geometry law does not uniquely identify the coordinates of the corresponding point of the second image corresponding to the point of interest of the first image.

  An object of the present invention is to provide a highly accurate and relatively light deviation of the coordinates of corresponding points in the second image with respect to the coordinates of the target point specified in the first image of the stereo image by image processing applying the PSA method. The corresponding point search process calculated with the calculation load can be realized.

The image processing apparatus according to the present invention is based on the first phase image and the second phase image that are phase images in the spatial frequency domain corresponding to the first image and the second image constituting the stereo image, respectively. An apparatus for executing a corresponding point search process for deriving the coordinates of a corresponding point in the second image corresponding to a point of interest specified in an image, and includes the following components.
(1) A first candidate element is a reference candidate area indicating a possible range of corresponding point deviation coordinates, which are coordinates of deviation of the corresponding point coordinates in the second image with respect to the coordinates of the target point in the first image. Is an epipolar line equivalent region specifying unit for specifying a part of the epipolar line equivalent region along the first straight line corresponding to the epipolar line corresponding to the attention point.
(2) The second component is a phase difference calculation unit that calculates a phase difference according to a frequency component between the first phase image and the second phase image.
(3) The third component is a first component in the reference candidate region obtained by applying a frequency component and the phase difference corresponding to the frequency component to a predetermined formula in the region corresponding to the epipolar line. It is an overlapping area specific | specification part which specifies the overlapping area which overlaps with 2 straight lines.
(4) A fourth component is a deviation coordinate deriving unit that derives the corresponding point deviation coordinates according to the coordinates of the overlapping region.

  In the image processing device according to the present invention, the overlapping area specifying unit may include the epipolar line equivalent area and the second straight line among a plurality of voting areas that are partial areas in which the reference candidate area is discretized. It is conceivable to specify the voting area including the overlapping part as the overlapping area.

In the image processing apparatus according to the present invention, the epipolar line equivalent region specifying unit may perform the processing shown in any of (1-1) to (1-3) below.
(1-1) The epipolar line equivalent area specifying unit specifies, as the epipolar line equivalent area, a set of voting areas that overlap the first straight line among the plurality of voting areas.
(1-2) The epipolar line equivalent area specifying unit sets, as the epipolar line equivalent area, a set of the voting areas existing within a predetermined range with respect to the first straight line among the plurality of voting areas. Identify.
(1-3) The epipolar line equivalent region specifying unit specifies a portion that coincides with the first straight line in the reference candidate region as the epipolar line equivalent region.

Further, in the image processing apparatus according to the present invention, it is conceivable that each component performs the processing shown in the following (2-1) and (2-2).
(2-1) The phase difference calculation unit and the overlap area specifying unit calculate the phase difference and specify the overlap area corresponding to the phase difference for each frequency component until a predetermined termination condition is satisfied. Run sequentially.
(2-2) The deviation coordinate deriving unit sequentially accumulates vote values for each of the voting regions in the overlapping region sequentially specified for each of a plurality of frequency components by the overlapping region specifying unit, and the reference candidate region The coordinates of the peak position of the integrated value of the voting value at are specified as the corresponding point deviation coordinates.

Further, in the image processing apparatus according to the present invention, when the epipolar line equivalent region specifying unit performs the process shown in (1-2), each component is the following (3-1) and (3-2). It is conceivable to perform the processing shown in FIG.
(3-1) The phase difference calculation unit and the overlap region specifying unit calculate the phase difference and specify the overlap region corresponding to the phase difference for each frequency component until a predetermined termination condition is satisfied. Run sequentially.
(3-2) The deviation coordinate deriving unit has a positional relationship with the first straight line with respect to each of the voting regions in the overlapping region sequentially specified for each of a plurality of frequency components by the overlapping region specifying unit. Accordingly, the weighted vote values are sequentially accumulated, and the coordinates of the peak position of the accumulated value of the vote values in the reference candidate region are specified as the corresponding point deviation coordinates.

The termination condition may be any of the following conditions (4-1) or (4-2).
(4-1) The termination condition is that an integrated value of the vote values in each of the vote areas satisfies a predetermined condition.
(4-2) The termination condition is that the phase difference is calculated for a predetermined number of frequency components and the overlapping region corresponding to the phase difference is specified.

In the image processing apparatus according to the present invention, it is conceivable that each component performs the processes shown in the following (5-1), (5-2), and (5-3).
(5-1) The epipolar line equivalent region specifying unit specifies a portion that coincides with the first straight line in the reference candidate region as the epipolar line equivalent region.
(5-2) The overlapping area specifying unit specifies an intersection between the first straight line and the second straight line as the overlapping area.
(5-3) The deviation coordinate deriving unit derives a coordinate obtained by integrating the coordinates of the plurality of intersections in the reference candidate region as the corresponding point deviation coordinate.

Further, in the image processing apparatus according to the present invention, it is conceivable that each component performs the processes shown in the following (6-1), (6-2), and (6-3).
(6-1) The epipolar line equivalent region specifying unit specifies a portion that coincides with the first straight line in the reference candidate region as the epipolar line equivalent region.
(6-2) The overlapping area specifying unit specifies an intersection between the first straight line and the second straight line as the overlapping area.
(6-3) The deviation coordinate deriving unit derives the coordinates of one intersection point in the reference candidate region as the corresponding point deviation coordinates.

In the image processing apparatus according to the present invention, it is conceivable that the epipolar-line-equivalent region specifying unit performs the processing shown in the following (7-1) or (7-2).
(7-1) The epipolar line-equivalent area specifying unit, based on the conversion information stored in advance in the storage unit, determines the position of the first straight line in the reference candidate area from the coordinates of the attention point in the first image. And the region corresponding to the epipolar line is specified based on the calculated position of the first straight line.
(7-2) The epipolar line equivalent region specifying unit specifies the epipolar line equivalent region according to information representing the epipolar line equivalent region stored in advance in a storage unit corresponding to the coordinate candidate of the target point.

  The image processing apparatus according to the present invention may further include a parallelization processing unit that sets the reference candidate region so that one coordinate axis in the coordinate system of the reference candidate region is parallel to the first straight line. Conceivable.

  Further, the present invention may be configured as an image processing program for causing a computer (processor) to execute processing executed by each component included in the image processing apparatus according to the present invention described above.

  In the present invention, in the reference candidate region, the corresponding point deviation coordinates are derived based on the coordinates of the overlapping region in which the region along the first straight line corresponding to the epipolar line and the straight line determined according to the frequency component overlap. The Here, the corresponding point deviation coordinates are coordinates of a deviation between the coordinates of the point of interest in the first image of the stereo image and the coordinates of the corresponding point in the second image of the stereo image. The reference candidate region is a region indicating a range that can be taken by the corresponding point deviation coordinates. This reference candidate area corresponds to a voting space in the conventional PSA method. In the present invention, the search range for corresponding points is limited to a part of the reference candidate region along the first straight line (a region corresponding to an epipolar line). Therefore, even when the search for corresponding points in the region corresponding to the epipolar line (identification of the overlapping region) is executed with high spatial resolution, the calculation load can be kept relatively low. Therefore, according to the present invention, it is possible to realize the corresponding point search process for calculating the deviation of the coordinates of the corresponding point with respect to the coordinates of the target point with high accuracy and relatively light calculation load.

  In addition, since only the set of voting areas that overlap the first straight line corresponding to the epipolar line is specified as the epipolar line equivalent area, the epipolar line equivalent area is more narrowly limited, and the effect of reducing the computation load is reduced. Increased further. This method is effective when the accuracy (reliability) of the position of the first straight line corresponding to the epipolar line is particularly high. It should be noted that the same effect can be obtained when a portion that coincides with the first straight line in the reference candidate region is specified as the epipolar line equivalent region.

  On the other hand, the position of the first straight line may be slightly deviated from the original position due to various errors. In that case, it is effective to specify a set of voting areas existing within a predetermined range with respect to the first straight line as the epipolar line equivalent area. Thereby, the possibility of the positional deviation of the first straight line is taken into account, and the corresponding point search with high accuracy based on the law of epipolar geometry can be performed.

  In the present invention, when the processes shown in (2-1) and (2-2) are executed, the voting process is performed as in the conventional PSA method. However, in the present invention, the area to be voted for is limited to be very narrow compared to the conventional PSA method. Therefore, the voting process in the present invention has a much lower calculation load than the voting process in the conventional PSA method.

  In the present invention, the process shown in (3-2) is performed when the region corresponding to the epipolar line has a certain width with respect to the first straight line corresponding to the epipolar line. This is a process of accumulating the voting values weighted according to the positional relationship. Thereby, the possibility that the position of the first straight line is slightly deviated from the original position is taken into consideration, and corresponding point search that emphasizes the law of epipolar geometry can be performed.

  Further, in the present invention, for example, the calculation load can be further reduced by terminating the voting process when the termination condition as shown in (4-1) or (4-2) is satisfied. In the present invention, based on the law of epipolar geometry, only a part of the reference candidate area that is likely to be a corresponding point deviation coordinate or a position close thereto is identified as an overlapping area. In other words, in the voting process, the voting values are intensively integrated only in a part of the area that is likely to be the corresponding point deviation coordinates or a position close thereto. Therefore, in the present invention, high accuracy is ensured even if the corresponding point deviation coordinates are specified based on the distribution of the integrated value of the vote values at the stage where the vote process for relatively few frequency components is completed.

  On the other hand, in the present invention, the voting process similar to the conventional PSA method is performed by performing the processes shown in (5-1) to (5-3) or (6-1) to (6-3). Accordingly, the corresponding point deviation coordinates can be derived.

  In addition, since the epipolar line equivalent region is specified by the process shown in (7-1), it is not necessary to prepare a memory for storing information on the epipolar line equivalent region in advance. On the other hand, by specifying the epipolar line equivalent region by the process shown in (7-2), it is possible to shorten the time required to specify the epipolar line equivalent region.

  In addition, since the image processing apparatus according to the present invention includes the parallelization processing unit, the calculation load for specifying the overlapping region is reduced, and the calculation load for the image processing is further reduced.

1 is a schematic block diagram of an image processing apparatus according to an embodiment of the present invention. It is a flowchart showing the procedure of the corresponding point search process by the image processing apparatus which concerns on 1st Embodiment of this invention. It is a schematic diagram which shows the 1st example of the epipolar line equivalent area | region in voting space. It is a schematic diagram which shows the 2nd example of the epipolar line equivalent area | region in voting space. It is a schematic diagram which shows the 3rd example of the epipolar line equivalent area | region in voting space. It is a schematic diagram which shows the 1st example of the overlap area | region with which the 1st example of the epipolar line equivalent area | region in a voting space overlaps with one candidate straight line. It is a schematic diagram which shows the 2nd example of the overlap area | region with which the 2nd example of the epipolar line equivalent area | region in a voting space overlaps with one candidate straight line. It is a schematic diagram which shows the 3rd example of the overlap area | region with which the 3rd example of an epipolar line equivalent area | region in a voting space overlaps with one candidate straight line. It is a schematic diagram which shows the 1st example of the overlapping area where the 1st example of an epipolar line equivalent area | region in a voting space overlaps with a some candidate straight line. It is a schematic diagram which shows the 2nd example of the overlap area | region with which the 2nd example of an epipolar line equivalent area | region in a voting space and a several candidate straight line overlap. It is a schematic diagram which shows the 3rd example of the overlapping area where the 3rd example of an epipolar line equivalent area | region in a voting space overlaps with a some candidate straight line. It is a flowchart showing the procedure of the corresponding point search process by the image processing apparatus which concerns on 2nd Embodiment of this invention. It is a schematic diagram of a reference candidate region that is a range that can be taken by corresponding point deviation coordinates. It is a schematic diagram which shows the straight line equivalent to the epipolar line in a reference | standard candidate area | region. It is a schematic diagram which shows the state where the straight line equivalent to the epipolar line in a reference | standard candidate area | region and one candidate straight line cross | intersect. It is a schematic diagram which shows the state which the straight line equivalent to the epipolar line in a reference | standard candidate area | region and a some candidate straight line cross | intersect. It is a schematic diagram which shows the voting space in PSA method. It is a schematic diagram which shows an example of the voting space containing one candidate straight line. It is a schematic diagram which shows an example of the voting space containing a some candidate straight line.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. The following embodiments are examples embodying the present invention, and do not limit the technical scope of the present invention. In the following description, unless otherwise specified, the term “image” is used in the same meaning as image data.

  First, the configuration of the image processing apparatus 10 according to the embodiment of the present invention will be described with reference to the block diagram shown in FIG. The image processing apparatus 10 has a hardware configuration that is employed in both a first embodiment and a second embodiment to be described later. As shown in FIG. 1, the image processing apparatus 10 includes an image input interface 11, a frame memory 12, an image processor 13, a ROM (Read Only Memory) (14), and a data output interface 15.

  The image input interface 11 is a circuit board that inputs an image obtained by imaging from each of the two cameras 1 and 2 that capture a stereo image and records the image in the frame memory 12. The two cameras 1 and 2 are arranged so as to capture the same imaging target from different directions, and thus the two images obtained by the two cameras 1 and 2 constitute a stereo image. Hereinafter, an image obtained by the first camera 1 is referred to as a first image, and an image obtained by the second camera is referred to as a second image.

  The frame memory 12 is a high-speed memory in which images are recorded. An image recorded in the frame memory 12 by the image input interface 11 is read or processed by the image processor 13.

  The image processor 13 includes a processor such as a DSP (Digital Signal Processor) or an MPU (Micro Processor Unit). The image processor 13 executes a corresponding point search process based on a stereo image by executing an image processing program stored in advance in the ROM (11).

  The image processing program includes program modules such as an epipolar processing module 131, a Fourier transform module 132, a phase image processing module 133, an overlapping region processing module 134, and a corresponding point coordinate calculation calculation module 135. The image processor 13 functions as an aggregate of a plurality of hardware that executes processes defined by the program modules 131 to 135 by developing and executing the program modules 131 to 135 in the main memory.

  That is, the image processing processor 13 is an epipolar processing unit that performs processing related to epipolar geometry, a Fourier transform unit that performs processing related to Fourier transform, a phase image processing unit that performs calculations related to phase images, and an overlap that performs calculations related to identification of overlapping regions, which will be described later. It functions as an area processing unit and a corresponding point coordinate calculation unit that calculates corresponding point coordinates.

  The ROM (14) is a memory that stores an image processing program executed by the image processor 13 and various data referred to by the image processor 13.

  The data output interface 15 is a circuit board that outputs image processing result data delivered from the image processing processor 13 to an external device of the image processing apparatus 10.

<< First Embodiment >>
Next, the corresponding point search process executed by the image processing apparatus according to the first embodiment of the present invention will be described with reference to the flowchart shown in FIG. In the flowchart, S1 to S11 represent identification codes of processing procedures. As described above, the image processing apparatus according to the present embodiment has the hardware configuration of the image processing apparatus 10 shown in FIG. In the following description, for convenience, the image processing apparatus according to the present embodiment is referred to as a first image processing apparatus.

  Before the corresponding point search process shown in FIG. 2 is started, the image input interface 11 records each data of the first image and the second image constituting the stereo image in the frame memory 12 in advance. Further, the corresponding point search process is executed for each one or a plurality of attention points (coordinates of the attention points) in the first image designated in advance. The information on the point of interest (coordinates of the point of interest) may be stored in advance in the ROM (14) or input from an external device through a data input interface (not shown), for example.

<Step S1>
In the corresponding point search process by the image processing apparatus 10, the image processor 13 sets the voting space 100 as shown in FIG. 17, that is, an array of a plurality of voting areas 20 (S1). More specifically, the image processor 13 secures a memory area in the frame memory 12 in which data corresponding to each voting area 20 of the voting space 100 is arranged. The processing in step S1 is realized by the image processor 13 executing the epipolar processing module 131, for example.

  As described above, the voting space 100 is an area indicating a possible range of the corresponding point deviation coordinates (dx0, dy0). The corresponding point deviation coordinates (dx0, dy0) are the coordinates of the deviation of the coordinates (x20, y20) of the corresponding points in the second image with respect to the coordinates (x10, y10) of the target point in the first image. Accordingly, one of the coordinate axes of the voting space 100 is a seat axis for the deviation dx of each coordinate value in the X-axis direction in the second image with respect to the coordinate value x0 in the X-axis direction of the target point in the first image, and the other is This is a sitting axis of deviation dy of each coordinate value in the Y-axis direction in the second image with respect to the coordinate value y0 in the Y-axis direction of the target point in the first image. In FIG. 17, the coordinate axis of the coordinate value dx is denoted by ΔX, and the coordinate axis of the coordinate value dy is denoted by ΔY.

  The first image processing apparatus is applied to information on the range (upper and lower boundary values) of the voting space 100 for specifying the voting space 100 and the number of voting areas 20 (number of sections) in each of the two coordinate axes. It is given in advance according to the target. For example, the information may be stored in advance in the ROM (14) or may be input from an external device through a data input interface (not shown). The voting space is an example of the reference candidate area.

<Step S2>
Next, the image processor 13 sets an epipolar line equivalent region, which is a partial region in the voting space 100 (S2). The processing in step S2 is realized by the image processor 13 executing the epipolar processing module 131. The image processor 13 that executes the epipolar processing module 131 is an example of the epipolar line equivalent region specifying unit.

  The epipolar line equivalent region is a part of the voting space 100 along a straight line corresponding to the epipolar line corresponding to the attention point. In addition, a straight line corresponding to the epipolar line corresponding to the point of interest is a line from the coordinate system of the second image to the coordinate system of the voting space 100 with respect to the epipolar line in the second image that is uniquely determined according to the coordinates of the point of interest. It is a straight line obtained by performing coordinate system conversion. Hereinafter, this straight line is referred to as a straight line 31 corresponding to an epipolar line.

  Hereinafter, a specific example of a method for setting the epipolar line equivalent region along the straight line 31 equivalent to the epipolar line will be described with reference to FIGS. 3 to 5. FIGS. 3 to 5 are schematic views showing the first to third examples of the epipolar line equivalent region 21 in the voting space 100, respectively. In FIGS. 3 to 5, the epipolar line equivalent area 21 is shaded with sand.

  The first example shown in FIG. 3 is an example in which the image processor 13 specifies a set of voting areas 20 that overlap with a straight line 31 equivalent to an epipolar line as the epipolar line equivalent area 21 among a plurality of voting areas 20.

  In the second example shown in FIG. 4, the image processor 13 sets a set of voting areas 20 existing within a predetermined range with respect to a straight line 31 corresponding to an epipolar line among a plurality of voting areas 20. This is an example specified as the epipolar line equivalent region 21. In the example shown in FIG. 4, the voting area 20 that overlaps the straight line 31 corresponding to the epipolar line and the voting area 20 adjacent to the voting area 20 are specified as the epipolar line equivalent area 21.

  Further, the third example shown in FIG. 5 is an example in which the image processor 13 specifies a part that coincides with the straight line 31 corresponding to the epipolar line in the voting space as the epipolar line equivalent region 21.

  The position of the epipolar line in the second image corresponding to the point of interest in the first image can be derived experimentally, and is based on the characteristics of the two cameras 1 and 2 that capture a stereo image and their positional relationship It can also be derived by theoretical calculation. Since this method is well known as shown in Non-Patent Document 3 and the like, description thereof is omitted here. Also, coordinate conversion from the coordinate system of the second image to the coordinate system of the voting space 100 can be performed based on known information.

  Therefore, the straight line 31 corresponding to the epipolar line can also be derived experimentally, and can also be derived by theoretical calculation based on the characteristics of the two cameras 1 and 2 and their positional relationship. For example, the following two methods are conceivable as the method by which the image processor 13 identifies the region corresponding to the epipolar line.

  In the first method, a straight line 31 corresponding to an epipolar line in the voting space 100 is calculated from the coordinates (x01, y01) of the point of interest in the first image based on conversion information stored in advance in a storage unit such as a ROM (14). This is a method of calculating the position (primary expression) of, and specifying the epipolar line equivalent region 21 along the straight line 31 based on the calculated position of the straight line 31. An example of a method for specifying the epipolar line equivalent region 21 based on the position of the straight line 31 corresponding to the epipolar line is as described above. According to the first method, it is not necessary to prepare a memory for storing information on the epipolar line equivalent region 21 in advance.

  Note that the conversion information used in the first method is not only stored as data referred to by the image processor 13 during execution of the program, but also defined in the program executed by the image processor 13. It is done. This conversion information is a very small amount of information compared to information that directly specifies the epipolar line equivalent region 21.

  In the second method, the epipolar line equivalent region 21 is associated with the candidate coordinates (x01, y01) of the target point and is stored in advance in the storage unit such as the ROM (14) according to the information representing the epipolar line equivalent region 21. It is a method to specify. According to the second method, it is possible to reduce the time required to specify the epipolar line equivalent region 21.

<Step S3>
Further, the image processor 13 performs a well-known discrete Fourier transform process on each data of the first image and the second image, thereby each of the first image in the frequency domain and the second image in the frequency domain. Data is calculated (S3). That is, the image processor 13 calculates a first frequency image corresponding to the first image and a second frequency image corresponding to the second image by discrete Fourier transform processing. Hereinafter, each pixel value in the first image is I1 (x, y), each pixel value in the second image is I2 (x, y), the first frequency image is F1 (u, v), and the second frequency image is F2. It is described as (u, v). Here, (u, v) is the frequency component of the image, that is, the coordinates in the two-dimensional coordinate system of the frequency domain. The process of step S3 is realized by the image processor 13 executing the Fourier transform module 132.

<Step S4>
Further, the image processor 13 sets the selection order of the frequency component (u, v) (S4). The frequency component (u, v) is selected in the order of the frequency components (u, v) applied to the processing when the subsequent processing from step S5 to step S9 is executed for each frequency component (u, v). ) Order. It is also conceivable to select only one representative frequency component (u, v) from among a plurality of frequency components (u, v).

  The selection order of the frequency components (u, v) is, for example, a predetermined order or a large cross spectrum between the first frequency image F1 (u, v) and the second frequency image F2 (u, v). The order may be considered. The cross spectrum represents the level of correlation between the first frequency image F1 (u, v) and the second frequency image F2 (u, v) in each frequency component (u, v). Therefore, setting the selection order of the frequency components (u, v) based on the size of the cross spectrum is between the first frequency image F1 (u, v) and the second frequency image F2 (u, v). Are sequentially selected from the frequency components (u, v) that increase the correlation of. The processing in step S4 is realized by the image processor 13 executing the phase image processing module 133.

<Step S5>
Then, the image processor 13 sequentially selects the frequency components (u, v) according to the order set in step S4 (S5). The image processor 13 executes the following processing from step S6 to step S10 for the selected frequency component (u, v), thereby performing processing from step S6 to step S10 until a predetermined end condition is satisfied. repeat.

<Step S6>
For each selection of the frequency component (u, v), the image processor 13 calculates the phase difference Δθ (u, v) between the first frequency image and the second frequency image at the selected frequency component (u, v). Calculate (S6). The processing in steps S5 and S6 is realized by the image processor 13 executing the phase image processing module 133.

  For the amplitude component Cr (u, v) of the first frequency image F1 (u, v) and the second frequency image F2 (u, v), the frequency component (u, v) is applied to the equation (1). Is calculated by The expression (1) is an expression for calculating a cross spectrum. The phase difference Δθ (u, v) between the first frequency image F1 (u, v) and the second frequency image F2 (u, v) applies the frequency component (u, v) to equation (2). Is calculated by

  The processes in steps S5 and S6 are realized by the image processor 13 executing the phase image processing module 133. The image processor 13 that executes the phase image processing module 133 is an example of the phase difference calculation unit.

<Step S7>
Next, the image processor 13 selects the frequency selected in the equation (3) using the frequency component (u, v), the phase difference Δθ (u, v), and the coordinates (dx, dy) in the voting space as variables. By applying the phase difference Δθ (u, v) calculated based on the component (u, v) and its frequency component, the coefficient of the candidate straight line equation (primary equation) that is a straight line in the voting space 100 is calculated. (S7). In Equation (3), M and N are the sizes (number of pixels) of the stereo image in the X-axis direction and the Y-axis direction, respectively, and are known constants.

  As shown in Non-Patent Document 1, etc., this equation (3) is calculated between the parameters of the frequency component (u, v), the phase difference Δθ (u, v), and the coordinates (dx, dy) in the voting space 100. This is a relational expression that holds.

<Step S8>
Further, the image processor 13 identifies an area that overlaps with the candidate straight line based on the result of the process in step S7 in the epipolar line equivalent area 21 set in step S2 (S8). Hereinafter, this area is referred to as an overlapping area 22.

  The processing of step S7 and step S8 is realized by the image processor 13 executing the overlapping area processing module 134. The image processor 13 that executes the overlapping area processing module 134 is an example of the overlapping area specifying unit.

  Hereinafter, a specific example of the method for identifying the overlapping region 22 will be described with reference to FIGS. 6 to 8. FIGS. 6 to 8 are schematic diagrams showing the first to third examples of the overlapping region 22 specified in step S8, respectively. 6 to 8, the overlapping area 22 is shaded with hatching.

  In any of the examples from FIG. 6 to FIG. 8, the image processor 13 overlaps the voting area 20 including the part where the epipolar line equivalent area 21 and the candidate straight line 31 overlap among the plurality of voting areas 20 in the voting space 100. This is an example specified as the region 21.

  FIG. 6 shows that a set of voting areas 20 that overlap with one candidate straight line 32 is set as an overlapping area 22 in the set of voting areas 20 that form the epipolar line equivalent area 21 (FIG. 3) in the first example. It is a schematic diagram which shows the example (1st example).

  FIG. 7 shows that a set of voting areas 20 that overlap with one candidate straight line 32 is set as an overlapping area 22 in the set of voting areas 20 that form the epipolar line equivalent area 21 (FIG. 4) in the second example. It is a schematic diagram which shows the example (2nd example).

  In FIG. 8, the voting area 20 including the portion (intersection) where the epipolar line equivalent area 21 (FIG. 5) and the single candidate straight line 32 coincide with the straight line 31 equivalent to the epipolar line is set as the overlapping area 22. It is a schematic diagram which shows an example (3rd example).

<Step S9>
Further, every time an overlapping area 22 corresponding to one candidate straight line 32 is specified, the image processor 13 accumulates a predetermined vote value for each voting area 20 included in the overlapping area 22. (S9). That is, the image processor 13 adds a predetermined voting value to the data stored in the data area in the frame memory 12 corresponding to each of the voting areas 20 defined as the overlapping areas 22. Note that the initial value of each data area in the frame memory 12 corresponding to the voting space 100 is zero.

  The voting value may be the same value for each of all the voting areas 20 specified as the overlapping area 22. In addition, in the second example (FIG. 7) in which the epipolar line equivalent region 21 having a relatively wide width with respect to the straight line 31 corresponding to the epipolar line is set, it is conceivable to employ different voting values depending on the position. .

  More specifically, the image processor 13 is larger as the position relative to the straight line 31 corresponding to the epipolar line is closer to each of the voting areas 20 in the overlapping area 22 sequentially specified for each frequency component (u, v). It is conceivable to sequentially accumulate weighted voting values. For example, a voting area 20 that overlaps with a straight line 31 corresponding to an epipolar line may be used as a reference voting area, and a larger voting value may be assigned as it is closer to the reference voting area.

<Step S10>
The image processor 13 determines whether or not a predetermined end condition is satisfied each time the processing from step S5 to step S9 is executed (S10). The image processor 13 selects the frequency component (u, v) (S5) and the phase difference Δθ (u, v) for each selected frequency component (u, v) until the termination condition is satisfied. Calculation (S6), identification of overlapping region 22 corresponding to the phase difference Δθ (u, v) (S8), and integration of vote values for overlapping region 22 (S9) are sequentially executed.

  Hereinafter, with reference to FIG. 9 to FIG. 11, voting processing (S5 to S9) is performed for a plurality of frequency components (u, v), whereby the overlapping region 22 that overlaps each of the plurality of candidate straight lines 32 is specified. Explain the situation. 9 to 11 correspond to FIGS. 6 to 8 described above, respectively. 9 to 1, the overlapping area 22 is shaded or blackened.

  FIG. 9 shows a set of voting areas 20 that overlap with each of the plurality of candidate straight lines 32 among the set of voting areas 20 forming the epipolar line equivalent area 21 (FIG. 3) in the first example. It is a schematic diagram which shows the example (1st example).

  FIG. 10 shows that a set of voting areas 20 that overlap each of a plurality of candidate straight lines 32 is set as an overlapping area 22 in the set of voting areas 20 that form the epipolar line equivalent area 21 (FIG. 4) in the second example. It is a schematic diagram which shows the example (2nd example).

  In FIG. 11, the voting area 20 including a portion (intersection) where the straight line 31 corresponding to the epipolar line and the plurality of candidate straight lines 32 each overlap with the epipolar line corresponding area 21 (FIG. 5) in the third example is defined as the overlapping area 22. It is a schematic diagram which shows the example (3rd example) set.

  The end condition is that the integrated value of the voting value in each voting area 20 satisfies a predetermined condition. The condition is, for example, that the integrated values P1 and P2 of the vote values satisfy the following expression (4). In Equation (4), H is the number of all frequency components (u, v), h is the number of frequency components (u, v) for which voting processing (voting value accumulation) has been completed, and Umax Is the maximum voting value per voting process, and P1 and P2 are the maximum integrated values of voting values at the time when the voting process is performed for h frequency components (u, v) And the second largest value. Note that, when the voting value is not weighted according to the position of the voting area 20, Umax is a voting value added for each voting area 20 in one voting, and is normally “1”.

  Satisfying the expression (4) means that the voting area 20 whose voting value integrated value is P1 at the present time is finally determined to have the maximum voting value integrated value. Therefore, when the corresponding point deviation coordinates (dx0, dy0) are derived based on the position of the voting area 20 where the integrated value of the voting value finally becomes the maximum, it is effective to use the expression (4) as an end condition. .

  In addition, other conditions may be considered as the end condition based on the integrated value of the vote value in each vote area 20. For example, the maximum integrated value P1 of the voting value is equal to or greater than a predetermined threshold, the maximum integrated value P1 of the voting value, and the second largest value P2 of the integrated voting value It is conceivable that the difference (P1-P2) is greater than or equal to a predetermined threshold value, or that both conditions are satisfied.

  In addition, for the predetermined number of frequency components (u, v), the end condition is the calculation (6) of the phase difference Δθ (u, v) and the overlapping region corresponding to the phase difference Δθ (u, v). It is also conceivable that 22 identifications (S8) and voting value accumulation (S9) have been performed. It is also conceivable that the end condition is that the voting process for one representative frequency component (u, v) has been completed, or that the voting process has been completed for all frequency components (u, v). .

<Step S11>
When the end condition is satisfied, the image processor 13 determines the corresponding point deviation coordinates (dx0, dy0) according to the coordinates of the voting area 20 in which the voting values are integrated, based on the integrated value of the voting values in each voting area 20. ) Is calculated (S11). More specifically, the image processor 13 calculates the coordinates of the peak position of the integrated value of the vote value in the vote space 100 as the corresponding point deviation coordinates (dx0, dy0).

  The processing from step S9 to step S11 is realized by the image processor 13 executing the corresponding point coordinate calculation module 135. The image processor 13 that executes the corresponding point coordinate calculation module 135 is an example of the deviation coordinate deriving unit.

  For example, the image processor 13 calculates the coordinates of the center of the voting area 20 where the integrated value of the voting values when the end condition is satisfied is the corresponding point deviation coordinates (dx0, dy0). Alternatively, the image processor 13 calculates the coordinates of the barycentric position of the overlapping region 22 as the corresponding point deviation coordinates (dx0, dy0) using the integrated value of the vote values when the end condition is satisfied as a weighting coefficient.

  9 to 11 show a state in which the voting area 20 having the maximum integrated voting value is specified as the peak position 220 (black portion) of the integrated voting value. When the end condition is satisfied, if there is one voting area 20 on which the voting value is accumulated, the coordinates of the center of the voting area 20 are calculated as corresponding point deviation coordinates (dx0, dy0). The

  In the process of step S11, the image processor 13 outputs the information of the corresponding point deviation coordinates (dx0, dy0) calculated as described above to the external device through the data output interface 15. Alternatively, the image processor 13 determines the coordinates (x02, y02) of the corresponding point in the second image based on the coordinates (x01, y01) of the target point in the first image and the corresponding point deviation coordinates (dx0, dy0). The information of the coordinates (x02, y02) is calculated and output to the external device through the data output interface 15. The process described above is the corresponding point search process in the first image processing apparatus.

  According to the corresponding point search processing of the first image processing apparatus, in the voting space 100 corresponding to the range that the corresponding point deviation coordinates (dx0, dy0) can take, the epipolar line equivalent region 21 along the straight line 31 equivalent to the epipolar line, Corresponding point deviation coordinates (dx0, dy0) are derived based on the coordinates of the overlapping area 22 where the candidate straight line 32 determined according to the frequency component (u, v) overlaps. Therefore, the search range of corresponding points, that is, the range of the voting area 20 that is the target of voting value accumulation, is limited to a part of the area (epipolar line equivalent area 21) along the straight line 31 equivalent to the epipolar line. Therefore, even when the corresponding point search (identification of the overlapping region 22) is executed with high spatial resolution, that is, when the number of sections of the voting region 20 is large, the calculation load of the corresponding point search process is kept relatively low. It is done. Therefore, according to the first image processing apparatus, it is possible to calculate the deviation of the coordinate of the corresponding point from the coordinate of the target point with high accuracy and a relatively light calculation load.

  Further, as in the first example shown in FIGS. 3, 6, and 9, only the set of voting areas 20 that overlap with the straight line 31 corresponding to the epipolar line is specified as the epipolar line equivalent area 21, thereby The line equivalent region 21 is more narrowly limited, and the effect of reducing the calculation load is further enhanced. This method is effective when the accuracy (reliability) of the position of the straight line 31 corresponding to the epipolar line is particularly high. The same applies to the case where the portion corresponding to the epipolar line equivalent line 31 in the voting space 100 is specified as the epipolar line equivalent region 21 as in the third example shown in FIGS. 5, 8, and 11. Has the effect of.

  On the other hand, the position of the straight line 31 corresponding to the epipolar line may be slightly deviated from the original position due to various errors. In that case, as in the second example shown in FIGS. 4, 7, and 10, a set of voting areas 20 existing within a predetermined range with respect to the straight line 31 corresponding to the epipolar line is equivalent to the epipolar line. It is effective to specify the area 21. By specifying the epipolar line equivalent region 21 having a relatively wide width in this way, the possibility of positional deviation of the straight line 31 is taken into consideration, and the corresponding point search with high accuracy based on the law of epipolar geometry is performed. It becomes possible. In particular, in step S9, by adding up the weighted voting values according to the positional relationship with the straight line 31, the possibility of positional deviation of the straight line 31 is taken into account, and corresponding points that place greater importance on the law of epipolar geometry Search is possible.

  Further, the calculation load can be further reduced by ending the voting process when the end condition is satisfied (S10). In the first image processing apparatus, based on the epipolar geometry law, only a part of the region that is likely to be the corresponding point deviation coordinate (dx0, dy0) or a position close to it in the voting space 100, It is specified as the overlapping area 22. In other words, in the voting process, the voting value is intensively integrated only in a part of the voting area 20 that is likely to be the corresponding point deviation coordinate (dx0, dy0) or a position close thereto. Therefore, in the first image processing apparatus, the corresponding point deviation (dx0, dy0) is specified based on the distribution of the integrated values of the voting values at the stage where the voting processing for relatively few frequency components (u, v) is completed. Even if done, high accuracy is ensured.

<< Second Embodiment >>
Next, the corresponding point search process executed by the image processing apparatus according to the second embodiment of the present invention will be described with reference to the flowchart shown in FIG. In the flowchart, S21 to S31 represent process procedure identification codes. As described above, the image processing apparatus according to the present embodiment has the hardware configuration of the image processing apparatus 10 shown in FIG. In the following description, for convenience, the image processing apparatus according to the present embodiment is referred to as a second image processing apparatus.

  The corresponding point search process of the second image processing apparatus is different from the corresponding point search process of the first image processing apparatus only in some processes. Hereinafter, the corresponding point search process of the second image processing apparatus will be mainly described in terms of differences from the corresponding point search process of the first image processing apparatus.

<Step S21>
In the corresponding point search process by the image processing apparatus 2, the image processor 13 sets a reference candidate region 101 as shown in FIG. 13 (S21). The reference candidate area 101 is an area indicating a possible range of the corresponding point deviation coordinates (dx0, dy0). More specifically, the image processor 13 sets information specifying the range of the reference candidate region 101 in the main memory. The processing in step S1 is realized by the image processor 13 executing the epipolar processing module 131, for example.

  Information for specifying the reference candidate region 101 is given in advance according to a target to which the second image processing apparatus is applied. For example, the information may be stored in advance in the ROM (14) or may be input from an external device through a data input interface (not shown).

<Step S22>
Next, the image processor 13 sets the epipolar line equivalent region 21 which is a partial region in the reference candidate region 101, similarly to the process of step S2 in the first image processing apparatus (S22). However, the epipolar line equivalent region 21 set in step S22 is a portion that coincides with the epipolar line equivalent straight line 31 in the reference candidate region 101, as in the third example shown in FIG.

  That is, the image processor 13 corresponds to the epipolar line in the reference candidate region 101 from the coordinates (x01, y01) of the target point in the first image based on the conversion information stored in advance in a storage unit such as the ROM (14). The position (linear expression) of the straight line 31 is calculated. The processing in step S22 is realized by the image processor 13 executing the epipolar processing module 131. The image processor 13 that executes the epipolar processing module 131 is an example of the epipolar line equivalent region specifying unit.

  FIG. 14 is a schematic diagram showing a straight line 31 corresponding to an epipolar line in the reference candidate region 101. A linear portion that coincides with the straight line 31 corresponding to the epipolar line is set as the epipolar line equivalent region 21. The process in step S22 is realized by the image processor 13 executing the epipolar processing module 131. The image processor 13 that executes the epipolar processing module 131 is an example of the epipolar line equivalent region specifying unit.

<Step S23>
Similarly to the process of step S3 in the first image processing apparatus, the image processor 13 converts the first frequency image, which is the first image in the frequency domain, to F1 (u, v) and the second frequency domain in the frequency domain. Each data of F2 (u, v) is calculated from the second frequency image which is an image (S23).

<Step S24>
Further, the image processor 13 sets the selection order of the frequency components (u, v) in the same manner as the process of step S4 in the first image processing apparatus (S24). It is also conceivable to select only one representative frequency component (u, v) from among a plurality of frequency components (u, v).

<Step S25>
Then, the image processor 13 sequentially selects the frequency components (u, v) according to the order set in step S24 (S25). The image processor 13 performs the processing from step S26 to step S30 until a predetermined end condition is satisfied by executing the following processing from step S26 to step S30 for the selected frequency component (u, v). repeat.

<Step S26>
Each time the frequency component (u, v) is selected, the image processor 13 calculates the phase difference Δθ (u, v) in the same manner as in step S6 in the first image processing apparatus (S26). The processing in steps S25 and S26 is realized by the image processor 13 executing the phase image processing module 133. The image processor 13 that executes the phase image processing module 133 is an example of the phase difference calculation unit.

<Step S27>
Next, the image processor 13 obtains the frequency component (u, v), the phase difference Δθ (u, v), and the coordinates (dx, dy) in the voting space in the same manner as in step S7 in the first image processing apparatus. By applying the selected frequency component (u, v) and the phase difference Δθ (u, v) calculated based on the frequency component to the equation (3) as a variable, a straight line in the reference candidate region 101 is obtained. The coefficient of the equation (primary equation) of the candidate straight line 32 is calculated (S27).

<Step S28>
Further, the image processor 13 calculates the coordinates of the intersection of the epipolar line equivalent region 21 set in step S22, that is, the straight line 31 equivalent to the epipolar line, and the candidate straight line 32 based on the result of the process in step S27 (S28). ). This intersection corresponds to the overlapping region 22 in the first embodiment. In other words, in step S <b> 28, the image processor 13 specifies the intersection of the straight line 31 corresponding to the epipolar line and the candidate straight line 32 as the overlapping region 22.

  The processing in step S27 and step S28 is realized by the image processor 13 executing the overlapping area processing module 134. The image processor 13 that executes the overlapping area processing module 134 is an example of the overlapping area specifying unit.

  FIG. 15 is a schematic diagram showing a situation in which the intersection of the epipolar line equivalent region 21 and one candidate straight line 32 that coincides with the straight line 31 equivalent to the epipolar line is set as the overlapping region 22.

<Step S29>
Further, every time an overlapping region 22 (intersection) corresponding to one candidate straight line 32 is specified, the image processor 13 records the coordinates of the intersection in the main memory (S29).

<Step S30>
The image processor 13 determines whether or not a predetermined end condition is satisfied, similar to the process of step S29 by the first image processing apparatus, every time the processes of step S25 to step S29 are executed. (S30). Then, the image processor 13 selects the frequency component (u, v) (S25) and the phase difference Δθ (u, v) for each selected frequency component (u, v) until the termination condition is satisfied. Calculation (S26) and identification and recording (S28, S29) of the overlapping area 22 (intersection) corresponding to the phase difference Δθ (u, v) are sequentially executed.

  The termination condition is, for example, that a predetermined number or more of overlapping regions 22 (intersection points) are concentrated and exist within a predetermined length range on the straight line 31 corresponding to the epipolar line.

  In addition, for a predetermined number of frequency components (u, v), the end condition is the calculation (26) of the phase difference Δθ (u, v) and the overlapping region corresponding to the phase difference Δθ (u, v). It is also conceivable that 22 (intersection) is identified (S28). Further, the processing of steps S26 to S29 is completed for one representative frequency component (u, v), or the processing of steps S26 to S29 is completed for all frequency components (u, v). An end condition may be considered.

  FIG. 16 is a schematic diagram showing a situation where the intersections between the epipolar line equivalent region 21 and each of the plurality of candidate straight lines 32 that coincide with the straight line 31 equivalent to the epipolar line are set as the overlapping region 22.

<Step S31>
When the end condition is satisfied, the image processor 13 calculates coordinates obtained by integrating the coordinates of the plurality of overlapping areas 22 (intersection points) in the reference candidate area 101 as corresponding point deviation coordinates (dx0, dy0). For example, it is conceivable to calculate the coordinates of the barycentric positions of a plurality of overlapping regions 22 (intersection points) as corresponding point deviation coordinates (dx0, dy0). In addition, when the process of step S26-S29 is performed only about one representative frequency component (u, v), the intersection of the straight line 31 equivalent to an epipolar line and one candidate straight line 32 is a corresponding point. Calculated as deviation coordinates (dx0, dy0).

  In step S31, the image processor 13 outputs the corresponding point deviation coordinate (dx0, dy0) information calculated as described above to the external device via the data output interface 15. Alternatively, the image processor 13 determines the coordinates (x02, y02) of the corresponding point in the second image based on the coordinates (x01, y01) of the target point in the first image and the corresponding point deviation coordinates (dx0, dy0). The information of the coordinates (x02, y02) is calculated and output to the external device through the data output interface 15. The process described above is the corresponding point search process in the second image processing apparatus.

  According to the corresponding point search processing of the second image processing device, as in the first image processing device, it is possible to calculate the deviation of the coordinates of the corresponding point with respect to the coordinates of the target point with high accuracy and a relatively light calculation load. It becomes. In particular, since the corresponding point search process of the second image processing apparatus does not require the voting process performed in the first image processing apparatus, the corresponding point deviation coordinates (dx0, dy0) can be calculated with a lighter calculation load.

<< Third Embodiment >>
In the embodiment described above, one of the coordinate axes (ΔX axis or ΔY axis) in the coordinate system of the voting space 100 or the reference candidate region 101 and the straight line 31 corresponding to the epipolar line are parallel to each other. It is conceivable to set the reference candidate region 101.

  That is, the image processor 13 executes a parallelizing process for making one coordinate axis (ΔX axis or ΔY axis) in the coordinate system of the voting space 100 or the reference candidate region 101 parallel to the straight line 31 corresponding to the epipolar line. Conceivable. In this case, the image processor 13 is realized by executing a program module for parallelization processing (not shown). The image processor 13 that executes the parallel processing program module is an example of the parallel processing unit.

  For example, before the corresponding point search processing shown in FIG. 2 or FIG. 12 is performed, the parallelization processing is performed by the angle formed by the epipolar line and one coordinate axis (X axis or Y axis) with respect to the second image. This is realized by applying a rotation process.

  By performing the parallelization process, the information specifying the straight line 31 corresponding to the epipolar line in the voting space 100 or the reference candidate region 101 is only the information on the ΔY-axis intercept or the information on the ΔX-axis intercept. Thereby, the calculation load required for setting the epipolar line equivalent region 21 and specifying the overlapping region 22 is further reduced.

DESCRIPTION OF SYMBOLS 1, 2 Camera 10 Image processing apparatus 11 Image input interface 12 Frame memory 13 Image processor 14 ROM
15 Data Output Interface 20 Voting Area 21 Epipolar Line Equivalent Area 22 Overlapping Area or Intersection 23 Specific Voting Area 131 Epipolar Processing Module 132 Fourier Transform Module 133 Phase Image Processing Module 134 Overlapping Area Processing Module 135 Corresponding Point Coordinate Calculation Module S1-S31 Processing Procedure identification code

Claims (15)

Based on the first phase image and the second phase image, which are phase images in the spatial frequency domain corresponding to each of the first image and the second image constituting the stereo image, the attention point designated in the first image An image processing device that executes corresponding point search processing for deriving coordinates of corresponding points in the corresponding second image,
Epipolar line corresponding to the target point in a reference candidate region indicating a possible range of the corresponding point deviation coordinate that is a coordinate of the deviation of the coordinate of the corresponding point in the second image with respect to the coordinate of the target point in the first image An epipolar line equivalent region specifying part for specifying a part of the epipolar line equivalent region along the first straight line corresponding to
A phase difference calculator that calculates a phase difference according to a frequency component between the first phase image and the second phase image;
In the region corresponding to the epipolar line, an overlapping region that overlaps a second straight line in the reference candidate region obtained by applying a frequency component and the phase difference corresponding to the frequency component to a predetermined formula is specified. An overlapping area specifying part;
A deviation coordinate deriving unit for deriving the corresponding point deviation coordinates according to the coordinates of the overlapping region;
An image processing apparatus comprising:   The overlapping region specifying unit includes the overlapping unit region including a portion where the epipolar line equivalent region and the second straight line overlap each other among a plurality of unit regions that are partial regions in which the reference candidate region is discretized. The image processing apparatus according to claim 1, wherein the image processing apparatus is specified as a region.   The image processing apparatus according to claim 2, wherein the epipolar line equivalent region specifying unit specifies, as the epipolar line equivalent region, a set of the unit regions that overlap the first straight line among the plurality of unit regions.   The epipolar line equivalent region specifying unit specifies, as the epipolar line equivalent region, a set of unit regions existing within a predetermined range with respect to the first straight line among the plurality of unit regions. An image processing apparatus according to 1.   The image processing apparatus according to claim 2, wherein the epipolar line equivalent region specifying unit specifies a portion that coincides with the first straight line in the reference candidate region as the epipolar line equivalent region. The phase difference calculation unit and the overlap region specifying unit sequentially execute calculation of the phase difference and specification of the overlap region corresponding to the phase difference for each frequency component until a predetermined end condition is satisfied,
The deviation coordinate deriving unit sequentially accumulates voting values for each of the unit regions in the overlapping region sequentially specified for each of a plurality of frequency components by the overlapping region specifying unit, and calculates the voting values in the reference candidate region. 6. The image processing apparatus according to claim 2, wherein coordinates of a peak position of an integrated value are specified as the corresponding point deviation coordinates. The phase difference calculation unit and the overlap region specifying unit sequentially execute calculation of the phase difference and specification of the overlap region corresponding to the phase difference for each frequency component until a predetermined end condition is satisfied,
The deviation coordinate deriving unit is weighted according to the positional relationship with the first straight line for each of the unit regions in the overlapping region sequentially specified for each of a plurality of frequency components by the overlapping region specifying unit. The image processing apparatus according to claim 4, wherein the vote values are sequentially accumulated, and the coordinates of the peak position of the accumulated value of the vote values in the reference candidate region are specified as the corresponding point deviation coordinates.   The image processing apparatus according to claim 6, wherein the end condition is that an integrated value of the vote values in each of the unit areas satisfies a predetermined condition.   8. The image processing according to claim 6, wherein the termination condition is that the phase difference is calculated and the overlapping region corresponding to the phase difference is specified for a predetermined number of frequency components. apparatus. The epipolar line equivalent region specifying unit specifies a portion coinciding with the first straight line in the reference candidate region as the epipolar line equivalent region,
The overlapping area specifying unit specifies an intersection between the first straight line and the second straight line as the overlapping area,
The image processing apparatus according to claim 1, wherein the deviation coordinate deriving unit derives, as the corresponding point deviation coordinates, a coordinate obtained by integrating the coordinates of the plurality of intersections in the reference candidate region. The epipolar line equivalent region specifying unit specifies a portion coinciding with the first straight line in the reference candidate region as the epipolar line equivalent region,
The overlapping area specifying unit specifies an intersection between the first straight line and the second straight line as the overlapping area,
The image processing apparatus according to claim 1, wherein the deviation coordinate deriving unit derives the coordinates of one intersection point in the reference candidate region as the corresponding point deviation coordinates.   The epipolar line equivalent region specifying unit calculates the position of the first straight line in the reference candidate region from the coordinates of the attention point in the first image based on conversion information stored in advance in a storage unit. The image processing apparatus according to claim 1, wherein the region corresponding to the epipolar line is specified based on the position of the first straight line.   The epipolar line equivalent region specifying unit specifies the epipolar line equivalent region according to information representing the epipolar line equivalent region stored in advance in a storage unit corresponding to the coordinate candidate of the target point. The image processing apparatus according to any one of 11.   The parallel processing part which sets the said reference candidate area | region so that the one coordinate axis in the coordinate system of the said reference candidate area | region and the said 1st straight line may become parallel is provided. Image processing device. Based on the first phase image and the second phase image, which are phase images in the spatial frequency domain corresponding to each of the first image and the second image constituting the stereo image, the attention point designated in the first image An image processing program for causing a computer to execute corresponding point search processing for deriving coordinates of corresponding points in the corresponding second image,
Epipolar line corresponding to the target point in a reference candidate region indicating a possible range of the corresponding point deviation coordinate that is a coordinate of the deviation of the coordinate of the corresponding point in the second image with respect to the coordinate of the target point in the first image Epipolar line equivalent region special processing for identifying a part of the epipolar line equivalent region along the first straight line corresponding to
A phase difference calculation process for calculating a phase difference according to a frequency component between the first phase image and the second phase image;
In the region corresponding to the epipolar line, an overlapping region that overlaps a second straight line in the reference candidate region obtained by applying a frequency component and the phase difference corresponding to the frequency component to a predetermined formula is specified. Overlapping area special processing,
Deviation coordinate derivation processing for deriving the corresponding point deviation coordinates according to the coordinates of the overlapping region;
An image processing apparatus program for causing a computer to execute.

Images