JP5476928B2 - Corresponding point search device, corresponding point searching method, and program - Google Patents

Description

  The present invention relates to a corresponding point search technique for searching for a corresponding point between two or more images.

  Conventionally, there is known a corresponding point search technique in which pixels are associated with each other between two images obtained by photographing the same subject from different viewpoints with a stereo camera or the like. In such a corresponding point search technique, an area-based corresponding point search process is generally performed using a two-dimensional area correlation between two images.

  In the area-based corresponding point search process, the following process is performed. One image is set as a reference, a window having a reference point as a center of gravity is set for the one image, and a window of the same size is set for the other image. Based on the correlation between the image areas related to the two windows, the corresponding point of the other image corresponding to the reference point is obtained. Such a process is performed while all pixels of one image are sequentially set as reference points.

  However, in the area-based corresponding point search process, corresponding points are searched according to the similarity between the image areas in the two windows. Therefore, the corresponding points are searched for an image area with low contrast. Is very difficult. Examples of the image region having a low contrast include an image region in which a surface of a white wall or an automobile body painted in one color is captured.

  Therefore, it is determined whether or not the image area has a low contrast (low contrast area) according to the presence or absence of an edge in the window, and if it is a low contrast area, a technique that is excluded from the corresponding point search processing target. It has been proposed (for example, Patent Document 1).

  In addition, the reliability related to the result of association between the two images is calculated for each reference point, and a reference point having a low reliability is obtained by using other pixels around the reference point as reference points. A technique has been proposed in which corresponding points are determined by an interpolation process using the obtained association results (for example, Patent Document 2).

JP-A-7-120255 Japanese Patent Laid-Open No. 2001-148012

  However, in the technique of the above-described Patent Document 1, a region that is excluded from the corresponding point search processing target is lacking in the result of association between two images. For this reason, for example, three-dimensional image data generated from the result of the association becomes difficult to handle. Such a problem becomes conspicuous when there are many regions excluded from the corresponding point search processing target.

  Moreover, in the technique of the above-mentioned Patent Document 2, the initial correspondence result obtained by calculation is not used for the reference point with low reliability. For this reason, the time required for the calculation to obtain the result of association not used is wasted.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a technique capable of searching for corresponding points with two or more images at a high speed and with as few missing points as possible.

In order to solve the above-described problem, a corresponding point search device according to a first aspect is a corresponding point search device that searches for corresponding points of a second image for each pixel of a first image. to generate a first low resolution image having a relationship with a reduced resolution for a generation means for generating a second low-resolution image having a relationship with a reduced resolution to the second image, the first First search means for setting a first reference point for the low-resolution image and performing a first search process for searching for the first corresponding point corresponding to the first reference point in the second low-resolution image; Is provided. Further, the corresponding point search device sets a second reference point corresponding to the first reference point for the first image, and uses the first search result by the first search process to set the second reference point. A second search means for performing a second search process for searching for a second corresponding point corresponding to the second reference point in two images, an area including the second reference point in the first image, and Calculating means for calculating an evaluation value related to a change in pixel value for an evaluation area corresponding to at least one of areas including a point corresponding to the second reference point in the second image; Further, the corresponding point search device prohibits execution of the second search process according to a comparison result between the evaluation value and the threshold, and prohibits execution of the second search process by the prohibition means. And determining the relationship between the second reference point and the second corresponding point based on the first search result, and if the prohibition means does not prohibit the execution of the second search process, Determination means for determining a relationship between the second reference point and the second corresponding point based on a second search result by the second search process .

  The corresponding point search device according to the second aspect is the corresponding point search device according to the first aspect, wherein the prohibiting means executes the second search process when the evaluation value is less than the threshold value. Is prohibited.

  The corresponding point search device according to the third aspect is the corresponding point search device according to the first or second aspect, wherein the evaluation value includes an evaluation value related to a contrast of a pixel value in the evaluation region.

  The corresponding point search device according to a fourth aspect is the corresponding point search device according to the third aspect, wherein the calculation means calculates a difference between a pixel value and a reference value related to each pixel included in the evaluation region. The sum is calculated as the evaluation value.

  The corresponding point search device according to the fifth aspect is the corresponding point search device according to the fourth aspect, wherein the calculation means calculates an average value of pixel values related to the evaluation region as the reference value.

  The corresponding point search device according to a sixth aspect is the corresponding point search device according to the third aspect, wherein the calculation means performs a process of extracting a contour of a subject for the evaluation region, and the evaluation The number of pixels occupied by the contour in the region is calculated as the evaluation value.

  The corresponding point search device according to a seventh aspect is the corresponding point search device according to the first or second aspect, wherein the evaluation value includes an evaluation value related to a distribution of pixel value patterns in the evaluation region. .

  The corresponding point search device according to an eighth aspect is the corresponding point search device according to the seventh aspect, wherein the calculation means performs a Fourier transform on the evaluation region, thereby obtaining a distribution of frequency components. The evaluation value is calculated based on the distribution of the frequency components.

  A corresponding point search device according to a ninth aspect is the corresponding point search device according to any one of the first to eighth aspects, wherein the determining means performs the second search process by the prohibiting means. When prohibited, an error range corresponding to the resolution of the first low-resolution image is added to determine the relationship between the second reference point and the second corresponding point.

  The corresponding point search device according to a tenth aspect is the corresponding point search device according to any one of the first to eighth aspects, wherein the determining means executes the second search process by the prohibiting means. When prohibited, the position of the second reference point with respect to the first image and the second image based on the first search result and an error range corresponding to the resolution of the first low-resolution image. The relationship between the second reference point and the second corresponding point is determined so that the shift amount between the second corresponding point and the position of the second corresponding point becomes the largest within the range of the error.

  The corresponding point search device according to an eleventh aspect is the corresponding point search device according to any one of the first to eighth aspects, wherein the determining means causes the prohibition means to execute the second search process. When prohibited, the position of the second reference point with respect to the first image and the second image based on the first search result and an error range corresponding to the resolution of the first low-resolution image. The relationship between the second reference point and the second corresponding point is determined so that the amount of deviation between the second corresponding point and the position of the second corresponding point becomes the smallest within the range of the error.

  The corresponding point search device according to a twelfth aspect is the corresponding point search device according to any one of the first to eleventh aspects, wherein at least one of the first and second search processes is performed as follows: Includes processing using phase-only correlation.

  A corresponding point search device according to a thirteenth aspect is the corresponding point search device according to any one of the first to twelfth aspects, wherein the determining means performs the second search process by the prohibiting means. If not prohibited, a relationship between the second reference point and the second corresponding point is determined according to the second search processing result, and further, for each pixel included in the first image, A relationship between the second reference point and the second corresponding point is determined based on one search processing result, or a relationship between the second reference point and the second corresponding point based on the second search processing result To decide.

  A corresponding point search device according to a fourteenth aspect is the corresponding point search device according to the thirteenth aspect, from one pixel of the first image to a peripheral pixel located around the one pixel. Regarding the halfway point, the relationship between the second reference point related to the one pixel obtained by the determining means and the second corresponding point, the second reference point related to the surrounding pixels obtained by the determining means, and the second reference point. Interpolation calculation means for deriving corresponding points in the second image by interpolation using the relationship between the two corresponding points is further provided.

The corresponding point search method according to the fifteenth aspect is a corresponding point search method for searching corresponding points of the second image for each pixel of the first image , wherein the resolution is reduced with respect to the first image. first the first to generate a low-resolution image, a generation step of generating a second low-resolution image having a relationship with a reduced resolution to the second image, the first low-resolution images in the 1 A first search step for setting a reference point and performing a first search process for searching for a first corresponding point corresponding to the first reference point for the second low-resolution image. Further, the corresponding point search method includes a region including a second reference point corresponding to the first reference point in the first image, and a point corresponding to the second reference point in the second image. For the evaluation region corresponding to at least one of the regions including, the calculation step for calculating the evaluation value related to the variation of the pixel value, and the first search result by the first search step is used to calculate the second image. A prohibiting step of prohibiting execution of a second search process for searching for a second corresponding point corresponding to the second reference point as a target according to a comparison result between the evaluation value and a threshold value. Further, in the corresponding point search method, when the execution of the second search process is prohibited by the prohibiting step, the relationship between the second reference point and the second corresponding point is based on the first search result. And when the execution of the second search process is not prohibited by the prohibiting step, the second search process is executed, and the second reference point is determined based on the second search result of the second search process. And a determination step of determining a relationship between the second corresponding points .

  The program according to the sixteenth aspect is executed by a control unit included in the corresponding point search apparatus, so that the corresponding point search apparatus is used as the corresponding point search apparatus according to any one of the first to fourteenth aspects. It is a program that makes it work.

  The corresponding point search device according to any one of the first to fourteenth aspects prohibits the execution of a more detailed corresponding point search process based on the evaluation value related to the change in the pixel value, and has a relatively low resolution. Based on the results of the search processing for a plurality of images, the relationship between the reference points and the corresponding points between the plurality of images is determined. Search is possible.

  With respect to an image region in which the corresponding point search device according to any of the third to sixth aspects predicts that the variation of the pixel value is small from the contrast in the image and that the appropriate corresponding point search process cannot be performed. Since execution of a more detailed corresponding point search process is prohibited, the search for corresponding points related to two or more images is speeded up.

  An image predicted by the corresponding point search device according to any of the seventh and eighth aspects to have a small variation in pixel value from the distribution of pixel value patterns and to be unable to perform an appropriate corresponding point search process. For a region, since execution of a more detailed corresponding point search process is prohibited, the search for corresponding points related to two or more images is speeded up.

  According to the corresponding point search device according to the ninth aspect, it is possible to obtain the result of the corresponding point search in consideration of the error range corresponding to the search accuracy.

  According to the corresponding point search device according to the tenth aspect, it is possible to apply to a use in which it is preferable that the parallax between two or more images is recognized as much as possible.

  According to the corresponding point search device according to the eleventh aspect, it is possible to apply to a use in which it is preferable that the parallax between two or more images is recognized as small as possible.

  According to the corresponding point search device according to the thirteenth aspect, it is possible to search for corresponding points in the second image at high speed and with no omission as much as possible for each pixel of the first image.

  With the corresponding point search device according to the fourteenth aspect, it is possible to improve the resolution related to the corresponding point search result.

  According to any of the corresponding point search method according to the fifteenth aspect and the program according to the sixteenth aspect, the execution of a more detailed corresponding point search process is prohibited on the basis of the evaluation value related to the fluctuation of the pixel value. Since the relationship between the reference points and the corresponding points between the plurality of images is determined based on the result of the search processing for a plurality of images with extremely low resolution, the lack of two or more images is as fast as possible It is possible to search for corresponding points that have no error.

It is a figure which shows schematic structure of the information processing system which concerns on one Embodiment and a modification. It is a block diagram which shows the principal part structure of the information processing system which concerns on one Embodiment and a modification. It is a figure which shows the functional structure implement | achieved in the control part which concerns on one Embodiment. It is a figure which illustrates a standard image and a reference image. It is a schematic diagram which illustrates the 1st reduction standard image and the 1st reduction reference image. It is a schematic diagram which illustrates the 2nd reduction standard image and the 2nd reduction reference image. It is a schematic diagram which illustrates the 3rd reduction standard image and the 3rd reduction reference image. It is a figure which shows typically the setting aspect of the search reference point with respect to a reference | standard image. It is a schematic diagram which shows the example of a setting of the reference point and process target point which concern on the 3rd layer. It is a schematic diagram which shows the example of a setting of the reference | standard area | region and comparison area | region which concerns on the 3rd layer. It is a schematic diagram which shows the example of a setting of the evaluation area | region and reference | standard area | region which concern on the 2nd layer. It is a schematic diagram which shows the example of a setting of the evaluation area | region and reference | standard area | region which concern on the 1st layer. It is a schematic diagram which shows the example of a setting of the evaluation area | region and reference area | region with respect to a 1st image. It is a schematic diagram which shows the example of a setting of the comparison area | region which concerns on the 2nd layer. It is a schematic diagram which shows the example of a setting of the comparison area | region which concerns on the 1st layer. It is a schematic diagram which shows the example of a setting of the comparison area with respect to a reference image. It is a figure for demonstrating concretely the search process of the corresponding point using POC method. It is a figure which illustrates distribution of the POC value which shows the correlation with a reference | standard area | region and a comparison area. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on one Embodiment. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on one Embodiment. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on one Embodiment. It is a figure which shows the functional structure implement | achieved in the control part which concerns on a modification. It is a figure for demonstrating an example of the interpolation calculation method which concerns on a modification. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on a modification. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on a modification.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

<(1) Configuration of information processing system>
FIG. 1 is a diagram showing a schematic configuration of an information processing system 1A according to an embodiment of the present invention, and FIG. 2 is a block diagram showing a main configuration of the information processing system 1A.

  The information processing system 1A includes a two-lens stereo camera 2 and an information processing apparatus 3A connected to the stereo camera 2 so as to be able to transmit data.

  The two-lens stereo camera 2 is provided with two imaging systems 21 and 22 each having an imaging device. The imaging systems 21 and 22 are spaced apart along a predetermined direction, and are configured to capture the subject OB in front of the camera from different viewpoints at the same timing. The two images captured at the same timing by the imaging systems 21 and 22 are so-called stereo images, and are transmitted to the information processing apparatus 3A via the data line CB.

  Of the two images constituting the stereo image, an image acquired by imaging of the imaging system 21 is appropriately referred to as a “first captured image” G1, and an image acquired by imaging of the imaging system 22 is appropriately referred to as “first image”. This is referred to as “2 captured images” G2. Accordingly, the first and second captured images G1 and G2 in which the same subject OB is captured by the two-lens stereo camera 2 are acquired.

  And in this embodiment, in order to avoid complication of explanation, the coordinate of the pixel which caught a certain part of a subject in the 1st picked-up image G1 acquired by image pick-up system 21, and the reference picture acquired by image pick-up system 22 In the following description, it is assumed that the imaging system 21 and the imaging system 22 are installed such that the coordinates of the pixel that captures the same part of the subject in G2 are the same in the Y direction and different in the X direction.

  The information processing apparatus 3A is configured by, for example, a personal computer (personal computer), and includes an operation unit 31 including a mouse and a keyboard, a display unit 32 configured by, for example, a liquid crystal display, and an interface for receiving data from the stereo camera 2 ( I / F) unit 33. The information processing apparatus 3A includes a storage unit 34, an input / output unit 35, and a control unit 36A.

  The storage unit 34 is composed of, for example, a hard disk or the like, and stores a program PGa for performing a corresponding point search operation described later.

  The input / output unit 35 includes, for example, a disk drive, receives the storage medium 9 such as an optical disk, and exchanges data with the control unit 36A.

  The control unit 36A includes a CPU 36a that functions as a processor and a memory 36b that temporarily stores information, and comprehensively controls each unit of the information processing apparatus 3A. In the control unit 36A, various functions, information processing, and the like are realized by reading and executing the program PGa in the storage unit 34.

  The program data stored in the storage medium 9 can be stored in the memory 36b of the control unit 36A via the input / output unit 35. Thereby, the stored program can be reflected in the operation of the information processing apparatus 3A.

  Further, the control unit 36A captures the same part of the subject between the two images (specifically, the first and second captured images G1 and G2) constituting the stereo image acquired by the stereo camera 2. A series of operations for searching for points (corresponding point search operations) are performed. This corresponding point search operation will be described later. Further, the control unit 36A calculates the three-dimensional position of the subject OB based on the principle of triangulation using information on corresponding points between images.

  The display unit 32 visually outputs a three-dimensional image of the subject OB based on the three-dimensional position of the subject OB calculated by the control unit 36A.

  In the present embodiment, in order to simplify the explanation, the aberration of the stereo camera 2 is well corrected, and the imaging systems 21 and 22 are set to be substantially parallel (preferably completely parallel). That is, the optical axes of the imaging systems 21 and 22 are set to be substantially parallel (preferably completely parallel), and the subjects captured by the first and second captured images G1 and G2 are the first and second captured images G1 and G1, respectively. It has substantially the same angular relationship (preferably completely the same angular relationship) with respect to the outer edge of G2. Further, when the actual configuration of the stereo camera 2 is not in such a condition, the stereo camera 2 may be converted into a stereo image captured under the same condition by image processing.

<(2) Corresponding point search operation>
In the corresponding point search operation according to the present embodiment, first, a set of images in which the resolution is reduced in a plurality of stages is generated on the basis of the set of the first captured image G1 and the second captured image G2 constituting the stereo image. The Next, for each pixel of the first captured image G1, a search for corresponding points is performed by capturing the same portion of the subject in order from a set of images with low resolution. At this time, the search result of the corresponding point for the set of images whose resolution is one step lower is used for the search for the corresponding point in the next step for the set of images whose resolution is one step higher. The However, before searching for the corresponding point in the next stage, it is evaluated whether or not the image is suitable for searching for the corresponding point so that the search for the corresponding corresponding point is not performed. Thus, the search for the corresponding point in the next stage is prohibited, and the corresponding point is determined based on the search result of the corresponding point in the previous stage.

  Hereinafter, a functional configuration for realizing such a corresponding point search operation, a basic principle related to the corresponding point search process, and an operation flow related to the corresponding point search operation will be sequentially described.

<(2-1) Functional configuration related to corresponding point search operation>
FIG. 3 is a diagram illustrating a functional configuration realized in the control unit 36A when the corresponding point search operation is executed. In the control unit 36A, for example, a functional configuration as shown in FIG. 3 is realized by reading and executing the program PGa stored in the storage unit 34.

  As shown in FIG. 3, the control unit 36A includes an image acquisition unit 361, a resolution conversion unit 362, a search reference point setting unit 363, a lower layer image designating unit 364, an initial parallax setting unit 365, an initial window as functional configurations. Setting unit 366, lower layer corresponding point searching unit 367, next layer image specifying unit 368, evaluation area setting unit 369, evaluation value calculating unit 370, search prohibiting unit 371, next layer parallax setting unit 372, next layer window setting unit 373, next It has a layer corresponding point search unit 374, a remaining layer determination unit 375, a corresponding point determination unit 376, a storage processing unit 377, and a remaining pixel determination unit 378.

  Hereinafter, each functional structure implement | achieved in 36 A of control parts is demonstrated.

<(2-1-1) Image acquisition unit>
The image acquisition unit 361 acquires a first captured image G1 and a second captured image G2 obtained by capturing the same subject from the storage unit 34 at the same timing.

  FIG. 4 is a schematic view illustrating the forms of the first and second captured images G1 and G2. In the first and second captured images G1, G2, a large number of pixels are arranged in a matrix. Specifically, a first predetermined number (here, 480) of pixels are arranged in the vertical direction (Y direction) and a second predetermined number (here, 640) in the horizontal direction (X direction). Pixels are arranged.

  In the first and second captured images G1 and G2, the upper left position is the origin, and the horizontal position of each pixel constituting the first and second captured images G1 and G2 is indicated by the X coordinate. The position in the direction is indicated by the Y coordinate. That is, in the first and second captured images G1 and G2, the position of each pixel is indicated by XY coordinates (x, y). For example, if the pixel is shifted rightward (X direction) by one pixel, the value of the X coordinate is 1. The value of the Y coordinate increases by one when it is shifted by one pixel in the downward direction (Y direction).

  In the corresponding point search operation, the first captured image G1 becomes a reference image (reference image), and the second captured image G2 becomes an image (reference image). Hereinafter, the first captured image G1 is referred to as a standard image G1, and the second captured image G2 is referred to as a reference image G2. Strictly speaking, the image acquisition unit 361 acquires data indicating the standard image G1 and the reference image G2. However, in this specification, the data indicating the standard image G1 and the standard image G1 itself are collectively referred to as the standard image G1. The data indicating the reference image G2 and the reference image G2 itself are collectively referred to as a reference image G2.

<(2-1-2) Resolution converter>
The resolution converting unit 362 generates an image (low-resolution standard image) having a relationship in which the resolution is gradually reduced with respect to the standard image G1 based on the standard image G1, and also uses the reference image G2 as a base. An image (low-resolution reference image) having a relationship in which the resolution is gradually reduced with respect to the reference image G2 is generated.

  Specifically, in the resolution conversion unit 362, every other vertical line among a plurality of columns of vertical pixels (vertical lines) and a plurality of rows of horizontal pixels (horizontal lines) constituting the reference image G1. And every other horizontal line is thinned out, and a reduced low-resolution reference image (first reduced reference image) G11 (FIG. 5) is generated. In the first reduced reference image G11, 320 pixels in the vertical direction and 240 pixels in the horizontal direction are arranged in a matrix.

  Then, the resolution conversion unit 362 thins out every other vertical line and every other horizontal line in the first reduced reference image G11, thereby further reducing the reduced-resolution reference image (second image). A reduced reference image) G12 (FIG. 6) is generated. In the second reduced reference image G12, 160 pixels in the vertical direction and 120 pixels in the horizontal direction are arranged in a matrix.

  Further, in the resolution conversion unit 362, every other vertical line and every other horizontal line in the second reduced reference image G12 are thinned, so that the reduced-resolution reference image (third image) is further reduced. A reduced reference image) G13 (FIG. 7) is generated. In the third reduced reference image G13, 80 pixels in the vertical direction and 60 pixels in the horizontal direction are arranged in a matrix.

  In addition, the resolution conversion unit 362 reduces the size by reducing every other vertical line and every other horizontal line among a plurality of vertical lines and a plurality of horizontal lines constituting the reference image G2. A low-resolution reference image (first reduced reference image) G21 (FIG. 5) is generated. In the first reduced reference image G21, 320 pixels in the vertical direction and 240 pixels in the horizontal direction are arranged in a matrix.

  Then, the resolution conversion unit 362 thins out every other vertical line and every other horizontal line in the first reduced reference image G21, thereby further reducing the reduced-resolution reference image (second image). A reduced reference image) G22 (FIG. 6) is generated. In the second reduced reference image G22, 160 pixels in the vertical direction and 120 pixels in the horizontal direction are arranged in a matrix.

  Further, the resolution conversion unit 362 thins out every other vertical pixel line and every other horizontal line in the second reduced reference image G22, thereby further reducing the reduced resolution reference image (first image). 3 reduced reference image) G23 (FIG. 7) is generated. In the third reduced reference image G23, 80 pixels in the vertical direction and 60 pixels in the horizontal direction are arranged in a matrix.

  In this way, the resolution conversion unit 362 uses the set of the standard image G1 and the reference image G2 as a reference, and the set of the first reduced reference image G11 and the first reduced reference image G21 whose one-step resolution is reduced is 1 It is generated as a set of low resolution images in the layer. In addition, a set of the second reduced reference image G12 and the second reduced reference image G22 in which the two-step resolution is reduced is formed as a set of second-layer low-resolution images. Further, a set of the third reduced reference image G13 and the third reduced reference image G23 in which the three-step resolution is reduced is formed as a set of low resolution images in the third layer (lowermost layer).

<(2-1-3) Search reference point setting section>
The search reference point setting unit 363 sets a reference point (reference point) Sp1 (FIG. 8) as a reference for the corresponding point search for the reference image G1.

  Specifically, the search reference point setting unit 363 first sets the upper left pixel of the reference image G1 as the reference point Sp1. Each time a corresponding point is detected from the reference image G2 with respect to the reference point Sp1, a new reference point Sp1 is set. Specifically, as shown by the arrow in FIG. 8, the reference point Sp1 is a predetermined pixel (here, from left to right (X direction) sequentially from the upper direction (+ Y direction) with respect to the reference image G1. In this case, the corresponding points corresponding to the respective reference points Sp1 are detected on the reference image G2 while being sequentially set while being shifted by one pixel.

  In other words, for the reference image G1, the reference points Sp1 are set in time sequence along the horizontal lines parallel to the X direction arranged from the -Y direction to the + Y direction. When the setting of the reference point Sp1 along one horizontal line parallel to the X direction is completed, the reference point Sp1 is time-sequentially along the next horizontal line parallel to the X direction located in the + Y direction for one pixel. Is set.

<(2-1-4) Lower layer image designation part>
The lower layer image designating unit 364 designates the lowest resolution image set (specifically, the third reduced standard image G13 and the third reduced reference image G23) as the set of images to be processed.

<(2-1-5) Initial parallax setting unit>
The initial parallax setting unit 365 temporarily sets a predetermined initial parallax serving as a reference for searching for corresponding points between the third reduced reference image G13 and the third reduced reference image G23. For example, the predetermined initial parallax is set to zero or the like.

  Then, as shown in FIG. 9, the initial parallax setting unit 365 sets a point (reference point) Sp13 corresponding to the reference point Sp1 set by the search reference point setting unit 363 for the third reduced reference image G13. At the same time, a point (processing target point) Pp23 to be subjected to arithmetic processing is set for the third reduced reference image G23. Here, the amount of deviation of the X coordinate between the position of the reference point Sp13 in the third reduced reference image G13 and the position of the processing point Pp23 in the third reduced reference image G23 corresponds to a predetermined initial parallax.

<(2-1-6) Initial window setting section>
As shown in FIG. 10, the initial window setting unit 366 sets a window (reference region) W13 including the reference point Sp13 as a center with respect to the third reduced reference image G13, and also sets the third reduced reference image G23. On the other hand, a window (comparison area) W23 including the processing target point Pp23 as a center is set. Note that the reference region W13 and the comparison region W23 are square regions of the same size, and for example, 17 pixels are arranged in the vertical direction and the horizontal direction.

<(2-1-7) Lower layer corresponding point search part>
The lower layer corresponding point search unit 367 searches for the corresponding point corresponding to the reference point Sp13 in the third reduced reference image G23. The lower layer corresponding point search unit 367 searches for corresponding points based on the correlation between the reference region W13 and the comparison region W23 obtained by using a phase-only correlation method (POC method) described later.

  That is, in the present embodiment, the initial parallax setting unit 365, the initial window setting unit 366, and the lower layer corresponding point search unit 367 set the reference point Sp13 for the third reduced reference image G13 and the third reduction reference. A process (initial search process) for searching for a corresponding point corresponding to the reference point Sp13 is performed on the image G23.

  Information indicating the search result by the lower layer corresponding point search unit 367 is appropriately output to the next layer parallax setting unit 372 or the corresponding point determination unit 376.

<(2-1-8) Next layer image designation part>
The next-layer image designating unit 368 designates, as a set of images to be processed, a set of images that are one step higher in resolution than the set of low-resolution images that have been subjected to the corresponding point search process immediately before.

  For example, if a corresponding point search process has been performed immediately before between the third reduced reference image G13 and the third reduced reference image G23, the next layer image specifying unit 368 causes the second reduced reference image G12 and the second reduced reference image G12 to be A reduced reference image G22 is designated. If the corresponding point search process has been performed between the second reduced reference image G12 and the second reduced reference image G22 immediately before, the next layer image specifying unit 368 and the first reduced reference image G11 and the first reduced reference image G22 A reduced reference image G21 is designated. Furthermore, if a corresponding point search process has been performed between the first reduced reference image G11 and the first reduced reference image G21 immediately before, the next layer image specifying unit 368 determines that the reference image G1 and the reference image G2 are It is specified.

<(2-1-9) Evaluation area setting section>
The evaluation area setting unit 369 sets an evaluation area for the image related to the reference image G1 among the set of images specified by the next layer image specifying unit 368.

  For example, if the second reduced standard image G12 and the second reduced reference image G23 related to the second layer are specified by the next layer image specifying unit 368, as shown in FIG. A reference point Sp12 corresponding to the reference point Sp1 in the second reduced reference image G12 is set. Then, an evaluation area Ae12 including the reference point Sp12 as a center point is set for the second reduced reference image G12.

  If the first reduced standard image G11 and the first reduced reference image G21 related to the first layer are specified by the next layer image specifying unit 368, as shown in FIG. 12, the evaluation region setting unit 369 A reference point Sp11 corresponding to the reference point Sp1 in the first reduced reference image G11 is set. Then, an evaluation area Ae11 that includes the reference point Sp11 as a center point is set for the first reduced reference image G11.

  Further, if the standard image G1 and the reference image G2 are designated by the next layer image designation unit 368, as shown in FIG. 13, the evaluation area setting unit 369 centers the standard point Sp1 in the standard image G1. An evaluation area Ae1 included as a point is set for the reference image G1.

  Note that the evaluation areas Ae1, Ae11, and Ae12 are square areas of the same size, and are configured, for example, by arranging 17 pixels in the vertical and horizontal directions.

<(2-1-10) Evaluation value calculator>
The evaluation value calculation unit 370 calculates an evaluation value related to the contrast of the pixel value for the image region included in the evaluation region set by the evaluation region setting unit 369.

  Specifically, as shown in the following Equation 1, the average value of the pixel values related to each pixel constituting the image area included in the evaluation area is calculated as a reference value, and the image area included in the evaluation area is A difference between a pixel value and a reference value relating to each pixel constituting the pixel is calculated, and a sum of the differences is calculated as an evaluation value CTR.

  In the above equation 1, the number of pixels in the X direction and the number of pixels in the Y direction constituting the evaluation area are indicated by w (here, 17), and the upper left pixel of the evaluation area is the origin, and the evaluation area The X coordinate of each pixel included in is represented by a variable i and the Y coordinate is represented by a variable j.

  Information indicating the evaluation value CTR calculated by the evaluation value calculation unit 370 is output to the search prohibition unit 371.

<(2-1-11) Search prohibited part>
The search prohibition unit 371 prohibits execution of the search process by the next layer corresponding point search unit 374 according to the comparison result between the evaluation value CTR calculated by the evaluation value calculation unit 370 and the threshold value Th. Specifically, when the evaluation value CTR is less than the threshold Th, execution of the search process by the next layer corresponding point search unit 374 is prohibited.

  When execution of the search process by the next layer corresponding point searching unit 374 is prohibited, a signal indicating that is output to the corresponding point determining unit 376. On the other hand, when the execution of the search process is not prohibited, a signal indicating that is output to the next layer parallax setting unit 372.

<(2-1-12) Next layer parallax setting unit>
The next-layer parallax setting unit 372 temporarily sets a parallax (reference parallax) serving as a reference for searching for corresponding points for the set of images specified by the next-layer image specifying unit 368.

  Here, when the set of images specified by the next layer image specifying unit 368 is a set of the second reduced reference image G12 and the second reduced reference image G22, the search result by the lower layer corresponding point search unit 367 is displayed. A corresponding reference parallax is set.

  Specifically, as shown in FIG. 14, the next-layer parallax setting unit 372 sets a point (processing target point) Pp22 that is a target of arithmetic processing for the second reduced reference image G22. At this time, the shift amount of the X coordinate between the position of the reference point Sp12 (FIG. 11) in the second reduced reference image G12 and the position of the processing target point Pp22 in the second reduced reference image G22 corresponds to the reference parallax. .

  If the set of images specified by the next layer image specifying unit 368 is a set of the first reduced reference image G11 and the first reduced reference image G21, the immediately previous search by the next layer corresponding point searching unit 374 is performed. A reference parallax corresponding to the result is set. At this time, the immediately preceding search result is a search result of corresponding points between the second reduced reference image G12 and the second reduced reference image G22.

  Specifically, the next layer parallax setting unit 372 sets the processing target point Pp21 for the first reduced reference image G21 as illustrated in FIG. At this time, the amount of deviation of the X coordinate between the position of the reference point Sp11 (FIG. 12) in the first reduced reference image G11 and the position of the processing target point Pp21 in the first reduced reference image G21 corresponds to the reference parallax. .

  When the set of images specified by the next layer image specifying unit 368 is a set of the reference image G1 and the reference image G2, the reference parallax according to the immediately preceding search result by the next layer corresponding point search unit 374 is used. Is set. At this time, the immediately preceding search result is a search result of corresponding points between the first reduced reference image G11 and the first reduced reference image G21.

  Specifically, the next layer parallax setting unit 372 sets the processing target point Pp2 for the reference image G2, as shown in FIG. At this time, the deviation amount of the X coordinate between the position of the reference point Sp1 (FIG. 13) in the reference image G1 and the position of the processing target point Pp2 in the reference image G2 corresponds to the reference parallax.

<(2-1-13) Next layer window setting section>
The next layer window setting unit 373 sets a window for the set of images specified by the next layer image specifying unit 368.

  Here, when the set of images specified by the next layer image specifying unit 368 is a set of the second reduced reference image G12 and the second reduced reference image G22, as shown in FIG. 11 and FIG. A window (reference area) W12 and a window (comparison area) W22 are set.

  Specifically, a reference region W12 that includes the reference point Sp12 as the center with respect to the second reduced reference image G12 is set, and a comparison region that includes the processing target point Pp22 as the center with respect to the second reduced reference image G22. W22 is set. In the present embodiment, the reference area W12 and the evaluation area Ae12 are the same area.

  When the set of images specified by the next layer image specifying unit 368 is a set of the first reduced reference image G11 and the first reduced reference image G21, as shown in FIG. 12 and FIG. A window (reference area) W11 and a window (comparison area) W21 are set.

  Specifically, a reference region W11 that includes the reference point Sp11 as the center with respect to the first reduced reference image G11 is set, and a comparison region that includes the processing target point Pp21 as the center with respect to the first reduced reference image G21. W21 is set. In the present embodiment, the reference area W11 and the evaluation area Ae11 are the same area.

  If the set of images specified by the next layer image specifying unit 368 is a set of the standard image G1 and the reference image G2, as shown in FIGS. 13 and 16, a window (reference region) W1 is used. And a window (comparison area) W2 is set.

  Specifically, a reference region W1 that includes the reference point Sp1 as the center is set for the reference image G1, and a comparison region W2 that includes the processing target point Pp2 as the center is set for the reference image G2. In the present embodiment, the reference area W1 and the evaluation area Ae1 are the same area.

  The reference areas W1, W11, and W12 and the comparison areas W2, W21, and W22 are square areas of the same size, and are configured by 17 pixels arranged in the vertical direction and the horizontal direction, for example.

<(2-1-14) Next layer corresponding point search unit>
The next layer corresponding point search unit 374 searches for a corresponding point for the set of images specified by the next layer image specifying unit 368. The next layer corresponding point search unit 374 employs a phase only correlation method (POC method).

  Here, when the set of images specified by the next layer image specifying unit 368 is a set of the second reduced reference image G12 and the second reduced reference image G22, the second reduced reference image G22 is targeted. Thus, a corresponding point corresponding to the reference point Sp12 is searched.

  When the set of images specified by the next layer image specifying unit 368 is a set of the first reduced reference image G11 and the first reduced reference image G21, the first reduced reference image G21 is targeted. A corresponding point corresponding to the reference point Sp11 is searched.

  Furthermore, when the set of images specified by the next layer image specifying unit 368 is a set of the standard image G1 and the reference image G2, the corresponding point corresponding to the reference point Sp1 is targeted for the reference image G2. Explored.

  That is, in the present embodiment, the reference point Sp12 is set with respect to the second reduced reference image G12 by the evaluation area setting unit 369, the next layer parallax setting unit 372, the next layer window setting unit 373, and the next layer corresponding point search unit 374. In addition to the setting, a search result by the lower layer corresponding point search unit 367 is used to search for a corresponding point corresponding to the reference point Sp12 in the second reduced reference image G22 (second layer search process). .

  In the present embodiment, the reference point Sp11 is set with respect to the first reduced reference image G11 by the evaluation area setting unit 369, the next layer parallax setting unit 372, the next layer window setting unit 373, and the next layer corresponding point search unit 374. A process of searching for a corresponding point corresponding to the reference point Sp11 in the first reduced reference image G21 by using the previous search result by the next layer corresponding point searching unit 374 while being set (first layer searching process) Is done.

  Furthermore, in this embodiment, the reference point Sp1 is set for the reference image G1 by the search reference point setting unit 363, the next layer parallax setting unit 372, the next layer window setting unit 373, and the next layer corresponding point search unit 374. In addition, a search process (final search process) for searching for a corresponding point corresponding to the reference point Sp1 is performed on the reference image G2 using the previous search result by the next layer corresponding point searching unit 374.

  Information indicating the search result by the next layer corresponding point searching unit 374 is appropriately output to the next layer parallax setting unit 372 or the corresponding point determining unit 376.

<(2-1-15) Remaining layer determination unit>
The remaining layer determination unit 375 determines whether or not there is an image set having a higher resolution than the set of images subjected to the previous search process by the next layer corresponding point search unit 374.

  If there is a set of images having a resolution higher than that of the previous search processing target, the remaining layer determination unit 375 outputs a signal indicating that to the next layer image designation unit 368. In response to the signal, the next-layer image designating unit 368 sets a set of images that are one step higher in resolution than the set of low-resolution images that have been subjected to search processing immediately before. Specified as

<(2-1-16) Corresponding point determination unit>
The corresponding point determination unit 376 determines a corresponding point corresponding to the reference point Sp1 set by the search reference point setting unit 363. That is, the relationship between the reference point Sp1 and the corresponding point is determined.

  Specifically, when the corresponding point determination unit 376 receives a signal indicating that execution of the search process is prohibited from the search prohibition unit 371, the corresponding point determination unit 376 immediately before the lower layer corresponding point search unit 367 or the next layer corresponding point search unit 374 The corresponding point corresponding to the reference point Sp1 is determined based on the search result. At this time, an error range in the search is appropriately added, and information indicating the relationship between the reference point Sp1 and the corresponding point is generated.

  More specifically, when the search prohibition unit 371 prohibits the execution of the search process by the next layer corresponding point search unit 374 based on the evaluation value CTR related to the evaluation region Ae12 of the second reduced reference image G12, Based on the search result by the corresponding point search unit 367, the corresponding point corresponding to the reference point Sp1 is determined.

  Specifically, if a corresponding point is detected from the third reduced reference image G23 immediately before by the lower layer corresponding point search unit 367, a plurality of pixels corresponding to the corresponding point are recognized from the reference image G2, and the plurality of pixels are detected. The position of the approximate center of the pixels is determined as a corresponding point (detected corresponding point).

  However, the resolution of the third reduced reference image G13 and the third reduced reference image G23 is 1/8 of the resolution of the reference image G1 and the reference image G2. Therefore, an error range (± 4 in this case) in the search according to the resolution of the third reduced reference image G13 and the third reduced reference image G23 is added to the information indicating the detected corresponding points. It becomes.

  Further, when the search prohibition unit 371 prohibits the execution of the search process by the next layer corresponding point search unit 374 based on the evaluation value CTR related to the evaluation area Ae11 of the first reduced reference image G11, the next layer corresponding point Based on the search result by the search unit 374, a corresponding point corresponding to the reference point Sp1 is determined.

  Specifically, if a corresponding point is detected from the second reduced reference image G22 immediately before by the next layer corresponding point search unit 374, a plurality of pixels corresponding to the corresponding point are recognized from the reference image G2, The position of the approximate center of the plurality of pixels is determined as a corresponding point (detected corresponding point).

  However, the resolution of the second reduced reference image G12 and the second reduced reference image G22 is ¼ of the resolution of the reference image G1 and the reference image G2. Therefore, an error range (± 2 in this case) in the search according to the resolution of the second reduced reference image G12 and the second reduced reference image G22 is added to the information indicating the detected corresponding points. It becomes.

  In addition, when the search prohibition unit 371 prohibits the execution of the search process by the next layer corresponding point search unit 374 based on the evaluation value CTR related to the evaluation area Ae1 of the reference image G1, the next layer corresponding point search unit 374 A corresponding point corresponding to the reference point Sp1 is determined based on the search result obtained by the above.

  Specifically, if a corresponding point is detected from the first reduced reference image G21 immediately before by the next layer corresponding point search unit 374, a plurality of pixels corresponding to the corresponding point are recognized from the reference image G2, The position of the approximate center of the plurality of pixels is determined as a corresponding point (detected corresponding point).

  However, the resolution of the first reduced standard image G11 and the first reduced reference image G21 is ½ of the resolution of the standard image G1 and the reference image G2. Therefore, an error range (± 1 in this case) in the search according to the resolution of the first reduced reference image G11 and the first reduced reference image G21 is added to the information indicating the detected corresponding points. It becomes.

  On the other hand, in the corresponding point determination unit 376, when a signal indicating that the execution of the search process is prohibited is not input from the search prohibition unit 371, based on the result of the final search process in the next layer corresponding point search unit 374, Corresponding points corresponding to the reference point Sp1 are determined. At this time, the result of the final search process is directly adopted as the relationship between the reference point Sp1 and the corresponding point.

<(2-1-17) Storage processing unit>
The storage processing unit 377 stores information indicating the relationship between the reference point Sp1 determined by the corresponding point determination unit 376 and the corresponding point in the storage unit 34.

<(2-1-18) Remaining pixel determination unit>
The remaining pixel determination unit 378 determines whether or not all pixels of the reference image G1 are set as the reference point Sp1 by the search reference point setting unit 363. Here, if all the pixels of the reference image G1 are not set as the reference point Sp1, a signal indicating that is output to the search reference point setting unit 363. On the other hand, if all the pixels of the standard image G1 have already been set as the standard point Sp1, the corresponding point search operation between the standard image G1 and the reference image G2 is terminated.

<(2-2) Basic principle related to search processing of corresponding points using POC method>
Here, the basic principle relating to the corresponding point search process using the POC method will be described.

  In the present embodiment, the corresponding point search process between the standard image G1 and the reference image G2, the corresponding point search process between the first reduced standard image G11 and the first reduced reference image G21, the second reduced standard image The corresponding point search process between G12 and the second reduced reference image G22 and the corresponding point search process between the third reduced reference image G13 and the third reduced reference image G23 are performed. In the search process, the same corresponding point search process using the POC method is performed.

  Therefore, in the following, the basic principle relating to the corresponding point search process using the POC method will be described, with the corresponding point search process between the reference image G1 and the reference image G2 as a representative example.

  In the standard image G1 and the reference image G2, a predetermined number (N) of pixels are arranged along the X direction to form a long side, and a predetermined number (M) along the Y direction different from the X direction. It is assumed that the short side is formed by arranging the pixels.

In the corresponding point search process, first, as shown in FIG. 13, a window (reference area) W1 including a reference point Sp1 designated on the reference image G1 as a central point is set on the reference image G1. On the other hand, as shown in FIG. 16, a window (comparison area) W2 having the same size as the window W1 is set as the reference image G2. In the reference area W1 and the comparison area W2, a large number of pixels are arranged in a matrix. Specifically, a predetermined number N ₁ of pixels are arranged along the X direction, and a predetermined number of N ₂ pixels are arranged along the Y direction. Next, a value indicating the correlation between the reference region W1 and the comparison region W2 (hereinafter referred to as “correlation value”) is calculated. Then, based on the correlation value (here, a POC value described later), a point (corresponding point) corresponding to the reference point Sp1 is detected from the reference image G2.

  FIG. 17 is a diagram for specifically explaining the corresponding point search processing using the POC method.

  In the corresponding point search process using the POC method, first, a window (reference area) W1 setting process T0a for the reference image G1 and a window (comparison area) W2 setting process T0b for the reference image G2 are performed. At this time, it is assumed that image areas corresponding to the reference area W1 and the comparison area W2 are expressed by the following equation (2).

Here, f (n ₁ , n ₂ ) in the above equation 2 indicates the reference region W1 on the reference image G1, and g (n ₁ , n ₂ ) in the above equation 2 indicates the comparison region on the reference image G2. W2 is shown. N ₁ and N ₂ are set as N ₁ = 2M ₁ +1 and N ₂ = 2M ₂ +1, for example.

  Next, two-dimensional Fourier transform processing T1a and T1b using an arithmetic expression expressed by the following equation 3 is performed on each image region corresponding to the reference region W1 and the comparison region W2 of the reference image G1 and the reference image G2. Is called.

Note that N ₁ and N ₂ are substituted for the subscript P of W in the above proviso of Equation ₃ , and ₁ and ₂ are substituted for the subscript s of k.

  For each image region subjected to such Fourier transform processing T1a and T1b, normalization processing T2a and T2b for removing the amplitude component of the image is obtained using an arithmetic expression expressed by the following equation (4). Each done.

  When the normalization processes T2a and T2b are completed, a synthesis process T3 using the arithmetic expression shown in the following Expression 5 is performed, and a two-dimensional inverse Fourier transform process T4 using the arithmetic expression shown in the Expression 6 is performed. Is called. Thereby, the correlation calculation between the images is performed, and the result (POC value) is output.

  Through the above processing, a calculation result (POC value) indicating the correlation between the reference area W1 and the comparison area W2 is obtained, and for example, a result (POC value) as shown in FIG. 18 is obtained.

In FIG. 18, the POC value at a location with high correlation in the window (N ₁ × N ₂ ) is large, and the position corresponding to the peak Jc of the POC value in the reference image W2 on the reference image G2 is shown. This corresponds to the corresponding point on the reference image G2 corresponding to the center point (reference point) Sp1 of the reference area W1 on the reference image G1. For this reason, by detecting the peak Jc of the POC value, the corresponding point on the reference image G2 corresponding to the reference point Sp1 on the reference image G1 is detected.

  According to the corresponding point search processing using the POC method, the amplitude component of the image is removed, and the correlation calculation is performed using only the phase component of the image. Is detected with high accuracy.

<(2-3) Operation flow related to corresponding point search operation>
19 to 21 are flowcharts showing the flow of the corresponding point search operation according to the present embodiment. This operation flow is realized by the control unit 36A reading and executing the program PGa. The operation flow is started in response to an instruction from the operation unit 31, for example, and proceeds to step ST1 in FIG.

  In step ST1, the image acquisition unit 361 acquires the standard image G1 and the reference image G2 from the storage unit 34.

  In step ST2, the resolution conversion unit 362 uses the reference image G1 as a base, and the first to third reduced reference images G11 to G13 in the first to third layers having a relationship in which the resolution is gradually reduced from the reference image G1. Are generated, and the first to third reduced reference images G21 to G23 of the first to third layers having a relationship in which the resolution is gradually reduced from the reference image G2 are generated based on the reference image G2. Is done.

  In step ST3, the reference point Sp1 is set for the reference image G1 by the search reference point setting unit 363. In step ST3, every time the process returns from step ST32 of FIG. 21, the search reference point setting unit 363 sets a pixel that has not yet been set as the reference point Sp1 in the reference image G1 as the reference point Sp1. The

  In step ST4, the lower layer image designating unit 364 designates the third reduced standard image G13 and the third reduced reference image G23 related to the lowest layer as a set of images to be processed.

  In step ST5, the initial parallax setting unit 365 temporarily sets a predetermined initial parallax between the third reduced reference image G13 and the third reduced reference image G23.

  Specifically, the initial parallax setting unit 365 sets a reference point Sp13 corresponding to the reference point Sp1 set in step ST3 for the third reduced reference image G13.

  In step ST6, the initial window setting unit 366 sets a window (reference region) W13 that includes the reference point Sp13 as a center for the third reduced reference image G13, and performs processing on the third reduced reference image G23. A window (comparison region) W23 including the target point Pp23 as a center is set.

  In step ST7, the lower layer corresponding point search unit 367 searches for the corresponding point corresponding to the reference point Sp13 in the third reduced reference image G23.

  As described above, in steps ST5 to ST7, the reference point Sp13 is set for the third reduced reference image G13, and the initial search for searching for the corresponding point corresponding to the reference point Sp13 in the third reduced reference image G23 is performed. Processing is performed.

  When the process of step ST7 is completed, the process proceeds to step ST11 of FIG.

  In step ST11, the next layer image designating unit 368 performs processing on an image set that is one step higher in resolution than the low-resolution image set that has been subjected to the corresponding point search process immediately before. Specified as a pair.

  For example, if the corresponding point search process has been performed immediately before between the third reduced reference image G13 and the third reduced reference image G23, the second reduced reference image G12 and the second reduced reference image G22 are designated. The

  If the corresponding point search process has been performed between the second reduced reference image G12 and the second reduced reference image G22 immediately before, the first reduced reference image G11 and the first reduced reference image G21 are designated. The

  Furthermore, if the corresponding point search process has been performed immediately before between the first reduced reference image G11 and the first reduced reference image G21, the reference image G1 and the reference image G2 are designated.

  In step ST12, the evaluation area setting unit 369 sets an evaluation area for the image related to the reference image G1 in the set of images specified in step ST11.

  For example, if the second reduced reference image G12 and the second reduced reference image G22 related to the second layer are designated in step ST11, the reference point Sp12 corresponding to the reference point Sp1 in the second reduced reference image G12 is set. Is set. Then, an evaluation area Ae12 including the reference point Sp12 as a center point is set for the second reduced reference image G12.

  Further, if the first reduced reference image G11 and the first reduced reference image G21 related to the first layer are designated in step ST11, the reference point Sp11 corresponding to the reference point Sp1 in the first reduced reference image G11 is set. Is set. Then, an evaluation area Ae11 including the reference point Sp11 as a center point is set for the first reduced reference image G11.

  Furthermore, if the standard image G1 and the reference image G2 are specified in step ST11, an evaluation area Ae1 including the standard point Sp1 in the standard image G1 as a central point is set for the standard image G1.

  In step ST13, the evaluation value calculation unit 370 calculates an evaluation value related to contrast for the image region included in the evaluation region set in step ST12. Here, the average value of the pixel values related to the image area included in the evaluation area is calculated as the reference value, and the sum of the differences between the pixel value related to each pixel constituting the image area included in the evaluation area and the reference value is evaluated. Calculated as value CTR.

  In step ST14, the search prohibition unit 371 determines whether or not the evaluation value CTR calculated in step ST13 is less than the threshold Th. Here, if evaluation value CTR is not less than threshold value Th, it will progress to step ST15, and if evaluation value CTR is less than threshold value Th, it will progress to step ST20.

  In step ST15, the next-layer parallax setting unit 372 temporarily sets a reference parallax as a reference for searching for corresponding points for the set of images specified in step ST11.

  For example, if a set of the second reduced reference image G12 and the second reduced reference image G22 is designated in step ST11, the processing target point Pp22 is set for the second reduced reference image G22.

  Further, if a set of the first reduced reference image G11 and the first reduced reference image G21 is specified in step ST11, the processing target point Pp21 is set for the first reduced reference image G21.

  Furthermore, if a set of the base image G1 and the reference image G2 is designated in step ST11, the processing target point Pp2 is set for the reference image G2.

  In step ST16, the next layer window setting unit 373 sets a window for each set of images specified in step ST11.

  For example, if a set of the second reduced reference image G12 and the second reduced reference image G22 is designated in step ST11, a window (reference area) W12 is set for the second reduced reference image G12, and the second A window (comparison area) W22 is set for the two reduced reference images G22.

  If a set of the first reduced reference image G11 and the first reduced reference image G21 is specified in step ST11, a window (reference area) W11 is set for the first reduced reference image G11, and the first A window (comparison area) W21 is set for one reduced reference image G21.

  Further, if a set of the standard image G1 and the reference image G2 is designated in step ST11, a window (standard region) W1 is set for the standard image G1, and a window (comparison region) is set for the reference image G2. ) W2 is set.

  In step ST17, the next layer corresponding point search unit 374 searches for a corresponding point for the set of images specified in step ST16.

  For example, if a set of the second reduced reference image G12 and the second reduced reference image G22 is specified in step ST11, the corresponding point corresponding to the reference point Sp12 for the second reduced reference image G22 is searched. Is called.

  If a set of the first reduced reference image G11 and the first reduced reference image G21 is designated in step ST11, the corresponding point corresponding to the reference point Sp11 for the first reduced reference image G21 is searched. Is called.

  Furthermore, if a set of the reference image G1 and the reference image G2 is designated in step ST11, a corresponding point corresponding to the reference point Sp1 for the reference image G2 is searched.

  In step ST18, the remaining layer determination unit 375 determines whether or not there is a set of images with higher resolution related to the upper layer than the set of images subjected to the search processing in the immediately preceding step ST17. Here, if a set of high-resolution images related to the upper layer exists, the process returns to step ST11. If a set of high-resolution images related to the upper layer does not exist, the process proceeds to step ST19. In addition, when progressing to step ST19, the search process in step ST17 immediately before becomes a final search process.

  In step ST19, the corresponding point determination unit 376 determines a corresponding point corresponding to the reference point Sp1 set in step ST3 based on the result of the final search process in step ST17, and the relationship between the reference point Sp1 and the corresponding point. Is generated. Here, the result of the final search process is directly adopted as the relationship between the reference point Sp1 and the corresponding point. And according to completion | finish of the process of this step ST19, it progresses to step ST31 of FIG.

  When the process of step ST19 is performed, the execution of the search process is not prohibited in step ST20 described later for the currently designated reference point Sp1.

  In step ST20, the search prohibition unit 371 prohibits execution of search processing by the next layer corresponding point search unit 374 for the set of images specified in the immediately preceding step ST11.

  Here, for example, if a set of the second reduced reference image G12 and the second reduced reference image G22 is designated in step ST11, the correspondence for the second reduced reference image G12 and the second reduced reference image G22 is targeted. Point search processing is prohibited.

  If a set of the first reduced reference image G11 and the first reduced reference image G21 is designated in step ST11, search for corresponding points for the first reduced reference image G11 and the first reduced reference image G21 is performed. Processing is prohibited.

  Furthermore, if a set of the standard image G1 and the reference image G2 is designated in step ST11, the corresponding point search process for the standard image G1 and the reference image G2 is prohibited.

  In step ST21, the corresponding point determination unit 376 determines a corresponding point corresponding to the reference point Sp1 set in step ST3.

  Here, for example, if a set of the second reduced reference image G12 and the second reduced reference image G22 is specified in the immediately preceding step ST11, the reference point Sp1 is based on the result of the corresponding point search process in step ST7. Corresponding points corresponding to are determined. At this time, information indicating the relationship between the reference point Sp1 to which the error range (± 4 in this case) in the search is added and the corresponding point is generated.

  If the set of the first reduced reference image G11 and the first reduced reference image G21 is designated in the immediately preceding step ST11, the second reduced reference image G12 and the second reduced reference image G22 in the immediately preceding step ST17 are specified. A corresponding point corresponding to the reference point Sp1 is determined based on the result of the corresponding corresponding point search process. At this time, information indicating the relationship between the reference point Sp1 to which the error range (± 2 in this case) in the search is added and the corresponding point is generated.

  Further, if a set of the standard image G1 and the reference image G2 is specified in the immediately preceding step ST11, the corresponding points for the first reduced reference image G11 and the first reduced reference image G21 in the immediately preceding step ST17 are set. Based on the result of the search process, a corresponding point corresponding to the reference point Sp1 is determined. At this time, information indicating the relationship between the reference point Sp1 to which the error range (± 1 in this case) in the search is added and the corresponding point is generated.

  When the process in step ST21 is completed, the process proceeds to step ST31 in FIG.

  In step ST31 of FIG. 21, the storage processing unit 377 stores the determination result in step ST19 or step ST21, that is, information indicating the relationship between the reference point Sp1 and the corresponding point in the storage unit 34.

  In step ST32, the remaining pixel determination unit 378 determines whether or not all the pixels of the reference image G1 have already been set as the reference point Sp1 in step ST3. Here, if all the pixels of the reference image G1 are not set as the reference point Sp1, the process returns to step ST3, and a pixel not yet set as the reference point Sp1 in the reference image G1 is set as the reference point Sp1. On the other hand, if all the pixels of the reference image G1 have already been set as the reference point Sp1, this operation flow ends.

  In this way, a plurality of pixels included in the standard image G1 are sequentially designated as the standard point Sp1, and between each of the third reduced standard image G13 and the third reduced reference image G23 with respect to each standard point Sp1. As a result of the corresponding point search process at, as a result of the corresponding point search process between the second reduced reference image G12 and the second reduced reference image G22, it is between the first reduced reference image G11 and the first reduced reference image G21. The relationship between the reference point Sp1 and the corresponding point is derived based on one of the result of the corresponding point search process in FIG. 3 and the result of the corresponding point search process between the reference image G1 and the reference image G2. .

  As described above, according to the information processing system 1A according to the embodiment of the present invention, the execution of the more detailed corresponding point search process is prohibited based on the evaluation value related to the contrast of the pixel value in the image, and the relative The relationship between the reference point Sp1 and the corresponding point between the reference image G1 and the reference image G2 is determined based on the result of the search process for a plurality of images with a particularly low resolution. For this reason, it is possible to search for corresponding points with two or more images at high speed and with as few missing points as possible.

  Since such processing is performed for each pixel of the standard image G1, the corresponding points of the reference image G2 are searched for at high speed and without omission as to each pixel of the standard image G1.

  In particular, more detailed corresponding point search processing is executed for an area where the variation of the pixel value is predicted from the contrast of the pixel value in the image and the appropriate corresponding point search processing cannot be performed. Is prohibited. As a result, useless computation is omitted, and the search for corresponding points related to two or more images is speeded up.

  In addition, when the relationship between the reference point Sp1 and the corresponding point between the reference image G1 and the reference image G2 is determined based on the result of search processing for a plurality of images with relatively low resolution, An error range corresponding to the search accuracy is added to the relationship between the reference point Sp1 and the corresponding point. As a result, a corresponding point search result that takes into account a range of errors according to the search accuracy is obtained.

<(3) Modification>
It should be noted that the present invention is not limited to the above-described embodiment, and various changes and improvements can be made without departing from the gist of the present invention.

  For example, in the above-described embodiment, each search process for corresponding points is performed using the POC method, but the present invention is not limited to this. For example, other methods such as a SAD (Sum of Absolute Differences) method may be adopted for at least one search process.

  In the above-described embodiment, the average value of the pixel values related to the evaluation region is calculated as the reference value in the evaluation value calculation unit 370, and the sum of the differences between the pixel value related to each pixel in the evaluation region and the reference value is calculated. Is calculated as the evaluation value CTR, but is not limited thereto.

  For example, a process of extracting an outline (edge) for an image area in the evaluation area is performed, and the total number of pixels (edge amount) occupied by the edge in the evaluation area is calculated as an evaluation value related to contrast. Also good.

  As the edge extraction method implemented here, various methods referred to as Sobel, Prewrite, etc. may be applied. Note that the sum of the number of pixels occupied by the edge (edge amount) may be obtained after the image (edge image) as a result of edge extraction is binarized based on the threshold value.

  In this way, the configuration for determining whether or not to perform the corresponding point search process for the next-stage resolution based on the evaluation value related to the contrast is the corresponding point search process that greatly affects the contrast, such as template matching. Is more suitable for configurations where

  Further, instead of calculating the evaluation value related to the contrast, the evaluation value related to the fluctuation of the pixel value may be calculated for the evaluation region by other methods.

  For example, a frequency component distribution (power spectrum) may be derived by performing Fourier transform on an image area within the evaluation area, and an evaluation value may be calculated based on the distribution. The evaluation value corresponds to an evaluation value related to the distribution of pixel value patterns in the evaluation region.

  Specifically, for example, the evaluation value may be calculated by the following method (I) or (II).

  (I) First, a Fourier transform is performed on an image area in the evaluation area, thereby obtaining a distribution (power spectrum) of frequency components related to the spatial frequency. Next, the sum of power (energy) in the power spectrum is calculated as an evaluation value.

  (II) First, a Fourier transform is performed on the image region in the evaluation region, thereby obtaining a distribution (power spectrum) of frequency components related to the spatial frequency. Next, the threshold value Th (ρ) set for each frequency is used as a reference, and the power (energy) in the power spectrum is binarized. Then, for the distribution of frequency components (binary distribution) obtained by binarizing the power spectrum, the sum of power (energy) is calculated as an evaluation value.

  In the method (II), for example, if an evaluation value that places more importance on the low-frequency region is obtained, an evaluation that suppresses the influence of so-called aliasing can be performed. In addition, in the corresponding point search process using the POC method, since the image in the window has a wide variety of frequency components, it is possible to perform a highly accurate search process. Therefore, the method (II) is used, It is preferable that a wide range of frequency components is emphasized and the evaluation value is calculated as appropriate.

  If such a configuration is adopted, for an image region in which it is predicted from the evaluation result related to the distribution of the pixel value pattern that the variation in the pixel value is small and an appropriate corresponding point search process cannot be performed, Execution of more detailed corresponding point search processing is prohibited. For this reason, similarly to the above-described embodiment, the search for corresponding points related to two or more images is accelerated.

  In the above-described embodiment, the reference point Sp1 and the corresponding point between the reference image G1 and the reference image G2 based on the result of the corresponding point search process for a plurality of relatively low resolution images. Is determined, an error range corresponding to the search accuracy is added to the relationship between the reference point Sp1 and the corresponding point. However, the present invention is not limited to this.

  For example, according to the use of the information obtained by the corresponding point search operation, based on the result of the corresponding point search processing for a plurality of images with relatively low resolution and the error range according to the resolution, The reference point Sp1 and the corresponding point are set such that the amount of deviation of the X coordinate between the position of the reference point Sp1 with respect to the reference image G1 and the position of the corresponding point with respect to the reference image G2 is the largest within the error range according to the resolution. The relationship may be determined.

  As an application in which such a configuration is suitable, a system that detects the approach of an obstacle using a stereo image obtained by a vehicle-mounted camera and prevents a rear-end collision can be considered. For such an application, it is preferable that parallax is detected to be larger because it is possible to detect the approach of an obstacle with strict eyes and maintain safety.

  Also, depending on the use of the information obtained by the corresponding point search operation, conversely, the result of the corresponding point search processing for a plurality of relatively low resolution images and the error range according to the resolution. Based on this, the reference point Sp1 is such that the amount of deviation of the X coordinate between the position of the reference point Sp1 with respect to the reference image G1 and the position of the corresponding point with respect to the reference image G2 is the smallest within the error range according to the resolution. And the relationship between corresponding points may be determined.

  As an application in which such a configuration is suitable, a system that detects and notifies a situation in which a person or the like is approaching a dangerous place using a stereo image obtained by a surveillance camera can be considered. For example, when a surveillance camera is provided near a dangerous place (pond, river, cliff, various facilities, etc.) toward the dangerous place, the closer the person or the like gets closer to the dangerous place, the more the stereo image A person etc. can be seen far away. Therefore, for such applications, it is preferable that the parallax is detected to be smaller because it is possible to strictly detect the approach to a dangerous place such as a person and to maintain safety. .

  In the above embodiment, the POC values are obtained discretely, but the present invention is not limited to this. For example, a corresponding point can be searched more finely by performing an interpolation calculation of POC values between adjacent pixels and estimating the position of the peak Jc with a sub-pixel size smaller than the size of one pixel. As a method for the interpolation calculation, a method for obtaining a parabolic function from the distribution of the POC values obtained discretely can be considered. According to such a configuration, the resolution in the corresponding point search result can be improved.

  However, from the viewpoint of suppressing an increase in the amount of calculation and speeding up the calculation, the reference point for each pixel is used rather than the search for the corresponding point at the subpixel size in the search process of the corresponding point between the images of each resolution. Only when the relationship between the point Sp1 and the corresponding point is determined, it is preferable that the corresponding point is detected with the size of the sub-pixel. For this reason, for example, there are various modes such as a mode in which a corresponding point is searched with a subpixel size only in the final search process, and a mode in which a corresponding point is searched with a subpixel size in a search process other than the final search process. Variations are possible.

  In the embodiment described above, corresponding pixels are detected in the reference image G2 for each pixel of the standard image G1, but the present invention is not limited to this. For example, in addition to associating the positions of pixels, a configuration in which corresponding points in the reference image G2 are detected with respect to the position of a portion (subpixel) between the pixels can be considered.

  In such a configuration, the corresponding point is detected with the subpixel size from the result of the final search process, or the corresponding point is detected with the subpixel size from the result of the search process other than the final search process. Various variations, such as an aspect, can be considered. According to such an aspect, the amount of information in the corresponding point search result, that is, the resolution can be improved.

  Here, a mode in which corresponding points are detected with the size of the sub-pixel from the result of the final search process will be described with a specific example.

  FIG. 22 is a diagram illustrating a functional configuration realized in the control unit 36B according to the present modification. In the information processing system 1B according to the present modification, the control unit 36A is changed to the control unit 36B to which an interpolation calculation unit 379B having a functional configuration is added, as compared with the information processing system 1A according to the one embodiment. In addition, the program PGa stored in the storage unit 34 is changed to the program PGb, and the information processing apparatus 3A is changed to the information processing apparatus 3B in accordance with the change. For this reason, the information processing system 1B according to the present modification has the same configuration in most part as the information processing system 1A according to the above-described embodiment. For this reason, about the same structure, the same code | symbol is attached | subjected and description is abbreviate | omitted and a different part is demonstrated below.

  The interpolation calculation unit 379B determines the relationship between the reference point Sp1 in pixel units obtained by the corresponding point determination unit 376 and the corresponding point, and pixels (peripheral pixels) located around the reference point Sp1 obtained by the corresponding point determination unit 376. An interpolation calculation is performed using the relationship between the reference point Sp1 and the corresponding point. As a result of this interpolation calculation, the interpolation calculation unit 379B is configured to perform a halfway point (intermediate point) from one pixel set as the reference point Sp1 in the reference image G1 to a peripheral pixel located around the one pixel. Corresponding points in the reference image G2 are derived.

  As the interpolation calculation, when the parallax determined for three consecutive pixels of the same Y coordinate in the reference image G1 shows a monotonous decrease or monotonous increase tendency, for example, between adjacent pixels It is only necessary to employ a calculation in which the parallax concerning the middle point is obtained by so-called interpolation. Further, when the parallax determined for three consecutive pixels of the same Y coordinate in the reference image G1 does not show a monotonous decrease or monotonous increase, various fittings or the like may be employed. .

  FIG. 23 is a diagram for explaining an example of the interpolation calculation method according to the present modification.

  Here, it is assumed that the parallax between the standard image G1 and the reference image G2 is obtained by the final search process for three consecutive points of the same Y coordinate in the standard image G1. Specifically, the parallax Va is obtained for a pixel whose X coordinate is Xa, the parallax Vb is obtained for a pixel whose X coordinate is Xb, and the parallax Vc is obtained for a pixel whose X coordinate is Xc. It shall be.

  Under such conditions, as shown in FIG. 23, three consecutive X coordinates and plots PLa to PLc relating to parallax are attached to a graph in which the horizontal axis indicates the X coordinate and the vertical axis indicates the parallax. . Next, a straight line (one-dot chain line) is drawn connecting the central plot PLb and a plot (plot PLa in FIG. 23) in which the parallax of the plots PLa and PLc on both sides differs greatly with respect to the plot PLb. Further, a straight line (one-dot chain line) having the opposite polarity to that of the straight line is drawn so as to pass through the remaining plot (plot PLc in FIG. 23). Then, a point where two straight lines intersect (in FIG. 23, the X coordinate is Xe and the parallax is Ve) is obtained. Information related to the combination of the X coordinate Xe and the parallax Ve regarding this point corresponds to the result obtained by the interpolation calculation.

  FIG. 24 is a flowchart illustrating an operation flow according to the corresponding point search operation in the information processing system 1B. This operation flow is realized by the control unit 36B reading and executing the program PGb. In addition, this operation | movement flow is compared with the operation | movement flow which concerns on the corresponding point search operation | movement based on the said one embodiment shown by FIGS. 19-21 until it progresses from step ST19 of FIG. 20 to step ST31 of FIG. In the meantime, steps ST191 and ST192 are added, and the other points are the same. Hereinafter, steps ST191 and ST192 which are different points will be described.

  In step ST191, the interpolation calculation unit 379B determines whether or not the final search process has been executed three times in succession for three consecutive pixels at the same Y coordinate of the reference image G1. If the final search process has been executed three times in succession, the process proceeds to step ST192. If the final search process has not been executed three times in succession, the process proceeds to step ST31 in FIG.

  In step ST192, an interpolation calculation is performed by the interpolation calculation unit 379B. In the interpolation calculation, as described above, the corresponding points are detected with the size of the sub-pixel based on the parallax between the reference image G1 and the reference image G2 obtained by the final search process executed three times in succession. Done. Here, the parallax related to one of the three pixels is derived in the immediately preceding step ST19.

  When step ST192 is completed, the process proceeds to step ST31 in FIG. 21. At this time, in step ST31, information related to the determination result in step ST19 and the result of the interpolation calculation in step ST192 is stored in the storage unit 34. Is done.

  Next, an aspect in which the corresponding points are detected with the size of the subpixel from the result of the search process other than the final search process will be described with a specific example.

  As shown in FIG. 22, the information processing system 1C according to the present modification includes an interpolation calculation unit 379C having a functional configuration as compared with the information processing system 1A according to the above-described embodiment. The program PGa stored in the storage unit 34 is changed to the program PGc, and the information processing apparatus 3A is changed to the information processing apparatus 3C in accordance with the change. . For this reason, the information processing system 1C according to the present modification has the same configuration in most part as the information processing system 1A according to the embodiment. For this reason, about the same structure, the same code | symbol is attached | subjected and description is abbreviate | omitted and a different part is demonstrated below.

  The interpolation calculation unit 379C has the same configuration as the above-described interpolation calculation unit 379B. However, the information used for the interpolation calculation is different in the interpolation calculation unit 379C compared to the interpolation calculation unit 379B. This difference will be described with reference to a flowchart showing an operation flow related to the corresponding point search operation in the information processing system 1C shown in FIG.

  In addition, this operation | movement flow is compared with the operation | movement flow which concerns on the corresponding point search operation | movement which concerns on the said one embodiment shown by FIGS. 19-21 until it progresses to step ST31 of FIG. 21 from step ST21 of FIG. In the meantime, step ST201 is added, and the other points are the same.

  In step ST201, interpolation calculation by the interpolation calculation unit 379C is executed. In the interpolation calculation, corresponding points are detected with the size of the sub-pixel based on the parallax determined for three consecutive pixels of the same Y coordinate in the reference image G1. Here, the parallax related to one of the three pixels is derived in the immediately preceding step ST21.

  When step ST201 is completed, the process proceeds to step ST31 in FIG. 21. At this time, in step ST31, information related to the determination result in step ST21 and the result of the interpolation operation in step ST201 is stored in the storage unit 34. Is done.

  In the above embodiment, the control unit 36A calculates the three-dimensional position of the subject OB based on the principle of triangulation using information on corresponding points between images, and based on the three-dimensional position of the subject OB. A three-dimensional image of the subject OB is visually output on the display unit 32, and various variations are conceivable as the display mode.

  For example, the three-dimensional position of the subject OB obtained from the relationship between the reference point Sp1 to which the error range corresponding to the search accuracy is added and the corresponding point corresponds to the reference point Sp1 to which no error range is added. A possible mode is one that is displayed in a manner distinguishable from the three-dimensional position of the subject OB obtained from the relationship with the point. This distinguishable aspect is realized by, for example, color difference or contour enhancement. This makes it easy for the user to distinguish between a portion with high search accuracy and a portion with low search accuracy.

  In addition, for example, in a configuration in which the approach of an obstacle is detected using a stereo image obtained by an in-vehicle camera and an obstacle with a risk of rear-end collision is displayed in a visually recognizable state, an error corresponding to the search accuracy is displayed. Regarding the three-dimensional position of the subject OB obtained from the relationship between the reference point Sp1 to which the range is added and the corresponding point, a threshold value for determining whether or not an obstacle is approaching is set strictly. Various embodiments are possible.

  Specifically, for example, when it is determined that there is a risk of a rear-end collision if the obstacle is within 20 m, a mode in which the threshold is changed from the default 20 m to 30 m can be considered. By such an aspect, the safety | security with respect to the approach of an obstruction increases.

  In addition, for example, when clustering is performed in which two points whose three-dimensional positions are at a distance less than the threshold are determined to be included in the image area that captures one object, the threshold corresponds to the search accuracy. A mode in which the range is changed according to the error range is conceivable.

  Specifically, for example, if the three-dimensional positions of two adjacent points are within 10 cm, the threshold value for clustering is set to ± 10 cm when it is determined that the object constitutes one object. On the other hand, a mode in which the threshold value related to the clustering is changed to ± 20 cm according to the error range corresponding to the search accuracy is conceivable.

  In the above-described embodiment, when the evaluation value CTR is calculated, the average value of the pixel values related to the evaluation region is calculated as the reference value. However, the present invention is not limited to this. For example, the reference value may be an average value of pixels in four or eight directions adjacent to each pixel.

  In the above-described embodiment, the pixels are thinned out every other horizontal line and every other vertical line for the base image G1 and the reference image G2, and the image is reduced step by step by 50%. Although the first to third reduced standard images G11 to G13 and the first to third reduced reference images G21 to G23 are generated, the present invention is not limited thereto.

  For example, the stepwise reduction rate need not be constant at 50%. However, the first reduced reference image G11 and the first reduced reference image G21 have the same resolution, the second reduced reference image G12 and the second reduced reference image G22 have the same resolution, and the third reduced reference image G13. And the third reduced reference image G23 need to have the same resolution.

  In the above-described embodiment, the pixel value is simply used to calculate the contrast-related evaluation value. However, the present invention is not limited to this. For example, if the stereo image is a color image using red (R), green (G), blue (B), or the like, pixel values are used for each color, and an evaluation value related to contrast is calculated. The determination using the evaluation value may be performed.

  In the above-described embodiment, the search for corresponding points between the two images (the base image G1 and the reference image G2) has been described. However, the present invention is not limited to this. The present invention can be applied to search for corresponding points between two or more images.

  In the above-described embodiment, an evaluation region is set for an image related to the reference image G1 (specifically, the reference image G1, the first reduced reference image G11, and the second reduced reference image G12). Although the evaluation value is calculated for the evaluation region, the present invention is not limited to this. For example, an evaluation area is set for an image related to the reference image G2 (specifically, the reference image G2, the first reduced reference image G21, and the second reduced reference image G22), and an evaluation value is calculated for the evaluation area. May be. At this time, the aspect etc. which the evaluation area | region is set so that process target point Pp2, Pp21, Pp22 is included as a center point etc. can be considered, for example.

  It goes without saying that all or a part of each of the above-described embodiment and various modifications can be appropriately combined within a consistent range.

1A, 1B, 1C Information processing system 2 Stereo camera 3A, 3B, 3C Information processing device 34 Storage unit 36A, 36B, 36C Control unit 361 Image acquisition unit 362 Resolution conversion unit 363 Search reference point setting unit 364 Lower layer image designating unit 365 Initial Parallax setting unit 366 Initial window setting unit 367 Lower layer corresponding point search unit 368 Next layer image designation unit 369 Evaluation region setting unit 370 Evaluation value calculation unit 371 Search prohibition unit 372 Second layer parallax setting unit 373 Next layer window setting unit 374 Next layer correspondence Point search unit 375 Remaining layer determination unit 376 Corresponding point determination unit 377 Storage processing unit 378 Remaining pixel determination unit 379B, 379C Interpolation calculation unit Ae1, Ae11, Ae12 Evaluation region G1 reference image G11 to G13 First to third reduced reference images G2 reference Images G21 to G23 First to third reduced reference images PGa , PGb, PGc Program Pp2, Pp21, Pp22, Pp23 Processing target point Sp1, Sp11, Sp12, Sp13 Reference point W1, W11, W12, W13 Reference region W2, W21, W22, W23 Comparison region

Claims (16)

A corresponding point search device for searching for corresponding points of the second image for each pixel of the first image,
To generate a first low resolution image having a relationship with a reduced resolution with respect to the first image, generating means for generating a second low-resolution image having a relationship with a reduced resolution to the second image When,
A first reference point is set for the first low-resolution image, and a first search process is performed for searching for a first corresponding point corresponding to the first reference point in the second low-resolution image. Search means;
A second reference point corresponding to the first reference point is set for the first image, and the second reference is set for the second image using a first search result obtained by the first search process. Second search means for performing a second search process for searching for a second corresponding point corresponding to the point;
An evaluation region corresponding to at least one of a region including the second reference point in the first image and a region including a point corresponding to the second reference point in the second image. A calculating means for calculating an evaluation value related to a change in pixel value;
Prohibiting means for prohibiting the execution of the second search process according to a comparison result between the evaluation value and the threshold;
When execution of the second search process is prohibited by the prohibition means, a relationship between the second reference point and the second corresponding point is determined based on the first search result, and the prohibition means If execution of the second search process is not prohibited, a determining unit that determines a relationship between the second reference point and the second corresponding point based on a second search result by the second search process ;
A corresponding point search device comprising: The corresponding point search device according to claim 1,
The prohibition means is
The corresponding point search device for prohibiting execution of the second search process when the evaluation value is less than the threshold value. The corresponding point search device according to claim 1 or 2,
The evaluation value is
The corresponding point search apparatus characterized by including the evaluation value which concerns on the contrast of the pixel value in the said evaluation area | region. The corresponding point search device according to claim 3,
The calculating means is
A corresponding point search apparatus that calculates a sum of differences between a pixel value and a reference value for each pixel included in the evaluation region as the evaluation value. The corresponding point search device according to claim 4,
The calculating means is
The corresponding point search apparatus characterized in that an average value of pixel values related to the evaluation area is calculated as the reference value. The corresponding point search device according to claim 3,
The calculating means is
A corresponding point search apparatus that performs a process of extracting a contour of a subject for the evaluation region, and calculates the number of pixels occupied by the contour in the evaluation region as the evaluation value. The corresponding point search device according to claim 1 or 2,
The evaluation value is
The corresponding point search apparatus characterized by including the evaluation value which concerns on distribution of the pattern of the pixel value in the said evaluation area | region. The corresponding point search device according to claim 7,
The calculating means is
A corresponding point search apparatus characterized by deriving a distribution of frequency components by performing a Fourier transform on the evaluation region, and calculating the evaluation value based on the distribution of the frequency components. The corresponding point search device according to any one of claims 1 to 8,
The determining means is
When the prohibition means prohibits the execution of the second search process, an error range corresponding to the resolution of the first low-resolution image is added, and the second reference point and the second corresponding point Corresponding point search device characterized by determining relationship. The corresponding point search device according to any one of claims 1 to 8,
The determining means is
When the prohibition means prohibits execution of the second search process, the first image is detected based on the first search result and an error range corresponding to the resolution of the first low-resolution image. The second reference point and the second correspondence are set such that the amount of deviation between the position of the second reference point and the position of the second corresponding point with respect to the second image is the largest within the range of the error. A corresponding point search device characterized by determining a relationship with a point. The corresponding point search device according to any one of claims 1 to 8,
The determining means is
When the prohibition means prohibits execution of the second search process, the first image is detected based on the first search result and an error range corresponding to the resolution of the first low-resolution image. The second reference point and the second correspondence are set such that a deviation amount between the position of the second reference point and the position of the second corresponding point with respect to the second image is minimized within the range of the error. A corresponding point search device characterized by determining a relationship with a point. The corresponding point search device according to any one of claims 1 to 11,
At least one of the first and second search processes is
A corresponding point search apparatus including processing using a phase only correlation method. The corresponding point search device according to any one of claims 1 to 12,
The determining means is
If the prohibition means does not prohibit the execution of the second search process, a relationship between the second reference point and the second corresponding point is determined according to the second search process result, and the first search process is further performed. For each pixel included in the image, a relationship between the second reference point and the second corresponding point is determined based on the first search processing result, or the first search processing result is determined based on the second search processing result. 2. A corresponding point search device, wherein a relationship between two reference points and the second corresponding point is determined. The corresponding point search device according to claim 13,
The second reference point and the second corresponding point relating to the one pixel obtained by the determining means with respect to an intermediate point from one pixel of the first image to a peripheral pixel located around the one pixel. And an interpolation calculation for deriving a corresponding point in the second image by an interpolation calculation using the relationship between the second reference point and the second corresponding point related to the surrounding pixel obtained by the determining means means,
The corresponding point search apparatus characterized by further including. A corresponding point search method for searching corresponding points of the second image for each pixel of the first image,
To generate a first low resolution image having a relationship with a reduced resolution with respect to the first image, generating step of generating a second low-resolution image having a relationship with a reduced resolution to the second image When,
A first reference point is set for the first low-resolution image, and a first search process is performed for searching for a first corresponding point corresponding to the first reference point in the second low-resolution image. A search step;
At least one of a region including a second reference point corresponding to the first reference point in the first image and a region including a point corresponding to the second reference point in the second image. A calculation step for calculating an evaluation value related to a change in pixel value for an evaluation region corresponding to
Executing a second search process for searching for a second corresponding point corresponding to the second reference point in the second image using the first search result in the first search step, the evaluation value and the threshold value A prohibition step prohibited according to the comparison result of
When execution of the second search process is prohibited by the prohibition step, a relationship between the second reference point and the second corresponding point is determined based on the first search result, and the prohibition step determines the relationship When the execution of the second search process is not prohibited, the second search process is executed, and the relationship between the second reference point and the second corresponding point is based on the second search result obtained by the second search process. a determination step of determining a,
A corresponding point search method comprising:   The program which makes the said corresponding point search apparatus function as a corresponding point search apparatus as described in any one of Claims 1-14 by being performed in the control part contained in a corresponding point search apparatus.

Images