JP2011170519A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a corresponding point search technology which can make compatible both the maintenance of accuracy and an increase in speed in corresponding point search processing with a plurality of images as objects. <P>SOLUTION: An image processing method comprises steps of: converting a first region of a first image to first frequency component information and converting a second region of a second image to second frequency component information; calculating a first correlation value indicating correlation between the first region and the second region on the basis of the first and second frequency component information; setting a use limit range of frequencies where the use of the frequency component information in calculation in a correlation calculation part should be limited on the basis of the first correlation value; converting a third region of a third image corresponding to the first image to third frequency component information and converting a fourth region of a fourth image corresponding to the second image to fourth frequency component information; and calculating a second correlation value indicating a correlation between the third region and the fourth region on the basis of frequency component information concerning frequencies outside the use limit range out of third and fourth frequency component information. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing technique.

  For two images taken from different viewpoints using a stereo camera, for each pixel of one image, a correlation (that is, a degree of similarity) between the two images is obtained, and the same based on the correlation There is known a technique (hereinafter, referred to as “corresponding point search technique”) in which a point (hereinafter referred to as “corresponding point”) in which this portion is captured is detected from the other image.

  With respect to the corresponding point search technique, each pixel of one image is voted on the voting space based on the phase difference component of the sine wave of each frequency between the two images, so that the other image A technique that requires the above corresponding point has been proposed (for example, Patent Document 1). Further, a technique in which weighting is added at the time of voting has been proposed (for example, Non-Patent Document 1). Furthermore, a technique has been proposed in which the amount of calculation is reduced by executing a corresponding point search process using a phase-only correlation method using a one-dimensional image region along an epipolar line (for example, Patent Documents). 2).

Japanese Patent No. 3468182 JP 2008-123141 A

IEICE transactions on information and systems E87-D (1) pp.58-65 (2004), KOUZUKI I (Kogakuin Univ., Tokyo, Jpn) KANEKO T (Kogakuin Univ., Tokyo, Jpn) ITO M (Kogakuin Univ., Tokyo, Jpn) "Precise and Reliable Image Shift Detection by a New Phase-Difference Spectrum Analysis (PSA) Method"

  However, each technique of the above-mentioned Patent Literature 1 and Non-Patent Literature 1 requires a long time for computation because the computation amount is enormous. In the technique of Patent Document 2, it is impossible to apply the technique to an area that is not included in the one-dimensional image area along the epipolar line, and to search for each corresponding point. Since the area used (that is, the amount of data) is too small, the accuracy in the corresponding point search process is reduced.

  The present invention has been made in view of the above problems, and provides a corresponding point search technique capable of both maintaining accuracy and improving speed in corresponding point search processing for a plurality of images. For the purpose.

  In order to solve the above problem, an image processing apparatus according to a first aspect converts a first region of a first image into first frequency component information and converts a second region of the second image into second frequency component information. A conversion unit for converting to a first correlation value, a correlation calculation unit for calculating a first correlation value indicating a correlation between the first region and the second region based on the first and second frequency component information, and the first correlation And a restriction unit that sets a frequency use restriction range in which use of frequency component information in the calculation in the correlation calculation part is restricted based on the value. In the image processing device, the conversion unit converts the third region of the third image corresponding to the first image into third frequency component information, and the fourth image of the fourth image corresponding to the second image. Four regions are converted into fourth frequency component information, and the correlation calculation unit is configured to generate the third region based on frequency component information relating to a frequency outside the use restriction range in the third and fourth frequency component information. And a second correlation value indicating a correlation between the first region and the fourth region.

  An image processing device according to a second aspect is the image processing device according to the first aspect, wherein the first image is generated by reducing the number of pixels from the third image according to a predetermined rule, and the first image A generation unit that generates the second image by reducing the number of pixels from the four images according to the predetermined rule.

  An image processing apparatus according to a third aspect is the image processing apparatus according to the first or second aspect, wherein the third and fourth images are a plurality of images obtained by capturing the same subject from different viewpoints. .

  The image processing apparatus according to the fourth aspect converts the first region of the first image into first frequency component information, converts the second region of the second image into second frequency component information, and converts the second image into the second frequency component information. Based on the first and second frequency component information, and a conversion unit that converts the third region of the third image into third frequency component information, and converts the fourth region of the third image into fourth frequency component information. A correlation calculation unit that calculates a first correlation value indicating a correlation between one region and the second region, and the use of frequency component information in the calculation by the correlation calculation unit is limited based on the first correlation value. And a limiting unit that sets a frequency use limiting range. Then, in the image processing device, the correlation calculation unit is configured to use the third region and the fourth region based on frequency component information relating to a frequency outside the use restriction range in the third and fourth frequency component information. A second correlation value indicating the correlation with the region is calculated.

  An image processing apparatus according to a fifth aspect is the image processing apparatus according to the fourth aspect, wherein the first to third images have a plurality of time-lapse changes of the same subject from the same viewpoint. It is an image.

  An image processing device according to a sixth aspect is the image processing device according to any one of the first to fifth aspects, wherein the limiting unit is configured to change the conversion unit as the first correlation value increases. The use restriction range is set so that use of frequency component information relating to a wider range of frequencies from the lowest frequency side among the obtained frequency component information is restricted.

  An image processing device according to a seventh aspect is the image processing device according to any one of the first to sixth aspects, wherein the limiting unit is configured to change the conversion unit as the first correlation value decreases. The use restriction range is set so that use of frequency component information relating to a wider range of frequencies from the highest frequency side among the obtained frequency component information is restricted.

  An image processing method according to an eighth aspect includes a first conversion step of converting a first region of a first image into first frequency component information and converting a second region of the second image into second frequency component information; Based on the first and second frequency component information, a first calculation step for calculating a first correlation value indicating a correlation between the first region and the second region, and based on the first correlation value, A setting step for setting a frequency use restriction range in which the use of frequency component information in the calculation in the correlation calculation unit is restricted, and a third region of the third image corresponding to the first image as third frequency component information A second conversion step of converting the fourth region of the fourth image corresponding to the second image into fourth frequency component information, and out of the use restriction range of the third and fourth frequency component information Frequency Based on the information, and a second calculation step of calculating a second correlation value indicating a correlation between the third region and the fourth region.

  An image processing method according to a ninth aspect includes a first conversion step of converting a first region of a first image into first frequency component information and converting a second region of the second image into second frequency component information; A first calculation step of calculating a first correlation value indicating a correlation between the first region and the second region based on the first frequency component information and the second frequency component information; and the first correlation A setting step for setting a use restriction range of a frequency in which use of frequency component information in the correlation calculation is restricted based on the value; a third region of the second image is converted into third frequency component information; A second conversion step of converting a fourth region of the image into fourth frequency component information; and the third component based on frequency component information of a frequency outside the use restriction range of the third and fourth frequency component information. Region and the fourth region And a second calculation step of calculating a second correlation value indicating the correlation between.

  A program according to a tenth aspect is executed by a control unit included in the information processing apparatus, thereby causing the information processing apparatus to function as the image processing apparatus according to any one of the first to seventh aspects. It is.

  The image processing apparatus according to any one of the first to third aspects also uses the correlation value indicating the degree of similarity between the plurality of images to determine the similarity between the plurality of images corresponding to the plurality of images. Since the amount of calculation when calculating the correlation value indicating the degree of the reduction is reduced, both the maintenance of accuracy and the improvement of the speed in the corresponding point search process for a plurality of images are compatible.

  Even with the image processing device according to any of the fourth and fifth aspects, the correlation between the second image and the third image is based on the correlation value indicating the degree of similarity between the first image and the second image. Since the amount of calculation when calculating the correlation value indicating the degree of similarity is reduced, both maintaining accuracy and improving speed in the corresponding point search processing for a plurality of images are compatible.

  With the image processing apparatus according to any of the sixth and seventh aspects, the search speed can be improved by reducing the amount of calculation.

  According to the image processing method according to the eighth aspect, the same effect as the image processing apparatus according to the first aspect can be obtained.

  According to the image processing method of the ninth aspect, the same effect as that of the image processing apparatus according to the fourth aspect can be obtained.

  According to the program according to the tenth aspect, an effect similar to that of the image processing apparatus according to any one of the first to seventh aspects can be obtained.

It is a figure showing a schematic structure of an information processing system concerning one embodiment. It is a block diagram which shows the principal part structure of the information processing system which concerns on one Embodiment. It is a figure which shows the functional structural example of the control part which concerns on one Embodiment. It is a schematic diagram which illustrates a standard image and a reference image. It is a schematic diagram which illustrates the 1st reduction standard image and the 1st reduction reference image. It is a schematic diagram which illustrates the 2nd reduction standard image and the 2nd reduction reference image. It is a schematic diagram which illustrates the 3rd reduction standard image and the 3rd reduction reference image. It is a figure which shows typically the setting aspect of the reference point with respect to a reference image. It is a schematic diagram which shows the example of a setting of the reference point and process target point which concern on the 3rd layer. It is a schematic diagram which shows the example of a setting of the reference point and process target point which concern on the 2nd layer. It is a schematic diagram which shows the example of a setting of the reference point which concerns on the 1st layer. It is a schematic diagram which shows the example of a setting of the process target point which concerns on the 1st layer. It is a schematic diagram which shows the example of a setting of the reference point in a reference image. It is a schematic diagram which shows the example of a setting of the process target point in a reference image. It is a schematic diagram which shows the example of a setting of the reference | standard area | region and comparison area | region which concerns on the 3rd layer. It is a schematic diagram which shows the example of a setting of the reference | standard area | region and comparison area | region which concerns on the 2nd layer. It is a schematic diagram which shows the example of a setting of the reference area | region which concerns on the 1st layer. It is a schematic diagram which shows the example of a setting of the comparison area | region which concerns on the 1st layer. It is a schematic diagram which shows the example of a setting of the reference | standard area | region with respect to a reference | standard image. It is a schematic diagram which shows the example of a setting of the comparison area with respect to a reference image. It is a figure which illustrates the relationship between a frequency and a weighting coefficient. It is a figure which illustrates the relationship between a frequency and a weighting coefficient. It is a schematic diagram which illustrates the relationship between a correlation value and a weighting coefficient. It is a schematic diagram which illustrates the relationship between a correlation value and a weighting coefficient. It is a schematic diagram which illustrates the relationship between a correlation value and a weighting coefficient. It is a figure which illustrates the voting space used by correlation calculation. It is a figure for demonstrating the voting method in correlation calculation. It is a figure for demonstrating the voting method in correlation calculation. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on one Embodiment. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on one Embodiment. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on one Embodiment. It is a figure which shows schematic structure of the information processing system which concerns on one modification. It is a block diagram which shows the principal part structure of the information processing system which concerns on one modification. It is a schematic diagram which illustrates a T1 frame image. It is a schematic diagram which illustrates a T2 frame image. It is a schematic diagram which illustrates a T3 frame image. It is a figure which shows the functional structural example of the control part which concerns on one modification. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on one modification. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on one modification. It is a flowchart which shows the flow of the corresponding point search operation | movement which concerns on one modification.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

<(1) Configuration of information processing system>
FIG. 1 is a diagram showing a schematic configuration of an information processing system 1A according to an embodiment of the present invention, and FIG. 2 is a block diagram showing a main configuration of the information processing system 1A.

  The information processing system 1A includes a two-lens stereo camera 2 and an information processing apparatus 3 connected to the stereo camera 2 so as to be able to transmit and receive data.

  The two-lens stereo camera 2 is provided with two imaging systems 21 and 22 each having an imaging device. The imaging systems 21 and 22 are spaced apart along a predetermined direction, and are configured to capture the subject OB in front of the camera from different viewpoints at the same timing. Two image data (hereinafter abbreviated as “image”) obtained by imaging at the same timing by the imaging systems 21 and 22 are so-called stereo images, and are transmitted to the information processing device 3 via the data line CB.

  Hereinafter, an image acquired by the imaging system 21 is referred to as a “first captured image” G1, and an image acquired by the imaging system 22 is referred to as a “second captured image” G2. That is, the first and second captured images G1 and G2 form a set of images in which the same subject OB is captured from different viewpoints.

  Here, in order to avoid complication of explanation, the coordinates of a pixel in which a certain part of the subject is captured in the first captured image G1, and the coordinates of a pixel in which the same part of the subject is captured in the second captured image G2. However, description will be made assuming that the imaging system 21 and the imaging system 22 are installed so as to be substantially the same in the vertical direction (Y direction) and different in the horizontal direction (X direction).

  In order to simplify the explanation, the aberration of the stereo camera 2 is corrected well, and the imaging systems 21 and 22 are set to be substantially parallel (preferably completely parallel). That is, the optical axes of the imaging systems 21 and 22 are set to be substantially parallel (preferably completely parallel). If the actual configuration of the stereo camera 2 does not meet such conditions, the image processing for the first and second captured images G1 and G2 may be converted into a stereo image captured under equivalent conditions. good.

  The information processing apparatus 3 is configured by, for example, a personal computer (personal computer), an operation unit 31 including a mouse and a keyboard, a display unit 32 including a liquid crystal display, and an interface for receiving data from the stereo camera 2 ( I / F) unit 33. In addition, the information processing device 3 includes a storage unit 34, an input / output unit 35, and a control unit 36.

  The storage unit 34 is configured by, for example, a hard disk or the like, and stores first and second captured images G1 and G2 obtained by imaging with the stereo camera 2. The storage unit 34 stores a program PGa for performing a corresponding point search operation described later.

  The input / output unit 35 includes, for example, a disk drive, receives the storage medium 9 such as an optical disk, and exchanges data with the control unit 36.

  The control unit 36 includes a CPU 36 a that functions as a processor and a memory 36 b that temporarily stores information, and comprehensively controls each unit of the information processing apparatus 3. The control unit 36 reads and executes the program PGa in the storage unit 34, thereby realizing various functions and various information processing. The program data stored in the storage medium 9 can be stored in the memory 36b via the input / output unit 35. This stored program can be appropriately reflected in the operation of the information processing apparatus 3.

  Further, the control unit 36 performs a series of operations (hereinafter referred to as “corresponding”) for searching for a point where the same part of the subject is captured between the first and second captured images G1 and G2 constituting the stereo image acquired by the stereo camera 2. Also referred to as “point search operation”.

  Further, the control unit 36 calculates the position of the subject OB in the three-dimensional space (hereinafter also referred to as “three-dimensional position”) based on the principle of triangulation using information obtained by the corresponding point search operation. The display unit 32 visually outputs an image (hereinafter, also referred to as “three-dimensional image”) representing the subject OB three-dimensionally based on the three-dimensional position of the subject OB calculated by the control unit 36. The

<(2) Corresponding point search operation>
In the corresponding point search operation according to the present embodiment, the following two steps (I) and (II) are sequentially performed in this order.

  (I) Based on a set of the first captured image G1 and the second captured image G2 constituting a stereo image, a set of images with a resolution reduced in a plurality of stages is generated.

  (II) For each pixel of the first captured image G1, in order from the set of images having a low resolution, a point of the same subject as a reference point (hereinafter referred to as “reference point”) of one image (hereinafter referred to as “reference point”) A process of searching for “corresponding point” from the other image (hereinafter also referred to as “corresponding point search process”) is performed. In each corresponding point search process, a degree of similarity (hereinafter also referred to as “correlation”) between two images is obtained using a frequency component, and a corresponding point corresponding to a reference point is detected according to the correlation. The “frequency component” referred to here includes the amplitude and phase of a sine wave of each frequency representing a spatial change in pixel values on the image.

  In step (II), in the corresponding point search process for a set of images with a certain resolution, the corresponding point search process for a set of images whose resolution is one step lower than the set of images with a certain resolution is performed. The result is used.

  Furthermore, in the step (II), in the corresponding point search process for a set of images of a certain resolution, the certain resolution is used in order to achieve both the maintenance of the accuracy and the speed improvement in the corresponding point search process. Used for calculation according to the level of the correlation value (hereinafter also referred to as “correlation value”) obtained by the corresponding point search processing for a set of images whose resolution is one step lower than the set of images The frequency component to be performed is limited. Here, the restriction of the frequency component is realized by multiplying the amplitude by a weighting coefficient described later.

In the present embodiment, an example in which a peak correlation value C _max described later is employed as a correlation value that contributes to the limitation of the frequency component used for the calculation will be described.

  Hereinafter, a functional configuration for realizing the corresponding point search operation and a flow of the corresponding point search operation will be sequentially described.

<(2-1) Functional configuration related to corresponding point search operation>
FIG. 3 is a diagram illustrating a functional configuration realized by the control unit 36 in order to execute the corresponding point search operation. Here, the functional configuration of the control unit 36 is described as being realized by executing the program PGa, but may be realized by a dedicated hardware configuration.

  As illustrated in FIG. 3, the control unit 36 includes an image acquisition unit 361, a resolution conversion unit 362, a search reference point setting unit 363, an image designation unit 364, a reference parallax setting unit 365, a window setting unit as functional configurations. 366, a frequency analysis unit 367, a frequency component restriction unit 368, a correlation calculation unit 369, and a corresponding point determination unit 370.

<(2-1-1) Image acquisition unit>
The image acquisition unit 361 acquires from the storage unit 34 the first captured image G1 and the second captured image G2 obtained by capturing the same subject at approximately the same timing.

  FIG. 4 is a schematic view illustrating the forms of the first and second captured images G1 and G2. In the first and second captured images G1, G2, a large number of pixels are arranged in a matrix. Specifically, a first predetermined number (here, 480) of pixels are arranged in the vertical direction (Y direction), and a second predetermined number (here, 640) of pixels are arranged in the horizontal direction (X direction). Arranged.

  Here, in the first and second captured images G1 and G2, the upper left position is the origin, the horizontal position in each pixel of the first and second captured images G1 and G2 is indicated by the X coordinate, and the vertical direction Is indicated by the Y coordinate. That is, in the first and second captured images G1 and G2, the position of each pixel is indicated by XY coordinates (x, y). For example, if the pixel is shifted rightward (X direction) by one pixel, the value of the X coordinate is 1. The value of the Y coordinate increases by one when it is shifted by one pixel in the downward direction (Y direction).

  In the corresponding point search operation, the first captured image G1 is used as a reference image (hereinafter also referred to as “reference image”), and the second captured image G2 is referred to as an image (hereinafter also referred to as “reference image”). Is done. Therefore, the first captured image G1 is referred to as a standard image G1, and the second captured image G2 is referred to as a reference image G2.

  Strictly speaking, the image acquisition unit 361 acquires data indicating the standard image G1 and the reference image G2. However, in this specification, the data indicating the standard image G1 and the standard image displayed based on the data are displayed. G1 itself is collectively referred to as “standard image G1”, and data indicating the reference image G2 and the reference image G2 itself displayed based on the data are collectively referred to as a reference image G2.

<(2-1-2) Resolution converter>
The resolution conversion unit 362 generates an image (hereinafter, also referred to as “low-resolution reference image”) having a relationship in which the resolution is gradually reduced with respect to the reference image G1 with the reference image G1 as a reference. Further, the resolution conversion unit 362 generates an image (hereinafter, also referred to as “low-resolution reference image”) having a relationship in which the resolution is gradually reduced with respect to the reference image G2 with the reference image G2 as a reference.

  Here, for the base image G1 and the reference image G2, a plurality of columns of pixels in the vertical direction (Y direction) (hereinafter referred to as “vertical lines”) and a plurality of rows of pixels in the horizontal direction (X direction), respectively. (Hereinafter, referred to as “horizontal line”), the low resolution standard image and the low resolution reference image are generated by thinning out the vertical line and the horizontal line according to a predetermined thinning rule.

  Here, the predetermined thinning rule is a rule indicating regularity for determining vertical lines to be thinned out and / or regularity for determining horizontal lines to be thinned out. For example, the predetermined thinning rule includes a rule indicating that one vertical line is thinned out for every predetermined number of vertical lines, and a rule indicating that one horizontal line is thinned out for every predetermined number of horizontal lines. included.

  In the following, an example in which both vertical lines and horizontal lines are thinned out is shown, but a line composed of a plurality of pixels of at least one of the vertical lines and horizontal lines (hereinafter also referred to as “pixel lines”) is omitted. It only has to be drawn.

  For the reference image G1, first, every other vertical line and every other horizontal line are thinned out from a plurality of vertical lines and a plurality of horizontal lines constituting the reference image G1. At this time, a reduced low-resolution reference image (hereinafter also referred to as “first reduced reference image”) G11 (FIG. 5) is generated. In the first reduced reference image G11, 240 pixels in the vertical direction and 320 pixels in the horizontal direction are arranged in a matrix.

  Next, the vertical line every other column and the horizontal line every other row are thinned out from the first reduced reference image G11 to further reduce the low-resolution reference image (hereinafter, “second reduced reference image”). G12 (FIG. 6) is also generated. In the second reduced reference image G12, 120 pixels in the vertical direction and 160 pixels in the horizontal direction are arranged in a matrix.

  Next, every other vertical line and every other horizontal line are thinned out from the second reduced reference image G12 to further reduce the low-resolution reference image (hereinafter referred to as “third reduced reference image”). G13 (FIG. 7) is generated. In the third reduced reference image G13, 60 pixels in the vertical direction and 80 pixels in the horizontal direction are arranged in a matrix.

  In this way, the first to third reduced reference images G11 to G13 corresponding to the reference image G1 are generated based on the reference image G1.

  On the other hand, for the reference image G2, first, every other vertical line and every other horizontal line are thinned out from a plurality of vertical lines and a plurality of horizontal lines constituting the reference image G2. At this time, a reduced low-resolution reference image (hereinafter also referred to as “first reduced reference image”) G21 (FIG. 5) is generated. In the first reduced reference image G21, 240 pixels in the vertical direction and 320 pixels in the horizontal direction are arranged in a matrix.

  Next, every other vertical line and every other horizontal line are thinned out from the first reduced reference image G21 to further reduce the low resolution reference image (hereinafter “second reduced reference image”). G22 (FIG. 6) is also generated. In the second reduced reference image G22, 120 pixels in the vertical direction and 160 pixels in the horizontal direction are arranged in a matrix.

  Next, every other vertical line and every other horizontal line are thinned out from the second reduced reference image G22, thereby further reducing the low-resolution reference image (hereinafter referred to as “third reduced reference image”). G23 (FIG. 7) is generated. In the third reduced reference image G23, 60 pixels in the vertical direction and 80 pixels in the horizontal direction are arranged in a matrix.

  In this way, the first to third reduced reference images G21 to 23 corresponding to the reference image G2 are generated based on the reference image G2.

  In other words, in the resolution conversion unit 362, a set of the first reduced reference image G11 and the first reduced reference image G21 (hereinafter referred to as “the first layer low resolution image of the first resolution image) from the set of the reference image G1 and the reference image G2. A set of the second reduced reference image G12 and the second reduced reference image G22 (hereinafter, also referred to as a “second set of low-resolution images”), and the third reduced reference image G13 and the third reduced reference image G22. A set with the reduced reference image G23 (hereinafter also referred to as “a set of low-resolution images in the third layer (lowermost layer)”) is generated.

  That is, in the resolution conversion unit 362, a set of the base image G1 and the reference image G2 is used as a reference, and a set of low-resolution images in the first layer in which the resolution is reduced by one step, and the resolution is reduced by two steps. A set of low-resolution images in the layer and a set of low-resolution images in the third layer (lowermost layer) with the resolution reduced by three levels are generated.

<(2-1-3) Search reference point setting section>
The search reference point setting unit 363 sets a reference point (reference point) Sp1 (FIG. 8) as a reference for the corresponding point search for the reference image G1.

  For example, first, the upper left pixel in the reference image G1 is set as the reference point Sp1. Each time a corresponding point is detected from the reference image G2 with respect to the reference point Sp1, a new reference point Sp1 is set.

  Specifically, as shown by the arrow in FIG. 8, the reference point Sp1 is a predetermined pixel (here, from left to right (X direction) sequentially from the upper direction (+ Y direction) with respect to the reference image G1. In this case, the corresponding points corresponding to the reference points Sp1 are sequentially detected on the reference image G2.

  In other words, with respect to the reference image G1, the reference point Sp1 is set sequentially in time from the −X side to the + X side along each horizontal line parallel to the X axis lined from the −Y side to the + Y side. The When the setting of the reference point Sp1 along one horizontal line is completed, the reference point Sp1 is sequentially set along the next horizontal line located on the + Y side for one pixel.

<(2-1-4) Image designation part>
The image designating unit 364 designates a set of images (hereinafter referred to as “processing target image pair”) to be subjected to corresponding point search processing. Here, each time a corresponding point is recognized for a certain low-resolution processing target image pair, an image set having a one-step resolution higher than a certain low-resolution image set in which the corresponding point is recognized is Specified as a processing target image pair.

  For example, first, a set of low-resolution images (specifically, the third reduced reference image G13 and the third reduced reference image G23) in the lowermost layer (third layer) is designated as a processing target image pair. Next, a set of low-resolution images in the second layer (specifically, the second reduced reference image G12 and the third reduced reference image G22) is designated as a processing target image pair. Next, a set of low-resolution images in the first layer (specifically, the first reduced reference image G11 and the first reduced reference image G21) is designated as a processing target image pair. Next, the standard image G1 and the reference image G2 are designated as a processing target image pair.

<(2-1-5) Reference parallax setting unit>
The reference parallax setting unit 365 temporarily sets parallax (hereinafter also referred to as “reference parallax”) as a reference for searching for corresponding points for each processing target image pair.

  Specifically, an initial value of the reference parallax is provisionally set for a set of low resolution images in the lowest layer (third layer). As an initial value of the reference parallax, for example, a predetermined value (for example, zero) in the X and Y directions is set.

  For example, as shown in FIG. 9, a reference point Sp13 corresponding to the reference point Sp1 set by the search reference point setting unit 363 is set for the third reduced reference image G13, and the third reduced reference image is displayed. A point (hereinafter also referred to as a “processing target point”) Pp23 to be subjected to arithmetic processing is set for G23. Here, the amount of deviation between the coordinates of the reference point Sp13 in the third reduced reference image G13 and the coordinates of the processing target point Pp23 in the third reduced reference image G23 corresponds to the initial value of the reference parallax.

  Next, the parallax obtained by the corresponding point search processing using the pair of low-resolution images in the third layer as the processing target image pair is compared with the pair of low-resolution images in the second layer as the next processing target image pair. , Temporarily set as the reference parallax. Here, since the resolution of the second layer is twice the resolution of the third layer, if the parallax is handled as a shift amount of the coordinate of the pixel, it corresponds to the parallax obtained by the corresponding point search process for the third layer. Twice the pixel coordinate shift amount is the pixel coordinate shift amount corresponding to the reference parallax set in the second layer.

  For example, as shown in FIG. 10, a reference point Sp12 corresponding to the reference point Sp1 is set for the second reduced reference image G12, and a processing point Pp22 is set for the second reduced reference image G22. Is done. Here, a set of low-resolution images in the third layer whose amount of deviation between the coordinates of the reference point Sp12 in the second reduced reference image G12 and the coordinates of the processing target point Pp22 in the second reduced reference image G22 is the processing target image This corresponds to the parallax obtained by the paired corresponding point search process.

  Next, the parallax obtained by the corresponding point search process using the pair of low-resolution images in the second layer as the processing target image pair becomes the pair of low-resolution images in the first layer as the next processing target image pair. On the other hand, it is temporarily set as the reference parallax. Here, since the resolution of the first layer is twice the resolution of the second layer, if the parallax is handled as a shift amount of the coordinate of the pixel, it corresponds to the parallax obtained by the corresponding point search process for the second layer. Twice the pixel coordinate shift amount is the pixel coordinate shift amount corresponding to the reference parallax set in the first layer.

  For example, as shown in FIG. 11, a reference point Sp11 corresponding to the reference point Sp1 is set for the first reduced reference image G11, and as shown in FIG. 12, for the first reduced reference image G21. The processing target point Pp21 is set. Here, a set of low-resolution images of the second layer in which the amount of deviation between the coordinates of the reference point Sp11 in the first reduced reference image G11 and the coordinates of the processing target point Pp21 in the first reduced reference image G21 is a processing target image. This corresponds to the parallax obtained by the paired corresponding point search process.

  Further, the parallax obtained by the corresponding point search processing using the pair of low-resolution images in the first layer as the processing target image pair becomes the pair of the reference image G1 and the reference image G2 as the next processing target image pair. On the other hand, it is temporarily set as the reference parallax. Here, since the resolution of the standard image G1 and the reference image G2 is twice the resolution of the first layer, if the parallax is handled as a shift amount of the coordinate of the pixel, it can be obtained by the corresponding point search process for the first layer. Twice the displacement amount of the pixel coordinates corresponding to the parallax is the displacement amount of the pixel coordinates corresponding to the reference parallax set in the reference image G1 and the reference image G2.

  For example, as shown in FIG. 13, a reference point Sp1 is set for the reference image G1, and a processing target point Pp2 is set for the reference image G2, as shown in FIG. Here, the amount of deviation between the coordinates of the reference point Sp1 in the reference image G1 and the coordinates of the position of the processing target point Pp2 in the reference image G2 corresponds to a pair of low resolution images in the first layer as a processing target image pair. This corresponds to the parallax obtained by the point search process.

  Here, the reference points Sp11, Sp12, Sp13 have been described as corresponding to the reference point Sp1, but this is because the reference point Sp1 and the reference points Sp11, Sp12, Sp13 are portions of the same subject. It means that it is a point that captures.

  Specifically, the relative positional relationship between the outer edge of the reference image G1 and the reference point Sp1, the relative positional relationship between the outer edge of the first reduced reference image G11 and the reference point Sp11, and the second reduced reference image G12. This means that the relative positional relationship between the outer edge of the third reference image Sp12 and the reference point Sp13 and the relative positional relationship between the outer edge of the third reduced reference image G13 and the reference point Sp13 are substantially the same (preferably completely the same). ing.

<(2-1-6) Window setting section>
The window setting unit 366 sets windows (hereinafter also referred to as “reference areas”) W1, W11 to W13 including the respective reference points Sp1, Sp11 to Sp13 as centers, and sets the processing target points Pp2, Pp21 to Pp23. Windows (hereinafter also referred to as “comparison areas”) W2, W21 to W23 included as centers are set. The reference areas W1, W11 to W13 and the comparison areas W2, W21 to W23 are square image areas of the same size, and are configured, for example, by arranging 32 pixels in the vertical direction and the horizontal direction, respectively.

  Specifically, as shown in FIG. 15, a reference region W13 including the reference point Sp13 as a center is set for the third reduced reference image G13, and for the third reduced reference image G23, A comparison area W23 including the processing target point Pp23 as a center is set.

  Also, as shown in FIG. 16, a reference region W12 that includes the reference point Sp12 as a center is set for the second reduced reference image G12, and a processing target point for the second reduced reference image G22. A comparison area W22 including Pp22 as the center is set.

  In addition, as shown in FIG. 17, a reference region W11 including the reference point Sp11 as a center is set for the first reduced reference image G11, and as shown in FIG. 18, the first reduced reference image is set. For G21, a comparison region W21 including the processing target point Pp21 as a center is set.

  Further, as shown in FIG. 19, a reference area W1 including the reference point Sp1 as a center is set for the reference image G1, and the reference image G2 is processed for the reference image G2 as shown in FIG. A comparison region W2 including the target point Pp2 as a center is set.

<(2-1-7) Frequency analysis section>
The frequency analysis unit 367 performs a process of converting an image into information indicating a frequency component related to a sine wave (hereinafter also referred to as “conversion process”). This conversion process is performed for each of the reference areas W1, W11 to W13, and for each of the comparison areas W2, W21 to W23.

  Here, it is assumed that the conversion process in the frequency analysis unit 367 is executed by a known Fourier transform for each image region. Here, the discrete areas where the reference values W1, W11 to W13 and the comparison areas W2, W21 to W23 are discretely arranged are given discrete pixel functions, and the discrete Fourier transform is performed. Is called.

  Specifically, information (hereinafter also referred to as “frequency component information”) indicating the frequency component of the sine wave related to the reference region W13 by two-dimensional Fourier transform for the reference region W13 by the frequency analysis unit 367 as a conversion unit. ) Is obtained, and frequency component information related to the comparison region W23 is obtained by two-dimensional Fourier transform for the comparison region W23. Further, the frequency component information related to the reference region W12 is obtained by the two-dimensional Fourier transform for the reference region W12, and the frequency component information related to the comparison region W22 is obtained by the two-dimensional Fourier transform for the comparison region W22. It is done.

  Further, the frequency component information related to the reference region W11 is obtained by the two-dimensional Fourier transform for the reference region W11, and the frequency component information related to the comparison region W21 is obtained by the two-dimensional Fourier transform for the comparison region W21. It is done. Furthermore, frequency component information related to the reference region W1 is obtained by two-dimensional Fourier transform for the reference region W1, and frequency component information related to the comparison region W2 is obtained by two-dimensional Fourier transform for the comparison region W2. It is done.

  In the present embodiment, since the frequency component of the two-dimensional image is handled, each frequency is, for example, a frequency obtained by combining the frequency u in the x direction and the frequency v in the y direction orthogonal to the x direction. . This synthesized frequency is indicated by frequency (u, v). Further, the conversion processing in the frequency analysis unit 367 may be executed by other methods such as so-called wavelet conversion.

<(2-1-8) Frequency component limiter>
<(2-1-8-1) Overview of frequency component restriction>
The frequency component restriction unit 368 restricts the frequency component information generated by the frequency analysis unit 367 according to a predetermined restriction rule, and then outputs the information to the correlation calculation unit 369. Thereby, in the calculation (hereinafter also referred to as “correlation calculation”) for obtaining the correlation value between the reference region and the comparison region in the correlation calculation unit 369, a part of the frequency component information generated by the frequency analysis unit 367. The remaining frequency component information is not used, and the remaining part of the frequency component information is selectively used.

  Specifically, with respect to one reference point Sp1, depending on the level of the correlation value obtained by the correlation calculation of the corresponding point search process for a certain processing target image pair, the certain processing target image pair However, the frequency component information used in the correlation calculation of the corresponding point search process for the processing target image pair whose resolution is one step higher is limited.

  For example, first, according to the level of the correlation value between the reference region W13 and the comparison region W23 obtained based on the frequency component information related to the reference region W13 and the comparison region W23 of the third layer, the reference of the second layer Frequency component information used in the correlation calculation for the region W12 and the comparison region W22 is limited.

  Next, according to the level of the correlation value between the reference region W12 and the comparison region W22 obtained based on the frequency component information related to the reference region W12 and the comparison region W22 of the second layer, the reference region of the first layer Frequency component information used in the correlation calculation for W11 and comparison region W21 is limited.

  Next, in comparison with the reference region W1 according to the level of the correlation value between the reference region W11 and the comparison region W21 obtained based on the frequency component information relating to the reference region W11 and the comparison region W21 of the first layer. Frequency component information used in the correlation calculation for the region W2 is limited.

  Specifically, by determining the weighting coefficient described below, a frequency range (hereinafter also referred to as “usage limited frequency range”) of frequency component information that is restricted for use in correlation calculation in the correlation calculation unit 369 is set. The

  In the following description, the corresponding point search process for a processing target image pair whose resolution is lower by one level on the basis of a processing target image pair having a certain resolution is also referred to as a “preceding point corresponding point search process”. In addition, the corresponding point search process for a processing target image pair whose resolution is one step higher than a processing target image pair having a certain resolution is also referred to as a “next-stage corresponding point search process”.

  Here, the setting of the use limited frequency range by determining the weighting coefficient will be described.

  When the correlation value obtained by a certain corresponding point search process is high, it is predicted that there is a corresponding point corresponding to the reference point in the vicinity of the processing target point in the next corresponding point search process. In such a case, in the corresponding point search process in the next stage, there is no need to search for a corresponding point at a position that is somewhat distant from the processing target point.

  Note that when the difference in image pattern between the reference region and the comparison region forming one processing target image pair is small, the relative frequency is higher than the frequency component of a relatively high frequency (hereinafter referred to as “high frequency”). The difference between the frequency component in the reference region and the frequency component in the comparison region is smaller for a frequency component having a lower frequency (hereinafter referred to as “low frequency”).

  From the above, when the correlation value obtained by the correlation calculation of a corresponding point search process is high, the corresponding point search process in the vicinity of the processing target point is performed with high accuracy in the next corresponding point search process. It ’s fine. In other words, the higher the correlation value obtained in a corresponding point search process, the lower the contribution of the low frequency frequency component to ensuring the search accuracy in the corresponding point search process at the next stage. Conversely, the lower the correlation value obtained in a corresponding point search process, the lower the contribution of high frequency frequency components to ensuring the search accuracy in the corresponding point search process at the next stage.

  Therefore, in a corresponding point search process, if the correlation value obtained in the preceding corresponding point search process is high, the frequency component limiting unit 368 reduces the weighting of the low frequency frequency component. Conversely, if the correlation value obtained by the corresponding point search process in the preceding stage is low, the frequency component limiting unit 368 reduces the weighting for the high frequency component.

  Specifically, as the correlation value obtained in the corresponding point search process in the previous stage increases, the frequency component related to a wider range of frequencies from the lowest frequency side among the frequency component information related to each frequency obtained by the frequency analysis unit 367. The use restricted frequency range is set so that the use of information is restricted. As a result, the search speed is improved by reducing the amount of calculation, and both the maintenance of accuracy and the speed improvement in the corresponding point search process are achieved.

  On the other hand, as the correlation value obtained in the corresponding point search process in the previous stage becomes smaller, the frequency component information related to a wider range of frequencies from the highest frequency side among the frequency component information related to each frequency obtained by the frequency analysis unit 367. The limited use frequency range is set so that the use is limited. As a result, the search speed is improved by reducing the amount of calculation, and both the maintenance of accuracy and the speed improvement in the corresponding point search process are achieved.

<(2-1-8-2) Specific example of frequency component limiting method>
Next, a specific example of the frequency component limiting method in the frequency component limiting unit 368 will be described.

Here, for each maximum value of correlation values (hereinafter also referred to as “peak correlation values”) C _max obtained by the corresponding point search process in the preceding stage, information indicating the relationship between the frequency and the weighting coefficient (that is, the weighting rule) is: Prestored in the storage unit 34 or the like. Then, the processing in the frequency component limiter 368 is performed while referring to this information.

  In the frequency component limiting unit 368, the following processes (i) to (iii) are performed in this order.

(i) A weighting rule corresponding to the peak correlation value C _max is adopted according to the peak correlation value C _max obtained by the corresponding point search processing in the preceding stage in the correlation calculation unit 369.

  (ii) A frequency coefficient (specifically amplitude information) related to each frequency is multiplied by a weighting coefficient in accordance with the weighting rule adopted in the process (i).

  (iii) Frequency component information indicating the frequency component related to each frequency after the weighting coefficient is multiplied to the amplitude information in the process (ii) is output to the correlation calculation unit 369.

In such a flow, the frequency component restriction unit 368 restricts the frequency component information generated by the frequency analysis unit 367 according to the weighting rule according to the peak correlation value C _max obtained by the previous correlation calculation.

FIG. 21 is a diagram illustrating the relationship between the frequency set for a certain peak correlation value C _max and the weighting coefficient (that is, the weighting rule).

  In FIG. 21, the vertical axis indicates the weighting coefficient, and the horizontal axis indicates the frequency. A thick dashed curve LL1 indicates the relationship between the frequency and the weighting coefficient for a weighting rule (hereinafter also referred to as “low frequency emphasis type rule”) in which low frequency frequency components are emphasized. Furthermore, a plurality of step-like line segments LH1 indicated by bold lines indicate the relationship between the frequency and the weighting coefficient for a weighting rule in which high frequency components are important (hereinafter also referred to as “high frequency priority rule”).

In the weighting process for the frequency component according to the weighting rule as shown in FIG. 21, the weighting coefficient W _{LOW of the low} frequency emphasis type rule and the weighting coefficient W _HIGH of the high frequency emphasis type rule are set for each discrete frequency. Of these, a relatively low weighting factor is employed.

For example, here, the discrete frequency is i (i = 0 to i _max ), and the low-frequency weighted rule weighting coefficient corresponding to both the peak correlation value C _max and the frequency i is W _LOW (i, C _max ), And let W _HIGH (i, C _max ) be the weighting coefficient of the high-frequency emphasis type rule corresponding to both the peak correlation value C _max and the frequency i. In addition, the amplitude of the frequency i and f _i.

In this case, the weighting coefficient W _LOW (i, C _max ) of the low-frequency emphasis type rule is adopted for the frequency i satisfying the relationship of the following equation (1), and the amplitude is expressed as shown in the following equation (2). A value obtained by multiplying f _i by the weighting coefficient W _LOW (i, C _max ) of the low frequency emphasis type rule becomes the weighted amplitude f _iAFTER .

W _LOW (i, C _max ) <W _HIGH (i, C _max ) (1)
f _iAFTER = W _LOW (i, C _max ) × f _i (2)

On the other hand, for the frequency i that does not satisfy the relationship of the above formula (1), the weighting coefficient W _HIGH (i, C _max ) of the high frequency emphasis type rule is adopted, and as shown by the following formula (3), the amplitude A value obtained by multiplying f _i by the weighting coefficient W _HIGH (i, C _max ) of the high-frequency-oriented rule is the weighted amplitude f _iAFTER .

f _iAFTER = W _HIGH (i, C _max ) × f _i (3)

Here, as shown in FIG. 21, the weighting coefficient becomes zero and a certain amount from the highest frequency side in a certain frequency range from the lowest frequency side (hereinafter also referred to as “low frequency side limited range”) RL1. In the frequency range (hereinafter also referred to as “high frequency side limited range”) RH1, the weighting coefficient becomes zero. Then, if the amplitude f _i is multiplied by a zero weighting coefficient, the amplitude f _iAFTER becomes zero, and the frequency component including the amplitude f _iAFTER loses its substance.

  For this reason, the use of the frequency component information related to the frequency in the low frequency side restriction range RL1 in the correlation calculation is restricted by the weighting coefficient, and the frequency component information related to the frequency in the high frequency side restriction range RH1 is used in the correlation calculation. To be restricted. Therefore, here, the low frequency side limit range RL1 and the high frequency side limit range RH1 correspond to the use limit frequency range.

Further, here, the larger the peak correlation value C _max obtained by the correlation calculation in the previous stage is, the wider the low frequency side limit range RL1 is. The smaller the peak correlation value C _max is, the higher the frequency side limit range is. The limited use frequency range is set so that RH1 is expanded.

FIG. 22 is a diagram showing a weighting rule in which the low frequency side limit range RL1 is expanded and the high frequency side limit range RH1 is narrowed compared to the weighting rule shown in FIG. The weighting rule shown in FIG. 22 corresponds to the weighting rule in the case where the peak correlation value C _max obtained by the previous correlation calculation is larger than the weighting rule shown in FIG.

Note that the relationship between the weighting factors W _LOW (i, C _max ), W _HIGH (i, C _max ) and the peak correlation value C _max is stored in the storage unit 34 in the form of a table or a formulated relational expression. Such an embodiment is conceivable.

Further, the relationship between the frequency i and the weighting factors W _LOW (i, C _max ), W _HIGH (i, C _max ) is weighted with the change of the frequency i as shown by the curve LL1 drawn by the thick broken line in FIG. The coefficients W _LOW (i, C _max ) and W _HIGH (i, C _max ) may be continuously changed, or like a plurality of step-like line segments LH 1 drawn by bold lines in FIG. A relationship in which the weighting coefficients W _LOW (i, C _max ) and W _HIGH (i, C _max ) change stepwise every time the discrete threshold value is exceeded along with the change in the frequency i may be used.

For each frequency i, the relationship between the correlation value (here, the peak correlation value C _max ) and the weighting coefficient is set, and according to the correlation value (here, the peak correlation value C _max ) obtained in the previous stage. The weighting coefficient for each frequency i may be uniquely determined. The relationship between the correlation value and the weighting coefficient includes a linear relationship (FIG. 23), a curvilinear relationship (FIG. 24), and a relationship in which the weighting coefficient is switched step by step every time the correlation value reaches the threshold value. Any of (FIG. 25) may be sufficient.

<(2-1-8-3) Output from frequency component limiter>
In the frequency component limiting unit 368, the frequency component calculated by the frequency analysis unit 367 is multiplied by the weighting coefficient set for each frequency i, and the frequency component information indicating the weighted frequency component is obtained as the correlation calculation unit 369. Is output for. That is, for each frequency i, frequency component information including amplitude information of the weighted amplitude _fiAFTER as _represented by the above equations (2) and (3) is output.

At this time, if a zero weighting coefficient is multiplied to the amplitude f _{i of the} frequency i, the amplitude f _iAFTER becomes zero, and the substance of the frequency component is lost, so the frequency component information for the frequency i is not output. It will be.

  As a result, the frequency component information is limited so that part of the frequency component information generated by the frequency analysis unit 367 is not used in the correlation calculation in the correlation calculation unit 369.

  When the correlation value of the previous stage is not obtained, that is, a set of low resolution images (specifically, the third reduced reference image G13 and the third reduced reference image G23) in the lowest layer (third layer) When the processing target image pair is designated, the frequency component restriction unit 368 does not restrict the frequency component information. That is, the frequency component information obtained by the conversion process in the frequency analysis unit 367 is output to the correlation calculation unit 369 as it is.

<(2-1-9) Correlation calculator>
The correlation calculation unit 369 calculates a correlation value between the reference region and the comparison region based on the frequency component information related to the reference region and the frequency component information related to the comparison region output from the frequency component restriction unit 368. That is, in the correlation calculation unit 369, the correlation value between the reference region and the comparison region is calculated using the frequency component information limited by the multiplication of the weighting coefficient in the frequency component limiting unit 368.

  Then, in the correlation calculation unit 369, for each reference point Sp1, the correlation value between the reference region W13 of the third layer and the comparison region W23, the correlation value between the reference region W12 of the second layer and the comparison region W22, A correlation value between the reference region W11 and the comparison region W21 of the first layer and a correlation value between the reference region W1 and the comparison region W2 are sequentially calculated.

  As a correlation calculation method for calculating a correlation value in the correlation calculation unit 369, for example, the method “Phase-Differnce Spectrum Analysis (PSA method)” proposed in Non-Patent Document 1 can be adopted.

  Hereinafter, a correlation calculation method using the PSA method will be described.

  Here, with respect to the set of the reference region and the comparison region that are the objects of the correlation calculation, the phase difference (phase difference) of the sine wave of the frequency (u, v) between the reference region and the comparison region is Δθ (u, v) is set, and the amounts of shift (parallax) in the X direction and Y direction between the position of the reference point in the reference area and the position of the corresponding point in the comparison area are dx and dy. Then, when the number of pixels in the X direction and the Y direction constituting the rectangular reference region and the comparison region that are subject to correlation calculation is M and N, respectively, the relationship of the following expression (4) is established.

    dy = − (N / M) × (u / v) × dx + (N / 2π) × Δθ (u, v) / v (4)

  In the above equation (4), values M and N indicating the sizes of the reference region and the comparison region are determined by design, and the phase difference Δθ (u, v) for each frequency (u, v) is obtained from the frequency component limiter 368. It is obtained from input frequency component information. For this reason, design values M and N are substituted into the above equation (4), and the phase difference Δθ (u, v) is substituted for each frequency (u, v). A linear function (hereinafter also referred to as “parallax linear function”) indicating the relationship between the parallax dx and the parallax dy with respect to the frequency (u, v) is obtained.

For this reason, in the correlation calculation using the PSA method, the following steps (A) and (B) are sequentially performed in this order, so that a correlation value (here, peak correlation) between the reference region and the comparison region is obtained. Value C _max ) is obtained.

  (A) According to the above equation (4), for each frequency (u, v), the phase difference Δθ (u, v) is converted into the parallax (dx, dy) related to the reference point and the corresponding point, and the parallax primary A function is obtained.

(B) Based on the parallax linear function of each frequency (u, v) obtained in step (A), a space of a predetermined size (hereinafter also referred to as “voting space”) with parallax dx as the horizontal axis and parallax dy as the vertical axis. ) Is voted for each coordinate. Thereby, the distribution of the integrated value of the number of votes for each coordinate of the voting space is obtained. At this time, the maximum value of the integrated value of the number of votes is obtained as the peak correlation value _Cmax .

  Here, voting for each coordinate in the voting space will be described.

  FIG. 26 is a schematic diagram illustrating a voting space D369.

  As illustrated in FIG. 26, the voting space D369 is configured by a two-dimensional spatial coordinate system in which the vertical axis and the horizontal axis are orthogonal to each other, the horizontal axis indicates the parallax dx, and the vertical axis indicates the parallax dy. The size of the voting space D369 may be determined according to the number of frequencies (u, v) to be handled and the size of the area to be searched for corresponding points.

  Further, for example, when searching for parallax (dx, dy) within a range of ± 1 pixel in increments of 0.1 pixels, the voting space D369 can be voted in increments of 0.1 pixels including the origin (0, 0). A coordinate point (hereinafter also referred to as “votable coordinate point”) is set. Specifically, as shown in FIG. 26, 21 voting possible coordinate points are provided in a matrix in the vertical direction (dy direction) and 21 in the horizontal direction (dx direction). In FIG. 26, each voteable coordinate point is schematically shown as a square area.

  27 and 28 are diagrams for explaining a voting method for the voting space D369.

  As shown in FIG. 27, for each frequency (u, v), a straight line (thick line in the figure) indicated by the parallax linear function is drawn in the voting space D369. For each frequency (u, v), voting values are given to all voting possible coordinate points through which a straight line related to the parallax linear function passes among a large number of voting possible coordinate points constituting the voting space D369. For example, for a straight line related to the parallax linear function shown in FIG. 27, voting values are given to a plurality of voting possible coordinate points as shown in FIG.

  For each frequency (u, v), when a voting value is given to a voting possible coordinate point, a voting value corresponding to the magnitude of the amplitude is given. Specifically, the vote value may be increased as the amplitude increases. For example, the voting value may be increased as the amplitude increases, or a threshold value of 1 or more is provided for the amplitude, and the voting value every time the threshold is exceeded when the amplitude increases. However, the aspect which increases in steps with 1, 2, 3, ... may be sufficient.

Then, for each frequency (u, v), voting values are added to the voting space D369 (ie, voting) sequentially, so that the voting values are integrated for each voteable coordinate point. At this time, the frequency component information related to the frequency (u, v) whose amplitude becomes zero by weighting is not used for assigning a vote value to the vote space D369. That is, the calculation of the correlation value indicating the correlation between the reference area and the comparison area of a certain resolution is set according to the peak correlation value C _max in the previous stage among the frequency component information of the reference area and the comparison area of the resolution. This is performed based on frequency component information of a frequency that does not belong to the use limited frequency range.

By providing such voting values, a distribution of voting value integrated values (hereinafter also referred to as “voting integrated values”) in the voting space D369 is obtained. The distribution of vote integrated values obtained here indicates the correlation between the base image and the reference image, and the maximum vote integrated value is the peak correlation value C _max . In addition, the voting integrated value given to the voteable coordinate point near the point where the straight lines related to the parallax linear function frequently intersect is the peak correlation value _Cmax .

In the correlation calculation unit 369, information indicating the distribution of vote integrated values in the voting space D369 is output to the corresponding point determination unit 370, and information indicating the peak correlation value C _max is output to the frequency component restriction unit 368. Is output.

<(2-1-10) Corresponding point determination unit>
Corresponding point determination unit 370 corresponds to the reference point in the reference region and the comparison region based on the information indicating the distribution of vote integrated values obtained by correlation calculation unit 369 for the pair of the reference region and the comparison region of each resolution. The amount of deviation (parallax) from the point is derived. Specifically, the parallax (dx, dy) corresponding to the maximum vote integrated value, that is, the peak correlation value C _max in the distribution of vote integrated values is derived. Then, the corresponding point determination unit 370 recognizes corresponding points corresponding to the respective reference points Sp11 to Sp13, and each reference point Sp1 based on the parallax (dx, dy) between the reference region W1 and the comparison region W2. The corresponding point on the reference image G2 corresponding to is determined.

  For example, the parallax between the reference area W13 and the comparison area W23 related to the low resolution of the third layer (lowermost layer) is derived. Further, the parallax between the reference area W12 and the comparison area W22 related to the low resolution of the second layer is derived. Further, the parallax between the reference area W11 and the comparison area W21 related to the low resolution of the first layer is derived. Then, the parallax between the reference area W1 and the comparison area W2 is derived. At this time, a corresponding point on the reference image G2 corresponding to the reference point Sp1 is determined.

Here, the peak correlation value C _max corresponds to a correlation value indicating the correlation between the reference point and the corresponding point from which the peak correlation value C _max is obtained for each resolution.

  Note that the parallax derived by the corresponding point determination unit 370 is appropriately output to the reference parallax setting unit 365 in order to set the next-stage reference parallax. Further, in response to the parallax being derived by the corresponding point determination unit 370, a signal is transmitted to the image designation unit 364 and the search reference point setting unit 363.

<(3) Corresponding point search operation flow>
FIGS. 29 to 31 are flowcharts illustrating the flow of corresponding point search operations realized in the information processing system 1A. For example, this operation flow is started in response to an operation on the operation unit 31 by the user, and the process proceeds to step ST1 in FIG.

  In step ST1, the image acquisition unit 361 acquires a stereo image composed of the base image G1 and the reference image G2.

  In step ST2, the resolution conversion unit 362 performs resolution conversion processing on the standard image G1 and the reference image G2. Specifically, first to third reduced reference images G11 to G13 with a reduced resolution are generated from the reference image G1, and first to third reduced reference images G21 to G23 with a reduced resolution are generated from the reference image G2. Generated.

  In step ST3, the reference point Sp1 is set for the reference image G1 by the search reference point setting unit 363. In step ST3, every time a corresponding point is detected from the reference image G2 for one reference point Sp1, the process returns from step ST22 of FIG. 31 described later, and a new reference point Sp1 is set. .

  In step ST4, the image specifying unit 364 converts the lower-layer (third layer) low-resolution image set (specifically, the third reduced reference image G13 and the third reduced reference image G23) into a processing target image pair. Specified as

  In step ST5, the reference parallax setting unit 365 sets a reference point Sp13 corresponding to the reference point Sp1 on the third reduced reference image G13, and sets a processing target point Pp23 on the third reduced reference image G23. . That is, the initial value of the reference parallax is set for the processing target image pair specified in step ST4.

  In step ST6, the window setting unit 366 sets a reference area W13 including the reference point Sp13 as a center for the third reduced reference image G13, and also sets a processing target point for the third reduced reference image G23. A comparison area W23 including Pp23 as a center is set.

  In step ST7, the frequency analysis unit 367 converts the reference region W13 set in step ST6 into frequency component information and the comparison region W23 set in step ST6 into frequency component information. The frequency component information obtained here is directly output to the correlation calculation unit 369 via the frequency component limiting unit 368.

In step ST8, the correlation calculation unit 369 performs a correlation calculation based on the frequency component information related to the reference region W13 and the frequency component information related to the comparison region W23 obtained in step ST7. By this correlation calculation, a distribution of vote integrated values is obtained, and a peak correlation value C _max indicating a correlation between the reference region W13 and the comparison region W23 is calculated.

In step ST9, the corresponding point determination unit 370 derives the parallax (dx, dy) corresponding to the peak correlation value _Cmax based on the distribution of the vote integrated values obtained in step ST8, and the reference point Sp13. Corresponding points on the corresponding third reduced reference image G23 are recognized. When the process of step ST9 is completed, the process proceeds to step ST11 of FIG.

  In step ST11 of FIG. 30, the image designation unit 364 designates a set of images belonging to a layer having a resolution higher than that of the previous processing target image pair as the processing target image pair.

  For example, if the processing target image pair in the previous stage is a third reduced standard image G13 and a third reduced reference image G23 that are a set of images with a third layer resolution, the second set of images with a resolution of the second layer. A 2 reduced standard image G12 and a second reduced reference image G22 are designated. Further, if the processing target image pair in the previous stage is the second reduced reference image G12 and the second reduced reference image G22 that are a set of images of the second layer resolution, the first set of images of the resolution of the first layer. One reduced standard image G11 and a first reduced reference image G21 are designated. Furthermore, if the first-stage processing target image pair is the first reduced standard image G11 and the first reduced reference image G21 that are sets of images of the first layer resolution, the standard image G1 and the reference image G2 are designated. .

  In step ST12, the reference parallax setting unit 365 sets the reference parallax for the processing target image pair specified in the latest step ST11. Here, the reference parallax corresponding to the previous stage parallax recognized in step ST9 or step ST17 is set.

  For example, a reference point Sp12 corresponding to the reference point Sp1 is set for the second reduced reference image G12 according to the parallax recognized for the resolution of the third layer, which is the previous stage, and the second reduced reference image G22. The processing target point Pp22 is set for. In addition, a reference point Sp11 corresponding to the reference point Sp1 is set for the first reduced reference image G11 according to the parallax recognized for the resolution of the second layer, which is the previous stage, and the first reduced reference image G21. The processing target point Pp21 is set for. Furthermore, a processing target point Pp2 based on the reference point Sp1 of the reference image G1 is set on the reference image G2 according to the parallax recognized for the resolution of the first layer, which is the previous stage.

  In step ST13, the window setting unit 366 uses the reference point set in the latest step ST12 and the processing target point as a reference for the processing target image pair specified in the latest step ST11, and compares it with the reference region. The area is set.

  For example, if the processing target image pair specified in the latest step ST11 is a set of the second reduced reference image G12 and the second reduced reference image G22, the reference point Sp12 set in the latest step ST12 and the processing target The reference area W12 is set for the second reduced reference image G12 and the comparison area W22 is set for the second reduced reference image G22 with the point Pp22 as a reference. Further, if the processing target image pair designated in the latest step ST11 is a set of the first reduced reference image G11 and the first reduced reference image G21, the reference point Sp11 set in the latest step ST12 and the processing target are set. Using the point Pp21 as a reference, a reference area W11 is set for the first reduced reference image G11, and a comparison area W21 is set for the first reduced reference image G21. Further, if the processing target image pair specified in the latest step ST11 is a set of the standard image G1 and the reference image G2, the reference point Sp1 and the processing target point Pp2 set in the latest step ST12 are used as the reference. Thus, the reference area W1 is set for the reference image G1, and the comparison area W2 is set for the reference image G2.

  In step ST14, the frequency analysis unit 367 converts the reference region set in the latest step ST13 into frequency component information, and converts the comparison region set in the latest step ST13 into frequency component information.

  For example, the reference region W12 set in the latest step ST13 is converted into frequency component information, and the comparison region W22 set in the latest step ST13 is converted into frequency component information. Further, the reference region W11 set in the latest step ST13 is converted into frequency component information, and the comparison region W21 set in the latest step ST13 is converted into frequency component information. Further, the reference region W1 set in the latest step ST13 is converted into frequency component information, and the comparison region W2 set in the latest step ST13 is converted into frequency component information.

In step ST15, the frequency component restriction unit 368 restricts the frequency component information generated in the latest step ST14 according to a weighting rule corresponding to the peak correlation value _Cmax obtained in the previous correlation calculation. Here, based on the peak correlation value C _max obtained in the previous correlation calculation, the use limited frequency range in which the use of the frequency component information in the correlation calculation is limited (for example, the low frequency side limited range RL1 and the high frequency side limited range RH1). Etc.) is set. Then, the frequency component information generated in the latest step ST14 is limited in accordance with the use limited frequency range, and then output to the correlation calculation unit 369.

  For example, if the reference region W12 and the comparison region W22 are set in the latest step ST13, the frequency component information related to the reference region W12 and the frequency component information related to the comparison region W22 are limited according to the use limited frequency range. It is output after. Further, if the reference region W11 and the comparison region W21 are set in the most recent step ST13, the frequency component information related to the reference region W11 and the frequency component information related to the comparison region W21 are limited according to the use restricted frequency range. It is output after. If the reference region W1 and the comparison region W2 are set in the most recent step ST13, the frequency component information related to the reference region W1 and the frequency component information related to the comparison region W2 are limited according to the use limited frequency range. Is output.

In step ST16, based on the frequency component information of the frequency outside the use limited frequency range output from the frequency component limiting unit 368 in step ST15 by the correlation calculation unit 369, the reference region and the comparison region set in the most recent step ST13 The correlation calculation is performed. By this correlation calculation, a distribution of vote integrated values is obtained, and a peak correlation value C _max indicating a correlation between the reference region and the comparison region is calculated.

For example, if frequency component information of frequencies outside the use restriction frequency range related to the reference region W12 and the comparison region W22 is output from the frequency component restriction unit 368 in the most recent step ST15, the reference region W12 and the comparison region W22 The distribution of vote integrated values and the peak correlation value C _max are obtained. In addition, if frequency component information of frequencies outside the use restriction frequency range related to the reference region W11 and the comparison region W21 is output from the frequency component restriction unit 368 in the latest step ST15, the reference region W11 and the comparison region W21 are output. The distribution of vote integrated values and the peak correlation value C _max are obtained. Further, if frequency component information of frequencies outside the use restriction frequency range related to the reference region W1 and the comparison region W2 is output from the frequency component restriction unit 368 in the most recent step ST15, the reference region W1 and the comparison region W2 are output. The distribution of vote integrated values and the peak correlation value C _max are obtained.

In step ST17, the corresponding point determination unit 370 derives the parallax (dx, dy) corresponding to the peak correlation value _Cmax based on the distribution of vote integrated values obtained in step ST16, and the latest step ST11. Corresponding points corresponding to the reference points are recognized for the processing target image pair specified in step (b).

  For example, if the processing target image pair designated in the most recent step ST11 is a set of the second reduced reference image G12 and the second reduced reference image G22, it corresponds to the reference point Sp12 on the second reduced reference image G12. Corresponding points on the second reduced reference image G22 are recognized. Further, if the processing target image pair specified in the most recent step ST11 is a set of the first reduced reference image G11 and the first reduced reference image G21, it corresponds to the reference point Sp11 on the first reduced reference image G11. Corresponding points on the first reduced reference image G21 are recognized. If the processing target image pair specified in the most recent step ST11 is a set of the standard image G1 and the reference image G2, the corresponding point on the reference image G2 corresponding to the standard point Sp1 on the standard image G1 is recognized. Is done.

  In step ST18, the corresponding point determination unit 370 determines whether or not there is a set of image layers whose resolution is one level higher than the processing target image pair specified in the most recent step ST11. Here, if there is a set of images of the layer one level higher than the resolution, the process proceeds to step ST11, and if there is no set of images of the layer one level higher than the resolution, the process proceeds to step ST19.

  For example, if the processing target image pair designated in the most recent step ST11 is a set of the second reduced reference image G12 and the second reduced reference image G22, the resolution corresponds to a set of images in the layer one level higher. There is a set of the first reduced standard image G11 and the first reduced reference image G21. In addition, if the processing target image pair designated in the most recent step ST11 is a set of the first reduced reference image G11 and the first reduced reference image G21, the resolution corresponds to a set of images in the layer one level higher. There is a set of a reference image G1 and a reference image G2. On the other hand, if the processing target image pair specified in the most recent step ST11 is a set of the standard image G1 and the reference image G2, there is no set of images of the layer whose resolution is one level higher.

  In step ST19, the corresponding point determination unit 370 determines the corresponding point on the reference image G2 recognized in the latest step ST17 as the corresponding point corresponding to the reference point Sp1 set in the latest step ST3. When the process of step ST19 is completed, the process proceeds to step ST21 of FIG.

  In step ST21 of FIG. 31, the control unit 36 associates the reference point Sp1 set in the most recent step ST3 with the corresponding point determined in the most recent step ST19 (FIG. 30) and stores it in the storage unit 34. Is done.

  In step ST22, the search reference point setting unit 363 determines whether all the pixels of the reference image G1 have already been set as the reference points Sp1 in step ST3. If all the pixels of the reference image G1 have not been set as the reference point Sp1, the process proceeds to step ST3 in FIG. 29, and the next reference point Sp1 is set. On the other hand, if all the pixels of the reference image G1 have already been set as the reference point Sp1, this operation flow is ended.

<(4) Summary of one embodiment>
As described above, according to the information processing system 1A according to the embodiment, based on the correlation value indicating the degree of similarity between the plurality of images, the plurality of different resolutions respectively corresponding to the plurality of images. The amount of correlation calculation between images is reduced. For this reason, both maintaining accuracy and improving speed in the corresponding point search processing for a plurality of images are compatible.

  As used herein, “another plurality of images” includes, for example, a plurality of image sets in which the resolutions are different from the plurality of image sets. However, it is not limited to this. The “different plural images” may be, for example, stereo images taken by the stereo camera 2 at slightly different times from the original plural images.

<(5) Modification>
It should be noted that the present invention is not limited to the above-described embodiment, and various changes and improvements can be made without departing from the gist of the present invention.

<(5-1) First modification>
For example, in the above-described embodiment, the corresponding point search operation between the standard image G1 obtained by imaging with the stereo camera 2 and the reference image G2 has been described, but the present invention is not limited thereto. For example, the technical idea of the present invention can also be applied to a single-eye camera, that is, a corresponding point search operation between a plurality of images that are continuously obtained in time by imaging from one viewpoint. Hereinafter, a specific example will be described.

<(5-1-1) Configuration of information processing system according to first modification>
FIG. 32 is a diagram illustrating a schematic configuration of an information processing system 1B according to the first modification, and FIG. 33 is a block diagram illustrating a main configuration of the information processing system 1B.

  In the information processing system 1B, the stereo camera 2 is replaced with a monocular camera 2B as compared with the information processing system 1A according to the above-described embodiment, and the other configuration is the same.

  However, in the information processing system 1B, n (GB is a natural number of 3 or more) images GB1 to GBn having different imaging times are acquired by sequential imaging by the camera 2B. Then, corresponding point search processing is performed between the n images GB1 to GBn. For this reason, various functions, various information processing, and the like realized by the control unit 36 are slightly different from the various functions, various information processing, and the like that are executed by the control unit 36 according to the embodiment.

  Specifically, as illustrated in FIG. 33, in the information processing system 1B, a program PGb different from the program PGa according to the above-described embodiment is stored in the storage unit 34. The program PGb is read and executed by the control unit 36, thereby realizing various functions and various information processing of the corresponding point search operation according to the present modification.

  The camera 2B, for example, has a fixed position and orientation with respect to the ground, and performs n time-sequential imaging to capture n images GB1 that capture temporal changes in the same subject OBB from the same viewpoint. Obtain ~ GBn.

  For example, if the camera 2B is constituted by a so-called high speed camera or the like, a large number of frame images can be obtained in a short time. At this time, even if the subject OBB is moving, a portion where the same subject OBB is captured is slightly shifted between temporally obtained frame images.

  The n images GB1 to GBn acquired in time sequence by the camera 2B are so-called time-series images, which are transmitted to the information processing device 3 via the data line CB and stored in the storage unit 34. .

  In the following, images obtained by imaging at time T1, time T2, time T3,..., Time Tn with the camera 2B are respectively referred to as “T1 frame image” GB1, “T2 frame image” GB2, and “T3 frame image” GB3. , ..., referred to as “Tn frame image” GBn.

  FIG. 34 is a schematic view illustrating the form of the T1 frame image GB1, FIG. 35 is a schematic view illustrating the form of the T2 frame image GB2, and FIG. 36 is a schematic view illustrating the form of the T3 frame image GB3. FIG. FIG. 34 to FIG. 36 show a state in which the image area (the hatched area in the figure) capturing the subject OBB is gradually shifted to the right.

  Further, as shown in FIG. 4, the T1 to Tn frame images GB1 to GBn are, for example, a large number of pixels arranged in a matrix form, similarly to the first and second captured images G1 and G2 according to the one embodiment. Configured. Specifically, a first predetermined number (here, 480) of pixels are arranged in the vertical direction (Y direction), and a second predetermined number (here, 640) of pixels are arranged in the horizontal direction (X direction). Arranged.

  Here, in the T1 to Tn frame images GB1 to GBn, as in the first and second captured images G1 and G2 according to the above-described embodiment, the upper left position is the origin, and the T1 to Tn frame images GB1 to GBn The horizontal position of each pixel is indicated by the X coordinate, and the vertical position is indicated by the Y coordinate. That is, in the T1 to Tn frame images GB1 to GBn, as in the first and second captured images G1 and G2 according to the one embodiment, the position of each pixel is indicated by XY coordinates (x, y). When the pixel shifts in the right direction (X direction), the value of the X coordinate increases by one, and when the pixel shifts by one pixel in the downward direction (Y direction), the value of the Y coordinate increases by one.

<(5-1-2) Functional configuration related to corresponding point search operation according to first modification>
FIG. 37 is a diagram illustrating a functional configuration realized by the control unit 36 in order to execute the corresponding point search operation of the present modification. Here, the functional configuration of the control unit 36 is described as being realized by executing the program PGb, but may be realized by a dedicated hardware configuration.

  As shown in FIG. 37, the control unit 36 has an image acquisition unit 361B, a search reference point setting unit 363B, a reference parallax setting unit 365B, a window setting unit 366B, a frequency analysis unit 367B, a frequency component restriction, as a functional configuration. Unit 368B, correlation calculation unit 369B, and corresponding point determination unit 370B.

<(5-1-2-1) Image Acquisition Unit According to First Modification>
The image acquisition unit 361B obtains a set of images (processing target image) to be searched for corresponding points from the T1 to Tn frame images GB1 to GBn acquired sequentially by the camera 2B and stored in the storage unit 34. Get a pair). Here, assuming that m is a natural number, when a corresponding point is recognized for the processing target image pair including the Tm frame image GBm and the T (m + 1) frame image GB (m + 1), the T (m + 1) frame image GB (m + 1) And a T (m + 2) frame image GB (m + 2) is acquired as the next processing target image pair.

  For example, first, a set of a T1 frame image GB1 and a T2 frame image GB2 is acquired as a processing target image pair. Next, a set of the T2 frame image GB2 and the T3 frame image GB3 is acquired as a processing target image pair. Next, a set of the T3 frame image GB3 and the T4 frame image GB4 is acquired as a processing target image pair.

  In the following description, the Tm frame image GBm and the Tm frame image GBm are defined with reference to the corresponding point search processing for the processing target image pair including the T (m + 1) frame image GB (m + 1) and the T (m + 2) frame image GB (m + 2). The corresponding point search process for the processing target image pair including the T (m + 1) frame image GB (m + 1) is referred to as a previous corresponding point search process. On the contrary, T (m + 1) frame image GB (m + 1) and T (m) are used with reference to the corresponding point search processing for the processing target image pair consisting of Tm frame image GBm and T (m + 1) frame image GB (m + 1). m + 2) Corresponding point search processing for a processing target image pair including the frame image GB (m + 2) is referred to as next-step corresponding point search processing.

  In the corresponding point search operation, when the corresponding point search process is performed between the T1 frame image GB1 and the T2 frame image GB2, the T1 frame image GB1 is used as a reference image (reference image) GB1, and T2 The frame image GB2 is referred to as an image (reference image) GB2. When the corresponding point search process is performed between the T2 frame image GB2 and the T3 frame image GB3, the T2 frame image GB2 is used as a reference image (reference image) GB2, and the T3 frame image GB3 is referred to. Image (reference image) GB3.

  That is, when the corresponding point search process is performed between the Tm frame image GBm and the T (m + 1) frame image GB (m + 1), the Tm frame image GBm is set as a reference image (reference image) GBm. The (m + 1) frame image GB (m + 1) is referred to as an image (reference image) GB (m + 1).

<(5-1-2-2) Search Reference Point Setting Unit According to First Modification>
Similarly to the search reference point setting unit 363 according to the above-described embodiment, the search reference point setting unit 363B sets a point (reference point) SpB1 (FIG. 8) serving as a reference for the corresponding point search for the T1 frame image GB1. Set.

  For example, first, the upper left pixel in the T1 frame image GB1 is set as the reference point SpB1. Next, the reference image GB1 and the reference image GB2, the reference image GB2 and the reference image GB3, the reference image GB3 and the reference image GB4,..., The reference image GB (n−1) and the reference. Corresponding points corresponding to the reference point SpB1 are detected from the T2 to Tn frame images GB2 to GBn in this order by sequentially performing the corresponding point searching process with the image GBn. Each time a corresponding point is detected from the Tn frame image GBn, a new reference point SpB1 is set on the T1 frame image GB1.

<(5-1-2-3) Reference Parallax Setting Unit According to First Modification>
Similarly to the reference parallax setting unit 365 according to the above-described embodiment, the reference parallax setting unit 365B temporarily sets a parallax (reference parallax) serving as a reference for searching for corresponding points for each processing target image pair.

  Specifically, an initial value of the standard parallax is provisionally set for a set of a T1 frame image (standard image) GB1 and a T2 frame image (reference image) GB2. As an initial value of the reference parallax, for example, a predetermined value (for example, zero) in the X and Y directions is set.

  For example, a reference point SpB1 (FIG. 13) is set for the T1 frame image (reference image) GB1, and a processing target point PpB2 (FIG. 14) is set for the T2 frame image (reference image) GB2. Here, the amount of deviation between the coordinates of the reference point SpB1 in the T1 frame image (reference image) GB1 and the coordinates of the processing point PpB2 in the T2 frame image (reference image) GB2 corresponds to the initial value of the reference parallax.

  Next, the parallax obtained by the corresponding point search process using the pair of the T1 frame image (reference image) GB1 and the T2 frame image (reference image) GB2 as the processing target image pair is the T2 frame image as the next processing target image pair. A (standard image) GB2 and a T3 frame image (reference image) GB3 are set temporarily as a standard parallax.

  For example, with respect to the T2 frame image (reference image) GB2, the corresponding points obtained by the corresponding point search process in the previous stage are set as the reference point SpB2 (FIG. 13), and the T3 frame image (reference image) GB3 is set. On the other hand, the processing target point PpB3 (FIG. 14) is set. Here, the disparity between the coordinates of the reference point SpB2 in the T2 frame image (reference image) GB2 and the coordinates of the processing target point PpB3 in the T3 frame image (reference image) GB3 is obtained by the corresponding point search process in the previous stage. It corresponds to.

  Then, the parallax obtained by the corresponding point search processing using the set of the T (m−1) frame image (reference image) GB (m−1) and the Tm frame image (reference image) GBm as the processing target image pair is A reference parallax is temporarily set for a set of a Tm frame image (reference image) GBm and a T (m + 1) frame image (reference image) GB (m + 1) as a processing target image pair.

  For example, with respect to the Tm frame image (reference image) GBm, the corresponding point obtained by the corresponding point search process in the previous stage is set as the reference point SpBm (FIG. 13) and the T (m + 1) frame image (reference image). ) A processing target point PpB (m + 1) (FIG. 14) is set for GB (m + 1). Here, the amount of deviation between the coordinates of the reference point SpBm in the Tm frame image (reference image) GBm and the coordinates of the processing target point PpB (m + 1) in the T (m + 1) frame image (reference image) GB (m + 1) is This corresponds to the parallax obtained by the corresponding point search process.

<(5-1-2-4) Window Setting Unit According to First Modification>
Similarly to the window setting unit 366 according to the above-described embodiment, the window setting unit 366B sets a window (reference region) WBm including the reference point SpBm as a center in the Tm frame image (reference image) GBm, and T ( m + 1) A window (comparison area) WB (m + 1) including the processing target point Pp (m + 1) as a center in the frame image (reference image) GB (m + 1) is set. The reference area WBm and the comparison area WB (m + 1) are square image areas of the same size, and are configured, for example, by arranging 32 pixels in the vertical direction and the horizontal direction, respectively.

<(5-1-2-5) Frequency analysis unit according to first modification>
Similar to the frequency analysis unit 367 according to the above-described embodiment, the frequency analysis unit 367B executes processing (conversion processing) for converting an image into information indicating a frequency component related to a sine wave. This conversion process is performed for each reference area WBm and also for each comparison area WB (m + 1). Note that the conversion process in the frequency analysis unit 367B is the same process as the conversion process in the frequency analysis unit 367 according to the above-described embodiment, and thus the description thereof is omitted here.

<(5-1-2-6) Frequency Component Limiting Unit According to First Modification>
Similarly to the frequency component limiting unit 368 according to the above embodiment, the frequency component limiting unit 368B limits the frequency component information generated by the frequency analyzing unit 367B according to a predetermined limiting rule, and then performs a correlation operation on the correlation calculation unit 369B. Output. Thereby, in the correlation calculation for obtaining the correlation value between the reference region WBm and the comparison region WB (m + 1) in the correlation calculation unit 369B, a part of the frequency components in the frequency component information generated by the frequency analysis unit 367B. Information is not used, and the remaining part of the frequency component information is selectively used.

Specifically, in the frequency component limiting unit 368B, as in the frequency component limiting unit 368 according to the above-described embodiment, the frequency component information generated by the frequency analyzing unit 367B is the peak correlation value obtained by the previous correlation calculation. It is limited according to a weighting rule according to _Cmax . Further, the frequency component restriction unit 368B determines the frequency range of the frequency component information (use limited frequency range) that is restricted in use in the correlation calculation in the correlation calculation unit 369B by determining the weighting coefficient for each frequency according to the weighting rule. ) Is set.

  The detailed frequency component limiting method and the processing according to the limiting method in the frequency component limiting unit 368B are the same as the detailed frequency component limiting method and the limiting method in the frequency component limiting unit 368 according to the above embodiment. Since the processing is the same as that described above, description thereof is omitted here.

<(5-1-2-7) Correlation Calculation Unit According to First Modification>
Correlation calculation unit 369B, similarly to correlation calculation unit 369 according to the above embodiment, frequency component information related to reference region WBm and frequency component information related to comparison region WB (m + 1) output from frequency component limiting unit 368B Based on the above, information indicating the distribution of vote integrated values for the reference area WBm and the comparison area WB (m + 1) is obtained, and a correlation value between the reference area WBm and the comparison area WB (m + 1) is calculated. That is, the correlation calculation unit 369B uses a part of the frequency component information obtained by being limited by multiplication of the weighting coefficient in the frequency component limiting unit 368B among the frequency component information generated by the frequency analysis unit 367B. Thus, a correlation value between the reference area WBm and the comparison area WB (m + 1) is calculated.

  Note that the calculation processing in the correlation calculation unit 369B is the same as the calculation processing in the correlation calculation unit 369 according to the above-described embodiment, and thus description thereof is omitted here.

<(5-1-2-8) Corresponding point determination unit according to first modification>
Corresponding point determination unit 370B, similarly to corresponding point determination unit 370 according to the above-described embodiment, for the set of reference region WBm and comparison region WB (m + 1), the voting integrated value obtained by correlation calculation unit 369B Based on the information indicating the distribution, a deviation amount (parallax) between the reference point SpBm in the reference region WBm and the corresponding point in the comparison region WB (m + 1) is derived. Then, the corresponding point determination unit 370B determines the corresponding point corresponding to the reference point SpB1 of the T1 frame image GB1 from each of the frame images GB2 to GBn based on the derived parallax.

  In addition, since the specific process in the corresponding point determination part 370B is the same as the process in the corresponding point determination part 370 which concerns on the said one Embodiment, description is abbreviate | omitted here.

<(5-1-3) Corresponding point search operation flow according to the first modification>
38 to 40 are flowcharts illustrating the flow of corresponding point search operations realized in the information processing system 1B. For example, this operation flow is started in response to an operation on the operation unit 31 by the user, and the process proceeds to Step SS1 in FIG.

  In step SS1, the search reference point setting unit 363B sets a reference value j to 1 when the reference point is set.

  In step SS2, the image acquisition unit 361B sets 1 as a reference value m when a processing target image pair is designated.

  In step SS3, the image acquisition unit 361B acquires the T1 frame image GB1 and the T2 frame image GB2 from the storage unit 34 as a processing target image pair according to the numerical value m (= 1) set in step SS2. At this time, the T1 frame image GB1 is set as the reference image GB1, and the T2 frame image GB2 is set as the reference image GB2.

  In step SS4, the search reference point setting unit 363B performs the jth operation on the T1 frame image (reference image) GB1 acquired in step SS3 according to the numerical value j most recently set in step SS1 or step SS23 (FIG. 40). The reference point SpB1 is set.

  In step SS5, the reference parallax setting unit 365B sets the processing point PpB2 for the T2 frame image (reference image) GB2 acquired in step SS3. That is, the reference parallax is set to the initial value.

  In step SS6, the window setting unit 366B sets a reference area WB1 including the reference point SpB1 as a center for the T1 frame image (reference image) GB1, and also sets the T2 frame image (reference image) GB2. A comparison area WB2 including the processing target point PpB2 as a center is set.

  In step SS7, the frequency analysis unit 367B converts the reference region WB1 of the T1 frame image (reference image) GB1 into frequency component information, and converts the comparison region WB2 of the T2 frame image (reference image) GB2 into frequency component information. Is done.

In step SS8, the correlation calculation unit 369B performs a correlation calculation based on the frequency component information related to the reference region WB1 and the frequency component information related to the comparison region WB2 obtained in step SS7. By this correlation calculation, a distribution of vote integrated values is obtained, and a peak correlation value C _max indicating a correlation between the reference area WB1 and the comparison area WB2 is calculated.

In step SS9, the corresponding point determination unit 370B derives the parallax (dx, dy) corresponding to the peak correlation value C _max based on the distribution of the vote integrated values obtained in step SS8, and also determines the reference point SpB1. Corresponding points on the corresponding reference image GB2 are recognized. When the process of step SS9 is completed, the process proceeds to step SS11 of FIG.

  In step SS11 of FIG. 39, the image acquisition unit 361B increments the numerical value m that is a reference when specifying the processing target image pair by one.

  In step SS12, the image acquisition unit 361B acquires the Tm frame image GBm and the T (m + 1) frame image GB (m + 1) from the storage unit 34 as a processing target image pair according to the numerical value m set in step SS11. At this time, the Tm frame image GBm is set as the reference image GBm, and the T (m + 1) frame image GB (m + 1) is set as the reference image GB (m + 1). Note that the Tm frame image GBm that has already been acquired from the storage unit 34 for use in the previous correlation calculation may be used as it is.

  In step SS13, the search reference point setting unit 363B sets the corresponding point on the Tm frame image GBm most recently determined in step SS9 (FIG. 38) or step SS19 (FIG. 39) as the reference point SpBm.

  In step SS14, the reference parallax setting unit 365B sets the processing target point PpB (m + 1) for the T (m + 1) frame image (reference image) GB (m + 1) acquired in step SS12. That is, the reference parallax is set.

  In step SS15, the window setting unit 366B sets a reference area WBm that includes the reference point SpBm as a center for the Tm frame image (reference image) GBm, and a T (m + 1) frame image (reference image) GB. A comparison area WB (m + 1) including the processing target point PpB (m + 1) as a center is set for (m + 1).

  In step SS16, the frequency analysis unit 367B converts the reference area WBm of the Tm frame image (reference image) GBm into frequency component information, and also compares the comparison area WB of the T (m + 1) frame image (reference image) GB (m + 1). (m + 1) is converted into frequency component information.

In step SS17, the frequency component restriction unit 368B restricts the frequency component information generated in the latest step SS16 according to a weighting rule corresponding to the peak correlation value _Cmax obtained in the previous correlation calculation. Here, based on the peak correlation value C _max obtained in the previous correlation calculation, the use limited frequency range in which the use of the frequency component information in the correlation calculation is limited (for example, the low frequency side limited range RL1 and the high frequency side limited range RH1). Etc.) is set. Then, the frequency component information generated in the latest step SS16 is limited according to the use limited frequency range, and then output to the correlation calculation unit 369B.

In step SS18, the correlation calculation unit 369B performs a correlation calculation based on the frequency component information related to the reference region WBm and the frequency component information related to the comparison region WB (m + 1), a part of which is limited in step SS17. . By this correlation calculation, a distribution of vote integrated values is obtained, and a peak correlation value C _max indicating a correlation between the reference area WBm and the comparison area WB (m + 1) is calculated.

In step SS19, the corresponding point determination unit 370B derives the parallax (dx, dy) corresponding to the peak correlation value C _max based on the distribution of the vote integrated values obtained in step SS18, and the reference point SpBm. Corresponding points on the corresponding reference image GB (m + 1) are determined.

  In step SS20, the image acquisition unit 361B determines whether or not the numerical value (m + 1) has reached the number n of frame images to be subjected to corresponding point search processing. Here, if the numerical value (m + 1) does not reach the number n, the process proceeds to step SS11, and the processes of steps SS11 to SS20 are repeated until the numerical value (m + 1) reaches the number n. Thereby, the corresponding points corresponding to one reference point Sp1 set in step SS4 are determined on each of the T2 to Tn frame images GB2 to GBn. On the other hand, if the numerical value (m + 1) has reached the number n, the process proceeds to step SS21 in FIG.

  In step SS21 of FIG. 40, the control unit 36 associates the reference point Sp1 set in the most recent step SS4 with each corresponding point determined in step SS19 (FIG. 39) and stores it in the storage unit 34. The

  In step SS22, the search reference point setting unit 363B determines whether all pixels of the T1 frame image (reference image) GB1 have already been set as the reference point SpB1 in step SS4. If all the pixels of the T1 frame image (reference image) GB1 have not yet been set as the reference point SpB1, the process proceeds to step SS23. On the other hand, if all the pixels of the T1 frame image (reference image) GB1 have already been set as the reference point SpB1, this operation flow ends.

  In step SS23, the search reference point setting unit 363B increments the reference value j when setting the reference point by one, and proceeds to step SS2 in FIG. Thereby, the corresponding points corresponding to the reference points SpB1 set on the T1 frame image GB1 are determined on the T2 to Tn frame images GB2 to GBn, respectively.

<(5-1-4) Summary of the first modification>
As described above, according to the information processing system 1B according to the first modification, the similarity between the T (m−1) frame image GB (m−1) and the Tm frame image GBm as the processing target image pair is similar. Based on the correlation value indicating the degree, the amount of correlation calculation between the Tm frame image GBm and the T (m + 1) frame image GB (m + 1) as the next processing target image pair is reduced. For this reason, both maintaining accuracy and improving speed in the corresponding point search processing for a plurality of images are compatible.

  In the present modification, the corresponding point search operation for time series images (specifically, T1 to Tn frame images GB1 to GBn) acquired sequentially in time by the camera 2B has been described as an example. Not limited to this. For example, the corresponding point search operation may be performed on three or more images capturing the same type of subject. However, considering the ease of setting the reference parallax and the like, it is preferable to use a time-series image for the corresponding point search operation because the degree of similarity between the images is high.

  Further, in the present modification, the corresponding point search process between the two images constituting the processing target image pair is simply performed, but the present invention is not limited to this. For example, when the movement of the subject captured in the time-series images (specifically, the T1 to Tn frame images GB1 to GBn) is large or the movement is not regular, each processing target image pair As in the above-described embodiment, it is more accurate to generate a set of images whose resolution is gradually reduced and perform the corresponding point search process sequentially from the low-resolution image set. It is preferable for maintaining the above.

<(5-2) Other variations>
In the above-described embodiment, the weighting coefficient for each frequency and the use-restricted frequency range are set according to the correlation value (for example, peak correlation value C _max ) obtained in the corresponding point search process in the previous stage. Not limited to this. For example, on the basis of a search process for a certain corresponding point, a correlation value obtained by a search process for corresponding points at one or more stages before the previous stage (for example, the previous stage) is also used, and a weighting coefficient for each frequency or A limited use frequency range may be set.

As a specific example, if the correlation value (for example, the peak correlation value C _max ) is continuously maintained at a value higher than a predetermined value two or more times, the processing target image pair used in the corresponding point search process is processed. It is estimated that the difference between the target point and the corresponding point is small. Therefore, in such a case, a mode is conceivable in which the weighting coefficient for the low frequency frequency component is automatically reduced and the low frequency side limit range RL1 is expanded.

Further, for example, the correlation value in the previous stage for limiting the frequency component information used in the correlation calculation may be a probability density of vote integrated values, or a disparity indicating a maximum vote value in the distribution of vote integrated values. It may be the sum of the vote integrated value and the maximum vote value related to one or more parallax around (dx, dy). In addition, the ratio of the peak correlation value C _max to the total sum of correlation values obtained between the reference area and the comparison area may be employed. That is, various values indicating correlation between a plurality of images obtained by the preceding correlation calculation may be adopted as the preceding correlation value for limiting the frequency component information used in the correlation calculation. Thus, by adding more information, the accuracy in the corresponding point search process can be improved regardless of the influence of noise or the like.

In the above-described embodiment, for each frequency i, a relatively low weighting factor is adopted among the weighting factor W _{LOW of the low} frequency emphasis type rule and the weighting factor W _HIGH of the high frequency emphasis type rule. Not limited to. For example, for each frequency i, the weighting coefficient according to one weighting rule may be selected from the weighting coefficients of three or more types of weighting rules.

As a specific example, the weighting coefficient W _LOW of the low frequency-oriented rules, in addition to the weighting factor W _HIGH high frequency-oriented rules, further weighting factor W _e in accordance with another weighting rule may be set. In this case, for each frequency i, the weighting factor W _LOW of the low frequency-oriented rule, the weighting coefficient W _HIGH high frequency-oriented rules and of the weighting factors W _e according to another weighting rule, relatively lowest weighting factor A mode in which is adopted is conceivable.

For example, if satisfying the relation of the following equation (4) is, W _e (i, C _max) is adopted as weighting coefficients, as represented by the following formula (5), the amplitude f _i W _e (i, A mode is conceivable in which the value multiplied by C _max ) becomes the weighted amplitude f _iAFTER .

_{W e (i, C max)} <W LOW (i, C max) <W HIGH (i, C max) ··· (4)
f _iAFTER = W _e (i, C _max ) × f _i (5)

Further, as another aspect relating to the adoption of the weighting coefficient, there is also an aspect in which, for each frequency i, the product of the weighting coefficient W _{LOW of the low} frequency emphasis type rule and the weighting coefficient W _HIGH of the high frequency emphasis type rule is adopted as the weighting coefficient. Conceivable.

  In the above-described embodiment, the use restriction frequency range corresponding to the correlation value is set for each frequency i. However, the use restriction frequency range includes a predetermined frequency range regardless of the correlation value. May be included. For example, it is assumed that an extremely low frequency or high frequency has a small contribution in improving the accuracy of the corresponding point search process. In such a case, a predetermined frequency range may be included in the use limited frequency range regardless of the correlation value.

  For example, in the above-described embodiment, for the set of low-resolution images in the third layer (lowermost layer), since the correlation value of the previous stage is not obtained, weighting and restriction on the frequency component information were not performed. It is not limited to this. For example, frequency component information related to extremely low frequencies and high frequencies among the frequency component information may be limited. That is, frequency component information related to extremely low frequencies and high frequencies may not be used in the correlation calculation for the set of low resolution images in the third layer (lowermost layer). Thereby, the calculation speed in the corresponding point search process can be further improved.

  In the embodiment described above, the vote values are integrated according to the amplitude multiplied by the weighting coefficient, but the present invention is not limited to this. For example, a configuration may be employed in which a weighting coefficient other than zero is attached to the frequency component information, and voting values corresponding to the magnitude of the weighting coefficient are integrated in the correlation calculation.

  In the above embodiment, the corresponding point search is performed between the two images, but the present invention is not limited to this. The technical idea of the present invention can also be applied to search for corresponding points between three or more images.

  In the embodiment described above, the example in which the PSA method is adopted as the correlation calculation has been described. However, the present invention is not limited to this. For example, as the correlation calculation using the frequency component information output from the frequency component limiter 368, various correlation calculations such as other voting methods and a phase only correlation method (POC method) may be employed. When the POC method is adopted, the upstream correlation value for limiting the frequency component information used in the correlation calculation is, for example, the peak value of the POC value obtained by the upstream correlation calculation.

  In the above embodiment, the vertical lines and the horizontal lines are thinned out from the standard image G1 and the reference image G2 according to a predetermined thinning rule, so that the first to third reduced standard images G11 to G13 and the first to first images are displayed. Although the three reduced reference images G21 to G23 are respectively generated, the present invention is not limited to this. For example, an average value of pixel values is obtained for a predetermined number (for example, four) of adjacent pixel clusters and processed as one pixel, and the first to third reduced reference images G11 to G13 are performed over the entire image. A method in which the first to third reduced reference images G21 to G23 are generated is also conceivable. That is, the number of pixels is reduced according to a predetermined rule from the standard image G1 and the reference image G2 by various methods such as known reduction and magnification change, and thus the first to third reduced standard images G11 to G13 and the first to first images. Three reduced reference images G21 to G23 may be generated.

  ◎ Regarding the two-dimensional Fourier transform, a method of obtaining the result of the two-dimensional Fourier transform using the result of the one-dimensional Fourier transform in the x direction and the result of the Fourier transform in the y direction is also considered. It is done. When such a method is employed, a configuration in which the frequency component in the x direction and the frequency component in the y direction are separately limited may be employed.

  It goes without saying that all or a part of each of the above-described embodiment and various modifications can be appropriately combined within a consistent range.

  For example, for a corresponding point search operation for a base image and a reference image that constitute a stereo image of each frame after obtaining a time-series stereo image, the resolution is reduced stepwise as in the above embodiment. A set of images is generated, and corresponding point search processing is sequentially performed from the set of low-resolution images. Further, as in the first modified example, a time-series reference image (or reference image) is targeted. The corresponding point search operation can be considered.

1A, 1B Information processing system 2 Stereo camera 2B Camera 3 Information processing device 34 Storage unit 36 Control unit 361, 361B Image acquisition unit 362 Resolution conversion unit 363, 363B Search reference point setting unit 364 Image specifying unit 365, 365B Reference parallax setting unit 366, 366B Window setting unit 367, 367B Frequency analysis unit 368, 368B Frequency component limiting unit 369, 369B Correlation calculation unit 370, 370B Corresponding point determination unit PGa, PGb program

Claims (10)

A converter that converts the first region of the first image into first frequency component information and converts the second region of the second image into second frequency component information;
A correlation calculating unit that calculates a first correlation value indicating a correlation between the first region and the second region based on the first and second frequency component information;
Based on the first correlation value, a restriction unit that sets a use restriction range of frequencies in which use of frequency component information in the calculation in the correlation calculation unit is restricted;
With
The conversion unit is
Converting the third region of the third image corresponding to the first image into third frequency component information, and converting the fourth region of the fourth image corresponding to the second image into fourth frequency component information;
The correlation calculation unit is
Calculating a second correlation value indicating a correlation between the third region and the fourth region based on frequency component information relating to a frequency outside the use restriction range in the third and fourth frequency component information. An image processing apparatus. The image processing apparatus according to claim 1,
A generating unit that generates the first image by reducing the number of pixels from the third image according to a predetermined rule, and generates the second image by reducing the number of pixels from the fourth image according to the predetermined rule;
An image processing apparatus further comprising: The image processing apparatus according to claim 1 or 2,
The third and fourth images are
An image processing apparatus comprising a plurality of images obtained by capturing the same subject from different viewpoints. The first region of the first image is converted into the first frequency component information, the second region of the second image is converted into the second frequency component information, and the third region of the second image is converted into the third frequency component information. A conversion unit for converting and converting the fourth region of the third image into fourth frequency component information;
A correlation calculating unit that calculates a first correlation value indicating a correlation between the first region and the second region based on the first and second frequency component information;
Based on the first correlation value, a restriction unit that sets a use restriction range of frequencies in which use of frequency component information in the calculation in the correlation calculation unit is restricted;
With
The correlation calculation unit is
Calculating a second correlation value indicating a correlation between the third region and the fourth region based on frequency component information relating to a frequency outside the use restriction range in the third and fourth frequency component information. An image processing apparatus. The image processing apparatus according to claim 4,
The first to third images are
An image processing apparatus comprising a plurality of images capturing temporal changes of the same subject from the same viewpoint. The image processing apparatus according to any one of claims 1 to 5, wherein:
The restriction unit is
As the first correlation value increases, the use restriction is such that use of frequency component information relating to a wider range of frequencies from the lowest frequency side among the frequency component information obtained by the conversion unit is restricted. An image processing apparatus that sets a range. The image processing apparatus according to any one of claims 1 to 6, wherein
The restriction unit is
As the first correlation value decreases, the use restriction range is such that use of frequency component information relating to a wider range of frequencies from the highest frequency side among the frequency component information obtained by the conversion unit is restricted. An image processing apparatus characterized by setting the value. A first conversion step of converting a first region of the first image into first frequency component information and converting a second region of the second image into second frequency component information;
A first calculation step of calculating a first correlation value indicating a correlation between the first region and the second region based on the first and second frequency component information;
Based on the first correlation value, a setting step for setting a frequency use restriction range in which use of frequency component information in the calculation in the correlation calculation unit is restricted;
A second region that converts the third region of the third image corresponding to the first image into third frequency component information and converts the fourth region of the fourth image corresponding to the second image into fourth frequency component information. A conversion step;
A second correlation value indicating a correlation between the third region and the fourth region is calculated based on frequency component information relating to a frequency outside the use restriction range in the third and fourth frequency component information. Two computation steps;
An image processing method comprising: A first conversion step of converting a first region of the first image into first frequency component information and converting a second region of the second image into second frequency component information;
A first calculation step of calculating a first correlation value indicating a correlation between the first region and the second region based on the first frequency component information and the second frequency component information;
A setting step of setting a frequency use restriction range in which use of frequency component information in the correlation calculation is restricted based on the first correlation value;
A second conversion step of converting the third region of the second image into third frequency component information and converting the fourth region of the third image into fourth frequency component information;
A second correlation value indicating a correlation between the third region and the fourth region based on frequency component information of a frequency outside the use restriction range in the third and fourth frequency component information; A calculation step;
An image processing method comprising:   A program that causes the information processing apparatus to function as the image processing apparatus according to any one of claims 1 to 7 by being executed by a control unit included in the information processing apparatus.

Images