JP5446516B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus that searches a reference image for a corresponding point of an attention point set in a standard image.

  In recent years, among a plurality of images related to a measurement object photographed from different viewpoints and a plurality of time-series images related to a moving measurement object photographed in time series from the same viewpoint, one image is a reference image, Using the image as a reference image, a corresponding point on the reference image corresponding to the point of interest on the standard image is searched, and based on the obtained corresponding point and point of interest, the three-dimensional shape of the measurement object is obtained, or the measurement target Devices for tracking the movement of objects are known. Here, since the existence range of corresponding points in the reference image cannot be specified in advance, it is common to perform a corresponding point search process in which the entire reference image is set as the corresponding point search range. As a technique for searching for corresponding points with high accuracy without setting the entire area of the reference image as a search range, a technique for searching for corresponding points by a multi-resolution strategy using a multi-resolution image has attracted attention.

  In Patent Document 1, in the corresponding point search using the multi-resolution image, the search result of the corresponding point in the reference image of the target resolution is used for the next corresponding point search in the reference image having one step higher resolution. In addition, when the reliability of corresponding points is low, a technique is disclosed that enables high-accuracy corresponding point search by correcting corresponding points with low reliability using information on surrounding corresponding points.

Japanese Patent Laid-Open No. 2001-148012

  However, in the image processing apparatus of Patent Document 1, the corresponding point search before correcting the corresponding point with low reliability is performed with a resolution equal to the pixel size of the reference image of the multi-resolution image. In addition, there is a problem in that there is a limit in accuracy in correcting corresponding points with low reliability.

  The present invention has been made to solve these problems, and provides a technique capable of improving the accuracy and reliability of the corresponding point search while suppressing the processing cost in the corresponding point search using the multi-resolution image. Objective.

In order to solve the above problems, the invention of claim 1 is a view picture processor, the original reference image input images of one of the two or more input images, other input image original reference image A plurality of reference images having different resolutions and a plurality of reference images by reducing the resolution of the original reference image and the original reference image so that the resolution decreases from the upper layer to the lower layer. Multi-resolution image creating means for generating each of the reference images in a hierarchical manner, and among the plurality of reference images and the plurality of reference images, images belonging to the same attention layer as attention reference images and attention reference images, Search reference point setting means for setting a search reference point as a reference for search processing for searching for a corresponding point of the target reference image corresponding to the target point of the target reference image to the target reference image; and the search reference point In the hierarchical structure of the corresponding point search means for searching for the corresponding point by performing the search process with reference to the reference point, and the plurality of reference images and the plurality of reference images, Control for controlling the search reference point setting means and the corresponding point search means so that the search processing is sequentially executed from the lower hierarchy to the upper hierarchy by sequentially setting from the higher hierarchy to the higher hierarchy. And the corresponding point search means searches for the corresponding point in the target reference image with sub-pixel resolution using a sub-pixel having a size smaller than a pixel size of the target reference image as an image representation unit. In addition, the search reference point setting means, based on the corresponding points searched with the sub-pixel resolution, a reference picture of a hierarchy one level higher than the target hierarchy. Set the search reference point in the multi-resolution image generation means, the resolution of the target reference picture and the target reference picture with respect to the reference image and the reference image of one upper layer of the target hierarchy, greater real than 1 The resolution is reduced to 1 / r using r, and the corresponding point search means is coarser than 1 / r ² of the pixel size of the target reference image, and the target reference image The corresponding points in the reference image of interest are searched with a search resolution finer than the pixel size.

The invention according to claim 2 is the image processing apparatus according to claim 1 , wherein the corresponding point search means is expressed by 1 / r of the pixel size of the target reference image using the real number r. The corresponding points in the target reference image are searched with a search resolution that can be obtained.

The invention according to claim 3 is the image processing apparatus according to claim 1 , wherein the corresponding point search means is expressed by 1 / r of a pixel size of the target reference image using the real number r. The corresponding points in the target reference image are searched with a search resolution finer than the search resolution.

The invention according to claim 4 is the image processing apparatus according to claim 1 , wherein the corresponding point search means is expressed by 1 / r of a pixel size of the reference image of interest using the real number r. The corresponding points in the target reference image are searched with a search resolution coarser than the search resolution.

The invention of claim 5 is the images processing apparatus, an image obtaining means for obtaining first input image among the two or more input image original reference image, the other input image as an original reference image , By reducing the resolution of the original standard image and the original reference image so that the resolution becomes lower from the upper hierarchy toward the lower hierarchy, the plurality of standard images and the plurality of reference images having different resolutions are respectively hierarchized. A multi-resolution image generating means for generating the target image, and an image belonging to the same target layer among the plurality of standard images and the plurality of reference images as a target standard image and a target reference image as a target point of the target standard image Search reference point setting means for setting a search reference point as a reference for search processing for searching for a corresponding point of the corresponding target reference image corresponding to the target reference image; and the search processing based on the search reference point. In the hierarchical structure of the corresponding point searching means for searching for the corresponding points and the plurality of reference images and the plurality of reference images, the target layer is directed from the lower layer to the upper layer. Control means for controlling the search reference point setting means and the corresponding point search means so that the search process is sequentially executed from a lower hierarchy to an upper hierarchy by sequentially setting The corresponding point search means searches for the corresponding point in the target reference image with a sub-pixel resolution using a sub-pixel having a size smaller than the pixel size of the target reference image as a unit of image representation, and the search criterion The point setting means, based on the corresponding points searched with the sub-pixel resolution, finds a search reference point in a reference image in a hierarchy one level higher than the target hierarchy. Constant, and the corresponding point searching section, said at pixel resolution is the pixel size of the target reference picture, the target reference pixel corresponding point for searching for a pixel corresponding point of pixel size in the target reference picture corresponding to the current point of the image A pixel search means for performing a search process; and a sub-pixel search means for performing a sub-pixel corresponding point search process for searching for a corresponding point of the target reference image at the sub-pixel resolution based on the pixel corresponding point. The image processing apparatus includes: a reliability calculation unit that calculates a reliability indicating a search accuracy of the pixel corresponding point based on a search result of the pixel corresponding point; and whether the reliability is lower than a predetermined value. A reliability determination unit that determines whether or not the reliability is lower than a predetermined value by the reliability determination unit. , Wherein the sub-pixel search means of the corresponding point searching section, disabling the sub-pixel corresponding point searching process.

The invention of claim 6 is the images processing apparatus, an image obtaining means for obtaining first input image among the two or more input image original reference image, the other input image as an original reference image , By reducing the resolution of the original standard image and the original reference image so that the resolution becomes lower from the upper hierarchy toward the lower hierarchy, the plurality of standard images and the plurality of reference images having different resolutions are respectively hierarchized. A multi-resolution image generating means for generating the target image, and an image belonging to the same target layer among the plurality of standard images and the plurality of reference images as a target standard image and a target reference image as a target point of the target standard image Search reference point setting means for setting a search reference point as a reference for search processing for searching for a corresponding point of the corresponding target reference image corresponding to the target reference image; and the search processing based on the search reference point. In the hierarchical structure of the corresponding point searching means for searching for the corresponding points and the plurality of reference images and the plurality of reference images, the target layer is directed from the lower layer to the upper layer. Control means for controlling the search reference point setting means and the corresponding point search means so that the search process is sequentially executed from a lower hierarchy to an upper hierarchy by sequentially setting The corresponding point search means searches for the corresponding point in the target reference image with a sub-pixel resolution using a sub-pixel having a size smaller than the pixel size of the target reference image as a unit of image representation, and the search criterion The point setting means, based on the corresponding points searched with the sub-pixel resolution, finds a search reference point in a reference image in a hierarchy one level higher than the target hierarchy. Constant, and the corresponding point searching section, said at pixel resolution is the pixel size of the target reference picture, the target reference pixel corresponding point for searching for a pixel corresponding point of pixel size in the target reference picture corresponding to the current point of the image A pixel search means for performing a search process; and a sub-pixel search means for performing a sub-pixel corresponding point search process for searching for a corresponding point of the target reference image at the sub-pixel resolution based on the pixel corresponding point. The image processing device includes: a first reliability indicating a search accuracy of the pixel corresponding point based on a search result of the pixel corresponding point search process; and the attention reference based on a search result of the sub pixel corresponding point search process. Calculating at least one of the second reliability levels indicating the search accuracy of the corresponding points searched with the sub-pixel resolution of the image; Determining at least one of reliability calculation means and whether or not the first reliability is lower than a first predetermined value and whether or not the second reliability is lower than a second predetermined value; Reliability determination means, wherein the multi-resolution image creation means has a resolution lower than that of the reference image and reference image of the hierarchy one level higher than the attention hierarchy, and is higher than the attention reference image and the attention reference image. Generating an auxiliary standard image and an auxiliary reference image in which the original standard image and the original reference image are reduced in resolution so as to increase the resolution, and the first reliability is generated by the reliability determination unit. Is determined to be lower than the first predetermined value, or the second reliability is lower than the second predetermined value, (a) the search reference point setting means refers to the attention reference The pixel pair of the image Based on a corresponding point or a corresponding point searched at the sub-pixel resolution of the target hierarchy, a search reference point serving as a reference for processing for searching for a corresponding point of the auxiliary reference image corresponding to the target point of the auxiliary reference image (B) the control means sets the auxiliary reference image and the auxiliary reference image as the target layer in the hierarchical structure of the plurality of reference images and the plurality of reference images, respectively. Is set as an intermediate hierarchy between the target hierarchy and the one hierarchy higher than the target hierarchy, and the target hierarchy is sequentially set from the intermediate hierarchy toward the higher hierarchy, so that the search processing is performed higher than the intermediate hierarchy. The search reference point setting means and the corresponding point search means are controlled so as to be executed sequentially toward the hierarchy.

According to the first to sixth aspects of the present invention, the search reference point of the reference image of the next higher hierarchy is set based on the corresponding point of the reference image of the target hierarchy searched with the sub-pixel resolution. as compared with the case where the corresponding points are not searched in the sub-pixel resolution, it is possible to set a highly reliable search reference point in a narrower range, and as a result, to be reduce the number of layers of the multi-resolution images, search criteria Since the search range of the corresponding points set based on the points can be narrowed, the corresponding points can be searched with high accuracy while suppressing the processing cost related to the corresponding points search.

According to the first aspect of the present invention, the subpixel resolution for obtaining the corresponding point is set to be finer than the pixel size of the reference image in the target layer and coarser than the pixel size of the reference image in the layer two levels higher than the target layer. Therefore, it is possible to efficiently perform the corresponding point search process by the multi-resolution strategy in which the images of each layer of the multi-resolution image are used without waste and the corresponding points are sequentially obtained for each layer.

According to the second aspect of the present invention, the search resolution having the same size as the pixel size of the reference image of the reference layer one level higher than the target layer, that is, the pixel size of the search reference point in the reference image one level higher than the target layer. Thus, corresponding points in the reference image of the target hierarchy can be searched, and corresponding point search based on the multi-resolution strategy can be efficiently performed at a suitable processing cost with respect to the hierarchical structure of the multi-resolution image.

According to the invention of claim 3 , since the corresponding point in the reference image in the target layer can be searched with a search resolution finer than the pixel size of the reference image in the layer one level higher than the target layer, the search is performed in the reference image in the target layer. By using the corresponding points, it becomes possible to verify the reliability of the corresponding points searched for in the reference image one level higher than the target layer, and the reliability of the corresponding point search can be increased.

According to the fourth aspect of the present invention, the search resolution is coarser than the pixel size of the reference image one level higher than the target layer, that is, the pixel size of the search reference point in the reference image one level higher than the target layer. By using the corresponding points in the searched reference image of the target layer, the position of the search reference point in the reference image of the layer one level higher than the target layer can be obtained with higher accuracy than when the corresponding point is not searched with subpixel resolution. Therefore, the corresponding point search can be performed with high accuracy while suppressing the processing cost related to the corresponding point search.

According to the invention of claim 5 , when the reliability of the pixel corresponding point searched with the pixel resolution is low, the sub-pixel corresponding point searching process for searching the corresponding point with the sub-pixel resolution is not performed, and the pixel corresponding point is searched. Since the failure of the subpixel corresponding point search process due to the low reliability can be prevented, the processing cost related to the corresponding point search can be suppressed.

According to the sixth aspect of the present invention, the probability that a correct corresponding point exists in the search range in the intermediate hierarchy can be obtained by expanding the angle of view of the search range in the intermediate hierarchy higher than one hierarchy higher than the target hierarchy. The corresponding points in the intermediate layer can be searched in a state where the information amount of the pixel is larger than that of the target layer and the perspective conflict is less likely to occur. The probability of successful search of the corresponding points can be increased, and therefore the probability of successful search of corresponding points based on the multi-resolution strategy can be increased.

It is a figure which shows schematic structure of the three-dimensional shape measuring apparatus using the image processing apparatus which concerns on embodiment. It is a functional block diagram showing the composition of the principal part of the image processing device concerning an embodiment. It is a figure which illustrates a multi-resolution reference image. It is a figure which illustrates a multi-resolution reference image. It is a figure explaining the example which the resolution scaling factor has on a corresponding point search process. It is a figure which illustrates the multi-resolution image of a reference image. It is a figure which shows the example of a setting of the attention point in a multiresolution reference image. It is a figure which shows the example of a setting of the search reference point in a multiresolution reference image. It is a figure explaining the outline | summary of a pixel corresponding point search process. It is a figure which shows the example of a setting of the reference window with respect to the search reference point of a reference image. It is explanatory drawing of the pixel corresponding point search process using SAD method. It is a figure which illustrates the scanning aspect of the reference window in SAD method. It is a figure which shows the procedure of the pixel corresponding point search process using POC method. It is a figure which shows the example of a POC value. It is a figure which shows the example of the sub pixel corresponding point search based on the pixel corresponding point with low reliability. It is explanatory drawing which shows the example of a subpixel corresponding point search process. It is explanatory drawing which shows the example of a subpixel corresponding point search process. It is a figure explaining the correspondence of the subpixel corresponding point of the attention hierarchy, and the search reference point of one higher hierarchy. It is a figure which shows the example of a setting of the search reference point of the one higher hierarchy with respect to the subpixel corresponding | compatible point of an attention hierarchy. It is a figure which shows the example of a setting of the search reference point of the one higher hierarchy with respect to the subpixel corresponding | compatible point of an attention hierarchy. It is a figure explaining the effect by the setting of a subpixel corresponding point. It is a figure which shows the example of a setting of the search reference point of the one higher hierarchy with respect to the subpixel corresponding | compatible point of an attention hierarchy. It is a figure explaining the effect by the setting of a subpixel corresponding point. 6 is a flowchart illustrating an outline of an operation of the image processing apparatus according to the embodiment. 6 is a flowchart illustrating an outline of an operation of the image processing apparatus according to the embodiment. It is a figure explaining the example of the process which calculates | requires the reliability of a subpixel corresponding point search process. 6 is a flowchart illustrating an outline of an operation of the image processing apparatus according to the embodiment.

  The image processing apparatus according to the present invention measures, for example, an object to be measured by two or more cameras, processes the obtained image, and obtains a line-of-sight equation for each camera passing through a point of interest on the object to be measured. A three-dimensional shape measuring device that measures the shape of the measurement object or a single camera measures the measurement object in time series, processes the obtained image, determines the amount of movement of the measurement object, The present invention is applied to an apparatus that compares a registered measurement object image with a measured image.

  Hereinafter, the image processing apparatus according to the present invention measures a measurement object using two cameras (also referred to as stereo cameras), and processes the obtained stereo image to measure the three-dimensional shape of the measurement object. An embodiment applied to a three-dimensional measuring apparatus will be described with reference to the drawings. For the sake of explanation, the coordinate axes are appropriately described in the drawings.

<Three-dimensional shape measuring device>
FIG. 1 is a diagram illustrating a schematic configuration of a three-dimensional shape measuring apparatus 1A using an image processing apparatus 3A according to the embodiment.

  As shown in FIG. 1, the three-dimensional shape measuring apparatus 1A includes a two-lens stereo camera 2 and an image processing apparatus 3A connected to the stereo camera 2 so as to be able to transmit data.

  The two-lens stereo camera 2 is provided with two imaging systems 21 and 22 each having an imaging device. The imaging systems 21 and 22 have substantially the same optical characteristics and electrical characteristics, are spaced apart along a predetermined direction, and the measurement object OB in front of the camera is synchronized from the same viewpoint at different timings. It is configured to take an image. The two images captured at the same timing by the imaging systems 21 and 22 are so-called stereo images, and are transmitted to the image processing device 3A via the data line DL.

  Here, of the two images constituting the stereo image, an image captured and acquired by the imaging system 21 is appropriately referred to as a “reference image”, and an image captured and acquired by the imaging system 22 is appropriately referred to as “reference”. This is called “image”.

  In addition, camera parameters such as the focal length, principal point position, distortion, and baseline length of the imaging systems 21 and 22 are calibrated in advance, and the measurement target imaged on the imaging elements of the imaging systems 21 and 22 by the camera parameters. A camera line-of-sight equation passing through a point on the image of the object OB and the principal point of the imaging system is determined.

  First, the image processing device 3A hierarchically repeats the resolution reduction of the standard image BIT and the reference image RIT related to the measurement object OB, so that the standard image BIT and the reference image RIT have a plurality of resolutions different from each other. A multi-resolution image composed of images is created, and corresponding point search processing is performed for searching for a corresponding point on the reference image RIT corresponding to the target point on the base image BIT using the multi-resolution image.

  Next, the image processing apparatus 3A creates a camera line-of-sight equation for the target point and the corresponding point using the searched coordinate information of the target point and the corresponding point and the camera parameters of the imaging systems 21 and 22, and this equation. The three-dimensionalization processing for obtaining the three-dimensional coordinates of the point of the measuring object OB corresponding to the target point is performed by solving the above. The image processing apparatus 3A calculates the three-dimensional shape of the measurement object OB by performing corresponding point search processing and three-dimensionalization processing for a plurality of attention points.

  Here, although the stereo camera is provided in the three-dimensional shape measuring apparatus 1A, for example, when three or more cameras are provided, one of the input images input to the three-dimensional shape measuring apparatus is provided. The input image is acquired as a standard image and the other input images are acquired as reference images, and the same processing as when using a stereo camera is performed to obtain the three-dimensional shape of the measurement object OB.

<Functional Description of Image Processing Device 3A>
FIG. 2 is a functional block diagram illustrating a configuration of a main part of the image processing apparatus 3A according to the embodiment.

  As shown in FIG. 2, the image processing apparatus 3A includes an operation unit 4, a display unit 5, a communication unit 6, an input / output unit 7, a storage unit 9, an image acquisition unit 10, a multi-resolution image creation unit 11, a search reference point. A setting unit 12, a corresponding point search unit 13, a reliability calculation unit 14, a reliability determination unit 15, and a control unit 16 are provided. By these functional blocks, multi-resolution image generation and corresponding points for the reference image BIT and the reference image RIT are provided. Search and three-dimensional processing are performed.

  Hereinafter, the operation unit 4 to the control unit 16 of the image processing apparatus 3A will be described.

◎ Operation unit 4 to storage unit 9:
The operation unit 4 includes, for example, a keyboard, a mouse, and operation buttons. The operation unit 4 includes an operation signal input to the control unit 16 and a resolution scaling factor RR and a search resolution SR related to each unit of the image processing apparatus 3A. The control parameters that are used and used for the purpose of setting control parameters are stored in the storage unit 9 by the control unit 16.

  The display unit 5 is configured by a liquid crystal display, for example, and can display a moving image or the like. Image information such as a captured stereo image or a generated three-dimensional image, various messages, and the like are displayed on the display unit 5. Control parameter setting information is also displayed on the display unit 5.

  The communication unit 6 is an interface for connecting to a LAN or the Internet, and includes, for example, a network adapter.

  The input / output unit 7 includes, for example, a multimedia drive, receives a storage medium 8 such as an optical disk, and exchanges data with the control unit 16. The stereo image processed by the image processing apparatus 3A can be input not only directly from the stereo camera 2, but also via the storage medium 8 or the network by the communication unit 6 or the input / output unit 7.

  The storage unit 9 includes, for example, a hard disk, a ROM, a RAM, and the like. Information output from each unit of the image processing apparatus 3A, control parameters set to control each unit of the image processing apparatus 3A, and calibration are performed. It is used for permanent storage of camera parameters and control programs, and for temporary storage of various information.

  Here, the control parameters include, for example, the number of layers T of the multi-resolution image, the resolution scaling factor RR, the window sizes WX and WY which are the horizontal and vertical sizes of the reference window and the reference window, and the resolution (“ SR and threshold value TH1 for determining the reliability of pixel corresponding point search processing.

  The image acquisition unit 10, multi-resolution image creation unit 11, search reference point setting unit 12, corresponding point search unit 13, reliability calculation unit 14, reliability determination unit 15 and control unit 16 described below are CPUs. It may be realized by executing a predetermined program, or may be realized by using a dedicated hardware circuit.

Image acquisition unit 10:
The image acquisition unit 10 acquires an image captured by the standard camera 21 and an image captured by the reference camera 22 and stores them in the storage unit 9 as a standard image BIT and a reference image RIT, respectively. Here, the standard image BIT and the reference image RIT are, for example, digital image data in which a plurality of pixels are arranged in a matrix and pixel values of each pixel are expressed with a predetermined number of gradations.

Multi-resolution image creation unit 11:
The multi-resolution image creation unit 11 performs multi-stage resolution reduction processing on the standard image BIT and the reference image RIT, thereby performing a multi-resolution standard image MB that is a multi-resolution image of the standard image BIT and a multi-resolution image of the reference image RIT. Are created and stored in the storage unit 9.

  FIG. 3 is a diagram illustrating a multi-resolution standard image MB, and FIG. 4 is a diagram illustrating a multi-resolution reference image MR. Regarding the multi-resolution standard image MB and the multi-resolution reference image MR, FIGS. 3A and 4A show the highest standard image BIT and the reference image RIT, respectively, as shown in FIGS. ) Respectively shows the entire multi-resolution standard image MB and multi-resolution reference image MR, and FIGS. 3C and 4C show the lowest-order standard image BI1 and reference image RI1, respectively.

  As shown in FIGS. 3 and 4, the multi-resolution standard image MB and the multi-resolution reference image MR are an aggregate of a plurality of reference images having different resolutions and a plurality of reference images having different resolutions, respectively. , It is hierarchically formed so that the resolution decreases as it goes from the upper layer to the lower layer.

  In the multi-resolution standard image MB and the multi-resolution reference image MR, the highest standard image BIT and the reference image RIT image are not reduced in resolution, and the lowest standard image BI1 and the reference image RI1 image have the lowest resolution. It is an image. The multi-resolution standard image MB and the multi-resolution reference image MR have the same number of layers T, and the standard image and the reference image in the same layer have the same resolution.

  Note that a reference image BIT that is a reference image taken by the reference camera 21 and that is also the highest-order image of the multi-resolution reference image MB and is not reduced in resolution is also referred to as an “original reference image” BIT, and is taken by the reference camera 22. A reference image RIT that is a reference image and is also the highest-order image of the multi-resolution reference image MR and has not been reduced in resolution is also referred to as an “original reference image” RIT.

  In addition, an image with higher resolution is referred to as an “upper” (or “upper hierarchy”) image, and an image with lower resolution than an image of a target hierarchy (also referred to as “target hierarchy”) of the multi-resolution image. Are referred to as “lower” (or “lower hierarchy”) images. Further, the ratio of the resolution reduction when the image of the target layer is reduced in resolution to create one lower layer image is also referred to as “resolution scaling factor”.

  The resolution scaling factor is given as the ratio of the number of pixels in the horizontal direction of the image in the lower layer to the number of pixels in the horizontal direction of the image in the upper layer among the two adjacent layers in the multi-resolution image. It is also given as the ratio of the number of pixels in the vertical direction of the lower layer image to the number of pixels in the vertical direction of the layer image. Usually, these two types of ratios are set to the same value.

  The multi-resolution image is repeatedly reduced in resolution by the resolution scaling factor RR until the number of layers of the multi-resolution image reaches the number of layers T, for example, according to the resolution scaling factor RR and the number of layers T preset and stored in the storage unit 9. Alternatively, it is created by repeatedly reducing the resolution with the resolution scaling factor RR until the number of pixels of the lowermost layer image becomes a predetermined value or less.

  Specifically, the multi-resolution image creation unit 11 first relates to multi-resolution image creation such as the number of layers T of the multi-resolution image, the resolution scaling factor RR, the base image BI1 in the lowest layer, and the number of pixels of the reference image RI1. The control parameter, the standard image BIT, and the reference image RIT are acquired from the storage unit 9, and the standard image BIT and the reference image RIT are respectively referred to as the standard image (also referred to as “target standard image”) BIm and the reference image of the target hierarchy. Set as RIm (also referred to as “attention reference image”) (step S2).

  When the setting of the target layer is completed, the multi-resolution image creating unit 11 multiplies the number of pixels in the horizontal direction and the vertical direction of the base image BIm and the reference image RIT by the resolution scaling factor RR, and sets the standard of the next lower layer of the target layer. The number of pixels in the horizontal direction and the vertical direction of the image BI (m−1) and the reference image RI (m−1) is obtained, and the storage areas of the base image BI (m−1) and the reference image RI (m−1) are stored. Secured in the unit 9 (step S10).

  When the storage area is secured, the multi-resolution image creation unit 11 averages the pixel values of one pixel of the reference image BI (m−1) and the pixel values of the pixels of the reference image BIm corresponding to this pixel. The pixel value is set in the storage area corresponding to the pixel, and the pixel value setting process is performed for all of the reference image BI (m−1) and the reference image RI (m−1). By performing the processing for the pixels, images of the base image BI (m−1) and the reference image RI (m−1) are created, and the base image BI (m−1) and the reference image RI (m−1) are stored in the target hierarchy. A new standard image BIm and reference image RIm are set (step S20).

  Hereinafter, step S10 and step S20 are repeated until the number of images reaches the number of layers T, or until the number of pixels of the base image BI1 and reference image RI1 in the lowermost layer is less than or equal to a predetermined value, so that The base image BIT and the reference image RIT acquired by the image acquisition unit 10 are hierarchically reduced in resolution so that the resolution becomes lower toward the hierarchy, and a plurality of hierarchical standard images and reference images having different resolutions are obtained. That is, the multi-resolution standard image MB and the multi-resolution reference image MR are created, and the multi-resolution standard image MB and the multi-resolution reference image MR are stored in the storage unit 9.

○ Resolution scaling factor RR:
FIG. 5 is a diagram for explaining an example of the influence of the resolution scaling factor RR on the corresponding point search process.

  An image RIm shown in FIG. 5 is a reference image of the target layer, and an image RI (m + 1) is a reference image one layer higher than the target layer. Here, each region partitioned by the grid shown in FIG. 5 represents individual pixels (also referred to as “pixels”) of the reference image RIm and the reference image RI (m + 1).

  Further, the resolution scaling factor RR when generating the reference image RIm from the reference image RI (m + 1) is 1/4. The pixel CPmT on the reference image RIm indicates a correct corresponding point with respect to a reference point set on the reference image of the target layer, and the pixel CPmF indicates an incorrect corresponding point.

  The pixel K (m + 1) T on the reference image RI (m + 1) indicates the correct search reference point set based on the correct corresponding point CPmT, and the pixel K (m + 1) F is based on the incorrect corresponding point CPmF. Indicates the set wrong search reference point.

  As shown in FIG. 5, the deviation between the correct corresponding point CPmT and the incorrect corresponding point CPmF is only one pixel in the diagonal direction of the pixel, but the search reference points K (m + 1) T and K (m + 1) F are The shift in the diagonal direction of these pixels is expanded to four pixels that are the reciprocal times of the resolution scaling factor. That is, the corresponding point search error in the lower layer is further expanded in the search for the search reference point in the upper layer.

  Since the corresponding point on the reference image RI (m + 1) is searched in the search range based on the search reference point set in the reference image RI (m + 1), the reference image RI is used by using the incorrect search reference point K (m + 1) F. Even if the corresponding point on (m + 1) is searched, the error of the corresponding point search in the lower-layer image RIm cannot be corrected, and the possibility that an erroneous search is performed increases.

  That is, when the resolution scaling factor RR is small, a small error in the corresponding point search in the lower layer image (low resolution image) leads to a large error in the corresponding point search in the upper layer image (high resolution image). Therefore, the accuracy of the corresponding point search is likely to be reduced.

  The resolution scaling factor RR is set by, for example, 2/3, 1/2, 1/3 or 1/4 (the resolution scaling factor RR is a real number larger than 1 (typically, displayed on the display unit 5). When expressed as 1 / r using a rational number r, a method of selecting r using an operation unit 4 from candidates in an appropriate range such as 3/2, 2, 3 or 4) is adopted. Alternatively, a method of setting a resolution scaling factor smaller than 1 other than the candidate value by an input from the operation unit 4 or the like may be employed. The set resolution scaling factor RR is stored in the storage unit 9.

  When the multi-resolution image creation unit 11 is a dedicated hardware circuit, the resolution scaling factor RR is set by, for example, a plurality of multi-resolution image creation units 11 respectively corresponding to a plurality of different resolution scaling factors RR. It may be set by a method of selecting the multi-resolution image creation unit 11 corresponding to a desired resolution scaling factor RR by switching a dip switch or the like.

  In a multi-resolution image, the lower the hierarchy image, the more “information reduction” occurs in each pixel in the image. Further, when the number of pixels in the search range based on the search reference point is constant regardless of the hierarchy of the multi-resolution image, the search range based on the search reference point is more as the image is in a lower hierarchy. Since the measurement object OB imaged in a wide field of view and the background image are included, so-called “distance competition” occurs, and a corresponding point search error is likely to occur.

  Similarly, the smaller the resolution scaling factor RR when the lower layer image is created by lowering the resolution of the upper layer image, the more “information amount reduction” occurs in the lower layer image and ”Occurs, and a corresponding point search error is likely to occur.

  Therefore, in order to suppress the above-described adverse effect that the corresponding point search error in the lower layer amplifies the corresponding point search error in the upper layer, as shown in the example of the multi-resolution image of the reference image shown in FIG. A plurality of resolution scaling factors RR may be used to create a small resolution scaling factor for an upper-layer image and a large resolution scaling factor for a lower-layer image. In the example of FIG. 6, the above-described adverse effect is suppressed by setting the resolution scaling factor on the upper layer side to 1/4 and the resolution scaling factor on the lower layer side to 1/2.

Search reference point setting unit 12:
The search reference point setting unit 12 determines whether the multi-resolution reference image MB and the multi-resolution reference image are based on the correspondence relationship between the multi-resolution standard image MB determined by the control parameter and each layer of the multi-resolution reference image MR. On each image of the target hierarchy specified by the control unit 9 in MR, the target point and the search reference point, which are the reference of the corresponding point search processing performed by the corresponding point search unit 13, have a resolution equal to the pixel size (“ Also referred to as “pixel resolution”). Here, the process of setting the search reference point is also referred to as “search reference point setting process”.

○ Regarding points of interest:
FIG. 7 is a diagram illustrating an example of setting a target point in the multi-resolution reference image MB. Nm shown in FIG. 7 is a point of interest on the reference image BIm of the target layer in the multi-resolution reference image MB, and N (m + 1) is a reference image BI (m + 1) one layer higher than the target layer. It is an attention point and corresponds to the attention point Nm. The attention point Nm is set by the search reference point setting unit 12 based on the correspondence relationship between the reference image BI (m + 1) and the reference image BIm and the coordinates of the attention point N (m + 1). By repeating this setting process one after another from the upper layer side to the lower layer side, the attention point in the reference image of each layer from the highest reference image BIT to the lowest reference image BI1 is set.

  Here, the attention point in the highest-order reference image BIT of the multi-resolution reference image MB is, for example, manual designation of feature points on the reference image BIT displayed on the display unit 5 by the mouse of the operation unit 4, or the reference image BIT. It is set by a method such as automatic designation based on edge positions automatically extracted by performing edge detection processing.

  Further, the attention point in the lower attention layer is manually designated with the mouse of the operation unit 4, for example, and the attention point in the upper layer is set based on the correspondence relationship of the pixel between the attention layer and the one higher hierarchy. A method of setting a point of interest on a reference image in each layer by sequentially performing processing from a lower layer to an upper layer may be employed.

  Note that the point of interest in the multi-resolution reference image MB does not necessarily have to be set by the search reference point setting unit 12. For example, the multi-resolution image creation unit 11 creates the multi-resolution reference image MB and You may perform the setting of the attention point on a reference | standard image.

○ About search reference points:
FIG. 8 is a diagram illustrating an example of setting search reference points in the multi-resolution reference image MR.
The CSm shown in FIG. 8 has a resolution (also referred to as “subpixel resolution”) in which a subpixel having a size smaller than the pixel size on the reference image RIm of the target layer in the multiresolution reference image MR is an image representation unit. K (m + 1) is a search reference point in the reference image RI (m + 1) one level higher than the target layer, and corresponds to the subpixel corresponding point CSm. . The search reference point K (m + 1) is set based on the pixel correspondence between the reference image RIm and the reference image RI (m + 1) and the coordinates of the subpixel corresponding point CSm.

  By the search reference point setting process described above, the search reference point setting unit 12 uses the search process reference in the search process for searching for the corresponding point corresponding to the target point on the reference image of the target layer on the reference image of the target layer. Is set on the reference image of the target hierarchy.

  A specific search reference point setting process will be described in a column of “correspondence between subpixel corresponding point and search reference point of one higher hierarchy” described later.

Corresponding point search unit 13:
The corresponding point search unit 13 includes a pixel search unit 13a and a sub-pixel search unit 13b.

○ Pixel search unit 13a:
The pixel search unit 13a performs search processing (also referred to as “pixel corresponding point search processing”) for searching for corresponding points with pixel resolution, so that the attention set on the reference image of the target layer in the multi-resolution reference image MB is obtained. Corresponding points corresponding to the points are searched on the reference image of the target layer in the multi-resolution reference image MR. Here, the corresponding points searched for with pixel resolution are also referred to as “pixel corresponding points”.

  In the pixel corresponding point search process, search windows (also referred to as a “reference window” and a “reference window”, respectively) on the reference image and the reference image of the target hierarchy based on the attention point and the search reference point, respectively. And a correlation value between images in the standard window and the reference window is obtained. The sizes of the standard window and the reference window are determined by the window sizes WX and WY that are control parameters acquired from the storage unit 9.

○ Sub-pixel search unit 13b:
The sub-pixel search unit 13b searches for the corresponding point with sub-pixel resolution based on the information about the pixel corresponding point searched by the pixel search unit 13a and its neighboring points, and the search resolution SR acquired from the storage unit 9. By performing the process (also referred to as “sub-pixel corresponding point search process”), the corresponding point corresponding to the target point is searched on the reference image of the target layer. Here, the corresponding points searched with subpixel resolution are also referred to as “subpixel corresponding points”.

  By the cooperation of the pixel search unit 13a and the subpixel search unit 13b, the corresponding point search unit 13 performs a pixel corresponding point search process and a subpixel corresponding point search process, and is set as a reference image of the target hierarchy. A corresponding point on the reference image of the target layer in the multi-resolution reference image MR corresponding to the target point is searched with sub-pixel resolution.

○ Search resolution SR:
The search resolution SR is, for example, a method of selecting by the operation unit 4 from a group of constant search resolutions such as 1/2 and 1/3 displayed on the display unit 5, and a multiplication coefficient displayed on the display unit 5. The storage unit is preset by a method of multiplying the coefficient selected by the operation unit 4 from the candidate group by the resolution scaling factor RR stored in the storage unit 9 or a method of inputting and setting a desired search resolution from the operation unit 4. 9 is stored.

  Here, the unit of the search resolution SR according to the present invention is “pixel size” (also referred to as “pixel length” or simply “pixel”), but in the description according to the present invention, The description of the unit of the search resolution SR is omitted as appropriate.

  The search resolution value “large” is also referred to as “coarse”, and conversely the search resolution value “small” is also referred to as “fine” search resolution.

The search resolution SR is limited to a range of RR ² <SR <1, and when a search resolution SR that does not fall within this range is set, a warning such as a warning message display on the display unit 5 is given by the control unit 16. This is done to prompt the user to re-enter the search resolution SR.

  In addition, when the corresponding point search unit 13 is a dedicated hardware circuit, the search resolution SR is set by, for example, dip among a plurality of corresponding point search units 13 respectively corresponding to a plurality of different search resolutions SR. It may be set by a method of selecting the corresponding point search unit 13 corresponding to the desired search resolution SR by switching the switch.

○ About the setting range of search resolution SR:
In order to perform corresponding point search with sub-pixel resolution, search resolution SR needs to be a value smaller than 1 corresponding to pixel resolution. If the search resolution SR is equal to or smaller than the square of the resolution scaling factor RR, the region of the subpixel corresponding point CSm searched on the reference image RIm of the target layer is the reference image RIm. Since the pixel size of the reference image RI (m + 2) higher by two layers is smaller than the pixel size, the reference image RI (m + 1) higher by one cannot be used effectively. For this reason, the setting range of the search resolution SR is limited to a range of RR ² <SR <1.

  That is, by using the search resolution SR in the above range, the subpixel resolution for obtaining the corresponding point is finer than the pixel size of the reference image in the target layer and higher than the pixel size of the reference image in the layer two higher than the target layer. Since it can be set roughly, it is possible to efficiently perform the corresponding point search process by the multi-resolution strategy in which the images of each layer of the multi-resolution image are used without waste and the corresponding points are sequentially obtained for each layer. .

◎ Reliability calculation unit 14 and reliability determination unit 15:
The reliability calculation unit 14 and the reliability determination unit 15 are functional blocks that determine the reliability of the pixel corresponding point search process in cooperation. The reliability calculation unit 14 performs a reliability calculation process of calculating the reliability of the pixel corresponding point search process based on the correlation value between the images of the reference window and the reference window related to the pixel corresponding points, and the reliability determination unit 15 The reliability is compared with a predetermined threshold TH1, and if the reliability is equal to or higher than the threshold TH1, it is determined that the reliability of the pixel corresponding point search process is high, and otherwise, the reliability of the pixel corresponding point search process is high. A reliability determination process for determining that the value is low is performed.

  As a result of the reliability determination process, the subpixel search unit 13b performs the subpixel corresponding point search process only when it is determined that the reliability of the pixel corresponding point search process is high. Implementation of the search process can be prevented.

  In the present embodiment, the reliability calculation unit 14 and the reliability determination unit 15 are independent functional blocks. For example, the corresponding point search unit 13 also functions as the reliability calculation unit 14 and is described later. 16 may also function as the reliability calculation unit 14.

  Also, for example, when it is confirmed in advance that the reliability of the pixel corresponding point search process is sufficiently high, the reliability calculation process and the reliability determination process may be omitted, and the reliability calculation unit 14 and the reliability The determination unit 15 may be omitted from the components of the image processing apparatus 3A.

Control unit 16:
The control unit 16 realizes information processing such as corresponding point search processing and three-dimensional data creation by comprehensively controlling each unit of the image processing apparatus 3A.

  In the corresponding point search process, the control unit 16 searches the target hierarchy in the multi-resolution standard image MB and the multi-resolution reference image MR that are targets of the corresponding point search process from the lower hierarchy to the higher hierarchy. While sequentially setting the setting unit 12 and the corresponding point searching unit 13, the corresponding reference point setting unit 12 and the corresponding points are executed so that the corresponding point search process is repeatedly executed hierarchically from the lower layer to the upper layer. The search unit 13 is controlled.

  With the configuration described above, the image processing apparatus 3A performs corresponding point search processing based on the multi-resolution strategy. Corresponding point search processing is performed with subpixel resolution, and a search reference point on the reference image RI (m + 1) of the next higher hierarchy is set based on the subpixel corresponding point on the searched reference image RIm of the target hierarchy. Therefore, it is possible to set a highly reliable search reference point in a narrower range than when the corresponding point is not searched with subpixel resolution. As a result, it is possible to employ a multi-resolution image with a reduced number of layers, a standard window and a reference window with a narrower range than when not searched with sub-pixel resolution. Therefore, it is possible to perform the corresponding point search with high accuracy while suppressing the processing cost related to the corresponding point search.

<Description of Operation of Image Processing Device 3A>
Next, the pixel corresponding point search process, the reliability determination process, the subpixel corresponding point search process, the search reference point setting process in the hierarchy one level higher than the target hierarchy, and the like of the image processing apparatus 3A will be described in more detail. The overall processing flow of the corresponding point search processing using the multi-resolution image performed by the device 3A will be described.

◎ Pixel correspondence point search processing:
FIG. 9 is a diagram illustrating an outline of the pixel corresponding point search processing performed by the pixel search unit 13a. The attention point Nm and the search reference point Km in FIG. 9 are respectively displayed on the reference image BIm of the attention layer set in the multiresolution reference image MB and the attention in the multiresolution reference image MR by the search reference point setting unit 12. The attention point and the search reference point set on the reference image RIm of the hierarchy.

  Based on the window sizes WX and WY stored in the storage unit 9, the pixel search unit 13a sets a reference window WBm for the attention point Nm and a reference window WRm for the search reference point Km, respectively. A pixel corresponding point CPm corresponding to the attention point Nm is obtained by obtaining a correlation between the image in WBm and the image in the reference window WRm.

  Note that the number of pixels in the horizontal direction (X direction) of each of the standard window WBm and the reference window WRm is the same, and the number of pixels in the vertical direction (Y direction) is also the same.

○ Reference window and reference window:
FIG. 10 is a diagram illustrating a setting example of the reference window WRm for the search reference point Km set in the reference image WRm.

  The reference window WRm illustrated in FIG. 10A has an odd number of horizontal pixels and vertical pixels, and the coordinates of the search reference point Km coincide with the center of gravity Wg of the reference window WRm.

  The reference window WRm illustrated in FIG. 10B has an even number of horizontal pixels and vertical pixels, and the coordinates of the search reference point Km are in the −X direction with respect to the center of gravity Wg of the reference window WRm. , Shifted by one pixel in the -Y direction.

  In addition to the example shown in FIGS. 10A and 10B, the horizontal and vertical pixel numbers of the base window WBm and the reference window WRm may be a pair of odd and even numbers (in no particular order).

  Here, as a technique for obtaining the correlation between the images of the standard window WBm and the reference window WRm, for example, a technique that does not perform frequency decomposition such as the SAD method (difference absolute value sum method), and a POC method (phase-only correlation method). ) Etc., there is a method for performing frequency decomposition, and when using a method for performing frequency decomposition, fast Fourier transform (FFT) is used to suppress the required number of computations, from the viewpoint of speeding up the processing. It is desirable that the number of pixels in the horizontal direction and the number of pixels in the vertical direction be the number of pixels represented by a power of two.

  Next, a technique for obtaining a correlation between images of the standard window WBm and the reference window WRm will be described using the SAD method and the POC method as examples.

○ SAD method:
FIG. 11 is a diagram illustrating a pixel corresponding point search processing procedure using the SAD method.

  An attention point Nm whose X coordinate is h2 is set on the reference image BIm of the attention layer shown in FIG. 11, and a reference window WBm is set for the attention point Nm. The X coordinates of both ends in the X direction of the reference window WBm are h1 and h3. A pixel corresponding point CPm corresponding to the target point Nm is shown on the reference image RIm of the target layer.

  Here, an example of obtaining the pixel corresponding point CPm using the SAD method will be described.

  On the reference image RIm, there are a plurality of search reference points for each pixel in the range of xs ≦ x ≦ xe from the search reference point Kms whose X coordinate is xs to the search reference point Kme whose X coordinate is xe. Is set, and only the search reference points Kms and Kme at both ends are described as representatives of the plurality of search reference points.

  In addition, a reference window WBm and a plurality of reference windows of the same size are set for each of the plurality of search reference points, and only reference windows WRms and WRme at both ends are described as representatives of the plurality of reference windows. ing.

  Here, in order to simplify the description in FIG. 11, the Y coordinate of the attention point Nm and the plurality of search reference points Kms to Kme are the same. Therefore, the reference window WBm and the reference windows WRms to WRme in the Y direction are the same. The coordinates are set so that there is no deviation. The attention point Nm and the Y coordinate of the pixel corresponding point CPm to be obtained are also set to coincide.

  Next, a correlation value between each image in each reference window of the reference windows WRms to WRme and an image in the standard window WBm is calculated. The amount of deviation (also referred to as “corresponding deviation amount”) in the X-direction and Y-direction coordinates of the reference window and the reference window is assumed to be xd and yd, respectively. In the example of FIG. 11, the range of xd is xs-h2 to xe-h2, and yd is 0.

  The pixel value of each pixel (xi, yj) in the reference window WBm is set to Img1 (xi, yj), and the corresponding shift amount in the X direction and Y direction is xd, yd. Each pixel (xi + xd, yj + yd) in the reference window WRm. Assuming that the pixel value of Img2 (xi + xd, yj + yd), the correlation value CORp between the image in the reference window WBm and the image in the reference window WRm is calculated by the calculation using the equations (1) and (2).

  While changing the corresponding deviation amount xd in the X direction in the range of xs−h2 to xe−h2 so that the reference window is scanned from WRms to WRme, each corresponding deviation amount xd is expressed by Expressions (1) and (2). When the correlation value CORp is calculated, the relationship between the X coordinate of the reference image RIm and the correlation value CORp is clarified as shown in a graph G1 in FIG.

  In the example of FIG. 11, since the maximum value of the correlation value CORp in the graph G1 is given at the coordinate xM, the reference window in which the X coordinate of the search reference point matches xM is the reference that gives the maximum value of the correlation value CORp. The search reference point of the reference window matches the pixel corresponding point CPm corresponding to the point of interest Nm.

  FIG. 12 is a diagram illustrating a scanning mode of the reference window WRm in the SAD method, and the reference window WRm is scanned over the entire area of the reference image RIm of the target hierarchy. Since the SAD method has a smaller calculation amount than the POC method and can perform high-speed processing, the corresponding point search processing by the SAD method is applied to the reference image RIm having a large number of pixels, and the whole area scanning shown in FIG. 12 is performed. Also shortens the processing time.

  Therefore, when the SAD method is used, the number of hierarchies of the multi-resolution image can be reduced as compared with the case where a method of performing frequency decomposition is used. In addition, the search process can be performed in a shorter time than the case of using the frequency decomposition method even if the corresponding point search is performed by scanning the reference window over the entire area of the image without obtaining a search reference point close to the corresponding point. Since it can be performed, the processing cost for obtaining the search reference point can be reduced.

○ POC method:
FIG. 13 is a diagram illustrating a pixel corresponding point search processing procedure using the POC method.

  The POC method is a correlation method using frequency decomposition and a correlation method in which amplitude components are suppressed. As a similar correlation method, there is known a DCT code limited correlation method (reference paper: “Fusion of image signal processing and image pattern recognition—DCT code limited correlation and its application”, Hitoshi Takatsuka). Since these correlation methods perform similarity calculation using a signal of only a phase component with an amplitude component suppressed from a frequency resolution signal of a pattern, a difference in imaging conditions in the stereo camera 2 for acquiring an image (here, The difference between the imaging conditions in the imaging systems 21 and 22) and the influence of noises and the like, and robust corresponding point search is possible.

  Further, since the SAD method outputs only one correlation value (CORp value) for one set of the standard window and the reference window, the reference window having the highest correlation value, that is, the correspondence deviation amount having the highest correlation value is obtained. In order to specify, it is necessary to scan a reference window over a plurality of reference windows, whereas a correlation method using frequency decomposition such as the POC method is performed on an image of a set of a reference window and a reference window. Thus, since the correlation value between the images having the highest correlation value is obtained together with the correlation value between the images, it is not necessary to scan the reference window when the pixel corresponding point exists in the reference window. .

  For this reason, according to the corresponding point search method using the POC method for the multi-resolution image, the lowest base image of the multi-resolution image having the smallest number of pixels and the entire image area of the reference image are set as the initial standard window and the reference window, respectively. After searching for the correct corresponding point, a reference window is set in the upper hierarchy based on the corresponding point, and a search process for finding the correct corresponding point in the hierarchy without scanning the reference window is performed over all the hierarchies. It becomes possible to repeat. Therefore, the corresponding point search method using the POC method is a method suitable for the corresponding point search processing for multi-resolution images.

  In the following, an example in which a corresponding point is searched using an image in the reference window WBm for the reference image BIm in a certain target layer and an image in the reference window WRm for the reference image RIm in the target layer as an object of the corresponding point search process. Next, the basic principle of the POC method will be described.

  Assume that the image area in the reference window WBm and the image area in the reference window WRm are expressed by the following equations (3) and (4).

  Here, f (n1, n2) in the expression (3) and g (n1, n2) in the expression (4) indicate the pixel values of the image area in the standard window WBm and the pixels in the reference window WRm. N1 and N2 are set as N1 = 2M1 + 1 and N2 = 2M2 + 1, for example.

  Next, for each image region in the standard window WBm and the reference window WRm, the two-dimensional Fourier transform processes T1a and T1b shown in FIG. 13 are performed using the arithmetic expressions shown in the expressions (5) and (6). Done.

  Note that N1 and N2 are substituted for the subscript P of W in the lower proviso of the formulas (5) and (6), and 1 and 2 are substituted for the subscript s of k. Here, the frequency decomposition process is not necessarily Fourier transform, and may include any of discrete cosine transform, discrete sine transform, wavelet transform, or Hadamard transform.

  For each image region subjected to such Fourier transform processing T1a and T1b, normalization processing T2a and T2b for removing the amplitude component of the image is performed using the arithmetic expression shown in equation (7). Is called.

  When the normalization processes T2a and T2b are completed, a synthesizing process (cross power spectrum calculation process) T3 using the arithmetic expression shown in the expression (8) is performed.

  When the synthesizing process T3 is completed, a two-dimensional inverse Fourier transform process T4 is performed using the arithmetic expression shown in the equation (9). As a result, the correlation calculation between the images in the standard window WBm and the reference window WRm is performed, and the result (POC value) is output.

  By the above processing, a calculation result (POC value) indicating the correlation between the image in the standard window WBm and the image in the reference window WRm is obtained. For example, a result (POC value) as shown in FIG. 14 is obtained.

  FIG. 14 is a diagram illustrating an example of the POC value. In FIG. 14, the POC value at a location with high correlation in the window (N1 × N2) is large, and the coordinates giving the peak S of the POC value are the image in the standard window WBm and the image in the reference window WRm. The correspondence deviation amount that gives the highest correlation is shown. Based on the correspondence shift amount, a pixel corresponding point Cpm in the reference window WRm corresponding to the attention point Nm in the standard window WBm is obtained.

  When the correlation value, that is, the value of the peak S of the POC value, which is the reliability, is determined not to be sufficiently large by the reliability determination process described later, the reference window WBm is shifted by half the window size, for example. Then, the reference window WBm is reset and the corresponding point search process using the POC method is performed again.

  As described above, according to the corresponding point search process using the POC method, the amplitude component of the image is removed, and the correlation calculation is performed using only the phase component of the image. Points are detected with high accuracy.

◎ Reliability judgment processing:
Next, the effect of the reliability determination process of the pixel corresponding point search performed by the reliability calculation unit 14 and the reliability determination unit 15 will be described.

  FIG. 15 is a diagram illustrating an example of sub-pixel corresponding point search based on pixel corresponding points with low reliability, and CPmT on the reference image RIm is correct pixel correspondence with respect to the reference point set on the reference image of the target hierarchy. CPmF indicates an erroneous pixel corresponding point, and AX and AY are the pixel lengths of the pixel corresponding point CPmF in the horizontal direction and the vertical direction, respectively, and the existence range of the pixel corresponding point CPmF in the reference image RIm Is shown.

  Here, although the correct subpixel corresponding point CSmT exists in the correct pixel corresponding point CPmT, the subpixel corresponding point is searched in the region of the pixel corresponding point. When the point CPmF is searched, the subpixel corresponding point is also erroneously searched for, for example, the subpixel corresponding point CSmF.

  In order to prevent such a waste of searching for an erroneous subpixel corresponding point, it is effective to determine the reliability of the pixel corresponding point search process prior to the subpixel corresponding point search process.

  That is, when the reliability of the pixel corresponding point is low by performing the reliability determination of the pixel corresponding point search process, the sub pixel corresponding point search process is not performed, and this is due to the low reliability of the pixel corresponding point. Since the failure of the subpixel corresponding point search process can be prevented, the processing cost related to the corresponding point search can be suppressed.

  Further, in this case, for example, based on the subpixel corresponding point CS (m−1) in the next lower layer of the target layer, the search reference point K (m + 1) in the upper layer of the target layer is determined, If the pixel corresponding point CP (m + 1) is searched, the region length of the subpixel corresponding point CS (m−1) is SR / RR times the pixel length of the target layer, and therefore the search resolution SR is the resolution scaling factor RR. And the pixel corresponding point CP (m + 1) based on the sub-pixel corresponding point CS (m−1) with a fine search resolution equal to or lower than that when searching based on the pixel corresponding point CPm. Can be searched.

  Here, for example, the CORp value in the SAD method and the POC value in the POC method indicate the level of correlation between the image in the reference window and the image in the reference window, and are obtained by the pixel corresponding point search process. The CORp value or the POC value for the pixel corresponding point can be used as it is as the reliability of the pixel corresponding point search obtained by the reliability calculation unit 14. That is, for example, when the reliability is used as it is without performing normalization processing or the like on the POC value, the corresponding point search unit 13 can also function as the reliability calculation unit 14. In addition, the reliability calculation unit 14 may perform the normalization process on the CORp value, the POC value, or the like that gives the pixel corresponding point CPm, and the reliability may be obtained.

◎ Subpixel corresponding point search processing:
FIGS. 16 and 17 show an example of the subpixel corresponding point search process performed by the subpixel search unit 13b, and FIG. 16 is an explanatory diagram of an example of performing the subpixel corresponding point search process by so-called equiangular fitting. is there. Here, the search resolution SR stored in the storage unit 9 is adopted as the subpixel resolution in the subpixel corresponding point search processing.

  RP2 shown in FIG. 16 corresponds to the pixel corresponding point searched by the pixel corresponding point search process, and RP1 and RP3 are adjacent pixels adjacent in the horizontal direction (X direction) with the pixel corresponding point in between. It corresponds to. Here, RP1, RP2, and RP3 are represented by drawing on the graph the reliability R1, R2, and R3 of the pixel corresponding point search processing for the respective coordinates X2-1, X2, and X2 + 1. Here, R2 is the reliability calculated by the reliability calculation unit 15. The provisional subpixel corresponding point RSt indicates a provisional subpixel corresponding point obtained in order to obtain the subpixel corresponding point RS1, and its coordinates are XSt and the reliability is RS.

  When there is a correspondence between the image in the reference window and the image in the reference window, the correlation between the image in the reference window and the image in the reference window is an autocorrelation between the images. For example, FIG. ), When R1> R3, the provisional subpixel corresponding point RS1 includes a straight line L2 connecting the point RP2 and the point RP3 in FIG. 16A and a straight line L2 passing through the point RP1. It is an intersection with a straight line L1 having an inclination with an opposite sign of the inclination. Therefore, the coordinate XSt is given by equation (10).

  Similarly, as shown in FIG. 16B, in the case of R1 ≦ R3, the coordinate XSt is given by the equation (11).

  The vertical coordinate YSt of the temporary subpixel corresponding point RSt is also the same as the coordinate XSt using the coordinates of pixels adjacent to the pixel corresponding point RP2 in the vertical direction and the reliability of the pixel corresponding point search process. It can be calculated.

  Here, the sub-pixel corresponding point RS1 is, for example, FIG. 17B in which the image area is equally divided by the area of the sub-pixel corresponding point determined by the pixel area of the image of the target layer and the search resolution SR. Among the regions exemplified in (1) above, a method for searching for a region including the coordinates of the calculated subpixel corresponding point RSt is obtained.

By these setting methods, the subpixel corresponding point region can be set corresponding to the search resolution SR set in the range of RR ² <SR <1 with respect to the resolution scaling factor RR. Note that since there is no higher hierarchy for the highest-order reference image RIT, the temporary sub-pixel corresponding point RSt may be adopted as the sub-pixel corresponding point RS1 in the highest-order reference image RIT.

  In the example of FIG. 16, subpixel corresponding point search processing using equiangular fitting is performed using the property of autocorrelation. For example, another fitting method using quadratic function approximation or a method using interpolation is used. For example, the subpixel corresponding point search process may be performed.

  Next, FIG. 17 shows the magnitude of the reliability of the pixel corresponding point search for the pixel corresponding point (X, Y) whose coordinates are given by (X, Y) on the reference image RIm of the target layer and its four neighboring pixels. The example of the technique which calculates | requires the coordinate of the subpixel corresponding point on the reference image RIm based on the relationship is shown.

  FIG. 17A shows a pixel corresponding point (X, Y) and four neighboring pixels (X-1, Y), (X + 1, Y), (X, Y-1), and (X, Y + 1). Indicated. Here, the reliability of the pixel corresponding point search and the pixel corresponding point search of the four neighboring pixels are R (X, Y), R (X-1, Y), R (X + 1, Y), R ( X, Y-1) and R (X, Y + 1).

  First, an example of a technique for obtaining the subpixel corresponding point with a search resolution of 1/3 of the pixel size will be described.

  FIG. 17B is a diagram illustrating nine coordinates that can be taken by a subpixel corresponding point when searching for a subpixel corresponding point with a search resolution of ３ of the pixel size.

  The reliability of the pixel corresponding point search for each of the three pixels (X-1, Y), (X, Y), and (X + 1, Y) in the horizontal direction centering on the pixel corresponding point shown in FIG. If the equation (12) holds for R (X-1, Y), R (X, Y), and R (X + 1, Y), the horizontal coordinate XS of the subpixel corresponding point is shown in FIG. ) Is the X of the three horizontal coordinates shown in FIG. 5 and when the formula (12) does not hold and the formula (13) holds, it is X + 1/3, the formula (12) does not hold, and If the equation (13) is not satisfied, the horizontal coordinate XS can be determined with a search resolution of 1/3 of the pixel size by assuming that X-1 / 3. Here, TH2 is a predetermined threshold value stored in the storage device 9.

  Similarly, the reliability R (X, Y, X) of the pixel corresponding point search for each of the three pixels (X, Y−1), (X, Y), and (X, Y + 1) in the vertical direction centered on the pixel corresponding point. If equation (14) holds for Y-1), R (X, Y) and R (X, Y + 1), the vertical coordinate YS of the subpixel corresponding point is shown in FIG. Of the three vertical coordinates, Y is Y, the expression (14) is not satisfied, and the expression (15) is satisfied, Y + 1/3, the expression (14) is not satisfied, and the expression (15) If Y 1/3 is not established, the vertical coordinate YS can be determined with a search resolution of 1/3 of the pixel size.

  Next, an example of a technique for obtaining the subpixel corresponding point with a search resolution ½ of the pixel size will be described.

  FIG. 17C is a diagram illustrating four coordinates that can be taken by a subpixel corresponding point when searching for a subpixel corresponding point with a search resolution that is ½ of the pixel size.

  First, the reliability R (X−1, Y) of the pixel corresponding point search of each of the two horizontal pixels (X−1, Y) and (X + 1, Y) across the pixel corresponding point shown in FIG. When the equation (16) is established for Y) and R (X + 1, Y), the horizontal coordinate XS of the subpixel corresponding point is the X− of the two horizontal coordinates shown in FIG. If it is 1/4 and equation (16) does not hold, it is possible to determine the horizontal coordinate XS with a search resolution of 1/2 the pixel size by setting X + 1/4.

  Similarly, the reliability R (X, Y, X) of the pixel corresponding point search for each of the two pixels (X, Y−1) and (X, Y + 1) in the horizontal direction across the pixel corresponding point shown in FIG. When the equation (17) is satisfied for Y-1) and R (X, Y + 1), the vertical coordinate YS of the subpixel corresponding point is Y-1 / 4, and the equation (17) is not satisfied. Is Y + 1/4, the vertical coordinate YS can be determined with a search resolution of 1/2 the pixel size.

  In addition, according to the above-described method for obtaining the subpixel corresponding point with the subpixel resolution of 1/2 or 1/3 of the pixel size, the pixel without directly calculating the coordinates of the subpixel corresponding point by fitting or interpolation. Since the coordinates of the subpixel corresponding point can be determined only by comparing the magnitude relationship of the reliability of the neighboring pixels of the corresponding point, the cost of the subpixel corresponding point search process can be reduced.

◎ Correspondence between subpixel corresponding point and search reference point of one higher hierarchy:
FIG. 18 is a diagram for explaining the correspondence between the subpixel corresponding point in the target layer and the search reference point in the next higher layer.

  FIG. 18A shows a state in which the subpixel corresponding point (XS, YS) is obtained with a search resolution of ½ the pixel size in the reference image RIm of the target layer. Here, the horizontal and vertical pixel numbers of the reference image RIm are Xn and Yn, the coordinates of the upper left corner pixel are (1, 1), and the upper left corner coordinates of the reference image RIm are (0.5, 0). .5). AX and AY indicate the pixel lengths in the horizontal direction and the vertical direction, respectively.

  In the description of the present embodiment, the coordinates of the pixels and corresponding points are given by the center of gravity (center) of the area of the pixels and corresponding points.

  FIG. 18B shows a state in which the search reference point (XK, YK) is set in the reference image RI (m + 1) that is one layer higher than the target layer based on the subpixel corresponding point (XS, YS). Is shown. Here, the horizontal and vertical pixel numbers of the reference image RI (m + 1) are Xm and Ym, the coordinates of the upper left corner pixel are (1, 1), and the upper left corner coordinates of the reference image RI (m + 1) are (0.5, 0.5). In addition, AX ′ and AY ′ indicate the pixel lengths in the horizontal direction and the vertical direction, respectively.

  The horizontal length LXS between the upper left corner of the reference image RIm and the subpixel corresponding point (XS, YS) is expressed by the equation (18), and the horizontal length LYS is expressed by the equation (19).

  When the resolution scaling factor when creating the reference image RIm from the reference image RI (m + 1) is set to RR, in order to associate the reference image RIm with the reference image RI (m + 1), the reference image RIm The scale needs to be enlarged to 1 / RR times. Therefore, the relationship between the pixel lengths AX and AY and the pixel lengths AX ′ and AY ′ is given by the equation shown in the proviso below the equation (19).

  The coordinates of the search reference point (XK, YK) set on the reference image RI (m + 1) are represented by the pixel resolution of the reference image RI (m + 1). Accordingly, the horizontal coordinate XK of the reference image RI (m + 1) is obtained by dividing the horizontal length LXS projected on the reference image RI (m + 1) by the pixel length AX ′, and then truncating the fractional part (20) It is given by the formula.

  Similarly, the horizontal coordinate YK is given by equation (21).

◎ Examples of setting search reference points:
Next, an example of setting the search reference point K (m + 1) on the reference image RI (m + 1) one level higher than the target layer with respect to the subpixel corresponding point CSm on the reference image RIm of the target layer will be described.

A. When the resolution scaling value is equal to the search resolution value:
FIG. 19 is a diagram illustrating a setting example of the search reference point K (m + 1) for the subpixel corresponding point CSm when the resolution scaling factor RR and the search resolution SR are both equal and ½.

  FIG. 19A shows a subpixel corresponding point CSm searched with a search resolution SR of 1/2 in the reference image RIm, and FIG. 19B shows a reference having twice the resolution of the reference image RIm. An image RI (m + 1) is shown, and FIG. 19C shows a search reference point K (m + 1) set in the reference image RI (m + 1).

  Here, as the coordinates of the search reference point K (m + 1) on FIG. 19 and FIGS. 20 to 23 described later, for example, the coordinates of the subpixel corresponding point CSm are substituted into the expressions (20) and (21). It is calculated by.

  As shown in FIG. 19, when the resolution scaling factor RR and the search resolution SR are equal, the region of the subpixel corresponding point CSm and the pixel region of the search reference point K (m + 1) match. That is, the corresponding point CSm in the reference image RIm in the target layer can be searched with the same search resolution as the pixel size of the search reference point K (m + 1) in the reference image RI (m + 1) in the layer one level higher than the target layer. Therefore, it is possible to efficiently perform a corresponding point search based on the multi-resolution strategy for the hierarchical structure of the multi-resolution image at a suitable processing cost.

  In this case, the setting of the search reference point K (m + 1) may be omitted, and the reference window W (m + 1) of the reference image RI (m + 1) may be set based on the coordinate information of the subpixel corresponding point CSm. .

B. If the search resolution is smaller than the resolution scaling factor:
FIG. 20 is a diagram illustrating a setting example of the search reference point K (m + 1) for the subpixel corresponding point CSm when the resolution scaling factor RR is 1/2 and the search resolution SR is 1/3.

  FIG. 20A shows a subpixel corresponding point CSm searched with a search resolution SR of 1/3 in the reference image RIm, and FIG. 20B shows a reference having twice the resolution of the reference image RIm. FIG. 20C shows the image RI (m + 1), and the reference image RIm enlarged to 1 / SR times is superimposed on the reference image RI (m + 1), and the subpixel corresponding point CSm is placed on the reference image RI (m + 1). FIG. 20D shows a projected reference point K (m + 1) set in the reference image RI (m + 1).

  FIG. 21 is a diagram for explaining the effect of setting the subpixel corresponding point CSm shown in FIG. 20, and the subpixel corresponding point CSm is projected onto the reference image RI (m + 1) as in FIG. ing. Here, the pixel corresponding points CP (m + 1) T and CP (m + 1) F are references that can be searched as pixel corresponding points in the reference window set based on the search reference point K (m + 1) in FIG. Two pixels on the image RI (m + 1) are illustrated.

  Since the pixel corresponding point closer to the sub-pixel corresponding point CSm is estimated to have higher reliability, in FIG. 21, CP (m + 1) T is a pixel corresponding point that has been correctly searched for, and CP (m + 1) It is estimated that F is a pixel corresponding point that has been searched in error.

  That is, when the search resolution SR is finer than the resolution scaling factor RR, as shown in FIG. 21, the reference image of the target layer with a search resolution finer than the pixel size of the reference image of the layer one level higher than the target layer. Can be searched for, so that it is possible to verify the reliability of the corresponding point searched on the reference image one level higher than the target layer based on the corresponding point searched for the reference image of the target layer. The reliability of the corresponding point search can be increased.

C. When the search resolution value is larger than the resolution scaling factor value:
FIG. 22 is a diagram illustrating a setting example of the search reference point K (m + 1) for the subpixel corresponding point CSm when the resolution scaling factor RR is 1/3 and the search resolution SR is 1/2.

  FIG. 22A shows a subpixel corresponding point CSm searched with a search resolution SR of 1/2 in the reference image RIm, and FIG. 22B shows a reference having a resolution three times that of the reference image RIm. FIG. 22C shows the image RI (m + 1). FIG. 22C shows the reference image RIm enlarged to 1 / SR times superimposed on the reference image RI (m + 1), and the subpixel corresponding point CSm is placed on the reference image RI (m + 1). FIG. 22D shows a projected reference point K (m + 1) set in the reference image RI (m + 1).

  FIG. 23 is a diagram for explaining the effect of setting the sub-pixel corresponding point CSm shown in FIG.

  FIG. 23A shows an example in which the subpixel corresponding point CSm shown in FIG. 22C is projected onto the search reference point K (m + 1) shown in FIG. , An arrow set in a direction from the coordinates of the search reference point K (m + 1) to the coordinates of the subpixel corresponding point CSm.

  In FIG. 23B, the search reference point K (m + 1), which is one level higher than the reference image RIm, sets the pixel corresponding point CPm on the reference image RIm when the subpixel corresponding point search process is not performed. An example of projection on the reference image RI (m + 1) is shown.

  As shown in FIG. 23B, when the subpixel corresponding point search process is not performed, the coordinates of the search reference point K (m + 1) and the pixel corresponding point CPm coincide. Therefore, even if it is necessary to correct the position of the search reference point K (m + 1) because the reliability of the searched pixel corresponding point is low, it is based on the coordinates of the pixel corresponding point CPm, region information, and the like. The direction of the correction cannot be estimated.

  On the other hand, when the subpixel corresponding point search process is performed, the coordinates of the search reference point K (m + 1) and the subpixel corresponding point CSm are shifted as shown in FIG. When correcting the position of the search reference point K (m + 1), for example, a method of setting the nearest pixel in the direction indicated by the arrow VK to the search reference point K (m + 1) as a new search reference point K (m + 1). Thus, the direction of resetting the search reference point K (m + 1) can be estimated based on the subpixel corresponding point CSm.

  That is, a corresponding point is searched with sub-pixel resolution by using a corresponding point in the reference image of the target layer searched with a search resolution coarser than the pixel size of the search reference point in the reference image one level higher than the target layer. It is possible to estimate the position of the search reference point in the reference image that is one level higher than the target layer with higher accuracy than in the case where the target layer is not performed. A point search can be performed.

◎ Overall processing flow:
Next, an overall processing flow of corresponding point search processing using a multi-resolution image performed by the image processing apparatus 3A will be described.

  24 and 25 are flowcharts showing an outline of the procedure of corresponding point search processing using a multi-resolution image performed by the image processing apparatus 3A according to the embodiment.

  First, the image acquisition unit 10 acquires an image captured by the standard camera 21 and an image captured by the reference camera 22 as a standard image BIT and a reference image RIT, respectively, and stores them in the storage unit 9 (step S30).

  When the image acquisition is completed, the multi-resolution image creating unit 11 stores control parameters related to multi-resolution image creation, such as the number T of layers of the multi-resolution image and the resolution scaling factor RR, the standard image BIT, and the reference image from the storage unit 9. RIT is acquired, the multi-resolution standard image MB and the multi-resolution reference image MR are created by performing multi-stage resolution reduction processing on the standard image BIT and the reference image RIT using the control parameters, and stored in the storage unit 9 (Step S40).

  When the creation of the multi-resolution image is completed, if the target layer is not set, the control unit 16 selects the lowest base image BI1 and the reference image RI1 of the multi-resolution standard image MB and the multi-resolution reference image MR as the target layer. If the standard image BIm and the reference image RIm are newly set, and there is an existing standard image BIm and the reference image RIm, the standard image BI (m + 1) and the reference image RI that are one layer higher than the standard image BIm and the reference image RIm. (M + 1) is reset as the reference image BIm and the reference image RIm of the new target layer, and the setting information is transmitted to the search reference point setting unit 12 and the corresponding point search unit 13 (step S50).

  When the setting of the target hierarchy is completed, the search reference point setting unit 12 acquires control parameters related to the creation of the multi-resolution image from the storage unit 9, and acquires the reference image BIm and the reference image RIm of the target hierarchy. Here, the attention point on the uppermost reference image RIT is set in advance on a feature point on the reference image RIT by the operation unit 4 or the like, and the search reference point setting unit 12 sets the uppermost reference image RIT. Based on the upper attention point and the correspondence relationship of the pixels between the reference images of the multi-resolution reference image MB determined by the control parameter, the attention point Nm is set on the reference image BIm, and the result is stored in the storage unit 9. (Step S60).

  When the setting of the attention point is completed, the search reference point setting unit 12 acquires the reference image RIm from the storage unit 9, sets the search reference point Km for the corresponding point search process on the reference image RIm, and stores the result. Store in the unit 9 (step S70).

  FIG. 25 is a flowchart showing an overview of the processing in step S70. Hereinafter, the processing procedure of step S70 will be described with reference to the flowchart of FIG.

  The search reference point setting unit 12 first acquires the reference image RIm based on the attention layer information set by the control unit 16, and confirms whether the reference image RIm is the lowest reference image RI1 (step S710). ).

  As a result of the confirmation in step S710, if the reference image RIm is the lowest reference image RI1, the search reference point setting unit 12 sets a predetermined pixel (X0, Y0) preset on the reference image RIm as a search reference point. It sets as Km (step S720), and returns to the flowchart of FIG. Here, the predetermined pixels (X0, Y0) are, for example, predetermined search reference points that can set the entire reference image RIm as a reference window.

  As a result of the confirmation in step S710, if the reference image RIm is not the lowest reference image RI1, the search reference point setting unit 12 corresponds to the subpixel in the reference image RI (m−1) that is one layer below the target layer. It is confirmed whether the point CS (m−1) is obtained (step S730).

  As a result of the confirmation in step S730, if the subpixel corresponding point CS (m−1) is obtained, the search reference point setting unit 12 sets the subpixel corresponding point CS (m−1), the expression (20), and ( The search reference point Km is set on the reference image RIm by the equation (21) (step S740), and the process returns to the flowchart of FIG.

  If the sub-pixel corresponding point CS (m−1) is not obtained as a result of the confirmation in step S730, the search reference point setting unit 12 determines the pixel corresponding point CP obtained on the reference image RI (m−1). The search reference point Km is set by (m-1), (20) and (21) (step S750), and the process returns to the flowchart of FIG.

  In the flowchart of FIG. 24, the processing is restarted from step S80, and the corresponding point search unit 13 stores the reference image BIm of the target hierarchy, the reference image RIm, the target point Nm, the search reference point Km, and the window size WX from the storage unit 9. , WY and search resolution SR are acquired, and the pixel search unit 13a sets the reference window WBm and the reference window WRm on the reference image BIm and the reference image RIm based on the search processing parameters (step). S80).

  Next, the pixel search unit 13a performs a pixel corresponding point search process for obtaining a correlation between the image in the standard window WBm and the image in the reference window WRm, and the reference image corresponding to the target point Nm on the standard image BIm. The pixel corresponding point CPm on RIm is searched, and information on the pixel corresponding point CPm and the correlation value between the image of the reference window and the reference window calculated in the process of obtaining the pixel corresponding point CPm are transmitted to the reliability calculation unit 14. (Step S90).

  Next, the reliability calculation unit 14 obtains the reliability of the pixel corresponding point CPm based on the acquired correlation value (step S100), and the reliability determination unit 15 stores the predetermined reliability stored in the storage unit 9. A reliability determination process for checking whether or not the threshold value TH1 is exceeded is performed, and the result is transmitted to the corresponding point search unit 13 (step S110). As a result of the reliability determination process, when the reliability of the pixel corresponding point CPm is less than the predetermined threshold TH1, the control unit 16 ends the corresponding point search process.

  As a result of the reliability determination process, if the reliability of the pixel corresponding point CPm is greater than or equal to the predetermined threshold TH1, the sub-pixel search unit 13b of the corresponding point search unit 13 determines the pixel corresponding point CPm and its corresponding point based on the search processing parameter. Sub-pixel corresponding point search processing using the coordinates and reliability of each neighboring pixel is performed to obtain a sub-pixel corresponding point CSm on the reference image RIm (step S120).

  When the subpixel corresponding point search processing is completed, the control unit 16 checks whether the corresponding point search processing has been completed for all layers. When the search process is finished and the search process for all layers is not finished, the process returns to step S50 (step S130).

  With the overall processing flow of the corresponding point search process described above, the corresponding point search process based on the multi-resolution strategy is performed from the lower layer to the upper layer, and finally corresponds to the attention point on the uppermost reference image BIT. Corresponding points on the reference image RIT are obtained.

<Modification>
The present invention is not limited to the above-described embodiments, and various changes and improvements can be made without departing from the scope of the present invention.

  For example, the reliability calculation unit 14 and the reliability determination unit 15 cooperate to perform at least one of the reliability determination of the pixel corresponding point search process and the reliability determination of the subpixel corresponding point search process, and the reliability As a result of the determination process, when it is determined that the reliability of at least one of the pixel corresponding point search process and the sub-pixel corresponding point search process in the target layer is low, for example, the image of the target layer and the one higher in the target layer By using an image having an intermediate resolution with the image in the hierarchy, the corresponding point search based on the corresponding point in the target hierarchy determined to have low reliability may be continued. Hereinafter, specific examples will be described.

<Regarding Corresponding Point Search of Image Processing Apparatus According to Modification>
In the search for corresponding points that define the search range, if there is no correct corresponding point in the search range, even if there is a large amount of pixel information and there is little distance conflict in the search range of the corresponding points, The search fails.

  Further, as described above, in a multi-resolution image, the lower the hierarchy, the smaller the resolution scaling factor when creating a multi-resolution image, the smaller the “ A decrease in the amount of information occurs, and so-called “distance competing” is likely to occur within the search range, so that a corresponding point search error is likely to occur.

Here, based on the search result of the attention layer for which the search for the corresponding point has failed, the search for the corresponding point in the one higher layer of the target layer can be continued, and the search for the corresponding point in the upper layer can be successfully performed. For example, there is a method of increasing the probability that a corresponding point exists in the search range while maintaining the information amount of pixels in the upper layer by increasing the number of pixels in the search range in the upper layer and expanding the search range. It is done.

  However, with this method, since the angle of view of the new search range is wider than the original search range in the upper hierarchy, perspective conflict is likely to occur, and the processing time also increases. Further, when the corresponding point search unit is configured by, for example, a hardware circuit, it is generally not easy to increase the number of pixels in the search range.

  Therefore, in the modified example, the original standard image and the original reference image are reduced in resolution so that the resolution is higher than that of the target layer and lower than that of the image one layer higher than the target layer. An image (also referred to as an “auxiliary reference image”) and a reference image (also referred to as an “auxiliary reference image”) are provided, the attention layer in each hierarchical structure of the multi-resolution standard image and the multi-resolution reference image, and one of the attention layers A search reference point is set in the auxiliary reference image based on the search result of the attention layer in which the search for the corresponding point has failed in the intermediate layer. Is set to provide a search range having the same number of pixels as the other layers, and corresponding points in the intermediate layer are searched.

  As a result of the search, when the reliability of the search in the intermediate hierarchy is high, a corresponding point search is performed in one higher hierarchy of the target hierarchy based on the corresponding points searched in the intermediate hierarchy, and the search in the higher hierarchy is performed. Processing is performed sequentially.

  According to the modification, the probability that a correct corresponding point exists in the search range in one of the intermediate hierarchies is set to one of the target hierarchies by expanding the angle of view of the search range in the intermediate hierarchies than one higher hierarchy of the target hierarchies. It is possible to search for corresponding points in the middle layer because it can be higher than the upper layer and can search for corresponding points in a state in which the amount of pixel information is larger than the target layer and perspective conflicts are less likely to occur. The probability of success can be increased, and therefore the probability of successful search for corresponding points based on the multi-resolution strategy can be increased.

<Functions of Image Processing Device 3B>
FIG. 2 is a diagram illustrating functional blocks of an image processing device 3B according to a modification.

  As shown in FIG. 2, the components of the image processing apparatus 3B are the same except for the multi-resolution image creation unit 11B, the search reference point setting unit 12B, the reliability calculation unit 14B, the reliability determination unit 15B, and the control unit 16B. It has the same components as the components of the image processing apparatus 3A shown in FIG.

  Here, the functions of the multi-resolution image creation unit 11B, the search reference point setting unit 12B, the reliability calculation unit 14B, the reliability determination unit 15B, and the control unit 16B of the image processing device 3B are respectively corresponding configurations in the image processing device 3A. Explain the difference from the function of the element.

Multi-resolution image creation unit 11B:
The multi-resolution image creating unit 11B has a function different from that of the multi-resolution image creating unit 11 as to the resolution scaling factor for the images of each layer excluding the lowest layer image of the created multi-resolution standard image MB and multi-resolution reference image MR. A function of creating a multi-resolution standard image MB ′ and a multi-resolution reference image MR ′ by storing the resolution in one step based on RR ′ and storing it in the storage unit 9 is provided.

  Here, a value larger than the resolution scaling factor RR is set in the resolution scaling factor RR ′. For example, when the resolution scaling factor RR is 1/4, for example, a value such as 1/3 or 1/2 is set in the resolution scaling factor RR ′.

  Therefore, the resolution of each layer of the multi-resolution standard image MB ′ and the multi-resolution reference image MR ′ is the same as the multi-resolution standard image MB and the multi-resolution reference that are the targets of the resolution reduction when the image of each layer is created The resolution is lower than the image of each layer of the image MR, and the resolution is higher than the image of the next lower layer of each image subjected to the resolution reduction.

  The multi-resolution standard image MB ′ and the multi-resolution reference image MR ′ are searched for corresponding points on the target reference image RIm of the multi-resolution reference image MR corresponding to the target point on the target standard image BIm of the multi-resolution standard image MB. When the search process to be performed fails, it is used to succeed the process of searching for the corresponding point of the auxiliary reference image corresponding to the attention point of the auxiliary reference image based on the failed search result.

  In addition to the method of creating the multi-resolution standard image MB ′ and the multi-resolution reference image MR ′ in advance as in the above example, for example, every time the corresponding point search process fails, the attention that has failed is generated. Adopting a method of creating a set of auxiliary reference images and auxiliary reference images in the intermediate layer by reducing the resolution in one step based on the resolution scaling factor RR ′ to the image one layer above the layer Also good.

  It should be noted that the images in the intermediate layer created by these methods have an original base image BIT and an original reference image RIT that have a higher resolution than the target layer corresponding to the intermediate layer and are one level higher than the target layer. The reference image (auxiliary reference image) and the reference image (auxiliary reference image) are reduced in resolution so that the resolution is lower than that of the image.

Search reference point setting unit 12B:
The search reference point setting unit 12B has a function different from the search reference point setting unit 12 in the case where the corresponding point search process fails in the target hierarchy of the multi-resolution standard image MB and the multi-resolution reference image MR, that is, the pixel corresponding point. When it is determined that the reliability of the search process is low or the reliability of the subpixel corresponding point search process is low, the searched pixel corresponding point or the subpixel corresponding point searched based on the pixel corresponding point And a function for setting a search reference point on the auxiliary reference image corresponding to the target hierarchy.

◎ Reliability calculation unit 14B and reliability determination unit 15B:
The reliability calculation unit 14B and the reliability determination unit 15B are functional blocks that determine the reliability of at least one of the pixel corresponding point search process and the subpixel corresponding point search process in cooperation.

  Specifically, the reliability calculation unit 14B represents the reliability of the first reliability calculation process for calculating the first reliability representing the reliability of the pixel corresponding point search process and the sub pixel corresponding point search process. At least one of the second reliability calculation processes for calculating the second reliability is performed.

  In addition, the reliability determination unit 15B compares the first reliability with a predetermined threshold TH3 and determines that the reliability of the pixel corresponding point search process is high if the first reliability is equal to or higher than the threshold TH3. Otherwise, the first reliability determination process for determining that the reliability of the pixel corresponding point search process is low, and the second reliability are compared with a predetermined threshold TH4, and the second reliability is equal to or higher than the threshold TH4. If so, it is determined that the reliability of the subpixel corresponding point search process is high, and if not, at least one of the second reliability determination processes is determined to determine that the reliability of the subpixel corresponding point search process is low. Here, the threshold value TH3 is a threshold value corresponding to the threshold value TH1 related to the reliability determination unit 15 of the image processing apparatus 3A, and the threshold value TH3 and the threshold value TH4 may be different values or the same value. Also good.

  In the present embodiment, the reliability calculation unit 14B and the reliability determination unit 15B are independent functional blocks. For example, the corresponding point search unit 13 also functions as the reliability calculation unit 14B, and will be described later. The unit 16B may also use the function of the reliability calculation unit 14B.

○ Second reliability calculation processing:
FIG. 26 is a diagram illustrating an example of a second reliability calculation process for obtaining the reliability (second reliability) of the subpixel corresponding point search process performed by the reliability calculation unit 14B.

  In the example shown in FIG. 26, the reliability of the subpixel corresponding point search process is obtained by fitting using a quadratic function. RP2 is a pixel corresponding point searched by the pixel corresponding point searching process performed by the pixel search unit 13a, and RP1 and RP3 are adjacent pixels adjacent in the horizontal direction (X direction) with the pixel corresponding point interposed therebetween. . The X coordinates X2-1, X2, and X2 + 1 and the pixel corresponding point search process reliability R1, R2, and R3 corresponding to the adjacent pixel RP1, the pixel corresponding point RP2, and the adjacent pixel RP3 are determined from the corresponding point search unit 13. It is supplied to the degree calculation unit 14B. The reliability R2 corresponds to the first reliability obtained by the reliability calculation unit 15B.

  The function f1 is a quadratic function fitted to the adjacent pixel RP1, the pixel corresponding point RP2, and the adjacent pixel RP3 shown in FIG. 26, and the pixel coordinates of the adjacent pixel and the pixel corresponding point and the reliability It is a function indicating the relationship. The function f1 is determined by substituting the respective X coordinates and reliability of the three points RP1, RP2, and RP3 into the quadratic function expressed by the equation (22) to obtain the coefficients a and b and the constant term c. Is done.

  In this example, since the function f1 is determined based on three points, the values of a, b, and c are uniquely determined. However, when four or more points are used, a, b are determined by the least square method. And the value of c may be obtained.

  When the function f1 is determined, the X coordinate XSt and the reliability RS of the temporary subpixel corresponding point RSt are obtained by obtaining the vertex thereof. In addition, you may obtain | require X coordinate XSt and reliability RS using the equiangular fitting similar to the example of FIG.

  Here, the difference between the coordinates of the temporary subpixel corresponding point RSt and the coordinates of the subpixel corresponding point RS1 obtained based on the temporary subpixel corresponding point RSt is the difference between the coordinates of the temporary subpixel corresponding point RSt and the pixel corresponding point RP2. Since it is usually smaller than the difference from the coordinates, the reliability RS can be adopted as the reliability of the subpixel corresponding point RS1. Alternatively, the reliability of the subpixel corresponding point RS1 may be calculated by substituting the coordinates of the subpixel corresponding point RS1 searched by the subpixel searching unit 13b into the obtained function f1.

  Further, when the function f1 is approximately obtained by the least square method for the pixel corresponding point RP2 and a large number of adjacent pixels, the deviation amount between the calculated function f1 and the pixel corresponding point RP2 and each adjacent pixel is calculated. Based on this, the reliability for the subpixel corresponding point RS1 may be obtained.

  Further, a reliability obtained by performing normalization processing or the like on the reliability for the subpixel corresponding point RS1 obtained based on the function f1 may be adopted as the reliability for the subpixel corresponding point RS1. .

  By the various methods described above, the reliability calculation unit 14B calculates the second reliability to obtain the reliability for the subpixel corresponding point, that is, the reliability (second reliability) for the subpixel corresponding point search process. Process.

Control unit 16B:
The control unit 16B has a function different from that of the control unit 16 of the image processing apparatus 3A when the corresponding point search fails in the target hierarchy of the multi-resolution standard image MB and the multi-resolution reference image MR, that is, the pixel corresponding point search in the target hierarchy. On the auxiliary reference image corresponding to the target hierarchy when it is determined that at least one of the processing reliability (first reliability) and the subpixel corresponding point search processing reliability (second reliability) is low A function for controlling the search reference point setting unit 12B so as to set a search reference point, and the auxiliary reference image and the auxiliary reference image in the respective hierarchical structures of the multi-resolution reference image MB and the multi-resolution reference image MR. , And setting the position to be a hierarchy (intermediate hierarchy) between the target hierarchy and the hierarchy one level higher than the target hierarchy. And the corresponding layer based on the multi-resolution strategy from the intermediate layer to the upper layer, while setting the target layer sequentially from the intermediate layer set to the new target layer to the upper layer. It has a function of controlling the search reference point setting unit 12B and the corresponding point search unit 13 so that the search process is performed.

<Operation of Image Processing Device 3B>
Hereinafter, an example of the operation of the image processing apparatus 3B will be described with reference to the drawings.

  FIG. 27 is a diagram illustrating an example of the overall processing flow of the image processing apparatus 3B. 27 are the same as those shown in FIG. 24 except for steps S102 to S128, and therefore the steps for performing the same processing are the same as those shown in FIG. A number is attached and description of operation | movement is abbreviate | omitted.

  As a result of the processing from step S30 to step S90 in FIG. 27, pixel corresponding point search processing is performed by the pixel search unit 13a of the corresponding point search unit 13, and the correlation value between the image in the reference window WBm and the image in the reference window WRm. (Reliability) is obtained by using the POC method or the SAD method for a plurality of corresponding deviation amounts, and the pixel corresponding to the correspondence deviation amount that gives the correlation value having the highest correlation is the attention on the reference image BIm. Search is made as a pixel corresponding point CPm on the reference image RIm corresponding to the point Nm (step S90). Further, the coordinate information and the correlation values for the pixel corresponding point CPm and its neighboring pixels are supplied to the sub-pixel search unit 13b and the reliability calculation unit 14B.

  The reliability calculation unit 14B obtains a first reliability representing the reliability of the pixel corresponding point search process by performing a process such as normalization on the correlation value corresponding to the supplied pixel corresponding point CPm as necessary. The first reliability calculation process is performed (step S102), and the acquired first reliability is supplied to the reliability determination unit 15B.

  Next, the subpixel search unit 13b of the corresponding point search unit 13 performs a subpixel corresponding point search process using the coordinates and correlation values (reliability) of the pixel corresponding point CPm and its neighboring pixels based on the search processing parameters. Then, the subpixel corresponding point CSm on the reference image RIm is searched (step S118).

  Further, the reliability calculation unit 14B obtains a correlation value (reliability) for the sub-pixel corresponding point CSm based on each coordinate information and each correlation value for the pixel corresponding point CPm and its neighboring pixels, and if necessary, A second reliability calculation process for obtaining a second reliability representing the reliability of the sub-pixel corresponding point search process by performing a process such as normalization is performed (step S121), and the acquired second reliability is trusted. It supplies to degree determination part 15B.

  The processing flow of FIG. 27 is an example in which both the pixel corresponding point CPm and the subpixel corresponding point CSm are searched, and both the first reliability and the second reliability are obtained by the reliability calculation unit 14B. However, the reliability calculation unit 14B may obtain the first reliability or the second reliability.

  Next, the reliability determination unit 15B uses the first reliability and the second reliability supplied from the reliability calculation unit 14B, and the predetermined threshold values TH3 and TH4 stored in the storage unit 9, The first reliability is determined by performing a first reliability determination process for checking whether or not the first reliability is greater than or equal to the threshold value TH3 and a second reliability determination process for determining whether or not the second reliability is greater than or equal to the threshold value TH4. Is less than the threshold value TH3 or the second reliability is less than the threshold value TH4, and the determination result is supplied to the corresponding point search unit 13 (step S122). The reliability determination unit 15B may perform at least one of the first reliability determination process and the second reliability determination process.

  The determination result supplied to the corresponding point search unit 13 is supplied to the control unit 16B, and when the first reliability related to the pixel corresponding point CPm is less than the predetermined threshold TH3, or the subpixel corresponding point CSm. If the second reliability related to is less than the predetermined threshold value TH4, the process proceeds to step S123; otherwise, the process proceeds to step S130.

  In step S123, the control unit 16B determines whether or not the current target layer is the auxiliary base image and the auxiliary reference image, that is, the intermediate layer. As a result of the determination, if the target layer is the intermediate layer, the corresponding point search using the auxiliary reference image and the auxiliary reference image of the intermediate layer has already been performed, and the corresponding point search of the previous target layer is performed. Following the failure, the corresponding point search in the current intermediate layer has also failed, so the control unit 16B ends the corresponding point search process.

  Further, when the target layer is the highest layer of the hierarchical structure of each of the multi-resolution standard image MB and the multi-resolution reference image MR, there is no auxiliary standard image and auxiliary reference image corresponding to the target layer. The control unit 16B ends the corresponding point search process.

  As a result of the determination in step S123, when the current target layer is neither the intermediate layer nor the highest layer, the multi-resolution image creating unit 11B follows the control from the control unit 16B to select one of the target layers. An auxiliary standard image and an auxiliary reference image corresponding to the target hierarchy are created by reducing the resolution of the higher-level standard image and reference image based on the resolution scaling factor RR ′ stored in the storage unit 9 (step S124). .

  As described above, the auxiliary standard image and the auxiliary reference image corresponding to the target hierarchy are, for example, the final images of the multi-resolution standard image MB and the multi-resolution reference image MR created in step S40 after the end of step S40. A multi-resolution standard image MB ′ and a multi-resolution reference image MR ′ are created in advance by performing one-step resolution reduction on the image of each layer excluding the lower layer image based on the resolution scaling factor RR ′, and the storage unit In step S124, the auxiliary standard image and the auxiliary reference image corresponding to the target hierarchy may be selected from the multi-resolution standard image MB ′ and the multi-resolution reference image MR ′.

  When the auxiliary reference image and the auxiliary reference image are created, the search reference point setting unit 12B is searched for in the pixel corresponding point CPm searched for the pixel corresponding point in step S90 or in step S118 according to the control from the control unit 16B. Based on the subpixel corresponding point CSm, a search reference point is set on the auxiliary reference image corresponding to the target layer (step S126).

  When step 102 is completed, the reliability determination unit 15B determines whether or not the first reliability is greater than or equal to the threshold value TH3. If the first reliability is greater than or equal to the threshold value TH3, the process is stepped. The process moves to S118, and if the first reliability is less than the threshold TH3, the process moves to Step S123. After the process in Step S124, the search reference point is added to the auxiliary reference image based on the pixel corresponding point in Step S126. May be set.

  When the setting of the search reference point on the auxiliary reference image is completed, the control unit 16B converts the auxiliary reference image and the auxiliary reference image into the attention layer and the attention in the hierarchical structures of the multi-resolution reference image MB and the multi-resolution reference image MR. After positioning the middle layer as a new layer of interest in the middle layer between one layer above the layer, the process proceeds to step S80 (step S118).

  In step S130, the control unit 16B checks whether or not the corresponding point search process has been completed for all the hierarchies. If the search process for all hierarchies has been completed, the control unit 16B ends the corresponding point search process. If the search process has not been completed, the process returns to step S50.

  With the above operation, the image processing device 3B according to the modification can perform the intermediate processing corresponding to the target layer based on the search result of the target layer in which the search has failed even when the search for the corresponding point in the target layer fails. By searching for corresponding points in the hierarchy, the probability of successful search for corresponding points based on the multi-resolution strategy can be increased.

DESCRIPTION OF SYMBOLS 1A 3D shape measuring apparatus 2 Stereo camera 3A, 3B Image processing apparatus 4 Operation part 5 Display part 6 Communication part 7 Input / output part 8 Storage medium 9 Storage part 10 Image acquisition part 11, 11B Multi-resolution image creation part 12, 12B Search Reference point setting unit 13 Corresponding point search unit 13a Pixel search unit 13b Sub-pixel search unit 14, 14B Reliability calculation unit 15, 15B Reliability determination unit 16, 16B Control unit 21, 22 Imaging system DL data line OB Measurement object BIT , BI (T-1), BI (m + 1), BIm, BI1 Reference image RIT, RI (T-1), RI (m + 1), RIm, RI1 Reference image MB Multi-resolution reference image MR Multi-resolution reference image Nm, N (M + 1) Attention point CPm, CPmT, CPmF Pixel corresponding point CSm, RS1 Subpixel corresponding point RSt Pixel corresponding points Km, K (m + 1), K (m + 1) T, K (m + 1) F Search reference point WBm Reference window WRm Reference window Wg Search window centroid RR Resolution scaling factor SR Search resolution WSX, WSY Search window size R1, R2, R3 Reliability AX, AY, AX ', AY' Pixel length

Claims (6)

Image acquisition means for acquiring one input image of two or more input images as an original reference image and another input image as an original reference image;
By reducing the resolution of the original reference image and the original reference image so that the resolution decreases from the upper layer to the lower layer, the plurality of reference images and the plurality of reference images having different resolutions are hierarchically arranged. A multi-resolution image creating means for generating
Of the plurality of reference images and the plurality of reference images, the images belonging to the same attention layer are used as the attention reference image and the attention reference image, and the corresponding point of the attention reference image corresponding to the attention point of the attention reference image is searched. A search reference point setting means for setting a search reference point serving as a reference for the search processing to the attention reference image;
Corresponding point search means for searching for the corresponding points by performing the searching process based on the search reference points;
In each hierarchical structure of the plurality of reference images and the plurality of reference images, the search process is sequentially set from a lower hierarchy to an upper hierarchy, whereby the search process is performed from a lower hierarchy to an upper hierarchy. Control means for controlling the search reference point setting means and the corresponding point search means so as to be executed sequentially toward
With
The corresponding point search means searches for the corresponding point in the target reference image with sub-pixel resolution in which a sub-pixel having a size smaller than the pixel size of the target reference image is an image expression unit,
The search reference point setting means sets a search reference point in a reference image of a layer one level higher than the target layer based on the corresponding points searched with the sub-pixel resolution ,
The multi-resolution image creating means uses a real number r in which the resolution of the target standard image and the target reference image with respect to the standard image and the reference image one level higher than the target layer is 1 / r. And reducing the resolution as expressed as
The corresponding point search means searches for the corresponding point in the target reference image with a search resolution coarser than 1 / r ² of the pixel size of the target reference image and finer than the pixel size of the target reference image. A featured image processing apparatus. An image processing apparatus according to claim 1 ,
Before SL corresponding point searching section, using the real r, in search resolution represented by 1 / r of the pixel size of the target reference picture, image, characterized by searching the corresponding point in the target reference picture Processing equipment. An image processing apparatus according to claim 1 ,
The corresponding point searching means uses the real number r to search for the corresponding point in the target reference image with a search resolution finer than the search resolution represented by 1 / r of the pixel size of the target reference image. A featured image processing apparatus. An image processing apparatus according to claim 1 ,
The corresponding point search means uses the real number r to search for the corresponding point in the target reference image with a search resolution coarser than the search resolution represented by 1 / r of the pixel size of the target reference image. A featured image processing apparatus. Image acquisition means for acquiring one input image of two or more input images as an original reference image and another input image as an original reference image;
By reducing the resolution of the original reference image and the original reference image so that the resolution decreases from the upper layer to the lower layer, the plurality of reference images and the plurality of reference images having different resolutions are hierarchically arranged. A multi-resolution image creating means for generating
Of the plurality of reference images and the plurality of reference images, the images belonging to the same attention layer are used as the attention reference image and the attention reference image, and the corresponding point of the attention reference image corresponding to the attention point of the attention reference image is searched. A search reference point setting means for setting a search reference point serving as a reference for the search processing to the attention reference image;
Corresponding point search means for searching for the corresponding points by performing the searching process based on the search reference points;
In each hierarchical structure of the plurality of reference images and the plurality of reference images, the search process is sequentially set from a lower hierarchy to an upper hierarchy, whereby the search process is performed from a lower hierarchy to an upper hierarchy. Control means for controlling the search reference point setting means and the corresponding point search means so as to be executed sequentially toward
With
The corresponding point search means searches for the corresponding point in the target reference image with sub-pixel resolution in which a sub-pixel having a size smaller than the pixel size of the target reference image is an image expression unit,
The search reference point setting means sets a search reference point in a reference image of a layer one level higher than the target layer based on the corresponding points searched with the sub-pixel resolution,
The corresponding point search means includes
Pixel search means for performing a pixel corresponding point search process for searching for a pixel corresponding point of the pixel size in the target reference image corresponding to the target point of the target reference image at a pixel resolution that is a pixel size of the target reference image;
Subpixel search means for performing a subpixel corresponding point search process for searching for a corresponding point of the target reference image at the subpixel resolution based on the pixel corresponding point;
The image processing apparatus includes:
A reliability calculation means for calculating a reliability indicating a search accuracy of the pixel corresponding point based on a search result of the pixel corresponding point;
Reliability determination means for determining whether or not the reliability is lower than a predetermined value;
Further comprising
When the reliability determination unit determines that the reliability is lower than a predetermined value, the subpixel search unit of the corresponding point search unit disables the subpixel corresponding point search process. An image processing apparatus. Image acquisition means for acquiring one input image of two or more input images as an original reference image and another input image as an original reference image;
By reducing the resolution of the original reference image and the original reference image so that the resolution decreases from the upper layer to the lower layer, the plurality of reference images and the plurality of reference images having different resolutions are hierarchically arranged. A multi-resolution image creating means for generating
Of the plurality of reference images and the plurality of reference images, the images belonging to the same attention layer are used as the attention reference image and the attention reference image, and the corresponding point of the attention reference image corresponding to the attention point of the attention reference image is searched. A search reference point setting means for setting a search reference point serving as a reference for the search processing to the attention reference image;
Corresponding point search means for searching for the corresponding points by performing the searching process based on the search reference points;
In each hierarchical structure of the plurality of reference images and the plurality of reference images, the search process is sequentially set from a lower hierarchy to an upper hierarchy, whereby the search process is performed from a lower hierarchy to an upper hierarchy. Control means for controlling the search reference point setting means and the corresponding point search means so as to be executed sequentially toward
With
The corresponding point search means searches for the corresponding point in the target reference image with sub-pixel resolution in which a sub-pixel having a size smaller than the pixel size of the target reference image is an image expression unit,
The search reference point setting means sets a search reference point in a reference image of a layer one level higher than the target layer based on the corresponding points searched with the sub-pixel resolution,
The corresponding point search means includes
Pixel search means for performing a pixel corresponding point search process for searching for a pixel corresponding point of the pixel size in the target reference image corresponding to the target point of the target reference image at a pixel resolution that is a pixel size of the target reference image;
Subpixel search means for performing a subpixel corresponding point search process for searching for a corresponding point of the target reference image at the subpixel resolution based on the pixel corresponding point;
The image processing apparatus includes:
Wherein based rather on the search result of the pixel corresponding point search process, first reliability indicating the search accuracy of the pixel corresponding point, and based on the search result of the sub-pixel corresponding point searching process, the sub of the target reference picture Reliability calculation means for calculating at least one of second reliability indicating the search accuracy of corresponding points searched with pixel resolution ;
Reliability determination means for determining at least one of whether the first reliability is lower than a first predetermined value and whether the second reliability is lower than a second predetermined value ;
Further comprising
The multi-resolution image creating means has the original standard so that the resolution is lower than that of the standard image and the reference image that are one level higher than the target hierarchy, and the resolution is higher than that of the target standard image and the target reference image. Generating an auxiliary reference image and an auxiliary reference image, each of which is an image and a reduced-resolution image of the original reference image,
When it is determined by the reliability determination means that the first reliability is lower than the first predetermined value, or the second reliability is lower than the second predetermined value,
(A) The search reference point setting means corresponds to the target point of the auxiliary reference image based on the pixel corresponding point of the target reference image or the corresponding point searched at the sub-pixel resolution of the target layer. A search reference point serving as a reference for processing for searching for a corresponding point of the auxiliary reference image is set in the auxiliary reference image,
(B) The control means converts the auxiliary standard image and the auxiliary reference image into one hierarchy higher than the target hierarchy and the target hierarchy in each hierarchical structure of the plurality of standard images and the plurality of reference images. The search process is sequentially executed from the intermediate hierarchy to the upper hierarchy by sequentially setting the attention hierarchy from the intermediate hierarchy to the upper hierarchy. And an image processing apparatus for controlling the search reference point setting means and the corresponding point search means .

Images