JP5316309B2 - Image processing apparatus and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve the processing performance of corresponding point search processing using a multi-resolution image. <P>SOLUTION: An apparatus includes an image acquiring means for acquiring an original standard image and an original reference image each including an image content corresponding to each other, a multi-resolution image generating means for hierarchically generating a plurality of images whose resolutions are different from each other by lowering resolutions of the acquired original standard image and the original reference image, a noticed point setting means for setting a noticed point to a noticed standard image belonging to a noticed hierarchy, a parallax specifying means for acquiring parallax between the noticed standard image and a noticed reference image about the noticed point, and a selection applying means for making (1) a first parallax specifying rule using image information within a window when the window including the noticed point is included within the noticed standard image, or (2) a second parallax specifying rule which is made lower in contribution degree from an outer area of the noticed standard image than the first parallax specifying rule when the window is protruded from the noticed standard image, and selectively applied when the parallax specifying means requires the parallax. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method for searching for a corresponding point of a target point set in a standard image from a reference image.

  Among a plurality of images related to a measurement object photographed from different viewpoints and a plurality of time-series images related to a moving measurement object photographed in time series from the same viewpoint, one image is a reference image and another image is Search for a corresponding point on the reference image corresponding to the target point on the standard image as a reference image using an area-based corresponding point search method using template matching, etc., and based on the obtained corresponding point and the target point There are known devices for obtaining the three-dimensional shape of a measurement object and devices for tracking the movement of the measurement object.

  In the area-based corresponding point search method, a search window (also referred to as a “reference window”) that has a point of interest approximately as a center is set on a reference image, and the corresponding point is expected to exist in an area on the reference image. Also, a search window (also referred to as a “reference window”) having the same size as the reference window is set.

  Next, a parallax specifying process for obtaining a shift amount (also referred to as “parallax”) between the images in the search window, in which the correlation between the images between the search windows is the highest, is performed, and the obtained parallax and the pixel coordinates of the target point are obtained. Then, a corresponding point search process is performed to obtain a corresponding point on the reference image corresponding to the point of interest on the standard image.

  In the area-based corresponding point search method, since the range of corresponding points in the reference image cannot normally be specified in advance, the corresponding point search is performed by successively setting reference windows so as to cover the entire area of the reference image. Processing is performed.

  In recent years, as one of the area-based corresponding point search methods, a base image and a reference image are sequentially reduced in resolution so that the resolution gradually decreases for each layer, thereby generating a plurality of layers of images. Each time, a corresponding point search method using a multi-resolution strategy in which corresponding points are searched in order from the lowest resolution layer to the highest resolution layer while using an area-based corresponding point search method has attracted attention.

  In the corresponding point search using the multi-resolution strategy, it is possible to determine an initial disparity that defines a reference window that is one layer above the target layer based on the disparity obtained in the target layer.

  Therefore, in the corresponding point search method using the multi-resolution strategy, the search can be performed while narrowing down the reference window setting area in each layer.

  Patent Document 1 discloses a technique of an image processing apparatus that uses corresponding point search based on a multi-resolution strategy.

Japanese Patent Laid-Open No. 2001-148012

  In the corresponding point search method based on the multi-resolution strategy, when the search window at the edge of the image protrudes from the image, the correlation between the search windows cannot be obtained with high accuracy. There is a problem that it gets worse and a problem that the processing time increases as the number of layers increases.

  In the image processing apparatus of Patent Document 1, when the reliability of the obtained corresponding point is low, the corresponding point with low accuracy is corrected by correcting the corresponding point with low reliability using information on surrounding corresponding points. Although a technology that enables search is disclosed, a measure for improving the search accuracy when the search region protrudes from the image at the edge of the image and a suppression of an increase in processing time accompanying an increase in the number of hierarchies No measures are disclosed.

  The present invention has been made to solve such a problem, and an object of the present invention is to provide a technique capable of improving the processing performance of corresponding point search processing using a multi-resolution image.

In order to solve the above problems, the invention of claim 1 is a images processing device, an image obtaining unit for obtaining the corresponding containing image content to the original reference image and the original reference image to each other, the upper layer A plurality of standard images and a plurality of reference images having different resolutions are generated in a hierarchical manner by reducing the resolution of the original standard image and the original reference image so that the resolution becomes lower toward the lower hierarchy from Multi-resolution image creating means, and images that belong to the same attention layer among the plurality of reference images and the plurality of reference images are set as attention reference images and attention reference images, and attention is set to the attention reference images. A point setting unit, a parallax specifying unit for determining the parallax between the target reference image and the target reference image for the target point, and the parallax specifying unit for determining the parallax 1) When the window containing the eye point fits in the target reference image, the first parallax specifying rule using the image information in the window is used. 2) When the window protrudes from the target reference image, the outside of the target reference image Selection application means for selectively applying a second parallax specifying rule whose contribution from the region is lower than that of the first parallax specifying rule, and the second parallax specifying rule is included in the reference image of interest. It is a rule that uses a parallax already specified for a point closer to the center than the point of interest.

The invention according to claim 2 is the image processing apparatus according to claim 1 , wherein the second parallax specifying rule is already set for one alternative point closer to the center than the point of interest in the point-of-interest reference image. It is a rule for obtaining the parallax for the attention point using the specified parallax.

The invention according to claim 3 is the image processing apparatus according to claim 1 , wherein the second parallax specifying rule is already set for a plurality of alternative points closer to the center than the attention point in the attention reference image. It is a rule for obtaining a parallax for the attention point by extrapolation of the specified parallax.

According to a fourth aspect of the present invention, there is provided image acquisition means for acquiring an original standard image and an original reference image including image contents corresponding to each other, and the original so that the resolution decreases from the upper hierarchy to the lower hierarchy. By reducing the resolution of the standard image and the original reference image, multi-resolution image creating means for hierarchically generating a plurality of standard images and a plurality of reference images having different resolutions, the plurality of standard images, Of the plurality of reference images, an image belonging to the same attention layer is set as the attention base image and the attention reference image, attention point setting means for setting the attention point in the attention base image, and the attention point is the attention reference image. Parallax specifying means for obtaining a parallax between the target reference image and the target reference image with respect to the target point when the target point is in a predetermined parallax specific region. Area ratio of the specific region and the reference image is characterized by an image processing apparatus which is set larger upper layer.

The invention according to claim 5 is the image processing device according to claim 4 , wherein the parallax specifying unit uses the image information in the window including the attention point, The parallax between the target standard image and the target reference image is obtained, and the parallax specifying region is a region in a range where the window fits in the target standard image, and an initial standard for obtaining the parallax in an upper layer The initial parallax used as is obtained by using the parallax obtained for the attention point in the parallax specifying region in the lower hierarchy.

Furthermore, the resolution as the invention of claim 6 is the images processing apparatus, an image acquiring unit that acquires a correspondence including image content to the original reference image and the original reference image to each other, toward the lower layer from the upper layer Multi-resolution image creation means for hierarchically generating a plurality of standard images and a plurality of reference images having different resolutions by lowering the resolution of the original standard image and the original reference image so that the Of the plurality of reference images and the plurality of reference images, an attention point setting means for setting an attention point in the attention reference image using an image belonging to the same attention layer as the attention reference image and the attention reference image, and the attention Parallax specifying means for obtaining a parallax between the target reference image and the target reference image for a point, and when the parallax specifying means obtains the parallax, 1) a window containing the target point When the dough fits in the target reference image, the first parallax specifying rule using the image information in the window is used. 2) When the window protrudes from the target reference image, the degree of contribution from the external region of the target reference image Selection applying means for selectively applying a second parallax specifying rule lower than the first parallax specifying rule, and a region to which the first rule is applied in the reference image of interest and the first An area composed of areas to which two rules are applied is set as a parallax specific area, and an area ratio between the parallax specific area and the reference image in each hierarchy is set to be higher in the upper hierarchy and the first parallax is set in the upper hierarchy. The initial parallax used in the specific rule is obtained using the parallax obtained at each point of interest for the lower layer.

Further, the invention of claim 7 is a method of acquiring an original standard image and original reference image including image contents corresponding to each other, and the original standard image so that the resolution decreases from the upper hierarchy to the lower hierarchy. And generating a plurality of standard images and reference images having different resolutions in a hierarchical manner by reducing the resolution of the original reference image, and the plurality of standard images and the plurality of reference images. Of these, an image belonging to the same attention layer as an attention standard image and an attention reference image, a step of setting an attention point on the attention reference image, and when the attention point is in a predetermined parallax specific region of the attention reference image Obtaining a parallax between the target standard image and the target reference image for the target point, and an area ratio between the parallax specifying region and the standard image in each layer is higher than the upper layer. Characterized in that an image processing method that is set to be larger.

According to the first to third aspects of the present invention, the second parallax specifying rule applied when the window containing the attention point protrudes from the reference image is applied when the window does not protrude from the reference image. Since the contribution degree of the extra-image information to the parallax specifying process is smaller than the one parallax specifying rule, the parallax specifying accuracy when the window protrudes from the reference image can be improved, and the corresponding point search process using the multi-resolution image can be performed. Processing performance can be improved.

According to the first to third aspects of the invention, the second parallax specifying rule obtains the parallax of the attention point by using the parallax already specified for the point from the center of the reference image rather than the attention point. Therefore, it is possible to improve the parallax specifying accuracy when the window protrudes from the reference image, and it is possible to improve the processing performance of the corresponding point search processing using the multi-resolution image.

According to the invention of claim 2 , since the second parallax specifying rule obtains the parallax of the attention point using the parallax already specified for one alternative point from the center of the reference image rather than the attention point, the second parallax is determined. It is possible to speed up the parallax specifying process using the specifying rule, and to improve the processing performance of the corresponding point searching process using the multi-resolution image.

According to the invention of claim 3 , since the second parallax specifying rule obtains the parallax of the attention point by extrapolation of the parallax already specified for a plurality of alternative points from the center of the reference image rather than the attention point. The continuity can improve the parallax identification accuracy when the window protrudes from the reference image, and can improve the processing performance of the corresponding point search processing using the multi-resolution image.

According to the invention described in the present application, in the second parallax specifying rule applied when the window including the attention point protrudes from the reference image, the protruding direction of the window is shorter than the window used in the first parallax specifying rule. Since the image information in the width window is used, the parallax identification accuracy according to the second parallax identification rule can be improved, and the processing performance of the corresponding point search process using the multi-resolution image can be improved.

According to the invention described in the present application , the second disparity specifying rule has a lower contribution than the first disparity specifying rule in the portion of the window containing the attention point that protrudes from the reference image. Can be improved as compared with the case where the first parallax specification rule is used, so that the parallax specification accuracy based on the second parallax specification rule can be improved, and the processing performance of the corresponding point search process using the multi-resolution image can be improved. Is possible.

According to the fourth to seventh aspects of the present invention, in the process in which the parallax specifying process is sequentially executed from the lower hierarchy to the upper hierarchy, the ratio between the parallax specifying area and the reference image is slightly increased as the hierarchy increases. Since the parallax specifying process can be performed while increasing the size gradually, the parallax specifying process can be accelerated by reducing the parallax specifying area, and the processing performance of the corresponding point search process using the multi-resolution image is improved. Is possible.

According to the invention of claim 5, the parallax of the target layer is based on the parallax obtained for each target point of the parallax specific region in which the window including the target point does not protrude from the image region of the reference image of the lower layer. By setting each initial parallax for each target point of the specific area, each initial parallax for the parallax specific area of the target layer can be set to an appropriate value, so that the parallax specifying process can be speeded up, It becomes possible to improve the processing performance of the corresponding point search processing using the multi-resolution image.

According to the invention of claim 6 , among the parallax specifying areas of the target hierarchy, the initial parallax for each target point of the area where the window containing the target point does not protrude from the image area of the target reference image is the lower level parallax. By setting based on each parallax obtained for each target point in the specific area, each initial parallax for the area where the window in the target layer does not protrude from the reference image can be set to an appropriate value. In addition, it is possible to speed up the parallax identification process, and it is possible to improve the processing performance of the corresponding point search process using the multi-resolution image.

It is a figure which shows schematic structure of the three-dimensional shape measuring apparatus using the image processing apparatus which concerns on embodiment. It is a figure which shows the correspondence of the attention point of a reference | standard image, and the corresponding point of a reference image. It is a functional block diagram showing an example of composition of a principal part of an image processing device concerning an embodiment. It is a figure which illustrates a multi-resolution reference image. It is a figure which illustrates a multi-resolution reference image. It is a figure which illustrates a multi-resolution reference image. It is a figure which illustrates a multi-resolution reference image. It is a figure which illustrates a multi-resolution reference image. It is a figure which illustrates a multi-resolution reference image. It is a figure explaining the example of the influence which the resolution scaling factor has on parallax specification processing. It is a figure which illustrates the multi-resolution image of a reference image. It is a figure explaining the 1st field and the 2nd field concerning an embodiment. It is a figure explaining the 1st field and the 2nd field concerning an embodiment. It is a figure which illustrates the 1st field and the 2nd field of a higher hierarchy and a lower hierarchy. It is a figure which illustrates the 1st field and the 2nd field of a higher hierarchy and a lower hierarchy. It is a figure explaining the setting of initial parallax. It is a figure explaining the setting of initial parallax. It is a figure explaining the outline | summary of a parallax specific process. It is a figure which shows the example of a setting of the reference window with respect to the search reference point of a reference image. It is a figure which shows the example of a setting of the reference window with respect to the search reference point of a reference image. It is explanatory drawing of the parallax specific process using a SAD method. It is a figure which illustrates the scanning aspect of the reference window in SAD method. It is a figure which shows the procedure of the parallax specific process using POC method. It is a figure which shows the example of a POC value. It is a figure which illustrates the window of the 1st field and the 2nd field. It is a figure which illustrates the window function of the 1st parallax specific processing concerning an embodiment. It is a figure which illustrates the window function of the 2nd parallax specific processing concerning an embodiment. It is a figure which illustrates the window function of the 2nd parallax specific processing concerning an embodiment. It is a figure which illustrates the window function of the 2nd parallax specific processing concerning an embodiment. It is a figure which shows the example of a setting of the window in a 2nd area | region. It is a figure which shows the example of the identification method of the parallax of a 2nd area | region. It is a figure which shows the example of the identification method of the parallax of a 2nd area | region. It is a figure which shows the example of the identification method of the parallax of a 2nd area | region. It is a figure which shows the example of a setting of the parallax specific area | region which concerns on a modification. It is a figure explaining the example of the setting method of the initial parallax which concerns on a modification. It is a figure which shows the example of a setting of the parallax specific area | region which concerns on a modification. 3 is a flowchart illustrating an outline of an operation of the image processing apparatus according to the embodiment. 3 is a flowchart illustrating an outline of an operation of the image processing apparatus according to the embodiment. 10 is a flowchart illustrating an outline of an operation of an image processing apparatus according to a modification. 10 is a flowchart illustrating an outline of an operation of an image processing apparatus according to a modification.

  The image processing apparatus according to the present invention measures, for example, an object to be measured by two or more cameras, processes the obtained image, and obtains a line-of-sight equation for each camera passing through a point of interest on the object to be measured. A three-dimensional shape measuring device that measures the shape of the measurement object or a single camera measures the measurement object in time series, processes the obtained image, determines the amount of movement of the measurement object, The present invention is applied to an apparatus that compares a registered measurement object image with a measured image.

  Hereinafter, the image processing apparatus according to the present invention measures a measurement object using two cameras (also referred to as stereo cameras), and processes the obtained stereo image to measure the three-dimensional shape of the measurement object. An embodiment applied to a three-dimensional measuring apparatus will be described with reference to the drawings. For the sake of explanation, the coordinate axes are appropriately described in the drawings.

<Three-dimensional shape measuring device>
FIG. 1 is a diagram illustrating a schematic configuration of a three-dimensional shape measuring apparatus 1A using an image processing apparatus 3A according to the embodiment.

  As shown in FIG. 1, the three-dimensional shape measuring apparatus 1A includes a two-lens stereo camera 2 and an image processing apparatus 3A connected to the stereo camera 2 so as to be able to transmit data.

  The two-lens stereo camera 2 is provided with two imaging systems 21 and 22 each having an imaging device. The imaging systems 21 and 22 have substantially the same optical characteristics and electrical characteristics, are spaced apart along a predetermined direction, and the measurement object OB in front of the camera is synchronized from the same viewpoint at different timings. It is configured to take an image. The two images captured at the same timing by the imaging systems 21 and 22 are so-called stereo images, and are transmitted to the image processing device 3A via the data line DL.

  Here, of the two images constituting the stereo image, an image captured and acquired by the imaging system 21 is appropriately referred to as a “reference image”, and an image captured and acquired by the imaging system 22 is appropriately referred to as “reference”. This is called “image”.

  In addition, camera parameters such as the focal length, principal point position, distortion, and baseline length of the imaging systems 21 and 22 are calibrated in advance, and the measurement target imaged on the imaging elements of the imaging systems 21 and 22 by the camera parameters. A camera line-of-sight equation passing through a point on the image of the object OB and the principal point of the imaging system is determined.

  FIG. 2 is a diagram illustrating the correspondence between the attention points of the standard image and the corresponding points of the reference image.

  First, the image processing device 3A hierarchically repeats the resolution reduction of the standard image BIT and the reference image RIT related to the measurement object OB, so that the standard image BIT and the reference image RIT have a plurality of resolutions different from each other. Corresponding by creating a multi-resolution image composed of images and specifying the shift amount (parallax) of the corresponding point CT on the reference image RIT corresponding to the target point NT on the standard image BIT using the multi-resolution image A process of searching for the point CT is performed.

  Next, the image processing apparatus 3A creates a camera line-of-sight equation regarding the target point NT and the corresponding point CT using the coordinate information of the searched target point NT and the corresponding point CT and the camera parameters of the imaging systems 21 and 22. Then, a three-dimensional process for obtaining the three-dimensional coordinates of the point of the measurement object OB corresponding to the target point CT is performed by solving these equations simultaneously.

  The image processing apparatus 3A calculates the three-dimensional shape of the measurement object OB by searching for corresponding points for a plurality of attention points and performing a three-dimensional process.

  Here, although the stereo camera is provided in the three-dimensional shape measuring apparatus 1A, for example, when three or more cameras are provided, one of the input images input to the three-dimensional shape measuring apparatus is provided. The input image is acquired as a standard image and the other input images are acquired as reference images, and the same processing as when using a stereo camera is performed to obtain the three-dimensional shape of the measurement object OB.

<Functional Description of Image Processing Device 3A>
FIG. 3 is a functional block diagram illustrating a configuration example of a main part of the image processing apparatus 3A according to the embodiment.

  As shown in FIG. 3, the image processing apparatus 3A includes an operation unit 4, a display unit 5, a communication unit 6, an input / output unit 7, a storage unit 9, an image acquisition unit 10, a multi-resolution image creation unit 11, an attention point setting. Unit 12, initial parallax setting unit 14, parallax specifying unit 15, and control unit 16, and these function blocks are used to create multi-resolution images related to the standard image BIT and the reference image RIT, search for corresponding points, and Perform 3D processing.

  Hereinafter, the operation unit 4 to the control unit 16 of the image processing apparatus 3A will be described.

◎ Operation unit 4 to storage unit 9:
The operation unit 4 includes, for example, a keyboard, a mouse, an operation button, and the like, and inputs operation signals to the control unit 16 and sets control parameters such as resolution scaling factor RR for each unit of the image processing apparatus 3A. The control parameters used and set for such purposes as are stored in the storage unit 9 by the control unit 16.

  The display unit 5 is configured by a liquid crystal display, for example, and can display a moving image or the like. Image information such as a captured stereo image or a generated three-dimensional image, various messages, and the like are displayed on the display unit 5. Control parameter setting information is also displayed on the display unit 5.

  The communication unit 6 is an interface for connecting to a LAN or the Internet, and includes, for example, a network adapter.

  The input / output unit 7 includes, for example, a multimedia drive, receives a storage medium 8 such as an optical disk, and exchanges data with the control unit 16. The stereo image processed by the image processing apparatus 3A can be input not only directly from the stereo camera 2, but also via the storage medium 8 or the network by the communication unit 6 or the input / output unit 7.

  The storage unit 9 includes, for example, a hard disk, a ROM, a RAM, and the like. Information output from each unit of the image processing apparatus 3A, control parameters set to control each unit of the image processing apparatus 3A, and calibration are performed. It is used for permanent storage of camera parameters and control programs, and for temporary storage of various information.

  Here, the control parameters include, for example, the number of layers T of the multi-resolution image, the resolution scaling factor RR, the window sizes WX and WY that are the horizontal and vertical sizes of the standard window and the reference window, and the attention points set in the standard image. Are the region information V1 for determining whether or not belongs to a predetermined parallax specifying region, a threshold for determining the reliability of the parallax specifying processing, and the like.

  The area information V1 includes information for specifying a parallax specifying area in the reference image specified by the area information V1 and selection information for parallax specifying processing to be applied to the specified parallax specifying area. Yes.

  These control parameters are set in advance by the operation unit 4 and stored in the storage unit 9.

  Note that the image acquisition unit 10, the multi-resolution image creation unit 11, the attention point setting unit 12, the region determination unit 13, the initial parallax setting unit 14, the parallax specifying unit 15 and the control unit 16 described below are a predetermined program executed by the CPU. It may be realized by executing or may be realized by using a dedicated hardware circuit.

Image acquisition unit 10:
The image acquisition unit 10 acquires an image captured by the standard camera 21 and an image captured by the reference camera 22 and stores them in the storage unit 9 as a standard image BIT and a reference image RIT, respectively. Here, the standard image BIT and the reference image RIT are, for example, digital image data in which a plurality of pixels are arranged in a matrix and pixel values of each pixel are expressed with a predetermined number of gradations.

  For example, VGA (640 pixels × 480 pixels) is used as the size of the standard image BIT and the reference image RIT.

Multi-resolution image creation unit 11:
The multi-resolution image creation unit 11 performs multi-stage resolution reduction processing on the standard image BIT and the reference image RIT, thereby performing a multi-resolution standard image MB that is a multi-resolution image of the standard image BIT and a multi-resolution image of the reference image RIT. Are created and stored in the storage unit 9.

  4 to 6 are diagrams illustrating the multi-resolution standard image MB, and FIGS. 7 to 9 are diagrams illustrating the multi-resolution reference image MR. Regarding the multi-resolution standard image MB and the multi-resolution reference image MR, FIGS. 4 and 7 show the respective highest-order standard images BIT and reference images RIT, and FIGS. 5 and 8 show the multi-resolution standard image MB and multi-resolution, respectively. FIG. 6 and FIG. 9 show the lowest reference image BI1 and reference image RI1, respectively, of the entire reference image MR.

  As shown in FIGS. 4 to 9, the multi-resolution standard image MB and the multi-resolution reference image MR are an aggregate of a plurality of standard images having different resolutions and a plurality of reference images having different resolutions, respectively. , It is hierarchically formed so that the resolution decreases as it goes from the upper layer to the lower layer.

  In the multi-resolution standard image MB and the multi-resolution reference image MR, the highest standard image BIT and the reference image RIT image are not reduced in resolution, and the lowest standard image BI1 and the reference image RI1 image have the lowest resolution. It is an image. The multi-resolution standard image MB and the multi-resolution reference image MR have the same number of layers T, and the standard image and the reference image in the same layer have the same resolution.

  Note that a reference image BIT that is a reference image taken by the reference camera 21 and that is also the highest-order image of the multi-resolution reference image MB and is not reduced in resolution is also referred to as an “original reference image” BIT, and is taken by the reference camera 22. A reference image RIT that is a reference image and is also the highest-order image of the multi-resolution reference image MR and has not been reduced in resolution is also referred to as an “original reference image” RIT.

  In addition, an image with a higher resolution is referred to as an “upper” (or “upper hierarchy”) image than an image of a multi-resolution image of interest (also referred to as “target hierarchy”), and an image with lower resolution. Are referred to as “lower” (or “lower hierarchy”) images. Further, the ratio of the resolution reduction when the image of the target layer is reduced in resolution to create one lower layer image is also referred to as “resolution scaling factor”.

  Here, the base image BIm (FIG. 5) and the reference image RIm (FIG. 8) are the base image and the reference image when the attention layer is set in the one upper layer of the lowest layer. Note that the reference image and the reference image of the attention layer are also referred to as “attention reference image” and “attention reference image”, respectively.

  The resolution scaling factor RR is given as a ratio of the number of pixels in the horizontal direction of the image in the lower layer to the number of pixels in the horizontal direction of the image in the upper layer among the two adjacent layers in the multi-resolution image. It is also given as the ratio of the number of pixels in the vertical direction of the lower layer image to the number of pixels in the vertical direction of the upper layer image. Usually, these two types of ratios are set to the same value.

  The multi-resolution image is repeatedly reduced in resolution by the resolution scaling factor RR until the number of layers of the multi-resolution image reaches the number of layers T, for example, according to the resolution scaling factor RR and the number of layers T preset and stored in the storage unit 9. Alternatively, it is created by repeatedly reducing the resolution with the resolution scaling factor RR until the number of pixels of the lowermost layer image becomes a predetermined value or less.

  Further, when the multi-resolution image creating unit 11 generates the multi-resolution standard image MB and the multi-resolution reference image MR for each layer, the initial parallax obtained by the initial parallax setting unit 14 for the attention point of the standard image of each layer. And as a storage area for storing the parallax specified by the parallax specifying unit 15, the number of pixels is the same as the number of pixels of the reference image of each layer, and the capacity is large enough to store the specified initial parallax and parallax. The stored image is secured in the storage unit 9.

○ About low resolution processing:
In the resolution reduction process, the multi-resolution image creating unit 11 first, for example, multi-resolution image such as the number T of layers of the multi-resolution image, the resolution scaling factor RR, the base image BI1 in the lowest layer, and the number of pixels of the reference image RI1. Control parameters related to creation, the standard image BIT, and the reference image RIT are acquired from the storage unit 9.

  Next, the multi-resolution image creating unit 11 multiplies the number of pixels in the horizontal and vertical directions of the standard image BIT and the reference image RIT by the resolution scaling factor RR, thereby subtracting the standard image BI (T−1) of one lower layer, The number of pixels in the horizontal direction and the vertical direction of the reference image RI (T-1) is obtained, and storage areas for the base image BI (T-1) and the reference image RI (T-1) are secured in the storage unit 9.

  When the storage area is secured, the multi-resolution image creation unit 11 averages the pixel values of one pixel of the reference image BI (T-1) and the pixel values of each pixel of the reference image BIT corresponding to this pixel. The pixel value is set and stored in the storage area corresponding to the pixel, and the pixel value setting process is performed for all of the reference image BI (T-1) and the reference image RI (T-1). By performing the processing for the pixels, images of the base image BI (T-1) and the reference image RI (T-1) are created.

  As described above, the correspondence relationship of pixels across the layers of the multi-resolution image is determined by the control parameters related to the creation of the multi-resolution image.

  Note that the correspondence relationship of pixels across hierarchies is stored in, for example, the storage unit 9 and used by each component of the image processing apparatus 3A as necessary.

  Hereinafter, the above-described resolution reduction processing is repeated until the number of images reaches the number of layers T, or until the number of pixels of the base image BI1 and the reference image RI1 in the lowermost layer is equal to or less than a predetermined value, from the upper layer. The reference image BIT and the reference image RIT acquired by the image acquisition unit 10 are hierarchically reduced in resolution so that the resolution becomes lower toward the lower layer, and a plurality of hierarchical reference images and references having different resolutions are used. An image, that is, a multi-resolution standard image MB and a multi-resolution reference image MR are created, and the multi-resolution standard image MB and the multi-resolution reference image MR are stored in the storage unit 9.

  In addition to the resolution reduction process described above, various techniques such as the Qubic-Convolution method can be employed as the resolution reduction process.

○ Resolution scaling factor RR:
FIG. 10 is a diagram for explaining an example of the influence of the resolution scaling factor RR on the parallax specifying process.

  A reference image RIm shown in FIG. 10 is a reference image of the target layer, and a reference image RI (m + 1) is a reference image one layer higher than the target layer. Here, each region divided by the grid shown in FIG. 10 indicates individual pixels (also referred to as “pixels”) of the reference image RIm and the reference image RI (m + 1).

  Further, the resolution scaling factor RR when generating the reference image RIm from the reference image RI (m + 1) is 1/4.

  The corresponding point CPmT on the reference image RIm indicates the corresponding point correctly identified with respect to the target point set on the base image of the target layer (“correct corresponding point”), and the corresponding point CPmF is the erroneously specified correspondence. Indicates a point ("wrong corresponding point").

  The search reference point K (m + 1) T on the reference image RI (m + 1) indicates the correct search reference point specified based on the correct corresponding point CPmT on the reference image RIm, and the search reference point K (m + 1) F is Indicates an incorrect search reference point specified based on an incorrect corresponding point CPmF on the reference image RIm.

  A reference point for setting a reference window is specified on the reference image of each target layer based on the target point set on the reference image of each target layer and the initial parallax for the target point. . In the present invention, this reference point is also referred to as a “search reference point”.

  As shown in FIG. 10, the difference between the correct corresponding point CPmT and the incorrect corresponding point CPmF is only one pixel in the diagonal direction of the pixel, but the search reference points K (m + 1) T and K (m + 1) F are The shift in the diagonal direction of these pixels is expanded to four pixels that are the reciprocal times of the resolution scaling factor. That is, the corresponding point search error in the lower hierarchy is further expanded with respect to the setting position of the reference window set in the upper hierarchy.

  Since the parallax for obtaining the corresponding point on the reference image RI (m + 1) is specified using the reference window set in the reference image RI (m + 1), the reference window set so as to deviate from an appropriate position is used. Thus, even if the corresponding point on the reference image RI (m + 1) is searched, the error in the corresponding point search in the reference image RIm in the lower layer cannot be corrected, and an erroneous search is performed in the reference image RI (m + 1). The potential increases.

  That is, when the resolution scaling factor RR is small, a small error in the corresponding point search in the lower layer image (low resolution image) leads to a large error in the corresponding point search in the upper layer image (high resolution image). Therefore, the accuracy of the corresponding point search is likely to be reduced.

  The resolution scaling factor RR is set by, for example, 2/3, 1/2, 1/3 or 1/4 (the resolution scaling factor RR is a real number larger than 1 (typically, displayed on the display unit 5). When expressed as 1 / r using a rational number r, a method of selecting r using an operation unit 4 from candidates in an appropriate range such as 3/2, 2, 3 or 4) is adopted. Alternatively, a method of setting a resolution scaling factor smaller than 1 other than the candidate value by an input from the operation unit 4 or the like may be employed. The set resolution scaling factor RR is stored in the storage unit 9.

  When the multi-resolution image creation unit 11 is a dedicated hardware circuit, the resolution scaling factor RR is set by, for example, a plurality of multi-resolution image creation units 11 respectively corresponding to a plurality of different resolution scaling factors RR. It may be set by a method of selecting the multi-resolution image creation unit 11 corresponding to a desired resolution scaling factor RR by switching a dip switch or the like.

  In a multi-resolution image, the lower the hierarchy image, the more “information reduction” occurs in each pixel in the image. In addition, when the number of pixels in the search window is constant regardless of the hierarchy of the multi-resolution image, the lower the hierarchy image, the more the measurement target imaged in the search window is captured in the search window. Since the OB and the background image are included, so-called “far / near conflict” occurs, and a corresponding point search error is likely to occur.

  Similarly, the smaller the resolution scaling factor RR when the lower layer image is created by lowering the resolution of the upper layer image, the more “information amount reduction” occurs in the lower layer image and ”Occurs, and a corresponding point search error is likely to occur.

  Accordingly, in order to suppress the above-described adverse effect of the corresponding point search error in the lower layer amplifying the corresponding point search error in the upper layer, as shown in the example of the multi-resolution image of the reference image shown in FIG. A plurality of resolution scaling factors RR may be used to create a small resolution scaling factor for an upper-layer image and a large resolution scaling factor for a lower-layer image. In the example of FIG. 11, the above-described adverse effect is suppressed by setting the resolution scaling factor on the upper layer side to ¼ and the resolution scaling factor on the lower layer side to ½.

◎ Attention point setting unit 12:
The attention point setting unit 12 sets the attention point on the attention reference image. For example, all the pixels on the reference image are set as the respective attention points. For each set target point, the corresponding point on the target reference image corresponding to the target point is obtained by the parallax specifying unit 15.

  The attention point is, for example, a method of manually setting feature points on the reference image displayed on the display unit 5 using the mouse of the operation unit 4 or the attention point setting unit 12 performs edge detection processing on the reference image. Alternatively, it may be selectively set by a method of automatically setting the extracted edge position.

  The attention point setting information is stored in the storage unit 9 as, for example, coordinate information of the attention point, and is appropriately used by the components of the image processing apparatus 3A.

Area determination unit 13:
The region determination unit 13 determines whether each target point set by the target point setting unit 12 belongs to a predetermined region on the target reference image determined based on the region information V1 stored in the storage unit 9. The obtained determination information is supplied to the initial parallax setting unit 14 and the parallax specifying unit 15 directly or indirectly via the control unit 16.

  In the present embodiment, the area information V1 is information that specifies two parallax specifying areas.

  Specifically, the region information V1 includes a parallax specifying region (referred to as a “first region”) configured by a point of interest in which a reference window set based on the region of interest does not protrude from the reference image, and a point of interest. This is information for specifying each of the parallax specifying areas (referred to as “second areas”) in which the reference window set based on the reference image protrudes from the reference image.

  12 and 13 are diagrams illustrating the first region and the second region specified in the attention reference image BIm based on the region information V1 according to the embodiment.

  In FIG. 12, three black circles are illustrated as attention points, and reference windows W1 to W3 having a predetermined size are set based on the attention points.

  Both the reference windows W1 and W2 do not protrude from the attention reference image BIm, but the reference window W3 protrudes from the attention reference image BIm.

  The first area Dm shown in FIG. 13 is an area defined by each attention point when the reference window set for the attention point does not protrude from the attention reference image BIm.

  The second area Em is an area defined by each attention point when the reference window set for the attention point protrudes from the attention reference image BIm.

  The boundary line Gm indicates the boundary line between the first region Dm and the second region Em.

  The area information V1 of the present embodiment is information that identifies the first area and the second area for each layer.

  FIG. 14 and FIG. 15 are diagrams illustrating the first region and the second region in each of the reference image BIm that is the upper layer and the reference image BI (m−1) that is one lower layer. .

  The reference image BIm shown in FIG. 14 exemplifies a 20 × 20 pixel reference image, and the reference image BI (m−1) shown in FIG. 15 has a resolution scaling factor RR from the reference image BIm. A reference image of 10 pixels × 10 pixels with a resolution reduced to 1/2 is illustrated.

  Here, the scale of the reference image BI (m−1) is adjusted and displayed so that the reference image BIm and the reference image BI (m−1) are displayed with the same image size.

  The reference window WBm shown in FIG. 14 is an example of a reference window of 5 pixels × 5 pixels set for the attention point Pa, and the reference window WBm shown in FIG. 15 is set for the attention point Pb. It is an example of a reference window of 5 pixels × 5 pixels.

  FIG. 14 shows a boundary line Gm between the first region and the second region in the reference image BIm of interest, and FIG. 15 shows the first region and the second region in the reference image BI (m−1). A boundary line Gm with the region is shown, and a boundary line Gm in the target reference image BIm is shown with an imaginary line.

  Here, in FIG. 15, the boundary line G (m−1) exists inside the boundary line Gm, and the area ratio of the first region with respect to the reference image between the reference image BIm and the reference image BI (m + 1). The reference image BIm is larger.

  In this way, when the images of each layer are displayed at the same scale, the apparent pixel size increases in the lower layer, whereas when the window size is the same between layers, the reference window extends beyond the reference image. For example, in FIG. 14 and FIG. 15, the number of pixels in each coordinate direction of the two regions is a fixed number such that there are two pixels in both the X-axis and Y-axis directions. The higher the hierarchy, the smaller the lower hierarchy.

Initial parallax setting unit 14:
The initial parallax setting unit 14 sets an initial parallax that is a reference for the parallax specifying process for specifying the parallax between the target point of the target reference image and the corresponding point of the target reference image corresponding to the target point, performed by the parallax specifying unit 15. Then, an initial parallax setting process for setting the attention point is performed. Further, the set initial parallax is stored in an area corresponding to the attention point in the parallax storage image.

  When the parallax is specified by the parallax specifying unit 15 for the attention point, the initial parallax stored in the parallax storage image is rewritten to the specified parallax, for example.

  FIGS. 16 and 17 illustrate the initial parallax setting process performed by the initial parallax setting unit 14 by taking, as an example, some pixels in the reference image BIm of interest and the reference image BI (m−1) of the one lower hierarchy. It is a figure to do.

  In FIG. 16, some pixels of the target reference image BIm and a part of the reference image BI (m−1) of one lower hierarchy in which the resolution of the target reference image BIm is reduced by halving the resolution scaling factor RR. The correspondence relationship with the other pixels is shown.

  In FIG. 16, since the resolution scaling factor RR is 1/2, the attention points of the two pixels in the hatched portion of the reference image BI (m−1) correspond to the attention points of the eight pixels in the hatched portion of the reference image BIm. ing.

  FIG. 17 shows two pixel attention points p1 and p2 in the hatched portion of the reference image BI (m−1), and eight pixel attention points p3 in the hatched portion of the attention reference image BIm corresponding to the attention points p1 and p2, respectively. The parallax storage images corresponding to p6 and p7 to P10 are shown.

  The parallaxes specified by the parallax specifying unit 15 for the attention points p1 and p2 are Q1 and Q2.

  The initial parallaxes set for the attention points p3 to p6 corresponding to the attention point p1 are q1 to q4, respectively, and the initial parallaxes q1 to q4 are obtained by multiplying the parallax Q1 by the inverse of the resolution scaling factor RR.

  Similarly, the initial parallaxes set for the attention points p7 to p10 corresponding to the attention point p2 are q5 to q8, respectively, and the initial parallax q5 to q8 is obtained by multiplying the parallax Q2 by the inverse of the resolution scaling factor RR. Desired.

  As described above, the initial parallax setting process for the target point of the target reference image includes the parallax specified for the target point of the reference image BI (m−1) in the lower hierarchy corresponding to the target point of the target reference image BIm. Is set based on

  In addition, when the initial parallax setting unit 14 can determine that the target point of the target reference image belongs to the region to which the parallax specification rule that does not use the initial parallax is applied, for example, by using the result of the region determination, The initial parallax may not be set for this attention point.

  It should be noted that due to the adopted method of setting the parallax identification area, the parallax for the attention point of the reference image in the lower hierarchy corresponding to the attention point of the attention reference image is not specified, or the creation of the multi-resolution image If there is no attention point of the reference image in the lower hierarchy corresponding to the attention point of the attention reference image due to the method, the initial parallax for these attention points of the attention reference image is the initial of the attention reference image. It is set by extrapolation calculation or the like based on the initial parallax for other attention points for which parallax is set.

  The setting of the initial parallax by extrapolation calculation will be described later in the description of the modification.

◎ Parallax specifying unit 15:
The disparity specifying unit 15 includes a target point set on the base image of the target layer in the multi-resolution standard image MB, and a corresponding point on the reference image of the target layer in the multi-resolution reference image MR corresponding to the target point. The corresponding point is searched for by performing a parallax specifying process for specifying the parallax.

  Further, the specified parallax is stored in a storage area corresponding to the attention point of the parallax stored image.

  In the present embodiment, the parallax specifying unit 15 determines that when the attention point of the reference image BIm belongs to the first region Dm shown in FIG. 12, that is, when the reference window that includes the attention point is within the attention reference image. The parallax for the target point is specified by the first parallax specifying process using the image information in the reference window and the image information of the reference window set as the target reference image based on the initial parallax for the target point.

  When the attention point belongs to the second region Em shown in FIG. 12, that is, when the reference window that includes the attention point protrudes from the attention reference image, the parallax specifying unit 15 The parallax for the target point is specified by the second parallax specifying process in which the degree of contribution to the parallax specifying process is lower than when the target point belongs to the first region.

  The second parallax specifying process in which the window including the attention point is applied to the attention point of the second region Em that protrudes from the reference image is applied to the attention point of the first region in which the reference window does not protrude from the reference image BIm. Since the contribution degree of the extra-image information to the parallax specifying process is smaller than that of the first parallax specifying process, it is possible to improve the parallax specifying accuracy when the window protrudes from the reference image, and the corresponding points using the multi-resolution image The processing performance of search processing can be improved.

  The first parallax specifying process and the second parallax specifying process are, for example, selectively switching the processing program for the first parallax specifying process and the second parallax specifying process according to the region to which the attention point belongs. Is done by.

  Further, the parallax specifying unit 15 includes processing devices for the first parallax specifying processing and the second parallax specifying processing, and selectively switches these processing devices according to the region to which the attention point belongs. You may go by.

  Regarding the switching of the processing program and the processing device described above, for example, the control unit 16 that has acquired the determination result of the region determination unit 13 uses the disparity specifying unit 15 or the like based on the selection information of the disparity specifying process included in the region information V1. This is done by controlling.

  Further, even when the parallax specifying unit 15 itself acquires the selection information of the parallax specifying process included in the region information V1, and switches the parallax specifying processing method based on the determination result supplied from the region determining unit 13. good.

  Note that the sizes of the standard window and the reference window used in the parallax specifying process are determined by the window sizes WX and WY that are control parameters acquired from the storage unit 9.

Control unit 16:
The control unit 16 implements information processing such as parallax identification processing and three-dimensional data creation by comprehensively controlling each unit of the image processing apparatus 3A.

  In the parallax specifying process, the control unit 16 moves the target hierarchy in the multi-resolution standard image MB and the multi-resolution reference image MR that are targets of the parallax specifying process from the lower hierarchy to the higher hierarchy. The parallax specifying process is executed repeatedly in a hierarchical manner from the lower layer to the upper layer while sequentially setting the respective units of the image processing apparatus 3A such as the initial parallax setting unit 14 and the parallax specifying unit 15. Each part of the image processing apparatus 3A is controlled.

  In addition, the control unit 16 acquires the area information V1 stored in the storage unit 9 and supplies the area information V1 to the area determination unit 13, and acquires the parallax specifying process identification information included in the area information V1. Based on the obtained determination information about the parallax specific area to which the target point belongs, information on the parallax specific process to be applied to each parallax specific area is acquired.

  Based on the relationship between the acquired parallax specifying area and the parallax specifying process applied to the parallax specifying area, the control unit 16 causes the parallax specifying unit 15 to perform the parallax specifying process according to the parallax specifying area to which the target point belongs. The parallax specifying unit 15 and the like are controlled to perform.

  In the parallax identification processing, as described above, control for selectively switching between the first parallax identification processing and the second parallax identification processing according to the affiliation region of the target point is also performed.

  With the configuration described above, the image processing apparatus 3A searches for corresponding points by performing a parallax specifying process based on a multi-resolution strategy.

<Description of Operation of Image Processing Device 3A>
Next, the parallax specifying process of the image processing apparatus 3A will be described in more detail, and then the overall processing flow of the corresponding point search process using the multi-resolution image performed by the image processing apparatus 3A will be described.

◎ First parallax specifying process:
FIG. 18 is a diagram for describing an overview of the first parallax specifying process performed by the parallax specifying unit 15 in the first region where the reference window set as the attention point does not protrude from the reference image. In FIG. 18, the attention point setting unit 12 sets the attention point Nm on the reference image BIm of the attention layer in the multi-resolution reference image MB.

  In FIG. 18, the search reference point Km is specified on the reference image RIm of the target layer in the multi-resolution reference image MR by the attention point Nm and the initial parallax set by the initial parallax setting unit 14 as the attention point Nm. Has been.

  The parallax specifying unit 15 sets the reference window WBm for the attention point Nm and the reference window WRm for the search reference point Km based on the window sizes WX and WY stored in the storage unit 9, respectively. By obtaining the correlation between the image in WBm and the image in the reference window WRm, the corresponding point CPm corresponding to the attention point Nm is obtained.

○ Reference window and reference window:
19 and 20 are diagrams illustrating an example of setting the standard window WBm set in the reference image WBm.

  In the reference window WBm illustrated in FIG. 19, the number of pixels in the horizontal direction and the number of pixels in the vertical direction are both odd numbers, and the coordinates of the attention point Nm coincide with the center of gravity Wg of the reference window WBm.

  The reference window WBm illustrated in FIG. 20 has an even number of pixels in the horizontal direction and the number of pixels in the vertical direction, and the coordinates of the attention point Nm are in the −X direction and the −Y direction with respect to the center of gravity Wg of the reference window WBm. 1 pixel.

  In addition to the examples shown in FIGS. 19 and 20, the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the reference window WBm may be odd and even numbers (in no particular order).

  Note that the number of pixels in the horizontal direction (X direction) of each of the standard window WBm and the reference window WRm is the same, and the number of pixels in the vertical direction (Y direction) is also the same.

  Here, as a technique for obtaining the correlation between the images of the standard window WBm and the reference window WRm, for example, a technique that does not perform frequency decomposition such as the SAD method (difference absolute value sum method), and a POC method (phase-only correlation method). ) Etc., there is a method for performing frequency decomposition, and when using a method for performing frequency decomposition, fast Fourier transform (FFT) is used to suppress the required number of computations, from the viewpoint of speeding up the processing. It is desirable that the number of pixels in the horizontal direction and the number of pixels in the vertical direction be the number of pixels represented by a power of two.

  Next, a technique for obtaining a correlation between images of the standard window WBm and the reference window WRm will be described using the SAD method and the POC method as examples.

○ SAD method:
FIG. 21 is a diagram illustrating a procedure of parallax specifying processing using the SAD method.

  An attention point Nm whose X coordinate is h2 is set on the reference image BIm of the attention layer shown in FIG. 21, and a reference window WBm is set for the attention point Nm. The X coordinates of both ends in the X direction of the reference window WBm are h1 and h3. A corresponding point CPm corresponding to the point of interest Nm is shown on the reference image RIm of the layer of interest.

  Here, an example in which the corresponding point CPm is obtained using the SAD method will be described.

  On the reference image RIm, there are a plurality of search reference points for each pixel in the range of xs ≦ x ≦ xe from the search reference point Kms whose X coordinate is xs to the search reference point Kme whose X coordinate is xe. Is set, and only the search reference points Kms and Kme at both ends are described as representatives of the plurality of search reference points.

  In addition, a reference window WBm and a plurality of reference windows of the same size are set for each of the plurality of search reference points, and only reference windows WRms and WRme at both ends are described as representatives of the plurality of reference windows. ing.

  Here, in order to simplify the description in FIG. 21, the Y coordinate of the attention point Nm and the plurality of search reference points Kms to Kme are the same, and accordingly, the reference window WBm and the reference windows WRms to WRme in the Y direction are the same. The coordinates are set so that there is no deviation. Further, the attention point Nm and the Y coordinate of the corresponding point CPm to be obtained are also set to coincide.

  Next, a correlation value between each image in each reference window of the reference windows WRms to WRme and an image in the standard window WBm is calculated. The shift amounts (also referred to as “parallax”) of the coordinates of the reference window and the reference window in the X direction and Y direction are referred to as parallax xd and yd, respectively. In the example of FIG. 21, the range of the parallax xd is xs-h2 to xe-h2, and the parallax yd is zero.

  The pixel value of each pixel (xi, yj) in the reference window WBm is Img1 (xi, yj), and each pixel (xi + xd, yj + yd) in the reference window WRm whose parallaxes are xd and yd in the X direction and Y direction. Assuming that the pixel value of Img2 (xi + xd, yj + yd), the correlation value CORp between the image in the reference window WBm and the image in the reference window WRm is calculated by the calculation using the equations (1) and (2).

  Correlation value CORp for each parallax xd according to equations (1) and (2) while changing parallax xd in the X direction in the range of xs-h2 to xe-h2 so that the reference window is scanned from WRms to WRme. Is calculated, the relationship between the X coordinate of the reference image RIm and the correlation value CORp is clarified as in the graph U1 shown in FIG.

  In the example of FIG. 21, since the maximum value of the correlation value CORp in the graph U1 is given at the coordinate xM, the reference window in which the X coordinate of the search reference point matches xM is the reference that gives the maximum value of the correlation value CORp. The search reference point of the reference window matches the corresponding point CPm corresponding to the point of interest Nm.

  FIG. 22 is a diagram illustrating a scanning mode of the reference window WRm in the SAD method, and the reference window WRm is scanned over the entire area of the reference image RIm of the target hierarchy. Since the SAD method has a smaller calculation amount than the POC method and can perform high-speed processing, even if the parallax specifying processing by the SAD method is applied to the reference image RIm having a large number of pixels, the entire area scanning shown in FIG. 22 is performed. Processing time is shortened.

  Therefore, when the SAD method is used, the number of hierarchies of the multi-resolution image can be reduced as compared with the case where a method of performing frequency decomposition is used. Also, the parallax identification process can be performed in a shorter time than when using the method of frequency decomposition even if the parallax identification is performed by scanning the reference window over the entire area of the image without obtaining a search reference point close to the corresponding point. Since it can be performed, the processing cost for obtaining the search reference point can be reduced.

○ POC method:
FIG. 23 is a diagram illustrating a parallax identification processing procedure using the POC method.

  The POC method is a correlation method using frequency decomposition and a correlation method in which amplitude components are suppressed. As a similar correlation method, there is known a DCT code limited correlation method (reference paper: “Fusion of image signal processing and image pattern recognition—DCT code limited correlation and its application”, Hitoshi Takatsuka). Since these correlation methods perform similarity calculation using a signal of only a phase component with an amplitude component suppressed from a frequency resolution signal of a pattern, a difference in imaging conditions in the stereo camera 2 for acquiring an image (here, The difference between the imaging conditions in the imaging systems 21 and 22) and the influence of noises and the like, and robust corresponding point search is possible.

  Further, since the SAD method outputs only one correlation value (CORp value) for one set of the standard window and the reference window, the reference window having the highest correlation value, that is, the parallax having the highest correlation value is specified. For this purpose, the reference window needs to be scanned over a plurality of reference windows, whereas the correlation method using the frequency decomposition such as the POC method is performed on a set of reference window and reference window images. Since the parallax between the images having the highest correlation value is obtained together with the correlation value between the images, it is not necessary to scan the reference window when the corresponding point exists in the reference window.

  For this reason, according to the corresponding point search method using the POC method for the multi-resolution image, the lowest base image of the multi-resolution image having the smallest number of pixels and the entire image area of the reference image are set as the initial standard window and the reference window, respectively. After searching for the correct corresponding point, a reference window is set in the upper hierarchy based on the corresponding point, and a search process for finding the correct corresponding point in the hierarchy without scanning the reference window is performed over all the hierarchies. It becomes possible to repeat. Therefore, the parallax specifying method using the POC method is a method suitable for the parallax specifying process for multi-resolution images.

  In the following, a case where the parallax for the target point is obtained using the image in the standard window WBm for the standard image BIm in a certain target layer and the image in the reference window WRm for the reference image RIm in the target layer as the target of the parallax specifying process. As an example, the basic principle of the POC method will be described.

  First, a standard window setting process T1a for setting the standard window WBm for the standard image BIm in the target layer and a reference window setting process T1b for setting the reference window WRm for the reference image RIm in the target layer are performed.

  Assume that the image area in the reference window WBm and the image area in the reference window WRm are expressed by the following equations (3) and (4).

Here shows a (3) of f0 (n _1, n ₂₎ and (4) of the formula g0 (n _1, n _2), the pixel values of the pixels in the reference window WBm and in the reference window WRm . N ₁ and N ₂ are set as N ₁ = 2M ₁ +1 and N ₂ = 2M ₂ +1, for example.

  Next, before performing frequency decomposition, window function processing T2a and T2b using a window function is performed in order to remove the influence of discontinuity at the edge of the window region.

  As the window function, for example, a Hanning window given by H (n1, n2) in the equation (5) is adopted, and the window function processes T2a and T2b are applied to the pixel values of the respective pixels in the standard window WBm and the reference window WRm. Expressions (6) and (7) are respectively given by multiplying the output value of the window function of Expression (5).

  As the window function, for example, a Hamming window, a Kaiser window, or the like is employed in addition to the Hanning window.

  Next, for each image area in the standard window WBm and the reference window WRm, the two-dimensional Fourier transform processes T3a and T3b shown in FIG. 23 are performed using the arithmetic expressions shown in the expressions (8) and (9). Done.

Note that N ₁ and N ₂ are substituted for the subscript P of W in the lower proviso of the equations (8) and (9), and ₁ and ₂ are substituted for the subscript s of k. Here, the frequency decomposition process is not necessarily Fourier transform, and may include any of discrete cosine transform, discrete sine transform, wavelet transform, or Hadamard transform.

  For each image region subjected to such Fourier transform processing T3a and T3b, normalization processing T4a and T4b for removing the amplitude component of the image is performed using the arithmetic expression shown in equation (10). Is called.

  When the normalization processes T4a and T4b are completed, a normalized cross power spectrum calculation process T5 using the arithmetic expression shown in the expression (11) is performed.

  When the normalized cross power spectrum calculation process T5 is completed, a two-dimensional inverse Fourier transform process T6 is performed using an arithmetic expression represented by the expression (12). As a result, the correlation calculation between the images in the standard window WBm and the reference window WRm is performed, and the result (POC value) is output.

  Through the above processing, a calculation result (POC value) indicating the correlation between the image in the standard window WBm and the image in the reference window WRm is obtained. For example, a result (POC value) as shown in FIG. 24 is obtained.

FIG. 24 is a diagram illustrating an example of the POC value. In FIG. 24, the POC value at a portion having a high correlation in the window (N ₁ × N ₂ ) is large, and the coordinates giving the peak S of the POC value are the image in the reference window WBm and the reference window WRm. The parallax with the highest correlation with the image is shown. The process for specifying the parallax is parallax specifying T7, and based on the specified parallax, the corresponding point Cpm in the reference window WRm corresponding to the point of interest Nm in the reference window WBm is obtained.

  As described above, according to the parallax specifying process using the POC method, the amplitude component of the image is removed, and the correlation calculation is performed using only the phase component of the image. Is detected with high accuracy.

◎ Second disparity specifying process:
Next, some examples of the technique for the second parallax specifying process applied when the attention point belongs to the second area will be described.

○ Second parallax identification processing method example 1:
An example of using the POC method will be described as Method Example 1 of the second parallax specifying process.

  FIG. 25 is a diagram illustrating a reference window of the first region Dm and the second region Em of the target reference image BIm, and a boundary line Gm shown in FIG. 25 is a boundary between the first region Dm and the second region Em. A line is shown.

  In FIG. 25, the reference window W1 is set based on the attention point of the first area Dm, and the reference window W2 having the same size as the reference window W1 is set based on the attention point of the second area Em. Protrudes from the reference image BIm.

  In the case where the POC method in the first parallax specifying process is applied as it is using the reference window W2, it is necessary to be able to perform a correlation operation even on a portion where the reference window W2 protrudes from the reference image BIm.

FIG. 26 is a diagram illustrating an example of the window function described using the equation (5) in the POC method in the first parallax specifying process, and the shape of the window function represented by the equation (5) is the center of gravity pc of the reference window. The direction in which the width of the reference window is 2M ₁ +1 is displayed as the function value WF0.

  In addition, the protruding portion Ex corresponds to a portion where the window function protrudes from the reference image BIm in FIG.

  Here, the correlation between search windows can be obtained by adopting, for example, a predetermined fixed value or a pixel value at the end of the image of the reference image as a provisional pixel value of a portion of the reference window W2 protruding from the reference image. A method of searching for corresponding points by obtaining values can be considered.

  However, in this case, in the window function of the expression (5) in the first parallax specifying process, the accuracy of the correlation calculation is likely to be lowered due to the temporary pixel value set in the portion of the protruding portion Ex.

  That is, when the POC method applied to the first region Dm is applied as it is, the accuracy of the parallax specifying process is likely to decrease due to the pixel value set at the portion where the reference window protrudes from the reference image.

  Therefore, as a first parallax specifying method example 1, in addition to the original function of the window function of removing the influence of discontinuity at the end of the window area, the window function of the first parallax specifying process is used. In contrast, the window function represented by the equation (13), which has a function of reducing the contribution of the window area protruding outside the image to the second parallax specifying process, is employed.

  FIG. 27 is a diagram illustrating the window function of the expression (13) as a window function used in the POC method of the second parallax specifying process according to the embodiment.

In FIG. 27, the function value WF0 of the window function in FIG. 26 is compressed in the width direction to become a function value WF1 having a width of 2L ₁ +1. The other symbols are the same as in FIG. 26, and the function value WF0 of the window function in FIG. 26 is also shown for comparison.

  As shown in FIG. 27, in the protruding portion Ex, the function value WF1 is smaller than the function value WF0.

  That is, in Method Example 1 of the second parallax specifying process, the contribution degree to the second parallax specifying process of the part of the reference window that includes the target point that protrudes from the reference image is determined as the first parallax specifying of this part. It is lower than the contribution to processing.

  As described above, in the technique example 1 of the second parallax specifying process, the accuracy of the parallax specifying process by the second parallax specifying process is suppressed by suppressing the contribution degree of the out-of-image information more than when the first parallax specifying process is used. It is possible to improve the processing performance of the parallax specifying process using the multi-resolution image.

  In addition, according to the method example 1, in the parallax specifying process using frequency analysis such as the phase correlation method, the window function which is indispensable for removing the signal discontinuity is set as the attention point of the second region. The function of reducing the contribution degree of the reference image of the reference image of the reference image to the disparity specifying process of the reference window as compared with the case where the window function of the first disparity specifying process is applied to the reference window for the same attention point. Therefore, it is possible to improve the parallax specifying accuracy for the attention point of the second region without adding new processing to the process of the parallax specifying process by the POC method, and to perform the parallax specifying process using the multi-resolution image. Processing performance can be improved.

Here, the same effect can be obtained by adopting a rectangular wave window function having a width of 2L ₁ +1 instead of the window function indicated by the function value WF1.

  FIG. 28 and FIG. 29 are diagrams illustrating another aspect of the window function of the technique example 1 of the second parallax specifying process according to the embodiment.

  28 and 29, function values WF2 and WF3 are employed in place of the function value WF1 in FIG. 27, and the other reference numerals are the same as those in FIG.

  As the function value WF2 shown in FIG. 28, the function value WF1 corresponding to the same portion is adopted as the function value on the right side of FIG. 28 from the center of gravity pc of the reference window in the direction in which the reference window protrudes from the reference image. The function value WF0 is employed as the function value for the pixel from the center of the reference image on the left side of FIG. 28 from the center of gravity pc of the reference window.

  The use of the window function indicated by the function value WF1 in FIG. 27 does not impair the usefulness of the present invention. However, if the window function indicated by the function value WF2 illustrated in FIG. In the reference window, the degree of contribution of the portion that protrudes from the reference image to the second parallax specifying process can be made lower than the contribution of the portion to the first parallax specifying process.

  Further, the same contribution degree as that of the window function WF0 used for the first parallax specifying process can be maintained for the pixels in the portion from the center of the reference window on the left side of FIG. 28 from the center of the reference image. Therefore, in the parallax specifying process, the pixel value of the pixel having higher reliability than the pixel of the protruding portion Ex from the center of the reference image can be effectively used, and the parallax specifying accuracy can be improved.

  In addition, the function value WF3 shown in FIG. 29 has a function value WF1 corresponding to the same part as the function value on the right side of FIG. 29 from the center of gravity pc of the reference window in the direction in which the reference window protrudes from the reference image. As the function value WF3 for the pixel from the center of the reference image on the left side of FIG. 28 from the center of gravity pc of the reference window, a function value having a higher contribution to the parallax specifying process than the function value WF0. Is adopted.

  That is, the window function indicated by the function value WF3 lowers the contribution of the portion of the reference window that protrudes from the reference image compared to the first parallax specifying process, and also contributes the portion of the reference window that is closer to the center of the reference image of interest. The degree is higher than that of the first parallax specifying process.

  If the window function indicated by the function value WF3 is used, the contribution degree to the second parallax specifying process of the portion of the reference window including the attention point that protrudes from the reference image is transferred to the first parallax specifying process of this portion. It can be made lower than the contribution degree.

  Further, the contribution of the pixel in the portion from the center of the reference image on the left side from the center of the reference image to the center of the reference image from the center of gravity pc of the reference window may be made higher than the window function WF0 used in the first parallax specifying process. Therefore, in the parallax specifying process, the pixel value of the pixel having higher reliability than the pixel of the protruding portion Ex from the center of the reference image can be used more effectively than the function value WF0, and the parallax specifying accuracy can be improved. Can be improved.

○ Method example 2 of the second parallax specifying process:
FIG. 30 is a diagram illustrating a setting example of the reference window W3 in the second region Em employed in the second parallax identification processing method example 2. Excluding the reference window W3, each region and boundary line in FIG. The reference window is indicated with the same reference as in FIG.

  In the reference window W3 shown in FIG. 30, the same attention point as that of the reference window W2 shown in FIG. 25 is set, and the protruding direction of the reference window W2 from the attention reference image BIm is shorter than that of the reference window W2. The disparity specifying process for the attention point in the second area is performed by the POC method using the image information in the window of the width.

  Therefore, in the second parallax specifying process in which the reference window including the target point is applied to the target point of the second region that protrudes from the reference image, the reference used in the first parallax specifying process for the protruding direction of the reference window. Since the image information in the window shorter than the window is used, the parallax identification accuracy by the second parallax identification process can be improved, and the processing performance of the corresponding point search process using the multi-resolution image can be improved. It becomes possible.

○ Method example 3 and method example 4 of the second parallax specifying process:
FIG. 31 is a diagram for explaining an example of the parallax specifying method of the second region in the technique example 3 of the second parallax specifying process.

  FIG. 32 and FIG. 33 are diagrams for explaining an example of the parallax specifying method of the second region in the technique example 4 of the second parallax specifying process.

  31 and 32, the parallax storage image SIm corresponding to the reference image BIm in FIG. 30 is shown. In the parallax storage image SIm, the same regions and boundary lines as the reference image BIm in FIG. 30 is indicated by the same reference numeral.

  In Method Example 3 and Method Example 4, first, the parallax for the attention point belonging to the first area is first calculated, and the parallax for the attention point in the second area is more than the reference image in the second area. Is identified using a parallax that has already been identified.

  In the second parallax specifying process according to the method example 3 and the method example 4, the parallax of the attention point is obtained by using the already specified parallax with respect to the point from the center of the reference image rather than the attention point. Thus, it is possible to improve the parallax specifying accuracy when the window set as the attention point protrudes from the reference image, and it is possible to improve the processing performance of the corresponding point search processing using the multi-resolution image.

  First, the method example 3 will be specifically described. The attention point p1 in FIG. 31 belongs to the first region, and the attention point p2 belongs to the second region.

  The parallax Q1 for the attention point p1 is a parallax that has already been specified by the first parallax specifying process.

  As the parallax Q2 for the attention point p2, the attention point p1 from the center of the reference image rather than the attention point p2 is used as an alternative point of the attention point p2, and the specified parallax Q1 for the alternative point is directly adopted as the parallax Q2.

  In addition, the parallax calculated for the alternative point is not only used as the parallax Q2, but for example, a parallax obtained by multiplying the parallax Q1 by a weighting factor corresponding to the interval between the points of interest p1 and p2 may be used as the parallax Q2. good.

  Moreover, the parallax about the other attention point is identified using the parallax Q1 about the attention point p1 and the parallax about the other attention point located between the attention point p1 and the attention point p2, and this parallax is used. You may specify the parallax about the attention point p2.

  That is, in Method Example 3, as the second parallax specifying process, the parallax for the target point using the parallax already specified for one alternative point closer to the center of the reference image than the target point in the target reference image. The process of specifying is performed.

  In the second parallax specifying process according to the technique example 3, the parallax of the target point is obtained using the parallax already specified for one alternative point from the center of the reference image rather than the target point.

  In this case, since only the parallax specified for one attention point is used, the second parallax specifying process by the method 3 can be accelerated, and the processing performance of the corresponding point search process using the multi-resolution image can be improved. It becomes possible to improve.

  Next, Method Example 4 will be specifically described. The attention points p1 to p4 in FIG. 32 belong to the first region, and the attention point p5 belongs to the second region.

  For the attention points p1 to p4 in the first region, the parallaxes Q1 to Q4 have already been specified by the first parallax specifying process. The identification of the parallax Q5 for the attention point p5 will be described with reference to FIG.

  FIG. 33 is a graph showing the parallaxes Q1 to Q4 with respect to the coordinates x1 to x4 in the X-axis direction of FIG. 32 for the attention points p1 to p4 shown in FIG.

  In FIG. 33, when specifying the parallax Q5 for the point of interest p5, a straight line passing through the coordinates of the parallaxes Q1 to Q4 is approximated, and the X coordinate x5 of the point of interest p5 is substituted into this line to obtain the parallax Q5. ing.

  That is, in Method Example 4, as the second parallax specifying process, the extrapolation of the parallax already specified for a plurality of alternative points closer to the center of the reference image than the target point in the target reference image is performed. Processing for specifying the parallax is performed.

  In the example shown in FIG. 33, linear approximation extrapolation is used. For example, instead of linear approximation, a curve may be approximated to perform extrapolation.

  In the second parallax specifying process according to the technique example 4, since the parallax of the target point is obtained by extrapolation of the parallax already specified for the plurality of alternative points from the center of the reference image rather than the target point, the plurality of target points are crossed. Parallax identification processing utilizing the continuity of parallax can be performed, the parallax identification accuracy for the attention point of the second region where the window protrudes from the reference image can be improved, and corresponding point search processing using a multi-resolution image It is possible to improve the processing performance.

◎ Overall processing flow:
Next, an overall processing flow of corresponding point search processing using a multi-resolution image performed by the image processing apparatus 3A will be described.

  FIG. 37 is a flowchart illustrating an outline of the procedure of the parallax specifying processing S100A using the multi-resolution image performed by the image processing device 3A according to the embodiment.

  In the flowchart of FIG. 37, first, the image acquisition unit 10 acquires an image captured by the standard camera 21 and an image captured by the reference camera 22 as a standard image BIT and a reference image RIT, respectively, and stores them in the storage unit 9. (Step S10).

  When the image acquisition is completed, the multi-resolution image creating unit 11 stores control parameters related to multi-resolution image creation, such as the number T of multi-resolution images and the resolution scaling factor RR, the standard image BIT, and the reference image from the storage unit 9. RIT is acquired, the multi-resolution standard image MB and the multi-resolution reference image MR are created by performing multi-stage resolution reduction processing on the standard image BIT and the reference image RIT using the control parameters, and stored in the storage unit 9 (Step S20).

  When the creation of the multi-resolution image is completed, if the target layer is not set, the control unit 16 uses the lowest base image BI1 and the reference image RI1 of the multi-resolution standard image MB and the multi-resolution reference image MR as the target layer. New standard image BIm and reference image RIm, and if there is an existing standard image BIm and reference image RIm, standard image BI (m + 1) and reference image RI that are one level higher than standard image BIm and reference image RIm. (M + 1) is reset as the reference image BIm and the reference image RIm of the new target layer, respectively, and the setting information is supplied to the initial parallax setting unit 14, the parallax specifying unit 15, and the like (step S30).

  When the setting of the attention layer is completed, the control unit 16 acquires the region information V1 of the first region and the second region for the attention layer from the storage unit 9, and supplies the region information V1 to the region determination unit 13 (step S40).

  Note that, by using the coordinates of the attention point and the search window sizes WX and WY without using the region information V1, it is determined whether the reference point for the attention point protrudes from the reference image. Since it can be determined whether it belongs to one area or the second area, in this case, the process of step S40 need not be performed.

  When the acquisition of the region information ends, the attention point setting unit 12 sets the back of the reference image pixel as the attention point (step S50).

  When the setting of the attention point is completed, the process proceeds to processing step S60A for specifying the parallax of the attention point.

  FIG. 38 is a flowchart illustrating an overview of the processing in step S60A for specifying the parallax of the target point according to the embodiment.

  First, the initial parallax setting unit 14 performs the initial parallax setting process described in the description column for the initial parallax setting unit 14 and sets the initial parallax for the point of interest (step S210).

  Note that, when the attention layer is the lowest layer, the initial disparity cannot be set using the disparity in the one lower layer, and therefore, for example, the initial disparity 0 is set for the attention point.

  When the initial parallax setting for the attention point is completed, the region determination unit 13 determines whether the attention point belongs to the parallax specific region defined by the region information V1 (step S220). In the present embodiment, specifically, it is determined whether the attention point belongs to the first region or the second region, and the determination result is supplied to the control unit 16 or the like.

  The control unit 16 acquires a determination result for the region to which the attention point belongs, and determines whether or not the reference window set for the attention point protrudes from the reference image, that is, whether or not the attention point belongs to the first region. (Step S230).

  As a result of the determination, if the target point belongs to the first area, the control unit 16 causes the parallax specifying unit 15 to execute the described first parallax specifying process, and returns the process to step S60A in FIG. S240).

  As a result of the determination, when the target point belongs to the second region, the control unit 16 causes the parallax specifying unit 15 to execute the described second parallax specifying process, and returns the process to step S60A in FIG. S250).

  When the process returns to step S60A, the control unit 16 determines whether or not the disparity specifying process has been completed for all the attention points in the attention layer, and as a result of the determination, the control unit 16 performs the disparity specifying process for all attention points in the attention layer. If not completed, the process returns to step S50. If the parallax identification process has been completed for all the points of interest in the layer of interest, the process proceeds to step S80 (step S70).

  In step S80, the control unit 16 determines whether or not the disparity specifying process has been completed for all the hierarchies. If not, the control unit 16 sets the upper hierarchy as the target hierarchy (step S30) and ends. If so, the control unit 16 ends the parallax specifying process S100A using the multi-resolution image.

  Through the above processing, the parallax identification processing according to the present embodiment is executed.

<Modification>
The present invention is not limited to the above-described embodiment, and various modifications and improvements can be made without departing from the scope of the present invention.

  For example, as the region information V1 in the storage unit 9, region information V1 may be set such that the area ratio of the parallax specifying process in each layer with respect to the reference image in each layer increases as the upper layer. Hereinafter, Modification 1 and Modification 2 will be described as specific examples.

  The image processing apparatuses according to the first and second modifications employ the same components as those of the image processing apparatus 3A according to the embodiment, and only the area information V1 stored in the storage unit 9 is set differently from the embodiment. It can be configured simply by changing to

  Therefore, in the first and second modifications, description of each component of the image processing apparatus according to the first and second modifications is omitted, and only the difference from the embodiment caused by the change of the region information V1 is described. This will be described below.

<Regarding Image Processing Apparatus According to Modification 1>
In the first modification, the region information V1 is set so that the parallax specifying region in the reference image of each layer is composed of only the first region described in the embodiment.

  Note that the parallax specifying region in the reference image of each layer is not limited to the maximum range that the first region can take, and the area ratio of the parallax specifying process in each layer with respect to the reference image in each layer is larger in the upper layer If the condition is satisfied, it may be a part of the maximum range that the first region can take.

  That is, the parallax specific area according to the modification 1 is an area in a range where the reference window set for the attention point of the attention reference image fits in the attention reference image.

  Further, as selection information for the parallax specifying process included in the region information V1, information for selecting the first parallax specifying process is employed.

  Therefore, the parallax specifying unit 15 in Modification 1 uses the image information in the standard window set based on the target point of the standard image to obtain the parallax between the target standard image and the target reference image for the target point. A first parallax specifying process is performed.

  FIG. 34 is a diagram illustrating a setting example of a parallax specifying region for a plurality of hierarchical reference images according to the first modification.

  FIG. 34 shows three consecutive layers of reference images BI (m−1), BIm, and BI (m + 1) from the lower layer to the upper layer.

  In the reference images BI (m−1), BIm, and BI (m + 1), parallax specific regions D (m−1), Dm, and D (m + 1) are set, respectively. This is the first region in the reference images BI (m−1), BIm, and BI (m + 1).

  Here, in FIG. 34, reference image BI (m−1) is set with reference to reference image BI (m + 1) so that BI (m−1), BIm, and BI (m + 1) are all displayed with the same image size. ) And the scale of the reference image BIm are adjusted and displayed.

  As described in FIG. 14 and FIG. 15 according to the description of the embodiment, when the parallax specifying region of the first modification is the maximum range of the first region of the reference image in each layer, the size of the window (number of pixels) If the shape is the same between the hierarchies, the area ratio of the parallax specifying process in each hierarchy with respect to the reference image in each hierarchy becomes larger as the upper hierarchy.

  In the first disparity specifying process employed in the first modification, it is necessary to set an initial disparity for setting a reference window for the reference window for the attention point of the disparity specifying area.

  Here, as mentioned in the description of the initial parallax setting unit 14 according to the embodiment, the setting of the initial parallax when the parallax is not specified at the target point in the lower layer that has a correspondence relationship with the target point in the target layer. A method will be described.

  FIG. 35 is a diagram for explaining an example of an initial parallax setting method according to the first modification.

  In FIG. 35, the reference image BIm of interest and the reference image BI (m−1) one level lower are shown.

  A region D (m−1) in the reference image BI (m−1) is a parallax specifying region including a point of interest constituting the first region, and a boundary line G (m−1) is a parallax specifying region D (m− This is the outer edge of 1).

  In addition, the parallax specifying region Dm in the reference image BIm is a region including a region surrounded by the boundary line G (m−1) and a hatched region, and the boundary line Gm is an outer edge of the parallax specifying region Dm. Is made.

  Here, the boundary line G (m−1) in the reference image BIm is set in the same range as the boundary line G (m−1) of the reference image BI (m−1).

  Accordingly, for the attention point in the region surrounded by the boundary line G (m−1) of the attention reference image BIm, based on the parallax specified in the reference image BI (m−1) of the lower hierarchy, the method of FIG. The initial parallax can be set by the same method.

  Next, in the hatched region of the reference image of interest, the initial parallax cannot be set by the method of FIG. 17 because the parallax has not been specified for the target point of the corresponding lower layer.

  Therefore, the initial parallax of the hatched area is set based on the initial parallax already set for the attention point in the area surrounded by the boundary line G (m−1) of the attention reference image BIm.

  Specifically, in the same manner as the method example 3 and the method example 4 of the second parallax specifying process for the embodiment described with reference to FIGS. 31 to 33, one or a plurality of already set initial parallaxes are set. Using the attention point as an alternative point, the initial parallax for the attention point in the hatched area where the initial parallax is not set is obtained by extrapolation or the like using the initial parallax of the alternative point.

  When the initial parallax is set for all the points of interest in the parallax specifying area Dm, the parallax is specified by the first parallax specifying process.

  Similarly, by repeating the initial parallax setting and the parallax specifying process for the upper layer above the reference image BIm, the higher resolution uses the multi-resolution image while increasing the area ratio of the parallax specifying region to the reference image. Can be performed.

<Regarding Image Processing Apparatus According to Modification 2>
The parallax specifying area specified by the area information V1 of the image processing apparatus according to the modification 2 is configured by the first area and the second area distributed outside the first area, similarly to the area information according to the embodiment. The

  The selection information of the parallax specifying process included in the area information V1 applies the first parallax specifying process for the attention point in the first area, and applies the second parallax specifying process for the attention point in the second area. It is stipulated that

  Here, a difference between the second region according to the second modification and the second region according to the embodiment will be described with reference to FIG.

  FIG. 36 is a diagram illustrating a setting example of the parallax identification area according to the second modification.

  In FIG. 36, three consecutive hierarchical reference images BI (m−1), BIm, and BI (m + 1) are arranged from the lower hierarchy to the upper hierarchy.

  The first areas D (m−1), Dm, and D (m + 1), and the second areas E (m−1), Em, and E (m + 1) are set as the respective reference images.

  As shown in FIG. 36, the area ratio of the parallax specific region composed of the first region and the second region to the reference image is set so as to increase as the higher layer.

  Since the initial region needs to be set with respect to the first region, it is set by the same method as in the embodiment and the first modification.

  That is, in the initial parallax setting processing for the attention point of the first region of the attention reference image BIm, the parallax specified for the attention point of the reference image BI (m−1) of the one lower hierarchy corresponding to this attention point is determined. When present, it is set based on this parallax.

  When there is no parallax specified for the target point of the reference image BI (m−1) in the lower hierarchy, the reference for which the initial parallax has already been set using the initial parallax setting method described in Modification 1 The initial parallax for the target point is set by extrapolation or substitution from the initial parallax for the target point of the image BIm.

  As the second parallax specifying process applied to the second region, Method Example 1 to Method Example 4 are employed as in the second parallax specifying process according to the embodiment.

  For the attention point in the second area, the setting of the initial parallax when the second parallax specifying process that requires the setting of the initial parallax is employed can be performed in the same manner as the setting of the initial parallax for the first area.

  That is, in the second modification, the initial parallax used in the first parallax specifying process for the upper hierarchy is obtained using the parallax obtained at each point of interest for the lower hierarchy.

  As described above, the parallax specifying area is set by the area information V1, the first parallax specifying process is specified for the attention point in the first area by the selection information of the parallax specifying process included in the area information V1, and the second In addition to specifying the parallax specifying process, it is also possible to set initial parallax for a point of interest that requires setting of initial parallax.

  Therefore, in the second modification, as in the first modification, the initial parallax setting process (corresponding point search process) using the multi-resolution image can be performed.

  If the methods of Modification 1 and Modification 2 are used, the ratio between the parallax identification area and the reference image is slightly increased as the hierarchy increases in the process in which the parallax identification processing is sequentially executed from the lower hierarchy to the upper hierarchy. Since the parallax specifying process can be performed while increasing the size gradually, the parallax specifying process can be accelerated by reducing the parallax specifying area, and the processing performance of the corresponding point search process using the multi-resolution image is improved. Is possible.

  Further, if the technique of the first modification is used, the reference window including the attention point is based on each parallax obtained for each attention point in the parallax specific area that does not protrude from the image area of the reference image in the lower hierarchy. By setting each initial parallax for each target point in each parallax specific region, it is possible to set each initial parallax for the parallax specific region in the target layer to an appropriate value, so that the parallax specific processing can be speeded up. It is possible to improve the processing performance of the corresponding point search processing using the multi-resolution image.

  If the method of the modification 2 is used, each initial parallax for each target point in a region where the reference window including the target point does not protrude from the image region of the target reference image among the parallax specifying regions of the target layer is converted to the lower layer By setting based on each parallax obtained for each target point in the parallax specific region, each initial parallax for a region where the window in the target layer does not protrude from the reference image can be set to an appropriate value. Therefore, it is possible to speed up the parallax specifying process and improve the processing performance of the corresponding point search process using the multi-resolution image.

<Overall processing flow of modification>
○ Overall processing flow of Modification Example 1:
FIG. 37 is a flowchart illustrating an outline of the procedure of the parallax specifying process S100B using the multi-resolution image performed by the image processing device 3A according to the first modification.

  Here, the parallax specifying process S100B of Modification 1 is the same as the parallax specifying process S100A using the multi-resolution image performed by the image processing device 3A according to the embodiment, except for the processing step S60B of specifying the parallax of the target point. Therefore, the description of the parallax specifying process S100B is omitted.

  FIG. 39 is a flowchart illustrating an overview of the process of process step S60B for specifying the parallax of the target point according to the first modification.

  In the flowchart of FIG. 39, first, the region determination unit 13 determines whether or not the attention point set by the attention point setting unit 12 belongs to the parallax specific region defined by the region information V1 (step S310). .

  Specifically, in the first modification, it is determined whether or not the reference window set as the attention point protrudes from the reference image, and the determination result is supplied to the control unit 16 or the like.

  The control unit 16 acquires a determination result for the region to which the attention point belongs, and determines whether or not the reference window set for the attention point protrudes from the reference image, that is, whether or not the attention point belongs to the first region. To do.

  As a result of the determination, if the target point does not belong to the first region, the control unit 16 returns the process to step S60B (FIG. 37) (step S320).

  If the target point belongs to the first region as a result of the determination, the initial parallax setting unit 14 sets the initial parallax for the target point by the setting method described above (step S330).

  When the initial parallax is set for the point of interest, the control unit 16 causes the parallax specifying unit 15 to execute the first parallax specifying process, and returns the process to step S60B (FIG. 37) (step S340).

  Through the above processing, the processing step S60B for specifying the parallax of the target point according to the first modification is performed.

○ Overall processing flow of Modification 2:
FIG. 37 is a flowchart illustrating an outline of the procedure of the parallax specifying process S100C using the multi-resolution image performed by the image processing device 3A according to the second modification.

  Here, the parallax specifying process S100C of Modification 2 is the same as the parallax specifying process S100A according to the embodiment except for the processing step S60C for specifying the parallax of the target point, and thus the description of the parallax specifying process S100C is omitted. To do.

  FIG. 40 is a flowchart illustrating an overview of the process of process step S60C for specifying the parallax of the target point according to Modification 2.

  In the flowchart of FIG. 40, first, the region determination unit 13 determines whether or not the attention point set by the attention point setting unit 12 belongs to the parallax specific region defined by the region information V1 (step S410). .

  Specifically, in the second modification, it is determined whether or not the reference window set as the attention point protrudes from the reference image, and whether or not the attention point belongs to the second region. Supplied to the unit 16 and the like.

  First, the control unit 16 acquires a determination result for the region to which the attention point belongs, and whether or not the reference window set for the attention point protrudes from the reference image, that is, whether or not the attention point belongs to the first region. Is determined (step S420).

  As a result of the determination, when the attention point belongs to the first area, the initial parallax setting unit 14 sets the initial parallax for the attention point of the first area by the setting method described above (step S430).

  When the initial parallax is set for the attention point, the control unit 16 causes the parallax specifying unit 15 to execute the first parallax specifying process, and returns the process to step S60C (FIG. 37) (step S440).

  As a result of the determination in step S420, if the attention point does not belong to the first area, the control unit 16 determines whether or not the attention point protrudes from the predetermined area of the second area (step S450).

  As a result of the determination, when the attention point belongs to the predetermined area, the initial parallax setting unit 14 sets the initial parallax for the attention point of the predetermined area by the setting method described above (step S460).

  When the initial parallax is set for the attention point, the control unit 16 causes the parallax specifying unit 15 to execute the second parallax specifying process, and returns the process to step S60C (FIG. 37) (step S470).

  If the result of determination in step S450 is that the target point does not belong to the predetermined area, the control unit 16 returns the process to step S60C (FIG. 37).

  Through the above processing, the processing step S60C for specifying the parallax of the target point according to the modified example 2 is performed.

DESCRIPTION OF SYMBOLS 1A 3D shape measuring apparatus 2 Stereo camera 3A Image processing apparatus 4 Operation part 5 Display part 6 Communication part 7 Input / output part 8 Storage medium 9 Storage part 10 Image acquisition part 11 Multi-resolution image creation part 12 Attention point setting part 13 Area determination Unit 14 initial parallax setting unit 15 parallax specifying unit 16 control unit 21, 22 imaging system DL data line OB measurement object BIT, BI (T−1), BI (m + 1), BIm, BI1 reference image RIT, RI (T− 1), RI (m + 1), RIm, RI1 Reference image MB Multi-resolution standard image MR Multi-resolution reference image Nm, N (m + 1) Attention point CPm, CPmT, CPmF Corresponding point Km, K (m + 1), K (m + 1) T , K (m + 1) F Search reference point SIm Parallax storage image WBm Reference window WRm Reference window Wg Search window centroid R Resolution magnification WX, WY search window size

Claims (7)

Image acquisition means for acquiring an original standard image and an original reference image including image contents corresponding to each other;
By reducing the resolution of the original reference image and the original reference image so that the resolution decreases from the upper layer to the lower layer, the plurality of reference images and the plurality of reference images having different resolutions are hierarchically arranged. A multi-resolution image creating means for generating
Of the plurality of reference images and the plurality of reference images, an attention point setting means for setting an attention point in the attention reference image using an image belonging to the same attention layer as the attention reference image and the attention reference image;
Parallax specifying means for obtaining parallax between the target reference image and the target reference image for the target point;
When the parallax specifying means obtains the parallax,
1) When a window containing the attention point fits in the attention reference image, a first parallax specifying rule using image information in the window is
2) When the window protrudes from the target reference image, a second parallax specifying rule in which the contribution from the external region of the target reference image is lower than the first parallax specifying rule,
Selective application means for selectively applying each;
Equipped with a,
The second parallax specific rules, the image processing apparatus according to claim rule der Rukoto utilizing parallax that have already been identified for the points near the center than the point of interest in the target reference picture. An image processing apparatus according to claim 1,
The second parallax specifying rule is a rule for obtaining a parallax for the attention point using a parallax already specified for one alternative point closer to the center than the attention point in the attention reference image. An image processing apparatus. An image processing apparatus according to claim 1 ,
The second parallax specifying rule is a rule for obtaining a parallax for the attention point by extrapolation of the parallax already specified for a plurality of alternative points closer to the center than the attention point in the attention reference image. An image processing apparatus. Image acquisition means for acquiring an original standard image and an original reference image including image contents corresponding to each other;
By reducing the resolution of the original reference image and the original reference image so that the resolution decreases from the upper layer to the lower layer, the plurality of reference images and the plurality of reference images having different resolutions are hierarchically arranged. A multi-resolution image creating means for generating
Of the plurality of reference images and the plurality of reference images, an attention point setting means for setting an attention point in the attention reference image using an image belonging to the same attention layer as the attention reference image and the attention reference image;
Parallax specifying means for obtaining a parallax between the target reference image and the target reference image for the target point when the target point is in a predetermined parallax specific region of the target standard image;
With
The image processing apparatus according to claim 1, wherein an area ratio between the parallax specifying region and the reference image in each layer is set to be higher in an upper layer . An image processing apparatus according to claim 4 ,
The parallax specifying means obtains a parallax between the target reference image and the target reference image for the target point using image information in a window including the target point;
The parallax identification area is an area in a range where the window fits in the target reference image,
An image processing apparatus, characterized in that an initial parallax used as an initial reference for obtaining the parallax in an upper hierarchy is obtained using parallax obtained for a point of interest in a parallax specific area in a lower hierarchy . Image acquisition means for acquiring an original standard image and an original reference image including image contents corresponding to each other;
By reducing the resolution of the original reference image and the original reference image so that the resolution decreases from the upper layer to the lower layer, the plurality of reference images and the plurality of reference images having different resolutions are hierarchically arranged. A multi-resolution image creating means for generating
Of the plurality of reference images and the plurality of reference images, an attention point setting means for setting an attention point in the attention reference image using an image belonging to the same attention layer as the attention reference image and the attention reference image;
Parallax specifying means for obtaining parallax between the target reference image and the target reference image for the target point;
When the parallax specifying means obtains the parallax,
1) When a window containing the attention point fits in the attention reference image, a first parallax specifying rule using image information in the window is
2) When the window protrudes from the target reference image, a second parallax specifying rule in which the contribution from the external region of the target reference image is lower than the first parallax specifying rule,
Selective application means for selectively applying each;
With
An area ratio between the parallax specific region and the reference image in each layer is defined as a parallax specific region in the target reference image, which is a region composed of a region to which the first rule is applied and a region to which the second rule is applied. , The higher the hierarchy is set,
An image processing apparatus , wherein an initial parallax used in the first parallax specifying rule for an upper layer is obtained using parallax obtained at each point of interest for a lower layer . Acquiring an original standard image and an original reference image including image contents corresponding to each other;
By reducing the resolution of the original reference image and the original reference image so that the resolution decreases from the upper layer to the lower layer, the plurality of reference images and the plurality of reference images having different resolutions are hierarchically arranged. A step of generating
Of the plurality of reference images and the plurality of reference images, a step of setting an attention point in the attention reference image using an image belonging to the same attention layer as the attention reference image and the attention reference image;
Obtaining a parallax between the target reference image and the target reference image for the target point when the target point is in a predetermined parallax specifying region of the target standard image;
With
The image processing method according to claim 1, wherein an area ratio between the parallax specifying region and the reference image in each layer is set to be higher in an upper layer .

Images