JP2011165117A - Apparatus, method and program for processing image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform accurate matching by simple processing even when a hidden area is present in a stereo-matching method for performing a corresponding point search by template matching. <P>SOLUTION: A boundary pixel designation part 21 designates a boundary pixel that is composed of one or more pixels corresponding to a boundary between the hidden area that is an area not projected in either of a first image and a second image by the effect of an object and a non-hidden area that is an area projected in both respectively in the first image and the second image. A correlation calculation range determination part 22 determines a correlation calculation range based on the respective boundary pixels in the first image and the second image. A correlation calculation part 13 calculates, for each of overall or partial combinations between a plurality of block images in the first image and a plurality of block images in the second image, a correlation coefficient of pixel value according to the correlation calculation range. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing device, an image processing method, and an image processing program, and more particularly to an image processing device, an image processing method, and an image processing program that perform stereo matching.

  As an example of a conventional stereo processing method, for example, Patent Document 1 (Japanese Patent Laid-Open No. 4-299474) discloses the following technique. That is, the correlation between the pixel values of the blocks of the two images A and B is calculated by the correlation calculation means. For each of the directions from A to B and from B to A, the correlation coefficients are ranked by the correlation coefficient ranking means. Further, the sum of the ranks in the A to B direction and the B to A direction is obtained by the correlation coefficient rank adding means. The sum of the ranks is integrated, and the minimum combination rank selection means obtains the corresponding combination that minimizes the sum of a certain penalty corresponding to the number of uncorresponding blocks.

  Here, for the calculation of the correlation coefficient, template matching using a correlation value between M × N size blocks in two images as shown in Non-Patent Document 1 is known, and matching is achieved. In general, the template position (corresponding point) is set as the center position (attention point) at the time. In this case, for example, odd numbers such as 11 and 13 are used for M and N, matching is obtained at the position of the center pixel (attention point), and processing that assumes that the attention point is obtained as a corresponding point is performed. .

  Patent Document 2 (Japanese Patent Laid-Open No. 11-183142) discloses the following technique. That is, in a three-dimensional image imaging method for measuring luminance information and distance information for a measured object, the measured object is photographed from a plurality of imaging positions to obtain a plurality of parallax images. Next, one of the parallax images is used as luminance information, and is necessary for stereo viewing from the image pair of the reference parallax image serving as the luminance information and the first parallax image at the imaging position closest to the imaging position. The first corresponding pixel search is performed, and each distance from the imaging position for each pixel of the luminance information is calculated from the parallax of the image pair within a range where the first occlusion does not occur, and stored as distance information. Next, the corresponding pixel search is performed again from the image pair of the reference parallax image and the second parallax image at the imaging position second closest to the imaging position, and each pixel is handled in a range where the second occlusion does not occur. Each distance to be calculated is calculated, and a distance information rewriting process for rewriting the already stored distance information is performed for the pixels for which the distance has been calculated. Next, in the distance information rewriting process, the process is sequentially repeated for image pairs of the reference parallax image and each parallax image whose imaging position is close to the third and subsequent positions. Then, the pixel in the range where the first occlusion occurs is stored as an occlusion pixel without distance information, and the pixel in the range where the first occlusion occurs is the nearest neighbor to this pixel. Assign pixel distance information.

JP-A-4-299474 JP-A-11-183142

New Edition Image Analysis Handbook Mikio Takagi, Yoshihisa Shimoda, University of Tokyo Press, pp.1669-1672

  A stereo matching method that uses a template image for matching has a problem in that if there is a sharp shape change in an object to be stereo processed, the accuracy of stereo matching in the vicinity of the changed portion may be reduced. .

  The reason is as follows. In other words, there is a hidden area (occlusion area) that is a part where there is no correspondence between both images around the shape change part. In the correlation coefficient calculation, the correlation coefficient value in the vicinity of the boundary portion between the hidden area and the non-hidden area is not necessarily maximized, and a stereo matching correspondence error occurs. Hereinafter, this will be described with reference to schematic views.

  FIG. 11 is a schematic diagram illustrating a method for capturing an L image and an R image. FIG. 12 is a diagram for explaining a difference in appearance between the L image and the R image in the stereo image.

  A subject BL shown in FIG. 11 is, for example, a building from which an aerial photograph is taken. Further, the stereo image shown in FIG. 12, that is, the L image (left image) and the R image (right image) are aerial photographs continuously taken at the positions L and R shown in FIG. That is, FIG. 12 shows a schematic diagram of an L image and an R image obtained when shooting is performed as shown in FIG.

  FIG. 13 is a diagram for explaining a hidden area in a stereo image. FIG. 14 is a diagram showing the left and right height images.

  FIG. 14 shows an example of the result of obtaining the height value by obtaining the corresponding points between the L image and the R image. The L-height image indicates a result obtained by obtaining points corresponding to each point of the L image from the R image and obtaining a positional deviation amount between the L image point and the corresponding point of the R image, that is, the parallax as the height. It is an image. Similarly, in the R height image, points corresponding to each point of the R image are obtained from the L image, and the amount of positional deviation between the points of the R image and the corresponding points of the L image, that is, the parallax is obtained as the height. It is an image which shows a result.

  Here, in the L-height image and the R-height image, there are left and right areas (hatched areas in FIG. 14) where the respective imaging ranges do not overlap and the height cannot be obtained. A portion where the shooting ranges of the L image and the R image do not overlap, that is, the hatched area in FIG. 14, can be easily dealt with by setting only the shooting range as a target of stereo matching.

  On the other hand, since the L-height image and the R-height image are hidden by the subject BL at the time of shooting, there are areas that are reflected in one image but not in the other image, that is, hidden areas OCL1 and OCL2. Exists.

  FIG. 15 is a diagram illustrating a block image in which correlation coefficient calculation is performed at the boundary between a hidden area and a non-hidden area. That is, FIG. 15 shows processing of a block image including a hidden area.

  Referring to FIG. 15, block image A is an image when the entire block centering on corresponding point P is shown in both the L image and the R image. In this case, it is established that the correlation coefficient between the block images in which the same object is captured is maximized.

On the other hand, in the block image B, the attention point P appears in both the L image and the R image, but a part of the region where the correlation coefficient is calculated corresponds to the hidden region OCL. That is, FIG.
When the case shown in FIG. 2 is considered, the correlation between the region F of the L image and the correlation between the region D and the region E of the R image are compared (the region D and the region E are hidden regions).

  For this reason, when there is a hidden area in the block image, generally, the possibility that the correlation coefficients match with each other is reduced, while on the other hand, a higher correlation may be obtained in combination with other block images, That is, there is a high possibility that an incorrect corresponding point will be obtained.

  Here, the technique described in Patent Document 1 is for performing accurate matching between images having occlusion by performing bidirectional matching when matching is performed for matching two images. However, a technique for performing high-precision matching beyond the technique described in Patent Document 1 is desired.

  Further, in the technique described in Patent Document 2, it is necessary to perform the corresponding pixel search process for a plurality of images, which increases the processing amount. In addition, when the occlusion occurs, only the process of assigning the distance information of the nearest pixel is performed, so that the matching rate around the hidden region cannot be improved in stereo matching.

  The present invention has been made to solve the above-described problems. The purpose of the present invention is to achieve high-accuracy matching with simple processing even in the presence of a hidden region in a stereo matching method for searching for corresponding points by template matching. An image processing apparatus, an image processing method, and an image processing program that can be performed are provided.

  In order to solve the above problems, an image processing apparatus according to an aspect of the present invention includes an image dividing unit for dividing each of a first image and a second image into a plurality of block images, and an influence of a subject. A boundary that is one or a plurality of pixels corresponding to a boundary between a hidden area that is not shown in any of the first image and the second image and a non-hidden area that is shown in both A boundary pixel designating unit for designating pixels in the first image and the second image, respectively, and a correlation calculation range is determined based on each of the boundary pixels in the first image and the second image. A correlation calculation range determining unit, and a combination of all or part of the plurality of block images in the first image and the plurality of block images in the second image For each combination, the correlation calculation unit for calculating the correlation coefficient of the pixel value according to the correlation calculation range, and the correlation coefficient value for each combination calculated by the correlation calculation unit, The first ranking is performed in descending order of the correlation coefficient from the block image in the first image to the block image in the second image, and the block image in the second image is related to the first in relation to the combination. A correlation coefficient ranking unit for performing the second ranking in descending order of the correlation coefficient with respect to the block image in each image, and for each combination, the sum of the first rank and the second rank is calculated. It is expressed by connecting the plurality of paths with a correlation coefficient rank adding unit for calculating, a path generating unit for generating a plurality of paths between the combinations. For each group of the combination, a sum calculated by the correlation coefficient rank addition unit is added, and a final addition rank selection unit for obtaining the group that minimizes the result of the addition using dynamic programming, Is provided.

In order to solve the above problems, an image processing method according to an aspect of the present invention includes (a) a step of dividing each of the first image and the second image into a plurality of block images; One or a plurality of pixels corresponding to a boundary between a hidden area that is not reflected in any of the first image and the second image and a non-hidden area that is reflected in any of the first image and the second image due to influence. Designating a certain boundary pixel in each of the first image and the second image, and (c) determining a correlation calculation range based on each boundary pixel in the first image and the second image. And (d) for each or some combination between the plurality of block images in the first image and the plurality of block images in the second image, (E) calculating the correlation coefficient of the pixel value according to the correlation calculation range; and (e) using the correlation coefficient value for each combination calculated in the step (d) above, The first ranking is performed in descending order of the correlation coefficient from the block image in the image to the block image in the second image, and the block image in the second image in the first image is related to the combination. Performing a second ranking in descending order of the correlation coefficient to the block image; (f) calculating a sum of the first ranking and the second ranking for each combination; and (g ) Generating a plurality of paths between each of the combinations; and (h) for each group of the combinations expressed by connecting the plurality of paths. The adds the sum calculated in step (f), and determining the group to which the result of the addition is minimized by using a dynamic programming.

  In order to solve the above problems, an image processing program according to an aspect of the present invention includes: (a) a step of dividing each of the first image and the second image into a plurality of block images; and (b) a subject image. One or a plurality of pixels corresponding to a boundary between a hidden area that is not reflected in any of the first image and the second image and a non-hidden area that is reflected in any of the first image and the second image due to influence. Designating a certain boundary pixel in each of the first image and the second image, and (c) determining a correlation calculation range based on each boundary pixel in the first image and the second image. (D) a combination of all or part of the plurality of block images in the first image and the plurality of block images in the second image. (E) calculating a correlation coefficient of pixel values according to the correlation calculation range; and (e) using the correlation coefficient value for each combination calculated in the step (d) above, The first ranking is performed in the descending order of the correlation coefficient from the block image in the first image to the block image in the second image, and the first image from the block image in the second image is related to the combination. Performing a second ranking in descending order of the correlation coefficient to the block image in the image; (f) calculating a sum of the first ranking and the second ranking for each of the combinations; (G) generating a plurality of paths between each of the combinations, and (h) a group of the combinations expressed by connecting the plurality of paths. Each in the above by adding the sum calculated in step (f), and a step of obtaining using dynamic programming the group result of the addition is minimized to the computer.

  According to the present invention, in a stereo matching method for searching for corresponding points by template matching, high-precision matching can be performed with simple processing even when a hidden region exists.

It is a figure for demonstrating the designation | designated method of the boundary pixel in the boundary of a hidden region and a non-hidden region. It is an enlarged view of the boundary part of a hidden region and a non-hidden region. It is a figure which shows the deformation | transformation of the correlation calculation range when a boundary pixel exists in a block image. 1 is a schematic configuration diagram of an image processing apparatus according to a first embodiment of the present invention. It is a block diagram which shows the control structure which the image processing apparatus which concerns on the 1st Embodiment of this invention provides. It is a flowchart which shows the operation | movement procedure at the time of the image processing apparatus which concerns on the 1st Embodiment of this invention performing a stereo matching process. It is a figure which shows the example of the correlation calculation range which the correlation calculation range determination part which concerns on the 1st Embodiment of this invention designates. It is a figure which shows the method of calculating | requiring a correlation matrix using a block image. It is a block diagram which shows the control structure which the image processing apparatus which concerns on the 2nd Embodiment of this invention provides. It is a flowchart which shows the operation | movement procedure at the time of the image processing apparatus which concerns on the 2nd Embodiment of this invention performing a stereo matching process. It is a schematic diagram which shows the imaging | photography method of L image and R image. It is a figure for demonstrating the difference in the appearance of the L image and R image in a stereo image. It is a figure for demonstrating the hidden area in a stereo image. It is a figure which shows a left-right height image. It is a figure which shows the block image in which a correlation coefficient calculation is performed in the boundary of a hidden area and a non-hidden area.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals and description thereof will not be repeated.

<First Embodiment>
[Overview]
The image processing apparatus according to the first embodiment of the present invention uses the boundary pixel designating unit 21 (to be described later) for designating the boundary between the hidden region and the non-hidden region, thereby providing a boundary point. It is intended to correct mishandling in.

  FIG. 1 is a diagram for explaining a method for specifying a boundary pixel at the boundary between a hidden area and a non-hidden area. FIG. 1 shows an example of the correspondence given by the boundary pixel designating unit 21. FIG. 2 is an enlarged view of a boundary portion between the hidden area and the non-hidden area.

  Referring to FIGS. 1 and 2, boundary pixel specifying unit 21 specifies edges corresponding to each other in the L image and the R image, that is, a line segment indicating the boundary between the hidden region and the non-hidden region. Here, the boundary pixel designating unit 21 designates α1 and α2 as corresponding boundaries in the L image and the R image, respectively, and designates β1 and β2 as corresponding boundaries in the L image and the R image, respectively.

  The correlation calculation range setting unit 22 to be described later changes the calculation range in the block image to be subjected to stereo matching when calculating the correlation coefficient for the matching process in the vicinity of the explicitly given boundary. Thereby, when a pixel corresponding to the boundary specified by the boundary pixel specifying unit 21 (hereinafter also referred to as a boundary pixel) is included in the matching window, that is, the correlation calculation range, the region including the attention point with the line segment as a boundary. It is possible to perform the matching determination by calculating the correlation coefficient only in the region including the attention point.

  FIG. 3 is a diagram illustrating a modification of the correlation calculation range when the boundary pixel exists in the block image. FIG. 3 shows a block image A in a normal correlation calculation range and a block image B after deformation by the correlation calculation range setting unit (matching window deformation unit) 22.

  In the deformed block image B, since the matching determination can be performed only between the regions of the L image and the R image determined to be the same region, the correlation coefficient value from which the influence of the hidden region H is removed is obtained. Can do. With such a configuration, highly accurate matching can be performed based on the correlation coefficient of only the region on the side including the attention point P.

  The image processing apparatus 201 according to the first embodiment of the present invention typically has a basic structure of a computer having a general-purpose architecture, and will be described later by executing a preinstalled program. Provide various functions. In general, such a program is stored in a recording medium such as a flexible disk and a CD-ROM (Compact Disk Read Only Memory) or distributed via a network or the like. When using such a general-purpose computer, an OS (Operating System) for providing basic functions of the computer is installed in addition to the application for providing the functions according to the present embodiment. It may be. In this case, the program according to the present embodiment may execute processing by calling necessary modules out of program modules provided as part of the OS in a predetermined order and / or timing. Good. That is, the program itself according to the present embodiment does not include the module as described above, and the process may be executed in cooperation with the OS. Therefore, the program according to the present embodiment may have a form that does not include the above-described module.

  Furthermore, the program according to the present embodiment may be provided by being incorporated in a part of another program. Even in this case, the program itself according to the present embodiment does not include a module included in the other program as described above, and the processing is executed in cooperation with the other program. That is, the program according to the present embodiment may be in a form incorporated in such another program.

  Alternatively, some or all of the functions provided by program execution may be implemented as a dedicated hardware circuit.

[Device configuration]
FIG. 4 is a schematic configuration diagram of the image processing apparatus according to the first embodiment of the present invention. Referring to FIG. 4, an image processing apparatus 201 includes a central processing unit (CPU) 101 that is an arithmetic processing unit, a main memory 102 and a hard disk 103 as a storage unit, an input interface 104, a display controller 105, data A reader / writer 106 and a communication interface 107 are provided. These units are connected to each other via a bus 121 so that data communication is possible.

  The CPU 101 performs various operations by developing programs (codes) stored in the hard disk 103 in the main memory 102 and executing them in a predetermined order. The main memory 102 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory), and stores data indicating various arithmetic processing results in addition to programs read from the hard disk 103. To do. The hard disk 103 is a non-volatile magnetic storage device, and stores various setting values in addition to programs executed by the CPU 101. The program installed in the hard disk 103 is distributed in a state of being stored in the recording medium 111 as will be described later. In addition to the hard disk 103 or instead of the hard disk 103, a semiconductor storage device such as a flash memory may be employed.

  The input interface 104 mediates data transmission between the CPU 101 and an input unit such as a keyboard 108, a mouse 109, and a touch panel (not shown). That is, the input interface 104 receives an external input such as an operation command given by a user operating the input unit.

The display controller 105 is connected to a display 110 that is a typical example of a display unit, and controls display on the display 110. That is, the display controller 105 displays the result of image processing by the CPU 101 to the user. The display 110 is, for example, an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube).

  The data reader / writer 106 mediates data transmission between the CPU 101 and the recording medium 111. That is, the recording medium 111 is distributed in a state where a program executed by the image processing apparatus 201 is stored, and the data reader / writer 106 reads the program from the recording medium 111. Further, the data reader / writer 106 writes the processing result in the image processing apparatus 201 into the recording medium 111 in response to the internal command of the CPU 101. The recording medium 111 is, for example, a general-purpose semiconductor storage device such as CF (Compact Flash) and SD (Secure Digital), a magnetic storage medium such as a flexible disk, or a CD-ROM (Compact Disk Read Only). Memory).

  The communication interface 107 mediates data transmission between the CPU 101 and a personal computer, a server device, or the like. The communication interface 107 typically has an Ethernet (registered trademark) or USB (Universal Serial Bus) communication function. Instead of installing the program stored in the recording medium 111 in the image processing apparatus 201, a program downloaded from a distribution server or the like via the communication interface 107 may be installed in the image processing apparatus 201.

  Further, the image processing apparatus 201 may be connected to another output device such as a printer as necessary.

[Control structure]
Next, a control structure for providing various functions in the image processing apparatus 201 will be described.

  FIG. 5 is a block diagram showing a control structure provided by the image processing apparatus according to the first embodiment of the present invention. Each block shown in FIG. 5 is provided by developing a program (code) stored in the hard disk 103 in the main memory 102 and causing the CPU 101 to execute it. Note that some or all of the modules shown in FIG. 5 may be provided by firmware implemented in hardware. Alternatively, part or all of the control structure shown in FIG. 5 may be realized by dedicated hardware and / or a wiring circuit.

  Referring to FIG. 5, the image processing apparatus 201 has, as its control structure, an image input unit 10, an image storage unit 11, an image division unit 12, a correlation calculation unit 13, and a correlation coefficient ranking unit 14. , A correlation coefficient rank addition unit 15, a path generation unit 16, a final addition rank selection unit 17, a boundary pixel designation unit 21, and a correlation calculation range determination unit 22.

  The image input unit 10, the image storage unit 11, the image division unit 12, the correlation calculation unit 13, the correlation coefficient ranking unit 14, the correlation coefficient rank addition unit 15, the path generation unit 16, and the final addition rank selection unit 17 are For example, the same operation as that shown in FIG.

  Specifically, the image input unit 10 inputs two images (L image and R image). The image storage unit 11 stores the image input to the image input unit 10. These images are sent, for example, from the recording medium 111 shown in FIG. 4 to the image input unit 10 via the data reader / writer 106, and the image input unit 10 stores these images in the main memory 102 that is the image storage unit 11. Alternatively, it is stored in the hard disk 103.

  The image dividing unit 12 divides the L image and the R image stored in the image storage unit 11 into a plurality of block images while allowing positional overlap. The plurality of block images in each image are provided such that partial areas of adjacent block images overlap each other. For example, as in the configuration shown in FIG. 4 of Patent Document 1, the image dividing unit 12 divides each of the L image and the R image for each scanning line, and a part of the portion corresponding to the divided scanning line overlaps. Then, it is divided into a plurality of block images arranged.

  The boundary pixel designating unit 21 receives input of boundary pixels that are pixels corresponding to the boundary in each of the L image and the R image.

  The correlation calculation range determination unit 22 uses the boundary pixel information obtained by the boundary pixel specification unit 21 to determine ranges (correlation calculation ranges) for performing correlation calculation around the point of interest in both the L image and the R image. decide.

  The correlation calculation unit 13 calculates the cross-correlation of pixel values included in the block image between the block image in the L image and the block image in the R image. Correlation calculation unit 13 is, for example, similar to the configuration shown in FIG. 4 of Patent Document 1, all or a plurality of block images in the L image and a plurality of block images in the R image existing on the scanning line at the same position. For each combination, the correlation coefficient of the pixel value is calculated according to the correlation calculation range.

  The correlation coefficient ranking unit 14 uses the value of the correlation coefficient for each combination of the block image in the L image and the block image in the R image obtained by the correlation calculation unit 13, and relates to these combinations. Ranking is performed in descending order of the correlation coefficient from the image to the block image in the R image and in descending order of the correlation coefficient from the R image to the L image.

  For each of these combinations, the correlation coefficient rank adding unit 15 ranks the block image in the R image from the block image in the L image obtained by the correlation coefficient ordering unit 140, and the L image from the same block image in the R image. Calculate the sum of ranks for the same block image in.

  The path generation unit 16 generates a plurality of paths between the combinations. For example, the path generation unit 16 generates all candidates for connection between corresponding points as paths, as in the configuration shown in FIG. That is, the path generation unit 16 generates not only the paths between the adjacent combinations but also the paths between the non-adjacent combinations for the above combinations.

  The final addition rank selection unit 17 obtains the best path from the paths generated by the path generation unit 16 by dynamic programming. For example, as in the configuration illustrated in FIG. 4 of Patent Document 1, the final addition rank selection unit 17 performs the correlation for each group of all combinations expressed by connecting the paths generated by the path generation unit 16. The sum of the ranks obtained by the number rank addition unit 15 is added.

  And the last addition order selection part 17 adds a fixed penalty according to the number of block images not included in the combination in each group, for example, similarly to the configuration shown in FIG. 4 of Patent Document 1, The group that minimizes this result is obtained using dynamic programming. That is, the final addition order selection unit 17 obtains the best path by the method shown in paragraphs [0052] to [0053] of Patent Document 1 and FIG.

[Operation]
Next, the operation of the image processing apparatus according to the first embodiment of the present invention will be described with reference to the drawings. In the first embodiment of the present invention, the stereo matching method according to the first embodiment of the present invention is performed by operating the image processing apparatus 201. Therefore, the description of the stereo matching method according to the first embodiment of the present invention is replaced with the operation description of the image processing apparatus 201 below. In the following description, FIG. 5 will be referred to as appropriate.

  FIG. 6 is a flowchart showing an operation procedure when the image processing apparatus according to the first embodiment of the present invention performs the stereo matching process.

  Referring to FIG. 6, first, the image input unit 10 inputs an L image and an R image that are targets of stereo processing (step S2).

  Next, the image storage unit 11 stores these L and R images (step S4). Here, the L image and the R image are parallelized images in which images are arranged in epipolar lines. The search for corresponding points in the subsequent processing is to obtain the correspondence between the two images as a whole by repeating the process of obtaining the corresponding points for each same row of the L image and the R image.

  Next, the image dividing unit 12 divides each of the L image and the R image into a plurality of block images. Specifically, the image dividing unit 12 divides each row of the L image and the R image acquired from the image storage unit 11 into a plurality of block images having an overlapping size of M × N. For example, when the corresponding points are obtained for each pixel, the block images are arranged so as to overlap each other with a shift of one pixel (step S6).

  Next, the boundary pixel specifying unit 21 acquires the L image and the R image from the image storage unit 11. Then, the boundary pixel specifying unit 21 is a pixel corresponding to the boundary between the hidden area that is an area that is not captured in either the L image or the R image due to the subject and the non-hidden area that is an area that is captured in both. Are designated in the L image and the R image, respectively. In the first embodiment of the present invention, the boundary pixel specifying unit 21 receives the specification of each boundary pixel by a user operation. Specifically, the boundary pixel designating unit 21 displays both images side by side in the image storage unit 11 shown in FIG. 4 as shown in FIG. 1, for example, and the user places the mouse 109 or the like at the corresponding location of both images. A boundary pixel is designated by accepting an operation for giving a line segment using it (step S8).

  Here, the boundary line is a boundary portion between the hidden area and the non-hidden area that is the target of attention, and is drawn inside the target of interest. As a result, a set of a plurality of boundary points, i.e., (i, L1)-(i, R1), (i, L2)-(i, R2),. A set of n boundary pixels, i, Ln)-(i, Rn) is given.

  Next, the correlation calculation range determination unit 22 determines the correlation calculation range based on each boundary pixel in the L image and the R image. Specifically, the correlation calculation range determination unit 22 determines a correlation calculation range for obtaining a correlation coefficient between the block images of the L image and the R image. Here, since the block image has a size of M × N as described above, when neither of the left and right block images includes the boundary pixel specified by the boundary pixel specifying unit 21, M × N The rectangular image is the correlation calculation range (step S10).

  Next, the correlation calculation unit 13 determines the correlation calculation range determined by the correlation calculation range determination unit 22 for all or some combinations between the plurality of block images in the L image and the plurality of block images in the R image. The correlation coefficient of the pixel value is calculated according to (Step S12).

Next, the correlation coefficient ranking unit 14 uses the correlation coefficient value calculated for each combination calculated by the correlation calculation unit 13 to correlate the block image in the L image to the block image in the R image with respect to the combination. The first ranking is performed in the descending order of the number, and the second ranking is performed in the descending order of the correlation coefficient from the block image in the R image to the block image in the L image with respect to the combination (step S14).

  Next, the correlation coefficient rank adding unit 15 calculates the sum of the first rank and the second rank for each combination (step S16).

  Next, the path generation unit 16 generates a plurality of paths between the combinations (step S18).

  Next, the final addition rank selection unit 17 adds the sum calculated by the correlation coefficient rank addition unit 15 for each group of combinations expressed by connecting a plurality of paths, and the addition result is minimized. Is determined using dynamic programming (step S20). From the group obtained by the final addition rank selection unit 17, the pixel position that becomes the corresponding point is obtained (step S22).

  FIG. 7 is a diagram illustrating an example of the correlation calculation range specified by the correlation calculation range determination unit according to the first embodiment of the present invention. FIG. 7 corresponds to FIG.

  Referring to FIG. 7, when the boundary pixel specified by the boundary pixel specifying unit 21 is included in the block image, the correlation calculation range determination unit 22 correlates only the area inside the boundary line including the point of interest P. The calculation range. Here, points on the boundary, that is, boundary pixels are included in the correlation calculation range.

  The correlation calculation range determination unit 22 sends the obtained correlation calculation range to the correlation calculation unit 13. The correlation calculation unit 13 obtains a correlation coefficient between the attention points of the L image and the R image, that is, the corresponding pixels, according to the correlation calculation range received from the correlation calculation range determination unit 22.

  FIG. 8 is a diagram illustrating a method for obtaining a correlation matrix using a block image. FIG. 8 shows an image of the correlation matrix.

  Referring to FIG. 8, the value of (i, j) in the correlation matrix indicates the degree of correlation between the i-th pixel in a certain row of the L image and the j-th pixel in the corresponding row of the R image. Show.

  The result obtained by the correlation calculation unit 13 is equivalent to obtaining a correlation matrix in which each position in the L image and the R image is represented by rows and columns and the correlation coefficient is a value.

  By the way, in the technique described in Patent Document 2, it is necessary to perform the corresponding pixel search process for a plurality of images, and there is a problem that the processing amount increases. In addition, when occlusion occurs, only the process of assigning distance information of the nearest pixel is performed, so there is a problem that the matching rate around the hidden region cannot be improved in stereo matching. . A technique for solving these problems and performing highly accurate matching beyond the technique described in Patent Document 1 is desired.

On the other hand, in the first embodiment of the present invention, the boundary pixel designating unit 21 is reflected in both the hidden area that is not reflected in either the L image or the R image due to the influence of the subject. A boundary pixel that is a pixel corresponding to a boundary with a non-hidden region that is a region that is present is designated in each of the L image and the R image. The correlation calculation range determination unit 22 determines the correlation calculation range based on each boundary pixel in the L image and the R image. The correlation calculating unit 13 uses a plurality of block images obtained by the image dividing unit 12 dividing each of the L image and the R image, and uses a plurality of block images in the L image and a plurality of block images in the R image. The correlation coefficient of the pixel value is calculated in accordance with the correlation calculation range for all or a part of the combinations between the two.

  As described above, in the first embodiment of the present invention, when there is a hidden region around the corresponding point, the calculation range of the correlation calculation is deformed by designating the boundary between the hidden region and the non-hidden region. As a result, it is possible to omit the correlation calculation for a portion that is a hidden region and does not clearly correspond to the pixel of the other image, thereby reducing the processing amount and improving the accuracy of the corresponding point search by the correlation calculation. .

  That is, among the components in the image processing apparatus according to the first embodiment of the present invention, the image dividing unit 12, the correlation calculating unit 13, the correlation coefficient ranking unit 14, the correlation coefficient rank adding unit 15, the path Even if there is a hidden region in the stereo matching method for searching for corresponding points by template matching, with the minimum configuration including the generation unit 16, the final addition rank selection unit 17, the boundary pixel specification unit 21, and the correlation calculation range determination unit 22. It is possible to achieve the object of the present invention to perform highly accurate matching with simple processing.

  In the first embodiment of the present invention, a pixel that is in contact with a boundary line between a hidden region and a non-hidden region and a pixel in the non-hidden region is designated as a boundary pixel, and the boundary pixel is included in the correlation calculation range. Although it is the configuration, it is not limited to this. A pixel that is in contact with the boundary line between the hidden region and the non-hidden region, and a pixel in the hidden region is designated as a boundary pixel, and the boundary pixel may not be included in the correlation calculation range. Further, the number of boundary pixels in each block image may be one or plural.

  Further, in the first embodiment of the present invention, in the flowchart shown in FIG. 6, the operation of step S8 is performed after the operation of step S6. However, the present invention is not limited to this. A configuration in which the operation of step S6 is performed after the above operation may be employed.

  Next, another embodiment of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals and description thereof will not be repeated.

<Second Embodiment>
The present embodiment relates to an image processing apparatus in which the boundary designation method is changed as compared with the image processing apparatus according to the first embodiment. The contents other than those described below are the same as those of the image processing apparatus according to the first embodiment.

  FIG. 9 is a block diagram showing a control structure provided by the image processing apparatus according to the second embodiment of the present invention. FIG. 10 is a flowchart showing an operation procedure when the image processing apparatus according to the second embodiment of the present invention performs the stereo matching process.

  Each block shown in FIG. 9 is provided by developing a program (code) stored in the hard disk 103 in the main memory 102 and causing the CPU 101 to execute it. Note that some or all of the modules shown in FIG. 9 may be provided by firmware implemented in hardware. Alternatively, part or all of the control structure shown in FIG. 9 may be realized by dedicated hardware and / or a wiring circuit.

Compared with the image processing apparatus 201 according to the first embodiment of the present invention, the image processing apparatus 202 has a control structure in which a boundary pixel detection unit 23 and a boundary pixel association unit 24 are used instead of the boundary pixel specifying unit 21. Is provided.

  The boundary pixel detection unit 23 detects boundary pixel candidates from the L image and the R image, respectively. That is, the boundary pixel detection unit 23 refers to the L image and the R image, and extracts a line segment that is determined to be a clear boundary line from each image (step S7).

  The boundary pixel association unit 24 specifies boundary pixels in the L image and the R image by associating the boundary pixel candidates in the L image with the boundary pixel candidates in the R image. In other words, the boundary pixel association unit 24 performs association between the boundary lines extracted in each of the L image and the R image (step S9).

  More specifically, the boundary pixel detection unit 23 uses, for example, an edge extraction technique that is well known as a two-dimensional image processing technique. That is, the boundary pixel detection unit 23 extracts all line segments having a certain level of strength by using the edge strength, and performs expansion and contraction processing on the extracted line segments to remove small noises. Thereafter, a line segment indicating a boundary is obtained by performing a thinning process. At this stage, the L image and the R image are processed independently, and line segments that are candidates for the boundary line are obtained from the respective images. The boundary pixel detection unit 23 sends the obtained line segment to the boundary pixel association unit 24 together with the L image and the R image.

  The boundary pixel associating unit 24 receives the line segment that is a candidate for the boundary line from the boundary pixel detection unit 23, and the L image and the R image, and for the line segment on the same scanning line, Based on the length and the degree of similarity around the line segment, the L image and the R image are associated with each other.

  The boundary pixel associating unit 24 determines the correlation calculation range in the same manner as the boundary pixel designating unit 21 according to the first embodiment of the present invention. Send to part 22. This correspondence is used to determine the range in which the boundary pixel detection unit 23 performs the correlation calculation.

  Since other configurations and operations are the same as those of the image processing apparatus according to the first embodiment, detailed description thereof will not be repeated here.

  As described above, in the second embodiment of the present invention, the boundary pixel detecting unit 23 and the boundary pixel associating unit 24 perform the same association as the boundary pixel specifying unit 21 in the first embodiment of the present invention. Can be performed. With such a configuration, it is possible to change the correlation calculation range by autonomously specifying the boundary without explicitly specifying the boundary pixel by the user or the like. As a result, the correlation calculation for the hidden region that does not clearly correspond to the other pixel can be omitted, so that the processing amount can be reduced and the accuracy of the corresponding point selection by the correlation calculation can be improved.

  In the second embodiment of the present invention, in the flowchart shown in FIG. 10, the operation of step S7 and step S9 is performed after the operation of step S6. However, the present invention is not limited to this. A configuration in which the operation of step S6 is performed after the operations of step S7 and step S9 may be adopted.

  The above embodiment should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  According to the present invention, in a stereo matching method for searching for corresponding points by template matching, high-precision matching can be performed with simple processing even when a hidden region exists. Therefore, the present invention can be applied to an application for measuring the height of land from an aerial photograph, for example, by stereo matching processing, and can also be applied to an application for perceiving depth by the visual function of a robot. Has industrial applicability.

  Although a part or all of the above embodiments can be described as the following supplementary notes, the scope of the present invention is not limited to the following supplementary notes.

[Appendix 1]
(A) dividing each of the first image and the second image into a plurality of block images; and (b) not appearing in either the first image or the second image due to the influence of the subject. A step of designating boundary pixels, which are one or a plurality of pixels, corresponding to the boundary between a hidden area that is an area and a non-hidden area that is reflected in both areas in the first image and the second image, respectively. When,
(C) determining a correlation calculation range based on each of the boundary pixels in the first image and the second image;
(D) a correlation coefficient of pixel values according to the correlation calculation range for every or some combination between the plurality of block images in the first image and the plurality of block images in the second image Calculating steps,
(E) Using the value of the correlation coefficient for each combination calculated in step (d), the phase from the block image in the first image to the block image in the second image for the combination The first ranking is performed in descending order of the number of relations, and the second ranking is performed in descending order of the correlation coefficient from the block image in the second image to the block image in the first image with respect to the combination. Performing steps,
(F) calculating the sum of the first rank and the second rank for each combination;
(G) generating a plurality of paths between each said combination;
(H) For each group of the combinations expressed by connecting the plurality of paths, the sum calculated in the step (f) is added, and the group that minimizes the result of the addition is dynamically added. An image processing method including a step of obtaining using a planning method.

[Appendix 2]
The image processing method according to appendix 1, wherein in the step (b), designation of each boundary pixel by a user operation is accepted.

[Appendix 3]
The step (b)
Detecting the boundary pixel candidates from the first image and the second image, respectively.
Designating the boundary pixels in the first image and the second image by associating the boundary pixel candidates in the first image with the boundary pixel candidates in the second image, respectively. The image processing method according to claim 1, further comprising:

[Appendix 4]
(A) dividing each of the first image and the second image into a plurality of block images; and (b) not appearing in either the first image or the second image due to the influence of the subject. A step of designating boundary pixels, which are one or a plurality of pixels, corresponding to the boundary between a hidden area that is an area and a non-hidden area that is reflected in both areas in the first image and the second image, respectively. When,
(C) determining a correlation calculation range based on each of the boundary pixels in the first image and the second image;
(D) a correlation coefficient of pixel values according to the correlation calculation range for every or some combination between the plurality of block images in the first image and the plurality of block images in the second image Calculating steps,
(E) Using the value of the correlation coefficient for each combination calculated in step (d), the phase from the block image in the first image to the block image in the second image for the combination The first ranking is performed in descending order of the number of relations, and the second ranking is performed in descending order of the correlation coefficient from the block image in the second image to the block image in the first image with respect to the combination. Performing steps,
(F) calculating the sum of the first rank and the second rank for each combination;
(G) generating a plurality of paths between each said combination;
(H) For each group of the combinations expressed by connecting the plurality of paths, the sum calculated in the step (f) is added, and the group that minimizes the result of the addition is dynamically added. An image processing program for causing a computer to execute a step obtained using a planning method.

[Appendix 5]
The image processing program according to appendix 4, wherein in the step (b), designation of each boundary pixel by a user operation is accepted.

[Appendix 6]
The step (b)
Detecting the boundary pixel candidates from the first image and the second image, respectively.
Designating the boundary pixels in the first image and the second image by associating the boundary pixel candidates in the first image with the boundary pixel candidates in the second image, respectively. The image processing program according to appendix 4, including:

DESCRIPTION OF SYMBOLS 10 Image input part 11 Image memory | storage part 12 Image division part 13 Correlation calculation part 14 Correlation coefficient ranking part 15 Correlation coefficient order | rank addition part 16 Path generation part 17 Final addition order | rank selection part 21 Boundary pixel designation | designated part 22 Correlation calculation range determination Unit 22 boundary pixel detection unit 23 boundary pixel association unit 101 CPU
DESCRIPTION OF SYMBOLS 102 Main memory 103 Hard disk 104 Input interface 105 Display controller 106 Data reader / writer 107 Communication interface 108 Keyboard 109 Mouse 110 Display 111 Recording medium 121 Bus 201 Image processing apparatus

Claims (9)

An image dividing unit for dividing each of the first image and the second image into a plurality of block images;
One or more corresponding to a boundary between a hidden area that is an area that is not shown in either the first image or the second image due to the influence of the subject and a non-hidden area that is an area that is shown in both A boundary pixel designating unit for designating a boundary pixel that is a pixel in each of the first image and the second image;
A correlation calculation range determination unit for determining a correlation calculation range based on each of the boundary pixels in the first image and the second image;
A correlation coefficient of a pixel value is calculated according to the correlation calculation range for every or some combination between the plurality of block images in the first image and the plurality of block images in the second image. A correlation calculation unit for
Using the correlation coefficient value for each combination calculated by the correlation calculation unit, the correlation coefficient from the block image in the first image to the block image in the second image with respect to the combination in descending order. And the second order in the descending order of the correlation coefficient from the block image in the second image to the block image in the first image with respect to the combination. A number ranking section;
A correlation coefficient rank adding unit for calculating a sum of the first rank and the second rank for each combination;
A path generator for generating a plurality of paths between each of the combinations;
For each group of the combinations expressed by connecting the plurality of paths, the sum calculated by the correlation coefficient rank adding unit is added, and the group that minimizes the result of the addition is determined by dynamic programming. An image processing apparatus comprising: a final addition rank selection unit for obtaining using   The image processing apparatus according to claim 1, wherein the boundary pixel specifying unit receives specification of each boundary pixel by a user operation. The boundary pixel designating part is
A boundary pixel detector for detecting the boundary pixel candidates from the first image and the second image, respectively.
By specifying the boundary pixels in the first image and the second image by associating the boundary pixel candidates in the first image with the boundary pixel candidates in the second image, respectively. The image processing apparatus according to claim 1, further comprising a boundary pixel associating unit. (A) dividing each of the first image and the second image into a plurality of block images; and (b) not appearing in either the first image or the second image due to the influence of the subject. A step of designating boundary pixels, which are one or a plurality of pixels, corresponding to the boundary between a hidden area that is an area and a non-hidden area that is reflected in both areas in the first image and the second image, respectively. When,
(C) determining a correlation calculation range based on each of the boundary pixels in the first image and the second image;
(D) a correlation coefficient of pixel values according to the correlation calculation range for every or some combination between the plurality of block images in the first image and the plurality of block images in the second image Calculating steps,
(E) Using the value of the correlation coefficient for each combination calculated in step (d), the phase from the block image in the first image to the block image in the second image for the combination The first ranking is performed in descending order of the number of relations, and the second ranking is performed in descending order of the correlation coefficient from the block image in the second image to the block image in the first image with respect to the combination. Performing steps,
(F) calculating the sum of the first rank and the second rank for each combination;
(G) generating a plurality of paths between each said combination;
(H) For each group of the combinations expressed by connecting the plurality of paths, the sum calculated in the step (f) is added, and the group that minimizes the result of the addition is dynamically added. An image processing method including a step of obtaining using a planning method.   The image processing method according to claim 4, wherein in the step (b), designation of each boundary pixel by a user operation is received. The step (b)
Detecting the boundary pixel candidates from the first image and the second image, respectively.
Designating the boundary pixels in the first image and the second image by associating the boundary pixel candidates in the first image with the boundary pixel candidates in the second image, respectively. The image processing method of Claim 4 containing these. (A) dividing each of the first image and the second image into a plurality of block images; and (b) not appearing in either the first image or the second image due to the influence of the subject. A step of designating boundary pixels, which are one or a plurality of pixels, corresponding to the boundary between a hidden area that is an area and a non-hidden area that is reflected in both areas in the first image and the second image, respectively. When,
(C) determining a correlation calculation range based on each of the boundary pixels in the first image and the second image;
(D) a correlation coefficient of pixel values according to the correlation calculation range for every or some combination between the plurality of block images in the first image and the plurality of block images in the second image Calculating steps,
(E) Using the value of the correlation coefficient for each combination calculated in step (d), the phase from the block image in the first image to the block image in the second image for the combination The first ranking is performed in descending order of the number of relations, and the second ranking is performed in descending order of the correlation coefficient from the block image in the second image to the block image in the first image with respect to the combination. Performing steps,
(F) calculating the sum of the first rank and the second rank for each combination;
(G) generating a plurality of paths between each said combination;
(H) For each group of the combinations expressed by connecting the plurality of paths, the sum calculated in the step (f) is added, and the group that minimizes the result of the addition is dynamically added. An image processing program for causing a computer to execute a step obtained using a planning method.   The image processing program according to claim 7, wherein in the step (b), designation of each boundary pixel by a user operation is accepted. The step (b)
Detecting the boundary pixel candidates from the first image and the second image, respectively.
Designating the boundary pixels in the first image and the second image by associating the boundary pixel candidates in the first image with the boundary pixel candidates in the second image, respectively. The image processing program according to claim 7, comprising:

Images