WO2017130546A1

WO2017130546A1 - Image processing device, image processing method, and computer-readable recording medium

Info

Publication number: WO2017130546A1
Application number: PCT/JP2016/084832
Authority: WO
Inventors: 喜宏山下
Original assignee: Ｎｅｃソリューションイノベータ株式会社
Priority date: 2016-01-29
Filing date: 2016-11-24
Publication date: 2017-08-03
Also published as: JP6601883B2; JPWO2017130546A1

Abstract

The image processing device 100 is equipped with: an image division unit 10 for setting, upon dividing a pair of images into image blocks, image block pairs each comprising a combination of an image block of one of the images and an image block of the other image; a correlation value calculation unit 20 for calculating a correlation value for each of the pairs; a coefficient setting unit 30 for setting, upon constructing a pair set comprising several selected pairs, a first coefficient (upper limit = 1) for each of the pairs such that the value of the coefficient becomes smaller as the distance between the pair and another selected pair increases; an integrated correlation value calculation unit 40 for calculating an integrated correlation value for each of the constituent pairs of the pair set by integrating a multiplication value obtained by multiplying the correlation value by the first coefficient; and an image block determination unit 50 for determining that the image blocks of the respective constituent pairs of the pair set having the smallest integrated correlation value match each other.

Description

Image processing apparatus, image processing method, and computer-readable recording medium

The present invention relates to an image processing apparatus and an image processing method for performing image processing using two images, and a computer-readable recording medium on which a program for realizing these is recorded.

Conventionally, as a method for automatically generating three-dimensional data, stereo matching processing is performed on the basis of pair images obtained from an artificial satellite or an aircraft, etc., and three-dimensional data [DSM (Digital Surface Model) data showing terrain] The method of generating is widely known. A pair image is an image obtained by photographing the same subject from different viewpoints with two cameras.

Specifically, in the stereo matching process, first, each image constituting the stereo image is divided into image blocks of a certain size, and then the image block of the other image that matches the image block of one image is specified. . Specifically, a correlation value between image blocks is calculated using a sum of luminance differences (SAD: SumSof Absolute Difference) or a square sum of luminance differences (SSD: Sum of Squared Difference), and the calculated correlation By using the value, a combination of matching image blocks is specified. Thereafter, the corresponding points are identified between the matching image blocks, and the depth and shape of the subject are obtained by performing triangulation using the parallax between the identified corresponding points.

By the way, when the pair image is an aerial image, a partial area of the subject may be visible in one image, but may be hidden in the other image. This hidden and invisible area is generally called an “occlusion area”. And when such an occlusion area | region has generate | occur | produced, the matching precision by a stereo matching process will fall.

For this reason, for example, Patent Document 1 discloses a stereo matching process that suppresses a decrease in matching accuracy using a correlation value calculated for each combination of image blocks. Specifically, in the stereo matching process disclosed in Patent Document 1, the following steps (a) to (g) are executed.

(A) A step of dividing one image A and the other image B constituting the pair image into image blocks.
(B) A step of setting a combination of image blocks and calculating a correlation value for each set combination.
(C) Using the correlation value calculated for each combination of image blocks, the first order for the image block on the A image in descending order of the correlation coefficient from the image block to the image block on the B image And assigning the second rank to the image blocks on the B image in descending order of the correlation coefficient from the image block to the image block on the A image.
(D) A step of calculating the sum of the first rank and the second rank for each combination of image blocks.
(E) A combination of image blocks is selected so that there are not a plurality of combinations including the same image block, and a penalty is given for each selected combination according to the distance between the combination and another selected combination adjacent to the combination. Process.
(F) A step of integrating the total and penalty of each selected combination.
(G) A step of identifying a combination of image blocks having the smallest integrated value and determining that the image blocks match in the identified combination of image blocks.

According to the stereo matching processing disclosed in Patent Document 1, a larger penalty is added as the distance between image blocks that are expected to match an image block of another image increases. For this reason, the situation where it is determined that the image block including the occlusion area matches the non-corresponding image block is avoided. According to the stereo matching process disclosed in Patent Document 1, it is considered that a decrease in matching accuracy is suppressed.

Patent No. 3027995

However, in the stereo matching process disclosed in Patent Document 1, the penalty increases as the distance between image blocks that are expected to match the image blocks of another image increases. For this reason, the larger the occlusion area, the fewer sets that are determined to match one image block and the other image block. In such a case, on the contrary, the matching accuracy is lowered.

An example of an object of the present invention is an image processing apparatus and an image processing method capable of solving the above-described problem and suppressing a decrease in matching accuracy even when an occlusion area is increased in stereo matching processing of a pair image. And providing a computer-readable recording medium.

In order to achieve the above object, an image processing apparatus according to one aspect of the present invention provides:
Each of the two target images is divided into image blocks of a set size, one of the image blocks obtained from one of the two images, and one of the image blocks obtained from the other image And an image dividing unit for setting a set of a plurality of image blocks,
A correlation value calculation unit for calculating a correlation value for each set of set image blocks;
A set of image blocks is selected from the set of the plurality of set image blocks so that a plurality of sets including the same image block do not exist, and a set set is constructed. Each time, the distance between the set and another selected set including an image block adjacent to the image block of the set in the setting direction is obtained, and the larger the obtained distance, the smaller the value, and the upper limit value. A coefficient setting unit that sets a first coefficient that becomes 1;
In each of the set sets, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient to obtain a multiplied value, and the obtained multiplied value is further accumulated, An integrated correlation value calculating unit for calculating an integrated correlation value;
The set set having the smallest integrated correlation value is specified, and in each set of image blocks constituting the specified set set, one image block and the other image block constituting the set coincide with each other. An image block determination unit that determines that
It is characterized by having.

In order to achieve the above object, an image processing method according to one aspect of the present invention includes:
(A) Each of the two target images is divided into image blocks of a set size, and one of the image blocks obtained from one of the two images and the image block obtained from the other image A set of a plurality of image blocks in combination with one of
(B) calculating a correlation value for each set of set image blocks; and
(C) selecting a set of image blocks from the set of the plurality of set image blocks so that a plurality of sets including the same image block do not exist, constructing a set, and further selecting the selected image For each set of blocks, determine the distance between the set and another selected set that includes an image block adjacent to the set of image blocks in the set direction, and the larger the calculated distance, the smaller the value, and Setting a first coefficient with an upper limit value of 1; and
(D) In each of the set sets, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient to obtain a multiplication value, and the obtained multiplication value is further integrated. And calculating the integrated correlation value,
(E) The set set that minimizes the integrated correlation value is specified, and in each set of image blocks constituting the specified set set, one image block and the other image block constituting the set are A step of determining that they match, and
It is characterized by having.

Furthermore, in order to achieve the above object, a computer-readable recording medium according to one aspect of the present invention is provided.
On the computer,
(A) Each of the two target images is divided into image blocks of a set size, and one of the image blocks obtained from one of the two images and the image block obtained from the other image A set of a plurality of image blocks in combination with one of
(B) calculating a correlation value for each set of set image blocks; and
(C) selecting a set of image blocks from the set of the plurality of set image blocks so that a plurality of sets including the same image block do not exist, constructing a set, and further selecting the selected image For each set of blocks, determine the distance between the set and another selected set that includes an image block adjacent to the set of image blocks in the set direction, and the larger the calculated distance, the smaller the value, and Setting a first coefficient with an upper limit value of 1; and
(D) In each of the set sets, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient to obtain a multiplication value, and the obtained multiplication value is further integrated. And calculating the integrated correlation value,
(E) The set set that minimizes the integrated correlation value is specified, and in each set of image blocks constituting the specified set set, one image block and the other image block constituting the set are A step of determining that they match, and
A program including an instruction for executing is recorded.

As described above, according to the present invention, it is possible to suppress a decrease in matching accuracy even when the occlusion area becomes large in the stereo matching process of a pair image.

FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram showing a specific configuration of the image processing apparatus according to the embodiment of the present invention. FIG. 3 is a diagram showing an example of a pair image divided in the embodiment of the present invention. FIG. 4 is a diagram for explaining an example of a set of image blocks set in the embodiment of the present invention. FIG. 5 is an explanatory diagram for explaining the coefficient setting processing according to the embodiment of the present invention. FIG. 6 is a diagram for explaining the effect of the coefficient setting process according to the embodiment of the present invention. FIG. 7 is a flowchart showing the operation of the image processing apparatus according to the embodiment of the present invention. FIG. 8 is a block diagram illustrating an example of a computer that implements the image processing apparatus according to the embodiment of the present invention.

(Embodiment)
Hereinafter, an image processing apparatus, an image processing method, and a program according to an embodiment of the present invention will be described with reference to FIGS.

[Device structure]
First, the configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to an embodiment of the present invention.

As shown in FIG. 1, the image processing apparatus 100 according to the present embodiment includes an image dividing unit 10, a correlation value calculating unit 20, a coefficient setting unit 30, an integrated correlation value calculating unit 40, and an image block determining unit 50. And.

The image dividing unit 10 divides each two target images into image blocks having a set size. Further, the image dividing unit 10 sets one set of a plurality of image blocks by combining one of the image blocks obtained from one of the two images and one of the image blocks obtained from the other image. .

The correlation value calculation unit 20 calculates a correlation value for each set of set image blocks. Further, the coefficient setting unit 30 selects a set of image blocks from among a set of a plurality of set image blocks so that a plurality of sets including the same image block do not exist, and constructs a set set. Further, for each set of selected image blocks, the coefficient setting unit 30 obtains a distance between the set and another selected set including an image block adjacent to the image block of the set in the setting direction. Then, the coefficient setting unit 30 sets, for each set of selected image blocks, a first coefficient whose value decreases as the obtained distance increases and the upper limit value is 1.

The integrated correlation value calculation unit 40 multiplies the correlation value by the first coefficient for each set of image blocks constituting each set set to obtain a multiplied value, and further integrates the obtained multiplied value. Then, the integrated correlation value is calculated.

The image block determination unit 50 identifies a set set having the smallest integrated correlation value. Further, the image block determination unit 50 determines that in each set of image blocks constituting the specified set set, one image block constituting the set matches the other image block.

As described above, in the present embodiment, the first coefficient is set such that the value decreases as the distance between image blocks that are expected to match the image blocks of another image increases. The upper limit of the first coefficient is 1, and the fluctuation range is suppressed. Then, the integrated correlation value for performing the coincidence determination is obtained by integrating the product of the correlation value and the first coefficient. For this reason, in the present embodiment, when the occlusion area becomes large, it is possible to avoid a situation in which the number of pairs determined to match one image block and the other image block is reduced. Will be suppressed.

Subsequently, the configuration and function of the image processing in the embodiment of the present invention will be described more specifically with reference to FIG. FIG. 2 is a block diagram showing a specific configuration of the image processing apparatus according to the embodiment of the present invention.

First, in the present embodiment, the image processing apparatus 100 is an apparatus that performs stereo matching processing. The two images to be processed are a pair image obtained by photographing the same object from different angles, specifically, a pair image obtained from an artificial satellite or an aircraft.

As shown in FIG. 2, in the present embodiment, the image processing apparatus 100 includes the image dividing unit 10, the correlation value calculating unit 20, the coefficient setting unit 30, the integrated correlation value calculating unit 40, shown in FIG. In addition to the image block determination unit 50, an image acquisition unit 60 is further provided.

The image acquisition unit 60 acquires a pair image to be processed. Specifically, the image acquisition unit 60 acquires each image data of a pair image transmitted via a network or the like, and inputs the acquired image data to the image dividing unit 10. In addition, the image acquisition unit 60 determines whether or not the epipolar lines of the pair images match in the horizontal direction. If they do not match, the image acquisition unit 60 performs at least one of rotation and projective transformation on one or both images. One is performed and correction is performed so that the epipolar lines coincide.

In the present embodiment, as shown in FIG. 3, the image dividing unit 10 forms a pair of images in the horizontal direction and columns in the vertical direction along the vertical direction and the horizontal direction, as shown in FIG. Divide into tiles. FIG. 3 is a diagram showing an example of a pair image divided in the embodiment of the present invention.

Specifically, in the example of FIG. 3, the pair image includes an image 201 and an image 202. As shown in FIG. 3, the image dividing unit 10 divides each of the image 201 and the image 202 into rectangular image blocks having a preset size. As a result, the image blocks are arranged along the horizontal and vertical directions of the image.

Further, in the present embodiment, the image dividing unit 10 selects one of the image blocks obtained from one image and one of the image blocks obtained from the other image so that the rows in which both are located are mutually connected. Select to respond. Then, the image dividing unit 10 combines the selected image blocks, sets a set of image blocks, and inputs the set set of image blocks to the correlation value calculation unit 20.

Specifically, as described above, in the present embodiment, the image acquisition unit 60 matches the epipolar lines of the pair images in the waterside direction. Therefore, as shown in FIG. 4, the image dividing unit 10 sets a set of image blocks by combining image blocks having the same number of rows. FIG. 4 is a diagram for explaining an example of a set of image blocks set in the embodiment of the present invention.

In the example of FIG. 4, the image dividing unit 10 first sets a set of image blocks using the image blocks selected from the first row of each of the image 201 and the image 202. That is, the image dividing unit 10 sets all combinations obtained by all the image blocks in the first row of the image 201 and all the image blocks in the first row of the image 202 as a set of image blocks. For example, the image block 203 is combined with each of the image blocks 205 to 209 in addition to the image block 204.

Subsequently, the image dividing unit 10 similarly sets the image block set by combining the image block of the image 201 and the image block of the image 202 for the second and subsequent rows of the image 201 and the image 202, respectively. Then, the image dividing unit 10 inputs the set of each image block set in this way to the correlation value calculating unit 20.

In the present embodiment, the correlation value calculation unit 20 calculates a correlation value for each set of input image blocks using, for example, an existing calculation method. Existing calculation methods include a calculation method using the sum of luminance differences (SAD: Sum of Absolute Difference), a calculation method using sum of squares of luminance differences (SSD: SumSof Squared Difference), and a normalized cross-correlation ( Examples include a calculation method using NCC: Normalized Cross-Correlation. In the present embodiment, the correlation value calculation method is not particularly limited, and a newly developed calculation method may be employed.

As described above, the coefficient setting unit 30 selects several image block sets from the set of a plurality of set image blocks, and sets the first coefficient for each selected set. Here, the coefficient setting process by the coefficient setting unit 30 will be specifically described with reference to FIG. FIG. 5 is an explanatory diagram for explaining a coefficient setting process according to the embodiment of the present invention. In FIG. 5, each circular figure shows a set of set image blocks, and a filled figure shows a selected set.

Specifically, each set shown in FIG. 5 includes a plurality of image blocks in a specific row of one image of a pair image and a plurality of image blocks in a corresponding row of the other image. . In FIG. 5, “y” indicates a position in a specific row of the image block of one image, and “x” indicates a position in a corresponding row of the image block of the other image. x and y are set so that the left end is “0” in the horizontal direction.

Then, in the example of FIG. 5, for example, if the first coefficient setting target is the set B (4, 8), the coefficient setting unit 30 uses the following formula 1, for example, A distance D to a set A (3, 4) including an image block adjacent to the B image block in the horizontal direction is calculated. In Equation 1, x and y indicate the position of each image block of the set for which the first coefficient is set, and jx and jy indicate the position of the set including adjacent image blocks in the horizontal direction. ing. Α is an arbitrary integer.

The coefficient setting unit 30 can set the first coefficient using, for example, the following formula 2. In the following Equation 2, the first coefficient is expressed as a jump coefficient W _jump . As can be seen from Equation 2 below, the upper limit of W _jump is 1, and the larger D is, the smaller the value of W _jump is. Note that β is an arbitrary integer.

In the present embodiment, the coefficient setting unit 30 also sets a second coefficient in addition to the jump coefficient W _jump . In this case, the coefficient setting unit 30, or the correlation value of the set of image blocks to be set is less than the threshold value T _jump, depending on whether the threshold value T _jump over, to set different second coefficient.

Specifically, when the correlation value is less than the threshold value T _jump , the coefficient setting unit 30 sets the second coefficient using the following _Equation 3. Further, the coefficient setting unit 30 sets the second coefficient using _Equation 4 when the correlation value is equal to or greater than the threshold value T _jump . In the following

equations

3 and 4, the second coefficient is represented as a low cost coefficient W _lowcost . α ₁ and α ₂ are arbitrary integers.

In this embodiment, the integrated correlation value calculation unit 40 multiplies the correlation value by the jump coefficient W _jump and the low cost coefficient W _lowcost for each set of image blocks constituting each set. Find the value. Specifically, if the correlation value of each selected set is R _i and the number of the selected set is T, the integrated correlation value calculation unit 40 calculates the integrated correlation value RA using the following formula 5. can do.

As shown in FIG. 5, in the present embodiment, the distance D is calculated only for a set including an image block adjacent on the left side. Therefore, in FIG. 5, since the distance is not calculated for the pair closest to the origin (lower left), neither the jump coefficient W _jump nor the low cost coefficient W _lowcost is calculated. Therefore, in the calculation of the integrated correlation value RA according to the above equation 6, only the correlation value R ₀ is integrated for the set closest to the origin (lower left).

Thus, in this embodiment, when the correlation value of a set of image blocks is high, that is, when the possibility that both image blocks are the same is low, the value of the jump coefficient W _jump is increased by the low cost coefficient W _lowcost. By determining the coefficient α ₁ of the low cost coefficient W _lowcost so that the correlation value becomes higher as a result, the correlation value is corrected. On the other hand, when the correlation value of the set of image blocks is low, that is, when there is a high possibility that both image blocks are the same, the low cost coefficient W _lowcost is reduced so that the value of the jump coefficient W _jump is reduced by the low cost coefficient W _lowcost . By determining the coefficient α ₂ , the correlation value is corrected so that the correlation value becomes lower as a result. For this reason, generation | occurrence | production of the problem shown in FIG. 6 is suppressed.

FIG. 6 is a diagram for explaining the effect of the coefficient setting process in the embodiment of the present invention. Also in FIG. 6, each circular figure shows a set of set image blocks, and a filled figure shows a selected set. Each set includes a plurality of image blocks in a specific row of one image of the pair image and a plurality of image blocks in a corresponding row of the other image.

In the example of FIG. 6, it is assumed that one image constituting the pair image matches the other image. In such a state, the integrated correlation value is minimum in the set set constructed by sets of image blocks having the same position in the horizontal direction. In other words, as shown in FIG. 6, when a hatched set is selected, the integrated correlation value is minimized.

Incidentally, in the situation of FIG. 6, the correlation value of the hatched group is low, but since the number of selected groups increases, the correlation value of the group a to group i can be obtained as it is. The integrated value may be higher than the correlation value of the set j. Therefore, the integrated correlation value when the set k and the set j are selected may be lower than the integrated correlation value when the set a to the set j are selected. It is determined that the image blocks match in k and set j.

On the other hand, in the present embodiment, when the correlation value of a set of image blocks is high, that is, when it is unlikely that both image blocks are the same, the value of the jump coefficient W _jump is set by the low cost coefficient W _lowcost. growing. For this reason, the situation where the group k and the group j are selected is avoided, and the situation where an erroneous determination is made is avoided.

Further, the coefficient setting unit 30 sets the setting when the parallax between the set of image blocks to be set and another selected set including an image block adjacent to the set of image blocks in the setting direction matches. a set of jump coefficient W _jump of the image block of interest, can be corrected so that the value becomes smaller.

Specifically, in the example of FIG. 6, if the setting target of the jump coefficient W _jump is a set i, the set i is between a set h including image blocks adjacent to the image blocks in the set i in the horizontal direction. The parallax has the same relationship. That is, when the two sets are adjacent to each other at an oblique 45 degree position in the matrix shown in FIG. 6, the two sets of parallax have the same relationship. In such a case, the coefficient setting unit 30 can correct the jump coefficient W _jump using, for example, the following Equation 6. In the following formula 6, b is an arbitrary integer.

For example, it is assumed that a subject of a pair image has a surface with few features such as a rooftop of a building and a road. In such a situation, in FIG. 6, the correlation value of the set j becomes larger than the correlation value of the set l, and there is a possibility that the set l is selected instead of the set j. On the other hand, when _Equation 6 is used, the jump coefficient W _jump of the set j is corrected and becomes small, so that the situation where the set l is selected by mistake is avoided.

[Device operation]
Next, the operation of the image processing apparatus 100 according to the embodiment of the present invention will be described with reference to FIG. FIG. 7 is a flowchart showing the operation of the image processing apparatus according to the embodiment of the present invention. In the following description, FIGS. 1 to 6 are referred to as appropriate. In the present embodiment, the image processing method is performed by operating the image processing apparatus 100. Therefore, the description of the image processing method in the present embodiment is replaced with the following description of the operation of the image processing apparatus 100.

As shown in FIG. 7, in the image processing apparatus 100, first, the image acquisition unit 60 acquires a pair image transmitted via a network or the like (step A1). Furthermore, the image acquisition unit 60 corrects one or both images so that the epipolar lines of the pair images match in the horizontal direction (step A2).

Next, the image dividing unit 10 divides each paired image into tiles so as to form horizontal rows and vertical columns along the vertical and horizontal directions, and generate image blocks. (Step A3).

Next, the image dividing unit 10 sets one set of image blocks by combining one image block obtained from one image and one image block obtained from the other image (step A4). Specifically, the image dividing unit 10 sets a set of image blocks by combining image blocks having the same number of rows.

Next, the correlation value calculation unit 20 calculates a correlation value for each set of input image blocks (step A5). Next, the coefficient setting unit 30 selects a set of image blocks from the set of a plurality of set image blocks so that a plurality of sets including the same image block do not exist, and builds a set set (step A6). Furthermore, the coefficient setting unit 30 sets a jump coefficient W _jump and a low cost coefficient W _lowcost for each selected set in each set set (step A7). The coefficient setting unit 30 corrects the jump coefficient W _jump as necessary.

Next, in each set set, the integrated correlation value calculation unit 40 multiplies the correlation value by the jump coefficient W _jump and the low cost coefficient W _lowcost for each set of image blocks to obtain a multiplication value. Integration is performed to calculate an integrated correlation value (step A8).

Thereafter, the image block determination unit 50 identifies the set set having the smallest integrated correlation value, and one image block and the other image block match in the set of image blocks constituting the specified set set. (Step A9). Thereafter, the image block determination unit 50 outputs a set of image blocks determined to match.

As described above, by executing steps A1 to A9, a set of matching image blocks in the pair image is specified. Thereafter, the image processing apparatus 100 identifies corresponding points between the matching image blocks, and performs triangulation using the parallax between the identified corresponding points. Furthermore, the image processing apparatus 100 identifies the depth and shape of the subject based on the result of triangulation.

[Effects of the embodiment]
As described above, in the present embodiment, the integrated correlation value is calculated using the correlation value multiplied by the jump coefficient W _jump. Therefore, when the occlusion area becomes large, the image block that is determined to match is determined. The situation where the number of pairs is reduced is avoided. Furthermore, in this embodiment, since the correlation value is also multiplied by the low cost coefficient W _lowcost , it is possible to suppress a situation in which it is determined that a set in which image blocks do not match is erroneously matched. According to the present embodiment, a decrease in matching accuracy is suppressed.

[program]
The program in the present embodiment may be a program that causes a computer to execute steps A1 to A9 shown in FIG. By installing and executing this program on a computer, the image processing apparatus 100 and the image processing method in the present embodiment can be realized. In this case, a CPU (Central Processing Unit) of the computer functions as the image dividing unit 10, the correlation value calculating unit 20, the coefficient setting unit 30, the integrated correlation value calculating unit 40, the image block determining unit 50, and the image acquiring unit 60. , Process.

Note that the program in the present embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer is any one of the image dividing unit 10, the correlation value calculating unit 20, the coefficient setting unit 30, the integrated correlation value calculating unit 40, the image block determining unit 50, and the image acquiring unit 60, respectively. May function.

Here, a computer that realizes the image processing apparatus 100 by executing the program according to the present embodiment will be described with reference to FIG. FIG. 8 is a block diagram illustrating an example of a computer that implements the image processing apparatus according to the embodiment of the present invention.

As shown in FIG. 8, the computer 110 includes a CPU 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These units are connected to each other via a bus 121 so that data communication is possible.

The CPU 111 performs various operations by developing the program (code) in the present embodiment stored in the storage device 113 in the main memory 112 and executing them in a predetermined order. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). Further, the program in the present embodiment is provided in a state of being stored in a computer-readable recording medium 120. Note that the program in the present embodiment may be distributed on the Internet connected via the communication interface 117.

Further, specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and an input device 118 such as a keyboard and a mouse. The display controller 115 is connected to the display device 119 and controls display on the display device 119.

The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, and reads a program from the recording medium 120 and writes a processing result in the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and another computer.

Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as a flexible disk, or CD- Optical recording media such as ROM (Compact Disk Read Only Memory) are listed.

Note that the image processing apparatus 100 according to the present embodiment can be realized by using hardware corresponding to each unit, instead of a computer in which a program is installed. Furthermore, part of the image processing apparatus 100 may be realized by a program, and the remaining part may be realized by hardware.

Some or all of the above-described embodiments can be expressed by the following (Appendix 1) to (Appendix 15), but is not limited to the following description.

(Appendix 1)
Each of the two target images is divided into image blocks of a set size, one of the image blocks obtained from one of the two images, and one of the image blocks obtained from the other image And an image dividing unit for setting a set of a plurality of image blocks,
A correlation value calculation unit for calculating a correlation value for each set of set image blocks;
A set of image blocks is selected from the set of the plurality of set image blocks so that a plurality of sets including the same image block do not exist, and a set set is constructed. Each time, the distance between the set and another selected set including an image block adjacent to the image block of the set in the setting direction is obtained, and the larger the obtained distance, the smaller the value, and the upper limit value. A coefficient setting unit that sets a first coefficient that becomes 1;
In each of the set sets, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient to obtain a multiplied value, and the obtained multiplied value is further accumulated, An integrated correlation value calculating unit for calculating an integrated correlation value;
The set set having the smallest integrated correlation value is specified, and in each set of image blocks constituting the specified set set, one image block and the other image block constituting the set coincide with each other. An image block determination unit that determines that
An image processing apparatus comprising:

(Appendix 2)
The coefficient setting unit further sets a second coefficient that increases the first coefficient when the correlation value of the set is less than a threshold, and when the correlation value of the set is equal to or greater than the threshold, Setting a second coefficient to reduce the first coefficient;
For each set set, the integrated correlation value calculation unit multiplies the correlation value by the first coefficient and the second coefficient for each set of image blocks constituting the set set, and obtains a multiplication value. Ask,
The image processing apparatus according to appendix 1.

(Appendix 3)
When the parallax of the set and another selected set including an image block adjacent to the image block of the set in the setting direction matches the set, the coefficient setting unit sets the first coefficient of the set To reduce the
The image processing apparatus according to appendix 1 or 2.

(Appendix 4)
The image processing apparatus according to any one of appendices 1 to 3, wherein the two images are pair images obtained by photographing the same object from different angles.

(Appendix 5)
When the two images are acquired and the epipolar lines of the acquired two images do not match in the horizontal direction, at least one of rotation and projective transformation is performed on one or both of the two images. And further comprising an image acquisition unit that matches the epipolar lines of the two images in the horizontal direction,
The image dividing unit divides each of the two images in a tile shape so that rows in the horizontal direction and columns in the vertical direction are formed along the vertical direction and the horizontal direction.
One of the image blocks obtained from the one image and one of the image blocks obtained from the other image are selected so that the rows in which they are located correspond to each other, and the selected image blocks To set the set of image blocks,
The image processing apparatus according to any one of appendices 1 to 4.

(Appendix 6)
(A) Each of the two target images is divided into image blocks of a set size, and one of the image blocks obtained from one of the two images and the image block obtained from the other image A set of a plurality of image blocks, and (b) calculating a correlation value for each set of set image blocks; and
(C) selecting a set of image blocks from the set of the plurality of set image blocks so that a plurality of sets including the same image block do not exist, constructing a set, and further selecting the selected image For each set of blocks, determine the distance between the set and another selected set that includes an image block adjacent to the set of image blocks in the set direction, and the larger the calculated distance, the smaller the value, and Setting a first coefficient with an upper limit value of 1; and
(D) In each of the set sets, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient to obtain a multiplication value, and the obtained multiplication value is further integrated. And calculating the integrated correlation value,
(E) The set set that minimizes the integrated correlation value is specified, and in each set of image blocks constituting the specified set set, one image block and the other image block constituting the set are A step of determining that they match, and
An image processing method characterized by comprising:

(Appendix 7)
In the step (c), when the correlation value of the set is less than the threshold, a second coefficient that increases the first coefficient is set, and the correlation value of the set is equal to or greater than the threshold And setting a second coefficient to reduce the first coefficient,
In the step (d), for each set set, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient and the second coefficient to obtain a multiplication value. Ask,
The image processing method according to attachment 6.

(Appendix 8)
In the step (c), when the parallax of the set and another selected set including an image block adjacent to the image block of the set in the setting direction matches, the first coefficient of the set To correct the value to be smaller,
The image processing method according to appendix 6 or 7.

(Appendix 9)
The image processing method according to any one of appendices 6 to 8, wherein the two images are pair images obtained by photographing the same object from different angles.

(Appendix 10)
(F) When the two images are acquired and one or both of the two images are not rotated or projected when the epipolar lines of the acquired two images do not match in the horizontal direction, Performing at least one to match the epipolar lines of each of the two images in the horizontal direction;
In the step (a), each of the two images is divided into tiles so that rows in the horizontal direction and columns in the vertical direction are formed along the vertical direction and the horizontal direction.
One of the image blocks obtained from the one image and one of the image blocks obtained from the other image are selected so that the rows in which they are located correspond to each other, and the selected image blocks To set the set of image blocks,
The image processing method according to any one of appendices 6 to 9.

(Appendix 11)
On the computer,
(A) Each of the two target images is divided into image blocks of a set size, and one of the image blocks obtained from one of the two images and the image block obtained from the other image A set of a plurality of image blocks, and (b) calculating a correlation value for each set of set image blocks; and
(C) selecting a set of image blocks from the set of the plurality of set image blocks so that a plurality of sets including the same image block do not exist, constructing a set, and further selecting the selected image For each set of blocks, determine the distance between the set and another selected set that includes an image block adjacent to the set of image blocks in the set direction, and the larger the calculated distance, the smaller the value, and Setting a first coefficient with an upper limit value of 1; and
(D) In each of the set sets, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient to obtain a multiplication value, and the obtained multiplication value is further integrated. And calculating the integrated correlation value,
(E) The set set that minimizes the integrated correlation value is specified, and in each set of image blocks constituting the specified set set, one image block and the other image block constituting the set are A step of determining that they match, and
The computer-readable recording medium which recorded the program containing the instruction | indication which performs this.

(Appendix 12)
In the step (c), when the correlation value of the set is less than the threshold, a second coefficient that increases the first coefficient is set, and the correlation value of the set is equal to or greater than the threshold And setting a second coefficient to reduce the first coefficient,
In the step (d), for each set set, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient and the second coefficient to obtain a multiplication value. Ask,
The computer-readable recording medium according to appendix 11.

(Appendix 13)
In the step (c), when the parallax of the set and another selected set including an image block adjacent to the image block of the set in the setting direction matches, the first coefficient of the set To correct the value to be smaller,
The computer-readable recording medium according to appendix 11 or 12.

(Appendix 14)
14. The computer-readable recording medium according to any one of appendices 11 to 13, wherein the two images are pair images obtained by photographing the same object from different angles.

(Appendix 15)
The program is stored in the computer.
(F) When the two images are acquired and one or both of the two images are not rotated or projected when the epipolar lines of the acquired two images do not match in the horizontal direction, Further comprising: performing at least one to match the epipolar lines of each of the two images in the horizontal direction;
In the step (a), each of the two images is divided into tiles so that rows in the horizontal direction and columns in the vertical direction are formed along the vertical direction and the horizontal direction.
One of the image blocks obtained from the one image and one of the image blocks obtained from the other image are selected so that the rows in which they are located correspond to each other, and the selected image blocks To set the set of image blocks,
15. A computer-readable recording medium according to any one of appendices 11 to 14.

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2016-016522 filed on January 29, 2016, the entire disclosure of which is incorporated herein.

As described above, according to the present invention, it is possible to suppress a decrease in matching accuracy even when the occlusion area becomes large in the stereo matching process of a pair image. The present invention is useful, for example, in a field that requires stereo matching.

DESCRIPTION OF SYMBOLS 10 Image division part 20 Correlation value calculation part 30 Coefficient setting part 40 Integrated correlation value calculation part 50 Image block determination part 60 Image acquisition part 100 Image processing apparatus 110 Computer 111 CPU
112 Main Memory 113 Storage Device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Device 119 Display Device 120 Recording Medium 121 Bus

Claims

Each of the two target images is divided into image blocks of a set size, one of the image blocks obtained from one of the two images, and one of the image blocks obtained from the other image And an image dividing unit for setting a set of a plurality of image blocks,
A correlation value calculation unit for calculating a correlation value for each set of set image blocks;
A set of image blocks is selected from the set of the plurality of set image blocks so that a plurality of sets including the same image block do not exist, and a set set is constructed. Each time, the distance between the set and another selected set including an image block adjacent to the image block of the set in the setting direction is obtained, and the larger the obtained distance, the smaller the value, and the upper limit value. A coefficient setting unit that sets a first coefficient that becomes 1;
In each of the set sets, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient to obtain a multiplied value, and the obtained multiplied value is further accumulated, An integrated correlation value calculating unit for calculating an integrated correlation value;
The set set having the smallest integrated correlation value is specified, and in each set of image blocks constituting the specified set set, one image block and the other image block constituting the set coincide with each other. An image block determination unit that determines that
An image processing apparatus comprising:
The coefficient setting unit further sets a second coefficient that increases the first coefficient when the correlation value of the set is less than a threshold, and when the correlation value of the set is equal to or greater than the threshold, Setting a second coefficient to reduce the first coefficient;
For each set set, the integrated correlation value calculation unit multiplies the correlation value by the first coefficient and the second coefficient for each set of image blocks constituting the set set, and obtains a multiplication value. Ask,
The image processing apparatus according to claim 1.
When the parallax of the set and another selected set including an image block adjacent to the image block of the set in the setting direction matches the set, the coefficient setting unit sets the first coefficient of the set To reduce the
The image processing apparatus according to claim 1.
The image processing apparatus according to any one of claims 1 to 3, wherein the two images are pair images obtained by photographing the same object from different angles.
When the two images are acquired and the epipolar lines of the acquired two images do not match in the horizontal direction, at least one of rotation and projective transformation is performed on one or both of the two images. And further comprising an image acquisition unit that matches the epipolar lines of the two images in the horizontal direction,
The image dividing unit divides each of the two images in a tile shape so that rows in the horizontal direction and columns in the vertical direction are formed along the vertical direction and the horizontal direction.
One of the image blocks obtained from the one image and one of the image blocks obtained from the other image are selected so that the rows in which they are located correspond to each other, and the selected image blocks To set the set of image blocks,
The image processing apparatus according to any one of claims 1 to 4.
(A) Each of the two target images is divided into image blocks of a set size, and one of the image blocks obtained from one of the two images and the image block obtained from the other image A set of a plurality of image blocks, and (b) calculating a correlation value for each set of set image blocks; and
(C) selecting a set of image blocks from the set of the plurality of set image blocks so that a plurality of sets including the same image block do not exist, constructing a set, and further selecting the selected image For each set of blocks, determine the distance between the set and another selected set that includes an image block adjacent to the set of image blocks in the set direction, and the larger the calculated distance, the smaller the value, and Setting a first coefficient with an upper limit value of 1; and
(D) In each of the set sets, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient to obtain a multiplication value, and the obtained multiplication value is further integrated. And calculating the integrated correlation value,
(E) The set set that minimizes the integrated correlation value is specified, and in each set of image blocks constituting the specified set set, one image block and the other image block constituting the set are A step of determining that they match, and
An image processing method characterized by comprising:
In the step (c), when the correlation value of the set is less than the threshold, a second coefficient that increases the first coefficient is set, and the correlation value of the set is equal to or greater than the threshold And setting a second coefficient to reduce the first coefficient,
In the step (d), for each set set, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient and the second coefficient to obtain a multiplication value. Ask,
The image processing method according to claim 6.
In the step (c), when the parallax of the set and another selected set including an image block adjacent to the image block of the set in the setting direction matches, the first coefficient of the set To correct the value to be smaller,
The image processing method according to claim 6 or 7.
The image processing method according to any one of claims 6 to 8, wherein the two images are pair images obtained by photographing the same object from different angles.
(F) When the two images are acquired and one or both of the two images are not rotated or projected when the epipolar lines of the acquired two images do not match in the horizontal direction, Performing at least one to match the epipolar lines of each of the two images in the horizontal direction;
In the step (a), each of the two images is divided into tiles so that rows in the horizontal direction and columns in the vertical direction are formed along the vertical direction and the horizontal direction.
One of the image blocks obtained from the one image and one of the image blocks obtained from the other image are selected so that the rows in which they are located correspond to each other, and the selected image blocks To set the set of image blocks,
The image processing method according to any one of claims 6 to 9.
On the computer,
(A) Each of the two target images is divided into image blocks of a set size, and one of the image blocks obtained from one of the two images and the image block obtained from the other image A set of a plurality of image blocks, and (b) calculating a correlation value for each set of set image blocks; and
(C) selecting a set of image blocks from the set of the plurality of set image blocks so that a plurality of sets including the same image block do not exist, constructing a set, and further selecting the selected image For each set of blocks, determine the distance between the set and another selected set that includes an image block adjacent to the set of image blocks in the set direction, and the larger the calculated distance, the smaller the value, and Setting a first coefficient with an upper limit value of 1; and
(D) In each of the set sets, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient to obtain a multiplication value, and the obtained multiplication value is further integrated. And calculating the integrated correlation value,
(E) The set set that minimizes the integrated correlation value is specified, and in each set of image blocks constituting the specified set set, one image block and the other image block constituting the set are A step of determining that they match, and
The computer-readable recording medium which recorded the program containing the instruction | indication which performs this.
In the step (c), when the correlation value of the set is less than the threshold, a second coefficient that increases the first coefficient is set, and the correlation value of the set is equal to or greater than the threshold And setting a second coefficient to reduce the first coefficient,
In the step (d), for each set set, for each set of image blocks constituting the set set, the correlation value is multiplied by the first coefficient and the second coefficient to obtain a multiplication value. Ask,
The computer-readable recording medium according to claim 11.
In the step (c), when the parallax of the set and another selected set including an image block adjacent to the image block of the set in the setting direction matches, the first coefficient of the set To correct the value to be smaller,
The computer-readable recording medium according to claim 11 or 12.
The computer-readable recording medium according to any one of claims 11 to 13, wherein the two images are pair images obtained by photographing the same object from different angles.
The program is stored in the computer.
(F) When the two images are acquired and one or both of the two images are not rotated or projected when the epipolar lines of the acquired two images do not match in the horizontal direction, Further comprising: performing at least one to match the epipolar lines of each of the two images in the horizontal direction;
In the step (a), each of the two images is divided into tiles so that rows in the horizontal direction and columns in the vertical direction are formed along the vertical direction and the horizontal direction.
One of the image blocks obtained from the one image and one of the image blocks obtained from the other image are selected so that the rows in which they are located correspond to each other, and the selected image blocks To set the set of image blocks,
The computer-readable recording medium according to any one of claims 11 to 14.