JP5712810B2 - Image processing apparatus, program thereof, and image processing method - Google Patents

Description

  The present invention relates to a corresponding point search technique for searching for a corresponding point of an attention point set in one image from another image.

  By searching for a corresponding point on another image corresponding to a point of interest on one image among a plurality of images related to the measurement object photographed from different viewpoints by performing block matching between both images, There is known an apparatus for obtaining a three-dimensional shape of an object to be measured based on a corresponding point and a point of interest.

  In the corresponding point search method using block matching, for example, a search block (also referred to as “reference block” or “reference window”) whose center is the attention point is set on one image, and A search block (also referred to as “reference block” or “reference window”) having the same size as the standard window is set in an area on the reference image in which the presence of a point is expected.

  Next, while the position of the reference block is shifted on another image, the correlation calculation of each image of the base block and the reference block is performed, and the correlation (also referred to as “similarity”) becomes the highest. A search block is identified. Then, corresponding points are specified in the specified search block.

  By the way, in the corresponding point search using block matching, for example, when there are a plurality of combinations of similar image contents between the images to be matched, there is a case where an erroneous correspondence in which an incorrect corresponding point is searched for may occur. is there.

  Patent Document 1 discloses an image processing apparatus that verifies the reliability of corresponding points searched by block matching. In the image processing apparatus, first, a corresponding point corresponding to the reference point is searched on the other image by block matching between the reference block containing the reference point on one image and another image. Next, if the block on the one image corresponding to the verification block on the other image including the corresponding point is specified, and the specified block matches the verification block, then the searched correspondence The reliability of a point is determined to be high (search success), and if they do not match, the reliability is determined to be low (miscorresponding).

JP 2004-184240 A

  However, although the image processing apparatus disclosed in Patent Document 1 can detect corresponding points searched for due to incorrect correspondence, there is a problem that it is not possible to search for corresponding points with higher reliability than corresponding points related to erroneous correspondence.

  The present invention has been made to solve these problems, and an object of the present invention is to provide a technique capable of improving the reliability of searched corresponding points.

In order to solve the above problem, an image processing apparatus according to a first aspect includes an acquisition unit that acquires a first image and a second image that include image contents corresponding to each other, and a plurality of images on the second image. A first setting unit that sets a first reference block, a first reference block that spatially encloses a point of interest set in advance on the first image, and a first of each of the plurality of first reference blocks A first computing unit that obtains a plurality of first index values respectively representing similarity of image contents between the first reference block and each of the plurality of first reference blocks by block matching; Based on one index value, an extraction unit that extracts a plurality of candidate points of corresponding points corresponding to the attention point in the second image, and only one of the plurality of candidate points spatially Multiple second standards to include Each of the plurality of candidate points is obtained by second block matching between a second setting unit for setting a lock, each of the plurality of second reference blocks, and a second reference block set in advance on the first image. A second calculation unit that obtains a plurality of second index values expressing suitability for selection as the corresponding points, and one of the plurality of candidate points based on the plurality of second index values. And a determining unit that determines the corresponding points by selecting.

  An image processing apparatus according to a second aspect is the image processing apparatus according to the first aspect, wherein the first setting unit spatially sets a plurality of search reference points set in advance on the second image. A plurality of first reference blocks are respectively set by applying a first rule relating to block setting in the image space to each of the plurality of search reference points as a reference point, and the second setting unit The plurality of second reference blocks by applying a second rule relating to block setting in the image space to each of the plurality of candidate points, with each of the plurality of candidate points as a spatial reference point. Each of the first rule and the second rule is characterized in that a block setting mode with respect to a spatial reference point is spatially different from each other.

  An image processing apparatus according to a third aspect is the image processing apparatus according to the second aspect, wherein the first rule and the second rule are the first rule and the first rule with respect to a common spatial reference point. The shapes of the two blocks set by applying the rule and the second rule are different from each other.

  An image processing device according to a fourth aspect is the image processing device according to the second aspect, wherein the first rule and the second rule are the first rule and the first rule with respect to a common spatial reference point. The spatial sizes of the two blocks set by applying the rule and the second rule are different from each other.

  An image processing apparatus according to a fifth aspect is the image processing apparatus according to the second aspect, wherein the first rule and the second rule are the first rule and the first rule with respect to a common spatial reference point. The positions of the two blocks set by applying the rule and the second rule are different from each other.

  An image processing device according to a sixth aspect is the image processing device according to any one of the second to fifth aspects, and the second rule is to spatially select a candidate point of interest among the plurality of candidate points. As the reference point, the second reference block set by applying the second rule to the attention candidate point is configured so that only the attention candidate point is included in the plurality of candidate points. This is a rule for setting the second reference block.

  An image processing apparatus according to a seventh aspect is the image processing apparatus according to the sixth aspect, wherein the second rule is set by applying the second rule to the attention candidate point. The second reference block is a rule for setting the second reference block according to the mutual positional relationship between the plurality of candidate points so that only the attention candidate point is included in the second reference block. It is characterized by that.

  An image processing device according to an eighth aspect is the image processing device according to the sixth aspect, wherein the second rule is set by applying the second rule to the attention candidate point. According to the spatial distribution state of the plurality of first index values in the image space of the second image, so that only the attention candidate point among the plurality of candidate points is included in the second reference block. It is a rule for setting the second reference block.

  An image processing device according to a ninth aspect is the image processing device according to any one of the first to eighth aspects, wherein the second calculation unit includes each of the plurality of second reference blocks, The second block matching is performed by performing block matching with the second reference block while shifting the position of the second reference block on the first image, and each of the plurality of candidate points is determined by the second block matching. A plurality of corresponding corresponding points are specified on the first image, respectively, and the plurality of second index values are acquired based on distances between the attention point and the corresponding points, respectively. Features.

  An image processing device according to a tenth aspect is the image processing device according to any one of the first to eighth aspects, wherein the second calculation unit includes each of the plurality of second reference blocks, Performing the second block matching by performing block matching with the second reference block while shifting the position of the second reference block on the first image, and based on the result of the second block matching, Obtaining the highest similarity with the second reference block for each of the second reference blocks, and obtaining the plurality of second index values based on the obtained similarity. To do.

  An image processing apparatus according to an eleventh aspect is the image processing apparatus according to any one of the first to eighth aspects, wherein the second reference block is set with the attention point as a spatial reference point. The second arithmetic unit performs the second block matching by block matching between each of the plurality of second reference blocks and the second reference block, and the plurality of second reference blocks. The plurality of second index values are respectively acquired based on the similarity between each of the first reference block and the second reference block.

  An image processing device according to a twelfth aspect is the image processing device according to any one of the sixth to eighth aspects, wherein the second reference block uses the attention point as a spatial reference point. Each of the plurality of candidate points is set in advance on the first image by applying each of the second rules when each of the plurality of candidate points is the target candidate point. And the second calculation unit includes: the second reference block corresponding to one candidate point among the plurality of candidate points; and the block corresponding to the first candidate point among the plurality of second reference blocks. The second block matching is performed by performing block matching while changing a candidate point set as the first candidate point among the plurality of candidate points, and based on the result of the second block matching. Then, each similarity with the corresponding second reference block is acquired for each of the plurality of second reference blocks, and the plurality of second index values are acquired based on the acquired similarities, respectively. It is characterized by that.

  The program according to the thirteenth aspect is executed by a computer mounted on the image processing apparatus, thereby causing the image processing apparatus to function as the image processing apparatus described in any one of the first to twelfth aspects. It is characterized by that.

An image processing method according to a fourteenth aspect includes a first reference block that spatially encloses a point of interest set in advance on a first image, and a second image including image content corresponding to the first image. A plurality of first representations of image content similarities between the first reference block and each of the plurality of first reference blocks by first block matching with each of the plurality of first reference blocks set above. A first calculation step of acquiring a first index value; an extraction step of extracting a plurality of candidate points of corresponding points corresponding to the attention point in the second image based on the plurality of first index values; the second block matching between respectively, the second reference block which is previously set on the first image of the plurality of second reference block enclosing each one point only, respectively spatially of the plurality of candidate points Further, for each of the plurality of candidate points, based on the second calculation step for obtaining a plurality of second index values expressing suitability for selection as the corresponding points, and the plurality of second index values, the plurality of the plurality of candidate points And determining a corresponding point by selecting one of the candidate points.

By the invention described in the present lever, a plurality of candidate points corresponding points corresponding to the target point on the first image by the first block matching are extracted, respectively, in the second image. Then, based on the result of the second block matching between the plurality of second reference blocks each spatially including the plurality of candidate points and the second reference block on the first image, the plurality of candidate points are selected from the plurality of candidate points. Since the corresponding points are determined, the reliability of the determined corresponding points can be improved.

1 is a functional block diagram illustrating a configuration example of an image processing system using an image processing apparatus according to an embodiment. It is a functional block diagram showing an example of composition of a principal part of an image processing device concerning an embodiment. It is a figure for demonstrating the outline | summary of the operation example of the image processing apparatus which concerns on embodiment. It is a figure for demonstrating the outline | summary of the operation example of the image processing apparatus which concerns on embodiment. It is a figure which illustrates the graph which shows a correspondence with the 1st reference block and the 1st index value. It is a figure which illustrates the graph which shows a response | compatibility with a 2nd reference | standard block and a 2nd index value. It is a figure which shows the example of a setting of a 1st reference | standard block and a 1st reference block. It is a figure which shows the example of a setting of a 2nd reference | standard block and a 2nd reference block. It is a figure explaining the example of the candidate point extraction process using SAD method. It is a figure which illustrates the scanning aspect of the reference block in SAD method. It is a figure which shows an example of the procedure of the candidate point extraction process using POC method. It is a figure which shows the example of a POC value. It is a figure which shows the example of a setting of a 2nd reference | standard block. It is a figure which shows the example of a setting of a 2nd reference | standard block. It is a figure which shows the example of a setting of a 2nd reference | standard block. It is a figure which shows the example of a setting of a 2nd reference | standard block. It is a figure which shows the example of a setting of a 2nd reference | standard block. It is a figure which shows the example of a setting of a 2nd reference | standard block. It is a figure which shows the example of a setting of a 2nd reference | standard block. It is a figure which shows the example of a setting of a 2nd reference | standard block. It is a figure for demonstrating an example of the 2nd block matching process. It is a figure for demonstrating an example of the 2nd block matching process. It is a figure for demonstrating an example of the 2nd block matching process. It is a figure which illustrates the operation | movement flow of the image processing apparatus which concerns on embodiment. It is a figure which illustrates the operation | movement flow of the image processing apparatus which concerns on embodiment.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, parts having the same configuration and function are denoted by the same reference numerals, and redundant description is omitted in the following description. Each drawing is schematically shown, and for example, the size and positional relationship of the display object on the image in each drawing are not accurately shown. In addition, image data and images displayed based on the image data are collectively referred to as “image” as appropriate. Further, in each drawing, the upper left pixel of the image is the origin, the direction along the long side of the image (here, the horizontal direction) is the X axis direction, and the direction along the short side of the image (here (The vertical direction) is the Y-axis direction. The right direction of each image is the + X direction, and the downward direction of each image is the + Y direction. 3, 4, 7 to 10, and FIGS. 13 to 23, two orthogonal XY axes are attached.

<About embodiment:>
<(1) About Image Processing System 100A>
FIG. 1 is a functional block diagram illustrating a configuration example of an image processing system 100A using the image processing apparatus 200A according to the embodiment. As shown in FIG. 1, the image processing system 100A mainly includes a stereo camera 300 and an image processing apparatus 200A. In the image processing system 100 </ b> A, the image processing apparatus 200 </ b> A acquires the first image 1 and the second image 2 acquired by photographing the subject 70 with the stereo camera 300. The image processing apparatus 200A processes the first image 1 and the second image 2, thereby corresponding points 25 (see FIG. 3) on the second image 2 corresponding to the attention points 21 (FIG. 3) on the first image 1. ). The attention point 21 and the corresponding point 25 respectively correspond to the same point on the subject 70, and the corresponding point information 35 (FIG. 2) such as coordinate information regarding the corresponding point 25 is stored in the storage device 46 of the image processing apparatus 200A. Is remembered. The image processing apparatus 200A acquires distance information of the subject 70 based on, for example, a shift amount (also referred to as “parallax”) between the attention point 21 and the corresponding point 25 in the image space.

<(1-1) Stereo Camera 300>
As shown in FIG. 1, the stereo camera 300 mainly includes a first camera 91 and a second camera 92. Each of the first camera 91 and the second camera 92 mainly includes an imaging optical system (not shown) and a control processing circuit having an image sensor with a predetermined number of pixels. Various operations of the stereo camera 300 are controlled based on control signals supplied from the image processing apparatus 200A via the input / output unit 41 and the data line DL.

  The first camera 91 and the second camera 92 are provided with a predetermined baseline length apart, and the stereo camera 300 displays the same subject 70 from different directions according to a control signal from the image processing device 200A. Shoot each one. The stereo camera 300 processes the light ray information from the photographed subject 70 in synchronization with the first camera 91 and the second camera 92, so that the stereo camera 300 has, for example, 3456 pixels × 2592 pixels, and the stereo of the subject. A first image 1 and a second image 2 constituting the image are generated. The first image 1 and the second image 2 are images obtained by shooting the subject 70 from different viewpoints, and include image contents corresponding to each other. The generated first image 1 and second image 2 are supplied to the input / output unit 41 of the image processing apparatus 200A via the data line DL.

<(1-2) About Image Processing Device 200A>
As shown in FIG. 1, the image processing apparatus 200 </ b> A mainly includes a CPU 11 </ b> A, an input / output unit 41, an operation unit 42, a display unit 43, a ROM 44, a RAM 45, and a storage device 46. This is realized by executing the program on a computer.

○ Input / output unit 41:
The input / output unit 41 includes an input / output interface such as a USB interface or a Bluetooth (registered trademark) interface, an interface for connecting to a LAN or the Internet such as a multimedia drive and a network adapter, and the like. Exchange data between the two. Specifically, the input / output unit 41 supplies, for example, various control signals for the CPU 11A to control the stereo camera 300 to the stereo camera 300 connected to the input / output unit 41 via the communication line DL or the like. To do. Further, the input / output unit 41 supplies the first image 1 and the second image 2 captured by the stereo camera 300 to the image processing apparatus 200A. The input / output unit 41 receives the first image 1 and the second image 2 in the image processing apparatus 200A by receiving a storage medium such as an optical disk in which the first image 1 and the second image 2 are stored in advance. Supply.

○ Operation unit 42:
The operation unit 42 includes, for example, a keyboard or a mouse. When the operator operates the operation unit 42, setting of various control parameters for the image processing apparatus 200A and various operation modes of the image processing apparatus 200A are performed. Settings are made. In addition, each functional unit of the image processing apparatus 200A is configured to perform processing according to each operation mode set from the operation unit 42.

○ Display unit 43:
The display unit 43 is configured by, for example, a liquid crystal display screen for three-dimensional display corresponding to a three-dimensional display method such as a parallax barrier method, and the first image 1 and the second image that form a stereoscopic image of the subject. 2 etc. are displayed on the display screen. The display unit 43 can also display various setting information regarding the image processing apparatus 200 </ b> A, a control GUI (Graphical User Interface), and the like as a two-dimensional image and text information so that the viewer can visually recognize them.

○ ROM44:
A ROM (Read Only Memory) 44 is a read only memory, and stores a program PG1 for operating the CPU 11A and the like. A readable / writable nonvolatile memory (for example, a flash memory) may be used instead of the ROM 44.

○ RAM45:
A RAM (Random Access Memory) 45 is a readable / writable volatile memory, and functions as an image storage unit that temporarily stores various images acquired or processed by the image processing apparatus 200A. The RAM 45 also functions as a work memory that temporarily stores processing information of the CPU 11A. Note that an image (image data) or the like is exchanged between the respective functional units in the CPU 11A, which will be described later, via the RAM 45.

○ Storage device 46:
The storage device 46 is configured by, for example, a readable / writable nonvolatile memory such as a flash memory, a hard disk device, or the like, and permanently records information such as various control parameters and various operation modes of the image processing device 200A. As the various control parameters, for example, first rule information 57 and second rule information 58 (FIG. 2) are stored by input via the operation unit 42 or the like. The first rule information 57 is information defining a first rule relating to setting of a block in the image space, and the second rule information 58 is information defining a second rule relating to setting of a block in the image space. The first rule information 57 and the second rule information 58 include, for example, parameters such as a block shape with respect to a spatial reference point in the image space, a spatial size, and block arrangement information with respect to the reference point. It is. The first rule information 57 and the second rule information 58 are acquired by the CPU 11A and are used for setting blocks in the image space (also referred to as “space blocks”). The CPU 11A can not only use the first rule information 57 and the second rule information 58 as they are stored in the storage device 46, but also according to the information of the image on which the block is set, etc. Each parameter can be changed and used. The storage device 46 also permanently stores the corresponding point information 35 acquired by the image processing device 200A.

○ CPU 11A:
The CPU (Central Processing Unit) 11 </ b> A is a control processing device that controls each functional unit of the image processing device 200 </ b> A, and executes control and processing according to the program PG <b> 1 stored in the ROM 44. As will be described later with reference to FIG. 2, the CPU 11A acquires the acquisition unit 12, the attention point setting unit 13, the first setting unit 14, the first calculation unit 15, the extraction unit 16, the second setting unit 17, and the second calculation unit 18. And also functions as the determination unit 19. The CPU 11A processes the first image 1 and the second image 2 with these function units and the like, and thereby on the second image 2 corresponding to the attention point 21 (FIG. 3) set on the first image 1. Corresponding point 25 (FIG. 3) is determined. The corresponding point information 35 regarding the determined corresponding point 25 is stored in the storage device 46 or the like, and is used by the CPU 11A to acquire distance information of the photographed subject. In addition, the CPU 11A controls the imaging operation of the stereo camera 300 and controls the display unit 43 to display various images, calculation results, various control information, and the like on the display unit 43.

  Further, the CPU 11 A, the input / output unit 41, the operation unit 42, the display unit 43, the ROM 44, the RAM 45, the storage device 46, and the like are electrically connected via a signal line 49. Therefore, for example, the CPU 11A can execute control of the stereo camera 300 via the input / output unit 41, acquisition of image information from the stereo camera 300, display on the display unit 43, and the like at a predetermined timing. In the configuration example shown in FIG. 2, the acquisition unit 12, the attention point setting unit 13, the first setting unit 14, the first calculation unit 15, the extraction unit 16, the second setting unit 17, the second calculation unit 18, and Each function unit such as the determination unit 19 is realized by executing a predetermined program PG1 by the CPU 11A. However, each of these function units may be realized by a dedicated hardware circuit, for example.

<(1-3) Configuration of Main Part of Image Processing Apparatus 200A>
FIG. 2 is a functional block diagram illustrating a configuration example of a main part of the image processing apparatus 200A according to the embodiment. As shown in FIG. 2, the main part of the image processing apparatus 200A includes a CPU 11A, an input / output unit 41, a storage device 46, and the like. The CPU 11A also includes the acquisition unit 12, the attention point setting unit 13, the first setting unit 14, the first calculation unit 15, the extraction unit 16, the second setting unit 17, the second calculation unit 18, the determination unit 19, and the like. Also functions as a functional unit.

  3 and 4 are diagrams for explaining an overview of processing performed by each functional unit of the image processing apparatus 200A. The attention point 21 (FIGS. 3 and 4) is an attention point set on the first image 1, and the first reference block 51 (FIG. 3) set on the first image 1 It is a block that contains spatially. Each of the first reference blocks 52a and 52b (FIG. 3) is an example of each block constituting a plurality of first reference blocks 52 (FIG. 3) set on the second image 2.

  FIG. 5 is a diagram illustrating a graph 81 illustrating the correspondence between each of the plurality of first reference blocks 52 and the first index value 71 (FIG. 2). The first index value 71 is an index value that represents the degree of similarity of image contents between the first reference block 51 and each of the plurality of first reference blocks 52. The threshold value TH1 (FIG. 5) is a predetermined threshold value for determining whether or not a corresponding point candidate. As shown in FIG. 5, among the first index values 71, there are two index values greater than the threshold value TH1, first index values 71a and 72a. The first index values 71a and 72a are the first index values 71 corresponding to the first reference blocks 52a and 52b (FIG. 3), respectively. As shown in FIG. 3, candidate points 23a and 23b are spatially included in the first reference blocks 52a and 52b, respectively. The candidate points 23a and 23b constitute a plurality of candidate points 23 (FIGS. 3 and 4) of the corresponding points 25 (FIGS. 3 and 4) on the second image 2 corresponding to the attention point 21.

  The second reference blocks 53a and 53b (FIG. 4) are blocks set on the second image 2 so as to include the candidate points 23a and 23b, respectively. The second reference blocks 53a and 53b constitute a plurality of second reference blocks 53 (FIG. 4). The second reference block 54 (FIG. 4) is one or more blocks set on the first image 1.

  FIG. 6 is a graph illustrating a correspondence relationship between the plurality of second reference blocks 53 (FIG. 4) and the second index value 72 (FIG. 2). In FIG. 6, the relationship between the second reference values 53 a and 53 b (FIG. 4) corresponding to the candidate points 23 a and 23 b (FIGS. 3 and 4) and the second index value 72 is shown. The second index value 72 is an index value expressing the aptitude of selection as the corresponding point 25 for each of the plurality of candidate points 23, that is, the candidate points 23a and 23b. As shown in the graph 82, the second index value 72a for the second reference block 53a is higher than the second index value 72b for the second reference block 53b. That is, in the example shown in the graph 82, the candidate point 23a included in the second reference block 53a is determined to be the corresponding point 25.

  Hereinafter, the acquisition unit 12, the attention point setting unit 13, the first setting unit 14, the first calculation unit 15, the extraction unit 16, the second setting unit 17, the second calculation unit 18, and the determination unit 19 of the image processing apparatus 200A. Will be described.

○ Acquisition unit 12:
The acquisition unit 12 acquires the first image 1 (FIG. 3) and the second image 2 (FIG. 3) supplied from the input / output unit 41. The acquired first image 1 and second image 2 include image contents corresponding to each other. From the acquisition unit 12, the attention point setting unit 13, the first setting unit 14, the first calculation unit 15, and the second It is supplied to the calculation unit 18.

○ Attention point setting unit 13:
The point-of-interest setting unit 13 sets the point of interest 21 (FIG. 3) on the first image 1 by extracting feature points on the image by various image processes or setting according to a predetermined rule, and sets the attention point set. The coordinates of the point 21 are acquired as attention point information 31 (FIG. 2). The acquired attention point information 31 is supplied to the first setting unit 14, the extraction unit 16, the second setting unit 17, and the second calculation unit 18.

○ First setting unit 14:
The first setting unit 14 sets a first reference block 51 (FIG. 3) that spatially encloses the attention point 21 based on the attention point information 31, and is the region information of the set first reference block 51. 1 reference block information 61 (FIG. 2) is acquired. In addition, the first setting unit 14 sets a plurality of first reference blocks 52 (FIG. 3) on the second image 2, and a plurality of first information that is region information regarding each of the set first reference blocks 52. Reference block information 62 (FIG. 2) is acquired. The acquired first reference block information 61 and the plurality of first reference block information 62 are supplied to the first calculation unit 15 and the extraction unit 16.

  The first setting unit 14 acquires from the storage device 46 the first rule information 57 that defines the first rule regarding the setting of blocks in the image space, and the first reference block 51 and the plurality of first reference blocks 52. Are used to set at least a plurality of first reference blocks 52. More specifically, the first setting unit 14 uses, for example, a plurality of search reference points set in advance on the second image 2 as spatial reference points for each of the plurality of search reference points. A plurality of first reference blocks 52 are set by applying the first rule, respectively.

  In addition, the first setting unit 14 normally applies the first rule to the setting of the first reference block 51 by the operation mode of the initial setting (default), but the first rule is changed according to the change of the operation mode. Even if other rules are applied, the usefulness of the present invention is not impaired. As a result of using the other rules, if the first reference block 51 and each first reference block 52 have different shapes or spatial sizes, prior to the first block matching in the first calculation unit 15 In order to make both shapes and spatial sizes equal, for example, image trimming, deformation, or expansion processing is performed. In the initial setting mode, the first rule related to the first rule information 57 and the second rule related to the second rule information 58, which will be described later, are spatially mutually different from each other in the block setting mode with respect to the spatial reference point. Is different. However, even if the setting modes of the blocks with respect to the spatial reference point of the first rule and the second rule are used spatially equal to each other according to the change of the operation mode, the usefulness of the present invention is impaired. It is not a thing.

○ First calculation unit 15:
The first computing unit 15 is a first reference block that spatially encloses the point of interest 21 based on the first image 1, the second image 2, the first reference block information 61, and the plurality of first reference block information 62. The first block matching between 51 and each of the plurality of first reference blocks 52 is performed. The first calculation unit 15 uses a first block matching to express a plurality of first index values 71 (FIG. 2) that express the similarity of the image contents between the first reference block 51 and each of the plurality of first reference blocks 52. ) To get. The plurality of acquired first index values 71 are supplied to the extraction unit 16 and the second setting unit 17.

○ Extraction unit 16:
Based on the attention point information 31, the plurality of first index values 71, the first reference block information 61, and the plurality of first reference block information 62, the extraction unit 16 corresponds to the attention point 25 corresponding to the attention point 21 (FIG. 3). Are extracted from the second image 2 respectively. The extraction unit 16 acquires a plurality of candidate point information 33 that is the coordinate information of each of the extracted plurality of candidate points 23, and the plurality of candidate point information 33 is obtained as the second setting unit 17, the second calculation unit 18, and the determination. It supplies to the part 19.

○ Second setting unit 17:
The second setting unit 17 sets a plurality of second reference blocks 53 that spatially include the plurality of candidate points 23 based on the plurality of candidate point information 33, and sets the region information of the second reference block 53 thus set. The second reference block information 63 (FIG. 2) is acquired. Further, the second setting unit 17 sets one or more second reference blocks 54 on the first image 1 by appropriately using the attention point information 31 according to the operation mode, and the set second reference block. For each of 54, one or more second reference block information 64 (FIG. 2), which is region information, is acquired. The acquired plurality of second reference block information 63 and one or more pieces of second reference block information 64 are supplied to the second calculation unit 18.

  Note that the second setting unit 17 acquires from the storage device 46 the second rule information 58 that defines the second rule relating to the setting of the blocks in the image space, and includes a plurality of second reference blocks 53 and one or more second blocks. It is used for setting at least a plurality of second reference blocks 53 among the reference blocks 54. More specifically, the second setting unit 17 applies the second rule to each of the plurality of candidate points 23 with each of the plurality of candidate points 23 as a spatial reference point. Two reference blocks 53 are set.

  The second setting unit 17 normally applies the second rule to the setting of one or more second reference blocks 54 according to the operation mode of the initial setting (default), but according to the change of the operation mode, Even if other rules other than the second rule are applied, the usefulness of the present invention is not impaired. When the second reference block 53 and the second reference block 54 are different in shape or spatial size as a result of using the other rule, prior to the second block matching in the second calculation unit 18, In order to make both shapes and spatial sizes equal, for example, image trimming, deformation, or expansion processing is performed.

○ Second calculation unit 18:
The second calculation unit 18 performs the second block matching between each of the plurality of second reference blocks 53 and the second reference block 54 set in advance on the first image 1, for each of the plurality of candidate points 23. A plurality of second index values 72 (FIG. 2) expressing suitability for selection as the corresponding points 25 are acquired. The second calculation unit 18 includes at least a first image 1, a second image 2, a plurality of second reference block information 63, and one or more second reference blocks among the information supplied from the other functional units. A plurality of second index values 72 are acquired based on the reference block information 64. Further, the plurality of acquired second index values 72 are supplied to the determination unit 19.

○ Determining unit 19:
The determination unit 19 determines the corresponding point 25 by selecting one of the plurality of candidate points 23 as the corresponding point 25 based on the plurality of candidate point information 33 and the plurality of second index values 72. . The determination unit 19 acquires the determined coordinates of the corresponding point 25 as the corresponding point information 35 and stores it in the storage device 46.

  As described above, in the image processing apparatus 200A, the plurality of candidate points 23 of the corresponding points 25 corresponding to the point of interest 21 on the first image 1 are extracted in the second image 2 by the first block matching. Then, based on the result of the second block matching between the plurality of second reference blocks 53 that spatially contain the plurality of candidate points 23 and the second reference block 54 on the first image 1, the plurality of candidates A corresponding point 25 is determined from the points 23. Therefore, the reliability as the corresponding point of the determined corresponding point 25 can be improved.

<(2) Operation of Image Processing Device 200A>
24 and 25, the image processing apparatus 200A according to the embodiment determines the corresponding point 25 (FIG. 4) on the second image 2 corresponding to the attention point 21 (FIG. 4) on the first image 1. It is a figure which shows one example of operation | movement flow S10A. The operation of the image processing apparatus 200A will be described below with reference to FIGS. 24 and 25 as appropriate.

<(2-1) Step S10: Image Acquisition Process>
When the first image 1 and the second image 2 are acquired and supplied to the image processing apparatus 200A by the photographing operation of the subject 70 by the stereo camera 300 according to the instruction from the image processing apparatus 200A, the acquisition unit 12 (FIG. 2). ) Acquires the first image 1 and the second image 2 (FIGS. 1 to 3) via the input / output unit 41 (step S <b> 10 in FIG. 24). The acquired first image 1 and second image 2 are supplied to the attention point setting unit 13, the first setting unit 14, the first calculation unit 15, and the second calculation unit 18.

<(2-2) Step S20: Step of Setting Interesting Point>
When the first image 1 and the second image 2 are acquired, the point-of-interest setting unit 13 (FIG. 2) performs the first operation by extracting feature points on the image by various image processing or setting by a predetermined rule. An attention point 21 (FIG. 3) is set on one image 1 (step S20 in FIG. 24), and the coordinates of the set attention point 21 are acquired as attention point information 31 (FIG. 2). The acquired attention point information 31 is supplied to the first setting unit 14, the extraction unit 16, the second setting unit 17, and the second calculation unit 18.

<(2-3) Step S30: First Reference Block Setting Process>
When the attention point information 31 is acquired, the first setting unit 14 (FIG. 2) sets the first reference block 51 (FIG. 3) that spatially includes the attention point 21 based on the attention point information 31 ( Step S30 in FIG. 24 is performed, and first reference block information 61 (FIG. 2), which is the set region information of the first reference block 51, is acquired. The acquired first reference block information 61 is supplied to the first calculation unit 15 and the extraction unit 16.

  FIG. 7 is a diagram illustrating a setting example of the first reference block 51 and the first reference block 52. As shown in FIG. 7, the first setting unit 14 sets the first reference block 52 by applying the first rule based on the first rule information 57, for example, using the point of interest 21 as a spatial reference point. Set on the first image 1 (FIG. 3). In the first rule information 57 (FIGS. 1 and 2) defining the first rule, for example, the shape of the block with respect to a spatial reference point in the image space, the spatial size, and the arrangement of the block with respect to the reference point Each parameter such as information is included. The parameters can be changed as appropriate, and the parameters at the time of initial setting can be changed as appropriate. The first rule relating to FIG. 7 is the first rule corresponding to the parameters at the time of initial setting.

<(2-4) Step S40: First Reference Block Setting Process>
When the first reference block 51 is set, the first setting unit 14 sets a plurality of first reference blocks 52 (FIG. 3) on the second image 2 (step S40 in FIG. 24), and sets the plurality of set A plurality of pieces of first reference block information 62 (FIG. 2), which is region information regarding each of the first reference blocks 52, is acquired. Specifically, for example, the first setting unit 14 first sets a plurality of preset search reference points on the second image 2, and then sets the plurality of set search reference points in space. A plurality of first reference blocks 52 are set as typical reference points. The plurality of first reference blocks 52 are set by applying the first rule to each of the plurality of search reference points. In the setting example of FIG. 7, the first setting unit 14 sets the first reference block 52 on the second image 2 (FIG. 3) according to the first rule, with the search reference point 22 as a spatial reference point. Yes. The first setting unit 14 supplies the acquired first reference block information 62 to the first calculation unit 15 and the extraction unit 16.

  For example, as shown in FIG. 10, the search reference point 22 may be set by being shifted in units of pixel size on the second image 2, and may be stepped every several pixels or irregularly discrete. It may be set by setting. Alternatively, the second image 2 may be divided by a grid, and each grid point may be set as the search reference point 22.

  In addition, when a method of performing frequency analysis is selected as a block matching method to be described later, it is set from the viewpoint of speeding up the processing by suppressing the necessary number of calculations using Fast Fourier Transform (FFT). The block size is preferably the number of pixels represented by the power of 2 as the number of pixels in the horizontal direction and the number of pixels in the vertical direction.

<(2-5) Step S50: First Block Matching Step>
When the first reference block 51 and the plurality of first reference blocks 52 are set, the first calculation unit 15 performs the first image 1, the second image 2, the first reference block information 61, and the plurality of first reference blocks. Based on the information 62, first block matching is performed between the first reference block 51 that spatially encloses the point of interest 21 and each of the plurality of first reference blocks 52 (step S50 in FIG. 24).

<(2-6) Step S60: Step of Obtaining First Index Value>
Next, the first calculation unit 15 represents a plurality of image contents similarities between the first reference block 51 and each of the plurality of first reference blocks 52 based on the calculation result of the first block matching. The first index value 71 (FIG. 2) is acquired (step S60 in FIG. 24). The plurality of acquired first index values 71 are supplied to the extraction unit 16 and the second setting unit 17. Steps S50 and S60 function as a first calculation step.

<(2-7) Step S70: Extraction Step of Corresponding Points>
When the plurality of first index values 71 are acquired, the extraction unit 16 based on the attention point information 31, the plurality of first index values 71, the first reference block information 61, and the plurality of first reference block information 62, A plurality of candidate points 23 (FIG. 3) of corresponding points 25 (FIG. 3) corresponding to the point of interest 21 are extracted from the second image 2 (step S70 in FIG. 24). The extraction unit 16 acquires a plurality of candidate point information 33 that is the coordinate information of each of the extracted plurality of candidate points 23, and the plurality of candidate point information 33 is obtained as the second setting unit 17, the second calculation unit 18, and the determination. It supplies to the part 19. Step S70 functions as an extraction process.

  Here, the first block matching related to steps S50 to S70 described above and the second block matching related to steps S100 to S110 described later are performed by block matching (pattern matching) processing using a correlation calculation method. As the correlation calculation method, for example, a correlation calculation method such as an NCC (Normalized Cross Correlation) method, an SAD (Sum of Absolute Difference) method, or a POC (Phase Only Correlation) method may be employed.

  Next, a method for extracting the candidate point 23 by first block matching for obtaining a correlation between the images of the standard block 51 and the reference block 52 will be described by taking as an example the case where the SAD method and the POC method are respectively employed.

○ Candidate point extraction processing by first block matching using the SAD method:
FIG. 9 is a diagram illustrating an example of candidate point extraction processing (candidate point search processing) by first block matching using the SAD method. On the first image 1 shown in FIG. 9, an attention point 21 whose X coordinate is h2 is set, and a reference block 51 is set for the attention point 21. The X coordinates of both ends of the reference block 51 in the X direction are h1 and h3. In addition, candidate points 23 corresponding to the attention point 21 are shown on the second image 2. Here, an example in which the candidate point 23 is extracted using the SAD method will be described.

  Further, as shown in FIG. 9, on the second image 2, from the search reference point 22s whose X coordinate is xs to the search reference point 22e whose X coordinate is xe, the X coordinate is xs ≦ x ≦ xe. A plurality of search reference points 22 (FIG. 3) are set for each pixel in the range. Only the search reference points 22 s and 22 e at both ends are described as representatives of the plurality of search reference points 22. In the plurality of search reference points 22, a plurality of reference blocks having the same size as the reference block 51 are set, and only reference blocks 52s and 52e at both ends are described as representatives of the plurality of reference blocks.

  Here, in order to simplify the description in FIG. 9, the Y coordinate of the attention point 21 and the plurality of search reference points 22 s to 22 e is the same. The coordinates are set so that there is no deviation. Further, the attention point 21 and the Y coordinate of the candidate point 23 to be obtained are also set to coincide.

  Next, a correlation value (similarity) between each image in each reference block of the reference blocks 52s to 52e and an image in the standard block 51 is calculated. The shift amounts (parallax) of the coordinates in the X and Y directions of the base block and the reference block are set as parallax xd and yd, respectively. In the example of FIG. 9, the range of the parallax xd is xs-h2 to xe-h2, and the parallax yd is zero.

  The pixel value of each pixel (xi, yj) in the reference block 51 is Img1 (xi, yj), and each pixel (xi + xd, yj + yd) in the reference block 52 in which the parallaxes in the X and Y directions are parallaxes xd and yd Assuming that the pixel value of Img2 (xi + xd, yj + yd), the correlation value CORp between the image in the reference block 51 and the image in the reference block 52 is calculated by the calculation using the equations (1) and (2).

  While changing the parallax xd in the X direction in the range of xs−h2 to xe−h2 so that the reference block is scanned from 52s to 52e, the correlation value CORp for each parallax xd according to the equations (1) and (2) Is calculated, the relationship between the X coordinate of the second image 2 and the correlation value CORp is clarified as in the graph U1 shown in FIG.

  In the example of FIG. 9, since the maximum value of the correlation value CORp in the graph U1 is given at the coordinate xM, the reference block whose X coordinate of the search reference point matches xM is the reference that gives the maximum value of the correlation value CORp. The search reference point of the reference block matches the candidate point 23 corresponding to the point of interest 21.

  FIG. 10 is a diagram illustrating a scanning mode of the reference block 52 in the SAD method, and the reference block 52 is scanned over the entire area of the second image 2. Since the SAD method has a smaller amount of computation and can perform high-speed processing than the POC method, the candidate point extraction processing by the SAD method is applied to the second image 2 having a large number of pixels, and the entire area scanning shown in FIG. 10 is performed. However, the processing time is shortened.

  Therefore, when using the SAD method, a method is used in which frequency analysis is performed even if parallax is specified by scanning a reference block over the entire image without setting a search reference point 22 close to the candidate point 23. Since the candidate point extraction process can be performed in a shorter time than the above, the processing cost for obtaining the search reference point 22 can be reduced.

  In addition to the SAD method, for example, the SSD (concentration difference sum of squares) method or the like may be adopted as the differential search method without impairing the usefulness of the present invention.

○ Candidate point extraction processing by first block matching using the POC method:
FIG. 11 is a diagram illustrating an example of a procedure of candidate point extraction processing (candidate point search processing) by first block matching using the POC method.

  The POC method is a correlation method using frequency analysis and a correlation method in which the amplitude component is suppressed. As a similar correlation method, there is known a DCT code limited correlation method (reference paper: “Fusion of image signal processing and image pattern recognition—DCT code limited correlation and its application”, Hitoshi Takatsuka). Since these correlation methods perform similarity calculation based on the signal of only the phase component with the amplitude component suppressed from the frequency analysis signal of the pattern, the difference in imaging conditions (here, the stereo camera 300 for acquiring an image) The difference between the shooting conditions of the first camera 91 and the second camera 92) and the influence of noise and the like, and robust candidate point search is possible.

  Further, since the SAD method outputs only one correlation value (CORp value) for one set of standard block and reference block, the reference block having the highest correlation value, that is, the parallax having the highest correlation value is specified. For this purpose, it is necessary to scan the reference block over a plurality of reference blocks, whereas the correlation method using the frequency analysis such as the POC method is performed on the image of one set of the reference block and the reference block. Since the parallax between the images having the highest correlation value is obtained together with the correlation value between the images, it is not necessary to scan the reference block when the candidate point exists in the reference block.

  In the following, an example in which a candidate point is obtained for a point of interest using an image in the standard block 51 for the first image 1 and an image in the reference block 52 for the second image 2 as candidates for candidate point extraction processing will be described. The basic principle of the POC method will be described.

  First, a reference block setting process T1a for setting the reference block 51 for the first image 1 and a reference block setting process T1b for setting the reference block 52 for the second image 2 are performed.

  Assume that the image area in the reference block 51 and the image area in the reference block 52 are expressed by the following equations (3) and (4).

Here shows a (3) of f0 (n _1, n ₂₎ and (4) of g0 (n _1, n _2), the pixel value of the pixel of the reference block and the reference block 52 51 . N ₁ and N ₂ are set as N ₁ = 2M ₁ +1 and N ₂ = 2M ₂ +1, for example.

  Next, before performing frequency analysis, window function processing T2a and T2b using a window function is performed in order to remove the influence of discontinuity at the end of the block region.

As the window function, for example, a Hanning window given by H (n ₁ , n ₂ ) in the equation (5) is adopted, and the window function processes T2a and T2b are the pixels of the pixels in the base block 51 and the reference block 52, respectively. The values are given by equations (6) and (7), respectively, which multiply the value by the output value of the window function of equation (5).

  As the window function, for example, a Hamming window, a Kaiser window, or the like is employed in addition to the Hanning window.

  Next, for each image region in the standard block 51 and the reference block 52, the two-dimensional Fourier transform processes T3a and T3b shown in FIG. 11 are performed using the arithmetic expressions shown in the expressions (8) and (9). Done.

Note that N ₁ and N ₂ are substituted for the subscript P of W in the lower proviso of the equations (8) and (9), and ₁ and ₂ are substituted for the subscript s of k. Here, the frequency analysis process does not necessarily need to be a Fourier transform, and may include any of a discrete cosine transform, a discrete sine transform, a wavelet transform, or a Hadamard transform.

  For each image region subjected to such Fourier transform processing T3a and T3b, normalization processing T4a and T4b for removing the amplitude component of the image is performed using the arithmetic expression shown in equation (10). Is called.

  When the normalization processes T4a and T4b are completed, a normalized cross power spectrum calculation process T5 using an arithmetic expression represented by the expression (11) is performed.

  When the normalized cross power spectrum calculation process T5 is completed, a two-dimensional inverse Fourier transform process T6 is performed using an arithmetic expression represented by the expression (12). Thereby, the correlation calculation between the images in the standard block 51 and the reference block 52 is performed, and the result (POC value) is output.

  Through the above processing, a calculation result (POC value) indicating the correlation between the image in the standard block 51 and the image in the reference block 52 is obtained. For example, a result (POC value) as shown in FIG. 12 is obtained.

FIG. 12 is a diagram illustrating an example of the POC value. In FIG. 12, the POC value at a location where the correlation is high in the block (N ₁ × N ₂ ) is large. The coordinate giving the peak S of the POC value indicates the parallax that gives the highest correlation between the image in the standard block 51 and the image in the reference block 52.

  In the candidate point extraction process using the POC method, the reference corresponding to the attention point 21 is performed by the candidate point extraction process T7 based on the parallax specified using the POC method and the position of the attention point 21 in the reference block 51. Candidate points 23 in block 52 are extracted.

  As described above, according to the candidate point extraction process using the POC method, the amplitude component of the image is removed, and the correlation calculation is performed based only on the phase component of the image. The candidate points are detected with high accuracy.

  In the image processing apparatus 200A, even if the attention point 21, the candidate point 23, the corresponding point 25, and the like are specified in units of pixel size, or specified in units of subpixels equal to or less than the pixel size, It does not detract from the usefulness of the invention. In each drawing of the present application, drawings corresponding to cases where attention points, candidate points, corresponding points, and the like are specified (extracted) in units of pixels are appropriately described.

<(2-8) Step S80: Second Reference Block Setting Step>
When the plurality of candidate points 23 are extracted, the second setting unit 17 sets the second reference blocks 53 that spatially include the plurality of candidate points 23 based on the plurality of candidate point information 33 to the second It is set on the image 2 (step S80 in FIG. 25), and the second reference block information 63 (FIG. 2) that is the area information of the set second reference block 53 is acquired. The plurality of acquired second reference block information 63 is supplied to the second calculation unit 18.

  More specifically, the second setting unit 17 applies the second rule to each of the plurality of candidate points 23 with each of the plurality of candidate points 23 as a spatial reference point. Two reference blocks 53 are set. The second rule information 58 (FIGS. 1 and 2) defining the second rule includes, for example, the shape of the block with respect to a spatial reference point in the image space, the spatial size, and the arrangement of the block with respect to the reference point. Each parameter such as information is included. The parameters can be changed as appropriate, and the parameters at the initial setting can be changed as appropriate.

  FIG. 8 is a diagram illustrating a setting example of the second reference block 53 and the second reference block 54. As shown in FIG. 8, the second setting unit 17 applies the second rule defined by the first rule information 58 with the candidate point 23 as a spatial reference point to thereby apply the second reference block 53 to the second reference block 53. Set on 2 images 2 (FIG. 3). Note that the second rule relating to FIG. 8 is the second rule corresponding to the parameters at the time of initial setting.

  As shown in the setting examples of FIGS. 7 and 8, the first rule and the second rule are spatially different from each other in the block setting mode with respect to a spatial reference point in the initial setting mode. However, even if the setting mode of the block with respect to the spatial reference point of the first rule and the second rule is used spatially equal to each other according to the change of the operation mode, the usefulness of the present invention is reduced. There is no loss.

  Next, various setting examples of the second reference block 53 will be described. 13 to 20 are diagrams illustrating examples of setting the second reference block 53, respectively. The first reference block 52 in FIGS. 13 to 20 is one of the plurality of first reference blocks 52 (FIG. 3) used for extracting the plurality of candidate points 23. The first reference blocks 52 in FIGS. 13 to 20 are set according to a common first rule.

  Further, in the first reference block 52 (FIGS. 13 to 20), the candidate point of interest 23S and the candidate point 23c are included as a part of the plurality of candidate points 23 extracted on the second image 2. Yes. The candidate point of interest 23S (FIGS. 13 to 20) is located at the center of the first reference block 52 (FIGS. 13 to 20) by the first block matching for the first reference block 52 (FIGS. 13 to 20). The extracted candidate points. The first reference block 51 (not shown) on the first image 1 subjected to the first block matching together with each of the first reference blocks 52 (FIGS. 13 to 20) is also set by the first rule. ing. The attention candidate point 23S (FIGS. 13 to 20) is a candidate point used as a spatial reference point when the second reference block 53 (FIGS. 13 to 20) is set.

  The candidate points 23c (FIGS. 13 to 20) are candidate points extracted by the first block matching for the first reference block 52 other than the first reference block 52 (FIGS. 13 to 20), respectively. It is.

  Note that the equidistant line 83 shown in FIGS. 13 to 16 is a line that is equidistant from each of the target candidate point 23S and the candidate point 23c in the image space of the second image 2.

  17 is a graph showing the distribution of the first index value 71 in the X-axis direction at a predetermined Y coordinate in the image space included in the first reference block 52 (FIG. 17). is there. Even if the predetermined Y coordinate is changed within the range of the first reference block 52 (FIG. 17), the graph V1 with respect to the changed Y coordinate is also a graph having substantially the same shape as the graph V1 (FIG. 17). Become.

  As shown in the graph V1, at the X coordinates xa and xc, the first index value 71 is a maximum value, and the first index value 71 at the coordinate xa is larger than the first index value 71 at the coordinate xc. ing. Further, the first index value 71 at the coordinate xb between the coordinates xa and xc is a local minimum value. The coordinate xb is the X coordinate of the boundary region 84 (FIGS. 17 to 20). Therefore, the boundary region 84 (FIGS. 17 to 20) is a region where the distribution of the first index value 71 in the X-axis direction is a minimum value.

  As the second rule used for setting the second reference block 53 (FIGS. 13 and 17), a rule in which the block setting mode with respect to the spatial reference point is equal to the first rule is adopted. Accordingly, the first reference block 52 (FIGS. 3 and 17) and the second reference block 53 (FIGS. 3 and 17) have the same spatial setting mode with respect to the candidate point of interest 23S. The second reference block 53 (FIGS. 3 and 17) also includes two candidate points 23, ie, the candidate point of interest 23S and the candidate point 23c.

  Therefore, when the second reference block 53 (FIGS. 13 and 17) is used for the second block matching, the second index value 72 corresponding to the target candidate point 23S is obtained by being influenced by the candidate point 23c. There is a possibility that an error caused by the candidate point 23 is included.

  Therefore, in the setting examples of the second reference block 53 shown in FIGS. 14 to 16 and FIGS. 18 to 20, the attention candidate point 23S is used as a spatial reference point and the second reference block 53 is set to the second reference block 53S. Each parameter of the second rule is adjusted so that only the target candidate point 23S among the plurality of candidate points 23 is included in the second reference block 53 set by applying the two rules. .

  For example, the second reference block 53 (FIGS. 14 to 16) includes a plurality of candidate points 23 so that the second reference block 53 includes only the candidate point of interest 23S among the plurality of candidate points 23. Are set according to the second rule for setting the second reference block according to the positional relationship. Specifically, in the first reference block 52 (FIGS. 14 to 16), the second reference block 53 (FIG. 14) is not included so that the region on the candidate point 23c side with respect to the equidistant line 83 is not included. To FIG. 16) are set.

  More specifically, the second reference block 53 (FIG. 14) is set to have a smaller spatial size than the first reference block 52 (FIG. 14) corresponding to the first rule. Further, the shape of the second reference block 53 (FIG. 15) is modified with respect to the first reference block 52 (FIG. 15), and the second reference block 53 (FIG. 16) is changed to the first reference block 52 (FIG. 16). ) Has been changed. Due to these changes, the second reference block 53 (FIGS. 14 to 16) includes a region on the candidate point 23c side of the equidistant line 83 in the first reference block 52 (FIGS. 14 to 16). Not included. Therefore, it is possible to reduce the possibility that the second index value 72 for each of the attention candidate points 23S (FIGS. 14 to 16) includes an error caused by another candidate point 23 such as the candidate point 23c. Further, if the second rule is a rule for setting the second reference block according to the mutual positional relationship between the plurality of candidate points 23, the second reference block 53 can be calculated by simple calculation based on the positional relationship. You can adjust the settings.

  Even if the second rule for setting the second reference block according to the mutual positional relationship between the plurality of candidate points 23 is not adopted, the block setting for the spatial reference point between the first rule and the second rule is performed. If the modes are spatially different from each other, it is possible to reduce the possibility that an error caused by another candidate point 23 is included, as in the examples of FIGS. Therefore, if the block setting mode with respect to the spatial reference point of the first rule and the second rule is spatially different from each other, the second rule is determined according to the mutual positional relationship between the plurality of candidate points 23. Even if it is not a rule for setting the second reference block, it does not impair the usefulness of the present invention.

  Further, for example, the second reference block 53 (FIGS. 18 to 20) includes the second image 2 so that only the attention candidate point 23S among the plurality of candidate points 23 is included in the second reference block 53. These are set by the second rule for setting the second reference block according to the spatial distribution state of the plurality of first index values 71 in the image space. Specifically, in the first reference block 52 (FIGS. 18 to 20), the second reference block 53 (FIGS. 18 to 20) is not included so as not to include the region on the candidate point 23c side with respect to the boundary region 84. FIG. 20) is set.

  More specifically, the spatial size of the second reference block 53 (FIG. 18) is set smaller than that of the first reference block 52 (FIG. 18) corresponding to the first rule. Further, the shape of the second reference block 53 (FIG. 19) is modified with respect to the first reference block 52 (FIG. 19), and the second reference block 53 (FIG. 20) is changed to the first reference block 52 (FIG. 20). ) Has been changed. With these changes, the second reference block 53 (FIGS. 18 to 20) includes an area on the candidate point 23c side of the boundary area 84 in the first reference block 52 (FIGS. 18 to 20). Not. Therefore, it is possible to reduce the possibility that the second index value 72 for each of the attention candidate points 23S (FIGS. 18 to 20) includes an error due to another candidate point 23 such as the candidate point 23c.

  Even if the second rule for setting the second reference block according to the spatial distribution state of the plurality of first index values 71 in the image space of the second image 2 is not adopted, the first rule and the second rule are used. If the block setting mode with respect to the spatial reference point is spatially different from each other, it is possible to reduce the possibility that an error caused by another candidate point 23 is included as in the examples of FIGS. obtain. Therefore, if the setting mode of the block with respect to the spatial reference point of the first rule and the second rule is spatially different from each other, the second rule is determined by the plurality of first indices in the image space of the second image 2. Even if it is not a rule for setting the second reference block according to the spatial distribution state of the value 71, it does not impair the usefulness of the present invention.

  If the second rule is a rule for setting the second reference block according to the spatial distribution state of the plurality of first index values 71 in the image space of the second image 2, the second rule is the first rule. Compared to the rule that sets the second reference block according to the spatial distribution state of the plurality of first index values 71 in the image space of the two images 2, the distribution state of the first index value 71 is more direct. It is possible to set the second reference block 53 corresponding to.

<(2-9) Step S90: Second Reference Block Setting Step>
When a plurality of second reference blocks 53 are set, the second setting unit 17 uses one or more second reference blocks on the first image 1 by appropriately using the attention point information 31 according to the operation mode. 54 is set (step S90 in FIG. 25), and for each of the set second reference blocks 54, one or more second reference block information 64 (FIG. 2) as region information is acquired. The acquired one or more pieces of second reference block information 64 are supplied to the second calculation unit 18.

  In the example shown in FIG. 8, the second setting unit 17 uses the attention point 21 as a spatial reference point and applies the second rule defined by the second rule information 58, for example, to thereby apply the second reference block. 54 is set on the first image 1 (FIG. 3).

<(2-10) Step S100: Second Block Matching Step>
When a plurality of second reference blocks 53 and one or more second reference blocks 54 are set, the second calculation unit 18 is set in advance on each of the plurality of second reference blocks 53 and on the first image 1. Second block matching with the second reference block 54 is performed (step S100 in FIG. 25). Similar to the first block matching, the second block matching is performed by block matching (pattern matching) processing using a correlation calculation method such as the SAD method. Even if the correlation calculation method used for the first block matching and the correlation calculation method used for the second block matching are different from each other, the usefulness of the present invention is not impaired.

<(2-11) Step S110: Second Index Value Acquisition Step>
Next, the second calculation unit 18 uses a plurality of second index values 72 (see FIG. 5) expressing suitability for selection as the corresponding points 25 for each of the plurality of candidate points 23 based on the calculation result of the second block matching. 2) is acquired (step S110 in FIG. 25). The plurality of acquired second index values 72 are supplied to the determination unit 19. Steps S100 and S110 function as a second calculation step.

  The second calculation unit 18 includes at least a first image 1, a second image 2, a plurality of second reference block information 63, and one or more second reference blocks among the information supplied from the other functional units. A plurality of second index values 72 are acquired based on the reference block information 64.

  21 to 23 are diagrams for explaining an example of the second index value 72 (FIG. 2) acquired by the second block matching process.

  In FIG. 21, second reference blocks 53a and 53b are set with the candidate points 23a and 23b extracted on the second image 2 as spatial reference points. The second reference blocks 54a and 54b set on the first image 1 are set in advance by the second setting unit 17 while being shifted in position along a plurality of arrows Y1 on the first image 1. These are some second reference blocks among the plurality of second reference blocks 54.

  In the example of FIG. 21, the second calculation unit 18 performs block matching between each of the plurality of second reference blocks 53a and 53b and the second reference block 54, and sets the position of the second reference block 54 to the first. The second block matching is performed by shifting on the image 1. Specifically, the second calculation unit 18 includes the second block among the plurality of second reference blocks 54 set in advance along the plurality of arrows Y1 on the first image 1 by the second setting unit 17. The second block matching is performed while shifting the position of the second reference block 54 to be matched.

  By the second block matching, the second calculation unit 18 uses the second reference block among the plurality of second reference blocks 54 set along the plurality of arrows Y1 on the first image 1 by the second setting unit 17. The second reference block 54a having the highest similarity with 53a is specified. The second calculation unit 18 acquires the similarity as the second index value 72a (FIG. 21). Similarly, the second calculation unit 18 identifies the second reference block 54b having the highest similarity with the second reference block 53b among the plurality of second reference blocks 54. Then, the second calculation unit 18 acquires the similarity as the second index value 72b (FIG. 21). That is, the second calculation unit 18 has the highest similarity with the second reference block on the second image 2 for each of the plurality of second reference blocks 53a and 53b based on the result of the second block matching. And a plurality of second index values 72a and 72b are acquired based on the acquired similarities. The second index value 72a is larger than the second index value 72b, and the candidate point 23a is determined as the corresponding point 25.

  Moreover, the 2nd calculating part 18 can acquire the 2nd index value 72 different from 2nd index value 72a, 72b according to an operation mode. Specifically, the second calculation unit 18 first, based on the identified second reference blocks 54a and 54b (FIG. 21), a plurality of points on the first image 1 respectively corresponding to the plurality of candidate points 23a and 23b. Corresponding points (also referred to as “reference corresponding points”) 24a and 24b are specified. Then, the second calculation unit 18 determines the distances between the attention point 21 (FIG. 21) on the first image 1 and the reference corresponding points 24 a and 24 b, as a plurality of second index values 72 c and 72 d (FIG. 21), respectively. The second index values 72c and 72d are the second index values 72 corresponding to the candidate points 23a and 23b, respectively. The second index value 72c is smaller than the second index value 72d, and the candidate point 23a is determined as the corresponding point 25.

  FIG. 22 is a diagram for explaining the second index value 72 acquired by a method different from the acquisition of the second index value 72 described above with reference to FIG. In FIG. 22, as in the setting example of FIG. 21, second reference blocks 53a and 53b are set on the second image 2, and candidate points 23a and 23b are extracted, respectively. The second reference block 54 (FIG. 22) is one second reference block set by the second setting unit 17 with the point of interest 21 (FIG. 22) as a spatial reference point.

  The second computing unit 18 performs second block matching by block matching between each of the plurality of second reference blocks 53a and 53b (FIG. 22) and the second reference block 54 (FIG. 22). And the 2nd calculating part 18 acquires each similarity of each of several 2nd reference | standard block 53a, 53b and the 2nd reference block 54 as 2nd index value 72e, 72f. The second index values 72e and 72f are second index values 72 corresponding to the candidate points 23a and 23b, respectively. The second index value 72e is larger than the second index value 72f, and the candidate point 23a is determined as the corresponding point 25.

  FIG. 23 is a diagram for explaining the second index value 72 acquired by a method different from the acquisition of the second index value 72 described above with reference to FIGS. 21 and 22. In FIG. 23, as in the setting example of FIG. 21, second reference blocks 53a and 53b are set on the second image 2, and candidate points 23a and 23b are extracted, respectively. Note that the second reference block 53a and the second reference block 53b are set in advance by the second setting unit 17 as in the setting examples described above with reference to FIGS. That is, the second reference block 53a (53b) is set by applying the second rule to the candidate point 23 (23b) with the candidate point 23a (23b) as a spatial reference point (target candidate point). Has been. The second reference block 53a (53b) includes only the candidate point 23a (23b) among the candidate points. Therefore, the second rule corresponding to the candidate point 23a and the second rule corresponding to the candidate point 23b have different parameters.

  In addition, the second setting unit 17 applies the second rule corresponding to the candidate point 23a and the second rule corresponding to the candidate point 23b to the attention point 21 with the attention point 21 as a spatial reference point. Thus, the second reference blocks 54a and 54b are set in advance on the first image 1 for each of the plurality of candidate points 23a and 23b.

  The second computing unit 18 includes a second reference block 54a (54b) corresponding to one candidate point 23a (23b) among the plurality of candidate points 23a and 23b, and one of the plurality of second reference blocks 53a and 53b. Block matching with the second reference block 53a (53b) corresponding to the candidate point 23a (23b) is performed. Therefore, the second calculation unit 18 performs the second block matching by performing the block matching while changing the candidate point set as one candidate point among the plurality of candidate points 23a and 23b.

  Then, based on the result of the second block matching, the second calculation unit 18 determines each similarity with the corresponding second reference block 54a, 54b for each of the plurality of second reference blocks 53a, 53b. Have acquired. The second calculation unit 18 acquires the acquired similarities as second index values 72g and 72h, respectively. The second index values 72g and 72h are the second index values 72 corresponding to the candidate points 23a and 23b, respectively. The second index value 72g is larger than the second index value 72h, and the candidate point 23a is determined as the corresponding point 25.

<(2-12) Step S120: Corresponding Point Determination Step>
When the plurality of second index values 72 are acquired, the determination unit 19 selects one of the plurality of candidate points 23 as a corresponding point based on the plurality of candidate point information 33 and the plurality of second index values 72. By selecting as 25, the corresponding point 25 is determined (step S120 in FIG. 25). The determination unit 19 acquires the determined coordinates of the corresponding point 25 as the corresponding point information 35 and stores it in the storage device 46. Step S120 functions as a determination process.

  As described above, according to the image processing apparatus 200A, the plurality of candidate points 23 of the corresponding points 25 corresponding to the point of interest 21 on the first image 1 are extracted in the second image 2 by the first block matching. . Then, based on the result of the second block matching between the plurality of second reference blocks 53 that spatially contain the plurality of candidate points 23 and the second reference block 54 on the first image 1, the plurality of candidates A corresponding point 25 is determined from the points 23. Therefore, the reliability as the corresponding point of the determined corresponding point 25 can be improved.

<About modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications can be made.

  For example, the image processing system 100A described above is configured by the image processing apparatus 200A in the image processing system 100A being executed by a general-purpose computer, but the image processing system 100A is replaced with the configuration. However, for example, it may be realized as a system including a stereo camera 300 and an image processing device 200A in one device such as a digital camera, a digital video camera, or a form information terminal.

100A image processing system 200A image processing apparatus 300 stereo camera 1 first image 2 second image 31 attention point information 33 candidate point information 35 corresponding point information 51 first reference block 52 first reference block 53 second reference block 54 second reference Block 61 First reference block information 62 First reference block information 63 Second reference block information 64 Second reference block information 71 First index value 72 Second index value 91 First camera 92 Second camera

Claims (14)

An acquisition unit for acquiring a first image and a second image including image contents corresponding to each other;
A first setting unit that sets a plurality of first reference blocks on the second image;
The first reference block and the plurality of the plurality of first reference blocks by first block matching between the first reference block and the plurality of first reference blocks spatially including the attention point set in advance on the first image. A first calculation unit that acquires a plurality of first index values each representing the similarity of the image content with each of the first reference blocks;
An extraction unit that extracts a plurality of candidate points of corresponding points corresponding to the attention point based on the plurality of first index values, respectively, in the second image;
A second setting unit that sets a plurality of second reference blocks that spatially include only one point of each of the plurality of candidate points;
The second block matching between each of the plurality of second reference blocks and a second reference block set in advance on the first image makes it possible to select each of the plurality of candidate points as the corresponding point. A second calculation unit that acquires a plurality of second index values expressing
A determination unit that determines the corresponding points by selecting one of the plurality of candidate points based on the plurality of second index values;
An image processing apparatus comprising: An image processing apparatus according to claim 1,
The first setting unit includes:
A plurality of search reference points set in advance on the second image are used as spatial reference points, and a first rule relating to block setting in the image space is applied to each of the plurality of search reference points. By setting each of the plurality of first reference blocks,
The second setting unit includes:
The plurality of second reference blocks are respectively applied by applying a second rule relating to block setting in the image space to each of the plurality of candidate points, each of the plurality of candidate points being a spatial reference point. Set,
The first rule and the second rule are:
An image processing apparatus characterized in that block setting modes with respect to a spatial reference point are spatially different from each other. An image processing apparatus according to claim 2,
The first rule and the second rule are:
An image processing apparatus, wherein two blocks set by applying the first rule and the second rule to a common spatial reference point are different from each other. An image processing apparatus according to claim 2,
The first rule and the second rule are:
An image in which two blocks set by applying the first rule and the second rule to a common spatial reference point have different spatial sizes. Processing equipment. An image processing apparatus according to claim 2,
The first rule and the second rule are:
An image processing apparatus, wherein positions of two blocks set by applying the first rule and the second rule with respect to a common spatial reference point are different from each other. An image processing apparatus according to any one of claims 2 to 5, wherein
The second rule is:
The plurality of candidate points in the second reference block set by applying the second rule to the candidate point of interest with the candidate point of interest as a spatial reference point among the plurality of candidate points An image processing apparatus characterized by a rule for setting the second reference block so that only the attention candidate point is included. An image processing apparatus according to claim 6,
The second rule is:
The plurality of candidates so that only the attention candidate point is included in the second reference block set by applying the second rule to the attention candidate point. An image processing apparatus characterized by a rule for setting the second reference block according to a mutual positional relationship between points. An image processing apparatus according to claim 6,
The second rule is:
The second image so that only the attention candidate point is included in the second reference block set by applying the second rule to the attention candidate point. An image processing apparatus characterized in that the second reference block is set according to a spatial distribution state of the plurality of first index values in the image space. An image processing apparatus according to any one of claims 1 to 8, comprising:
The second arithmetic unit is
Performing the second block matching by performing block matching between each of the plurality of second reference blocks and the second reference block while shifting the position of the second reference block on the first image;
A plurality of corresponding points respectively corresponding to the plurality of candidate points are specified on the first image by the second block matching, and based on distances between the attention point and the corresponding points. An image processing apparatus, wherein each of the plurality of second index values is acquired. An image processing apparatus according to any one of claims 1 to 8, comprising:
The second arithmetic unit is
Performing the second block matching by performing block matching between each of the plurality of second reference blocks and the second reference block while shifting the position of the second reference block on the first image;
Based on the result of the second block matching, for each of the plurality of second reference blocks, obtain the highest similarity with the second reference block,
The image processing apparatus, wherein the plurality of second index values are respectively acquired based on the acquired similarities. An image processing apparatus according to any one of claims 1 to 8, comprising:
The second reference block is one block in which the attention point is set as a spatial reference point;
The second arithmetic unit is
The second block matching is performed by block matching between each of the plurality of second reference blocks and the second reference block, and each similarity between each of the plurality of second reference blocks and the second reference block is determined. An image processing apparatus that acquires each of the plurality of second index values based on the image data. An image processing apparatus according to any one of claims 6 to 8, wherein
The second reference block is
The plurality of candidate points by applying each of the second rules when each of the plurality of candidate points is the target candidate point, with the target point as a spatial reference point, respectively. Are preset on the first image for each of the
The second arithmetic unit is
Block matching between the second reference block corresponding to one candidate point among the plurality of candidate points and the block corresponding to the one candidate point among the plurality of second reference blocks is performed by the plurality of candidate points. Performing the second block matching by changing the candidate point set as the one candidate point in
Based on the result of the second block matching, for each of the plurality of second reference blocks, obtain the similarity with the corresponding second reference block,
The image processing apparatus, wherein the plurality of second index values are respectively acquired based on the acquired similarities. By being executed on a computer mounted on the image processing apparatus,
A program for causing the image processing apparatus to function as the image processing apparatus according to any one of claims 1 to 12. A first reference block that spatially encloses a point of interest set in advance on the first image, and a plurality of first reference blocks set on a second image including image contents corresponding to the first image. A first calculation step of acquiring a plurality of first index values respectively representing similarity of image contents between the first reference block and each of the plurality of first reference blocks by first block matching with When,
An extracting step of extracting a plurality of candidate points of corresponding points corresponding to the attention point in the second image based on the plurality of first index values;
By second block matching between each of a plurality of second reference blocks that spatially include only one point among the plurality of candidate points, and a second reference block preset on the first image A second calculation step of obtaining a plurality of second index values expressing suitability for selection as the corresponding points for each of the plurality of candidate points;
A determination step of determining the corresponding points by selecting one of the plurality of candidate points based on the plurality of second index values;
An image processing method comprising:

Images