JP2012198712A - Image processing system, image processing method and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system estimating a corresponding point of a sub-pixel level with high accuracy.SOLUTION: The image processing system includes: block shape setting means for setting a bock shape for performing a matching block by using one of two images as a criterion image and the other as a reference image; criterion image block position setting means for setting a block position in the criterion image based on the block shape set by the block shape setting means; reference image block position setting means for setting a plurality of blocks in the vicinity of a position corresponding to the block position set by the criterion image block position setting means in the reference image; and an evaluation value arithmetic means for calculating a correlation evaluation value between the block set in the criterion image and the plurality of blocks set by the reference image block position setting means. The block shape setting means sets the block shape according to the shape of an object included in the criterion image.

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

  In recent years, for example, as a method for acquiring a three-dimensional shape from an image photographed by a camera, a method using two images photographed from different viewpoints like a stereo camera is known. In this method, two images are associated with each other, parallax is obtained from points corresponding to each other, and distance information to the subject is acquired. As a method for searching for corresponding points, for example, a block including a point of interest is set for a point of interest on one base image, and a plurality of blocks of the same size are set on the other reference image in units of pixels. To do. Correlation values are calculated between the block on the reference image set in this way and each block on the reference image, and the corresponding point is searched by searching for the block on the reference image having the highest correlation value.

  Here, in order to increase the distance resolution in the three-dimensional image, the resolution of the parallax may be increased. In order to increase the resolution of the parallax, it is only necessary to perform the association with an accuracy finer than the pixel level. Therefore, for example, pixel association at the sub-pixel level is performed. Note that the pixel association at the sub-pixel level is performed by, for example, a method of estimating the corresponding point at the sub-pixel level by performing processing (fitting processing) for applying the correlation value obtained at the pixel level to the sub-pixel estimation model. It has been.

  Conventionally, in order to estimate corresponding points with sub-pixel accuracy, the correlation value between pixels is applied to a linear equation or a curved equation based on the positional relationship between the position having the highest correlation value and the surrounding correlation values. There is known a method of interpolating and estimating a peak position and a peak value of a correlation value (for example, see Patent Document 1). A method of selecting a subpixel estimation model based on the peak position and the degree of coincidence between the correlation value around the peak position and the subpixel estimation model is known (see, for example, Patent Document 2). Also, a function of a subpixel estimation model that takes the sum of a polygonal line and a parabola to estimate corresponding points at the subpixel level using four correlation values is known (see, for example, Non-Patent Document 1).

  Furthermore, the variance and average value of parallax are obtained by MAP estimation assuming that the distribution of parallax, noise, and image differential in the window is a normal distribution, and the window that minimizes the parallax variance is determined as the window for performing the correlation operation. The method is known (for example, refer nonpatent literature 2). In addition, a parallax calculation method is known in which resolution is improved in the vicinity of disparity by setting an adaptive window reflecting discontinuity of disparity (see, for example, Patent Document 3). In addition, in order to improve the detection accuracy of subpixel parallax, a compound eye imaging device that adds correlation values obtained from a plurality of stereo images is known (see, for example, Patent Document 4).

  By the way, since the stereo camera described above is configured by using, for example, two cameras, the shape of the stereo camera becomes large, and there may be a case where the stereo camera cannot be installed at a desired location. Therefore, for example, an ultra-small stereo camera that uses a lens array to acquire images from a plurality of viewpoints with one image sensor and obtains distance information from the plurality of images has been studied. In addition, for example, it has been studied that two image sensors are formed adjacent to each other on a silicon wafer and a lens array is mounted thereon to obtain images with different viewpoints from the respective sensors.

  On the other hand, unlike the conventional stereo camera, the ultra-small stereo camera described above has a very short base length (distance between lenses). For example, even when the base line length of a conventional stereo camera using two cameras is about 10 cm, it is about several millimeters with an ultra-small stereo camera. Here, when the baseline length is small as a distance measurement principle using a stereo camera, the detectable parallax is also small. That is, when a parallax that can be detected by a stereo camera having a base length of about 10 cm when a subject at a predetermined distance is photographed is about 50 pixels, for example, in the case of a stereo camera having a base length of about 2 mm, Assuming that the focal length does not change, the detectable parallax is about one pixel. The degree of the parallax distance measurement accuracy given to the sub-pixel level is about 1% in the case of the former camera, but is almost 100% in the case of the latter camera. That is, in the case of an ultra-small stereo camera with a small baseline length, the distance measurement accuracy greatly depends on the sub-pixel level parallax detection (corresponding point marking) accuracy.

  Therefore, the above-described sub-pixel level parallax detection accuracy is not a problem in a conventional stereo camera, but is an important problem in a miniature stereo camera. Here, the problem given by the sub-pixel level parallax detection accuracy described above will be described with reference to FIGS. 1 and 2.

  FIG. 1 is a diagram for explaining a method of performing block matching using a standard image obtained by photographing an object at a predetermined distance and a reference image. FIG. 2 is a diagram for explaining a case where the block set as the reference image is shifted by a predetermined pixel with respect to the block of the base image.

  The above-described method for obtaining a correlation value for each block in order to search for corresponding points is called block matching, and a square is usually used as the block shape. When the block shape is square, it is easy to handle software and hardware operations because it is vertically and horizontally symmetrical. By the way, in view of the actual object shape in the world, the shape is often composed of a horizontal line and a vertical line.

  Here, assuming that FIG. 1A is a reference image and FIG. 1B is a reference image, the reference image and the reference image have a one-pixel parallax in the left-right direction. A square block centered on point A is set in the reference image shown in FIG. 1A, and a plurality of blocks having the same shape and size are set in the reference image shown in FIG. Do.

  The block position of the reference image having the maximum correlation value with respect to the block of the standard image shown in FIG. 2A is a block position surrounded by a broken line of the reference image shown in FIG. That is, the point A ′ of the reference image shown in FIG. 2A ′ is a corresponding point in pixel units that is closest to the point A of the standard image shown in FIG.

  The block position of the reference image when the block of the reference image shown in FIG. 2 (A ′) is shifted to the left by one pixel is the block position of the reference image shown in FIG. 2 (B ′). In this case, the left end portion of the block of the reference image shown in FIG. 2 (B ′) has a pixel (image) luminance value that is significantly different from the left end portion of the block of the standard image shown in FIG. 2 (B). In addition to the decrease in correlation corresponding to the difference in pixel luminance value near the center shown in (B) and FIG. 2 (B ′), the correlation further decreases (for example, SAD (Sum of Absolute Difference: corresponding pixel of each block) In the case of the sum of absolute values of the difference of values) or SSD (Sum of Squared Difference: the sum of squares of the difference of the pixel values corresponding to each block), the evaluation value becomes large).

  On the other hand, as shown in FIG. 2C ′, the correlation when the block position of the reference image is shifted to the right by one pixel from the block of the standard image shown in FIG. The difference between the pixel luminance values of the block boundaries between the standard image and the reference image shown in FIG. Therefore, the correlation is reduced by a value corresponding to the difference between the pixel luminance values near the center.

  Here, FIG. 3 is a diagram in which the correlation between the block of the base image and the block of the reference image shown in FIG. 2 is plotted on a graph. 3A to 3C show quadratic curve fitting.

  In FIG. 3A, the horizontal axis represents the search pixel position, and the vertical axis represents the NCC (Normalize Cross Correlation: normalized cross-correlation method: normalizes the product sum of corresponding pixel values of each block) evaluation value. Yes. In FIG. 3B, the horizontal axis indicates the search pixel position, and the vertical axis indicates the SAD evaluation value and the SSD evaluation value. In FIG. 3C, the horizontal axis indicates the search pixel position, and the vertical axis indicates the correlation value.

  In the case of FIG. 3A, the evaluation value becomes maximum when the correlation is maximum, and in the case of FIG. 3B, the evaluation value becomes minimum when the correlation is maximum.

  As described above, since the standard image and the reference image shown in FIG. 2 have a one-pixel parallax, ideally, the correlation between the reference image and the reference image shown in FIG. In the example of C), the pixel is symmetric with respect to the point of the search pixel position 1), and no parallax occurs at the subpixel level.

  On the other hand, in the diagrams of FIGS. 3A and 3B, parallax is generated at the sub-pixel level without being symmetrical with respect to the corresponding point when the correlation is maximum. . That is, when sub-pixel level estimation is performed for the corresponding point search, the sub-pixel value does not become “0” by any method of quadratic curve fitting or equiangular straight line fitting, and a value other than “0”. Will have.

  This is a detection error when subpixel estimation is performed for the corresponding point search, which is a big problem for the above-described ultra-small stereo camera.

  Note that the methods disclosed in Patent Document 1, Patent Document 2, and Non-Patent Document 1 described above describe a method for improving subpixel estimation accuracy. However, in any method, the above-described correlation value is obtained. It does not mention the problems. In addition, in the methods described in Non-Patent Document 2 and Patent Document 3 described above, the shape and size of the block are changed in order to improve the accuracy of matching, but the block is rectangular. The problem that the boundary of the block coincides with the boundary of the subject and affects the above-described correlation is not solved. Further, in the method disclosed in Patent Document 4, the block shape is similarly rectangular.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing apparatus that estimates corresponding points at a subpixel level with high accuracy.

  The present invention is an image processing apparatus for achieving the above object, a block shape setting means for setting a block shape for performing a matching block using one of two images as a standard image and the other as a reference image, and the block Corresponding to the reference image block position setting means for setting the block position in the reference image based on the block shape set by the shape setting means, and the block position set by the reference image block position setting means in the reference image Reference image block position setting means for setting a plurality of blocks in the vicinity of the position to be detected, a block set in the standard image, and a correlation evaluation value between the plurality of blocks set by the reference image block position setting means Evaluation value calculation means for calculating, and the block shape setting means And sets the block-shaped according to the shape of Murrell object.

  The present invention is also an image processing method, wherein a block shape setting procedure for setting a block shape for performing a matching block using one of two images as a standard image and the other as a reference image is set by the block shape setting procedure. A reference image block position setting procedure for setting a block position in the reference image based on the block shape, and a position near the position corresponding to the block position set by the reference image block position setting procedure in the reference image. Reference image block position setting procedure for setting a plurality of blocks, evaluation value calculation for calculating a correlation evaluation value between a block set in the base image and a plurality of blocks set by the reference image block position setting procedure The block shape setting procedure includes a shape of an object included in the reference image. And sets the block shape according to.

  Further, the present invention is an image processing program for causing a computer to function as the above-described image processing apparatus.

  According to the present invention, it is possible to estimate a corresponding point at a subpixel level with high accuracy.

It is a figure for demonstrating the method of performing block matching using the standard image which image | photographed the object in a predetermined distance, and a reference image. It is a figure for demonstrating the case where the block set to the reference image has shifted | deviated the predetermined pixel with respect to the block of a reference | standard image. FIG. 3 is a diagram in which a correlation between a block of a standard image and a block of a reference image shown in FIG. 2 is plotted on a graph. It is a figure which shows the structure of the stereo camera apparatus containing the image processing apparatus which concerns on this embodiment. It is a figure which shows the image image | photographed with the lens array shown in FIG. It is a figure for demonstrating the matching block which concerns on this embodiment. It is a figure for demonstrating the example from which the matching block which concerns on this embodiment has shifted | deviated the predetermined pixel. It is a figure which shows the function structure of the correlation calculating means which concerns on this embodiment. It is a figure which shows an example of the hardware constitutions which implement | achieve the function of the image processing apparatus which concerns on this embodiment. It is a flowchart which shows the flow of the correlation calculation process which concerns on this embodiment. It is a figure for demonstrating the estimation method of the sub pixel using the value obtained by the correlation calculating means. It is a figure for demonstrating the other matching block which concerns on this embodiment.

  Hereinafter, embodiments of the present invention will be described in detail.

<Configuration of stereo camera device>
A stereo camera apparatus including the image processing apparatus according to the present embodiment will be described with reference to FIGS. 4 and 5. FIG. 4 shows a configuration of a stereo camera apparatus including the image processing apparatus according to the present embodiment. FIG. 5 shows an image captured by the lens array.

  As shown in FIG. 4, the stereo camera device 100 according to the present embodiment is, for example, one ultra-small stereo camera, and is configured to include an imaging unit 10 and an image processing device 11.

  The imaging means 10 has a lens array, an image sensor, and a light shielding wall, and images images with different viewpoints on the left and right. The left image L shown in FIG. 5 is an image taken by the right lens B shown in FIG. 4, and the right image R shown in FIG. 5 is an image taken by the left lens C shown in FIG.

  The image processing apparatus 11 is configured to include a shading correction unit 20, a distortion correction unit 30, a correlation calculation unit 40, a subpixel parallax estimation unit 50, and a distance calculation unit 60.

  The shading correction unit 20 performs shading correction on the image acquired from the imaging unit 10. The shading correction means 20 corrects, for example, the difference in brightness between the left and right cameras and the darkness around the image. For example, the shading correction unit 20 may prepare a shading correction table in which a value to be multiplied by the luminance value of the pixel is stored in advance, and correct the brightness of the pixel by multiplying the value of the table by the luminance value.

  The distortion correction unit 30 performs distortion correction on the image subjected to the shading correction by the shading correction unit 20. The distortion correction unit 30 corrects, for example, distortion of each of the left and right images. The distortion correction unit 30 corrects, for example, an image captured with distortion derived from the characteristics of the lens into an image without distortion as captured by a pinhole camera. The distortion correction unit 30 may perform distortion correction using a known method, for example, using a lookup table prepared in advance, and uses a correction polynomial that approximates the distortion state using a higher-order polynomial. The distortion may be removed.

  The correlation calculation means 40 calculates the correlation between the left and right images using block matching for the left and right images whose distortion has been corrected by the distortion correction means 30. At this time, one of the left and right images is set as a standard image, and the other is set as a reference image. In the following example, the correlation calculation means 40 uses the left image L (image taken by the right lens B) shown in FIG. 5 as a reference image and the right image R (image taken by the left lens C). ) As a reference image, the correlation value of the left and right images is calculated while performing block matching. However, the present invention is not limited to this, and the reference image and the reference image may be reversed. A specific correlation calculation method by the correlation calculation means 40 will be described later.

  The subpixel parallax estimation unit 50 estimates the subpixel parallax based on the correlation evaluation value and the extreme value position obtained by the correlation calculation unit 40. For example, the sub-pixel parallax estimation unit 50 receives a correlation evaluation value sequence as an input, and sets a correlation evaluation value that gives a minimum value (for example, SSD, SAD) or a maximum value (NCC) of the parallax evaluation value, and left and right of the evaluation value. A correlation evaluation value at a position shifted by one pixel is acquired.

  The sub-pixel parallax estimation unit 50 estimates the sub-pixel position of the corresponding point from the three evaluation values calculated as described above. For example, when the SAD is used for block matching, the subpixel parallax estimation unit 50 estimates the subpixel position of the corresponding point using equiangular straight line fitting. Further, the sub-pixel parallax estimation means 50 estimates the sub-pixel value of the corresponding point using quadratic curve fitting when SSD or NCC is used. An estimation method using equiangular straight line fitting and quadratic curve fitting will be described later.

  The distance calculation unit 60 calculates the distance to the subject (object) using the subpixel position (subpixel level parallax) of the corresponding point estimated by the subpixel parallax estimation unit 50. The distance calculating means 60 calculates the distance to the subject at the point A described above using, for example, Z = Bf / (D-offset) as an expression for calculating the distance to the subject. Here, Z means the distance to the subject, B means the base length (distance between lenses), f means the focal length, and D means the parallax.

  The distance calculation means 60 calculates the distance for all pixels whose distance can be calculated using the above-described formula. Thereby, a distance image is obtained.

<About the matching block according to this embodiment>
Next, the matching block according to the present embodiment will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram for explaining the matching block according to the present embodiment. FIG. 7 is a diagram for explaining an example in which the matching block according to this embodiment is shifted by a predetermined pixel.

  As shown in FIG. 6A, a point for searching for a corresponding point in the reference image is set as a point A, and a block centered on the point A is set. In the present embodiment, for example, a rhombus is used as an example of the shape of the block, and the size is set with reference to the diagonal of the rhombus. For example, the diagonal is 16 pixels in the horizontal direction and 16 pixels in the vertical direction. Note that the number of pixels is not limited to this.

  Next, as shown in FIG. 6B, a plurality of blocks having the same size and the same shape as the blocks set in FIG. 6A are set on the reference image.

  The correlation calculation means 40 calculates the correlation between the rhombus block centered on the point A of the standard image and a plurality of blocks set on the reference image, and obtains the evaluation value. In the present embodiment, due to the nature of the stereo image, the point corresponding to the point A of the left image L (reference image) is at the same position as the position corresponding to the point A on the right image R (reference image). It exists or exists only on the right side of the position.

  Therefore, as shown in FIG. 6B, the positions of a plurality of blocks of the reference image are to be measured to the right from the position moved several pixels (at least one pixel) to the left on the horizontal line at the same position as the point A. The pixels corresponding to the shortest distance are set by moving one pixel at a time to a position further moved by several pixels (at least one pixel).

  As described above, the block positions of extra several pixels set to the left and right are necessary for each pixel, for example, to estimate the sub-pixel parallax, and the reference image and the reference image have an offset due to lens manufacturing error or the like This is to prepare for the case.

  As a method for calculating the correlation evaluation value by the correlation calculation means 40, SAD, SSD, NCC, or the like that is conventionally used may be used. In the case of the block shape shown in FIG. 6, even when an image as shown in FIG. 1 is taken, a comparison different from the example of FIG. 2 is performed. For example, in FIGS. 7A to 7C, the rhombus blocks are compared with each other, and it is possible to reduce a large difference in pixel luminance values that reduce the correlation near the block end.

  That is, the cause of lowering the correlation at the block end in FIG. 2 described above is greatly reduced, and only the positional deviation near the center is reflected in the correlation, and a desired correlation evaluation value can be obtained. .

<Functional configuration of correlation calculation means>
Next, the functional configuration of the correlation calculation means 40 described above will be described with reference to FIG. FIG. 8 shows a functional configuration of the correlation calculation means according to this embodiment. As shown in FIG. 8, the correlation calculation means 40 includes a block shape setting means 41, a standard image block position setting means 42, a reference image block position setting means 43, an evaluation value calculation means 44, and an extreme value detection means 45. It is comprised so that it may have.

  The block shape setting unit 41 sets, for example, the shape (block shape) and size of a calculation block for performing a matching block using one of the two images as a standard image and the other as a reference image. Here, the block shape setting means 41 sets the block shape according to the shape of the object that is the subject included in the reference image when setting a block centering on a predetermined point for which the corresponding point search is performed in the reference image. It is good to set.

  Specifically, the block shape setting unit 41 sets a shape that does not include a vertical line along the boundary portion of the target object as a block shape. For example, the block shape is a rhombus shape or a hexagon that does not include a vertical line. It may be set to a square shape, a circular shape, an elliptical shape, or the like. For example, when the block shape is a rhombus shape, the block shape setting unit 41 sets a block shape having 16 pixels in the horizontal direction and 16 pixels in the vertical direction with a predetermined point as a center.

  Further, the block shape setting means 41 may set at least one block shape according to the object among a plurality of block shapes having different preset shapes.

  The reference image block position setting unit 42 sets a block position in the reference image based on the block shape set by the block shape setting unit 41.

  The reference image block position setting unit 43 sets a plurality of blocks in the vicinity of the position corresponding to the block position set by the standard image block position setting unit 42 in the reference image. The reference image block position setting means 43 moves several pixels (at least one pixel or more) to the left on the horizontal line at the same position as a predetermined point (point A in the example of FIG. 6), for example, as described in FIG. The setting is made by moving one pixel at a time to a position where several pixels (at least one pixel or more) are moved to the pixel corresponding to the shortest distance to be measured to the right from the position.

  The evaluation value calculation means 44 performs predetermined calculations on the blocks in the reference image set by the reference block position setting means 42 and the plurality of blocks set in the reference image by the reference image block position setting means 43, respectively. Thus, the correlation evaluation value is calculated. The evaluation value calculation means 44 uses the above-described SAD, SSD, NCC, etc. as the predetermined calculation for calculating the correlation evaluation value, and obtains the above-described correlation evaluation value between each block.

  The extreme value detection means 45 detects the extreme value and its position from the plurality of correlation evaluation values obtained by the evaluation value calculation means 44, and outputs the plurality of correlation evaluation values and the position of the extreme value.

<Hardware configuration>
Next, a hardware configuration for realizing the functions of the above-described image processing apparatus 11 will be described with reference to FIG. FIG. 9 shows an example of a hardware configuration that implements the functions of the image processing apparatus according to the present embodiment.

  The image processing apparatus 11 shown in FIG. 9 includes an input device 71, an output device 72, a drive device 73, an auxiliary storage device 74, a memory device 75, and an arithmetic processing unit (CPU: Central Processing Unit) that performs various controls. ) 76, a network connection device 77, and a recording medium 78, which are connected to each other via a system bus B.

  The input device 71 has operation buttons and the like operated by the user, and inputs various operation signals such as execution of a program from the user. The output device 72 has a display or the like for displaying various windows and data necessary for performing the processing in the present invention, and can display the program execution progress, results, and the like by the control program of the CPU 76.

  Here, the execution program installed in the image processing apparatus 11 is provided by the recording medium 78 or the like. The recording medium 78 on which the program is recorded can be set in the drive device 73, and the execution program included in the recording medium 78 is installed from the recording medium 78 to the auxiliary storage device 74 via the drive device 73.

  The recording medium 78 is, for example, a CD-ROM, a recording medium for optically, electrically, or magnetically recording information such as a flexible disk or a magneto-optical disk, a ROM, a flash memory, or the like. Various types of recording media such as a semiconductor memory for electrical recording can be used.

  The auxiliary storage device 74 is a storage means such as a hard disk, and can store an execution program according to the present invention, a control program provided in a computer, various data, etc., and perform input / output as necessary.

  The memory device 75 stores an execution program or the like read from the auxiliary storage device 74 by the CPU 76. The memory device 75 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  The CPU 76 controls processing of the entire computer, such as various operations and input / output of data with each hardware component, based on a control program such as an OS (Operating System) and an execution program stored in the memory device 75. Thus, the above-described functions and correlation calculation processing described later can be realized. Various kinds of information required during the execution of the program can be acquired from the auxiliary storage device 78 and stored.

  The network connection device 77 obtains an execution program from another terminal connected to the communication network by connecting to a communication network or the like, or an execution result obtained by executing the program or the execution program itself in the present invention. Can be provided to other terminals.

  With the hardware configuration as described above, it is possible to realize each function described above and a correlation calculation process described later efficiently at a low cost without requiring a special device configuration. Further, by installing the program, each function described above and each correlation calculation process described later can be easily realized.

<About correlation calculation processing>
Next, the flow of the correlation calculation process by the correlation calculation means 40 described above will be described with reference to FIG. FIG. 10 is a flowchart showing the flow of correlation calculation processing according to the present embodiment.

  As shown in FIG. 10, when the standard image and the reference image corrected by each of the correction units described above are input to the correlation calculation unit 40 (S10), the block shape setting unit 41 determines the block shape used for block matching. Set (S11).

  Next, the reference image block position setting unit 42 sets the block position of the reference image using the block shape set by the block shape setting unit 41 (S12).

  Next, the reference image block position setting unit 43 sets a plurality of block positions in the vicinity of the position corresponding to the block position set by the standard image block position setting unit 41 in the reference image (S13).

  Next, the evaluation value calculation unit 44 determines a block in the reference image set by the reference block position setting unit 42 and a plurality of blocks set in the reference image by the reference image block position setting unit 43, respectively. A correlation evaluation value is calculated by performing an operation (S14).

  Next, it is determined whether or not processing has been performed on all the pixels input by the correlation calculation means 40 (S15). If it is determined that processing has not been performed for all pixels (NO in S15), the process returns to S12, and the same processing is performed for the other pixels.

  If it is determined that processing has been performed for all pixels (YES in S15), extreme value detection means 45 determines the extreme value and its position from a plurality of correlation evaluation values obtained by evaluation value calculation means 44. Is detected (S16), a plurality of correlation evaluation values and their extreme positions are output (S17), and the process is terminated.

<Subpixel estimation method>
Next, a method for estimating a subpixel based on the correlation evaluation value and the extreme value position obtained by the correlation calculation means 40 will be described with reference to FIG. FIG. 11 is a diagram for explaining a subpixel estimation method using values obtained by the correlation calculation means. Note that FIG. 11A is a diagram showing fitting using an equiangular straight line. The horizontal axis indicates the search pixel position, and the vertical axis indicates the SAD evaluation value.

  In order to estimate the subpixel position of the corresponding point from the diagram shown in FIG. 11A, the subpixel parallax estimation unit 50 firstly has P1 having the smallest SAD evaluation value and two SAD evaluation values adjacent to P1 ( A straight line connecting P0 and P0 having a large evaluation value is drawn. Next, a straight line that passes through P2 and is parallel to a straight line obtained by horizontally inverting the straight line connecting P0 and P1 is drawn. Next, the subpixel parallax estimation means 50 estimates the intersection of two straight lines as the subpixel position of the corresponding point.

  FIG. 11B is a diagram showing fitting using a quadratic curve. The horizontal axis indicates the search pixel position, and the vertical axis indicates the SSD evaluation value.

  In order to estimate the subpixel position of the corresponding point from the diagram shown in FIG. 11B, the subpixel parallax estimation means 50 first uses the three points (P0, P1, P2) shown in FIG. A quadratic curve is fitted and the vertex of the quadratic curve is determined. The subpixel parallax estimation means 50 estimates the position of this vertex as the subpixel position of the corresponding point.

  As described above, the sub-pixel parallax estimation means 50 can detect the sub-pixel level parallax (corresponding point).

<Other block shapes>
Next, an example of another matching block according to the present embodiment will be described with reference to FIG. FIG. 12 is a diagram for explaining another matching block according to the present embodiment. As mentioned above, most of the world's landscapes and human-made structures have their boundaries in the vertical direction. effective.

  FIG. 12A shows a case where the block shape is circular. Since the 16 × 16 pixel rhombus block shape described above has a smaller number of pixels used for correlation than a 16 × 16 pixel square and is likely to be affected by noise, for example, a circular block shape is used. Block matching may be performed.

  FIG. 12B shows a case where the block shape is a hexagon. In the case of a hexagonal block shape, for example, the number of pixels close to a square block of 16 × 16 pixels can be secured, so that it is possible to achieve both noise resistance and subpixel estimation accuracy.

  In addition to the block shape described above, for example, a boundary portion of an object such as a subject (by rotating a predetermined angle (for example, 60 ° clockwise rotation)) with reference to the center of a polygon such as a star or a square For example, a shape that is not parallel to the contour shape may be generated.

  In the present embodiment, a plurality of the block shapes described above are stored in advance, and depending on the shape (for example, contour shape or pattern) or the type (for example, person, building, mountain, moon, etc.) of the subject, A shape with a high degree of mismatch may be selected, or the user may be able to select a block shape as appropriate. In the present embodiment, at least one block shape among the plurality of block shapes described above is set, and when a plurality of block shapes are selected, they may be switched according to, for example, the position and time, and are included in the image. It may be changed as the subject to be changed.

  As described above, according to the embodiment of the present invention, an optimum correlation is obtained by performing a corresponding point search using block matching for setting a block having a shape according to a subject in the world, and is highly accurate. It is possible to estimate sub-pixel level corresponding points (sub-pixel level parallax) and perform highly accurate sub-pixel estimation. In addition, the above-described estimation of corresponding points at the sub-pixel level may be used for a distance measuring device using a stereo camera, a three-dimensional shape recognition device, or the like.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims. Can be changed.

DESCRIPTION OF SYMBOLS 10 Imaging means 11 Image processing apparatus 20 Shading correction means 30 Distortion correction means 40 Correlation calculation means 41 Block shape setting means 42 Reference image block position setting means 43 Reference image block position setting means 44 Evaluation value calculation means 45 Extreme value detection means 50 Sub Pixel parallax estimation means 60 Distance calculation means 71 Input device 72 Output device 73 Drive device 74 Auxiliary storage device 75 Memory device 76 Arithmetic processing device 77 Network connection device 78 Recording medium 100 Stereo camera device

JP 2001-195597 A JP 2009-282635 A JP-A-5-256613 Japanese Patent No. 4382156

Motoki Arai, Kazuhiko Sumi, Takashi Matsuyama, “Optimization of correlation function and subpixel estimation method in image block matching”, Information Processing Society of Japan, May 2004, CVIM-144-5, pp.33-40 "A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment", Takeo Kanade and Masatoshi Okutomi, IEEE PAMI, Vol. 16, NO.9, Sep. 1994

Claims (6)

A block shape setting means for setting a block shape for performing a matching block using one of the two images as a standard image and the other as a reference image;
Reference image block position setting means for setting a block position in the reference image based on the block shape set by the block shape setting means;
In the reference image, reference image block position setting means for setting a plurality of blocks in the vicinity of a position corresponding to the block position set by the standard image block position setting means;
Evaluation value calculation means for calculating a correlation evaluation value between a block set in the reference image and a plurality of blocks set by the reference image block position setting means;
The image processing apparatus, wherein the block shape setting means sets the block shape according to the shape of an object included in the reference image.   The image processing apparatus according to claim 1, wherein the block shape setting unit sets a shape that does not include a vertical line along a boundary portion of the object as the block shape.   The block shape setting means sets the block shape to at least one of a rhombus shape, a polygonal shape not including a vertical line along a boundary portion of the object, a circular shape, and an elliptical shape. The image processing apparatus according to claim 2.   The said block shape setting means sets at least 1 block shape according to the said object among several block shapes from which the preset shape differs, The any one of Claim 1 thru | or 3 characterized by the above-mentioned. An image processing apparatus according to 1. A block shape setting procedure for setting a block shape for performing a matching block using one of the two images as a standard image and the other as a reference image;
A reference image block position setting procedure for setting a block position in the reference image based on the block shape set by the block shape setting procedure;
In the reference image, a reference image block position setting procedure for setting a plurality of blocks in the vicinity of a position corresponding to the block position set by the standard image block position setting procedure;
An evaluation value calculation procedure for calculating a correlation evaluation value between a block set in the reference image and a plurality of blocks set by the reference image block position setting procedure;
The block shape setting procedure sets the block shape according to the shape of an object included in the reference image.   An image processing program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 4.

Images