JP2020021126A - Image processing device and control method thereof, distance detection device, imaging device, program - Google Patents

Abstract

To reduce an error of a correlation value, and acquire the parallax amount with higher accuracy.SOLUTION: A distance detection device 101 acquires a plurality of image data having viewpoints made different from each other by an imaging unit. An arithmetic processing unit 105 sets a standard image region on an image of first image data as a standard image, among the plurality of acquired image data, and sets a reference image region on the image of second image data as a reference image. The arithmetic processing unit 105 performs correlation calculation between the standard image region and the reference image region, and calculates a parallax amount from correlation value groups obtained by changing the reference image region. The arithmetic processing unit 105 determines a shape of the standard image region so that a luminance change value of an adjoining pixel at a boundary part in a parallax direction of the standard image region is less than a predetermined threshold, and performs the correlation calculation between the reference image regions having the shape corresponding to the shape of the standard image region.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing technique for calculating a parallax amount using a captured image.

  As a method of calculating distance information from a captured image, there is a method of obtaining distance information by obtaining a parallax amount from a correlation value between a plurality of images obtained from different viewpoints. Specifically, a process of extracting an image signal of a partial region including a pixel of interest in one image as a signal of a reference image region and extracting an image signal of a partial region in the other image as a reference image signal from among the images forming the set Is performed. The extraction position of the reference image signal is changed, and the correlation value at each position is calculated. By obtaining the position having the highest correlation among the correlation values at each position, the amount of parallax at the pixel of interest can be calculated. In addition, a sub-pixel estimation method for estimating a higher-resolution parallax amount using a correlation value of a pixel value between a position having the highest correlation and a position of a reference image adjacent to the position is performed. In this case, an area where the pixel value of the captured image greatly changes depends on the shape of the subject and the like, and an error occurs if the amount of parallax is calculated under the condition that the area and the end of the reference image area overlap. Patent Literature 1 discloses a method of reducing an error in a block matching process by selecting a shape of a reference image area so that the shape of an object and the shape of the reference image area do not overlap.

Japanese Patent No. 5782766

When selecting the shape of the reference image area so that the shape of the target object does not match the shape of the reference image area, for example, for a subject such as a vertical line, select the shape of the reference image area so as to be circular. The error still remains. Even if the shape of the target object does not match the shape of the reference image area, if a luminance change remains at the boundary in the parallax search direction of the reference image area, a reference adjacent to the position of the high correlation value by the sub-pixel estimation method An error may occur in the correlation value at the image position.
SUMMARY An advantage of some aspects of the invention is to provide an image processing apparatus that can reduce a correlation value error and obtain a more accurate amount of parallax.

  An image processing apparatus according to an embodiment of the present invention is an image processing apparatus that performs image processing on a plurality of image data having different viewpoints, wherein the obtaining unit obtains the plurality of image data, and the obtaining unit obtains the plurality of image data. Setting a reference image area in the image of the first image data as the reference image and setting a reference image area in the image of the second image data as the reference image among the plurality of image data; Calculation processing means for calculating a parallax amount of the plurality of image data by performing a correlation calculation between the image data and the reference image area. The arithmetic processing unit determines the reference image region having a shape in which a luminance change value of an adjacent pixel at a boundary portion in the parallax direction of the reference image region is smaller than a threshold, and the determined reference image region and the reference The correlation operation is performed with the reference image area having a shape corresponding to the shape of the image area.

  ADVANTAGE OF THE INVENTION According to the image processing apparatus of this invention, the error of a correlation value can be reduced and a more highly accurate parallax amount can be acquired.

FIG. 2 is a schematic diagram illustrating an example of a distance detection device according to the first embodiment. FIG. 2 is a schematic diagram illustrating a configuration example of an imaging device according to the first embodiment. FIG. 5 is a diagram illustrating a distance detection method according to the first embodiment. FIG. 5 is an explanatory diagram of a method for determining a shape of a reference image area according to the first embodiment. FIG. 6 is a diagram illustrating an example of a distance detection result according to the first embodiment. FIG. 14 is an explanatory diagram of a method for determining a shape of a reference image area according to the second embodiment.

  Each embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the following embodiments, as an example of an apparatus to which the image processing apparatus of the present invention is applied, an apparatus that acquires a plurality of image data having different viewpoints by an imaging unit and calculates a parallax amount or distance information corresponding to the parallax amount is described. explain.

[First Embodiment]
FIG. 1 shows an outline of a distance detection device 101 of the present embodiment. A distance detection device 101 including an imaging unit captures an image of a subject 102. The distance detection device 101 includes an imaging optical system 103, an image sensor 104, an arithmetic processing unit 105, and a memory unit 106.

  The imaging optical system 103 is an imaging optical system that forms light from a subject and forms an image of the subject on an imaging surface of the imaging element 104. The imaging optical system 103 includes a plurality of lens groups and a stop (not shown), and has an exit pupil at a position at a predetermined distance from the image sensor 104. Note that FIG. 1 shows the optical axis 110 of the imaging optical system 103, and defines three orthogonal X-axis, Y-axis, and Z-axis. In this specification, the optical axis 110 is parallel to the Z axis, and the X axis and the Y axis orthogonal to the Z axis are perpendicular to each other, and are perpendicular to the optical axis.

  A plurality of pixel units are arranged in the image sensor 104. A specific description will be given with reference to FIG. FIG. 2A is a cross-sectional view schematically illustrating a configuration of each pixel portion. The pixel portion has a micro lens 201, a color filter 202, and photoelectric conversion units 203A and 203B. The image sensor 104 is provided with spectral characteristics of R (red), G (green), and B (blue) according to the wavelength band detected by the color filter 202 for each pixel unit. Is arranged. In this specification, for convenience, each pixel portion is described as being arranged on an XY plane. On the substrate 204, photoelectric conversion units 203A and 203B having sensitivity in a wavelength band to be detected are formed. The photoelectric conversion units 203A and 203B are formed using photodiodes. In addition, each pixel portion includes a wiring (not shown).

  FIG. 2B is a view of the exit pupil of the imaging optical system 103 from the intersection (center image height) between the optical axis 110 and the image sensor 104. The first light flux mainly passing through the first pupil partial region 210 enters the photoelectric conversion unit 203A. The second light flux mainly passing through the second pupil partial region 220 enters the photoelectric conversion unit 203B. The first pupil partial region 210 and the second pupil partial region 220 are different regions in the exit pupil. The photoelectric conversion units 203A and 203B generate an A image and a B image by photoelectrically converting the incident light flux, respectively. The signals of the A image and the B image acquired by the photoelectric conversion units 203A and 203B are transmitted to the arithmetic processing unit 105. The calculation processing unit 105 executes a distance measurement calculation process, and calculates a distance value from the distance detection device 101 to the subject 102. The data of the calculated distance value is stored in the memory unit 106. An image signal obtained by adding the signal of the A image and the signal of the B image can be used as a captured image signal of the subject 102.

  In FIG. 2B, the position of the center of gravity of the first pupil partial region 210 (the position of the first center of gravity) is indicated by a point 211, and the position of the center of gravity of the second pupil partial region 220 (the position of the second center of gravity) is indicated by a point. 221. The axis 200 is an axis extending in the X-axis direction through the points 211 and 221.

  In the present embodiment, the position of the first center of gravity indicated by the point 211 is eccentric (moved) along the axis 200 from the center of the exit pupil. On the other hand, the second center of gravity position indicated by the point 221 is eccentric (moved) along the axis 200 in a direction opposite to the first center of gravity position. The direction connecting the point 211 indicating the first position of the center of gravity and the point 221 indicating the position of the second center of gravity is called a pupil division direction. The distance between the point 211 and the point 221, that is, the distance between the centers of gravity is the base line length 222. The positions of the A image and the B image change in the same direction as the pupil division direction (X-axis direction in the present embodiment) due to defocus. The relative position change amount between the A image and the B image, that is, the parallax amount between the A image and the B image is an amount corresponding to the defocus amount. Therefore, the amount of parallax between the A image and the B image can be acquired by a method described later, and the amount of parallax can be converted into a defocus amount or a distance by a known conversion method.

  With reference to FIG. 3, the distance measurement calculation according to the present embodiment will be described. FIG. 3A is a flowchart illustrating the flow of the processing. The following processing is realized by a CPU (Central Processing Unit) included in the control unit of the distance detection device 101 executing a predetermined program.

  In “image acquisition” shown in S301 of FIG. 3A, a process of photographing a subject and saving an image signal is performed. The captured image is acquired as a set of an image including an A image and a B image having parallax, and each image data is stored in the memory unit 106.

  The distance measurement calculation processing shown in S302 to S305 is performed by the calculation processing unit 105. The positional relationship between the reference image area and the reference image area set in each step will be described with reference to FIG. FIG. 3B is a diagram schematically illustrating the A image 310A and the B image 310B. The Z axis is an axis perpendicular to the plane of FIG. The X axis is a horizontal axis, and the Y axis is a vertical axis.

  In “calculation of reference image region shape” shown in S302 of FIG. 3A, a process of determining the shape of the reference image region is executed. The pixel 320 in the A image 310A of FIG. 3B is a target for which distance calculation is performed, and is hereinafter referred to as a target pixel. A process of extracting a partial region including the target pixel 320 and its neighboring pixels is performed, and an image region 311 having an initial shape is obtained. Hereinafter, the image area 311 is referred to as an initial reference image area. In this specification, an example in which a target pixel is set on the A image 310A will be described. That is, although the A image 310A is used as a reference image and the B image 310B is used as a reference image, a pixel of interest may be set on the B image 310B and the A image 310A may be used as a reference image. Details of the process of S302 will be described later with reference to FIG.

  In the “correlation calculation” shown in S303, a process of calculating a correlation value between the A image 310A and the B image 310B is executed. A reference image area of the shape of the target pixel of the A image determined in S302 is acquired. Next, a process of extracting an area having the same shape as the reference image area on the B image is performed, and a reference image area 313 is obtained. In FIG. 3B, the reference image area 313 is displayed as a square for convenience of explanation, but is actually an image area having the same shape as the shape determined in S302. A process of moving the position where the reference image region is extracted on the B image along the X-axis direction that is the same as the pupil division direction, and calculating a correlation value between the reference image region and the reference image region at each movement amount is performed. As a result, a correlation value composed of a data sequence of correlation values for each movement amount is obtained. The direction in which the reference image area is moved to perform the correlation operation is called a parallax calculation direction. By setting the parallax calculation direction to the same direction as the pupil division direction, it is possible to correctly calculate the amount of parallax generated according to the subject distance of the A image and the B image. For the calculation of the correlation value, a general calculation method such as SAD (Sum of Absolute Difference) or SSD (Sum of Squared Difference) can be used.

  In the “parallax amount calculation” shown in S304, a process of calculating the parallax amount from the correlation value obtained in S303 is executed. The amount of parallax can be calculated using an existing method. For example, a process of extracting a data sequence of the correlation value corresponding to the movement amount that provides the highest correlation among the correlation values and the movement amount in the vicinity thereof is performed. The parallax amount can be calculated by estimating the movement amount having the highest correlation with sub-pixel accuracy using a known interpolation method. The known interpolation method is, for example, a conformal straight line fitting method when SAD is used for calculating a correlation value, and a parabolic fitting method when SSD is used. Various interpolation methods are used depending on the method of calculating the correlation value, but the parallax amount may be calculated with sub-pixel accuracy using any interpolation method.

  In the “distance value calculation” shown in S305, a process of converting the parallax amount calculated in S304 into a defocus amount or a subject distance is performed. The conversion from the parallax amount to the defocus amount can be performed by using a geometric relationship using the base line length. The conversion from the defocus amount to the subject distance may be performed using the imaging relationship of the imaging optical system 103. Alternatively, it may be converted into a defocus amount or a subject distance by multiplying the parallax amount by a predetermined conversion coefficient. With such a method, the distance value at the pixel of interest can be calculated.

  The distribution data group corresponding to the distance value includes an image shift amount map, a defocus map, and the like in addition to the distance map. A plurality of image data having different viewpoints have a parallax, and an image shift amount or a defocus amount corresponding to the parallax amount can be acquired. For example, in application to an imaging apparatus, the control unit controls the focus adjustment of the imaging optical system using the defocus amount. That is, autofocus control based on the imaging surface phase difference method is performed based on the relative positional shift amount between a plurality of images having different viewpoints. Alternatively, the image processing unit in the imaging device generates data such as a distance map, an image shift amount map, and a defocus map by using the acquired plurality of image data. These data are depth information in the depth direction of the captured image, and can be used in, for example, a subject area recognition process, a viewpoint change process, a blur correction process, and the like. In addition, the image processing unit in the imaging device generates an image for three-dimensional display (stereoscopic display) using the acquired plurality of pieces of image data having different viewpoints, and outputs the image to the display unit and the recording unit.

  Next, with reference to FIG. 4, a description will be given of a processing example of “reference image area shape calculation” shown in S302 of FIG. FIG. 4A is a flowchart illustrating an example of the process of S302. FIG. 4B shows an example of an image for explaining the “reference image region shape calculation” process. In the figure, pixels whose pixel values are greater than or equal to the threshold are displayed in white, pixels whose pixel values are less than the threshold are displayed in black, white pixels represent high luminance pixels, and black pixels represent low luminance pixels . The axis perpendicular to the plane of FIG. 4B is the Z axis, the horizontal axis orthogonal to the Z axis is the X axis (the left side is defined as the positive direction), and the vertical axis orthogonal to the Z axis is the Y axis. (The upper side is defined as the positive direction).

  The upper diagram in FIG. 4B shows a partial image near the pixel of interest. The area inside the bold frame shown in gray indicates the initial reference image area (5 × 5 pixel area). In the present embodiment, regarding the initial reference image area, the initial shape (the shape of the image area 311) in FIG. 3B will be described as a square, but any shape can be selected as the initial shape.

  In step S400 of FIG. 4A, the boundary of the initial shape in the parallax search direction in the reference image area is obtained, and the process is sequentially performed on pixels near the boundary. In the present embodiment, the parallax search direction is the same as the pupil division direction, and is the X-axis direction. The initial shape is the shape of the initial reference image area. In the upper part of FIG. 4B, arrows are described at the left boundary of the thick line frame L shown in gray. Similarly, there may be a boundary portion on the right side of the thick line frame L.

  In the “brightness change calculation” process of S401, a process of calculating a brightness change of an adjacent pixel at the boundary of the initial shape is performed. The pixel indicated by the end point of the arrow in the upper diagram of FIG. 4B is the target pixel. In the “brightness change threshold process” of S402, the brightness change value calculated in S401 is compared with a threshold. If the luminance change value of the target pixel is equal to or larger than the threshold, the process proceeds to S403. If the luminance change value is smaller than the threshold, the process proceeds to S404. More specifically, in the second row from the top in the upper diagram of FIG. 4B, the neighboring pixels at the boundary of the initial shape are all low-luminance pixels, and the luminance change value is less than the threshold value. Move to the processing of. On the other hand, in the third row from the top in the upper part of FIG. 4B, adjacent pixels at the boundary of the initial shape are low-luminance pixels and high-luminance pixels. In this case, since the luminance change value is equal to or larger than the threshold, the process proceeds to S403.

  In the “shape deformation process” of S403, a process of deforming the shape of the reference image area from the initial shape is performed. That is, a process of moving the boundary in the parallax search direction is performed. Then, the flow shifts to S401, where a luminance change calculation at the boundary after the movement is performed. Next, a luminance change threshold process is performed in S402. Until a boundary portion smaller than the threshold value is found, the process of moving the boundary portion in the parallax search direction is repeatedly executed. This will be specifically described with reference to an example shown in FIG.

  In the upper part of FIG. 4B, the luminance change value is equal to or larger than the threshold value in the third row from the top (the second row in the initial shape). That is, of the two pixels adjacent to the boundary, the left pixel is a black low-luminance pixel, and the right pixel is a white high-luminance pixel. In this case, as shown by the thick line frame LL in the lower diagram of FIG. 4B, in the third row (the second row in the initial shape), the boundary is shifted by one pixel in the −X direction (right direction). Is performed. As a result, both the left and right pixels on the boundary become high-luminance pixels, and the luminance change value is less than the threshold. Similar processing is performed on the other boundary portions. When it is determined in S402 that the luminance change value is less than the threshold value in any of the boundary portions, the process proceeds to S404.

  The boundary is determined in the “shape determination processing” of S404. The shape of the reference image area is determined by performing the process of determining the shape at the boundaries in all the parallax search directions. For example, in the upper diagram of FIG. 4B, the second row and the fourth to sixth rows from the top (the first row and the third to fifth rows in the initial shape) and the brightness change value at the right boundary portion are different. Since it is less than the threshold value, no deformation process is performed on the initial shape, and the boundary is determined without any change. In this case, in the lower diagram of FIG. 4B, the shape of the portion corresponding to the second row and the fourth to sixth rows from the top in the thick line frame LL does not change.

  In S405, it is determined whether or not the processing for all the boundaries of the initial shape has been completed. When the processing is completed, the shape of the reference image area 312 is determined with respect to the image area 311 having the initial shape in FIG. If the processing has not been completed, the process returns to step S400 to continue the processing of the next boundary portion.

  The threshold value for the luminance change value used in S402 may be determined in any manner. The accuracy is improved as the threshold value is reduced, but the luminance change value often does not become less than the threshold value even if the shape deformation processing in S403 is performed. As a result, the amount of calculation may increase. In that case, a setting process for increasing the threshold value is performed each time the shape deformation process is performed. A process of comparing the calculated luminance change value with the changed threshold value is performed. If the luminance change value is less than the threshold value, the shape is determined at that time. As a result, the amount of shape deformation can be suppressed, and a balance between a reduction in error and an increase in the amount of calculation can be achieved. The luminance change value to be compared with the changed threshold value may be the minimum value of the values calculated so far. Alternatively, a method of limiting the amount of deformation of the shape itself may be adopted. Even if the process of calculating the brightness change value by performing the shape deformation process of S403 is executed at a predetermined threshold number of times, if the brightness change value does not become less than the threshold value, the shape deformation process is terminated, and the amount of calculation is increased. Can be suppressed. In that case, a process of selecting a shape deformation amount that has a minimum brightness change value from among the calculated brightness change values is performed. That is, the shape having the minimum luminance change value is determined as the shape of the reference image area. After the calculation amount is limited so as not to exceed the upper limit, the optimum shape deformation amount can be determined from a plurality of shape deformation amounts.

  In FIG. 4B, as an example of the shape deformation processing, the case where the boundary is changed to the minus side in the X-axis direction has been described. However, the boundary may be changed to the plus side in the X-axis direction. May be deformed to both sides. The order of transformation is not limited to a particular order.

  Regarding the size of the reference image area (that is, the number of pixels), the smaller the size, the lower the noise resistance of the pixel signal, the larger the error, and the more the calculation amount. It is desirable that the initial reference image area be set with a balance between noise and calculation amount. Even if the shape deformation processing is performed, it is desirable to perform the deformation so that the size of the reference image area after the shape deformation processing becomes equal before and after the deformation. Equal means that the size of the initial reference image area is equal to the size of the reference image area after the shape deformation processing, or that the size difference between the two is less than the threshold. For this purpose, a process is performed in which the number of boundaries when deforming to the plus side in the X-axis direction and the number of boundaries when deforming to the minus side are the same or set to be close to the same number. For example, there is a method in which shape deformation to the positive side and shape deformation to the negative side are alternately performed for each row. Alternatively, there is a method in which the direction of shape deformation is set to different directions at the left boundary and the right boundary of the initial reference image area. In this method, the shape is deformed outward (or inward) at the left boundary of the initial reference image area, and the shape is deformed inward (or outside) at the right boundary. In addition, any method may be used to make the size of the initial reference image area equal to the size of the reference image area after shape deformation.

  Further, the shape deformation direction may be changed in accordance with the shape deformation amount up to the previous process in the loop process in the “reference image region shape calculation” of S302 shown in FIG. The loop process is being executed in S405 in S400 to S405, and the process is continued. When processing a new boundary portion of the initial shape, the total value of the shape deformation amount by the shape deformation process (S403) performed up to the previous process is calculated. For example, a process of calculating an added value of the shape deformation amount is performed for each loop process. If the calculated addition value is a positive value, the shape deformation processing to the minus side is performed. If the calculated addition value is a negative value, the shape deformation processing to the plus side is performed. By this processing, the shape deformation processing can be performed so that the difference between the size (the number of pixels) of the initial reference image area and the size of the deformed reference image area approaches zero. Alternatively, a method may be used in which the size of the initial reference image area is set to a minimum size from the viewpoint of noise, and the shape deformation processing is always performed in a direction to increase the size. Conversely, a method may be used in which the size of the initial reference image area is set to the maximum size that allows the amount of calculation, and the shape deformation processing is always performed in a direction in which the size decreases.

  In the present embodiment, by performing the shape deformation processing of the reference image area according to the brightness change value at the boundary of the reference image area in the parallax search direction, it is generated in relation to the brightness distribution of the subject and the position of the reference image area. Therefore, the error of the correlation value can be reduced, and more accurate ranging can be realized.

  Here, the principle of generation of an error and the fact that the error is reduced by the present embodiment will be described. First, with reference to FIGS. 5A, 5B, and 5C, a description will be given of the reason why an error occurs when the shape of the reference image area is not changed (when the initial reference shape is used). In the following description, it is assumed that the A image and the B image are images having the same shading and have no parallax.

  FIG. 5A is a diagram illustrating a positional relationship between the reference image area 502 and the reference image area 503 in the A image 501. The A image 501 is a pattern image of a vertical stripe shape in which the brightness changes in the left-right direction. In the following, the boundary portions 504 and 505 at which light and dark are switched in the A image 501 are referred to as image edges. In the reference image area 502, there are boundaries 504 and 505, which are image edges. On the other hand, a boundary portion 505 that is an image edge exists in the reference image region 503, and the right end of the reference image region 503 overlaps the boundary portion 504.

  FIG. 5B illustrates each correlation value calculated by moving the reference image area and performing a correlation operation with the reference image area 502. The horizontal axis represents the movement amount of the reference image area, and the vertical axis represents the correlation value. Here, the lower the correlation value, the higher the correlation between images. The correlation values C0, Cp, and Cm are correlation values when the position of the reference image area is moved by an amount corresponding to 0, +1 pixel, and -1 pixel, respectively. When the movement amount is zero, the reference image and the reference image match, and the correlation value C0 becomes a low value. When the reference image area is moved by an amount corresponding to +1 pixel or −1 pixel, a difference occurs between the reference image area and the reference image area at the boundary portions 504 and 505, which are image edges, and the correlation values Cp, Cm both increase. At this time, the correlation values Cp and Cm have the same value. By calculating the correlation curve 510 by interpolating the correlation value and calculating the movement amount (parallax amount) having the highest correlation, the parallax amount indicated by reference numeral 511, that is, the correct value (parallax amount 0) is obtained.

  FIG. 5C illustrates each correlation value calculated by moving the reference image area and performing a correlation operation with the reference image area 503. The setting of the horizontal axis and the vertical axis is the same as in FIG. In the case of FIG. 5C, the boundary 504, which is an image edge, and the right end of the reference image area 503 overlap. When the movement amount is zero, the reference image and the reference image match, and the correlation value C0 is low. When the movement amount is an amount corresponding to +1 pixel, the correlation value Cp increases as in the case of the reference image area 502. On the other hand, when the movement amount is an amount corresponding to −1 pixel, a difference between the reference image and the reference image occurs only at the boundary portion 505 which is an image edge. Therefore, the increase in the correlation value Cm corresponding to −1 pixel is slightly increased. Since the correlation value is asymmetric on the plus side and the minus side of the movement amount, the amount of parallax (see reference numeral 512) obtained by interpolating each correlation value is different from the correct value (parallax amount 0). Occurs.

  Next, the reason why the error is reduced by the present embodiment will be described. In the present embodiment, when the image edge and the right end of the reference image area overlap, as in the case of the reference image area 503, the shape deformation processing of the reference image area is performed. The shape of the reference image area changes so that the image edge and the right end of the reference image area do not overlap. This will be specifically described with reference to FIG.

  A reference image area 506 shown in FIG. 5D is an area after the shape deformation processing corresponding to the reference image area 503. The right end of the reference image area 503 is deformed, so that the reference image area 506 has a shape that does not overlap the image edge. Therefore, as in FIG. 5B, the correlation values Cp and Cm when the movement amount is an amount corresponding to +1 pixel and −1 pixel have the same value. Therefore, a correct value (parallax amount 0) can be obtained. More specifically, the shape is set so that a change in the shape of the reference image region does not cause a large change in luminance at the boundary in the parallax direction of the reference image region. For example, even if the shape of the reference image area is rhombic or circular, it does not overlap with the image edge, but a part of the boundary where a large luminance change occurs at the boundary in the parallax direction remains, which is a factor that causes an error. Become. Therefore, in the present embodiment, the error can be further reduced by executing the shape deformation processing of the reference image area so that a large luminance change does not occur at each boundary in the parallax direction. For example, it is possible to reduce the calculation error of the amount of parallax that occurs when an area in which the pixel value of the captured image greatly changes and an end (boundary) in the left-right direction of the reference image area overlaps.

  According to the present embodiment, in distance measurement using a captured image, errors in correlation values can be reduced and more accurate distance information can be obtained.

[Second embodiment]
Next, a second embodiment of the present invention will be described. In the present embodiment, differences from the first embodiment will be mainly described, and description of the same items as those in the first embodiment will be omitted. The manner of omitting such description is the same in the embodiments described later.

  In “reference image region shape calculation” shown in S302 of FIG. 3A, a process of determining the shape of the reference image region based on a difference value (brightness change value) between adjacent pixels is performed. When many pixels need to be calculated as the target pixel, the same calculation may be performed many times as the calculation of the luminance change value of the adjacent pixel. Therefore, in the present embodiment, an example will be described in which the calculation amount is reduced by calculating the luminance change values of adjacent pixels in advance, and the shape of the reference image area is efficiently calculated.

  FIG. 6 is a flowchart illustrating “reference image region shape calculation” of S302 in the present embodiment. The process of S610 is a process of calculating a luminance change value of an adjacent pixel on the A image. This calculation process is executed in advance before the process of S302 is performed. The data of the calculation result in S610 is stored in, for example, the memory unit 106. In S610, the calculation process is executed over the entire pixel area where the brightness change value of the adjacent pixel on the A image, that is, the brightness change value of the target pixel and its surrounding pixels may be calculated a plurality of times. The luminance change value of an adjacent pixel is calculated, for example, as an absolute value of a difference between adjacent pixel values.

  In S302, as in FIG. 4A, the calculation is repeatedly performed along the boundary of the shape of the initial reference image in the disparity search direction. In the “minimum value search processing” of S601 following S400, processing of searching for the minimum value of the luminance change value of the adjacent pixel calculated in advance in S610 is performed. A process of extracting a luminance change value of an adjacent pixel corresponding to an allowable shape change range from the boundary of the shape of the initial reference image is executed. The minimum value of the luminance change values of the extracted adjacent pixels is searched.

  In the “reference image region shape determination processing” of S602, the position of the reference image region is determined by setting a position on the A image corresponding to the pixel position that is the minimum luminance change value searched for in S601 as a boundary. Is done.

  In order to make the size of the reference image area approximately constant among a plurality of target pixels, the shape change range allowable from the boundary of the shape of the initial reference image described above may be changed for each loop process. For example, there is a method of changing the shape change range for each row, and a method of changing the shape change range between the right boundary and the left boundary.

  Alternatively, S601 may be set as threshold processing instead of searching for the minimum value, and a pixel position where the luminance change value is less than the threshold may be searched. Thereby, the shape of the reference image area can be determined in S602 from the pixel position candidates where the luminance change value of the adjacent pixel is less than the threshold value. For example, by selecting a position closest to the shape of the initial reference image area, the shape of the reference image area closer to the shape of the initial reference image area can be determined. Further, the size can be adjusted by selecting a pixel position according to the already deformed shape deformation amount at each boundary position of the reference image region so that the size of the reference image region is approximately constant among the plurality of target pixels. .

  Further, the position at which the “reference image region shape calculation” of S302 is performed may be determined in advance based on the luminance change value of the adjacent pixel calculated in advance in S610. From the information on the luminance change value of the adjacent pixel, it is possible to determine in advance the pixel position where the luminance change value is equal to or larger than the threshold value in each target pixel. A process of extracting a pixel position where the luminance change value is equal to or greater than the threshold value and extracting a target pixel at which the boundary of the shape of the initial reference image region in the parallax direction overlaps is executed, and the process of S302 is performed using only the target pixel. Be executed. The process shifts to S303 without performing the process of S302 except for the image of interest. As a result, the amount of calculation can be reduced.

  According to the present embodiment, it is possible to more efficiently calculate the shape of the reference image area in the shape deformation processing on the initial reference image area.

[Third embodiment]
Next, a third embodiment of the present invention will be described. In the present embodiment, a stereo camera is used instead of a pupil division type image sensor having a plurality of photoelectric conversion units for each microlens. An imaging device capable of acquiring a plurality of image data with different viewpoints can be realized by two or more imaging optical systems and their corresponding imaging devices. In this case, the design flexibility of the base line length is improved, and the distance measurement resolution is improved.

  In the present embodiment, for example, a distance detection device is configured by using the imaging device and the projection device as separate devices. By combining with a projection device that irradiates a subject with light of a predetermined pattern, more accurate distance measurement can be performed. For example, by projecting a vertical line chart as shown in FIGS. 5A and 5D onto a subject, distance measurement can be performed even in a region where the subject has no texture. However, since the scene is likely to have an error as described above, the error can be reduced by executing the shape deformation processing of the reference image area described above. It is preferable that the control device is configured using a CPU provided in the imaging device from the viewpoint of miniaturization of the device.

  The present embodiment can be applied to a parallax amount detection device that detects a parallax amount in addition to calculating a distance from an imaging device to a subject. In this case, in FIG. 3, “calculation of distance value” in S305 can be omitted. In the parallax amount detection device, based on the detected parallax amount, a process of cutting out an image area of a subject near the in-focus position from a captured image is performed. The parallax amount detection apparatus is configured to output the parallax amount as it is without converting the parallax amount into a distance in the distance detection apparatus 101 of the above embodiment, and the other configuration is the same as the distance detection apparatus 101.

  In the digital camera, feedback control is performed on the imaging optical system 103 and the image sensor 104 based on the detected amount of parallax, and for example, more accurate automatic focus adjustment can be realized. In this case, the control unit of the image pickup optical system and the control unit of the image pickup device, or any one of the control units, calculate the movement amount according to the correction distance at an arbitrary angle of view. For example, the control unit calculates an image shift amount or a defocus amount corresponding to the parallax amount, and controls a driving unit that drives a focus unit in the imaging optical system to perform focus adjustment. Alternatively, the control unit calculates an image shift amount or a defocus amount corresponding to the parallax amount, and controls the driving unit of the image sensor to move the image sensor to perform focus adjustment. Since a more accurate amount of parallax can be obtained, more accurate focusing can be performed with a single feedback control on the subject. Further, since more accurate field information can be obtained using the distance information (depth information) corresponding to the parallax amount, it is possible to perform optimal flash photography using a strobe (not shown).

  In addition, the device of the present embodiment is mounted on a robot, a car, a drone, or the like capable of autonomously creating an action plan, and can be used as information acquisition means for recognizing an external environment. Using the acquired distance information, the external environment recognizing unit performs a process of converting the distance information into external environment recognition data. The system in this case includes an imaging device, an external environment recognition unit that processes the captured image data, an action plan creation unit, and an actuator control unit. The action plan creation unit acquires the external environment recognition data output from the external environment recognition unit, and creates an action plan according to the data and a given purpose in advance. The actuator control unit controls the actuator according to the action plan created by the action plan creation unit. The actuator is a prime mover, an electric motor, a driving unit of a wheel, a leg mechanism, or the like. In realizing autonomous movement of a robot or a moving device, the imaging device according to the present embodiment can acquire distance information with reduced errors, and thus has an effect of being able to more stably recognize an external environment.

  According to the present embodiment, faster and more accurate distance detection or parallax amount detection is possible in application to various systems using the distance detection device or the parallax amount detection device.

[Other Embodiments]
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

Reference Signs List 101 distance detection device 103 imaging optical system 104 imaging device 105 arithmetic processing unit

Claims (24)

An image processing apparatus that performs image processing on a plurality of image data having different viewpoints,
Acquisition means for acquiring the plurality of image data;
A reference image area is set in the image of the first image data that is the reference image, and a reference image area is set in the image of the second image data that is the reference image, of the plurality of pieces of image data acquired by the acquisition unit. Setting, calculating processing means for calculating the amount of parallax of the plurality of image data by a correlation operation between the reference image area and the reference image area,
The arithmetic processing unit determines the reference image region having a shape in which a luminance change value of an adjacent pixel at a boundary portion in the parallax direction of the reference image region is smaller than a threshold, and the determined reference image region and the reference An image processing apparatus, wherein the correlation operation is performed with the reference image area having a shape corresponding to the shape of the image area. The arithmetic processing means,
Setting means for setting an initial shape of the reference image area,
Calculation means for calculating a luminance change value of an adjacent pixel at a boundary portion in the parallax direction of the reference image region that is the initial shape,
When the brightness change value calculated by the calculation unit is equal to or more than the threshold, a deformation unit that changes the shape of the reference image region by changing the position of the boundary of the reference image region that is the initial shape, The image processing apparatus according to claim 1, comprising: The calculating means calculates a luminance change value of an adjacent pixel at a boundary in the parallax direction of the reference image area deformed by the deforming means,
The arithmetic processing unit, when the luminance change value calculated by the calculation unit is less than the threshold, determines a position of a boundary of the reference image area deformed by the deformation unit. 3. The image processing device according to 2. The deformation unit limits the amount of deformation when deforming the shape of the reference image region, and when the luminance change value calculated by the calculation unit does not fall below the threshold, the shape of the reference image region is deformed. The image processing apparatus according to claim 3, wherein the processing is stopped. When the brightness change value calculated by the calculation means does not become less than the threshold value, the deformation means determines a shape having a minimum brightness change value among the brightness change values calculated by the calculation means as the reference. The image processing apparatus according to claim 4, wherein the image processing apparatus determines the shape of the image area. The image processing apparatus according to claim 2, wherein the arithmetic processing unit sets the threshold value to be large each time the deformation unit performs a deformation process on the shape of the reference image area. . The size of the reference image area that is the initial shape is equal to the size of the reference image area after the deformation processing by the deformation unit, or the size of the reference image area that is the initial shape and the size of the reference image area after the deformation processing by the deformation unit. The image processing apparatus according to any one of claims 2 to 6, wherein a size difference from the size of the reference image area is smaller than a threshold value. The image processing apparatus according to claim 2, wherein the deforming unit changes a direction in which a shape of the reference image area is deformed for each row. The said deformation | transformation means sets the direction which deform | transforms the shape of the said reference | standard image area to the direction of a different side by the 1st boundary of a parallax direction, and the 2nd boundary. The image processing apparatus according to claim 1. The said deformation | transformation means changes the direction which deform | transforms the shape of the said reference image area according to the total value of the deformation amount by the already-performed deformation | transformation processing. The Claims any one of Claims 2 thru | or 7 characterized by the above-mentioned. Image processing device. The said deformation | transformation means always makes the size of the said reference image area changed for every deformation process larger or smaller than the size of the reference | standard image area which is the said initial shape. The image processing device according to any one of claims 1 to 4. The image processing apparatus according to claim 2, wherein the deforming unit performs a process of deforming a shape of the reference image area using a luminance change value of the adjacent pixel calculated in advance by the calculating unit. . The deformation unit searches for a minimum value of the luminance change value of the adjacent pixel calculated in advance by the calculation unit, and the reference image area corresponding to a pixel position that is the minimum value of the searched luminance change value. The image processing apparatus according to claim 12, wherein the position is set as a boundary. The image according to claim 12, wherein the deforming unit determines the shape of the reference image area based on a pixel position where a luminance change value of the adjacent pixel calculated in advance by the calculating unit is smaller than a threshold. Processing equipment. The deformation unit determines a shape of the reference image region by selecting a pixel position closer to the shape of the reference image region, which is the initial shape, from among the pixel positions where the luminance change value of the adjacent pixel is less than the threshold value. The image processing apparatus according to claim 14, wherein: The deformation unit selects the pixel position according to the deformation amount that has been deformed at the boundary position of the reference image region among the pixel positions where the luminance change value of the adjacent pixel is less than the threshold, and The image processing apparatus according to claim 14, wherein a size is adjusted to determine a shape of the reference image area. With a luminance change value of the adjacent pixel calculated in advance by the calculation unit, a determination unit that determines whether to perform a deformation process of the shape of the reference image region,
When the determination unit determines that the shape of the reference image area is to be deformed by the determination unit, the deformation unit selects a pixel position where the luminance change value is equal to or greater than a threshold to determine the shape of the reference image area. The image processing apparatus according to claim 12, wherein: The determining unit determines that the deformation processing of the shape of the reference image region is not performed when the luminance change value of the adjacent pixel is less than a threshold value at the boundary of the reference image region that is the initial shape. The image processing apparatus according to claim 17, wherein An image processing apparatus according to any one of claims 1 to 18,
Imaging means,
The arithmetic processing means calculates distance information corresponding to the amount of parallax from a plurality of correlation values calculated by performing a correlation operation between the reference image region and the reference image region by changing the reference image region. Distance detecting device. The imaging unit includes a plurality of microlenses, and a plurality of photoelectric conversion units corresponding to each microlens,
20. The distance detecting apparatus according to claim 19, wherein the acquisition unit acquires a plurality of image data having different viewpoints from the imaging unit. An image processing apparatus according to any one of claims 1 to 18,
Imaging means;
Control means for calculating a defocus amount corresponding to the parallax amount and controlling focus adjustment of the imaging optical system. The imaging unit includes a plurality of microlenses, and a plurality of photoelectric conversion units corresponding to each microlens,
22. The imaging apparatus according to claim 21, wherein the acquisition unit acquires a plurality of pieces of image data having different viewpoints from the imaging unit. A control method executed by an image processing apparatus that performs image processing of a plurality of image data having different viewpoints,
An obtaining step of obtaining the plurality of image data,
A setting step of setting a reference image area in the image of the first image data as the reference image and setting a reference image area in the image of the second image data as the reference image among the plurality of acquired image data sets When,
A calculating step of calculating a parallax amount of the plurality of image data by a correlation operation between the reference image area and the reference image area,
In the setting step, a process of determining the reference image region having a shape in which a luminance change value of an adjacent pixel in a boundary portion of the reference image region in the parallax direction is smaller than a threshold value is performed. The control method of an image processing device, wherein the correlation operation is performed between the reference image region and the reference image region having a shape corresponding to the shape of the reference image region. A program for causing a computer to execute each step of the control method according to claim 23.

Images