JP2019207185A - Image processing device, and imaging device and method - Google Patents

Abstract

To provide an image processing device, and an imaging device and method capable of acquiring highly accurate distance information.SOLUTION: An image processing device 100 includes acquisition means 110 and calculation means 120. The acquisition means acquires a second image of a first color component in which a blur function included in a first image affected by the aberration of an optical system is represented in non-point symmetry, and a third image of a second color component different from the first color component. The calculation means calculates a distance to a subject included in the first image according to a correlation between a plurality of fourth images obtained by adding a plurality of different blurs to the second image and the third image. The correlation is calculated on the basis of the aberration.SELECTED DRAWING: Figure 4

Description

  Embodiments described herein relate generally to an image processing apparatus, an imaging apparatus, and a method.

  For example, it is known to acquire distance information to a subject by imaging the subject from different directions with two cameras called stereo cameras.

  In recent years, a technique has been proposed in which distance information to an object is acquired from an image obtained by imaging the object with a single camera.

JP 2014-026051 A

  Incidentally, the accuracy of the distance information to the subject described above may be lowered due to blur or color shift caused by lens aberration (deviation from ideal image formation in the optical system).

  Therefore, the problem to be solved by the present invention is to provide an image processing device, an imaging device and a method capable of obtaining highly accurate distance information.

  The image processing apparatus according to the embodiment includes an acquisition unit and a calculation unit. The acquisition means includes a second image of the first color component in which the blur function is expressed asymptotically included in the first image affected by the aberration of the optical system, and a second image different from the first color component. A third image of two color components is acquired. The calculation means is configured to determine a subject included in the first image according to a correlation between a plurality of fourth images obtained by adding a plurality of different blurs to the second image and the third image. Calculate the distance. The correlation is calculated based on the aberration.

1 is a diagram illustrating an example of a hardware configuration of an imaging apparatus according to an embodiment. The figure for demonstrating an example of a filter. The figure which shows an example of the transmittance | permeability characteristic of a 1st filter area | region and a 2nd filter area | region. 2 is a diagram illustrating an example of a functional configuration of an imaging device. FIG. 6 is a flowchart illustrating an example of a processing procedure of the imaging apparatus. The figure for demonstrating the relationship between the distance to a to-be-photographed object and the blur shape which arises in an image. The figure which shows that the magnitude | size of the blur shape of a target image changes according to distance. The figure which shows that the magnitude | size of the blur shape of a reference | standard image changes according to distance. The figure which shows an example of a blur correction | amendment kernel. The figure for demonstrating the principle of a curvature of field. The figure for demonstrating the principle of a curvature of field. The figure which shows an example of distribution of the index value in case the aberration which arises by curvature of field is not correct | amended. The figure which shows an example of distribution of the index value at the time of correct | amending the aberration which arises by a curvature of field. 9 is a flowchart illustrating an example of a processing procedure of the imaging apparatus according to the second embodiment. The figure for demonstrating an example of the filter which concerns on 3rd Embodiment. The figure which shows the example of the transmittance | permeability characteristic of a 1st filter area | region and a 2nd filter area | region. 6 is a flowchart illustrating an example of a processing procedure of the imaging apparatus. The figure which shows an example of distribution of the index value in case the aberration which arises by curvature of field is not correct | amended. The figure which shows an example of distribution of the index value at the time of correct | amending the aberration which arises by a curvature of field. 10 is a flowchart illustrating an example of a processing procedure of an imaging apparatus according to a fourth embodiment. FIG. 11 is a block diagram illustrating an example of a structure of a moving object including an imaging device. The perspective view which shows an example of the external appearance of a motor vehicle provided with an imaging device. The perspective view which shows an example of the external appearance of a drone provided with an imaging device. The perspective view which shows an example of the external appearance of a robot provided with an imaging device. The perspective view which shows an example of the external appearance of a robot arm provided with an imaging device.

Hereinafter, each embodiment will be described with reference to the drawings.
(First embodiment)
First, the first embodiment will be described. FIG. 1 is a block diagram illustrating a hardware configuration of the imaging apparatus according to the first embodiment.

  As shown in FIG. 1, the imaging apparatus 100 includes a filter 10, a lens 20, an image sensor 30, a CPU (Central Processing Unit) 40, a memory 50, a communication I / F 60, a display 70, and a memory card. And a slot 80.

  The image sensor 30, CPU 40, memory 50, communication I / F 60, display 70, and memory card slot 80 are connected by a bus.

  In the example shown in FIG. 1, the arrow toward the filter 10 represents the incidence of light. The filter 10 is installed in the optical system of the imaging device 100. Note that the optical system of the imaging apparatus 100 includes the lens 20, the image sensor 30, and the like. That is, the filter 10 may be installed in the lens 20 or in the opening of the imaging device 100, or may be installed between the lens 20 and the image sensor 30.

  The filter 10 receives light reflected by the subject, and the incident light passes through the filter 10 and the lens 20. The light transmitted through the filter 10 and the lens 20 reaches the image sensor 30 and is received by the image sensor 30. The image sensor 30 generates an image composed of a plurality of pixels by converting the received light into an electrical signal (photoelectric conversion). In the following description, an image generated by the image sensor 30 is referred to as a captured image for convenience.

  The image sensor 30 is realized by, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. The image sensor 30 is, for example, a sensor (R sensor) that detects light in the red (R) wavelength band, a sensor (G sensor) that detects light in the green (G) wavelength band, and a blue (B) wavelength band. A sensor image (R image, G image, and B image) corresponding to each wavelength band (color component) is received by receiving light in the corresponding wavelength band by each sensor. Is generated. That is, the above-described captured image includes sensor images of R image, G image, and B image.

  The CPU 40 is a hardware processor that comprehensively controls the operation of the imaging apparatus 100. Specifically, the CPU 40 executes a program stored in the memory 50 or the like and controls the operation of the entire imaging apparatus 100.

  The memory 50 is a rewritable nonvolatile storage device such as an HDD (Hard Disk Drive) or a NAND flash memory. The memory 50 stores, for example, programs related to control of the imaging apparatus 100 and various data used for processing.

  The communication I / F 60 is an interface that controls communication with an external device and input of various instructions by a user, for example. The display 70 includes a liquid crystal display, a touch panel, and the like. The memory card slot 80 is configured such that a portable storage medium such as an SD memory card or an SDHC memory card can be inserted and used. When a storage medium is inserted into the memory card slot 80, data can be written to and read from the storage medium.

  Next, an example of the filter 10 will be described with reference to FIG. The filter 10 is a color filter and transmits light in a specific wavelength band. In the example shown in FIG. 2, the filter 10 includes a first filter region 11 and a second filter region 12.

  As shown in FIG. 2, the center of the filter 10 coincides with the optical center of the imaging device 100. The first filter region 11 and the second filter region 12 each have a shape that is asymmetric with respect to the optical center. In other words, the first filter region 11 and the second filter region 12 are configured not to be point-symmetric when the center point (the center of gravity position) of the filter region that is the optical center is a symmetric point. . Note that the first filter region 11 and the second filter region 12 do not overlap and constitute the entire region of the filter 10.

  In the example shown in FIG. 2, each of the first filter region 11 and the second filter region 12 has a semicircular shape in which the circular filter 10 is divided by a line segment passing through the optical center. The first filter region 11 is, for example, a yellow (Y) filter region, and the second filter region 12 is, for example, a cyan (C) filter region. In this case, the first filter region 11 transmits light in the red wavelength band and light in the green wavelength band, and attenuates light in the blue wavelength band. The second filter region 12 transmits light in the green wavelength band and light in the blue wavelength band, and attenuates light in the red wavelength band.

  The filter 10 has two or more color filter regions. Each of the color filter regions has an asymmetrical shape with respect to the optical center of the imaging apparatus 100. A part of the wavelength band of light transmitted through one color filter region and a part of the wavelength band of light transmitted through the other color filter area overlap, for example. That is, the wavelength band of light transmitted through one color filter region may include, for example, the wavelength band of light transmitted through another color filter region.

  When a subject is imaged through such a filter 10, sensor images (R image, G image, and B image) corresponding to each wavelength band having different blur functions are generated. The blur function is a function that represents the blur shape of an image captured using a point light source as a subject, and is also referred to as a point spread function (PSF).

  In addition, the 1st filter area | region 11 and the 2nd filter area | region 12 change the condensing power of the filter which changes the transmittance | permeability of arbitrary wavelength bands, the polarization filter which passes the polarized light of arbitrary directions, or arbitrary wavelengths band. A microlens may be used. Specifically, filters that change the transmittance of an arbitrary wavelength band include primary color filters (RGB), complementary color filters (CMY), color correction filters (CC-RGB / CMY), infrared or ultraviolet cut filters, ND filters, It may be a shielding plate. In the case where the first filter region 11 and the second filter region 12 are microlenses, the blur function changes due to a bias in the distribution of the condensed light rays by the lens 20.

  Here, FIG. 3 shows the first filter region 11 and the first filter region 11 when the first filter region 11 is a yellow (Y) filter region and the second filter region 12 is a cyan (C) filter region. The example of the transmittance | permeability characteristic of 2 filter area | regions is shown. FIG. 3 shows that the first filter region 11 transmits red light and green light and attenuates blue light as described above. FIG. 3 shows that the second filter region 12 transmits green light and blue light and attenuates red light.

  FIG. 4 shows an example of a functional configuration of the imaging apparatus 100 according to the present embodiment. As illustrated in FIG. 4, the imaging apparatus 100 includes an imaging unit 110 and an image processing unit 120. The imaging unit 110 corresponds to an imaging optical system and includes the filter 10, the lens 20, and the image sensor 30 described above.

  The image processing unit 120 is a functional unit that calculates the distance from the image including the subject imaged by the imaging unit 110 to the subject. In the present embodiment, a part or all of the image processing unit 120 is realized by causing a computer such as the CPU 40 to execute a program, that is, by software. A program to be executed by a computer may be stored and distributed in a computer-readable storage medium, or may be downloaded to the imaging apparatus 100 through a network. Part or all of the image processing unit 120 may be realized by hardware such as an IC (Integrated Circuit), or may be realized as a combination configuration of software and hardware.

  The image sensor 30 photoelectrically converts the light transmitted through the filter 10 and the lens 20 and sends an electric signal to the image processing unit 120. Although FIG. 4 shows a configuration in which the lens 20 is provided between the filter 10 and the image sensor 30, the filter 10 may be provided between the lens 20 and the image sensor 30 as described above. When there are a plurality of lenses 20, the filter 10 may be provided between the two lenses. The filter 10 may be provided inside the lens 20 or may be provided on the surface of the lens 20. That is, the filter 10 only needs to be provided at a position where the image sensor 30 can receive light transmitted through the filter 10 and generate an image.

  The image sensor 30 includes a first sensor 31, a second sensor 32, and a third sensor 33. For example, the first sensor 31 is an R sensor that detects light in the red wavelength band, the second sensor 32 is a G sensor that detects light in the green wavelength band, and the third sensor 33 is light in the blue wavelength band. B sensor for detecting

  Each of the first sensor 31, the second sensor 32, and the third sensor 33 generates a sensor image based on the detected light. Specifically, the first sensor 31 generates an R image based on the light in the red wavelength band detected by the first sensor 31. The second sensor 32 generates a G image based on the light in the green wavelength band detected by the second sensor 32. The third sensor 33 generates a B image based on the light in the blue wavelength band detected by the third sensor 33. Thereby, the image sensor 30 can generate a captured image including an R image, a G image, and a B image.

  Here, the second sensor 32 detects the light in the green wavelength band that has passed through both the first filter region 11 and the second filter region 12 as described above. For this reason, the G image generated by the second sensor 32 is an image that is brighter than other images (R image and B image), has less noise, and is less affected by the provision of the filter 10. I can say that. On the other hand, the R image generated by the first sensor 31 and the B image generated by the second sensor 33 are images generated from light transmitted through one of the first filter region 11 and the second filter region 12. Therefore, it is different from the G image.

  As described above, a captured image (an image generated by the image sensor 30) including the R image, the G image, and the B image generated by the first sensor 31, the second sensor 32, and the third sensor 33 is obtained from the image sensor 30. It is output to the image processing unit 120.

  As shown in FIG. 4, the image processing unit 120 includes a sensor control unit 121, a distance calculation unit 122, and an aberration information storage unit 123.

  The sensor control unit 121 controls the image sensor 30 and acquires captured images (R image, G image, and B image) output from the image sensor 30. Note that the R image, the G image, and the B image included in the captured image acquired by the sensor control unit 121 are images having different blur functions, for example, the target image and the blur function in which the blur function is changed to be astigmatic. Includes a reference image that is point-symmetric.

  The distance calculation unit 122 calculates the distance to the subject included in the captured image according to the correlation between the target image to which different blur is added based on the blur correction kernel described later and the reference image (that is, the distance to the subject). Get information).

  Here, the captured images (R image, G image, and B image) are affected by the aberration of the imaging optical system described above. For this reason, in this embodiment, the distance to the subject is calculated in consideration of the influence of the aberration of the imaging optical system.

  The aberration information storage unit 123 stores in advance information used when calculating the distance to the subject while correcting the aberration of the imaging optical system (hereinafter referred to as aberration information).

  In FIG. 4, the image processing unit 120 is described as being included in the imaging device 100, but the image processing unit 120 may be realized as a device (image processing device) separate from the imaging device 100. .

  Hereinafter, an example of a processing procedure of the imaging apparatus 100 according to the present embodiment will be described with reference to the flowchart of FIG. 5 is executed by the image processing unit 120 included in the imaging apparatus 100.

  Here, for example, when a subject is imaged by the imaging apparatus 100, a captured image including the subject is output from the image sensor 30 as described above.

  In this case, the sensor control unit 121 acquires the captured image output from the image sensor 30 (step S1). Note that the captured image acquired in step S1 includes sensor images (R image, G image, and B image) generated by each of the first sensor 31, the second sensor 32, and the third sensor 33 described above. At least one of the sensor images has the blur function changed to asymmetry by the filter 10. That is, the filter area (the first filter area 11 and the second filter area 12) of the filter 10 attenuates arbitrary light out of the light received by the image sensor 30, and is biased toward the distribution of light collection. As a result, the blur function of the sensor image can be changed. In the example of the filter 10 described with reference to FIG. 2, the B image and the R image correspond to images in which the blur function has been changed to asymmetry by the filter 10.

  Next, the sensor control unit 121 acquires a target image from the captured image acquired in step S1 (step S2). In the present embodiment, the target image is a sensor image generated by receiving light that has been attenuated (absorbed) by the first filter region 11 of the filter 10 and transmitted through the second filter region 12, for example, a B image. It is. The B image has a semicircular blur function, and the blur shape is a semicircular shape.

  In the following description, the case where the target image is a B image will be described. However, the target image is attenuated by the second filter region 12 and the sensor image generated by receiving the light transmitted through the first filter region 11 ( R image).

  The sensor control unit 121 acquires a reference image from the captured image acquired in step S1 (step S3). In the present embodiment, the reference image is a sensor image generated by receiving light transmitted through the first filter region 11 and the second filter region 12 of the filter 10, for example, and is a G image, for example. The G image has a circular blur function, and the blur shape is circular.

  Here, with reference to FIG. 6, the relationship between the distance to the subject and the blurred shape (blurring function) generated in the image will be described. The left column of FIG. 6 shows the combination of the lens 20 and the filter 10, the image sensor 30, and the subject when the imaging apparatus 100 is viewed from above (that is, the positive direction of the Y axis parallel to the dividing direction of the filter 10). The positional relationship is shown. The middle column and the right column in FIG. 6 represent blur shapes of the target image (B image) and the reference image (G image) formed on the image sensor 30 when the subject shown in the left column is imaged. Yes.

  In the following description, the distance from the focus position (hereinafter referred to as the focus position) in the imaging apparatus 100 to the subject is referred to as a distance d. The distance d is a positive value when the position of the subject is far from the focus position with respect to the imaging apparatus 100 with the focus position as the reference (0), and the position of the subject with respect to the imaging apparatus 100 is the focus position. If it is closer, the value is negative.

  First, it is assumed that the position of the subject is farther than the focus position, that is, the distance d> 0. In this case, since the subject is not in focus, the target image and the reference image are blurred as shown in the upper part of FIG.

  Further, the blur shape 201a of the target image when the distance d> 0 is, for example, an asymmetrical shape biased to the right as compared with the blur shape 202a of the point-symmetric reference image. The blur shape 202a of the reference image (G image) is point-symmetric because the first filter region 11 and the second filter region 12 of the filter 10 transmit green light almost evenly. On the other hand, the blur shape 201a of the target image (B image) is an asymmetrical shape (a shape biased to the right) because the first filter region 11 of the filter 10 attenuates blue light and the second filter region. 12 is for transmitting blue light.

  The blur shape described in the present embodiment refers to a blur shape generated in a predetermined range including a specific pixel. The same applies to the following description.

  Next, it is assumed that the position of the subject matches the focus position, that is, the distance d = 0. As shown in the middle part of FIG. 6, when the distance d = 0, no blur occurs in the target image and the reference image.

  Furthermore, it is assumed that the position of the subject is closer than the focus position, that is, the distance d <0. In this case, as shown in the lower part of FIG. 6, since the subject is not in focus, the target image and the reference image are blurred.

  Further, as described above, the target image (B image) is an image generated based on light (blue light) mainly transmitted through the second filter region 12, but the target image when the distance d <0 is satisfied. As shown in the lower part of FIG. 6, the blur shape 201 b is a shape that is biased to the left as compared with the blur shape 202 b of the reference image, for example.

  That is, the blur shape 201b is an astigmatic shape similar to the blur shape 201a described above, and is a shape obtained by inverting the blur shape 201a about a straight line parallel to the Y-axis direction.

  On the other hand, the blur shape 202b of the reference image in this case is a point-symmetric shape similar to the blur shape 202a of the reference image described above.

  As described above, in the target image, the blur shape changes according to the distance d. Specifically, if the distance d> 0, the blur shape of the target image changes to a semicircle shape (asymmetry shape) with the left side of the blur shape of the reference image missing, and the distance d <0. If there is, the shape of the reference image changes to a semicircular shape (asymmetry shape) with a missing right side of the blurred shape.

  Although not shown in FIG. 6, the size (width) of the blurred shape in the target image and the reference image depends on the distance | d |. FIG. 7 shows that the size of the blur shape of the target image changes according to the distance | d |. FIG. 8 shows that the size of the blurred shape of the reference image changes according to the distance | d |. That is, the size of the blur shape becomes larger (wider) as the distance | d | is larger.

  Here, in the present embodiment, sensor images having different blur functions are generated by receiving light through the filter 10 having the first filter region 11 and the second filter region 12 as described above. In the sensor image (target image and reference image), blur (image degradation) due to lens aberration (aberration of the imaging optical system) different from blur due to the filter 10 may occur.

  When the distance to the subject is calculated based on the target image and the reference image that are affected by the lens aberration in this way, the calculation accuracy of the distance may be reduced. Therefore, in calculating the distance to the subject, it is necessary to remove (correct) the influence of lens aberration. There are various lens aberrations, but in this embodiment, for example, aberrations caused by field curvature are targeted.

  Incidentally, aberrations that do not directly affect the calculation of the distance to the subject (that is, do not affect the blur shapes of the target image and the reference image) are preferably removed before the distance calculation processing. Aberrations that do not directly affect the calculation of the distance to the subject include, for example, lateral chromatic aberration and distortion.

  Returning to FIG. 5 again, the distance calculation unit 122 corrects the lateral chromatic aberration and distortion in the target image and the reference image based on the aberration information stored in the aberration information storage unit 123 (step S4).

  Note that lateral chromatic aberration refers to chromatic aberration in which the imaging position differs due to a difference in refractive index due to the wavelength band (color component) of light. According to this lateral chromatic aberration, a color shift in the image height direction occurs in the image. Further, according to the distortion aberration, there occurs a positional shift that the shape of the subject plane and the shape on the image plane (image) are not similar.

  Here, the aberration information stored in the aberration information storage unit 123 includes, for example, correction for lateral chromatic aberration and distortion aberration that holds horizontal and vertical correction amounts (deviation amounts) corresponding to local regions in the image. A table (hereinafter referred to as a first correction table).

  In step S4, referring to such a first correction table, the color shift caused by the chromatic aberration of magnification and the positional shift caused by the distortion aberration are corrected with respect to the target image and the reference image.

  In the present embodiment, it has been described that both the lateral chromatic aberration and the distortion aberration are corrected. However, depending on the influence of the aberration, for example, only one of the lateral chromatic aberration and the distortion aberration may be corrected, or the lateral chromatic aberration and the distortion aberration may be corrected. It is good also as a structure which does not correct | amend both. Specifically, for example, a lens having a small focal length (wide angle of view) has a large influence of distortion, whereas a lens having a large focal length (telephoto) has a larger influence of lateral chromatic aberration than the influence of distortion. large. For this reason, for example, when the focal length of the lens is smaller than a predetermined value, the distortion is corrected, and when the focal length of the lens is larger than a predetermined value, the lateral chromatic aberration is corrected. It may be configured. Whether or not to correct the lateral chromatic aberration or distortion may be determined according to other lens characteristics (such as the lens member).

  When the process of step S4 is executed, the process after step S5 is executed for each pixel constituting the target image and the reference image (captured image). Hereinafter, a pixel that is a target of processing in step S5 and subsequent steps will be described as a target pixel. In the following description, the blur shape of the target image is a blur shape within a predetermined range including the target pixels constituting the target image, and the blur shape of the reference image includes the target pixels constituting the reference image. It is assumed that the shape of blur is within a predetermined range.

  In this case, the distance calculation unit 122 generates a corrected image by correcting (changing) the blur shape of the target image by adding different blurs to the target image (step S5).

  Here, the blur shape between the target image and the reference image changes according to the distance d to the subject included in the captured image as described above. For this reason, in this embodiment, it is assumed that the distance to the subject included in the captured image is an arbitrary distance d, and a plurality of blur correction kernels (blur change filters) created for different distances d are prepared. By generating a corrected image in which the blur shape of the target image is corrected using the plurality of blur correction kernels, a blur correction kernel having a higher correlation between the generated corrected image and the reference image is specified, and the specified The distance to the subject is calculated by obtaining the distance d corresponding to the blur correction kernel. The blur correction kernel is a convolution function for correcting the blur shape of the target image to the blur shape of the reference image.

  Hereinafter, the correction image generation process in step S5 described above will be specifically described. First, an image (sensor image) that is not affected by the filter 10 is considered. In this case, if the distance from the subject included in the sensor image Ix to the focus position is d, the sensor image Ix is an ideal sensor image Iy with less blur and the sensor image that changes according to the distance d as described above. Using the blur function f (d), it can be expressed by the following equation (1).

  Note that the blur function f (d) of the sensor image described above is determined by the opening shape of the imaging apparatus 100 and the distance d. With respect to the distance d, with respect to the focus position as described above, d> 0 when the subject exists farther than the focus position, and d <0 when the subject exists closer to the focus position. When the aperture shape of the imaging apparatus 100 is a circular shape (point-symmetric shape), the blur function f (d) does not change before and after the focus position, so the blur function f (d) is a large distance d. It can be expressed as a Gaussian function in which the blur width varies depending on the depth | d |. The blur function f (d) may be expressed as a pull box function in which the blur width changes depending on the magnitude | d | of the distance d.

Next, consider an image (target image and reference image) affected by the filter 10. Here, the target image Ix _o can be expressed by the following equation (2) using a blur function f _o (d) determined from the characteristics of the aperture shape and the filter region, as in the equation (1). it can.

The reference image Ix _r can be expressed by the following expression (3) using the blur function f _r (d) determined from the characteristics of the aperture shape and the filter region, as in the expression (1).

Here, in the case of the filter 10 (first filter region 11 and second filter region 12) shown in FIG. 2, the blur function f _o (d) of the target image (B image) is the light attenuation in the first filter region 11. Due to the influence of the above, as described with reference to FIG. 6 described above, the shape changes to before and after the focus position d = 0. In addition, as shown in FIG. 7 described above, the blur function f _o (d) of the target image is x> 0 and the first filter region is d> 0 when d> 0 where the subject is far from the focus position. According to the light attenuation, a Gaussian function of the attenuated blur width | d | Similarly, the blur function f _o (d) of the target function is attenuated according to the light attenuation in the first filter region 11 when x <0 when d <0 where the subject is closer to the focus position. The Gaussian function of the blurred blur width | d |.

On the other hand, in the case of the filter 10 shown in FIG. 2, since the reference image (G image) is not affected by the first filter region 11 and the second filter region 12, the blur function f _r (d) = f of the reference image. (D).

By adding a blur in the target image Ix _o, the blurring function for the blur shape of the target image Ix _o match the blurring shape of the reference image Ix _r, defined as the blur correction kernel f _{c (d).} In this case, the reference image Ix _r described above can be expressed by the following expression (4) using the blur correction kernel f _c (d) and the target image Ix _o .

The blur correction kernel f _c (d) of the equation (4) is obtained by using the above-described equations (2) to (4), the blur function f _r (d) of the reference image Ix _{r and} the blur function f _o of the target image Ix _o. Using (d), it can be expressed as in the following formula (5).

In Equation (5), f _o ⁻¹ (d) is an inverse filter of the blur function f _o (d) of the target image. In this embodiment, it is possible to calculate the blur correction kernel f _c (d) by analyzing from the blur function of the reference image Ix _r and the target image Ix _o . The blur shape (blur function) of the target image Ix _o can be corrected to various blur shapes assuming an arbitrary distance d by using the blur correction kernel f _c (d).

  Here, FIG. 9 shows an example of the blur correction kernel described above. FIG. 9 shows a blur correction kernel that adds blur to the blur function of the target image when the distance d <0.

The blur correction kernel f _c (d) shown in FIG. 9 passes through the center point of the line segment that divides the first filter region 11 and the second filter region 12, and is linear in the direction perpendicular to the line segment. This corresponds to the blur function distributed in the vicinity of the straight line.

If a corrected image obtained by correcting the blur shape of the target image Ix _o using the blur correction kernel f _c (d) at an arbitrary distance d is Ix ′ _o (d), the corrected image Ix ′ _o (d) is It can be expressed by equation (6).

In step S5 described above, the distance calculation unit 122 adds the blur to the target image Ix _o using the blur function f _c (d) corresponding to the arbitrary distance d by using the equation (6), and the corrected image Ix ′ _O (d) can be generated.

Returning to FIG. 5 again, the distance calculation unit 122 compares the corrected image Ix ′ _o (d) generated in step S5 with the reference image Ix _r, and corrects the corrected image Ix ′ _o (d) and the reference image Ix _r. The degree of coincidence of the blurred shape is calculated (step S6). In step S6, the degree of coincidence of the blur shape is calculated based on the aberration of the optical system, as will be described later. As the degree of matching blur shape, the correlation between the corrected image Ix' _o and the reference image Ix _r in the rectangular region of any size around the target pixel need be computed. For calculating the degree of coincidence of the blur shape, for example, SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), NCC (Normalized Cross-Correlation), ZNCC (Zero-mean Normalized Cross-Correlation), Color Alignment Measure The edge image ZNCC or the like can be used. In the present embodiment, it is assumed that ZNCC that is robust to brightness fluctuations is used.

Next, the distance calculation unit 122 determines whether the blurred shape of the corrected image Ix ′ _o (d) and the reference image Ix _r match based on the matching degree calculated in step S6 (step S6). S7). Note that “the blurred shape matches” may include not only the case where the blurred shape completely matches, but also the case where the degree of matching is equal to or greater than a predetermined threshold. When an index using ZNCC, which will be described later, is used as the degree of coincidence, it may be determined in the determination process in step S7 that the blurred shapes match when the index value is less than a threshold value.

When it is determined that the blurred shape of the corrected image Ix ′ _o (d) and the reference image Ix _r match (YES in step S7), the distance d in the corrected image Ix ′ _o (d) is included in the captured image. Calculated as the distance to the subject.

On the other hand, when it is determined that the blurred shape of the corrected image Ix ′ _o (d) and the reference image Ix _r do not match (NO in step S7), the process returns to the above step S5 and is repeated. In this case, the process of step S5 is executed by changing the distance d in the blur correction kernel f _c (d).

In FIG. 5, although it has been described that the process ends when it is determined in step S7 that the blurred shape of the corrected image Ix ′ _o (d) and the reference image Ix _r match, for example, it is assumed. The degree of coincidence may be calculated for all distances d, and the distance d having the highest degree of coincidence may be calculated as the distance to the subject.

In other words, the distance calculation unit 122 can calculate the distance to the subject by obtaining the distance d that gives the highest correlation between the corrected image Ix ′ _o (d) and the reference image Ix _r .

  In addition, although the case where the process after step S5 is performed about a target pixel was demonstrated here, when the distance to a to-be-photographed object is calculated about the said target pixel, also about pixels other than the said target pixel after step S5 Processing is executed. Thereby, the distances to all the subjects included in the captured image (RGB image) can be calculated.

The coincidence degree calculation process in step S6 shown in FIG. 5 will be specifically described below. In step S6, the reference image Ix _r included in the captured image generated by the image sensor 30 and the distance d are assumed (that is, the blur is added to the target image Ix _o using the blur correction kernel f _c (d)). The index L (d) is calculated in the local region (the above-described rectangular region of any size) centered on the target pixel of the RGB image generated from the corrected image Ix ′ _o (d). When the reference image Ix _r as described above is assumed G image _{I G,} the target image Ix _o is a B image _{I B,} index L (d) is calculated by the following equation (7).

In the equation (7), _I'B (d) is a corrected image obtained by correcting the blurring shape of the B image _{I B} using the blur correction kernel _f c (d), _{I G} is G image (reference image) It is. Further, ∇I is an edge image of the blurred image I. The index L (d) in the equation (7) indicates that the smaller the value is, the higher the matching degree of the images is. That is, in the present embodiment, this index L (d) can be used as the degree of coincidence.

  Here, as described above, there is an aberration in the actual lens 20, and the calculation (estimation) accuracy of the distance to the subject in the present embodiment may be lowered. Note that the above-mentioned field curvature is assumed as the aberration that directly affects the calculation of the distance to the subject. The field curvature refers to monochromatic aberration in which, for example, the condensing surface is curved due to the influence of a spherical lens, and the image surface becomes spherical.

  Hereinafter, the principle of field curvature will be described with reference to FIGS. 10A and 10B. FIG. 10A is a diagram for explaining the influence of aberration caused by field curvature when the filter 10 according to the present embodiment is not provided. FIG. 10B is a diagram for explaining the influence of aberration caused by field curvature when the filter 10 according to the present embodiment is provided.

  As described above, the curvature of field, which is a monochromatic aberration, is a phenomenon in which when a plane object is imaged, even if the center of the image sensor is in focus, the image is out of focus at the end of the captured image.

  When the filter 10 in the present embodiment is not provided, the lens transmits all light regardless of the wavelength band, so that the target image (B image) and the reference image (G image) are shown in FIG. 10A. The blur function observed at and has the same shape.

  On the other hand, when the filter 10 according to the present embodiment is provided (that is, when the first filter region 11 that blocks a part of the wavelength band exists), as shown in FIG. The blur function of the target image (B image) changes.

  As described above, the field curvature is originally monochromatic aberration, but the change in the blur function shown in FIG. 10B is caused by the insertion of the filter 10 as chromatic aberration.

  In the present embodiment, in order to calculate the degree of coincidence with high accuracy and reduce the error while mitigating (correcting) the influence of the chromatic aberration generated in this way, The calculated index L (d) is calculated. Note that c in the equation (8) is a correction amount of deviation due to field curvature.

  Here, FIG. 11A shows the distribution of the index value (index L (d)) calculated using the above equation (7) when the aberration caused by the curvature of field is not corrected. . On the other hand, FIG. 11B shows a distribution of index values when the aberration caused by the curvature of field is corrected, that is, the index value distribution calculated using the above equation (8).

  When the aberration caused by the curvature of field is not corrected, as shown in FIG. 11A, the minimum index value (that is, d having the highest degree of coincidence) is deviated from the true value.

  On the other hand, as shown in FIG. 11B, for example, an index value is obtained by using a correction table for field curvature (hereinafter referred to as a second correction table) in which the correction amount of deviation due to field curvature is defined. By changing (shifting) the correspondence relationship of the distance d, it is possible to correct the shift due to the curvature of field. This second correction table is stored in advance in the aberration information storage unit 123 as the aberration information described above.

  Here, since the correction processing of the aberration caused by the curvature of field is performed in each pixel, a second correction table for the entire image (that is, a correction amount is defined for each pixel constituting the image) is prepared. Shall. The second correction table is created by imaging the evaluation pattern in advance and measuring the amount of deviation from the plane. In order to obtain the shift amount of the entire image from the measurement points obtained in this way, model approximation such as a polynomial approximation curve or Zernike polynomial is used. If the model coefficient is known, the correction amount of the entire image can be calculated by development. For this reason, the second correction table stored in the aberration information storage unit 123 as the aberration information described above may be a model coefficient or a correction value of all the pixels after development.

  Since the aberration caused by the curvature of field may change according to the distance to the subject (imaging distance), the second correction table may be switched according to the distance. In this case, for example, the index L (d) is calculated by the following equation (9).

  In this equation (9), c (d) is a correction amount of deviation due to field curvature, and the correction amount is changed according to the distance d. c (d) is specified by preparing a plurality of second correction tables corresponding to the distance and referring to the second correction table corresponding to the distance closest to the distance d among the plurality of second correction tables. Alternatively, it may be obtained by linear interpolation from a plurality of second correction tables corresponding to distances close to the distance d, or may be a value developed by model approximation in the distance direction.

  As described above, in the present embodiment, the blur function included in the captured image (first image) affected by the aberration (lens aberration) of the imaging optical system is expressed, for example, in blue (first) Color component) target image (second image) and, for example, a green (second color component) reference image (third image) in which the blur function is expressed in point symmetry, The distance to the subject included in the captured image is calculated according to the correlation between the plurality of images (corrected images) obtained by adding different blurs and the reference image. In this embodiment, the correlation between images for calculating the distance to the subject is calculated based on the aberration of the imaging optical system. The aberration of the imaging optical system is imaged using the filter 10. Aberrations caused by curvature of field that affect the calculation of the distance to the subject.

  In other words, in the present embodiment, the blur shape of the target image changed according to the filter 10 installed in the optical system of the imaging apparatus 100 is the blur correction kernel assuming the distance d (the blur function of the target image is used as the reference image). A corrected image corrected by a convolution function for correcting to a blur function is generated, and a subject (captured image) included in the captured image is obtained by obtaining a distance d at which the correlation between the generated corrected image and the reference image becomes higher. When calculating the distance to the subject shown in (2), the deviation due to the curvature of field measured in advance is corrected.

  Note that the blur correction kernel can be calculated by analyzing the blur function of the target image and the blur function of the reference image. In the present embodiment, a plurality of blur correction kernels are used for correcting blur of a subject at an arbitrary distance. It is assumed that a blur correction kernel is prepared in advance.

  In the present embodiment, with such a configuration, the distance is calculated using the correlation between the image where the blur function and the sampling position coincide with each other while correcting the shift caused by the lens aberration (field curvature). Distance information can be obtained, and distance estimation with less error can be realized. Furthermore, in this embodiment, since blur information that is a result of convolution of spatial information is used, the result of distance estimation is stable, and distance estimation that is not affected by a repetitive pattern or hidden surface is realized. Can do.

  In the present embodiment, it is possible to generate a distance image or the like processed based on the distance (distance information) to the subject calculated as described above. Specifically, as the distance image, for example, according to the distance to the subject, it is possible to generate an image in which the subject existing in the foreground is brighter and the subject existing in the back is darker.

  Further, in the present embodiment, it is possible to correct a color shift caused by lateral chromatic aberration or a position shift caused by distortion aberration with respect to the target image and the reference image used for calculating the distance to the subject in advance. Therefore, more accurate distance information can be obtained.

  In the present embodiment, the second filter region 12 is described as a cyan filter region. However, when the target image is a B image and the reference image is a G image as in the process shown in FIG. The second filter region 12 may be another filter region as long as it transmits light in the green and blue wavelength bands.

  For example, when the second filter region 12 is a transparent filter region or the like that transmits light in all the wavelength bands, the red wavelength band light is also transmitted in the first filter region 11 and the same as the green wavelength band light. It passes through both of the second filter regions 12. In this case, the R image may be used as the reference image. Also, using both the G image and the R image as reference images, an index (degree of coincidence) is calculated for each of the G image and the R image, and the distance to the subject is calculated using a weighted average of the index. You may do it.

  Further, as described in the present embodiment, when the first filter region is the yellow filter region and the second filter region 12 is the cyan filter region, the light in the red wavelength band is the first filter region 11. , And is attenuated by the second filter region 12, the blur function of the R image is also changed to astigmatism similarly to the blur number of the B image. In this case, the R image may be the target image instead of the B image. Note that the blur function (blurred shape) of the R image is a shape obtained by inverting the blurred shape of the B image about a straight line parallel to the Y-axis direction.

  Note that the filter 10 in the present embodiment may have any number of filter regions as long as the filter 10 is not point-symmetric. Further, the filter 10 may be configured by one filter region or a plurality of filter regions as long as the blur function of the sensor image can be changed.

  In the present embodiment, the first filter region 11 and the second filter region 12 have a shape obtained by dividing the filter 10 by an arbitrary straight line, and the straight line passes through the optical center. According to such a configuration, the dimension of the blur correction kernel can be reduced, and the blur function of the sensor image can be changed even when an aperture mechanism such as a shield for adjusting the amount of light is inserted. Structure.

  In the present embodiment, the image sensor 30 has been described as including the first sensor 31 (R sensor), the second sensor 32 (G sensor), and the third sensor 33 (B sensor). By having the above sensors and passing light through the filter 10, it is possible to generate a target image in which the blur function is changed to asymmetry, a reference image that is at least one type of sensor image, and the like. What is necessary is just to be comprised. The two or more sensors are, for example, a combination of two or more sensors such as R sensor and G sensor, G sensor and B sensor, R sensor and B sensor, R sensor, G sensor, and B sensor among RGB sensors. . That is, the image sensor 30 may be configured such that, for example, one of two or more sensors generates a target image, and another sensor generates a reference image.

  In the present embodiment, the image sensor 30 is mainly described as including an RGB sensor, but the wavelength band of the sensor included in the image sensor 30 is not limited thereto.

(Second Embodiment)
Next, a second embodiment will be described. Note that the hardware configuration, filter, and functional configuration of the imaging apparatus according to this embodiment are the same as those in the first embodiment described above, and therefore will be described with reference to FIGS. 1, 2, 4, and the like as appropriate. In the following description, detailed description of the same parts as those of the first embodiment described above will be omitted, and parts different from those of the first embodiment will be mainly described.

  In the first embodiment described above, the chromatic aberration of magnification is corrected. However, in the first embodiment, the chromatic aberration of magnification that varies depending on the distance is not supported.

  Therefore, this embodiment is different from the first embodiment described above in that the magnification chromatic aberration correction is performed at the timing of calculating the distance to the subject.

  Hereinafter, an example of a processing procedure of the imaging apparatus 100 according to the present embodiment will be described with reference to the flowchart of FIG. Note that the processing illustrated in FIG. 12 is executed by the image processing unit 120 included in the imaging apparatus 100.

  First, the processes of steps S11 to S13 corresponding to the processes of steps S1 to S3 shown in FIG. 5 described above are executed.

  Next, the distance calculation unit 122 corrects the chromatic aberration of magnification in the target image and the reference image based on the aberration information (first correction table) stored in the aberration information storage unit 123 (step S14).

  When the process of step S14 is executed, the processes of steps S15 to S17 corresponding to the processes of steps S5 to S7 shown in FIG. 5 are executed. However, in step S14, the steps S15 and S16 are executed. The color shift caused by the chromatic aberration of magnification is corrected by the correction amount corresponding to the distance d used in FIG.

  In this case, for the correction amount corresponding to the distance d, for example, a plurality of first correction tables corresponding to the distance are prepared, and the first correction corresponding to the distance closest to the distance d among the plurality of first correction tables is prepared. The value may be specified by referring to the table, may be obtained by linear interpolation from a plurality of first correction tables corresponding to distances close to the distance d, or may be a value developed by model approximation in the distance direction It is good.

In the present embodiment, the chromatic aberration of magnification is corrected by a correction amount corresponding to the distance d. Therefore, the blurred shape of the corrected image Ix ′ _o (d) and the reference image Ix _r do not match in step S17. (NO in step S17), the process returns to step S14 and the process is repeated.

  In the present embodiment, by executing the above-described processing, it is possible to correct lateral chromatic aberration with a correction amount corresponding to the distance d at the timing of calculating the distance to the subject.

  Further, in the first embodiment described above, it has been described that the processing after step S5 is executed for each pixel constituting the target image and the reference image. However, in the present embodiment, the target image and the reference image are configured. The process after step S14 is performed about each pixel to perform.

  Although omitted in FIG. 12, the processing for correcting distortion described in the first embodiment may be executed after the processing in step S13.

  As described above, in the present embodiment, it is possible to obtain more accurate distance information by correcting the magnification chromatic aberration in consideration of the point that the influence of the magnification chromatic aberration varies depending on the distance, and the error can be obtained. Less distance estimation can be realized.

(Third embodiment)
Next, a third embodiment will be described. In the first embodiment described above, the sensor image (eg, G image) is described as the reference image. However, in the present embodiment, an image obtained by adding blur to any one of the sensor images is used as the reference image. This is different from the first embodiment described above. The hardware configuration of the imaging apparatus according to the present embodiment is the same as that of the first embodiment described above, and will be described with reference to FIG. 1 as appropriate. In the following description, detailed description of the same parts as those of the first embodiment will be omitted, and parts different from those of the first embodiment will be mainly described.

  First, an example of the filter 10 in the present embodiment will be described with reference to FIG. Here, differences from the filter 10 in the first embodiment described above will be described.

  As shown in FIG. 13, the filter 10 in the present embodiment includes a first filter region 13 and a second filter region 14.

  In the present embodiment, the first filter region 13 and the second filter region 14 have different transmittance characteristics in any wavelength band of the RGB sensor of the image sensor 30. Specifically, description will be made assuming that the first filter region 13 is a yellow color correction filter (CC-Y) and the second filter region 14 is a cyan color correction filter (CC-C).

  FIG. 14 shows an example of the transmittance characteristics of the first filter region 13 and the second filter region 14 described above.

  As shown in FIG. 14, the first filter region 13 transmits light in all wavelength bands of red light, green light, and blue light. Further, the second filter region 14 transmits light in all wavelength bands of red light, green light, and blue light.

  However, the transmittance of each wavelength band is different between the first filter region 13 and the second filter region 14. In addition, the 1st filter area | region 13 and the 2nd filter area | region 14 change the transmittance | permeability of arbitrary wavelength bands, the polarization filter which passes the polarized light of arbitrary directions, or the condensing power of arbitrary wavelength bands It may be a micro lens. Specifically, filters that change the transmittance of an arbitrary wavelength band include primary color filters (RGB), complementary color filters (CMY), color correction filters (CC-RGB / CMY), infrared or ultraviolet cut filters, ND filters, It may be a shielding plate. In the case where the first filter region 13 and the second filter region 13 are microlenses, the blur function changes due to a bias in the distribution of the condensed light rays by the lens 20.

  The imaging apparatus 100 according to the present embodiment can obtain distance information indicating the distance to the subject by imaging an arbitrary subject via the filter 10 having the above-described configuration.

  Note that the functional configuration of the imaging apparatus 100 according to this embodiment is the same as that of the first embodiment described above, but in this embodiment, the distance calculation unit 122 included in the image processing unit 120 is prepared in advance. Subject included in captured image according to correlation between target image (corrected image) with different blur added based on multiple blur correction kernels and reference image with blur added to any sensor image (target image) The distance to is calculated.

  Hereinafter, an example of a processing procedure of the imaging apparatus 100 according to the present embodiment will be described with reference to the flowchart of FIG. The process illustrated in FIG. 15 is executed by the image processing unit 120 included in the imaging apparatus 100.

  First, the process of step S21 corresponding to the process of step S1 shown in FIG. 5 described above is executed.

  Next, the sensor control unit 121 acquires a target image from the captured image acquired in step S21 (step S22).

  Here, the blur function of the target image changes asymmetrically by the filter 10 (the first filter region 13 and the second filter region 14). As a result, blurring of the target image is caused when the position of the subject exists farther than the focus position (d> 0) and when the position of the subject exists closer than the focus position (d <0). Astigmatism is blurred, and the shape is different before and after the focus position. Specifically, light observed in each wavelength band of the first sensor 31 (R sensor), the second sensor 32 (G sensor), and the third sensor 33 (B sensor) is transmitted to the first filter region 13 and the first filter region 13. The deviation of the blur shape of the target image is determined depending on which of the two filter regions 13 has passed. Note that when the position of the subject exists at the focus position (d = 0), the target image is an image with no blur.

  In the case of the filter 10 shown in FIG. 13, the first filter region 13 and the second filter region 14 cause the red wavelength band light, the green wavelength band light, and the blue wavelength band light to all be reflected. Attenuated, passed, and received by the image sensor 30. For this reason, in this embodiment, all the sensor images (R image, G image, and B image) included in the captured image acquired in step S21 can be the target images. Note that the target image may not be all images of the R image, the G image, and the B image. The sensor control unit 121 acquires a sensor image selected from a plurality of sensor images included in the captured image as a target image.

  Of the light passing through the filter 10, blue light is more absorbed and attenuated by the CC-Y filter (first filter region) than by the CC-C filter (second filter region 13). For this reason, when the position of the subject exists farther than the focus position (d> 0), the B image is blurred to the right as compared with the circular shape, and the position of the subject is closer to the focus position. In the image (d <0), the B image is blurred to the left as compared with the circular shape.

  Of the light passing through the filter 10, red light is more absorbed and attenuated by the CC-C filter (second filter region 14) than by the CC-Y filter (first filter region 13). For this reason, when the position of the subject exists farther than the focus position (d> 0), the R image is blurred to the left as compared with the circular shape, and the position of the subject is closer to the focus position. In the case (d <0), the R image is blurred to the right as compared with the circular shape.

  Further, among the light passing through the filter 10, green light is more absorbed and attenuated by the CC-Y filter (first filter region 13) than by the CC-C filter (second filter region 14). For this reason, when the position of the subject exists farther than the focus position (d> 0), the G image is blurred to the right as compared to the circular shape, and the position of the subject is closer to the focus position. In the G image (d <0), the G image is blurred to the left as compared with the circular shape.

  In the following description, it is assumed that all sensor images (R image, G image, and B image) included in the captured image are acquired as target images in step S22.

  When the process of step S22 is executed, the process of step S23 corresponding to the process of step S4 shown in FIG. 5 is executed.

  Next, the process after step S24 is performed for every pixel which comprises a target image (captured image). Hereinafter, description will be made assuming that the pixel to be processed in step S24 and subsequent steps is the target pixel. In the following description, the blur shape of the target image is a blur shape within a predetermined range including pixels constituting the target pixel.

  In this case, the distance calculation unit 122 generates a reference image in which the blur shape of at least one target image (hereinafter referred to as a first target image) among the target images acquired in step 22 is corrected to a blur of an arbitrary shape. (Step S24).

  In step S24, the distance calculation unit 122 assumes that the distance to the subject is an arbitrary distance d, and uses the blur correction kernel created for the distance d to generate a reference image obtained by correcting the blur shape of the first target image. Generate. Note that the process of correcting the blur shape of the first target image to a blur of an arbitrary shape is the same as the process of step S5 shown in FIG.

  In the following description, it is assumed that the blur shape of the reference image generated in step S24 is a circular shape, but the blur shape of the reference image may be a blur shape of an arbitrary sensor image (target image), Other shapes different from these may be used.

Here, the blur correction kernel f ′ _c (d) for generating the reference image by correcting the blur shape of the arbitrarily selected first target image Ix _o is the blur function f of the reference image in the above-described equation (5). _By setting _r (d) to f ′ _r (d), it can be expressed as the following formula (10).

The reference image Ix ′ _r (d) obtained by correcting the blur of the first target image Ix _o is expressed by the following equation (11) using the blur correction kernel f ′ _c (d) represented by the equation (10). Can be expressed as

In the present embodiment, as described above, it is possible to generate the reference image Ix ′ _r (d) when it is assumed that the distance to the subject is d.

  Next, the distance calculation unit 122 generates a corrected image in which the blur shape of a target image other than the first target image (hereinafter referred to as a second target image) among the target images acquired in step S22 is corrected ( Step S25). Note that the process of correcting the blurred shape of the second target image is the same as the process of step S5 shown in FIG.

Specifically, a blur correction kernel f for adding blur to the arbitrarily selected second target image and matching the blur shape of the second target image with the blur shape of the reference image generated in step S24. ′ _C (d) can be obtained by the above-described equation (10). The distance calculation unit 122 generates a corrected image in which the blur shape of the second target image is corrected using the obtained blur correction kernel f ′ _c (d).

Here, if a corrected image obtained by correcting the blur shape of the second target image Ix _o using the blur correction kernel f ′ _c (d) is Ix ′ _o (d), the corrected image Ix ′ _o (d) is (12).

  In the present embodiment, as described above, it is possible to generate the corrected image Ix′o (d) when it is assumed that the distance to the subject is d.

  Note that the processes in steps S24 and S25 described above are executed for each combination of target images acquired in step S22. Specifically, when each of the R image, the G image, and the B image included in the captured image is acquired as the target image as described above, for example, the first target image is the G image and the second target image. Steps S24 and S25 are executed with the first target image as the B image and the second target image as the G image, and the processes at Steps S24 and S25 are executed with the first target image as the R image and the second target image. The processes of steps S24 and S25 are executed with B as the B image.

Next, the distance calculation unit 122 compares the corrected image Ix ′ _o (d) generated in step S25 with the reference image Ix ′ _r (d), and the corrected image Ix ′ _o (d) and the reference image Ix ′. _The degree of coincidence of the blurred shape with _r (d) is calculated (step S26). In step S26, the degree of coincidence of the blur shape is calculated based on the aberration of the optical system, as will be described later.

  Hereinafter, the coincidence degree calculation process in step S26 will be described in detail. For example, SSD, SAD, NCC, ZNCC, Color Alignment Measure, and ZNCC of an edge image can be used to calculate the degree of coincidence of the blur shape. Here, ZNCC that is robust to variations in brightness is used. The case will be described.

In step S26, the index L (d) is a local region centered on the target pixel of the RGB image composed of the reference image Ix ′ _r (d) and the corrected image Ix ′ _o (d) assuming the distance d. Calculate As described above, when the processes of steps S24 and S25 are executed for each combination of target images (R image, G image, and B image), the index L (d) is calculated by the following equation (13). The

In Expression _{(13), I'R (d} ) is a corrected image or the reference image by correcting the blurring shape of the R image _{I R} with a blur correction kernel _f'c (d). I ′ _G (d) and I ′ _B (d) are obtained by converting the R image of I ′ _R (d) into a G image and a B image. Further, ∇I is an edge image of the blurred image I. The index L (d) in the equation (13) indicates that the smaller the value is, the higher the matching degree of the images is. In the present embodiment, this index L (d) can be used as the degree of coincidence.

  In Expression (13), the weight of the combination of sensor images (colors) is an arithmetic average, but the transmittance characteristics of the filter 10 (first filter region 13 and second filter region 14) and each sensor (first It is good also as a weighted average which considered the spectral sensitivity characteristic of the sensor 31, the 2nd sensor 32, and the 3rd sensor 33).

  Here, there is an aberration in the actual lens 20, and the calculation (estimation) accuracy of the distance to the subject in the present embodiment may be lowered. Note that curvature of field is assumed as an aberration that directly affects the calculation of the distance to the subject. Since the field curvature principle and the like are as described in the first embodiment, detailed description thereof is omitted here.

In the present embodiment, in order to calculate a degree of coincidence with high accuracy and reduce distance with less error while reducing (correcting) the influence of aberration (chromatic aberration) caused by field curvature, the following equation (14) ) To calculate the above-described index L (d). Note that c _RG , c _GB, and c _BR in Equation (14) are correction amounts for deviation due to field curvature.

  Here, FIG. 16A shows the distribution of the index value (index L (d)) calculated using the above equation (13) when the aberration caused by the curvature of field is not corrected. . On the other hand, FIG. 16B shows an index value when the aberration caused by the curvature of field is corrected, that is, a distribution of index values calculated using the above-described equation (14).

  When the aberration caused by the curvature of field is not corrected, as shown in FIG. 16A, the minimum index value (that is, d having the highest degree of coincidence) for each combination of sensor images is shifted from the true value. ing.

  Here, in the present embodiment, a correction table for field curvature (hereinafter referred to as a third correction table) in which the correction amount of deviation due to field curvature is defined is prepared in advance for each combination of sensor images. It is assumed that This third correction table may be stored in advance in the aberration information storage unit 123 as aberration information.

  According to this, as shown in FIG. 16B, a third correction table prepared for each combination of sensor images (that is, an RG correction table, a GB correction table, and a BR correction table). By changing (shifting) the correspondence between the index value and the distance d using, the deviation due to the curvature of field can be corrected.

  Here, since the correction processing of the aberration caused by the curvature of field is performed in each pixel, it is assumed that a third correction table for the entire image is prepared. The third correction table is created by imaging the evaluation pattern in advance and measuring the amount of deviation from the plane. In order to obtain the shift amount of the entire image from the measurement points obtained in this way, model approximation such as a polynomial approximation curve or Zernike polynomial is used. If the model coefficient is known, the correction amount of the entire image can be calculated by development. For this reason, the third correction table stored in the aberration information storage unit 123 as the aberration information described above may be a model coefficient or a correction value of all the pixels after development.

  Since the aberration caused by the curvature of field may change according to the distance to the subject (imaging distance), the third correction table may be switched according to the distance. In this case, for example, the index L (d) is calculated by the following equation (15).

In this equation (15), c _RG (d), c _GB (d), and c _BR (d) are correction amounts for deviation due to field curvature, and the correction amounts are changed according to the distance d. For c _RG (d), c _GB (d), and c _BR (d), a plurality of third correction tables corresponding to the distance are prepared, and according to the distance closest to the distance d among the plurality of third correction tables. May be specified by referring to the third correction table, or may be obtained by linear interpolation from a plurality of third correction tables corresponding to distances close to the distance d, or by model approximation with respect to the distance direction. It may be an expanded value.

The distance calculation unit 122 determines whether the blurred shapes of the corrected image Ix ′ _o (d) and the reference image Ix ′ _r (d) match based on the degree of matching calculated in step S <b> 26 ( Step S27). As described above, when the index L (d) using ZNCC is used as the degree of coincidence, if the index L (d) is less than the threshold, the corrected image Ix ′ _o (d) and the reference image Ix ′ _r ( It is determined that the blurred shape matches with d).

When it is determined that the blurred shape of the corrected image Ix ′ _o (d) and the reference image Ix ′ _r (d) match (YES in step S27), the distance d is the distance to the subject included in the captured image. Is calculated as

On the other hand, when it is determined that the blurred shape of the corrected image Ix ′ _o (d) and the reference image Ix ′ _r (d) do not match (NO in step S27), the process returns to the above step S24 and the process is repeated. It is. In this case, the reference image and the corrected image are generated at different distances d.

In FIG. 15, the description has been made assuming that the processing is ended when it is determined in step S <b> 27 that the corrected image Ix ′ _o (d) and the reference image Ix ′ _r (d) match. For example, the degree of coincidence is calculated for all assumed distances d, and the distance d having the highest degree of coincidence (for example, the distance d with the smallest index L (d)) is calculated as the distance to the subject. Good.

In other words, the distance calculation unit 122 can calculate the distance to the subject by obtaining the distance d that gives the highest correlation between the corrected image Ix ′ _o (d) and the reference image Ix ′ _r (d). it can.

  In addition, although the case where the process after step S24 is performed about a target pixel was demonstrated here, when the distance to a to-be-photographed object is calculated about the said target pixel, also about pixels other than the said target pixel after step S24 Processing is executed. Thereby, the distances to all the subjects included in the captured image (RGB image) can be calculated.

  As described above, in the present embodiment, a corrected image obtained by adding a plurality of different blurs to the second target image and a reference image obtained by adding blur to the first target image (third image) ( The distance to the subject is calculated according to the correlation with the fifth image). Specifically, in the present embodiment, a reference image and a corrected image are generated by correcting the blur shape of a plurality of target images changed according to the filter 10 to an arbitrary blur shape assuming a distance d, and The distance to the subject is calculated by obtaining the distance d at which the correlation between the reference image and the corrected image becomes higher.

  In the present embodiment, with such a configuration, it is possible to obtain highly accurate distance information as in the first embodiment described above, and it is possible to realize distance estimation with less error.

(Fourth embodiment)
Next, a fourth embodiment will be described. Note that the hardware configuration, filter, and functional configuration of the imaging apparatus according to this embodiment are the same as those in the first embodiment described above, and therefore will be described with reference to FIGS. 1, 2, 4, and the like as appropriate. In the following description, detailed description of the same parts as those of the first embodiment described above will be omitted, and parts different from those of the first embodiment will be mainly described.

  Although the third embodiment has been described as correcting the lateral chromatic aberration, the third embodiment does not support the lateral chromatic aberration that varies depending on the distance.

  Therefore, this embodiment is different from the first embodiment described above in that the magnification chromatic aberration correction is performed at the timing of calculating the distance to the subject.

  Hereinafter, an example of a processing procedure of the imaging apparatus 100 according to the present embodiment will be described with reference to the flowchart of FIG. Note that the processing illustrated in FIG. 17 is executed by the image processing unit 120 included in the imaging apparatus 100.

  First, the processes of steps S31 and S32 corresponding to the processes of steps S21 and S22 shown in FIG. 15 are executed.

  Next, the distance calculation unit 122 corrects the chromatic aberration of magnification between the target images based on the aberration information (first correction table) stored in the aberration information storage unit 123 (step S33).

  In addition, when the process of step S33 is performed, the process of step S34-S37 equivalent to the process of step S24-S27 shown in FIG. 15 mentioned above is performed, However, In step S33, the said step S34 and S35 are performed. The color shift caused by the chromatic aberration of magnification is corrected by the correction amount corresponding to the distance d used in FIG.

  In this case, for the correction amount corresponding to the distance d, for example, a plurality of first correction tables corresponding to the distance are prepared, and the first correction corresponding to the distance closest to the distance d among the plurality of first correction tables is prepared. The value may be specified by referring to the table, may be obtained by linear interpolation from a plurality of first correction tables corresponding to distances close to the distance d, or may be a value developed by model approximation in the distance direction It is good.

In the present embodiment, since the chromatic aberration of magnification is corrected by a correction amount corresponding to the distance d, the blurred shape between the corrected image Ix ′ _o (d) and the reference image Ix ′ _r (d) in step S37. Are determined not to match (NO in step S37), the process returns to step S33 and the process is repeated.

  In the present embodiment, by executing the above-described processing, it is possible to correct lateral chromatic aberration with a correction amount corresponding to the distance d at the timing of calculating the distance to the subject.

  Further, in the third embodiment described above, it has been described that the processing after step S24 is executed for each pixel constituting the target image. However, in this embodiment, step S33 is performed for each pixel constituting the target image. The subsequent processing is executed.

  Although omitted in FIG. 17, the process for correcting distortion described in the first embodiment may be executed after the process of step S32.

  As described above, in the present embodiment, it is possible to obtain more accurate distance information by correcting the magnification chromatic aberration in consideration of the point that the influence of the magnification chromatic aberration varies depending on the distance, and the error can be obtained. Less distance estimation can be realized.

  According to at least one embodiment described above, it is possible to provide an image processing apparatus, an imaging apparatus, and a method that can obtain highly accurate distance information.

  The information including the processing procedure, control procedure, specific name, various data, parameters, and the like described above can be arbitrarily changed unless otherwise specified. Each component of the illustrated apparatus is functionally conceptual and does not necessarily have to be physically configured as illustrated. That is, the specific form of the distribution or integration of the devices is not limited to the illustrated one, and all or a part thereof is functionally or physically distributed or arbitrarily distributed in arbitrary units according to various burdens or usage conditions. Can be integrated.

  In addition, the function of the image processing unit 120 included in the imaging device 100 according to each of the above-described embodiments can be realized by using a general-purpose computer device as basic hardware, for example. The executed program (image processing program) has a module configuration including the above-described functions. The program to be executed may be provided by being recorded in a computer-readable storage medium such as a CD-ROM, CD-R, DVD or the like in an installable or executable format file, a ROM, etc. May be provided in advance.

(Application examples)
Hereinafter, some application examples to which the imaging apparatus 100 having the above-described configuration is applied will be described.

  FIG. 18 illustrates a functional configuration example of the moving object 300 including the imaging device 100. The mobile object 300 can be realized as, for example, an automobile having an automatic driving function, an unmanned aircraft, an autonomous mobile robot, or the like. Unmanned aerial vehicles are airplanes, rotary wing aircraft, gliders, airships that humans cannot ride on, and can be made to fly by remote control or autopilot, such as drones (multicopters), radio control aircraft, Includes helicopters for spraying agricultural chemicals. Autonomous mobile robots include a mobile robot such as an automated guided vehicle (AGV), a cleaning robot for cleaning a floor, a communication robot for performing various guidance for visitors. The moving body 300 may include not only a robot body that moves but also an industrial robot having a drive mechanism for moving and rotating a part of the robot, such as a robot arm.

  As illustrated in FIG. 18, the moving body 300 includes, for example, an imaging device 100, a control signal generation unit 301, and a drive mechanism 302. At least the imaging unit 110 of the imaging apparatus 100 is installed so as to image a subject in the traveling direction of the moving body 300 or a part thereof, for example.

  As shown in FIG. 19, when the moving body 300 is an automobile 300 </ b> A, the imaging unit 110 can be installed as a so-called front camera that images the front, and can also be installed as a so-called rear camera that captures the rear when back. Of course, both of these may be installed. Further, the imaging unit 110 may be installed also as a function as a so-called drive recorder. That is, the imaging unit 110 may be a recording device.

  Next, FIG. 20 shows an example in which the moving body 300 is a drone 300B. The drone 300 </ b> B includes a drone body 310 corresponding to the drive mechanism 302 and four propeller units 311, 312, 313, and 314. Each propeller part 311, 312, 313, 314 has a propeller and a motor. When the drive of the motor is transmitted to the propeller, the propeller is rotated, and the drone 300B is lifted by the lifting force by the rotation. An imaging unit 110 (or the imaging device 100 including the imaging unit 110) is mounted on, for example, the lower portion of the drone body 310.

  FIG. 21 shows an example in which the moving body 300 is an autonomous mobile robot 300C. Below the mobile robot 300C, a power unit 320 corresponding to the drive mechanism 302 and including a motor, wheels, and the like is provided. The power unit 320 controls the rotation speed of the motor and the direction of the wheels. The mobile robot 300 </ b> C can move in an arbitrary direction by rotating the wheels installed on the road surface or the floor surface by transmitting the motor drive and controlling the direction of the wheels. The imaging unit 110 can be installed, for example, on the head of the humanoid mobile robot 300C so as to capture the front. Note that the imaging unit 110 may be installed so as to capture the rear and left and right, or a plurality of imaging units 110 may be installed so as to capture a plurality of directions. Also, dead reckoning can be performed by providing at least the imaging unit 110 in a small robot with a small space for mounting a sensor or the like and estimating the position of the subject.

  In the case of controlling the movement and rotation of a part of the moving body 300, as shown in FIG. 22, the imaging unit 110, for example, the tip of the robot arm 300D or the like so as to image an object held by the robot arm 300D. May be installed. The image processing unit 120 estimates a three-dimensional shape of an object to be grasped and a position where the object is placed. Thereby, an accurate gripping operation of the object can be performed.

  The control signal generation unit 301 outputs a control signal for controlling the drive mechanism 302 based on the position of the subject output from the imaging device 100. The driving mechanism 302 drives the moving body 300 or a part of the moving body by a control signal. The drive mechanism 302 is, for example, moving, rotating, accelerating, decelerating, adjusting thrust (lifting force), changing a traveling direction, switching between a normal operation mode and an automatic operation mode (collision avoidance mode), At least one of the operations of a safety device such as an air bag is performed. For example, when the distance to the subject is less than the threshold, the drive mechanism 302 moves, rotates, accelerates, adjusts thrust (lift), changes direction to approach the object, and from the automatic operation mode (collision avoidance mode). At least one of the switching to the normal operation mode may be performed.

  The drive mechanism 302 of the automobile 300A is, for example, a tire. The drive mechanism 302 of the drone 300B is, for example, a propeller. The drive mechanism 302 of the mobile robot 300C is, for example, a leg portion. The drive mechanism 302 of the robot arm 300D is a support unit that supports the tip provided with the imaging unit 110, for example.

  The moving body 300 may further include a speaker or a display to which information (distance information) related to the position of the subject from the image processing unit 120 is input. The speaker or the display outputs sound or an image relating to the position of the subject. The speaker and the display are connected to the imaging device 100 by wire or wirelessly. Furthermore, the moving body 300 may include a light emitting unit to which information regarding the position of the subject from the image processing unit 120 is input. For example, the light emitting unit is turned on or off according to information on the position of the subject from the image processing unit 120.

  As another example, for example, when the moving body 300 is a drone, when a map (three-dimensional shape of an object) is created, a structure survey of a building or terrain, an inspection such as a crack or an electric wire breakage is performed from above. The imaging unit 110 acquires an image obtained by photographing the target, and determines whether the distance to the subject is equal to or greater than a threshold value. Based on the determination result, the control signal generation unit 301 generates a control signal for controlling the thrust of the drone so that the distance from the inspection target is constant. Here, the thrust includes lift. The drive mechanism 302 operates the drone based on the control signal, so that the drone can fly in parallel with the inspection target. When the moving body 300 is a drone for monitoring, a control signal for controlling the drone thrust may be generated so as to keep the distance from the object to be monitored constant.

  Further, when the drone flies, the imaging unit 110 acquires an image obtained by photographing the ground direction, and determines whether or not the distance from the ground is equal to or greater than a threshold value. Based on the determination result, the control signal generation unit 301 generates a control signal for controlling the thrust of the drone so that the height from the ground becomes a specified height. The drive mechanism 302 operates the drone based on the control signal, so that the drone can fly at a specified height. If it is a drone for agricultural chemical spraying, it will become easy to spray agricultural chemicals uniformly by keeping the height of the drone from the ground constant.

  Further, when the moving body 300 is a drone or a car, the imaging unit 110 obtains an image of the surrounding drone or a front car during the drone linked flight or the car regime run, It is determined whether the distance is greater than or equal to a threshold value. Based on the determination result, the control signal generation unit 301 generates a control signal for controlling the drone thrust and the speed of the car so that the distance from the surrounding drone and the car ahead is constant. The drive mechanism 302 operates the drone or the vehicle based on the control signal, so that the drone linked flight and the vehicle regime can be easily performed. When the moving body 300 is an automobile, the threshold value may be changed by receiving an instruction from the driver via the user interface so that the driver can set the threshold value. As a result, the vehicle can be driven at a distance between the vehicles preferred by the driver. Alternatively, the threshold may be changed according to the speed of the automobile in order to maintain a safe inter-vehicle distance from the automobile ahead. The safe inter-vehicle distance depends on the speed of the car. Therefore, the threshold value can be set longer as the speed of the automobile is higher. In addition, when the moving body 300 is an automobile, a predetermined distance in the traveling direction is set as a threshold, and when an object appears in front of the threshold, a brake is automatically activated, or safety such as an airbag is set. The control signal generator 301 may be configured to generate a control signal for automatically operating the apparatus. In this case, a safety device such as an automatic brake or an airbag is provided in the drive mechanism 302.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 10 ... Filter, 11 ... 1st filter area | region, 12 ... 2nd filter area | region, 20 ... Lens, 30 ... Image sensor, 31 ... 1st sensor, 32 ... 2nd sensor, 33 ... 3rd sensor, 40 ... CPU, 50 ... Memory, 60 ... Communication I / F, 70 ... Display, 80 ... Memory card slot, 110 ... Imaging unit (optical system), 120 ... Image processing unit (image processing apparatus), 121 ... Sensor control unit, 122 ... Distance calculation Part, 123... Aberration information storage part.

Claims (10)

A second image of the first color component in which the blur function is expressed asymptotically and included in the first image affected by the aberration of the optical system, and a second color component different from the first color component. Acquisition means for acquiring three images;
Calculation for calculating the distance to the subject included in the first image according to the correlation between the plurality of fourth images obtained by adding a plurality of different blurs to the second image and the third image Means and
The correlation is calculated based on the aberration.   The image processing apparatus according to claim 1, wherein the aberration of the optical system includes an aberration caused by curvature of field. Further comprising first correction means for correcting color shift caused by lateral chromatic aberration with respect to the second image and the third image;
The calculating means is configured to detect the subject up to the subject according to a correlation between a plurality of fourth images obtained by adding a plurality of different blurs to the corrected second image and the corrected third image. The image processing apparatus according to claim 2, wherein the distance is calculated. A second correcting means for correcting a positional shift caused by distortion with respect to the second image and the third image;
The calculating means is configured to detect the subject up to the subject according to a correlation between a plurality of fourth images obtained by adding a plurality of different blurs to the corrected second image and the corrected third image. The image processing apparatus according to claim 2, wherein the distance is calculated.   The calculation means includes a blur correction kernel having a high correlation between a fourth image obtained by adding blur and the third image among a plurality of blur correction kernels that add a plurality of different blurs to the second image. The image processing apparatus according to any one of claims 1 to 4, wherein a distance corresponding to the specified blur correction kernel is calculated by specifying.   The blur correction kernel is a convolution function for correcting the blur function of the second image to the blur function of the third image, and is an arbitrary value calculated from the blur function of the second image and the blur function of the third image. The image processing apparatus according to claim 5, wherein a plurality of blur correction kernels for correcting blur of a subject at a distance of 5 are prepared in advance.   The calculation means is configured to determine a correlation between a plurality of fourth images obtained by adding a plurality of different blurs to the second image and a fifth image obtained by adding blurs to the third image. The image processing apparatus according to claim 1, wherein a distance to the subject is calculated. The image processing apparatus according to any one of claims 1 to 7,
An imaging device comprising an image sensor,
The image sensor receives light that has passed through a filter that changes a blur function to asymmetry, a second image of a first color component whose blur function is changed to asymmetry by the filter, and the first color An imaging device that generates a first image including a third image having a second color component different from the component. The image sensor includes two or more sensors of a first sensor that receives red light, a second sensor that receives green light, and a third sensor that receives blue light,
The second image is generated by one of two or more sensors included in the image sensor,
The imaging device according to claim 8, wherein the third image is generated by another sensor of two or more sensors included in the image sensor. A second image of the first color component in which the blur function is expressed asymptotically and included in the first image affected by the aberration of the optical system, and a second color component different from the first color component. Acquiring three images;
Calculating a distance to a subject included in the first image according to a correlation between the second image and the third image to which different blur is added, and
The correlation is calculated based on the aberration.

Images