JP2018084571A - Processing device, imaging device, and automatic control system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a processing device capable of outputting and controlling related to the reliability of distance to a subject obtained from an image.SOLUTION: According to an embodiment, a processing device includes an acquisition unit, a distance calculation unit, and a reliability calculation unit. The acquisition unit acquires a first image of a subject including a blur of a shape indicated by a symmetrical first blur function and a second image of the subject including a blur of the shape indicated by an asymmetrical second blur function. The distance calculation unit calculates distance to the subject on the basis of correlation between the first blur function and the second blur function. The reliability calculation unit calculates the reliability of the distance on the basis of a level of the correlation.SELECTED DRAWING: Figure 5

Description

  Embodiments described herein relate generally to a processing apparatus, an imaging apparatus, an automatic control system, an imaging system, and a distance information acquisition method.

  In recent years, an image processing technique for obtaining a distance to a subject from an image has attracted attention.

JP-A-2015-194486 JP 2004-340714 A Japanese Patent No. 4997525 Utility Model Registration No. 3179836 JP, 2006-241744, A

  The problem to be solved by the present invention is to provide a processing device, an imaging device, an automatic control system, an imaging system, and a distance information acquisition method capable of performing output and control related to the reliability of the distance from the image to the subject. That is.

  According to the embodiment, the processing device includes an acquisition unit, a distance calculation unit, and a reliability calculation unit. The acquisition unit includes a first image of a subject including a blur having a shape indicated by a symmetric first blur function, and a second image of the subject including a blur having a shape indicated by an asymmetric second blur function. And get. The distance calculation unit calculates a distance to the subject based on a correlation between the first blur function and the second blur function. The reliability calculation unit calculates the reliability of the distance based on the degree of correlation.

FIG. 2 is a block diagram illustrating a hardware configuration example of the imaging apparatus according to the embodiment. The figure which shows the structural example of the filter which concerns on embodiment. The figure which shows an example of the transmittance | permeability characteristic of the filter area | region which concerns on embodiment. The figure for demonstrating the light ray change by the color opening which concerns on embodiment, and the shape of a blur. FIG. 2 is a block diagram illustrating an example of a functional configuration of the imaging apparatus according to the embodiment. The figure which shows an example of the blurring function of the reference | standard image which concerns on embodiment. The figure which shows an example of the blurring function of the object image which concerns on embodiment. The figure which shows an example of the blurring correction filter which concerns on embodiment. The 1st figure for demonstrating the reliability calculation process in embodiment. The 2nd figure for demonstrating the reliability calculation process in embodiment. The 3rd figure for demonstrating the reliability calculation process in embodiment. FIG. 6 is a fourth diagram for explaining reliability calculation processing in the embodiment. The figure for demonstrating the distance and correlation value in stereo matching. The figure for demonstrating an example of the calculation method of the curvature of a correlation function in embodiment. The figure which shows an example of the output format of the distance and reliability of distance in embodiment. The figure which shows the other example of the output format of the distance and reliability of distance in embodiment. 5 is a flowchart illustrating an example of a flow of image processing in the embodiment. The block diagram which shows the function structural example of the robot which concerns on embodiment. The figure for demonstrating one operation example based on the reliability of the distance in the robot which concerns on embodiment. The block diagram which shows the function structural example of the mobile body which concerns on embodiment. The 1st figure for demonstrating one operation example based on the reliability of the distance in the mobile body which concerns on embodiment. The 2nd figure for demonstrating one operation example based on the reliability of the distance in the mobile body which concerns on embodiment. The block diagram which shows the function structural example of the monitoring system which concerns on embodiment. The figure for demonstrating the example of 1 process based on the reliability of the distance in the monitoring system which concerns on embodiment. The figure for demonstrating one process example based on the distance in the monitoring system which concerns on embodiment. The figure which shows the example of 1 presentation of the distance in the monitoring system which concerns on embodiment. The figure which shows the example of 1 display of the message in the monitoring system which concerns on embodiment. The figure which shows the example of installation of the imaging device which concerns on 2nd Embodiment. The figure which shows an example of rotation of the imaging device which concerns on 2nd Embodiment. The block diagram which shows an example of the electrical constitution of the system which concerns on 2nd Embodiment. The block diagram which shows an example of the function structure for the distance calculation of the system which concerns on 2nd Embodiment, and display control. FIG. 6 is a diagram illustrating an example of a subject photographed by an imaging apparatus according to a second embodiment. FIG. 6 is a diagram illustrating an example of a color filter of an imaging apparatus according to a second embodiment. The figure which shows the 1st modification of the color filter of the imaging device which concerns on 2nd Embodiment. FIG. 10 is a diagram illustrating a second modification of the color filter of the imaging apparatus according to the second embodiment. FIG. 10 is a diagram illustrating a third modification of the color filter of the imaging device according to the second embodiment. FIG. 10 is a diagram illustrating a fourth modification of the color filter of the imaging device according to the second embodiment. The figure which shows the example of installation of the imaging device which concerns on 3rd Embodiment. The figure which shows an example of rotation of the imaging device which concerns on 3rd Embodiment. The figure which shows an example of the 1st, 2nd main axis | shaft in the imaging device which concerns on 3rd Embodiment. The block diagram which shows an example of a function structure of the monitoring system which concerns on the 1st application example of embodiment. The figure which shows the usage example of a monitoring system. The figure which shows an example of a function structure of the automatic door system which concerns on the 2nd application example of embodiment. The figure which shows the operation example of an automatic door system. The figure which shows an example of the motor vehicle door opening / closing system which concerns on the modification of an automatic door system. The block diagram which shows an example of a function structure of the mobile body control system which concerns on the 3rd application example of embodiment. The figure which shows the robot which is an example of the mobile body of the 3rd application example. The block diagram which shows an example of a function structure in case the mobile body of a 3rd application example is a drone. The block diagram which shows an example of a function structure in case the mobile body of a 3rd application example is a motor vehicle.

  Hereinafter, embodiments will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a hardware configuration example of the imaging apparatus according to the embodiment. The imaging apparatus 100 has a function of capturing an image and processing the captured image. The imaging apparatus 100 is incorporated in, for example, a camera, a mobile phone or a smartphone having a camera function, a portable information terminal such as a PDA (Personal Digital Assistant, Personal Data Assistant), a personal computer having a camera function, or various electronic devices. It can be realized as a system.

  As illustrated in FIG. 1, the imaging device 100 includes, for example, a filter 10, a lens 20, an image sensor 30, an image processing unit, and a storage unit. The image processing unit is configured by a circuit such as a CPU 40, for example. The storage unit includes, for example, a RAM 50 and a nonvolatile memory 90. The imaging apparatus 100 may further include a memory card slot 60, a display 70, and a communication unit 80. For example, the image sensor 30, the CPU 40, the RAM 50, the memory card slot 60, the display 70, the communication unit 80, and the nonvolatile memory 90 can be connected to each other via the bus 110.

  The image sensor 30 receives light transmitted through the filter 10 and the lens 20, and generates an image by converting the received light into an electrical signal (photoelectric conversion). For example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) is used for the image sensor 30. The image sensor 30 includes, for example, an image sensor (first sensor 31) that receives red (R) light, an image sensor (second sensor 32) that receives green (G) light, and blue (B ). The imaging element (third sensor 33) that receives the light of Each image sensor receives light in a corresponding wavelength band, and converts the received light into an electrical signal. A color image can be generated by A / D converting the electrical signal. It is also possible to generate an R image, a G image, and a B image using electrical signals for the red, green, and blue image sensors. That is, a color image, an R image, a G image, and a B image can be generated simultaneously. In other words, the imaging apparatus 100 can obtain a color image, an R image, a G image, and a B image by one imaging.

  The CPU 40 is a processor that controls operations of various components in the imaging apparatus 100. The CPU 40 executes various programs loaded into the RAM 50 from the nonvolatile memory 90 that is a storage device. The non-volatile memory 90 can also store images generated by the image sensor 30 and processing results of the images.

  Various portable storage media such as an SD memory card and an SDHC memory card can be inserted into the memory card slot 60. When a storage medium is inserted into the memory card slot 60, data can be written to and read from the storage medium. The data is, for example, image data or distance data.

  The display 70 is, for example, an LCD (Liquid Crystal Display). The display 70 displays a screen image based on a display signal generated by the CPU 40 or the like. The display 70 may be a touch screen display. In that case, for example, a touch panel is arranged on the upper surface of the LCD. The touch panel is an electrostatic capacitance type pointing device for performing input on an LCD screen. The touch position on the screen where the finger is touched and the movement of the touch position are detected by the touch panel.

  The communication unit 80 is an interface device configured to execute wired communication or wireless communication. The communication unit 80 includes a transmission unit that transmits signals in a wired or wireless manner and a reception unit that receives signals in a wired or wireless manner.

  FIG. 2 is a diagram illustrating a configuration example of the filter 10. The filter 10 includes, for example, a first filter area 11 and a second filter area 12 that are two color filter areas. The center of the filter 10 coincides with the optical center 13 of the imaging device 100. The first filter region 11 and the second filter region 12 each have a shape that is asymmetric with respect to the optical center 13. For example, the filter regions 11 and 12 do not overlap each other, and the two filter regions 11 and 12 constitute the entire filter region. In the example shown in FIG. 2, each of the first filter region 11 and the second filter region 12 has a semicircular shape in which the circular filter 10 is divided by a line segment passing through the optical center 13. The first filter region 11 is, for example, a yellow (Y) filter region, and the second filter region 12 is, for example, a cyan (C) filter region.

  The filter 10 has two or more color filter regions. Each of the color filter regions has an asymmetric shape with respect to the optical center of the imaging device. A part of the wavelength region of light transmitted through one color filter region and a part of the wavelength region of light transmitted through another color filter region overlap, for example. The wavelength region of light transmitted through one color filter region may include, for example, the wavelength region of light transmitted through another color filter region. Below, it demonstrates using the filter 10 of FIG. 2 as an example.

  By arranging such a filter 10 in the opening of the camera, a color opening which is a structure opening in which the opening is divided into two colors is formed. Based on the light rays that pass through the color aperture, the image sensor 30 generates an image. The lens 20 may be disposed between the filter 10 and the image sensor 30 on the optical path of the light incident on the image sensor 30. The filter 10 may be disposed between the lens 20 and the image sensor 30 on the optical path of the light incident on the image sensor 30. When a plurality of lenses 20 are provided, the filter 10 may be disposed between the two lenses 20.

  More specifically, the light in the wavelength band corresponding to the second sensor 32 passes through both the yellow first filter region 11 and the cyan second filter region 12. The light in the wavelength band corresponding to the first sensor 31 passes through the yellow first filter region 11 and does not pass through the cyan second filter region 12. The light in the wavelength band corresponding to the third sensor 33 passes through the cyan second filter region 12 and does not pass through the yellow second filter region 12.

  Note that light in a certain wavelength band passes through a filter or filter region. The filter or filter region transmits light in that wavelength band with a high transmittance, and the filter or filter region attenuates light in that wavelength band (ie This means that the reduction in the amount of light) is extremely small. In addition, the fact that light in a certain wavelength band does not pass through the filter or filter area means that the light is shielded by the filter or filter area. This means that the attenuation of light in the wavelength band by the filter or the filter region is extremely large. For example, a filter or filter region attenuates light by absorbing light in a certain wavelength band.

  FIG. 3 is a diagram illustrating an example of transmittance characteristics of the first filter region 11 and the second filter region 12. As shown in FIG. 3, in the transmittance characteristic 21 of the yellow first filter region 11, light in the wavelength band corresponding to the R image and G image is transmitted with high transmittance, and the wavelength band corresponding to the B image is transmitted. Little light is transmitted. In the transmittance characteristic 22 of the cyan second filter region 12, light in the wavelength band corresponding to the B image and the G image is transmitted with high transmittance, and light in the wavelength band corresponding to the R image is almost transmitted. Not.

Therefore, the light in the wavelength band corresponding to the R image is transmitted only through the first filter region 11 for yellow, and the light in the wavelength band corresponding to the B image is transmitted only through the second filter region 12 for cyan. the shape of blur in the image and B image, depending on the distance d to the subject, and more particularly, changes according to the difference between the distance d and the focal length d _f. Further, each filter region is so asymmetric shape with respect to the optical center, the shape of the blurring of the R and B images, depending on whether the subject is there in front of the focusing distance d _f, or in the back. That is, the blur shapes on the R image and the B image are biased.

  With reference to FIG. 4, a description will be given of a light ray change caused by the color aperture in which the filter 10 is disposed and a blur shape.

When the object 15 is in the back than focusing distance _{d f (d> d f)} , the image captured by the image sensor 30 blur occurs. The blur function (PSF: Point Spread Function) indicating the shape of the blur on the image is different for each of the R image, the G image, and the B image. That is, the blur function 101R of the R image shows a blur shape biased to the left side, the blur function 101G of the G image shows a blur shape without bias, and the blur function 101B of the B image has a blur shape biased to the right. Show.

When the subject 15 is at the in-focus distance d _f (d = d _f ), the image captured by the image sensor 30 is hardly blurred. The blur function indicating the shape of blur on this image is substantially the same for the R image, the G image, and the B image. In other words, the blur function 102R for the R image, the blur function 102G for the G image, and the blur function 102B for the B image indicate the shape of the blur without deviation.

When the object 15 is in front of the focusing distance _{d f (d <d f)} , the image captured by the image sensor 30 blur occurs. The blur function indicating the blur shape on the image is different for each of the R image, the G image, and the B image. That is, the blur function 103R of the R image shows a blur shape biased to the right, the blur function 103G of the G image shows a blur shape without bias, and the blur function 103B of the B image has a blur shape biased to the left. Show.

  In the present embodiment, the distance to the subject is calculated using such characteristics.

  FIG. 5 is a block diagram illustrating a functional configuration example of the imaging apparatus 100.

  As illustrated in FIG. 5, the imaging apparatus 100 includes an image processing unit 41 in addition to the filter 10, the lens 20, and the image sensor 30 described above. An arrow from the filter 10 to the image sensor 30 indicates a light path. An arrow from the image sensor 30 to the image processing unit 41 indicates a path of an electric signal. The image processing unit 41 includes, for example, an image acquisition unit 411, a distance calculation unit 412, a reliability calculation unit 413, and an output unit 414. The image processing unit 41 may be realized in part or in whole by software (program) or a hardware circuit.

  The image acquisition unit 411 acquires, as a reference image, a G image that shows a blur shape with no blurring function (Point spread function, PSF). In addition, one or both of an R image and a B image showing a blur shape with a defocused blur function is acquired as a target image. For example, the target image and the reference image are images captured at the same time by one imaging device.

  The distance calculation unit 412 calculates a distance to a subject in the image by obtaining a blur correction filter that has a higher correlation with the reference image when added to the target image among the plurality of blur correction filters. The distance calculation unit 412 may output a distance image from the further calculated distance. The plurality of blur correction filters are functions that add different blurs to the target image. Here, first, the details of the distance calculation processing by the distance calculation unit 412 will be described.

  The distance calculation unit 412 generates a corrected image in which the blur shape of the target image is corrected by adding different blurs to the target image based on the acquired target image and the reference image. In this embodiment, a corrected image in which the blur shape of the target image is corrected is generated using a plurality of blur correction filters created assuming that the distance to the subject in the image is an arbitrary distance. The distance to the subject is calculated by obtaining a distance that provides a higher correlation with the reference image. A method for calculating the correlation between the corrected image and the reference image will be described later.

  The blur function of the captured image is determined by the opening shape of the image capturing apparatus 100 and the distance between the subject position and the focus position. FIG. 6 is a diagram illustrating an example of a blur function of a reference image according to the embodiment. As shown in FIG. 6, since the aperture shape through which the wavelength region corresponding to the second sensor transmits is a circular shape that is a point-symmetrical shape, the blur shape indicated by the blur function does not change before and after the focus position. The blur width changes depending on the distance between the subject and the focus position. Such a blur function indicating the shape of the blur can be expressed as a Gaussian function in which the blur width varies depending on the distance between the subject position and the focus position. Note that the blur function may be expressed as a pillbox function in which the blur width changes depending on the distance between the position of the subject and the focus position.

On the other hand, FIG. 7 is a diagram illustrating an example of a blur function of the target image according to the embodiment. The center (x ₀ , y ₀ ) = ( ₀ , ₀ ) in each figure. As shown in FIG. 7, a blur function of the target image (e.g. R image) in the case of d> d _f in farther than the subject focus position, the optical attenuation at the first filter region 11 in x> 0 It can be expressed as a Gaussian function with a reduced blur width. Further, the subject be in a proximal than the focus position of the d <d _f, can be expressed as a Gaussian function width of blur is attenuated by the light attenuation of the first filter region 11 in x <0.

  Further, by analyzing the blur function of the reference image and the blur function of the target image, a plurality of blur correction filters for correcting the blur shape of the target image to the blur shape of the reference image can be obtained.

  FIG. 8 is a diagram illustrating an example of a blur correction filter according to the embodiment. Note that the blur correction filter shown in FIG. 8 is a blur correction filter when the filter 10 shown in FIG. 2 is used. As shown in FIG. 8, the blur correction filter is distributed on the straight line (near the straight line) that passes through the center point of the line segment at the boundary between the first filter region 11 and the second filter region 12 and is orthogonal to the line segment. To do. The distribution is a mountain-shaped distribution as shown in the figure in which the peak point (position and height on the straight line) and the spread from the peak point differ for each assumed distance. The blur shape of the target image can be corrected to various blur shapes assuming an arbitrary distance using a blur correction filter. That is, a corrected image assuming an arbitrary distance can be generated.

  The distance calculation unit 412 obtains the distance at which the blur shape between the generated corrected image and the reference image is the closest or the same from each pixel of the captured image. The degree of coincidence of the blur shapes may be calculated by calculating the correlation between the corrected image and the reference image in a rectangular area having an arbitrary size centered on each pixel. The degree of coincidence of blur shapes may be calculated using an existing similarity evaluation method. The distance calculation unit 412 calculates a distance that gives the highest correlation between the corrected image and the reference image, and calculates the distance to the subject that appears in each pixel.

  For example, existing similarity evaluation methods include SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), NCC (Normalized Cross-Correlation), ZNCC (Zero-mean Normalized Cross-Correlation), Color Alignment Measure, etc. Use it. In the present embodiment, Color Alignment Measure is used that utilizes the characteristic that the color components of a natural image have a locally linear relationship. In Color Alignment Measure, an index representing a correlation is calculated from the dispersion of the color distribution in the local border centered on the target pixel of the captured image.

  As described above, the distance calculation unit 412 generates a corrected image in which the blur shape of the target image corresponding to the filter region is corrected by the blur correction filter assuming the distance, and the correlation between the generated corrected image and the reference image is greater. The distance to the subject is calculated by obtaining the distance that increases.

  The reliability calculation unit 413 calculates the reliability of the distance calculated by the distance calculation unit 412 as described above. Next, details of the reliability calculation processing by the reliability calculation unit 413 will be described.

  For example, as shown in FIG. 9, when the position of the subject (A) is far from the focus position, the blur of the subject image (B) is biased to the right side (C1) in the B image and to the left side in the R image. (C2). In the G image, symmetrical blur appears. The horizontal axes of the subject (A) and the subject image (B) in FIG. 9 are the same dimensions as the horizontal axes of C1, C2, D1, D2, and E. The vertical axes of C1, C2, D1, and D2 indicate the amount of color components related to blur.

  Thus, for one or both of the B image and the R image, the optimum blur is selected from the blur correction filters (D1, D2) for each distance in order to match the blur shape with the blur shape of the G image (E). By searching for the correction filter, the distance to the subject (A) can be obtained.

  FIG. 10 is a diagram illustrating the blur shape (C1, C2) and the corrected blur shape (E) of the subject image (B) illustrated in FIG. 9 as cross-sectional waveforms. As shown in FIG. 11, for one or both of the B image and the R image, the search problem of searching for a blur correction amount for matching the blur shape with the blur shape of the G image (A1, A2) is convex optimal. Problem. That is, the correlation function between the blurred shape of the B image or R image and the blurred shape of the G image after correction by the blur correction filter for each distance (FIG. 9: D1, D2) is a convex function.

  Further, as shown in FIG. 12, the curvature of the correlation function, which is a convex function, increases when the number of blur correction filters that obtain a correlation value equal to or higher than the threshold value is small, that is, when the dispersion of the solution is small, When the variation of the solution is large, it becomes small. The reliability calculation unit 413 calculates the reliability of the distance calculated by the distance calculation unit 412 based on the curvature of the correlation function. On the other hand, when calculating the distance from two images to the subject by stereo matching, the search problem of searching for a corresponding point on the other image corresponding to the target point on one image is, for example, FIG. This is not a convex optimization problem. Therefore, in the case of stereo matching, it is difficult to obtain the reliability of distance using the correlation value.

  An example of a method for calculating the curvature of the correlation function will be described with reference to FIG.

For example, a quadratic function is fitted (least square method) as a curve obtained from each correlation value (r _n ) when applying the blur correction filter (f _n ) for each distance, and the curvature of the correlation function is calculated using the quadratic function. A secondary coefficient is used. In this case, the curvature can be calculated from three points (p ₁ , p ₂ , p ₃ ).

More specifically, for the above three points (p ₁ , p ₂ , p ₃ ),
r ₁ = c + bf ₁ + af ₁ ² (Formula 1)
r ₂ = c + bf ₂ + af ₂ ² (Formula 2)
r ₃ = c + bf ₃ + af ₃ ² (Formula 3)
And the values of a, b, and c are calculated from these three equations. That is, a which is a secondary coefficient as a curvature is calculated.

  Moreover, the curvature calculated in this way is converted into reliability (0 to 1) by, for example, the following (Expression 4) using a Gaussian function.

Reliability = 1−exp (− (curvature ² / 2σ ² )) (Formula 4)
The reliability calculation unit 413 obtains a correlation value at the time of applying the blur correction filter processing for each distance calculated in the process of calculating the distance to the subject from the distance calculation unit 412, and for example, the distance calculation unit 412 by the method described above. The reliability of the distance calculated by the above is calculated.

  The output unit 414 outputs output data in which the distance calculated by the distance calculation unit 412 and the reliability of the distance calculated by the reliability calculation unit 413 are associated with each other. For example, as illustrated in FIG. 15, the output unit 414 outputs the distance calculated for each pixel and the reliability of the distance in a map format that is arranged in a positional relationship with the image. Alternatively, the output unit 414 outputs the distance calculated in pixel units and the reliability of the distance in a list format arranged in the order based on the coordinates set on the image, for example, as shown in FIG. Also good. The output unit 414 is not limited to the formats shown in FIGS. 15 and 16, and may output the distance calculated for each pixel and the reliability of the distance in any format.

  For example, the distance (distance map) and the reliability (reliability map) may be output separately in the map format as described above. Further, either or both of the two map data may be connected after the RGB three-image data as output data. Alternatively, either one or both of the two map data may be connected after three data of YUV (luminance signal, color difference signal [Cb], color difference signal [Cr]).

  Further, for example, the distance (distance list) and the reliability (reliability list) may be separately output in the list format as described above. Further, one or both of the two list data may be connected to the RGB three-image data as output data. Alternatively, either one or both of the two list data may be connected after the three YUV data.

Moreover, the form which outputs a reliability may be the display which shows a reliability with a pop-up, for example, when the position on a distance image is designated. The distance at the specified position on the distance image may be displayed in a pop-up, or the distance information and the reliability may be displayed in a pop-up on the color image.

  The distance need not be calculated for all pixels in the image. For example, a subject whose distance is to be detected may be specified in advance. The identification can be performed by, for example, image recognition or designation by user input. On the other hand, the reliability may not be calculated for all the pixels for which the distance is required. For example, the reliability may be calculated for a specific subject or a subject close to the distance, and the reliability may not be calculated for a far subject.

  Further, in the map format as described above, when the distance and the reliability are output simultaneously as shown in FIG. 15, when the distance map and the reliability map are output separately, or in the list format as described above. As shown in FIG. 16, in the case where the distance and the reliability are output at the same time or when the distance list and the reliability list are output separately, all the calculated distances are not included in the output data. May be. For example, when the reliability is lower than a predetermined value or relatively low, the distance may not be included in the output data.

  In addition, output data that can be output in various forms such as a map, a list, and the like may be output to the display 70, for example.

  FIG. 17 is a flowchart illustrating an example of the flow of image processing in the embodiment.

  The image acquisition unit 411 is an image generated by the image sensor 30 and is transmitted without being attenuated by either the first filter region or the second filter region of the light transmitted through the filter region of the filter 10. A reference image formed by the light to be acquired is acquired (step A1). The image acquisition unit 411 is also an image generated by the image sensor 30 and is imaged by light transmitted through the filter region of the filter 10 after being attenuated and transmitted by the first filter region, for example. Target image is acquired (step A2). Here, it is assumed that an image formed by light that is attenuated and transmitted through the first filter region is acquired as the target image, but the image acquisition unit 411 is attenuated and transmitted through the second filter region. An image formed by light that is attenuated by the first filter region and an image formed by light that is attenuated and transmitted by the first filter region and the light that is attenuated and transmitted by the second filter region. You may acquire two with the image to be imaged.

  The distance calculation unit 412 uses the blur correction filter to generate a corrected image in which the blur shape of the target image is corrected (step A3), and calculates a correlation value between the blur of the corrected image and the blur of the reference image (step A3). A4). Correction image generation and correlation value calculation are performed by the number of blur correction filters for each distance. The distance calculation unit 412 calculates the distance to the subject based on the calculated correlation value (step A5). More specifically, the distance to the subject is obtained by searching for the blur correction filter having the highest correlation between the generated corrected image and the reference image from among the blur correction filters for each distance. Alternatively, a blur correction filter that generates a corrected image having a higher correlation with the reference image or the like than other correction filters may be searched.

  Further, the reliability calculation unit 413 calculates the curvature of the correlation function based on the correlation value when the blur correction filter process is applied for each distance calculated in the process of calculating the distance to the subject (step A6). The reliability calculation unit 413 calculates the reliability of the distance calculated by the distance calculation unit 412 based on the calculated curvature (step A7).

  Then, the output unit 414 outputs the distance calculated by the distance calculation unit 412 and the reliability of the distance calculated by the reliability calculation unit 413 in association with each other (step A8).

  As described above, according to the present embodiment, the reliability of the distance from the image to the subject can be calculated as a value reflecting the actual reliability.

  In the above description, the curvature of the correlation function between the corrected image generated by the blur correction filter for each distance and the reference image is calculated, and this curvature is converted into distance reliability. The correlation value between the corrected image and the reference image is regarded as a probability (0 to 1) that the distance associated with the blur correction filter used to generate the corrected image is a correct value, and the calculated correlation value itself is used. It is good also as the reliability of distance. More specifically, among the plurality of blur correction filters, the correlation value of the blur correction filter having the highest correlation between the correction image and the reference image may be used as the distance reliability.

  For example, when converting curvature into distance reliability, the highest correlation value may be used as a weight, and the higher the correlation value, the higher the distance reliability. Alternatively, the edge direction of the subject image, more specifically, the edge gradient direction of the pixel to be processed is used as a weight, and the direction is the boundary between the first filter region and the second filter region of the filter 10. The more reliable the distance, the higher the reliability of the distance. Even when the correlation value itself is the reliability of the distance, the edge direction of the subject image may be used as the weight. Alternatively, the edge strength of the subject image, more specifically, the strength of the edge gradient of the pixel to be processed may be used as a weight, and the higher the strength, the higher the reliability of the distance.

  In the above description, the example in which the distance to the subject is calculated from the image by changing the blur function of the image with the filter 10 has been described. However, for example, the received incident light called a 2PD sensor or the like is detected for each pixel. By using an image sensor that divides the image into left and right parts, it is possible to obtain two images in which at least one of the blur functions is changed without using the filter 10, and to obtain a distance from the correlation between the blur shapes of these two images. It can be calculated. Even in this case, the above-described method for calculating the reliability of the distance can be applied.

  Next, some examples of a system to which the imaging apparatus 100 configured as described above and outputting the distance to the subject and the reliability of the distance are applied will be described.

<Automatic control system: Robot>
FIG. 18 is a block diagram illustrating a functional configuration example of the robot 200 according to the embodiment. Here, it is assumed that the robot 200 is an industrial robot installed in a production line that can produce a plurality of types of products, for example. The robot 200 is not limited to the installation type, and may be an autonomously moving robot such as an AGV (Automatic Guided Vehicle). The robot 200 can also be realized for non-industrial use such as a cleaning robot for cleaning the floor and a communication robot for performing various guidance to visitors.

  As illustrated in FIG. 18, the robot 200 includes an imaging device 100, a control unit 201, a drive mechanism 202, and a rotation mechanism 203. In addition, the imaging device 100 is attached to the rotation mechanism 203.

  First, the control unit 201 controls the drive mechanism 202 based on the distance to the subject that is the work target, which is output from the imaging device 100. The drive mechanism 202 is, for example, a robot arm for attaching a member to an object or picking up an object and transporting it to a predetermined place. Secondly, the control unit 201 controls the rotation mechanism 203 based on the reliability of the distance output together with the distance from the imaging device 100. FIG. 19 is a diagram for explaining the control of the rotation mechanism 203 by the control unit 201.

  In general, when the edge direction of the subject image and the direction of the boundary line between the first filter region and the second filter region of the filter 10 coincide with each other, the reliability of the distance increases. On the other hand, when these directions are orthogonal, the reliability of the distance is low. Therefore, the control unit 201 controls the rotation mechanism 203 so that the reliability of the distance output from the imaging apparatus 100 is high. More specifically, for example, when many edges of a subject image appear in the vertical direction due to the shape and pattern of the subject, the orientation of the subject, etc., as shown in FIG. The rotation mechanism 203 is controlled so that the boundary line with the filter area of the filter 10 is in the vertical direction, and when many edges of the subject image appear in the horizontal direction, for example, as shown in FIG. The rotation mechanism 203 is controlled so that the boundary line between the filter area and the second filter area is in the horizontal direction.

  For example, the control unit 201 first causes the imaging device 100 to perform pre-imaging, and based on the reliability of the distance output from the imaging device 100 during this pre-imaging, the rotation angle at which the imaging device 100 should be rotated by the rotation mechanism 203. To derive. In pre-imaging, the imaging apparatus 100 does not calculate the distance and reliability of the distance for all the pixels on the image, but calculates a certain number of sampled pixels, distance and the reliability of the distance. You may make it do. In addition, the control unit 201 uses, for example, the average value of these as the reliability of the distance in the pre-imaging. As a method for deriving the rotation angle, for example, when the reliability of the distance in the pre-imaging is less than the threshold, various methods such as 90 degrees can be applied. After rotating the imaging apparatus 100 by the rotation mechanism 203, the control unit 201 causes the imaging apparatus 100 to perform main imaging.

  Alternatively, the control unit 201 may cause the image capturing apparatus 100 to continuously perform image capturing while rotating the image capturing apparatus 100 using the rotation mechanism 203, and may employ an image having the highest reliability of distance.

  FIG. 19 shows an example in which the rotation mechanism 203 is provided on the drive mechanism 202. However, it is not essential to provide the imaging device 100 on the drive mechanism 202. Therefore, the rotation mechanism 203 includes the drive mechanism 202. May be provided separately.

  The technique of rotating the imaging apparatus 100 so that the reliability of the distance is high can be applied to a moving body other than the robot 200. When the moving body is a rotatable flying body such as a drone, for example, the control unit 301 may control the drive mechanism 302 to rotate the entire moving body 300 regardless of the rotating mechanism.

  The imaging apparatus 100 may include a rotation mechanism that rotates the filter 10 with respect to the image sensor 30. The rotation mechanism rotates the filter in one plane around the optical center, for example. With the rotation of the filter 10, a highly reliable distance can be acquired.

<Automatic control system: moving body>
FIG. 20 is a block diagram illustrating a functional configuration example of the moving object 300 according to the embodiment. Here, it is assumed that the moving body 300 is, for example, an automobile. The moving body 300 is not limited to a vehicle including an automobile, and can be realized as various things such as a flying body such as a drone or an airplane, a ship, a robot such as an AGV or a cleaning robot as long as it has a driving mechanism for movement. Can be done. Furthermore, the moving body 300 may be an automatic door.

  As shown in FIG. 20, the moving body 300 has a control system. The control system includes two imaging devices 100 (100-1 and 100-2), a control unit 301, and a drive mechanism 302. Here, it is assumed that two imaging devices 100 are provided, but three or more imaging devices 100 may be provided. The control system may be mounted on the moving body 300 or the moving body may be controlled remotely. The control unit 301 may directly control the driving mechanism 302 or may indirectly control by wireless. As shown in FIG. 21, the two imaging devices 100 are installed so as to image a subject in the traveling direction of the moving body 300, for example. In addition, as a form installed so that the to-be-photographed object of the moving body 300 may be imaged, it can be installed as a so-called front camera that images the front, or it can be installed as a so-called rear camera that images the back during back. Of course, both of these may be installed. In addition, the imaging apparatus 100 may be installed so as to function as a so-called drive recorder. That is, the imaging device 100 may be a recording device.

  The control unit 201 controls the drive mechanism 302 based on the distance and reliability output from each of the two imaging devices 100. For example, the control unit 201 controls the drive mechanism 302 based on the distance with higher reliability among the distances obtained from the two imaging devices 100. Alternatively, the control unit 201 controls the drive mechanism 302 based on the distances obtained by weighting the distances obtained from the two imaging devices 100 with the respective reliability. The control is, for example, stopping, decelerating, or accelerating the moving moving body 300 or starting moving the stopped moving body 300 when approaching a subject in the traveling direction up to a predetermined distance. Alternatively, the control unit 201 may control the driving mechanism 302 to stop, decelerate, accelerate, and start moving the moving body 300 when the subject is separated by a predetermined distance or more. Alternatively, the control unit 201 switches from the normal drive mode to the collision avoidance mode when the subject approaches a predetermined distance, or switches from the collision avoidance mode to the normal drive mode when the subject is separated by a predetermined distance or more. May be controlled. The predetermined distance may be changed according to the reliability, for example.

  The reliability may be obtained for each image or for each region on the image. In the former case, for example, an image having a higher average value of distance reliability is employed. In the latter case, for example, the reliability value of the distance is compared for each corresponding pixel between two images, and the higher value is adopted. Thereby, the distance to the subject can be acquired more accurately. The drive mechanism is, for example, a motor or an engine for driving a tire, a roller, or a propeller.

  Next, with reference to FIG. 22, an example of using distance reliability in the moving body 300 will be described.

  Here, it is assumed that the control unit 201 controls the drive mechanism 302 to stop the moving body 300 when approaching a subject in the traveling direction up to a predetermined distance as one of the controls of the drive mechanism 302. When the control unit 201 acquires the distance to the subject and the reliability of the distance from the imaging apparatus 100, the control unit 201 calculates a lower limit value of the distance that is one end of the error range based on the distance and the reliability of the distance. Then, the control unit 201 controls the drive mechanism 302 using this lower limit value instead of the distance output from the imaging device 100. The lower limit value is calculated as a value having a smaller difference from the distance as the reliability of the distance is higher, and is calculated as a value having a larger difference from the distance as the reliability of the distance is lower.

  For example, as shown in FIG. 22, even when a distance longer than the actual distance to the subject is calculated and output by the imaging apparatus 100, the lower limit value of the distance calculated from the distance reliability is set. By using it, it is possible to prevent a situation in which the moving body 300 stops, decelerates, avoids a collision, turns around, or operates a safety device such as an airbag.

  Note that the lower limit value of the distance calculated from the distance and the reliability of the distance may be used when the number of the imaging device 100 is one. Further, the present invention is not limited to the moving body 300, and is effective for the robot 200 described with reference to FIGS. 19 and 20, for example.

<Monitoring system>
FIG. 23 is a block diagram illustrating a functional configuration example of the monitoring system 400 according to the embodiment. Here, it is assumed that the monitoring system 400 is a system for grasping the flow of people in, for example, a store for each time zone.

  As illustrated in FIG. 23, the monitoring system 400 includes the imaging device 100, a control unit 401, and a user interface unit 402. The imaging device 100 and the control unit 401 may be connected via a wired or wireless network.

  The control unit 401 causes the imaging device 100 to continuously perform imaging, and first displays an image captured by the imaging device 100 via the user interface unit 402. The user interface unit 402 executes display processing on a display device, for example, and input processing from, for example, a keyboard or a pointing device. The display device and the pointing device may be an integrated device such as a touch screen display.

  Secondly, based on the distance to the subject and the reliability of the distance, which is sequentially output from the imaging apparatus 100, the control unit 401, for example, which part of the passage is walking in which direction, etc. Are analyzed, and the analysis result is recorded in a storage device such as an HDD (Hard Disk Drive). This analysis is not necessarily executed in real time, and may be executed as a batch process using the distance to the subject stored in the storage device and the reliability of the distance.

  Now, for example, as shown in FIG. 24, it is assumed that imaging is performed by the imaging apparatus 100 in a state where two three-dimensional objects exist in the imaging range. Further, for example, it is assumed that an image showing a three-dimensional object that does not actually exist is easily captured due to various factors such as background and lighting. In this case, if a three-dimensional object is recognized based on the distance distribution in the image, for example, a three-dimensional object may be misrecognized as a three-dimensional object (A) and may be a tracking target, for example.

  On the other hand, for example, when calculating the distribution of distances in the image, it is possible to prevent a situation in which a three-dimensional object is erroneously recognized as a three-dimensional object by excluding distances with low reliability. (B).

  Next, with reference to FIG. 25, a description will be given of an example of utilizing distance in tracking the subject image in the image recognized as described above.

  Assume that a person moves from left to right as viewed from the imaging apparatus 100 and another person tries to move from right to left (A). In addition, the two persons are different in height, and the person having a shorter height is positioned in front of the person having a shorter height than the person having a higher height, and as a result, the size of the subject image on the image. Are substantially in agreement.

  If these two people continue to move as they are, the subject images on the image overlap at a certain point (B), and then move left and right (C). In such a case, if the subject image is tracked only by image recognition without using the distance, for example, when the subject image intersects, the tracking target may be mistaken and the two people may mistrack as if they made a U-turn. .

  On the other hand, by using the distance, it is possible to prevent a situation in which the tracking target is mistaken when the subject images intersect.

  Another example of using the distance in tracking the subject image in the image recognized as described above is, for example, that the subject is moving toward the door and has approached a predetermined distance. When detected, the door is automatically opened.On the other hand, if it is detected that the subject is moving away from the door and the distance is more than a predetermined distance, an automatic door system that automatically occupies the door is used. Can be mentioned.

  FIG. 26 is a diagram illustrating an example of a distance using the reliability.

  As described above, the control unit 401 displays an image captured by the imaging device 100 via the user interface unit 402. In addition, the control unit 401 acquires the distance to the subject and the reliability of the distance from the imaging device 100. Furthermore, when an arbitrary position on the image is designated by a pointing device, for example, the control unit 401 receives event information including the coordinates from the user interface unit 402.

  Upon receiving this event information, the control unit 401, based on the distance calculated in the imaging apparatus 100 for the pixel corresponding to the coordinates and the reliability of the distance, the lower limit value of the distance and the upper limit of the distance. Calculate the limit value. Then, the control unit 201 pops up a range of distance considering not only the distance but also an error, for example, near the pointer of the pointing device via the user interface unit 402.

  That is, it is possible to provide a GUI (graphical user interface) capable of presenting the distance to the subject so that the error range can be recognized when the position on the image in which the subject is shown is pointed out.

  Further, the user interface unit 402 simultaneously displays at least one of a G image in which left-right symmetric blur appears, a B image and R image in which left-right asymmetric blur appears, and a color image (RGB image), and a distance image. When a position on the image, the B image, the R image, or the color image is designated, a GUI may be provided that displays the distance and the reliability on the distance image.

  Such a GUI is also useful when the imaging apparatus is configured with a stand-alone electronic device such as a tablet computer or a smartphone. For example, the GUI may be provided as a distance measurement tool that captures an image with an electronic device and displays a distance to a subject by performing a touch operation on a touch screen display on which the image is displayed.

  If the distance to the subject can be acquired in units of pixels, the length of each part of the subject can be calculated using these distances. Therefore, for example, a measuring tool capable of measuring the size of the furniture displayed in the store by taking an image of the furniture can be realized in the stand-alone electronic device. As described above, the reliability of the distance is determined by the edge direction of the subject image, more specifically, the edge gradient direction of the pixel to be processed, the first filter region and the second filter region of the filter 10. Depending on the direction of the boundary line. Therefore, when the reliability of the distance is less than the threshold, for example, as shown in FIG. 27, the rotation angle is indicated and the electronic device is rotated to capture an image so that a more accurate distance is calculated. You may provide GUI which displays the message to urge on a display etc. Alternatively, the display shows the current orientation of the electronic device as a bar-shaped figure such as a needle or arrow, and the direction that makes it easy to calculate the correct distance is shown as a figure on a bar such as a needle-valley arrow, or the rotation angle or direction of rotation. May be represented by an arrow.

  As described above, according to the present embodiment, it is possible to perform output and control related to the reliability of the distance from the image to the subject using the curvature of the correlation function.

  Hereinafter, other embodiments will be described with reference to the drawings. The disclosure is merely an example, and the invention is not limited by the contents described in the following embodiments. Variations readily conceivable by those skilled in the art are naturally included in the scope of the disclosure. In order to make the explanation clearer, in the drawings, the size, shape, and the like of each part may be schematically expressed by changing the actual embodiment. Corresponding elements in the drawings are denoted by the same reference numerals, and detailed description may be omitted.

[Second Embodiment]
As described above, the distance from the image taken by the imaging device, which is a monocular camera having a color opening, to the subject is obtained. The color aperture is configured by arranging a color filter including at least two color filter regions in the aperture of the imaging device. An image sensor generates an image based on light rays that pass through the color aperture. In the example shown in FIG. 2, the first and second filter regions have a semicircular shape in which a circular filter is divided into left and right (horizontal directions) by a vertical line segment passing through the optical center. For example, the first filter area is a yellow (Y) filter area, and the second filter area is a cyan (C) filter area. Light in the wavelength band of the green (G) image is transmitted through the first and second filter areas, but light in the wavelength band of the red (R) image is transmitted through only the first filter area, and the wavelength of the blue (B) image. The band light passes through only the second filter region. Therefore, the blur shape of the G image does not change according to the distance to the subject, but the blur shape of the R image and the B image changes according to the distance to the subject. Specifically, the blur between the R image and the B image is biased to the right or left depending on whether the subject is in front of or behind the in-focus distance.

  The difference in shape of the blur of the R, G, and B images changed by the color opening has a one-to-one correspondence with the distance to the subject. Therefore, a blur correction filter for correcting the blur shape of the R image and the B image changed by the color opening to the blur shape of the G image is prepared. The blur correction filter corresponds to the distance to the subject. Then, the blur correction filter is calculated into an R image and / or a B image, and the distance is determined based on the correlation between the corrected image and the G image.

  As shown in FIG. 8, the blur correction filter is distributed in the vicinity of the horizontal direction orthogonal to the vertical direction (division direction) that divides the filter region into first and second filter regions. To correct the R image and / or the B image with such a correction filter is to convolve the R image and / or the B image and the correction filter in the horizontal direction. Therefore, the convolution result is the same at any assumed distance at the edge in the horizontal direction orthogonal to the filter area dividing direction of the color aperture (the density gradient direction is the vertical direction), and the distance may not be determined. In order to reduce such a possibility, the imaging apparatus according to the second embodiment prevents the filter area division direction of the color opening from being orthogonal to the direction of the edge included in the subject, in other words, the filter area division direction of the subject. It is configured to be installed so as not to coincide with the concentration gradient direction.

[Imaging device installation example]
FIG. 28 shows an installation example of the imaging apparatus according to the second embodiment. The imaging device of the second embodiment can be applied to a monitoring system or the like. FIG. 28 shows an example in which an imaging device 502 having a color opening 504 is attached to the ceiling of a room via a fixture capable of tilting / panning / rolling. Since tilt / pan is not directly related to the embodiment, these functions can be omitted. Furthermore, as will be described later, the roll function can also be omitted. As shown in FIG. 2, the X axis and the Y axis are axes in the plane of the color opening. The Z axis is an axis in the optical axis direction of the imaging device 502.

  The tip of a columnar arm 508 is fixed at the center of the rear end of the imaging device 502. The axis of the arm 508 coincides with the optical axis of the imaging device 502. The rear end of the arm 508 is inserted into the tip of an arm 512 that is concentric with the arm 508 and has a larger diameter than the arm 508. Therefore, the arm 508 (and the imaging device 502) can be rotated in the clockwise and counterclockwise directions around the optical axis (also referred to as a roll axis) while being inserted into the arm 512. Although the clockwise direction and the counterclockwise direction vary depending on the direction of the line of sight, in this specification, the rotation direction is defined while the subject is viewed from the imaging device 502. That is, the rotation of the subject along the X axis along the Y axis is referred to as clockwise rotation. The roll angle in each of the clockwise direction and the counterclockwise direction (angle based on the vertical direction) does not need to be 90 degrees or more, and may be about 45 degrees. The rotation of the arm 508 is suppressed by a screw or the like, and the roll angle is fixed. Adjust the roll angle so that the filter area division direction of the color aperture does not intersect the direction of the edge included in the subject, in other words, the filter area division direction does not match the density gradient direction of the subject. Can do.

  The rear end of the arm 512 is pivotally supported by the lower end of the arm 514 in the vertical direction. This axis is also called a tilt axis. For this reason, the arm 512 (and the arm 508 and the imaging device 502) can be tilted in the vertical direction. The rotation of the arm 512 is suppressed by a screw or the like, and the tilt angle is fixed.

  The upper end of the arm 514 is concentric with the arm 514 and is inserted into the lower end of the arm 516 having a larger diameter than the arm 514. For this reason, the arm 514 (and the arms 512 and 508 and the imaging device 502) can be pan-rotated in the horizontal direction while being inserted into the arm 516. The rotation of the arm 514 is suppressed by a screw or the like, and the pan angle is fixed. The upper end of the arm 516 is integrated with the mounting plate 520.

  The tilt angle and pan angle may be fixed at specific angles before installation of the imaging apparatus, or may be adjusted by moving the arms 512 and 514 so that a desired field of view is photographed after installation.

  FIG. 28 shows an example of a fixture to be attached to the ceiling. However, if the arm 516 is bent in the horizontal direction, or if the horizontal arm is further connected to the arm 516, it can be attached to a telephone wall or street light on the wall of the room or the road. Is also possible.

[Rotation of imaging device]
FIG. 29 shows an example of rotation of the imaging apparatus according to the second embodiment. Here, it is assumed that the filter division direction of the color opening is the vertical direction, and the edge whose distance may be difficult to calculate is the horizontal edge. Therefore, a straight line representing the vertical direction is assumed. When the roll angle of the image pickup apparatus 502 is 0 degree, that is, when the vertical direction of the image pickup apparatus 502 coincides with the vertical direction, as shown in FIG. The straight lines representing the division directions are parallel. In this state, it may be difficult to calculate the distance between the edges in the horizontal direction perpendicular to the straight line representing the filter dividing direction. As shown in FIG. 29B, if the imaging device 502 (arm 508) is rotated around the optical axis so that the vertical direction of the imaging device 502 does not coincide with the vertical direction, a straight line representing the vertical direction is obtained. The straight line projected on the filter surface and the straight line representing the division direction of the filter can be made non-parallel. As a result, the horizontal edge included in the subject and the filter division direction are not orthogonal, and the distance between the horizontal edges can be calculated.

  In this case, the roll angle may be larger than 0 degree. If the roll angle is 90 degrees, the edge distance in the vertical direction may be difficult to calculate. The roll angle may be about 45 degrees. Edges that are difficult to calculate are unavoidable, and it is up to the user to decide which edge is calculated and which edge is difficult to calculate.

  After installation, the subject can be photographed to calculate the distance, and an appropriate angle can be obtained by trial and error while adjusting the roll angle while viewing the result. After installation, the roll angle may be determined based on an index such as reliability as shown in FIGS. Alternatively, when the directions of various edges included in the subject are known in advance, an appropriate roll angle is determined in advance so that the directions of all the edges are not orthogonal to the filter division direction, and the imaging device is fixed to the angles. It may be installed after leaving. Furthermore, when an appropriate roll angle is obtained in advance from the direction of a known edge included in the subject, as shown in FIG. 28, if the imaging device can be attached to a ceiling or the like while being rotated around the optical axis. A roll rotation mechanism is not essential. However, if a roll rotation mechanism is provided, it is possible to easily cope with a change in the direction of the edge included in the subject. If the roll rotation mechanism is not provided, it may be installed again when the direction of the edge included in the subject changes.

  The roll rotation mechanism is not limited to the example of FIG. In FIG. 28, the roll rotation axis coincides with the optical axis of the imaging device 502, but the imaging device may be installed so as to rotate the imaging device around an axis other than the optical axis. For example, in FIG. 28, instead of attaching the imaging device 502 to the tip of the arm 508, a holder for mounting the imaging device 502 may be attached to the surface of the arm 508. In this case, when the arm 508 is rotated, the captured image is tilted about the axis of the arm 508 outside the screen. In the case of FIG. 28, when the arm 508 is rotated, the captured image is tilted about the optical axis in the screen.

[System block diagram]
FIG. 30 is a block diagram illustrating an example of an electrical configuration of the imaging apparatus 502 of the second embodiment. The second embodiment is a system including an imaging unit (sometimes referred to as a camera) and a processing device (processing unit). The imaging unit 505 includes, for example, an image sensor 542, a photographing lens 538, and a color filter 536. Light from the subject (broken arrows in the figure) enters the image sensor 542 through a photographing lens 538 formed of a plurality of lenses (for convenience, one is shown). The image sensor 542 photoelectrically converts incident light and outputs an image signal (which may be a moving image or a still image), and is a CCD (Charge Coupled Device) type image sensor or a CMOS (Complementary Metal Oxide Semiconductor) type image. Any sensor such as a sensor may be used. The taking lens 538 may include, for example, a plurality of lenses, and any of the lenses may be movable along the optical axis in order to adjust the focal point. A color filter 536 is formed at the opening (the main point or in the vicinity thereof) of the photographing lens 538. The photographing lens 538 to which the color filter 536 is added is also referred to as a color aperture lens 504. Although an example in which a color filter is disposed on the entire surface of the opening of the photographing lens 538 has been shown, the color filter may not be disposed on the entire surface of the opening. For example, the opening may be configured by a color filter region and a region where no color filter is provided.

  The processing device includes a CPU (Central Processing Unit) 544, a nonvolatile storage unit 546 such as a flash memory or a hard disk drive, and a volatile memory 548 such as a RAM (Random Access Memory). The processing device may further include a communication interface 550, a display 556, and a memory card slot 552. The image sensor 542, the CPU 544, the nonvolatile storage unit 546, the volatile memory 548, the communication interface 550, the display 556, the memory card slot 552, and the like are connected to each other via the bus 34.

  The imaging unit and the processing device may be separate or integrated. In the case of integration, both may be realized as an electronic device with a camera such as a smartphone or a tablet. In the case of a separate body, a signal output from an imaging unit realized as a single-lens reflex camera or the like may be input to a processing device realized as a personal computer or the like. The imaging unit and the processing device can communicate, for example, wirelessly or by wire.

  The CPU 544 controls the overall operation of the system. For example, the CPU 544 executes a shooting control program, a distance calculation program, a display control program, and the like stored in the nonvolatile storage unit 546, and implements functional blocks for shooting control, distance calculation, display control, and the like. Thereby, the CPU 544 controls not only the image sensor 542 of the imaging apparatus but also the display 556 and the like. Note that functional blocks for shooting control, distance calculation, display control, and the like are not realized by the CPU 544 but may be realized by dedicated hardware. As an example, the distance calculation program obtains the distance to the subject shown in each pixel of the captured image based on the principle described above.

  The nonvolatile storage unit 546 includes a hard disk drive, a flash memory, and the like. The display 556 includes a liquid crystal display, a touch panel, or the like. The display 556 displays the captured image in color, for example, and the distance information obtained for each pixel in a specific form, for example, as a distance image (also referred to as a distance map) in which the captured image is colored according to the distance. indicate. The distance information may be displayed not in the distance image but in a table format such as a distance-position correspondence table.

  For example, a volatile memory 548 composed of SDRAM (Synchronous Dynamic Random Access Memory) or the like stores programs related to control of the entire system, various data used for processing, and the like.

  The communication I / F 550 is an interface that controls communication with an external device and input of various instructions by a user using a keyboard, operation buttons, and the like. The captured image and the distance information are not only displayed on the display 556 but may be transmitted to the outside via the communication I / F 550 and used by an external device whose operation is controlled based on the distance information. An example of the external device is a driving support system such as an automobile or a drone, a monitoring system that monitors intrusion of a suspicious person, and the like. The processing apparatus may perform a part of the processing for obtaining the distance from the image signal and the rest may be performed by an external device such as a host, and the distance information may be obtained by sharing with a plurality of devices.

  A portable storage medium such as an SD (Secure Digital) memory card or an SDHC (SD High-Capacity) memory card can be inserted into the memory card slot 552. The captured image and the distance information may be stored in a portable storage medium, and the information on the portable storage medium may be read by another device so that the captured image and the distance information are used by the other device. Alternatively, an image signal captured by another imaging device may be input to the processing device of the present system via a portable storage medium in the memory card slot 552, and the distance may be calculated based on the image signal. Furthermore, an image signal captured by another imaging apparatus may be input to the processing apparatus of the present system via the communication I / F 550.

  FIG. 31 is a functional block diagram for distance calculation and display control by the CPU 544. The CPU 544 includes, for example, a captured image acquisition unit 562, a distance image acquisition unit 564, inclination correction units 566 and 568, and a rotation angle / rotation center acquisition unit 570. The output of the image sensor 542 is supplied to the captured image acquisition unit 562, and a captured image of the subject is obtained. The captured image is supplied to an inclination correction unit 566 and a distance image acquisition unit 564. The distance image acquisition unit 564 obtains a distance image for each pixel of the photographed image based on the distance calculation program, and obtains a distance image. The distance image is an image representing the distance or the size of the blur corresponding to the distance. The distance image is supplied to the inclination correction unit 568. The inclination correction units 566 and 568 rotate the captured image and the distance image about the rotation center supplied from the rotation angle / rotation center acquisition unit 570 by the rotation angle supplied from the rotation angle / rotation center acquisition unit 570. The rotation angle / rotation center acquisition unit 570 calculates the rotation center and rotation angle of the imaging unit 505 (imaging device 502) (for example, the clockwise direction is positive and the counterclockwise report is negative), and the inclination correction units 566 and 568 indicate the rotation angle. Rotate the image in the opposite direction. The rotation angle / rotation center acquisition unit 570 may acquire the rotation center and the rotation angle by tracking the feature points in a plurality of consecutive captured images, or the value measured by the user in advance The rotation center and the rotation angle may be acquired by inputting to the rotation center acquisition unit 570. An image rotated by the inclination correction units 566 and 568 is displayed on the display 556.

[Image tilt correction]
An example of inclination correction will be described with reference to FIG. After the imaging device 502 is installed on the ceiling or the like with the roll angle of the imaging unit 505 set to 0 degree, a distance image is displayed based on the calculated distance. For example, when the user observes the distance image and determines that the distance between the horizontal edges is not correct, the imaging unit 505 rotates around the optical axis. As shown in FIG. 28, when the imaging unit 505 is integrated with the imaging device 502, the imaging unit 505 is rotated by rotating the imaging device 502. However, when only the imaging unit 505 is rotatable, the imaging device 502 is fixed. However, only the imaging unit 505 may be rotated. When the imaging unit 505 rotates clockwise about the optical axis, the captured image acquired by the image acquisition unit 562 becomes a captured image with the horizontal line rotated clockwise as shown in FIG. That is, the straight line whose vertical direction is projected on the filter surface is inclined clockwise from the vertical direction. This may be left as it is, but it is uncomfortable to observe as an image. The tilt correction unit 566 rotates the captured image counterclockwise, and generates a corrected image in which the horizontal direction of the display frame and the display 56 matches the horizontal direction of the image, as shown in FIG. Similarly, the inclination of the distance image is corrected by the inclination correction unit 568. It is not always necessary to perform tilt correction. In some cases, it is preferable to perform tilt correction when displaying an image. However, when only a calculated distance is used and no distance image is displayed, tilt correction is often unnecessary.

[Example of color filter]
FIG. 33 shows an example of the color filter 536 according to the second embodiment. The filter region 580 at the center of the color filter 536 is composed of, for example, a first filter region 580A and a second filter region 580B that are two color filter regions. The center of the filter region 580 coincides with the optical center 582 of the imaging unit 505. Each of the first and second filter regions 580A and 580B has a shape that is asymmetric with respect to the optical center 582. The first and second filter regions 580A and 580B do not overlap, and the first and second filter regions 580A and 580B constitute the entire region of the filter region 580. The first and second filter regions 580A and 580B each have a semicircular shape in which the circular filter region 580 is divided by a line segment passing through the optical center 582.

  A straight line that is perpendicular to the line segment at the midpoint of the line segment that connects the centroids of the first and second filter regions 580A and 580B is defined as a straight line that represents the division direction of the filter. When the first and second filter regions 580A and 580B have the same shape and the same size, the straight line representing the filter dividing direction is a straight line that actually divides the filter region 580 shown in FIG. Diameter).

  The filter regions 580A and 580B are color filters that transmit light of different specific wavelength bands. The filter area 580 including the filter areas 580A and 580B transmits a color common to the areas 580A and 580B. When the transmitted light is collected on the image sensor 542, unevenness may occur. In order to reduce the unevenness of collected light on the image sensor 542, the filter surface of the filter region 580 may be installed in parallel with the imaging surface of the image sensor 542.

  The first filter region 580A transmits the first wavelength region that is a part of the wavelength region received by the image sensor 542. For example, as shown in FIG. 33B, the first filter region 580A is a yellow (Y) filter that transmits light in the wavelength band corresponding to the R image and light in the wavelength band corresponding to the G image. The second filter region 580 </ b> B transmits a second wavelength region different from the first wavelength region in the color of light received by the image sensor 542. For example, as shown in FIG. 33B, the second filter region 580B is a magenta (M) filter that transmits light in the wavelength region corresponding to the B image and light in the wavelength region corresponding to the R image. Part of the first wavelength region and part of the second wavelength region overlap. The common color that both the Y and M filters transmit is R. In general, C, M, and Y complementary color filters are more sensitive than R, G, and B primary color filters, so that even if the transmittance in the same wavelength band is the same, more light is transmitted. It has been known.

  The combination of the first filter region 580A and the second filter region 580B is not limited to the above, and the first filter region 580A transmits light in the wavelength region corresponding to the R image and light in the wavelength region corresponding to the G image. In the filter, the second filter region 580B may be a cyan (C) filter that transmits light in the wavelength region corresponding to the B image and light in the wavelength region corresponding to the G image, and the first filter region 580A corresponds to the R image. The second filter region 580B transmits light in the wavelength region corresponding to the B image and light in the wavelength region corresponding to the G image by the M filter that transmits the light in the wavelength region corresponding to the B image and the light in the wavelength region corresponding to the B image. The C filter may be used, or the first filter region 580A may be any one of the C, M, and Y filters, and the second filter region 580B may be a transparent filter that transmits light of all colors. Further, in FIG. 33, the right side is the first filter region 580A and the left side is the second filter region 580B, but the right side is the second filter region 580B and the left side is the first filter region 580A. Good.

  The first and second filter regions 580A and 580B are a filter that changes the transmittance in an arbitrary wavelength band, a polarizing filter (polarizing plate) that transmits polarized light in an arbitrary direction, and a microlens that changes a condensing power in an arbitrary wavelength band. It may be. For example, filters for changing the transmittance in an arbitrary wavelength band include R, G, B primary color filters, C, M, Y complementary color filters, color correction filters (CC-RGB / CMY), infrared / ultraviolet cut filters, ND It may be a filter or a shielding plate. When the first and second filter regions 580A and 580B are microlenses, the blur function changes due to a deviation in the light collection by the photographing lens 538.

  FIG. 33B shows an example of transmittance characteristics of the first filter region 580A and the second filter region 580B. The transmittance characteristic 586A of the first filter region 580A that is a yellow filter is such that light in the wavelength band corresponding to the R image and G image is transmitted with high transmittance, and light in the wavelength band corresponding to the B image is hardly transmitted. Indicates. The transmittance characteristic 586B of the second filter region 580B which is a magenta filter is such that light in the wavelength band corresponding to the B image and the R image is transmitted with high transmittance, and light in the wavelength band corresponding to the G image is hardly transmitted. Indicates. Therefore, the light in the wavelength band corresponding to the R image is transmitted through both the first and second filter regions 580A and 580B, while the light in the wavelength band corresponding to the G image is transmitted only through the first filter region 580A. Since the light in the wavelength band corresponding to the image is transmitted only through the second filter region 580B, the shape of the blur on the G image and the B image changes according to the distance to the subject. Since each filter region has an asymmetric shape with respect to the optical center, the blur shape on the G image and the B image differs depending on whether the subject is in front of or behind the focus distance. That is, the blur shape on the G image and the B image is biased. Therefore, a blur correction filter for correcting the blur shape of the G image and the B image to the blur shape of the R image is prepared for each distance, and the blur correction filter at the assumed distance is calculated to the G image and / or the B image. The distance can be determined based on the correlation between the corrected image and the R image.

[Modification of color filter]
The filter area 580 in FIG. 33 is divided so that the entire area is constituted by the first and second filter areas 580A and 580B. As a modification, an example in which the filter area is divided into first, second, and third filter areas will be described.

  FIG. 34 shows a first modification of the color filter according to the second embodiment. As shown in FIGS. 34A and 34B, the filter region 590 includes a first filter region 590A, a second filter region 590B, and a third filter region 590c. As in the example of FIG. 33, the first and second filter regions 590A and 590B transmit a plurality of different colors, and the third filter region 590C transmits a common color that transmits both the first and second filter regions. To do. When the first and second filter regions 590A and 590B are Y, M or M, Y filters, the third filter region 590C is an R filter. When the first and second filter regions 590A and 590B are Y, C or C, Y filters, the third filter region 590C is a G filter. When the first and second filter regions 590A and 590B are C, M or M and C filters, the third filter region 590C is a B filter. The third filter region 590C may be a transparent filter that transmits all the colors R, G, and B.

  In FIG. 34 (a), the first and second filter regions 590A and 590B are located on the left and right of a straight line passing through the optical center 592 and extending along the Y axis, respectively, and the shape is a circle in contact with each other at the optical center 592. . The third filter region 590C is a region other than the first and second filter regions 590A and 590B. In FIG. 34 (b), the first and second filter regions 590A and 590B are located on the left and right sides of a straight line passing through the circular optical center 592 and extending along the Y axis, respectively, and the shapes are elliptical shapes that touch each other at the optical center 592. It is. In FIGS. 34A and 34B, the first and second filter regions 590A and 590B are arranged so as to be symmetrical with respect to a straight line passing through the optical center 592 and extending along the Y axis. Although it is an area | region of a magnitude | size, two area | regions of a different magnitude | size may be arrange | positioned by axisymmetrical. The shape of the first and second filter regions 590A and 590B is not limited to a circle and an ellipse, but may be a triangle, a rectangle, or a polygon.

  The first and second filter regions 590A and 590B are straight lines representing the filter dividing direction, which is a straight line that passes through the midpoint of the line segment that connects the centroids of the first and second filter regions 590A and 590B and is orthogonal to the line segment. When the line is symmetrical with respect to the Y axis, the line passes through the optical center 592 and extends along the Y axis.

  In the first modification of FIG. 34, the first and second filter regions are in contact with each other at the optical center, but a second modification in which the first and second filter regions are not in contact with each other will be described. FIG. 35 shows a second modification of the color filter according to the second embodiment. As shown in FIGS. 35A and 35B, the filter region 600 includes a first filter region 600A, a second filter region 600B, and a third filter region 600C. As in the example of FIGS. 33 and 34, the first and second filter regions 600A and 600B transmit light of different wavelength regions that partially overlap, and the third filter region 600C has both the first and second filter regions. Transmits light in the transmitted wavelength region. Examples of the colors of the first and second filter regions are the same as in the first modification.

  In FIG. 35A, the shapes of the first and second filter regions 600A and 600B are crescents located on the left and right of the straight line extending through the optical center 602 and extending along the Y axis. The third filter region 600C other than the first and second filter regions 600A and 600B is elliptical. In FIG. 35B, the circular filter region 600 is divided into three by two straight lines extending along the Y axis, the center is the third filter region 600C, and both sides of the third filter region 600C are the first. , Second filter regions 600A and 600B. The circular filter region 600 may be divided into three by two wavy lines instead of two straight lines. The two dividing lines may not be parallel. Further, the three divisions are not limited to three equal parts, and the sizes of the three regions are arbitrary.

  The first and second filter regions 600A and 600B are straight lines that represent the dividing direction of the filter, which is a straight line that passes through the midpoint of the line segment that connects the centroids of the first and second filter regions 600A and 600B and is orthogonal to the line segment. A line symmetric with respect to the Y axis is a straight line passing through the optical center 602 and extending along the Y axis.

  FIG. 36 shows a third modification of the color filter according to the second embodiment. The filter region 610 includes first and second filter regions 610A and 610B that are semicircular as in the filter region 580 of FIG. A plurality of third filter regions 610C are provided inside the first and second filter regions 610A and 610B. The shape, number, and arrangement of the third filter region 610C are arbitrary.

  The first and second filter regions 610A and 610B are straight lines representing the filter dividing direction, which is a straight line that passes through the midpoint of the line segment connecting the centroids of the first and second filter regions 610A and 610B and is orthogonal to the line segment. When the third filter region 610C is line symmetric with respect to the Y axis, the line is symmetric with respect to the Y axis, and is a straight line extending along the Y axis through the optical center 612.

  FIG. 37 shows a fourth modification of the color filter according to the second embodiment. The fourth modification is an example in which the first and second filter regions are not in contact with each other in the second embodiment shown in FIG. In FIG. 37 (a), the first and second filter regions 620A and 620B are positioned apart from each other on the left and right of a straight line that passes through the optical center 622 and extends along the Y axis, and have a circular shape. In FIG. 37 (b), the first and second filter regions 620A and 620B are spaced apart from each other on the left and right of a straight line passing through the optical center 622 and extending along the Y axis, and the shape is a square. The shape of the first and second filter regions 620A and 620B is not limited to a square, but may be a triangle, a rectangle, or a polygon, or a plurality of first and second filter regions 620A and 620B may be provided, or asymmetrical on the left and right. May be provided.

  The first and second filter regions 620A and 620B are straight lines representing the dividing direction of the filter, which is a straight line passing through the midpoint of the line segment connecting the centroids of the first and second filter regions 620A and 620B and orthogonal to the line segment. A line symmetric with respect to the Y axis is a straight line passing through the optical center 622 and extending along the Y axis.

  In summary, the filter region includes a first filter region and a second filter region in which the transmitted wavelength regions partially overlap. The filter region may further include a third filter region that transmits a wavelength region that is transmitted by both the first filter and the second filter. For example, the third filter region may transmit light in a wavelength region transmitted through the first filter region and a wavelength region transmitted through the second filter region. For example, the filter region may be composed of an arbitrary number and type of filter regions in addition to the first and second filter regions. Any number and types of filter regions may be selected from R filters, G filters, B filters, Y filters, C filters, M filters, and transparent filters.

  According to the second embodiment, the image capturing unit 505 is rotated about the optical axis, and the image capturing unit 505 or the image capturing device 502 is installed so that the filter region dividing direction of the color aperture is an arbitrary direction. Since it is orthogonal to the dividing direction, the direction of the edge, which is difficult to calculate the distance to the subject, can be changed. Therefore, the direction of the edge for which it is difficult to calculate the distance can be set as the direction of the edge that is not included or hardly included in the subject. For example, it is possible to prevent a situation in which it is difficult to calculate the distance regarding the edge in the horizontal direction.

  18 and 19, the roll rotation may be an electric rotation, and the imaging unit 505 or the imaging device 502 may be automatically rotated according to an index relating to the calculated reliability of distance.

[Third Embodiment]
FIG. 38 shows an installation example of the imaging apparatus 502 according to the third embodiment. The third embodiment can also be applied to a monitoring system. In the second embodiment, an obliquely lower part is photographed from a ceiling, a wall, a pillar, and the like, but a third embodiment in which a photograph is taken directly below from the ceiling will be described. That is, the optical axis of the imaging unit 505 is parallel to the vertical direction. The tip of a columnar arm 524 is fixed at the center of the rear end of the imaging device 502. The axis of the arm 524 coincides with the optical axis of the imaging unit 505 or the imaging device 502. The rear end of the arm 524 is inserted into the tip of an arm 526 that is concentric with the arm 524 and has a larger diameter than the arm 524. For this reason, the arm 524 (and the imaging device 502) can be rotated in the clockwise and counterclockwise directions around the optical axis (also referred to as a roll axis) while being inserted into the arm 526. The upper end of the arm 526 is integrated with the mounting plate 530.

  Similar to the second embodiment, the roll angle only needs to be greater than 0 degrees in both the clockwise and counterclockwise directions. If the roll angle is 90 degrees, the edge distance in the vertical direction may be difficult to calculate. The roll angle may be about 45 degrees. Edges that are difficult to calculate are unavoidable, and it is up to the user to decide which edge is calculated and which edge is difficult to calculate.

  After installation, the subject can be photographed to calculate the distance, and an appropriate angle can be obtained by trial and error while adjusting the roll angle while viewing the result. After installation, the roll angle may be determined based on an index such as reliability as shown in FIGS. Alternatively, when the directions of various edges included in the subject are known in advance, the roll angle can be set so that the division direction of the filter region is not orthogonal to the most edge directions included in the subject. Alternatively, an appropriate roll angle is calculated in advance so that the direction of all edges is not orthogonal to the filter division direction, in other words, the filter region division direction does not coincide with the density gradient direction of the subject. The image pickup apparatus 502 may be installed so as to match the angle. Further, when an appropriate roll angle is obtained in advance from the direction of a known edge included in the subject, when the mounting plate 530 is attached to the ceiling, the imaging device 502 is operated with the imaging unit 505 rotated about the optical axis. If it can be attached, a roll rotation mechanism as shown in FIG. 38 is not essential. However, if a roll rotation mechanism is provided, it is possible to easily cope with a change in the direction of the edge included in the subject. If the roll rotation mechanism is not provided, the imaging unit 505 or the imaging device 502 may be installed again in an arbitrary direction when the direction of the edge included in the subject changes.

[Rotation of imaging unit]
FIG. 39 shows an example of rotation of the imaging unit 505 or the imaging device 502 according to the third embodiment. In the third embodiment, since the optical axis of the imaging unit 505 or the imaging apparatus 502 is along the vertical direction, when the vertical direction is projected onto the filter surface, it becomes a point instead of a straight line. For this reason, the definition of the condition for which it is difficult to calculate the distance is different from the definition of the second embodiment. In the third embodiment, two orthogonal axes (referred to as principal axes) are defined in the scene of the subject. The main axis is set based on the main structure of the scene. For example, as shown in FIG. 40A, when the imaging range is indoors, the first main axis and the second main axis can be set along two sides of a generally rectangular floor surface. For example, a plane parallel to the first main axis and the second main axis is parallel to the floor surface. Further, when the person is moving, the first main axis and the second main axis may be set along the moving direction and the direction orthogonal thereto. As shown in FIG. 40B, when the imaging range includes a road, a corridor, and the like, the first main axis and the second main axis can be set along the extending direction of the road, the corridor, and the direction orthogonal thereto. For example, a plane parallel to the first main axis and the second main axis is parallel to a road or a hallway. Further, when a car or a person is moving, the first main axis and the second main axis may be set along the moving direction and the direction orthogonal thereto. When the imaging range includes a moving object such as an automobile or a person, for example, a plane parallel to the first principal axis and the second principal axis is parallel to a plane on which the automobile or person moves. When the roll angle of the imaging unit 505 or the imaging device 502 is 0 degree, as shown in FIG. 39A, a straight line obtained by projecting the first principal axis onto the filter surface or a straight line obtained by projecting the second principal axis onto the filter surface, The straight lines representing the division directions are parallel. In this state, it may be difficult to calculate the distance of the edge in the second principal axis direction. As shown in FIG. 39B, the imaging unit 505 or the imaging device 502 (arm 524) is rotated around the optical axis, and a straight line representing the filter dividing direction is a straight line obtained by projecting the first principal axis onto the filter surface. The second principal axis is not parallel to the straight line projected on the filter surface, that is, the straight line representing the filter dividing direction intersects the straight line obtained by projecting the first principal axis on the filter surface and the straight line projected on the filter surface on the second principal axis. be able to.

  Thereby, the edge in the direction of the first principal axis and the second principal axis included in the subject and the filter division direction are not orthogonal, and the distance between the edges in the direction of the first principal axis and the second principal axis can be calculated. In this case, the roll angle is larger than 0 degree and smaller than 90 degrees. The roll angle is, for example, 55 degrees or less. The roll angle is, for example, 35 degrees or more.

  Also in the third embodiment, similarly to the second embodiment, the image capturing unit 505 or the image capturing apparatus 502 is rotated about the optical axis, and the filter region dividing direction of the color opening is rotated, thereby dividing the filter region. Since the direction is orthogonal to the direction, the direction of the edge where the distance to the subject is difficult to calculate can be changed. Therefore, an edge whose distance is difficult to calculate can be an edge in a direction that is not included or hardly included in the subject. For example, it is possible to prevent a situation in which it is difficult to calculate the distance related to the edges in the direction of the first principal axis and the second principal axis in the subject.

  In the above-described embodiment, the display of the distance image has been described as an example of the output form of the distance information. In addition to the distance to the subject shown in each pixel, the maximum value / minimum value / center value / average value of the distance of the subject shown in the entire image may be output. Further, not only the distance image of the entire image but also the distance of a part of the image may be output.

  The following information can be obtained by processing the blur of the image signal of each pixel using the distance information. Generates all-in-focus images in which the image signals of all pixels are in focus, and refocused images in which subject areas that are different from those at the time of shooting are in focus, and those that are in focus at the time of shooting are out of focus. can do. It is also possible to extract an object at an arbitrary distance and recognize the extracted object. Furthermore, the behavior of the object can also be estimated by following a change in the distance of the recognized object to the present.

  In the embodiment, the distance information is displayed so as to be recognized by the user in the processing device. However, the present invention is not limited to this, and the distance information may be output to another device and used by the other device. According to the embodiment, a captured image and distance information can be acquired using a monocular camera without using a stereo camera, and the compact and lightweight monocular camera can be applied to various fields.

[Application example 1: Monitoring system]
The monitoring system detects an intruder into the space where the image is being taken by the imaging device and issues an alarm. FIG. 42 shows an example of a monitoring system, which is a system for grasping the flow of people and vehicles in a parking lot for each time zone. The monitoring system is used not only for parking lots but also for monitoring various subjects moving within a range captured by an imaging device such as a flow of people in a store.

  FIG. 41 shows a block diagram of the monitoring system. The imaging device 502 is input to the processing device 632 corresponding to both the captured image acquisition unit 562 and the distance image acquisition unit 564 of FIG. The captured image output from the processing device 632 and distance information for each pixel are input to the person detection unit 634. The person detection unit 634 detects a person or a moving body based on a change in distance. The detection result is supplied to the area entry / exit detection unit 636. The area intrusion / exit detection unit 636 determines whether or not a person or a moving body has entered a specific area within a predetermined range from the imaging apparatus 502 based on the detected distance to the person or moving body, or a person or a movement from the specific area. Determine if the body has left. The area intrusion / exit detection unit 636 may be, for example, a flow of a person such as a person entering a certain reference distance, a person exiting a certain reference distance, a car entering a certain reference distance, a vehicle For example, the flow of the vehicle such as having exited from a certain reference distance may be analyzed, and the analysis result may be recorded in a storage device such as an HDD (Hard Disk Drive). The person detection unit 634 and the area entry / exit detection unit 636 may be included in the CPU, for example. The process of detecting a person or moving body and the process of determining whether a person or moving body has entered a specific area within a predetermined range or whether a person or moving body has left the specific area are: A single process may be performed collectively.

 FIG. 42 shows a usage example of the monitoring system. Using the imaging device 502 installed in the parking lot, it is possible to monitor the movement of people and cars in the parking lot. The specific area can be set to a part of the imageable area.

When intrusion / exit of a person or a moving body is detected, a predetermined warning is issued by the user interface unit 638. The warning is, for example, display by a display unit or sound output by a speaker. The user interface unit 638 also executes input processing from, for example, a keyboard or a pointing device. When the user interface unit 638 includes a display device and a pointing device, the user interface unit 638 may be, for example, a touch screen display in which the display device and the pointing device are integrated.

  Instead of issuing a warning, the surveillance camera can take other actions. For example, a space in front of the automatic door may be photographed with a camera, and the door may be opened when a person moves into the space.

[Application example 2: Automatic door system]
FIG. 43 shows a functional configuration of an automatic door system including the imaging device 502. The automatic door system includes an imaging device 502, a control signal generation unit 642, a drive mechanism 644, and a door unit 646.

  The control signal generation unit 642 is the same as the monitoring unit 630 in FIG. That is, the control signal generation unit 642 includes the functions of the captured image acquisition unit 562, the distance image acquisition unit 564 in FIG. 31, the person detection unit 634, and the area intrusion / exit detection unit 636 in FIG. It is determined whether it is in front or behind, and based on the determination result, a control signal relating to the opening / closing of the door portion 646 is generated, and the generated control signal is output to the drive mechanism 644. More specifically, the control signal generation unit 642 is free of the control signal or the door unit 646 for opening the door unit 646 based on the determination result indicating that the subject is closer to the reference distance. A control signal for maintaining the state is generated and output to the drive mechanism 644. In addition, the control signal generation unit 642 maintains the control signal for closing the door unit 646 or the closed state of the door unit 646 based on the determination result indicating that the subject is behind the reference distance. A control signal for generating the signal is generated and output to the drive mechanism 644.

  The drive mechanism 644 has, for example, a motor, and opens and closes the door portion 646 by transmitting the drive of the motor to the door portion 646. The drive mechanism 644 is operated based on the control signal generated by the control signal generation unit 642 so that the door unit 646 is opened or closed.

  FIG. 44 shows an operation example of the automatic door system. An imaging device 502 is installed using a fixture as shown in FIG. 28, for example, above the door 646, which is a position where a pedestrian or the like moving in front of the door 646 can be photographed. In other words, the imaging device 502 is installed so that an image obtained by bird's-eye view of a passage in front of the door portion 646 can be acquired.

  The reference distance in the control signal generation unit 642 is set to a certain distance from the door unit 646 on the front surface of the door unit 646, for example. Since the optical axis of the imaging device 502 is oblique to the floor surface, a plane 652 that is orthogonal to the floor surface but is oblique to the optical axis of the imaging device 502 is set as a reference distance surface. The imaging device 502 installed above determines whether the pedestrian 650 is in front of or behind this plane 652.

  In the example shown in FIG. 44A, it is determined that the pedestrian 106 is in front of the reference plane 652. Based on the determination result, the control signal generation unit 642 generates a control signal for opening the door unit 646 and outputs the control signal to the drive mechanism 644. The drive mechanism 644 operates so that the door unit 646 is opened based on the control signal received from the control signal generation unit 642.

  In the example shown in FIG. 44B, it is determined that the pedestrian 106 is behind the reference plane 652. Based on the determination result, the control signal generation unit 642 generates a control signal for closing the door unit 646 and outputs the control signal to the drive mechanism 644. The driving mechanism 644 operates based on the control signal received from the control signal generation unit 642 so that the door unit 646 is in a closed state.

  Such an automatic door system can also be applied to control of an automobile door. As shown in FIG. 45, an imaging device 502A for photographing the right side surface of the automobile and an imaging device 502B for photographing the left side surface are provided on the windshield of the automobile. When it is determined that the person has changed from the back to the front of the first plane set at the first distance from the imaging devices 502A and 502B, the door may be opened. If it is determined that the person has changed from the back to the front of the second plane set at the second distance from the imaging devices 502A and 502B, the door will not be opened even if the door is opened from inside the company. Also good. The second distance can be a distance shorter than the first distance, for example. This is to prevent an accident that the door and the person or the object come into contact with each other by opening the door when the person or the object is approaching the automobile.

[Application Example 3: Mobile Control System]
FIG. 46 illustrates a functional configuration example of the moving object 670 including the imaging device 502. Here, it is assumed that the moving body 670 is a robot that moves autonomously, such as a mobile robot (Automated Guided Vehicle), a cleaning robot, or a communication robot. The moving body 670 is not limited to such a robot, and may be realized as various things such as a vehicle including an automobile, a flying body such as a drone or an airplane, a ship, and the like as long as it has a driving mechanism for movement. Further, the moving body 670 may include not only a robot body that moves, but also an industrial robot having a driving mechanism for moving and rotating a part of the robot, such as a robot arm.

  As illustrated in FIG. 46, the moving body 670 includes an imaging device 502, a control signal generation unit 672, and a drive mechanism 674. As shown in FIG. 47, the imaging device 502 is installed so as to image a subject in the traveling direction of the moving body 670, for example. As a form installed so as to image the subject in the traveling direction of the moving body 670, it can be installed as a so-called front camera that images the front, or it can be installed as a so-called rear camera that images the rear when back. Of course, both of these may be installed. Further, the imaging device 502 may be installed so as to function as a so-called drive recorder. That is, the imaging device 502 may be a recording device. Note that when the movement and rotation of a part of the moving body 670 is controlled, the imaging device 502 may be installed at the tip of the robot arm or the like so as to image an object gripped by the robot arm, for example.

  The control signal generation unit 672 is the same as the processing device 632 of FIG. 41, and based on the distance to the subject, the mobile body 670 or a part thereof is accelerated, decelerated, stopped, collision avoidance, direction change, air bag safety, etc. A control signal for at least one of the operation of the device is generated.

  As with the control signal generation unit 643 in FIG. 43, the control signal generation unit 672 determines whether the subject has entered a specific area within the predetermined range or has left the specific area based on the distance to the subject. May be determined. In this case, based on the determination result indicating that the subject is in front of the reference distance, the control signal generation unit 672 includes deceleration, collision avoidance, direction change away from the subject, and operation of the safety device. A control signal for at least one may be generated. In addition, the control signal generation unit 672 can generate a control signal related to at least one of acceleration and a change in direction toward the subject based on the determination result that the subject is closer to the reference distance. The control signal generation unit 672 outputs the generated control signal to the drive mechanism 674.

  The drive mechanism 674 operates the moving body 670 based on this control signal. That is, the driving mechanism 674 operates based on the control signal so that at least one of acceleration, deceleration, collision avoidance, direction change, and operation of a safety device such as an airbag is performed by the moving body 670 or a part thereof. To do. Such a configuration is suitable for, for example, movement of a robot or automatic driving of a car that requires real-time control.

  When the moving body 670 is a drone, the imaging device 502 acquires an image obtained by photographing the inspection target and detects the distance to the subject when the inspection such as a crack from the sky or a wire breakage occurs. Determine whether it is in front or behind. Based on the detection result or determination result, the control signal generation unit 672 generates a control signal for controlling the drone thrust so that the distance from the inspection target is constant. The drive mechanism 674 operates the drone based on this control signal, so that the drone can fly in parallel with the inspection target.

  In addition, when the drone flies, the imaging device 502 acquires an image obtained by capturing the direction of the ground, detects the height of the drone from the ground, or whether the height from the ground is in front of the reference distance or in the back. It is determined whether or not. The control signal generation unit 672 generates a control signal for controlling the thrust of the drone so that the height from the ground becomes the specified height based on the detection result or the determination result. The drive mechanism 674 operates the drone based on the control signal, so that the drone can fly at a specified height.

  Further, when the moving body 670 is a drone or a car, the imaging device 502 acquires an image of a surrounding drone or a car in front of the drone or a car during a cooperative flight of the drone or a car regime, and The distance is detected, or it is determined whether the drone or the car is in front of or behind the reference distance. Based on the detection result or the determination result, the control signal generation unit 672 generates a control signal for controlling the drone thrust and the vehicle speed so that the distance from the surrounding drone and the vehicle ahead is constant. Generate. The drive mechanism 674 operates the drone or the automobile based on the control signal, so that the drone can perform the cooperative flight or the automobile regime run easily.

  FIG. 48 is a block diagram showing an example of drone movement control that can avoid an obstacle. The output of the imaging device 502 is input to the processing device 680 including the functions of both the captured image acquisition unit 562 and the distance image acquisition unit 564 of FIG. The captured image output from the processing device 680 and the distance information for each pixel are input to the obstacle recognition unit 682. The drone's travel route is automatically determined once the destination and current location are known. The drone includes a GPS 686, and the destination information and the current location information are input to the movement route calculation unit 684. The travel route information output from the travel route calculation unit 684 is input to the obstacle recognition unit 682 and the flight control unit 688. The flight control unit 688 rotates and steers the rotor.

  The obstacle recognition unit 682 extracts an object within a certain distance from the drone based on the captured image and the distance information. The detection result is supplied to the movement route calculation unit 684. When an obstacle is detected, the movement route calculation unit 684 corrects the movement route determined from the destination and the current location to a movement route having a smooth trajectory that can avoid the obstacle.

  Thereby, even when an unexpected obstacle appears in the air, it is possible to automatically avoid the obstacle and safely fly the drone to the destination. FIG. 48 is applicable not only to drones but also to mobile robots (Automated Guided Vehicles), cleaning robots, and the like whose traveling routes are determined. In the case of a cleaning robot, the route itself is not determined, and rules such as turning and retreating may be determined when an obstacle is detected. Even in this case, the configuration of FIG. 48 can be applied to the detection and avoidance of obstacles.

  FIG. 49 shows an example of a system configuration when the embodiment is applied to an automobile. The output of the imaging device 502 is input to the processing device 692 including the functions of both the captured image acquisition unit 562 and the distance image acquisition unit 564 of FIG. The processing device 692 outputs the captured image and distance information for each pixel. The captured image and distance information are input to the pedestrian / vehicle detection unit 694. The pedestrian / vehicle detection unit 694 sets an object perpendicular to the road in the captured image as a pedestrian / vehicle candidate area based on the captured image and the distance information. The pedestrian / vehicle detection unit 694 calculates a feature amount for each candidate region, and compares the feature amount with a large number of reference data obtained in advance from a large amount of sample image data, thereby obtaining a pedestrian / vehicle. Can be detected. When a pedestrian / vehicle is detected, a warning 698 may be issued to the driver, or the automatic brake 696 may be activated to decelerate or stop the automobile.

  The imaging device 502 is not limited to the front camera in the driver's seat, but may be a side camera attached to the side mirror or a rear camera attached to the rear glass. In the case of a side camera and a rear camera, instead of detecting a pedestrian / vehicle, an obstacle when moving backward and parking may be detected. In the case of a front camera, in recent years, a drive recorder has been developed that records on the SD card or the like a scene in front of the automobile taken by a camera attached to the windshield of the automobile. By applying the camera of the embodiment to the camera of this drive recorder, it is possible to obtain distance information in addition to a captured image in front of the automobile without providing a separate camera in the vehicle.

  Although it is not a moving body but is stationary, it can also be applied to a device provided with a moving unit, such as a manufacturing robot. If an obstacle is detected according to the distance from the arm that grips, moves, or processes the part, the movement of the arm may be limited.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 10 ... Filter, 11 ... 1st filter area | region, 12 ... 2nd filter area | region, 20 ... Lens, 30 ... Image sensor, 40 ... CPU, 41 ... Image processing part, 50 ... RAM, 60 ... Memory card slot, 70 DESCRIPTION OF SYMBOLS ... Display, 80 ... Communication part, 90 ... Non-volatile memory, 100 ... Imaging device, 110 ... Bus, 200 ... Robot, 201 ... Control part, 202 ... Drive mechanism, 203 ... Rotation mechanism, 300 ... Moving body, 301 ... Control , 302 ... Drive mechanism, 400 ... Monitoring system, 401 ... Control unit, 402 ... User interface unit, 411 ... Image acquisition unit, 412 ... Distance calculation unit, 413 ... Reliability calculation unit, 414 ... Output unit, 502 ... Imaging Device, 504 ... Color opening.

Embodiments of the present invention, the processing unit relates to an imaging apparatus and an automatic control system.

An object of the present invention is to provide is to provide a processing apparatus, an imaging apparatus and an automatic control system that can be output and control over the reliability of the distance obtained from the image.

JP-A-2015-194486 JP 2004-340714 A Japanese Patent No. 4997525 Utility Model Registration No. 3179836 JP 2006-102733 A

Claims (26)

Acquisition of a first image of a subject including a blur of a shape indicated by a symmetric first blur function and a second image of the subject including a blur of a shape indicated by an asymmetric second blur function And
A distance calculating unit that calculates a distance to the subject based on a correlation between the first blur function and the second blur function;
A reliability calculation unit that calculates the reliability of the distance based on the degree of correlation;
A processing apparatus comprising: The distance calculation unit calculates the distance based on a correlation between each of a plurality of correction images generated by the second image and each of a plurality of blur correction filters, and the first image,
The reliability calculation unit calculates the reliability of the distance based on a curvature of a correlation function between each of the plurality of corrected images and the first image;
The processing apparatus according to claim 1.   The reliability calculation unit calculates the reliability of the distance based on a curvature of a correlation function between each of the plurality of corrected images and the first image, and an edge direction of the subject image or an edge strength of the subject image. The processing apparatus according to claim 2.   The processing apparatus further includes an output unit that outputs a map, and the map includes the plurality of second points corresponding to the plurality of first points in either the first image or the second image. The processing apparatus according to claim 1, wherein the distance of the first point and the reliability of the distance are indicated.   The processing apparatus further includes an output unit for outputting a list, and the list indicates the coordinates of the first image or the second image, the distance at the coordinates, and the reliability of the distance. Item 2. The processing apparatus according to Item 1. The processing device further includes an output unit for outputting output data,
The output data includes color image data acquired by the acquisition unit, distance data including the distances of a plurality of points on the image indicated by the color image data, and reliability of the distances of the plurality of points. The processing apparatus according to claim 1, further comprising reliability data that includes the data. The processing apparatus according to any one of claims 1 to 6,
An imaging unit that captures the first image and the second image;
An imaging apparatus comprising:   The imaging device according to claim 7, wherein the first image and the second image are images captured by the same imaging device at the same time.   9. The display unit according to claim 7, further comprising a display unit capable of displaying a display image including the distance corresponding to the position on the first image or the second image and the reliability of the distance. The imaging device described in 1. An input unit for receiving a designation of a position on the display image;
The display unit displays the distance of the position on the first image or the second image corresponding to the position on the display image designated by the input unit and the reliability of the distance;
The imaging device according to claim 9.   11. The imaging device according to claim 9, wherein the display unit outputs a message prompting the user to rotate the imaging device when the reliability of the distance is equal to or less than a threshold value. The imaging device according to any one of claims 7 to 11,
Based on the distance and the reliability of the distance, a control unit that controls the driving mechanism of the moving body;
An automatic control system for a moving body.   The automatic control system according to claim 12, wherein the control unit controls the drive mechanism using a lower limit value of the distance calculated from the distance and the reliability of the distance.   The automatic control system according to claim 13, wherein the control unit stops, decelerates, accelerates, or starts moving by controlling the drive mechanism based on the lower limit value. The imaging device is attached to a rotation mechanism,
The automatic control system according to claim 12, wherein the control unit controls the rotation mechanism so that reliability of the distance obtained from the imaging device is higher. A camera including at least an opening including a filter including a first filter region and a second filter region;
The camera is installed such that a first straight line obtained by projecting a straight line representing a vertical direction onto the filter and a second straight line representing a dividing direction of the first filter region and the second filter region of the filter are not parallel to each other. Means to
An imaging apparatus comprising:   The imaging apparatus according to claim 16, further comprising a processing unit that rotates a captured image based on an angle formed by the first straight line and the second straight line. An imaging device according to claim 17,
A display unit for displaying the rotated photographed image. A camera including at least a filter including at least a first filter region and a second filter region in an opening, wherein a first straight line obtained by projecting a straight line representing a vertical direction onto the filter, a first filter region of the filter, Photographing a subject using a camera in which the second straight line representing the division direction of the two filter areas is non-parallel,
Obtaining distance information from the filter to the subject from a captured image of the subject;
A distance information acquisition method comprising:   The distance information acquisition method according to claim 19, further comprising rotating a captured image based on an angle formed by the first straight line and the second straight line.   The distance information acquisition method according to claim 20, further comprising displaying the rotated photographed image. A camera including at least an opening including a filter including a first filter region and a second filter region;
A first straight line obtained by projecting a straight line representing a first principal axis included in a photographed image output from the camera onto the filter, and a straight line representing a second principal axis perpendicular to the first principal axis included in the photographed image are used as the filter. Means for installing the camera so that a second straight line projected onto the first straight line and a third straight line representing the dividing direction of the first filter region and the second filter region of the filter are non-parallel,
An imaging apparatus comprising:   The imaging device according to claim 22, wherein the first main axis and the second main axis are along two orthogonal sides of a floor or wall of a room included in the imaging range.   The imaging apparatus according to claim 22, wherein the first main axis or the second main axis is along an extending direction of a road or a traveling direction of a vehicle included in an imaging range.   The imaging device according to any one of claims 22 to 24, further comprising a processing unit that rotates the captured image based on an angle formed by the first straight line and the second straight line.   26. The imaging apparatus according to claim 25, further comprising a display unit that displays the rotated photographed image.

Images