JP2013106217A - Depth estimating/imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide imaging technique for acquiring depth information in a method different from a conventional one.SOLUTION: A depth estimating/imaging apparatus includes: a first optical filter 1a and a second optical filter 1b, in which transmission factors to light in a specified wavelength region periodically change in a first direction parallel to an imaging face of an imaging element 2. The first optical filter 1a and the second optical filter 1b confront each other, and periodical changes of the transmission factors to the specified wavelength region are mutually shifted by 1/4 period.

Description

  The present invention relates to a monocular three-dimensional imaging technique for acquiring depth information of a subject using one optical system and one imaging device.

  In recent years, there has been a remarkable increase in functionality and performance of digital cameras and digital movies that use a solid-state imaging device (hereinafter referred to as “imaging device”) such as a CCD or CMOS. In particular, with the advancement of semiconductor manufacturing technology, the pixel structure in an image sensor has been miniaturized. As a result, higher integration of pixels and drive circuits of the image sensor has been achieved. For this reason, in a few years, the number of pixels of the image sensor has increased significantly from about 1 million pixels to over 10 million pixels. Furthermore, the quality of the image obtained by imaging has improved dramatically. On the other hand, with respect to the display device, a thin liquid crystal display or a plasma display enables high-resolution and high-contrast display without taking up space, and high performance is realized. Such a flow of improving the quality of video is spreading from a two-dimensional image to a three-dimensional image. In recent years, polarized glasses are required, but high-quality three-dimensional display devices are being developed.

  Regarding a three-dimensional imaging technique, there is a method of acquiring a right-eye image and a left-eye image using an imaging system including two cameras, as a typical method having a simple configuration. In such a so-called two-lens imaging method, since two cameras are used, the imaging apparatus becomes large and the cost can be high. Therefore, a method (monocular imaging method) for acquiring a plurality of images having parallax (hereinafter sometimes referred to as “multi-viewpoint images”) using one camera has been studied.

  For example, Patent Documents 1 and 2 disclose a method of acquiring a multi-viewpoint image using two polarizing plates whose transmission axes are orthogonal to each other and a rotating polarizing filter. Patent Documents 3 to 5 disclose methods for acquiring a multi-viewpoint image using a diaphragm (light flux limiting plate) provided with a plurality of color filters.

  The methods disclosed in Patent Documents 1 to 5 are mainly used when a multi-viewpoint image is generated by a monocular camera. On the other hand, there is a technique that can acquire depth information using a monocular camera equipped with a plurality of microlenses and can freely change the focal position of the acquired image based on the information. Such a technique is called light field photography, and a monocular camera using it is called a light field camera. In a light field camera, a plurality of microlenses are arranged on an image sensor. Each microlens is disposed so as to cover a plurality of pixels. After imaging, the depth of the subject can be estimated by calculating information related to the direction of incident light from the acquired image information. Such a camera is disclosed in Non-Patent Document 1, for example.

  In a light field camera, depth information can be calculated. However, since the resolution is determined by the number of microlenses, there is a problem that the resolution is lower than the resolution determined by the number of pixels of the image sensor. In response to this problem, Patent Document 6 discloses a technique for improving resolution using two imaging systems. In this technology, incident light is divided into two, and each of the divided incident lights is imaged by an imaging system having a microlens group arranged with a spatially shifted pitch of 1/2 pitch, and then the acquired images are synthesized. To improve the resolution. However, this technique requires two imaging systems and has problems in terms of size and cost.

  A technique for switching between a normal imaging mode and a mode based on light field photography using a single imaging system is disclosed in Japanese Patent Application Laid-Open No. 2004-151867. According to this technique, a microlens whose focal length changes according to the applied voltage is used, and the focal length of the microlens is set to infinity in the former mode, and is set to a predetermined distance in the latter mode. The With such a mechanism, an image with high resolution and depth information can be obtained. However, this method requires an advanced control technique for controlling the focal length of the microlens.

JP-A-62-291292 JP-A-62-2217790 JP 2002-344999 A JP 2009-276294 A JP 2003-134533 A Japanese Patent Laid-Open No. 11-98532 JP 2008-167395 A

Ren Ng, et al, “Light Field Photography with a Hand-held Plenological Camera”, Stanford Tech Report CTSR 2005-02

  In the light field camera, depth information can be obtained, but there is a problem that the resolution of the image is lowered. In order to solve the problem, there are problems that two imaging systems are required or the focal length of the microlens has to be controlled as in the techniques of Patent Documents 6 and 7 described above.

  In the embodiment of the present invention, in view of the above problems, an imaging technique capable of acquiring depth information using an optical system and signal processing different from those of the conventional techniques is provided. Furthermore, in a preferred embodiment of the present invention, there is provided an imaging technique that can acquire an image with no reduction in resolution and depth information by one imaging.

  The depth estimation imaging apparatus according to the present invention includes an imaging element in which a plurality of photosensitive cells are arranged in a matrix on an imaging surface, an optical lens arranged to form an image on the imaging surface, the optical lens, and the optical lens. A first optical filter disposed between the imaging elements and having a transmittance with respect to light in a specific wavelength range periodically changing in a first direction parallel to the imaging surface; and the first optical filter The transmittance with respect to the light in the specific wavelength range, which is arranged between the image sensor and the image sensor, periodically changes in the first direction, and the periodic change of the transmittance is in the first optical filter. A second optical filter shifted by ¼ period compared to a periodic change in transmittance with respect to light in the specific wavelength range, and a signal processing unit for processing pixel signals output from the plurality of photosensitive cells. I have.

  In one embodiment, the signal processing unit generates information indicating the depth of a subject based on a spatial frequency characteristic of pixel signals output from the plurality of photosensitive cells.

  In one embodiment, the signal processing unit further includes a memory in which information defining a correspondence relationship between the spatial frequency characteristics of the pixel signals output from the plurality of photosensitive cells and the depth of the subject is recorded. The information indicating the depth is generated by referring to the information defining the correspondence.

  In one embodiment, a distance between the first optical filter and the second optical filter at a position corresponding to a specific row or column in the array of the plurality of photosensitive cells is the specific row or column. Is different from the distance between the first optical filter and the second optical filter at a position corresponding to a row or a column adjacent to.

  In one embodiment, the first optical filter and the second optical filter are arranged at positions corresponding to every n rows (n is an integer of 1 or more) or every n columns of the array of the plurality of photosensitive cells. Has been.

  In one embodiment, the spatial change in the first direction of the transmittance of the first optical filter and the second optical filter with respect to the light in the specific wavelength range is represented by a trigonometric function or a rectangular function. .

In one embodiment, when a and b are positive real numbers satisfying a + b <1, b−a> 0, and ω ₀ is a positive real number, light in the specific wavelength band of the first optical filter A function representing a spatial change of the transmittance with respect to the first direction is approximately represented by asinω ₀ x + b, and the first transmittance of the second optical filter with respect to the light in the specific wavelength range is represented by A function representing a spatial change in the direction is approximately represented by acos ω ₀ x + b.

  In one embodiment, when the plurality of photosensitive cells are divided into a plurality of blocks each including u × v photosensitive cells arranged in u rows and v columns (u and v are integers of 2 or more), The first optical filter and the second optical filter are arranged so as to cover each block, and the transmittance is changed in the first direction by one period or more on each block.

  In one embodiment, in each block, the spatial change in the first direction of the transmittance of the first optical filter and the second optical filter with respect to the light in the specific wavelength range has a plurality of different periods. It is expressed as a linear combination of periodic functions.

  In one embodiment, the lower limit of the specific wavelength range is longer than 650 nm.

  In one embodiment, the depth estimation imaging apparatus further includes a third optical filter that is disposed closer to the subject than the first optical filter and transmits only visible light, and the specific wavelength range is Included in the visible light wavelength range.

  In one embodiment, the signal processing unit includes a pixel signal output from each photosensitive cell when the first and second optical filters are not provided, and the first and second optical filters. When a filter is provided, a visible light image is obtained by correcting pixel signals output from the plurality of photosensitive cells based on information indicating a ratio with pixel signals output from the photosensitive cells. Generate.

  In one embodiment, the depth estimation imaging device includes a fourth optical filter arranged on the side closer to the subject than the first optical filter, in which a plurality of light shielding portions and a plurality of transparent portions are alternately arranged. In addition, in the portion of the first optical filter on which the light beam that has passed through each transparent portion is incident, the transmittance for light in the specific wavelength region changes by one period or more in the first direction.

  According to the embodiment of the present invention, imaging is performed through two optical filters in which the spatial distribution of transmittance with respect to light in a specific wavelength range is represented by a periodic function and the phases are different from each other by π / 2 (1/4 period). For this reason, the influence of the periodic change pattern of the transmittance of these optical filters appears in the acquired pixel signal. Since these optical filters are arranged apart from each other in the optical axis direction, the condensing state of incident light can be grasped from the pattern of the image modulated by these optical filters. As a result, the depth information of the subject can be calculated. On the other hand, if an operation based on the transmission characteristics of the two optical filters is performed on the acquired pixel signal, a normal image can be acquired.

3 is a plan view showing optical filters 1a and 1b in Embodiment 1. FIG. It is an A-A 'line sectional view in Drawing 1A, and is a figure showing light flux condensed on an image pick-up surface. It is a graph which shows the change of the x direction of the transmittance | permeability with respect to the light of the 1st wavelength range of the optical filters 1a and 1b. FIG. 1B is a cross-sectional view taken along line A-A ′ in FIG. FIG. 1B is a cross-sectional view taken along the line A-A ′ in FIG. 1A and shows a light beam that is condensed before the imaging surface. FIG. 6 is a plan view showing optical filters 1a and 1b in a modification of the first embodiment. FIG. 6 is a cross-sectional view showing optical filters 1a and 1b and a plurality of photosensitive cells 10 according to another modification of the first embodiment. 6 is a plan view showing optical filters 1a and 1b in Embodiment 2. FIG. It is a B-B 'line sectional view in Drawing 2A. FIG. 2C is a cross-sectional view taken along line C-C ′ in FIG. 2A. It is a top view which shows the light shielding filter 1c in Embodiment 3, and the optical filters 1a and 1b. FIG. 5B is a cross-sectional view taken along line A-A ′ in FIG. 5A. FIG. 10 is a cross-sectional view showing optical filters 1a and 1b and a plurality of photosensitive cells 10 in a modification of the third embodiment. It is a block diagram which shows the structure of the imaging device in embodiment of this invention. 1 is a perspective view schematically showing an optical system configuration of an imaging apparatus in an embodiment of the present invention.

  Hereinafter, a depth estimation imaging device (hereinafter, simply referred to as “imaging device”) according to an embodiment of the present invention will be described with reference to the drawings. In the following description, the same or corresponding elements are denoted by the same reference symbols throughout the drawings.

(Embodiment 1)
First, the imaging device according to the first embodiment of the present invention will be described. FIG. 6 is a block diagram illustrating the overall configuration of the imaging apparatus according to the present embodiment. The imaging apparatus according to the present embodiment is a digital electronic camera, and includes an imaging unit 100 and a signal processing unit 200 that generates a signal (image signal) indicating an image based on a signal generated by the imaging unit 100. ing. The imaging device may have a function of generating not only a still image but also a moving image.

  The imaging unit 100 includes a solid-state imaging device 2 (hereinafter simply referred to as “imaging device”) including a plurality of photosensitive cells (pixels) arranged in a matrix on the imaging surface, and a wavelength region close to infrared light. An optical filter unit 1 that removes part of the light, an optical lens 3 for forming an image on the imaging surface of the imaging element 2, and a visible light transmission filter 4 that transmits only visible light are provided. The optical filter unit 1 includes two types of optical filters and is disposed on the imaging surface of the imaging device 2. Each of the two types of optical filters of the optical filter unit 1 is designed to absorb or reflect a part of light in the wavelength range of approximately 650 nm to 700 nm and transmit light in other wavelength ranges in general. On the other hand, the visible light transmission filter 4 is designed to transmit almost all visible light including light having a wavelength range of 650 nm to 700 nm and hardly transmit infrared light and ultraviolet light other than visible light. In this specification, an electromagnetic wave having a wavelength range of about 400 nm to 700 nm is referred to as “visible light”. The light of 650 nm to 700 nm described above is included in “visible light” in this specification, but is hardly visible to humans and may be treated as infrared light. In the following description, light of about 650 nm to 700 nm may be referred to as “light in the first wavelength range”.

  The imaging unit 100 also generates a basic signal for driving the imaging device 2, receives an output signal from the imaging device 2, and sends the signal to the signal processing unit 200. Signal generation / reception And an element driving unit 6 that drives the image sensor 2 based on the basic signal generated by the unit 5.

  The image sensor 2 is typically a CCD or CMOS sensor, and is manufactured by a known semiconductor manufacturing technique. The signal generation / reception unit 5 and the element driving unit 6 are composed of an LSI such as a CCD driver, for example. A plurality of photosensitive cells are arranged in a matrix on the imaging surface of the imaging device 2. Each photosensitive cell typically includes a photodiode and outputs a photoelectric conversion signal (pixel signal) corresponding to the intensity of received light.

  The signal processing unit 200 processes the signal output from the imaging unit 100 to generate an image signal and information indicating the depth of the subject (depth information), and various data used for generating the image signal. And an interface (IF) unit 8 for sending the generated image signal and depth information to the outside. The image processing unit 7 can be suitably realized by a combination of hardware such as a known digital signal processor (DSP) and software that executes image processing including image signal generation processing. The memory 30 is configured by a DRAM or the like. The memory 30 records the signal obtained from the imaging unit 100 and temporarily records the image data generated by the image processing unit 7 and the compressed image data. These image data are sent to a recording medium (not shown) or a display unit via the interface unit 8.

  Note that the imaging apparatus of the present embodiment may include known components such as an electronic shutter, a viewfinder, a power source (battery), and a flashlight, but a description thereof is omitted because it is not particularly necessary for understanding the present invention. In addition, the above configuration is an example, and in the present embodiment, components other than the optical filter unit 1, the image sensor 2, and the image processing unit 7 can be used by appropriately combining known elements.

  Hereinafter, the configuration of the imaging unit 100 will be described in more detail. In the following description, when explaining the position and direction of the area in the imaging unit 100, xyz coordinates shown in the figure are used. Note that terms such as “upper” and “lower” in the following description are based on the referenced drawings and do not necessarily reflect actual positional relationships.

  FIG. 7 is a diagram schematically illustrating an arrangement relationship of the lens 3, the visible light transmission filter 4, the optical filter unit 1, and the imaging element 2 in the imaging unit 100. The lens 3 may be a lens unit composed of a plurality of lens groups, but is illustrated as a single lens in FIG. 7 for simplicity. The lens 3 is a known lens, and collects incident light and forms an image on the imaging unit of the image sensor 2 regardless of the presence or absence of the visible light transmission filter 4. 7 is merely an example, and the present invention is not limited to such an example. For example, the arrangement relationship between the lens 3 and the visible light transmission filter 4 may be switched.

FIG. 1A is a plan view showing a part of the imaging surface of the imaging device 2 on which the optical filter unit 1 is arranged. Two types of optical filters 1a and 1b are arranged to face a plurality of photosensitive cells arranged in a matrix on the imaging surface. FIG. 1B is a cross-sectional view taken along the line AA ′ in FIG. 1A and shows a situation where the optical filters 1 a and 1 b are arranged on the plurality of photosensitive cells 10. The transmittance of the optical filters 1a and 1b with respect to the light in the first wavelength range periodically changes in the x direction. The frequency of this periodic change is ω ₀ . The phase of the periodic change of the transmittance of the optical filter 1a and the optical filter 1b with respect to the light in the first wavelength region is different by π / 2 (1/4 period).

Here, when the transmittances T1a and T1b of the optical filters 1a and 1b with respect to the light in the first wavelength range are expressed by equations, they are approximately expressed by the following equations 1 and 2, respectively. However, in both formulas, a and b are positive real numbers that satisfy a + b <1, ba−a> 0.

  FIG. 1C is a graph showing T1a and T1b. Here, the origin of the x coordinate may be any position, and may be set at the center or end of the optical filter unit 1, for example. The parameters a and b may be set to arbitrary values as long as the above conditions are satisfied. As described above, the spatial change in the x direction of the transmittance of the optical filters 1a and 1b in the present embodiment with respect to the light in the first wavelength band is represented by a trigonometric function.

  The optical filters 1a and 1b having such transmission characteristics can be manufactured using a known pigment or dye that mainly absorbs light in a wavelength range of 650 nm to 700 nm, for example. By adjusting the content of the pigment or dye according to the x-coordinate, it is possible to realize the transmission characteristics shown in Equation 1 and Equation 2.

  The optical filter 1b is disposed immediately above the photosensitive cell 10, and the optical filter 1a is disposed on the optical filter 1b at a distance d. Incident light passes through the optical filter 1a and the optical filter 1b in this order, and enters the photosensitive cell 10 for photoelectric conversion. Each photosensitive cell 10 outputs a pixel signal corresponding to the amount of received light by photoelectric conversion.

  With the above configuration, light incident on the imaging device during exposure is imaged on the imaging surface of the imaging device 2 through the lens 3, the visible light transmission filter 4, and the optical filter unit 1, and photoelectrically converted by each photosensitive cell 10. Is done. The pixel signal output from each photosensitive cell 10 is sent to the signal processing unit 200 via the signal generation / reception unit 5. The image processing unit 7 in the signal processing unit 200 generates an image based on the transmitted signal and calculates depth information.

  Hereinafter, depth information and image generation processing by the image processing unit 7 will be described. In the following description, a normal image signal using visible light may be referred to as a “normal image” or a “visible light image”.

  First, a method for calculating depth information will be described. The light that has passed through the lens 3 and the visible light transmission filter 4 passes through the optical filter 1a. At this time, with respect to the light in the first wavelength region, a light pattern modulated by the transmittance characteristic represented by Equation 1 in the x direction is generated due to the transmission characteristic of the optical filter 1a. That is, the intensity distribution of the light in the first wavelength region immediately after passing through the optical filter 1a is almost proportional to Equation 1. On the other hand, visible light outside the first wavelength band is transmitted through the optical filter 1a without being modulated. The light that has passed through the optical filter 1a further passes through the optical filter 1b. In the optical filter 1b, for the light in the first wavelength region, a light pattern modulated by the transmittance characteristic represented by Expression 2 is generated in the x direction. That is, the intensity distribution of the light in the first wavelength range that has passed through the optical filter 1b is substantially proportional to the product of the function expressed by Equation 1 and the function expressed by Equation 2 when the light can be regarded as parallel light. On the other hand, visible light outside the first wavelength range passes through the optical filter 1b without being modulated. The light transmitted through the optical filter 1b is photoelectrically converted by the light sensing cell 10. Although the above-described modulated light pattern is fine, the position of the optical filters 1a and 1b and the light sensing so that the frequency can be determined by analyzing the signal of the light sensing cell 10. The number and size of the cells 10 are adjusted in advance.

  In the present embodiment, the light collecting state on the imaging surface of the imaging device 2, that is, the surface on which the plurality of photosensitive cells 10 are arranged, varies depending on the distance (depth) from the lens 3 to the subject. Depth is required. For example, when light incident from a certain part of the subject forms an image on the imaging surface of the image sensor 2, according to wave optics, the incident light can be treated as parallel light as shown in FIG. 1B. it can. On the other hand, light incident from different depths forms an image behind or in front of the imaging surface as shown in FIG. 1D or FIG. 1E and cannot be treated as parallel light. In the example shown in FIG. 1D, the incident light forms an image behind the photosensitive cell 10, and in the example shown in FIG. 1E, the incident light forms an image in front of the photosensitive cell 10. Since the optical filters 1a and 1b have different arrangements (z-coordinates) on the optical axis, an optical pattern that is transmitted through the optical filter 1a and modulated by the transmittance represented by Equation 1 is incident on the optical filter 1b. The pattern varies depending on the light condensing state. If the frequency of the optical pattern on the optical filter 1b is ω, ω = ωo in the state shown in FIG. 1B, ω> ωo in the state shown in FIG. 1D, and ω <ωo in the state shown in FIG. 1E. .

From the above, the AC component of the light pattern due to the light photoelectrically converted by the photosensitive cell 10 is proportional to the value Ac represented by the following Equation 3.

In Equation 4, the fourth term sin (ω−ωo) x represents the amount of change in the light pattern that occurs while the light travels between the optical filters 1a and 1b, and is an odd function. For this reason, by detecting the component of the frequency (ω−ωo) corresponding to the fourth term from the pixel signal output from each photosensitive cell 10, it is determined how much light is focused or diverged. be able to. Specifically, for example, a low-frequency sin (ω−ωo) x component is extracted using a low-pass filter that removes a frequency component equal to or higher than ωo from the pixel signals output from the plurality of photosensitive cells 10, and the frequency analysis is performed. This is done by detecting the sign and frequency. Based on the detected code and frequency, the condensing state of incident light can be determined, and the distance to the subject can be estimated. In the case of ω <ω _0, the sin ωx component in Equation 4 remains, but ω−ω ₀ is significantly smaller than ω, and therefore only the low frequency component of the signal after the low-pass filter needs to be analyzed. Further, instead of ω0, for example, a low-pass filter that removes frequency components of ω0 / 2 or more may be used.

  Here, information (for example, a database) indicating a correspondence relationship between the value of the component of sin (ω−ωo) x and the distance to the subject is obtained in advance by experiments or simulations, and is recorded in a recording medium such as the memory 30. Has been. The image processing unit 7 can obtain the depth of the subject from the pixel signal with reference to the information.

  Here, the frequency ω depends on the distance d in the optical axis direction of the optical filters 1a and 1b. In the present embodiment, the distance d is set so that the frequency ω becomes a value close to ωo. In addition, since the above calculation is effective for a portion of a subject that does not have a low-frequency image frequency band close to (ω−ωo), in this embodiment, such a subject portion is subjected to depth calculation. set to target. However, about 650 nm to 700 nm, which is the wavelength range of light used for the depth calculation described above, corresponds to the red to infrared wavelength range. Therefore, if an RGB color image sensor is used, the spatial frequency of the subject image is affected. There is a possibility that depth information can be calculated. This is because the image of the light pattern and the subject image can be separated from an image of red (R) pixels and an image of green (G) pixels or blue (B) pixels. Therefore, the image processing unit 7 may estimate the depth information from the image by the red pixel and generate a normal image from the image by the green pixel and the blue pixel.

  Next, a process for acquiring a normal image (visible light image) with no resolution reduction will be described. In the present embodiment, the visible light transmission filter 4 is included in the imaging optical system. Therefore, if the optical filter unit 1 is not mounted, a normal image with visible light without a reduction in resolution can be acquired. However, since the optical filter unit 1 is actually mounted, light modulation occurs in a wavelength range close to red light to infrared light. As a result, the light pattern modulated by the optical filters 1a and 1b is superimposed on the subject image. Therefore, in the present embodiment, a ratio of pixel signals (hereinafter referred to as “light pattern removal ratio”) from an image when the optical filter unit 1 is not mounted in advance and an image when the optical filter unit 1 is mounted. Call). For example, using white light having a constant intensity, a ratio S1 / S2 between the pixel signal S1 when there is no optical filter unit 1 and the pixel signal S2 when there is the optical filter unit 1 is used as a light pattern removal ratio. Each time it is recorded in advance in the memory 30 or the like. The image processing unit 7 removes the influence of the light modulation by the optical filters 1a and 1b by converting the signal acquired from each photosensitive cell 10 using the light pattern removal ratio. For example, when the above S1 / S2 is the light pattern removal ratio, an image signal is obtained by multiplying each pixel signal acquired by imaging by S1 / S2. In this way, the image processing unit 7 acquires a normal image with no resolution reduction.

  As described above, according to the present embodiment, two types of optical filters 1a and 1b whose transmittance periodically changes in the x direction and whose periodicities are different from each other by about π / 2 are provided on the photosensitive cell 10. They are arranged at a predetermined distance d in the y direction. As a result, the pattern of light transmitted through the optical filters 1a and 1b (the intensity distribution in the x direction) is modulated to reflect the condensed state of incident light. By analyzing the spatial frequency characteristics of the pixel signal, the condensing state of incident light can be detected, and the distance to the subject can be estimated. In addition, a normal image with no reduction in resolution can be acquired by calculation using the light pattern removal ratio.

  In this embodiment, the optical filters 1a and 1b for attenuating a part of light in the wavelength range of about 650 nm to 700 nm close to infrared light are used. However, the present invention is not limited to this, and light in other wavelength ranges is used. An optical filter that attenuates a part of the filter may be used. For example, by using an optical filter whose transmittance for blue light in a wavelength range of about 400 nm to 500 nm, green light in a wavelength range of about 500 nm to 600 nm, and infrared light in a wavelength range exceeding 700 nm is periodically different in the x direction. Also good. However, the modulation of light by the optical filter is preferably performed for light having a long wavelength range where the influence of diffraction is small, for example, light having a lower limit of 650 nm or more. When using an optical filter whose transmittance with respect to infrared light having a wavelength exceeding 700 nm is periodically changed, an optical filter having transmission characteristics up to the wavelength range of the infrared light is used instead of the visible light transmitting filter 4. It is necessary to provide it.

  In addition, the transmission characteristics of the optical filters 1a and 1b in the present embodiment are expressed by using a sin function and a cos function, but if they are periodic functions, optical filters having transmission characteristics expressed by other functions are used. There is no problem. An arbitrary periodic function can be expressed by a linear combination of a plurality of sine functions or cos functions having different frequencies by Fourier series expansion. Therefore, any optical filter having any transmission characteristic can be used as long as it can be decomposed into Fourier components and processed. It may be used.

  Furthermore, the transmittance of the optical filters 1a and 1b periodically changes in the x direction. However, the direction of the periodic change need not be the x direction, and may be any direction. The image processing unit 7 can generate depth information based on the pattern of the pixel signal in the direction regardless of the direction of the period change.

  Further, as shown in FIG. 2, the optical filters 1a and 1b may be arranged corresponding to every other row of the pixel array of the image sensor 2, or every n rows (n is an integer of 2 or more). It may be arranged correspondingly. However, in this case, regarding the depth estimation, the accuracy in the vertical direction (y direction) is reduced to 1/2 or 1 / (n + 1). Alternatively, the optical filters 1a and 1b may be arranged every other column or every n columns of the pixel array. In this case, the optical filters 1a and 1b are configured such that the transmittance with respect to light in the first wavelength region periodically changes in the y direction.

  Further, as shown in FIG. 3, optical filters 1a and 1b in which infrared light absorbing portions 1ab_1 and transparent portions 1ab_2 are alternately arranged may be used. In this example, the distribution in the x direction of the transmittance for light in a wavelength region close to infrared light is represented by a step function. That is, the infrared light absorption unit 1ab_1 transmits light in a wavelength range of about 650 nm to 700 nm with a constant transmittance regardless of the x coordinate, and the transparent unit 1ab_2 has a large amount of visible light including the above wavelength range. Make the part transparent. Even in such a configuration, depth information can be obtained based on the same principle as long as the two optical filters 1a and 1b are arranged to be shifted from each other by a quarter period in the x direction. Note that the individual infrared light absorber 1ab_1 and the transparent portion 1ab_2 are provided so as to cover, for example, u × v photosensitive cells 10 each having u rows and v columns (u and v are integers of 2 or more) as one block. It is done. In such a configuration, by processing the pixel signal for each block, it is possible to obtain information on the depth of the portion of the subject corresponding to each block. Such a configuration for processing in units of blocks is not limited to the configuration shown in FIG. 3 but can also be applied to configurations using the optical filters 1a and 1b shown in FIG. 1B and FIG. In that case, it is preferable that the transmittance of the optical filters 1a and 1b corresponding to one block with respect to the light in the first wavelength band changes by one period or more in at least one direction. This is because in order to detect the frequency characteristics of signals detected by the u × v photosensitive cells included in one block, it is preferable that a change of one period or more occurs in one block.

  Note that the transmittances of the individual infrared light absorbing portions 1ab_1 and the transparent portions 1ab_2 are not limited to being expressed by step functions, and the transmittance for light in the first wavelength range may vary depending on the position within the range. Good. In addition, even in a configuration in which the transmittance for light in the first wavelength region is different depending on the position in one block, a block including a plurality of photosensitive cells may be processed as a basic configuration.

  In the present embodiment, the pixel array of the image sensor 2 is a square lattice array, but is not limited to this array, and may be, for example, an oblique array. When the oblique light type array is used, the direction of the array of the photosensitive cells 10 and the direction of the periodic change of the transmittance in the optical filters 1a and 1b intersect. Even in such a case, depth information can be obtained by processing signals output from a plurality of photosensitive cells arranged in the same direction as the direction of periodic change in transmittance.

(Embodiment 2)
Next, an imaging apparatus according to the second embodiment of the present invention will be described. In this embodiment, compared with Embodiment 1, only the structure of the optical filter unit 1 is different. The following description will be focused on differences from the imaging apparatus according to the first embodiment, and description of overlapping items will be omitted.

  FIG. 4A is a plan view showing a part of the image pickup surface of the image pickup device 2 on which the optical filter unit 1 is arranged in the present embodiment. 4B is a cross-sectional view taken along line B-B ′ in FIG. 4A, and FIG. 4C is a cross-sectional view taken along line C-C ′ in FIG. 4A. The optical filter unit 1 in the present embodiment includes optical filters 1a and 1b as in the optical filter unit 1 in the first embodiment, but the structure of the optical filter 1a is different. In the present embodiment, the distance between the optical filter 1 a and the optical filter 1 b is different for each row of the pixel array of the image sensor 2. Their distance is d for odd rows and almost zero for even rows. That is, the optical filter 1a has a structure in which unevenness is repeated in the y direction at intervals of one pixel. The z coordinate of the portion of the optical filter 1a corresponding to the odd row is the same as the z coordinate of the optical filter 1a in the first embodiment, but the portion of the optical filter 1a corresponding to the even row is substantially in close contact with the optical filter 1b. ing.

In the image sensor 2, if the y coordinate of the odd-numbered row is expressed as 2m-1, and the y-coordinate of the even-numbered row is expressed as 2m (m is a natural number), the coordinates (x, 2m-1) and the coordinates (x, 2m) respectively Functions To and Te indicating the spatial change of transmittance with respect to light in one wavelength region are expressed by the following Expressions 5 and 6, respectively. Further, the difference To−Te is expressed by Expression 7.

From the above, the difference processing result of the pixel signals of adjacent two lines is proportional to T ₀ -T _e of the formula 7, by removing the high frequency components by the low pass filter for removing ωo or more frequency components, sin ( The component of (ω−ωo) x can be extracted. Since this process can detect the condensing state of incident light as in the process in the first embodiment, the distance to the subject can also be estimated. Note that, in a general subject, the pixel signals of two adjacent rows are substantially equal, and therefore the image information of the subject can be removed by the difference between the pixel signals of two adjacent rows. As a result, unlike Embodiment 1, depth information can be calculated regardless of the spatial frequency of the subject image. Note that the process of acquiring a normal image without a resolution reduction is the same as the process in the first embodiment, and thus the description thereof is omitted.

  As described above, according to the present embodiment, the distance between the optical filters 1 a and 1 b is different every other row of the pixel array of the image sensor 2, and these distances are set to d or 0. Furthermore, by performing differential processing of the pixel signals of two adjacent rows as signal processing and detecting the low frequency band components, it is possible to identify the condensing state of the incident light and estimate the distance to the subject.

  In this embodiment, the optical filters 1a and 1b that attenuate part of light in the wavelength range of about 650 nm to 700 nm close to infrared light are used. However, the optical filter attenuates part of light in other wavelength ranges. A filter may be used. Further, the transmission characteristics of the optical filters 1a and 1b are not limited to the transmission characteristics expressed by using the sin function and the cos function, and any transmission characteristics of the optical filter can be used as long as they are expressed by a periodic function. Good. Further, in the present embodiment, the distance between the optical filters 1a and 1b is changed every other row, but there is no problem if the distance is changed every n rows (n is an integer of 2 or more). Alternatively, the optical filters 1a and 1b are arranged so that the transmittance for light in a specific wavelength region periodically changes in the y direction, and the distance between the optical filters 1a and 1b every other row or every n rows. May be changed.

(Embodiment 3)
Next, a third embodiment of the present invention will be described. The imaging apparatus according to the present embodiment is different from the imaging apparatus according to the first embodiment in that it further includes a light shielding filter 1c disposed so as to cover the optical filter 1a. Hereinafter, the description will focus on the differences from the first embodiment, and a description of the overlapping items will be omitted.

  FIG. 5A is a plan view illustrating a part of the imaging surface of the imaging device 2 in which the optical filters 1a and 1b and the light shielding filter 1c are arranged. 5B is a cross-sectional view taken along line A-A ′ in FIG. 5A. As shown in FIGS. 5A and 5B, a light shielding filter 1c in which light shielding portions 1c_1 and transparent portions 1c_2 are alternately arranged is arranged so as to cover the optical filter 1a.

  The light shielding portion 1c_1 is formed of a member that absorbs or reflects most of the light. On the other hand, the transparent portion 1c_1 is formed of a transparent member such as glass or air. By providing such a light shielding filter 1c, it is possible to prevent unnecessary light from entering from the surroundings, so that the accuracy of signal processing is improved.

  Since the image signal generation process and the depth information generation process in the present embodiment are the same as those in the first embodiment, detailed description thereof is omitted.

  In this embodiment, each of the light-shielding portion 1c_1 and the transparent portion 1c_2 is square, and has a size corresponding to 25 pixels (= 5 × 5). It is not restricted to such an example. Any shape and size may be used as long as the size of the transparent part 1c_1 in a certain direction is larger than the size of one cycle of the transmittance of the optical filter 1a with respect to the light in the first wavelength region.

  As described above, in the present embodiment, the light shielding filter 1c in which the plurality of light shielding portions 1c_1 and the plurality of transparent portions 1c_2 are alternately arranged is disposed closer to the subject than the optical filter 1a. The light incident from the subject first passes through the transparent portion 1c_2, then passes through the optical filters 1a and 1b and enters the plurality of photosensitive cells 10. In the portion of the optical filter 1a where the light beam transmitted through each transparent portion 1c_2 is incident, the transmittance for the light in the first wavelength region changes by one period or more in the x direction. Can be obtained. In the present embodiment, unnecessary light from the surroundings is blocked by the light shielding portion 1c_1, so that there is a remarkable effect that the accuracy of signal processing is improved as compared with the configuration of the first embodiment.

  In the present embodiment, a configuration different from the configuration shown in FIG. 5B may be adopted as long as the same light shielding effect as the above configuration is obtained. For example, the configuration shown in FIG. 5C may be adopted. In the configuration shown in FIG. 5C, a part of the optical filter 1a is a light shielding part 1a_1 formed from a light shielding member, and the other part is a light transmitting part 1a_2 formed by a member similar to the optical filter 1a of the first embodiment. It is. The positions of the light shielding part 1a_1 and the light transmitting part 1a_2 respectively correspond to the positions of the light shielding part 1c_1 and the transparent part 1c_2 in FIG. 5B, and only their z coordinates are different. Even with such a configuration, the same effect as the configuration shown in FIG. 5B can be obtained.

  In the above embodiment, the optical filter 1b is close to the imaging surface of the imaging device 2, but the optical filter 1b may be separated from the imaging surface. Even if the optical filter 1b is separated from the imaging surface by a predetermined distance, depth information can be obtained on the same principle by correcting the signal processing in consideration of the distance.

  In the above embodiment, the image processing unit 7 incorporated in the imaging device performs image processing. However, another device independent of the imaging device may execute the image processing. For example, a signal that is acquired by the imaging device having the imaging unit 100 in each of the above embodiments is input to another device (image processing device), and a program that defines the signal calculation processing is incorporated in the image processing device. The same effect can be obtained by causing the computer to execute. When causing an external image processing apparatus to perform image processing, the imaging apparatus may not include an image processing unit. In this specification, such an imaging apparatus that itself does not generate depth information but outputs a signal for generating depth information is also referred to as a “depth estimation imaging apparatus”.

  The depth estimation imaging apparatus according to the present invention is effective for all cameras using an imaging element. For example, it can be used for consumer cameras such as digital cameras and digital video cameras, and industrial solid-state surveillance cameras.

DESCRIPTION OF SYMBOLS 1 Optical filter part 1a, 1b Optical filter 1a_1 Light-shielding part 1a_2 Light transmission part 1ab_1 Infrared light absorption part 1ab_2 Transparent part 1c Light-shielding filter 1c_1 Light-shielding part 1c_2 Light-transmission part 2 Solid-state image sensor 3 Lens 4 Visible light transmission filter 5 Signal generation / Receiving unit 6 Element driving unit 7 Image processing unit 8 Interface unit 10 Photosensitive cell 30 Memory 100 Imaging unit 200 Signal processing unit

Claims (13)

An imaging device in which a plurality of photosensitive cells are arranged in a matrix on the imaging surface;
An optical lens arranged to form an image on the imaging surface;
A first optical filter disposed between the optical lens and the image sensor, wherein the transmittance for light in a specific wavelength range is periodically changed in a first direction parallel to the imaging surface;
The transmittance for the light in the specific wavelength range, which is disposed between the first optical filter and the imaging element, periodically changes in the first direction, and the periodic change in the transmittance is A second optical filter that is shifted by a quarter of a period compared to a periodic change in transmittance for light in the specific wavelength region in the first optical filter;
A signal processing unit that processes pixel signals output from the plurality of photosensitive cells;
A depth estimation imaging apparatus comprising:   The depth estimation imaging apparatus according to claim 1, wherein the signal processing unit generates information indicating a depth of a subject based on a spatial frequency characteristic of pixel signals output from the plurality of photosensitive cells. A memory in which information defining a correspondence relationship between the spatial frequency characteristics of the pixel signals output from the plurality of photosensitive cells and the depth of the subject is recorded;
The depth estimation imaging apparatus according to claim 2, wherein the signal processing unit generates information indicating the depth by referring to information defining the correspondence.   A distance between the first optical filter and the second optical filter at a position corresponding to a specific row or column in the array of the plurality of photosensitive cells is a row adjacent to the specific row or column or The depth estimation imaging device according to any one of claims 1 to 3, wherein a distance between the first optical filter and the second optical filter at a position corresponding to a column is different.   The first optical filter and the second optical filter are arranged at positions corresponding to every n rows (n is an integer of 1 or more) or every n columns of the array of the plurality of photosensitive cells. Item 4. The depth estimation imaging apparatus according to any one of Items 1 to 3.   The spatial change in the first direction of the transmittance of the first optical filter and the second optical filter with respect to light in the specific wavelength range is represented by a trigonometric function or a rectangular function. The depth estimation imaging device according to any one of claims 5 to 6. When a and b are positive real numbers that satisfy a + b <1, b−a> 0, and ω ₀ is a positive real number,
A function representing a spatial change in the first direction of the transmittance of the first optical filter with respect to the light in the specific wavelength range is approximately represented by asinω ₀ x + b.
A function representing a spatial change in the first direction of the transmittance of the second optical filter with respect to the light in the specific wavelength range is approximately represented by acosω ₀ x + b.
The depth estimation imaging device according to claim 6. When dividing the plurality of photosensitive cells into a plurality of blocks each including u × v photosensitive cells arranged in u rows and v columns (u and v are integers of 2 or more),
The first optical filter and the second optical filter are arranged so as to cover each block, and the transmittance is changed in the first direction by one period or more on each block.
The depth estimation imaging device according to claim 1.   In each block, the spatial change in the first direction of the transmittance of the first optical filter and the second optical filter with respect to the light in the specific wavelength range is a linear function of a periodic function having a plurality of different periods. The depth estimation imaging device according to any one of claims 1 to 8, represented by a combination.   The depth estimation imaging apparatus according to claim 1, wherein a lower limit of the specific wavelength range is longer than 650 nm. A third optical filter that is disposed closer to the subject than the first optical filter and transmits only visible light;
The specific wavelength range is included in the visible wavelength range,
The depth estimation imaging device according to claim 1.   The signal processing unit is provided with a pixel signal output from each photosensitive cell and the first and second optical filters when it is assumed that the first and second optical filters are not provided. A visible light image is generated by correcting pixel signals output from the plurality of photosensitive cells based on information indicating a ratio with a pixel signal output from each photosensitive cell in a case where The depth estimation imaging device according to any one of 1 to 11. A fourth optical filter that is arranged closer to the subject than the first optical filter, and in which a plurality of light shielding portions and a plurality of transparent portions are alternately arranged;
In the portion of the first optical filter on which the light beam that has passed through each transparent portion is incident, the transmittance for light in the specific wavelength region has changed by one period or more in the first direction.
The depth estimation imaging device according to claim 1.