JP6636663B2 - Imaging device and image generation method - Google Patents

Description

  The present invention relates to an imaging device and an image generation method, and more particularly to a technique effective for improving the performance of an imaging device.

  Digital cameras mounted on smartphones and the like need to be thinner. As a technique for reducing the thickness of a digital camera of this type, for example, there is a technique for obtaining an object image without using a lens (for example, see Patent Document 1).

  In this technology, a special diffraction grating substrate is attached to an image sensor, and the light transmitted through the diffraction grating substrate is obtained from a projection pattern generated on the image sensor to obtain the incident angle of the incident light by an inverse problem operation, thereby obtaining the external environment. This is to obtain an image of the object.

US Patent Application Publication No. 2014/0253781

  In Patent Document 1 described above, the pattern of the diffraction grating formed on the upper surface of the substrate attached to the image sensor is a special grating pattern such as a spiral shape. Then, the image of the object is obtained by solving the inverse problem for reproducing the image from the projection pattern received by the image sensor, but the calculation when solving the inverse problem becomes complicated. There is.

  If the calculation is complicated, there is a problem that the processing time naturally increases and the time required for displaying the photograph also increases. In order to perform arithmetic processing at high speed, a high-performance CPU or the like is used. However, in that case, there is a possibility that the cost and power consumption of the digital camera may increase.

  An object of the present invention is to provide a technique capable of reducing the cost and power consumption of an imaging device by facilitating detection of an incident angle of a light beam transmitted through a lattice substrate.

  The above and other objects and novel features of the present invention will become apparent from the description of the present specification and the accompanying drawings.

  The outline of a typical invention among the inventions disclosed in the present application will be briefly described as follows.

  That is, a typical imaging device has an image sensor, a modulator, and an image processing unit. The image sensor converts an optical image captured by a plurality of pixels arranged in an array on the imaging surface into an image signal and outputs the image signal. The modulator is provided on a light receiving surface of the image sensor, and modulates light intensity. The image processing unit performs image processing on an image signal output from the image sensor.

  The modulator also has a grating substrate, a first grating pattern, and a second grating pattern. The first grating pattern is formed on a first surface of the grating substrate near the light receiving surface of the image sensor. The second lattice pattern is formed on a second surface opposite to the first surface.

  The first grid pattern and the second grid pattern are each composed of a plurality of concentric circles. The modulator modulates the intensity of the light transmitted through the second grating pattern with the first grating pattern and outputs the modulated light to the image sensor.

  In particular, the plurality of concentric circles in the first lattice pattern and the second lattice pattern are respectively formed from a plurality of concentric circles in which the pitch of the concentric circles becomes smaller in inverse proportion to the reference coordinates at the center of the concentric circles. The pitch of the concentric circles becomes finer in inverse proportion to the reference coordinates at the center of the concentric circles.

  The effects obtained by typical aspects of the invention disclosed in the present application will be briefly described as follows.

  (1) The processing time until the acquisition of the object image can be reduced.

  (2) The hardware cost of the imaging device can be reduced.

FIG. 2 is an explanatory diagram illustrating an example of a configuration in the imaging device according to the first embodiment; FIG. 2 is an explanatory diagram illustrating an example of imaging by the imaging device in FIG. 1. 2 is a flowchart illustrating an outline of image processing by an image processing circuit included in the imaging device of FIG. 1. It is explanatory drawing which shows an example of the in-plane shift | offset | difference of the projection image from the front surface of a double-sided lattice board | substrate by oblique incident parallel light to the back surface. It is a schematic diagram explaining generation of a moire fringe and frequency spectrum when the axes of the lattice pattern of the double-sided lattice substrate are aligned. It is explanatory drawing which shows an example of the double-sided lattice board | substrate which shifted the axis | shaft of the grid pattern on the front side and the grid pattern on the back side. FIG. 3 is a schematic diagram illustrating generation of moiré fringes and a frequency spectrum when a lattice pattern is shifted. It is explanatory drawing which shows the calculation result of the spatial frequency spectrum image at the time of irradiating with a total of 10 lights of a normal incidence plane wave and nine other plane waves of different incident angles. It is a bird's-eye view which shows the calculation result of the spatial frequency spectrum image at the time of irradiating with a total of ten lights of a normal incidence plane wave and nine other plane waves of different incident angles. FIG. 4 is an explanatory diagram illustrating an angle formed by light from each point constituting an object with respect to an image sensor. FIG. 9 is an explanatory diagram illustrating an example of a spatial frequency spectrum when the lattice pattern is shifted in the horizontal direction. FIG. 4 is an explanatory diagram illustrating an example of a spatial frequency spectrum when a lattice pattern is shifted in a vertical direction. 13 is an explanatory diagram illustrating an example of a configuration in an imaging device according to Embodiment 3. FIG. 14 is a flowchart illustrating an outline of image processing by an image processing circuit included in the imaging device in FIG. 13. FIG. 11 is an explanatory diagram showing that when an object to be imaged is at a finite distance, the projection of the lattice pattern on the front surface onto the back surface is enlarged more than the lattice pattern. 15 is an explanatory diagram illustrating an example of a configuration in an imaging device according to Embodiment 4. FIG. FIG. 15 is an explanatory diagram showing an example of a configuration in a double-sided lattice substrate according to a fifth embodiment. FIG. 17 is an external view showing an example of a portable information terminal according to a sixth embodiment.

  In the following embodiments, when necessary for the sake of convenience, the description will be made by dividing into a plurality of sections or embodiments, but unless otherwise specified, they are not irrelevant to each other, and one is the other. Of some or all of the above, details, supplementary explanations, and the like.

  Further, in the following embodiments, when referring to the number of elements (including the number, numerical value, amount, range, etc.), a case where it is particularly specified and a case where it is clearly limited to a specific number in principle, etc. Except, the number is not limited to the specific number, and may be more than or less than the specific number.

  Furthermore, in the following embodiments, the constituent elements (including element steps, etc.) are not necessarily essential unless otherwise specified or considered to be essential in principle. Needless to say.

  Similarly, in the following embodiments, when referring to the shapes, positional relationships, and the like of the components, the shapes are substantially the same, unless otherwise specified, and in cases where it is considered that it is not clearly apparent in principle. And the like. This is the same for the above numerical values and ranges.

  In all the drawings for describing the embodiments, the same members are denoted by the same reference numerals in principle, and the repeated description thereof will be omitted.

  Hereinafter, embodiments will be described in detail.

(Embodiment 1)
<Configuration example of imaging device>
FIG. 1 is an explanatory diagram illustrating an example of a configuration of the imaging device 101 according to the first embodiment.

  The imaging apparatus 101 acquires an image of an external object without using a lens for forming an image. As shown in FIG. 1, the imaging apparatus 101 includes a modulator 102, an image sensor 103, and an image processing circuit 106. I have.

  The modulator 102 is fixed in close contact with the light receiving surface of the image sensor 103, and has a configuration in which grating patterns 104 and 105 are formed on a grating substrate 102a. The lattice substrate 102a is made of a transparent material such as glass or plastic.

  In the modulator 102, a lattice pattern 104 serving as a second lattice pattern is formed on the surface of the lattice substrate 102a. The surface of the lattice substrate 102a is the second surface. The grid pattern 104 is formed of concentric grid patterns in which the spacing between the grid patterns, that is, the pitch, becomes smaller in inverse proportion to the radius from the center toward the outside.

  On the back surface of the lattice substrate 102a, that is, on the surface in contact with the light receiving surface of the image sensor 103, a lattice pattern 105 serving as a first lattice pattern is formed. The back surface of the lattice substrate 102a is the first surface.

  Similarly to the lattice pattern 104, the lattice pattern 105 is formed of a concentric lattice pattern in which the pitch of the lattice pattern becomes smaller in inverse proportion to the radius from the center toward the outside.

  The lattice pattern 104 and the lattice pattern 105 are formed by evaporating aluminum or the like by, for example, a sputtering method used in a semiconductor process. Shading is given by the pattern with and without the aluminum deposited.

  The formation of the lattice patterns 104 and 105 is not limited to this, and may be formed by shading, for example, by printing with an inkjet printer or the like.

  The light transmitted through the grating patterns 104 and 105 has its light intensity modulated by the grating patterns. The transmitted light is received by the image sensor 103. The image sensor 103 includes, for example, a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.

  On the surface of the image sensor 103, pixels 103a serving as light receiving elements are regularly arranged in a grid pattern. The image sensor 103 converts a light image received by the pixel 103a into an image signal which is an electric signal. The image signal output from the image sensor 103 is subjected to image processing by an image processing circuit 106 as an image processing unit, and is output to a monitor display 107 or the like.

<Imaging example of imaging device>
FIG. 2 is an explanatory diagram illustrating an example of imaging by the imaging device 101 in FIG. FIG. 2 shows an example in which the subject 301 is photographed by the imaging device 101 and is displayed on the monitor display 107.

  As shown in the figure, when photographing the subject 301, one surface of the modulator 102, specifically, the surface of the lattice substrate 102a on which the lattice pattern 104 is formed faces the subject 301. Shooting is performed.

<Image processing example of image processing circuit>
Subsequently, an outline of image processing by the image processing circuit 106 will be described.

  FIG. 3 is a flowchart illustrating an outline of image processing by the image processing circuit 106 included in the imaging apparatus 101 in FIG.

  First, the moire fringe image output from the image sensor 103 is subjected to a two-dimensional Fourier transform (FFT: Fast Fourier Transform) operation for each of RGB (Red Green Blue) components to obtain a frequency spectrum (step S101). ).

  Then, after extracting data of one side frequency of the frequency spectrum by the processing of step S101 (step S102), an image is obtained by calculating the intensity of the frequency spectrum (step S103).

  Then, noise removal processing is performed on the obtained image (step S104), and subsequently, contrast enhancement processing (step S105) and the like are performed. Thereafter, the color balance of the image is adjusted (step S106), and the image is output as a captured image.

  Thus, the image processing by the image processing circuit 106 ends.

<Principle of imaging device imaging>
Next, the imaging principle of the imaging device 101 will be described.

  First, the concentric lattice patterns 104 and 105 whose pitches become smaller in inverse proportion to the radius from the center are defined as follows. It is assumed that a laser interferometer or the like causes a spherical wave close to a plane wave to interfere with a plane wave used as reference light.

  When the radius from the reference coordinate, which is the center of the concentric circle, is r, and the phase of the spherical wave there is φ (r), this is determined using a coefficient β that determines the magnitude of the wavefront bend.

Can be expressed as

  In spite of the spherical wave, it is represented by the square of the radius r because it is a spherical wave close to a plane wave and can be approximated only by the lowest order of expansion. When a plane wave interferes with light having this phase distribution,

The intensity distribution of interference fringes as shown in FIG.

  this is,

Concentric stripes with a bright line at the radial position satisfying If the pitch of the stripe is p,

It can be seen that the pitch decreases in inverse proportion to the radius.

  Such stripes are called Fresnel zone plates. A grid pattern having a transmittance distribution proportional to the intensity distribution defined in this way is used as the grid patterns 104 and 105 shown in FIG.

  As shown in FIG. 4, it is assumed that parallel light enters the modulator 102 having a thickness t having such a lattice pattern on both sides at an angle θ0. In terms of geometrical optics, when the refraction angle in the modulator 102 is θ, the light multiplied by the transmittance of the front surface grating is incident on the back surface with a shift of δ = t · tan θ, and the center of the two concentric gratings is temporarily If they are formed to be aligned, the transmittance of the lattice on the back surface is shifted by δ and multiplied.

  At this time,

Is obtained.

  It can be seen that the fourth term of this expansion formula creates a stripe pattern at equal intervals straight in the direction of the displacement of the two grids over the overlapping region. Such a fringe generated at a relatively low spatial frequency by the superposition of the fringes is called a Moire fringe.

  Such straight, regularly spaced fringes produce sharp peaks in the spatial frequency distribution obtained by two-dimensional Fourier transform of the detected image. From the frequency value, the value of δ, that is, the incident angle θ of the light beam can be obtained.

  It is clear that such moiré fringes uniformly obtained over the entire surface are generated at the same pitch irrespective of the direction of displacement due to the symmetry of the concentric lattice arrangement. Such stripes are obtained because the grating pattern is formed by the Fresnel zone plate, and it is considered impossible to obtain uniform stripes over the entire surface with other grating patterns.

  In the second term, it can be seen that fringes in which the intensity of the Fresnel zone plate is modulated by Moiré fringes are generated. However, since the frequency spectrum of the product of the two fringes is a convolution of each Fourier spectrum, a sharp peak is obtained. Absent.

  From equation (5), only the components having sharp peaks

And its Fourier spectrum is

become that way.

  Here, F represents a Fourier transform operation, u and v are spatial frequency coordinates in the x and y directions, and δ with parentheses is a delta function. From this result, it can be seen that in the spatial frequency spectrum of the detected image, the peak of the spatial frequency of the moire fringes occurs at the position of u = ± δβ / π.

  This is shown in FIG. In FIG. 5, from the left to the right, a schematic diagram of the arrangement of the light beam and the modulator 102, a moiré fringe, and a spatial frequency spectrum are shown. 5A shows a case where a light beam is incident at an angle θ from the left side, and FIG. 5B shows a case where a light beam is incident at an angle θ from the right side. .

  The axes of the grating pattern 104 formed on the front side of the modulator 102 and the grating pattern 105 formed on the back side of the modulator 102 are aligned. In FIG. 5A, since the shadows of the grid pattern 104 and the grid pattern 105 match, no moire fringes occur.

  In FIG. 5B and FIG. 5C, the same moire occurs because the displacement between the grid pattern 104 and the grid pattern 105 is equal, and the peak positions of the spatial frequency spectrum also match. It is impossible to determine whether the incident angle is the case of FIG. 5B or FIG. 5C.

  In order to avoid this, the two grating patterns 104 and 105 are preliminarily illuminated as shown in FIG. It is necessary to keep the offset relative to the axis.

  Assuming that the relative shift between the shadows of the two gratings with respect to the on-axis normal plane wave is δ0, the shift δ caused by the plane wave at the incident angle θ is

Can be expressed as

  At this time, the peak of the spatial frequency spectrum of the moiré fringe of the light beam at the incident angle θ is on the plus side of the frequency.

Position.

  Assuming that the size of the image sensor is S and the number of pixels in the x direction and the y direction of the image sensor is N, the spatial frequency spectrum of the discrete image by the fast Fourier transform (FFT) is from −N / (2S) to + N / ( 2S).

  From this, considering that the incident angle on the plus side and the incident angle on the minus side are equally received, the spectral peak position of the moiré fringe due to the vertically incident plane wave (θ = 0) is the origin (DC: DC component) position. And, for example, the center position between the frequency position on the + side end, that is,

It is appropriate to use the spatial frequency position of

  Therefore, the relative center displacement of the two gratings is

Is appropriate.

  FIG. 7 is a schematic diagram illustrating the generation of moiré fringes and the frequency spectrum when the lattice patterns 104 and 105 are shifted from each other.

  As in FIG. 5, the left side shows the arrangement of the light beam and the modulator 102, the central row shows the moire fringes, and the right side shows the spatial frequency spectrum. 7A shows a case where the light beam is vertically incident, FIG. 7B shows a case where the light beam is incident at an angle θ from the left side, and FIG. This is a case where light is incident at θ.

  The grid pattern 104 and the grid pattern 105 are previously shifted by δ0. Therefore, moiré fringes also occur in FIG. 7A, and a peak appears in the spatial frequency spectrum.

  As described above, the shift amount δ0 is set such that the peak position appears at the center of the spectrum range on one side from the origin. At this time, in FIG. 7 (b), the deviation δ is in a direction in which the deviation is further increased, and in FIG. 7 (c), the deviation is in a direction in which the deviation is smaller. Can be determined from the peak position of the spectrum.

  The spectrum image of this peak is a luminescent spot indicating a light beam at infinity, and is nothing but an image captured by the imaging device 101 in FIG.

  When the maximum angle of incidence of parallel light that can be received is θmax,

Therefore, the maximum angle of view that can be received by the imaging device 101 is

Given by

  By analogy with image formation using a general lens, considering that parallel light having an angle of view θmax is focused at the end of the image sensor and received, the effective focal length of the imaging device 101 without a lens is:

Can be considered as equivalent to

  As shown in equation (2), it is assumed that the transmittance distribution of the grating pattern basically has a sinusoidal characteristic, but such a component is considered as a fundamental frequency component of the grating pattern. For example, it is conceivable to binarize the transmittance of the grating pattern, change the duty of the grating region having a high transmittance and the duty of the region having a low transmittance, and increase the width of the region having a high transmittance to increase the transmittance.

  In the above description, the incident light beam has only one incident angle at the same time. However, in order for the imaging device 101 to actually act as a camera, it is assumed that light beams at a plurality of incident angles are incident simultaneously. There must be.

  The light of such a plurality of incident angles already overlaps the images of the plurality of front side gratings at the time of being incident on the back side grating pattern. If they generate moire fringes mutually, there is a concern that noise may be generated that hinders detection of moire fringes with the lattice pattern 105 as a signal component.

  However, actually, the overlap of the images of the lattice pattern 104 does not cause a peak of the moire image, and the peak occurs only at the overlap with the lattice pattern 105 on the back side.

  The reason will be described below.

  First, a significant difference is that the overlapping of the shadows of the lattice pattern 104 on the front side by the light beams at a plurality of incident angles is not a product but a sum. In the overlap between the shadow of the grid pattern 104 and the grid pattern 105 due to the light at one incident angle, the intensity distribution of the light that is the shadow of the grid pattern 104 is multiplied by the transmittance of the grid pattern 105 to obtain the grid on the back side. The light intensity distribution after passing through the pattern 105 is obtained.

  On the other hand, the overlap of the shadows caused by the lights having different angles incident on the lattice pattern 104 on the front surface side is not the product but the sum, because the lights are overlapped. In the case of sum,

The distribution is obtained by multiplying the distribution of the grating of the original Fresnel zone plate by the distribution of the moire fringes.

  Therefore, the frequency spectrum is represented by the overlap integral of each frequency spectrum. Therefore, even if the moire spectrum has a sharp peak by itself, a ghost of the frequency spectrum of the Fresnel zone plate is actually generated at that position. That is, no sharp peak occurs in the spectrum.

  Therefore, the spectrum of the moiré image detected even when light having a plurality of incident angles is applied is always only the moiré of the product of the lattice pattern 104 on the front side and the lattice pattern 105 on the back side, and the single lattice pattern 105 Therefore, only one peak of the spectrum is detected for one incident angle.

<Confirmation of shooting principle>
Hereinafter, the results of a simulation performed to confirm the principle are shown in FIGS. 8 and 9.

  FIG. 8 is an explanatory diagram showing a calculation result of a spatial frequency spectrum image when illuminated with a total of 10 lights of a vertical incident plane wave and nine other plane waves having different incident angles. FIG. 9 is a bird's-eye view showing a calculation result of a spatial frequency spectrum image when illuminated with a total of 10 lights of a normal incidence plane wave and nine other plane waves having different incident angles.

  In each case, the sensor size of the image sensor 103 is 20 mm □, the viewing angle θmax = ± 70 °, the incident-side and exit-side lattice coefficients β = 50 (rad / mm2), δ0 = 0.8 mm, the number of pixels is 1024 × 1024, and the modulator 102 is used. When the substrate thickness is 1 mm and the substrate refractive index is 1.5, a vertical incident plane wave, incident light of θx = 50 ° and θy = 30 °, incident light of θx = −30 ° and θy = 70 °, incident light of θx = 10 °, θy = −20 °, incident light of θx = 20 °, θy = 30 °, incident light of θx = 30 °, θy = −40 °, θx = −10 °, incident light at θy = 40 °, incident light at θx = −20 °, θy = −30 °, incident light at θx = −30 °, θy = 0 °, and θx = 40 °, θy = 50 ° It is a spectrum when a total of 10 plane waves of incident light and incident light are incident.

  FIG. 8 is a black-and-white inverted image of the spectrum image, and FIG. 9 is a bird's-eye view showing the luminance of the spectrum image. The original moiré image itself is omitted because it has a fine lattice pitch and is invisible even when displayed as a drawing in this specification.

  In the drawing, the center represents the DC component, and the periphery represents the entire spatial frequency spectrum region of ± N / 2S. Since the DC component has a large value, it is masked and removed, and only the peak component to be detected is displayed. Furthermore, the contrast is emphasized because the peak width of the spectrum is narrow and it is difficult to recognize the spectrum as it is.

  In FIG. 8, the positions of the signal peaks are indicated by circles. In the bird's-eye view of FIG. 9, since the drawing line cannot be displayed without passing through the peak as it is, the result obtained by applying an averaging filter of the mesh size is displayed.

  In each case, basically, 10 peaks can be detected as a total of 20 peaks on both the positive and negative sides of the origin. In this case, the pitch of the outermost periphery of the lattice pattern was about 6 μm, and the effective focal length was about 12.4 mm.

  Here, the correspondence between parallel light, which has been described so far, and light from an actual object will be schematically described with reference to FIG.

  FIG. 10 is an explanatory diagram for explaining an angle formed by light from each point constituting the object with respect to the image sensor.

  Light from each point constituting the subject 301 is strictly a spherical wave from a point light source, and is transmitted to the modulator 102 and the image sensor 103 (hereinafter, referred to as a grating sensor integrated substrate in FIG. 10) of the imaging device 101 in FIG. Incident.

  At this time, when the grid sensor integrated substrate is sufficiently small or sufficiently far from the subject 301, it can be considered that the incident angle of light illuminating the grid sensor integrated substrate is the same from each point.

  From the relationship that the spatial frequency displacement Δu of the moire with respect to the minute angular displacement Δθ obtained from the equation (9) is 1 / S or less, which is the minimum resolution of the spatial frequency of the image sensor, the condition that Δθ can be regarded as parallel light is

Can be expressed as

  From this, if Δθ <0.18 °, this is a condition that can be realized at a distance of 6 m from the subject if the sensor size is 20 mm.

  From the analogy of the above results, it can be seen that the imaging device of the present invention can form an image at an object at infinity.

  As described above, an object image of the outside world can be obtained by a simple operation such as fast Fourier transform (FFT). This makes it possible to reduce the processing time until the acquisition of the object image.

  Further, since a high-performance arithmetic processing device is not required, hardware cost of the imaging device 101 can be reduced. Further, power consumption of the imaging device 101 can be reduced by shortening the processing time of the calculation. Can be.

(Embodiment 2)
<Overview>
In the first embodiment, the image output from the imaging device 101 is vertically long. However, in the second embodiment, a case where the output image is horizontally long will be described.

  In the first embodiment, as described above, the grid pattern 104 and the grid pattern 105 are formed so as to be shifted in the x (horizontal) direction of the image sensor 103. In other words, the rectangular image output from the image processing circuit 106 is formed so as to be shifted in the long side direction.

<Example of forming lattice pattern>
FIG. 11 is an explanatory diagram showing an example of the spatial frequency spectrum when the lattice patterns 104 and 105 are shifted in the horizontal direction.

  At this time, the shape of the image sensor 103 is a square, and the pixel pitch is the same in both the x and y directions. In this case, as shown on the right side of FIG. 11, the spatial frequency spectrum of the output of the image sensor is such that the image is reproduced left and right separately within the frequency range of both x and y ± N / S. Become.

  However, in the example shown in FIG. 11, the image is basically limited to a vertically long area. Generally, an image obtained by a digital camera or the like is a horizontally long rectangle having an aspect ratio of 3: 2 or 4: 3, for example. Therefore, as an arrangement of the lattice patterns 104 and 105 suitable for a horizontally long rectangle, for example, the arrangement shown in FIG. 12 is desirable.

  FIG. 12 is an explanatory diagram showing an example of a spatial frequency spectrum when the lattice patterns 104 and 105 are shifted in the vertical direction.

  As shown in FIG. 12, the lattice patterns 104 and 105 are formed so as to be shifted in the vertical direction of the image sensor, that is, in the y direction of the image sensor. In other words, the grid pattern 104 and the grid pattern 105 are formed so as to be shifted in the short side direction of the rectangular image output from the image processing circuit 106. As a result, the image formed in the spatial frequency space of the image sensor output is vertically separated as shown on the right side of FIG.

  As described above, the image output from the imaging device 101 can be made horizontally long. As a result, an image can be obtained in the same manner as a general digital camera, so that the versatility of the imaging apparatus 101 can be improved.

(Embodiment 3)
<Overview>
In the modulators 102 according to the first and second embodiments, the grating pattern 104 and the grating pattern 105 having the same shape are formed on the front surface and the back surface of the grating substrate 102a so as to be shifted from each other. The image was detected from the spatial frequency spectrum of the image.

  The lattice pattern 105 on the back side is an optical element that modulates the intensity of light incident on the image sensor 103 in close contact therewith. Therefore, by setting the sensitivity of the image sensor effectively in consideration of the transmittance of the lattice pattern 105 on the rear surface side, moire can be virtually generated in the processed image.

<Configuration example of imaging device>
FIG. 13 is an explanatory diagram illustrating an example of a configuration of the imaging device 101 according to the third embodiment.

  The imaging device 101 of FIG. 13 differs from the imaging device 101 of FIG. 1 of the first embodiment in that the lattice pattern 105 shown in FIG. 1 is not formed on the back side of the lattice substrate 102a. Other configurations are the same as those in FIG.

  With the configuration of FIG. 13, the number of grid patterns formed on the grid substrate 102a can be reduced by one. Thereby, the manufacturing cost of the modulator 102 can be reduced.

  However, in this case, the pitch of the pixels 103a included in the image sensor 103 may be fine enough to sufficiently reproduce the pitch of the grid pattern, or coarse enough so that the pitch of the grid pattern can be reproduced at the pixel pitch of the image sensor 103. is necessary.

  When the grid pattern is formed on both sides of the grid substrate 102a, the pitch of the grid pattern does not necessarily need to be resolvable by the pixels 103a of the image sensor 103, and only the moiré image needs to be resolvable. Therefore, the pitch of the lattice pattern can be determined independently of the pixel pitch.

  However, when a grid pattern is reproduced by the image sensor 103, the resolution of the grid pattern and the resolution of the image sensor 103 need to be equal. Therefore, the image processing circuit 106 is provided with an intensity modulation circuit 106c corresponding to the back side grid pattern 105 (FIG. 1) for generating moire for the output image of the image sensor 103.

<Image processing example of image processing circuit>
FIG. 14 is a flowchart illustrating an outline of image processing by the image processing circuit 106 included in the imaging apparatus 101 in FIG.

  14 differs from the flowchart of FIG. 3 of the first embodiment in the processing of step S201. In the process of step S201, the above-described intensity modulation circuit 106c generates a moiré fringe image corresponding to the lattice pattern on the back side with respect to the image output from the image sensor 103.

  Thereafter, the processing of steps S202 to S208 in FIG. 14 is the same as the processing of steps S101 to S107 in FIG. 3 of the first embodiment, and a description thereof will not be repeated.

  As described above, by providing the intensity modulation circuit 106c, the same effect as changing the lattice pattern 105 (FIG. 1) on the back side can be obtained, and the detection light does not necessarily have to be parallel light. It is possible.

<About focusing>
FIG. 15 is an explanatory diagram showing that when the object to be imaged is at a finite distance, the projection of the lattice pattern 104 on the front surface onto the back surface is enlarged more than the lattice pattern 104.

  As shown in FIG. 15, when a spherical wave from a point 1301 constituting the object irradiates the lattice pattern 104 on the surface side and its shadow 1302 is projected on the lower surface, the image projected on the lower surface is , Are expanded almost uniformly.

  Therefore, if the transmittance distribution of the lattice pattern on the back side (corresponding to the lattice pattern 105 in FIG. 1) designed for parallel light is multiplied as it is, linear moire fringes at equal intervals will not occur. However, if the grid on the lower surface is enlarged in accordance with the uniformly enlarged shadow of the grid pattern 104 on the front surface side, equally-spaced linear Moire fringes are generated again for the enlarged shadow 1302. be able to.

  Thus, light from the point 1301 at a distance that is not necessarily infinity can be selectively reproduced. Thereby, focusing can be performed, and it is possible to perform imaging by focusing on an arbitrary position instead of imaging at infinity as described in the first embodiment.

  As described above, the convenience of the imaging device 101 can be improved.

(Embodiment 4)
In the fourth embodiment, a technique for changing the lattice pattern 104 on the front surface side in FIG. 1 will be described.

<Configuration example and operation example of imaging device>
FIG. 16 is an explanatory diagram illustrating an example of a configuration of the imaging device 101 according to the fourth embodiment.

  The imaging device 101 of FIG. 16 differs from the imaging device 101 of FIG. 1 of the first embodiment in that a liquid crystal unit 108 is newly provided in the modulator 102 and a focus position designation input unit 109 is newly added. It has just been provided. Note that the configuration of the image sensor 103 in FIG. 16 is the same as that in FIG.

  The liquid crystal unit 108 has a configuration in which a liquid crystal layer (not shown) is provided on a glass substrate (not shown) on which a transparent electrode or the like is formed, for example. The liquid crystal layer is formed so as to be sandwiched between the glass substrate and the lattice substrate 102a. ing.

  An arbitrary lattice pattern 1403 is displayed on the liquid crystal layer, and the lattice pattern 1403 functions as the lattice pattern 104 on the front side. A grating pattern 105 is formed on the back side of the grating substrate 102a in the modulator 102, as in FIG.

  The focus position designation input unit 109 is an input unit for setting a focus position which is information such as a distance to a subject, and is connected to the image processing circuit 106. Further, the image processing circuit 106 is provided with a liquid crystal drive circuit 106a and a lattice pattern generation circuit 106b.

  The lattice pattern generation circuit 106b generates a lattice pattern optimal for focusing based on the focus position input from the focus position designation input unit 109. The liquid crystal driving circuit 106a controls the display by applying a voltage to the transparent electrode formed on the glass substrate so that the grid pattern generated by the grid pattern generating circuit 106b is displayed on the liquid crystal layer of the liquid crystal unit 108.

  Basically, light from a point 1301 at a finite distance closer to infinity than the infinity is divergent light. Therefore, in order to have the same size on the back side as the grid pattern 105, the front side grid pattern 104 must be It is only necessary to display the pattern slightly smaller than the pattern 105.

  As described above, higher-speed focusing can be performed.

(Embodiment 5)
In the fifth embodiment, another example of the front-side lattice pattern formed on the double-sided lattice substrate will be described.

<Example of forming lattice pattern>
FIG. 17 is an explanatory diagram showing an example of the configuration of the modulator 102 according to the fifth embodiment.

  In the first embodiment, the grating pattern 104 (FIG. 1) of the modulator 102 is formed by, for example, printing or sputtering. However, the modulator 102 of FIG. 17 includes a cylindrical lens 110. The lattice pattern 105 on the back side of the lattice substrate 102a is the same as that in FIG. 1 of the first embodiment.

  In this case, the cylindrical lenses 110 are arranged on the surface of the grating substrate 102a of the modulator 102 so as to have a pattern similar to the grating pattern 104 of FIG. The cylindrical lens 110 is a lens whose surface is formed by a cylindrical surface, has a curvature of a convex lens in the vertical direction, and has no curvature in the horizontal direction.

  As described above, by forming the lattice pattern with the cylindrical lens 110, the loss of the light amount can be significantly reduced. For example, as described in the first embodiment, in a grid pattern with shading by a printing pattern or the like, the printed portion of the grid pattern blocks light, and the light amount is greatly reduced. become.

  On the other hand, in the case of the cylindrical lens 110, the light use efficiency can be improved without blocking the light.

  As described above, the S / N ratio (Signal-to-Noise ratio) of the imaging apparatus 101 can be increased, and thus the drawing performance can be improved.

(Embodiment 6)
<Configuration example of portable information terminal>
In the sixth embodiment, a portable information terminal configured using the imaging device 101 according to the fifth embodiment will be described.

  FIG. 18 is an external view showing an example of the portable information terminal 200 according to the sixth embodiment.

  The portable information terminal 200 is, for example, a smartphone or the like. Note that the portable information terminal 200 is not limited to a smartphone, but may be any portable terminal having a built-in camera such as a tablet.

  An imaging device 101 is built in the portable information terminal 200. An opening window 202 is provided on the back surface of the portable information terminal 200, and the modulator 102 of FIG. 16 is provided inside the portable information terminal 200 so as to be close to the opening window 202.

  On one long side of the portable information terminal 200, a focus adjustment knob 201 is provided. The knob 201 corresponds to the focus position designation input unit 109 according to the fourth embodiment.

  The focus position is set by turning the knob 201, and an arbitrary lattice pattern 1403 is displayed on the liquid crystal layer of the liquid crystal unit 108 in FIG. 16 according to the set focus position. As a result, an image of an object at an arbitrary distance can be captured.

  The imaging device 101 can increase the effective focal length according to the expression (14) shown in the first embodiment. Therefore, the aperture can be enlarged while the imaging device 101 is kept thin.

  In the case of a digital camera for a smartphone using a general lens, the aperture of the lens must be reduced in order to reduce the thickness of the portable information device. As a result, the focal length becomes short, and the image is not sufficiently blurred.

On the other hand, in the imaging device 101, since the opening can be enlarged as described above, a beautiful blur can be obtained. As described above, the portable information terminal 200 with high delineation performance can be realized.

  As described above, the invention made by the inventor has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and various modifications can be made without departing from the gist of the invention. Needless to say.

  Note that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described above.

  In addition, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of one embodiment can be added to the configuration of another embodiment. . In addition, it is possible to add, delete, or replace another configuration with respect to a part of the configuration of each embodiment.

101 Imaging device 102 Modulator 102a Lattice substrate 103 Image sensor 103a Pixel 104 Lattice pattern 105 Lattice pattern 106 Image processing circuit 106a Liquid crystal drive circuit 106b Lattice pattern generation circuit 106c Intensity modulation circuit 107 Monitor display 108 Liquid crystal unit 109 Focus position designation input unit 110 Cylindrical lens 200 Portable information terminal 201 Knob 202 Opening window

Claims (19)

An image sensor that converts light received by the plurality of pixels arranged on the light receiving surface into an image signal and outputs the image signal;
A modulator arranged on the light receiving surface side of the image sensor and modulating light,
An image processing unit that performs image processing on an image signal output from the image sensor;
With
The modulator has two concentric patterns disposed opposite to each other, and modulates incident light with the two concentric patterns.
The imaging device, wherein the image sensor receives light modulated by the modulator. The imaging device according to claim 1,
An imaging apparatus, wherein a pitch of concentric circles in each of the two concentric patterns becomes narrower outward from a center of the concentric circles. The imaging device according to claim 1,
The imaging device, wherein the image processing unit calculates a frequency spectrum by performing a two-dimensional Fourier transform on an image signal output from the image sensor. The imaging device according to claim 2,
An imaging apparatus, wherein the center positions of the concentric circles in the two concentric patterns are shifted from each other. The imaging device according to claim 4,
An imaging device, wherein the center positions of the concentric circles in the two concentric patterns are shifted from each other in a short side direction of an image output from the image processing unit. The imaging device according to claim 1,
An imaging device, wherein one of the two concentric patterns is formed by a cylindrical lens. The imaging device according to claim 1,
The imaging device, wherein the modulator has a transparent substrate, and the concentric pattern is formed on both surfaces of the transparent substrate. In an imaging device that generates an image of a subject,
It has a first concentric pattern and a second concentric pattern arranged opposite to each other, and modulates the light of the subject with the first concentric pattern and the second concentric pattern to form moiré. A modulator for forming an optical image including fringes;
An image sensor that receives an optical image including the moiré fringe formed by the modulator, converts the optical image into an image signal, and outputs the image signal.
An image processing unit that performs image processing including Fourier transform on an optical image including the moiré fringes output from the image sensor to generate an image of the subject,
An imaging device comprising: The imaging device according to claim 8,
An imaging apparatus, wherein a pitch of concentric circles in each of the first concentric pattern and the second concentric pattern decreases from the center of the concentric circle toward the outside. The imaging device according to claim 8,
An imaging device, wherein a center position of a concentric circle in the first concentric pattern is shifted from a center position of a concentric circle in the second concentric pattern. The imaging device according to claim 10,
An imaging apparatus, wherein a center position of a concentric circle in the first concentric pattern and a center position of a concentric circle in the second concentric pattern are shifted in a short side direction of an image output from the image processing unit. The imaging device according to claim 8,
An imaging device, wherein the first concentric pattern is located on the image sensor side, the second concentric pattern is located on the subject side, and the second concentric pattern is formed by a cylindrical lens. . The imaging device according to claim 8,
The modulator has a transparent substrate, and the first concentric pattern is formed on a surface of the transparent substrate facing the light receiving surface of the image sensor, and the second concentric pattern is formed on an opposite surface. Imaging device. An image generation method for generating an image of a subject using an image sensor,
A modulator having a first concentric pattern and a second concentric pattern facing each other modulates light of the subject to form an optical image having moiré fringes,
The image sensor receives an optical image having the moiré fringes formed by the modulator, converts the optical image into an image signal, and outputs the image signal.
An image generating method, wherein an image processing unit performs signal processing including Fourier transform on an image signal of the optical image having the moiré fringe output from the image sensor and generates an image of the subject using the signal processing. The image generation method according to claim 14,
An image generation method, wherein a pitch of concentric circles in each of the first concentric pattern and the second concentric pattern becomes narrower outward from a center of the concentric circle. The image generation method according to claim 14,
An image generation method, wherein a center position of a concentric circle in the first concentric pattern is shifted from a center position of a concentric circle in the second concentric pattern. The image generation method according to claim 14,
An image generation method, wherein a center position of a concentric circle in the first concentric pattern and a center position of a concentric circle in the second concentric pattern are shifted in a short side direction of an image output from the image processing unit. The image generation method according to claim 14,
The image generating method, wherein the second concentric pattern is formed by a cylindrical lens. The image generation method according to claim 14,
The first concentric pattern is located on the image sensor side, the second concentric pattern is located on the subject side, and the second concentric pattern is formed by a cylindrical lens; Method.