JP2020065265A - Imaging apparatus - Google Patents

Abstract

To reduce cost and power consumption of an imaging apparatus, by facilitating detection of the incident angle of a beam of light permeating a lattice substrate.SOLUTION: An imaging apparatus 101 has an image sensor 103 for converting an optical image taken in by multiple pixels arranged in an array on the imaging surface, into an image signal and outputting, a modulator 102 provided on a light-receiving face of the image sensor 103, and modulating the intensity of light, and an image processing circuit 106 performing image processing of an image signal outputted from the image sensor 103. The modulator 102 has a lattice substrate 102a, a lattice pattern 105 formed on the back side of the lattice substrate 102a close to the light-receiving face of the image sensor 103, and a lattice pattern 104 formed on a surface facing the back side. The lattice patterns 104, 105 are constituted of multiple concentric circles, respectively. The modulator 102 performs intensity modulation of the light transmitting the lattice pattern 104 by the lattice pattern 105 before outputting it to the image sensor 103.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image pickup apparatus and an image generation method, and more particularly to a technique effective for improving the performance of the image pickup apparatus.

  A digital camera mounted on a smartphone or the like needs to be thin. As a technology for reducing the thickness of this type of digital camera, for example, there is one that obtains an object image without using a lens (see, for example, Patent Document 1).

  In this technique, a special diffraction grating substrate is attached to the image sensor, and the incident angle of the incident light is obtained by an inverse problem operation from the projection pattern of the light transmitted through the diffraction grating substrate generated on the image sensor. It is an image of an object.

U.S. Patent Application Publication No. 2014/0253781

  In Patent Document 1 described above, the pattern of the diffraction grating formed on the upper surface of the substrate attached to the image sensor is a special grating pattern such as a spiral shape. Then, the image of the object is obtained by solving the inverse problem for reproducing the image from the projection pattern received by the image sensor, but the calculation when solving this inverse problem becomes complicated. There is.

  If the calculation is complicated, there is a problem that the processing time naturally becomes long and the time until the photograph is displayed becomes long. A high-performance CPU or the like is used to perform the arithmetic processing at high speed, but in that case, there is a possibility that the cost and power consumption of the digital camera increase.

  An object of the present invention is to provide a technique capable of reducing the cost and power consumption of an image pickup device by facilitating the detection of the incident angle of a light beam that passes through a grating substrate.

  The above and other objects and novel features of the present invention will be apparent from the description of the present specification and the accompanying drawings.

  The following is a brief description of the outline of the typical invention disclosed in the present application.

  That is, a typical imaging device has an image sensor, a modulator, and an image processing unit. The image sensor converts an optical image captured by a plurality of pixels arranged in an array on the imaging surface into an image signal and outputs the image signal. The modulator is provided on the light receiving surface of the image sensor and modulates the intensity of light. The image processing unit performs image processing of the image signal output from the image sensor.

  The modulator also has a grating substrate, a first grating pattern, and a second grating pattern. The first grid pattern is formed on the first surface of the grid substrate adjacent to the light receiving surface of the image sensor. The second lattice pattern is formed on the second surface facing the first surface.

  The first lattice pattern and the second lattice pattern are each composed of a plurality of concentric circles. The modulator intensity-modulates the light transmitted through the second grating pattern with the first grating pattern and outputs the light to the image sensor.

  In particular, the plurality of concentric circles in the first lattice pattern and the second lattice pattern are each formed of a plurality of concentric circles in which the pitch of the concentric circles becomes fine in inverse proportion to the reference coordinate that is the center of the concentric circles. The pitch of the plurality of concentric circles becomes fine in inverse proportion to the pitch of the concentric circles with respect to the reference coordinate that is the center of the concentric circles.

  The effects obtained by the typical ones of the inventions disclosed in the present application will be briefly described as follows.

  (1) The processing time until the acquisition of the object image can be shortened.

  (2) The hardware cost of the imaging device can be reduced.

FIG. 3 is an explanatory diagram showing an example of a configuration of the image pickup apparatus according to the first embodiment. It is explanatory drawing which shows an example of imaging | photography by the imaging device of FIG. 3 is a flowchart showing an outline of image processing by an image processing circuit included in the imaging device of FIG. 1. It is explanatory drawing which shows an example of the in-plane displacement of the projected image from the front surface to the back surface of the double-sided grating substrate by obliquely incident parallel light. It is a schematic diagram explaining generation | occurrence | production of a moire fringe and frequency spectrum when the axis of the lattice pattern of a double-sided lattice board | substrate is aligned. It is an explanatory view showing an example of a double-sided grid substrate formed by shifting the axes of the grid pattern on the front surface side and the grid pattern on the back surface side. It is a schematic diagram explaining generation | occurrence | production of a moire fringe and a frequency spectrum when it arrange | positions by shifting a lattice pattern. It is explanatory drawing which shows the calculation result of the spatial frequency spectrum image when it irradiates with the light of a total of 10 light of a normal incidence plane wave and the other 9 plane waves of different incident angles. It is a bird's-eye view which shows the calculation result of the spatial frequency spectrum image when irradiating with a total of 10 light of a normal incidence plane wave and the other 9 plane waves of different incident angles. It is explanatory drawing explaining the angle which the light from each point which comprises an object makes with respect to an image sensor. It is explanatory drawing which shows an example of the spatial frequency spectrum when it shifts to a grid pattern horizontal direction. It is an explanatory view showing an example of a spatial frequency spectrum when a lattice pattern is shifted in the vertical direction. FIG. 16 is an explanatory diagram showing an example of a configuration of an image pickup apparatus according to a third embodiment. 14 is a flowchart showing an outline of image processing by an image processing circuit included in the imaging device of FIG. 13. It is explanatory drawing which shows that the projection to the back surface of the grating | lattice pattern by the side of a surface is expanded rather than this grating | lattice pattern, when the object to image-form is a finite distance. FIG. 16 is an explanatory diagram showing an example of a configuration of an image pickup device according to a fourth embodiment. FIG. 16 is an explanatory diagram showing an example of a configuration of a double-sided grid substrate according to a fifth embodiment. FIG. 28 is an external view showing an example of a mobile information terminal according to the sixth embodiment.

  In the following embodiments, when there is a need for convenience, the description will be divided into a plurality of sections or embodiments, but unless otherwise specified, they are not unrelated to each other, and one is the other. There is a relation of some or all of modified examples, details, supplementary explanations, and the like.

  In addition, in the following embodiments, when referring to the number of elements (including the number, numerical value, amount, range, etc.) of the elements, when explicitly stated, and in principle, the number is clearly limited to a specific number, etc. However, the number is not limited to the specific number, and may be equal to or more than the specific number.

  Further, in the following embodiments, the constituent elements (including element steps and the like) are not necessarily essential unless otherwise specified or in principle considered to be essential. Needless to say.

  Similarly, in the following embodiments, when referring to shapes, positional relationships, etc. of constituent elements, etc., the shapes thereof are substantially the same, unless otherwise specified or in principle not apparently. And the like, etc. are included. This also applies to the above numerical values and ranges.

  In all the drawings for explaining the embodiments, the same members are designated by the same reference numerals in principle and their repeated description is omitted.

  Hereinafter, embodiments will be described in detail.

(Embodiment 1)
<Example of configuration of imaging device>
FIG. 1 is an explanatory diagram showing an example of the configuration of the image pickup apparatus 101 according to the first embodiment.

  The imaging device 101 acquires an image of an object in the outside world without using an imaging lens, and includes a modulator 102, an image sensor 103, and an image processing circuit 106 as shown in FIG. There is.

  The modulator 102 is in close contact with and fixed to the light receiving surface of the image sensor 103, and has a configuration in which lattice patterns 104 and 105 are formed on a lattice substrate 102a. The lattice substrate 102a is made of a transparent material such as glass or plastic.

  In the modulator 102, a grating pattern 104 which is a second grating pattern is formed on the surface of the grating substrate 102a. Further, the surface of the lattice substrate 102a becomes the second surface. The lattice pattern 104 is formed of concentric circular lattice patterns in which the intervals of the lattice patterns, that is, the pitches are reduced in inverse proportion to the radius from the center toward the outside.

  Further, on the back surface of the grid substrate 102a, that is, the surface in contact with the light receiving surface of the image sensor 103, a grid pattern 105 serving as a first grid pattern is formed. The back surface of the lattice substrate 102a becomes the first surface.

  Like the lattice pattern 104, this lattice pattern 105 is also a concentric lattice pattern in which the pitch of the lattice pattern narrows in inverse proportion to the radius from the center toward the outside.

  The lattice pattern 104 and the lattice pattern 105 are formed by depositing aluminum or the like by, for example, a sputtering method used in a semiconductor process. The tint is created by the pattern with and without aluminum deposition.

  Note that the formation of the grid patterns 104 and 105 is not limited to this, and may be formed by printing with an ink jet printer or the like with different shades.

  The intensity of light transmitted through the lattice patterns 104 and 105 is modulated by the lattice pattern. The transmitted light is received by the image sensor 103. The image sensor 103 includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

  On the surface of the image sensor 103, pixels 103a, which are light receiving elements, are regularly arranged in a grid pattern. The image sensor 103 converts an optical image received by the pixel 103a into an image signal which is an electric signal. The image signal output from the image sensor 103 is subjected to image processing by an image processing circuit 106 which is an image processing unit and output to a monitor display 107 or the like.

<Example of shooting with image pickup device>
FIG. 2 is an explanatory diagram showing an example of shooting by the image pickup apparatus 101 of FIG. FIG. 2 shows an example in which the subject 301 is photographed by the imaging device 101 and displayed on the monitor display 107.

  As shown in the figure, when the subject 301 is photographed, one surface of the modulator 102, specifically, the surface of the lattice substrate 102a on which the lattice pattern 104 is formed, is faced to the subject 301. Will be taken.

<Example of image processing by image processing circuit>
Subsequently, an outline of image processing by the image processing circuit 106 will be described.

  FIG. 3 is a flowchart showing an outline of image processing by the image processing circuit 106 included in the image pickup apparatus 101 of FIG.

  First, the moire fringe image output from the image sensor 103 is subjected to a two-dimensional Fourier transform (FFT) operation for each color RGB (Red Green Blue) component to obtain a frequency spectrum (step S101). ).

  Then, after the data of the one side frequency of the frequency spectrum is cut out by the process of step S101 (step S102), the intensity of the frequency spectrum is calculated (step S103) to acquire an image.

  Then, noise removal processing is performed on the obtained image (step S104), and then contrast enhancement processing (step S105) is performed. After that, the color balance of the image is adjusted (step S106) and output as a captured image.

  With the above, the image processing by the image processing circuit 106 is completed.

<Principle of photographing of image pickup device>
Next, the shooting principle of the image pickup apparatus 101 will be described.

  First, the concentric lattice patterns 104 and 105 whose pitch becomes fine in inverse proportion to the radius from the center are defined as follows. In a laser interferometer, it is assumed that a spherical wave close to a plane wave interferes with a plane wave used as reference light.

  When the radius from the reference coordinate that is the center of the concentric circles is r and the phase of the spherical wave there is φ (r), this is calculated using the coefficient β that determines the magnitude of the bending of the wavefront,

Can be expressed as

  Despite the spherical wave, it is represented by the square of the radius r because it is a spherical wave close to a plane wave and can be approximated only by the lowest order of expansion. When a plane wave interferes with light with this phase distribution,

An intensity distribution of interference fringes such as

  this is,

It becomes a concentric stripe with bright lines at radial positions that satisfy. If the stripe pitch is p,

It can be seen that the pitch becomes narrower in inverse proportion to the radius.

  Such stripes are called Fresnel zone plates. Lattice patterns having a transmittance distribution proportional to the intensity distribution thus defined are used as the lattice patterns 104 and 105 shown in FIG.

  It is assumed that collimated light is incident on the modulator 102 having the thickness t and having such a lattice pattern formed on both sides at an angle θ 0 as shown in FIG. In terms of geometrical optics with the refraction angle in the modulator 102 as θ, the light multiplied by the transmittance of the grating on the front surface is incident on the back surface with a deviation of δ = t · tan θ, and the centers of the two concentric circular gratings are assumed. If they are formed in line, the transmittances of the gratings on the back surface are shifted by δ and are multiplied.

  At this time,

An intensity distribution such as

  It can be seen that the fourth term of this expansion formula forms straight striped patterns at equal intervals in the direction of displacement of the two lattices over the entire overlapping region. The fringes generated at a relatively low spatial frequency due to the superposition of such fringes are called moire fringes.

  Such straight stripes at equal intervals produce sharp peaks in the spatial frequency distribution obtained by the two-dimensional Fourier transform of the detected image. The value of δ, that is, the incident angle θ of the light ray can be obtained from the value of the frequency.

  Due to the symmetry of the concentric lattice arrangement, it is clear that the moire fringes uniformly obtained on the entire surface at equal intervals occur at the same pitch regardless of the deviation direction. Such stripes are obtained because the lattice pattern is formed by the Fresnel zone plate, and it is considered impossible to obtain uniform stripes on the entire surface with other lattice patterns.

  In the second term as well, it can be seen that the intensity of the Fresnel zone plate produces fringes modulated by moire fringes, but the frequency spectrum of the product of the two fringes is the convolution of the respective Fourier spectra, so a sharp peak is obtained. Absent.

  From equation (5), only the components with sharp peaks are

, The Fourier spectrum is

become that way.

  Here, F represents a Fourier transform operation, u and v are spatial frequency coordinates in the x direction and the y direction, and δ with parentheses is a delta function. From this result, it can be seen that in the spatial frequency spectrum of the detected image, the peak of the spatial frequency of moire fringes occurs at the position of u = ± δβ / π.

  This is shown in FIG. In FIG. 5, from left to right, a schematic view of the arrangement of the light beam and the modulator 102, moire fringes, and spatial frequency spectrum are shown. 5A shows the case of vertical incidence, FIG. 5B shows the case of incident light rays from the left side at an angle θ, and FIG. 5C shows the case of incident light rays from the right side at an angle θ. .

  The axes of the grating pattern 104 formed on the front surface side and the grating pattern 105 formed on the back surface side of the modulator 102 are aligned. In FIG. 5A, since the shadows of the lattice pattern 104 and the lattice pattern 105 coincide with each other, moire fringes do not occur.

  In FIG. 5B and FIG. 5C, the same moiré occurs because the lattice pattern 104 and the lattice pattern 105 have the same displacement, and the peak positions of the spatial frequency spectrum also match. It becomes impossible to determine whether the incident angle of is in the case of FIG. 5B or in the case of FIG. 5C.

  In order to avoid this, as shown in FIG. 6, for example, the two grating patterns 104 and 105 are preliminarily arranged so that the shadows of the two grating patterns are shifted and overlap even with respect to a light beam that is vertically incident on the modulator 102. It is necessary to shift relative to the axis.

  When the relative displacement of the shadows of the two gratings with respect to the vertically incident plane wave on the axis is δ0, the displacement δ caused by the plane wave with the incident angle θ is

Can be expressed as

  At this time, the peak of the spatial frequency spectrum of the moire fringes of the ray of incidence angle θ is

Position.

  When the size of the image sensor is S and the number of pixels in the x direction and the y direction of the image sensor are both N, the spatial frequency spectrum of the discrete image by the Fast Fourier Transform (FFT) is -N / (2S) to + N / ( 2S).

  From this, considering that the incident angle on the plus side and the incident angle on the minus side are received evenly, the spectral peak position of the moire fringe due to the vertically incident plane wave (θ = 0) is the origin (DC: direct current component) position. And, for example, the central position between the + side end frequency position, that is,

The spatial frequency position of is appropriate.

  Therefore, the relative center misalignment of the two gratings is

Is appropriate.

  FIG. 7 is a schematic diagram for explaining the generation of moire fringes and the frequency spectrum when the grid pattern 104 and the grid pattern 105 are arranged in a shifted manner.

  Similar to FIG. 5, the left side shows the layout of the light beam and the modulator 102, the center row shows the moire fringes, and the right side shows the spatial frequency spectrum. Further, FIG. 7A shows a case where the light ray is vertically incident, FIG. 7B shows a case where the light ray is incident from the left side at an angle θ, and FIG. 7C is a case where the light ray is incident from the right side. This is a case of incident at θ.

  The lattice pattern 104 and the lattice pattern 105 are arranged in advance with a shift of δ0. Therefore, Moire fringes also occur in FIG. 7A, and a peak appears in the spatial frequency spectrum.

  As described above, the shift amount δ0 is set so that the peak position appears in the center of the spectrum range on one side from the origin. At this time, in FIG. 7B, the deviation δ is further increased, and in FIG. 7C, the deviation δ is decreased. Therefore, unlike FIGS. 5A and 5B, FIG. 7B and FIG. The difference can be identified from the peak position of the spectrum.

  That is, the spectrum image of this peak is a bright spot indicating a light flux at infinity, which is nothing but an image taken by the image pickup apparatus 101 in FIG.

  If the maximum incident angle of parallel light that can be received is θmax,

Therefore, the maximum angle of view that can be received by the imaging device 101 is

Given in.

  From the analogy with the image formation using a general lens, considering that parallel light having an angle of view θmax is focused and received at the end of the image sensor, the effective focal length of the image pickup apparatus 101 without a lens is

Can be considered equivalent to.

  As shown in the equation (2), it is assumed that the transmittance distribution of the lattice pattern basically has a sinusoidal characteristic, but such a component should be included as the fundamental frequency component of the lattice pattern. For example, it is conceivable to binarize the transmittance of the grating pattern, change the duty of the grating area having high transmittance and the area of low transmittance, and widen the width of the area having high transmittance to increase the transmittance.

  In the above description, the incident light rays have only one incident angle at the same time. However, in order for the image pickup apparatus 101 to actually act as a camera, it is assumed that light having a plurality of incident angles is simultaneously incident. There must be.

  The light of such a plurality of incident angles already overlaps the images of the plurality of front side gratings at the time of being incident on the back side grating pattern. If these mutually generate moire fringes, there is a concern that they may become noises that hinder the detection of moire fringes with the lattice pattern 105 that is a signal component.

  However, in reality, the overlapping of the images of the lattice pattern 104 does not generate the peak of the moire image, and the peak occurs only in the overlapping with the lattice pattern 105 on the back surface side.

  The reason will be described below.

  First, the overlapping of the shadows of the grating pattern 104 on the surface side due to light rays having a plurality of incident angles is a sum, not a product, which is a big difference. When the shadow of the grating pattern 104 and the grating pattern 105 are overlapped by the light of one incident angle, the intensity distribution of the light which is the shadow of the grating pattern 104 is multiplied by the transmittance of the grating pattern 105 to obtain the grating on the back surface side. A light intensity distribution after passing through the pattern 105 is obtained.

  On the other hand, the overlapping of the shadows due to the lights incident on the surface-side grating pattern 104 at different angles is not the product but the sum, because the lights overlap. In the case of sum,

As described above, the distribution of the original Fresnel zone plate lattice is multiplied by the distribution of Moire fringes.

  Therefore, the frequency spectrum is represented by the overlap integral of each frequency spectrum. Therefore, even if the moiré spectrum has a sharp peak alone, in reality, only a ghost of the frequency spectrum of the Fresnel zone plate occurs at that position. That is, no sharp peak appears in the spectrum.

  Therefore, the spectrum of the moiré image detected even when light with a plurality of incident angles is input is always only the moiré of the product of the front side grating pattern 104 and the back side grating pattern 105, and the grating pattern 105 is single. Therefore, the number of detected peaks of the spectrum is only one for one incident angle.

<Confirmation of shooting principle>
The results of the simulation performed to confirm the principle are shown in FIGS. 8 and 9.

  FIG. 8: is explanatory drawing which shows the calculation result of the spatial frequency spectrum image at the time of irradiating with 10 light of a normal incident plane wave and the other 9 plane waves of different incident angles. FIG. 9 is a bird's-eye view showing the calculation result of the spatial frequency spectrum image when irradiating with a total of 10 lights of the normal incident plane wave and the other 9 plane waves having different incident angles.

  In each case, the sensor size of the image sensor 103 is 20 mm □, the viewing angle θmax = ± 70 °, the entrance-side and exit-side lattice coefficients β = 50 (rad / mm 2), δ0 = 0.8 mm, the number of pixels 1024 × 1024, and the modulator 102. At a substrate thickness of 1 mm and a substrate refractive index of 1.5, a normal incident plane wave, incident light of θx = 50 °, θy = 30 °, incident light of θx = −30 °, θy = 70 °, Incident light of θx = 10 °, θy = −20 °, incident light of θx = 20 °, θy = 30 °, incident light of θx = 30 °, θy = −40 °, θx = −10 °, Incident light of θy = 40 °, incident light of θx = −20 °, θy = −30 °, incident light of θx = −30 °, θy = 0 °, and θx = 40 °, θy = 50 ° It is a spectrum when incident light and a total of 10 plane waves are made incident.

  FIG. 8 is a black and white inverted image of the spectrum image, and FIG. 9 is a bird's-eye view showing the brightness of the spectrum image. The original moire image itself is omitted because the lattice pitch is fine and cannot be visually recognized even when displayed as the drawings in this specification.

  In the figure, the center shows the DC component, and the periphery shows the entire region of the spatial frequency spectrum region of ± N / 2S. Since the DC component has a large value, it is masked and removed, and only the peak component to be detected is displayed. Furthermore, since the peak width of the spectrum is narrow as it is and it is difficult to visually recognize it, the contrast is emphasized.

  Further, in FIG. 8, the position of the signal peak is displayed surrounded by a circle. The bird's-eye view of FIG. 9 cannot be displayed as it is because the drawn line does not pass through the peaks, so the result obtained by applying the mesh size averaging filter is displayed.

  All of them basically indicate that 10 peaks can be detected as a total of 20 peaks on both positive and negative sides of the origin. In this case, the pitch of the outermost periphery of the lattice pattern was about 6 μm, and the effective focal length was about 12.4 mm.

  Here, the correspondence between the parallel light, which has been described to be detected so far, and the light from the actual object, will be schematically described with reference to FIG.

  FIG. 10 is an explanatory diagram for explaining the angle formed by the light from each point forming the object with respect to the image sensor.

  Strictly speaking, the light from each point constituting the subject 301 is, as a spherical wave from a point light source, transmitted to the modulator 102 and the image sensor 103 (hereinafter, referred to as a grating sensor integrated substrate in FIG. 10) of the image pickup apparatus 101 in FIG. Incident.

  At this time, when the grating sensor integrated substrate is sufficiently small or far from the subject 301, the incident angles of the light illuminating the grating sensor integrated substrate can be regarded as the same from each point.

  Since the spatial frequency displacement Δu of the moire with respect to the minute angular displacement Δθ obtained from the equation (9) is 1 / S or less, which is the minimum resolution of the spatial frequency of the image sensor, the condition that Δθ can be regarded as parallel light is as follows.

Can be expressed as

  From this, if Δθ <0.18 °, this is a condition that can be realized if the sensor size of 20 mm is 6 m away from the subject.

  From the analogy of the above results, it can be seen that an image can be formed on an object at infinity by the image pickup apparatus of the present invention.

  As described above, the object image of the outside world can be obtained by a simple calculation such as the fast Fourier transform (FFT). As a result, the processing time until the acquisition of the object image can be shortened.

  Further, since a high-performance arithmetic processing device is not required, the hardware cost of the image pickup device 101 can be reduced, and further, the power consumption of the image pickup device 101 can be reduced by shortening the calculation processing time. You can

(Embodiment 2)
<Outline>
In the first embodiment, the image output from the image pickup apparatus 101 is vertically long, but in the second embodiment, a case where the output image is horizontally long will be described.

  In the first embodiment, as described above, the grid pattern 104 and the grid pattern 105 are formed so as to be shifted in the x (horizontal) direction of the image sensor 103. In other words, the rectangular image output from the image processing circuit 106 is formed so as to be displaced in the long side direction.

<Lattice pattern formation example>
FIG. 11 is an explanatory diagram showing an example of the spatial frequency spectrum when the lattice patterns 104 and 105 are laterally shifted.

  At this time, the image sensor 103 has a square shape, and its pixel pitch is the same in both the x direction and the y direction. In this case, as shown in the right side of FIG. 11, in the spatial frequency spectrum of the output of the image sensor, the image is reproduced separately in the left and right within the frequency range of both x and y ± N / S. Become.

  However, in the example shown in FIG. 11, the image is basically limited to the vertically long area. In general, an image acquired by a digital camera or the like is a horizontally long rectangle having an aspect ratio of 3: 2 or 4: 3, for example. Therefore, as the arrangement of the lattice patterns 104 and 105 suitable for a horizontally long rectangle, for example, the arrangement shown in FIG. 12 is desirable.

  FIG. 12 is an explanatory diagram showing an example of the spatial frequency spectrum when the lattice patterns 104 and 105 are vertically shifted.

  As shown in FIG. 12, the lattice patterns 104 and 105 are formed so as to be displaced in the vertical direction of the image sensor, that is, in the y direction of the image sensor. In other words, the grid pattern 104 and the grid pattern 105 are formed so as to be shifted in the short side direction of the rectangular image output from the image processing circuit 106. As a result, the image formed in the spatial frequency space of the image sensor output is vertically separated as shown on the right side of FIG.

  As described above, the image output from the imaging device 101 can be horizontally long. As a result, an image can be obtained as in a general digital camera, and thus the versatility of the image pickup apparatus 101 can be improved.

(Embodiment 3)
<Outline>
In the modulators 102 of the first and second embodiments, the grating pattern 104 and the grating pattern 105 having the same shape are formed on the front surface and the back surface of the grating substrate 102a so as to be offset from each other, so that the angle of the incident parallel light is moiré fringes. The image was constructed by detecting from the spatial frequency spectrum of.

  The lattice pattern 105 on the back surface side is an optical element that closely adheres to the image sensor 103 and modulates the intensity of the incident light. Therefore, by effectively setting the sensitivity of the image sensor in consideration of the transmittance of the lattice pattern 105 on the back surface side, it is possible to virtually generate moire in the processed image.

<Example of configuration of imaging device>
FIG. 13 is an explanatory diagram showing an example of the configuration of the image pickup apparatus 101 according to the third embodiment.

  The image pickup apparatus 101 of FIG. 13 differs from the image pickup apparatus 101 of FIG. 1 of the first embodiment in that the lattice pattern 105 shown in FIG. 1 is not formed on the back surface side of the lattice substrate 102a. Other configurations are the same as those in FIG. 1, and thus description thereof will be omitted.

  With the configuration shown in FIG. 13, the number of grid patterns formed on the grid substrate 102a can be reduced by one. Thereby, the manufacturing cost of the modulator 102 can be reduced.

  However, in this case, the pitch of the pixels 103a included in the image sensor 103 may be fine enough to sufficiently reproduce the pitch of the grid pattern, or may be coarse enough that the pitch of the grid pattern can be reproduced with the pixel pitch of the image sensor 103. is necessary.

  When the grid pattern is formed on both sides of the grid substrate 102a, the pitch of the grid pattern does not necessarily have to be resolved by the pixels 103a of the image sensor 103, and only the moire image needs to be resolved. Therefore, the pitch of the lattice pattern can be determined independently of the pixel pitch.

  However, when the image sensor 103 reproduces a lattice pattern, the lattice pattern and the image sensor 103 need to have the same resolution. Therefore, the image processing circuit 106 is provided with an intensity modulation circuit 106c corresponding to the back-side lattice pattern 105 (FIG. 1) for generating moire on the output image of the image sensor 103.

<Example of image processing by image processing circuit>
FIG. 14 is a flowchart showing an outline of image processing by the image processing circuit 106 included in the image pickup apparatus 101 of FIG.

  The flowchart of FIG. 14 differs from the flowchart of FIG. 3 of the first embodiment in the processing of step S201. In the processing of step S201, the intensity modulation circuit 106c described above generates a moire fringe image corresponding to the lattice pattern on the back surface side with respect to the image output from the image sensor 103.

  Since the processes of steps S202 to S208 of FIG. 14 are the same as the processes of steps S101 to S107 of FIG. 3 of the first embodiment, the description thereof will be omitted here.

  In this way, by providing the intensity modulation circuit 106c, the same effect as making the lattice pattern 105 (FIG. 1) on the back surface side variable can be obtained, and the detection light does not necessarily have to be parallel light. It is possible.

<About focusing>
FIG. 15 is an explanatory diagram showing that the projection of the grating pattern 104 on the front surface to the back surface is magnified by the grating pattern 104 when the object to be imaged is at a finite distance.

  As shown in FIG. 15, when a spherical wave from a point 1301 forming an object illuminates the surface side grating pattern 104 and its shadow 1302 is projected on the lower surface, the image projected on the lower surface is , Almost uniformly expanded.

  Therefore, if the transmittance distribution of the back-side grating pattern (corresponding to the grating pattern 105 in FIG. 1) designed for parallel light is multiplied as it is, linear moire fringes at equal intervals do not occur. However, if the lattice on the lower surface is enlarged in accordance with the uniformly enlarged shadow of the lattice pattern 104 on the front surface side, linear moire fringes with equal intervals are again generated in the enlarged shadow 1302. be able to.

  Thereby, the light from the point 1301 at a distance which is not necessarily infinity can be selectively reproduced. With this, focusing becomes possible, and it is possible to perform shooting by focusing on an arbitrary position instead of shooting at infinity as shown in the first embodiment.

  As described above, the convenience of the imaging device 101 can be improved.

(Embodiment 4)
In the fourth embodiment, a technique for changing the lattice pattern 104 on the front surface side in FIG. 1 will be described.

<Structure example and operation example of imaging device>
FIG. 16 is an explanatory diagram showing an example of the configuration of the image pickup apparatus 101 according to the fourth embodiment.

  The image pickup apparatus 101 in FIG. 16 differs from the image pickup apparatus 101 in FIG. 1 of the first embodiment in that a liquid crystal unit 108 is newly provided in the modulator 102 and a focus position designation input unit 109 is newly provided. It has just been established. Note that the configuration of the image sensor 103 in FIG. 16 is the same as that in FIG. 1, so description thereof will be omitted.

  The liquid crystal unit 108 is composed of, for example, a glass substrate (not shown) on which transparent electrodes and the like are provided, and a liquid crystal layer (not shown) is also provided. The liquid crystal layer is formed so as to be sandwiched between the glass substrate and the lattice substrate 102a. ing.

  An arbitrary lattice pattern 1403 is displayed on this liquid crystal layer, and the lattice pattern 1403 acts as the lattice pattern 104 on the surface side. On the back surface side of the grating substrate 102a in the modulator 102, a grating pattern 105 is formed as in FIG.

  The focus position designation input unit 109 is an input unit that sets a focus position, which is information such as a distance to a subject, and is connected to the image processing circuit 106. Further, the image processing circuit 106 is provided with a liquid crystal drive circuit 106a and a lattice pattern generation circuit 106b.

  The lattice pattern generation circuit 106b generates an optimal lattice pattern for focusing on the basis of the focus position input from the focus position designation input unit 109. The liquid crystal drive circuit 106a controls the display by applying a voltage to the transparent electrode formed on the glass substrate so that the lattice pattern generated by the lattice pattern generation circuit 106b is displayed on the liquid crystal layer of the liquid crystal unit 108.

  Light from a point 1301 at a finite distance, which is basically closer to infinity, is divergent light. Therefore, in order to make the back surface side grid pattern 105 and the back surface have the same size, the front surface side grid pattern 104 is gridded. It suffices to display the pattern 105 slightly smaller than the pattern 105.

  From the above, it is possible to achieve faster focusing.

(Embodiment 5)
In the fifth embodiment, another example of the front side lattice pattern formed on the double-sided lattice substrate will be described.

<Lattice pattern formation example>
FIG. 17 is an explanatory diagram showing an example of the configuration of the modulator 102 according to the fifth embodiment.

  In the first embodiment, the grating pattern 104 (FIG. 1) of the modulator 102 is formed by, for example, printing or sputtering, but in the modulator 102 of FIG. 17, it is configured by the cylindrical lens 110. The grid pattern 105 on the back surface side of the grid substrate 102a is the same as that in FIG. 1 of the first embodiment.

  In this case, the cylindrical lenses 110 are formed on the surface of the grating substrate 102a of the modulator 102 so as to have the same pattern as the grating pattern 104 of FIG. The cylindrical lens 110 is a lens having a cylindrical surface, and has a convex lens curvature in the vertical direction and no curvature in the horizontal direction.

  As described above, by forming the lattice pattern with the cylindrical lens 110, it is possible to significantly reduce the loss of light amount. For example, as described in the first embodiment, in a grid pattern in which light and shade are added by a printing pattern or the like, the printed portion of the grid pattern blocks light, resulting in a large loss of light quantity. become.

  On the other hand, in the case of the cylindrical lens 110, the light utilization efficiency can be improved without blocking the light.

  As described above, since the S / N ratio (Signal-to-Noise ratio) in the image pickup apparatus 101 can be increased, the drawing performance can be improved.

(Embodiment 6)
<Example of mobile information terminal configuration>
In the sixth embodiment, a mobile information terminal configured by using the image pickup apparatus 101 according to the fifth embodiment will be described.

  FIG. 18 is an external view showing an example of mobile information terminal 200 according to the sixth embodiment.

  The mobile information terminal 200 is, for example, a smartphone. The mobile information terminal 200 is not limited to a smartphone, and may be a mobile terminal having a built-in camera such as a tablet.

  The mobile information terminal 200 has an imaging device 101 built therein. An opening window 202 is provided on the back surface of the portable information terminal 200, and the modulator 102 of FIG. 16 is provided inside the portable information terminal 200 so as to be close to the opening window 202.

  Further, a knob 201 for focus adjustment is provided on the side surface of the mobile information terminal 200 on one long side. The knob 201 corresponds to the focus position designation input unit 109 of the fourth embodiment.

  The focus position is set by turning the knob 201, and an arbitrary lattice pattern 1403 is displayed on the liquid crystal layer of the liquid crystal unit 108 in FIG. 16 according to the set focus position. As a result, an image of an object at an arbitrary distance can be taken.

  The image pickup apparatus 101 can increase the effective focal length according to the equation (14) shown in the first embodiment. Therefore, the aperture can be increased while keeping the imaging device 101 thin.

  In the case of a smart phone digital camera using a general lens, the aperture of the lens has to be small in order to reduce the thickness of the portable information device. Therefore, the focal length becomes short, and the image is flat and no blur can be obtained.

On the other hand, in the image pickup apparatus 101, since the opening can be made large as described above, it is possible to produce a beautiful blur. As described above, the portable information terminal 200 having high depiction performance can be realized.

  Although the invention made by the present inventor has been specifically described based on the embodiment, the invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Needless to say.

  It should be noted that the present invention is not limited to the above-described embodiment, and various modifications are included. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described.

  Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. . Further, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

101 Imaging device 102 Modulator 102a Lattice substrate 103 Image sensor 103a Pixel 104 Lattice pattern 105 Lattice pattern 106 Image processing circuit 106a Liquid crystal drive circuit 106b Lattice pattern generation circuit 106c Intensity modulation circuit 107 Monitor display 108 Liquid crystal unit 109 Focus position designation input unit 110 Cylindrical lens 200 Personal digital assistant 201 Knob 202 Open window

Claims (9)

An image sensor that converts light received by a plurality of pixels arranged on the light receiving surface into an image signal and outputs the image signal,
A transparent modulator arranged on the light-receiving surface side of the image sensor to modulate light,
An image processing circuit that performs image processing of an image signal output from the image sensor,
Equipped with
A first concentric pattern is formed on one surface of the modulator, and the incident light is modulated by the first concentric pattern,
The image sensor receives light modulated by the modulator and outputs an image signal including an image corresponding to the first concentric pattern,
The image processing circuit,
An image signal including an image corresponding to the first concentric pattern output from the image sensor is input,
Generate a second concentric pattern image,
Generating a moire fringe image using an image corresponding to the first concentric circular pattern included in the input image signal and an image of the second concentric circular pattern,
A process including a two-dimensional Fourier transform is performed on the moire fringe image to generate an image,
Imaging device. The image pickup apparatus according to claim 1,
The imaging device, wherein the pitch of the concentric circles of the first concentric pattern becomes narrower from the center of the concentric circles toward the outside. The image pickup apparatus according to claim 1,
The imaging device, wherein the pitch of the concentric circles in each of the first concentric pattern and the second concentric pattern becomes narrower from the center of the concentric circle toward the outside. The image pickup apparatus according to claim 1,
The image processing circuit forms the second concentric pattern so that the center position of the concentric circle in the image of the second concentric pattern is displaced from the center position of the concentric circle in the image corresponding to the first concentric pattern. An imaging device to generate. The imaging device according to claim 4,
The center position of the concentric circles in the image corresponding to the first concentric pattern and the center position of the concentric circles in the image of the second concentric pattern are displaced from each other in the short side direction of the image output from the image processing circuit. The imaging device. The image pickup apparatus according to claim 1,
The image pickup device according to claim 1, wherein the first concentric pattern is formed by a cylindrical lens. The image pickup apparatus according to claim 1,
The image processing circuit generates an image by performing a process of calculating a frequency spectrum of the moire fringe image by a process of two-dimensional Fourier transforming the moire fringe image and a process of calculating an intensity of the frequency spectrum of the moire fringe image. An imaging device. An image sensor that converts light received by a plurality of pixels arranged on the light receiving surface into an image signal and outputs the image signal,
A transparent modulator arranged on the light-receiving surface side of the image sensor to modulate light,
An image processing circuit that performs image processing of an image signal output from the image sensor,
Equipped with
A first grating pattern is formed on one surface of the modulator, and the incident light is modulated by the first grating pattern,
The image sensor receives the light modulated by the modulator and outputs an image signal including an image corresponding to the first grating pattern,
The image processing circuit,
An image signal including an image corresponding to the first lattice pattern output from the image sensor is input,
Generate an image of the second grid pattern,
Generating an image including an interference image based on the image corresponding to the first lattice pattern included in the input image signal and the image of the second lattice pattern,
Generating an image by performing a predetermined signal processing on the image including the interference image,
Imaging device. The imaging device according to claim 8,
The predetermined signal processing includes a process of performing a two-dimensional Fourier transform on the image including the interference image to calculate a frequency spectrum, and a process of calculating the intensity of the frequency spectrum of the image including the interference image to generate an image. , Imaging device.