JP6770164B2 - Imaging device - Google Patents

Description

The present invention relates to an image pickup apparatus and an image generation method, and more particularly to a technique effective for improving the performance of the image pickup apparatus.

Digital cameras mounted on smartphones and the like need to be made thinner. As a thinning technique of this kind of digital camera, for example, there is a technique for obtaining an object image without using a lens (see, for example, Patent Document 1).

In this technique, a special diffraction grating substrate is attached to an image sensor, and the incident angle of the incident light is obtained by inverse problem calculation from the projection pattern in which the light transmitted through the diffraction grating substrate is generated on the image sensor. It obtains an image of an object.

U.S. Patent Application Publication No. 2014/0253781

In Patent Document 1 described above, the pattern of the diffraction grating formed on the upper surface of the substrate to be attached to the image sensor is a special lattice pattern such as a spiral pattern. Then, the image of the object is obtained by solving the inverse problem for reproducing the image from the projection pattern received by the image sensor, but the problem that the calculation when solving this inverse problem becomes complicated. There is.

If the calculation is complicated, there is a problem that the processing time naturally becomes long and the time until the photograph is displayed becomes long. A high-performance CPU or the like is used to perform arithmetic processing at high speed, but in that case, there is a risk that the cost of the digital camera will increase and the power consumption will increase.

An object of the present invention is to provide a technique capable of reducing the cost and power consumption of an image pickup apparatus by facilitating the detection of the incident angle of a light ray passing through a lattice substrate.

The above and other objects and novel features of the present invention will become apparent from the description and accompanying drawings herein.

A brief outline of the typical inventions disclosed in the present application is as follows.

That is, a typical image pickup device has an image sensor, a modulator, and an image processing unit. The image sensor converts an optical image captured by a plurality of pixels arranged in an array on the imaging surface into an image signal and outputs the image signal. The modulator is provided on the light receiving surface of the image sensor and modulates the intensity of light. The image processing unit performs image processing of the image signal output from the image sensor.

The modulator also has a grid substrate, a first grid pattern, and a second grid pattern. The first lattice pattern is formed on the first surface of the lattice substrate close to the light receiving surface of the image sensor. The second grid pattern is formed on the second surface facing the first surface.

The first grid pattern and the second grid pattern are each composed of a plurality of concentric circles. The modulator outputs the light transmitted through the second grid pattern to the image sensor by intensity-modulating the light with the first grid pattern.

In particular, the plurality of concentric circles in the first grid pattern and the second grid pattern are formed from a plurality of concentric circles in which the pitch of the concentric circles becomes finer in inverse proportion to the reference coordinates at the center of the concentric circles. For a plurality of concentric circles, the pitch of the concentric circles becomes finer in inverse proportion to the reference coordinates at the center of the concentric circles.

Among the inventions disclosed in the present application, the effects obtained by typical ones will be briefly described as follows.

(1) The processing time until the acquisition of the object image can be shortened.

(2) The hardware cost of the imaging device can be reduced.

It is explanatory drawing which shows an example of the structure in the image pickup apparatus according to Embodiment 1. FIG. It is explanatory drawing which shows an example of the photographing by the image pickup apparatus of FIG. It is a flowchart which shows the outline of the image processing by the image processing circuit which the image pickup apparatus which FIG. 1 has. It is explanatory drawing which shows an example of the in-plane deviation of the projected image from the front surface to the back surface of a double-sided lattice substrate by obliquely incident parallel light. It is a schematic diagram explaining the generation of moire fringes and the frequency spectrum when the axes of the lattice pattern of a double-sided lattice substrate are aligned. It is explanatory drawing which shows an example of the double-sided lattice substrate formed by shifting the axis of the lattice pattern on the front surface side and the lattice pattern on the back surface side. It is a schematic diagram explaining the generation of moire fringes and the frequency spectrum when the lattice pattern is staggered. It is explanatory drawing which shows the calculation result of the spatial frequency spectrum image when irradiating with a total of 10 lights of a vertically incident plane wave and 9 other plane waves of different incident angles. It is a bird's-eye view which shows the calculation result of the spatial frequency spectrum image when irradiating with a total of 10 lights of a vertically incident plane wave and 9 other plane waves of different incident angles. It is explanatory drawing explaining the angle which light from each point which constitutes an object makes with respect to an image sensor. It is explanatory drawing which shows an example of the spatial frequency spectrum when the lattice pattern is shifted in the lateral direction. It is explanatory drawing which shows an example of the spatial frequency spectrum when the lattice pattern is shifted in the vertical direction. It is explanatory drawing which shows an example of the structure in the image pickup apparatus according to Embodiment 3. It is a flowchart which shows the outline of the image processing by the image processing circuit which the image pickup apparatus which FIG. 13 has. It is explanatory drawing which shows that the projection on the back surface of the lattice pattern on the front surface side is enlarged from the lattice pattern when the object to be imaged is at a finite distance. It is explanatory drawing which shows an example of the structure in the image pickup apparatus according to Embodiment 4. It is explanatory drawing which shows an example of the structure in the double-sided lattice substrate by Embodiment 5. It is an external view which shows an example of the mobile information terminal according to Embodiment 6.

In the following embodiments, when necessary for convenience, the description will be divided into a plurality of sections or embodiments, but unless otherwise specified, they are not unrelated to each other, and one is the other. It is related to some or all of the modified examples, details, supplementary explanations, etc.

In addition, in the following embodiments, when the number of elements (including the number, numerical value, quantity, range, etc.) is referred to, when it is specified in particular, or when it is clearly limited to a specific number in principle, etc. Except, the number is not limited to the specific number, and may be more than or less than the specific number.

Furthermore, in the following embodiments, the components (including element steps, etc.) are not necessarily essential unless otherwise specified or clearly considered to be essential in principle. Needless to say.

Similarly, in the following embodiments, when the shape, positional relationship, etc. of the constituent elements are referred to, the shape is substantially the same, except when it is clearly stated or when it is considered that it is not so clearly in principle. Etc., etc. shall be included. This also applies to the above numerical values and ranges.

Further, in all the drawings for explaining the embodiment, the same members are, in principle, given the same reference numerals, and the repeated description thereof will be omitted.

Hereinafter, embodiments will be described in detail.

(Embodiment 1)
<Configuration example of imaging device>
FIG. 1 is an explanatory diagram showing an example of the configuration in the image pickup apparatus 101 according to the first embodiment.

The image pickup apparatus 101 acquires an image of an object in the outside world without using a lens to form an image, and is composed of a modulator 102, an image sensor 103, and an image processing circuit 106 as shown in FIG. There is.

The modulator 102 is closely fixed to the light receiving surface of the image sensor 103, and has a configuration in which lattice patterns 104 and 105 are formed on the lattice substrate 102a, respectively. The lattice substrate 102a is made of a transparent material such as glass or plastic.

In the modulator 102, a lattice pattern 104, which is a second lattice pattern, is formed on the surface of the lattice substrate 102a. Further, the surface of the lattice substrate 102a becomes the second surface. The grid pattern 104 is composed of a concentric grid pattern in which the spacing between the grid patterns, that is, the pitch becomes narrower in inverse proportion to the radius from the center toward the outside.

Further, a lattice pattern 105 as a first lattice pattern is formed on the back surface of the lattice substrate 102a, that is, the surface on the side in contact with the light receiving surface of the image sensor 103. The back surface of the lattice substrate 102a is the first surface.

Similar to the grid pattern 104, the grid pattern 105 is also composed of a concentric grid pattern in which the pitch of the grid pattern becomes narrower in inverse proportion to the radius from the center toward the outside.

The lattice pattern 104 and the lattice pattern 105 are formed by depositing aluminum or the like by, for example, a sputtering method used in a semiconductor process. Shading is done by patterns with and without aluminum deposition.

The formation of the lattice patterns 104 and 105 is not limited to this, and may be formed by adding shading by printing with, for example, an inkjet printer.

The intensity of light transmitted through the grid patterns 104 and 105 is modulated by the grid pattern. The transmitted light is received by the image sensor 103. The image sensor 103 includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

Pixels 103a, which are light receiving elements, are regularly arranged in a grid pattern on the surface of the image sensor 103. The image sensor 103 converts an optical image received by the pixel 103a into an image signal which is an electric signal. The image signal output from the image sensor 103 is image-processed by the image processing circuit 106, which is an image processing unit, and output to the monitor display 107 or the like.

<Example of shooting with an imaging device>
FIG. 2 is an explanatory diagram showing an example of photographing by the image pickup apparatus 101 of FIG. FIG. 2 shows an example in which the subject 301 is photographed by the image pickup apparatus 101 and displayed on the monitor display 107.

As shown in the figure, when the subject 301 is photographed, one surface of the modulator 102, specifically the surface of the lattice substrate 102a on which the lattice pattern 104 is formed, faces the subject 301. Is taken.

<Image processing example of image processing circuit>
Subsequently, an outline of image processing by the image processing circuit 106 will be described.

FIG. 3 is a flowchart showing an outline of image processing by the image processing circuit 106 included in the image pickup apparatus 101 of FIG.

First, a two-dimensional Fourier transform (FFT) calculation is performed on each moire fringe image output from the image sensor 103 for each component of color RGB (Red Green Blue) to obtain a frequency spectrum (step S101). ).

Subsequently, after cutting out the data of the one-sided frequency of the frequency spectrum by the processing of step S101 (step S102), the intensity calculation of the frequency spectrum is performed (step S103) to acquire an image.

Then, the obtained image is subjected to noise removal processing (step S104), followed by contrast enhancement processing (step S105) and the like. After that, the color balance of the image is adjusted (step S106) and the image is output as a captured image.

As a result, the image processing by the image processing circuit 106 is completed.

<Principle of imaging device>
Subsequently, the imaging principle in the image pickup apparatus 101 will be described.

First, the concentric grid patterns 104 and 105 whose pitch becomes finer in inverse proportion to the radius from the center are defined as follows. In a laser interferometer or the like, it is assumed that a spherical wave close to a plane wave interferes with a plane wave used as reference light.

When the radius from the reference coordinate, which is the center of the concentric circle, is r and the phase of the spherical wave there is φ (r), this is determined by using the coefficient β that determines the magnitude of the curve of the wave surface.

Can be expressed as.

Despite the spherical wave, it is represented by the square of the radius r because it is a spherical wave close to a plane wave and can be approximated only by the lowest order of expansion. When a plane wave interferes with light having this phase distribution,

The intensity distribution of the interference fringes is obtained.

this is,

It becomes a concentric stripe with a bright line at a radius position that satisfies. If the stripe pitch is p,

Is obtained, and it can be seen that the pitch becomes narrower in inverse proportion to the radius.

Such stripes are called Fresnel zone plates. The lattice patterns having the transmittance distribution proportional to the intensity distribution defined in this way are used as the lattice patterns 104 and 105 shown in FIG.

It is assumed that parallel light is incident on the modulator 102 having a thickness t having such a lattice pattern formed on both sides at an angle θ0 as shown in FIG. Geometrically, with the refraction angle in the modulator 102 as θ, the light multiplied by the transmittance of the front lattice is incident on the back surface with a deviation of δ = t · tan θ, and the centers of the two concentric lattices are assumed to be. If they are formed together, the transmittances of the lattices on the back surface are shifted by δ and multiplied.

At this time,

The intensity distribution is obtained.

It can be seen that the fourth term of this expansion formula creates straight, evenly spaced striped patterns in the direction of deviation of the two grids over the overlapping region. The fringes generated at a relatively low spatial frequency by superimposing such fringes are called moire fringes.

Such straight, evenly spaced fringes produce sharp peaks in the spatial frequency distribution obtained by the two-dimensional Fourier transform of the detected image. From the value of the frequency, the value of δ, that is, the incident angle θ of the light ray can be obtained.

From the symmetry of the concentric grid arrangement, it is clear that the moire fringes obtained uniformly over the entire surface at equal intervals occur at the same pitch regardless of the direction of deviation. The reason why such stripes are obtained is that the lattice pattern is formed by the Fresnel zone plate, and it is considered impossible to obtain uniform stripes on the entire surface with other lattice patterns.

In the second term, it can be seen that the intensity of the Fresnel zone plate is modulated by moire fringes, but the frequency spectrum of the product of the two fringes is a convolution of each Fourier spectrum, so a sharp peak can be obtained. Absent.

From equation (5), only the components with sharp peaks

When taken out as, the Fourier spectrum is

become that way.

However, here, F represents the operation of the Fourier transform, u and v are the spatial frequency coordinates in the x and y directions, and δ with parentheses is the delta function. From this result, it can be seen that the peak of the spatial frequency of the moire fringes occurs at the position of u = ± δβ / π in the spatial frequency spectrum of the detected image.

The situation is shown in FIG. In FIG. 5, from left to right, a layout diagram of the light beam and the modulator 102, a moire fringe, and a schematic diagram of the spatial frequency spectrum are shown, respectively. 5 (a) shows a vertical incident, FIG. 5 (b) shows a case where a light ray is incident from the left side at an angle θ, and FIG. 5 (c) shows a case where a light ray is incident from the right side at an angle θ. ..

The grid pattern 104 formed on the front surface side and the grid pattern 105 formed on the back surface side of the modulator 102 have axes aligned with each other. In FIG. 5A, since the shadows of the grid pattern 104 and the grid pattern 105 match, moire fringes do not occur.

In FIGS. 5 (b) and 5 (c), the same moire occurs because the deviations between the lattice pattern 104 and the lattice pattern 105 are equal, and the peak positions of the spatial frequency spectra also match. It becomes impossible to determine whether the incident angle of FIG. 5 (b) is the case of FIG. 5 (b) or the case of FIG. 5 (c).

In order to avoid this, the shadows of the two grid patterns are shifted and overlapped with respect to the light rays perpendicularly incident on the modulator 102, for example, as shown in FIG. 6, the two grid patterns 104 and 105 are illuminated in advance. It is necessary to shift it relative to the axis.

When the relative deviation of the shadows of the two lattices with respect to the vertically incident plane wave on the axis is δ0, the deviation δ caused by the plane wave at the incident angle θ is

It can be expressed as.

At this time, the peak of the spatial frequency spectrum of the moire fringes of the light beam having an incident angle θ is on the positive side of the frequency.

It becomes the position of.

Assuming that the size of the image sensor is S and the number of pixels in the x and y directions of the image sensor is N, the spatial frequency spectrum of the discrete image by the fast Fourier transform (FFT) is from -N / (2S) to + N / (. Obtained in the range of 2S).

From this, considering that the positive-side incident angle and the negative-side incident angle are received evenly, the spectral peak position of the moire fringes due to the vertically incident plane wave (θ = 0) is the origin (DC: DC component) position. And, for example, the center position with the frequency position at the + side end, that is,

It is appropriate to use the spatial frequency position of.

Therefore, the relative center misalignment of the two grids is

Is appropriate.

FIG. 7 is a schematic diagram illustrating the generation of moire fringes and the frequency spectrum when the grid pattern 104 and the grid pattern 105 are arranged in a staggered manner.

Similar to FIG. 5, the left side shows the arrangement of the light beam and the modulator 102, the center row shows the moire fringes, and the right side shows the spatial frequency spectrum. Further, FIG. 7A shows a case where the light ray is vertically incident, FIG. 7B shows a case where the light ray is incident at an angle θ from the left side, and FIG. 7C shows a case where the light ray is angled from the right side. This is the case of incident at θ.

The grid pattern 104 and the grid pattern 105 are arranged in advance by shifting by δ0. Therefore, moire fringes also occur in FIG. 7A, and peaks appear in the spatial frequency spectrum.

As described above, the shift amount δ0 is set so that the peak position appears in the center of the spectrum range on one side from the origin. At this time, in FIG. 7 (b), the deviation δ is further increased, and in FIG. 7 (c), the deviation is decreased. Therefore, unlike FIG. 5, FIGS. 7 (b) and 7 (c) are shown. The difference can be discerned from the peak position of the spectrum.

The spectral image of this peak is a bright spot showing a luminous flux at infinity, which is nothing but an image taken by the image pickup apparatus 101 of FIG.

Assuming that the maximum angle of incidence of parallel light that can be received is θmax,

Therefore, the maximum angle of view that can be received by the image pickup device 101 is

Given in.

By analogy with imaging using a general lens, considering that parallel light with an angle of view θmax is focused at the edge of the image sensor and received, the effective focal length of the image pickup device 101 without a lens is

Can be considered to be equivalent to.

As shown in Eq. (2), the transmittance distribution of the lattice pattern is basically assumed to have sinusoidal characteristics, but such a component is included as the fundamental frequency component of the lattice pattern. For example, it is conceivable to binarize the transmittance of the lattice pattern, change the duty of the lattice region having high transmittance and the duty of the region having low transmittance, and widen the width of the region having high transmittance to increase the transmittance.

In the above description, the incident light rays are only one incident angle at the same time, but in order for the imaging device 101 to actually act as a camera, it is assumed that light of a plurality of incident angles is incident at the same time. There must be.

Light with such a plurality of incident angles already overlaps the images of the plurality of front-side grids when they are incident on the back-side grid pattern. If these cause moire fringes with each other, there is a concern that noise may hinder the detection of moire fringes with the lattice pattern 105, which is a signal component.

However, in reality, the overlap of the images of the grid pattern 104 does not cause the peak of the moire image, and the peak is generated only by the overlap with the grid pattern 105 on the back surface side.

The reason will be explained below.

First, the major difference is that the overlap of the shadows of the grid patterns 104 on the surface side due to the light rays having a plurality of incident angles is not a product but a sum. In the overlap between the shadow of the grid pattern 104 and the grid pattern 105 due to the light of one incident angle, the light intensity distribution that is the shadow of the grid pattern 104 is multiplied by the transmittance of the grid pattern 105 to obtain the grid on the back side. The light intensity distribution after passing through the pattern 105 is obtained.

On the other hand, the overlap of shadows due to a plurality of light incidents on the surface side lattice pattern 104 at different angles is not a product but a sum because the light overlaps. In the case of sum

The distribution is obtained by multiplying the distribution of the grid of the original Fresnel zone plate by the distribution of moiré fringes.

Therefore, the frequency spectrum is represented by the overlap integral of each frequency spectrum. Therefore, even if the moiré spectrum alone has a sharp peak, in practice, only a ghost of the frequency spectrum of the Fresnel zone plate occurs at that position. That is, no sharp peaks occur in the spectrum.

Therefore, the spectrum of the moire image that is detected even when light with a plurality of incident angles is input is always only the moire of the product of the lattice pattern 104 on the front surface side and the lattice pattern 105 on the back surface side, and the lattice pattern 105 is single. Therefore, there is only one peak of the detected spectrum for one incident angle.

<Confirmation of shooting principle>
Below, the results of the simulation performed to confirm the principle are shown in FIGS. 8 and 9.

FIG. 8 is an explanatory diagram showing a calculation result of a spatial frequency spectrum image when irradiated with a total of 10 lights of a vertically incident plane wave and nine other plane waves having different incident angles. FIG. 9 is a bird's-eye view showing the calculation results of the spatial frequency spectrum image when irradiated with a total of 10 lights of a vertically incident plane wave and nine other plane waves having different incident angles.

In each case, the sensor size of the image sensor 103 is 20 mm □, the viewing angle is θmax = ± 70 °, the incident side and exit side lattice coefficients β = 50 (rad / mm2), δ0 = 0.8 mm, the number of pixels is 1024 × 1024, and the modulator 102. When the substrate thickness is 1 mm and the refractive index of the substrate is 1.5, the vertically incident plane wave, the incident light of θx = 50 ° and θy = 30 °, and the incident light of θx = -30 ° and θy = 70 °. Incident light of θx = 10 °, θy = -20 °, incident light of θx = 20 °, θy = 30 °, incident light of θx = 30 °, θy = -40 °, θx = -10 °, CTy = 40 ° incident light, θx = -20 °, θy = -30 ° incident light, θx = -30 °, θy = 0 ° incident light, θx = 40 °, θy = 50 ° It is a spectrum when a total of 10 plane waves of incident light are incident.

FIG. 8 is a black-and-white inverted image of the spectrum image, and FIG. 9 is a bird's-eye view showing the brightness of the spectrum image. The original moire image itself has a fine grid pitch and cannot be visually recognized even if it is displayed as a drawing of the present specification, and thus is omitted.

In the figure, the center is the DC component, and the periphery is the entire area of the spatial frequency spectrum region of ± N / 2S. Since the DC component has a large value, it is masked and removed, and only the peak component to be detected is displayed. Furthermore, since the peak width of the spectrum is narrow and difficult to see as it is, the contrast is emphasized.

Further, in FIG. 8, the position of the signal peak is indicated by being surrounded by a circle. Since the bird's-eye view of FIG. 9 cannot be displayed as it is without passing through the peak, the result of applying the mesh size averaging filter is displayed.

In each case, it is basically shown that 10 peaks can be detected as a total of 20 peaks on both the positive and negative sides of the origin. In this case, the pitch of the outermost periphery of the lattice pattern was about 6 μm, and the effective focal length was about 12.4 mm.

Here, the correspondence between the parallel light, which has been described so far for detection, and the light from an actual object will be schematically described with reference to FIG.

FIG. 10 is an explanatory diagram illustrating an angle formed by light from each point constituting the object with respect to the image sensor.

Strictly speaking, the light from each point constituting the subject 301 is sent to the modulator 102 and the image sensor 103 of the image pickup apparatus 101 of FIG. 1 (hereinafter referred to as a grid sensor integrated substrate in FIG. 10) as spherical waves from a point light source. Incident.

At this time, when the grid sensor integrated substrate is sufficiently small or sufficiently far from the subject 301, it can be considered that the incident angles of the light illuminating the grid sensor integrated substrate are the same from each point.

Since the spatial frequency displacement Δu of the moire with respect to the minute angular displacement Δθ obtained from the equation (9) is 1 / S or less, which is the minimum resolution of the spatial frequency of the image sensor, the condition that Δθ can be regarded as parallel light is

It can be expressed as.

From this, if Δθ <0.18 °, this is a condition that can be realized if the sensor size is 20 mm and the distance from the subject is 6 m.

From the analogy of the above results, it can be seen that the imaging apparatus of the present invention can form an image on an object at infinity.

As described above, an object image in the outside world can be obtained by a simple calculation such as a fast Fourier transform (FFT). As a result, the processing time until the acquisition of the object image can be shortened.

Further, since a high-performance arithmetic processing unit is not required, the hardware cost of the imaging apparatus 101 can be reduced, and further, the power consumption of the imaging apparatus 101 can be reduced by shortening the arithmetic processing time. Can be done.

(Embodiment 2)
<Summary>
In the first embodiment, the image output from the image pickup apparatus 101 is vertically long, but in the second embodiment, the case where the output image is horizontally long will be described.

In the first embodiment, as described above, the grid pattern 104 and the grid pattern 105 are formed so as to be shifted in the x (horizontal) direction of the image sensor 103. In other words, it is assumed that the rectangular image output from the image processing circuit 106 is formed so as to be displaced in the long side direction.

<Example of grid pattern formation>
FIG. 11 is an explanatory diagram showing an example of a spatial frequency spectrum when the lattice patterns 104 and 105 are laterally shifted.

At this time, the shape of the image sensor 103 is square, and the pixel pitch thereof is the same in both the x-direction and the y-direction. In this case, as shown on the right side of FIG. 11, the spatial frequency spectrum of the output of the image sensor shows that the images are reproduced separately on the left and right within the frequency range of ± N / S for both x and y. Become.

However, in the example shown in FIG. 11, the image is basically limited to a vertically long area. Generally, an image acquired by a digital camera or the like is a horizontally long rectangle having an aspect ratio of 3: 2 or 4: 3, for example. Therefore, as the arrangement of the lattice patterns 104 and 105 suitable for the horizontally long rectangle, for example, the one shown in FIG. 12 is desirable.

FIG. 12 is an explanatory diagram showing an example of the spatial frequency spectrum when the lattice patterns 104 and 105 are vertically shifted.

As shown in FIG. 12, the grid pattern 104 and the grid pattern 105 are formed so as to be displaced in the vertical direction of the image sensor, that is, in the y direction of the image sensor. In other words, the grid pattern 104 and the grid pattern 105 are formed so as to be offset in the short side direction of the rectangular image output from the image processing circuit 106. As a result, the images formed in the spatial frequency space of the image sensor output are separated into upper and lower parts as shown on the right side of FIG.

As described above, the image output from the image pickup apparatus 101 can be made horizontally long. As a result, an image can be obtained in the same manner as a general digital camera, so that the versatility of the image pickup apparatus 101 can be enhanced.

(Embodiment 3)
<Summary>
In the modulator 102 of the first and second embodiments, the lattice patterns 104 and the lattice patterns 105 having the same shape are formed on the front surface and the back surface of the lattice substrate 102a by shifting them from each other, so that the angle of the incident parallel light is moire fringes. The image was constructed by detecting from the spatial frequency spectrum of.

The lattice pattern 105 on the back surface side is an optical element that closely adheres to the image sensor 103 and modulates the intensity of incident light. Therefore, by effectively setting the sensitivity of the image sensor in consideration of the transmittance of the grid pattern 105 on the back surface side, moire can be virtually generated in the processed image.

<Configuration example of imaging device>
FIG. 13 is an explanatory diagram showing an example of the configuration in the image pickup apparatus 101 according to the third embodiment.

The image pickup device 101 of FIG. 13 differs from the image pickup device 101 of FIG. 1 of the first embodiment in that the lattice pattern 105 shown in FIG. 1 is not formed on the back surface side of the lattice substrate 102a. Since the other configurations are the same as those in FIG. 1, the description thereof will be omitted.

By adopting the configuration of FIG. 13, the lattice pattern formed on the lattice substrate 102a can be reduced by one surface. Thereby, the manufacturing cost of the modulator 102 can be reduced.

However, in this case, the pitch of the pixels 103a of the image sensor 103 may be fine enough to reproduce the pitch of the grid pattern, or coarse enough to reproduce the pitch of the grid pattern with the pixel pitch of the image sensor 103. is necessary.

When the grid pattern is formed on both sides of the grid substrate 102a, the pitch of the grid pattern does not necessarily have to be resolved by the pixels 103a of the image sensor 103, and only the moire image can be resolved. Therefore, the pitch of the grid pattern can be determined independently of the pixel pitch.

However, when the image sensor 103 reproduces the grid pattern, the resolutions of the grid pattern and the image sensor 103 need to be the same. Therefore, the image processing circuit 106 is provided with an intensity modulation circuit 106c corresponding to the grid pattern 105 (FIG. 1) on the back surface side for generating moire on the output image of the image sensor 103.

<Image processing example of image processing circuit>
FIG. 14 is a flowchart showing an outline of image processing by the image processing circuit 106 included in the image pickup apparatus 101 of FIG.

The difference between the flowchart in FIG. 14 and the flowchart in FIG. 3 in the first embodiment is the process in step S201. In the process of step S201, the intensity modulation circuit 106c described above generates a moire fringe image corresponding to the lattice pattern on the back surface side of the image output from the image sensor 103.

Hereinafter, the processes of steps S202 to S208 of FIG. 14 are the same as the processes of steps S101 to S107 of FIG. 3 of the first embodiment, and thus the description thereof will be omitted here.

By providing the intensity modulation circuit 106c in this way, the same effect as making the lattice pattern 105 (FIG. 1) on the back surface side variable can be obtained, and the detection light does not necessarily have to be parallel light. It is possible.

<About focusing>
FIG. 15 is an explanatory diagram showing that when the object to be imaged is at a finite distance, the projection of the lattice pattern 104 on the front surface side onto the back surface is enlarged from the lattice pattern 104.

As shown in FIG. 15, when the spherical wave from the point 1301 constituting the object irradiates the lattice pattern 104 on the surface side and the shadow 1302 is projected on the lower surface, the image projected on the lower surface is , Enlarged almost uniformly.

Therefore, if the transmittance distribution of the lattice pattern on the back surface side (corresponding to the lattice pattern 105 in FIG. 1) designed for parallel light is multiplied as it is, linear moire fringes at equal intervals will not occur. However, if the grid on the lower surface is enlarged to match the shadow of the uniformly enlarged grid pattern 104 on the front surface side, linear moire fringes at equal intervals are generated again with respect to the enlarged shadow 1302. be able to.

As a result, the light from the point 1301 at a distance not necessarily infinity can be selectively reproduced. As a result, focusing becomes possible, and shooting can be performed by focusing on an arbitrary position instead of shooting at infinity as shown in the first embodiment.

As described above, the convenience of the image pickup apparatus 101 can be enhanced.

(Embodiment 4)
In the fourth embodiment, a technique for making the lattice pattern 104 on the surface side of FIG. 1 variable will be described.

<Configuration example and operation example of imaging device>
FIG. 16 is an explanatory diagram showing an example of the configuration in the image pickup apparatus 101 according to the fourth embodiment.

The image pickup device 101 of FIG. 16 differs from the image pickup device 101 of FIG. 1 of the first embodiment in that the modulator 102 is newly provided with the liquid crystal unit 108 and the focus position designation input unit 109 is newly provided. It has just been set up. Since the configuration of the image sensor 103 in FIG. 16 is the same as that in FIG. 1, the description thereof will be omitted.

The liquid crystal portion 108 has a configuration in which a liquid crystal layer (not shown) is also provided on a glass substrate (not shown) on which a transparent electrode or the like is formed, and the liquid crystal layer is formed so as to be sandwiched between the glass substrate and the lattice substrate 102a. ing.

An arbitrary grid pattern 1403 is displayed on the liquid crystal layer, and the grid pattern 1403 acts as a grid pattern 104 on the surface side. A lattice pattern 105 is formed on the back surface side of the lattice substrate 102a in the modulator 102 as in FIG.

The focus position designation input unit 109 is an input unit for setting a focus position, which is information such as a distance to a subject, and is connected to an image processing circuit 106. Further, the image processing circuit 106 is provided with a liquid crystal drive circuit 106a and a grid pattern generation circuit 106b.

The grid pattern generation circuit 106b generates a grid pattern optimal for focusing based on the focus position input from the focus position designation input unit 109. The liquid crystal drive circuit 106a applies a voltage to the transparent electrode formed on the glass substrate so that the grid pattern generated by the grid pattern generation circuit 106b is displayed on the liquid crystal layer of the liquid crystal unit 108 to control the display.

Basically, the light from the point 1301 at a finite distance, which is closer than infinity, is divergent light. The display may be slightly smaller than the pattern 105.

From the above, it is possible to enable faster focusing.

(Embodiment 5)
In the fifth embodiment, another example of the lattice pattern on the surface side formed on the double-sided lattice substrate will be described.

<Example of grid pattern formation>
FIG. 17 is an explanatory diagram showing an example of the configuration in the modulator 102 according to the fifth embodiment.

In the first embodiment, the lattice pattern 104 (FIG. 1) of the modulator 102 is formed by, for example, printing or a sputtering method, but the modulator 102 of FIG. 17 is composed of a cylindrical lens 110. The lattice pattern 105 on the back surface side of the lattice substrate 102a is the same as that of FIG. 1 of the first embodiment.

In this case, the cylindrical lenses 110 are arranged on the surface of the lattice substrate 102a in the modulator 102 so as to have a pattern similar to the lattice pattern 104 of FIG. The cylindrical lens 110 is a lens whose surface is formed of a cylindrical surface, has a curvature of a convex lens in the vertical direction, and has no curvature in the horizontal direction.

By forming the lattice pattern with the cylindrical lens 110 in this way, the loss of the amount of light can be significantly reduced. For example, as described in the first embodiment, in a grid pattern in which shades are added by a print pattern or the like, the printed portion of the grid pattern blocks light, resulting in a large loss of light quantity. become.

On the other hand, in the case of the cylindrical lens 110, the light utilization efficiency can be improved without blocking the light.

As described above, the S / N ratio (Signal-to-Noise ratio) in the image pickup apparatus 101 can be increased, so that the drawing performance can be improved.

(Embodiment 6)
<Configuration example of mobile information terminal>
In the sixth embodiment, a mobile information terminal configured by using the image pickup apparatus 101 according to the fifth embodiment will be described.

FIG. 18 is an external view showing an example of the mobile information terminal 200 according to the sixth embodiment.

The mobile information terminal 200 is, for example, a smartphone or the like. The mobile information terminal 200 is not limited to a smartphone, and may be a portable terminal having a built-in camera such as a tablet.

The personal digital assistant 200 has a built-in image pickup device 101. An opening window 202 is provided on the back surface of the mobile information terminal 200, and the modulator 102 of FIG. 16 is provided inside the mobile information terminal 200 so as to be close to the opening window 202.

Further, a knob 201 for focusing adjustment is provided on one long side side surface of the portable information terminal 200. The knob 201 corresponds to the focus position designation input unit 109 of the fourth embodiment.

The focus position is set by turning the knob 201, and an arbitrary grid pattern 1403 is displayed on the liquid crystal layer of the liquid crystal unit 108 of FIG. 16 according to the set focus position. As a result, it is possible to take an image of an object at an arbitrary distance.

The image pickup apparatus 101 can increase the effective focal length according to the equation (14) shown in the first embodiment. Therefore, the opening can be enlarged while keeping the image pickup apparatus 101 thin.

In the case of a digital camera for smartphones using a general lens, the aperture of the lens must be reduced in order to reduce the thickness of the information mobile device. Therefore, the focal length is shortened, and the image is flat and cannot be blurred.

On the other hand, in the image pickup apparatus 101, since the opening can be enlarged as described above, it is possible to produce a beautiful blur, and thus it is possible to realize the portable information terminal 200 having high depiction performance.

Although the invention made by the present inventor has been specifically described above based on the embodiment, the present invention is not limited to the embodiment and can be variously modified without departing from the gist thereof. Needless to say.

The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations.

It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. .. Further, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

101 Imaging device 102 Modulator 102a Lattice substrate 103 Image sensor 103a Pixel 104 Lattice pattern 105 Lattice pattern 106 Image processing circuit 106a Liquid crystal drive circuit 106b Lattice pattern generation circuit 106c Intensity modulation circuit 107 Monitor display 108 Liquid crystal unit 109 Focus position designation input unit 110 Cylindrical lens 200 Mobile information terminal 201 Knob 202 Opening window

Claims (9)

An image sensor that converts the light received by a plurality of pixels arranged on the light receiving surface into an image signal and outputs it.
A transmissive modulator arranged on the light receiving surface side of the image sensor to modulate light,
An image processing circuit that performs image processing of the image signal output from the image sensor, and
With
A first concentric pattern is formed on one surface of the modulator, and the light incident on the first concentric pattern is modulated.
The image sensor receives the light modulated by the modulator and outputs an image signal including an image corresponding to the first concentric pattern.
The image processing circuit
An image signal including an image corresponding to the first concentric pattern output from the image sensor is input.
Generate an image of the second concentric pattern and
A moire fringe image is generated by using the image corresponding to the first concentric pattern included in the input image signal and the image of the second concentric pattern.
A process including a two-dimensional Fourier transform is performed on the moire fringe image to generate an image.
Imaging device. In the imaging apparatus according to claim 1,
An imaging device in which the pitch of the concentric circles of the first concentric pattern becomes narrower from the center of the concentric circles toward the outside. In the imaging apparatus according to claim 1,
An imaging device in which the pitch of concentric circles in each of the first concentric pattern and the second concentric pattern becomes narrower from the center of the concentric circles toward the outside. In the imaging apparatus according to claim 1,
The image processing circuit performs the second concentric pattern so that the center position of the concentric circles in the image of the second concentric pattern deviates from the center position of the concentric circles in the image corresponding to the first concentric pattern. An imaging device to generate. In the imaging apparatus according to claim 4,
The center position of the concentric circles in the image corresponding to the first concentric pattern and the center position of the concentric circles in the image of the second concentric pattern are deviated from each other in the short side direction of the image output from the image processing circuit. There is an image pickup device. In the imaging apparatus according to claim 1,
The first concentric pattern is an imaging device formed by a cylindrical lens. In the imaging apparatus according to claim 1,
The image processing circuit generates an image by performing a process of calculating the frequency spectrum of the moire fringe image by a two-dimensional Fourier transform process of the moire fringe image and a process of calculating the intensity of the frequency spectrum of the moire fringe image. Imaging device. An image sensor that converts the light received by a plurality of pixels arranged on the light receiving surface into an image signal and outputs it.
A transmissive modulator arranged on the light receiving surface side of the image sensor to modulate light,
An image processing circuit that performs image processing of the image signal output from the image sensor, and
With
A first grid pattern is formed on one surface of the modulator, and the light incident on the first grid pattern is modulated.
The image sensor receives the light modulated by the modulator and outputs an image signal including an image corresponding to the first grid pattern.
The image processing circuit
An image signal including an image corresponding to the first grid pattern output from the image sensor is input.
Generate an image of the second grid pattern and
An image including an image corresponding to the first grid pattern included in the input image signal and an interference image based on the image of the second grid pattern is generated.
A predetermined signal processing is performed on the image including the interference image to generate an image.
Imaging device. In the imaging apparatus according to claim 8,
The predetermined signal processing includes a process of calculating a frequency spectrum by performing a two-dimensional Fourier transform on an image including the interference image, and a process of calculating the intensity of the frequency spectrum of the image including the interference image to generate an image. , Imaging device.