JP2021005743A - Imaging apparatus - Google Patents

Abstract

To create a captured image of uniform picture quality regardless of an incident angle of light.SOLUTION: An imaging apparatus includes a modulator having an imaging lattice pattern, and modulating intensity of incident light, an image sensor for creating a projection image by converting the incident light, passed through the converter, into an electric signal, an image processing part for creating a development image by development processing using a cross-correlation function with a development lattice pattern corresponding to the imaging lattice pattern, and an attention field angle setting part for setting an attention field angle for the image processing part, where the image processing part changes the development processing according to the attention field angle thus set.SELECTED DRAWING: Figure 1

Description

The present invention relates to an imaging device.

Mobile terminals such as smartphones are equipped with camera modules, and as mobile terminals become thinner, the camera modules are also required to be thinner. As a technique for reducing the thickness of a digital camera, for example, Patent Document 1 describes an imaging device that realizes a thinning by not using a lens.

JP-A-2018-061109

The image pickup apparatus described in Patent Document 1 causes light transmitted through a modulator having a predetermined pattern to be incident on an image sensor, and obtains an incident angle of the incident light from a projection pattern generated on the image sensor by cross-correlation calculation. , It is possible to obtain an captured image in which the subject is captured.

However, in the image pickup apparatus, when the light is obliquely incident, the entire projection pattern cannot be acquired by the image sensor, so that the obtained captured image is deteriorated as compared with the case where the light is vertically incident.

The present invention has been made in view of such a situation, and an object of the present invention is to enable a captured image having uniform image quality to be generated regardless of the incident angle of light.

The present application includes a plurality of means for solving at least a part of the above problems, and examples thereof are as follows.

In order to solve the above problems, the image pickup apparatus according to one aspect of the present invention has an imaging lattice pattern, a modulator that modulates the intensity of incident light, and an electric signal that transmits light that has passed through the modulator and is incident. Image processing that generates a developed image by a development process using an image sensor that generates a projected image by converting to, and a developing process that uses a mutual correlation function between the projected image and a developing lattice pattern corresponding to the imaging lattice pattern. The image processing unit includes a unit and an angle of view setting unit that sets the angle of view range of interest with respect to the image processing unit, and the image processing unit changes the development process according to the set angle of view range of interest. It is characterized by that.

According to one aspect of the present invention, it is possible to generate a captured image having uniform image quality regardless of the incident angle of light.

Issues, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

FIG. 1 is a diagram showing a configuration example of an imaging device that is the basis of each embodiment of the present invention. FIG. 2 is a diagram showing a first configuration example of the modulator in the image pickup apparatus of FIG. FIG. 3 is a diagram showing a second configuration example of the modulator in the image pickup apparatus of FIG. FIG. 4 is a diagram showing an example of a state of imaging by an imaging device. FIG. 5 is a diagram showing an example of a Gabor zone plate. FIG. 6 is a diagram showing an example of a Fresnel zone plate. FIG. 7 is a diagram for explaining that the parallel light obliquely incident on the modulator shifts on the image sensor. FIG. 8 is a diagram showing an example of a projected image. FIG. 9 is a diagram showing an example of a development grid pattern. FIG. 10 is a diagram showing an example of a developed image. FIG. 11 is a diagram for explaining the angle of view of the image pickup apparatus. FIG. 12 is a diagram showing an example of an imaging grid pattern. FIG. 13 is a diagram showing an example of an imaging grid pattern. FIG. 14 is a diagram showing a configuration example of a functional block included in the image processing unit. FIG. 15 is a flowchart illustrating an example of image processing. FIG. 16 is a diagram for explaining an angle formed by light from each point constituting the subject with respect to the sensor. FIG. 17 is a diagram showing an example of a state in which an imaging grid pattern is irradiated on an image sensor when the subject is at an infinite distance. FIG. 18 is a diagram showing an example of a state in which an imaging grid pattern is irradiated on the image sensor when the subject is at a finite distance. FIG. 19 is a diagram showing a modified example of the image pickup apparatus of FIG. FIG. 20 is a flowchart illustrating an example of image processing according to the modification of FIG. 19. FIG. 21 is a diagram showing an example of a state in which the image pickup grid pattern is not partially projected on the image sensor. FIG. 22 is a diagram showing an example of a projected image of the image pickup grid pattern corresponding to the state of FIG. 21. FIG. 23 is a diagram showing an example of a development grid pattern. FIG. 24 is a diagram showing a developed image corresponding to the projected image of FIG. 22. FIG. 25 is a diagram showing that the projection of the image pickup grid pattern with the expanded size covers the entire image sensor. FIG. 26 is a diagram showing an example of a projected image corresponding to the expanded grid pattern for imaging. FIG. 27 is a diagram showing an example of an expanded grid pattern for development. FIG. 28 is a diagram showing a developed image corresponding to the projected image of FIG. 26. FIG. 29 is a diagram showing a configuration example of an imaging device according to the first embodiment of the present invention. FIG. 30 is a diagram showing that the entire image pickup grid pattern reduced in size is projected onto the image sensor. FIG. 31 is a diagram showing an example of a projected image corresponding to an imaging grid pattern whose size has been reduced. FIG. 32 is a diagram showing an example of a projected image corresponding to an imaging grid pattern whose size has been reduced. FIG. 33 is a diagram showing a state in which the incident light is diffracted from the opening of the image pickup grid pattern and projected onto the image sensor. FIG. 34 is a diagram showing a state in which incident light is diffracted and projected onto an image sensor from adjacent openings of an imaging grid pattern. FIG. 35 is a diagram showing a configuration example of an imaging device according to a second embodiment of the present invention.

Hereinafter, a plurality of embodiments of the present invention will be described with reference to the drawings. In addition, in all the drawings for explaining each embodiment, in principle, the same members are designated by the same reference numerals, and the repeated description thereof will be omitted. In addition, in the following embodiments, the components (including element steps and the like) are not necessarily essential unless otherwise specified or clearly considered to be essential in principle. Needless to say. In addition, when saying "consisting of A", "consisting of A", "having A", and "including A", other elements are excluded unless it is clearly stated that it is only that element. It goes without saying that it is not something to do. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of a component or the like, the shape is substantially the same unless otherwise specified or, in principle, it is considered that it is not clearly the case. Etc., etc. shall be included.

<Structure example of the image pickup apparatus 101 which is the basis of each embodiment of the present invention>
FIG. 1 shows a configuration example of an image pickup apparatus 101 according to each embodiment of the present invention.

The image pickup apparatus 101 takes an image of the subject without using a lens for forming an image of light from the subject on the image sensor, and outputs the captured image obtained as a result to the image display unit 107.

The image pickup apparatus 101 includes a modulator 102, an image sensor 103, and an image processing unit 106.

The modulator 102 has an imaging grid pattern 104 on its upper surface (incident surface), and the imaging grid pattern 104 modulates the intensity of light passing through the modulator 102. The image sensor 103 generates an image signal (hereinafter, referred to as a sensor image or a projected image) according to the light incident through the modulator 102 by photoelectric conversion.

Next, FIG. 2 shows a first configuration example of the modulator 102. The first configuration example of the modulator 102 includes a grid substrate 102a and a grid pattern 104 for imaging, the grid substrate 102a is closely fixed to the light receiving surface of the image sensor 103, and the grid pattern 104 for imaging is the grid substrate 102a. It is formed on the upper surface of.

When imaging with visible light is assumed, a material transparent to visible light such as glass or plastic may be used for the lattice substrate 102a. The imaging lattice pattern 104 is formed by depositing a metal such as aluminum or chromium that blocks visible light on the upper surface of the lattice substrate 102a by, for example, a sputtering method used in a semiconductor process. The image-taking grid pattern 104 is shaded by a portion where metal is vapor-deposited and a portion where metal is not vapor-deposited.

The method of forming the image-imaging grid pattern 104 is not limited to the vapor deposition of metal by a sputtering method or the like, and what if the light transmittance can be modulated by, for example, printing with an inkjet printer or the like to add shading. May be formed in.

Further, when imaging with light other than visible light is assumed, the lattice substrate 102a is provided with a material transparent to the light to be imaged (for example, in the case of far infrared light, for example, germanium, silicon, chalcogenide). Etc.), and a material that blocks light to be imaged may be used for the imaging lattice pattern 104.

Next, FIG. 3 shows a second configuration example of the modulator 102. A second configuration example of the modulator 102 is composed of a support member 102b and an imaging grid pattern 104, the imaging grid pattern 104 is formed of a thin film, and the thin film (imaging grid pattern 104) is formed by the support member 102b. It is held at a predetermined distance from the image sensor 103.

Return to FIG. The image sensor 103 includes, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. On the light receiving surface of the image sensor 103, a plurality of pixels 103a, which are light receiving elements, are regularly arranged in a grid pattern. The image sensor 103 generates a sensor image (projected image) by converting the light incident on each pixel 103a through the modulator 102 into an electric signal.

The image pickup grid pattern 104 is a striped pattern composed of a plurality of concentric circles, and is formed so that the distance (also referred to as pitch or opening) between adjacent circles becomes narrower in inverse proportion to the radius from the center toward the outside. ing.

The image processing unit 106 performs image processing including development processing on the projected image, and outputs the captured image obtained as a result to the image display unit 107.

<State of imaging by the imaging device 101>
Next, FIG. 4 shows a state of imaging by the imaging device 101. As shown in the figure, the image pickup apparatus 101 can be used for an application in which the subject 401 is imaged, subjected to predetermined image processing, and the captured image obtained as a result is displayed on the image display unit 107.

<Imaging principle in the imaging device 101>
Next, the imaging principle in the imaging device 101 will be described.

First, a striped pattern composed of a plurality of concentric circles as an imaging grid pattern 104 is defined. In a laser interferometer or the like, it is assumed that a spherical wave close to a plane wave interferes with a plane wave used as reference light. When the center of the concentric circle is the reference coordinate, the radius from the reference coordinate is r, and the phase of the spherical wave at the position of the radius r from the reference coordinate is φ (r), the phase φ (r) is the magnitude of the curve of the wave surface. Using the coefficient β that determines, it can be expressed as in the following equation (1).

Despite the spherical wave, the phase φ (r) is represented by the square of the radius r because the spherical wave is close to a plane wave and can be approximated only by the lowest order of expansion. By interfering a plane wave with light having this phase distribution, the intensity distribution of the interference fringes represented by the following equation (2) can be obtained.

Equation (2) is a concentric striped pattern having bright lines at the position of radius r satisfying the following equation (3).

When the pitch of the striped pattern is p, the relationship of the following equation (4) can be obtained.

From equation (4), it can be seen that the pitch p becomes narrower in inverse proportion to the radius r. Such a concentric striped pattern is called a Gabor zone plate or a Fresnel zone plate.

FIG. 5 shows an example of a Gabor zone plate corresponding to the formula (2). FIG. 6 shows an example of a Fresnel zone plate in which I (r) of the formula (2) is binarized with 1 as a threshold value.

In the following, in order to simplify the explanation, only the x direction will be described using a mathematical formula, but the y direction will be expanded into two dimensions in the x direction and the y direction by processing in the same manner as in the x direction. It is possible.

Next, FIG. 7 is a diagram for explaining that the parallel light obliquely incident on the modulator 102 shifts on the image sensor 103.

As shown in the figure, when the thickness of the modulator 102 is d and the incident angle of the parallel light in the x direction is θ ₀ , if the refraction angle in the modulator 102 is θ, the geometrical optics Is, the light multiplied by the transmittance of the imaging lattice pattern 104 is shifted by k = d · tan θ and incident on the image sensor 103. Then, the image sensor 103 generates a projected image having an intensity distribution represented by the following equation (5).

Note that Φ _F in the equation (5) indicates the initial phase of the interference fringe intensity distribution in the equation (2).

FIG. 8 shows an example of a projected image of the imaging grid pattern 104 generated by the image sensor 103. As can be seen from FIG. 8, the imaging grid pattern 104 is projected by shifting it by k from the center of the image sensor 103.

FIG. 9 shows an example of a developing grid pattern 105 for use in the developing process for the projected image of FIG. FIG. 10 shows an example of a developed image obtained as a result of a developing process on the projected image of FIG. Bright spots with a shift amount of k appear in the developed image.

In the present embodiment, a correlation development method is adopted as the development process for the projected image, and the developed image is obtained by calculating the cross-correlation function between the projected image (FIG. 8) and the development grid pattern 105 (FIG. 9).

In general, if the cross-correlation function is calculated by a two-dimensional convolution operation, the amount of calculation becomes large. Therefore, in the present embodiment, the Fourier transform is used to perform the calculation so that the amount of calculation in the calculation of the cross-correlation function is smaller. The principle of calculating the cross-correlation function using the Fourier transform will be described below. However, the operation of the cross-correlation function may be performed by a two-dimensional convolution operation.

For the developing grid pattern 105, the same Gabor zone plate or Fresnel zone plate as the imaging grid pattern 104 is used. Therefore, the developing grid pattern 105 can be expressed by the following equation (6) using the initial phase Φ _B.

Since the developing grid pattern 105 is only used for calculation in the developing process and is not used for transmitting light like the imaging grid pattern 104, it may have a negative value. Therefore, the equation (6) representing the developing grid pattern 105 does not need to be offset by adding 1 as in the equation (2) representing the imaging grid pattern 104.

The results of the Fourier transform of equations (5) and (6) are as shown in the following equations (7) and (8).

In equations (7) and (8), F [] represents the Fourier transform operation, and u represents the frequency coordinates in the x direction. Δ () in equation (7) is a delta function. The same applies to F [], u, and δ () in the following equations.

What is important in the equations (7) and (8) is that the component representing the Fresnel zone plate or the Gabor zone plate is contained even after the Fourier transform. Therefore, the development grid pattern after the Fourier transform may be directly generated based on the equations (4) and (5). This makes it possible to reduce the amount of calculation in the image processing unit 106.

Next, the following equation (9) is obtained by multiplying the equation (7) and the equation (8).

The term e ^(-iku) represented by the exponential function in the equation (9) is a signal component, and the point spread function shown in the following equation (10) can be obtained by ^{performing an} inverse Fourier transform on this term.

Equation (10) represents a bright spot at a position deviated by k from the reference coordinates on the original x-axis, as in the developed image shown in FIG. This bright spot indicates a luminous flux at infinity, and the developed image (FIG. 9) is nothing but an image captured by the image pickup apparatus 101 (FIG. 1).

From the equation in the second term on the right side of the equation (9), when Φ _F = Φ _B = π / 4, the initial phase of the sin is 0. Further, in this case, the first term on the right side of the equation (9) becomes 0, and the term unnecessary for development can be reduced. If the development process is performed under these conditions, it is possible to obtain a developed image with less noise during development.

The development process for the projected image is not limited to the above-mentioned correlation development method, and a moire development method may be adopted.

<Angle of view in the imaging device 101>
Next, FIG. 11 is a diagram for explaining the angle of view in the image pickup apparatus 101.

The maximum angle of incidence of parallel light that can be detected by the image sensor 103, that is, the angle of view, is such that the peak of the cross-correlation function between the projected image of the imaging lattice pattern 104 and the developing lattice pattern 105 is the edge of the image sensor 103. This is the case. Therefore, as shown in the figure, the angle represented by the line connecting the center of the imaging grid pattern 104 and the end of the image sensor 103 is the maximum angle of view and is expressed as the following equation (11).

Further, in order to receive parallel light having an angle of view θ _max so as to be in focus at the edge of the image sensor 103, the effective focal length of the image pickup apparatus 101 without a lens can be inferred from the image formation by a general lens. Is considered to correspond to the following equation (12).

Here, from the equation (11), it can be seen that the angle of view θ _max can be changed by the thickness d of the modulator 102. Therefore, for example, when the modulator 102 has the configuration shown in FIG. 3 (second configuration example), if the length of the support member 102b is variable, the angle of view can be changed during imaging. It will be possible.

It is assumed that the transmittance distribution of the imaging grid pattern 104 basically fluctuates like a sine wave as shown in the equation (2), but the present invention is not limited to this. As the fundamental frequency component, there may be a component that fluctuates like a sine wave.

Next, FIGS. 12 and 13 show an example of an imaging grid pattern 104 having a component that fluctuates like a sine wave as a fundamental frequency component.

FIG. 12 shows the ternary transmittance of the imaging grid pattern 104. FIG. 13 shows the imaging grid pattern 104 in which the duty of the high transmittance region and the low transmittance region is changed to widen the width of the high transmittance region and increase the transmittance. In the case of the imaging grid pattern 104 shown in FIG. 13, the effect of suppressing the diffraction in the imaging grid pattern 104 can be obtained, and the deterioration of the projected image can be reduced.

<Configuration example of image processing unit 106>
Next, FIG. 14 shows a configuration example of a functional block included in the image processing unit 106.

The image processing unit 106 includes a two-dimensional FFT calculation unit 1301, a development grid FFT image generation unit 1302, an inter-lattice multiplication unit 1303, an inverse two-dimensional FFT calculation unit 1304, a noise removal unit 1305, a contrast enhancement unit 1306, and a color balance adjustment. Each functional block of the unit 1307 is provided.

The image processing unit 106 is composed of, for example, a general computer provided with a CPU (Central Processing Unit), a memory, a storage, a communication interface, and the like, and the above-mentioned functional block is realized by the CPU executing a program. Each functional block of the image processing unit 106 may be realized by hardware such as an image processing circuit.

The two-dimensional FFT calculation unit 1301 performs a two-dimensional FFT (fast Fourier transform) on the projected image input from the image sensor 103, and outputs the Fourier transform image of the projected image obtained as a result to the interstitial multiplication unit 1303. To do. The development grid FFT image generation unit 1302 generates a Fourier transform image of the development grid pattern 105 and outputs it to the inter-grid multiplication unit 1303. The Fourier transform image of the development grid pattern 105 does not need to be calculated for each frame (output of the image sensor 103), and may be stored in a memory or the like. This makes it possible to reduce the amount of calculation performed by the image processing unit 106.

Further, in the above description, the FFT is adopted as the Fourier transform, but instead of the FFT, a discrete cosine transform (DCT) or the like may be adopted. This makes it possible to reduce the amount of calculation as compared with the FFT.

The inter-grid multiplication unit 1303 multiplies the Fourier transform image of the projected image and the Fourier transform image of the development grid pattern 105 for each element and outputs the result to the inverse two-dimensional FFT calculation unit 1304. The inverse two-dimensional FFT calculation unit 1304 performs an inverse two-dimensional FFT with respect to the input from the interstitial multiplication unit 1303, and outputs the developed image obtained as a result to the noise removal unit 1305. The noise removing unit 1305 performs noise removing processing on the developed image input from the inverse two-dimensional FFT calculation unit 1304 and outputs it to the contrast enhancing unit 1306. The contrast enhancement unit 1306 performs contrast enhancement processing on the developed image from which noise has been removed and outputs the developed image to the color balance adjustment unit 1307. The color balance adjustment unit 1307 performs color balance adjustment processing on the developed image in which the contrast is emphasized, and outputs the captured image obtained as a result to the image display unit 107.

<Image processing by image processing unit 106>
Next, FIG. 15 is a flowchart illustrating an example of image processing by the image processing unit 106.

The image processing is started, for example, when a projected image is input from the image sensor 103 to the image processing unit 106.

First, the two-dimensional FFT calculation unit 1301 performs two-dimensional FFT on the projected image input from the image sensor 103, and outputs the Fourier transform image of the projected image obtained as a result to the interstitial multiplication unit 1303 (step). S1). Next, the development grid FFT image generation unit 1302 generates a Fourier transform image of the development grid pattern 105 by calculating the equation (8) and outputs it to the inter-grid multiplication unit 1303 (step S2).

Next, the inter-lattice multiplication unit 1303 multiplies the Fourier transform image of the projected image and the Fourier transform image of the developing lattice pattern 105 for each element and outputs them to the inverse two-dimensional FFT calculation unit 1304 (step S3). Next, the inverse two-dimensional FFT calculation unit 1304 performs the inverse two-dimensional FFT with respect to the input from the interstitial multiplication unit 1303, and outputs the developed image obtained as a result to the noise removal unit 1305 (step S4).

Next, the noise removing unit 1305 performs noise removing processing on the developed image input from the inverse two-dimensional FFT calculation unit 1304 and outputs it to the contrast enhancing unit 1306 (step S5). Next, the contrast enhancement unit 1306 performs contrast enhancement processing on the developed image from which noise has been removed and outputs the developed image to the color balance adjustment unit 1307 (step S6). Next, the color balance adjusting unit 1307 performs a color balance adjusting process on the developed image in which the contrast is emphasized, and outputs the captured image obtained as a result to the image display unit 107 (step S7). With the above, the image processing by the image processing unit 106 is completed.

In the image processing described above, the two-dimensional FFT is used for the calculation of the cross-correlation function, but the two-dimensional convolution calculation may be used instead of the two-dimensional FFT.

<Incident angle of light from the subject>
In the above description, it is assumed that light is incident on the image pickup apparatus 101 from only one direction at the same time. However, when the image pickup apparatus 101 actually captures a subject, light is incident on the image sensor 103 from a plurality of angles at the same time. In this way, the light that is simultaneously incident on the image pickup apparatus 101 at different angles of incidence is such that the projected images of the plurality of imaging grid patterns 104 are already overlapped when they are incident on the image sensor 103.

However, the incident light that is incident at the same time under illumination by a general light source is incoherent and does not interfere with each other, so that each incident light can be considered independently.

Next, the correspondence between the parallel light, which has been described so far for detection, and the light from an actual subject will be described with reference to FIG. FIG. 16 is a diagram for explaining an angle formed by light from each point constituting the subject with respect to the image sensor 103.

Strictly speaking, the light from each point constituting the subject 401 passes through the modulator 102 of the image pickup apparatus 101 as a spherical wave from a point light source and is incident on the image sensor 103. Hereinafter, the modulator 102 and the image sensor 103 are collectively referred to as a grid sensor integrated substrate 1401.

When the grid sensor integrated substrate 1401 is sufficiently small or sufficiently far from the subject 401, the incident angles of the light incident on the grid sensor integrated substrate 1401 can be regarded as the same from each point. Under this condition, the imaging device 101 can image a subject at infinity.

FIG. 17 shows a state in which the image pickup grid pattern 104 is projected onto the image sensor 103 by the light of the spherical wave from the point 1501 constituting the subject when the subject to be imaged is at an infinite distance.

As shown in the figure, the light of the spherical wave from the point 1501 constituting the subject at infinity becomes a plane wave while propagating a sufficiently long distance. Therefore, the projected image 1502 obtained by the image sensor 103 has substantially the same shape as the imaging grid pattern 104. Therefore, by performing a cross-correlation calculation on the projected image 1502 with the developing grid pattern 105, a developed image representing a single bright spot can be obtained.

<Principle of imaging a finite distance subject>
Next, FIG. 18 shows a state in which the imaging grid pattern 104 is projected onto the image sensor 103 by the spherical waves from the points 1601 constituting the subject when the subject to be imaged is at a finite distance.

As shown in the figure, when the spherical wave from the point 1601 constituting the subject passes through the imaging grid pattern 104 and is projected onto the image sensor 103, the projected image 1602 obtained by the image sensor 103 is almost one. It will be expanded like this. The enlargement ratio α can be calculated by the following equation (13) using the distance f from the imaging grid pattern 104 to the point 1601.

Therefore, even if the cross-correlation function is calculated using the developing grid pattern 105 having the same size as the imaging grid pattern 104, which is calculated on the assumption that the incident light is parallel light, a single bright spot. It is not possible to obtain a developed image representing.

In order to obtain a developed image representing a single bright spot, the cross-correlation function is calculated after the developing grid pattern 105 is also enlarged according to the projected image 1602 in which the imaging grid pattern 104 is uniformly enlarged. Just do it. Specifically, the coefficient β in the equation (8) representing the Fourier transform image of the development grid pattern 105 may be replaced with β / α ² .

As a result, the light from the point 1601 of the subject located at a distance not necessarily infinity can be selectively developed. In other words, the imaging device 101 can perform imaging by focusing on an arbitrary position.

Further, the imaging device 101 (FIG. 1) may be modified so that the image can be focused at an arbitrary distance after imaging.

FIG. 19 shows a modified example of the image pickup apparatus 101. In the modified example, an image storage unit 1701 and a focus setting unit 1703 are added to the image pickup apparatus 101 (FIG. 1), and the focus can be adjusted to an arbitrary distance after imaging.

The image storage unit 1701 is arranged between the image sensor 103 and the image processing unit 106. The image storage unit 1701 temporarily holds the projected image output from the image processing unit 106 and outputs it to the image processing unit 106.

The focus setting unit 1703 includes a user interface such as a button, a dial, and a GUI (Graphical User Interface) for receiving the focus distance input from the user, and image-processes the focus distance information indicating the received focus distance. Output to unit 106. The image processing unit 106 changes the size of the developing grid pattern 105 used for calculating the cross-correlation function with the projected image according to the input focus distance information.

Next, FIG. 20 is a flowchart for explaining an example of image processing according to a modified example (FIG. 19) of the image pickup apparatus 101.

The image processing is started, for example, when a projected image is input from the image sensor 103 to the image processing unit 106.

First, the two-dimensional FFT calculation unit 1301 performs two-dimensional FFT on the projected image input from the image sensor 103, and outputs the Fourier transform image of the projected image obtained as a result to the interstitial multiplication unit 1303 (step). S11). Next, the developing grid FFT image generation unit 1302 calculates the enlargement ratio α according to the equation (13) based on the focus distance information input from the focus setting unit 1703 (step S12). Next, the development grid FFT image generation unit 1302 generates a Fourier transform image of the development grid pattern 105 by substituting the coefficient β in the equation (8) with β / α ² and performing an operation between the grids. Output to the multiplication unit 1303 (step S13).

Since the processing of steps S14 to S18 is the same as that of steps S3 to S7 of FIG. 15, the description thereof will be omitted. This completes the image processing according to the modified example of the image pickup apparatus 101.

According to the image processing, it is possible to generate an captured image focused at an arbitrary distance based on the projected image after imaging and output it to the image display unit 107.

<Explanation of missing and corrected projection image of grid pattern 104 for imaging>
In the description so far, it has been assumed that the entire image pickup grid pattern 104 is projected onto the image sensor 103, but depending on the incident angle of light, the image pickup grid pattern 104 is partially projected onto the image sensor 103. It can happen that it is not projected.

FIG. 21 shows an example of a state in which the imaging grid pattern 104 is not partially projected onto the image sensor 103 when the image sensor 103 and the imaging grid pattern 104 have the same size.

For example, when there is a point (not shown) constituting the subject directly above the image sensor 103, the imaging grid pattern 104 is projected onto the image sensor 103 without being chipped by the light from that point. However, as shown in the figure, a part of the projection 1902 of the imaging grid pattern 104 by the light from the point 1901, which does not exist directly above the image sensor 103, protrudes from the image sensor 103, and the image sensor 103 The obtained projected image has a part of the image pickup grid pattern 104 missing.

FIG. 22 shows an example of the projected image 2001 of the imaging grid pattern 104 obtained by the image sensor 103 in the state of FIG. 21. As shown in the figure, the projected image 2001 lacks a part of the imaging grid pattern 104.

FIG. 23 shows an example of the developing grid pattern 105 corresponding to the imaging grid pattern 104 shown in FIG. FIG. 24 shows a developed image 2002 obtained by performing a developing process on the projected image 2001 shown in FIG. 22 using the developing grid pattern 105 shown in FIG. 23.

As shown in FIG. 24, in the developed image 2002, the bright spots 2003 corresponding to the points 1901 (FIG. 21) constituting the subject are extended in the x direction. This is because the amount of information in the x direction in the projected image 2001 is smaller than the amount of information in the y direction, and the point spread function (PSF) in the x direction becomes dull.

Therefore, in order to prevent the reduction of information due to the chipping of the projected image 2001, a method of increasing the size of the imaging grid pattern 104 and a method of reducing the size of the imaging grid pattern 104 can be considered.

<Method of expanding the imaging grid pattern 104>
FIG. 25 shows that the projection of the image pickup grid pattern 104 with the expanded size covers the entire image sensor 103.

As shown in the formula (11), the angle of view of the image pickup apparatus 101 can be calculated by using the size S of the image sensor 103 and the distance d between the image sensor 103 and the image pickup grid pattern 104. Further, the field of view (FOV) of the image pickup apparatus 101 can be calculated by the following equation (14) by further using the distance f between the image pickup grid pattern 104 and the points 2102 constituting the subject.

From the relationship between the position of the point 2102 existing at the outermost end of the field of view FOV and the size and position of the image sensor 103, the size l of the imaging grid pattern 104 that does not cause a chip in the projected image at all angles of view is determined by the following equation (15). Can be calculated by.

Hereinafter, the imaging grid pattern 104 expanded to the size l will be referred to as an imaging grid pattern 2101.

FIG. 26 shows an example of the projected image 2201 when the imaging grid pattern 2101 is used. As shown in the figure, when the imaging grid pattern 2101 is used, the projection covers the entire image sensor 103, so that it can be seen that a projection image 2201 without chips can be obtained.

FIG. 27 shows an example of the developing grid pattern 2202 corresponding to the imaging grid pattern 2101. As shown in the figure, the developing grid pattern 2202 has the same size as the imaging grid pattern 2101.

FIG. 28 shows a developed image 2203 obtained by a developing process using the developing grid pattern 2202 shown in FIG. 27 with respect to the projected image 2201 shown in FIG. 26.

As shown in FIG. 28, in the developed image 2203, as compared with the developed image 2002 shown in FIG. 24, the extension of the bright spot 2204 corresponding to the point 1901 (FIG. 21) constituting the subject in the x direction is increased. It can be seen that the image quality is high. This is because there is no difference between the amount of information in the x direction and the amount of information in the y direction in the projected image 2201, and the dullness of the point spread function is eliminated.

As described above, if the size of the image pickup grid pattern 104 is expanded so as to cover the entire angle of view of the image pickup apparatus 101, deterioration of the developed image due to chipping of the projected image can be suppressed.

However, as described above, when the size of the developing grid pattern 2202 is expanded according to the expanded size of the imaging grid pattern 2101, the larger each size is, the heavier the calculation load on the image processing unit 106 is and the more the development processing is performed. Will take time.

Therefore, next, the configuration of the image pickup apparatus capable of suppressing the increase in the calculation load in the image processing unit 106 will be described.

<Structure example of the image pickup apparatus 101 ₁ according to the first embodiment of the present invention>
FIG. 29 shows a configuration example of the image pickup apparatus 1011 according to the _first embodiment of the present invention. The image pickup apparatus 101 ₁ sets an angle range of interest in the entire angle of view, and adaptively changes the size of the development grid pattern 2202 according to the angle range of interest to calculate the amount of calculation required for the development process. It is possible to suppress the increase and improve the image quality in the angle range of interest.

The image pickup apparatus 101 ₁ is an image pickup apparatus 101 (FIG. 1) to which the attention angle of view setting unit 2301 is added. The angle of view setting unit 2301 is composed of a user interface such as a button, a dial, and a GUI (Graphical User Interface) for receiving an angle range of the entire angle of view to be noticed, which is input by the user from the outside, and the accepted angle. The angle of view range θp representing the range is output to the image processing unit 106. The attention angle of view setting unit 2301 may be a communication interface that communicates with a PC, a smartphone, or the like. In that case, the attention angle of view range θp input from the PC or the like is output to the image processing unit 106.

The image processing unit 106 in the image pickup apparatus 101 ₁ generates a development grid pattern 2202 having a size corresponding to the attention angle range θp input from the attention angle of view setting unit 2301, and uses the generated development grid pattern 2202. Perform development processing. The size of the developing grid pattern 2202 according to the angle of view range θp of interest can be calculated by the following equation (16).

According to the image pickup apparatus 101 ₁ , the size of the development grid pattern 2202 is adaptively changed according to the angle of view range θp of interest to perform the development process. Therefore, attention is paid to suppressing an increase in the amount of calculation required for the development process. It is possible to improve the image quality in the angle range.

<Method of reducing the imaging grid pattern 104>
In the first embodiment described above, attention is paid to the suppression of an increase in the amount of calculation required for the development process by using the expanded grid pattern 2101 for imaging and adaptively changing the size of the grid pattern 2202 for development. The image quality has been improved in the angle range.

Next, as a second embodiment of the present invention, a method of reducing the size of the image pickup grid pattern 104 to improve the image quality of the developed image when light is incident from an oblique direction will be described. Hereinafter, the imaging grid pattern 104 whose size has been reduced is referred to as an imaging grid pattern 2401.

FIG. 30 shows that the entire image pickup grid pattern 2401 is projected onto the image sensor 103.

As shown in the figure, when the size of the image pickup grid pattern 2401 is smaller than that of the image sensor 103, the point 2402 located in front of the image sensor 103 and the point 2403 located at the end of the angle of view of the image sensor 103. The light from each of them projects the entire image pickup grid pattern 2401 onto the image sensor 103 without chipping.

FIG. 31 shows an example of the projected image 2501 obtained by the light from the point 2402 passing through the imaging grid pattern 2401 and incident on the image sensor 103. FIG. 32 shows an example of the projected image 2502 obtained by the light from the point 2403 passing through the imaging grid pattern 2401 and incident on the image sensor 103.

According to FIGS. 31 and 32, the projected images 2501, 502 include a region (black region) in which the imaging grid pattern 2401 is not projected, but the entire imaging grid pattern 2401 is projected. Can be confirmed. Further, when the projected images 2501, 502 are compared, it can be seen that the imaging grid patterns 2401 are projected in the same size and shape. Therefore, the image quality of the developed image obtained by performing the developing process on each of the projected images 2501, 502 is the same.

The size of the developing grid pattern 105 used for the developing process for each of the projected images 2501, 502 is the same size as the projection of the imaging grid pattern 2401 in the projected images 2501, 502, and is the size of the imaging grid pattern 2401. Depending on the case, it may be smaller than the projected image 2501. This makes it possible to improve the image quality within the angle of view of interest while reducing the amount of calculation required for the development process.

<Restriction on the narrowest pitch>
In the first and second embodiments described above, the size of the imaging grid pattern 104 is changed. When the size of the imaging grid pattern 104 is changed, the pitch of the concentric striped pattern is limited.

The pitch width of the concentric striped pattern forming the imaging grid pattern 104 in the present embodiment is related to the resolution of the developed image, and the finer the pitch, the higher the resolution of the developed image. However, if the pitch is too fine, the shadow that reaches the image sensor 103 spreads due to diffraction and cannot be separated.

Therefore, the pitch of the concentric striped pattern forming the imaging grid pattern 104 needs to be wide enough to be separated even when diffraction occurs.

FIG. 33 shows a state in which incident light is diffracted from one aperture of the imaging grid pattern 104 and projected onto the image sensor 103. In the figure, when the incident light ray 2601 passes through the imaging grid pattern 104, it spreads at a diffraction angle W _θ corresponding to the aperture width q, and is projected on the image sensor with a width of W _q . Assuming that the distance between the imaging grid pattern 104 and the image sensor 103 is d and the wavelength of the incident ray 2601 is λ, the width W _q can be calculated by using an approximation as follows.

In this region where the width W _q is large, the image pickup grid pattern 104 is not projected on the image sensor 103, and as a result, the amount of information in the projected image is insufficient, leading to a decrease in resolution.

Therefore, consider the width of the imaging grid pattern 104 in which a region not projected by diffraction is not generated.

FIG. 34 shows a state in which incident light opens from an adjacent aperture of the imaging grid pattern 104 and is projected onto the image sensor 103. Here, the aperture width q can be calculated by the following equation (18) using the distance r from the center of the imaging grid pattern 104 and the coefficient β.

Strictly speaking, the width q of the adjacent openings is different because the distance r from the center of the imaging grid pattern 104 is different, but the difference of the distance r is sufficiently small in the adjacent openings, so that the opening width q Can be regarded as the same.

In the case of the figure, the incident light is projected onto the image sensor 103 from two openings separated by a distance of 2q from the center. In the region where the projections from these two openings overlap, the projected imaging grid pattern cannot be handled as information, which causes a decrease in the resolution of the developed image.

Therefore, the coefficient β is adjusted so that the projections from two adjacent openings do not overlap. Rayleigh limits can be used for this.

The Rayleigh limit indicates the limit at which a light source of the same brightness can be recognized as two light sources when they are seen close to each other, and the definition is that the first dark line diffracted from the first light source is used. , Corresponds to the case where the center of the second light source is reached. When this is applied to the imaging grid pattern 104, the resolution is the limit when the width of the pattern projected from one aperture reaches 4q, and the aperture width q can be calculated by the following equation (19).

Further, using the equation (18), the coefficient β of the lattice pattern having the pitch of the resolution limit can be calculated by the following equation (20).

As can be seen from the equation (20), the optimum coefficient β depends on the size S of the image sensor 103 and the distance d between the image sensor 103 and the imaging grid pattern 104, that is, based on the equation (20). By adjusting the coefficient β by changing the size S of the sensor 103 and the distance d between the image sensor 103 and the imaging grid pattern 104, it is possible to effectively utilize the resolution of the image sensor 103.

<Structure example of the image pickup apparatus 101 ₃ according to the third embodiment of the present invention>
Next, FIG. 35 shows a configuration example of the image pickup apparatus 1013 according to the _third embodiment of the present invention. Similar to the image pickup device 101 ₁ , the image pickup apparatus 101 ₃ sets an angle range of interest in the entire angle of view, and adaptively changes the size of the developing lattice pattern 2202 according to the angle range of interest. It is possible to suppress an increase in the amount of calculation required for the development process and improve the image quality in the angle range of interest.

The image pickup apparatus 101 ₃ is an image pickup apparatus 101 (FIG. 1) to which the attention angle of view storage unit 2801 is added. The attention angle of view storage unit 2801 is composed of, for example, a semiconductor memory or the like, and the attention angle of view range θp representing the angle range of interest in the entire angle of view, which is stored in advance (during manufacturing, etc.), is stored in the image processing unit 106. Output. The attention angle of view storage unit 2801 corresponds to the attention angle of view setting unit of the present invention.

Similar to the image pickup device 101 ₁ , the image processing unit 106 in the image pickup apparatus 101 ₃ generates a development grid pattern 2202 having a size corresponding to the angle of view range θp input from the angle of view storage unit 2801, and the generated development. Development processing is performed using the grid pattern 2202.

According to the imaging apparatus 101 _3, since the development process by changing the size of the developing grating pattern 2202 adaptively in accordance with the target angle range theta] p, the suppression of the increase in the calculation amount required for the development, attention It is possible to improve the image quality in the angle range.

Further, according to the image pickup device 101 ₃ , unlike the image pickup device 101 ₁ , it is not necessary for the user to input the angle range of the entire angle of view to be noticed from the outside, so that the user's operation can be simplified accordingly. Can be done.

However, if the imaging apparatus 101 _3, needs to be stored attention angle range θp advance in the target angle storage unit 2801. Further, if the imaging apparatus 101 _3, if the distance between the imaging grating pattern 104 and the image sensor 103 d, the center position coordinates of the imaging grid pattern 104, such as refractive index of the modulator 102 is varied by external factors such as, It is necessary to update the attention angle range θp stored in the attention angle of view storage unit 2801 to a value corresponding to those fluctuations.

The present invention is not limited to the above-described embodiment, and further includes various modifications. In addition, each of the above-described embodiments has been described in detail in order to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to those including all the components described above. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

In addition, the control lines and information lines indicate those that are considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

101, 101 _{1 to} 101 ₃ ... Imaging device, 102 ... Modulator, 102a ... Lattice substrate, 102b ... Support member, 103 ... Image sensor, 103a ... Pixel, 104 ... -Imaging lattice pattern, 105 ... Development lattice pattern, 106 ... Image processing unit, 107 ... Image display unit, 1301 ... Two-dimensional FFT calculation unit, 1302 ... Development lattice FFT image Generation unit, 1303 ... Interstitial multiplication unit, 1304 ... Inverse two-dimensional FFT calculation unit, 1305 ... Noise removal unit, 1306 ... Contrast enhancement unit, 1307 ... Color balance adjustment unit, 1401. .. Lattice sensor integrated substrate, 1501 ... point, 1502 ... projected image, 1601 ... point, 1602 ... projected image, 1701 ... image storage unit, 1703 ... focus setting unit, 1901 ... Point, 1902 ... Projection, 2001 ... Projection image, 2002 ... Developed image, 2003 ... Bright point, 2101 ... Imaging grid pattern, 2102 ... Point, 2201 ... -Projection image 2202 ... Development grid pattern 2203 ... Development image, 2204 ... Bright spot, 2301 ... Attention angle of view setting unit 2401 ... Imaging grid pattern, 2402, 2403. .. Point, 2501,502 ... Projected image, 2601 ... Incident light, 2801 ... Attention angle of view storage

Claims (9)

A modulator that has a grid pattern for imaging and modulates the intensity of incident light,
An image sensor that generates a projected image by converting the light that has passed through the modulator and incident light into an electrical signal.
An image processing unit that generates a developed image by a developing process using a cross-correlation function between the projected image and a developing grid pattern corresponding to the imaging grid pattern.
An angle of view setting unit that sets the angle of view range of interest for the image processing unit, and
With
The image processing unit is an image pickup apparatus characterized in that the development process is changed according to the set angle of view range of interest. The imaging device according to claim 1.
The image processing unit is an image pickup apparatus that changes the size of the development grid pattern used in the development processing according to the set angle of view range of interest. The imaging device according to claim 2.
An image pickup apparatus characterized in that the size of the image pickup grid pattern is larger than the size of the image sensor. The imaging device according to claim 2.
An image pickup apparatus characterized in that the size of the image pickup grid pattern is smaller than the size of the image sensor. The imaging device according to claim 1.
The image pickup grid pattern is a striped pattern composed of a plurality of concentric circles.
An imaging device characterized in that the striped opening becomes finer in inverse proportion to the distance from the center of the concentric circles. The imaging device according to claim 5.
An imaging device characterized in that the width of the primary diffracted light from the minimum aperture width of the striped pattern forming the image-taking grid pattern is four times or less the minimum aperture width. The imaging device according to claim 1.
The attention angle of view setting unit is an image pickup apparatus characterized in that the attention angle of view range input by the user is set for the image processing unit. The imaging device according to claim 1.
The attention angle of view setting unit is an image pickup apparatus characterized in that the attention angle of view range stored in advance is set for the image processing unit. The imaging device according to claim 1.
The image processing unit is an image pickup apparatus characterized in that a captured image is generated by performing noise removal processing, contrast enhancement processing, and color balance processing on the developed image.

Images