JP2021005786A - Imaging apparatus, imaging method, transmitting device of imaging system and receiving device of imaging system - Google Patents

Abstract

To appropriately remove noise due to a point spread function with a vibrating side lobe that can occur in an image obtained as a result of imaging.SOLUTION: An imaging apparatus comprises: an imaging unit that images an object and outputs an image in which grid noise due to a side lobe in a point spread function during imaging occurs; and a grid noise removing unit that removes the grid noise from the image on the basis of the grid noise information. The imaging apparatus may comprise a grid noise information output unit that outputs the grid noise information to the grid noise removing unit.SELECTED DRAWING: Figure 25

Description

The present invention relates to an imaging device, an imaging method, a transmitting device of an imaging system, and a receiving device of an imaging system.

An imaging device to be mounted on a device such as an IoT (Internet of Things) device or a wearable device is required to be thin and low in cost because the installation space is limited.

As a technique for reducing the thickness of an imaging device, for example, Patent Document 1 proposes an imaging method capable of obtaining an object image without using a lens.

Further, Patent Document 2 proposes a technique for capturing an infrared image of a finger with a thin image pickup device and improving the authentication accuracy when the vein pattern detected from the infrared image is used for authentication.

JP-A-2018-061109 Japanese Unexamined Patent Publication No. 2015-0226228

According to the imaging method described in Patent Document 1, for development, the projected image obtained by passing the light of an object through a modulator composed of a pattern for imaging and incident on an image sensor is subjected to a pattern for imaging. A developed image of an object can be obtained by performing a cross-correlation calculation using the pattern of. However, since the point spread function at the time of imaging has a side lobe that vibrates, there is a problem that noise caused by the side lobe is generated in the developed image.

Further, in the technique described in Patent Document 2, noise that hinders the detection of veins is removed from the infrared image obtained by capturing the finger. The noise source here assumes wrinkles on the fingers. In order to remove noise, high frequency components are removed by Top-hat conversion. Specifically, taking advantage of the fact that skin wrinkles are often narrower than the blood vessel width, the length of the structural element is empirically adjusted to remove wrinkle components to the extent possible. However, this technique has a problem that a thin shadow of a blood vessel is erroneously determined to be a wrinkle and removed. Another problem is that the contrast of the shadows of clear blood vessels is reduced.

The present invention has been made in view of such a situation, and makes it possible to appropriately remove noise caused by a point spread function having a vibrating side lobe that may occur on an image obtained as a result of imaging. The purpose.

The present application includes a plurality of means for solving at least a part of the above problems, and examples thereof are as follows.

In order to solve the above problems, the imaging device according to one aspect of the present invention is an imaging unit that images an object and outputs a developed image in which grid noise due to a side lobe in the point spread function at the time of imaging may occur. A grid noise removing unit that removes the grid noise from the developed image based on the grid noise information is provided.

According to one aspect of the present invention, it is possible to appropriately remove noise caused by a point spread function having a vibrating side lobe, which may occur in an image obtained as a result of imaging.

Issues, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

FIG. 1 is a diagram showing a configuration example of an imaging device that is the basis of each embodiment of the present invention. FIG. 2 is a diagram showing a configuration example around the imaging pattern. FIG. 3 is a diagram showing another configuration example around the imaging pattern. FIG. 4 is a diagram showing an example of a Gabor zone plate as an imaging pattern. FIG. 5 is a diagram showing an example of a Fresnel zone plate as an imaging pattern. FIG. 6 is a diagram for explaining that the parallel light obliquely incident on the pattern substrate shifts on the image sensor. FIG. 7 is a diagram showing an example of a projected image. FIG. 8 is a diagram showing an example of a developing pattern. FIG. 9 is a diagram showing an example of a developed image by the correlation development method. FIG. 10 is a diagram showing an example of moire fringes by the moire development method. FIG. 11 is a diagram showing an example of a developed image by the moire development method. FIG. 12 is a diagram showing an example of a combination of initial phases in the fringe scan calculation process. FIG. 13 is a diagram showing an example of an imaging pattern generated by spatially combining a plurality of imaging patterns having different initial phases. FIG. 14 is a flowchart illustrating an example of the fringe scan calculation process. FIG. 15 is a flowchart illustrating an example of development processing by the correlation development method. FIG. 16 is a flowchart illustrating an example of development processing by the moire development method. FIG. 17 is a diagram for explaining that the imaging pattern is projected at the same magnification when the object is at an infinite distance. FIG. 18 is a diagram for explaining that the imaging pattern is enlarged and projected when the object is at a finite distance. FIG. 19 is a diagram showing an example of a point spread function when the image sensor is small. FIG. 20 is a diagram showing an example of a point spread function when the image sensor is large. FIG. 21 is a diagram showing an example of a single-luminance object. FIG. 22 is a diagram showing an imaging result of the object of FIG. 21. FIG. 23 is a diagram showing a one-dimensional profile of the object of FIG. 21. FIG. 24 is a diagram showing a one-dimensional profile of the imaging result of the object of FIG. 21. FIG. 25 is a diagram showing a configuration example of an imaging device according to the first embodiment of the present invention. FIG. 26 is a diagram showing a first configuration example of the grid noise information output unit. FIG. 27 is a diagram showing a second configuration example of the grid noise information output unit. FIG. 28 is a diagram showing a configuration example of an imaging device according to a second embodiment of the present invention. FIG. 29 is a diagram showing an example of an imaging pattern including a plurality of basic imaging patterns. FIG. 30 is a diagram showing a configuration example of an imaging device according to a third embodiment of the present invention. FIG. 31 is a diagram showing a first configuration example of the grid noise information storage unit. FIG. 32 is a diagram showing a second configuration example of the grid noise information storage unit. FIG. 33 is a diagram showing a configuration example of an imaging device according to a fourth embodiment of the present invention. FIG. 34 is a diagram showing a configuration example of the imaging unit of FIG. 33. FIG. 35 is a diagram showing a configuration example of an imaging device according to a fifth embodiment of the present invention. FIG. 36 is a diagram showing a configuration example of an imaging device according to a sixth embodiment of the present invention. FIG. 37 is a diagram showing a configuration example of an image pickup apparatus according to a seventh embodiment of the present invention. FIG. 38 is a diagram showing a configuration example of an image pickup apparatus according to an eighth embodiment of the present invention.

Hereinafter, a plurality of embodiments of the present invention will be described with reference to the drawings. In addition, in all the drawings for explaining each embodiment, in principle, the same members are designated by the same reference numerals, and the repeated description thereof will be omitted. In addition, in the following embodiments, the components (including element steps and the like) are not necessarily essential unless otherwise specified or clearly considered to be essential in principle. Needless to say. In addition, when saying "consisting of A", "consisting of A", "having A", and "including A", other elements are excluded unless it is clearly stated that it is only that element. It goes without saying that it is not something to do. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of a component or the like, the shape is substantially the same unless otherwise specified or, in principle, it is considered that it is not clearly the case. Etc., etc. shall be included.

<Structure example of imaging device 101>
FIG. 1 shows a configuration example of an image pickup apparatus 101 which is a basis of each embodiment of the present invention described below. The image pickup apparatus 101 includes an image pickup unit 102 and a controller 108.

The image pickup unit 102 includes an image sensor 103, a pattern substrate 104, an image pickup pattern 105, and an image processing unit 106. The imaging unit 102 images an object without using a lens for forming an optical image of the object, generates an image of the object by a predetermined development process, and outputs the captured image to the controller 108.

The controller 108 outputs the captured image input from the imaging unit 102 to an external device such as a computer or a recording device. When outputting the captured image to the external device, the controller 108 converts the data format of the captured image so as to be compatible with an interface for connecting to the external device (for example, USB (Universal Serial Bus) or the like). The controller 108 can be realized by, for example, a general computer including a CPU, a memory, a storage, and a communication interface.

Next, FIG. 2 shows a configuration example around the imaging pattern 105. The lower surface of the pattern substrate 104 is fixed in close contact with the light receiving surface of the image sensor 103, and the image pickup pattern 105 is formed on the upper surface (light incident surface) of the pattern substrate 104.

The pattern substrate 104 is made of a material that is transparent to visible light, such as glass or plastic. The imaging pattern 105 is formed by depositing a metal such as aluminum or chromium on the upper surface of the pattern substrate 104 by, for example, a sputtering method used in a semiconductor process.

The imaging pattern 105 is shaded by providing a portion where a metal such as aluminum is vapor-deposited and a portion where the metal such as aluminum is not vapor-deposited so that the transmittance of passing visible light can be modulated. Has been done. That is, the imaging pattern 105 corresponds to the modulator of the present invention.

However, the formation of the imaging pattern 105 is not limited to the vapor deposition of metal. Any means may be used as long as it can realize modulation of the transmittance of visible light, such as printing with an inkjet printer or the like.

Further, although visible light has been described as an example here, when imaging light other than visible light, the pattern substrate 104 is used as a material that transmits the light (for example, in the case of far infrared light, germanium or silicon. , Calcogenide, etc.), and the imaging pattern 105 may be formed of a material that blocks the light.

Next, FIG. 3 shows another configuration example around the imaging pattern 105. In the configuration example of FIG. 2, the imaging pattern 105 was formed on the upper surface of the pattern substrate 104. In the configuration example of FIG. 3, the imaging pattern 105 is formed of a thin film, and the thin film is held by the support member 301 at a predetermined distance from the image sensor 103.

The angle of view captured by the imaging device 101 can be changed depending on the thickness of the pattern substrate 104. Therefore, in the case of the configuration example shown in FIG. 3, the imaging angle of view can be changed by providing a mechanism capable of changing the length of the support member 301.

Pixels 103a, which are light receiving elements, are regularly arranged in a grid pattern on the surface of the image sensor 103. The image sensor 103 outputs a sensor image (hereinafter, also referred to as a projected image) composed of pixel signals corresponding to the incident light obtained as a result of photoelectric conversion by each pixel 103a to the image processing unit 106.

The image processing unit 106 generates an captured image by performing image processing such as fringe scan calculation processing and development processing on the projected image input from the image sensor 103, and outputs the captured image to the controller 108. The image processing unit 106 is realized by, for example, a dedicated image processing circuit or a computer.

<Principle of imaging by imaging device 101>
Here, the imaging principle by the imaging device 101 will be described.

First, the imaging pattern 105 will be described. The imaging pattern 105 is a pattern (pattern) composed of a plurality of concentric circles whose intervals become finer in inverse proportion to the distance from the center. The imaging pattern 105 is expressed by the following equation (1) using the center of the concentric circle as the reference coordinate, the radius r from the reference coordinate, and the coefficient β.

Since the imaging pattern 105 is used for actually transmitting light, it is inconvenient if it has a negative value, so 1 is added to the COS value to offset it. The imaging pattern 105 modulates the transmittance of transmitted light in proportion to the equation (1). Such a striped pattern consisting of a plurality of concentric circles is called a Gabor zone plate, a Fresnel zone plate, or the like.

FIG. 4 shows an example of a Gabor zone plate corresponding to the formula (1). FIG. 5 shows an example of a Fresnel zone plate in which I (r) of the formula (1) is binarized with 1 as a threshold value.

In the following, in order to simplify the explanation, only the x direction will be described using a mathematical formula, but the y direction will be expanded into two dimensions in the x direction and the y direction by processing in the same manner as in the x direction. It is possible.

Next, FIG. 6 is a diagram for explaining that the parallel light obliquely incident on the pattern substrate 104 shifts on the image sensor 103.

As shown in the figure, when parallel light is incident on the pattern substrate 104 having a thickness d on which the imaging pattern 105 is formed at an angle θ ₀ in the x direction, the refraction angle in the pattern substrate 104 is determined. If θ is set, geometrically, the light multiplied by the transmittance of the imaging pattern 105 is shifted by k = d · tan θ and incident on the image sensor 103. Then, the image sensor 103 generates a projected image having an intensity distribution represented by the following equation (2).

Φ in the equation (2) is the initial phase of the transmittance distribution of the imaging pattern 105 represented by the equation (1).

Next, FIG. 7 shows an example of a projected image obtained by passing through the imaging pattern 105 represented by the equation (1) and incident on the image sensor 103. As shown in the figure, the projected image is deviated by k as shown in the equation (2) as compared with the imaging pattern 105.

Here, the development process in the image processing unit 106 will be described. In the development process in the image processing unit 106, the development process is executed by either the correlation development method or the moire development method. FIG. 8 is an example of a developing pattern 801 used in the developing process.

First, the correlation development method will be described. In the correlation development method, a developed image is obtained by calculating a cross-correlation function between the projected image of light passing through the imaging pattern 105 (FIG. 7) and the developing pattern 801 (FIG. 8).

FIG. 9 shows an example of a developed image by the correlation development method. As shown in the figure, bright spots with a shift amount k appear in the developed image by the correlation development method.

In general, if the cross-correlation function is calculated by a two-dimensional convolution operation, the amount of calculation becomes large. Therefore, in the present embodiment, the Fourier transform is used to perform the calculation so that the amount of calculation in the calculation of the cross-correlation function is smaller. The principle of calculating the cross-correlation function using the Fourier transform will be described below.

As the developing pattern 801 (FIG. 8) for calculating the cross-correlation function with the projected image, a Gabor zone plate or a Fresnel zone plate is used as in the imaging pattern 105. Therefore, the developing pattern 801 can be expressed by the following equation (3) using the initial phase Φ.

Since the development pattern 801 is only used for calculation in the development process and is not used for transmitting light like the imaging pattern 105, it may have a negative value. Therefore, the equation (3) representing the developing pattern 801 does not need to be offset by adding 1 as in the equation (1) representing the imaging pattern 105.

The results of the Fourier transform of equations (1) and (3) are as shown in the following equations (4) and (5).

In equations (4) and (5), F represents the Fourier transform operation, and u represents the frequency coordinates in the x direction. Δ in equation (4) is a delta function. The same applies to F, u, and δ in the following equations.

What is important in the equations (4) and (5) is that even after the Fourier transform, the components representing the Fresnel zone plate and the Gabor zone plate are included. Therefore, the development pattern after the Fourier transform may be directly generated based on the equations (4) and (5). This makes it possible to reduce the amount of calculation.

Next, the following equation (6) is obtained by multiplying the equation (4) and the equation (5).

The term e ^(-iku) represented by the exponential function in the equation (6) is a signal component, and by ^{performing an} inverse Fourier transform on this term, the point spread function shown in the following equation (7) can be obtained.

Equation (7) represents a bright spot at a position deviated by k from the reference coordinates on the original x-axis, as in the developed image shown in FIG. This bright spot indicates a luminous flux at infinity, and the developed image (FIG. 9) is nothing but an image captured by the imaging unit 102.

In addition, since the term (6) also has a term other than the signal component, which becomes noise, noise cancellation is performed by fringe scan calculation processing (details will be described later).

Further, in the case of the correlation development method, the imaging pattern 105 and the developing pattern 801 are not limited to the Fresnel zone plate and the Gabor zone plate as long as the autocorrelation function of the patterns has a single peak, for example. It may be a random pattern.

Next, the moire development method will be described. In the moire development method, moire fringes are generated by multiplying the projected image of light passing through the imaging pattern 105 (FIG. 7) and the developing pattern 801 (FIG. 8), and the moire manuscript is subjected to Fourier transform. Obtain a developed image.

FIG. 10 shows an example of moire fringes obtained by multiplying the projected image of light passing through the imaging pattern 105 (FIG. 7) and the developing pattern 801 (FIG. 8). The moire fringes are represented by the following equation (8).

The third term in the expansion formula of the formula (8) is a signal component, and as shown in FIG. 10, the direction of pattern deviation between the projected image (FIG. 7) and the developing pattern 801 (FIG. 8) (drawing). It can be seen that striped patterns at equal intervals orthogonal to the lateral direction are generated in the entire region where the projected image (FIG. 7) and the developing pattern 801 (FIG. 8) overlap. Moire fringes are fringes that occur at a relatively low spatial frequency due to the superposition of such fringes.

By performing a two-dimensional Fourier transform on the third term in the expansion equation of the equation (8), the point spread function shown in the following equation (9) can be obtained.

Further, from the equation (9), it can be seen that the peak of the spatial frequency occurs at the position of u = ± kβ / π in the spatial frequency spectrum of the moire fringes.

FIG. 11 shows an example of a developed image obtained by Fourier transforming moire fringes. As shown in the figure, bright spots with a shift amount of ± kβ / π appear in the developed image by the moire development method. This bright spot indicates a luminous flux at infinity, and is nothing but an image captured by the imaging unit 102.

In addition, since the term (8) also has a term other than the signal component, which becomes noise, noise cancellation is performed by fringe scan calculation processing (details will be described later).

Further, in the case of the moire development method, the patterns of the imaging pattern 105 and the developing pattern 801 can be applied to the Fresnel zone plate or the Gabor zone plate as long as the moire fringes obtained by multiplying the two have a single frequency. The pattern is not limited, and may be, for example, an elliptical pattern.

<Noise cancellation based on fringe scan calculation>
Next, a method of canceling noise caused by terms other than the signal components in the equations (6) and (8) by the fringe scan calculation process will be described.

In order to perform the fringe scan calculation process, it is necessary to perform imaging using a plurality of imaging patterns 105 having different initial phases Φ.

FIG. 12 shows an example of four types of imaging patterns 105 having different initial phases Φ, and shows patterns 105 having initial phases Φ of 0, π / 2, π, 3π / 2 in order from the left side. ..

Four sensor images are imaged using four types of imaging patterns 105 having different initial phases Φ (0, π / 2, π, 3π / 2), and a complex number sensor image (10) is calculated according to the following equation (10). Hereinafter referred to as a complex sensor image) can be obtained.

The development pattern 801 corresponding to the complex sensor image is as shown in the following equation (11).

Although the equation (11) includes a complex number, the development pattern 801 corresponding to the complex sensor image does not actually transmit light and is only used for the fringe scan calculation process, so even if it is a complex number. no problem.

In the case of the moire development method, the equation (10) representing the complex sensor image is multiplied by the equation (11) representing the development pattern 801 to obtain the following equation (12).

Since the term exp (2iβkx) represented by the exponential function in the equation (12) is the signal component and there is no term other than the signal component as in the equation (8), it can be seen that the noise is removed.

On the other hand, in the case of the correlation development method, the equation (10) representing the complex sensor image and the development pattern 801 are Fourier transformed, respectively, to obtain the following equations (13) and (14).

Next, the following equation (15) is obtained by multiplying the equation (13) and the equation (14).

Since the term exp (-iku) represented by the exponential function in the equation (15) is the signal component and there is no term other than the signal component as in the equation (6), it can be seen that the noise is removed.

In the above description of noise cancellation by the fringe scan calculation process, four imaging patterns 105 having different initial phases Φ are used, but the number of imaging patterns 105 used for the fringe scan is not limited to four. For example, when using N (N is an integer of 2 or more) imaging patterns 105, the initial phase Φ may be set by dividing the angle [rad] between 0 and 2π into N equal parts.

In order to perform imaging using a plurality of imaging patterns 105 having different initial phases Φ, a method of switching the imaging patterns 105 by time division and a method of arranging a plurality of imaging patterns 105 having different initial phases Φ by spatial division. There is.

As a method of switching the imaging pattern 105 by time division, for example, a liquid crystal display element or the like capable of switching and displaying a plurality of imaging patterns 105 having different initial phases Φ may be used as the imaging pattern 105. Imaging is performed in synchronization with the switching timing of the liquid crystal display element and the shutter timing of the image sensor 103, and the four sensor images obtained as a result are output to the image processing unit 106, and the image processing unit 106 performs fringe scan calculation processing. Just execute.

On the other hand, in the method of arranging a plurality of imaging patterns 105 having different initial phases Φ by spatial division, a plurality of imaging patterns 105 having different initial phases Φ are spatially combined to generate one imaging pattern 105. Then, imaging may be performed using the one imaging pattern 105, and the sensor image obtained as a result may be divided into four by the image processing unit 106, and then the fringe scan calculation process may be executed.

FIG. 13 is an example of an imaging pattern 105 in which four imaging patterns having different initial phases Φ shown in FIG. 12 are spatially combined. It should be noted that the four imaging patterns having different initial phases Φ constituting one imaging pattern 105 as shown in FIG. 13 are hereinafter referred to as basic imaging patterns.

<Image processing by image processing unit 106>
Next, FIG. 14 is a flowchart illustrating an example of fringe scan calculation processing by the image processing unit 106.

The fringe scan calculation process is started, for example, when a sensor image is input from the image sensor 103 to the image processing unit 106. When a plurality of sensor images captured by using a plurality of imaging patterns 105 having different initial phases Φ are input from the image sensor 103 (hereinafter, referred to as a time-divided fringe scan), the initial phase When a single sensor image captured using one imaging pattern 105 in which a plurality of basic imaging patterns having different Φs are spatially combined is input (hereinafter, referred to as a space division fringe scan). There is.

First, in the case of the spatial division fringe scan, the image processing unit 106 divides the sensor image into a plurality of parts according to the number of basic imaging patterns forming one imaging pattern 105 (step S1). In the case of time-division fringe scan, step S1 is omitted.

Next, the image processing unit 106 initializes the complex sensor image for output (step S2). Next, the image processing unit 106 acquires the sensor image corresponding to the first initial phase Φ as the processing target (step S3), and multiplies the sensor image by exp (iΦ) corresponding to the initial phase Φ (step). S4), the multiplication result is added to the complex sensor image (step S5).

Next, the image processing unit 106 determines whether or not the sensor images corresponding to all the initial phases Φ are processed targets (step S6). Here, when it is determined that the sensor images corresponding to all the initial phases Φ are not processed (NO in step S6), the image processing unit 106 returns the processing to step S3 and repeats steps S3 to S6. .. For example, as shown in FIG. 12, when the initial phase Φ is 0, π / 2, π, 3π / 2, steps S3 to S6 are repeated four times. After that, when it is determined that the sensor images corresponding to all the initial phases Φ are the processing targets (YES in step S6), the image processing unit 106 outputs the complex sensor image for the development processing described below. (Step S7). This completes the fringe scan calculation process by the image processing unit 106.

Next, the development process by the image processing unit 106 will be described.

FIG. 15 is a flowchart illustrating an example of image processing including development processing adopting a correlation development method by the image processing unit 106.

The image processing is started in response to the input of the complex sensor image from the fringe scan calculation processing described above.

First, the image processing unit 106 performs a two-dimensional FFT (Fast Fourier Transform) calculation on the complex sensor image input from the fringe scan calculation process (step S11).

Next, the image processing unit 106 generates a development pattern 801 to be used for the development process, multiplies it by the complex sensor image after the two-dimensional FFT calculation (step S12), and executes the inverse two-dimensional FFT calculation (step S13). ). Since the calculation result in step S13 is a complex number, the image processing unit 106 generates a developed image by converting the calculation result into an absolute value or by taking out the real part and converting the image to be imaged into a real number and imaging it. (Step S14).

Next, the image processing unit 106 performs contrast enhancement processing (step S15), color balance adjustment processing (step S16), and the like on the obtained developed image, and outputs the captured image to the controller 108. This completes the image processing.

Next, FIG. 16 is a flowchart illustrating an example of image processing including development processing adopting a moire development method by the image processing unit 106.

The image processing is started in response to the input of the complex sensor image from the fringe scan calculation processing described above.

First, the image processing unit 106 generates a development pattern 801 to be used for the development process, multiplies it by the complex sensor image input from the fringe scan calculation process (step S21), and performs a two-dimensional FFT (fast Fourier transform). A frequency spectrum is obtained by performing an operation (step S22), and data in a necessary frequency domain is cut out from this frequency spectrum (step S23).

Since the data in the required frequency region is a complex number, the image processing unit 106 then converts the data into an absolute value, or takes out the real part and converts the image to be imaged into a real number and images the developed image. Is generated (step S24).

Next, the image processing unit 106 performs contrast enhancement processing (step S25), color balance adjustment processing (step S26), and the like on the obtained developed image, and then outputs the captured image to the controller 108. This completes the image processing.

<Principle of imaging finite-distance objects>
Up to this point, the description has been made on the assumption that the object to be imaged exists at an infinite distance. FIG. 17 shows how the imaging pattern 105 is projected onto the image sensor 103 when the object to be imaged exists at an infinite distance.

As shown in the figure, the spherical wave from the point 1701 constituting the object at infinity becomes a plane wave while propagating a sufficiently long distance, passes through the imaging pattern 105, and is incident on the image sensor 103. .. Therefore, the projected image 1702 obtained by the image sensor 103 has substantially the same shape as the imaging pattern 105. As a result, it is possible to obtain a single bright spot by performing a developing process on the projected image 1702 using a developing pattern.

Next, a case where the object to be imaged exists at a finite distance will be described.

FIG. 18 shows how the imaging pattern 105 is projected onto the image sensor 103 when the object to be imaged exists at a finite distance.

As shown in the figure, the spherical wave from the point 1801 constituting the object at a finite distance passes through the imaging pattern 105 in the state of the spherical wave and is incident on the image sensor 103. Therefore, in the projected image 1802 obtained by the image sensor 103, the imaging pattern 105 is enlarged substantially uniformly. The enlargement ratio α can be calculated by the following equation (16) using the distance f from the point 1801 to the imaging pattern 105.

In this way, when the object to be imaged exists at a finite distance, the projected image 1802 is a magnifying power α of the imaging pattern 105, and thus a developing pattern designed for parallel light. When the development process is performed using the above as it is, a single bright spot cannot be obtained.

In such a case, if the development pattern 801 is enlarged and used in accordance with the projected image 1802, a single brightness is obtained as in the case where the object to be imaged exists at a finite distance. You can get points.

Further, in such a case, the developing pattern 801 can be corrected by replacing the coefficient β in the equation (3) with β / α ² . As a result, the light from the point 1801 at a distance not necessarily infinity can be selectively reproduced on the developed image. As a result, the image can be focused on an arbitrary position.

<Generation of grid noise>
In the above description, an ideal situation is assumed in which the size of the image sensor 103 is infinitely large. Therefore, it was assumed that the projected image of the imaging pattern 105 also continued infinitely to the outside.

However, since the size of the image sensor 103 is finite, a realistic situation in which the size of the image sensor 103 is finite will be described below.

When the size of the image sensor 103 is finite, the point spread function represented by the equations (7) and (9) changes, and grid noise is generated in the developed image. The grid noise generated in the developed image will be described below.

The generation of grid noise in the developed image depends on the design value of the imaging unit 102. First, it will be described that the point spread function changes when the size of the image sensor 103 is finite.

When the size of the image sensor 103 is finite, the size of the projected image detected on the image sensor 103 is also finite. When the image sensor 103 has a width from −s to + s in the x direction, the equation (2) representing the projected image of the imaging pattern 105 is as shown in the following equation (17).

The rectangular function rect (x) in the equation (17) is as shown in the following equation (18).

In this case, as shown in FIG. 6, a point spread representing a developed image when parallel light is incident on a pattern substrate 104 having a thickness d on which an imaging pattern 105 is formed at an angle θ ₀ in the x direction. The function changes from equation (7) to equation (19).

That is, the point spread function changes from the δ function to sin (2βs (x + k)) / (2βs (x + k)) · cos (β (x + k) ² ), which is a function having a side lobe. Here, the side lobe refers to a remarkable vibration component that appears at least once in addition to the main lobe.

19 and 20 show an example of a point spread function having a side lobe when the size of the image sensor 103 is different, and FIG. 19 shows a point spread corresponding to the smaller size of the image sensor 103. It is a function, and FIG. 20 is a point spread function corresponding to the larger size of the image sensor 103.

As can be seen from FIGS. 19 and 20, the smaller the size of the image sensor 103, the slower the convergence of the vibration of the side lobe. On the other hand, the smaller the size of the image sensor 103, the wider the main lobe. Although not shown, the coarser the imaging pattern 105 of the side lobe, the slower the convergence of the vibration. On the other hand, the coarser the imaging pattern 105, the wider the main lobe.

Hereinafter, the point spread function when k = 0 in the equation (19) is referred to as PSF (x).

Next, it will be described that the point spread function has side lobes and grid noise is generated in the developed image under predetermined conditions.

First, consider the case of imaging a quadrangular object having a single brightness (for example, white). FIG. 21 shows an example of a single-luminance quadrangular object. When the object has a width of 2 L in the x direction and the object is represented by a rectangular function rect (x / 2 L), the developed image r (x) is represented by the following equation (20). Here, the developed image r (x) obtained by imaging and photographing a quadrangular object having a single brightness is particularly referred to as a grid (x).

FIG. 22 is a developed image obtained by imaging the object shown in FIG. 21. FIG. 23 is a one-dimensional profile of the single-luminance object shown in FIG. FIG. 24 is a one-dimensional profile of the developed image shown in FIG.

As can be seen from FIG. 24, the influence of the vibration of the side lobe of the point spread function becomes remarkable around the edge portion of the object whose brightness changes rapidly. Specifically, dark lines and bright lines are repeated at a distance from the edge determined by the shape of the side lobe. This is grid noise.

On the other hand, when the object does not have a single brightness, the noise generated changes in thickness and appearance position, and is not grid-like. Expressed by a mathematical formula, when the object is not a single luminance and its luminance distribution is a function f (x), the developed image r (x) is expressed by the following equation (21).

However, if the object has a brightness distribution close to a single brightness, the position of the grid noise can be approximated by calculation. When the object has a luminance distribution close to a single luminance, the function f (x) representing the luminance distribution is expressed by the following equation (22) using a minute amount h.

O (x) in the equation (22) is obtained by subtracting the constant component from the luminance distribution of the object. The developed image r (x) corresponding to this is expressed by the following equation (23).

Grid noise is predominant in the developed image r (x) represented by the equation (23), and its appearance is represented by grid (x). As described above, when the object has a brightness distribution close to a single brightness, grid noise appears at a fixed position depending on the design value of the imaging unit 102.

Next, an image pickup device capable of removing grid noise generated in the developed image will be described.

<Structure example of the image pickup apparatus 101 ₁ according to the first embodiment of the present invention>
FIG. 25 shows a configuration example of the image pickup apparatus 1011 according to the _first embodiment of the present invention. The image pickup apparatus 101 ₁ is an image pickup apparatus 101 (FIG. 1) in which a grid noise removal unit 107 and a grid noise information output unit 1071 are added between the image pickup unit 102 and the controller 108. Of the components of the image pickup device 101 ₁ , those that are common to the components of the image pickup device 101 are designated by the same reference numerals, and the description thereof will be omitted.

The grid noise removing unit 107 removes the grid noise generated in the developed image r (x) input from the image processing unit 106 based on the grid noise information grid (x), and outputs the grid noise to the controller 108. The grid noise removing unit 107 is realized by, for example, a dedicated image processing circuit or a computer.

The grid noise information output unit 1071 generates grid noise information grid (x) based on information input from the outside and outputs it to the grid noise removal unit 107.

Next, FIG 26 shows a first configuration example of a grid noise information output section 1071 in the image pickup apparatus 101 _1.

The first configuration example of the grid noise information output unit 1071 includes an imaging unit design value input unit 1072 and a grid noise information calculation unit 1073.

The imaging unit design value input unit 1072 comprises, for example, an input interface such as a USB port, and the imaging unit design value input from the outside (keyboard, semiconductor memory, PC (Personal Computer), etc.) is input to the grid noise information calculation unit 1073. Output.

Here, the image pickup unit design values include, for example, the size of the image sensor 103, the image pickup pattern information, the development pattern information, the field of view size, the distance f between the object and the image pickup pattern 105, and the image pickup pattern 105. It is assumed that the distance d and the like with the image sensor 103 are included.

The grid noise information calculation unit 1073 calculates the grid noise information grid (x) required for grid noise removal according to the equation (20) based on the image pickup unit design value input from the image pickup unit design value input unit 1072, and grid noise. Output to the removal unit 107.

As shown in the following equation (24), the grid noise removing unit 107 divides the developed image r (x) by the grid noise information grid (x) to obtain the developed image r'(x) after removing the grid noise. Calculate.

As is clear from the comparison between the equation (23) representing the developed image r (x) before removing the grid noise and the equation (24), the equation (23) was (grid noise + object). , Equation (24) is (1 + an object whose brightness is slightly changed), and it can be seen from Equation (24) that grid noise is removed.

As shown in FIG. 24, the rate of variation in the brightness of the grid noise in the field of view (dispersion of the brightness with respect to the absolute value of the brightness) is sufficiently smaller than 1, so that the amount of change in the brightness of the object is also sufficiently small. ..

Therefore, for an object represented by the equation (22), the effect of grid noise removal exceeds the effect of changing the brightness of the object, so that the grid noise removal according to the present embodiment works effectively.

In particular, for example, in the case of detecting a human finger vein, the grid noise removal by the present embodiment is used as a means for suppressing erroneous detection of the finger vein due to the grid noise generated in the infrared image of the human finger. Very effective. This is because the infrared image of the finger shows several veins with inconspicuous edges in a background with almost single brightness.

In the grid noise removal according to the present embodiment, the grid noise information calculated on the assumption of a single-luminance object can also be used for grid noise removal of a more general object.

However, unlike a single brightness, such as a checkerboard pattern, grid noise removal according to the present embodiment cannot be applied to an object not represented by the equation (22). When the grid noise removal according to the present embodiment is performed on the developed image obtained by capturing such an object, extra noise due to the grid noise removal will be further generated.

However, this extra noise that can occur in the developed image of the object can be used to determine whether or not the grid noise removal according to the present embodiment is used. That is, when a plurality of checkered patterns having different grid sizes are imaged and noise is generated in the developed image, it can be determined that the grid noise removal according to the present embodiment is used. It is also possible to use an object whose brightness gradually changes at the edge of the object, such as a two-dimensional normal distribution, instead of the checkered pattern.

Next, FIG. 27 shows a second configuration example of the grid noise information output section 1071 in the image pickup apparatus 101 _1.

A second configuration example of the grid noise information output unit 1071 includes an actual measurement image pickup result input unit 1074 and a grid noise information extraction unit 1075.

The actual measurement imaging result input unit 1074 comprises, for example, an input interface such as a USB port, and outputs the actual measurement imaging result input from the outside (semiconductor memory, PC, etc.) to the grid noise information extraction unit 1075. The actual measurement imaging result is a developed image in which grid noise is generated.

The grid noise information extraction unit 1075 extracts the grid noise information grid (x) from the actual measurement imaging result and outputs it to the grid noise removal unit 107. Specifically, the grid noise information extraction unit 1075 extracts the grid noise information grid (x) from the developed image as the actual measurement imaging result by using the built-in high-pass filter. Further, when the signal of the actually measured imaging result is weak, a process for smoothing the actually measured imaging result is required.

Here, as the simplest example of the actual measurement imaging result, a developed image obtained by imaging an object having a single brightness can be mentioned. Since this developed image directly corresponds to the grid noise information grid (x), the grid noise information extraction unit 1075 does not perform any particular processing, and the input developed image is used as the grid noise information grid (x) as it is in the grid noise removal unit. Output to 107.

However, the object to be imaged in order to obtain the actual measurement imaging result does not have to be limited to a single-luminance object. The grid noise information grid (x) is obtained by inputting the result of imaging an arbitrary known object into the actual measurement imaging result input unit 1074 and comparing the imaging result with the known object information by the grid noise information extraction unit 1075. It is also possible. Specifically, the point spread function PSF (x) is calculated from the imaging result and known object information, and the grid noise information grid (x) is calculated according to the equation (20). If necessary, a high-pass filter or smoothing is applied to the grid noise information grid (x).

Comparing the first configuration example (FIG. 26) and the second configuration example (FIG. 27) of the grid noise information output unit 1071, if the discrepancy between the theory and the actual machine can be ignored, it is affected by noise or the like. No Grid denoising based on the calculation by the first configuration is effective. On the contrary, when the discrepancy between the theory and the actual machine cannot be ignored, it is effective to remove the grid noise according to the reality by the second configuration example, although it is affected by noise and the like.

<Structure example of the image pickup apparatus 1012 according to the _second embodiment of the present invention>
Next, FIG. 28 shows a configuration example of the image pickup apparatus 1012 according to the _second embodiment of the present invention. The image pickup device 101 ₂ replaces the image pickup unit 102 in the image pickup device 101 ₁ (FIG. 25) with the image pickup unit 102 ₂ . Of the components of the image pickup device 101 ₂ , those that are common to the components of the image pickup device 101 and the image pickup device 101 ₁ are designated by the same reference numerals, and the description thereof will be omitted.

The image pickup unit 102 ₂ has the same size of the image pickup pattern 105, the size of the image sensor 103, and the distance between the image pickup pattern 105 and the image sensor 103 as compared with the image pickup unit 102 (FIG. 1). However, the difference is that the imaging pattern 105 is composed of a plurality of basic imaging patterns 1051, and the image sensor 103 is composed of a plurality of small image sensor units 1031.

With such a configuration, the imaging unit 102 ₂ can image a wider field of view as compared with the imaging unit 102.

FIG. 29 shows an example of an imaging pattern 105 composed of a plurality of basic imaging patterns 1051 adopted in the imaging unit 102 ₂ . In the case of the figure, the imaging pattern 105 is composed of nine basic imaging patterns 1051 having the same initial phase Φ, but the number of basic imaging patterns 1051 forming the imaging pattern 105 is not limited to nine.

Further, in the image pickup unit 102 ₂ , both the image pickup pattern 105 and the image sensor 103 do not necessarily have to be divided into a plurality of parts. For example, an imaging pattern 105 composed of one basic imaging pattern 1051 and an image sensor 103 composed of a plurality of small image sensor units 1031 may be combined, or an imaging pattern 105 composed of a plurality of basic imaging patterns 1051 may be combined. It may be combined with an image sensor 103 that is not subdivided.

<Structure example of the image pickup apparatus 101 ₃ according to the third embodiment of the present invention>
Next, FIG. 30 shows a configuration example of the image pickup apparatus 1013 according to the _third embodiment of the present invention.

In the image pickup apparatus 101 ₁ (FIG. 25), the grid noise information grid (x) generated based on the information input from the outside of the image pickup apparatus 101 ₁ is input to the grid noise removal unit 107. In contrast, the imaging apparatus 101 _3, grid noise removing unit 107 has a built-in grid noise information storage unit 1076 in advance (during manufacture) holding grid noise information grid (x). The grid noise information storage unit 1076 outputs the grid noise information grid (x) to be held to the grid noise removal unit 107. The grid noise information storage unit 1076 corresponds to the grid noise information output unit of the present invention.

Of the components of the image pickup device 101 ₃ , those that are common to the components of the image pickup device 101 and the image pickup device 101 ₁ are designated by the same reference numerals, and the description thereof will be omitted.

For imaging apparatus 101 _3, grid noise removal unit 107 can remove the grid noise by using the grid noise information grid (x) of the grid noise information storage unit 1076 holds.

Next, FIG 31 shows a first configuration example of a grid noise information storage unit 1076 in the image pickup apparatus 101 _3.

The first configuration example of the grid noise information storage unit 1076 includes an imaging unit design value storage unit 1077 and a grid noise information calculation unit 1073. The imaging unit design value storage unit 1077 outputs the imaging unit design value stored in advance (during manufacturing, etc.) to the grid noise information calculation unit 1073. The grid noise information calculation unit 1073 calculates and holds the grid noise information grid (x) based on the image pickup unit design value from the image pickup unit design value storage unit 1077, and outputs the grid noise information grid (x) to the grid noise removal unit 107 as appropriate.

Next, FIG. 32 shows a second configuration example of a grid noise information storage unit 1076 in the image pickup apparatus 101 _3.

A second configuration example of the grid noise information storage unit 1076 includes an actually measured imaging result storage unit 1078 and a grid noise information extraction unit 1075. The actual measurement imaging result storage unit 1078 outputs the actual measurement imaging result stored in advance (during manufacturing, etc.) to the grid noise information extraction unit 1075. The grid noise information extraction unit 1075 extracts and holds the grid noise information grid (x) from the actual measurement imaging result, and outputs the grid noise information grid (x) to the grid noise removal unit 107 as appropriate.

<Structure example of the image pickup apparatus 1014 according to the _fourth embodiment of the present invention>
Next, FIG. 33 shows a configuration example of the image pickup apparatus 1014 according to the _fourth embodiment of the present invention.

The imaging units 102 and 102 ₂ in the first to third embodiments have a configuration that does not include a lens. However, the grid noise removal according to the present embodiment effectively acts on the grid noise generated in the developed image when the point spread function has a side lobe regardless of the presence or absence of the lens in the imaging unit.

Therefore, the imaging unit 102 ₄ in the image pickup apparatus 101 _4, without limiting the presence or absence of the lens is defined as the point spread function has sidelobes. Of the components of the image pickup device 101 ₄ , those that are common to the components of the image pickup device 101 _{1 and the} like are designated by the same reference numerals, and the description thereof will be omitted.

Figure 34 illustrates a configuration example of the imaging unit 102 ₄ in the image pickup apparatus 101 _4. The imaging unit 102 ₄ includes an organic EL (Electro Luminescence) display 1021 and a camera module 1022. The camera module 1022 includes a lens, an image sensor, and an image processing unit (all not shown). The camera module 1022 itself does not generate side lobes in the point spread function (including cases where it occurs to a negligible extent).

Configuration of the imaging unit 102 ₄ shown in the figure, for example, mounted on a smartphone can be applied in the case of imaging an object (fingerprint of the user) with the rear surface of the display surface of the display.

In the imaging unit 102 ₄ shown in the figure, the camera in the module 1022 point spread function sidelobes but does not occur, the influence of diffraction due to the organic EL display 1021, an organic EL display 1021 a camera module 1022 is light that has passed through the Side lobes occur in the point spread function at the stage imaged by.

That is, the imaging unit 102 ₄ includes the entire configuration in which a transformant (in the case of the figure, the organic EL display 1021) capable of generating a side lobe in the point spread function is arranged between the camera module 1022 and the object. The imaging unit 102 ₄ outputs an image in which grid noise is generated due to the side lobe of the point spread function to the grid noise removing unit 107.

The configuration after the imaging unit 102 ₄ in the image pickup apparatus 101 ₄ may be the same as the image pickup device 101 ₁ (Fig. 25) As shown in FIG. 33, the imaging apparatus 101 ₃ (Figure 30) It may be similar to.

By the way, in the above-described first to fourth embodiments, since the imaging unit 102 and the subsequent configurations (grid noise removing unit 107 and controller 108) are integrated, the size of the entire imaging device is large. .. In addition, cost and power consumption may increase.

Hereinafter, embodiments that can be reduced in size will be described.

<Structure example of the image pickup apparatus 1015 according to the _fifth embodiment of the present invention>
Next, FIG. 35 shows a configuration example of the image pickup apparatus 1015 according to the _fifth embodiment of the present invention. Among the components of the imaging apparatus 101 _5, the same reference numerals for those that are common to the components of the first to fourth embodiments described above, description thereof will be omitted.

The image pickup apparatus 101 ₅ has a configuration in which the components included in the image pickup apparatus 101 ₁ (FIG. 25) are divided into a transmission unit 2001 and a reception unit 2002, and the grid noise information output unit 1071 is arranged on the transmission unit 2001 side. Imaging device 101 _5, the transmission unit 2001 and reception unit 2002, each imaging system of the present invention, the transmitting apparatus, and corresponds to the receiving device.

Transmission unit in the image pickup 101 ₅ 2001 includes an imaging unit 102, grid noise information output unit 1071, and a data transmission unit 2003. Receiving unit 2002 in the image pickup apparatus 101 ₅ includes a grid noise removing unit 107, the controller 108, and a data receiving unit 2004.

The data transmission unit 2003 transmits the developed image r (x) input from the imaging unit 102 and the grid noise information grid (x) input from the grid noise information output unit 1071 to the data reception unit 2004. The data receiving unit 2004 outputs the received developed image r (x) and the grid noise information grid (x) to the grid noise removing unit 107.

Data communication between the data transmitting unit 2003 and the data receiving unit 2004 is performed according to a predetermined wired communication standard or a predetermined wireless communication standard. A recording medium may be used for data communication between the data transmission unit 2003 and the data reception unit 2004.

Further, if the data transmitting unit 2003 and the data receiving unit 2004 can be connected via a bidirectional communication network represented by the Internet, for example, the receiving unit 2002 may be realized by a server on a so-called cloud network. May be good. In that case, one receiving unit 2002 may perform grid noise removal corresponding to a plurality of transmitting units 2001.

According to the imaging apparatus 101 _5, compared to the size of the imaging device 101 ₁ (FIG. 25), since the size of the transmission unit 2001, which includes the imaging unit 102 can be made smaller and lighter, increasing the degree of freedom in the arrangement of the transmission unit 2001 be able to.

<Structure example of the image pickup apparatus 1016 according to the _sixth embodiment of the present invention>
Next, FIG. 36 shows a configuration example of the image pickup apparatus 1016 according to the _sixth embodiment of the present invention. Among the components of the imaging apparatus 101 _6, the same reference numerals for those that are common to the components of the first to fifth embodiments described above, description thereof will be omitted.

Imaging device 101 _6, the components included in the imaging apparatus 101 ₃ (Figure 30), it is divided into a transmission unit 2001 and reception unit 2002 has a configuration of arranging the grid noise information storage unit 1076 in the transmitting unit 2001 side. Imaging device 101 ₆ corresponds to the imaging system of the present invention.

According to the imaging apparatus 101 _6, compared to the size of the imaging device 101 ₃ (Figure 30), since the size of the transmission unit 2001, which includes the imaging unit 102 can be made smaller and lighter, increasing the degree of freedom in the arrangement of the transmission unit 2001 be able to.

<Structure example of the image pickup apparatus 1017 according to the _seventh embodiment of the present invention>
Next, FIG. 37 shows a configuration example of the image pickup apparatus 1017 according to the _seventh embodiment of the present invention. Among components of the imaging apparatus 101 _7, the same reference numerals for those that are common to the components of the first to sixth embodiments described above, description thereof will be omitted.

The image pickup apparatus _{10 1 7} has a configuration in which the components included in the image pickup apparatus 101 ₁ (FIG. 25) are divided into a transmission unit 2001 and a reception unit 2002, and the grid noise information output unit 1071 is arranged on the reception unit 2002 side. Imaging device 101 ₇ corresponds to the imaging system of the present invention.

Transmission unit 2001 in the image pickup apparatus 101 ₇ includes an imaging unit 102, and a data transmission unit 2003,. Receiving unit 2002 in the image pickup apparatus 101 ₅ includes a grid noise removing unit 107, the controller 108, the grid noise information output unit 1071, and a data receiving unit 2004.

Data transmission unit in the imaging apparatus 101 ₇ 2003 transmits developed image r input from the imaging unit 102 (x) to the data reception unit 2004. The data receiving unit 2004 outputs the received developed image r (x) to the grid noise removing unit 107.

According to the imaging apparatus 101 _7, compared with the imaging device 101 ₅ (Fig. 35), by the amount of the grid noise information output section 1071, since the size of the transmission unit 2001, which includes the imaging unit 102 can be made smaller and lighter, the transmission unit The degree of freedom of arrangement of 2001 can be increased.

<Structure example of the image pickup apparatus 1018 according to the _eighth embodiment of the present invention>
Next, FIG. 38 shows a configuration example of the image pickup apparatus 1018 according to the _eighth embodiment of the present invention. Among the components of the imaging apparatus 101 _8, the same reference numerals for those that are common to the components of the first to seventh embodiments described above, description thereof will be omitted.

Imaging apparatus 101 ₈ has a configuration in which the components included in the imaging apparatus 101 ₃ (Figure 30), it is divided into a transmission portion 2001 and receiving portion 2002, were placed grid noise information storage unit 1076 in the reception unit 2002 side. Imaging apparatus 101 ₈ corresponds to the imaging system of the present invention.

According to the imaging apparatus 101 _8, compared with the imaging apparatus 101 ₆ (FIG. 36), since the size of the transmission unit 2001, which includes the imaging unit 102 can be made smaller and lighter, it is possible to increase the degree of freedom in the arrangement of the transmission unit 2001 it can.

The present invention is not limited to the above-described embodiment, and further includes various modifications. In addition, each of the above-described embodiments has been described in detail in order to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to those including all the components described above. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

In addition, the control lines and information lines indicate those that are considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

101, 101 ₁ to 101 ₈ ... imaging _apparatus, 102, 102 2, ₁₀₂ 4, ... imaging unit, 103 ... image sensor, 104 ... patterned substrate, 105 ... imaging patterns, 1051, ... Basic imaging pattern, 106 ... Image processing unit, 107 ... Grid noise removal unit, 108 ... Controller, 301 ... Support member, 801 ... Development pattern, 1021 ... Organic EL display, 1022 ... camera module, 1031 ... small image sensor unit, 1071 ... grid noise information output unit, 1072 ... imaging unit design value input unit, 1073 ... grid noise information calculation unit, 1074 ... Actual measurement imaging result input unit, 1075 ... Grid noise information extraction unit, 1076 ... Grid noise information storage unit, 2001 ... Transmission unit, 2002 ... Reception unit, 2003 ... Data transmission Unit, 2004 ... Data receiving unit

Claims (19)

An imaging unit that images an object and outputs an image in which grid noise due to side lobes in the point spread function during imaging may occur.
A grid noise removing unit that removes grid noise from the image based on the grid noise information,
An imaging device characterized by comprising. The imaging device according to claim 1.
A grid noise information output unit that outputs the grid noise information to the grid noise removal unit,
An imaging device characterized by comprising. The imaging device according to claim 1.
The imaging unit
A modulator consisting of an imaging pattern including at least one basic imaging pattern and modulating the intensity of incident light,
An image sensor that generates a projected image by converting the light that has passed through the modulator and incident light into an electrical signal.
An image processing unit that generates the image by a development process using the projected image and a development pattern corresponding to the image pickup pattern.
An imaging device characterized by having. The imaging device according to claim 3.
The image processing unit is an image pickup apparatus characterized in that noise cancellation is performed by a fringe scan calculation process using a plurality of projection images that have passed through a plurality of the imaging patterns having different initial phases. The imaging device according to claim 3.
The image sensor is an image pickup apparatus including a plurality of small image sensor units. The imaging device according to claim 1.
The imaging unit
An intermediate that causes a side lobe in the point spread function,
A camera module that generates the image based on the incident light that has passed through the intermediate.
An imaging device characterized by having. The imaging device according to claim 6.
The intermediate is an image pickup apparatus characterized by being an organic EL display. The imaging device according to claim 2.
The grid noise information output unit
An imaging unit design value input unit that inputs an imaging unit design value related to the imaging unit from the outside,
A grid noise information calculation unit that calculates the grid noise information based on the image pickup unit design value,
An imaging device characterized by having. The imaging device according to claim 2.
The grid noise information output unit
An actual measurement result input unit that inputs an actual measurement image result of actually imaging the object from the outside,
A grid noise information extraction unit that extracts the grid noise information from the actual measurement imaging result,
An imaging device characterized by having. The imaging device according to claim 2.
The grid noise information output unit
A grid noise information calculation unit that calculates the grid noise information based on an imaging unit design value related to the imaging unit, which is held in advance.
An imaging device characterized by having. The imaging device according to claim 2.
The grid noise information output unit
A grid noise information extraction unit that extracts the grid noise information from the actual measurement imaging result of actually imaging the object, which is held in advance.
An imaging device characterized by having. It is an imaging method using an imaging device.
An imaging step that images an object and outputs an image in which grid noise due to side lobes in the point spread function during imaging may occur.
A grid noise removal step for removing the grid noise from the image based on the grid noise information,
An imaging method comprising. The transmitting device in an imaging system including a transmitting device and a receiving device.
An imaging unit that images an object and outputs an image in which grid noise due to side lobes in the point spread function during imaging may occur.
A data transmitting unit that transmits the grid noise information and the image to the receiving device having a grid noise removing unit that removes the grid noise from the image based on the grid noise information.
A grid noise information output unit that outputs the grid noise information to the data transmission unit,
A transmitter characterized by comprising. The transmitting device according to claim 13.
The grid noise information output unit is a transmission device characterized in that the grid noise information is generated based on information input from the outside and output to the data transmission unit. The transmitting device according to claim 13.
The grid noise information output unit is a transmission device characterized in that the grid noise information is generated based on the information held in advance and output to the data transmission unit. The receiving device in an imaging system including a transmitting device and a receiving device.
An imaging unit that images an object and outputs an image in which grid noise due to side lobes in the point spread function during imaging may occur, a data transmission unit that transmits grid noise information and the image, and the grid noise. A grid noise information output unit that outputs information to the data transmission unit, and a data reception unit that receives the grid noise information and the image transmitted from a transmission device including the data transmission unit.
A grid noise removing unit that removes the grid noise from the image based on the received grid noise information,
A receiving device characterized by comprising. The receiving device in an imaging system including a transmitting device and a receiving device.
It is transmitted from a transmission device including an image pickup unit that images an object and outputs an image in which grid noise due to a side lobe in the point spread function at the time of imaging may be generated, and a data transmission unit that transmits the image. The data receiving unit that receives the image and
A grid noise removing unit that removes the grid noise from the image based on the grid noise information,
A grid noise information output unit that outputs the grid noise information to the grid noise removal unit,
A receiving device characterized by comprising. The receiving device according to claim 17.
The grid noise information output unit is a receiving device characterized in that the grid noise information is generated based on information input from the outside and output to the grid noise removing unit. The receiving device according to claim 17.
The receiving device is characterized in that the grid noise information output unit generates the grid noise information based on the information held in advance and outputs the grid noise information to the grid noise removing unit.

Images