WO2020059181A1 - Imaging device and imaging method - Google Patents

Abstract

Provided is technology for obtaining accurate developed images by correcting imaging device assembly errors, photographing pattern manufacturing errors, and focus adjustment distortions. An imaging device characterized by comprising: an image sensor for converting light into electrical signals and generating sensor images; a modulator for modulating the intensity of the light detected by the image sensor on the basis of a photographing pattern; and a parameter storage unit for storing parameters used to execute a prescribed correction process upon a plurality of sensor images, from among said sensor images, that have been photographed using different photographing patterns.

Description

Imaging device and imaging method

The present invention relates to an imaging device and an imaging method. The present invention claims the priority of Japanese Patent Application No. 2018-173930 filed on September 18, 2018, and the contents described in the application for the designated countries in which weaving by reference to a document is permitted. Incorporated by reference into this application.

背景 As a background art of this technical field, there is JP-A-2018-61109 (Patent Document 1). This publication states that “a modulator having a first pattern and modulating the intensity of light, an image sensor that converts light transmitted through the modulator into image data and outputs the image data, And an image processing unit for restoring an image based on a cross-correlation operation with pattern data indicating the second pattern. There is a description of a technique for noise-cancelling an image obtained by “space-dividing the image data” and “dividing the image data output from the image sensor into regions corresponding to the pattern arrangement of the modulator”.

JP 2018-61109 A

In the technique described in Patent Document 1, patterns having different initial phases are arranged with a shift (for example, among patterns A, B, C, and D, the center position of pattern B is shifted, and pattern C is rotated. If the upper surface of the substrate is not parallel to the sensor surface (for example, is inclined), the projection pattern received by the image sensor will be displaced or distorted, and the image will not be developed. There is a problem that a correct developed image cannot be obtained.

An object of the present invention is to provide a technique for obtaining a correct developed image by correcting an assembly error of an imaging device, a manufacturing error of a photographing pattern, and distortion of focus adjustment.

Although the present application includes a plurality of means for solving at least a part of the above-described problems, examples thereof are as follows. In order to solve the above problem, an imaging device according to one embodiment of the present invention converts an light into an electric signal, generates an image of a sensor, and an image sensor that detects the light based on an imaging pattern. A modulator that modulates intensity; and a parameter storage unit that stores a parameter used to execute a predetermined correction process on a plurality of the sensor images captured by the different capturing patterns among the sensor images.

According to the present invention, it is possible to provide a technique for obtaining a correct developed image by correcting an assembling error of the imaging device, a manufacturing error of a photographing pattern, and a distortion of focus adjustment. Problems, configurations, and effects other than those described above will be apparent from the following description of the embodiments.

FIG. 1 is a diagram illustrating a configuration example of an imaging device according to a first embodiment of the present invention. FIG. 3 is a diagram illustrating a configuration example of an imaging module according to the first embodiment. FIG. 4 is a diagram illustrating a configuration example of another imaging module according to the first embodiment. FIG. 4 is a diagram illustrating an example of a photographing pattern and a developing pattern according to the first embodiment. FIG. 6 is a diagram illustrating another example of the photographing pattern and the developing pattern according to the first embodiment. FIG. 9 is a diagram illustrating an example in which an in-plane shift occurs in a projected image from a pattern substrate surface to an image sensor due to obliquely incident parallel light. FIG. 3 is a diagram illustrating an example of a projected image of a photographing pattern. FIG. 4 is a diagram illustrating an example of a development pattern. FIG. 6 is a diagram illustrating an example of a developed image by a correlation development method. It is a figure showing an example of a moire fringe by a moire development system. FIG. 4 is a diagram illustrating an example of a developed image by a moiré developing method. FIG. 9 is a diagram illustrating an example of a combination of imaging patterns of an initial phase in a fringe scan. FIG. 4 is a diagram illustrating an example of a shooting pattern of a space division fringe scan. FIG. 9 is a diagram illustrating an example of a processing flow of fringe scan. FIG. 7 is a diagram illustrating an example of a processing flow of a development process by a correlation development method. FIG. 4 is a diagram illustrating an example of a processing flow of a development process by a moire development method. FIG. 4 is a diagram illustrating an example of projection of a shooting pattern when an object is at an infinite distance. FIG. 9 is a diagram illustrating an example of an enlarged projection of a photographing pattern when an object is at a finite distance. FIG. 6 is a diagram illustrating a configuration example of an imaging device according to a second embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a positional relationship between a point light source and an imaging device. It is a figure showing the example of the sensor image according to the distance of the point light source. FIG. 9 is a diagram illustrating an example of pattern center coordinates when a point light source is at infinity. FIG. 9 is a diagram illustrating an example of pattern center coordinates when a point light source is at a finite distance. FIG. 9 is a diagram illustrating an example of correcting a sensor image when a point light source is at a finite distance. FIG. 5 is a diagram illustrating an example of a sensor image when a shooting pattern is rotating. FIG. 9 is a diagram illustrating an example of pattern center coordinates when a shooting pattern is rotating. It is a figure showing the example of the state where the pattern for photography is inclined in the thickness direction. It is a figure showing the example of the sensor image of the state where the pattern for photography was inclined in the thickness direction. FIG. 4 is a diagram illustrating an example of pattern center coordinates in a state where a shooting pattern is inclined in a thickness direction. FIG. 9 is a diagram illustrating a configuration example of an imaging device according to a third embodiment of the present invention. FIG. 14 is a diagram illustrating a configuration example of an imaging device according to a modification of the third embodiment of the present invention. FIG. 9 is a diagram illustrating an example of a processing flow of calibration. FIG. 11 is a diagram illustrating an example of a sensor image distortion correction processing flow. FIG. 14 is a diagram illustrating a configuration example of an imaging device according to a fourth embodiment of the present invention. FIG. 7 is a diagram illustrating an example of a distortion correction processing flow of a development pattern. It is a figure showing the example of composition of the imaging device concerning a 5th example of the present invention. FIG. 9 is a diagram illustrating an example of an image when dirt or the like is attached to a shooting pattern. It is a figure showing an example of a luminance distribution of an ideal sensor image. FIG. 7 is a diagram illustrating an example of a luminance distribution of a defective sensor image. FIG. 9 is a diagram illustrating an example of defect detection by inversion pattern synthesis.

In the following embodiments, when necessary for the sake of convenience, the description will be made by dividing into a plurality of sections or embodiments, but unless otherwise specified, they are not unrelated to each other, and one is the other. Of some or all of the above, details, supplementary explanations, and the like.

Further, in the following embodiments, when referring to the number of elements (including the number, numerical value, amount, range, etc.), a case where it is particularly specified and a case where it is clearly limited to a specific number in principle, etc. Except, the number is not limited to the specific number, and may be more than or less than the specific number.

Further, in the following embodiments, the constituent elements (including element steps, etc.) are not necessarily essential unless otherwise specified or considered to be essential in principle. Needless to say.

Similarly, in the following embodiments, when referring to the shapes, positional relationships, and the like of the components, the shapes are substantially the same, unless otherwise specified, and in cases where it is considered that it is not clearly apparent in principle. And the like. This is the same for the above numerical values and ranges.

In addition, in all the drawings for describing the embodiments, the same members are denoted by the same reference numerals in principle, and the repeated description thereof will be omitted. Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Generally, digital cameras mounted on information devices such as in-vehicle cameras, wearable devices, and smartphones are often required to be thinner and lower in cost. For example, an imaging method has been proposed which realizes a thinner and lower cost by obtaining an object image without using a lens. In such a technique, an imaging method for obtaining an image of an object by pasting a special lattice pattern in front of an image sensor and solving an inverse problem for developing an image from a projection pattern received by the image sensor. There is. In this method, the operation for solving the inverse problem by signal processing becomes complicated, and the processing load is high, so that the hardware requirement specification of the information equipment is increased.

FIG. 1 is a diagram showing a configuration example of an imaging apparatus according to a first embodiment of the present invention. The image capturing apparatus 101 acquires an image of an external object without using a lens to form an image. As shown in FIG. 1, the image capturing module 102, a fringe scan processing unit 106, an image processing unit 107, and a controller 108 It is composed of FIG. 2 shows an example of the imaging module 102.

FIG. 2 is a diagram illustrating a configuration of the imaging module according to the first embodiment. The imaging module 102 includes an image sensor 103, a pattern substrate 104, and an imaging pattern 105. The pattern substrate 104 is fixed in close contact with the light receiving surface of the image sensor 103, and a pattern 105 for photographing is formed on the pattern substrate 104. The pattern substrate 104 is made of a material that is transparent to visible light, such as glass or plastic.

(4) The photographing pattern 105 is a concentric lattice pattern in which the distance between the lattice patterns, that is, the pitch, is reduced in inverse proportion to the radius from the center toward the outside. The imaging pattern 105 is formed by depositing a metal such as aluminum or chromium by, for example, a sputtering method used in a semiconductor process. Shading is given by the pattern with and without the metal deposited. Note that the formation of the photographing pattern 105 is not limited to this, and may be formed by shading, for example, by printing with an inkjet printer or the like. Further, here, the visible light has been described as an example. However, for example, when performing imaging of far-infrared rays, the pattern substrate 104 is made of a material that is transparent to far-infrared rays such as germanium, silicon, and chalcogenide. A material that is transparent to the wavelength to be used may be used, and the imaging pattern 105 may be made of a material that blocks light such as metal.

The pattern substrate 104 and the imaging pattern 105 can also be said to be modulators that modulate the intensity of light incident on the image sensor 103. Note that, here, a method of forming the imaging pattern 105 on the pattern substrate 104 in order to realize the imaging module 102 has been described. However, the imaging module 102 may be realized by a configuration as shown in FIG.

FIG. 3 is a diagram illustrating a configuration example of another imaging module according to the first embodiment. In the configuration example shown in FIG. 3, the imaging pattern 105 is formed in a thin film and is held by the support member 301.

In this apparatus, the angle of view can be changed depending on the thickness of the pattern substrate 104. Therefore, for example, if the pattern substrate 104 has the configuration shown in FIG. 3 and has a function of changing the length of the support member 301, it is possible to change the angle of view during shooting and shoot.

画素 On the surface of the image sensor 103, the pixels 103a, which are light receiving elements, are regularly arranged in a grid pattern. The image sensor 103 converts a light image received by the pixel 103a into an image signal which is an electric signal.

(4) The intensity of light transmitted through the imaging pattern 105 is modulated by the pattern, and the transmitted light is received by the image sensor 103. The image sensor 103 is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

(4) The image signal output from the image sensor 103 is subjected to processing such as noise removal by the fringe scan processing unit 106, and the data processed by the image processing unit 107 is output to the controller 108. When outputting the output to a host computer or an external recording medium, the controller 108 converts the data format so as to conform to an interface such as USB (Universal Serial Bus) and outputs the converted data.

Next, the principle of photographing in the imaging device 101 will be described. First, the photographing pattern 105 is a concentric pattern in which the pitch becomes finer in inverse proportion to the radius from the center, and using the radius r and the coefficient β from the reference coordinates that is the center of the concentric circle,

Can be defined as It is assumed that the photographing pattern 105 is transmittance-modulated in proportion to this equation. In the following, for simplicity, only the x-axis direction will be described using mathematical expressions. However, it is possible to consider two-dimensionally by similarly considering the y-axis direction.

プ レ ー ト A plate having such stripes is called a Gabor zone plate (GZP: Gabor Zone Plate) or a Fresnel zone plate (FZP: Fresnel Zone Plate).

FIG. 4 is a diagram showing an example of a photographing pattern and a developing pattern according to the first embodiment. Specifically, FIG. 4 is an example of a Gabor zone plate represented by the above equation (1).

FIG. 5 is a diagram showing another example of the photographing pattern and the developing pattern according to the first embodiment. Specifically, this is an example of a Fresnel zone plate using a pattern obtained as a result of binarizing Expression (1) with a threshold value of 1.

The patterned substrate 104 having a thickness of d that photographing pattern 105 is formed, a parallel beam by an angle theta ₀ in the x-axis direction as shown in FIG. 6 is a incident. In terms of geometrical optics, when the refraction angle in the pattern substrate 104 is θ, the light multiplied by the transmittance of the surface grating enters the image sensor 103 with a shift of k = d · tan θ. At this time,

The projection image having the intensity distribution as described above is detected on the image sensor 103. Note that Φ indicates the initial phase of the transmittance distribution of the above equation (1). FIG. 7 shows an example of a projected image of the photographing pattern 105.

FIG. 7 is a view showing an example of a projected image of the photographing pattern. As shown in FIG. 7, when parallel light is incident as shown in FIG. 6 using the photographing pattern 105, the image is projected onto the image sensor 103 by shifting by k as in the above equation (2). Is done. This is the output from the imaging module 102.

(4) Next, the image processing unit 107 performs a developing process. Among them, the developing process by the correlation developing method and the moiré developing method will be described.

In the correlation developing method, the image processing unit 107 calculates a cross-correlation function between the projected image of the photographing pattern 105 shown in FIG. 7 and the developing pattern 801 shown in FIG. A bright spot having a shift amount k can be obtained. In general, when the cross-correlation calculation is performed by a two-dimensional convolution calculation, the calculation amount increases.

FIG. 8 is a diagram showing an example of a development pattern. Specifically, the development pattern 801 has a pattern similar to the Gabor zone plate shown in FIG. 4 or the FZP (Fresnel zone plate) shown in FIG. That is, in the present embodiment, the development pattern 801 does not need to have a physical entity, and may be present as information used in image processing.

FIG. 9 is a diagram showing an example of a developed image by the correlation developing method. When the image is developed by the correlation development method, as described above, a developed image in which a certain bright point is shifted by k can be obtained.

Here, the principle of an example in which the cross-correlation operation is not performed by the two-dimensional convolution operation but is performed by using the Fourier transform will be described using mathematical expressions. First, since the development pattern 801 uses a Gabor zone plate or a Fresnel zone plate similarly to the imaging pattern 105, the development pattern 801 uses the initial phase Φ,

Can be expressed as Since the development pattern 801 is used in the image processing, it is not necessary to offset it by 1 as in the above equation (1), and there is no problem even if it has a negative value.

フ ー The Fourier transforms of the above equations (2) and (3) are respectively

become that way. Here, F represents an operation of Fourier transform, u is a frequency coordinate in the x direction, and δ with parentheses is a delta function. What is important in this equation is that the equation after the Fourier transform is also a Fresnel zone plate or a Gabor zone plate. Therefore, the development pattern after Fourier transform may be directly generated based on this equation. As a result, the amount of calculation can be reduced.

Next, when the above equation (4) and the above equation (5) are multiplied,

Becomes The term exp (-iku) represented by this exponential function is a signal component, and when this term is Fourier-transformed,

, And a bright spot can be obtained at the position of k on the original x-axis. This bright point indicates a light beam at infinity, and is nothing but a captured image obtained by the imaging device 101 in FIG.

In such a correlation development system, a pattern not limited to the Fresnel zone plate or the Gabor zone plate, for example, a random pattern may be realized as long as the autocorrelation function of the pattern has a single peak.

Next, in the moiré developing method, a moiré fringe as shown in FIG. 10 is generated by multiplying the projected image of the photographing pattern 105 shown in FIG. 7 by the developing pattern 801 shown in FIG. Is subjected to Fourier transform, so that a bright spot having a shift amount of (kβ / π) as shown in FIG. 11 can be obtained.

FIG. 10 is a diagram showing an example of moire fringes by the moire development method. Specifically, as shown in FIG. 10, the result of multiplication of the projected image of the photographing pattern 105 shown in FIG. 7 and the development pattern 801 shown in FIG. 8 is obtained as moire fringes.

と When this moiré fringe is shown by a formula

Becomes It can be seen that the third term of this expansion formula is a signal component, and that a striped pattern that is straight and equidistant in the direction of displacement between the two patterns is formed over the entire overlapping region. A fringe generated at a relatively low spatial frequency due to the superposition of such fringes is called a Moire fringe.

２ The two-dimensional Fourier transform of the third term is

become that way. Here, F represents an operation of Fourier transform, u is a frequency coordinate in the x direction, and δ with parentheses is a delta function. From this result, it can be seen that in the spatial frequency spectrum of the moiré fringes, the peak of the spatial frequency occurs at the position of (u = ± kβ / π). This bright point indicates a light beam at infinity, and is nothing but a captured image obtained by the imaging device 101 in FIG.

FIG. 11 is a view showing an example of a developed image by the moiré developing method. In the developed image shown in FIG. 11, bright points are obtained at the positions of u = ± kβ / π.

In the moiré developing method, as long as the moiré fringes obtained by shifting the pattern have a single frequency, the moire fringe may be realized by a pattern other than the Fresnel zone plate or the Gabor zone plate, for example, an elliptical pattern.

<Noise Cancellation> In the conversion from the above equation (6) to the above equation (7) and the conversion from the above equation (8) to the above equation (9), the discussion focused on the signal component. Terms other than the signal component hinder development as noise. Therefore, noise cancellation based on fringe scanning is effective.

For fringe scanning, it is necessary to use a plurality of patterns having different initial phases Φ as the imaging pattern 105. FIG. 12 shows an example of a plurality of patterns.

FIG. 12 is a diagram showing an example of a combination of imaging patterns of an initial phase in a fringe scan. Here, sensor images taken using four phases of (Φ = 0), (Φ = π / 2), (Φ = π), and (Φ = 3π / 2), that is, phases shifted by π / 2, Is calculated according to the following equation, a complex sensor image (complex sensor image) is obtained. Expressing this as an equation,

Becomes Here, the complex development pattern 801 is

Can be expressed as Since the developing pattern 801 is used in the fringe scan processing, there is no problem even if it is a complex number.

に お い て In the moiré developing method, when the above equation (10) and the above equation (11) are multiplied,

Becomes Since the term exp (2iβkx) represented by the exponential function is a signal component and an unnecessary term such as the above equation (8) does not occur, it can be seen that noise has been canceled.

Similarly, when the correlation development method is also confirmed, the Fourier transforms of the above equations (10) and (11) are

become that way. Next, when the above equation (13) is multiplied by the above equation (14),

Becomes Since the term exp (-iku) represented by the exponential function is a signal component and an unnecessary term as in the above equation (8) does not occur, it can be seen that noise has been canceled.

Note that, in the above example, a plurality of development patterns of four phases shifted by π / 2, such as (Φ = 0), (Φ = π / 2), (Φ = π), and (Φ = 3π / 2) Has been described, Φ may be set so as to equally divide an angle between 0 and 2π, and is not limited to this phase.

As a method of realizing the photographing using a plurality of patterns described above, a method of switching patterns by time division in fringe scan processing and a method of switching patterns by space division can be considered.

In order to realize the time-division fringe scan, for example, a liquid crystal display element which can electrically switch and display a plurality of initial phases shown in FIG. 12 may be used as the photographing pattern 105 in FIG. The switching timing of the liquid crystal display element and the shutter timing of the image sensor 103 are controlled in synchronization, and after acquiring four images in time series, the fringe scan processing unit 106 performs a fringe scan operation.

On the other hand, in order to realize the space division fringe scan, it is necessary to use an imaging pattern 105 having a plurality of initial phases as shown in FIG. After obtaining one image as a whole, the fringe scan processing unit 106 divides the image into four images corresponding to the respective patterns of the initial phase, and performs a fringe scan operation.

FIG. 13 is a diagram showing an example of a photographing pattern of the space division fringe scan. The imaging pattern 1300 of the space division fringe scan is an application example of the imaging pattern 105, and is spatially divided on an XY plane (a plane parallel to the image sensor 103) to form a plurality of initial phases ((Φ = 0), (Φ = π / 2), (Φ = π), (four phases with (Φ = 3π / 2)). Subsequently, the fringe scan calculation in the fringe scan processing unit 106 will be described.

FIG. 14 is a diagram showing an example of a processing flow of the fringe scan. First, the fringe scan processing unit 106 acquires sensor images based on a plurality of photographing patterns output from the image sensor 103. In the case where space-division fringe scanning is adopted, the acquired sensor images are converted according to individual photographing patterns. Since it is necessary to divide the image, a plurality of sensor images are obtained by dividing the image into a predetermined area (step 1401). When time-division fringe scan is adopted, division is not performed because a plurality of sensor images with different photographing patterns are obtained as time passes.

Next, the fringe scan processing unit 106 initializes a complex sensor image for output (step 1402).

{Then, the fringe scan processing unit 106 acquires the sensor image of the first initial phase Φ (step 1403).

{Then, the fringe scan processing unit 106 multiplies exp (iΦ) according to the initial phase Φ (step 1404).

{Then, the fringe scan processing unit 106 adds the multiplication result to the complex sensor image (step 1405).

The fringe scan processing unit 106 repeats the processing from step 1403 to step 1405 for the number of initial phases used (step 1406). For example, in the fringe scan using four phases as shown in FIG. 12, the fringe scan processing unit 106 determines the initial phases (Φ = 0), (π / 2), (π), and (3π / 2), respectively. Repeat 4 times in total.

Then, the fringe scan processing unit 106 outputs a complex sensor image (step 1407). The processing by the fringe scan processing unit 106 in steps 1401 to 1407 described above corresponds to the above equation (10). Next, image processing in the image processing unit 107 will be described.

FIG. 15 is a diagram showing an example of a processing flow of the development processing by the correlation development method. First, the image processing unit 107 obtains a complex sensor image output from the fringe scan processing unit 106, and performs a two-dimensional fast Fourier transform (FFT) on the complex sensor image (Step 1501). .

Next, the image processing unit 107 generates a predetermined development pattern 801 to be used for the development process, and multiplies the complex sensor image subjected to the two-dimensional FFT operation (step 1502).

{Circle around (2)} Then, the image processing unit 107 performs an inverse two-dimensional FFT operation (step 1503). The result of this operation is a complex number.

Therefore, the image processing unit 107 converts the result of the inverse two-dimensional FFT operation into an absolute value or a real part, converts the image to be photographed into a real number, and develops the image (step 1504).

Then, the image processing unit 107 performs contrast enhancement processing on the obtained developed image (step 1505). Further, the image processing unit 107 performs color balance adjustment (step 1506) and the like, and outputs the captured image. The above is the development processing by the correlation development method.

FIG. 16 is a diagram showing an example of a processing flow of a developing process by the moiré developing method. First, the image processing unit 107 acquires the complex sensor image output from the fringe scan processing unit 106, generates a predetermined development pattern 801 used for the development process, and multiplies the complex sensor image by the complex sensor image (step 1601). ).

{Circle around (2)} Then, the image processing unit 107 obtains a frequency spectrum by a two-dimensional FFT operation (step 1602).

{Circle around (4)} Then, the image processing unit 107 cuts out data of a necessary frequency region from the frequency spectrum obtained in step 1602 (step 1603).

の The subsequent realization processing in step 1504, the contrast enhancement processing in step 1505, and the color balance adjustment processing in step 1506 are the same as the processing in steps 1504 to 1506 shown in FIG.

<<< Principle of Photographing Finite-Distance Object> Next, FIG. 17 shows the manner in which the photographic pattern 105 is projected onto the image sensor 103 when the subject described above is sufficiently far (in the case of infinity).

FIG. 17 is a diagram illustrating an example of projection of a photographing pattern when an object is at an infinite distance. A spherical wave from a point 1701 constituting a distant object becomes a plane wave while propagating a sufficiently long distance and irradiates the imaging pattern 105. When a projected image 1702 is projected on the image sensor 103, the projected image becomes It has almost the same shape as the pattern 105 for photographing. As a result, a single luminescent spot can be obtained by performing development processing on the projection image 1702 using the development pattern. On the other hand, imaging of an object at a finite distance will be described.

FIG. 18 is a diagram showing an example of an enlarged projection of the photographing pattern when the object is at a finite distance. When the object to be imaged is at a finite distance, the projection of the imaging pattern 105 onto the image sensor 103 is enlarged more than the imaging pattern 105. When a spherical wave from a point 1801 constituting an object irradiates the imaging pattern 105 and a projected image 1802 is projected on the image sensor 103, the projected image is enlarged substantially uniformly. Note that this enlargement factor α is calculated using the distance f from the photographing pattern 105 to the point 1801.

It can be calculated as follows. For this reason, if an object at a finite distance is subjected to development processing using a development pattern designed for parallel light as it is, a single bright point cannot be obtained.

Therefore, if the developing pattern 801 is enlarged in accordance with the uniformly enlarged projection image of the photographing pattern 105, a single bright point can be obtained again for the enlarged projection image 1802. it can. For this purpose, the coefficients of the development patterns 801 beta, it is possible to correct by the beta / alpha ^2. Thus, light from point 1801 at a distance that is not necessarily infinity can be selectively reproduced. As a result, it is possible to perform imaging while focusing on an arbitrary position. In other words, when photographing is performed with the focus on the subject, the magnification of the development pattern 801 may be determined.

According to this configuration, it is also possible to focus on an arbitrary distance after shooting. FIG. 19 shows the configuration in this case.

FIG. 19 is a diagram showing a configuration of an imaging device according to the second embodiment of the present invention. The imaging device according to the second embodiment basically has the same configuration as the imaging device according to the first embodiment. However, what differs from the first embodiment is the presence of a focus setting unit 1901. The focus setting unit 1901 receives the setting of the focus distance by using a knob provided on the imaging apparatus 101 or a GUI (Graphical User Interface) of a smartphone, and outputs the focus distance information to the image processing unit 107.

Further, the fact that focus adjustment after shooting is possible means that the image processing unit 107 has depth information, and various functions such as autofocus and distance measurement can be realized in the image processing unit 107. .

To realize such a function as focus setting, it is necessary to freely change the coefficient β of the developing pattern 801. However, as in the processing in the fringe scan processing unit 106 described in the present embodiment, By performing the fringe scan operation, the processing using the development pattern 801 can be performed independently, and the processing can be simplified.

<Problem of Space-Division Fringe Scan: Subject Distance> As described with reference to FIG. 18, when the object to be imaged is at a finite distance, the projection of the imaging pattern 105 onto the image sensor 103 is larger than that of the imaging pattern 105. You. This can be a problem when using space division fringe scanning.

FIG. 20 is a diagram illustrating an example of a positional relationship between a point light source and an imaging device. As shown in FIG. 20, a case where the object to be imaged is at a far distance f (point light source 2001) and a case where it is at a short distance f '(point light source 2001') are considered.

FIG. 21 is a diagram illustrating an example of a sensor image for each distance of a point light source. As shown in FIG. 21, light from an object to be imaged is applied to the imaging pattern 105 of the space division fringe scan, and the image projected when the shadow is projected on the image sensor 103 is the same as the object to be imaged. It changes according to the case of the distance f between the pattern and the case of the distance f ′. In the space division fringe scan, it is necessary that the fringe scan processing unit 106 divides the image into four in correspondence with the respective patterns of the initial phase. Is changing. For example, the center of the concentric circle of the first quadrant 1302 is shifted to the upper right in the first quadrant 2102 at the distance f ′. As described above, if each quadrant is always divided at the same place, a shift occurs, and the effect of the fringe scan is reduced.

を To solve this problem, after performing image correction using the enlargement ratio α in Expression 16, the data may be divided into four images in each quadrant. In each of FIGS. 22 and 23, the center of each pattern when the center of the sensor image in FIG. 21 is arranged at the origin is indicated by x.

FIG. 22 is a diagram showing an example of pattern center coordinates when the point light source is at infinity. In FIG. 22, the coordinates of the pattern center 2201 in the first quadrant are represented by (x0, y0).

FIG. 23 is a diagram showing an example of the pattern center coordinates when the point light source is at a finite distance. In FIG. 23, the coordinates of the pattern center 2301 in the first quadrant are represented by (x1, y1).

Since the coordinates (x1, y1) of the pattern center 2301 in FIG. 23 are enlarged at the enlargement rate α from the coordinates (x0, y0) of the pattern center 2201 in FIG. 22, the coordinates (x1, y1) are changed to the coordinates (x0, y1). y0)

The coordinate conversion may be performed using a matrix M that specifies the amount of movement on the sensor surface as shown in FIG. FIG. 24 shows a sensor image obtained as a result of the conversion.

FIG. 24 is a diagram illustrating an example of correcting a sensor image when the point light source is at a finite distance. As shown in FIG. 24, the image is reduced and the same position is always the center of the pattern. Therefore, in this example, if the image is divided into four in each quadrant, a sensor image in each quadrant can be obtained. Since the image is reduced, the obtained sensor image is smaller than the image size before correction. In this case, a constant such as 0 may be complemented in the margin area 2401 or may not be used as the NAN value.

As described above, by performing coordinate transformation with reference to the center of the concentric circle using the matrix M shown in Expression (17), the projection enlargement that occurs when the point light source is at a finite distance, that is, when the subject distance is at the finite distance, Eliminate problems.

<Problem of space division fringe scan: pattern fixation accuracy> However, it may not be enough to correct the displacement due to the subject distance. This is the case where the accuracy at the time of assembling the imaging device 101 and the accuracy of manufacturing the imaging pattern 105 are not sufficient, and a shift has occurred. FIG. 25 shows an example of a sensor image in a case where the photographing pattern 105 is attached with a shift.

FIG. 25 is a diagram illustrating an example of a sensor image when the shooting pattern is rotating. In this example, an example of a sensor image in a case where the photographing pattern 105 is attached by being rotated by an angle −θ about the origin is shown.

方法 A method for correcting this will be described. First, the center coordinates 2501 to 2504 of each pattern are calculated by cross-correlation calculation with a reference pattern (for example, an image obtained by enlarging the photographing pattern 105 at an enlargement ratio α obtained from the subject distance). Next, the coordinates of the center of gravity O of four points of the center coordinates 2501 to 2504 are calculated, and are arranged so that the center of gravity O overlaps the origin as shown in FIG.

FIG. 26 is a diagram showing an example of the pattern center coordinates when the imaging pattern is rotating. At this time, to convert the coordinates (x1, y1) of the pattern center 2601 in the first quadrant of FIG. 26 to the coordinates (x0, y0) of the pattern center 2201 in FIG.

The coordinate transformation may be performed using a matrix H that specifies the amount of rotation movement on the sensor surface as shown in FIG. The reason why the matrix M is used in the equation (18) is to correct an image according to the distance f of the point light source 2001 used. As a result, the image of FIG. 25 is corrected by rotating the angle θ, and the image always has the same pattern center. Therefore, in the example of FIGS. 25 and 26, the image may be divided in each quadrant.

The matrix H shown in Expressions (18) and (19) is a matrix having 2 × 2 elements, which is generally called an affine matrix, and has a relationship between four coordinates (in this example, the center coordinate of each pattern). Elements can be calculated from 2201 to 2204 and center coordinates 2501 to 2504). Although two points are correctly used for calculating the affine matrix, a more accurate affine matrix can be obtained by applying the least squares method using information of four points.

Next, another example in which the accuracy in assembling the imaging device is low will be described. For example, as shown in FIG. 27, a case is considered where the imaging patterns 105 are arranged obliquely shifted in the thickness direction. FIG. 28 shows an example of a sensor image when the point light source 2001 is photographed in this state.

FIG. 28 is a diagram illustrating an example of a sensor image in a state where the photographing pattern is inclined in the thickness direction. In this case, as shown in FIG. 28, the sensor image has a trapezoidal distortion. The trapezoidal distortion cannot be completely corrected by the affine matrix as in the equation (18), but the trapezoidal distortion can be corrected more easily and accurately by using the homography matrix.

方法 A method for correcting this will be described. First, the center coordinates 2801 to 2804 of each pattern are calculated by cross-correlation calculation with a reference pattern (for example, an image obtained by enlarging the photographing pattern 105 at an enlargement ratio α obtained from the subject distance). Next, the coordinates of the center of gravity O of the four points of the center coordinates 2801 to 2804 are calculated, and are arranged so that the center of gravity O overlaps the origin as shown in FIG.

FIG. 29 is a diagram illustrating an example of pattern center coordinates in a state where the photographing pattern is inclined in the thickness direction. At this time, to convert the coordinates (x1, y1) of the pattern center 2801 in the first quadrant in FIG. 29 to the coordinates (x0, y0) of the pattern center 2201 in FIG.

The coordinates may be converted using a matrix H that specifies the amount of movement on the sensor surface as shown in FIG. The matrix H is a matrix having a 3 × 3 element generally called a homography matrix, and elements are determined from a relationship of four coordinates (in this example, center coordinates 2201 to 2204 and center coordinates 2801 to 2804 of each pattern). It is possible to calculate. The reason why the matrix M is used is to correct the image according to the distance f 'of the point light source 2001' used. As a result, the image in FIG. 28 is corrected for trapezoidal distortion (tilt) and rotation, and is always an image having the same pattern center. Therefore, in this example, the image may be divided in each quadrant.

The configuration and processing for realizing the above-described distortion correction will be described with reference to FIGS. FIG. 30 shows a configuration for realizing distortion correction.

FIG. 30 is a diagram illustrating a configuration example of an imaging device according to a third embodiment of the present invention. The imaging device shown in FIG. 30 is basically the same as the imaging device according to the second embodiment shown in FIG. 19, but is partially different. The following mainly describes the differences.

The imaging apparatus according to the third embodiment includes a correction value storage unit 3001 as a parameter storage unit that measures and stores correction parameters including a matrix H and a matrix M for distortion correction, and a distortion using the correction parameters. A fringe scan processing unit 3002 for performing correction is provided. As shown in FIG. 31, the correction value storage unit 3001 may be included in the imaging module 102. In that case, the correction value storage unit 3001 operates as a parameter output unit that outputs a correction value, that is, a parameter for correction, in response to a request via an API (Application, Programming, Interface).

FIG. 31 is a diagram illustrating a configuration example of an imaging device according to a modification of the third embodiment of the present invention. Considering that the correction parameters have individual differences due to the mounting variation of the imaging module 102, in a configuration in which the imaging module 102 is separated from the imaging device 101, the correction value storage unit is provided in the imaging module 102 as shown in FIG. The form with 3001 is more versatile. In any of the embodiments, the method of calculating the correction value (the calibration method) is common. First, a correction value calculation method (calibration method) will be described.

FIG. 32 is a diagram illustrating an example of a processing flow of calibration. First, the operator of the calibration irradiates a point light source 2001 on the optical axis of the imaging module 102 toward the imaging module 102 as shown in FIG. 20 (step 3201). The point light source 2001 may be any light source such as an LED (Light Emitting Diode) light source or the sun that knows infinity or the distance to a subject, such as the sun.

(4) Then, the operator of the calibration causes the imaging module 102 to acquire a sensor image (Step 3202).

{Then, the fringe scan processing unit 3002 calculates the center coordinates of FZA (Fresnel \ Zone \ Aperture) (step 3203). More specifically, the fringe scan processing unit 3002 calculates the center coordinates of each pattern (for example, an image obtained by enlarging the photographing pattern 105 at an enlargement factor α obtained from the subject distance) by using a cross-correlation operation. : Center coordinates 2801 to 2804) are calculated.

Then, the fringe scan processing unit 3002 calculates the center of gravity O from the central coordinate group (step 3204). Then, the fringe scan processing unit 3002 arranges the center of gravity O at the origin.

フ Then, the fringe scan processing unit 3002 calculates a correction matrix H (step 3205). Specifically, the fringe scan processing unit 3002 calculates a correction matrix based on the relationship between the displacements of the four center coordinates (eg, the rotation relationship between the center coordinates 2201 to 2204 and the center coordinates 2801 to 2804 and the trapezoidal distortion). A 3 × 3 element of the homography matrix H that becomes H is calculated.

{Then, the fringe scan processing unit 3002 stores the correction matrix H and the position of the center of gravity O as correction parameters in the correction value storage unit 3001 (step 3206).

The above is the calibration process. According to the calibration processing, the matrix H for correcting the sensor image and the position of the center of gravity O can be specified, and stored in the correction value storage unit 3001 as a parameter.

Subsequently, the fringe scan processing unit 3002 performs distortion correction processing.

FIG. 33 is a diagram illustrating an example of a sensor image distortion correction processing flow. First, the fringe scan processing unit 3002 reads the position of the center of gravity O from the correction value storage unit 3001, and moves the center of gravity O of the sensor image to the coordinates serving as the origin (step 3301).

Next, the fringe scan processing unit 3002 performs correction using the matrix M (step 3302). Specifically, the fringe scan processing unit 3002 obtains information on the focusing distance f ′ from the focus setting unit 1901 and performs correction using a matrix M corresponding to the distance f ′.

フ Then, the fringe scan processing unit 3002 performs correction using the matrix H (step 3303). Specifically, the fringe scan processing unit 3002 performs correction using the matrix H acquired from the correction value storage unit 3001.

After that, the same processing as the processing of steps 1401 to 1407 of the processing flow of the fringe scan shown in FIG. 14 is performed on the sensor image corrected by the matrix H.

In this example, the matrix H and the movement of the position of the center of gravity O are processed separately. However, when a homography matrix of a 3 × 3 matrix is used as in Expression (20), the 3 × 3 matrix is used. Since it is possible to include the correction amount of the movement of the center of gravity O in the matrix, it is not always necessary to perform the processing separately.

Also, the description has been given on the assumption that the calibration is performed in advance. However, when a point light source or the sun is detected, the correction parameters are recalculated while photographing, and the information in the correction value storage unit 3001 is updated. Is also good. That is, the calibration is not limited to the preliminary execution, and the calibration may be performed as needed under the use environment.

The above is the imaging apparatus according to the third embodiment. According to the imaging apparatus according to the third embodiment, an error in assembling the imaging apparatus and an error in producing a photographic pattern are corrected, and distortion associated with focus adjustment is corrected, thereby achieving high-precision space division. A fringe scan can be performed.

In the configuration of the third embodiment, the distortion is corrected before the fringe scan of the sensor image (the processes of steps 3301 to 3303 are performed before the processes of steps 1401 to 1407 in the fringe scan processing unit 3002). However, when the sensor image is transmitted to another device wirelessly or by wire, it may be desirable that the information to be transmitted is unprocessed in consideration of preventing loss of information. That is, it may be preferable to perform the distortion correction processing after the fringe scan. A method of correcting distortion after fringe scanning of a sensor image will be described with reference to FIGS.

FIG. 34 is a diagram illustrating a configuration example of an imaging device according to a fourth embodiment of the present invention. The imaging device according to the fourth embodiment is substantially the same as the imaging device according to the third embodiment, but is partially different. The imaging device according to the fourth embodiment is different from the imaging device according to the third embodiment in that the information in the correction value storage unit 3001 is read and used by the image processing unit 3401. That is, the imaging apparatus according to the fourth embodiment does not correct the distortion of the sensor image, but adds the distortion corresponding to the sensor image to the development pattern 801. In the third embodiment, the matrix H for converting the coordinates (x1, y1) into the coordinates (x0, y0) is calculated.

, A matrix H ′ and a matrix M ′ for transforming the coordinates (x0, y0) having the reverse transformation into the coordinates (x1, y1) are calculated.

現 像 The flow of the development process in this case is the same as the flow of the development process by the correlation development method in FIG. 15 described above, but is partially different. In the fourth embodiment, the generation of the development pattern 801 in step 1502 of the development processing by the correlation development method is performed according to the processing flow shown in FIG.

FIG. 35 is a diagram showing an example of a distortion correction processing flow of a developing pattern. The distortion correction processing for the development pattern is started in step 1502 of the development processing using the correlation development method.

Before executing the distortion correction processing flow of the development pattern, the image processing unit 3401 converts the matrix H ′ of the equation (21) into a distortion adding processing in advance similarly to the example of the calibration processing flow illustrated in FIG. Is calculated in advance. Specifically, the image processing unit 3401 obtains a 3 × 3 element of a homography matrix H ′ that becomes a correction matrix H ′ based on a relationship between four coordinates (eg, center coordinates 2201 to 2204 and center coordinates 2801 to 2804). Is calculated in advance.

Then, at the time of development, that is, in the distortion correction processing of the development pattern, the image processing unit 3401 generates a concentric pattern image for space division fringe scanning as shown in FIG. 20 (step 3501).

Then, the image processing unit 3401 moves the center of gravity O (step 3502). Specifically, the image processing unit 3401 reads the position of the center of gravity O from the correction value storage unit 3001, and moves the center of the pattern image of the concentric circle for the space division fringe scan by the position of the center of gravity O from the coordinates with the origin as the origin. .

{Then, the image processing unit 3401 performs the correction using the matrix H ′ acquired from the correction value storage unit 3001 (step 3503). Specifically, a concentric pattern image for space division fringe scanning is converted using the matrix H 'read from the correction value storage unit 3001.

{Then, the image processing unit 3401 performs the correction using the matrix M ′ obtained from the correction value storage unit 3001 (Step 3504). Specifically, the image processing unit 3401 obtains the focusing distance f ′ from the focus setting unit 1901, obtains information of a matrix M ′ corresponding to the distance f ′ from the correction value storage unit 3001, and performs correction. carry out.

Then, the image processing unit 3401 divides the pattern image into four (Step 3505). The subsequent processing is the same as the development processing of FIG. 15 in which the complex sensor image subjected to the two-dimensional FFT operation is multiplied.

In this method, since the development pattern 801 is distorted, the moiré development method does not produce moire at the time of point light source development at a single frequency but blurs the image. Therefore, it can be said that the correlation development method has higher affinity and is more desirable than the moiré development method.

The above is the imaging apparatus according to the fourth embodiment. According to the method and configuration according to the fourth embodiment, it is possible to correct an error in assembling an imaging device and an error in manufacturing a photographing pattern, and correct distortion related to focus adjustment, thereby achieving high-precision space. A divided fringe scan can be performed.

In the third and fourth embodiments, the method of correcting an error in assembling the imaging device and an error in manufacturing a photographing pattern and correcting a distortion related to focus adjustment has been described. However, no correction has been taken into account for defects, dirt, and dust in the imaging pattern and the image sensor. In the fifth embodiment, a method of protecting the defect will be described with reference to FIGS. First, the problem of this defect will be clarified using FIG.

FIG. 37 is a diagram showing an example of an image in the case where dirt or the like is attached to the photographing pattern. In the case of the time-division fringe scan, a defect appears at the same position in all four patterns, so that the defect is recorded at one position as a sensor image. In the case of the space division fringe scan, since a defect exists only in the quadrant of the pattern 3803 like the defect 3801, matching with other patterns cannot be obtained. By mapping the luminance distribution along the dotted line for the quadrant of the pattern 3802 and the quadrant of the pattern 3803 in FIG. 37, the graphs shown in FIGS. 38 and 39 are obtained.

FIG. 38 is a diagram showing an example of the luminance distribution of an ideal sensor image. That is, as shown in the pattern 3802, a signal component in which the luminance decreases in the mask portion of the concentric pattern can be obtained.

FIG. 39 is a diagram illustrating an example of the luminance distribution of a defective sensor image. That is, as shown in a pattern 3803, a signal component whose luminance is reduced in the mask portion of the concentric pattern can be obtained, but the luminance is reduced irrespective of the mask portion due to a part of the defect 3801. Note that, of course, the degree of decrease in luminance differs depending on the degree of the defect 3801. That is, if the defect is minor, the degree of decrease in luminance is small, and if the defect is large, the degree of decrease in luminance is large.

で は In the state shown in FIG. 37, the fringe scan of the defective portion is not performed correctly, and the error spreads to the entire image, and the image quality deteriorates (a significant image cannot be obtained). This is because the luminance difference in the defective area is generally larger than the luminance difference in the signal component, and the degree of influence is large. By specifying the defect position and excluding it from the development processing, the influence of the defect is reduced, and the influence of the S / N ratio is reduced. FIG. 36 shows the configuration of the fifth embodiment.

FIG. 36 is a diagram illustrating a configuration example of an imaging device according to a fifth embodiment of the present invention. The imaging device according to the fifth embodiment is basically the same as the imaging device according to the fourth embodiment, but is partially different.

撮 像 The imaging apparatus according to the fifth embodiment differs from the imaging apparatus according to the fourth embodiment in that a defect detection unit 3601 is provided and an image processing unit 3602 performs a defect protection process. The defect detection unit 3601 acquires the sensor image from the fringe scan processing unit 106 and, when detecting an area having a luminance lower than the defect detection threshold as shown in FIGS. 38 and 39, outputs a defect signal specifying the area. At the same time, it is stored in the correction value storage unit 3001. That is, information for specifying the position where the defect signal is output is stored in the correction value storage unit 3001. Such information is not limited to, for example, information for specifying coordinates on a defective sensor one by one, but may be information for specifying a rectangular area on the sensor or the center and radius of a minimum circle including a defective part. Good.

The simplest defect detection method by the defect detection unit 3601 is as described above. However, when the amplitude of the signal component is large, it may be difficult to distinguish between a defect and a signal. Therefore, the defect detection unit 3601 determines a relationship in which the phases are shifted by π, such as a combination of sensor images of Φ = 0 and π or a combination of sensor images of Φ = π / 2 and 3π / 2 in a fringe scan, that is, a concentric circle. The defect may be determined by adding the two sensor images in which black and white are inverted. This processing is desirably performed between steps 1401 and 1402 of the fringe scan processing in FIG. FIG. 40 shows the result of performing this addition on the signals shown in FIGS. 38 and 39.

FIG. 40 is a diagram showing an example of defect detection by inversion pattern synthesis. As shown in FIG. 40, the normal signal component is canceled by cancellation, and only the luminance fluctuation component remains. In response to this response, the defect detection unit 3601 outputs a defect signal and stores a correction value in an area where the remaining luminance is below or above the defect detection threshold, using a defect detection threshold within the range of ± constant of the average value. This is stored in the unit 3001. According to this method, stable defect detection can be performed regardless of the subject. This defect detection may be performed for each of all the sensor images, or may be performed for each pair with the inverted pattern as a pair.

{Circle around (2)} The image processing unit 3602 performs a process of not using a portion corresponding to the defect signal of the development pattern as a NAN value, that is, a data mask process, using the defect signal output from the defect detection unit 3601.

When the defect signal is applied not to the sensor image but to the development pattern of the image processing unit 3602, the matrix H ′ and the matrix M ′ are used to correspond to the defect signal as in the fourth embodiment. It is desirable to correct the position at which it is performed.

Also, the defect signal is input not to the image processing unit 3602 but to the fringe scan processing unit 3002 of FIG. 30 of the third embodiment, and the same position of all the patterns of the sensor image (the relative position between the pattern center and the defect signal is the same) May be performed without using the NAN value as the NAN value.

The fifth embodiment has been described above. According to the imaging apparatus according to the fifth embodiment, errors in assembling the imaging apparatus and errors in manufacturing the imaging pattern are corrected, and not only distortions related to focus adjustment are corrected, but also the imaging pattern is corrected. By performing correction for defects, dirt, and dust of the image sensor and the image sensor, it is possible to perform more accurate space division fringe scan.

Note that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described above.

In addition, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of one embodiment can be added to the configuration of another embodiment.

に つ い て In addition, a part of the configuration of each embodiment can be added, deleted, or replaced with another configuration.

In addition, each of the above-described configurations, functions, processing units, processing means, and the like may be partially or entirely realized by hardware, for example, by designing an integrated circuit. In addition, the above-described configurations, functions, and the like may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files for realizing each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.
In addition, control lines and information lines are shown as necessary for the description, and do not necessarily indicate all control lines and information lines on a product. In fact, it can be considered that almost all components are connected to each other.

The above has been a description of the present invention, focusing on the embodiments.

101 imaging apparatus, 102 imaging module, 103 image sensor, 103a pixel, 104 pattern substrate, 105 imaging pattern, 106 fringe scan processing unit 107, image processing unit, 108, controller, 301, support member, 801, development pattern, 1300, photography pattern, 1302, first quadrant, 1701, 1801 .. Point, 1702, 1802... Projected image, 1901... Focus setting unit, 2001, 2001 ′, light source, 2101, second quadrant, 2102, first quadrant, 2103, 3rd quadrant, 2104... Fourth quadrant, 2401... Blank area, 2201, 301, 2501, 2601, 280... Pattern center of first quadrant, 25 2, 2602, 2802: pattern center in the second quadrant, 2503, 2603, 2803 ... pattern center in the third quadrant, 2504, 2604, 2804 ... pattern center in the fourth quadrant, 3001 ... correction Value storage unit, 3002 ... fringe scan processing unit, 3401 ... image processing unit, 3601 ... defect detection unit, 3602 ... image processing unit, 3801 ... defect, 3802 ... fourth quadrant , 3803... Pattern in the first quadrant.

Claims (13)

1. An image sensor that converts light into an electric signal and generates a sensor image;
   A modulator that modulates the intensity of light detected by the image sensor based on a shooting pattern,
   A parameter storage unit that stores a parameter used for performing a predetermined correction process on a plurality of the sensor images captured by the different capturing patterns among the sensor images,
   An imaging device comprising:
2. The imaging device according to claim 1,
   A parameter output unit that outputs the parameter as required,
   An imaging device comprising:
3. The imaging device according to claim 1,
   The parameter includes information for specifying the amount of movement of the image sensor on the sensor surface for each of the photographing patterns,
   An imaging device characterized by the above-mentioned.
4. The imaging device according to claim 1,
   The parameters include information for correcting trapezoidal distortion based on the center of a concentric circle of the imaging pattern, as information for specifying the amount of movement on the image sensor surface for each imaging pattern.
   An imaging device characterized by the above-mentioned.
5. The imaging device according to claim 1,
   The imaging pattern is provided with a concentric pattern,
   The parameters include, as information for specifying the amount of movement on the sensor surface of the image sensor for each of the imaging patterns, the amount of movement of the center of gravity between the centers of concentric circles of the imaging pattern, and the center of rotation as the center of rotation. Information that specifies information corresponding to the rotation angle to be included.
   An imaging device characterized by the above-mentioned.
6. The imaging device according to claim 1,
   The predetermined correction processing is a homography conversion,
   The parameter is information for specifying a 3 × 3 matrix used for the homography conversion.
   An imaging device characterized by the above-mentioned.
7. The imaging device according to claim 1,
   The predetermined correction process is an affine transformation,
   The parameter is information for specifying a 2 × 2 matrix used for the affine transformation.
   An imaging device characterized by the above-mentioned.
8. The imaging device according to claim 1,
   A defect detection unit that detects a defect site in the sensor image,
   The predetermined correction process is a process for excluding the defective portion,
   The parameter is information for identifying the defect site,
   An imaging device characterized by the above-mentioned.
9. The imaging device according to claim 8,
   The defect detection unit detects a portion having an output of a predetermined value or more as the defective portion by combining the sensor images obtained by the imaging pattern having an inverted pattern among the imaging patterns,
   An imaging device characterized by the above-mentioned.
10. The imaging device according to claim 1,
    A fringe scan processing unit that performs the predetermined correction process,
    The fringe scan processing unit performs the predetermined correction process on each of the plurality of sensor images captured by the different imaging patterns among the sensor images,
    An imaging device characterized by the above-mentioned.
11. The imaging device according to claim 1,
    An image processing unit that performs the predetermined correction process,
    The image processing unit performs the predetermined correction process on a predetermined development pattern to be applied to a composite image of a plurality of the sensor images photographed by different ones of the sensor images among the sensor images,
    An imaging device characterized by the above-mentioned.
12. The imaging device according to any one of claims 1 to 11, wherein
    The parameter is provided according to the distance to the subject,
    An imaging device characterized by the above-mentioned.
13. An imaging method using an imaging device,
    The imaging device,
    A modulation step of modulating the intensity of the light transmitted through the shooting pattern,
    An image generation step of converting the modulated light into an electric signal by an image sensor and generating a sensor image,
    A correction unit that reads, from a predetermined storage unit, parameters used for performing a predetermined correction process on a plurality of the sensor images captured by the different imaging patterns among the sensor images, and executes the correction process by a fringe scan processing unit. Steps and
    An imaging method characterized in that:

Images