JP7097787B2 - Imaging device and imaging method - Google Patents

Description

The present invention relates to an image pickup apparatus and an image pickup method.

Japanese Patent Application Laid-Open No. 2018-61109 (Patent Document 1) is available as a background technique in this technical field. In this publication, "a modulator having a first pattern and modulating the intensity of light, an image sensor that converts the light transmitted through the modulator into image data and outputting the image data, the image data, and the first. There is a description of an image pickup device that "is characterized by having an image processing unit that restores an image based on mutual correlation calculation with pattern data showing two patterns", and further "a pattern in which the initial phase of the modulator is different". There is a description of a technique for noise canceling from an image in which "the image data output from the image sensor is divided into areas according to the pattern arrangement of the modulator" by "spatial division".

Japanese Unexamined Patent Publication No. 2018-61109

In the technique described in Patent Document 1, patterns having different initial phases are arranged so as to be offset (for example, among patterns A, B, C, and D, the center position of pattern B is displaced, and pattern C is rotated. If the upper surface of the substrate is not parallel to the sensor surface (for example, it is tilted), the projection pattern received by the image sensor will be misaligned or distorted, and even if the image is developed. There is a problem that a correct developed image cannot be obtained.

An object of the present invention is to provide a technique for obtaining a correct developed image by correcting an assembly error of an image pickup apparatus, a manufacturing error of a pattern for photographing, and a distortion of focus adjustment.

The present application includes a plurality of means for solving at least a part of the above problems, and examples thereof are as follows. In order to solve the above problems, the image pickup apparatus according to one aspect of the present invention is an image sensor that converts light into an electric signal to generate a sensor image, and an image sensor that detects light based on an imaging pattern. It has a modulator that modulates the intensity, and a parameter storage unit that stores parameters used for executing a predetermined correction process for a plurality of the sensor images captured by different imaging patterns among the sensor images.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a technique for obtaining a correct developed image by correcting an assembly error of an image pickup apparatus, a manufacturing error of a pattern for photographing, and a distortion of focus adjustment. Issues, configurations and effects other than those described above will be clarified by the following description of the embodiments.

It is a figure which shows the structural example of the image pickup apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows the structural example of the image pickup module which concerns on 1st Example. It is a figure which shows the structural example of another image pickup module which concerns on 1st Embodiment. It is a figure which shows the example of the pattern for photography and the pattern for development which concerns on 1st Example. It is a figure which shows another example of the pattern for photography and the pattern for development which concerns on 1st Example. It is a figure which shows the example which the in-plane deviation occurs in the projection image from the surface of a pattern substrate to an image sensor by diagonally incident parallel light. It is a figure which shows the example of the projection image of the pattern for photography. It is a figure which shows the example of the development pattern. It is a figure which shows the example of the developed image by the correlation development method. It is a figure which shows the example of the moire fringe by the moire development method. It is a figure which shows the example of the developed image by the moire development method. It is a figure which shows the example of the combination of the shooting pattern of the initial phase in a fringe scan. It is a figure which shows the example of the shooting pattern of the space division fringe scan. It is a figure which shows the example of the processing flow of a fringe scan. It is a figure which shows the example of the processing flow of the development processing by a correlation development method. It is a figure which shows the example of the processing flow of the development processing by a moire development method. It is a figure which shows the projection example of the pattern for photography when the object is at an infinite distance. It is a figure which shows the magnifying projection example of a pattern for photography when an object is at a finite distance. It is a figure which shows the structural example of the image pickup apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows the example of the positional relationship between a point light source and an image pickup apparatus. It is a figure which shows the example of the sensor image by the distance of a point light source. It is a figure which shows the example of the pattern center coordinates when a point light source is at infinity. It is a figure which shows the example of the pattern center coordinates when a point light source is at a finite distance. It is a figure which shows the correction example of a sensor image when a point light source is at a finite distance. It is a figure which shows the example of the sensor image when the pattern for photography is rotating. It is a figure which shows the example of the pattern center coordinates when the pattern for photography is rotated. It is a figure which shows the example of the state which the pattern for photography is inclined in the thickness direction. It is a figure which shows the example of the sensor image in the state that the pattern for photography is inclined in the thickness direction. It is a figure which shows the example of the pattern center coordinates in the state where the pattern for photography is inclined in the thickness direction. It is a figure which shows the structural example of the image pickup apparatus which concerns on 3rd Embodiment of this invention. It is a figure which shows the structural example of the image pickup apparatus which concerns on the modification of the 3rd Embodiment of this invention. It is a figure which shows the example of the process flow of calibration. It is a figure which shows the example of the distortion correction processing flow of a sensor image. It is a figure which shows the structural example of the image pickup apparatus which concerns on 4th Embodiment of this invention. It is a figure which shows the example of the distortion correction processing flow of a development pattern. It is a figure which shows the structural example of the image pickup apparatus which concerns on 5th Embodiment of this invention. It is a figure which shows the example of the image in the case where dirt or the like adheres to a pattern for photography. It is a figure which shows the luminance distribution example of an ideal sensor image. It is a figure which shows the example of the luminance distribution of a sensor image with a defect. It is a figure which shows the example of the defect detection by the inversion pattern synthesis.

In the following embodiments, when necessary for convenience, the description will be divided into a plurality of sections or embodiments, but unless otherwise specified, they are not unrelated to each other, one of which is the other. It is related to some or all of the modified examples, details, supplementary explanations, etc.

Further, in the following embodiments, when the number of elements (including the number, numerical value, quantity, range, etc.) is referred to, when it is specified in particular, or when it is clearly limited to a specific number in principle, etc. Except for this, the number is not limited to the specific number, and may be more than or less than the specific number.

Furthermore, in the following embodiments, the components (including element steps and the like) are not necessarily essential unless otherwise specified or clearly considered to be essential in principle. Needless to say.

Similarly, in the following embodiments, when the shape, positional relationship, etc. of the constituent elements are referred to, the shape is substantially the same, except when it is clearly stated or when it is considered that it is not clearly the case in principle. Etc., etc. shall be included. This also applies to the above numerical values and ranges.

Further, in all the drawings for explaining the embodiment, the same members are in principle the same reference numerals, and the repeated description thereof will be omitted. Hereinafter, examples of the present invention will be described with reference to the drawings.

In general, digital cameras mounted on information devices such as in-vehicle cameras, wearable devices, and smartphones are often required to be thinner and lower in cost. For example, an imaging method has been proposed that realizes thinning and cost reduction by obtaining an object image without using a lens. In such a technique, an imaging method is used in which a special grid pattern is attached in front of an image sensor, and an inverse problem for developing an image from a projection pattern received by the image sensor is solved to obtain an image of an object. There is. In this method, the calculation when solving the inverse problem by signal processing becomes complicated, and the processing load is high, so that the hardware requirement specifications of the information device become high.

<Principle for photographing an infinity object> FIG. 1 is a diagram showing a configuration example of an image pickup apparatus according to a first embodiment of the present invention. The image pickup apparatus 101 acquires an image of an object in the outside world without using a lens for forming an image, and as shown in FIG. 1, the image pickup module 102, the fringe scan processing unit 106, the image processing unit 107, and the controller 108. It is composed of. FIG. 2 shows an example of the image pickup module 102.

FIG. 2 is a diagram showing a configuration of an imaging module according to the first embodiment. The image pickup module 102 is composed of an image sensor 103, a pattern substrate 104, and a shooting pattern 105. The pattern substrate 104 is closely fixed to the light receiving surface of the image sensor 103, and a pattern 105 for photographing is formed on the pattern substrate 104. The pattern substrate 104 is made of a material transparent to visible light such as glass and plastic.

The photographing pattern 105 is composed of a concentric grid pattern in which the spacing of the grid patterns, that is, the pitch becomes narrower in inverse proportion to the radius from the center toward the outside. The photographing pattern 105 is formed by depositing a metal such as aluminum or chromium by, for example, a sputtering method used in a semiconductor process. Light and shade are added by the pattern with metal vapor deposition and the pattern without metal deposition. The formation of the photographing pattern 105 is not limited to this, and may be formed by adding shades by printing with, for example, an inkjet printer. Further, although visible light has been described as an example here, for example, when photographing far infrared rays, the pattern substrate 104 is a material to be photographed, for example, a material transparent to far infrared rays such as germanium, silicon, and chalcogenide. A material that is transparent to the wavelength to be used may be used, and a material that blocks light such as metal may be used for the photographing pattern 105.

Further, it can be said that the pattern substrate 104 and the photographing pattern 105 are modulators that modulate the intensity of the light incident on the image sensor 103. Although the method of forming the photographing pattern 105 on the pattern substrate 104 in order to realize the image pickup module 102 is described here, it can also be realized by the configuration as shown in FIG.

FIG. 3 is a diagram showing a configuration example of another imaging module according to the first embodiment. In the configuration example shown in FIG. 3, the photographing pattern 105 is formed on a thin film and held by the support member 301.

In this apparatus, the shooting angle of view can be changed by the thickness of the pattern substrate 104. Therefore, for example, if the pattern substrate 104 has the configuration shown in FIG. 3 and has a function of changing the length of the support member 301, it is possible to change the angle of view at the time of shooting.

Pixels 103a, which are light receiving elements, are regularly arranged in a grid pattern on the surface of the image sensor 103. The image sensor 103 converts an optical image received by the pixel 103a into an image signal which is an electric signal.

The intensity of the light transmitted through the photographing pattern 105 is modulated by the pattern, and the transmitted light is received by the image sensor 103. The image sensor 103 includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Sensor) image sensor.

The image signal output from the image sensor 103 is subjected to processing such as noise removal by the fringe scan processing unit 106, and the image-processed data by the image processing unit 107 is output to the controller 108. When the output is output to a host computer or an external recording medium, the controller 108 converts the data format and outputs the data so as to be compatible with an interface such as USB (Universal Serial Bus).

Subsequently, the imaging principle in the image pickup apparatus 101 will be described. First, the photographing pattern 105 is a concentric pattern in which the pitch becomes finer in inverse proportion to the radius from the center, and the radius r from the reference coordinate which is the center of the concentric circle and the coefficient β are used.

と定義することができる。撮影用パターン１０５は、この式に比例して透過率変調されているものとする。なお、以降、簡単化のためにｘ軸方向についてのみ数式で説明するが、同様にｙ軸方向について考慮することで２次元に展開して考えることが可能である。

Can be defined as. It is assumed that the photographing pattern 105 is transmittance-modulated in proportion to this equation. Hereinafter, for the sake of simplicity, only the x-axis direction will be described by a mathematical formula, but it is possible to expand and think in two dimensions by similarly considering the y-axis direction.

Plates with such stripes are called Gabor Zone Plates (GZPs) or Fresnel Zone Plates (FZPs).

FIG. 4 is a diagram showing an example of a photographing pattern and a developing pattern according to the first embodiment. Specifically, FIG. 4 is an example of a Gabor zone plate represented by the above equation (1).

FIG. 5 is a diagram showing another example of the photographing pattern and the developing pattern according to the first embodiment. Specifically, it is an example of a Fresnel zone plate using a pattern obtained as a result of binarizing the above equation (1) with a threshold value of 1.

As shown in FIG. 6, it is assumed that parallel light is incident on the pattern substrate 104 having a thickness d on which the photographing pattern 105 is formed at an angle θ ₀ in the x-axis direction. Geometrically and optically, with the refraction angle in the pattern substrate 104 as θ, the light multiplied by the transmittance of the lattice on the surface is incident on the image sensor 103 with a deviation of k = d · tan θ. At this time,

のような強度分布を持つ投影像が画像センサ１０３上で検出される。なお、Φは上式（１）の透過率分布の初期位相を示す。この撮影用パターン１０５の投影像の例を図７に示す。

A projected image having such an intensity distribution is detected on the image sensor 103. Note that Φ indicates the initial phase of the transmittance distribution in the above equation (1). An example of the projected image of the photographing pattern 105 is shown in FIG.

FIG. 7 is a diagram showing an example of a projected image of a shooting pattern. As shown in FIG. 7, when the parallel light as shown in FIG. 6 is incident using the photographing pattern 105, the image sensor 103 is projected on the image sensor 103 with a shift of k as shown in the above equation (2). Will be done. This is the output from the image pickup module 102.

Next, the image processing unit 107 performs the development processing, and the development processing by the correlation development method and the moire development method will be described.

In the correlation development method, the image processing unit 107 calculates a cross-correlation function between the projected image of the photographing pattern 105 shown in FIG. 7 and the developing pattern 801 shown in FIG. 8, and is shown in FIG. A bright spot with a shift amount k can be obtained. In general, if the cross-correlation operation is performed by a two-dimensional convolution operation, the amount of calculation becomes large.

FIG. 8 is a diagram showing an example of a developing pattern. Specifically, the developing pattern 801 has the same pattern as the Gabor zone plate shown in FIG. 4 and the FZP (Fresnel zone plate) shown in FIG. That is, in the present embodiment, the development pattern 801 does not have to have a physical substance, and may exist as information used in image processing.

FIG. 9 is a diagram showing an example of a developed image by the correlation development method. When developed by the correlation development method, as described above, it is possible to obtain a developed image in which a certain bright spot is shifted by k.

Here, an example in which the cross-correlation operation is performed by using the Fourier transform without performing the two-dimensional convolution operation will be described using a mathematical formula. First, since the developing pattern 801 uses a Gabor zone plate or a Fresnel zone plate as in the photographing pattern 105, the developing pattern 801 uses the initial phase Φ.

と表せる。なお、現像用パターン８０１は画像処理内で使用するため、上式（１）のように１でオフセットさせる必要はなく、負の値を有していても問題ない。

Can be expressed as. Since the developing pattern 801 is used in the image processing, it is not necessary to offset it by 1 as in the above equation (1), and there is no problem even if it has a negative value.

The Fourier transforms of the above equations (2) and (3) are, respectively.

のようになる。ここで、Ｆはフーリエ変換の演算を表し、ｕはｘ方向の周波数座標、括弧を伴うδはデルタ関数である。この式で重要なことはフーリエ変換後の式もまたフレネルゾーンプレートやガボールゾーンプレートとなっている点である。よって、この数式に基づいてフーリエ変換後の現像用パターンを直接的に生成してもよい。これにより演算量を低減可能である。

become that way. Here, F represents the operation of the Fourier transform, u is the frequency coordinate in the x direction, and δ with parentheses is the delta function. What is important in this equation is that the equation after the Fourier transform is also a Fresnel zone plate or a Gabor zone plate. Therefore, the development pattern after the Fourier transform may be directly generated based on this mathematical formula. This makes it possible to reduce the amount of calculation.

Next, when the above equation (4) and the above equation (5) are multiplied,

となる。この指数関数で表された項ｅｘｐ（－ｉｋｕ）が信号成分であり、この項をフーリエ変換すると、

Will be. The term exp (-iku) represented by this exponential function is a signal component, and when this term is Fourier transformed,

のように変換され、元のｘ軸においてｋの位置に輝点を得ることができる。この輝点が無限遠の光束を示しており、図１の撮像装置１０１により得られる撮影像にほかならない。

It is possible to obtain a bright spot at the position of k on the original x-axis. This bright spot indicates a luminous flux at infinity, and is nothing but a photographed image obtained by the image pickup apparatus 101 of FIG.

In such a correlation development method, as long as the autocorrelation function of the pattern has a single peak, it may be realized by a pattern not limited to the Fresnel zone plate or the Gabor zone plate, for example, a random pattern.

Next, in the moire development method, by multiplying the projected image of the photographing pattern 105 shown in FIG. 7 and the developing pattern 801 shown in FIG. 8, moire fringes as shown in FIG. 10 are generated. By Fourier transforming, a bright spot having a shift amount of (kβ / π) as shown in FIG. 11 can be obtained.

FIG. 10 is a diagram showing an example of moire fringes by the moire development method. Specifically, as shown in FIG. 10, the result of multiplication of the projected image of the photographing pattern 105 shown in FIG. 7 and the developing pattern 801 shown in FIG. 8 is obtained as moire fringes.

If this moire fringe is expressed by a mathematical formula,

となる。この展開式の第３項が信号成分であり、２つのパターンのずれの方向にまっすぐな等間隔の縞模様を重なり合った領域一面に作ることがわかる。このような縞と縞の重ね合わせによって相対的に低い空間周波数で生じる縞をモアレ縞と呼ぶ。

Will be. It can be seen that the third term of this expansion formula is the signal component, and straight striped patterns at equal intervals in the direction of deviation of the two patterns are formed on the entire overlapping region. The fringes generated at a relatively low spatial frequency by superimposing such fringes are called moire fringes.

The two-dimensional Fourier transform of this third term is

のようになる。ここで、Ｆはフーリエ変換の演算を表し、ｕはｘ方向の周波数座標、括弧を伴うδはデルタ関数である。この結果から、モアレ縞の空間周波数スペクトルにおいて、空間周波数のピークが（ｕ＝±ｋβ／π）の位置に生じることがわかる。この輝点が無限遠の光束を示しており、図１の撮像装置１０１により得られる撮影像にほかならない。

become that way. Here, F represents the operation of the Fourier transform, u is the frequency coordinate in the x direction, and δ with parentheses is the delta function. From this result, it can be seen that the peak of the spatial frequency occurs at the position (u = ± kβ / π) in the spatial frequency spectrum of the moire fringes. This bright spot indicates a luminous flux at infinity, and is nothing but a photographed image obtained by the image pickup apparatus 101 of FIG.

FIG. 11 is a diagram showing an example of a developed image by the moire development method. In the developed image shown in FIG. 11, a bright spot is obtained at the position of u = ± kβ / π.

In the moire development method, as long as the moire fringes obtained by shifting the pattern have a single frequency, a pattern not limited to the Fresnel zone plate or the Gabor zone plate, for example, an elliptical pattern may be realized.

<Noise Cancellation> In the conversion from the above equation (6) to the above equation (7), and in the conversion from the above equation (8) to the above equation (9), we focused on the signal components, but in reality Terms other than the signal component hinder development as noise. Therefore, noise cancellation based on fringe scan is effective.

For the fringe scan, it is necessary to use a plurality of patterns having different initial phases Φ as the photographing pattern 105. FIG. 12 shows an example of a plurality of patterns.

FIG. 12 is a diagram showing an example of a combination of shooting patterns of the initial phase in the fringe scan. Here, the sensor image taken using four phases of (Φ = 0), (Φ = π / 2), (Φ = π), and (Φ = 3π / 2), that is, phases shifted by π / 2. Is calculated according to the following equation to obtain a complex sensor image (complex sensor image). Expressing this as an expression,

となる。ここで、複素数の現像用パターン８０１は、

Will be. Here, the complex number development pattern 801 is

と表せる。現像用パターン８０１はフリンジスキャン処理内で使用するため、複素数であっても問題ない。

Can be expressed as. Since the developing pattern 801 is used in the fringe scan process, there is no problem even if it is a complex number.

In the moire development method, when the above equation (10) and the above equation (11) are multiplied,

となる。この指数関数で表された項ｅｘｐ（２ｉβｋｘ）が信号成分であり、上式（８）のような不要な項が発生しないため、ノイズがキャンセルされていることが解る。

Will be. It can be seen that the noise is canceled because the term exp (2iβkx) represented by this exponential function is the signal component and the unnecessary term as in the above equation (8) does not occur.

Similarly, when confirming the correlation development method, the Fourier transforms of the above equations (10) and (11) are, respectively.

のようになる。次に、上式（１３）と上式（１４）を乗算すると、

become that way. Next, when the above equation (13) and the above equation (14) are multiplied,

となる。この指数関数で表された項ｅｘｐ（－ｉｋｕ）が信号成分であり、上式（８）のような不要な項が発生しないため、ノイズがキャンセルされていることが解る。

Will be. It can be seen that the noise is canceled because the term exp (-iku) represented by this exponential function is a signal component and an unnecessary term as in the above equation (8) does not occur.

In the above example, a plurality of development patterns having four phases shifted by π / 2, which are (Φ = 0), (Φ = π / 2), (Φ = π), and (Φ = 3π / 2). However, Φ may be set so as to equally divide the angle between 0 and 2π, and is not limited to this phase.

As a method of realizing the above-mentioned shooting with a plurality of patterns, a method of switching the pattern by time division in the fringe scan process and a method of switching the pattern by spatial division can be considered.

In order to realize the time-division fringe scan, for example, a liquid crystal display element capable of electrically switching and displaying a plurality of initial phases shown in FIG. 12 may be used as the photographing pattern 105 in FIG. 1. The switching timing of the liquid crystal display element and the shutter timing of the image sensor 103 are controlled in synchronization, and after acquiring the four images in time series, the fringe scan processing unit 106 performs the fringe scan calculation.

On the other hand, in order to realize the spatial division fringe scan, it is necessary to use the photographing pattern 105 having a plurality of initial phases as shown in FIG. After acquiring one image as a whole, the fringe scan processing unit 106 divides the image into four images corresponding to the pattern of each initial phase, and performs the fringe scan calculation.

FIG. 13 is a diagram showing an example of a shooting pattern of the spatial division fringe scan. The space-divided fringe scan imaging pattern 1300 is an application example of the imaging pattern 105, but is spatially divided on an XY plane (a plane parallel to the image sensor 103) and has a plurality of initial phases ((Φ =). It is a pattern for photography including 0), (Φ = π / 2), (Φ = π), and (Φ = 3π / 2). Subsequently, the fringe scan operation in the fringe scan processing unit 106 will be described.

FIG. 14 is a diagram showing an example of a fringe scan processing flow. First, the fringe scan processing unit 106 acquires sensor images based on a plurality of shooting patterns output from the image sensor 103, but when the spatial division fringe scan is adopted, the acquired sensor images are obtained according to individual shooting patterns. Since it is necessary to divide the image, it is divided into a predetermined area to obtain a plurality of sensor images (step 1401). When the time-division fringe scan is adopted, since a plurality of sensor images with different shooting patterns are obtained according to the passage of time, the division is not performed.

Next, the fringe scan processing unit 106 initializes the complex sensor image for output (step 1402).

Then, the fringe scan processing unit 106 acquires the sensor image of the first initial phase Φ (step 1403).

Then, the fringe scan processing unit 106 multiplies exp (iΦ) according to the initial phase Φ (step 1404).

Then, the fringe scan processing unit 106 adds the multiplication result to the complex sensor image (step 1405).

The fringe scan processing unit 106 repeats the processes from step 1403 to step 1405 by the number of initial phases used (step 1406). For example, in the fringe scan using four phases as shown in FIG. 12, the fringe scan processing unit 106 has the initial phases (Φ = 0), (π / 2), (π), and (3π / 2), respectively. Repeat 4 times in total.

Then, the fringe scan processing unit 106 outputs a complex sensor image (step 1407). The processing by the fringe scan processing unit 106 in steps 1401 to 1407 described above is a processing corresponding to the above equation (10). Next, the image processing in the image processing unit 107 will be described.

FIG. 15 is a diagram showing an example of a processing flow of development processing by a correlation development method. First, the image processing unit 107 acquires the complex sensor image output from the fringe scan processing unit 106, and performs a two-dimensional fast Fourier transform (FFT) operation on the complex sensor image (step 1501). ..

Next, the image processing unit 107 generates a predetermined development pattern 801 to be used for the development process, and multiplies the complex sensor image calculated by the two-dimensional FFT (step 1502).

Then, the image processing unit 107 performs an inverse two-dimensional FFT operation (step 1503). The result of this calculation is a complex number.

Therefore, the image processing unit 107 converts the image to be photographed into a real number and develops it by converting it into an absolute value or extracting the real part from the calculation result of the inverse two-dimensional FFT calculation (step 1504).

After that, the image processing unit 107 performs contrast enhancement processing on the obtained developed image (step 1505). Further, the image processing unit 107 performs color balance adjustment (step 1506) and the like, and outputs the captured image. The above is the development process by the correlation development method.

FIG. 16 is a diagram showing an example of a processing flow of development processing by a moire development method. First, the image processing unit 107 acquires the complex sensor image output from the fringe scan processing unit 106, generates a predetermined development pattern 801 to be used for the development process, and multiplies the complex sensor image (step 1601). ).

Then, the image processing unit 107 obtains a frequency spectrum by a two-dimensional FFT calculation (step 1602).

Then, the image processing unit 107 cuts out data in a necessary frequency region from the frequency spectrum obtained in step 1602 (step 1603).

Subsequent realization processing in step 1504, contrast enhancement processing in step 1505, and color balance adjustment processing in step 1506 are the same as the processing in steps 1504 to 1506 shown in FIG. 15, and thus the description thereof will be omitted.

<Principle of photographing a finite-distance object> Next, FIG. 17 shows a state of projection of the photographing pattern 105 on the image sensor 103 when the subject described above is sufficiently far away (in the case of infinity).

FIG. 17 is a diagram showing an example of projection of a shooting pattern when an object is at an infinite distance. When the spherical wave from the point 1701 constituting a distant object becomes a plane wave while propagating a sufficiently long distance and irradiates the photographing pattern 105, and the projected image 1702 is projected on the image sensor 103, the projected image is It has almost the same shape as the photographing pattern 105. As a result, it is possible to obtain a single bright spot by performing a developing process on the projected image 1702 using a developing pattern. In comparison with this, imaging of a finite-distance object will be described.

FIG. 18 is a diagram showing an example of magnified projection of a photographing pattern when an object is at a finite distance. When the object to be imaged is at a finite distance, the projection of the photographing pattern 105 on the image sensor 103 is magnified more than that of the photographing pattern 105. When a spherical wave from a point 1801 constituting an object illuminates the photographing pattern 105 and the projected image 1802 is projected onto the image sensor 103, the projected image is magnified almost uniformly. The enlargement ratio α is determined by using the distance f from the photographing pattern 105 to the point 1801.

のように算出できる。このため、有限距離の物体に対して、平行光に対して設計された現像用パターンをそのまま用いて現像処理したのでは、単一の輝点を得ることができない。

It can be calculated as follows. For this reason, it is not possible to obtain a single bright spot by developing a finite-distance object using the developing pattern designed for parallel light as it is.

Therefore, if the development pattern 801 is enlarged to match the uniformly enlarged projection image of the photographing pattern 105, a single bright spot can be obtained again with respect to the enlarged projection image 1802. can. For this purpose, the coefficient β of the developing pattern 801 can be corrected by setting it to β / α ² . This makes it possible to selectively reproduce the light from the point 1801 at a distance that is not necessarily infinity. This makes it possible to focus on an arbitrary position and shoot. That is, when shooting is performed with the focus focused on the subject, the magnification of the developing pattern 801 may be determined.

Further, according to this configuration, it is possible to focus on an arbitrary distance after shooting. The configuration in this case is shown in FIG.

FIG. 19 is a diagram showing a configuration of an image pickup apparatus according to a second embodiment of the present invention. The image pickup apparatus according to the second embodiment basically has the same configuration as the image pickup apparatus according to the first embodiment. However, what is different from the first embodiment is the existence of the focus setting unit 1901. The focus setting unit 1901 receives the focus distance setting by the knob provided in the image pickup device 101, the GUI (Graphical User Interface) of the smartphone, and the like, and outputs the focus distance information to the image processing unit 107.

Further, the fact that the focus can be adjusted after shooting in this way means that the image processing unit 107 has various functions such as autofocus and distance measurement because it has depth information. ..

In order to realize such a function such as focus setting, it is necessary to freely change the coefficient β of the developing pattern 801. However, it is similar to the processing in the fringe scan processing unit 106 described in this embodiment. By performing the fringe scan operation on the surface, the processing using the developing pattern 801 can be performed independently, and the processing can be simplified.

<Problem of spatial division fringe scan: subject distance> As described in FIG. 18, when the object to be imaged is at a finite distance, the projection of the photographing pattern 105 on the image sensor 103 is enlarged from that of the photographing pattern 105. To. This can be a problem when using spatially divided fringe scans.

FIG. 20 is a diagram showing an example of the positional relationship between the point light source and the image pickup device. As shown in FIG. 20, consider the case where the object to be imaged is at a long distance f (point light source 2001) and the case where the object to be imaged is at a short distance f'(point light source 2001').

FIG. 21 is a diagram showing an example of a sensor image for each distance of a point light source. As shown in FIG. 21, when the light from the object to be imaged is applied to the photographing pattern 105 of the spatial division fringe scan and the shadow is projected onto the image sensor 103, the image projected is the object to be imaged and the photographed object. It changes depending on the case of the distance f between the pattern and the case of the distance f'. In the spatial division fringe scan, it is necessary for the fringe scan processing unit 106 to divide into four sheets corresponding to the pattern of each initial phase, but the center of the pattern in each quadrant depends on the distance f or f'with the subject. It's changing. For example, the center of the concentric circles in the first quadrant 1302 is displaced to the upper right in the first quadrant 2102 in the case of the distance f'. In this way, if each quadrant is always divided at the same place, a shift will occur and the effect of the fringe scan will be reduced.

In order to solve this problem, after performing image correction using the enlargement ratio α of the equation 16, the data may be divided into four images for each quadrant. In each of FIGS. 22 and 23, the center of each pattern when the center of the sensor image of FIG. 21 is arranged at the origin is indicated by x.

FIG. 22 is a diagram showing an example of pattern center coordinates when the point light source is at infinity. In FIG. 22, the coordinates of the pattern center 2201 in the first quadrant are represented by (x0, y0).

FIG. 23 is a diagram showing an example of pattern center coordinates when the point light source is at a finite distance. In FIG. 23, the coordinates of the pattern center 2301 in the first quadrant are represented by (x1, y1).

Since the coordinates (x1, y1) of the pattern center 2301 in FIG. 23 are expanded from the coordinates (x0, y0) of the pattern center 2201 in FIG. 22 at an enlargement factor α, the coordinates (x1, y1) are coordinated (x0, y1). To convert to y0)

のようにセンサ面上の移動量を特定する行列Ｍを使用して座標変換すればよい。変換した結果得られるセンサ画像を図２４に示す。

The coordinates may be transformed using the matrix M that specifies the amount of movement on the sensor surface as in. The sensor image obtained as a result of the conversion is shown in FIG.

FIG. 24 is a diagram showing a correction example of the sensor image when the point light source is at a finite distance. As shown in FIG. 24, the image is reduced and the same position is always the center of the pattern. Therefore, in this example, if the image is divided into four in each quadrant, a sensor image in each quadrant can be obtained. Since the image is reduced, the obtained sensor image is smaller than the image size before correction. In this case, the margin area 2401 may be complemented with a constant such as 0 or may not be used as a NAN value.

In this way, by transforming the coordinates based on the center of the concentric circles using the matrix M shown in Eq. (17), the projection that occurs when the point light source is at a finite distance, that is, when the subject distance is a finite distance, is expanded. The problem can be solved.

<Problem of spatial division fringe scan: Pattern fixing accuracy> However, it may not be enough to correct the deviation due to the subject distance. It is a case where the accuracy when assembling the image pickup apparatus 101 and the manufacturing accuracy of the photographing pattern 105 are not sufficient and a deviation occurs. FIG. 25 shows an example of a sensor image when the photographing pattern 105 is attached in a misaligned manner.

FIG. 25 is a diagram showing an example of a sensor image when the photographing pattern is rotated. In this example, an example of a sensor image when the photographing pattern 105 is attached by rotating by an angle −θ with respect to the origin is shown.

A method for correcting this will be described. First, the center coordinates 2501 to 2504 of each pattern are calculated by cross-correlation calculation with a reference pattern (for example, an image obtained by enlarging the photographing pattern 105 from the subject distance at an enlargement ratio α). Next, the coordinates of the center of gravity O of the four points of the center coordinates 2501 to 2504 are calculated and arranged so that the center of gravity O overlaps with the origin as shown in FIG. 26.

FIG. 26 is a diagram showing an example of the pattern center coordinates when the photographing pattern is rotated. At this time, in order to convert the coordinates (x1, y1) of the pattern center 2601 in the first quadrant of FIG. 26 to the coordinates (x0, y0) of the pattern center 2201 of FIG.

のようにセンサ面上の回転の移動量を特定する行列Ｈを使用して座標変換すればよい。なお、式（１８）において行列Ｍを使用しているのは、使用する点光源２００１の距離ｆに応じて画像を補正するためである。この結果、図２５の画像は角度θ回転することで補正され、常に同じパターン中心となる画像になるため、図２５、図２６の例であれば各象限で画像を分割すればよい。

The coordinates may be transformed using the matrix H that specifies the amount of rotation on the sensor surface as in. The reason why the matrix M is used in the equation (18) is to correct the image according to the distance f of the point light source 2001 to be used. As a result, the image of FIG. 25 is corrected by rotating the angle θ, and the image always has the same pattern center. Therefore, in the example of FIGS. 25 and 26, the image may be divided in each quadrant.

The matrix H shown in the equations (18) and (19) is a matrix having 2 × 2 elements generally called an affine matrix, and has a relationship of four coordinates (in this example, the center coordinates of each pattern). It is possible to calculate the elements from 2201 to 2204 and the center coordinates 2501 to 2504). The points used to calculate the affine matrix are correctly two points, but by applying the least squares method using the information of four points, it is possible to obtain a more accurate affine matrix.

Next, another example in which the accuracy when assembling the image pickup apparatus is low will be described. For example, consider a case where the photographing pattern 105 is arranged diagonally in the thickness direction as shown in FIG. 27. FIG. 28 shows an example of a sensor image when the point light source 2001 is photographed in this state.

FIG. 28 is a diagram showing an example of a sensor image in a state where the photographing pattern is tilted in the thickness direction. In this case, as shown in FIG. 28, the sensor image has trapezoidal distortion. The trapezoidal distortion cannot be completely corrected by the affine matrix as in Eq. (18), but it is possible to correct the trapezoidal distortion more easily and accurately by using the homography matrix.

A method for correcting this will be described. First, the center coordinates 2801 to 2804 of each pattern are calculated by cross-correlation calculation with a reference pattern (for example, an image obtained by enlarging the photographing pattern 105 from the subject distance at the enlargement ratio α). Next, the coordinates of the center of gravity O of the four points of the center coordinates 2801 to 2804 are calculated and arranged so that the center of gravity O overlaps with the origin as shown in FIG. 29.

FIG. 29 is a diagram showing an example of pattern center coordinates in a state where the photographing pattern is tilted in the thickness direction. At this time, in order to convert the coordinates (x1, y1) of the pattern center 2801 of the first quadrant of FIG. 29 to the coordinates (x0, y0) of the pattern center 2201 of FIG.

のようにセンサ面上の移動量を特定する行列Ｈを使用して座標変換すればよい。この行列Ｈは一般的にホモグラフィ行列と呼ばれる３×３の要素を持つ行列であり、４つの座標の関係（この例では各パターンの中心座標２２０１～２２０４と中心座標２８０１～２８０４）から要素を算出することが可能である。なお、行列Ｍを使用しているのは、使用する点光源２００１´の距離ｆ´に応じて画像を補正するためである。結果、図２８の画像は台形歪み（アオリ）および回転が補正され、常に同じパターン中心となる画像になるため、この例であれば各象限で画像を分割すればよい。

The coordinates may be transformed using the matrix H that specifies the amount of movement on the sensor surface as in. This matrix H is a matrix having 3 × 3 elements generally called a homography matrix, and the elements are selected from the relationship of four coordinates (in this example, the center coordinates 2201 to 2204 and the center coordinates 2801 to 2804 of each pattern). It is possible to calculate. The matrix M is used in order to correct the image according to the distance f'of the point light source 2001' to be used. As a result, the image of FIG. 28 is corrected for trapezoidal distortion (tilt) and rotation, and always becomes an image having the same pattern center. Therefore, in this example, the image may be divided in each quadrant.

The configuration and processing for realizing the above distortion correction will be described with reference to FIGS. 30 to 33. FIG. 30 shows a configuration that realizes distortion correction.

FIG. 30 is a diagram showing a configuration example of an image pickup apparatus according to a third embodiment of the present invention. The image pickup apparatus shown in FIG. 30 is basically the same as the image pickup apparatus according to the second embodiment shown in FIG. 19, but is partially different. The differences will be mainly described below.

The image pickup apparatus according to the third embodiment uses a correction value storage unit 3001 as a parameter storage unit for measuring and storing a correction parameter including a matrix H and a matrix M for distortion correction, and a distortion using the correction parameter. A fringe scan processing unit 3002 for performing correction is provided. As shown in FIG. 31, the correction value storage unit 3001 may be included in the image pickup module 102. In that case, the correction value storage unit 3001 operates as a parameter output unit that outputs a correction value, that is, a parameter for correction, in response to a request via an API (Application Programming Interface).

FIG. 31 is a diagram showing a configuration example of an image pickup apparatus according to a modified example of the third embodiment of the present invention. Considering that there are individual differences in the correction parameters due to variations in the mounting of the image pickup module 102, in a configuration in which the image pickup module 102 is separated from the image pickup device 101, a correction value storage unit is stored in the image pickup module 102 as shown in FIG. 31. The form accompanied by 3001 is more versatile. In all the embodiments, the correction value calculation method (calibration method) is common. First, a method of calculating a correction value (calibration method) will be described.

FIG. 32 is a diagram showing an example of a calibration processing flow. First, as shown in FIG. 20, the calibrator performs the irradiation of the point light source 2001 on the optical axis of the image pickup module 102 toward the image pickup module 102 (step 3201). The point light source 2001 may be any light source such as an LED (Light Emitting Diode) light source or the sun, as long as it is at infinity or the distance to the subject is known.

Then, the calibrator causes the image pickup module 102 to acquire the sensor image (step 3202).

Then, the fringe scan processing unit 3002 calculates the center coordinates of the FZA (Fresnel Zone Aperture) (step 3203). Specifically, the fringe scan processing unit 3002 performs cross-correlation calculation with a reference pattern (for example, an image obtained by enlarging the photographing pattern 105 with an enlargement ratio α obtained from the subject distance) to obtain the center coordinates of each pattern (eg,). : Center coordinates 2801 to 2804) are calculated.

Then, the fringe scan processing unit 3002 calculates the center of gravity O from this center coordinate group (step 3204). Then, the fringe scan processing unit 3002 arranges the center of gravity O at the origin.

Then, the fringe scan processing unit 3002 calculates the correction matrix H (step 3205). Specifically, the fringe scan processing unit 3002 is a correction matrix based on the relationship between the deviations of the four center coordinates (eg, the relationship between the rotation of the center coordinates 2201 to 2204 and the relationship between the center coordinates 2801 to 2804 and the trapezoidal distortion). A 3 × 3 element of the homography matrix H to be H is calculated.

Then, the fringe scan processing unit 3002 stores the correction matrix H and the position of the center of gravity O as correction parameters in the correction value storage unit 3001 (step 3206).

The above is the calibration process. According to the calibration process, the matrix H for correcting the sensor image and the position of the center of gravity O can be specified and stored as parameters in the correction value storage unit 3001.

Subsequently, the fringe scan processing unit 3002 performs the distortion correction processing.

FIG. 33 is a diagram showing an example of a distortion correction processing flow of the sensor image. First, the fringe scan processing unit 3002 reads the position of the center of gravity O from the correction value storage unit 3001 and moves the center of gravity O of the sensor image to the coordinates that are the origin (step 3301).

Next, the fringe scan processing unit 3002 performs correction by the matrix M (step 3302). Specifically, the fringe scan processing unit 3002 acquires information on the distance f'to be focused from the focus setting unit 1901, and performs correction using the matrix M corresponding to the distance f'.

Then, the fringe scan processing unit 3002 performs correction by the matrix H (step 3303). Specifically, the fringe scan processing unit 3002 performs correction by the matrix H acquired from the correction value storage unit 3001.

After that, the same processing as the processing of steps 1401 to 1407 of the fringe scan processing flow shown in FIG. 14 is performed on the sensor image corrected by the matrix H.

In this example, the movement of the position of the matrix H and the center of gravity O is processed separately, but when a 3 × 3 matrix homography matrix is used as in the equation (20), 3 × 3 is used. Since it is possible to include the correction amount of the movement of the center of gravity O in the matrix, it is not always necessary to process them separately.

In addition, although the calibration was explained on the premise that it is performed in advance, the correction parameters are recalculated while shooting when a point light source or the sun is detected, and the information in the correction value storage unit 3001 is updated. May be good. That is, the calibration is not limited to the prior implementation, and the calibration may be performed at any time in the usage environment.

The above is the image pickup apparatus according to the third embodiment. According to the image pickup apparatus according to the third embodiment, the error in assembling the image pickup apparatus and the error in the production of the shooting pattern are corrected, and the distortion related to the focus adjustment is corrected to achieve high-precision spatial division. It becomes possible to carry out a fringe scan.

In the configuration of the third embodiment, the distortion is corrected before the fringe scan of the sensor image (in the fringe scan processing unit 3002, the processes of steps 3301 to 3303 are performed before the processes of steps 1401 to 1407. However, when the sensor image is transmitted to another device wirelessly or by wire, it may be desirable that the information to be transmitted is unprocessed in consideration of preventing information loss. That is, it may be preferable to perform the distortion correction process after the fringe scan. A method of correcting distortion after fringe scanning of a sensor image will be described with reference to FIGS. 34 and 35.

FIG. 34 is a diagram showing a configuration example of the image pickup apparatus according to the fourth embodiment of the present invention. The image pickup apparatus according to the fourth embodiment is substantially the same as the image pickup apparatus according to the third embodiment, but there are some differences. The image pickup apparatus according to the fourth embodiment is different from the image pickup apparatus according to the third embodiment in that the image processing unit 3401 reads out and uses the information of the correction value storage unit 3001. That is, the image pickup apparatus according to the fourth embodiment does not correct the distortion of the sensor image, but adds the distortion corresponding to the sensor image to the developing pattern 801. In the third embodiment, the matrix H for converting the coordinates (x1, y1) into the coordinates (x0, y0) was calculated, but in this embodiment, the matrix H is calculated.

のように、発想が逆の変換となる座標（ｘ０，ｙ０）を座標（ｘ１，ｙ１）に変換するための行列Ｈ’、行列Ｍ’を算出する。

As described above, the matrix H'and the matrix M'for converting the coordinates (x0, y0) whose ideas are the opposite conversion to the coordinates (x1, y1) are calculated.

The flow of the development process in this case is the same as the process flow of the development process by the correlation development method of FIG. 15 described above, but is partially different. In the fourth embodiment, the generation of the developing pattern 801 in the step 1502 of the developing process by the correlation developing method is carried out in the process flow shown in FIG. 35.

FIG. 35 is a diagram showing an example of a distortion correction processing flow of a developing pattern. The distortion correction process of the development pattern is started in step 1502 of the development process by the correlation development method.

Before executing the distortion correction processing flow of the developing pattern, the image processing unit 3401 preliminarily applies the matrix H'of the equation (21) for distortion addition, as in the example of the calibration processing flow shown in FIG. 32. Calculate as a matrix of. Specifically, the image processing unit 3401 is a 3 × 3 element of the homography matrix H'that becomes the correction matrix H'from the relationship of the four coordinates (eg, the center coordinates 2201 to 2204 and the center coordinates 2801 to 2804). Is calculated in advance.

Then, at the time of development, that is, in the distortion correction processing of the development pattern, the image processing unit 3401 generates a pattern image of concentric circles for spatial division fringe scan as shown in FIG. 20 (step 3501).

Then, the image processing unit 3401 moves the center of gravity O (step 3502). Specifically, the image processing unit 3401 reads the position of the center of gravity O from the correction value storage unit 3001 and moves the center of the concentric circle pattern image for spatial division fringe scan by the position of the center of gravity O from the coordinates as the origin. ..

Then, the image processing unit 3401 performs the correction by the matrix H'acquired from the correction value storage unit 3001 (step 3503). Specifically, the matrix H'read from the correction value storage unit 3001 is used to convert a pattern image of concentric circles for spatial division fringe scan.

Then, the image processing unit 3401 performs the correction by the matrix M'acquired from the correction value storage unit 3001 (step 3504). Specifically, the image processing unit 3401 acquires the focus distance f'from the focus setting unit 1901, acquires the information of the matrix M'corresponding to the distance f'from the correction value storage unit 3001, and corrects the image. implement.

Then, the image processing unit 3401 divides the pattern image into four (step 3505). The subsequent processing is the same as the development processing of FIG. 15 for multiplying the complex sensor image calculated by the two-dimensional FFT.

Since the development pattern 801 is distorted in this method, in the moire development method, the moire at the time of point light source development does not become a single frequency and the image is blurred. Therefore, it can be said that the correlation development method has a higher affinity and is preferable to the moire development method.

The above is the image pickup apparatus according to the fourth embodiment. According to the method / configuration according to the fourth embodiment, a highly accurate space is obtained by correcting an error in assembling an image pickup device and an error in making a pattern for photographing, and also correcting a distortion related to focus adjustment. It is possible to perform a split fringe scan.

In the third and fourth embodiments, a method of correcting an error in assembling an image pickup apparatus, an error in producing a pattern for photographing, and a distortion in focus adjustment has been described. However, corrections for defects, stains, and dust in the shooting pattern and the image sensor were not taken into consideration. In the fifth embodiment, this defect protection method will be described with reference to FIGS. 36 to 40. First, the problem of this defect will be clarified with reference to FIG. 37.

FIG. 37 is a diagram showing an example of an image in the case where stains or the like are attached to the photographing pattern. In the case of time-division fringe scan, defects appear at the same positions in all four patterns, so that defects are recorded in one place as a sensor image. In the case of the spatial division fringe scan, since the defect exists only in the quadrant of the pattern 3803 like the defect 3801, it cannot be matched with other patterns. By mapping the luminance distributions along the dotted lines for the quadrant of pattern 3802 and the quadrant of pattern 3803 of FIG. 37, the graphs shown in FIGS. 38 and 39 can be obtained, respectively.

FIG. 38 is a diagram showing an example of brightness distribution of an ideal sensor image. That is, as shown in the pattern 3802, it is possible to obtain a signal component in which the brightness of the mask portion of the concentric pattern is reduced.

FIG. 39 is a diagram showing an example of luminance distribution of a defective sensor image. That is, as shown in the pattern 3803, a signal component whose brightness is reduced for the mask portion of the concentric pattern can be obtained, but the brightness is reduced irrespective of the mask portion due to a part of the defect 3801. As a matter of course, the degree of decrease in luminance differs depending on the degree of defects 3801. That is, if the defect is minor, the degree of decrease in luminance is small, and if the defect is severe, the degree of decrease in luminance is large.

In the state shown in FIG. 37, the fringe scan of the defective portion is not performed correctly, and the error spreads to the entire image, resulting in deterioration of image quality (significant image cannot be obtained). This is because the luminance difference in the defect region is generally larger than the luminance difference of the signal component, and the degree of influence is large. By specifying the defect position and excluding it from the developing process, the influence of the defect becomes smaller and the influence of the S / N ratio becomes worse. The configuration of the fifth embodiment is shown in FIG.

FIG. 36 is a diagram showing a configuration example of the image pickup apparatus according to the fifth embodiment of the present invention. The image pickup apparatus according to the fifth embodiment is basically the same as the image pickup apparatus according to the fourth embodiment, but there are some differences.

The image pickup apparatus according to the fifth embodiment is different from the image pickup apparatus according to the fourth embodiment in that it includes a defect detection unit 3601 and that the image processing unit 3602 performs defect protection processing. The defect detection unit 3601 acquires a sensor image from the fringe scan processing unit 106, and as shown in FIGS. 38 and 39, when it detects a region having a brightness lower than the defect detection threshold value, it outputs a defect signal specifying the region. At the same time, it is stored in the correction value storage unit 3001. That is, the information for specifying the position where the defect signal is output is stored in the correction value storage unit 3001. Such information is not limited to, for example, information that specifies the coordinates on the defective sensor one by one, but may be information that identifies a rectangular area on the sensor, or even the center and radius of the smallest circle that includes the defective part. good.

The simplest defect detection method by the defect detection unit 3601 is as described above, but when the amplitude of the signal component is large, it may be difficult to distinguish between the defect and the signal. Therefore, the defect detection unit 3601 has a relationship of π-shifted phase, that is, concentric circles, such as a combination of sensor images of Φ = 0 and π in fringe scan, or a combination of sensor images of Φ = π / 2 and 3π / 2. Defects may be discriminated by adding two sensor images in which black and white are inverted. It is desirable that this process be performed between steps 1401 and 1402 of the fringe scan process of FIG. The results of performing this addition on the signals shown in FIGS. 38 and 39 are shown in FIG. 40.

FIG. 40 is a diagram showing an example of defect detection by inversion pattern synthesis. As shown in FIG. 40, the normal signal component is canceled by the offset and only the luminance fluctuation component remains. In response to this response, the defect detection unit 3601 outputs a defect signal and stores the correction value in the region where the residual brightness is below or above the defect detection threshold value using the defect detection threshold value within the range of ± constant of the average value. It is stored in the unit 3001. According to this method, stable defect detection is possible regardless of the subject. This defect detection may be performed for each of all the sensor images, or the inversion pattern may be paired and performed for each pair.

Then, in the image processing unit 3602, using the defect signal output from the defect detection unit 3601, a process of not using the portion corresponding to the defect signal of the developing pattern as a NAN value, that is, a data mask process is performed.

When the defect signal is applied not to the sensor image but to the development pattern of the image processing unit 3602, it corresponds to the defect signal by using the matrix H'and the matrix M'as in the fourth embodiment. It is desirable to correct the position to be used.

Further, the defect signal is input not to the image processing unit 3602 but to the fringe scan processing unit 3002 of FIG. 30 of the third embodiment, and all the patterns of the sensor image have the same relative positions (the relative positions of the pattern center and the defect signal are the same). A process that does not use the NAN value may be performed.

The fifth embodiment has been described above. According to the image pickup apparatus according to the fifth embodiment, not only the error in assembling the image pickup apparatus and the error in the production of the photographing pattern are corrected, and the distortion related to the focus adjustment is corrected, but also the photographing pattern is corrected. It is possible to perform more accurate spatial division fringe scan by correcting defects, stains, and dust in the image sensor.

The present invention is not limited to the above examples, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations.

Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment.

Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.
In addition, the control lines and information lines indicate those that are considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

The present invention has been described above with a focus on examples.

101 ... Imaging device, 102 ... Imaging module, 103 ... Image sensor, 103a ... Pixels, 104 ... Pattern substrate, 105 ... Shooting pattern, 106 ... Fringe scan processing unit , 107 ... Image processing unit, 108 ... Controller, 301 ... Support member, 801 ... Development pattern, 1300 ... Shooting pattern, 1302 ... First quadrant, 1701, 1801 ... .. Point, 1702, 1802 ... Projection image, 1901 ... Focus setting unit, 2001, 2001'... Light source, 2101 ... Second quadrant, 2102 ... First quadrant, 2103 ... Third quadrant, 2104 ... Fourth quadrant, 2401 ... Margin area, 2201,2301,251,260,2801 ... First quadrant pattern center, 2502,2602,2802 ... Second quadrant Pattern center, 2503, 263, 2803 ... Third quadrant pattern center, 2504, 264, 2804 ... Fourth quadrant pattern center, 3001 ... Correction value storage unit, 3002 ... Fringe scan processing unit 3,401 ... Image processing unit, 3601 ... Defect detection unit, 3602 ... Image processing unit, 3801 ... Defect, 3802 ... Fourth quadrant pattern, 3803 ... First quadrant pattern ..

Claims (10)

An image sensor that converts light into an electrical signal and generates a sensor image,
A modulator that modulates the intensity of light detected by the image sensor based on a shooting pattern, and
A parameter storage unit that stores parameters used for executing a predetermined correction process for a plurality of the sensor images captured by different imaging patterns among the sensor images.
Have,
The parameter includes information for correcting trapezoidal distortion with reference to the center of concentric circles of the imaging pattern as information for specifying the amount of movement on the image sensor surface for each imaging pattern.
An imaging device characterized by this. An image sensor that converts light into an electrical signal and generates a sensor image,
A modulator that modulates the intensity of light detected by the image sensor based on a shooting pattern, and
A parameter storage unit that stores parameters used for executing a predetermined correction process for a plurality of the sensor images captured by different imaging patterns among the sensor images.
Have,
The shooting pattern is provided with a concentric pattern.
The parameters include the amount of movement of the center of gravity between the centers of concentric circles of the imaging pattern and the center of rotation as the center of rotation as information for specifying the amount of movement of the image sensor on the sensor surface for each imaging pattern. Contains information that identifies information that corresponds to the rotation angle to be
An imaging device characterized by this. An image sensor that converts light into an electrical signal and generates a sensor image,
A modulator that modulates the intensity of light detected by the image sensor based on a shooting pattern, and
A parameter storage unit that stores parameters used for executing a predetermined correction process for a plurality of the sensor images captured by different imaging patterns among the sensor images.
Have,
The predetermined correction process is a homography transformation, and is
The parameter is information for specifying the 3 × 3 matrix used for the homography transformation.
An imaging device characterized by this. An image sensor that converts light into an electrical signal and generates a sensor image,
A modulator that modulates the intensity of light detected by the image sensor based on a shooting pattern, and
A parameter storage unit that stores parameters used for executing a predetermined correction process for a plurality of the sensor images captured by different imaging patterns among the sensor images.
Have,
The predetermined correction process is an affine transformation.
The parameter is information for specifying the 2 × 2 matrix used for the affine transformation.
An imaging device characterized by this. An image sensor that converts light into an electrical signal and generates a sensor image,
A modulator that modulates the intensity of light detected by the image sensor based on a shooting pattern, and
A parameter storage unit that stores parameters used for executing a predetermined correction process for a plurality of the sensor images captured by different imaging patterns among the sensor images.
Have,
A defect detection unit for detecting a defect portion of the sensor image is provided.
The predetermined correction process is a process of excluding the defective portion.
The parameter is information for identifying the defect site, and is
The defect detection unit synthesizes the sensor images obtained by the imaging pattern having an inversion pattern among the imaging patterns, and detects a portion having an output equal to or higher than a predetermined value as the defect portion.
An imaging device characterized by this. It is an imaging method using an imaging device.
The image pickup device
A modulation step that modulates the intensity of the light transmitted through the shooting pattern,
An image generation step of converting the modulated light into an electric signal by an image sensor to generate a sensor image, and
The fringe scan processing unit reads from a predetermined storage unit a parameter used to execute a predetermined correction processing for a plurality of the sensor images captured by different imaging patterns among the sensor images, and executes the correction processing. Steps and
And carry out
The parameter includes information for correcting trapezoidal distortion with reference to the center of concentric circles of the imaging pattern as information for specifying the amount of movement on the image sensor surface for each imaging pattern.
An imaging method characterized by that. It is an imaging method using an imaging device.
The image pickup device
A modulation step that modulates the intensity of the light transmitted through the shooting pattern,
An image generation step of converting the modulated light into an electric signal by an image sensor to generate a sensor image, and
The fringe scan processing unit reads from a predetermined storage unit a parameter used to execute a predetermined correction processing for a plurality of the sensor images captured by different imaging patterns among the sensor images, and executes the correction processing. Steps and
And carry out
The shooting pattern is provided with a concentric pattern.
The parameters include the amount of movement of the center of gravity between the centers of concentric circles of the imaging pattern and the center of rotation as the center of rotation as information for specifying the amount of movement of the image sensor on the sensor surface for each imaging pattern. Contains information that identifies information that corresponds to the rotation angle to be
An imaging method characterized by that. It is an imaging method using an imaging device.
The image pickup device
A modulation step that modulates the intensity of the light transmitted through the shooting pattern,
An image generation step of converting the modulated light into an electric signal by an image sensor to generate a sensor image, and
The fringe scan processing unit reads out from a predetermined storage unit a parameter used for executing a predetermined correction process for a plurality of the sensor images captured by different imaging patterns among the sensor images, and executes the predetermined correction process. Correction steps to be performed and
And carry out
The predetermined correction process is a homography transformation, and is
The parameter is information for specifying the 3 × 3 matrix used for the homography transformation.
An imaging method characterized by that. It is an imaging method using an imaging device.
The image pickup device
A modulation step that modulates the intensity of the light transmitted through the shooting pattern,
An image generation step of converting the modulated light into an electric signal by an image sensor to generate a sensor image, and
The fringe scan processing unit reads out from a predetermined storage unit a parameter used for executing a predetermined correction process for a plurality of the sensor images captured by different imaging patterns among the sensor images, and executes the predetermined correction process. Correction steps to be performed and
And carry out
The predetermined correction process is an affine transformation.
The parameter is information for specifying the 2 × 2 matrix used for the affine transformation.
An imaging method characterized by that. It is an imaging method using an imaging device.
The image pickup device
A modulation step that modulates the intensity of the light transmitted through the shooting pattern,
An image generation step of converting the modulated light into an electric signal by an image sensor to generate a sensor image, and
The fringe scan processing unit reads out from a predetermined storage unit a parameter used for executing a predetermined correction process for a plurality of the sensor images captured by different imaging patterns among the sensor images, and executes the predetermined correction process. Correction steps to be performed and
And carry out
A defect detection unit for detecting a defect portion of the sensor image is provided.
The predetermined correction process is a process of excluding the defective portion.
The parameter is information for identifying the defect site, and is
The defect detection unit synthesizes the sensor images obtained by the imaging pattern having an inversion pattern among the imaging patterns, and detects a portion having an output equal to or higher than a predetermined value as the defect portion.
An imaging method characterized by that.

Images