JP2019087796A - Imaging apparatus - Google Patents

Abstract

To provide an imaging apparatus which has a simple constitution, a thin thickness, and an expanded field of view in closeup.SOLUTION: An imaging apparatus 101 comprises: a modulator 3001 which modulates light intensity by using a compound eye lattice pattern 3003 having plural basic lattice patterns 3002; plural image sensors 3004 which convert light having passed through the modulator into image data and output the image data; and an image processing unit 3006 which performs image processing of plural pieces of image data outputted from the image sensors.SELECTED DRAWING: Figure 30

Description

  The present invention relates to an imaging device.

  Imaging devices mounted on devices such as IoT (Internet of things) devices and wearable devices are required to be thinner and lower in cost in order to be incorporated in a limited space. Therefore, for example, as disclosed in Patent Document 1, an imaging method capable of obtaining an object image without using a lens has been proposed. For example, Patent Document 2 proposes an “imaging device that enables a finger authentication device with a small thickness that can be installed in a portable information device” using a microlens array.

US Patent Application Publication No. 2015/0219808 JP, 2011-203792, A

  The above-mentioned Patent Document 1 is an imaging method in which a special grid pattern is disposed in front of an image sensor, and an image of an object is obtained by solving an inverse problem by signal processing using an oblique shadow pattern of the grid pattern received by the image sensor. . Although this imaging method enables thinning of an imaging device, there is a problem that the operation for obtaining an image becomes complicated.

  Further, in the above-mentioned Patent Document 2, “The microlens array of the former stage and the diaphragm array are opposed to each other, the diaphragm of the diaphragm array is provided in the vicinity of the focal point of the light from the subject of the microlens array of the former stage. By simultaneously performing light focusing with an array, this is an imaging method that “makes the optical system used for the finger vein authentication device small and can realize a small and thin finger vein authentication device”. Although this imaging method enables thinning of an image pickup apparatus capable of close-up photography, it is particularly thin in that it requires a distance for focusing with a lens and a space for arranging two lens arrays. Has become the limit of

  An object of the present invention is to provide an imaging device which has a simple configuration, is thin, and has a wide field of view at close-up time.

  Although this application contains multiple means to solve at least one part of the said subject, if the example is given, it is as follows.

  One embodiment of the present invention is an imaging device, which comprises: a modulator that modulates the intensity of light using a multi-eye grid pattern having a plurality of basic grid patterns; and converting light transmitted through the modulator into image data A plurality of image sensors for outputting and an image processing unit for performing image processing of a plurality of image data output from the plurality of image sensors are provided.

  According to the present invention, it is possible to provide an imaging device which has a simple configuration, is thin, and enlarges the field of view at the time of close-up shooting.

  Problems, configurations, and effects other than those described above will be apparent from the description of the embodiments below.

It is a figure showing an example of composition of an imaging device concerning a 1st embodiment. It is a figure which shows the structural example of a modulator. It is a figure which shows the other structural example of a modulator. It is a figure which shows a mode that the object of the external world is image | photographed using an imaging device. It is a flowchart which shows an example of the image processing of an image processing part. It is a figure explaining that the projection image from the grating | lattice board | substrate surface to a back surface by oblique incidence parallel light produces in-plane shift | offset | difference. It is a figure explaining generation | occurrence | production and frequency spectrum of a moire fringe when the axis of the grating | lattice on both sides of a grating | lattice board | substrate aligns, (a) shows the case of incident angle: 0, (b) shows the case of incident angle: + θ. (C) shows the case of the incident angle: -θ. It is a figure which shows an example of the grating | lattice board | substrate which shifted and arrange | positioned the axis | shaft of a surface grating | lattice and a back surface grating | lattice. It is a figure explaining generation | occurrence | production and frequency spectrum of a moire fringe in case the grating | lattice of both sides of a grating | lattice board | substrate is shifted and arrange | positioned, (a) shows the case of incident angle: 0, (b) shows the case of incident angle: + θ (C) shows the case of the incident angle: -θ. It is a figure which shows an example of a grating | lattice pattern. It is a figure which shows the other example of a grating | lattice pattern. It is a figure which shows the further another example of a grating | lattice pattern. It is a figure explaining the angle which the light from each point which comprises an object makes with respect to an image sensor. It is a figure explaining that a front side lattice pattern is projected, when an object is at infinite distance. It is a figure which shows the example of the moire fringes produced | generated when an object exists in infinite distance. It is a figure explaining that a front side lattice pattern is expanded, when an object is in finite distance. It is a figure which shows the example of the moire fringes produced | generated when an object exists in a finite distance. It is a figure which shows the example of the moire fringe which correct | amended the back side grating | lattice pattern, when an object is at a finite distance. It is a figure which shows the structural example of the imaging device which implement | achieves a back side grid | lattice pattern by image processing. It is a figure which shows the structural example of a modulator. It is a flowchart which shows an example of the image processing of an image processing part. It is a figure which shows the structural example of the imaging device which implement | achieves a time division fringe scan. It is a figure which shows the example of the grating | lattice pattern in time division fringe scan, (a) shows the case of a phase (ΦF, = 0 = 0), (b) shows the case of a phase (ΦF, == π / 2) (C) shows the case of phase ((F, == π), and (d) shows the case of phase (ΦF, = 3 = 3π / 2). It is a figure which shows the structural example of the modulator which implement | achieves a time division fringe scan, (a) shows the case of a phase ((PHI) F, (PHI) = 0), (b) is a phase ((PHI) F, (PHI) = pi / 2) Case (c) shows the case of phase (ΦF, == π), and (d) shows the case of phase (ΦF, = 3 = 3π / 2). It is a flowchart which shows an example of the image processing of the image processing part which implement | achieves a time division fringe scan. It is a figure showing an example of composition of an imaging device which realizes space division fringe scan. It is a figure which shows the example of the grating | lattice pattern in space division | segmentation fringe scan. It is a figure explaining field angle _thetamax2 . It is a figure explaining the range and the visual field of the grating | lattice pattern which the light from a certain point which comprises an object irradiates, when an object is in short distance. It is a figure which shows the structural example of the imaging device which implement | achieves wide-field-ization by the combination of the grating | lattice pattern of a compound eye, and several image sensors. It is a figure which shows an example of the lattice pattern of a compound eye. It is a figure explaining the lattice pattern of a compound eye, and the visual field by a plurality of image sensors. It is a flowchart which shows an example of the image processing of an image processing part. It is a figure explaining an example of the synthetic | combination method. It is a figure explaining an example of the synthetic | combination method. It is a flowchart which shows the other example of the image processing of an image processing part. It is a figure explaining the example of arrangement of an image sensor. It is a figure explaining the other example of arrangement of an image sensor. It is a figure explaining the optimization method of the size of an image sensor. It is a figure explaining an example of a gaze direction. It is a figure explaining the other example of a gaze direction. It is a figure which shows an example of the concentric grid pattern projected on a rectangular light-receiving surface. It is a figure which shows an example of the elliptical grid | lattice pattern projected on a rectangular light-receiving surface. It is a figure which shows the other example of the concentric grid pattern projected on a rectangular light-receiving surface. It is a figure which shows the structural example of the finger vein imaging device using the lattice pattern of a compound eye. It is sectional drawing which shows the structural example of the liquid crystal display element as a modulator. It is a figure which shows the structural example of the finger vein imaging device using the lattice pattern of a monocular.

  In the following embodiments, when it is necessary for the sake of convenience, it will be described divided into a plurality of sections or embodiments, but they are not unrelated to each other unless specifically stated otherwise, one is the other Some or all of the variations, details, supplementary explanations, etc. are in a relation.

  Further, in the following embodiments, when referring to the number of elements etc. (including the number, numerical value, quantity, range, etc.), unless explicitly stated otherwise or in principle when clearly limited to a specific number etc. The number is not limited to the specific number, and may be more or less than the specific number.

  Furthermore, in the following embodiments, it is needless to say that the constituent elements (including element steps and the like) are not necessarily essential unless particularly clearly stated and considered to be obviously essential in principle. Yes.

  Similarly, in the following embodiments, when referring to the shapes, positional relationships, etc. of the components etc., the shapes, etc. are substantially excluded unless specifically stated otherwise and in cases where it is considered that this is not clearly apparent in principle. It includes those that are similar or similar to The same applies to the above numerical values and ranges.

  Further, in all the drawings for describing the embodiment, the same reference numeral is attached to the same member in principle, and the repeated description thereof will be omitted. Hereinafter, each embodiment of the present invention will be described using the drawings.

First Embodiment
<Principle of shooting an infinite object>
FIG. 1 is a view showing an example of the arrangement of an imaging apparatus 101 according to the first embodiment. The image pickup apparatus 101 acquires an image of an external object without using a lens for forming an image, which is provided in a normal camera, and includes a modulator 102, an image sensor 103, and an image processing unit 106. It is done.

  FIG. 2 is a diagram showing a configuration example of the modulator 102. As shown in FIG. The modulator 102 is closely fixed to the light receiving surface of the image sensor 103 (FIG. 1), and the grating substrate 102 a and the first grating pattern 104 and the second grating formed on the upper and lower surfaces of the grating substrate 102 a. And a pattern 105. The lattice substrate 102a is made of, for example, a transparent material such as glass or plastic. Hereinafter, the image sensor 103 side of the lattice substrate 102a is referred to as the back surface, and the opposite surface, that is, the imaging target side is referred to as the front surface.

  These grating patterns 104 and 105 are formed of concentric grating patterns in which the spacing of the grating patterns, that is, the pitch becomes narrower in inverse proportion to the radius from the center toward the outside. The grating patterns 104 and 105 are formed, for example, by vapor-depositing a metal such as aluminum or chromium by a sputtering method used in a semiconductor process. The metal is tinted by the deposited and undeposited patterns.

  The formation of the grid patterns 104 and 105 is not limited to vapor deposition, and may be formed by shading, for example, by printing with an inkjet printer or the like. Furthermore, although visible light is described here as an example, for example, in the case of imaging with far infrared radiation, the lattice substrate 102a is made of a material transparent to far infrared radiation, such as germanium, silicon, chalcogenide, or the like. That is, a material that is transparent to the wavelength to be imaged can be used for the grating substrate 102a, and a material such as metal that blocks the wavelength to be imaged can be used for the grating patterns 104 and 105. .

  Although the method of forming the grating patterns 104 and 105 on the grating substrate 102 a to realize the modulator 102 has been described here, the present invention is not limited thereto. FIG. 3 is a diagram showing another configuration example of the modulator 102. As shown in FIG. The modulator 102 can also be realized by a configuration in which the grating patterns 104 and 105 are formed in a thin film and held by a support member 102b provided instead of the grating substrate 102a.

  The image sensor 103 is formed of, for example, a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. On the surface of the image sensor 103, pixels 103a which are light receiving elements are regularly arranged in a lattice. The light transmitted through the grating patterns 104 and 105 has its intensity modulated by the grating pattern and is received by the image sensor 103. The image sensor 103 converts the light image received by the pixel 103 a into an image signal which is an electrical signal. An image signal (also referred to as “image data”) output from the image sensor 103 is subjected to image processing by the image processing unit 106 and output to the image display unit 107 or the like. The image processing unit 106 may store the processed image data in a storage device (not shown) included in the imaging apparatus 101 or may output the processed image data to an external host computer or a recording medium.

  Note that the imaging device 101 can include, for example, a processor, a memory, a communication device, a processing circuit, and the like. The imaging apparatus 101 may also include an input / output interface connected to an external device such as Universal Serial Bus (USB) or High-Definition Multimedia Interface (HDMI). The image processing unit 106 may be realized by, for example, a dedicated image processing circuit, or may be realized by a processor that executes a program.

  FIG. 4 is a view showing how an external object is photographed using the image pickup apparatus 101 of FIG. FIG. 4 illustrates an example in which the subject 401 is photographed by the imaging device 101 and displayed on the image display unit 107. When the subject 401 is photographed, the surface of the modulator 102, specifically, the surface of the lattice substrate 102a on which the first lattice pattern 104 is formed, is photographed with the object 401 facing the subject 401. It will be.

  Subsequently, an outline of image processing by the image processing unit 106 will be described. FIG. 5 is a flowchart illustrating an example of the image processing of the image processing unit 106.

  First, the image processing unit 106 performs a two-dimensional Fourier transform operation such as Fast Fourier Transform (FFT) on the moiré fringe image output from the image sensor 103 for each color RGB (Red Green Blue) component. The frequency spectrum is determined by the development processing according to (501). Subsequently, the image processing unit 106 extracts data of a necessary frequency region in the frequency spectrum obtained by the process of step 501 (502), and calculates an intensity of the frequency spectrum (503) to acquire an image. Then, the image processing unit 106 performs noise removal processing on the obtained image (504), and then performs contrast enhancement processing (505) and the like. After that, the image processing unit 106 adjusts the color balance of the image (506) and outputs it as a photographed image. Thus, the image processing by the image processing unit 106 is completed.

  Subsequently, the imaging principle in the imaging device 101 will be described.

  First, concentric lattice patterns 104 and 105 in which the pitch is reduced in inverse proportion to the radius from the center are defined as follows. In a laser interferometer or the like, it is assumed that a spherical wave close to a plane wave interferes with a plane wave used as a reference beam. Assuming that the radius from the reference coordinate which is the center of the concentric circle is r, and the phase of the spherical wave there is ((r), the spherical wave uses a coefficient β which determines the size of the curvature of the wavefront,

と表せる。

It can be expressed.

  Despite the spherical wave, the reason that it is expressed by the square of the radius r is that it can be approximated only by the lowest order of development because it is a spherical wave close to a plane wave. When a plane wave is made to interfere with light having this phase distribution,

のような干渉縞の強度分布が得られる。これは、

An intensity distribution of interference fringes such as this is,

を満たす半径位置で明るい線を持つ同心円の縞となる。縞のピッチをｐとすると、

It becomes concentric stripes with bright lines at radial positions that satisfy Assuming that the pitch of stripes is p,

が得られ、ピッチは、半径に対して反比例して狭くなっていくことがわかる。このような縞を持つプレートは、フレネルゾーンプレートやガボールゾーンプレートと呼ばれる。本実施形態では、式（２）で定義される強度分布に比例した透過率分布をもった格子パターンを、図１に示した格子パターン１０４，１０５として用いる。

It can be seen that the pitch becomes narrower in inverse proportion to the radius. Plates having such stripes are called Fresnel zone plates or Gabor zone plates. In the present embodiment, a grating pattern having a transmittance distribution in proportion to the intensity distribution defined by Equation (2) is used as the grating patterns 104 and 105 shown in FIG.

FIG. 6 is a view for explaining the occurrence of an in-plane deviation of a projection image from the front surface to the back surface of the grating substrate by oblique incident and parallel light. It is assumed that parallel light is incident at an angle θ ₀ on the modulator 102 of thickness t in which the grating patterns 104 and 105 are formed on both sides. Assuming that the angle of refraction in the modulator 102 is θ, geometrically, light multiplied by the transmittance of the grating on the surface deviates by δ = t · tan θ and is incident on the back surface, for example, the centers of two concentric gratings. If they are formed in line, the transmittance of the grating on the back side is shifted by δ and multiplied. At this time,

のような強度分布が得られる。この展開式の第４項が、２つの格子のずれの方向にまっすぐな等間隔の縞模様を、２つの格子の重なり合った領域一面に作ることがわかる。このような縞と縞の重ね合わせによって相対的に低い空間周波数で生じる縞は、モアレ縞と呼ばれる。このようにまっすぐな等間隔の縞は、検出画像の２次元フーリエ変換によって得られる空間周波数分布に鋭いピークを生じる。その周波数の値からδの値、すなわち光線の入射角θを求めることが可能となる。このような全面で一様に等間隔で得られるモアレ縞は、同心円状の格子配置の対称性から、ずれの方向によらず同じピッチで生じることは明らかである。このような縞が得られるのは、格子パターンをフレネルゾーンプレートまたはガボールゾーンプレートで形成したことによるものであるが、全面で一様に等間隔なモアレ縞が得られるのであればどのような格子パターンを使用してもよい。

An intensity distribution like is obtained. It can be seen that the fourth term of this expansion formula creates equally spaced stripes straight in the direction of the two grid offsets, in the overlapping area of the two grids. The fringes produced at relatively low spatial frequencies due to such superposition of fringes and fringes are called moire fringes. Such straight equidistant stripes produce sharp peaks in the spatial frequency distribution obtained by the two-dimensional Fourier transform of the detected image. It is possible to determine the value of δ, that is, the incident angle θ of a ray from the value of the frequency. It is apparent from the symmetry of the concentric lattice arrangement that moiré stripes obtained uniformly at equal intervals over such an entire surface occur at the same pitch regardless of the direction of deviation. Such fringes can be obtained because the grating pattern is formed by the Fresnel zone plate or the Gabor zone plate, but any grating can be obtained as long as uniformly spaced moiré stripes can be obtained over the entire surface. Patterns may be used.

  Here, only the component having a sharp peak is

のように取り出すと、そのフーリエスペクトルは、

And its Fourier spectrum is

のようになる。ここで、Ｆはフーリエ変換の演算を表し、ｕおよびｖは、ｘ方向およびｙ方向の空間周波数座標、括弧を伴うδはデルタ関数である。この結果から、検出画像の空間周波数スペクトルにおいて、モアレ縞の空間周波数のピークがｕ＝±δβ／πの位置に生じることがわかる。その様子を図７に示す。

become that way. Here, F represents a Fourier transform operation, u and v are spatial frequency coordinates in the x and y directions, and δ with parentheses is a delta function. From this result, it can be seen that the spatial frequency peak of the moire fringes occurs at the position of u = ± δβ / π in the spatial frequency spectrum of the detected image. The situation is shown in FIG.

  FIG. 7 is a view for explaining the generation of moire fringes and the frequency spectrum when the axes of the gratings on both sides of the grating substrate are aligned. In FIG. 7, from left to right, the layout of the light beam and the modulator 102, moiré fringes, and a schematic diagram of the spatial frequency spectrum are shown, respectively. When (a) in FIG. 7 is normal incidence, and (b) in FIG. 7 is light rays incident at an angle θ from the left side, (c) in FIG. Each is shown.

  The first grating pattern 104 formed on the front surface side of the modulator 102 and the second grating pattern 105 formed on the back surface side have the same axis. In FIG. 7A, since the shadows of the first grating pattern 104 and the second grating pattern 105 coincide with each other, no moiré fringes occur.

  In (b) of FIG. 7 and (c) of FIG. 7, the same moire occurs because the first grating pattern 104 and the second grating pattern 105 have the same displacement, and the peak position of the spatial frequency spectrum is also coincident. From the spatial frequency spectrum, it can not be determined whether the incident angle of the light beam is the case of (b) of FIG. 7 or the case of (c) of FIG. An example of how to avoid this is shown in FIG.

FIG. 8 is a view showing an example of a lattice substrate in which the axes of the front surface lattice and the back surface lattice are offset. As shown in FIG. 8, the two grating patterns 104 and 105 are shifted relative to the optical axis in advance so that the shadows of the two grating patterns shift and overlap even with respect to the light beam vertically incident on the modulator 102. It is necessary to When the relative shift of the shadows of the two gratings with respect to the on-axis normal incident plane wave is δ ₀ , the shift δ caused by the plane wave at the incident angle θ is

のように表せる。このとき、入射角θの光線のモアレ縞の空間周波数スペクトルのピークは周波数のプラス側では、

It can be expressed as At this time, the peak of the spatial frequency spectrum of the moire fringes of the ray of incident angle θ is on the positive side of the frequency,

の位置となる。画像センサの大きさをＳ、画像センサのｘ方向およびｙ方向の画素数を共にＮとすると、２次元フーリエ変換による離散画像の空間周波数スペクトルは、−Ｎ／（２Ｓ）から＋Ｎ／（２Ｓ）の範囲で得られる。このことから、プラス側の入射角とマイナス側の入射角を均等に受光することを考えれば、垂直入射平面波（θ＝０）によるモアレ縞のスペクトルピーク位置は、原点（ＤＣ：直流成分）位置と、例えば＋側端の周波数位置との中央位置、すなわち、

Position. Assuming that the size of the image sensor is S, and the numbers of pixels in the x direction and y direction of the image sensor are both N, the spatial frequency spectrum of the discrete image by two-dimensional Fourier transform is -N / (2S) to + N / (2S) In the range of From this, considering that the incident angle on the positive side and the incident angle on the negative side are received equally, the spectral peak position of the moire fringe due to the vertical incident plane wave (θ = 0) is the origin (DC: DC component) position And the center position of, for example, the frequency position of the + side end, ie,

の空間周波数位置とするのが妥当である。したがって、２つの格子の相対的な中心位置ずれは、

It is appropriate to set the spatial frequency position of Thus, the relative center offset of the two grids is

とするのが妥当である。

It is appropriate to

  FIG. 9 is a diagram for explaining the generation of moire fringes and the frequency spectrum in the case where the gratings on both sides of the grating substrate are arranged in a staggered manner. Similar to FIG. 7, from left to right, the layout of the light beam and the modulator 102, moiré fringes, and a schematic diagram of the spatial frequency spectrum are shown, respectively. FIG. 9 (a) shows the case where the light beam is vertically incident, FIG. 9 (b) shows the case where the light beam is incident from the left at an angle θ, FIG. 9 (c) shows the light beam incident from the right at the angle θ The respective cases are shown.

The first grating pattern 104 and the second grating pattern 105 are arranged in advance by being shifted by δ ₀ . Therefore, moire fringes occur also in (a) of FIG. 9 and a peak appears in the spatial frequency spectrum. The shift amount δ ₀ is set so that the peak position appears at the center of the spectral range on one side from the origin as described above. At this time, in (b) of FIG. 9, the shift δ becomes larger, and in (c) of FIG. 9, the shift δ becomes smaller, so unlike (b) of FIG. The difference from (c) in FIG. 9 can be determined from the peak position of the spectrum. The spectral image of this peak is a bright spot that indicates a light flux at infinity, that is, an image captured by the imaging apparatus 101 of FIG. 1.

Assuming that the maximum angle of incidence of collimated light that can be received is θ _max

より、撮像装置１０１にて受光できる最大画角は、

Therefore, the maximum angle of view that can be received by the imaging device 101 is

で与えられる。

Given by

_Assuming that parallel light with an angle of view θ _max is focused at the end of the image sensor and received from an image sensor using an ordinary lens, the effective focal length of the imaging device 101 without the lens is ,

に相当すると考えることができる。

It can be considered to be equivalent to

  Here, it can be understood from the equation (13) and the equation (14) that the angle of view can be changed by the thickness t of the modulator 102 and the size S of the image sensor 103. Therefore, for example, if the modulator 102 has the configuration of FIG. 3 and has a function (for example, mechanical structure) capable of changing the length of the support member 102b, it is also possible to change the angle of view at the time of photographing Become.

  Although the fast Fourier transform has been described as an example of the method of calculating the spatial frequency spectrum from the moiré fringes, the present invention is not limited to this, and may be realized using discrete cosine transform (DCT) or the like. It is also possible to further reduce the amount of computation.

  Also, the transmittance distribution of the grating patterns 104 and 105 has been described on the assumption that there is a sinusoidal characteristic as shown in the equation (2), but such a component is used as the fundamental frequency component of the grating pattern. It should be included. For example, it is possible to binarize the transmittance of the grating pattern as shown in FIG. 10 (a diagram showing an example of a grating pattern), and as shown in FIG. 11 (a diagram showing another example of a grating pattern) It is also possible to increase the transmittance by increasing the width of the high transmittance region by changing the duty of the high ratio grating region and the low ratio region. As a result, an effect such as suppression of diffraction from the grating pattern can be obtained, and deterioration of a photographed image can be reduced.

  The grating patterns 104 and 105 may be realized by phase modulation instead of transmittance modulation. For example, as shown in FIG. 12 (a diagram showing still another example of the grating pattern), forming the grating substrate 102a with a cylindrical lens 1201 produces an intensity modulation pattern as shown in the figure on the image sensor 103. Imaging, as in the previous discussions. As a result, it is possible to reduce the light quantity loss due to the shielding portion of the grating pattern 104, to improve the light utilization efficiency, and to obtain the effect of suppressing the diffraction from the grating pattern. Although realized with a lens in FIG. 12, it is also possible to realize with a phase modulation element having the same effect.

  In the above description, all the incident light rays are simultaneously incident at one incident angle. However, in order for the imaging device 101 to actually function as a camera, light beams of a plurality of incident angles are simultaneously incident. Must be assumed. The light of such a plurality of incident angles already overlaps the images of the plurality of front side gratings at the time of entering the grating pattern on the back side. If these images mutually generate moire fringes, there is a concern that the moire fringes become noise that interferes with the detection of moire fringes with the second grating pattern 105 that is a signal component. However, in reality, the overlapping of the images of the first grid pattern 104 does not cause a peak of the moiré image, and the peak is generated only by the overlapping with the second grid pattern 105 on the back surface side. The reason is described below.

  First of all, the overlapping of the shadows of the first grating pattern 104 on the surface side with the rays of incident angles is the difference, not the product but the sum. Regarding the overlap between the shadow of the first grating pattern 104 and the second grating pattern 105 due to the light of one incident angle, the second grating pattern 105 has a light intensity distribution that is the shadow of the first grating pattern 104. The light intensity distribution after passing through the second grating pattern 105 on the back surface side can be obtained by multiplying the transmittance of

  On the other hand, overlapping of shadows due to light of different angles incident on the first grating pattern 104 on the front side is not a product but a sum because the light is overlapped. In the case of sum,

のように、もとのフレネルゾーンプレートの格子の分布に、モアレ縞の分布を乗算した分布となる。したがって、その周波数スペクトルは、それぞれの周波数スペクトルの重なり積分で表される。そのため、たとえモアレのスペクトルが単独で鋭いピークをもったとしても、実際上、その位置にフレネルゾーンプレートの周波数スペクトルのゴーストが生じるだけである。つまり、スペクトルに鋭いピークは生じない。したがって、複数の入射角の光を入れても検出されるモアレ像のスペクトルは、常に表面側の第１の格子パターン１０４と裏面側の第２の格子パターン１０５との積のモアレだけであり、第２の格子パターン１０５が単一である以上、検出されるスペクトルのピークは１つの入射角に対して１つだけとなるのである。

Thus, the distribution of the grating of the original Fresnel zone plate is multiplied by the distribution of the moire fringes. Therefore, the frequency spectrum is represented by the overlap integral of each frequency spectrum. Therefore, even if the moire spectrum alone has a sharp peak, in practice, only ghosting of the frequency spectrum of the Fresnel zone plate occurs at that position. That is, no sharp peak occurs in the spectrum. Therefore, the spectrum of the moiré image detected even when light at a plurality of incident angles is always the moiré of the product of the first grating pattern 104 on the front side and the second grating pattern 105 on the back side, As long as the second grating pattern 105 is single, the peak of the detected spectrum is only one for one incident angle.

  Here, the correspondence between the parallel light that has been described so far and the light from the actual object will be schematically described using FIG.

  FIG. 13 is a diagram for explaining the angles that the light from each point constituting the object forms with the image sensor. The light from each point constituting the subject 401 is strictly a spherical wave from a point light source, and the modulator 102 and the image sensor 103 (hereinafter referred to as a lattice sensor integrated substrate 1301 in FIG. 13) of the imaging device 101 of FIG. Incident to At this time, when the lattice sensor integrated substrate 1301 is sufficiently small or far from the subject 401, the incident angles of light illuminating the lattice sensor integrated substrate 1301 from each point can be regarded as the same.

  The condition that Δθ can be regarded as parallel light from the relationship that the spatial frequency displacement Δu of the moire with respect to the minute angular displacement Δθ obtained from the equation (9) is 1 / S or less which is the minimum resolution of the spatial frequency of the image sensor is

のように表せる。この条件下であれば、無限遠の物体を本実施形態の撮像装置が撮像可能である。

It can be expressed as Under this condition, the imaging device according to the present embodiment can capture an object at infinity.

<Principle of shooting a finite distance object>
Here, imaging for an object at infinity described above will be described again. FIG. 14 is a diagram for explaining that the front side grid pattern is projected when the object is at an infinite distance. FIG. 14 shows the projection onto the back surface of the first grid pattern 104 on the front surface side. A spherical wave from point 1401 constituting an object at infinity becomes a plane wave while propagating a sufficiently long distance and illuminates the first grating pattern 104 on the surface side, and its projection image 1402 is projected to the lower surface Ru. In this case, the projected image has substantially the same shape as the first grating pattern 104. As a result, by multiplying the projected image 1402 by the transmittance distribution of the grating pattern on the back side (corresponding to the second grating pattern 105 in FIG. 2), it is generated when FIG. 15 (the object is at infinite distance) It is possible to obtain equidistant linear moiré fringes as shown in FIG.

  On the other hand, imaging of an object at a finite distance will be described. FIG. 16 is a diagram for explaining that the front side grid pattern is enlarged when the object is at a finite distance. FIG. 16 shows the projection onto the back surface of the first grid pattern 104 on the front surface side. The spherical wave from the point 1601 constituting the object illuminates the first grating pattern 104 on the surface side, and the projection image 1602 is projected on the lower surface. In this case, the projected image is magnified almost uniformly. Note that this enlargement ratio α is calculated using the distance d from the first grid pattern 104 to the point 1601.

のように算出できる。

It can be calculated as

Therefore, Fig. 17 (showing an example of moire fringes generated when an object is at a finite distance) is obtained by multiplying the transmittance distribution of the lattice pattern on the back side designed for parallel light as it is. Thus, equally spaced linear moiré fringes do not occur. However, if the second grating pattern 105 is enlarged in accordance with the shadow of the uniformly enlarged first grating pattern 104 on the surface side, the enlarged projected image 1602 is again shown in FIG. As shown in the drawing showing an example of moire fringes in which the back side lattice pattern is corrected, when L is at a finite distance, equally spaced linear moire fringes can be generated. For this purpose, correction is possible by setting the coefficient β of the second grid pattern 105 to β / α ² .

  Thereby, it is possible to selectively develop the light from the point 1601 at a distance which is not necessarily at infinity. By this, it is possible to focus at an arbitrary position and to shoot.

<Simplified structure>
Next, a method of simplifying the configuration of the above-described modulator 102 will be described. In the modulator 102, the first grating pattern 104 and the second grating pattern 105 having the same shape are respectively formed on the front and back surfaces of the grating substrate 102a so as to be offset from each other. The image is developed by detecting the angle of the parallel light incident upon image processing from the spatial frequency spectrum of the moire fringes. The second grating pattern 105 on the back surface side is an optical element that modulates the intensity of the light that comes in close contact with the image sensor 103 and is the same grating pattern regardless of the incident light. Therefore, the configuration of the imaging apparatus 101 may be changed as follows. Description will be made focusing on differences from FIG.

  FIG. 19 is a diagram showing a configuration example of an imaging device 101 that realizes the back side grid pattern by image processing. The imaging apparatus 101 includes a modulator 1901 in which the second grating pattern 105 is omitted, and an image processing unit 1902 including an intensity modulating unit 1903 that performs processing corresponding to the second grating pattern 105.

  FIG. 20 is a diagram showing a configuration example of the modulator 1901. The modulator 1901 is composed of a grating substrate 102 a and a first grating pattern 104 formed on the surface of the grating substrate 102 a. Compared to FIG. 2, the lattice pattern formed on the lattice substrate 102a can be reduced by one. Thereby, the manufacturing cost of the modulator can be reduced, and the light utilization efficiency can be further improved.

  FIG. 21 is a flowchart showing an example of the image processing of the image processing unit 1902. The difference from the flowchart of FIG. 5 is the process of step 2101 before step 501. The processes in steps 501 to 506 are the same as the corresponding processes in FIG.

  In the process of step 2101, the above-described intensity modulation unit 1903 generates a moiré fringe image corresponding to the light transmitted through the lattice pattern 105 on the back side with respect to the image output from the image sensor 103. Specifically, the intensity modulation unit 1903 generates the grid pattern 105 on the back side and multiplies the image of the image sensor 103 because the calculation corresponding to the equation (5) may be performed. Furthermore, if the grid pattern 105 on the back side is a binarized pattern as shown in FIG. 10 or FIG. 11, the intensity modulation unit 1903 merely sets the value of the image sensor 103 in the area corresponding to black to 0. Good. Thereby, it is possible to suppress the calculation load of multiplication operation and the size of the multiplication circuit.

  In this case, the pitch of the pixels 103a (FIG. 20) of the image sensor 103 is fine enough to reproduce the pitch of the first grid pattern 104, or the pitch of the first grid pattern 104 is the pixel 103a. It is necessary to be coarse enough to reproduce on the pitch of When the grid pattern is formed on both sides of the grid substrate 102a, the pitch of the grid pattern does not necessarily have to be resolved by the pixels 103a of the image sensor 103, and only the moiré image needs to be resolved. However, when the grid pattern is reproduced by image processing, the grid pattern and the resolution of the image sensor 103 need to be equal.

  Also, the processing corresponding to the second grating pattern 105 is realized by the intensity modulation unit 1903 as described above. However, the present invention is not limited to this, and the second grating pattern 105 is an optical element which is in close contact with the sensor and modulates the intensity of the incident light, so that the sensitivity of the sensor is effectively the transmittance of the second grating pattern 105 It can also be realized by setting in consideration of

<Noise cancellation>
In the explanation so far, the story was advanced focusing on equation (6) in which only the component having a sharp peak was extracted from equation (5), but in actuality, terms other than the fourth term of equation (5) are noise It becomes. Therefore, noise cancellation based on fringe scanning is effective.

First, in the interference fringe intensity distribution of equation (2), assuming that the initial phase of the first grating pattern 104 is _{B F} and the initial phase of the second grating pattern 105 is _{B B} , equation (5) is

のように表せる。ここで、三角関数の直交性を利用し、

It can be expressed as Here, utilizing the orthogonality of the trigonometric function,

式（１９）のように式（１８）をΦ_Ｆ、Φ_Ｂに関して積分すると、ノイズ項がキャンセルされ単一周波数の定数倍の項が残ることになる。前述の議論から、これをフーリエ変換すれば、空間周波数分布にノイズのない鋭いピークを生じることになる。

When equation (18) is integrated with respect to _{F F} and _{B B} as in equation (19), the noise term is canceled and a term which is a constant multiple of a single frequency remains. From the above discussion, if this is subjected to Fourier transform, a spatial frequency distribution will have a sharp peak without noise.

Here, equation (19) is shown in the form of an integral, but in fact, the same effect can be obtained by calculating the sum of combinations of _{F F} and _{B B.} Φ _F and _{B B} may be set so as to equally divide the angle between 0 and 2π, and quartered as {0, π / 2, π, 3π / 2}, {0, π / 3 , 2π / 3} may be equally divided into three.

Furthermore, equation (19) can be simplified. In equation (19), _{F F} and _{B B} are calculated so as to be changed independently, but Φ _F = Φ _B, that is, even if the same phase is applied to the initial phases of lattice patterns 104 and 105, the noise term can be canceled . If _{F F} = Φ _B = Φ in equation (19), then

となり、ノイズ項がキャンセルされ単一周波数の定数倍の項が残ることになる。また、Φは０〜２πの間の角度を等分するように設定すればよく、｛０、π／２、π、３π／２｝のように４等分すればよい。

And the noise term is canceled, leaving a term of a constant multiple of the single frequency. Further, Φ may be set to equally divide the angle between 0 and 2π, and may be equally divided into four as {0, π / 2, π, 3π / 2}.

  Also, even if the phase is {0, π / 2} without using equal division, the noise term can be canceled and further simplified. First, when the second grid pattern 105 is implemented by the image processing unit 1902 as shown in the configuration of FIG. 19, since the grid pattern 105 can handle negative values, equation (18) is

となる（Φ_Ｆ＝Φ_Ｂ＝Φ）。格子パターン１０５は既知であるため、この式（２１）から格子パターン１０５を減算し、Φ＝｛０、π／２｝の場合について加算すれば、

(Φ _F = Φ _B = Φ). Since the lattice pattern 105 is known, if the lattice pattern 105 is subtracted from the equation (21) and に つ い て = {0, π / 2}, the addition is:

のようにノイズ項がキャンセルされ単一周波数の定数倍の項が残ることになる。

As a result, the noise term is canceled and a term of a constant multiple of a single frequency remains.

Moreover, the first grating pattern 104 and the second grating pattern 105 as described above, had separated two developed image occurring spatial frequency space by shifting pre [delta] _0. However, this method has a problem that the number of pixels of the developed image is halved. Therefore, a method for avoiding duplication of developed image without offset [delta] _0. Instead of cos in the fringe scan of equation (19),

のようにｅｘｐを用い複素平面上で演算する。これによりノイズ項がキャンセルされ単一周波数の定数倍の項が残ることになる。式（２３）中のｅｘｐ（２ｉβδｘ）をフーリエ変換すれば、

Operate on the complex plane using exp as shown below. This cancels the noise term and leaves a term of a constant multiple of a single frequency. If exp (2iβδx) in equation (23) is Fourier transformed,

となり、式（７）のように２つのピークを生じず、単一の現像画像を得られることが判る。このように、格子パターン１０４，１０５をずらす必要もなくなり、画素数を有効に使用可能となる。

It can be seen that a single developed image can be obtained without producing two peaks as in equation (7). As described above, it is not necessary to shift the grid patterns 104 and 105, and the number of pixels can be effectively used.

  The configuration for performing the noise cancellation method based on the above fringe scan will be described with reference to FIGS. In the fringe scan, it is necessary to use a plurality of patterns having different initial phases as at least the grating pattern 104. In order to realize this, there are a method of switching patterns in time division (FIGS. 22 to 25) and a method of switching patterns in space division (FIGS. 26 to 27).

  FIG. 22 is a diagram showing a configuration example of an imaging device 101 that realizes time-division fringe scanning. This imaging device 101 has a modulator 2201 instead of the modulator 102 of FIG. The imaging device 101 further includes a modulator control unit 2202 and an image processing unit 2203.

FIG. 23 is a view showing an example of a grid pattern in time division fringe scanning. The modulator 2201 is configured of, for example, a liquid crystal display element or the like which can electrically switch and display a plurality of initial phases shown in FIG. 23 (that is, the grid pattern can be changed). In the patterns of FIGS. 23A to 23D, the initial phase Φ _F or the phase difference Φ is {0, π / 2, π, 3π / 2}, respectively. An example of the electrode arrangement in the liquid crystal display element of the modulator 2201 for realizing this is shown in FIG.

FIG. 24 is a diagram showing a configuration example of a modulator 2201 that realizes time-division fringe scanning. Concentric electrodes are provided in the modulator 2201 so as to divide one cycle of the grating pattern into four, and electrodes are connected every four lines from the inside, and four electrodes are drawn out from the outer peripheral portion as drive terminals. ing. By changing over time the voltage states applied to these four electrodes by the modulator control unit 2202 in two states of “0” and “1”, the initial phase Φ _F or Φ of the lattice pattern is shown in FIG. It becomes possible to switch to {0, π / 2, π, 3π / 2} as in (d). Note that, in FIG. 24, an electrode to which "1" is applied shaded represents shielding light, and an electrode to which "0" indicated white is applied corresponds to transmitting light.

  FIG. 25 is a flowchart illustrating an example of image processing of the image processing unit 2203 that realizes time-division fringe scanning. The difference from the flowchart of FIG. 21 is the processing of steps 2501 to 2504 prior to step 501. The processing in steps 501 to 506 is the same as the corresponding processing in FIG.

  First, the image processing unit 2203 resets the addition result at the beginning of the fringe scan operation (2501). Next, when the equation (20) is satisfied, the image processing unit 2203 sets the same initial phase as that of the grating pattern 104 used for imaging (2502), and generates the grating pattern 105 having the initial phase. The output image of the image sensor 103 is multiplied (2101). The image processing unit 2203 adds the multiplication result for each initial phase pattern (2503). The image processing unit 2203 repeats the processing of the above steps 2502 to 2503 for all the initial phase patterns (NO in 2504). If the processing has been completed for all phases (YES in 2505), the image processing unit 2203 advances the processing to step 501. Although the above flowchart has been described by taking equation (20) as an example, it is possible to apply the same to equations (19), (22) and (23).

  FIG. 26 is a diagram showing a configuration example of an imaging device 101 that realizes space division fringe scanning. The imaging apparatus 101 has a modulator 2601 instead of the modulator 102 of FIG. The imaging apparatus 101 further includes an image dividing unit 2602 and an image processing unit 2203.

  FIG. 27 is a diagram showing an example of a grid pattern in space division fringe scanning. The modulator 2601 has a plurality of patterns of initial phases two-dimensionally arranged, for example, a pattern of the initial phases FF or Φ of FIG. 27 is {0, π / 2, π, 3π / 2}, respectively. The image dividing unit 2602 divides the output image of the image sensor 103 into area images corresponding to the pattern arrangement of the modulator 2601 and sequentially transmits the image to the image processing unit 2203. In the example of FIG. 26, the image sensor output is divided into 2 × 2 areas. Since fringe scanning based on equation (20) requires four phases, modulator 2601 has a 2 × 2 pattern arrangement. Since the fringe scan based on the equation (22) can be realized with two phases, the modulator 2601 can be realized with a 1 × 2 pattern arrangement, and the image sensor output is also divided into 1 × 2 regions accordingly. The subsequent processing of the image processing unit 2203 is the same as the processing of FIG. 22 which is time-division fringe scanning, and thus the description thereof is omitted.

  By using this space division fringe scan, it is not necessary to electrically switch the grid pattern as in the case of the time division fringe scan modulator 2201, and the modulator can be manufactured inexpensively. However, the use of space division fringe scanning effectively reduces the resolution because the image is divided. Therefore, time division fringe scanning is suitable when the resolution needs to be increased.

<Problems in close-up photography>
Next, problems in close-up shooting will be described. First, the angle of view is redefined. In the above description, the peak position u (Equation (9)) of the spatial frequency spectrum of the moire fringe of the light beam at the incident angle θ is larger than the upper limit frequency N / (2S) determined by the pixel pitch of the image sensor 103 In the case, the angle of view θ _max was defined (equation (13)). On the other hand, when β is small or the pixel pitch (S / N) of the image sensor 103 is small and the peak position u is smaller than the upper limit frequency, the definition of the angle of view is different.

FIG. 28 is a diagram for explaining the angle of view θ _max2 . The maximum angle of view that can be developed is θ _max2 , which is the slope of a straight line passing through the reference coordinates of the concentric lattice pattern that constitutes the first lattice pattern 104 and the end of the image sensor 103. Therefore, the angle of view θ _max2 is calculated using the size S of the image sensor 103 and the thickness t of the modulator in imaging at a distance of d from the first grid pattern.

のように表せる。また、視野の直径Ａは、θ_ｍａｘ２を用いて、

It can be expressed as Also, the diameter A of the field of view is θ _max2

となる。

It becomes.

  Here, at the time of close-up shooting, it is a problem that the field of view is narrowed due to the scattering angle from the object to be imaged or the CRA (Chief Ray Angle) characteristic of the image sensor.

First, the limitation of the field of view due to the scattering angle from the object to be imaged will be described. FIG. 29 is a diagram for explaining the range and the field of view of the grid pattern irradiated by the light from a certain point constituting the object when the object is at a short distance. FIG. 29 shows the spread angle (scattering angle) θ _s of light from a certain point of the object. When the scattered light from the points 2901 to 2903 constituting the object irradiates the first grating pattern 104, the diameter B of the illuminated area in the first grating pattern 104 is calculated using the scattering angle θ _s

となる。ｄは点２９０１乃至点２９０３から第１の格子パターン１０４までの距離である。このように、接写時には第１の格子パターン１０４のうち照射領域しか使用することができない。なお、現実には散乱光強度は散乱角が大きくなるに従い徐々に減衰するものであるが、図２９では簡単化のため、照射領域のみ散乱光が到達しているとしている。

It becomes. d is the distance from the point 2901 to the point 2903 to the first grid pattern 104. Thus, only the illumination area of the first grid pattern 104 can be used during close-up photography. In practice, the scattered light intensity is gradually attenuated as the scattering angle increases. However, in FIG. 29, it is assumed that the scattered light has reached only the irradiation area for the sake of simplification.

As described above, when the incident light passes through the reference coordinates of the concentric lattice pattern constituting the first lattice pattern 104, imaging becomes possible. Therefore, while it is possible to capture points 2901 and 2902 where scattered light passes the reference coordinates, it is not possible to capture points 2903 and the like that do not pass the reference coordinates. As in FIG. 29, when θ _s ≦ θ _{max 2} , the angle of view is determined by the scattering angle θ _s , and the diameter A _s of the visual field is

となる。

It becomes.

Next, the limitation of the visual field by the CRA characteristic of the image sensor will be described. The received light quantity of the image sensor has an angular dependence called CRA characteristic, depending on the arrangement of the wiring on the front surface or the back surface of the light receiving element and the design of the micro lens array disposed on the normal front surface of the light receiving element. Therefore, assuming that incident light can be received up to the incident angle θ _i based on the CRA characteristic, in imaging using an image sensor such that θ _i ≦ θ _{max 2} , the angle of view is determined by θ _i and the field of view _Ai Is

となる。ただし、θ_ｓ≦θ_ｉの場合の視野は式（２８）となる。

It becomes. However, the visual field in the case of θ _s ≦ θ _i is equation (28).

Note that the incident angle theta _i is advisable to determine an angle as a half width at half maximum of the CRA characteristics, not limited thereto, the peripheral light amount attenuation hardly occurs by designing a smaller angle than the half width at half maximum, from the half width at half maximum Designing at a large angle increases the field of view.

<Method of expanding the field of view when close-up>
Then, the method to expand a visual field at the time of close-up photography is demonstrated.

  FIG. 30 is a diagram showing an example of the configuration of an imaging device that achieves wide field of view by a combination of a grid pattern of multiple eyes and a plurality of image sensors. Unlike in FIG. 19, the imaging device 101 includes a modulator 3001 in which a compound eye grid pattern 3003 is formed, an image sensor 3004, and an image processing unit 3006 including an image combining unit 3007. The multi-eye grid pattern 3003 is configured by arranging a plurality of basic patterns (basic grid patterns) 3002 composed of concentric grid patterns in which the pitch of stripes becomes narrower as going from the center to the edge. The image sensor 3004 is configured by arranging a plurality of image sensors 3005.

  FIG. 31 is a view showing an example of a lattice pattern of a compound eye. In this example, a compound eye grid pattern 3003 is realized by arranging the basic patterns 3002 without overlapping each other in 3 × 3. In this case, the image sensors 3005 are arranged in 3 × 3 so as to correspond to the basic patterns 3002.

FIG. 32 is a view for explaining the lattice pattern of the compound eye shown in FIG. 30 and the field of view of a plurality of image sensors. Of the plurality of basic patterns 3002 and the plurality of image sensors 3005, the n-th field of view of the combination of the n-th basic pattern 3002 and the n-th image sensor 3005 is _An , the angle of view is θ _n , and the n-th and n + 1-th It is assumed that the center-to-center distance of the basic patterns 3002 is q _n , and the number of the plurality of basic patterns 3002 and the plurality of image sensors 3005 in the one-dimensional direction is N. At this time, the combined visual field A _N is

となる。つまり、基本パターン３００２と画像センサ３００５の組み合わせ（基本ユニットともいう）を複数用いることで、それぞれの組み合わせで撮像できる視野を合成し拡大することができる。

It becomes. That is, by using a plurality of combinations (also referred to as basic units) of the basic pattern 3002 and the image sensor 3005, it is possible to combine and expand the visual field that can be imaged by each combination.

However, if the distance d from the compound-eye grating pattern 3003 to the imaging object is short, the field of view of the basic pattern 3002 is reduced, a gap in the synthesis field A _N. To obtain a field of view continuous gap does not occur, _{A n} is as long or _{q n,} it may be set to d and _{q n} which satisfies _{A n = 2d · tanθ n ≧} q n.

  Subsequently, image processing for imaging using the above-described image sensor 3004 and compound eye grid pattern 3003 will be described with reference to FIGS.

  FIG. 33 is a flowchart showing an example of the image processing of the image processing unit 3006. What differs from the flowchart of FIG. 21 is the process of step 3301 added between step 504 and step 505. The image processing unit 3006 performs the processing of step 2101 to step 504 for each of the sensor images output from the plurality of image sensors 3005. At step 3301, the image combining unit 3007 of the image processing unit 3006 rearranges the output images of step 504 according to the arrangement of the image sensors 3005, and combines them.

34 and 35 are diagrams for explaining an example of the combining method of step 3301. FIG. _{A N} in FIG. 34 is a synthetic field _{A N} in FIG. 32, _{A n} and _{A n + 1} is the field _{A n} and field _{A n + 1} by n-th and (n + 1) th basic pattern 3002 in FIG. 32. Further, the output image of step 504 in FIG. 33 is represented as an output image 3401 and an output image 3402. FIG. 35 shows a composite image of the output image 3401 and the output image 3402. In step 3301, at least one of the output image 3401 and the output image 3402 is shifted and added by the number of pixels corresponding to the center-to-center distance q _n between the n-th and n + 1-th basic patterns 3002 to obtain one composite image. Be

Next, luminance correction after combination will be described using FIG. An area 3501 indicates an area where the field of view _An and the field of view _{An + 1} of the output image 3401 and the output image 3402 overlap with each other. The luminance of the area 3501 after combination is the sum of the output image 3401 and the output image 3402. Therefore, in order to eliminate the difference in luminance between the area 3501 and the other area where the field of view does not overlap and to obtain a smooth luminance distribution, it is necessary to correct the luminance of the area 3501. When the luminances of the area 3501 of the output image 3401 and the area 3501 of the output image 3402 coincide with each other, correction can be performed by reducing the luminance of the area 3501 after combination to 1/2.

However, since the actual luminance has a distribution that attenuates from the center to the end due to the influence of CRA or the like, the luminances of the area 3501 of the output image 3401 and the area 3501 of the output image 3402 do not match, and the luminance after combining Even if 1⁄2 is set to 1⁄2, the luminance difference can not be corrected. Therefore, in the image of either the output image 3401 or the output image 3402, a region 3501 where the visual field _An and the visual field A _{n + 1} overlap is cut out and synthesized, or pixels corresponding to q _n / 2 from both ends It is sufficient to cut out the images by number and remove the overlapping part. Thereby, the luminance difference can be reduced without performing the luminance correction.

FIG. 36 is a flowchart showing another example of the image processing of the image processing unit 3006. What differs from the flowchart of FIG. 33 is the processing of steps 3601 and 3602 prior to step 501. In FIG. 36, step 3301 is omitted. In step 3601, the image combining unit 3007 of the image processing unit 3006 has a plurality of sensor images output from the plurality of image sensors 3005 by the number of pixels corresponding to the center-to-center distance q _n of the nth and n + 1th basic patterns 3002. Shift and add to combine. In step 3602, the image processing unit 3006 performs intensity modulation on the combined image in step 3601 using a second compound eye grid pattern formed of a plurality of patterns corresponding to each of the plurality of basic patterns 3002.

  Although the configuration relating to the multi-eye grid pattern of FIG. 30 has been described based on FIG. 19, it is also possible to perform noise cancellation by fringe scan by applying this configuration to other configurations such as FIG. 22.

  In addition, although the basic pattern 3002 and the arrangement in the one-dimensional direction of the image sensor 3005 are described focusing on the basic pattern 3002 and the image sensor 3005 in FIG. 32 and the like, this configuration is similarly applicable to a two-dimensional arrangement by expanding in the orthogonal direction. is there.

  Further, the number of basic patterns 3002 is not limited to that illustrated either. Since the field of view is determined by the number of basic patterns 3002, it may be appropriately changed according to the size of the imaging target.

  Further, since the field of view widens when the distance from the basic pattern 3002 to the imaging target is increased, the number of basic patterns 3002 can be reduced in this way.

  FIG. 37 is a view for explaining an arrangement example of each image sensor 3005. FIG. 37 shows an arrangement in which the directions (directions of wiring) of the image sensors 3005 are aligned. In a general image sensor, a wire for transmitting a signal is emitted from the side of the light receiving surface. In order to dispose a plurality of image sensors 3005, it is necessary to space the image sensors 3005 so that interference between the wires and overlapping of the wires on the light receiving surface 3701 do not occur, and the light receiving surfaces can not be brought close to each other. is there.

  FIG. 38 is a view for explaining another arrangement example of the image sensor. FIG. 38 shows an arrangement in which the direction (direction of wiring) of each image sensor 3005 is directed to the outside of the image sensor 3004. Although the position of the wiring is different depending on the type of the image sensor, by arranging the orientation of at least a part of the image sensor 3005 to be different from the orientation of the other image sensor 3005, the restriction of the distance between the light receiving surfaces by the wiring is avoided. be able to.

  According to the above method and configuration, by using a plurality of combinations of the image sensor 3005 and the basic pattern 3002, it is possible to combine and expand the visual fields that can be imaged by each.

  According to the present embodiment, it is possible to provide an imaging device which has a simple configuration, is thin, and enlarges the field of view at the time of close-up shooting.

Second Embodiment
<Optimization of image sensor size>
In the present embodiment, a method for efficiently using pixels and reducing the amount of calculation by optimizing the size of the image sensor 3005 will be described.

  FIG. 39 is a diagram for explaining a method of optimizing the size of the image sensor. The size and number of light receiving surfaces of the image sensor 3005 necessary for imaging using the compound eye grating pattern 3003 will be described with reference to FIG. In the following description, the image sensor 3005 corresponds to the image sensor 3901 or the image sensor 3903.

Region C, when the angle theta _n is determined by theta _s or _{θ i (θ n ≦ θ max2} ), it indicates a region on the light receiving surface of the image sensor 3901 required to obtain a visual field _{A n.} A light beam with an incident angle of θ _n enters region C on image sensor 3901. In order to obtain the field of view _An , the size S of the light receiving surface of the image sensor 3901 may be C or more, and when S and C are equal, S can be minimized and the light receiving surface can be efficiently used. Further, by minimizing the light receiving surface of the image sensor 3901, it is possible to reduce the pixels other than the area C which is unnecessary for development, and therefore the amount of calculation can be reduced.

Here, C is the angle of view θ _n and the distance t from the compound eye grid pattern 3003 to the image sensor 3901.

と求められる。よって、ｔを小さくし、Ｃを小さくすることで、Ｓを小さくすることができる。

Is required. Therefore, S can be reduced by reducing t and C.

Further, to reduce the scattering angle theta _s of the imaging object, or also by using the maximum incident angle theta _i small image sensor 3901 that can be received by the image sensor 3901 which is determined by the CRA characteristics, can be reduced S. However, in this case, the field of view _An also becomes smaller. Therefore, as described in the first embodiment, it is necessary to widen the field of view by, for example, increasing the number of combinations of the image sensor 3005 and the basic pattern 3002.

  According to the above method and configuration, it is possible to optimize the size and number of image sensors 3005 and reduce the amount of calculation.

In FIG. 39, the image sensor 3901 is disposed to correspond to the basic pattern 3902 one by one, but even if one image sensor 3903 for receiving light rays of the visual field _{An + 1} and visual field _{An + 2} is arranged, for example Good. In this case, the number of image sensors 3005 can be reduced. Further, components having different sizes such as the image sensor 3901 and the image sensor 3903 may be combined and disposed.

Third Embodiment
<Design of gaze direction>
In the present embodiment, the gaze direction in imaging using the compound eye grid pattern 3003 will be described.

FIG. 40 is a view for explaining an example of the gaze direction of imaging by the compound eye lattice pattern 3003. A _n represents the field of view can be obtained by the n-th basic pattern 4001 when angle by the equation (25) is determined. Further, here, the reference coordinates of the concentric lattice pattern constituting the basic pattern 4001 and the center coordinates of the light receiving surface of the image sensor 3005 do not match. The central coordinates of the light receiving surface of the image sensor 3005 are intersection points of diagonal lines connecting the four corners of the light receiving surface. As described in the first embodiment, the angle of view determined by the equation (25) is the slope of a straight line passing through the end of the light receiving surface of the image sensor 3004 and the reference coordinates of the concentric lattice pattern constituting the basic pattern 4001. Become. Thus, gaze direction is the vector connecting the two points and the center of the reference coordinates and the field A _n of concentric grating pattern constituting the basic pattern 4001. That is, the gaze direction can be changed by setting at the time of design of the reference coordinates of the concentric lattice pattern forming the basic pattern 4001 and the position of the central coordinates of the light receiving surface of the image sensor 3004.

FIG. 41 is a diagram for explaining another example of the gaze direction, and shows a configuration for changing the gaze direction for each basic pattern. The difference between the configuration of FIG. 40 and the configuration of FIG. 41 is the position of the reference coordinates of the concentric pattern forming the basic pattern 4101 and the basic pattern 4102. In FIG. 41, these reference coordinates are shifted in the direction in which the center-to-center distance l between the concentric lattice patterns of the basic pattern 4101 and the basic pattern 4102 is closer. By thus changing the center coordinates of the concentric lattice pattern for each basic pattern, it is possible to change the gaze direction of imaging by each basic pattern. Depending on the type of the image sensor 3005, the center coordinates of the light receiving surface of the adjacent image sensor 3004 may not be able to be brought close due to the arrangement of the wiring extending from the light receiving surface. Even in such a case, by changing the position of the reference coordinates of the basic pattern 3002, the field of view A _N can be adjusted to the imaging target, and a continuous field of view A _N can be obtained.

  According to the above-described method and configuration, the gaze direction can be changed by setting the reference coordinates of the concentric lattice pattern constituting the basic pattern 3002, and the arrangement method of the image sensor 3005 is restricted. It is possible to direct the field of view to the imaging object.

Fourth Embodiment
In the present embodiment, imaging using an elliptical lattice pattern as the basic pattern 3002 will be described.

  The light receiving surface of the image sensor 3005 may be rectangular depending on the type. FIG. 42 is a view showing an example of a concentric lattice pattern projected on a rectangular light receiving surface. In FIG. 42, a concentric lattice pattern in which the pitch of stripes becomes narrower as going from the center to the end is used as the basic pattern 3002. In this case, the number of circles to be projected is different in the long side direction and the short side direction of the light receiving surface, and a lattice pattern with a finer pitch is projected in the long side direction.

  As described in the first embodiment, the fineness of the pitch of the concentric lattice pattern is determined by the coefficient β, and the resolution after development is determined by β as in equation (12). That is, the resolution after development depends on the fineness of the pitch of the pattern projected on the light receiving surface. Therefore, as in the projection pattern of FIG. 42, the fineness of the pitches in the long side direction and the short side direction do not match (a pattern of finer pitch outside the long side direction does not exist in the short side direction). The resolution after development will not match either. For the above reasons, when a lattice pattern concentric with a rectangular light receiving surface is used, there arises a problem that the resolution in the short side direction is lower than that in the long side direction.

  FIG. 43 is a diagram showing an example of an elliptical grid pattern projected on a rectangular light receiving surface. In FIG. 43, an elliptical lattice pattern in which the pitch of stripes becomes narrower as it goes from the center to the end is used as a basic pattern 3002. The major axis direction and the minor axis direction of the ellipse are set to coincide with the long side direction and the short side direction of the rectangle, respectively. Further, the lengths of the major axis and the minor axis of the ellipse may be set according to the aspect ratio of the long side and the short side of the rectangle, or may be set independently of the aspect ratio. In addition, the stripe pitch in the minor axis direction of the ellipse is set to be smaller than the pitch of the corresponding stripe in the major axis direction (that is, the minimum value of the stripe pitch in the minor axis direction of the ellipse is longer Not equal to the minimum value of the corresponding fringe pitch in the axial direction). In this case, as compared with FIG. 42, the number of ellipses projected in the short side direction is increased, and the number of ellipses obtained in the long side direction and the short side direction is uniform. That is, by using the elliptical grid pattern, even when the image sensor 3005 having a rectangular light receiving surface is used, a reduction in resolution in the short side direction can be prevented.

  Furthermore, the diffraction of incident light in a concentric grating pattern in which the stripe pitch narrows from the center toward the edge causes a reduction in the contrast of the projection pattern, causing a reduction in resolution and noise. The influence of diffraction depends on the pitch of the fringes of the lattice pattern, so when using the lattice pattern of FIG. 43, the influence of diffraction differs between the long side direction and the short side direction. On the other hand, the pitch of stripes in the minor axis direction matches the pitch of corresponding stripes in the major axis direction (ie, the minimum value of the pitch of stripes in the minor axis direction and the minimum pitch of corresponding stripes in the major axis direction) The long side direction and the short side direction are obtained by aligning the stripe pitch of the grid pattern projected on the rectangular light receiving surface with the long side direction and the short side direction of the light receiving surface using an elliptical grid pattern in which the values match. The influence of diffraction can be made constant in the side direction.

  FIG. 44 is a view showing another example of a concentric lattice pattern projected on a rectangular light receiving surface. FIG. 4443 shows the case where the pitches of the patterns in the long side direction and the short side direction are made constant by using a concentric lattice pattern. By using a concentric grating pattern, the influence of diffraction in the long side direction and the short side direction can be made constant, but in that case, the area where the pattern is projected on the light receiving surface is limited to a small size. As a result, pixels on the light receiving surface can not be used efficiently. On the other hand, by using an elliptical grid pattern, it is possible to use the pixels on the entire light receiving surface, and it is possible to increase the number of sampling points of stripes having the same pitch as the concentric grid pattern, thereby improving image quality. it can.

  According to the above method and configuration, when the image sensor 3005 having a rectangular light receiving surface is used, it is possible to prevent the resolution reduction in the short side direction.

Fifth Embodiment
In the present embodiment, a configuration for realizing a thin finger vein authentication device capable of close-up photography will be described. Hereinafter, the finger vein imaging device which constitutes a part of the finger vein authentication device will be described.

  In order to realize a finger vein authentication apparatus, it is necessary to secure a field of view for obtaining an image necessary for authentication in the finger vein imaging apparatus. FIG. 45 is a view showing a configuration example of a finger vein imaging apparatus using a lattice pattern of multiple eyes. The finger vein imaging apparatus includes a modulator 4501 in which a multi-eye grid pattern 4502 including a plurality of basic patterns 4503 is formed, a plurality of image sensors 4504, and a plurality of light sources 4505.

FIG. 45 shows a configuration necessary to obtain a continuous visual field A _N = 12 mm × 10 mm when the distance d from the imaging target to the compound eye grid pattern 4502 is 2 mm. The basic patterns 4503 are arranged 2 × 2 at intervals of center-to-center distances qc = 7.5 mm and qr = 5.0 mm, and each image sensor 4504 also has the number and arrangement of the basic patterns 4503. Correspondingly, 2 × 2 pieces are arranged.

Here, assuming that d = 2 mm and the angle of view θ _i obtained from the CRA characteristic of each image sensor 4504 is θ _i = 50 °, the equation is obtained from N = 2, qc = 7.5 mm and qr = 5.0 mm. From (30), it can be seen that A _N = 12 mm × 10 mm, and the intended visual field can be obtained.

  Further, in the configuration of FIG. 45, since the light receiving surface of the image sensor 4504 is rectangular, the developed image can be obtained as described in the fourth embodiment by making the pattern forming the basic pattern 4503 an elliptical grid pattern. It is possible to prevent the resolution from decreasing in the direction of the short side of.

  As the light source 4505, a near infrared light source can be used. In FIG. 45, the light source 4505 irradiates the finger to be imaged from the lower side. Of course, the light source 4505 may be disposed above the finger, and light transmitted through the finger may be obtained by the image sensor 4504.

  Further, in order to perform the time division fringe scan described in the first embodiment, the modulator 4501 may be configured by a liquid crystal display element as shown in FIG. FIG. 46 is a cross-sectional view showing a configuration example of a liquid crystal display element as the modulator 4501. The liquid crystal display element mainly includes a liquid crystal layer 4601, two polarizing plates 4602 disposed above and below the liquid crystal layer 4601, and two glass substrates 4603 disposed so as to sandwich the liquid crystal layer 4601. . As a polarizing plate used for a general liquid crystal display or the like, one corresponding to visible light is used, but as the polarizing plate 4602 used for the modulator 4501, it is necessary to use one corresponding to the wavelength of the light source 4505. The two polarizing plates 4602 have transmission and absorption characteristics according to the polarization direction of the light of the wavelength of the light source 4505.

  FIG. 47 is a view showing a configuration example of a finger vein imaging device using a lattice pattern of single eyes. Although the configuration using the compound eye lattice pattern 4502 is described in FIG. 45, a finger vein imaging apparatus may be used by using the lattice pattern 4702 of a single eye and the image sensor 4703. In this case, the distance d from the subject to the grid pattern 4702 is set large, and the field of view determined by the equation (26), (28) or (29) is enlarged to obtain an image necessary for authentication. Can secure the field of view. The image sensor 4703 may be configured by arranging a plurality of image sensors as shown in FIG.

  According to the above configuration, it is possible to secure a field of view necessary for the finger vein authentication device even in close-up photography and to realize a thin finger vein authentication device.

  The present invention has been described above using a plurality of embodiments. Of course, the present invention is not limited to the above embodiment, but includes various modifications. For example, the above-described embodiment is described in detail to explain the present invention in an easy-to-understand manner, and is not necessarily limited to one having all the described configurations.

  Further, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Moreover, it is possible to add, delete, and replace other configurations for part of the configurations of the respective embodiments.

  Further, each of the configurations, functions, processing units, processing means, etc. described above may be realized by hardware, for example, by designing part or all of them with an integrated circuit. Further, each configuration, function, etc. described above may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files for realizing each function is stored in a memory, a hard disk, a recording device such as a solid state drive (SSD), or a recording medium such as an IC card, an SD card, or a digital versatile disc (DVD). It can be put.

  Further, control lines and information lines indicate what is considered to be necessary for the description, and not all control lines and information lines in the product are necessarily shown. In practice, almost all configurations may be considered to be mutually connected.

  The present invention is not limited to an imaging device and an imaging method, and various aspects such as an imaging system, a developing method, a computer readable program, an image processing circuit, a finger vein authentication device, a finger vein imaging device, and a finger vein authentication method. Can be provided.

101: imaging device, 102: modulator, 102a: lattice substrate, 102b: support member, 103: image sensor, 103a: pixel, 104: first lattice pattern, 105: second lattice pattern, 106: image processing unit 107 image display unit 401 object 1201 cylindrical lens 1301 lattice sensor integrated substrate 1401 point 1402 projected image 1601 point 1602 projected image 1901 modulator 1902 image processing Unit 1903 Intensity modulator 2201 Modulator 2203 Image processor 2202 Modulator controller 2601 Modulator 2602 Image divider 2901 point 2902 point 2903 point 3001 ... Modulator, 3002 ... Basic pattern, 3003 ... Compound eye grid pattern, 3004 ... Image sensor, 3005 ... Image center , 3006 ... image processing unit, 3007 ... image combining unit, 3401 ... output image, 3402 ... output image, 3501 ... region, 3701 ... light receiving surface, 3901 ... image sensor, 3902 ... basic pattern, 3903 ... image sensor, 4001 ... basic Pattern, 4002: Basic pattern, 4101: Basic pattern, 4102: Basic pattern, 4501: Modulator, 4502: Compound eye grid pattern, 4503: Basic pattern, 4504: Image sensor, 4505: Light source, 4601: Liquid crystal layer, 4602: Polarized light Plate, 4603 ... glass substrate, 4702 ... lattice pattern, 4703 ... image sensor

Claims (11)

A modulator that modulates the light intensity using a compound eye grating pattern having a plurality of basic grating patterns;
A plurality of image sensors for converting light transmitted through the modulator into image data and outputting the image data;
An image processing unit that performs image processing on a plurality of image data output from the plurality of image sensors;
An imaging apparatus comprising: The imaging apparatus according to claim 1,
The basic grid pattern and the image sensor are constituted by a plurality of basic units corresponding to each other,
The plurality of basic units each have a different field of view,
An imaging device characterized by The imaging apparatus according to claim 1,
The size of each of the plurality of image sensors is the distance from the compound eye grid pattern to the image sensor, the number of the image sensors, the scattering angle of light from the object to be imaged, or the incident light intensity of the image sensor Determined based on the angle of view determined by the angle dependency
An imaging device characterized by The imaging apparatus according to claim 3,
The angle of view determined by the number of basic grid patterns and the scattering angle of light from the imaging target or the angle dependency of the incident light intensity of the image sensor, and the distance from the imaging target to the compound eye grid pattern The field of view is determined based on
An imaging device characterized by The imaging apparatus according to claim 2,
The gaze direction can be set by the reference coordinates of the concentric pattern forming the basic grid pattern and the center coordinates of the image sensor.
An imaging device characterized by The imaging apparatus according to claim 1,
The distance from the compound eye grid pattern to the imaging target is determined based on the distance between the basic grid patterns and the scattering angle or angle of view of light from the imaging target.
An imaging device characterized by The imaging apparatus according to claim 1,
The basic grid pattern is composed of a plurality of elliptical patterns
An imaging device characterized by The imaging apparatus according to claim 7, wherein
The plurality of elliptical patterns have the same minimum pitch between the plurality of elliptical patterns in the major axis direction and the minor axis direction.
An imaging device characterized by The imaging apparatus according to claim 7, wherein
The lengths of the major and minor axes of the plurality of elliptical patterns are determined according to the aspect ratio of the light receiving surface of the image sensor.
An imaging device characterized by A modulator that modulates the light intensity;
An image sensor that converts light transmitted through the modulator into image data and outputs the image data;
An image processing unit that performs image processing of image data output from the image sensor;
An image pickup apparatus comprising: a light source for emitting near infrared light to an image pickup object. The imaging apparatus according to claim 10, wherein
The modulator is a liquid crystal display element, and includes a first polarizing plate between the object to be imaged and the liquid crystal layer, and a second polarizing plate between the liquid crystal layer and the image sensor.
The first polarizing plate and the second polarizing plate have transmission and absorption characteristics with respect to the wavelength of the light source.
An imaging device characterized by

Images