JP6521470B1 - Information acquisition device - Google Patents

Abstract

An information acquisition apparatus capable of obtaining both a depth map and an all-in-focus image at high speed with high accuracy. The optical system unit 10 is configured to move a focal position, and includes an optical system unit 10 that forms an image of an object, and an imaging unit 21 that photoelectrically converts the formed image. A sensor unit 20 outputting, as an image, a signal obtained by time-integrating a product with a reference signal based on the weight function, and an image processing unit 40 generating an all-in-focus image and a depth map of the object based on the output image; The optical system unit 10 forms an image of an object while moving the focal position, and the sensor unit 20 uses the constant function and the first function group as the predetermined load function to generate the time-integrated signal. The normal image and the first correlation image group are output, and the image processing unit 40 generates an omnifocal image based on the normal image, and the depth based on the generated omnifocal image and the first correlation image pair Generate a map . [Selected figure] Figure 1

Description

  The present invention relates to an information acquisition apparatus capable of obtaining both a depth map and an all-in-focus image at high speed and with high accuracy by using a time correlation image sensor.

  Various methods have been proposed for three-dimensional shape measurement using a camera in which various industrial applications exist.

  For example, there are those by stereo reconstruction using two or more cameras and / or images, and those by which the evaluation function is calculated for a three-dimensional volume (visual volume). For stereo reconstruction, there is, for example, "three-dimensional shape measuring method and apparatus" disclosed in Patent Document 1, and for calculating an evaluation function with respect to a visual volume, for example, "distance information output apparatus and three-dimensional There is a "shape recovery device". These are accompanied by point matching, non-linear optimization, and the like, and there is a problem that the calculation cost is high. There is also the problem of requiring multiple cameras and / or images.

  On the other hand, a method of realizing three-dimensional shape measurement by calculating the distance (depth) to the object passively with one camera, that is, without requiring additional processing such as irradiation of a special pattern light to the object As, the DFD (Depth From Defocus) which estimates depth from the amount of blurring of a picture is known. The depth estimation using the DFD is, for example, disclosed in “the distance information acquisition device, the imaging device, and the program” disclosed in Patent Document 3 and Non-Patent Document 1. However, since the DFD uses a blurred image, there is a problem that the texture or the like on the object is blurred.

  Here, as a method for obtaining an omnifocal image free from image blurring, for example, in the “all-in-focus image generation method” disclosed in Patent Document 4, wavelet transformation is performed on a plurality of images with different focal positions to generate multiresolution images By doing this, a probability distribution relating to the focal position is generated, and the focal position of each pixel position is determined to minimize the evaluation function relating to the probability distribution, thereby obtaining an omnifocal image. However, the method requires a plurality of images with different focal positions, and there is a problem that the calculation cost such as minimization of the evaluation function is high. In addition, as another approach to Patent Document 4 and the like, a depth map is estimated from an image including only one blur, and then the deconvolution operation using PSF corresponding to the depth is performed to calculate an omnifocal image. Although possible, there is a problem that the calculated omnifocal image is unstable because the depth map in DFD is not accurate.

  Furthermore, as related methods, in the methods disclosed in Non-Patent Document 1 and Patent Document 5, it is an object to extend the depth of field (DoF; depth of field; depth range considered to be in focus), For a single blurred image obtained by an operation of imaging while turning the focus ring (the position and / or direction of the imaging surface) of the camera, under predetermined imaging conditions, a PSF (point) corresponding to the imaging operation It is possible to obtain an omnifocal image (that is, an image in which the DoF is expanded) by applying the deconvolution operation using the depth-invariant PSF by using the fact that the spread function is almost depth-invariant. However, the depth map can not be obtained in this method because of the use of the depth invariant property.

  Further, as another method of passive depth measurement, there is a coded aperture (Coded Aperture) for embedding information in the aperture by configuring the aperture as a special shape, which is disclosed, for example, in Non-Patent Document 2. However, although the coding aperture method can obtain a relatively clear all-in-focus image, it can be obtained by another method (due to the restriction that the layer structure of the depth map is simple). There is a problem that the depth map to be processed becomes discontinuous, and in some cases, manual designation operation is required for the layered depth map as intermediate data.

JP-A-6-74723 JP, 2008-015863, A JP, 2010-117593, A JP, 2013-84152, A WO 2009/120718 JP 10-281868 A

S. Kuthirummal, H. Nagahara, C. Zhou and S. K. Nayar, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 99, Mar. 2010. Levin, Anat, et al. "Image and depth from a conventional camera with a coded aperture." ACM transactions on graphics (TOG) 26.3 (2007): 70. Kaneda, K., Ishida, S., Ishida, A., & Nakamae, E. (1992). Image processing and synthesis for extended depth of field of optical microscopes. The Visual Computer, 8 (5-6), 351- 360.

  As described above, the related art has a problem in that it is difficult to achieve the following. That is, it is difficult in the prior art that passive measurement enables both (1) distance information to be reconstructed simultaneously and densely in depth map and (2) omnifocal image simultaneously at high speed and high accuracy. Met. Here, realizing both of these at high speed with high accuracy enables various industrial applications such as three-dimensional modeling, digital heritage, and the like.

  An object of the present invention is to provide an information acquisition apparatus capable of obtaining both a depth map and an all-in-focus image at high speed and with high accuracy, in view of the problems of the prior art.

In order to achieve the above object, the present invention is an information acquisition apparatus, which is configured to move a focal position, and an optical system unit that forms an image of an object, and an imaging unit that photoelectrically converts the formed image. A sensor unit that outputs, as an image, a signal obtained by time-integrating a product of the photoelectrically converted signal and a predetermined weight function, and an all-in-focus image of the object and a depth map based on the output image. An image processing unit, wherein the optical system unit forms an image of the object while moving the focal position, and the sensor unit uses the constant function and the first function group as the predetermined load function to perform the time The normal image and the first correlation image pair are respectively output as the integrated signals, and the image processing unit generates an omnifocal image based on the normal image, and the generated omnifocal image and the first correlation image The And generating a depth map based on.

  According to the present invention, both the depth map and the omnifocal image can be obtained at high speed with high accuracy.

It is a functional block diagram of the information acquisition device concerning one embodiment. It is a flowchart of operation | movement of the information acquisition apparatus which concerns on one Embodiment. It is a figure which shows typically the relationship of the main variable which appears in each type | formula relevant to an integrating | accumulating point spread function (IPSF). It is a circuit diagram of a sensor part (time correlation image sensor) for each individual image sensor as an example. It is a graph which shows typically the position which changes mutually interlockingly, and a load function as an example. It is the graph which confirmed the behavior which function IP ₁ shows numerically. It is a circuit diagram of a sensor part (time correlation image sensor) for each individual image sensor as an example. It is a figure which shows the polygonal line function as an example of a load function.

  FIG. 1 is a functional block diagram of an information acquisition apparatus 1 according to an embodiment. As illustrated, the information acquisition device 1 includes an optical system unit 10, a sensor unit 20, an AD conversion unit 30, and an image processing unit 40. As illustrated here, the optical system unit 10, the sensor unit 20, and the image processing unit 40 further include functional units having the following schematic functions. (Details of each functional unit will be described later.)

The optical system unit 10 includes an optical element mechanism 11 including one or more lenses and the like, an actuator and the like, and an object formed on the imaging surface 21 by driving the optical element mechanism 11 and / or the imaging surface 21. the imaging position of the image IM G of OB adjustably configuration comprising, namely, a drive unit 12 which is adjustable of focus position of the optical element mechanism 11.

The sensor unit 20 can be configured as a time correlation image sensor, which is an existing technology disclosed as “time correlation detection type image sensor” or the like in, for example, Patent Document 6 mentioned above, and in order to realize the time correlation image sensor as a configuration, the plurality of the image pickup device is configured to be arranged on a plane, an imaging plane 21 for outputting a photoelectric conversion signal obtained image IM G of the object OB by photoelectrically converting each element to the integrating circuit 23 A reference signal generation unit 22 for outputting each reference signal to the integration circuit 23, and an integration circuit 23 for obtaining each image (analog signal format) as a signal obtained by integrating the product of the photoelectric conversion signal and the reference signal. Note that the integration corresponds to obtaining a time correlation between the photoelectric conversion signal and the reference signal. Although details will be described later, in the integration circuit 23, a first correlation configured as a pair of a normal image g ₀ , a real part image Re (g ₁ ) and an imaginary part image Im (g ₁ ) by integrating with each reference signal An image pair (to be described later) which is the source of the image pair g ₁ and second correlation image groups s ₁ , s ₂ , and s ₃ are obtained and output to the AD conversion unit 30.

The AD conversion unit 30 outputs, to the image processing unit 40, digital images obtained by AD-converting the respective images (analog signal format) obtained from the integration circuit 23. Here, as shown in the drawing, an image pair that is the basis of the digitized normal image g ₀ and the first correlated image pair g ₁ is output to the conversion unit 45, and the digitized second correlated image group s ₁ , s ₂ , s ₃ are output to the phase determination unit 43.

  The image processing unit 40 includes an omnifocal image generation unit 41, a phase estimation unit 42, a phase determination unit 43, a depth map estimation unit 44, and a conversion unit 45. Note that the image processing unit 40 can be realized by a CPU or the like as a hardware configuration, and each functional unit performs processing on a digital image (and related digital data), so an image etc. It is omitted to mention that it is digital.

Conversion unit 45 uses the image pair as a normal image g _0, and the first correlation image pair g ₁ of the original obtained from the AD conversion unit 30, from each image pair comprising a first correlation image pair g ₁ of the original obtaining a first correlation image pair g ₁ by subtracting the normal image g _0. With conversion unit 45 outputs a first correlation image pair g ₁ obtained the to the phase estimation section 43 outputs the normal image g ₀ to omnifocal image generating unit 41 as it is.

The all-in-focus image generation unit 41 obtains an all-in-focus image r by performing Wiener convolution on the normal image g ₀ obtained from the conversion unit 45 (which is the same as that obtained from the AD conversion unit 30). The focused image r is output to the phase estimation unit 42 as well as being the first output from the image processing unit 40. Phase in the phase estimation section 42 first pixel position of the correlation image pair g ₁ using omnifocal image r obtained from the first correlation image pair g ₁ and omnifocal image generating unit 41 obtained from the conversion unit 45 (that The details are described later), and the estimated phase candidate is output to the phase determination unit 43. The phase determination unit 43 uses the second correlation image group s ₁ , s ₂ , s ₃ obtained from the AD conversion unit 30 to obtain phase candidates at each pixel position obtained from the phase estimation unit 42 (a plurality of phase candidates From among the phase candidates, one phase is determined, and the determined one phase is output to the depth map generation unit 44. The depth map generation unit 44 calculates the depth map (depth map corresponding to the omnifocal image r obtained by the omnifocal image generation unit 41) using the phase determined at each pixel position obtained from the phase determination unit 43, The depth map is used as the second output from the image processing unit 40.

  In the configuration of FIG. 1 described above, the AD conversion unit 30 may be included in the sensor unit 20.

FIG. 2 is a flowchart of the operation of the information acquisition device 1 according to an embodiment. In step S1, captures an image of the object OB while changing the focal position by the driving by the drive unit 12 in the optical system unit 10 (i.e., performs imaging while changing the blur condition of the image IM G of the target OB), and at the same time After each image g ₀ , g ₁ (the original one), s ₁ , s ₂ , s ₃ is acquired while outputting a predetermined reference signal group from the reference signal generation unit 22 in the sensor unit 20, the process proceeds to step S 2 And proceed.

In step S 2, each of the images g ₀ , g ₁ (the original ones), s ₁ , s ₂ , s ₃ acquired in step S ₁ is converted into a digital image by the AD conversion unit 30 and then each unit of the image processing unit 40 The processing in FIG. 2 outputs the omnifocal image and the depth map corresponding to the imaging in step S1, and the flow of FIG. 2 ends.

  The above flow of FIG. 2 can be executed for each time determined at a predetermined frame rate, and an omnifocal image and depth map can be output in real time at the frame rate.

  Hereinafter, the details of the processing contents of each part of the information acquisition apparatus 1 of FIG. 1 operating as shown in FIG. 2 will be described. First, the principle of the present invention will be described. The present invention is equivalent to an extension of the existing method disclosed in Non-Patent Document 1 and Patent Document 5 described above, that is, a method using a depth-invariant PSF at the time of imaging an object while changing the focus position. . (In the present invention, imaging while changing the focus position is performed in step S1 of FIG. 2.) That is, depth information that can not be acquired by the existing method using the depth invariant PSF is It is the present invention that acquisition is enabled by the use of a time correlation image sensor which acquires time correlation with a reference signal.

Specifically, assuming that an integrated PSF used in the existing method as a depth invariant PSF is referred to as IPSF (integrated PSF) and is expressed by the formula IP ₀ , the principle of the present invention is to calculate the IP ₀ This is further extended by introducing and using IP ₁ as IPSF which also takes into consideration the effect of time correlation calculation with the reference signal. Hereinafter, after briefly describing IP ₀ used in the existing method, IP ₁ used in the present invention will be further described.

(Description of IP _{0 in} the existing method)
Assuming that f is a focal length, d is a distance to a target (that is, depth), and v is a distance between a lens when in focus and an imaging device (imaging surface), the following lens equation holds.
1 / f = 1 / d + 1 / v
Here, it is assumed that a position p at time t (a position p corresponding to p = v when in focus) is given as follows by moving the lens in the direction of the image pickup element at a constant speed s. Here, p ₀ is the position at time t = 0.
p = p ₀ + st
Then, the blur size σ (t) at time t is as follows if the distance to the object is sufficiently large compared to the movement amount. Here, D is the size of the opening.
σ (t) = (D / v) | vp (t) |
The following is obtained by setting a = D (vp ₀ ) / v and b = Ds (df) / fd to simplify the expression. (Here, particularly, the a and b depend on the depth d.)
σ (t) = | a-bt |

  FIG. 3 shows a schematic example of the relationship of the main variables appearing in each of the above equations. In gray in FIG. 3 are the lens, the aperture and the light beam when in focus.

The PSF at the depth d can be modeled as the following equation (1) by Gaussian using the above blur amount σ = σ (d, t), and in the existing method, the following equation (2) IP ₀ is defined as IPSF obtained by integrating this over time over an exposure time t (0 ≦ t ≦ T) corresponding to one frame, and it is found that this is not dependent on the depth d . Then, since the normal image g ₀ obtained by constant-velocity movement imaging in the existing method is given by the convolution of the omnifocal image r and IP ₀ as shown in the following equation (3), the omnidirectional It is possible to determine the image r. (Note that, as described later, the normal image g ₀ and the omnifocal image r in the present invention are similar to this, and the same relationship holds, so g ₀ and r are used as the same variable names.) Or less x and the like in the formula (1) to (3) and later for example in the formula r (x) of the two-dimensional vector x = (X, Y) denote the pixel position as ∈R ^2.

(Description of IP ₁ used in the present invention)
In the present invention, with the above-described existing method, the depth d to an object that can not be acquired due to the information being lost is obtained. Here, the time (focusing time) at which focus is achieved is defined as time t = τ such that σ (d, t) = 0. By this definition, the depth d can be expressed as the following equation (4).
d = {(p ₀ + sτ) f} / {f- (p ₀ + sτ)} (4)

Although the outline description has already been mentioned, in the present invention, it is specifically obtained by modulating with the complex sine wave e ^jωt by the time correlation image sensor (that is, the sensor unit 20) as shown in the following equation (5) The correlation image g ₁ (first correlation image pair g ₁ ) is used as a complex modulation image to be generated. Here, in equation (5) By using the (IP _1, which corresponds to an extension of the IP ₀ of the formula (2) of the existing methods) function IP ₁ defined by the next equation (6), expresses the correlation image g ₁ in convolution of the IP ₁ defined the the all-focus image r.

Further, in the present invention, in particular, it is used that P ₁ (x) is a PSF having a two-dimensional x∈R ² independent of the depth d, and the following equation (6-1) holds.
IP ₁ ∝e ^jωτ P ₁ (x) (6-1)
That is, IP ₁ (x, d) in the present invention, which is introduced as a formal definition by extending IP ₀ of the existing method by being given by Equation (6), corresponds to the right side of Equation (6-1) It is proportional (∝) and therefore independent of the depth d. That is, “IP ₁ (x, d) = IP ₁ (x)”. Therefore, the principle of depth map acquisition of the present invention is as follows. That is, the depth constancy in function IP ₁ having a function form of Equation (6-1), is a constant in the phase (the integral of the correlation image g ₁ given by the convolution of equation (5), given in terms of e ^Jomegatau because it is the phase) is assumed to have the relationship between the focusing time tau, it is possible from the correlation image g ₁ get focus time tau at each pixel location x, and each pixel from the relationship in equation (4) It is possible to obtain the depth d of the position x.

In addition, the explanation that IP ₁ does not depend on the depth d will be described later in the supplementary explanation, and in the following, the detailed explanation of each part of the information acquisition device 1 of FIG. . The detailed description will be made in order for each of the following three topics.
<Regarding the sensor unit 20>
Regarding the relationship between imaging while moving the focus position of the optical system unit 10 and the reference signal in the sensor unit 20
Regarding Process of Obtaining All-In-Focus Image r and Depth Map in Image Processing Unit 40

<Regarding the sensor unit 20>
FIG. 4 is a circuit diagram of the sensor unit 20 for each photodiode PD as an example of an individual imaging element (an imaging element corresponding to each pixel of an image) constituting the imaging unit 21. As shown in the figure, each common photodiode PD is connected in parallel with six common element groups E _i (i = 1, 2,..., 6) for obtaining time correlation integral with respective reference signals. . Each element group E _i (i = 1, 2,..., 6) is configured as follows as shown. That is, a transistor Q _i (i = 1, 2,..., 6) for obtaining a current proportional to the current signal I of the photodiode PD (that is, a photoelectric conversion signal proportional to the light amount) and the reference signal V _i applied to the gate. And a capacitor C _i (i = 1, 2,..., 6) for storing the proportional current are connected in series, and are branched and connected between the transistor Qi and the capacitor Ci, The integrated value (accumulated charge) of the proportional current stored in the capacitor C _i in synchronization with the frame rate of the image in response to the readout signal ysel _i (1, 2,..., 6) applied is output out _i ( Each element group _Ei is configured to further include a switch SW _i (i = 1, 2,..., 6) of a transistor read out as i = 1, 2,. The reference signal V _i in the sensor unit 20 as a time correlation image sensor is common to all imaging elements (all pixels of an image) constituting the imaging unit 21.

By configuring as shown in FIG. 4, in each element group E _i , the output out _i is modulated by the weighting function f _i (i = 1, 2,..., 6) as in the following equation (7). The time integral of the product of the applied reference signal V _i (i = 1, 2,..., 6) (ie, applied voltage V _i ) and the current I of the photodiode PD, ie, obtained as proportional to the time correlation value Can. k is a proportional coefficient, which is common to each element group E _i . (In other words, each element characteristic (each characteristic of the transistor Q _i and the capacitor C _i etc. of the element group E _i is common to each other))

  In the above equation (7), t (0 ≦ t ≦ T) is used as the exposure time corresponding to one frame of the image, as in the case of the above equations (1) to (6).

The correspondence relationship between the circuit configuration of FIG. 4 and the sensor unit 20 of FIG. 1 as a functional block diagram as described above is as follows. As described above, each pixel of the imaging unit 21 of FIG. 1 corresponds to the photodiode PD of FIG. 4. Although the configuration corresponding to the reference signal generation unit 22 of FIG. 1 is not included in the circuit of FIG. 4 based on the correspondence for each pixel, the reference signal generated by the reference signal generation unit 22 It is input to the gate of each transistor Q _i as _i . Then, further under the corresponding per the pixel configuration corresponding to the integration circuit 23 of FIG. 1 is a respective element groups E _i in Figure 4, and outputs the images as respective integral values out _i.

As described above, the sensor unit 20 itself is configured as a time-correlated image sensor which is an existing technology disclosed in the above-mentioned Non-Patent Document 6 etc., and in one embodiment of the present invention, referring to FIG. This is utilized by inputting the reference signal V _i modulated with six weighting functions f _i as shown in FIG.

<Relationship between imaging while moving the focal position of the optical system unit 10 and the load function at the sensor unit 20>
That relationship corresponds to the details of step S1 of FIG. In one embodiment, the relationship can be set at an exposure time t (0 ≦ t ≦ T) for one frame as in the following equations (8-0) to (8-6).

FIG. 5 schematically shows, as [8-0] to [8-6], positions p and load functions f _i which change with time in conjunction with each other in the above equations (8-0) to (8-6). It is shown as a graph. In the exposure time t (0 ≦ t ≦ T) of one frame, the expression (8-0) corresponds to the lens and the imaging device in the optical system unit 10 in the same manner as in the description of IP ₀ and IP ₁ described above. The equation (8-1) to (8-6) indicates that the exposure time t (0.ltoreq.t.ltoreq.T) in conjunction with the movement. Among them, an example of weighting functions f _{1 to} f ₆ for modulating the reference signals V _{1 to} V ₆ input from the reference signal generation unit 22 is given.

Here, in particular, by outputting the time correlation integral of the above-mentioned equation (7) in the integration circuit 23 using the weight function f ₁ (example of the constant function ) given by the equation (8-1), the normal image g ₀ You can get It should be noted that if the load function f ₁ = 1 (that is, a constant) in the equation (7), it corresponds to an image in a normal meaning by accumulating the current of the photodiode PD as it is over the frame time. , “Normal image g ₀ ”.

Similarly, using the weight functions f ₂ and f ₃ (example of the first function group ) given by the equations (8-2) and (8-3), the time correlation integral of the above equation (7) is integrated By outputting at 23, the real part image Re (g ₁ ) and the imaginary part image Im (g ₁ ) in the first correlated image pair g ₁ can be obtained respectively. That is, the cosine function and the sine function of the equations (8-2) and (8-3) are phase terms e ^{jωτ in the} above-mentioned equations (5) and (6) according to Euler's formula ( ^ejωτ = cosωτ + jsinωτ). Corresponding to The frequency ω (the angular frequency ω of the complex sine wave) of Equations (8-2) and (8-3) is determined so that ω = 2π / T, that is, the corresponding period is the exposure time T. . As is apparent from equations (8-2) and (8-3) given by a cosine function and a sine function of frequency ω as an example, in one embodiment, the weighting functions f ₂ and f ₃ have exposure time T It can be given as the first orthogonal function group in.

Similarly, using the weighting functions f ₄ , f ₅ and f ₆ (examples of the second function group ) given by the equations (8-4), (8-5) and (8-6), the above equation ( By outputting the time correlation integral of 7) in the integration circuit 23, the respective images s ₁ , s ₂ and s ₃ of the second correlation image group can be obtained. As apparent from the specific functional forms of the equations (8-4), (8-5) and (8-6), in one embodiment, the reference signals f ₄ , f ₅ and f ₆ are at the exposure time T. It can be given as a second orthogonal function group.

The relationship between the weight function f _i and the reference signal V _i can be given as follows to give the reference signal V _i modulated by the weight function f _i . Here, V _a , V _b and v are constant voltages (voltages that become constant regardless of time), and V _a > v (absolute value).
V ₁ = V _a + v · f ₁
V ₂ = V _a + v · f ₂
V ₃ = V _a + v · f ₃
V ₄ = V _b · f ₄
V ₅ = V _b · f ₅
V ₆ = V _b · f ₆

<Regarding subtraction processing in conversion unit 45>
The relationship between the above reference signals V ₁ , V ₂ and V ₃ and specific functional forms of the load functions f ₁ , f ₂ and f ₃ of the equations (8-1), (8-2) and (8-3) more apparent, subtracts the ordinary image g0 obtained the conversion unit 45, from each of the first correlation image pair g ₁ of the underlying image pairs obtained as a reference signal V ₂ and V ₃ as the reference signal V ₁ it is, it is possible to obtain a first correlation image pair g ₁ as follows. The subtraction may be performed for each pixel given as a digital value. Incidentally, The following will describe a first correlation image pair g ₁ in the form of a time correlation integration in each pixel following the to equation (7) (analog value), the first as a digital value, the conversion unit 45 and thus obtaining a correlation image pair g _1.

Regarding Process of Obtaining All-In-Focus Image r and Depth Map in Image Processing Unit 40
Hereinafter, the details of the processing contents of the units 41, 42, and 43 of the image processing unit 40 will be described in order.

In the all-in-focus image generation unit 41, noise is added to the image in which the normal image g ₀ is obtained by convolution of the all-in-focus image r and IP ₀ as the depth invariant integration PSF of Equation (2). The omnifocal image r is calculated by the deconvolution as the reverse operation based on the relationship (the relationship of the following equation (9)) of being an object. Specifically, for example, the omnifocal image r may be generated by Wiener deconvolution using a Wiener filter, which is an existing method, and Fourier transform R (u of the omnifocal image r as an original image given by the following equation (10) , v) may be inverse Fourier transformed to generate an omnifocal image r.
g ₀ (x) = IP ₀ (x) * r (x) + n (x) (9)
R (u, v) = H * S / {| F [IP ₀ ] | ² S + N} (10)

In the above equations (9) and (10), x is a two-dimensional pixel position as described above, n (x) is noise, and * is a convolution operation. Further, F [·] is a Fourier transform of a signal etc., and, for example, F [IP ₀ ] is a Fourier transform of IP ₀ . _{Also, H = F [IP 0]} , R = F [r], a N = F [n], respectively may be determined as the Fourier transform.

Incidentally, as already described as those disclosed in Non-Patent Document 1 and Patent Document 5 supra, functions IP ₀ given by the above equation (2) is substantially the depth unchanged, and the position of the image It is a PSF model equation that it has the property that its functional form is not substantially changed by x (it has a shape similar to a two-dimensional Gaussian having approximately the same height and approximately the same spread centered on the position x) Empirically by giving a specific numerical value to the integrated value (ie, the equation (2)) obtained by integrating (the pillbox function as a simple model and the one modeled respectively by the Gaussian function of the equation (1)) Are known. That is, although it has been described that x of the function IP ₀ (x) has the center of the image at the optical axis center (image center and x = 0 (origin)) as shown in FIG. Even when the center deviates from the image center x = 0 and becomes x ₀ , the function form is (approximately) the same, and it is empirically found that the function form IP ₀ (xx ₀ ) is simply translated (approximate to) It is known to. Therefore, also in the present invention, the specific functional form (the functional form that becomes constant independently of the depth d and the position x) of the function IP _{0 in} the optical system unit 10 is determined in advance as the predetermined function of the optical system unit 10 Is calculated as a model based on numerical data etc. under the camera parameters of the above and a predetermined moving range set by the equation (8-0), and the deconvolution operation based on the above equations (9) and (10) is performed Thus, an omnifocal image r can be obtained.

The phase estimation unit 42 generates the first correlated image pair g ₁ (real part image) from the equation (5) of the convolution operation by the function IP ₁ described above and the equation (6-1) giving a specific function form of the function IP ₁ Re (g ₁₎ and the imaginary part image Im (g ₁₎₎ based on the following equation holds (11) and (12) with respect to, for the first pixel position x of the correlation image pair g _1, the phase tau ( Since the phase ωτ is exactly in proportion but the relationship is proportional, the candidate of the phase τ will be estimated as follows.
Re (g ₁ ) = cos (ωτ) P ₁ * r (11)
Im (g ₁ ) = sin (ωτ) P ₁ * r (12)

Equations (11) and (12) above indicate that the real part image Re (g ₁ ) and the imaginary part image Im (g ₁ ) are common images P obtained by the convolution of the function P ₁ and the omnifocal image r. It represents that it becomes an image obtained by multiplying ₁ * r by cos (ωτ) and sin (ωτ), respectively. Here, cos (ωτ) and sin (ωτ) as values to be multiplied are pixel position dependent through focusing time τ. (Note that although the proportionality coefficient in equation (6-1) is 1, but the phase candidate estimation result by the following calculation is determined by the ratio of equations (11) and (12), It does not depend on the basic value.)

Specifically, candidates for the phase τ may be estimated as follows based on the equations (11) and (12). That is, considering that the mathematical relational expression “cos ² θ + sin ² θ = 1” is applied, an image given by the sum of squares of the respective expressions (11) and (12) (ie, expression (11) An image obtained by squaring the pixel value at each pixel position x of (12) and (12) and adding the both together can be obtained as in the following equation (13).
| g ₁ | ² = | Re (g ₁ ) ² + j · Im (g ₁ ) ² |
= | Re (g ₁ ) | ² + | Im (g ₁ ) | ²
= | cos (ωτ) P ₁ * r | ² + | sin (ωτ) P ₁ * r | ²
= {cos ² (ωτ) + sin ² (ωτ)} · | P ₁ * r | ²
= | P ₁ * r | ² (13)

Furthermore, by taking the square of the equation (13), an absolute value image | g ₁ | can be obtained as in the following equation (14). As apparent from the equations (13) and (14), the absolute value image | g ₁ | means that the pixel value of each pixel position x is the real part image Re (g ₁ ) of the equations (11) and (12). And it becomes an image given by the squared sum of each pixel value of the position x of the imaginary part image Im (g ₁ ).
| g ₁ | = | P ₁ * r | (14)

Therefore, it is possible to obtain a plurality of candidates τ _[candidates] (x) of the phase τ at each pixel position x as satisfying all the above equations (11), (12) and (14). Expressing in a formula, a set given by the following formula (15) or (16) is a candidate to be obtained. Here, a set of elements A is described by the notation {A; condition satisfied by A}.
{τ _[candidate] (x); “| cos (ωτ _[candidate] (x)) | = Re (g ₁ (x)) / | g ₁ (x) |
And “| sin (ωτ _[candidate] (x)) | = Im (g ₁ (x)) / | g ₁ (x) |”} (15)
{τ _[candidate] (x); | tan (ωτ _[candidate] (x)) | = | Im (g ₁ (x)) / Re (g ₁ (x)) |} (16)
In addition, Formula (16) can also be written like the following Formula (17) using arctangent arctan2.
ωτ _[candidate] (x) = ωτ ₁ (x), π-ωτ ₁ (x), π + ωτ ₁ (x), 2π-ωτ ₁ (x) (17)
Here, ωτ ₁ (x) = arctan 2 (Re (g ₁ (x)) / | g ₁ (x) |, Im (g ₁ (x)) / | g ₁ (x) | The phase) ωτ ₁ (x) is in the first quadrant, ie 0 ≦ ωτ ₁ (x) ≦ π / 2.

  In addition, as is clear from the known graph form of the tangent function (absolute value) of equation (16), the elements of the set (that is, the number of candidates) are generally two candidates in the range of 0 ≦ ωτ <2π. Of τ can exist. Moreover, when calculating | requiring a candidate from Formula (16), calculation of the absolute value image of Formula (14) may be abbreviate | omitted. The existence of τ as two candidates is as follows.

That is, when the solution represented by (17) is substituted into (11) and (12), respectively, the following [1] to [4] are obtained.
[1] cos (ωt) P1r = Re (g1)
sin (ωt) P1r = Im (g1)

[2] cos (π-ωt) P1r = −cos (ωt) P1r = Re (g1)
sin (π-ωt) P1r = sin (ωt) P1r = Im (g1)

[3] cos (π + ωt) P1r = -cos (ωt) P1r = Re (g1)
sin (.pi. +. omega.t) P1r = -sin (.omega.t) P1r = Im (g1)

[4] cos (2π-ωt) P1r = cos (ωt) P1r = Re (g1)
sin (2π-ωt) P1r = -sin (ωt) P1r = Im (g1)

Therefore, under the assumption that g1 ≠ 0, [1] and [3] (when Re (g1) and Im (g1) have the same sign) or [2] and [4] (Re (g1) and Im (g1) )) Can only exist at the same time. In other words, the candidates are narrowed to two immediately from the signs of Re (g1) and Im (g1). (This is because (11) → (17) holds, but (17) → (11) does not necessarily hold.)

The phase determination unit 43 selects one of four or two candidates τ _[candidates] (x) obtained by the phase estimation unit 42 at each pixel position x. Determined as corresponding to As described in the outline description, the second correlation image group s ₁ , s ₂ , s ₃ is referred to in the determination.

Here, with regard to the second correlation image group s ₁ , s ₂ , s ₃ , the weighting functions f ₄ , f ₅ , f ₆ for obtaining them respectively are the aforementioned equations (8-4) to (8-6) and As shown in [8-4] to [8-6] of FIG. 5 as an example of the graph model, specifically, the exposure (usually, among the whole time t (0 ≦ t ≦ T) of the one frame is This is a weighting function having a function of limiting the same constant exposure as the image to only the first half 1/3, the middle 1/3 and the second half 1/3.

Therefore, in the phase determination unit 43, specifically, an image captured by limiting the exposure period to one third of the whole by the weight functions f ₄ , f ₅ , and f ₆ (ie, exposure) By comparing the respective images of the second correlation image group s ₁ , s ₂ , s ₃ respectively corresponding to a normal image of a limited period), it is judged to be the clearest (less blurred) at each pixel position x After identifying the image, one candidate τ such that the weighting functions f ₄ , f ₅ , f ₆ (of the identified ones) corresponding to the image exist within the period given as the exposure period. _[Candidate] (x) can be determined as corresponding to the in-focus position.

Here, any existing method may be used to determine which of the images of the second correlated image group s ₁ , s ₂ , s ₃ is the clearest with respect to each pixel position x. For example, frequency analysis may be performed on the predetermined region in the vicinity including the pixel position x by Fourier transform or the like to determine that the image determined to have the largest number of high frequency components is the sharpest.

Also, for example, the clearest image may be determined in the following procedures 1 to 4 similar to the method in Non-Patent Document 3 by Kanada et al.
(Procedure 1) A smoothing filter is applied to each image of the second correlation image group s ₁ , s ₂ , s ₃ to form images s _{1 [smoothing]} , s _{2 [smoothing]} , s _{3 [smoothing]} . obtain.
(Procedure 2) Each image h _i (i = 1, 2, 3) corresponding to the difference from the original image by the smoothing is determined as follows.
h _i = | s _i -s _{i [smoothing]} |
(Procedure 3) The image h _{i [smoothing]} is obtained by further smoothing the image h _i corresponding to the obtained difference.
(Procedure 4) At each pixel position x, the corresponding pixel values h _{i [smoothing]} (x) of a plurality of images h _{i [smoothing]} (i = 1, 2, 3) are compared, and the pixel value is maximum It is determined that one obtained by the above is obtained from the clearest image s _{i [max]} . If it writes in a formula, it is as follows.
i [max] (x) = argmax {hi _[smoothing] (x)}

  The depth map generation unit 44 generates a depth map using the in-focus time τ (x) which is proportional to the phase ωτ (x) of each pixel position x determined by the phase determination unit 43 as described above. Specifically, the depth map can be generated by calculating the depth d of each pixel position x according to the aforementioned equation (4).

  As described above, according to the present invention, it is possible to obtain both an omnifocal image and a depth map at high speed and with high accuracy by passive measurement using a time correlation image sensor. The following is a few supplementary notes of the present invention.

Regarding the First Another Embodiment of Phase Determination Unit 43
In the above, one candidate τ _[candidate] (x) is determined under the three weighting functions f ₄ , f ₅ and f ₆ corresponding to dividing the exposure period into three equal parts. another embodiment which extends Similarly, four or more of the N weighting function F _1, F _2, ..., using what the exposure period as F _N corresponds to N equal parts, one candidate tau _{[ Candidate]} (x) may be determined. That is, each load function F _i (i = 1, 2,..., N) can be written as follows. Here, the entire time for one frame is set to t (0 ≦ t ≦ T), similarly to the load functions of the above-mentioned equations (8-1) to (8-6).
F _i (t) = 1 ((i-1) · T / N <t <i · T / N)
F _i (t) = 0 (else)

In this case, the configuration shown in FIG. 4 as the configuration of each imaging element (photodiode PD) of the imaging unit 21 has N-3 additional elements connected in parallel similarly to the element groups E _{1 to} E ₆ . similar element group _{E 7, E 8, ... E} N + 3 a configuration in which a (not shown), N number of element groups E _4, E _5, ... the exposure period in E _{N + 3} to N equal parts In particular, the above-described load functions F _i corresponding to the above may be input as reference signals. Similarly, the determination of the clearest image may be performed on the second correlation image group formed of N images. Although N ≧ 4 in the above description, an embodiment of N = 2 is also possible.

Regarding Second Alternative Embodiment of Phase Determination Unit 43
As another embodiment, the phase determination unit 43 may determine _one of the phase candidates corresponding to the in-focus position without referring to the second correlation image group s ₁ , s ₂ , s _3. . That is, if it is known in advance that the depth at each position x in the image is within a predetermined range as the information given manually or the like, the information corresponding to the in-focus position may be determined by the known information. . For example, if it is known in advance that the acquisition target object of the depth map captured in the whole image is 1 m away from the camera on average and the range of the unevenness is ± 10 cm, the predetermined range is 90 cm Those within the range of up to 110 cm may be established as corresponding to the in-focus position.

<Regarding Movement of Focus Position of Optical System Unit 10 and Synchronization with Reference Signal in Reference Signal Generation Unit 22>
The optical system unit 10 moves the focus position at a substantially constant velocity (equal velocity determined under control processing with predetermined accuracy) by the drive unit 12 configured by any existing actuator mechanism or the like, and an image of the object Can be formed. As indicated by a line L1 in FIG. 1, a drive signal of the actuator (for example, a drive signal by pulse width modulation) and a reference signal by the reference signal generation unit 22 configured by any existing function generator or the like By controlling them so as to be synchronized with each other, synchronization operation as illustrated in FIG. 5 is possible. Although the control part as the said hardware is not shown in figure in FIG. 1, the said control part is realizable with arbitrary existing technologies, such as CPU and an exclusive circuit.

Regarding Setting of Movement of Focus Position by Optical System Unit 10
The predetermined movement range according to the equation (8-0) is set according to the predetermined camera parameters of the optical system unit 10 or the like so that the image is focused only once in the assumed depth range of the object OB to be imaged. do it. As predetermined parameters for setting the movement range, an initial position p ₀ and a speed s under an exposure time t (0 <t <T) may be set.

<Description of Function IP ₁ Given in Function Form of Expression (6-1) and Having Depth Invariant Property>
The formal definition gives the function form of the function IP ₁ given by the equation (6).
IP ₁ ∝e ^jωτ P ₁ (x) (6-1)

Here, the fact that the function form is the equation (6-1) can be particularly viewed in the present invention by carrying out a specific analysis on the function IP ₁ of the formal definition equation (6). It is the nature of the issue. In the analysis itself, the function IP _{0 in} the equation (2) of the existing method (Non-Patent Document 1 and Patent Document 5) has a property of depth invariance, giving specific numerical values to the model equation It is based on the same approach as published, specifically as briefly described below.

First, consider a case where a point spread function is given by a pillbox function ピ ル as a simple model as follows. That is, it is a pillbox function having a delta function ε <<< 1 within the range of DoF (the depth of field which can be regarded as being in focus), and a pillbox function of the radius c otherwise. Turn τ is the in-focus time as described above.
P (x, d, t) = Π (x, ε) (if | t-τ | <ε)
P (x, d, t) = Π (x, c) (else)
Here, the pillbox function Π is as follows.
Π (x, c) = 1 / (2πc ² ) (if | x | <c)
Π (x, c) = 0 (else)

By substituting the above model equation into the equation (6) of the definition and performing calculation, the calculation process on the way is omitted, but it is possible to obtain the function form IP ₁ as an analytical solution as follows. The analytical solution conforms to the function form of equation (6-1), that is, it is proportional to the phase term e ^jωτ , and the remaining terms are depth-invariant. It can confirm.

In the above, it has been shown that when a point spread function is modeled by a pillbox function と し て as a simple model, the function form is as shown in equation (6-1), but a more accurate model is modeled by Gaussian. It can be confirmed numerically that the same functional form is obtained. When modeling with Gaussian, the function IP ₁ can not be written in an analytical solution as in the case of the pillbox function, but can be written as definition including equation (6) as follows.

FIG. 6 is a diagram separately showing [1] to [4] a graph numerically confirming that IP ₁ written including the above-mentioned integral exhibits the behavior of the functional form as in equation (6-1) It is. [1] and [2] are graphs in which the real part and imaginary part of the function IP ₁ are plotted, and although the position x dependence is a two-dimensional dependence, only the one-dimensional change is cut out and shown here. Also, phase dependency is shown at θ = 2πt / T. It can be read from [1] and [2] that the form of the phase dependence is approximately cos θ and sin θ, and the function form of the position x dependence is constant considering the effects of the phase dependence.

Further, [3] and [4] in FIG. 6 indicate that the phase term e is numerically confirmed that IP ₁ written including the above-mentioned integral is depth-invariant as in equation (6-1). respectively real part and the imaginary part of the function IP ₁ / e ^{j? t} divided by ^{j? t,} shows a graph group that has been plotted for various depths d. Here, in both [3] and [4], it can be read that the graph shapes plotted for various depths d substantially overlap, that is, the graph shapes become almost the same, and therefore The depth invariant nature of the function IP ₁ can be confirmed numerically.

Although the qualitative property as described above is used as the principle of the calculation in the phase estimation unit 42 for the function IP ₁ , substantially necessary input is necessary in the details of the calculation in the phase estimation unit 42. data as is apparent from that only first correlation image pair g ₁ which is obtained from the sensor unit 20, a function IP ₀ used when generating the all-focus image r in omnifocal image generating unit 41 They are different and need not be given a specific function form of the function IP _1.

<Regarding another embodiment (second embodiment) related to use of reference signal etc.>
In the second embodiment, the circuit configuration of the integrating circuit 23 etc. for each pixel is connected in parallel by connecting six common element groups E _i (i = 1, 2,..., 6) as shown in FIG. Instead of simultaneously determining the original image pair of the normal image g ₀ and the first correlated image pair g ₁ and the second correlated image group s ₁ , s ₂ , s ₃ in a single pixel, May be

That is, as shown in FIG. 7 instead of the configuration of FIG. 4, the circuit configuration of the integration circuit 23 etc. for each pixel is not six but three common element groups E _i (i = 1, 2, 3). It is set as parallel connection. Here, in the configuration of FIG. 7, each element E _i is the same as that described in FIG. 4, and only the number of elements E _i arranged in parallel is different in FIG. 4 and FIG. The explanation is omitted. When the configuration of FIG. 7 is used, unlike the configuration of FIG. 4, an original image pair of the normal image g ₀ and the first correlation image pair g _{1 and} a second correlation image group s ₁ , s ₂ , since it is impossible to determine s ₃ simultaneously, the following procedures may be used instead of obtaining the same time. That is, as in the case of the known Bayer arrangement adopted in a single-plate color camera, predetermined adjacent pixels are different signals (different signals such as R signal or B signal to G signal) in the same pixel of the image. Use the method adopted for acquiring.

Specifically, in one of the adjacent pixels on the imaging surface 21, the original image pair of the normal image g ₀ and the first correlation image pair g ₁ is obtained by the 3-parallel circuit configuration of FIG. The second correlated image groups s ₁ , s ₂ , and s ₃ may be obtained by the circuit configuration of three parallels in FIG. 7 in the other of the adjacent pixels. In this case, by using the same reference signal as that described with reference to FIG. 4 for each image in the circuit configuration of FIG. 7 in the same manner, it can be similarly obtained as an integral output corresponding to the reference signal.

That is, when obtaining the original image pair of the normal image g _{0 and} the first correlated image pair g ₁ , the above-described equation (8-1) is applied to the terminals V ₁ , V ₂ and V ₃ in the 3-parallel circuit configuration of FIG. , (8-2), by inputting a reference signal V _1, V _2, V ₃ modulated by the weighting function f _1, f _2, f ₃ given by (8-3), the output out _1, out ₂ , out ₃ , an original image pair of the normal image g ₀ and the first correlation image pair g ₁ can be acquired. Similarly, in order to obtain the second correlation image group s ₁ , s ₂ , s ₃ , the above-described equations (8-4), respectively, are applied to the terminals V ₁ , V ₂ , V ₃ in the three parallel circuit configuration of FIG. By inputting the reference signals V ₄ , V ₅ and V ₆ modulated by the weight functions f ₄ , f ₅ and f ₆ given by (8-5) and (8-6), the outputs out ₁ , out ₂ and Each of the second correlation image groups s ₁ , s ₂ , and s ₃ can be acquired in out ₃ .

According to the second embodiment described above, one of the adjacent pixels acquires three images of the original image pair of the normal image g ₀ and the first correlation image pair g ₁ , and the other has the second correlation image group s ₁ , s ₂ , and s ₃ , the circuit configuration can be simplified as the configuration of FIG. 7 (three parallel) instead of the configuration of FIG. 4 (six parallel), and cost reduction can be achieved. It becomes.

  Here, adjacent pixels that acquire the other three images (that is, different signals) may be regarded as the same pixel (the same pixel at the same position) in the acquired image, or the adjacent pixels may be treated. It may be treated as (another pixel at an adjacent position), and another signal which is not acquired may be acquired by applying the interpolation which is an existing technology at the pixel position. When interpolation is applied, spatial resolution twice as high as when treated as the same pixel can be obtained. The application of the interpolation may be performed in the interpolation unit by providing the image processing unit 40 with an “interpolation unit” (not shown in FIG. 1). The interpolation unit may apply interpolation processing to each digital image obtained from the AD conversion unit 30 and then output the digital image interpolated to each of the units 41, 42, and 43.

Regarding yet another embodiment (third embodiment) relating to use of reference signal etc.
A third embodiment will be described which corresponds to a modification of the reference signal in the second embodiment using the circuit configuration of FIG. 7 as described above. The gist of the third embodiment is as follows. That is, as a weight function to be used, “f ₁ = 1, f ₂ = cos ωt, f ₃ = sin ωt” given by the above equations (8-1), (8-2) and (8-3) is the result The reference signals V ₁ , V ₂ , and V ₃ as input signals to the terminals are, for example, three-phase symmetrical signals, and “V ₁ = sin ωt, V ₂ = sin (ωt + 2π / 3), V By adopting “ ₃ = sin (ωt + 4π / 3)”, first, the first image group G ₁ , G ₂ , G ₃ is obtained at the outputs out ₁ , out ₂ , out ₃ , and these are respectively converted into AD conversion units Digitize at 30 as well. Then, the conversion process of the conversion unit 45 is changed as follows. That is, processing of obtaining _three images of the normal image g ₀ , the real part image Re (g ₁ ) and the imaginary part image Im (g ₁ ) by converting the first image group G ₁ , G ₂ , G ₃ Change to After that, the same process as described above may be performed.

That is, in the third embodiment, “f ₁ = 1, f ₂ = cos ωt, f ₃ = sin ωt” as the weight function as in the other embodiments above is eventually realized by the conversion processing. There is an effect that symmetrical ones can be used for the reference signal. The second correlation image groups s ₁ , s ₂ and s ₃ are the same as in the second embodiment described above, in the adjacent pixels according to the circuit configuration of FIG. And (8-6) can be obtained using the reference signals (ie, symmetrical signals) modulated by the weighting functions f ₄ , f ₅ , and f ₆ .

The details of the third embodiment will be described below. First, in the circuit configuration of FIG. 7, the multiplication characteristic of equation (21) is obtained from the relationship of equation (20) below. Here, each I _i (i = 1, 2, 3) is a drain current in the transistor Q _i , q is a charge, κ is a gate coefficient, k is a Boltzmann constant, and T is an absolute temperature. Further, the V upper line (the sum of V and the upper line) on the right side of the equation (21) represents the average value “(V ₁ + V ₂ + V ₃ ) / 3” of the reference signal (applied voltage).

In the third embodiment, a system g _a (t) and g _b (t) of appropriate weighting functions is prepared such that an orthonormal system is formed in one frame time 0 <t <T, and the following equation (22) To determine reference signals V ₁ , V ₂ and V ₃ as small signals from a predetermined bias voltage (predetermined DC component) and obtain first image groups G ₁ , G ₂ and G ₃ respectively. it can. (In the equation (22), the DC component is not described, and only the difference from the DC component is described.) For example, prepare g _a (t) = cos ωt and g _b (t) = sin ωt Thus, the following reference signals V ₁ , V ₂ and V ₃ can be defined as three-phase symmetrical.

The conversion unit 45, by performing the inverse operation based on the above relationship, the first image group G _1, G _2, G ₃ normal image g ₀ from the real part image Re (g ₁₎ and the imaginary part image Im (g ₁ The three images of) can be obtained. That is, the normal image g ₀ can be obtained as an average image of the first image group G ₁ , G ₂ , G ₃ as follows.
g ₀ = (G ₁ + G ₂ + G ₃ ) / 3
Further, using the determined normal image g ₀ , the real part image Re (g ₁ ) can be determined as follows, for example, using the relationship of the first expression of (22).
Re (g ₁ ) = G ₁ -g ₀
Further, using the determined normal image g ₀ and the real part image Re (g ₁ ), for example, the imaginary part image Im (g ₁ ) is determined as follows using the relationship of the second expression of (22) Can.
Im (g ₁ ) = (G ₂ −g ₀ + Re (g ₁ ) / 2) · (2 / √3)

Regarding Another Embodiment of the Converting Unit 45
In the embodiment to obtain a first original image pair correlation image pair g ₁ in the integration circuit 23, a first correlation image pair by subtracting the normal image g ₀ in the conversion unit 45 after undergoing the process of the AD converter 30 g ₁ I was trying to get Another embodiment of this is omitted directly, so as to obtain a first correlation image pair g ₁ by performing the subtraction processing in the analog value by the output or the like of the voltage difference, the subtraction processing of the conversion unit 45 in the integrating circuit 23 You may do so.

Regarding Another Embodiment of Weighting Functions f ₂ and f ₃ for Obtaining First Correlated Image Pair g ₁ >
With regard to the weighting functions f ₂ and f ₃ , the cases of using the cosine function as the real part and the sine function as the imaginary part for obtaining a complex modulation image as in equations (8-2) and (8-3) will be described. did. These have the property that the phase differs from each other in real space with the frame period 0 <t <T as the common period, and the phase estimation section 42 enables the phase estimation by the property. Therefore, not only the cosine function and sine function with respect to the weighting function f ₂ and f _3, and other functions, i.e., the frame time 0 <t <other functions different phases such as a common period of T with the properties May be used. For example, it is possible to use a line function shown a schematic graph example (function connecting the peak positions of the cosine function and a sine function by a straight line, respectively) as a dashed line and a solid line in FIG. 8 as a weighting function f ₂ and f _3.

<Regarding the image processing unit 40>
The image processing unit 40 can also be realized by software. In this case, the image processing unit 40 is realized by the computer by reading and executing a program that causes the computer to function as the image processing unit 40 in the computer. The computer may adopt a known hardware configuration such as a CPU (central processing unit), memory, and various I / Fs, and the CPU executes instructions corresponding to the functions of the respective units of the image processing unit 40. It will be. Further, the computer further includes a GPU (graphics processing device) capable of performing parallel processing at a higher speed than the CPU, and a program is loaded to the function of all or any part of the image processing unit 40 in place of the CPU. It may be performed.

  Furthermore, all or some of the functions of the image processing unit 40 may be realized by an ASIC and / or an FPGA or other dedicated circuit.

1 ... Information acquisition device
DESCRIPTION OF SYMBOLS 10 ... Optical system part, 11 ... Optical element mechanism, 12 ... Drive part
20: sensor unit, 21: imaging unit, 22: reference signal generation unit, 23: integration circuit
30 ... AD converter
40: image processing unit, 41: omnifocal image generation unit, 42: phase estimation unit, 43: phase determination unit, 44: depth map estimation unit

Claims (6)

An optical system unit configured to be capable of moving a focal position and forming an image of an object;
A sensor unit including an imaging unit that photoelectrically converts the formed image, and outputting, as an image, a signal obtained by time-integrating a product of the photoelectrically converted signal and a reference signal based on a predetermined load function;
An image processing unit that generates an omnifocal image of an object and a depth map based on the output image;
The optical system unit forms an image of an object while moving the focal position,
The sensor unit outputs a normal image and a first correlation image pair , respectively, as the time-integrated signal using the constant function and the first function group as the predetermined load function.
The image processing unit generates an omnifocal image based on the normal image, and generates a depth map based on the generated omnifocal image and the first correlated image pair. An optical system unit configured to be capable of moving a focal position and forming an image of an object;
A sensor unit including an imaging unit that photoelectrically converts the formed image, and outputting, as an image, a signal obtained by time-integrating a product of the photoelectrically converted signal and a reference signal based on a predetermined load function;
An image processing unit that generates an omnifocal image of an object and a depth map based on the output image;
The optical system unit forms an image of an object while moving the focal position,
The sensor unit outputs a first image group as the time-integrated signal using a constant function and a first function group as the predetermined load function.
The image processing unit obtains a normal image and a first correlation image pair by converting the first image group, and generates an omnifocal image based on the normal image, and generates the generated omnifocal image And generating a depth map based on the first correlated image pair. In the sensor unit, functions different in phase from each other are used as the first function,
The image processing unit estimates a focusing time at each pixel position based on mutually different phases reflected in the first correlated image pair , and generates the depth map based on the estimated focusing time. The information acquisition device according to claim 1 or 2, characterized by In the sensor unit, as the predetermined load function, a second function group is used as a time-integrated signal using a second function group having a constant value only during mutually different predetermined periods and having a value of zero outside the predetermined period. Output further correlation image groups,
The image processing unit estimates candidates for focusing time at each pixel position based on mutually different phases reflected in the first correlated image pair , and further, from among the estimated candidates, the second one 4. The information acquisition apparatus according to claim 3, wherein one of the correlated image groups in the predetermined period corresponding to the image where the corresponding pixel position is most sharp is determined as one corresponding to the focusing time. In the optical system unit, an image of an object is formed while moving a focal position at a substantially constant speed by an actuator;
The information acquisition apparatus according to any one of claims 1 to 4, wherein the predetermined reference signal is given by synchronizing with a drive control signal of the actuator.   In the image processing unit, all images from the normal image are generated by a deconvolution operation using a predetermined point spread function invariant to depth, corresponding to the case of forming an image of an object while moving the focus position in the optical system unit. The information acquisition apparatus according to any one of claims 1 to 5, wherein a focused image is generated.

Images