JP7065761B2 - Distance measuring device and distance measuring method - Google Patents

Description

The present invention relates to a distance measuring device and a distance measuring method.

Patent Document 1 describes "a plurality of pixels arranged in an array on the image pickup surface" as a configuration for realizing a high-performance image pickup device that performs focus adjustment (refocus), auto focus, distance measurement, etc. after shooting. An image sensor that converts the optical image captured in the image into an image signal and outputs it, a modulator provided on the light receiving surface of the image sensor to modulate the light intensity, and an image signal output from the image sensor temporarily. The modulator includes a first grid pattern composed of a plurality of concentric circles, comprising an image storage unit for storing the image and a signal processing unit for performing image processing of the image signal output from the image storage unit. The signal processing unit generates a moire fringe image by modulating the image signal output from the image sensor with a second lattice pattern composed of a plurality of concentric circles. " The imaging device is disclosed.

International Publication No. 2017/149678

In the image pickup apparatus described in Patent Document 1, in order to measure the distance, the distance information is extracted based on the high and low contrast of the image pickup result. However, in the configuration of the image pickup apparatus of the present invention, it is difficult to generate accurate distance information.

The present invention has been made in view of such a situation, and an object of the present invention is to enable accurate distance information to be generated in a distance measurement calculation of an image pickup device or the like.

The present application includes a plurality of means for solving at least a part of the above problems, and examples thereof are as follows. In order to solve the above problems, the distance measuring device according to one aspect of the present invention is projected onto the image sensor based on an image sensor that converts light into an electric signal and generates a sensor image and an imaging pattern. A modulator that modulates the light intensity, a complex sensor image generator that generates a complex sensor image having a complex number from the sensor image, and an evaluation index array generation that generates an evaluation index array based on the phase information of the complex sensor image. The evaluation index sequence generation unit includes an evaluation pattern including a plurality of different development patterns according to the focus distance, and a distance information generation unit that generates distance information using the evaluation index sequence. Is generated, a plurality of developed images are generated by performing a mutual correlation calculation between the complex sensor image and the plurality of developed patterns, and the plurality of developed images are laminated in order of close focus distance in the plane and depth directions. A three-dimensional complex information sequence having a It is characterized in that the stacked three-dimensional array is used as the evaluation index array.

According to the present invention, accurate distance information can be generated. Issues, configurations and effects other than those described above will be clarified by the following description of the embodiments.

It is a block diagram which shows the structural example of the distance measuring apparatus which is 1st Embodiment which concerns on this invention. It is a figure which shows the structural example of the image pickup part. It is a figure which shows the other structural example of the image pickup part. It is a figure which shows an example of the pattern for photography or the pattern for development. It is a figure which shows an example of the pattern for photography or the pattern for development. It is a figure for demonstrating that the projection image from the surface of a pattern substrate by diagonally incident parallel light to an image sensor causes in-plane deviation. It is a figure which shows the projection image of an example of a pattern for photography. It is a figure which shows an example of the development pattern. It is a figure which shows an example of the developed image by a correlation development method. It is a figure which shows the example of the combination of the initial phase in a fringe scan. It is a figure which shows an example of a pattern for photography. It is a figure which shows the configuration example of the distance measuring apparatus in the case of a time division fringe scan. It is a figure which shows the configuration example of the distance measuring apparatus in the case of a space division fringe scan. It is a figure which shows that the pattern for photography is projected when the object is at an infinite distance. It is a figure which shows that the pattern for photography is enlarged when an object is at a finite distance. It is a figure which shows the generation method of a complex sensor image. It is a figure which shows the example of the complex sensor image. It is a figure which shows the example of the evaluation pattern. It is a figure which shows an example of a distance measurement. It is a figure which shows the example of the evaluation index arrangement. It is a figure which shows an example of the profile of a complex developed image. It is a figure which shows an example of the profile of a complex developed image. It is a figure which shows an example of the profile of a complex developed image. It is a figure which shows an example of the profile of a complex developed image. It is a figure which shows an example of the profile of a complex developed image. It is a figure which shows an example of how the value of the array element of an evaluation index array changes according to the pitch of an evaluation pattern. It is a flowchart explaining an example of a distance measurement process. It is a figure which shows an example of the state of sampling an evaluation index array. It is a figure which shows an example of the distance image which data-converted the distance information. It is a figure which shows an example of the complex sensor image generation method. It is a block diagram which shows the structural example of the distance measuring apparatus which is 2nd Embodiment which concerns on this invention. It is a figure which shows an example of the state of the imaginary part absolute value and a sampling point of a complex information array. It is a figure which shows an example of the state of the imaginary part absolute value and a sampling point of a complex information array. It is a figure which shows an example of the state of the imaginary part absolute value and a sampling point of a complex information array. It is a figure which shows an example of the result of plotting the phase angle of the cross-correlation operation result of a complex sensor image and the evaluation pattern with respect to the pitch coefficient β'. It is a block diagram which shows an example of the structure of the evaluation index sequence generation part. It is a figure which shows an example of the generation method of a complex information array. It is a figure which shows an example of the structure of the evaluation index array conversion part. It is a figure which shows an example of the moire fringe by the moire development method. It is a figure which shows an example of the pattern for photography and the pattern for generation of an evaluation index array. It is a figure which shows an example of the pattern for photography and the pattern for generation of an evaluation index array. It is a figure which shows an example of the transparent electrode in the liquid crystal display element. It is a figure which shows an example of the voltage application pattern of a transparent electrode in a liquid crystal display element. It is a figure which shows an example of the voltage application pattern of a transparent electrode in a liquid crystal display element. It is a figure which shows an example of the voltage application pattern of a transparent electrode in a liquid crystal display element. It is a figure which shows an example of the voltage application pattern of a transparent electrode in a liquid crystal display element. It is a figure which shows an example of how the value of the array element of an evaluation index array changes according to the pitch of an evaluation pattern. It is a figure which shows an example of how the value of the array element of an evaluation index array changes according to the pitch of an evaluation pattern. It is a figure which shows an example of the state of the smoothing pattern generation. It is a figure which shows an example of the complex information array when the evaluation pattern which is not smoothed is used. It is a figure which shows an example of the complex information array when the smoothing pattern is used. It is a figure which shows an example of the state of the weighted smoothing pattern generation. It is a figure which shows an example of the weighting of a weighted smoothing pattern generation. It is a figure which shows an example of the user interface of the distance measurement software. It is a figure which shows an example of the smoothing pattern generation in the depth direction. It is a figure which shows an example of the smoothing pattern generation in the depth direction. It is a figure which shows an example of the smoothing pattern generation in the depth direction. It is a figure which shows an example of the smoothing pattern generation in the depth direction. It is a figure which shows an example of the smoothing pattern generation in the depth direction. It is a figure which shows the profile of the complex development image. It is a figure which shows an example of a smoothing kernel. It is a figure which shows an example of the kernel width of a smoothing kernel. It is a figure which shows an example of the profile of a complex developed image. It is a figure which shows an example which differentiated the profile of a complex development image. It is a figure which shows an example of the complex differential pattern generation. It is a flowchart explaining an example of the distance measurement processing by in-plane smoothing. It is a figure which shows an example of the differential smoothing pattern generation. It is a flowchart explaining an example of a distance measurement process. It is a figure which shows another example of a differential pattern.

In the following embodiments, when necessary for convenience, the description will be divided into a plurality of sections or embodiments, but unless otherwise specified, they are not unrelated to each other, one of which is the other. It is related to some or all of the modified examples, details, supplementary explanations, etc.
Further, in the following embodiments, when the number of elements (including the number, numerical value, quantity, range, etc.) is referred to, when it is specified in particular, or when it is clearly limited to a specific number in principle, etc. Except for this, the number is not limited to the specific number, and may be more than or less than the specific number.

Furthermore, it goes without saying that, in each embodiment, the components (including element steps and the like) are not necessarily essential unless otherwise specified or clearly considered to be essential in principle. stomach. Similarly, in the following embodiments, when the shape, positional relationship, etc. of the constituent elements are referred to, the shape is substantially the same, except when it is clearly stated or when it is considered that it is not clearly the case in principle. Etc., etc. shall be included. This also applies to the above numerical values and ranges. Further, in all the drawings for explaining each embodiment, the same members are designated by the same reference numerals in principle, and the repeated description thereof will be omitted.

Hereinafter, examples of the present invention will be described with reference to the drawings.

<First embodiment>
<Principle of shooting infinity objects>
FIG. 1 shows a configuration example of the distance measuring device 101 according to the first embodiment of the present invention.

The distance measuring device 101 acquires an image of an object in the outside world without using a lens to form an image, and as shown in FIG. 1, the image pickup unit 102, the image processing unit 106, the controller 107, and the distance measuring variable. It is composed of a setting unit 111.

The image processing unit 106 includes a complex sensor image generation unit 108, an evaluation index sequence generation unit 109, and a distance information generation unit 110.

FIG. 2 is a diagram showing a configuration example of the imaging unit. The image pickup unit 102 includes an image sensor 103, a pattern substrate 104, and a shooting pattern 105.

One of the planes of the pattern substrate 104 is fixed in close contact with the light receiving surface of the image sensor 103, and the pattern 105 for photographing is formed on the other plane. The pattern substrate 104 is made of a material transparent to visible light such as glass and plastic.

The photographing pattern 105 is formed by depositing a metal such as aluminum or chromium by, for example, a sputtering method used in a semiconductor process. Light and shade are added by the aluminum-deposited pattern and the non-deposited pattern.

The formation of the photographing pattern 105 is not limited to this, and any means may be used as long as it can realize modulation of the transmittance, for example, by printing with an inkjet printer or the like to add light and shade. .. Further, although visible light has been described as an example here, for example, when photographing far infrared rays, the pattern substrate 104 is a material to be photographed, for example, a material transparent to far infrared rays such as germanium, silicon, and chalcogenide. A material that is transparent to the wavelength to be used may be used, and a material that blocks the photographing pattern 105 may be used.

Although the method of forming the photographing pattern 105 on the pattern substrate 104 has been described here, it may be formed as shown in FIG.

FIG. 3 is a diagram showing another configuration example of the imaging unit. The imaging unit 102 forms an imaging pattern 105 on a thin film and is held by a support member 301. In the distance measuring device 101, the shooting angle of view can be changed by the thickness of the pattern substrate 104. Therefore, for example, if the pattern substrate 104 has the configuration shown in FIG. 3 and has a function of changing the length of the support member 301, it is possible to change the angle of view at the time of shooting.

Pixels 103a, which are light receiving elements, are regularly arranged in a grid pattern on the surface of the image sensor 103. The image sensor 103 converts an optical image received by the pixel 103a into an image signal which is an electric signal. The image signal output from the image sensor 103 is image-processed and output by the image processing unit 106, which is an image processing unit.

In the above configuration, when photographing, the light intensity of the light transmitted through the photographing pattern 105 is modulated by the pattern, and the transmitted light is received by the image sensor 103. As for the image signal output from the image sensor 103, the data processed by the image processing unit 106 is output to the controller 107. When the output of the image processing unit 106 is output to a host computer or an external recording medium, the controller 107 converts the data format so as to be compatible with an interface such as USB and outputs the data.

Subsequently, the image capturing principle in the distance measuring device 101 will be described. First, the photographing pattern 105 is a concentric pattern in which the pitch becomes finer in inverse proportion to the radius from the center. Here, the photographing pattern 105 is defined as the following equation (1) by using the radius r from the reference coordinates which are the centers of the concentric circles and the pitch coefficient β.

It is assumed that the photographing pattern 105 is transmittance-modulated in proportion to the equation (1).

Plates with stripes such as the photographing pattern 105 are called Gabor zone plates or Fresnel zone plates. FIG. 4 shows an example of a Gabor zone plate of the formula (1), and FIG. 5 shows an example of a Fresnel zone plate obtained by binarizing the formula (1) with a threshold value of 1.

From here onward, for the sake of simplicity, only the x-axis direction will be described by a mathematical formula, but it is possible to expand and think in two dimensions by similarly considering the y-axis direction.

As shown in FIG. 6, it is assumed that parallel light is incident on the pattern substrate 104 having a thickness d on which the photographing pattern 105 is formed at an angle θ ₀ in the x-axis direction. Geometrically, with the refraction angle in the pattern substrate 104 as θ, the light multiplied by the transmittance of the lattice on the surface is incident on the image sensor 103 with a deviation of k = d · tan θ. At this time, the image having an intensity distribution as shown in the following equation (2) is detected on the image sensor 103.

Note that Φ indicates the initial phase of the transmittance distribution in the equation (1). The projected image of the photographing pattern 105 is projected with a k-shift as shown in the equation (2). This is the output of the image pickup unit 102.

In the development of the captured image, a bright spot (FIG. 9) having a shift amount k is obtained by calculating a cross-correlation function between the projected image of the photographing pattern 105 (FIG. 7) and the developing pattern 801 (FIG. 8). ..

In general, if the cross-correlation operation is performed by a two-dimensional convolution operation, the amount of calculation becomes large. Therefore, an example of using a Fourier transform with a small amount of calculation will be described using a mathematical formula.

First, since the developing pattern 801 uses a Gabor zone plate or a Fresnel zone plate like the photographing pattern 105, the developing pattern 801 can be expressed by the following equation (3) using the initial phase Φ. can.

Since the development pattern 801 is used in the image processing by the image processing unit 106, it is not necessary to offset it by 1 as in the equation (1), and there is no problem even if it has a negative value.

The Fourier transforms of the equations (2) and (3) are as shown in the following equations (4) and (5), respectively.

Here, F represents the operation of the Fourier transform, u is the frequency coordinate in the x direction, and δ with parentheses is the delta function. What is important in the equations (4) and (5) is that the equation after the Fourier transform is also a Fresnel zone plate or a Gabor zone plate. Therefore, the development pattern after the Fourier transform may be directly generated based on the equations (4) and (5). This makes it possible to reduce the amount of calculation.

Next, the following equation (6) is obtained by multiplying the equation (4) and the equation (5).

The term exp (-iku) expressed by the exponential function in the equation (6) is a signal component, and when this term is Fourier transformed, it is transformed as in the following equation (7), and the position is k at the original x-axis. You can get a bright spot.

This bright spot indicates a luminous flux at infinity, and is an image taken by the distance measuring device 101.

In the correlation development method, as long as the autocorrelation function of the pattern has a single peak, it may be realized by a pattern not limited to the Fresnel zone plate or the Gabor zone plate, for example, a random pattern.

<Noise cancellation>
By the way, the conversion from the above-mentioned equation (6) to the equation (7) has been described focusing on the signal component, but in reality, the terms other than the signal component hinder the development and cause noise. Therefore, noise cancellation based on fringe scan is performed. Specifically, when the multiplication result of the equation (6) is integrated with respect to the initial phase Φ as shown in the following equation (8) by using the orthogonality of the trigonometric function, the noise term is canceled and the signal term is canceled. Only a constant multiple of can be left.

Although the equation (8) is shown in the form of integration, in reality, the initial phases Φ (Φ = 0, Φ = π / 2, Φ) of a plurality of types (4 types in the case of the same figure) shown in FIG. 10 are shown. The same effect can be obtained by calculating the sum of the combinations of = π, Φ = 3π / 2). Like this combination, the phase Φ may be set so as to equally divide the angle between 0 and 2π.

In the fringe scan described above, it is necessary to use a plurality of patterns having different initial phases Φ as the photographing pattern 105. To achieve this, there are a method of switching patterns by time division and a method of switching patterns by spatial division.

FIG. 12 shows the configuration of the distance measuring device 101 when the pattern is switched by time division to realize the fringe scan. As the photographing pattern 1201, a liquid crystal display element or the like capable of electrically switching and displaying a plurality of initial phases Φ shown in FIG. 10 may be used. The switching timing of the liquid crystal display element and the shutter timing of the image sensor 103 are controlled in synchronization with the modulator control unit 1202, and after acquiring four images, a fringe scan operation is performed.

When performing a fringe scan using a liquid crystal display element, the electrodes for driving the liquid crystal display element may be configured as shown in FIG. 42, for example. The electrode is configured as a concentric transparent electrode so as to divide one cycle of the photographing pattern 105 into four, and is connected to the outer electrode every four from the inside, and finally the terminal portion 4201 of the electrode. Four electrodes are output as drive terminals. One cycle of the photographing pattern 105 corresponds to one set of a black ring (or a black circle) adjacent to the white ring of the photographing pattern shown in FIG. In order to actually change the initial phase by applying a predetermined voltage to these, the voltage states applied to the four electrodes are temporally switched between two states of "0" and "1", and the initial phase shown in FIG. 43 is shown. The state of phase Φ = 0, the state of initial phase Φ = π / 2 shown in FIG. 44, the state of initial phase Φ = π shown in FIG. 45, and the state of initial phase Φ = 3π / 2 shown in FIG. 46 are displayed. This is achieved by changing the light transmission rate of the liquid crystal display element.

On the other hand, FIG. 13 shows the configuration of the distance measuring device 101 in the case where the pattern is switched by spatial division to realize the fringe scan. As the photographing pattern 1301, the photographing pattern 1301 having a plurality of initial phases shown in FIG. 11 may be used. Then, after acquiring one image, the image segmentation unit 1302 divides the image into four images corresponding to the pattern of each initial phase, and then performs a fringe scan operation.

<Distance measurement principle>
FIG. 14 shows the projection of the photographing pattern 105 onto the image sensor 103 when the subject is far away (when the object is at an infinite distance) described above. When the spherical wave from the point 1401 constituting the distant object becomes a plane wave while propagating a sufficiently long distance and irradiates the photographing pattern 105, and the projected image 1402 is projected on the image sensor 103, the projected image is It has almost the same shape as the photographing pattern 105. Therefore, it is possible to obtain a single bright spot by performing a developing process on the projected image 1402 using the developing pattern 801. Next, imaging of a finite-distance object will be described.

FIG. 15 is a diagram showing that when the object to be imaged is at a finite distance, the projection of the photographing pattern 105 on the image sensor 103 is magnified more than that of the photographing pattern 105. When a spherical wave from a point 1501 constituting an object illuminates the photographing pattern 105 and the projected image 1502 is projected onto the image sensor 103, the projected image is magnified almost uniformly. The size ratio α between the projected image and the photographing pattern can be calculated by the following equation (9) using the distance f from the photographing pattern 105 to the point 1501.

Therefore, if the development process is performed using the development pattern 801 designed for parallel light as it is, an in-focus image cannot be developed. Therefore, if the development pattern 801 is multiplied by the pitch coefficient β'expressed in the following equation (10) to be magnified according to the projected image of the uniformly enlarged image shooting pattern 105, the image in focus can be obtained. Development becomes possible.

Therefore, it is possible to measure the distance to the image pickup target by obtaining the enlargement ratio α ² of the development pattern 801 that can reproduce the image in focus at the time of development.

In the lens focus method, which measures the distance based on the out-of-focus of the image taken by a normal camera, the distance information is extracted mainly by calculating the focus position where the contrast of the image becomes high, but the distance measuring device 101 uses the distance measuring device 101. The point spread function is not a Gaussian function type shape like a normal camera, but a Sinc function type shape that vibrates positively and negatively. Therefore, in the method of simply extracting the distance information based on the contrast of the image, the error included in the extracted distance information becomes large, and high-precision distance measurement cannot be performed.

Therefore, in the distance measuring device 101, the enlargement ratio α ² of the developing pattern 801 in focus is obtained by using an evaluation index different from the contrast of the developed image, and the distance information is extracted based on the enlargement ratio α ² . do.

A distance measuring method using the distance measuring device 101 configured as described above will be described. The distance measuring device 101 supplies the sensor image (image signal) generated by the image sensor 103 to the image processing unit 106, and generates distance information by predetermined image processing. In the configuration for performing the time-division fringe scan shown in FIG. 12, the modulator control unit 1202 switches and displays a plurality of initial phases Φ shown in FIG. 10 on the photographing pattern 1201 to display four images. An image is taken and four sensor images are supplied to the image processing unit 106.

On the other hand, in the configuration in which the spatial division fringe scan shown in FIG. 13 is performed, one sensor image is divided into four regions by the image segmentation unit 1302, and the plurality of initial phases Φ shown in FIG. 10 are used. After generating the captured sensor image, the sensor image is supplied to the image processing unit 106.

In the image processing unit 106, the complex sensor image generation unit 108, the evaluation index sequence generation unit 109, and the distance information generation unit 110 sequentially perform predetermined processing. That is, the complex sensor image generation unit 108 generates the complex sensor image 1602 based on the sensor image 1601 which is the output of the image sensor 103.

FIG. 16 is a diagram showing a method of generating a complex sensor image. Here, the _four sensor images are _IS1 , _IS2 , IS3, _{IS4 ,} and the initial _phases of the photographing pattern 105 corresponding to _each sensor image are _Φ1 , Φ2, Φ3, and Φ4. For example, when the shooting patterns shown in FIGS. 10 and 11 are used, Φ ₁ = 0, Φ ₂ = π / 2, Φ ₃ = π, and Φ ₄ = 3π / 2 can be used. The complex sensor image generation unit 108 generates the complex sensor image 1602 based on the four sensor images 1601 which are the outputs of the image sensor 103. Then, as shown in the following equation (11), the complex is obtained by adding the results of multiplying each sensor image by exp (iΦ ₁ ), exp (iΦ ₂ ), exp (iΦ ₃ ), and exp (iΦ ₄ ), respectively. Generate a sensor image 1602.

Next, the evaluation index array generation unit 109 generates the evaluation index array 1603 based on the complex sensor image 1602 which is the output of the complex sensor image generation unit 108. The method of generating the evaluation index sequence 1603 will be described with reference to FIGS. 17, 18, 20, 25, 36, and 37.

FIG. 17 is a diagram showing an example of a complex sensor image. When the subject is a point light source, the complex sensor image 1602 becomes information including a real part and an imaginary part.

When the pitch coefficient when the photographing pattern 105 is enlarged at the enlargement ratio α ² is set to β', the evaluation index sequence generation unit 109 changes β'as shown in FIG. 18 (β'> β 0, β). '= β0, β'<β0) Cross-correlation calculation is performed between the evaluation index sequence generation pattern 1801 and the complex sensor image 1602. 21 to 25 show an example of the cross-correlation calculation result.

FIG. 36 is a block diagram showing an example of the configuration of the evaluation index sequence generation unit. The evaluation index sequence generation unit 109 includes an evaluation pattern generation unit 3601, an evaluation pattern storage unit 3602, a cross-correlation calculation unit 3603, and an evaluation index sequence conversion unit 3604.

The evaluation pattern generation unit 3601 generates a plurality of development patterns 801 having different pitch coefficients β'for evaluation. The plurality of development patterns for evaluation are evaluation index sequence generation patterns (hereinafter, may be referred to as evaluation patterns) 1801. The evaluation pattern generation unit 3601 determines the pitch coefficient β'of the development pattern 801 by using the input from the distance measurement variable setting unit 111.

The evaluation pattern generation unit 3601 stores the generated evaluation pattern 1801 in the evaluation pattern storage unit 3602. The evaluation pattern storage unit 3602 is a readable / writable storage area, and can be realized by, for example, a non-volatile storage element.

The cross-correlation calculation unit 3603 reads the evaluation pattern 1801 from the evaluation pattern storage unit 3602 each time the complex sensor image 1602 input from the complex sensor image generation unit 108 is input.

FIG. 37 is a diagram showing an example of a method for generating a complex information array. As shown in FIG. 37, the cross-correlation calculation unit 3603 performs a cross-correlation calculation with all the patterns in the evaluation pattern 1801 for each complex sensor image 1602 to generate a complex developed image, and arranges the complex developed image. Generates a complex information array 3701.

For example, when the evaluation pattern 1801 consists of N (N is a natural number) patterns (pitch coefficients are _β'1 , _β'2 , ..., _β'N , respectively), the cross-correlation calculation unit 3603 is one complex number. The sensor image 1602 is subjected to cross-correlation calculation with N evaluation patterns 1801 N times for convenience, and N complex developed images are generated. The mutual correlation calculation unit 3603 superimposes N complex developed images in the depth direction (stacking direction in which the focus distances from the distance measuring device 101 are stacked in order of increasing focus distance on the extension of the direction of the subject 1901 as seen from the distance measuring device 101). It is output as a stacked three-dimensional complex information array 3701.

The evaluation index array conversion unit 3604 generates and outputs an evaluation index array 1603 used as an evaluation index for distance measurement using the complex information array 3701. An example of the evaluation index sequence 1603 is shown in FIG.

FIG. 20 is a diagram showing an example of an evaluation index array. The evaluation index array 1603 is a three-dimensional array having an evaluation array of two-dimensional image information according to the information in the depth direction.

As shown in FIG. 18, with respect to the development pattern 801 having a different pitch coefficient β'generated by the evaluation pattern generation unit 3601, the pitch coefficient β ₀ is a coefficient of the image shooting pattern projected image projected on the sensor image 1601. .. It should be noted in the results of the intercorrelation calculation shown in FIGS. 21 to 25 that when the pitch coefficient β'matches the pitch coefficient β ₀ , the imaginary component of the intercorrelation calculation result shown by the broken line becomes zero. Therefore, the amplitude of the imaginary component increases as the difference in pitch coefficients increases. Therefore, searching for the pitch coefficient β'where the imaginary number component becomes zero is a process in the distance information generation unit 110, which will be described later.

Therefore, the evaluation index array conversion unit 3604 stores and outputs the imaginary absolute value of the complex information array 3701 in the evaluation index array 1603. Assuming that the number of pixels of each complex developed image in the complex information array 3701 is L × M, as shown in FIG. 20, the evaluation index array 1603 is composed of N L × M two-dimensional arrays 3 It becomes a dimensional array.

Finally, the distance information generation unit 110 generates the distance information 2901 based on the evaluation index array 1603 which is the output of the evaluation index array conversion unit 3604. As described above, the evaluation index array 1603 becomes zero when the pitch coefficient β'matches the pitch coefficient β ₀ , but may not be zero in the case of a general subject.

FIG. 26 is a diagram showing an example of how the values of the array elements of the evaluation index array change according to the pitch of the evaluation pattern. In other words, FIG. 26 shows the pitch coefficient β'dependence in some element of the evaluation index array 1603.

In FIG. 26, the (l, m) th sequence elements 2001, 2002, and 2003 are extracted from the N L × M sequences shown in FIG. 20, and these N sequence elements are plotted. create.

From FIG. 26, it can be read that the array elements of the evaluation index array 1603 do not become zero due to the influence of the surrounding elements, but show the minimum value under the condition that β'= β ₀ . In this way, the enlargement factor α ² satisfying the equation (10) is obtained from β'at the point where the array element of the evaluation index array 1603 is the minimum value, and the distance f from the equation (9) to the subject 1901 is calculated. Can be done.

The controller 107 outputs the distance information 2901, which is the output of the distance information generation unit 110, after performing data conversion according to the intended use. For example, as shown in FIG. 19, when the distance information 2901 is generated based on the sensor image of the subject 1901 and the distance image is displayed on the display, the data is converted into the distance image as shown in FIG. 29 and output. In this case, in the distance image of FIG. 29, the distance information of the area 2902 corresponding to the subject 1901 is output as the distance f to the subject. As shown in FIG. 29, the controller 107 may output a distance image that overlaps the distance information 2901 and the normal image obtained by shooting the shooting target. When the distance information 2901 is used for controlling the mechanical device, the operation amount for achieving the target control amount may be calculated from the distance information 2901, and the calculated operation amount may be output. Further, the distance measurement variable may be input to the distance measurement variable setting unit 111 via the user interface, and the data format of the data output from the controller 107 may be changed according to the input variable.

FIG. 27 is a flowchart illustrating an example of the distance measurement process. When the distance measurement process is started, first, in step 2701, the evaluation pattern generation unit 3601 generates an evaluation pattern 1801 composed of N development patterns 801 having different β's.

Next, in step 2702, the image sensor 103 acquires the sensor image 1601. Then, in step 2703, the complex sensor image generation unit 108 generates the complex sensor image 1602 based on the sensor image 1601 acquired in step 2702.

Then, in step 2704, the cross-correlation calculation unit 3603 performs a cross-correlation calculation between the complex sensor image 1602 generated in step 2703 and the evaluation pattern 1801 generated in step 2701 as shown in FIG. 37. , Generates a complex information array 3701 containing a plurality of complex developed images.

Then, in step 2705, the evaluation index sequence conversion unit 3604 generates the evaluation index sequence 1603 based on the complex information sequence 3701. The evaluation index array 1603 is generated as a three-dimensional array as shown in FIG.

Then, in step 2706, the distance information generation unit 110 generates distance information 2901 based on the evaluation index array 1603 generated in step 2705, outputs the distance information 2901 to the controller 107, and ends the distance measurement process. ..

The values of the array elements of the evaluation index array 1603 shown by the curve in FIG. 26 are sampled at the same number of sampling points N as the number N of the development patterns 801 included in the evaluation pattern 1801 generated in step 2701. Therefore, as shown by the black dots in FIG. 28, the value of the evaluation index can be obtained discretely with respect to β'. In step 2706, β'with the smallest evaluation index among the N sampled β's may be searched for. Further, the accuracy of the distance information 2901 may be improved by performing curve fitting or interpolation processing on the number of data points in FIG. 28.

In the description described later, four images are used as the sensor image 1601, but as shown in FIG. 30, if there are at least two images, the distance can be measured. In this case, for example, the complex sensor image generation unit 108 generates the complex sensor image 1602 after generating the sensor image 3003 from which the low frequency components 3001 and 3002 of the respective images are removed from the two images of the sensor image 1601. .. The low frequency component 3001 and the low frequency component 3002 may be removed by, for example, offset removing the average luminance of the sensor image 1601 or by performing a high-pass filter process on the sensor image 1601 to remove the low frequency component. You may go.

By measuring the distance from two sensor images 1601, it is possible to shorten the acquisition time of the sensor image in the case of time-divided fringe scan, and in the case of spatially divided fringe scan, the number of divisions of the sensor image. By reducing the number of, the resolution of the sensor image after division is improved, and the resolution of the distance information 2901 can be increased.

The distance may be measured from the three sensor images 1601, or the noise may be further reduced by using five or more images.

The low frequency component refers to a frequency component lower than the spatial frequency of the signal component included in the sensor image.

Further, in the above description, the pitch coefficient β'that minimizes the absolute value of the imaginary part of the complex information sequence 3701 is obtained. The ratio of the value to the absolute value of the real part may be used as an index at the time of generating the distance information 2901. In this case, the distance information 2901 can be generated based on the pitch coefficient β'where the array elements of the evaluation index array 1603 are the maximum or the minimum.

Further, instead of the imaginary part absolute value of the complex information array 3701, the imaginary part of the complex information array 3701 may be stored in the evaluation index array 1603. By doing so, the array elements of the evaluation index array 1603 become values having positive and negative signs, and the position of the object to be photographed is located in front of or behind the focus position from one L × M array. It becomes possible to determine the relationship.

Further, the method of determining a plurality of β'used for generating the development pattern 801 in the evaluation pattern 1801 is based on the distance measurement range and the distance measurement resolution input to the distance measurement variable setting unit 111 via the user interface. The evaluation index sequence generation unit 109 may select β'and determine the number and intervals of sampling points shown in FIG. 28.

Further, in the generation of the evaluation index array 1603, the cross-correlation calculation between the complex sensor image 1602 and the development pattern 801 was used, but the complex sensor image 1602 is multiplied by the development pattern 801 as shown in FIG. 39. The evaluation index sequence 1603 may be generated by performing the Fourier transform of the moire fringes after generating the moire fringes.

Further, the configuration using the Gabor zone plate and the Fresnel zone plate shown in FIGS. 4 and 5 as the photographing pattern 105 and the developing pattern 801 has been described, but the elliptical pattern shown in FIG. 40 is used as the photographing pattern. It may be used as 105 or a development pattern 801. With such a configuration, even when a rectangular sensor is used as the image sensor 103, it becomes possible to shoot with a pattern of the optimum pitch for the size and resolution of the sensor in the vertical and horizontal directions, and the resolution of the distance information 2901. Can be optimized. When performing time-division fringe scan with an elliptical pattern, the concentric transparent electrode pattern shown in FIG. 42 may be replaced with an elliptical one.

Further, the random pattern shown in FIG. 41 may be used as the photographing pattern 105 or the evaluation pattern 1801. With such a configuration, it is possible to conceal the evaluation pattern 1801 for measuring the distance and enhance the security.

According to the configuration / method described above, the distance information of the subject can be measured by the distance measuring device 101 having a simple configuration shown in FIG.

<Second embodiment>
<Non-search type ranging minimum value calculation type>
FIG. 31 is a block diagram showing a configuration example of the distance measuring device according to the second embodiment of the present invention. The distance measuring device 101 according to the second embodiment is basically the same as the distance measuring device 101 according to the first embodiment, but there are some differences. Specifically, the difference is that the distance measurement range input unit 3101 is added to the distance measurement device 101 according to the second embodiment, and the evaluation index array is an evaluation index array generation pattern that satisfies a predetermined condition. The pitch coefficient β'of 1801 is calculated non-searchably. As a result, the distance information 2901 is generated at high speed, so that non-search type distance measurement can be performed.

In the distance measuring device 101 according to the second embodiment, the distance measuring range input unit 3101 receives the distance measuring range input by the user and supplies it to the distance measuring variable setting unit 111. The user inputs the distance measurement range by, for example, specifying the latest distance _RN and the farthest distance _LF among the distance measurement ranges.

The distance measurement variable setting unit 111 determines the pitch coefficients of a plurality of development patterns 801 included in the evaluation pattern 1801 generated by the evaluation pattern generation unit 3601 based on the recent distance L _N and the farthest distance _LF . It is output to the evaluation index sequence generation unit 109.

In the evaluation index array generation unit 109, the evaluation pattern generation unit 3601 generates the evaluation pattern 1801 according to the determined pitch coefficient, and stores the evaluation pattern 1801 in the evaluation pattern storage unit 3602. The cross-correlation calculation unit 3603 generates and outputs a complex information array 3701 by the cross-correlation calculation of the input complex sensor image and each development pattern 801 in the evaluation pattern 1801. The evaluation index array conversion unit 3604 generates and outputs the evaluation index array 1603 based on the complex information array 3701.

The distance information generation unit 110 generates distance information 2901 based on the evaluation index array 1603, which is the output of the evaluation index array generation unit 109.

FIG. 32 is a diagram showing an example of the state of the imaginary part absolute value and the sampling point of the complex information array 3701.

The distance information generation unit 110 calculates the distance information 2901 only from the sampling points 3201 to 3204. The values at the sampling points 3201 and 3202 are the imaginary absolute values of the complex developed image generated by the mutual correlation calculation between the development pattern 801 of β'so that f <L _N and the complex sensor image 1602, and the sampling points 3203. The values of, 3204 are imaginary absolute values of the complex developed image generated by the mutual correlation calculation between the development pattern 801 of β'and the complex sensor image 1602 such that _f > LF.

The distance information generation unit 110 calculates β'that minimizes the absolute value of the imaginary part of the complex developed image based on the values of these four points. For example, by calculating the straight line 3205 passing through the sampling points 3201 and 3202, calculating the straight line 3206 passing through the sampling points 3203 and 3204, and calculating the intersection of the straight line 3205 and the straight line 3206, the imaginary part of the complex developed image is absolute. The distance information 2901 may be generated by deriving β'that minimizes the value.

FIG. 33 is a diagram showing an example of the state of the imaginary part absolute value and the sampling point of the complex information array. As shown in FIG. 33, the number of sampling points is reduced to three, a straight line 3304 passing through the sampling points 3301 and 3302 is calculated, and then the straight line 3305 having a slope obtained by reversing the slope sign of the straight line 3304 and passing through the point 3303. May be calculated and the distance information 2901 may be generated from the intersection of the straight line 3304 and the straight line 3305.

FIG. 34 is a diagram showing an example of the state of the imaginary absolute value and the sampling point of the complex information array. As shown in FIG. 34, after the distance measurement variable setting unit 111 selects three β'that L _N < _f <LF, the distance information generation unit 110 performs sampling, which is an evaluation index of three points. Distance information 2901 may be generated by performing curve fitting on points 3401 to 3403 with a parabola 3404.

As described above, the distance measuring device 101 has lower accuracy than the distance measurement using five or more sampling points as shown in FIG. 28 by reducing the sampling points as shown in FIGS. 32, 33, and 34. However, it is possible to generate distance information at a higher speed.

In addition, after the distance information with coarse accuracy is generated by the non-search type distance measurement, the distance information is generated by performing the high accuracy search type distance information generation within the narrow distance measurement range obtained by the non-search type distance measurement. The generation speed and the improvement of the accuracy may be compatible.

<Third embodiment>
<Non-search type ranging zero intersection calculation type>
The distance measuring device according to the third embodiment of the present invention will be described. The distance measuring device according to the third embodiment is configured in the same manner as the distance measuring device 101 (FIG. 1), except that distance information is generated based on the phase angle of the evaluation index array.

In order to generate the distance information 2901 with fewer sampling points than the non-search type distance measurement, it is conceivable to use an evaluation index array that changes linearly with respect to the pitch coefficient β'as shown in FIG. 35.

FIG. 35 is a diagram showing the results of plotting the phase angle of the cross-correlation calculation result of the complex sensor image 1602 and the evaluation pattern 1801 with respect to the pitch coefficient β'of the evaluation pattern 1801.

For example, the imaginary component and the phase angle of the complex developed image obtained by the cross-correlation calculation between the complex sensor image 1602 and the developing pattern 801 change linearly with respect to the pitch coefficient β'in a certain region centered on β'= β ₀ . do.

Therefore, in the evaluation index array generation unit 109, the imaginary component or the phase angle of the complex information array 3701 obtained by the cross-correlation calculation of the evaluation pattern 1801 consisting of the complex sensor image 1602 and the development pattern 801 of two arbitrary β's. Is set as the evaluation index array 1603, and as shown in FIG. 35, a straight line 3503 passing between the two sampling points 3501 and 3502 is calculated. After that, β'where the straight line 3503 intersects zero is calculated, and the distance information 2901 is generated.

If the distance information is generated as described above, the pitch coefficient β'used for generating the evaluation index array 1603 can be reduced to two, and high-speed distance information generation becomes possible.

The distance information generation unit 110 converts the value of the evaluation index array element into a value that changes linearly with respect to the enlargement ratio α ² or the distance f instead of the pitch coefficient β', and bases the enlargement ratio α ² . The distance information 2901 may be generated to reduce the amount of calculation.

<Fourth Embodiment>
<Depth smoothing pattern>
A distance measuring device according to a fourth embodiment of the present invention will be described. The distance measuring device according to the fourth embodiment is configured in the same manner as the distance measuring device 101 (FIG. 1), but by using a smoothing pattern in which development patterns in the depth direction (subject direction) are superimposed. It differs in that it expands the range of focus positions.

FIG. 47 is a diagram showing an example of how the values of the array elements of the evaluation index array change according to the pitch of the evaluation pattern. In this way, the values of the array elements of the evaluation index array 1603 change so as to draw a predetermined function when the pitch coefficient β', that is, the focus position is changed. In the graph, the solid line shows the case where the imaginary part of the complex information array 3701 is used as the evaluation index array 1603, and the broken line shows the case where the absolute value of the imaginary part of the complex information array 3701 is used.

When searching for the minimum value, the complex information array 3701 contains a complex developed image focused at a position of f ₁ or less (closer to f ₁ ) or f ₂ or more (remote than f ₂ ). If so, the value of the array element of the evaluation index array 1603 becomes almost zero in the region of f1 or _less or f2 or _more . Therefore, if the minimum value is found in the region of f ₁ or less or f ₂ or more, erroneous distance information 2901 may be output.

Further, in the case of the above-mentioned zero intersection calculation type, if the focus positions of the two complex developed images included in the complex information array 3701 are outside the range between f _{3 and} f ₄ , it is irrelevant to the original in-focus position. Zero crosses at the position of. Therefore, there is a possibility that distance information 2901 different from the original subject distance will be generated.

In consideration of these possibilities, in the present embodiment, more accurate distance measurement is realized by expanding the range of the focus position that can be used for the distance measurement calculation.

In the fourth embodiment, as the plurality of developing patterns constituting the evaluation pattern 1801, a smoothing pattern generated by adding a plurality of patterns similar to the photographing pattern but having different pitch coefficients β'is used. This will be described with reference to FIG. 49.

FIG. 49 is a diagram showing an example of the state of smoothing pattern generation. As the actual part of the developing pattern 801 of the evaluation pattern 1801, a smoothing pattern 4905 which is similar to the photographing pattern 105 and has a plurality of patterns 4901 to 4904 having different pitch coefficients β'are added.

Further, as the imaginary part of the developing pattern 801 of the evaluation pattern 1801, a smoothing pattern 4910 which is similar to the photographing pattern 105 and is obtained by adding a plurality of patterns 4906 to 4909 having different pitch coefficients β'is used. ..

Here, an example of creating a smoothing pattern by adding four patterns for both the real part and the imaginary part is shown, but the number of patterns to be added is not limited to four, and any number of two or more can be added. good. Further, instead of adding a plurality of patterns, a smoothing pattern may be generated directly from the equation obtained by integrating the equation (3) with respect to β to improve the calculation efficiency.

The complex developed image generated by the cross-correlation calculation between each of the smoothing patterns 4905 and 4910 and the complex sensor image 1602 has a complex information array 3701 generated by using the evaluation pattern 1801 shown in FIG. It is the same as the one smoothed in the depth direction.

As a method for determining β'used for creating a smoothing pattern in the distance measurement variable setting unit 111, a plurality of methods can be considered, and all of them are effective.

FIG. 55 is a diagram showing an example of smoothing pattern generation in the depth direction. FIG. 55 shows an example of a β'selection method when the evaluation pattern is composed of N smoothing patterns. FIG. 55 shows that a sampling number (1 to N) is set on the vertical axis and a focus distance to the subject is set on the horizontal axis, and a region indicating one focus distance range is set for each sampling number. The first region 5501, the second region 5502, the third region 5503, and the fourth region 5504 correspond to the first, second, third, and Nth sampling numbers, respectively.

That is, the first region 5501, the second region 5502, the third region 5503, and the fourth region 5504 each indicate the range of the focus distance f included in the smoothing pattern in the depth direction. A plurality of patterns similar to the shooting pattern but having different pitch coefficients β'are added in each region to create a smoothing pattern. That is, each range indicates the range of the focus position.

FIG. 48 is a diagram showing an example of how the values of the array elements of the evaluation index array change according to the pitch of the evaluation pattern. By using the evaluation pattern smoothed in the depth direction as described above, the focus range that can be used for distance measurement by minimum value search is expanded to f _{1'to f 2 '} , and it is used for distance measurement by zero intersection calculation. The possible focus range can also be expanded to f _3'to f ₄ '. In addition, by using the above smoothing pattern, smoothing in the depth direction can be performed at the same time as the cross-correlation calculation. Therefore, compared to the case of smoothing in the depth direction after generating a complex developed image, the calculation processing per frame It is possible to reduce the load and time.

Here, FIGS. 50 and 51 show contour plots when the complex information array when the point light source is photographed is cut out along the plane 3702 extending in the depth direction of FIG. 37.

FIG. 50 is a diagram showing an example of a complex information array when an unsmoothed evaluation pattern is used. As shown in FIG. 50, when a single pattern similar to the photographing pattern is used as the developing pattern, the spread of the point light source according to the defocus becomes steep as shown in FIG. 50.

FIG. 51 is a diagram showing an example of a complex information array when a smoothing pattern is used. As shown in FIG. 51, when the smoothing pattern is used as a developing pattern, the spread of the point light source according to the defocus becomes slow. Therefore, by using the smoothing pattern, not only the focus range that can be used for distance measurement is expanded, but also the influence of blur propagation from other pixels in the plane can be suppressed, and the disturbance of the distance measurement result is suppressed. There is an effect that makes it possible to do.

<Weighting when creating a smoothing pattern in the depth direction>
In FIG. 49, a plurality of patterns 4901 to 4904 and a plurality of patterns 4906 to 4909 are added with the same weight when the smoothing pattern is generated. However, the present invention is not limited to this, and as shown in FIG. 52, each pattern may be multiplied by different weighting coefficients W1 (5201) to W4 (5204).

FIG. 53 is a diagram showing an example of weighting for weighted smoothing pattern generation. Each of the weighting coefficients W1 (5201) to W4 (5204) is evaluated even when the smoothing range is widened by taking a value along the Gaussian function type weight 5301 as shown in FIG. 53, for example. It is possible to prevent the values of the array elements of the index array from not increasing monotonically with respect to the focus position. Further, by adding the weighting coefficients W1 (5201) to W4 (5204) along the asymmetrical weight 5302, it is possible to suppress the position where the value of the element of the evaluation index array intersects with zero.

<Diversification of the range of responsibility for smoothing patterns in the depth direction>
As shown in FIG. 55, the method of taking the smoothing range of the smoothing pattern is that the first region 5501, the second region 5502, the third region 5503, and the fourth region 5504 are equidistant at equal intervals. It is not limited to the range.

FIG. 56 is a diagram showing an example of smoothing pattern generation in the depth direction. As shown in FIG. 56, the smoothing ranges may overlap between the regions of the smoothing pattern. That is, in FIG. 56, a developing pattern of overlapping focus positions is included between the first region 5501 and the second region 5502, and the second region 5502 and the third region 5503, respectively. However, even in that case, it can be adopted without any problem.

FIG. 57 is a diagram showing an example of smoothing pattern generation in the depth direction. As shown in FIG. 57, the smoothing range may be gradually expanded according to the remoteness of the focus position. That is, the smoothing range may be narrowed as the focus position is closer to the front. Generally, the farther the focus position is, the deeper the depth of field of the developed image is. Therefore, depending on how to take such a smoothing range, the smoothing effect of the same degree is obtained for both short-distance and long-distance objects. Can be brought about.

Further, the smoothing range may be gradually expanded according to the proximity of the focus position. This is because, in general, the closer the focus position is, the more detailed detection of the focus position is often required.

FIG. 58 is a diagram showing an example of smoothing pattern generation in the depth direction. As shown in FIG. 58, when the intervals between the smoothing patterns are evenly spaced on a logarithmic scale, the smoothing range may be selected so as to have the same length on the logarithmic scale.

FIG. 59 is a diagram showing an example of smoothing pattern generation in the depth direction. As shown in FIG. 59, when a negative focus position is included in the smoothing range, that is, when the focus position is closer to the sensor surface than the photographing pattern 105 (when the distance f-averaging range <0). , The pitch coefficient diverges infinitely, which may result in an error. Even in this case, the error can be avoided by not including the portion surrounded by the broken line (when the value is negative f) in the first region 5501 in the smoothing pattern.

FIG. 54 is a diagram showing an example of a user interface of the distance measurement software. This user interface is a screen displayed on the display of the distance measuring device 101 or the display of a computer associated therewith. The distance measuring device 101 uses this screen to determine the minimum value (Deepth Min), maximum value (Deptth Max), smoothing range (Deepth Averaging Range), and smoothing resolution (Deepth Resolution) of the distance measurement range. Accept input. The distance measurement range (distance measurement range) is the distance from the minimum value (Deepth Min) to the maximum value (Deptth Max) of the distance measurement range.

The input minimum value, maximum value, smoothing range, and smoothing resolution are passed to the evaluation index sequence generation unit 109 by the distance measurement variable setting unit 111. The evaluation index sequence generation unit 109 selects the pitch coefficient β'used for generating the development image of the evaluation pattern 1801 based on the input value. For example, when searching for the minimum value, the evaluation index sequence generation unit 109 selects the first of the N development patterns so as to correspond to the focus position below the minimum value, and the Nth sheet is the maximum value. Select an arbitrary or predetermined value so as to correspond to the above focus position. The input minimum value of the distance measurement range (Depth Min) is the median value of the smoothing range in which the focus distance is the shortest, and the maximum value of the distance measurement range (Depth Max) is the longest in the focus distance. It is the median value of the smoothing range.

Further, when performing the zero intersection calculation process, the evaluation index sequence generation unit 109 selects the first of the two development patterns so as to correspond to the focus position below the minimum value, and the second is the maximum. Select to correspond to the focus position above the value. Further, for the smoothing range (focus width in charge of each region), the evaluation index sequence generation unit 109 uses the input smoothing range (Deptth Averaging Range). Further, the evaluation index sequence generation unit 109 uses the input smoothing resolution (Depth Resolution) for the degree of overlap of the smoothing ranges, that is, the distance between the medians of the smoothing ranges.

In addition to the value of the focus range itself, the evaluation index sequence generation unit 109 accepts a value that specifies the focus range, such as a ratio to the focus position, as the smoothing range to be input, and is shown in FIGS. 57 and 58. A smoothing pattern may be created in such a smoothing range.

In this way, the smoothing is defined as smoothing in the stacking direction, and the two-dimensional arrays constituting the evaluation index array may have different smoothing ranges in the depth direction (stacking direction), or may be smoothed in the stacking direction. The closer the focus distance is, the narrower the range is, or the closer the focus distance is, the wider the range is.

As described above, according to the fourth embodiment, by using a smoothing pattern in which development patterns in the depth direction (subject direction) are superimposed, it is possible to expand the range of the focus position, which is higher. Accurate distance measurement can be realized.

<Fifth Embodiment>
<In-plane smoothing>
The distance measuring device according to the fifth embodiment of the present invention will be described. The distance measuring device according to the fifth embodiment is configured in the same manner as the distance measuring device 101 (FIG. 1), except that the complex information array is smoothed in a plane.

FIG. 60 is a diagram showing a profile of a complex developed image. FIG. 60 shows a cross-sectional plot of an imaginary part of a complex developed image developed with a development pattern in which β'> β ₀ at the time of shooting with a point light source by line 6001.

Since the imaginary part of the complex developed image vibrates as it moves away from the center part and the sign is inverted, there are pixels in the plane whose value becomes zero even when the image is out of focus (hereinafter referred to as the zero point). .. Since the pixels corresponding to these zero points may cause an error in the distance measurement result, it is desirable to suppress the influence of the zero point.

FIG. 38 shows an example of the configuration in the evaluation index array conversion unit. The evaluation index array conversion unit 3604 of the evaluation index sequence generation unit 109 includes a smoothing kernel generation unit 3801, a smoothing kernel storage unit 3802, an imaginary part absolute value conversion unit 3803, and an in-plane smoothing calculation unit 3804. include.

The complex information array 3701 input from the cross-correlation calculation unit 3603 is input to the imaginary part absolute value conversion unit 3803. The imaginary part absolute value conversion unit 3803 outputs the absolute value of the imaginary part of each complex developed image in the complex information array 3701.

This will be described with reference to the graph of FIG. When the complex developed image shown by the line 6001 in FIG. 60 is input to the imaginary part absolute value conversion unit 3803, the imaginary part absolute value conversion unit 3803 displays the imaginary part absolute value array shown in the line 6002 shown in FIG. 60. Output. There may still be a zero point at the output of line 6002. Therefore, the in-plane smoothing calculation unit 3804 performed in-plane smoothing by calculation using the smoothing kernel stored in the smoothing kernel storage unit 3802, and the zero point shown by the line 6003 in FIG. 60 was removed. The imaginary absolute value smoothing array is output as the evaluation index array 1603.

As described above, the complex information array is smoothed after the absolute value transformation in the plane, so that the zero point can be removed and the error of the distance information can be suppressed.

The smoothing kernel used for smoothing in the in-plane smoothing calculation unit 3804 is generated by the smoothing kernel generation unit 3801 according to the smoothing kernel width input from the distance measurement variable setting unit 111. It is stored in the smoothing kernel storage unit 3802, and is passed to the in-plane smoothing calculation unit 3804 in accordance with the smoothing process in the in-plane smoothing calculation unit 3804.

Further, the shape of the kernel function generated by the smoothing kernel generation unit 3801 can be determined based on the input from the distance measurement variable setting unit 111. For example, the smoothing kernel generation unit 3801 determines the smoothing kernel as shown in the example of FIG.

FIG. 61 is a diagram showing an example of a smoothing kernel. The smoothing kernel generator 3801 may determine a rectangular or cylindrical kernel function as shown in the cross section plot 6101, or may determine a cone kernel function as shown in the cross section plot 6102. Alternatively, a Gaussian function type kernel function as shown in the cross section plot 6103 may be determined.

Further, the kernel width of the smoothing kernel is also determined by the smoothing kernel generation unit 3801 based on the input from the distance measurement variable setting unit 111 as in the example of FIG. 62.

FIG. 62 is a diagram showing an example of the kernel width of the smoothed kernel. The width of the line 6001 of the complex developed image shown in FIG. 60 is generally inversely proportional to the pitch coefficient β'of the developing pattern. Therefore, a kernel-width smoothing kernel shown in FIG. 62 may be created for each of the β'of the plurality of development patterns generated by the evaluation pattern generation unit 3601.

When the smoothing pattern (smoothing pattern in the depth direction) in the fourth embodiment is used as the development pattern, the smoothing kernel generation unit 3801 is based on a typical β'in the smoothing range. Just determine the kernel width. It should be noted that the same smoothing kernel pattern may be used for smoothing the absolute values of the imaginary parts of all the complex developed images.

FIG. 66 is a flowchart illustrating an example of a distance measurement process by in-plane smoothing. In step 6601, the evaluation pattern generation unit 3601 generates the evaluation pattern 1801.

Then, in step 6602, the smoothing kernel generation unit 3801 generates a smoothing kernel and stores it in the smoothing kernel storage unit 3802. Then, in step 6603, the image sensor 103 acquires the sensor image. Using the sensor image acquired by the image sensor 103, the complex sensor image generation unit 108 generates the complex sensor image 1602 in step 6604.

In step 6605, the cross-correlation calculation unit 3603 performs a cross-correlation calculation between the complex sensor image 1602 and the evaluation pattern 1801 to generate a complex information array 3701.

In step 6606, the imaginary part absolute value conversion unit 3803 extracted the imaginary part absolute value of the complex information sequence 3701, and the in-plane smoothing calculation unit 3804 generated the imaginary part absolute value of the complex information sequence 3701 in step 6602. Smoothing is performed using a smoothing kernel to generate an evaluation index sequence 1603.

In step 6607, the distance information generation unit 110 generates and outputs distance information using the evaluation index array 1603.

The above is an example of the distance measurement process when performing in-plane smoothing using the smoothing kernel. By performing in-plane smoothing, it is possible to reduce the distance measurement error inside and outside the edge. The developing pattern constituting the evaluation pattern 1801 generated in step 6601 may be a pattern similar to the photographing pattern, or may be similar to the photographing pattern described in the fourth embodiment. It may be a smoothing pattern in which a plurality of patterns are added together.

By generating the smoothing pattern as a development pattern, the effect of expanding the range-finding range by the smoothing in the depth direction described in the fourth embodiment and the measurement by the smoothing in the in-plane direction described in the fifth embodiment are performed. Both the effect of improving the distance accuracy and the effect can be obtained.

Further, the timing of generating the smoothed kernel in step 6602 is not limited to immediately after step 6601, and may be performed at any time before the execution of step 6606.

<Sixth Embodiment Differentiation>
The distance measuring device according to the sixth embodiment of the present invention will be described. The distance measuring device according to the sixth embodiment is configured in the same manner as the distance measuring device 101 (FIG. 1), but differs in that an in-plane differential operation is performed on the complex developed image.

FIG. 63 is a diagram showing an example of a profile of a complex developed image. This profile shows the appearance of the imaginary part of the complex developed image when a general subject (landscape, etc.) is photographed. When there is a change in luminance such as step response 6301 in the subject, the imaginary part of the complex developed image at β'<β ₀ has a left-right asymmetric shape as shown in plot 6302. On the other hand, the imaginary part of the complex developed image at β'> β ₀ has an asymmetrical shape with the left and right sides inverted from the plot 6302, as shown in the plot 6303. In the case of a general subject, since the left region of the step response 6301 is not necessarily zero, the offset component 6304 is added as a floor to the imaginary portion of the complex developed image.

As a result, since the value of the complex developed image does not change with respect to β'at the edge portion (vertical broken line in the figure) of the step response 6301 as it is, the distance information can be generated by the above-mentioned minimum value search and zero intersection calculation. not. Further, on the outside (left side) of the edge, the value of the distance information becomes larger than the actual distance regardless of whether the minimum value search or the calculation of the zero intersection is used under the influence of the offset component 6304. On the other hand, on the inside (right side) of the edge, the value of the distance information becomes smaller than the actual distance regardless of whether the minimum value search or the calculation of the zero intersection is used under the influence of the offset component 6304.

To summarize the above, in the case of a general subject, it is shown that distance information cannot be generated at the edge part itself, the subject is farther than it actually is outside the edge, and the subject is closer than it actually is inside the edge. Distance information will be generated. That is, even though the subjects composed of two different colors are equidistant, different distances may be calculated depending on the color portion.

FIG. 64 is a diagram showing an example of differentiating the profile of a complex developed image. As shown in FIG. 64, even in the profile of the general subject shown in FIG. 63, the asymmetry and the offset component of the edge portion of the complex developed image can be removed by differentiating the complex developed image. Therefore, it is possible to simultaneously solve the error of the distance information generation at the edge portion and the distance information inside and outside the edge.

In order to apply such a differential process, it is conceivable that the cross-correlation calculation unit 3603 performs a cross-correlation calculation using the complex differential pattern and the complex sensor image. A method of generating such a complex differential pattern will be described with reference to FIG. 65.

FIG. 65 is a diagram showing an example of complex differential pattern generation. The complex differential pattern can be said to be an example of the development pattern 801 of the evaluation pattern 1801. The evaluation pattern generation unit 3601 uses the difference between the pattern 6501 having a pitch coefficient β'similar to the photographing pattern 105 and the pattern 6502 having the same pitch coefficient as the pattern 6501 as the actual part of the developing pattern, and is complex. Generate a differential pattern 6503. However, the pattern centers of the pattern 6501 and the pattern 6502 are displaced by at least one pixel or more in either direction.

The evaluation pattern generation unit 3601 uses the difference between the pattern 6504 with a pitch coefficient β'similar to the photographing pattern 105 and the pattern 6504 with the same pitch coefficient as the imaginary part of the development pattern, and is complex. Generate a differential pattern 6506. However, it is assumed that the pattern center of the pattern 6504 of the imaginary part coincides with the pattern center of the pattern 6501 of the real part, and the pattern center of the pattern 6505 of the imaginary part coincides with the pattern center of the pattern 6502 of the real part.

Using the complex differential pattern consisting of the complex differential patterns 6503 and 6506 generated as described above, the cross-correlation calculation unit 3603 performs a cross-correlation operation with the complex sensor image 1602, whereby the profile as shown in FIG. 64 is performed. It is possible to generate a complex developed image having. Then, by generating the evaluation index array 1603 using the differentiated complex developed image, it is possible to improve the accuracy of the distance information generated by the distance information generation unit 110, particularly the accuracy of the distance information of the edge portion. Become. By performing the cross-correlation operation using the complex differential pattern as described above, it is possible to omit the process of applying the differential filter after generating the complex developed image. That is, it is possible to reduce the processing load and time for each frame.

The distance measurement flow using the above complex differential pattern is basically the same as the flow shown in FIG. 27. The partial difference is that the evaluation pattern 1801 generated in step 2701 is a complex differential pattern. In the process of performing cross-correlation calculation using a complex differential pattern, either smoothing in the depth direction by cross-correlation calculation using a smoothing pattern or in-plane smoothing using a smoothing kernel is performed. Both may be applied in combination.

<Seventh Embodiment: Smoothing in the depth direction and differentiation>
The seventh embodiment relates to an embodiment relating to a distance measuring device that combines smoothing in the depth direction by a smoothing pattern in the fourth embodiment and calculation by a complex differential pattern in the sixth embodiment. Is. The method of generating the evaluation pattern carried out by the distance measuring device according to the seventh embodiment will be described below.

The evaluation pattern according to the seventh embodiment is generated by the evaluation pattern generation unit 3601 in step 2701 shown in FIG. 27. Specifically, as shown in FIG. 67, the evaluation pattern generation unit 3601 adds a plurality of complex differential patterns generated from patterns having different pitch coefficients β'in both the real part and the imaginary part of the evaluation pattern 1801. To generate a differential smoothing pattern.

FIG. 67 is a diagram showing an example of differential smoothing pattern generation. In the differential smoothing pattern generation, the evaluation pattern generation unit 3601 generates the differential smoothing pattern 6705 by adding the real parts 6701 to 6704 of the complex differential pattern as the real part of the development pattern of the evaluation pattern 1801. Generate. Similarly, the evaluation pattern generation unit 3601 generates a differential smoothing pattern 6710 generated by adding the imaginary parts 6706 to 6709 of the complex differential pattern as the imaginary part of the development pattern of the evaluation pattern 1801.

The pitch coefficients of the two patterns used to create the complex differential patterns 6701 to 6704 are different from each other in the complex differential patterns 6701 to 6704. Further, the complex differential pattern 6706 of the imaginary part corresponding to the complex differential pattern 6701 of the real part is generated from the pattern of the same pitch coefficient. The same applies to the complex differential patterns 6702 to 6704 and the complex differential patterns 6707 to 6709.

As described in the fourth embodiment, the differential pattern used to generate the differential smoothing pattern 6705 and the differential smoothing pattern 6710 is not limited to four, and any number of two or more are used. It doesn't matter. Further, the differential smoothing pattern may be directly generated by integrating Eq. (3) with respect to β and performing differential processing. By doing so, the amount of calculation can be suppressed.

The above is the distance measuring device according to the seventh embodiment. According to the distance measuring device according to the seventh embodiment, the range of the focus position can be expanded and the distance can be measured with high accuracy even if there is an edge.

<Eighth Embodiment: Depth smoothing, in-plane smoothing and differentiation>
The eighth embodiment includes smoothing in the depth direction in the fourth embodiment, in-plane smoothing by a smoothing filter using a smoothing cursor in the fifth embodiment, and sixth embodiment. It is an embodiment relating to a distance measuring device that combines an operation based on a complex differential pattern in a form. The method of generating the evaluation pattern carried out by the distance measuring device according to the eighth embodiment will be described below with reference to FIG. 66.

In the distance measuring method according to the eighth embodiment, basically the same processing as that of the distance measuring device according to the fifth embodiment shown in FIG. 66 is performed. However, the evaluation pattern according to the eighth embodiment is generated as a complex differential pattern or a differential smoothing pattern by the evaluation pattern generation unit 3601 in step 6601 shown in FIG. Specifically, the evaluation pattern generation unit 3601 is a complex number obtained by shifting the center of a pattern having the same pitch coefficient β'by one pixel or more in a predetermined direction in both the real part and the imaginary part of the evaluation pattern 1801. Generate a differential pattern. Alternatively, the evaluation pattern generation unit 3601 generates a differential smoothing pattern by adding a plurality of complex differential patterns generated from patterns having different pitch coefficients β'in both the real part and the imaginary part of the evaluation pattern 1801. ..

Further, as the complex differential pattern, in addition to the method shown in FIG. 65, for example, a method as shown in FIG. 69 can be adopted. FIG. 69 is a diagram showing another example of the differential pattern. The evaluation pattern generation unit 3601 may be generated so as to shift by one pixel or more in the vertical direction to obtain the difference between the two patterns, or to shift by one pixel or more in the diagonal direction to obtain the difference between the two patterns. May be generated in. Alternatively, the evaluation pattern generation unit 3601 may be generated by a convolution operation with the Laplacian kernel shown in the following equations (12) and (13).

As described above, according to the distance measuring device according to the eighth embodiment, it is possible to measure the distance quickly and accurately while suppressing the amount of calculation.

<Integration of a plurality of evaluation patterns in the ninth embodiment>
The ninth embodiment is the distance information obtained as a result of the calculation by a plurality of evaluation patterns in the rangefinder billion device according to each embodiment shown in the first embodiment to the eighth embodiment. It is an embodiment related to a distance measuring device that integrates and obtains distance information. The distance measuring method carried out by the distance measuring device according to the ninth embodiment will be described below with reference to FIG. 68.

The distance measuring method of the distance measuring device according to the ninth embodiment is basically the same as the distance measuring method of the distance measuring device according to the first embodiment, but there are some differences. Hereinafter, the differences will be mainly described.

FIG. 68 is a flowchart illustrating an example of the distance measurement process. First, the evaluation pattern generation unit 3601 generates two or more evaluation patterns (for example, two types) in step 6801, and uses one of them and the complex sensor image 1602 in steps 6803 to 6806. The distance information 1 is calculated by performing a cross-correlation calculation. Similarly, the distance information 2 is calculated by performing a cross-correlation calculation using the remaining one in steps 6807 to 6809 and the complex sensor image 1602. If there are more evaluation patterns, the distance information is calculated for the remaining evaluation patterns in the same manner.

In step 6810, the distance information 1 and the distance information 2 are integrated and output as one distance information. As the integration method, for example, a method of calculating the average value of the distance information 1 and the distance information 2 in the same pixel can be considered. When a plurality of distance information is obtained, it can be integrated by using various statistical values such as a mean value, a mode value, and a median value in the same manner.

In addition, the evaluation pattern generation unit 3601 may generate a plurality of evaluation patterns differentiated in different directions in the in-plane direction in step 6801.

The above is an example of the distance measuring device according to the ninth embodiment. According to the distance measuring device according to the ninth embodiment, more accurate distance information can be obtained.

The present invention is not limited to the above-described embodiment, and various modifications are possible. For example, each of the above-described embodiments has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations.

Further, it is possible to replace or add a part of the configuration of one embodiment to the configuration of another embodiment.

Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

In addition, the control lines and information lines indicate what is considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

101 ... Distance measuring device, 102 ... Imaging unit, 103 ... Image sensor, 103a ... Pixel, 104 ... Pattern board, 105 ... Shooting pattern, 106 ... Image processing unit , 107 ... Controller, 108 ... Complex sensor image generation unit, 109 ... Evaluation index array generation unit, 110 ... Distance information generation unit, 111 ... Distance measurement variable setting unit, 801 ... Development pattern, 1201 ... Shooting pattern, 1202 ... Modulator control unit, 1301 ... Shooting pattern, 1302 ... Image division unit, 1401 ... Point, 1402 ... Projected image, 1501 ... Point, 1502 ... Projection image, 1601 ... Sensor image, 1602 ... Complex sensor image, 1603 ... Evaluation index array, 1801 ... Evaluation pattern, 1901 ... Subject, 2901 ... Distance information, 3101 ... Distance measurement range input unit, 3601 ... Evaluation pattern generation unit, 3602 ... Evaluation pattern storage unit, 3603 ... Mutual correlation calculation unit, 3604 ... Evaluation index array conversion unit, 3701 ... complex information array, 3801 ... smoothing kernel generation unit, 3802 ... smoothing kernel storage unit, 3803 ... imaginary part absolute value conversion unit, 3804 ... surface Internal smoothing calculation unit

Claims (15)

An image sensor that converts light into an electrical signal and generates a sensor image,
A modulator that modulates the intensity of light projected onto the image sensor based on a shooting pattern, and
A complex sensor image generator that generates a complex sensor image having a complex number from the sensor image,
An evaluation index array generation unit that generates an evaluation index array based on the phase information of the complex sensor image,
It has a distance information generation unit that generates distance information using the evaluation index array, and has.
The evaluation index sequence generation unit is
Generates an evaluation pattern that includes multiple different development patterns according to the focus distance.
A plurality of developed images are generated by performing a cross-correlation operation between the complex sensor image and the plurality of the developing patterns.
By stacking the plurality of developed images in order of close focus distance, a three-dimensional complex information array having a plane and a depth direction is generated.
The complex information array is smoothed in at least one direction of the three dimensions in a predetermined range to generate a plurality of two-dimensional arrays, and the three-dimensional array obtained by laminating the two-dimensional arrays in the stacking direction is referred to as the evaluation index array. do,
A distance measuring device characterized by this. The distance measuring device according to claim 1.
The developing pattern is similar to the photographing pattern and has a similar shape.
The distance information generation unit generates the distance information using the enlargement ratio of the development pattern from the photographing pattern.
A distance measuring device characterized by this. The distance measuring device according to claim 1.
The developing pattern is generated by adding a plurality of patterns that are similar to the photographing pattern but have different enlargement ratios.
The evaluation index sequence generation unit is
By performing a cross-correlation operation with the development pattern, the complex information array is smoothed in the stacking direction.
A distance measuring device characterized by this. The distance measuring device according to claim 3.
The evaluation index sequence generation unit generates the development pattern by weighting a plurality of patterns having different enlargement ratios with different weights.
A distance measuring device characterized by this. The distance measuring device according to claim 1.
The smoothing is smoothing in the stacking direction.
The two-dimensional array constituting the evaluation index array has different smoothing ranges in the stacking direction.
A distance measuring device characterized by this. The distance measuring device according to claim 5.
The developing pattern is generated by adding a plurality of patterns that are similar to the photographing pattern but have different enlargement ratios.
The evaluation index sequence generation unit is
By performing a cross-correlation operation with the development pattern, the complex information array is smoothed in the stacking direction.
A distance measuring device characterized by this. The distance measuring device according to claim 1.
The developing pattern is similar to the photographing pattern and has a similar shape.
The evaluation index sequence generation unit is
The imaginary absolute value of the complex information array is smoothed in the in-plane direction of the plane.
A distance measuring device characterized by this. The distance measuring device according to claim 1.
The developing pattern is similar to the photographing pattern and has a similar shape.
The evaluation index sequence generation unit is
The absolute value of the imaginary part of the complex information array is smoothed in the in-plane direction of the plane.
The kernel width of the smoothing kernel used for the smoothing is specified by using the enlargement ratio of the development pattern from the shooting pattern.
A distance measuring device characterized by this. The distance measuring device according to claim 1.
The evaluation index sequence generation unit is
The evaluation index array is generated using the result of performing a differential operation on the complex information array in at least one direction in the plane.
A distance measuring device characterized by this. The distance measuring device according to claim 1.
The evaluation index sequence generation unit is
The evaluation index array is generated by using the result of performing a differential operation on the complex information array in at least one direction in the plane.
A difference between at least two patterns that are similar to the photographing pattern and have a positional deviation that is translated by at least one pixel or more from each other is generated as the developing pattern.
The differential operation is performed by performing a cross-correlation operation with the complex sensor image using the development pattern.
A distance measuring device characterized by this. The distance measuring device according to claim 1.
The evaluation index sequence generation unit is
A plurality of the evaluation index arrays are generated by generating a plurality of the evaluation patterns that differentiate in different directions in the in-plane direction.
The distance information generation unit is
Using the plurality of evaluation index sequences, the plurality of distance information is generated, and the plurality of distance information is generated.
The distance information is generated by integrating the plurality of distance information.
A distance measuring device characterized by this. The distance measuring device according to claim 1.
Each array element of the evaluation index array is a set of numerical values that can take positive and negative values.
The developing pattern is a similar figure obtained by enlarging the photographing pattern at a predetermined magnification.
The distance information generation unit specifies the enlargement ratio at which each of the array elements becomes zero, and generates the distance information according to the enlargement ratio.
A distance measuring device characterized by this. The distance measuring device according to claim 1.
The developing pattern is a similar figure obtained by enlarging the photographing pattern at a predetermined magnification.
Each sequence element of the evaluation index sequence changes in proportion to the enlargement ratio.
A distance measuring device characterized by this. The distance measuring device according to claim 1.
The developing pattern is a similar figure obtained by enlarging the photographing pattern at a predetermined magnification.
It has a distance measurement variable setting unit that sets the distance measurement range and the distance measurement resolution according to the distance to the distance measurement target.
The evaluation index sequence generation unit generates the evaluation pattern by selecting the enlargement ratio of the development pattern from the photographing pattern according to the distance measurement range and the distance measurement resolution.
A distance measuring device characterized by this. A sensor image generation step that converts light into an electrical signal and generates a sensor image,
A modulation step that modulates the intensity of the light projected onto the image sensor based on the imaging pattern.
A complex sensor image generation step of generating a complex sensor image having a complex number from the sensor image,
An evaluation index array step that generates an evaluation index array based on the phase information of the complex sensor image, and
It has a distance information generation step for generating distance information using the evaluation index array.
In the evaluation index sequence generation step, an evaluation pattern including a plurality of different development patterns according to the focus distance is generated.
A plurality of developed images are generated by performing a cross-correlation operation between the complex sensor image and the plurality of the developing patterns.
By stacking the plurality of developed images in order of close focus distance, a three-dimensional complex information array having a plane and a depth direction is generated.
The complex information array is smoothed in at least one direction of the three dimensions in a predetermined range to generate a plurality of two-dimensional arrays, and the three-dimensional array obtained by laminating the two-dimensional arrays in the stacking direction is referred to as the evaluation index array. do,
A distance measurement method characterized by this.

Images