KR20220161014A - Apparatus and method for detph estimation using structured light and holographic camera - Google Patents

Abstract

An apparatus and a method for measuring depth using structured light and a holographic camera are disclosed. The depth measuring apparatus according to an embodiment of the present invention includes: a structured light output part configured to output structured light of a set pattern to an object; and a holographic camera that creates a holographic image for the structured light reflected from the object, obtains phase information based on an interference fringe generated according to a pattern of the structured light, and obtains depth information on the surface of the object based on the phase information, thereby capable of precisely measuring the depth of a microscopic surface of an object by using the structured light and the holographic camera.

Description

Apparatus and method for measuring depth using structured light and holographic camera

The present invention relates to a depth measuring device and method for measuring the surface depth (curvature, step, etc.) of an object using structured light and a holographic camera.

Methods for acquiring depth information of an object include a stereoscopic camera method, a structured light method, a time-of-flight (ToF) camera method, and the like.

The stereoscopic camera method measures depth through trigonometry by comparing the parallax of images obtained through two cameras using binocular parallax. This has the advantage of being able to simply extract depth information, but has the disadvantage of low accuracy because it uses multiple parallax.

The ToF camera method has the advantage of obtaining accurate depth information at a short distance because it emits infrared rays and measures the time it takes for them to reflect and return, but it is difficult to manufacture the sensor and has limitations in the shooting range or area.

The structured light method irradiates a light source with a special pattern to an object and estimates the depth information by calculating the change in the structured light pattern according to the unevenness of the object. The downside is that estimation is difficult. In addition, there is a limitation in that interference light such as laser light must be used as structured light to measure the surface depth of an object.

The present invention provides a depth measuring device and method capable of precisely measuring the microscopic surface depth of an object using structured light and a holographic camera and obtaining surface depth information of an object using non-coherent light as structured light. is to provide

A depth measuring device according to an embodiment of the present invention includes: a structured light output unit configured to output structured light of a set pattern to an object; and generating a holographic image of the structured light reflected from the object, acquiring phase information based on an interference fringe generated according to a pattern of the structured light, and obtaining depth information about the surface of the object based on the phase information. It includes; a holographic camera configured to acquire.

The holographic camera includes: a polarizer for polarizing structured light reflected from the object; a geometric phase lens separating a wavefront of the structured light passing through the polarizer and modulating a phase; and a polarization image sensor for measuring a hologram pattern from the structured light passing through the geometric phase lens.

The holographic camera may further include a depth detector configured to detect a surface depth of the object through auto-focus alignment of the holographic pattern.

The depth detection unit: evaluates sharpness of the hologram according to various numerical reconstruction depths through numerical reconstruction to find a focused propagation length; It may be configured to measure the surface depth of the object by finding a numerical reconstruction depth having the highest sharpness of the hologram.

The structured light may include incoherent light having a binary stripe pattern having a constant period.

A depth measuring method according to an embodiment of the present invention includes: generating a hologram pattern with respect to structured light reflected from an object by a holographic camera; and obtaining, by the holographic camera, phase information based on the interference fringe of the hologram pattern, and obtaining depth information about the surface of the object based on the phase information.

The generating of the hologram pattern may include: polarizing the structured light reflected from the object by a polarizer; separating a wavefront of the structured light passing through the polarizer and modulating a phase thereof by a geometric phase lens; and measuring a hologram pattern from the structured light passing through the geometric phase lens by a polarization image sensor.

Obtaining the depth information may include: detecting a surface depth of the object through auto-focus alignment of the hologram pattern.

The detecting of the surface depth may include: evaluating sharpness of the hologram according to various numerical reconstruction depths through numerical reconstruction to find a focused propagation length; and measuring the surface depth of the object by finding a numerical reconstruction depth having the highest sharpness of the hologram.

According to an embodiment of the present invention, a computer program recorded on a computer-readable recording medium to execute the depth measurement method is provided.

According to an embodiment of the present invention, by using the characteristics of holography for measuring interference fringes, it is possible to precisely measure the depth of a life-size target and acquire amplified depth information, thereby improving accuracy and resolution. Surface depth information of an object can be obtained by using non-coherent light as structured light.

1 is a conceptual diagram of a depth measuring device according to an embodiment of the present invention.
2 is a side view schematically illustrating a depth measuring device according to an embodiment of the present invention.
3 is a flowchart of a depth measurement method according to an embodiment of the present invention.
4 is a flowchart illustrating step S120 of FIG. 3 in more detail.
5 is a flowchart illustrating step S130 of FIG. 3 in more detail.
6 is an experimental photograph for confirming a focus extraction result according to a position of a structured light hologram according to an embodiment of the present invention.
7 is a diagram showing a focus extraction result according to a position of a structured light hologram according to an embodiment of the present invention.
8 shows a result of restoring a single binary stripe pattern and estimating the depth of a pattern part using the Tenengrad algorithm.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs. It is provided to completely inform the person who has the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numbers designate like elements throughout the specification.

The present invention uses structured light and a holographic camera to obtain phase information through interference fringes of structured light patterns as well as distortion according to the distance of structured light patterns, and analyze them to provide high-resolution and accurate micro-depth information. A depth measuring device and method are presented. 1 is a conceptual diagram of a depth measuring device according to an embodiment of the present invention. 2 is a side view schematically illustrating a depth measuring device according to an embodiment of the present invention. Referring to FIGS. 1 and 2 , a depth measuring device 100 according to an embodiment of the present invention includes a structured light output unit 110 and a holographic camera 120 .

The structured light output unit 110 is configured to output structured light 112 of a set pattern to the object 10 . The structured light 112 may be configured as a light source having a special pattern. For example, it may be set to have a binary stripe pattern in which parallel white and black straight lines are arranged at regular intervals. In an embodiment of the present invention, since distortion and focus of structured light are measured using interference fringe phase information of a hologram pattern, it is possible to obtain fine surface depth information of an object even when a general light source other than laser light is used as structured light.

The holographic camera 120 generates a holographic image (hologram pattern) for the structured light 114 reflected from the object 10, and obtains phase information based on the interference pattern of the hologram pattern generated according to the pattern of the structured light. depth information (surface step information) on the surface 12 of the object 10 may be measured by acquiring and analyzing the obtained phase information. The holographic camera 120 may include an optical system capable of obtaining amplified depth information compared to actual depth information.

The holographic camera 120 may include a condensing lens 122 , a polarizer 124 , a geometric phase lens 126 , a polarization image sensor 128 , and a depth detector 130 . The condensing lens 122 condenses the structured light 114 reflected from the object. The condensing lens 122 may be composed of a convex lens. A polarizer 124 polarizes the structured light 114 reflected from the object.

A geometric phase lens 126 separates the wavefront of the structured light 114 passing through the polarizer 124 and modulates the phase. The geometric phase lens 126 may be implemented as a passive liquid crystal device that simultaneously separates and modulates a wavefront. In a geometric phase lens, a phase change occurs due to a change in the polarization state of light according to the birefringence characteristics of liquid crystal, and the wavefront of the incident light is modulated accordingly. The geometric phase lens is manufactured using a hologram imaging technique, and a twin-image of the lens surface to be recorded is recorded together to show lens characteristics having both negative and positive focal lengths. A polarized image sensor 128 measures a hologram pattern from structured light 114 passing through a geometric phase lens 126 .

If such an optical system is used, phase information can be acquired based on natural light and hologram patterns can be measured even in an incoherent light source, so that it can be used in an environment where a general projector is applied as the structured light output unit 110. In addition, by combining a structured light system that irradiates a special pattern, it is possible to estimate the surface depth of an object by evaluating the sharpness of the hologram pattern according to the restoration length through hologram measurement and numerical reconstruction of the structured light pattern reflected on the surface of the object.

The depth detector 130 may detect the surface depth of the object 10 by analyzing phase information of the structured light pattern through autofocusing of the hologram pattern. The depth detector 130 evaluates the sharpness of the hologram according to various numerical reconstruction depths through numerical reconstruction to find the focused propagation length, and determines the numerical reconstruction depth with the highest hologram sharpness on the surface of the object 10. depth can be measured. The depth detection unit 130 finds a restoration position with the highest sharpness of the hologram through numerical restoration of the hologram pattern using techniques such as angular spectrum method (ASM) and Fresnel propagation. You can measure the (x, y, z) position of an object.

For example, in the case of using the structured light of the binary stripe pattern, the structured light of the binary stripe pattern is irradiated to the surface 12 of the object 10, and the white pattern of the structured light is reflected on the object to measure the observed hologram. Depth information can be measured through the focused propagation length by analyzing the sharpness according to the numerical reconstruction of the linear pattern. At this time, the total number of exposures is determined according to the pattern period of the structured light, and the precision of the structured light pattern is a factor determining the periodicity of the pattern.

As a result of the auto-focus alignment algorithm for the hologram pattern, the maximum and minimum values of the auto-focus alignment matrix according to the distance are determined. As a method for finding such an autofocus alignment matrix, a method through the overall distribution of gray levels of the image, a method using correlation, a method using differentiation, and a method using cosine similarity are used. The methods used can be applied.

In the case of the depth measuring device according to the embodiment of the present invention, since a laser light source is not used, there is no speckle noise, and instead, Gaussian noise is generated by aberrations and optical characteristics of the geometric lens and external penetrating light. ) may occur, it is effective to use an autofocusing algorithm that is strong against noise.

For this purpose, the use of the Tenengrad algorithm is preferred. The Tenengrad algorithm is a variation of the primary differentiation technique using the local gradient of an image, and is a method that precedes smoothing for noise reduction before obtaining the local gradient. By applying the Tenengrad algorithm, different Sobel operators can be used according to the shape of binary stripes of structured light. The focus calculation process of the Tenengrad method using the horizontal Sobel operator (O _{Sobel_horizontal} ), the vertical Sobel operator (O _{Sobel_vertical} ), and the horizontal/vertical Sobel operator (O _Sobel ) is as follows.

(M, N: 홀로그램의 픽셀 개수)

(M, N: number of pixels of hologram)

Since the Sobel operator checks the gradient in only one of the vertical and horizontal directions, the focus can be calculated using the same Sobel operator as the pattern direction of the structured light. At this time, if only a focus matrix having a specific threshold value or higher is used, a problem in which sharpness is detected in a part without an object can be prevented. I _f (x, y) means the intensity of the hologram restored at position f by numerical reconstruction, which can be obtained from the following equation through the angular spectrum method.

Here, E ₀ (x ₀ , y ₀ ) is the optical field of the object when z = 0, F is the Fourier angular spectral transform function, λ is the wavelength of structured light, and f _x /f _y are each The x-axis/y-axis Fourier frequency, the y-axis Fourier frequency, and z are the distance from the object. In this case, one holographic image may be divided into blocks of several sub-images having smaller sizes. Using multiple structured light patterns, depth information (F ₁ , F ₂ , F ₃ , F ₄ ) according to each pattern can be obtained, and the depth information (Fsub _m,n ) of each sub-image is summed to obtain the following Depth information (F _k ) according to each pattern can be obtained by the same formula.

3 is a flowchart of a depth measurement method according to an embodiment of the present invention. Referring to FIGS. 1 to 3 , the structured light output unit 110 may output structured light 112 of a set pattern to the object 10 (S110). The structured light 112 may be a light source having a special pattern, for example, a light source having a binary stripe pattern. The holographic camera 120 may generate a holographic image (hologram pattern) for the structured light 114 reflected from the object 10 (S120).

4 is a flowchart illustrating step S120 of FIG. 3 in more detail. 1 to 4, the structured light 114 reflected from the object can be collected by the condensing lens 122 and then the structured light 114 reflected from the object can be polarized by the polarizer 124 ( S122). The geometric phase lens 126 may separate the wavefront of the structured light 114 passing through the polarizer 124 and modulate the phase (S124). The polarization image sensor 128 may measure the hologram pattern from the structured light 114 passing through the geometric phase lens 126 (S126).

Through this process, it is possible to obtain phase information based on natural light and to measure a hologram pattern even in an incoherent light source, so that it can be used in an environment where a general projector is applied as the structured light output unit 110. In addition, by combining a structured light system that irradiates a special pattern, it is possible to estimate the surface depth of an object by evaluating the sharpness of the hologram pattern according to the restoration length through hologram measurement and numerical reconstruction of the structured light pattern reflected on the surface of the object.

The holographic camera 120 obtains phase information based on the interference fringe of the hologram pattern generated according to the pattern of structured light, and analyzes the obtained phase information to measure depth information about the surface 12 of the object 10. It can (S130). The depth detector 130 may detect the surface depth of the object 10 by analyzing phase information of the structured light pattern through autofocusing of the hologram pattern.

는 아래의 수식에 따라 산출될 수 있다.5 is a flowchart illustrating step S130 of FIG. 3 in more detail. Referring to FIGS. 1 to 3 and 5 , the depth detector 130 evaluates the sharpness of the hologram according to various numerical reconstruction depths through numerical reconstruction to find a focused propagation length (S132), and then evaluates the hologram sharpness. The surface depth of the object 10 can be measured for the numerical restoration depth with the highest sharpness (S134). The depth detection unit 130 finds a restoration position with the highest sharpness of the hologram through numerical reconstruction of the hologram pattern using techniques such as angular spectrum method (ASM) and Fresnel propagation. You can measure the (x, y, z) position of an object. In one embodiment, depth reconstruction coordinates for the surface of the object

can be calculated according to the formula below.

In the above formula, x ₁ , y ₁ are coordinates obtained by the polarization image sensor before restoration, d ₁ , z ₁ are coefficients according to the depth (distance) between the geometric phase lens and the object before restoration, d ₂ , A, B is the wavelength of the structured light and the coefficient associated with the holographic camera, λ is the wavelength of the structured light (center wavelength), and f _gp is the focal length of the geometric phase lens. For example, in the case of using the structured light of the binary stripe pattern, the structured light of the binary stripe pattern is irradiated to the surface 12 of the object 10, and the white pattern of the structured light is reflected on the object to measure the observed hologram. Depth information can be measured through the focused propagation length by analyzing the sharpness according to the numerical reconstruction of the linear pattern. As a method for finding an auto-focus alignment matrix, a method through the overall distribution of gray levels of an image, a method using correlation, a method using differentiation, and a method using cosine similarity are utilized. One method and the like can be applied.

6 is an experimental photograph for confirming a focus extraction result according to a position of a structured light hologram according to an embodiment of the present invention. 7 is a diagram showing a focus extraction result according to a position of a structured light hologram according to an embodiment of the present invention. The focus matrix of the hologram obtained through the single line image was analyzed using the sobel operator. The focus matrix as shown in FIG. 7 (b) was derived by numerical reconstruction according to the depth shown in FIG. It can be confirmed that a position without an object does not have a focus matrix value higher than a certain level.

An experiment was conducted to verify the performance of the depth measuring device and method according to an embodiment of the present invention. A structured light pattern was irradiated onto the object using a laser projector with an all-in-focus function. At this time, the factor affecting the measurement is the relative ratio of the white pattern and the black pattern. Since the holographic camera reads the reflection of the white pattern on the surface of an object, it is important to have the width of the white pattern so that the interference pattern has sufficient brightness to be recorded. In addition, the ratio between the white pattern and the black pattern, which is determined by the width of the black pattern, serves to ensure that each pattern can be observed separately without interference according to the width of the white pattern.

The ratio of the white pattern to the black pattern used in the experiment was 1:4, and the surface was read through four stages of exposure to obtain structured light patterns for all areas. In this experiment, a phase shift technique was used to detect the phase of the modulated wavefront through polarization change in order to remove the light source and pair image information. After measuring the light intensity information at each step, it is possible to obtain a complex hologram with the light source and pair image information removed by calculating it.

The focal length of the geometric phase lens used in the experiment is 371 mm based on the wavelength of 532 nm, and the distance between the geometric phase lens and the polarization image sensor is about 7.5 mm. The distance between the object and the geometric phase lens was about 100 mm, and a 50 mm convex lens was used as a preceding optical system for ease of recording. A color polarization image sensor from Lucid Vision with a resolution of 2448 × 2048 was used as the polarization image sensor used to acquire the hologram, and a green wavelength of 532 nm was used for numerical reconstruction of the hologram.

8 shows reconstruction results (20a, 20b, 20c, 20d) for a single binary stripe pattern and results (30a, 30b, 30c, 30d) of estimating the depth of a pattern portion using the Tenengrad algorithm. It can be confirmed that the curvature of the model can be measured through a depth map 40 obtained by merging the depth results obtained from each binary stripe pattern into one. The depth search range searched in the experiment measured sharpness while restoring the range between 330 mm and 380 mm, and the area showing meaningful depth values was measured in the range between 330 mm and 350 mm.

As described above, according to the depth measurement apparatus and method according to an embodiment of the present invention, it is possible to precisely measure the depth of a life-size object by using the characteristics of holography for measuring interference fringes, and to obtain amplified depth information. can In addition, according to an embodiment of the present invention, accuracy and resolution compared to depth information acquired through a conventional depth information obtaining system can be improved. In addition, when light of a specific wavelength is used as a structured light source, it can also be used for purposes such as measuring the depth of an object that responds only to a specific wavelength.

Depth measuring device and method according to an embodiment of the present invention can freely use the equipment from the outside through a simple configuration compared to conventional surface inspection equipment, precision surface measurement work in the industrial field, product production inspection equipment, user point of view tracking and face recognition It can be used in various fields such as In particular, it can be applied to inspection equipment such as production of large products or devices requiring high precision that cannot be inspected using general inspection equipment.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved. Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

10: object
12: surface
100: depth measuring device
110: structured light output unit
112, 114: structured light
120: holographic camera
122 condensing lens
124: polarizer
126 geometric phase lens
128: polarization image sensor
130: depth detector

Claims (11)

a structured light output unit configured to output structured light of a set pattern to an object; and
A holographic image is generated for the structured light reflected from the object, phase information is obtained based on an interference fringe generated according to a pattern of the structured light, and depth information on the surface of the object is obtained based on the phase information. Depth measuring device comprising a; holographic camera configured to acquire. According to claim 1,
The holographic camera:
a polarizer polarizing the structured light reflected from the object;
a geometric phase lens separating a wavefront of the structured light passing through the polarizer and modulating a phase; and
A depth measuring device including a polarization image sensor for measuring a hologram pattern from the structured light passing through the geometric phase lens. According to claim 2,
The holographic camera:
The depth measuring device further comprising a; depth detection unit configured to detect the surface depth of the object through the auto-focus alignment of the hologram pattern. According to claim 3,
The depth detector:
Evaluate the sharpness of the hologram according to various numerical reconstruction depths through numerical reconstruction to find a focused propagation length;
A depth measuring device configured to measure the surface depth of the object by finding a numerical reconstruction depth having the highest sharpness of the hologram. According to any one of claims 1 to 4,
The structured light includes non-coherent light having a binary stripe pattern having a constant period. generating a hologram pattern for structured light reflected from an object by a holographic camera; and
and obtaining, by the holographic camera, phase information based on the interference fringe of the hologram pattern, and obtaining depth information about the surface of the object based on the phase information. According to claim 6,
The step of generating the hologram pattern is:
polarizing the structured light reflected from the object by a polarizer;
separating a wavefront of the structured light passing through the polarizer and modulating a phase thereof by a geometric phase lens; and
and measuring a hologram pattern from the structured light passing through the geometric phase lens by a polarization image sensor. According to claim 7,
The step of obtaining the depth information is:
and detecting a surface depth of the object through auto-focus alignment of the hologram pattern. According to claim 8,
Detecting the surface depth comprises:
Evaluating sharpness of a hologram according to various numerical reconstruction depths through numerical reconstruction to find a focused propagation length; and
and measuring a surface depth of the object by finding a numerical reconstruction depth having the highest sharpness of the hologram. According to claim 6,
The depth measurement method of claim 1 , wherein the structured light is set to have a stripe pattern having a constant period. A computer program recorded on a computer-readable recording medium to execute the depth measurement method of claim 6.

Images