KR100588296B1 - Method and system of structural light based 3d depth imaging using signal separation coding and error correction thereof - Google Patents

Abstract

A method and system for measuring a three-dimensional distance image, the method comprising: irradiating a light beam from a projection means to an object to be measured and photographing the light beam from an image receiving means to measure a three-dimensional distance as an image. Encoding a signal by assigning a unique sender address to a corresponding signal, transmitting a signal by projecting a plurality of light patterns through a projection means, receiving the encoded signal at an image receiving means, received And dividing the signal to recover the address, and determining the pixel position of the target object using the sender address and the restored address.

By using the above-described three-dimensional distance image measuring method and system, even if the signal received by the image receiving means overlaps, even if the deformation due to the geometry of the object can be accurately separated, even in the presence of noise in the surrounding environment You can get a strong distance image.

Structured light, 3D distance image, signal separation coding, orthogonal code, hierarchical orthogonal code

Description

Method and System of Structural Light Based 3D Depth Imaging Using Signal Separation Coding and Error Correction Thereof}

1 is a conceptual diagram illustrating a three-dimensional measuring method using structured light;

2 is a conceptual diagram illustrating the mixing of signals according to geometric relations of an object;

3 is a flowchart illustrating a three-dimensional measurement method to which a signal separation method according to the present invention is applied;

4 is a conceptual diagram illustrating a hierarchical orthogonal code as an example of code that may be used for signal separation coding according to the present invention;

5 is a conceptual diagram illustrating a process of separating and applying a hierarchical orthogonal code in signal separation coding according to the present invention;

6 and 7 are diagrams illustrating signal separation maps obtained as a result of signal separation on a specific epipolar line and generation of a dual photography image using the same;

8 and 9 show dual photography with signal separation coding.

* Description of Symbols for Major Symbols in Drawings *

100: target object 200: image plane of the projection means

300: image plane of the image receiving means 320: epipolar line

The present invention relates to a method and system for 3D depth imaging, and in particular, a new technique called signal separation coding can be applied to accurately reconstruct a deformed signal, thereby enabling more accurate 3D distance image measurement. An image measuring method and system are provided.

In general, a 3D depth imaging method using structural light has recently attracted attention because it is suitable for sensing a 3D environment in service robotics. The basic principle of distance image measurement using structured light, which is an active stereo technique, is to irradiate an object with a projection means such as a projector, and then shoot an object irradiated with the image by means of an image receiving means such as a camera. By observing how much the light is distorted, the distance is calculated by calculating the distance to the object.

1 is a schematic diagram for explaining the principle of a structured light-based three-dimensional distance image measurement system. As shown in FIG. 1, that in order to measure the three-dimensional position of a point x in the object (100), connected to a point p on the image side of the origin O _p and the projection means of projecting means (retinal plane) (200) A straight line and an intersection point where a straight line connecting the origin O _c of the image receiving means and the point q on the image surface 300 of the image receiving means meet. Therefore, if the projector and the camera are calibrated, a distance image can be obtained by calculating coordinates of the x point as a pair of address values in each image plane of the point p and the point q . That is, the core of the method for measuring the distance image in such a stereo technique is to determine the pixel correspondence point in the received image and the projected image. Once the corresponding point is determined, the distance can be easily calculated by simple geometry.

For the accuracy of the distance measurement, the light pattern projected by the projection means is such that the spatial and / or temporal address of the signal detected by the image receiving means allows the pixel to uniquely determine the pixel correspondence point of the corresponding projection means. Errors are coded spatially and / or temporally in time sequence on a pixel array.

As the prior art of such a coding method, four types of direct coding, spatial coding, temporal coding, and hybrid coding method can be cited.

The direct coding method directly uses gray and / or color levels in coding and calculates a disparity image through one pattern frame. This method has the advantage of being fast, but it is less accurate and less robust against changes in ambient lighting or noise.

The spatial coding method uses a specially designed and coded pattern, arranged on a pixel array. De Brujin sequences and pseudorandom codes are used. This coded pattern is used to provide address information for the pixels from adjacent pixels. Spatial coding obtains a parallax image from a pattern of one or two frames. This method has high speed and has improved robustness in error correction because of using address information, but is vulnerable to signal deformation or complex objects because it uses spatially arranged address information for pixel correspondence.

Temporal coding uses coding patterns arranged along the time axis. Binary code, N code, and gray code according to line shifting have been proposed. In general, temporal coding has higher accuracy than spatial coding because coding can be coded using black and white signals without being constrained in order. However, because they use a sequence of frames, they are not suitable for fast changing scenes.

Hybrid coding uses a mixture of temporal coding and spatial coding. This coding method can obtain a robust distance image, but it cannot be used for a very complicated environment because it has the disadvantages of spatial coding. Spatial coding and hybrid coding are appropriate methods for objects with continuous surfaces, and transposition of address sequences can occur at discrete surfaces.

This conventional coding method focuses on designing spatial and / or temporal code in terms of computing a single pixel based on address information. However, the conventional coding method has a fundamental limitation in calculating a higher accuracy pixel correspondence point required for accurate distance image measurement. In particular, it ignores the fact that the address information is displaced in the case of complex boundary surfaces such as occluding and shadowing boundaries, or pixel correspondence, and thus, accuracy of distance image measurement for a complex object is very low. This is a result of the conventional method focusing only on coding the received signal based on the address information, so that the degree of distortion of the signal that makes the address information inaccurate is not taken into account in the image receiving means. Furthermore, even when using a long sequence of time codes that provide each pixel on the projection means side with a unique address, it is not effective in dealing with the deformation of the signal.

In addition, the signal received at the image receiving means is mixed with the neighboring and even distant pixels of the projection means, as well as system / environmental noise, such as scattering of light, changes in reflectance, and changes in ambient lighting. Consideration should be given to the mixing of. The conventional method overlooks this point.

Accordingly, the present inventors have come up with a new coding method called "signal separation technique" to solve the above problems.

An object of the present invention is to solve the problems of the prior art as described above, even if a plurality of signals received by the image receiving means is applied to the signal separation coding method that can accurately separate the original signal To provide a method and system for measuring 3D distance image based on structured light.

Another object of the present invention is to provide a signal separation coding method and system for determining a correspondence point capable of accurately separating a signal even when the signal is distorted by a complicated boundary of an object in 3D distance image measurement.

It is still another object of the present invention to provide a signal separation coding method and system for determining a corresponding point in structured light-based three-dimensional distance image measurement having robustness to surrounding systems and environmental noise.

Another object of the present invention is to provide a technique that can easily achieve the interpretation of the physical transformation between the artificial light source and the object and the camera.

In order to achieve the above object, the structured light-based three-dimensional distance image measuring method according to the present invention is a method of measuring a three-dimensional distance as an image by irradiating a light beam from the projection means to the object to be measured, and by imaging the light beam from the image receiving means WHEREIN: Encoding a signal by assigning a unique sender address to a signal corresponding to each pixel of said projection means, projecting a plurality of light patterns through said projection means, and transmitting a signal, said image receiving means Receiving the encoded signal, restoring an address by separating the received signal, and determining a pixel position of a target object by using the sender address and the restored address. do.

In the structured light-based three-dimensional distance image measuring method according to the present invention, the separating of the received signal may include calculating candidate values of possible addresses corresponding to the sender address and among the candidate values. An error correction step comprising determining an address according to a rule for obtaining an accurate address, wherein the rule includes a rule in which a value of a restored address gradually increases or decreases when a signal is obtained from an inclined plane of an object, It is a group consisting of a rule that the value of the restored address is not continuous when a signal is obtained, and that the value of the restored address is not continuous and disappears when a signal disappears from the shadow of an object and a part disappears or is displaced. The error correction step may be selected from the rule group.

In the structured light-based three-dimensional distance image measuring method according to the present invention, the separating of the received signal may include calculating candidate values of possible addresses corresponding to a sender address and performing self-selection among the candidate values. Defining an evaluation function by simultaneously considering candidates of address values of a location and a candidate set of address values at a neighboring location, and then determining an address using a search method that maximizes or minimizes the evaluation function. Characterized in that it comprises a.

In the structured light-based three-dimensional distance image measuring method according to the present invention, the separating of the received signal comprises calculating candidate values of possible addresses corresponding to a sender address and the candidate values. In order to determine an address within a plurality of candidate sets, an error is formed by first calculating an address value of a location having a high reliability, and then fixing the value of the location and then determining the candidate while gradually determining a value of another area. And a correction step.

In addition, in the structured light-based three-dimensional distance image measuring method according to the present invention, the light pattern is a binary light pattern, a color light pattern, a light pattern having a gray scale brightness value, the ultraviolet or infrared region of the non-visible region It is an optical pattern or a mixture pattern of these, It is characterized by the above-mentioned.

In the structured light-based three-dimensional distance image measuring method according to the present invention, in the encoding of the signal, signals that are orthogonal to each other between adjacent signals of the projection means are used.

In the structured light-based three-dimensional distance image measuring method according to the present invention, the orthogonal signals are divided into one or more hierarchical signals, and using codes orthogonal to each other in each layer. It features.

In the structured light-based three-dimensional distance image measuring method according to the present invention, the orthogonal signals divide the entire signal into one or more hierarchical signals, and are not orthogonal to the codes orthogonal to each other in each layer. It is characterized by using a mixture of cords.

In the structured light-based three-dimensional distance image measuring method according to the present invention, in the step of encoding the signal, a pseudo orthogonal signal having a property orthogonal to each other between adjacent signals of the projection means is used.

In the structured light-based three-dimensional distance image measuring method according to the present invention, in the step of encoding the signal, signals that are statistically independent from each other between adjacent signals of the projection means are used.

In order to achieve the above object, a three-dimensional distance image measuring method according to the present invention is a method for measuring a three-dimensional distance image in a system consisting of a projector and a camera, radiating a pattern from the projector to an object, the pattern Receiving this by the camera, calculating the pixel correspondence relationship between the projector and the camera through signal separation on each epipolar line, and after executing the calculating step, generating a new image at the projector viewpoint Characterized in that.

In addition, in order to achieve the above object, the stereo image restoration method according to the present invention is a method for restoring an image in a system consisting of a projector and a camera, comprising: radiating a pattern from the projector to an object and receiving the pattern into the camera; Calculating a pixel correspondence relationship between the projector and the camera through signal separation on each epipolar line, and after executing the calculating, generating a new image at the projector's point of view. The stereo image may be reconstructed by using an image and an image newly generated by the projector.

In order to achieve the above object, the measurement distance restoration method according to the present invention is a method for restoring the measurement distance in a system consisting of a projector and a camera, and assigns a unique sender address to a signal corresponding to each pixel of the camera. Transmitting a signal by projecting a plurality of light patterns through the projector, receiving the encoded signal from the camera, recovering an address by separating the received signal, 3. A method of measuring a 3D distance image based on a structured light, comprising: determining a pixel position of a target object using the restored address; and radiating a pattern from the projector to an object, receiving the pattern from the camera. Step, connect between the projector and the camera through signal separation on each epipolar line The measurement distance may be restored by simultaneously using a reconstruction method for reconstructing a stereo image including calculating a small correspondence relationship and after executing the calculating, generating a new image from the viewpoint of the projector. .

In order to achieve the above object, the structured light-based three-dimensional distance image measuring system according to the present invention includes projection means for irradiating light rays to an object to be measured, image receiving means for photographing light rays irradiated from the projection means, and the measurement object. Processing means for measuring a three-dimensional distance of an object as an image, the processing means assigning a unique sender address to a signal corresponding to each pixel of the object to be measured; Projecting a pattern to encode a signal including a sender address, receiving the encoded signal at the image receiving means, separating the received signal to recover an address, and recovering the address and the sender address Determining the pixel position of the target object by using the received address.

The above and other objects and novel features of the present invention will become more apparent from the description of the specification and the accompanying drawings.

First, the concept of the present invention will be described.

In the three-dimensional distance image measuring method according to the present invention, it is important to determine the exact correspondence between the pixel position in the projection means such as the projector and the pixel position in the image receiving means such as the camera. Once the corresponding points are determined, the three-dimensional data of the object can be easily calculated by simple geometry. Since the corresponding point in the stereo image exists on the epipolar line 320 shown in Fig. 1, it is sufficient to search only above the isopolar line. In particular, in the case of parallel stereo in which the projection means and the image receiving means are parallel, the calculation is simplified because the epipolar lines coincide with the rows or columns of the image plane. When the projection means and the image receiving means are not parallel, parallel stereo may be configured through a calibration and a parallelization process. This correction and parallelization process can be easily performed by those skilled in the art.

In the general structured light technique, the projection means radiates a series of patterns to an object, and reconstructs a 3D image by analyzing an image acquired through the image receiving means. At this time, it is necessary to calculate the exact pixel correspondence relationship between the projection means and the image receiving means, but in practice, a problem arises in that signals transmitted from the projection means are mixed and received by the image receiving means due to various error sources.

This mixing of signals can be explained according to the geometry of the surface portion of the object 100. 2 shows a mixed model of a signal according to the geometry of an object surface portion. Although the surface portion of the target object 100 is flat and parallel to the image planes 200 and 300 on the transmitting and receiving side as shown in FIG. 2A, the size between the light beam projected by the projection means and the reception area of the image receiving means The difference in position results in mixing of the signals. As shown in FIG. 2B, when the surface portion of the target object 100 is inclined toward the projection means (Out-Slant), an image corresponding to a plurality of light rays from adjacent pixels of the image surface 200 of the projection means is corresponding. It may be incident on the pixel of the image plane 300 of the receiving means. On the contrary, as shown in FIG. 2C, if the surface portion of the target object 100 is inclined away from the image plane 200 of the projection means (In-Slant), pixels adjacent to the image plane 200 of the projection means are inclined. Few rays of light can be incident on the pixels of the image plane 300 of the corresponding image receiving means.

In addition, as shown in (d) of FIG. 2, when there are separate objects separated around the target object (transposition), each of the two separated surface portions of the adjacent surface and the separate object far away, respectively. Transpose of the code occurs. In addition, in the case of (e) of FIG. 2 (out-discontinuity), the code is deleted by the reception area on the image surface 300 of the image receiving means and the shadow of the projection means. In case of FIG. 2F (In-Discontinuity), when there is a sharp discontinuity point of distance at the boundary of the cutting plane of the target object, the plurality of light rays are not only adjacent pixels but also corresponding pixels from distant image receiving means. It can overlap and be incident.

In addition to the mixing of such signals, the signals received by the image receiving means may be modified by system and environmental noise such as changes in surface reflectance, dissipation of light on the object surface, and changes in ambient lighting. The change in surface reflectance changes the signal's intensity, and the dissipation of light on the object's surface causes blurring or mixing of the signal. In addition, a change in intensity of a signal may occur due to a change in ambient lighting. Due to such a mixing of signals, the pixel point correspondence between the projection means and the image receiving means has an inaccurate relationship. The inaccuracy of the correspondence point causes a fundamental error in obtaining the 3D distance image in the structured light technique.

The signal separation coding method according to the present invention is designed to accurately restore the pixel correspondence point address on the projection means side through the process of separating the mixed signals with respect to such a mixture of signals. The fundamental difference from the conventional coding method is to introduce a process of separating the pattern signal received by the image receiving means. Therefore, in order to facilitate separation of signals, a plurality of light patterns are irradiated as an encoding process in the projection means.

In the present invention, the process of mixing and separating signals can be modeled as follows.

1) Mixing of Signals: First, when a signal is mixed with a neighboring signal, it may be assumed that the pixel brightness on the projection means side is a weighted sum of its brightness value and the brightness generated in the neighboring pixels. That is, the relationship between the projection means side signal and the image receiving means signal can be represented by the linear mixed signal model Y = XW . X ^∈ R ^(fxm) and Y ^∈ R ^(fxm) are respectively the transmitted code x _i = ( x ₁ ,…, x _f ) ^T sent from the projection means and the mixed image signal y _i = ( y ₁ , ..., y _f ) is a matrix consisting of ^T , and W ^∈ R ^(fxm) is a mixing matrix composed of mixing coefficients. F denotes the length of the signal and m denotes the number of pixels, respectively. For example, when using a time coding method, f means the number of frames.

2) Signal Separation: If X is a normal orthogonal matrix, then the mixing matrix is easily calculated as W = X ^T Y. Thus, if a pattern is transmitted using an orthogonal signal, even if the signals are mixed, if its value is greater than the neighboring signal value, then the index value of the largest code projected on the orthogonal codes code ( y _j ) = argmax (j) W _{i , j} can be determined.

This model is a generalized model without considering the geometrical relationship between the projection means and the image receiving means, and the search for pixel correspondence between the actual projection means and the image receiving means is epipolar rather than the whole image as in the case of stereo vision. Only above the line. In particular, as described above, in the parallel stereo condition, since the epipolar line and the row or column of the projection means and the image receiving means coincide, the search may be performed only based on the row or column in the image. For example, when placing them horizontally, the epipolar lines and their rows coincide.

In the present invention, 3D distance image measurement is performed by using an encoding, decoding, and error correction method using the concept of signal separation.

EMBODIMENT OF THE INVENTION Hereinafter, the structure of this invention is demonstrated according to drawing.

<Example 1>

3 is a flowchart illustrating a three-dimensional measurement method applying the signal separation method according to the present invention.

As shown in Fig. 3, in step S1, a unique sender address is assigned to the signal corresponding to each pixel of the object to be measured. Then, in step S2, a plurality of light patterns are projected through the projection means to encode a signal including the sender address. After receiving each pattern in step S3, the original signal is separated through a signal separation process in step S4. The corresponding point is calculated from the signal recovered in step S5.

 This step is executed by processing means for processing the collected data. In other words, the processing means used in the present invention is achieved by an ordinary computer system as having a function of executing the steps S1 to S5 as a processor.

Next, specific examples of the encoding, decoding process, and error detection and correction of a signal used in the present invention will be described.

Signal encoding

In the present invention, in order to calculate the pixel correspondence point through signal separation, an appropriate signal coding method should be assumed. As an embodiment of the coding method that enables signal separation, a plurality of patterns are projected on the time axis. In particular, an encoding method using signals orthogonal to each other between adjacent signals on the time axis may be used, and a method of hierarchically arranging orthogonal signals may be introduced to reduce the number of frames.

The structured optical system applied in the present invention may be regarded as a communication system including an information source, an encoding of the information source, a channel, a noise source, a decoding, and a receiver. When transmitting the encoded addresses in the projection means, the distortion of the signal by the geometry of the object and the system / environment can be thought of as the same as the noise added to the channel by the noise. When signals are received at the image receiving means, it is possible to recover the original signal from the noise-containing signals. If the order of the signals sent from the projection means is known, the disparity with the received signal can be calculated, and thus the 3D image can be reconstructed.

In a structured optical system using an orthogonal signal according to an embodiment of the present invention, a communication process may be divided into an encoding process and a decoding process. In the encoding process, a unique address value is first assigned to the pixel position of each image on the projection means side. In this case, since the structured light system can consider epipolar line geometric conditions as well as stereo vision, it is not necessary to give unique addresses to the entire image, but only uniquely addresses rows or columns of the image. do. Assuming that N pixels are placed on one epipolar line, the transmission signal of the projection means is composed of N addresses.

S = {s ₁ , s ₂ ,... , S _N } ∈N.

Binary code can be used for channel coding that is robust against environmental noise. That is, the binary code corresponding to the source, B = { b ₁ , b ₂ ,... , b _N } can be defined. If a full orthogonal binary code (i.e., < b _i , b _j > = 0, i ≠ j) is used, the number of images to be obtained by the image receiving means and the code length must be the same, so that an accurate distance image is obtained. Although it can be done, the calculation takes a long time. As described above, in order to solve the problem of increasing the number of frames, the present invention may use hierarchical orthogonal coding (HOC) for hierarchically arranging orthogonal signals. This HOC technique focuses on keeping the length of the signal as short as possible while maintaining the nature of the orthogonal code.

4 shows an example of such a hierarchical orthogonal code. As shown in Figure 4, the entire signal is divided into one or more layers, each layer consisting of a set of orthogonal signals. In the encoding process, a signal having N lengths is divided into L layers, and H orthogonal signals are configured in each layer. The entire signal is not orthogonal, but the signals are orthogonal to each other within a certain region (a range uniquely specified in the upper layer) within each layer. For example, suppose the HOC uses four layers (L = 4) and uses four orthogonal signals (H ₁ = H ₂ = H ₃ = H ₄ = 4) in each layer as well. The number of signals is 256 (H ₁ × H ₂ × H ₃ × H ₄ = 4 ⁴ = 256), and the code length is 16 (H ₁ + H ₂ + H ₃ + H ₄ = 16). That is, in order to recover the address, the image receiving means requires 16 frames of image.

Separation of signals

In the signal separation coding method according to the present invention, the signal separation process is a process of separating the original signal from the mixed signal and obtaining the address values transmitted by the projection means. Let the i-th position of the spatio-temporal image of the f-frame obtained by the image receiving means be represented by I (i, t), and the brightness value of the pixel at that position as the vector y _i = ( y ₁ , ..., y _f ) ^T. If the HOC has L layers, the vector y may be represented by one extension vector y _i = ( b ₁ , b _{2 ,} ..., B _L ) ^T composed of image vectors representing brightness values in each layer. The subscript of the image vector b _j indicates the j th layer. Since H orthogonal signals are used in each layer, the vector b is expressed as a linear combination of orthogonal signals, that is, b = Xc . Here, c and X represent coefficient vectors and orthogonal code matrices, respectively. That is, the vector b representing the brightness value at a specific position includes other signals due to the geometrical properties of the object and the environment and the reflection of the object surface.

In the L-th layer, the coefficient vector c (i) at the i-th position is the inner product of the transposed orthogonal matrix and the signal vector b (i) measured at that position, i.e. c (i) = X ^T b (i) Can be calculated as Through this process, the expansion coefficient vector c = ( c ₁ , c _{2 ,} …, c _L ) ^T corresponding to the expansion vector y _i = ( b ₁ , b _{2 ,} …, b _L ) ^T is calculated for the entire layer. You can get it. Due to the hierarchical structure of the HOC, the total number of possible addresses in the i th position is H ^L. That is, there are 256 possible candidates in all four layers and four orthogonal signals.

A conceptual diagram of such a signal separation process is shown in FIG. 5. In Fig. 5A, 16 frames are obtained at the i-th position on the epipolar line of the camera image. Then we take the dot product of the orthogonal matrix in (b) to separate the original signal, and (c) choose the possible code to determine (d) the correct address. Here, the process of determining the correct address is usually based on the strength of the signal is the simplest method. In general, the address is determined by the signal with the maximum signal strength.

Once the address is determined, distance images can be measured by simple geometry. Since the method of measuring distance using the original pixel position and the received pixel position is obvious in the art to which the present invention pertains, a detailed description thereof will be omitted.

Error detection and correction

Since a received signal is a mixture of multiple signals, just having maximum strength does not necessarily mean that it always provides the correct address. Therefore, when determining the correct address among possible candidate values, it is possible to correct more errors by using not only the signal strength but also the statistical distribution of the strength and the restored candidate address values in the neighbor position. Through this error correction process, more robust coding is possible for various noises.

In the present invention, the error correction method includes error correction by a rule, error correction by an evaluation function, and a mixing method of an existing coding scheme and an orthogonal coding.

(1) Correction of Error by HOC-Code Transition Rule

The error correction by the rule is that the restored address values represent a systematic pattern. That is, a series of address values restored on the epipolar line represents a systematic pattern because it reflects the geometric structure of objects and environments. The change of candidate addresses obtained through the signal separation process has the same pattern. Based on this principle, an error correction method can be used that provides parallax as accurately as possible. If we have candidate values of addresses at every location, it is possible to use this to detect and correct incorrect addresses.

Candidate code or set of DMA pixel addresses ({D _{i -k} , p _{i -k} }),... , ({D _{i -1} , p _{i -1} }), ({D _i , p _i }), ({D _{i +1} , p _{i +1} }),... , ({D _{i + k} , p _{i + k} }) are the camera pixels C _{i -k,.} , C _{i -1} , C _i , C _{i + 1} ,... , Obtained along the epipolar line by signal separation for C _{i + k} . A code represented by a 16-bit HOC code is assigned a priority index p based on the level of the relative signal strength calculated during signal separation.

Accordingly, the code transition rule for detecting an error is as follows.

Plane rule: set ({D _{i -k} , p _{i -k} }),... , ({D _{i -1} , p _{i -1} }), ({D _i , p _i }), ({D _{i +1} , p _{i + 1} }),... , ({D _{i + k,} p _{i + k})} has the same number of elements, respectively, while moving from C _i to C _{i + k -k} shows a continuous change of the pixel address.

Slanted surface rule: set ({D _{i -k} , p _{i -k} }),... , ({D _{i -1} , p _i-1 }), ({D _i , p _i }), ({D _{i +1} , p _{i +1} }),... , ({D _{i + k} , p _{i + k} }) gradually increases or decreases. Incrementally increasing or decreasing treats the same address repeated as the light beam projected on the inclined plane is enlarged, or the overlapping of consecutive addresses occurs as the light beam projected on the inclined plane is aggregated. In general, successive changes in pixel addresses remain intact, even as elements increase or decrease.

Occluding rule: set ({D _{i -k} , p _{i -k} }),... , ({D _{i -1} , p _{i -1} }), ({D _i , p _i }), ({D _{i + 1} , p _{i +1} }),... , ({D _{i + k} , p _{i + k} }) indicates a drastic change in the pixel address where the deletion of the address occurs.

Shadow / shading rule: set ({D _{i -k} , p _{i -k} }),... , ({D _{i -1} , p _i-1 }), ({D _i , p _i }), ({D _{i +1} , p _{i +1} }),... , ({D _{i + k} , p _{i + k} }) increases or decreases from 0 to 0 when entering or exiting a shadow.

(2) Error correction by evaluation function

The image measuring process according to the present invention is a complex process related to selecting a set of possible codes and selecting an address having the highest reliability of a location address from the set. In this process, the reliability indicator function can be used.

When the HOC uses the L layer and H orthogonal codes, the total number of address value candidates at a specific position becomes L ^H. Although it is possible to consider all cases, a more simplified method can be applied. In order to reduce the complexity of the calculation, one can assume that a signal is created with a mixture of at most two dominant signals. Using this assumption, only 2 ^L candidates can be considered, not L ^H candidates.

That is, first, two representative signals are selected using the signal strength in each layer of the HOC. Second, in order to determine the most reliable address, reliability indicators can be defined as factors such as the magnitude of the signal at the location, uncertainty due to differences from neighboring signals, and continuity reflecting the structural relationship between the object and the environment. have. That is, the code code (y _i) of y _i in the image position can be determined by the continuity of the calculated distance and the uncertainty of the position adjacent to reflect the size difference between the signal level and neighboring signals y _i. That is, if the weight vector of y _i is w _i = ( w ₁ ,…, w _f ) ^T , and the binary code is c _i = code ( y _i ), the cost function h can be defined as have.

In Equation 1, Ci (j) and w _i = ( w ₁ , ..., w _f ) ^T represent a decoded signal and a weight vector, respectively. The terms CC _i (j) w _i (j) and ∑ (w _i log (1 / w _i )) correspond to entropy measurements for the signal magnitude and uncertainty measurements of the orthogonal code Ci, respectively. The function g (C _i , C _k ) is a measure corresponding to the geometric continuity at the location of the distance information. These functions can be defined in various ways. For example, a simple slope can be used. λ1 and λ2 are coefficients for adjusting the degree of each factor.

(3) Error correction by mixing method

In order to reduce the number of images to be acquired and the calculation amount, a time coding such as a conventional gray code may be mixed with orthogonal coding. The orthogonal code at this time is used to correct an error in time coding. That is, the orthogonal code can determine the position of the pixel more accurately than the time code, and since the time code has more uncertainty in the lower bits, the orthogonal code can be used to correct the error in the local window area.

Add an orthogonal code with a length of K (e.g., three frames) before or after conventional time coding. Because orthogonal codes use only a small length of signal, it is not possible to fully determine the magnetic position in the entire image, but it is possible to determine the magnetic position in a locally adjacent region. That is, positioning can be performed among K-1, K, and K + 1. Using this property, if the position reconstructed using the orthogonal code and the position obtained using the time code do not match, a method of correcting the local position specified by the orthogonal code can be used.

In the present invention, an example of coding using an orthogonal code and a hierarchical orthogonal code is described, but the present invention is not necessarily limited to such a code, and a pseudo orthogonal code, a mixture of orthogonal codes and other codes, and statistically independent signals are described. It is also possible to recover the original signal using various codes, such as.

In addition, the signal separation coding method according to the present invention may be used by mixing a conventional coding method. That is, after signal separation coding is used for some of the multiple optical patterns, the signal separation coding method of the present invention may be used to correct the reconstructed address value without using signal separation.

<Example 2>

The signal separation coding scheme according to the first embodiment is a method of estimating a signal mixture between a projector and a camera pixel to determine a correspondence of the pixels. In the process of separating the mixed signals, the result of projecting the received signal onto a set of hierarchical orthogonal signals is represented in the projector-camera signal space. Calculation of the three-dimensional distance image was a process of determining a signal on the projector side corresponding to each pixel of the camera in the projector-camera signal space.

In the second embodiment, the Helmholtz complementarity principle and the signal separation that the light source received from the light source reflected from the object surface can be interpreted in the same way as the same light source is reflected from the receiving side and reflected from the object surface to the position of the light source. Using coding techniques, we present (1) a technique for generating an image from a projector's point of view without explicit reconstruction of three dimensions, and (2) simultaneously calculating the pixel-by-pixel correspondence in the generated stereo image and structured light. Present a new way to restore it.

In conventional dual photography, a time-consuming analysis method is used to analyze geometric relationships between a projector, an object, and a camera. The present invention uses a two-step analysis method different from the conventional method. First, a signal encoded to have hierarchical orthogonality in consideration of epipolar constraint in stereo vision is generated and radiated through a projector.

The mixed signal received by the camera reflecting the object is separated by a decoding technique for separating the original signal, and a technique for efficiently estimating the conversion relationship between the projector and the camera is applied based on the strength of the separated signal.

Through this process, an image is generated from the perspective of the projector and the image is merged with the distance image from the structured light, thereby simultaneously generating the image from the perspective of the projector and estimating the 3D distance image.

In other words, the conventional method has been limited to objects that do not have large properties such as reflection, transmission, and absorption in a limited environment such as a studio. One of the main reasons is to approach the creation of three-dimensional images and computational models only by interpretation of physical properties between light sources, objects, and sensors (e.g. shape from shading, photometric stereo, etc.) or by geometric relationships only. (E.g. passive or active stereo, structure from motion, etc.)

In Embodiment 2 according to the present invention, (1) analysis in terms of radiometry, which consists of scattering, reflection and reception at the light source and the object surface, and (2) the two-dimensional image of the projector Overcoming the limitations of existing methods by presenting a method of incorporating the geometric interpretation of projection into three-dimensional space, epipolar constraints between projectors, objects and cameras, and projection from three-dimensional space to the camera two-dimensional image plane.

Therefore, in Embodiment 2, on the contrary, when estimating the signal of the camera side corresponding to each position of the projector side, a new image viewed from the projector side can be obtained as shown in FIG. In FIG. 9, the unreconstructed area in the upper and lower parts is caused because the field of view of the projector and the field of view of the camera do not overlap, and the noise is generated because only the maximum value of the mixed signals is used for calculation.

6 and 7 are diagrams showing signal separation maps (FIG. 6) obtained as a result of signal separation on a specific epipolar line and generation of a dual photography image using the same.

FIG. 6 shows the relationship between the separated signals as the relationship between the set of signals (horizontal axis) sent from the projector and the set of signals (vertical axis) received from the camera. The brightness in the figure refers to the strength of the correspondence. In the enlarged small area, the correspondence between the projector and the camera signal is not 1: 1, but M: N. The simplest way to determine the correspondence of signals between the projector and the camera is to select the signal of the projector with the highest intensity for each address on the camera pixel side.

7 shows the principle of generating a new image at the projector side using this signal relationship. Contrary to FIG. 6, if all the protector sides determine the address of the camera side, a new image (right side of FIG. 7) viewed by the projector may be used using the same. That is, the brightness values for the x and y positions of the newly obtained image are

dualimage (x, y): = ref (arg max (j) Ty (j, x), y) ref denotes the image of the projector side, and Ty denotes a mixed separation map in the yth column (epipolar line).

8 and 9 show dual photography with signal separation coding.

That is, FIG. 8 is a gray scale image of the inside of the refrigerator taken from the camera side, and FIG. 9 is a new view image generated from the projector side by using signals separated on all epipolar lines. Since the camera-projector is positioned up and down, respectively, when the image is taken, it can be observed that the position of the air circulation port on the rear background is located further downward when viewed from the projector side than the camera side.

In other words, according to the second embodiment, the analysis on the physical side and the analysis on the geometric side are integrated, so that three-dimensional image acquisition due to scattering, reflection, absorption, and transmission of various object surfaces that are easily overlooked when only the geometric side is considered The difficulty of model generation can be partially solved and, conversely, if only the interpretation in terms of physical aspects is taken into account, enormous calculations are required to interpret the physical transformations between artificial light sources and objects and cameras that are difficult to implement with a sensing system. You can solve the problem.

As mentioned above, although the invention made by this inventor was demonstrated concretely according to the said Example, this invention is not limited to the said Example and can be variously changed in the range which does not deviate from the summary.

As described above, according to the structured light-based three-dimensional distance image measuring method and system according to the present invention, even if the signal received from the image receiving means overlap, deformation due to the geometry of the object can be accurately separated In addition, the effect of obtaining a distance image that is robust even in the presence of noise in the surrounding environment is obtained.

In addition, according to the method and system for measuring a 3D distance image according to the present invention, when generating an image on the projector side by a conventional method, a problem that requires enormous calculation can be solved. It is possible to obtain a three-dimensional image robustly to noise by using three-dimensional geometric information simultaneously.

Claims (15)

In the method of irradiating a light beam from the projection means to the object to be measured and taking the light beam from the image receiving means to measure the three-dimensional distance as an image, Encoding a signal by assigning a unique sender address to a signal corresponding to each pixel of the projection means, Projecting a plurality of light patterns through the projection means to transmit a signal, Receiving the encoded signal by the image receiving means; Recovering an address by separating the received signal; and And determining a pixel position of a target object by using the sender address and the reconstructed address. The method of claim 1, Separating the received signal Calculating candidate values of possible addresses corresponding to the sender address; An error correction step comprising determining an address according to a rule for obtaining an accurate address among the candidate values; A rule that gradually increases or decreases the value of the restored address when a signal is obtained from the slope of the object, When a signal is obtained at the cutting plane of an object, the value of the restored address is not continuous, and part of the rule disappears or transposes. When the signal disappears from the shadow of an object, the value of the restored address is not continuous but disappears. The error correction step is selected from the rule group structured light-based three-dimensional distance image measuring method. The method of claim 1, Separating the received signal Calculating candidate values of possible addresses corresponding to the sending address; Defining an evaluation function by simultaneously considering candidates of the address value of the own position among the candidate values and a candidate set of address values at a neighboring location, and then determining an address using a search method that maximizes or minimizes the evaluation function. Method for measuring a three-dimensional distance image based on structured light, characterized in that it comprises an error correction step consisting of. The method of claim 1, Separating the received signal Calculating candidate values of possible addresses corresponding to the sending address; In order to determine an address within a plurality of candidate sets of candidate values, an address value of a location having high reliability is first calculated, and then the value of the location is fixed, and then the candidate is gradually determined to be a value of another area. The structured light-based three-dimensional distance image measuring method comprising the step of determining the error correction step. The method according to any one of claims 2 to 4, The light pattern may be a binary light pattern, a color light pattern, a light pattern having a gray scale brightness value, a light pattern in an ultraviolet or infrared region in a non-visible region, or a mixed pattern thereof. Distance image measurement method. The method according to any one of claims 2 to 4, And encoding the signals by using signals that are orthogonal to each other between adjacent signals of the projection means. The method of claim 6, The orthogonal signals divide the entire signal into one or more hierarchical signals, and use codes that are orthogonal to each other in each layer. The method of claim 6, The orthogonal signals divide the entire signal into one or more hierarchical signals and use a mixture of codes that are orthogonal to each other and codes that are not orthogonal to each other in each layer, and use the structured light-based three-dimensional distance image measurement. Way. The method according to any one of claims 2 to 4, And a pseudo orthogonal signal having a property orthogonal to each other between adjacent signals of the projection means in the encoding of the signal. The method according to any one of claims 2 to 4, And in the encoding of the signal, signals which are statistically independent from each other between adjacent signals of the projection means. A method of measuring three-dimensional distance image in a system consisting of a projector and a camera, Radiating a pattern from the projector to an object, Receiving the pattern with the camera, Calculating pixel correspondence relations between the projector and the camera through signal separation on each epipolar line; and And generating a new image from the perspective of the projector after executing the calculating step. As a method of restoring images in a system consisting of a projector and a camera, Radiating a pattern from the projector to an object, Receiving the pattern with the camera, Calculating pixel correspondence relations between the projector and the camera through signal separation on each epipolar line; and After executing the calculating step, generating a new image at the projector view; And restoring a stereo image using the image obtained from the camera and the image newly generated by the projector. A method of restoring measurement distance in a system consisting of a projector and a camera. Encoding a signal by assigning a unique sender address to a signal corresponding to each pixel of the camera; Transmitting a signal by projecting a plurality of light patterns through the projector, Receiving the encoded signal at the camera, Recovering the address by separating the received signal; and A 3D distance image measuring method based on structured light including determining a pixel position of a target object using a sender address and the reconstructed address; Radiating a pattern from the projector to an object, Receiving the pattern with the camera, Calculating pixel correspondence relations between the projector and the camera through signal separation on each epipolar line; and And after the calculating, restoring the measurement distance by simultaneously using a restoration method for restoring a stereo image including generating a new image at the perspective of the projector. Projection means for irradiating light to the object to be measured, Image receiving means for photographing the light beam irradiated from the projection means; Processing means for measuring a three-dimensional distance of the object to be measured as an image, wherein the processing means Encoding a signal by assigning a unique sender address to a signal corresponding to each pixel of the projection means, Projecting a plurality of light patterns through the projection means to transmit a signal, Receiving the encoded signal by the image receiving means; Recovering an address by separating the received signal; and And determining a pixel position of a target object using the sender address and the reconstructed address in sequence. The method according to claim 1 or 14, wherein The projection means is a projector, The image receiving means is a three-dimensional distance image measuring system, characterized in that the camera.

Images