KR102684986B1 - Device for 3D image reconstruction using rectangular grid projection - Google Patents

Abstract

A square grid pattern is projected onto the subject with a beam projector, and the projected grid pattern is observed by observing the subject from different angles. The observed 2D grid information is converted into 3D grid data using a projection-detection algorithm, and implemented as a 3D object using a commercially available generator.

Description

3D image reconstruction device using rectangular grid projection {Device for 3D image reconstruction using rectangular grid projection}

The present disclosure is a 3D image reconstruction device that can reconstruct a 3D image by processing the data obtained by projecting a square grid pattern onto a subject with a single projector and observing it with a single camera with groundbreaking efficiency using a unique algorithm. It's about.

In general, 3D image reconstruction devices are used to produce 3D images in movies or games, or to inspect the manufacturing status of measurement objects in manufacturing industries such as semiconductors.

The most basic method of reconstructing the movement of an object in three dimensions on a computer (3D motion capture) is to attach a sensor to the object and receive information about the movement from it and combine it. This of course assumes that the object can be accessed in advance, and requires sensors and a special recording environment.

If such advance access is not possible in principle, movement can be captured using two or more cameras or one camera and an auxiliary device. Microsoft's Kinector uses an infrared device and a special pattern projector, and although it has no problem understanding gestures, the fundamental complexity of the technology prevents it from capturing accurate shapes in real time, especially when the subject moves quickly.

This is a widely used method developed in the 1990s that uses multiple photos or converts light generated from a light source into a standard pattern and projects it onto the object. The light transformed according to the shape of the object is compared with the standard pattern to create a three-dimensional image of the object. There is a Moire method to reconstruct .

The advantage of the moiré method is that measurements can be made without contact with the object, so there is less risk of damaging the object. However, the widely used phase shift projection moiré method had the problem of time delay because it required shooting multiple two-dimensional images to reconstruct a scene. In addition, the moiré method could only be used in a controllable environment because the grid must be placed close to the target object to obtain an interference pattern.

Therefore, the present disclosure is a device that reconstructs a three-dimensional image even in an environment where the subject cannot be controlled by efficiently processing the data obtained by one projector projecting a square grid pattern onto the subject and observing it with a single camera using a unique algorithm. In providing.

In order to achieve the above object, a three-dimensional image reconstruction system according to an embodiment of the present disclosure includes a projector that projects a square grid pattern onto a target object, a camera that photographs the projected pattern, and the camera and the projector. Calibrate, set a coordinate system including the camera, image plane, projection point, and grid using the two-dimensional image data obtained from the camera, and project straight lines passing through each grid point and projection origin onto the image plane. It includes an electronic device that corresponds grid points appearing in a two-dimensional image with the grid points using a method and measures three-dimensional coordinates of grid points appearing in the target object.

Meanwhile, a method of reconstructing a 3D image by projecting a square grid according to an embodiment of the present disclosure includes the steps of calibrating a camera and a projector, and projecting a square grid pattern onto a target object using the projector. , setting a coordinate system including a camera, an image plane, a projection point, and a grid using a two-dimensional video image obtained by photographing the projected pattern, and projecting straight lines passing through each grid point and the projection origin onto the image plane. It includes matching grid points shown in the two-dimensional video image with grid points in the grid pattern, and measuring three-dimensional coordinates of the grid points shown in the target object.

In this case, the calibrating step involves projecting a grid onto a device with three or more checkerboard plastic plates attached to each other to have an angle of 90 degrees to 130 degrees, and using the camera to obtain images and standards for the projected grid. The camera and the projector can be calibrated using a calibration algorithm.

Meanwhile, in the projecting step, a grid pattern may be projected onto the surface of the target object using the projector.

In this case, the projecting step may project the grid points on the surface of the target object along with some of the grid points having brighter or different colors than other grids.

Meanwhile, setting the coordinate system includes detecting grid points in a two-dimensional image from two-dimensional image data obtained through the camera, and storing location information of the detected grid points in the two-dimensional image, and It may include setting the coordinates of the image plane, projection origin, and grid of the two-dimensional image by setting a three-dimensional coordinate system using the camera as the origin.

Meanwhile, the matching step includes a straight line projection step of creating a bundle of straight lines by projecting a straight line passing through each grid point on the grid and the projection origin onto the image plane, from a grid point on the image to find a corresponding grid point on the grid. A dotted line distance measurement step of measuring the distance of the straight line bundles obtained in the straight line projection step, and a straight line that minimizes the distance value obtained in the dotted line distance measurement step is found and connects the projection origin and the grid point projected by the straight line in the image image. It may include a step of corresponding grid points on the image with straight grid points.

In this case, if there is a grid point that is brighter or has a different color than other grid points, the step of measuring the dotted line distance from the grid point and the step of matching the grid points can be performed with priority.

Meanwhile, the step of measuring the three-dimensional coordinates corresponds to the grid points shown in the two-dimensional image using a method of projecting straight lines passing through each grid point and the projection origin onto the image plane. From the correspondence between the grid points of the 3D image and the grid points on the grid, the 3D coordinates of the point where the straight line connecting the origin and the 2D image and the straight line connecting the projection origin and the grid points on the grid intersect can be obtained.

According to the various embodiments of the present disclosure as described above, the three-dimensional shape of the measurement object can be precisely reconstructed at high speed using an algorithm that consists only of a projector that projects a square grid and a camera that photographs it and processes it in one shot. You can. While the existing moiré method obtains coordinates by analyzing the interference pattern of light passing through a grid, the present disclosure is a fundamentally different and efficient method of obtaining coordinates directly from the grid image itself. In particular, while the existing moiré method requires placing a grid at a certain angle close to the target object to obtain an interference pattern, the present invention places the grid together with a light source, so it can be applied to objects moving at a distance (for example, a car passing through a toll gate). You can.

1 is a diagram for explaining an image reconstruction system according to an embodiment of the present disclosure, and
Figure 2 is a flowchart for explaining an image reconstruction method according to an embodiment of the present disclosure.

Since these embodiments can be modified in various ways and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, and should be understood to include all transformations, equivalents, and substitutes included in the disclosed spirit and technical scope. In describing the embodiments, if it is determined that a detailed description of related known technology may obscure the point, the detailed description will be omitted.

Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. Terms are used only to distinguish one component from another.

The terms used in this application are merely used to describe specific embodiments and are not intended to limit the scope of rights. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, “comprises.” Or “made up.” Terms such as are intended to designate the presence of features, numbers, steps, operations, components, parts, or a combination thereof described in the specification, but are intended to indicate the presence of one or more other features, numbers, steps, operations, components, parts, or It should be understood that the existence or addition possibility of combinations of these is not excluded in advance.

In an embodiment, a ‘module’ or ‘unit’ performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' are integrated into at least one module and implemented by at least one processor (not shown), except for 'modules' or 'units' that need to be implemented with specific hardware. It can be.

Hereinafter, the present disclosure will be described in detail using the attached drawings.

1 is a diagram for explaining an image reconstruction system according to an embodiment of the present disclosure.

Referring to FIG. 1, the image reconstruction system 1000 may be comprised of a projector 100, a camera 200, and an electronic device 300.

The projector 100 projects a square grid pattern onto the subject 10. This projector 100 can operate in a laser diode method or a beam projector method, and can project a square grid pattern onto the target object, that is, the subject 10. Here, the subject 10 may be referred to as a target object, and may be a fixed object or a moving object.

The camera 200 photographs the grid pattern projected on the subject 10 and outputs digital image data. Specifically, the camera 200 is arranged to be spaced apart from the projector 100 and generates a captured image (i.e., a two-dimensional video image) of the subject 10 on which a grid pattern is projected at an angle different from that of the projector 100 to be transmitted to the electronic device. It can be output as (300).

The electronic device 300 calibrates the camera 200 and the projector 110. Specifically, the electronic device 300 controls the projector 110 to project the grid on a device on which three or more checkerboard plastic plates are attached to have an angle of 90 degrees to 130 degrees, and the checkerboard plate on which the grid is projected is attached. The camera 200 can be controlled so that the device takes pictures. Additionally, the electronic device 300 can calibrate the camera 200 and the projector 110 using the image provided by the camera 200 and a standard calibration algorithm.

Additionally, the electronic device 300 can control the projector 110 to project a square grid pattern onto the target object, and control the camera 200 to capture an image of the target object onto which the grid pattern is projected during the process. At this time, the electronic device 300 may control the projector 110 to make some of the plurality of grid points brighter than other grid points or to project a square grid pattern with a different color. These unique grid points may be generated by the projector 110 or by a separate laser beam from the projector 110.

And when the electronic device 300 receives a two-dimensional image of the target object, it sets a coordinate system. Specifically, the electronic device 300 may detect grid points in a two-dimensional image from two-dimensional image data obtained through the camera 200 and store location information of the detected grid points in the two-dimensional image. Additionally, the electronic device 300 can set the coordinates of the image plane, projection origin, and grid of the two-dimensional image by setting a three-dimensional coordinate system with the camera as the origin.

Additionally, the electronic device 300 may correspond to the grid points of the grid pattern projected by the projector and the grid points in the two-dimensional image generated by the camera 200. Specifically, the electronic device 300 can correspond grid points appearing in a two-dimensional video image with grid points in a grid pattern using a method of projecting straight lines passing through each grid point and the projection origin onto the image plane.

For example, each grid point on the grid and the straight line passing through the projection origin are projected onto the image plane to create a bundle of straight lines, and the distance of the previously obtained bundle of straight lines is measured from the grid point on the image to find the corresponding grid point. Then, by finding a straight line that minimizes the distance value obtained in the dotted line distance measurement step, the grid point on the image can be matched to the grid point of the straight line connecting the grid point projected by the straight line in the image image and the projection origin.

On the other hand, if there is a grid point in the two-dimensional image image that is brighter than other grid points or has a different color from other grid points, the distance measurement operation and the operation of matching grid points are given priority over other grid points. It can be done.

Then, the electronic device 300 measures the three-dimensional coordinates of grid points appearing on the target object. Specifically, the electronic device 300 corresponds to the grid points shown in the two-dimensional image using a method of projecting straight lines passing through each grid point and the projection origin onto the image plane. From the correspondence between grid points and grid points on the grid, the three-dimensional coordinates of the point where the straight line connecting the origin and the two-dimensional image and the straight line connecting the projection origin and the grid points on the grid intersect can be obtained.

Alternatively, the electronic device 300 may receive digital image data from the camera 200 and reconstruct a 3D image. Specifically, the electronic device 300 stores straight line information passing through each intersection of the projection origin of the projector and the square grid and measures three-dimensional coordinates using an algorithm that searches for the intersection of the grid corresponding to the intersection projected on the object. can do.

More specifically, the electronic device 300 generates a sample grid square. Then, the electronic device 300 performs data fitting. Specifically, the electronic device 300 can calculate the equations of the lines connecting the light source and the grid from the positions of the projector and the grid and use this to calculate which part of the grid the sample grid square originally came from.

Then, the electronic device 300 generates a plurality of 3D samples. Specifically, the electronic device 300 may generate a plurality of 3D image samples by repeating the above process for a plurality of sample grid squares, taking observation errors into account.

And the electronic device 300 creates a 3D model. Specifically, the electronic device 300 can generate the sample that matches the most among the plurality of previously generated 3D samples as an actual 3D model with a high probability. The created 3D model can be used for virtual reality, medical imaging, etc.

As described above, the image reconstruction system according to this embodiment can immediately obtain the (3-dimensional) depth of a 2-dimensional grid pattern by analyzing a simple pattern in a unique and innovatively efficient way, without analyzing complex patterns in a complicated way. Therefore, it can be used to capture 3D images of a subject in real time.

In particular, it is a much more advanced technology than existing systems that are conceptually/technically complex, such as using two or more beam projectors or projecting special patterns, and have large data processing time and costs.

In the above, only the brief configuration of the image reconstruction system 1000 has been described, but the image reconstruction system 1000 may further include additional configurations in addition to the configuration described above, and any one of the above-described camera and projector may be a configuration within an electronic device. there is.

Figure 2 is a flowchart for explaining an image reconstruction method according to an embodiment of the present disclosure.

Referring to FIG. 2, first, light from a light source is passed through a grid to create a grid pattern on the surface of the subject (S510).

Afterwards, the subject with the grid pattern is photographed with a camera, and then the coordinate values of the grid points are obtained (S520).

And the 3D image is reconstructed (S530). Specifically, it is possible to create a sample grid square, calculate the equations of the lines connecting the light source and the grid from the positions of the projector and the grid, and use this to calculate which part of the original grid the sample grid square comes from. Also, considering the observation error, the above process can be repeated for a number of sample grid squares to generate a number of 3D image samples. And among the multiple 3D samples previously created, the sample that matches the most can be created as an actual 3D model with a high probability.

As described above, the image reconstruction method according to this embodiment can immediately obtain the (3-dimensional) depth of a 2-dimensional grid pattern by analyzing a simple pattern in a unique and innovatively efficient way, without analyzing complex patterns in a complicated way. Therefore, it can be used to capture 3D images of a subject in real time.

Meanwhile, the image reconstruction method according to the above-described embodiment may be implemented as a program and provided to an electronic device. In particular, a program including an image reconstruction method may be stored and provided in a non-transitory computer readable medium.

A non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specifically, the various applications or programs described above may be stored and provided on non-transitory readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc.

In addition, although the preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and the technical field to which the disclosure pertains without departing from the gist of the present disclosure as claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical idea or perspective of the present disclosure.

1000: Image reconstruction system 100: Projector
200: camera 300: electronic device

Claims (9)

In a method of reconstructing a 3D image by projecting a square grid,
calibrating the camera and projector;
Projecting a square grid pattern onto a target object using the projector;
Setting a coordinate system including a camera, an image plane, a projection point, and a grid using a two-dimensional video image obtained by photographing the projected pattern;
Corresponding grid points appearing in the two-dimensional video image with grid points in the grid pattern using a method of projecting straight lines passing through each grid point and the projection origin onto the image plane; and
It includes; measuring three-dimensional coordinates of grid points appearing on the target object,
The corresponding step is,
A straight line projection step of creating a bundle of straight lines by projecting a straight line passing through each grid point on the grid and the projection origin onto an image plane;
A dotted line distance measurement step of measuring the distance of the straight line bundles obtained in the straight line projection step from the grid point on the image to find the corresponding grid point on the grid;
A method including the step of finding a straight line that minimizes the distance value obtained in the dotted line distance measurement step and corresponding the grid point on the image with a grid point on a straight line connecting the grid point projected with the straight line in the image image and the projection origin. delete delete According to paragraph 1,
The projection step is,
A method of projecting a grid pattern on the surface of the target object using the projector, wherein some of a plurality of grid points within the grid pattern are brighter or have a different color than other grid points. delete delete According to paragraph 1,
If there is a grid point that is brighter or has a different color than other grids, a method of first performing the step of measuring the dotted line distance from the grid point and the step of matching the grid points. In a method of reconstructing a 3D image by projecting a square grid,
calibrating the camera and projector;
Projecting a square grid pattern onto a target object using the projector;
Setting a coordinate system including a camera, an image plane, a projection point, and a grid using a two-dimensional video image obtained by photographing the projected pattern;
Corresponding grid points appearing in the two-dimensional video image with grid points in the grid pattern using a method of projecting straight lines passing through each grid point and the projection origin onto the image plane; and
It includes; measuring three-dimensional coordinates of grid points appearing on the target object,
The step of measuring the three-dimensional coordinates is,
The difference between the grid points of the two-dimensional image and the grid points on the grid obtained from the step of corresponding the grid points shown in the two-dimensional image with the grid points using a method of projecting straight lines passing through each grid point and the projection origin onto the image plane. A method of obtaining the 3D coordinates of the point where the straight line connecting the origin and the 2D image and the straight line connecting the projection origin and grid points on the grid intersect from the correspondence. In the image reconstruction system,
A projector that projects a square grid pattern onto a target object;
a camera that photographs the projected pattern; and
Calibrate the camera and the projector, set a coordinate system including the camera, image plane, projection point, and grid using two-dimensional image data obtained from the camera, and set a coordinate system including the camera, image plane, projection point, and grid, and pass through each grid point and projection origin. An electronic device that corresponds grid points appearing in a two-dimensional image with grid points in the grid pattern using a method of projecting straight lines onto an image plane, and measures three-dimensional coordinates of grid points appearing in the target object; ,
The electronic device is,
A straight line projection is performed to create a bundle of straight lines by projecting a straight line passing through each grid point on the grid and the projection origin onto the image plane, and the straight line projection is performed from a grid point on the image to find a corresponding grid point. Perform measurement of the dotted line distance, which measures the distance of the bundle of straight lines obtained by doing so, find the straight line that minimizes the dotted line distance, and create a grid on the image as a straight grid point connecting the grid point that was projected with the straight line in the image image and the projection origin. An image reconstruction system that matches grid points appearing in the two-dimensional image with grid points in the grid pattern by matching points.