KR102629889B1 - Projection Mapping Method through Calibration Process Using Depth Rgb Data - Google Patents

Abstract

The projection mapping method through a calibration process using depth RGB data according to the present invention formalizes the relationship between each pixel constituting the pattern of light emitted through a beam projector and the light emitted from a LIDAR including an RGB camera. Step (a) of deriving a reference relational expression, step (b) of extracting a point cloud for the actual object of the target object through the LIDAR, based on the reference relational expression derived in step (a), a processor of a computer Step (c) of deriving a projection equation that formalizes the relationship between each pixel included in the pattern of light irradiated through the beam projector from the point cloud of the target object extracted in step (b) through and (c) It includes step (d) of performing projection mapping for the virtual object through the beam projector based on the projection equation derived in step (d).

Description

Projection Mapping Method through Calibration Process Using Depth Rgb Data}

The present invention relates to a projection mapping method, and more specifically, to a projection mapping method that can perform mapping without error through a calibration process using depth RGB data.

Projection Mapping is a technology that makes an object that exists in reality appear to have a different personality by projecting an image made of light onto the surface of the object and changing it.

Such projection mapping uses a projector to project light onto 3D objects such as buildings and statues, giving users a unique visual and psychological experience, encouraging people to participate, and providing a deeper user immersive experience.

Recently, such projection mapping technology has been actively applied in the fields of augmented reality (AR) and mixed reality (MR), and accordingly, the demand for application of projection mapping technology, especially for long-distance, large-scale, and non-planar targets, is increasing. am.

However, existing projection mapping methods are mainly effective when applied at short distances, and have the disadvantage that projection is not accurate when the mapping target is non-planar or long distances.

Additionally, the projection mapping method using a camera system has a problem in that the possibility of calibration errors occurring is very high because there is perspective distortion and lens distortion that occurs in the camera and projector lens.

Therefore, a method to solve these problems is required.

Korean Patent No. 10-1465497

The present invention is an invention made to solve the problems of the prior art described above, and has the purpose of enabling error-free projection mapping for long-distance, large-scale, and non-planar targets.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

The projection mapping method through a calibration process using depth RGB data of the present invention to achieve the above-mentioned purpose involves irradiating each pixel constituting the pattern of light irradiated through a beam projector and a LiDAR including an RGB camera. Step (a) of deriving a standard relational expression by formulating the relationship between rays, step (b) of extracting a point cloud for the real object of the target object through the lidar, and the standard relational expression derived in step (a) Based on this, (c) deriving a projection equation that formalizes the relationship between each pixel included in the pattern of light emitted through the beam projector from the point cloud of the target object extracted in step (b) through a computer processor. and a step (d) of performing projection mapping on the virtual object through the beam projector based on the projection equation derived in step (c).

At this time, step (a) may be performed two or more times before step (b) is performed.

And the step (a) is the step (a-1) of irradiating a preset light pattern to an arbitrary projectile through the beam projector, and the beam projector in step (a-1) through the lidar. Step (a-2) of scanning the pattern of light irradiated through the beam projector and extracting a point cloud for each pixel constituting the light pattern, and irradiating the light through the beam projector in step (a-1) through the processor of the computer. It may include a step (a-3) of deriving a reference relational expression by formulating the relationship between each pixel constituting the light pattern and the light ray emitted from the LIDAR in step (a-2).

Additionally, in step (a-2), only the point cloud for the pattern area of light irradiated through the beam projector can be extracted from the entire area scanned through the LIDAR.

Meanwhile, step (a-3) is step (a-3-1) of converting the point cloud extracted in step (a-2) into a 2D image, and step (a-3-1) of converting the point cloud extracted in step (a-2) into a 2D image. Step (a-3-2) of calculating center values for patterns appearing in 2D images, 3D position including a plurality of 3D position values through the center values calculated in step (a-3-2) Step (a-3-3) of deriving a value set and step (a-3-4) of deriving a 3-dimensional straight line equation by combining a plurality of 3-dimensional position values derived in step (a-3-3). may include.

In addition, step (a-3-1) can use a spherical coordinate system in the process of converting the point cloud extracted in step (a-2) into a 2D image.

And in step (a-3-2), the center value for patterns appearing in the 2D image can be calculated using the Harris Corner Detect method.

The projection mapping method through a calibration process using depth RGB data of the present invention to solve the above problems uses depth RGB data acquired using lidar to determine the relationship between all pixels of light emitted from a beam projector. Since it can be formalized, it is free from the distortion of the beam projector lens, and therefore has the advantage of enabling error-free projection mapping even when the target object for projection mapping has a large scale or is located over a long distance.

Accordingly, the present invention can be used for large-scale projection mapping used in performances, exhibitions, and advertisements, and when using a device that expands the angle of light, such as a convex lens, a larger area can be covered with only a small number of beam projectors. , it has the advantage of being able to dramatically reduce the calibration process itself, which was previously performed 10 to 20 times, to 2 times.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a diagram showing the entire process of a projection mapping method through a calibration process using depth RGB data according to an embodiment of the present invention;
Figure 2 is a diagram schematically showing the configuration of a system for performing a projection mapping method through a calibration process using depth RGB data according to an embodiment of the present invention;
Figure 3 is a diagram showing the detailed process of step (a) in the projection mapping method through a calibration process using depth RGB data according to an embodiment of the present invention; and
Figure 4 is a diagram showing the detailed process of step (a-3) in the projection mapping method through a calibration process using depth RGB data according to an embodiment of the present invention.

In this specification, when a component (or region, layer, portion, etc.) is referred to as being “on,” “connected to,” or “coupled to” another component, it is directly placed/on the other component. This means that they can be connected/combined or a third component can be placed between them.

Identical reference numerals refer to identical elements. Additionally, in the drawings, the thickness, proportions, and dimensions of components are exaggerated for effective explanation of technical content.

“And/or” includes all combinations of one or more that the associated configurations may define.

Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

Additionally, terms such as “below,” “on the lower side,” “above,” and “on the upper side” are used to describe the relationship between the components shown in the drawings. The above terms are relative concepts and are explained based on the direction indicated in the drawings.

Unless otherwise defined, all terms (including technical terms and scientific terms) used in this specification have the same meaning as commonly understood by a person skilled in the art to which the present invention pertains. Additionally, terms such as those defined in commonly used dictionaries should be construed as having a meaning consistent with their meaning in the context of the relevant technology, and unless interpreted in an idealized or overly formal sense, are explicitly defined herein. do.

Terms such as “include” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but do not include one or more other features, numbers, or steps. , it should be understood that it does not exclude in advance the possibility of the existence or addition of operations, components, parts, or combinations thereof.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

Figure 1 is a diagram showing the entire process of a projection mapping method through a calibration process using depth RGB data according to an embodiment of the present invention.

Figure 2 is a diagram schematically showing the configuration of a system for performing a projection mapping method through a calibration process using depth RGB data according to an embodiment of the present invention. In the following description, the number of each component is as follows. Based on the drawing.

As shown in FIG. 1, the projection mapping method through a calibration process using depth RGB data according to an embodiment of the present invention includes steps (a) to (d).

Step (a) is a process of deriving a standard relational expression by formulating the relationship between each pixel constituting the pattern of light irradiated through the beam projector 10 and the ray irradiated from the LIDAR 20 including the RGB camera. .

This step is a process for deriving a reference relational expression for calculating the relationship between the target object 1 to be scanned and each pixel included in the pattern of light irradiated through the beam projector 10. The beam projector ( 10), the relationship between each pixel constituting the pattern of light emitted and the light emitted from the LIDAR 20 including the RGB camera is formalized.

At this time, step (a) can be performed twice before step (b), which dramatically reduces the calibration process itself, which was previously performed 10 to 20 times.

And this step (a) may include a plurality of processes in detail.

Figure 3 is a diagram showing the detailed process of step (a) in the projection mapping method through a calibration process using depth RGB data according to an embodiment of the present invention.

As shown in Figure 3, step (a) of the projection mapping method through a calibration process using depth RGB data according to an embodiment of the present invention is in detail steps (a-1) to (a-3). may include.

First, step (a-1) is a process of irradiating a preset light pattern to an arbitrary projectile 15 through the beam projector 10.

In this process, a certain pattern is irradiated onto a random projector 15, such as a screen, using the beam projector 10.

And, as described above, step (a) can be performed twice, so in this process, the projectile 15 can be installed in two places at a certain interval.

In step (a-2), the pattern of light irradiated through the beam projector 10 in step (a-1) is scanned through the LIDAR 20 including an RGB camera, and each pixel constituting the light pattern is scanned. This is the process of extracting the point cloud.

That is, in this process, a scan is performed with the LIDAR 20 while being projected onto a projector 15 such as a screen, and the point cloud is extracted as a pts file. The file consists of x, y, z in 3D space and r, g, and b values corresponding to those values.

At this time, this step may be performed by extracting only the point cloud for the pattern area of light irradiated through the beam projector 10 from the entire area scanned through the LIDAR 20.

This is to reduce the amount of data and memory usage, and extracts only the point cloud for the pattern area among numerous point clouds. This is because overload may occur if all of the numerous point clouds, which exceed approximately 40 million, are used.

To do this, look at the point cloud for the entire space from the top and create the equation of a second-order straight line, 'y>ax+b', so that the section where the pattern is fired belongs to.

At this time, two coordinates are selected to create an equation, and each coordinate can be appropriately selected to include the pattern area, which is the user's area of interest.

Then, using the (x_1,y_1) and (x_2,y_2) values of the selected coordinates, a and b are obtained through the formula y-y_1=(y_2-y_1)/(x_2-x_1)·(x-x_1). The point cloud extracted using the formula is the data to be used in the subsequent process.

Next, in step (a-3), each pixel constituting the pattern of light irradiated through the beam projector 10 in step (a-1) is combined with the LIDAR ( This is the process of deriving a standard relational expression by formulating the relationship between the rays irradiated in 20).

And this step (a-3) may include a plurality of processes in detail.

Figure 4 is a diagram showing the detailed process of step (a-3) in the projection mapping method through a calibration process using depth RGB data according to an embodiment of the present invention.

As shown in Figure 4, step (a-3) of the projection mapping method through a calibration process using depth RGB data according to an embodiment of the present invention is in detail steps (a-3-1) to (a). -May include steps 3-4).

Step (a-3-1) is the process of converting the point cloud extracted in step (a-2) into a 2D image.

Through this step, when you select a point in a 2D image, you can find out where that part is in 3D coordinates.

In particular, point clouds are expressed in three-dimensional coordinates and are made up of space. In order to image detect a point cloud like this, 3D must be mapped into a 2D image.

In this embodiment, step (a-3-1) can be done by using a spherical coordinate system in the process of converting the point cloud extracted in step (a-2) into a 2D image.

This is because the LiDAR 20 performs measurements while rotating, and the image captured by the camera inside the scanner of the LiDAR 20 uses equirectangular projection, so projection is performed in a spherical coordinate system. It is intuitive to do and accurate mapping is possible.

Next, step (a-3-2) is the process of calculating center values for the patterns that appear in the 2D image converted by step (a-3-1).

In this process, in a 2D image converted to a spherical coordinate system formula, the center value for the patterns appearing in the 2D image is calculated using the Harris Corner Detect method for each point.

And step (a-3-3) is a process of deriving a 3D position value set including a plurality of 3D position values through the center values calculated in step (a-3-2).

As described above, in this embodiment, in order to perform step (a) twice, the screen, which is the projector 15, was installed in two places at a certain interval, so a total of two sets of three-dimensional position values can be created. .

Next, in step (a-3-4), a process of deriving a 3D straight line equation is performed by combining a plurality of 3D position values derived in step (a-3-3).

Based on the three-dimensional straight line equation obtained through this process, 2,073,600 line segments can be found through interpolation between each line segment. The reason why interpolation is necessary is to find the line segments of the rays for all pixels of the projector since only the information of some rays is known.

Once all the line segments are known, during the subsequent projection mapping process, you will be able to know which line segment the 3D value of the desired target touches, allowing simple and accurate mapping. Also, the above pattern can be used as is when interpolating.

Through the above process, the process of formulating the relationship between each pixel constituting the pattern of light irradiated through the beam projector 10 and the ray irradiated from the LIDAR 20 including the RGB camera to derive a standard relational expression is performed. It comes true.

Afterwards, in step (b), a process of extracting a point cloud for the actual target object (1) is performed through the LIDAR (20).

In this process, the actual object 1 on which projection mapping will be performed is prepared, and the point cloud for the target object 1 is extracted by scanning it through the LIDAR 20.

And in step (c), based on the reference relational expression derived in step (a), the point cloud of the target object 1 extracted in step (b) through the processor of the computer is irradiated through the beam projector 10. This is the process of deriving a projection equation that formalizes the relationship between each pixel included in the light pattern.

In this process, through the three-dimensional coordinates of the target object (1) extracted through the lidar (20) in step (b) described above, it is determined which ray of the beam projector (10) each corresponding point falls on, or which Find the closest distance to the ray of light.

In other words, this process is a process of obtaining a 2D pixel value after determining which of a plurality of vectors has the closest distance between the 3D coordinates obtained from the target object (1) for which projection mapping is desired.

Next, step (d) is a process of performing projection mapping on the virtual object 2 through the beam projector 10 based on the projection equation derived in step (c).

As described above, the present invention can formalize the relationship between all pixels of light emitted from the beam projector 10 by utilizing the depth RGB data obtained using the LIDAR 20, so the lens of the beam projector 10 It is free from distortion, and therefore error-free projection mapping is possible even when the target object (1) to be projection mapped has a large scale or is located at a long distance.

Accordingly, the present invention can be used for large-scale projection mapping used in performances, exhibitions, and advertisements. In particular, when using a device that widens the angle of light, such as a convex lens, a larger area can be covered with only a small number of beam projectors (10). Not only that, but it also has the advantage of being able to dramatically reduce the calibration process itself, which previously took 10 to 20 times, to just 2 times.

As described above, the preferred embodiments according to the present invention have been examined, and the fact that the present invention can be embodied in other specific forms in addition to the embodiments described above without departing from the spirit or scope thereof is recognized by those skilled in the art. It is self-evident to them. Therefore, the above-described embodiments are to be regarded as illustrative and not restrictive, and thus the present invention is not limited to the above description but may be modified within the scope of the appended claims and their equivalents.

1: Target object
2: Virtual object
10: Beam projector
15: Projectile
20: LIDAR

Claims (7)

Step (a) of deriving a standard relational expression by formulating the relationship between each pixel constituting the pattern of light irradiated through a beam projector and the ray irradiated from a LIDAR including an RGB camera;
Step (b) of extracting a point cloud for the actual target object through the LIDAR;
Based on the reference relationship derived in step (a), each pixel included in the pattern of light emitted through the beam projector from the point cloud of the target object extracted in step (b) through the processor of the computer Step (c) of deriving a projection equation that formalizes the relationship; and
Step (d) of performing projection mapping on the virtual object through the beam projector based on the projection equation derived in step (c);
Includes,

In step (a),
Step (a-1) of irradiating a preset light pattern to an arbitrary projectile through the beam projector;
Step (a-2) of scanning the pattern of light irradiated through the beam projector in step (a-1) through the LIDAR and extracting a point cloud for each pixel constituting the light pattern; and
Through the computer's processor, the relationship between each pixel constituting the pattern of light irradiated through the beam projector in step (a-1) and the ray irradiated from the LIDAR in step (a-2) is formulated to form a reference relational expression. Deriving step (a-3);
Including,

In step (a-3),
Step (a-3-1) of converting the point cloud extracted in step (a-2) into a 2D image;
Step (a-3-2) of calculating center values for patterns appearing in the 2D image converted by step (a-3-1);
Step (a-3-3) of deriving a 3D position value set including a plurality of 3D position values through the center values calculated in step (a-3-2); and
Step (a-3-4) of deriving a three-dimensional straight line equation by combining a plurality of three-dimensional position values derived in step (a-3-3);
Including,
Projection mapping method through a calibration process using depth RGB data.
According to paragraph 1,
Step (a) is performed two or more times before step (b) is performed,
Projection mapping method through a calibration process using depth RGB data. delete According to paragraph 1,
In step (a-2),
Extracting only the point cloud for the pattern area of light irradiated through the beam projector from the entire area scanned through the LIDAR,
Projection mapping method through a calibration process using depth RGB data.
delete According to paragraph 1,
The step (a-3-1) uses a spherical coordinate system in the process of converting the point cloud extracted in step (a-2) into a 2D image.
Projection mapping method through a calibration process using depth RGB data. According to paragraph 1,
The step (a-3-2) calculates the center value for the patterns appearing in the 2D image using the Harris Corner Detect method.
Projection mapping method through a calibration process using depth RGB data.