KR101031062B1 - Method for projecting video data for materializing mixed reality - Google Patents

Abstract

PURPOSE: A method for projecting video data for materializing mixed reality is provided to materialize mixed reality with improved quality by minimizing the distortion of video projected to a three-dimensional background. CONSTITUTION: A method for projecting video data for materializing mixed reality is as follows. A three-dimensional background that video data will be projected is modeled(S100). The image of the three-dimensional background is obtained by projecting a black and white image to the three-dimensional background(S110). The obtained image is projected to the modeled three-dimensional background(S120). The three-dimensional coordinate of the three-dimensional background is corrected using the shape and brightness of the black and white pattern of the obtained image(S130).

Description

Image Projection Method for Realizing Complex Reality {METHOD FOR PROJECTING VIDEO DATA FOR MATERIALIZING MIXED REALITY}

The present invention relates to an image projection method for realizing a composite reality, and more particularly, to an image projection method for minimizing distortion of an image by correcting an image projected on a stereoscopic background.

Recently, a mixed reality technology for synthesizing real-life images and computer graphics images in three dimensions has been in the spotlight. In general, the composite reality technology is implemented by synthesizing a photorealistic background to a three-dimensional object by computer graphics, or by combining a photorealistic object into a virtual space by computer graphics.

The system for realizing such a composite reality is expanding its target area. Recently, the system has been separated from the real concept of composite reality in which real-life images are taken and synthesized with virtual objects or backgrounds on a computer. The subject area has been expanded to the concept of broad projection of computer graphics actively controlled by movement or characteristics on a real background.

In particular, a complex reality can be realized by projecting an image onto a three-dimensional real background such as a structure. For example, a complex reality in which real-life and computer graphics are combined by projecting the image so that the computer graphic of the completed structure overlaps the real structure during construction. Can be implemented.

However, for this purpose, a desired image must be projected without distortion in a building having a three-dimensional structure. As shown in Figure 1, the building or the structure is not only the outer surface of the structure is composed of three-dimensional, but also the color of each surface is not constant.

In order to project an image on such a stereoscopic background, an appropriate number of projection apparatuses are used according to the size and shape of a building or structure. However, it is very difficult to allow an image to be displayed in a complex geometric object or a real space with colors or textures without distortion of color, brightness, shape, or the like.

Accordingly, the present invention has been made to solve the above problems, an object of the present invention is to provide an image projection method for realizing a composite reality that can minimize the distortion of the image projected on a three-dimensional background.

Another object of the present invention is to provide an image projection method for realizing a composite reality that can easily grasp the characteristics of the three-dimensional background.

It is still another object of the present invention to provide an image projection method for realizing a composite reality that can perform correction of image data to be projected according to the characteristics of a stereoscopic background.

The present invention having the above object comprises the steps of: (A) three-dimensional modeling the three-dimensional background to project the image data; (B) projecting a black and white pattern image onto the three-dimensional background, and then obtaining an image of the three-dimensional background on which the black and white pattern image is projected; (C) projecting the image obtained in the step (B) to the three-dimensional background model modeled in the step (A); (D) correcting three-dimensional coordinates of the feature points of the three-dimensional background model by using the shape and brightness of the black and white pattern of the image obtained in the step (B); (E) calculating position and direction information of each triangular plane composed of three adjacent feature points by using three-dimensional coordinates of the feature points corrected in step (D); And (F) determining the pixel of the image data to be projected on each triangle plane by using the position and direction information of the triangle plane calculated in the step (E), and correcting the contrast of the pixel to be projected on each triangle plane. It is carried out including.

In this case, the image projection method may include: (L) projecting a color pattern image onto the stereoscopic background, and then obtaining an image of the stereoscopic background onto which the color pattern image is projected; (M) calculating each color information of the triangular plane calculated in the step (E) by using the image obtained in the step (L); And (N) correcting the colors of the pixels determined to be projected on the respective triangular planes by using the color information of the triangular planes calculated in the step (M).

The image projection method may further include projecting the image data corrected in steps (F) and (N) onto the stereoscopic background.

Further, step (F) is performed by adding or subtracting the contrast correction value of the pixel to be projected onto each triangle plane calculated using the position and the direction information of the triangle plane to the contrast value of each pixel of the image data before correction. Step N) may be performed by adding or subtracting the color correction value of the pixel to be projected onto each triangle plane calculated using the color information of the triangle plane to the color value of each pixel of the image data before the correction.

Further, in the steps (F) and (N), when the image data to be projected on the stereoscopic background is moving image data, the contrast correction value and the color correction value for each pixel are included in each image included in the moving image data. It may also be performed by adding or subtracting the contrast values and the color values of all the pixels at the same position of the frame.

Steps (B) and (L) may be performed at the same angle and position as the position and angle at which the corrected image data is projected.

Wherein (B) to (F) and (L) to (M) are individual for each point where the corrected image data is projected when the corrected image data is projected at a plurality of points with respect to the stereoscopic background. It may be performed as.

The image projection method may further include performing an edge blending process for gradually mixing the images with respect to portions where the image data projected from the plurality of points overlap each other.

As described in detail above, according to the image projection method for realizing the composite reality according to the present invention, the following effects can be expected.

That is, by minimizing the distortion of the image projected on the three-dimensional background, there is an advantage that can realize a higher level of complex reality.

In addition, the image projection method for realizing the complex reality according to the present invention has the advantage of reducing the time, labor and cost required for the preparation process for image projection by easily grasping the characteristics of the three-dimensional background to correct the image to be projected .

1 is an exemplary view showing an example of projecting an image on a stereoscopic background using an image projection system.
Figure 2a is a block diagram showing the configuration of an image projection system according to an embodiment of the present invention.
Fig. 2B is a block diagram showing the configuration of an image projection system according to another embodiment of the present invention.
3 is a flowchart illustrating a method of projecting an image onto a stereoscopic background according to an exemplary embodiment of the present invention.
4A is an exemplary view of projecting a pattern image on a three-dimensional background.
Figure 4b is an exemplary view showing a change in brightness according to the projection distance of the pattern image.
5 is an exemplary view of projecting an image corrected according to the color of a three-dimensional background.

Hereinafter, an image projection system and an image projection method using the same according to the present invention will be described in detail with reference to the accompanying drawings.

Figure 2a is a block diagram showing the configuration of an image projection system according to an embodiment of the present invention, Figure 2b is a block diagram showing the configuration of an image projection system according to another embodiment of the present invention.

As shown in FIG. 2A, the image projection system 100 according to the exemplary embodiment of the present invention includes a control device 10 for controlling the operation of the entire system. The control device 10 projects an image on a stereoscopic background according to a user's command, and corrects the projected image to project the corrected image again.

The image data provided by the control device is transferred to the plurality of projection devices 14 through the distribution device 12. The distribution device 12 distributes the data allocated to each projection device 14 among the image data provided from the control device 10 to each projection device 14.

Accordingly, the projection device 14, which has been allocated data from the distribution device 12, outputs the image data allocated at each position as a beam and projects the stereoscopic background.

In this case, according to the projection device 14, a monitoring device 16 for monitoring the image projected on the stereoscopic background in real time may be separately installed in each projection device (14).

The projection apparatus 14 may be provided with a camera, respectively, in order to acquire an image in which the image is projected at the same angle of view as the beam output from the projection apparatus 14 in real time.

Accordingly, an image or a moving image photographed by a camera respectively installed in the projection device 14 is output to the monitoring device 16 to monitor whether image data is projected as desired on a stereoscopic background.

The user may check the image output through the monitoring device 16 and correct the projection angle of the projected image data, the color, brightness, etc. of the image data in real time through the control device 10.

On the other hand, the image projection system 100 according to another embodiment of the present invention receives the server 20 for providing the image data to be projected on the stereoscopic background and the image data provided from the server 20, as shown in Figure 2b And a control device 22 for transmitting to each projection device 24.

The control device 22 processes the received image data according to the position where each projection device 24 is installed and the projection angle of the beam, and transmits the received image data to the projection device 24. Accordingly, one control device 22 may be provided for each projection device 24.

In addition, each of the control device 22 may be provided with a monitoring device for monitoring an image or a video in which the image data projected by the projection device 24 is displayed on a stereoscopic background.

In addition, the projection device 24 may be provided with a camera for acquiring in real time an image of a stereoscopic background in which an image is projected at the same angle of view as the beam output from the projection device 14.

Accordingly, the control device 22 can check an image or a moving picture photographed by a camera respectively installed in the projection device 14.

In outputting image data to a stereoscopic background using the image projection system as described above, the colors, textures, brightness, etc. of the respective images output from the plurality of projection devices 14 and 24 and projected onto the stereoscopic background are unified with each other. In order to minimize distortion of the image due to the shape or color of the stereoscopic background, a process of preliminarily correcting the image data to be projected according to the attributes of the stereoscopic background should be performed before projecting the image data onto the stereoscopic background.

Hereinafter, a process of correcting image data and projecting it on a stereoscopic background will be described in detail with reference to the accompanying drawings.

3 is a flowchart illustrating a method of projecting an image onto a stereoscopic background according to an exemplary embodiment of the present invention. FIG. 4A is an exemplary view of projecting a pattern image onto a stereoscopic background. FIG. 4B is a diagram illustrating a projection distance of a pattern image. It is an exemplary diagram showing a change in brightness. 5 is an exemplary view of projecting an image corrected according to the color of the stereoscopic background.

As shown in FIG. 3, in the image projection method using an image projection system on a stereoscopic background, a first step of modeling a schematic shape of a stereoscopic background to project an image by using an application program for performing three-dimensional modeling such as CAD is performed. It starts from S100.

In step 100, only the external appearance of the rough three-dimensional background is modeled by using the design drawings of the three-dimensional background such as buildings and structures or the measured values.

Then, the projection apparatuses 14 and 24 are installed on the same position and angle as the position to project image data later, and then the pattern image is projected on the stereoscopic background through the projection apparatuses 14 and 24, so that the pattern image is In operation S110, an image of a projected stereoscopic background is obtained.

The pattern image projected through the projection apparatuses 14 and 24 in operation 110 may be, for example, a black and white pattern image including an image in which black and white squares cross each other in a checkerboard shape and an image inverted thereof, and red, green, and blue colors. A color pattern image including a pattern including a color is included.

The projection apparatuses 14 and 24 project the black and white pattern image and the color pattern image on the stereoscopic background a plurality of times, respectively, and acquire the image of the stereoscopic background on which each pattern image is projected. The reason for projecting the black and white pattern image and the color pattern image, respectively, is as follows. First, the reason for projecting the black and white pattern image is a camera that matches the position of the projection device 14 and the camera installed in the projection device 14, and adjusts the position value of the camera in the real space and the position of the virtual camera in the virtual space. To perform the calibration (Camera Calibration), and to know the x, y, z information on the area of the beam output from the projection device 14, and to find the brightness value of the area of the beam in the real space It is for. In this way, it is possible to correct the contrast of the pixel by determining the pixel of the image data to be projected on the stereoscopic background in a subsequent correction process. In addition, the reason for projecting the color pattern image is to obtain color information of the surface on which the image data is projected, so that the color, texture, brightness, etc. of the projected image are unified with each other. This makes it possible to correct the color of the pixels determined to be projected onto the stereoscopic background in a subsequent correction process.

In this case, in order to obtain an image of a stereoscopic background on which a pattern image is projected, the projection image is provided integrally or separately from the projection apparatuses 14 and 24 at the same position and angle as the beam projection positions and angles of the projection apparatuses 14 and 24. Use a camera device.

Accordingly, the images acquired in step 110 are all acquired at the same position and angle as the position at which the projection apparatuses 14 and 24 project the image data to be actually projected later on the stereoscopic background.

Then, in operation 120 (S120), the image acquired in operation 110 is projected onto the three-dimensional model of the three-dimensional modeled model in operation 100. Here, the image projected onto the three-dimensional modeled three-dimensional model is to sequentially project the image obtained by projecting the black and white pattern image and the image obtained by projecting the color pattern image.

First, the image obtained by projecting the black and white pattern image is projected. That is, in operation 120, the coordinates of the plurality of feature points of the three-dimensional modeled stereoscopic background model are corrected from the result of projecting the stereoscopic background image on which the black and white pattern image is projected onto the stereoscopic background model in step 120 (S130). ) Is performed.

The number of feature points may be appropriately selected by the user.

The more feature points are extracted from the three-dimensional background model, the more the three-dimensional background model is similar to the actual three-dimensional background, so that the quality of the projected image is higher. However, if too many feature points are extracted, the coordinates of each feature point are calculated, and the projection is performed accordingly. Since there is a disadvantage that the amount of data to be processed to correct the image to be increased, the number of feature points to be extracted can be appropriately selected by the user.

The feature points may not be intensively distributed at a specific position of the stereoscopic background model, but may be evenly distributed over the entire region of the stereoscopic background model, and the position may be selected by the user.

At this time, the coordinates are corrected by using the shape of the distorted pattern and the changed brightness value by projecting a constant black and white pattern image onto the three-dimensional background.

For example, as shown in FIG. 4A, when a rectangular pattern is projected onto a three-dimensional background, the projected pattern is distorted according to an angle formed by each surface of the three-dimensional background.

Therefore, an angle formed by each surface of the three-dimensional background on which the pattern is projected may be calculated by changing the position of the points included in each pattern.

Furthermore, as shown in FIG. 4B, even when the image of the same pattern is output as a beam, the brightness of the reflected pattern varies according to the distance of the three-dimensional background reflecting the same. The longer the distance, the lower the reflectivity and the darker the pattern is displayed.

Therefore, in operation 130, the image obtained by projecting the black and white pattern image is projected onto the three-dimensional modeled stereoscopic background model, and then the three-dimensional coordinates of the feature points are more accurately corrected by using the shape or brightness value of the pattern where each feature point is located.

In operation 140, the direction information of the triangular plane formed by three adjacent feature points is calculated by using three-dimensional coordinate values of the feature points corrected in operation 130.

Next, the image obtained by projecting the color pattern image is projected. That is, in step 150 (S150), the color of each triangular plane whose direction information is calculated in step 140 using an image obtained by projecting a color pattern image of the plurality of images acquired in step 110 onto a stereoscopic background. Calculate the information.

That is, the saturation of the red, green, and blue light of each triangular plane is calculated.

The color information is calculated in operation 150 to prevent the color from being distorted in the projected image due to the color difference of each surface when the stereoscopic background does not have one constant color.

For example, when the image is projected on the checkered stereoscopic background as shown in FIG. 5, the projected image and the pattern of the stereoscopic background are combined to obtain a desired image.

Therefore, color distortion of the image may be minimized by dividing the image to be projected into a plurality of units in advance to correct saturation.

Accordingly, the 160th step of correcting the brightness and the color of the image data to be actually projected on the three-dimensional background using the direction information of each triangular plane calculated in step 140 and the color information of each triangular plane calculated in step 150. S160 is performed.

In operation 160, the pixels to be projected on the respective triangular planes of the feature points of the image data to be projected are determined, and the brightness and the color of the pixels are corrected in consideration of angles and colors of the triangular planes.

That is, the pixels of the image data to be projected on the respective triangular planes are determined according to the positions and directions of the triangular planes calculated in operation 140, and the contrast and the color of the pixels determined to be projected on the triangular planes are corrected.

At this time, in determining the pixel of the image data to be projected on each triangular plane, the range of pixels to be projected on one triangular plane is determined according to the position of the triangular plane and the degree of inclination of the triangular plane.

For example, if one triangular plane is slightly inclined in a direction horizontal to the beam's projection direction, fewer pixels are projected into the triangle plane than pixels projected on the triangle plane perpendicular to the projection direction of the beam with the same width. It can be determined to be projected.

And in correcting the contrast and color of the pixels determined to be projected to each triangular plane, by adding or subtracting the contrast value and color value to be corrected to the original contrast value and color value of each pixel included in the image data, As described above, even when image data having a large data throughput is projected onto a stereoscopic background, the image data can be quickly corrected according to the characteristics of the stereoscopic background.

In operation S170, projection of the pattern image and projection of the image data corrected in operation 160 on the stereoscopic background is performed at the same position and angle as the acquired position and angle. In operation 170, a process of additionally correcting the projected image data in real time may be performed together.

According to an exemplary embodiment, various real-time effects may be added to the image data projected through the control devices 10 and 22 while projecting image data corrected on the stereoscopic background in operation 170.

In other words, if a specific effect is added to the three-dimensional modeled three-dimensional background model in real time, the effect may be reflected in the image data projected on the three-dimensional background.

For example, effects such as light and shadows being formed in a specific direction on a stereoscopic background, virtual objects appearing or moving by computer graphics, and effects such that images drawn in real time are projected on the stereoscopic background, etc. Various additional effects can be added.

Also, in correcting the image data in operation 160, edge blending using a gradation effect to gradually blend the images of the overlapped portions of the images projected by the plurality of projection devices 14 and 24. You can also do this. To this end, after projecting the image using the projection apparatuses 14 and 24 on the stereoscopic background in advance, the overlapping part is detected, and the contrast and the colors of the pixels to be projected to the overlapping part of the image must be corrected.

The rights of the present invention are not limited to the embodiments described above, but are defined by the claims, and those skilled in the art can make various modifications and adaptations within the scope of the claims. It is self-evident.

100: image projection system 10: controller
12: distribution device 14: projection device
16: monitoring device 20: server
22: control unit 24: projection unit

Claims (8)

(A) three-dimensional modeling of the three-dimensional background to project the image data;
(B) projecting a black and white pattern image onto the three-dimensional background, and then obtaining an image of the three-dimensional background on which the black and white pattern image is projected;
(C) projecting the image obtained in the step (B) to the three-dimensional background model modeled in the step (A);
(D) correcting three-dimensional coordinates of the feature points of the three-dimensional background model by using the shape and brightness of the black and white pattern of the image obtained in the step (B);
(E) calculating position and direction information of each triangular plane composed of three adjacent feature points by using three-dimensional coordinates of the feature points corrected in step (D); And
(F) determining the pixel of the image data to be projected on each triangular plane by using the position and direction information of the triangular plane calculated in step (E), and correcting the contrast of the pixel to be projected on each triangular plane. Image projection method comprising the. The method of claim 1,
The image projection method,
(L) projecting a color pattern image onto the stereoscopic background, and then obtaining an image of the stereoscopic background onto which the color pattern image is projected;
(M) calculating each color information of the triangular plane calculated in the step (E) by using the image obtained in the step (L); And
And (N) correcting the colors of the pixels determined to be projected on the respective triangular planes by using the color information of the triangular planes calculated in the step (M). Projection method. The method of claim 2,
The image projection method,
And projecting the image data corrected in steps (F) and (N) onto the stereoscopic background. The method of claim 3,
Step (F) is,
It is performed by adding or subtracting the contrast correction value of the pixel to be projected to each triangle plane calculated using the position and the direction information of the triangle plane to the contrast value of each pixel of the image data before correction,
Step (N),
And adding or subtracting a color correction value of a pixel to be projected onto each triangle plane by using color information of the triangle plane to a color value of each pixel of the image data before correction. The method of claim 4, wherein
Step (F) and step (N),
When the image data to be projected on the stereoscopic background is moving image data, the contrast correction value and the color correction value for each pixel are applied to the contrast values and the color values of all pixels at the same position of each image frame included in the moving image data. The image projection method characterized in that it is performed by adding and subtracting. The method of claim 3,
Step (B) and step (L),
And the corrected image data is performed at the same angle and position as the projected position and angle. The method of claim 6,
(B) to (F) and (L) to (M),
And when the corrected image data is projected at a plurality of points against the stereoscopic background, the corrected image data is individually performed for each point to be projected. The method of claim 7, wherein
The image projection method,
And performing an edge blending process for gradually mixing the images with respect to the portions where the image data projected from the plurality of points overlap each other.

Images