KR101409619B1 - Geometric Correction Apparatus and Method based on Recursive Bezier Patch Sub-Division - Google Patents

Abstract

A geometry correction device and method based on regression Bezier patch partitioning is disclosed. According to an embodiment of the present invention, there is provided a geometric correction method including: receiving a first image of a black screen projected by a projector on a projection surface from a camera; Receiving a second image of the predetermined pattern projected on the projection surface by the projector from the camera; Subtracting the first image from the second image to generate a third image; And performing a geometric correction on the predetermined pattern to correct a distortion between the preset pattern and the third image.

Description

[0001] The present invention relates to a geometric correction apparatus and a method based on a regression Bezier patch division,

Embodiments of the present invention relate to a geometry correction apparatus and method, and more particularly to a geometry correction apparatus and method based on a regression Bezier patch division.

Recently, with the emergence of projection technology, immersive projection environments have been established. Immersive displays are becoming a future technology for high-definition media content technology.

The goal of wide viewing angle and high resolution media content is to deliver the ultimate immersive experience by conveying the 'presence' feeling to the user. On the other hand, a large, flat surface, such as a theater's large screen, can provide a good immersion experience, but higher order surfaces such as curved screens, dome shaped screens, The ability to create a visual environment can provide a more immersive experience.

For example, to provide the user with a more immersive experience, a non-flat projection surface 120 may be provided as shown in FIG. That is, the projector 110 can project the media content onto the curved projection surface 120 without being flat. However, in this case, since the projection surface 120 is not flat, the image 130 projected on the projection surface 120 may be displayed differently from the original image. That is, the image 130 projected onto the projection surface 120 may be distorted.

Therefore, there is a need for research on a geometrical correction technique for directly viewing the contents projected on such a non-flat projection surface.

According to an embodiment of the present invention, there is provided a geometric correction method including: receiving a first image of a black screen projected by a projector on a projection surface from a camera; Receiving a second image of the predetermined pattern projected on the projection surface by the projector from the camera; Subtracting the first image from the second image to generate a third image; And performing a geometric correction on the predetermined pattern to correct a distortion between the preset pattern and the third image.

According to an aspect of the present invention, performing the geometric correction includes extracting a feature point from the third image; Moving the extracted minutiae to the domain of the predetermined pattern projected through the projector from the domain of the third image photographed through the camera; Applying a Bezier transform to the third image based on the moved minutiae; And performing the geometric correction by applying the Bezier transformation recursively.

A geometry correction apparatus according to an embodiment of the present invention includes a projector for receiving a first image of a black screen projected on a projection surface from a camera and projecting a second pattern A receiving unit for receiving an image from the camera; A generator for subtracting the first image from the second image to generate a third image from which undesired artifacts have been removed; And a processor for performing a geometric correction on the preset pattern to correct a distortion between the preset pattern and the third image.

By photographing an image of a predetermined pattern projected on a non-flat projection surface through a camera and performing a Bezier transformation on the captured image, the contents projected on a non-flat projection surface are not distorted, .

Also, according to embodiments of the present invention, geometric correction can be performed without camera calibration or extraction of intrinsic or extrinsic parameters. In addition, geometric correction can be automatically performed without requiring a person to take any action to perform the geometric correction.

Figure 1 is a view of a projector that projects media content onto a non-flat projection surface.
Figure 2 is a diagram showing a rectangular surface with nine data points.
3 and 4 are views showing a Bezier patch according to an embodiment of the present invention.
5A is a flowchart illustrating a geometric correction method according to an embodiment of the present invention.
5B is a diagram illustrating an operation of receiving an image from a camera according to an exemplary embodiment of the present invention.
6 is a diagram illustrating an operation for performing a geometric correction by the geometric correction method according to an embodiment of the present invention.
7 is a diagram illustrating an operation of executing a runtime application according to an embodiment of the present invention.
8 is a diagram illustrating a configuration of a geometric correction apparatus according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. Like reference symbols in the drawings denote like elements.

The geometric correction method according to an exemplary embodiment of the present invention uses a Recursive Bezier Patch Sub-Division to perform geometric correction on a non-flat quadric projection surface, Can be performed.

To explain the geometric correction method according to the present invention, a Bezier patch will be described first.

Bezier surfaces can be of the kind of polynomial surfaces that use Bernstein polynomials to create a 2D surface using a separate control point. In this case, the position of each control point can determine the shape and curvature of the Bezier surface.

The complete rectangular surface is a 2nd degree Bezier surface with nine control points-nine control points are four corner control points, four edge control points, and one control point. (Center control point). At this time, when any one of the plurality of control points is moved from the initial position, the surface may be modified correspondingly. Also, a surface point near the control point may be subjected to a stronger force.

The secondary Bézier surface on the (u, v) -plane of the (n, m) order is composed of two mutually orthogonal parametric curves, B (u) and B (v) And can be defined as a control point of (n + 1) (m + 1).

Each Bézier parameter value is determined within a range between 0 and 1, which can be expressed as in Equation 1.

Equation 1

이고,

와

은 u 매개축(u parametric axis) 및 v 매개축(v parametric axis) 상의 베지어 곡선을 나타내며,

는 제어점들의 세트(set)를 나타낸다.At this time,

ego,

Wow

Represents a Bezier curve on a u-parametric axis and v-parametric axis,

Represents a set of control points.

The geometry correction apparatus and method according to an embodiment of the present invention can utilize the following three characteristics of the Bézier surface. The three characteristics are: (1) the bezier surface is located inside the convex hull of the control points, (2) the bezier surface point at the corner is located just below the corresponding control point (This property does not apply to control points other than the control points in the corner), (3) a point at which a Bezier curve can be obtained if any one of the intermediate axis values has a constant value Thus, the edge of the Bezier surface can be seen as a Bezier curve in which one of the intermediate axes has a value of 0 or 1).

Figure 2 is a diagram showing a rectangular surface with nine data points.

Referring to FIG. 2, a rectangular grid 210 may include nine data points. The nine data points may include four corner data points, four edge data points, and one center data point. At this time, if the square grid 210 is distorted by a second order transformation, it can be deformed like the distorted rectangular surface 220.

On the other hand, a second order Bezier surface can be fitted to the data points on the secondary surface by estimating the position of the control points, given the set of data points on the two-dimensional surface. For example, the quadrangle 310 represents a quadratic Bezier surface, and the distorted quadrangle 320 represents a deformed quadrangle by moving the control points of the secondary Bezier surface.

The position of the control points can be estimated through the same method as in Equation 2

Equation 2

Where t represents the Bezier parameter and B (t) represents the position of a previously known data point on the surface. Referring to FIG. 3, the corner control points of P0 311, P2 313, P6 and P8 coincide with the four corner data points.

On the other hand, since the edge control point P1 312 is located between P0 311 and P2 313, the t value in equation (2) is 0.5, and B (t) is the edge data point corresponding to P1 As shown in FIG.

Similarly, the positions of the edge control points of P3, P5 and P7 can be calculated using the corresponding data points.

In addition, the face control point of P4 314 can be estimated by repeatedly adjusting the center pixel of the estimated Bézier surface to match the center of the image.

In the case of surfaces with high dimensional distortion, more control points are required for mapping and compensating surfaces. However, this results in increased mathematical complexity, which raises the computation time. However, the geometric correction method according to an embodiment of the present invention can obtain a level of surface mapping similar to a high dimensional level without requiring a high level.

That is, the geometric correction method according to an embodiment of the present invention can start from a smooth surface using sparse initial known points. And, to obtain high accuracy of the surface mesh, a more dense estimate of the surface on the secondary plane can be regressively generated.

More specifically, all Bezier patches can be defined using a secondary Bezier surface with nine control points. The first level of sub-division can be obtained by dividing each of the four Bezier patches on the secondary basal surface into four partial patches (sub-patches) or quadrants. At this time, each of the divided partial patches has nine control points (four corner control points, four edge control points, and one center control point). Referring to FIG. 4, the rectangle 410 represents a base level patch in the first level division state. On the other hand, as shown in the distorted rectangle 420, high control can be provided in the case of a high level division. That is, the dotted line 430 shown in FIG. 4 shows the deformation in the 0 level split state, which shows a lower control when compared to the outline of the distorted rectangle 420.

In the second level division, similarly to the above-described operation, each of the Bezier patches in the first level division state is divided again into four partial patches, thereby having 16 Bezier surface patches.

At this time, the control point at the boundary between the two partial patches can operate as a common control point commonly applied to both partial patches. Therefore, the edge control point in the i-th level split state becomes the corner control point in the (i + 1) -th level split state.

Hereinafter, the operation of the geometric correction method according to an embodiment of the present invention will be described with reference to FIGS. 5A and 5B. FIG.

The geometric correction method according to an embodiment of the present invention can be applied to the offline calibration step and the online rendering step. At this time, the geometric correction method may be applied once during the offline correction step, and may be applied every frame during the online rendering step.

The camera 530 may capture the content projected onto the projection surface 510 (Projection Surface). The projection surface 510 may be a flat, non-flat, uneven quadric projection surface. Therefore, when content without geometry correction is projected onto the projection surface 510, the projected content may be distorted and projected in a different shape from the original shape of the content. Therefore, the geometric correction method according to an embodiment of the present invention can estimate a deformation occurring in a projected pattern due to a surface using a regressive Bezier patch division.

A projector 520 may be arbitrarily placed in front of the projection surface 510 within a surface boundary. In some embodiments, a plurality of projectors 520 may be provided and provided as a multi projector. That is, the geometric correction method according to an embodiment of the present invention can be equally applied to a multi-projector environment.

In addition, the camera 530 may be placed in a position that may include all areas of the projection surface 510. [

When the camera 530 and the projector 520 are set as described above, the projector 520 can project a black screen onto the projection surface 510. [ At this time, the camera 530 can take a black screen projected on the projection surface 510 by the projector 520, and can transmit the first image 531, which has taken a black screen, to the geometry correction apparatus. Also, the geometry correction method may receive the first image 531 from the camera 530 (501).

The first image 531 of the black screen may be used to remove environmental effects, surface effects, and camera effects. At this time, for example, the environmental influence may be ambient light by another light source. Also, the surface effect may be a black point on the projection surface 510 that may cause a feature point detection failure. Also, the camera effect can be a hue spread and a defective pixel.

When the reception of the first image 531 on the black screen is completed, the projector 520 can project a predetermined pattern onto the projection surface 510. [ The predetermined pattern according to an exemplary embodiment may be a rectangular grid consisting of points. At this time, the number of points forming the square lattice can be selected based on the level of Bezier division.

The camera 530 can take a predetermined pattern projected on the projection surface 510 by the projector 520 and transmit the second image 532 of the preset pattern to the geometry correction device. The geometry correction method may receive the second image 532 from the camera 530 (502).

When the first image 531 and the second image 532 are received from the camera 530, the geometric correction method subtracts the first image 531 from the second image 532, A third image 533 from which unwanted artifacts have been removed may be generated 503.

In the geometric correction method, a geometric correction may be performed on a preset pattern in order to correct the distortion between the preset pattern and the third image 533 (504).

Hereinafter, with reference to FIG. 6, an operation of performing a geometric correction on a predetermined pattern to correct distortion according to an embodiment of the present invention will be described in detail.

The geometry correction method may extract feature points from the third image to perform the geometric correction (610). In some embodiments, the geometry correction method may extract feature points from the third image using Harris corner detection.

In addition, the geometric correction method may move 620 the extracted minutiae to the domain of the predetermined pattern projected through the projector from the domain for the third image photographed through the camera.

More specifically, the geometric correction method may relate the third image photographed through the camera to a predetermined pattern projected through the projector. That is, due to the shape of the projection surface, distortion occurs between the preset pattern and the third image, and therefore, the geometric correction method is required to perform an operation of associating the two images with each other.

For example, assuming that a predetermined pattern projected by the projector is in the domain P (x, y) and the third image captured by the camera is in the domain C (x, y) In order to associate the third image with a predetermined pattern on the P (x, y) domain, the geometric correction method may calculate homography. According to an embodiment, the geometric correction method is a method in which four corner feature points (which may be bezier corner control points) of a third image are corresponded to four corner points of a preset pattern It is possible to calculate the homography through the above method.

In addition, when the homography is calculated, the geometric correction method can move the extracted minutiae to the domain for the preset pattern from the domain for the third image using the calculated homography. That is, the geometric correction method can move all the feature points extracted for the third image to the domain for the predetermined pattern by using the homography.

Once all minutiae are moved, the geometric correction method can apply a Bezier transformation to the third image based on the shifted minutiae. In addition, the geometric correction method can perform geometric correction by applying the Bezier transformation recursively.

The geometry correction method according to one aspect of the present invention can set the entire third image as a single second order Bezier patch in order to apply Bezier transform to the third image. At this time, the secondary Bezier patch may include four corner feature points operating as four corner control points.

The geometry correction method can calculate the position of the control point for the secondary Bezier patch based on the moved minutiae (630). According to an embodiment, the geometric correction method first computes the position of a corner control point for a secondary Bezier patch, and calculates an edge control point and a center control point based on the calculated position of the corner control point, Respectively.

In order to correct the distortion (second-order distortion, 2nd degree distortion) due to the quadric surface when all of the positions of the control points are computed, the geometric correction method is used to perform a Bezier transformation Bezier transformation can be applied (640).

Also, in the case of a plane with geometric distortions that can not be corrected using a 2nd degree Bezier approximation, the geometric correction method is divided into four partial patches Operation can be performed. At this time, each partial patch has nine control points. The geometry correction method also calculates the position of the control points for each partial patch (650) and applies the Bezier transform to all feature points in the partial patch, Correction may be performed (660). At this time, ith level sub-patches generated by performing the dividing operation i times can be treated as independent secondary surfaces.

According to one aspect of the present invention, the geometric correction method can also perform (670, 680) a segmentation operation on a small portion of the projection surface (670, 680) and thus the segmentation level can be increased.

Also, when applying Bezier transform to all the feature points in the partial patch, the geometry correction method can apply Bezier transform in the reverse order of the dividing operation. For example, the geometry correction method may apply Bezier transform first for a partial patch with the highest split level, then sequentially apply Bezier transform from the next split level to the lowest split level.

According to one aspect of the present invention, the geometric correction method can record information on a partial patch to a look-up table while performing an operation of applying a Bezier application recursively. The recorded look-up table can then be used to execute a run time application.

7, the geometric correction method according to an exemplary embodiment uses a look-up table so that the contents 710 projected on the projection surface by the projector 730 can be projected without distortion on the projection surface A runtime application can be executed using OpenGL (Open Graphics Library) which warp each frame of the content. The frame 720 represents a twisted frame to which a look-up table is applied so that it is not distorted at the projection surface.

8 is a diagram illustrating a configuration of a geometric correction apparatus according to an embodiment of the present invention.

Referring to FIG. 8, a geometry correction apparatus 800 according to an embodiment of the present invention may include a receiving unit 810, a generating unit 820, and a processing unit 830.

The receiving unit 810 can receive the first image from the camera.

A first image of a black screen can be used to eliminate environmental, surface, and camera effects. At this time, for example, the environmental influence may be ambient light by another light source. Also, the surface effect may be a black dot on the projection surface, which may cause a feature point detection failure. Also, the camera effect can be a hue spread and a defective pixel.

When the reception of the first image for the black screen is completed, the projector can project a predetermined pattern onto the projection surface. The predetermined pattern according to an exemplary embodiment may be a rectangular grid consisting of points. At this time, the number of points forming the square lattice can be selected based on the level of Bezier division.

The camera can take a predetermined pattern projected by the projector onto the projection surface and transmit the second image of the preset pattern to the geometry correction device 800. [ The receiving unit 810 can receive the second image from the camera.

Upon receiving the first image and the second image from the camera, the generating unit 820 subtracts the first image from the second image, thereby generating a third image from which unwanted artifacts have been removed Can be generated.

In addition, the processing unit 830 may perform geometric correction on a preset pattern in order to correct the distortion between the preset pattern and the third image.

Hereinafter, the operation of the processing unit 830 to perform the geometric correction on a predetermined pattern for correcting the distortion will be described in detail.

The processing unit 830 may extract feature points from the third image to perform geometric correction. In some embodiments, the processing unit 830 may extract feature points from the third image using Harris corner detection.

In addition, the processing unit 830 can move the extracted minutiae to the domain of the preset pattern projected through the projector from the domain for the third image photographed through the camera.

In detail, the processing unit 830 may relate the third image photographed through the camera to a predetermined pattern projected through the projector. That is, due to the shape of the projection surface, distortion occurs between the preset pattern and the third image, so it is required that the processing unit 830 perform an operation of associating the two images.

For example, assuming that a predetermined pattern projected by the projector is in the domain P (x, y) and the third image captured by the camera is in the domain C (x, y) In order to associate the third image with a predetermined pattern on the P (x, y) domain, the processing unit 830 may calculate homography. According to an embodiment, the processing unit 830 may include four corner feature points (which may be bezier corner control points) of the third image, corresponding to four corner points of a predetermined pattern It is possible to calculate the homography through the method.

In addition, when the homography is calculated, the processing unit 830 can use the computed homography to move the extracted minutiae to the domain for the preset pattern from the domain for the third image. That is, the processing unit 830 can move all the feature points extracted for the third image to the domain for the predetermined pattern by using the homography.

When all of the minutiae are moved, the processing unit 830 may apply the Bezier transformation to the third image based on the minutiae that have been moved. In addition, the processing unit 830 can perform geometric correction by applying a Bezier transformation recursively.

The processing unit 830 according to an aspect of the present invention may set the entire third image to be a single second order Bezier patch in order to apply the Bezier transformation to the third image. At this time, the secondary Bezier patch may include four corner feature points operating as four corner control points.

The processing unit 830 can calculate the position of the control point for the secondary Bezier patch based on the moved minutiae. According to an embodiment, the processing unit 830 first computes the position of the corner control point for the secondary Bezier patch and calculates an edge control point and a center control point based on the calculated position of the corner control point ) Can be calculated, respectively.

When all of the positions of the control points are calculated, the processing unit 830 performs a Bezier transformation on all the feature points in the secondary Bezier patch to correct the distortion (second-order distortion, 2nd degree distortion) due to the quadric surface, (Bezier transformation) can be applied.

Also, in the case of a plane having geometric distortions that can not be corrected using a 2nd degree Bezier approximation, the processing unit 830 divides the secondary Bezier patch into four partial patches It is possible to perform the dividing operation. At this time, each partial patch has nine control points. In addition, the processing unit 830 calculates the position of the control point for each partial patch, applies a Bezier transformation to all the feature points in the partial patch, and applies a Bezier transformation to the geometric correction for the preset pattern Can be performed. At this time, ith level sub-patches generated by performing the dividing operation i times can be treated as independent secondary surfaces.

According to one aspect of the present invention, the processing unit 830 can perform the dividing operation recursively on a small portion of the projection surface, and therefore, the dividing level can be increased.

Also, when applying Bezier transform to all the feature points in the partial patch, the processing unit 830 can apply Bezier transform in the reverse order of the order of performing the dividing operation. For example, the processing unit 830 may first apply a Bezier transformation to a partial patch having the highest division level, and then sequentially apply a Bezier transformation from the division level to the lowest division level.

According to one aspect of the present invention, the processing unit 830 may write information on the partial patch to the look-up table while performing an operation of applying the Bezier application recursively. The recorded look-up table can then be used to execute a run time application.

In other words, the processing unit 830 according to an exemplary embodiment may use a look-up table to twist every frame of the content so that the content that the projector projects on the projection surface may be projected without distortion on the projection surface (warp) You can run a run time application using OpenGL (Open Graphics Library).

Embodiments according to the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as a hard disk, a floppy disk, and a magnetic tape; optical media such as CD-ROM and DVD; magnetic recording media such as a floppy disk; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

110: PROJECTOR
120: projection surface
130: Projected image

Claims (10)

Receiving a first image of a black screen projected by a projector on a projection surface from a camera;
Receiving a second image of the predetermined pattern projected on the projection surface by the projector from the camera;
Subtracting the first image from the second image to generate a third image; And
And performing a geometric correction on the predetermined pattern to correct distortion between the preset pattern and the third image,
Wherein the step of performing the geometric correction comprises:
The entirety of the third image is set as one secondary Bézier patch, the secondary Bézier patch is divided into four partial patches, and the control points for each of the four partial patches are divided into a rectangular grid . ≪ / RTI > The method according to claim 1,
The step of performing the geometric correction
Extracting feature points from the third image;
Moving the extracted minutiae to a domain for the predetermined pattern projected through the projector from the domain for the third image photographed through the camera;
Applying a Bezier transform to the third image based on the moved minutiae; And
Performing the geometric correction by applying the Bezier transformation recursively
/ RTI > 3. The method of claim 2,
The step of extracting the feature points
And extracting the feature points using Harris corner detection. 3. The method of claim 2,
The step of moving the extracted feature points
Calculating homomorphisms of four corners of the third image through four corner points of the predetermined pattern; And
Moving the extracted feature point from the domain for the third image to the domain for the predetermined pattern using the computed homography
/ RTI > 3. The method of claim 2,
Wherein applying the Bezier transform to the third image comprises:
Calculating a position of a control point for the set secondary Bezier patch based on the moved minutiae; And
Applying the Bezier transform to all feature points in the secondary Bezier patch
/ RTI > 6. The method of claim 5,
The step of applying the Bezier transformation recursively to perform the geometric correction comprises:
Calculating a position of a control point for each of the four partial patches of the divided secondary Bézier patch; And
Applying the Bezier transform to all the specific points within each of the four partial patches
/ RTI > The method according to claim 6,
Applying the Bezier transform to all the particular points within each of the four partial patches
And applying the Bezier transform in an order reverse to the order of performing the dividing operation. The method according to claim 6,
Writing information on the partial patch to a look-up table
The geometric correction method further comprising: 9. The method of claim 8,
Wherein the step of performing the geometric correction comprises:
(Open Graphics Library) is applied to every frame of the content on the basis of the look-up table so that the content projected on the projection surface by the projector can be projected without distortion on the projection surface step
The geometric correction method further comprising: A receiving unit that receives a first image of a black screen projected by a projector on a projection surface from a camera and receives a second image of the preset pattern projected by the projector on the projection surface from the camera;
A generator for subtracting the first image from the second image and generating a third image from which the artifact is removed; And
And a processor for performing a geometric correction on the predetermined pattern to correct a distortion between the preset pattern and the third image,
Wherein,
The entirety of the third image is set as one secondary Bézier patch, the secondary Bézier patch is divided into four partial patches, and the control points for each of the four partial patches are divided into a quadrangle A geometry correction device formed in a lattice.

Images