KR20220139139A - Device, method and computer program for correcting posture of camera based on image - Google Patents

Abstract

The present invention relates to a device for correcting a posture of a camera based on an image, comprising: a key frame setting unit setting a key frame based on two or more frames included in the image; a feature point extraction unit extracting a feature point from a target frame included in the image; an estimation unit estimating a posture value on the target frame based on the extracted feature points of the target frame and the key frame; a conversion unit converting the feature points of the target frame into three-dimensional feature points based on the estimated posture value; a classification unit matching the converted three-dimensional feature points with one or more reference planes drawn from the key frame and classifying the converted three-dimensional feature points into one or more groups; a plane configuration unit configuring one or more target planes drawn from the target frame based on the three-dimensional feature points included in the classified groups; and a correction unit correcting the posture value for the target frame based on a normal vector on the reference plane. The present invention is able to more precisely estimate a posture of a camera.

Description

DEVICE, METHOD AND COMPUTER PROGRAM FOR CORRECTING POSTURE OF CAMERA BASED ON IMAGE

The present invention relates to an apparatus, a method, and a computer program for correcting the posture of a camera through an image.

In the image-based posture estimation, a feature point is extracted from a captured image, and the camera posture (eg, movement value and rotation value, etc.) is estimated using the extracted feature point. For example, each corner of an object included in the image may be extracted as a feature point, and a movement value and a rotation value of the camera may be calculated from the extracted feature point.

In this regard, the prior art Korean Patent Publication No. 1942646 relates to a real-time camera posture estimation apparatus based on image feature points, and performs primary matching between key points for an object in a photographed image and key points previously stored for the object. , discloses a configuration for estimating a camera posture by performing secondary matching on key points of an object of photographed images based on a primary matching result.

However, in the prior art, since the camera posture is estimated only by matching between key points stored in advance for an object and key points for an object in a photographed image, the posture of the camera is accurately determined through the photographed image of the camera, which may include various variables. There are limits to the estimation.

Korea Patent Publication No. 1942646 (Registered on January 21, 2019) Korean Patent Publication No. 1880185 (Registered on July 13, 2018)

The present invention is to solve the problems of the prior art described above, and to provide a posture correction device capable of estimating a posture from an image more accurately by estimating the posture of a camera from an image and performing correction of the estimated posture in parallel do.

Another object of the present invention is to provide a posture correction device capable of estimating a plane from an image and estimating the posture of a camera more stably using the estimated plane.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention is an apparatus for correcting the posture of a camera through an image, a key for setting a key frame based on at least two frames included in the image frame setting unit; a feature point extraction unit for extracting a feature point from the target frame included in the image; an estimator for estimating a posture value with respect to the target frame based on the extracted feature points of the target frame and the key frame; a transformation unit converting the feature points of the target frame into three-dimensional feature points based on the estimated posture value; a classification unit for classifying the transformed 3D feature points into at least one group by matching them with at least one reference plane derived from the key frame; Correcting an attitude value for the target frame based on a plane component constituting at least one target plane derived from the target frame based on the three-dimensional feature points included in the classified group and a normal vector to the reference plane It is possible to provide a posture correcting device including a correcting unit.

Another embodiment of the present invention provides a method of correcting a camera posture through an image, the method comprising: setting a key frame based on at least two frames included in the image; extracting a feature point from a target frame included in the image; estimating a posture value of the target frame based on the extracted feature points of the target frame and the key frame; converting the feature point of the target frame into a three-dimensional feature point based on the estimated posture value; classifying the transformed 3D feature points into at least one group by matching them with at least one reference plane derived from the key frame; constructing at least one target plane derived from the target frame based on three-dimensional feature points included in the classified group, and correcting a posture value for the target frame based on a normal vector to the reference plane It is possible to provide a posture correction method comprising a.

Another embodiment of the present invention is a computer program stored in a computer-readable recording medium including a sequence of instructions for correcting the posture of a camera through an image, wherein the computer program is stored on the image when executed by a computing device. A key frame is set based on at least two included frames, a feature point is extracted from a target frame included in the image, and an attitude toward the target frame based on the extracted feature point of the target frame and the key frame feature point estimating a value, converting the feature point of the target frame into a three-dimensional feature point based on the estimated posture value, and matching the converted three-dimensional feature point with at least one reference plane derived from the key frame to obtain at least one Classify into groups, configure at least one target plane derived from the target frame based on 3D feature points included in the classified group, and pose values for the target frame based on a normal vector to the reference plane It is possible to provide a computer program stored in a computer-readable recording medium, including a sequence of instructions to correct the.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, the posture of the camera is estimated using the feature points extracted from at least two frames included in the image, the three-dimensional feature points and the plane, and the correction of the estimated posture is performed in parallel By doing so, it is possible to provide a posture correcting device capable of more accurately estimating the posture of the camera.

In addition, by estimating a plane from at least two frames included in an image and correcting the estimated camera posture using the estimated plane, it is possible to provide a posture correcting apparatus capable of estimating the camera posture more stably.

1 is a configuration diagram of a posture correcting apparatus according to an embodiment of the present invention.
FIG. 2 is an exemplary diagram for describing feature point matching between frames according to an embodiment of the present invention.
3 is an exemplary diagram for explaining a process of classifying 3D feature points into at least one group according to an embodiment of the present invention.
4 is an exemplary diagram for explaining a process of correcting a posture value of a frame based on a normal vector with respect to a reference plane according to an embodiment of the present invention.
5 is an exemplary diagram for explaining the limitation of point cloud data according to key frame candidates according to an embodiment of the present invention.
6 is an exemplary view for explaining posture value correction according to an embodiment of the present invention.
7 is a flowchart of a posture value correction method according to an embodiment of the present invention.
8 is an exemplary view for explaining before and after correction of the posture value of the frame according to an embodiment of the present invention.
9 is a flowchart of a posture correction method according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in this specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the corresponding server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram of a posture correcting apparatus according to an embodiment of the present invention. Referring to FIG. 1 , the posture correcting apparatus 100 includes a key frame setting unit 110 , a feature point extracting unit 120 , an estimating unit 130 , a transforming unit 140 , a classifying unit 150 , and a planar construction unit ( 160 ), a correction unit 170 , and a candidate setting unit 180 . However, the above components 110 to 180 are merely illustrative of components that can be controlled by the posture correcting apparatus 100 .

Each component of the posture correction apparatus 100 of FIG. 1 is generally connected through a network. For example, as shown in FIG. 1 , the posture correcting apparatus 100 may be connected simultaneously or at intervals of time.

A network refers to a connection structure that allows information exchange between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW: World). Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound communication, Visible Light Communication (VLC), LiFi, and the like, but are not limited thereto.

The posture correction apparatus 100 according to an embodiment of the present invention estimates the posture of the camera using feature points, three-dimensional feature points, and planes extracted from at least two frames included in an image, and performs correction of the estimated posture in parallel. can be done with

Also, the posture correcting apparatus 100 may more stably estimate the camera posture by estimating a plane from at least two frames included in the image and correcting the estimated camera posture using the estimated plane.

Hereinafter, each configuration of the posture correcting apparatus 100 will be described in detail.

The key frame setting unit 110 according to an embodiment of the present invention may set a key frame based on at least two frames included in an image. The key frame setting unit 110 may set a frame included in an image as a key frame according to a preset standard. Here, the process of setting the key frame may correspond to the initialization process. The key frame setting unit 110 may define a reference origin through an initialization process, and may generate a reference key frame, reference point cloud data, and a reference plane.

The key frame may be a reference frame for estimating and correcting an attitude value of a target frame, which will be described later. For example, the key frame may be a frame at which the camera image is switched.

For example, the key frame setting unit 110 matches the feature points of the first frame (f _t-1 ) and the second frame (f _t ), and uses matching information including information on the matched feature points. When the number of matched feature points in the process of estimating the posture value of the second frame f _t is a constant number, the first frame f _t-1 and the second frame f _t may be set as key frames. Here, the first frame f _t-1 and the second frame f _t may be immediately adjacent frames or may be temporally separated frames. Even in frames separated in time, when the number of matched feature points between the first frame f _t-1 and the second frame f _t is a predetermined number or more, both frames may be set as reference key frames. If the number of matched feature points is less than a certain number in the process of estimating the posture value of the second frame f _t using the matching information, the key frame setting unit 110 performs the above-described process using the next frame. can be repeated.

The key frame setting unit 110 may estimate the posture value of the second frame f _t based on the first frame f _t-1 by using the matching information. Specifically, the key frame setting unit 110 uses the first frame (f _t-1 ) and the second frame (f _t ) to perform homography or fundamental matrix decomposition using the second frame. Rotation and movement information of the frame f _t can be estimated. At this time, the key frame setting unit 110 does not match between the first frame f _t-1 and the second frame f _t in the calculation process of homography or fundamental matrix decomposition. Dots can be removed. In this process, if a predetermined number of matched feature points between the first frame f _t-1 and the second frame f _t is not maintained, the above-described process may be repeated using the next frame.

In the process of estimating the posture value of the second frame f _t , the key frame setting unit 110 sets the feature point as a three-dimensional feature point using the matched feature point and the estimated posture value when a certain number of matched feature points exist. can be converted to The generated 3D feature point may correspond to initial point cloud data.

That is, when a certain number of matched feature points exist, initialization may be performed. At this time, the matched feature point of the second frame f _t may be converted into a three-dimensional feature point based on the center of the first frame f _{t-1 ,} and the center of the first frame f t-1 is It can be defined as a reference origin. The defined origin may be the origin of the point cloud data, and may be the origin of the estimated posture value, that is, the estimated camera position information and rotation information.

The key frame setting unit 110 may generate a reference plane for the second frame f _t by using the transformed 3D feature point. For example, the key frame setting unit 110 may generate a vertical or horizontal plane for the second frame f _t by using the converted 3D feature points.

The key frame setting unit 110 may store the key frame setting process, that is, information about the key frame generated during the initialization process, the three-dimensional feature point, and the reference plane in the point cloud data. Point cloud data according to an embodiment of the present invention may include key frame feature points, narration information on key frame feature points, 3D key frame feature points, and normal vectors with respect to a reference plane. For example, the point cloud data may set the center of the first frame f _t-1 as the origin of the point cloud data. That is, the origin of the point cloud data may be set through an initialization process, and the location of the point cloud data may be defined based on the set origin.

The keypoint extractor 120 according to an embodiment of the present invention may extract a keypoint from a target frame included in an image. For example, the keypoint extractor 120 may extract a pixel that is easy to identify as a keypoint even if the shape, size, and position of an image, such as a corner portion of the target frame, or the viewpoint and lighting of a camera change. For example, the feature extraction unit 120 uses algorithms such as Harris Corner Detection, Adaptive and Generic Corner Detection Based on the Accelerated Segment Test, and FAST Corner Detection. Thus, a feature point can be extracted from the target frame.

The feature point extractor 120 may define the extracted feature point using a feature description. For example, the keypoint extractor 120 may generate a descriptor by detecting a direction component and a size of the extracted keypoint. For example, the keypoint extractor 120 may generate a descriptor for the keypoint extracted from the target frame by using algorithms such as ORB, SIFT, SURF, BRIEF, and BRISK.

The estimator 130 according to an embodiment of the present invention may estimate the posture value of the target frame based on the extracted feature points of the target frame and the key frame. Here, the posture value for the target frame may include position information and rotation information of a camera in which the frame was taken. For example, the estimator 130 may match the same feature point by comparing the feature point description information defined in the target frame with the feature point description information of the key frame. For keypoint matching, a matching method may vary depending on the keypoint description algorithm.

FIG. 2 is an exemplary diagram for describing feature point matching between frames according to an embodiment of the present invention. Referring to FIG. 2 , the estimator 130 may match the same feature point by comparing the feature point description information of the key frame 210 with the feature point description information of the target frame 220 . The estimator 130 may estimate the posture value A of the target frame 220 based on the key frame 210 using matching information including information on the matched feature points. The estimated posture value A may include a rotation value and a movement value of the target frame 220 based on the key frame 210 .

The estimator 130 uses the key frame 210 and the target frame 220 to estimate rotation and movement information of the target frame 220 through homography or fundamental matrix decomposition. can The estimator 130 may remove an inconsistent matching point between the key frame 210 and the target frame 220 in a calculation process of homography or fundamental matrix decomposition.

In this process, if the matched feature points between the key frame 210 and the target frame 220 do not maintain a predetermined number, the above-described process may be repeated using the next frame. The estimator 130 according to an embodiment of the present invention is configured to match the feature points of any one key frame 210 temporally close to the target frame 220 among the key frames 210 with the feature points of the target frame 220 . can For example, if the estimator 130 fails to maintain a predetermined number of matched feature points between the key frame 210 and the target frame 220 , the key frame 210 is temporally close to the target frame 220 . Based on (210), the above-described process may be repeatedly performed.

The conversion unit 140 according to an embodiment of the present invention may convert the feature point of the target frame 220 into a 3D feature point based on the estimated posture value. For example, in the process of calculating homography or fundamental matrix decomposition, the transform unit 140 may perform a target matching feature point between the key frame 210 and the target frame 220 if a certain number or more. The following process for correcting the estimated posture value of the frame 220 may be performed.

The conversion unit 140 matches the extracted feature points of the target frame 220 and the key frame 210, and when the number of the matched feature points of the target frame 220 is greater than or equal to a preset number, the extracted target frame ( 220) can be converted into a three-dimensional feature point.

3 is an exemplary diagram for explaining a process of classifying 3D feature points into at least one group according to an embodiment of the present invention. Referring to FIG. 3 , the classification unit 150 according to an embodiment of the present invention matches the transformed 3D feature point with at least one reference plane 311 , 312 , 313 derived from the key frame 310 to at least It can be classified into one group (321, 322, 323).

For example, the classification unit 150 may classify 3D feature points of the target frame 320 matching the first reference plane 311 of the key frame 310 into the first group 321 , and the key frame The three-dimensional feature points of the target frame 320 matching the second reference plane 312 of 310 may be classified into a second group 322 , and the third reference plane 313 of the key frame 310 and 3D feature points of the matching target frame 320 may be classified into a third group 323 .

The plane configuration unit 160 according to an embodiment of the present invention configures at least one target plane derived from the target frame 320 based on the 3D feature points included in the classified groups 321 , 322 , 323 . can For example, when the number of 3D feature points included in the classified groups 321 , 322 , and 323 is a predetermined number or more, the plane configuration unit 160 may configure the target plane in the corresponding group. As an example, when the number of feature points required to configure the target plane is three, the planar configuration unit 160 includes a second group 323 including three three-dimensional feature points in the target frame 320 shown in FIG. 3 . and a third group 323 including four 3D feature points may configure a target plane based on the 3D feature points.

4 is an exemplary diagram for explaining a process of correcting a posture value of a frame based on a normal vector with respect to a reference plane according to an embodiment of the present invention. Referring to FIG. 4 , the correction unit 170 may correct the posture value of the target frame based on the normal vector 411 with respect to the reference plane 410 . For example, the corrector 170 may correct the posture value of the target frame by comparing the target plane 420 of the target frame with the reference plane 410 of the key frame. Specifically, the corrector 170 may apply the normal vector 411 of the reference plane 410 as the normal vector 421 of the target plane 420 .

The posture correcting apparatus 100 according to an embodiment of the present invention may further include a candidate setting unit 180 . The candidate setting unit 180 may set the target frame as a key frame candidate when the number of key frame points of the matched target frame is less than a preset ratio with respect to the key frame key frame key points. For example, in the process of estimating the posture of the target frame, the candidate setting unit 180 may use the target frame as a key frame candidate if the number of matched feature points between the feature points of the target frame and the key frame is less than a predetermined number. have.

5 is an exemplary diagram for explaining the limitation of point cloud data according to key frame candidates according to an embodiment of the present invention. Referring to FIG. 5 , the candidate setting unit 180 may define the point cloud data 520 for the key frame based on the posture value 510 of the key frame candidate. For example, the candidate setting unit 180 sets the point cloud data 520 based on the movement value and rotation value of the key frame candidate, that is, the angle of view in the user terminal corresponding to the key frame candidate and the distance between the user terminal and the feature point. can be limited By limiting the point cloud data 520 based on the posture value 510 of the key frame candidate, the candidate setting unit 180 may improve the processing speed of the key frame candidate.

)가 하기 수학식 1에 해당하는 경우, 키 프레임 후보자를 키 프레임으로 설정할 수 있다. The key frame setting unit 110 may match the limited point cloud data with the feature points of the key frame candidates, and when the number of the matched key frame candidates corresponds to a preset number, the key frame candidates may be set as key frames. For example, the key frame setting unit 110 determines the number (

) corresponds to Equation 1 below, a key frame candidate may be set as a key frame.

<Equation 1>

)가 하기 수학식 2에 해당하는 경우, 키 프레임 후보자에 대한 자세값 보정을 수행할 수 있다. On the other hand, when the number of feature points of the matched key frame candidate exceeds a preset number, the corrector 170 is configured to adjust the posture value for the key frame candidate based on the normal vector to at least one reference plane corresponding to the point cloud data. can be corrected. For example, the correction unit 170 determines the number of feature points (

) corresponds to Equation 2 below, posture value correction may be performed on the key frame candidate.

<Equation 2>

For example, the correction unit 170 may correct the posture value of the key frame candidate by comparing the reference plane of the limited point cloud data with the plane of the key frame candidate. Specifically, the corrector 170 may apply the normal vector of the reference plane of the limited point cloud data as the normal vector of the plane with respect to the key frame candidate.

)을 보정할 수 있고, 보정된 자세값(

)을 이용하여 보정값(

)을 생성할 수 있다. 예를 들어, 보정부(170)는 하기 수학식 3을 이용하여 보정값(

)을 생성할 수 있다. The correcting unit 170 determines the posture value ( ) of the key frame candidate based on the point cloud data.

) can be corrected, and the corrected posture value (

) using the correction value (

) can be created. For example, the correction unit 170 uses the following Equation 3 to calculate the correction value (

) can be created.

<Equation 3>

)과 점군 데이터에 기초하여 보정된 키 프레임 후보자의 자세값(

)을 이용하여 보정값(

)을 생성할 수 있다. 보정부(170)는 생성된 보정값(

)에 기초하여 기존에 추정 및 보정된 대상 프레임의 자세값을 수정할 수 있다. Referring to Equation 3, the correction unit 170 sets the posture value (

) and the posture value of the key frame candidate corrected based on the point cloud data (

) using the correction value (

) can be created. The correction unit 170 generates the correction value (

) based on the previously estimated and corrected posture value of the target frame may be corrected.

6 is an exemplary view for explaining posture value correction according to an embodiment of the present invention. The correction unit 170 may correct an algorithm such as a bundle adjustment method using previously stored key frames, newly designated key frames, point cloud data, and feature points, and may accumulate them in the point cloud data. Referring to FIG. 6 , the plane configuration unit 160 may configure vertical or horizontal planes 621 , 622 , and 623 using the corrected point cloud data. The plane configuration unit 160 may accumulate and store normal vector information for the configured plane in the point cloud data.

The corrector 170 may estimate and correct the overall posture value using the newly designated keyframes 611 , 612 , 613 , and 614 . For example, the correction unit 170 corrects the estimated posture value of the target frame using the newly designated key frames 611 , 612 , 613 , and 614 that are accumulated and stored in the point cloud data, and accumulates the correction value in the point cloud data. can be included

7 is a flowchart of a posture value correction method according to an embodiment of the present invention. In summary, the posture correcting apparatus 100 according to the present invention sets a key frame through an initialization process ( S710 ) and an image ( S711 ) obtained by using the set key frame, referring to FIG. 7 ( a ). By extracting the feature points of the target frame included in and matching (S712), it is possible to estimate the posture value of the target frame (S713). The posture correcting apparatus 100 may correct the previously estimated posture value of the target frame based on the normal vector of the reference plane by comparing the reference plane derived from the key frame with the target plane derived from the target frame ( S714 ). . Meanwhile, the posture correcting apparatus 100 may check whether the target frame corresponds to a key frame candidate ( S715 ).

Referring to (b) of FIG. 7 , when the number of feature points of the target frame matching the key frame feature points is less than a preset ratio, the posture correcting apparatus 100 sets the target frame as a key frame candidate (S720). Also, the set key frame candidate may correct the posture value based on the point cloud data. The posture correcting apparatus 100 matches the key frame candidate's feature points with the feature points of the point cloud data (S721), and when the number of matched feature points (S722) exceeds a preset number (refer to Equation 2), in the point cloud data The posture value of the key frame candidate may be corrected (S723) based on the normal vector of the reference plane. On the other hand, when the number of matched feature points corresponds to a preset number (refer to Equation 1), the posture correcting apparatus 100 may set a key frame candidate as a key frame (S724). The posture correcting apparatus 100 may correct the previously estimated posture value of the entire frame based on the set key frame ( S725 ). The posture correcting apparatus 100 may store the set key frame and information on the key frame, the 3D key frame, and the plane generated in the process of setting the key frame in the point cloud data (S726). The posture correcting apparatus 100 may generate a correction value (refer to Equation 3) by using the posture value of the key frame candidate and the corrected posture value of the key frame candidate (S727). The posture correcting apparatus 100 may correct the previously estimated posture value of the target frame using the generated correction value and the set key frame ( S730 ). The posture correction process of FIG. 7A and the posture correction process of FIG. 7B may be performed in parallel.

8 is an exemplary view for explaining before and after correction of the posture value of the frame according to an embodiment of the present invention. Referring to FIG. 7 , the posture value 810 estimated using the feature point matching information between frames and the posture value 820 corrected using the reference plane may be checked. The point cloud data 830 may include a corrected posture value.

As described above, since the posture correcting apparatus 100 according to the present invention estimates the posture through the image and corrects the estimated posture in parallel, it is possible to more accurately estimate the posture of the camera. Accordingly, the user can enjoy more realistic AR contents through the estimated and corrected posture.

9 is a flowchart of a posture correction method according to an embodiment of the present invention. The posture correction method illustrated in FIG. 9 includes steps processed in time series according to the embodiments illustrated in FIGS. 1 to 8 . Accordingly, even if the description is omitted below, it is also applied to the method of correcting the posture through the image in the posture correcting apparatus according to the embodiment shown in FIGS. 1 to 8 .

In operation S910, the posture correcting apparatus may set a key frame based on at least two frames included in the image.

In operation S920, the posture correcting apparatus may extract a feature point from a target frame included in the image.

In operation S930, the posture correcting apparatus may estimate the posture value of the target frame based on the extracted feature points of the target frame and the key frame.

In operation S940, the posture correcting apparatus may convert the feature point of the target frame into a 3D feature point based on the estimated posture value.

In operation S950, the posture correcting apparatus may classify the transformed 3D feature point into at least one group by matching it with at least one reference plane derived from the key frame.

In operation S960, the posture correcting apparatus may configure at least one target plane derived from the target frame based on the 3D feature points included in the classified group.

In operation S970, the posture correcting apparatus may correct the posture value of the target frame based on the normal vector with respect to the reference plane.

In the above description, steps S910 to S970 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between the steps may be switched.

The method of correcting a posture through an image in the posture correcting apparatus described with reference to FIGS. 1 to 9 is a computer program stored in a computer readable recording medium executed by a computer or a recording medium including instructions executable by the computer. can also be implemented. In addition, the method of correcting the posture through an image in the posture correcting apparatus described with reference to FIGS. 1 to 9 may be implemented in the form of a computer program stored in a computer-readable recording medium that is executed by a computer.

A computer-readable recording medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable recording medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The foregoing description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: posture correction device
110: key frame setting unit
120: feature point extraction unit
130: estimator
140: conversion unit
150: classification unit
160: planar component
170: correction unit
180: candidate setting unit

Claims (17)

In the device for correcting the posture of the camera through the image,
a key frame setting unit for setting a key frame based on at least two frames included in the image;
a feature point extraction unit for extracting a feature point from the target frame included in the image;
an estimator for estimating a posture value with respect to the target frame based on the extracted feature points of the target frame and the key frame;
a transformation unit converting the feature points of the target frame into three-dimensional feature points based on the estimated posture value;
a classification unit for classifying the transformed 3D feature points into at least one group by matching them with at least one reference plane derived from the key frame;
a plane constructing unit configured to configure at least one target plane derived from the target frame based on three-dimensional feature points included in the classified group; and
A correction unit for correcting an attitude value of the target frame based on a normal vector with respect to the reference plane
Posture correction device comprising a.
The method of claim 1,
The conversion unit matches the extracted feature points of the target frame and the key frame, and if the number of the matched feature points of the target frame is greater than or equal to a preset number, converts the extracted feature points of the target frame into the three-dimensional feature points The thing to do is a posture correction device.
3. The method of claim 2,
A candidate setting unit configured to set the target frame as a key frame candidate when the number of feature points of the matched target frame is less than a preset ratio with respect to the key frame feature points
Further comprising a, posture correction device.
4. The method of claim 3,
The candidate setting unit,
The posture correcting apparatus of claim 1, wherein the point cloud data for the key frame is defined based on the posture value of the key frame candidate.
5. The method of claim 4,
The key frame setting unit,
matching the limited point cloud data with the feature points of the key frame candidate, and setting the key frame candidate as the key frame when the number of the matched key frame candidate feature points corresponds to a preset number Device.
6. The method of claim 5,
The correction unit,
When the number of feature points of the matched key frame candidate exceeds a preset number, correcting the posture value of the key frame candidate based on a normal vector to at least one reference plane corresponding to the point cloud data , posture correction device.
The method of claim 1,
wherein the conversion unit matches a feature point of any one key frame temporally close to the target frame among the key frames with a feature point of the target frame.
7. The method of claim 6,
The point cloud data includes a feature point of the key frame, description information on the feature point of the key frame, a three-dimensional feature point of the key frame, and a normal vector to the reference plane.
In the method of correcting the posture of the camera through the image,
setting a key frame based on at least two frames included in the image;
extracting a feature point from a target frame included in the image;
estimating a posture value of the target frame based on the extracted feature points of the target frame and the key frame;
converting the feature point of the target frame into a three-dimensional feature point based on the estimated posture value;
classifying the transformed 3D feature points into at least one group by matching them with at least one reference plane derived from the key frame;
constructing at least one target plane derived from the target frame based on the 3D feature points included in the classified group; and
correcting the posture value for the target frame based on a normal vector with respect to the reference plane
Posture correction method comprising a.
10. The method of claim 9,
The step of converting the feature point of the target frame into a three-dimensional feature point,
matching the feature point of the extracted target frame and the feature point of the key frame;
converting the extracted feature points of the target frame into the three-dimensional feature points when the number of feature points of the matched target frame is equal to or greater than a preset number;
That comprising a, posture correction method.
11. The method of claim 10,
setting the target frame as a key frame candidate when the number of the matched feature points of the target frame is less than a preset ratio with respect to the key frame feature points
Further comprising a, posture correction method
12. The method of claim 11,
defining point cloud data for the key frame based on the posture value of the key frame candidate
Further comprising a, posture correction method.
13. The method of claim 12,
matching the limited point cloud data with the feature points of the key frame candidates; and
setting the key frame candidate as the key frame when the number of feature points of the matched key frame candidate corresponds to a preset number;
Further comprising a, posture correction method.
14. The method of claim 13,
correcting the posture value of the key frame candidate based on a normal vector to at least one reference plane corresponding to the point cloud data when the number of feature points of the matched key frame candidate exceeds a preset number;
Further comprising a, posture correction method.
10. The method of claim 9,
The step of estimating the posture value for the target frame,
The method of claim 1, wherein a feature point of any one key frame that is temporally close to the target frame among the key frames is matched with a feature point of the target frame.
15. The method of claim 14,
The point cloud data includes a feature point of the key frame, description information on the feature point of the key frame, a three-dimensional feature point of the key frame, and a normal vector to the reference plane.
In a computer program stored in a computer-readable recording medium including a sequence of instructions for correcting a camera posture through an image,
When the computer program is executed by a computing device,
setting a key frame based on at least two frames included in the image,
extracting feature points from the target frame included in the image,
estimating the posture value of the target frame based on the extracted feature points of the target frame and the key frame,
Transforming the feature point of the target frame into a three-dimensional feature point based on the estimated posture value,
Classifying the transformed 3D feature point into at least one group by matching it with at least one reference plane derived from the key frame,
constructing at least one target plane derived from the target frame based on the three-dimensional feature points included in the classified group,
A computer program stored in a computer-readable recording medium comprising a sequence of instructions for correcting a posture value with respect to the target frame based on a normal vector with respect to the reference plane.

Images