KR20200036353A - Apparatus and method for estimating 3d pose - Google Patents

Abstract

Disclosed is a 3D posture estimation device capable of easily extracting a marker and accurately recognizing the marker by using a 3D marker including an identification part and a 3D information part. The 3D posture estimation device comprises: a camera generating an image by imaging an object including a 3D marker; and a control part extracting the 3D marker from the image and calculating a 3D conversion relation between the camera and the 3D marker by using 3D information included in the 3D marker.

Description

3D posture estimation device and 3D posture estimation method {APPARATUS AND METHOD FOR ESTIMATING 3D POSE}

The present invention relates to a 3D posture estimation apparatus and a 3D posture estimation method, and more specifically, a 3D posture estimation apparatus and 3D to calculate a 3D transformation relationship between a camera and a 3D marker using a 3D marker It is about a posture estimation method.

In computer vision, virtual reality, etc., artificial markers are used for tracking cameras and synthesizing camera images and virtual graphics. Marker types include Maxi Code, QR Code, Intersense, AR ToolKit AR Tag, and Aruco. AR ToolKit and Aruco are commonly used. These markers can be used to reconstruct the three-dimensional shape of the object.

Since the marker is composed of white and black, in the related art, in order to extract the marker region, a region with high contrast in black and white is set as a candidate, and the presence or absence of the marker is determined using contour information. However, since the Aruco marker has a simple pattern, it is easy to extract the marker, whereas the white area is provided with a smaller area than the black area, so it is difficult to extract the Aruco marker from the image. In particular, when the distance between the camera and the marker is far, the size of the marker is small, so it is difficult to extract the marker, and accordingly, there is a problem in that the marker is not recognized or the marker is incorrectly recognized.

An object of the present invention is to provide a 3D posture estimation apparatus and a 3D posture estimation method that can easily extract a marker and accurately recognize a marker using a 3D marker consisting of an identification unit and a 3D information unit.

A 3D posture estimation apparatus according to an embodiment of the present invention for achieving the above object is a 3D posture estimation apparatus for 3D estimation of an object using a 3D marker, the object including the 3D marker And a controller for extracting the three-dimensional marker from the image and a camera generating an image by capturing the image, and calculating a three-dimensional transformation relationship between the camera and the three-dimensional marker using the three-dimensional information included in the three-dimensional marker. Includes.

Here, the 3D marker includes an identification unit and a 3D information unit, and the controller uses the 3D marker extraction unit to extract the 3D marker from the image, and the identification unit of the 3D marker, to Correction unit for correcting rotation and inclination of a 3D marker, 3D information acquisition unit for obtaining 3D information of the 3D marker, and 3D of the 3D marker using the corrected 3D information unit of the 3D marker It may include a calculation unit for calculating a three-dimensional transformation relationship between the camera and the three-dimensional marker using the three-dimensional information of the previously stored camera and the dimensional information.

Here, the 3D marker may be provided in a rectangular shape, and the 3D information unit may be located inside the 3D marker, and the identification unit may be located outside the 3D marker.

Here, a plurality of identification units may be present at each corner of the 3D marker.

Here, the identification unit may be provided in the form of a chess board.

In addition, the 3D marker extracting unit may convert the image into a binarized image and extract the 3D marker by detecting the identification unit in the converted binarized image.

Here, the correction unit may remove polygons except for the rectangle using the Douglas-Peucker Algorithm from the binarized image.

In addition, the calculation unit may calculate a 3D transformation relationship between the camera and the 3D marker using the corner coordinate values of the 3D information unit and the corner coordinate values of the identification unit.

On the other hand, the three-dimensional attitude estimation method according to an embodiment of the present invention, in a three-dimensional attitude estimation method for estimating the three-dimensional attitude of an object using a three-dimensional marker, including the three-dimensional marker using a camera Generating an image by imaging an object, extracting the 3D marker from the image, and calculating a 3D transformation relationship between the camera and the 3D marker using 3D information included in the 3D marker It may include steps.

Here, the 3D marker includes an identification unit and a 3D information unit, and the calculating of the 3D transformation relationship uses the identification unit of the 3D marker to correct rotation and inclination of the 3D marker. Using the corrected three-dimensional information unit of the three-dimensional marker, obtaining three-dimensional information of the three-dimensional marker and using the three-dimensional information of the three-dimensional marker and the previously stored three-dimensional information of the camera And calculating a 3D transformation relationship between the camera and the 3D marker.

Here, the 3D marker may be provided in a rectangular shape, and the 3D information unit may be located inside the 3D marker, and the identification unit may be located outside the 3D marker.

Here, a plurality of identification units may be present at each corner of the 3D marker.

Here, the identification unit may be provided in the form of a chess board.

Further, the extracting of the 3D marker may include converting the image into a binarized image and extracting the 3D marker by detecting the identification unit in the converted binarized image.

Here, the correcting may include removing polygons excluding rectangles using the Douglas-Peucker Algorithm from the binarized image.

In addition, in the calculating of the 3D transform relationship, a 3D transform relationship between the camera and the 3D marker may be calculated using a corner coordinate value of the 3D information unit and a corner coordinate value of the identification unit.

According to various embodiments of the present invention as described above, a 3D marker can be accurately extracted, and accordingly, a 3D transformation relationship between a camera and a 3D marker can be more accurately calculated.

1 is a block diagram showing the configuration of a 3D posture estimation apparatus according to an embodiment of the present invention.
2 is a block diagram showing in detail the configuration of a 3D posture estimation apparatus according to an embodiment of the present invention.
3 is a view showing a three-dimensional marker according to an embodiment of the present invention.
4 to 6 are diagrams illustrating a method of extracting a 3D marker from an image according to an embodiment of the present invention.
7 to 9 are diagrams illustrating a method of calculating a 3D transformation relationship between a camera and a 3D marker according to an embodiment of the present invention.
10 is a flowchart illustrating a 3D attitude estimation method according to an embodiment of the present invention.

Other advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and general knowledge in the technical field to which the present invention pertains. It is provided to fully inform the holder of the scope of the invention, and the invention is only defined by the scope of the claims.

If not defined, all terms (including technical or scientific terms) used herein have the same meaning as generally accepted by universal technology in the prior art to which this invention belongs. Terms defined by general dictionaries can be interpreted as having the same meaning as meaning in the text of the present application and / or related descriptions, and are not conceptualized or over-formally interpreted, even if not explicitly defined herein. Will not.

The terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, 'includes' and / or various conjugations of this verb, such as 'includes', 'includes', 'includes', 'includes', etc., refer to the stated composition, ingredients, components, The steps, operations and / or elements do not exclude the presence or addition of one or more other compositions, ingredients, components, steps, operations and / or elements.

The term 'and / or' used herein refers to each of the listed configurations or various combinations thereof.

In order to estimate a 3D attitude in computer vision, virtual reality, etc., markers having 3D information are located at a specific viewpoint or a specific position of each image. There are various types of markers. In particular, ARToolkit markers and Aruco markers are frequently used as markers representing 3D information. Among them, the Aruco marker has a simple pattern, so it is easy to extract the marker from the image, while the white area for storing 3D information in the marker is provided with a smaller area than the black background, so the distance between the camera and the marker There was a problem that the marker was not extracted normally as it got farther away.

The present invention is to solve the problems of the above-described Aruco marker, by adding a separate identification unit to the existing Aruco marker, so that the camera can more easily recognize the marker, the marker using the corner coordinates of the identification unit By correcting the inclination, rotation, or the like, 3D information can be more accurately extracted from the marker. Hereinafter, a method of estimating a 3D attitude using a 3D marker according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 10.

1 is a block diagram showing the configuration of a 3D posture estimation apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the 3D posture estimation apparatus 100 includes a camera 110 and a control unit 120. The camera 110 photographs an object including a 3D marker to generate an image containing the 3D marker. Also, the camera 110 may generate an image including a 3D marker by imaging an object including the 3D marker.

The controller 120 extracts a 3D marker from the image generated by the camera 110 and calculates a 3D transformation relationship between the camera and the 3D marker using 3D information included in the 3D marker. The 3D marker may include an identification unit and a 3D information unit. The specific form of the 3D marker will be described later with reference to FIG. 7. As shown in FIG. 2, the controller 120 may include a 3D marker extraction unit 121, a correction unit 122, a 3D information acquisition unit 123, and a calculation unit 124.

The 3D marker extraction unit 121 extracts a 3D marker from the image. Specifically, as shown in FIG. 3, an image generated by the camera is converted into a black and white gray level image, and then converted into a binarized image using an adaptive threshold technique. Subsequently, filtering may be applied to extract the contours of the binarized image and remove contours not related to the 3D marker. In addition, the 3D marker extraction unit 121 may identify a coded 3D marker using a border of a black square background in a binarized image.

The correction unit 122 may correct rotation and inclination of the 3D marker using the identification unit of the 3D marker. Specifically, the correction unit 122 may remove the polygonal shape excluding the rectangle by using the Douglas-Peucker Algorithm from the contour information extracted from the binarized image. The correction unit 121 sets a contour existing in the binarized image to which the Douglas Packer algorithm is applied as a marker candidate, and converts the image plane to an ortho-normal plane of the marker using coordinate values of corner points (Homography ) To remove the camera's perspective transformation. In addition, the correction unit 122 may generate a lookup table that divides the binarization matrix of the marker into a rectangular grid and assigns a value of 0 or 1 according to the pixel value inside the grid, as shown in FIG. 4. The rotation angle of the marker may be recognized using a lookup table corresponding to the expressed binarization code.

The 3D information acquisition unit 123 acquires 3D information of the 3D marker using the 3D information unit of the corrected 3D marker. The calculation unit 124 may calculate a 3D transformation relationship between the camera and the 3D marker using 3D information of the 3D marker obtained from the 3D information unit. Specifically, as shown in FIG. 5, the calculation unit 124 calculates the homography using the corner coordinate values of the 3D information unit and the corner coordinate values of the identification unit, and then applies an inverse matrix of the homography transformation to the standard model, and A 3D transform relationship between 3D markers can be calculated. Since the calculation unit 124 calculates the homography using the corner coordinates of the identification unit in addition to the corner coordinates of the 3D information unit, the attitude error of the camera as shown in FIG. 6 does not occur, and the 3D mark can be extracted more accurately. Accordingly, it is possible to accurately calculate a 3D transformation relationship between the camera and the 3D marker.

Referring to FIG. 7, the 3D marker 200 according to an embodiment of the present invention may include an identification unit 210 and a 3D information unit 220. The identification unit 210 is disposed outside the 3D marker 200, and may be provided in plural for each corner of the 3D marker 200. Further, the identification unit 210 is provided in the form of a chess board, so that coordinate values of corner points can be easily extracted. The 3D information unit 220 may be located inside the 3D marker 200 and store 3D information of the 3D marker 200. Since the 3D marker 200 according to an embodiment of the present invention includes the identification unit 210 in addition to the 3D information unit 220, the 3D marker 200 can be more easily extracted, and the 3D marker ( The degree of rotation and tilt of 200) can be more accurately identified.

In addition, the 3D information acquisition unit 123 determines the coordinate values of the corner points of the chessboard shape of the 3D markers in the camera image, and then determines the 3D transformation relationship between the camera and the 3D markers. The coordinate system of the marker can be defined. Specifically, the 3D information acquisition unit 123 may convert a 3D world coordinate system to a 3D camera coordinate system to define a 3D position and rotation relationship of the 3D marker based on the origin of the camera. Thereafter, the 3D camera coordinate system may be converted into a 2D image coordinate system, and a 3D transformation relationship between a 2D image coordinate system and a 3D world coordinate system may be calculated using internal parameters of a previously stored camera. If there are multiple 3D markers, the coordinate values of the corner points of the chessboard shape can be calculated to find the corresponding points of the 2D image coordinates and 3D world coordinates of each 3D marker, and a PnP (Perspective n-Point) problem solving algorithm Can be used to calculate a 3D transformation equation. In addition, a solution that minimizes reprojection errors can be calculated using the Levenberg-Marquardt algorithm. In addition, the position of the camera or the position of the 3D marker can be estimated using the 3D transformation relationship between the camera and the 3D marker, and the position of the target point can be estimated by assuming that the 3D marker is attached to the target point. .

Referring to FIG. 8, when a plurality of 3D markers are present, when a coordinate value of the identification unit of the 3D marker is found, a corresponding point between image coordinates (U) and world coordinates (W) of each 3D marker can be known, and PnP problem The 3D transformation equation can be calculated by solving. In addition, it is possible to minimize reprojection errors based on Levenberg-Marquardt optimization. After calculating the three-dimensional transformation relationship between the camera and the three-dimensional marker, the position of the camera or the position of the three-dimensional marker can be estimated, and it is assumed that the three-dimensional marker is attached to the target point and the position of the target point can be estimated. . The result of the extraction of the 3D marker area attached to the target point and the definition of the coordinate system may be shown as in FIG. 9.

10 is a flowchart illustrating a 3D attitude estimation method according to an embodiment of the present invention.

Referring to FIG. 10, first, an object including a 3D marker is imaged to generate an image (S1010).

Subsequently, a 3D marker is extracted from the image (S1020). Here, the 3D marker may include an identification unit and a 3D information unit, the identification unit may be provided outside the 3D marker, and the 3D information unit may be provided inside the 3D marker. Specifically, an image generated by the camera is converted into a black and white gray level image, and then converted into a binary image using an adaptive threshold technique. Subsequently, filtering may be applied to extract the contours of the binarized image and remove contours not related to the 3D marker. In addition, a coded 3D marker can be identified by using a black square background border in the binarized image.

Subsequently, a 3D transformation relationship between the camera and the 3D marker is calculated using the 3D information included in the 3D marker (S1030). Specifically, the rotation and inclination of the 3D marker is corrected using the identification unit of the 3D marker, and after obtaining the 3D information of the 3D marker using the 3D information unit of the corrected 3D marker, the 3D marker Using the 3D information and the 3D information of the previously stored camera, a 3D transformation relationship between the camera and the 3D marker can be calculated.

According to various embodiments of the present invention as described above, a 3D marker can be accurately extracted, and accordingly, a 3D transformation relationship between a camera and a 3D marker can be more accurately calculated.

Meanwhile, a non-transitory computer readable medium in which a program for sequentially performing a 3D attitude estimation method according to an embodiment of the present invention is stored may be provided.

The non-transitory computer-readable medium refers to a medium that stores data semi-permanently and that can be read by a computer, rather than a medium that stores data for a short time, such as registers, caches, and memory. Specifically, the various applications or programs described above may be stored and provided in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention claimed in the claims. In addition, various modifications can be implemented by those skilled in the art, and these modifications should not be individually understood from the technical idea or prospect of the present invention.

100: 3D attitude estimation device
110: camera
120: control unit

Claims (16)

In the three-dimensional posture estimation apparatus for three-dimensional estimation of an object using a three-dimensional marker,
A camera that captures an object including the three-dimensional marker and generates an image; And
And a control unit for extracting the 3D marker from the image and calculating a 3D transformation relationship between the camera and the 3D marker using 3D information included in the 3D marker. According to claim 1,
The 3D marker includes an identification unit and a 3D information unit,
The control unit,
A 3D marker extractor for extracting the 3D marker from the image;
A correction unit for correcting rotation and inclination of the 3D marker using the identification unit of the 3D marker;
A 3D information acquisition unit that acquires 3D information of the 3D marker using the corrected 3D information unit of the 3D marker; And
And a calculator configured to calculate a 3D transformation relationship between the camera and the 3D marker using 3D information of the 3D marker and 3D information of the camera stored in advance. According to claim 2,
The 3D marker is provided in a rectangular shape,
A 3D posture estimation apparatus in which the 3D information unit is located inside the 3D marker and the identification unit is located outside the 3D marker. According to claim 3,
The identification unit, a three-dimensional attitude estimation apparatus that is present in a plurality of each corner of the three-dimensional marker. According to claim 4,
The identification unit, a three-dimensional posture estimation device provided in the form of a chess board. According to claim 3,
The 3D marker extraction unit,
A 3D posture estimation apparatus that converts the image into a binarized image and extracts the 3D marker by detecting the identification unit in the converted binarized image. The method of claim 6,
The correction unit,
A 3D posture estimation apparatus for removing polygons excluding rectangles using the Douglas-Peucker Algorithm from the binarization image. According to claim 3,
The calculation unit,
A 3D posture estimation apparatus for calculating a 3D transformation relationship between the camera and the 3D marker using a corner coordinate value of the 3D information unit and a corner coordinate value of the identification unit. In the 3D attitude estimation method for estimating the 3D attitude of an object using a 3D marker,
Generating an image by imaging an object including the three-dimensional marker using a camera;
Extracting the three-dimensional marker from the image; And
And calculating a 3D transformation relationship between the camera and the 3D marker using 3D information included in the 3D marker. The method of claim 9,
The 3D marker includes an identification unit and a 3D information unit,
The step of calculating the three-dimensional transformation relationship,
Correcting rotation and inclination of the three-dimensional marker by using the identification unit of the three-dimensional marker;
Obtaining 3D information of the 3D marker by using a corrected 3D information unit of the 3D marker; And
And calculating a 3D transformation relationship between the camera and the 3D marker using 3D information of the 3D marker and 3D information of the previously stored camera. The method of claim 10,
The 3D marker is provided in a rectangular shape,
A 3D posture estimation method in which the 3D information unit is located inside the 3D marker and the identification unit is located outside the 3D marker. The method of claim 11,
The identification unit, a method for estimating a three-dimensional attitude that a plurality of each corner of the three-dimensional marker exists. The method of claim 12,
The identification unit is a three-dimensional attitude estimation method provided in the form of a chess board. The method of claim 11,
Extracting the three-dimensional marker,
Converting the image into a binarized image; And
And detecting the identification unit from the converted binarized image and extracting the 3D marker. The method of claim 14,
The correction step,
A method of estimating a 3D posture, comprising removing polygons excluding rectangles using a Douglas-Peucker Algorithm from the binarized image. The method of claim 11,
The step of calculating the three-dimensional transformation relationship,
A 3D posture estimation method for calculating a 3D transformation relationship between the camera and the 3D marker using the corner coordinate values of the 3D information unit and the corner coordinate values of the identification unit.

Images