KR102261498B1 - Apparatus and method for estimating the attitude of a picking object - Google Patents

Abstract

An estimation device is provided. The estimation device includes: an acquisition unit for acquiring a two-dimensional image of an object; an extractor configured to extract a plurality of vectors having a similarity with respect to an object region representing an object included in the two-dimensional image higher than a reference value; a determination unit determining whether to use depth information of the object; and an estimation unit for estimating a posture of the object using the two-dimensional image, wherein the estimation unit operates in a first mode for extracting a plurality of first vectors according to a similarity ranking among the plurality of vectors according to a determination result of the determination unit, or extracts a plurality of second vectors according to an inverse similarity ranking among the plurality of vectors may operate in the second mode. Therefore, it is possible to accurately estimate the posture of the picking object while excluding or minimizing the dependence on depth information.

Description

Apparatus and method for estimating the attitude of a picking object

The present invention relates to an apparatus and method for estimating the posture of an object to be picked by a robot.

In order to control a robot that processes or processes an object according to the object, a sensor for detecting an object may be used.

The processing accuracy of the robot on the object may be related to the measurement precision of the sensor. In order to improve the processing accuracy of the robot, the higher the measurement precision of the sensor, the more advantageous.

However, since a sensor having high measurement accuracy is very expensive, popularization is difficult. Also, even when a sensor having high measurement accuracy is provided, a situation in which measurement accuracy is lowered due to diffuse reflection or the like may occur depending on the type of object.

In particular, when a low precision distance measuring means and a depth measuring means are applied in the surrounding environment, it may be difficult to grasp the posture of the picking object. The posture of the picking object may be one of the factors that must be grasped essential in the picking operation.

Korean Patent Application Laid-Open No. 2019-0072285 discloses a technique for grasping the position of a selected gripping object through a picking view camera and gripping the gripping object using a gripper.

Korean Patent Publication No. 2019-0072285

SUMMARY OF THE INVENTION It is an object of the present invention to provide an estimation apparatus and an estimation method for accurately estimating a posture of a picking target while excluding or minimizing dependence on depth information.

According to an embodiment of the present invention, an estimation apparatus is provided. The estimation apparatus may include: an acquisition unit configured to acquire a two-dimensional image of an object; an extractor configured to extract a plurality of vectors having a similarity with respect to an object region representing the object included in the two-dimensional image higher than a reference value; a determination unit determining whether to use the depth information of the object; and an estimator for estimating the posture of the object by using the two-dimensional image.

The estimator operates in a first mode for extracting a plurality of first vectors according to a similarity ranking among the plurality of vectors according to a determination result of the determination unit, or extracts a plurality of second vectors according to an inverse similarity ranking among the plurality of vectors may operate in the second mode.

The extraction unit generates the plurality of vectors in an order similar to the target region using at least one of a deep learning technique and a similarity determination technique, or the plurality of vectors in an order similar to the target region from the three-dimensional model of the target may be extracted, or the plurality of vectors may be extracted from a database in which the plurality of vectors are stored in advance in a similar order to the target area.

The extraction unit divides the entire angular range that the object can take into a plurality of angular ranges, and extracts the plurality of vectors having a similarity higher than the reference value corresponding to each of the divided angles in comparison with the object area. have.

The extractor may detect the target region in the 2D image using a deep learning-based target detection algorithm.

The extractor may localize the target area.

The extractor may apply the target region as an input of a pre-learned deep learning generation model.

The extractor may measure the similarity between the embedded source vector and the storage vector stored in the codebook using the target region as a source in the encoder part of the deep learning generation model.

The extractor may extract the plurality of vectors having a high similarity to the target region from among the stored vectors according to the measured similarity.

According to another embodiment of the present invention, an estimation apparatus is provided. The estimation apparatus may include: an extractor configured to extract a plurality of first vectors having a similarity with respect to an object region representing an object from a two-dimensional image higher than a first reference value; and an estimator for applying the postures of the plurality of first vectors extracted by the extraction unit to the three-dimensional model of the object.

The estimator may obtain a plurality of 2D rendering images from the 3D model to which the postures of the plurality of first vectors are applied.

The estimator may compare the plurality of 2D rendered images with the target region.

The estimator may estimate a specific posture of a specific first vector used to extract a specific 2D rendered image having the highest similarity to the object region as the posture of the object.

The estimator may 2D render the 3D model to which the postures of the plurality of vectors are applied by the number of the plurality of vectors.

The estimator may apply a BBox (Bounding Box) method to limit the object area within a bounding box area, or apply a masking method to limit the object area to a segmentation area.

The estimator may two-dimensionally match the object area limited to the bounding box area or the segmentation area with the 2D rendering image.

The estimator considers the specific 2D rendered image having the highest score of the 2D matching as the correct answer, and sets the specific posture of the specific vector used to extract the specific 2D rendered image as the posture of the object. can be estimated

An estimation apparatus according to another embodiment of the present invention is provided. The estimating apparatus may include: an extractor configured to extract a plurality of vectors in which a similarity with respect to an object region included in a 2D image satisfies a second reference value; and an estimator for estimating the posture of the object by using the two-dimensional image and depth information of the object.

The extractor may extract an initial vector having the highest degree of similarity to the target region from among the plurality of vectors, and extract a plurality of second vectors in an order of decreasing similarity to the target region from among the plurality of vectors.

The estimator may estimate the posture of the object by using at least one of a posture corresponding to the initial vector, postures of the plurality of second vectors, and the depth information.

The estimator may apply the postures of the initial vector and the postures of the plurality of second vectors to the 3D model of the object, respectively.

The estimator may generate as many model point clouds as the number of the initial vector and the plurality of second vectors by using the three-dimensional model to which the posture of the initial vector is applied and the three-dimensional model to which the postures of the plurality of second vectors are applied. have.

The estimator may generate a target point cloud by using the depth information.

The estimator may correct a plurality of the model point clouds in a direction to follow the single target point cloud using a three-dimensional matching technique.

The estimator may generate a plurality of 2D rendered images by applying the corrected postures of the plurality of model point clouds to the 3D model.

The estimator may estimate a posture of a specific 2D rendered image having the highest similarity to the target region among a plurality of the 2D rendered images as the posture of the object.

An estimation method according to another embodiment of the present invention is provided. The estimation method may include: acquiring a two-dimensional image of an object; extracting a plurality of vectors having a similarity with respect to an object region representing the object included in the two-dimensional image higher than a reference value; determining whether to use depth information of the object; and executing the first mode when the depth information is not used, and executing the second mode when the depth information is used.

The first mode may include: extracting a plurality of first vectors having a similarity to the target region higher than a first reference value from among the plurality of vectors; applying a posture of each of the plurality of first vectors to the three-dimensional model of the object; obtaining a plurality of two-dimensional rendering images from a plurality of three-dimensional models to which the plurality of postures of the first vector are applied; comparing the plurality of two-dimensional rendered images with the target area; The method may include estimating a specific posture of a specific first vector used to extract a specific 2D rendered image having the highest similarity to the target region as the posture of the object.

In the second mode, an initial vector having the highest degree of similarity to the object region is extracted from among a plurality of vectors, and a plurality of second vectors in an order in which the degree of similarity to the object region is lower than a second reference value from among the plurality of vectors extracting the vector; applying the postures of the initial vector and the postures of the plurality of second vectors to the three-dimensional model of the object, respectively; generating model point clouds as many as the number of the initial vector and the plurality of second vectors by using the three-dimensional model to which the posture of the initial vector is applied and the three-dimensional model to which the postures of the plurality of second vectors are applied; generating a target point cloud by using the depth information; correcting a plurality of the model point clouds in a direction to follow the single target point cloud using a three-dimensional matching technique; generating a plurality of two-dimensional rendering images by applying the corrected postures of the plurality of model point clouds to the three-dimensional model; and estimating, as the posture of the object, a posture of a specific 2D rendered image having the highest similarity to the target region among the plurality of 2D rendered images.

According to the present invention, in an environment in which a depth camera is excluded or a low-cost, low-capacity depth camera is applied, the posture of the picking object may be estimated using a two-dimensional image-based camera.

According to the present invention, a 6D posture of an object provided with a 3D model may be estimated using an inexpensive red, green, blue (RGB) camera or an RGB-D (depth) camera.

1 is a block diagram illustrating an estimation apparatus according to an embodiment of the present invention.
2 is a schematic diagram illustrating an operation of an extraction unit according to an embodiment of the present invention.
3 is a schematic diagram illustrating an operation of an estimation apparatus operating in a first mode.
4 is a schematic diagram illustrating an object area t treated with a bounding box according to an embodiment of the present invention.
5 is a schematic diagram illustrating an operation of an estimation apparatus operating in a second mode.
6 is a flowchart illustrating an estimation method according to an embodiment of the present invention.
7 is a flowchart illustrating a first mode according to an embodiment of the present invention.
8 is a flowchart illustrating a second mode according to an embodiment of the present invention.
9 is a diagram illustrating a computing device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In the present specification, duplicate descriptions of the same components will be omitted.

Also, in this specification, when it is said that a certain element is 'connected' or 'connected' to another element, it may be directly connected or connected to the other element, but other elements in the middle It should be understood that there may be On the other hand, in this specification, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that the other element does not exist in the middle.

In addition, the terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention.

Also, in this specification, the singular expression may include the plural expression unless the context clearly dictates otherwise.

Also, in this specification, terms such as 'include' or 'have' are only intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and one or more It should be understood that the existence or addition of other features, numbers, steps, operations, components, parts or combinations thereof is not precluded in advance.

Also in this specification, the term 'and/or' includes a combination of a plurality of listed items or any of a plurality of listed items.

Also, in this specification, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted.

1 is a block diagram illustrating an estimation apparatus according to an embodiment of the present invention. 2 is a schematic diagram showing the operation of the extraction unit 330 according to an embodiment of the present invention.

As shown in the drawing, the estimation apparatus according to an embodiment of the present invention may include an acquirer 310 , an extractor 330 , a determiner 350 , and an estimator 370 .

The acquisition unit 310 may acquire the 2D image i of the object 90 . The 2D image may represent a 2D flat image. For example, the two-dimensional image i may include a two-dimensional RGB (red, green, blue) image, a two-dimensional black-and-white image, and the like. The acquisition unit 310 may include a red, green, blue (RGB) camera, a black-and-white camera, and the like that generate the two-dimensional image i. Alternatively, the acquisition unit 310 may include a communication means for receiving a two-dimensional RGB image, a two-dimensional black-and-white image, or the like from an RGB camera, a black-and-white camera, and the like.

The extractor 330 may extract a plurality of vectors having a similarity higher than a reference value with respect to the object region t representing the object included in the 2D image i. The two-dimensional image i may include a photographed state of an actual object. In this case, the area occupied by the photographing image of the object in the two-dimensional image may correspond to the object area t.

A 'vector' described herein may refer to a three-dimensional shape briefly representing an object or an object region t using feature points extracted from the object region t. Alternatively, the 'vector' may refer to a flat image such as a plan view of the corresponding stereoscopic image. Alternatively, the 'vector' may represent the object or the area t of the object as it is, or a three-dimensional model of the object. The 'vector' may indicate a state in which a posture is applied to a three-dimensional shape briefly representing an object, an original shape of the object, a three-dimensional model of the object, or the like. The 'vector' may include a posture.

The 'posture' may include a rotation angle, a position, etc. based on a specific arrangement state of the object. As an example, the 'posture' may include at least one of six degrees of freedom (six degrees of freedom) positions.

When three coordinate axes, an x-axis, a y-axis, and a z-axis, which are orthogonal to each other are defined, a total of six degrees of freedom positions may be defined. The six degrees of freedom positions may include an x-axis position, a y-axis position, a z-axis position, a rotation angle about the x-axis, a rotation angle about the y-axis, and a rotation angle about the z-axis.

The extraction unit 330 may extract a plurality of vectors similar to the object region t from the three-dimensional model or database of the written object using a deep learning technique or the like.

As an example, the extractor 330 may generate a plurality of vectors in an order similar to the target region t by using at least one of a deep learning technique and a similarity determination technique. As an example, the extractor 330 may extract a vector having a similarity with respect to an object region higher than a reference value through various similarity calculation techniques for the image. Alternatively, the extraction unit 330 may extract a plurality of vectors in an order similar to the object region t from the three-dimensional model of the object, or extract a plurality of vectors in an order similar to the object region t from a database in which a plurality of vectors are stored in advance. have. The vector may include information representing the posture. The posture included in the plurality of vectors extracted by the extraction unit 330 may be used for estimating the posture of the actual object.

The extraction unit 330 of the present invention may extract a plurality of vectors having a similarity higher than a reference value, instead of extracting only a single vector having the highest similarity with respect to the target region t. Since various errors may occur in the process of grasping the posture of an actual object having a three-dimensional shape using only a two-dimensional image, it is possible to reduce the error by using a plurality of vectors having a similarity higher than the reference value.

Meanwhile, a plurality of vectors having a similarity higher than the reference value may theoretically exist infinitely. This is because the angle corresponding to the characteristic of the vector can be divided into infinity. In this case, since a plurality of vectors having very close angles include almost similar errors, it may not be meaningful to extract a plurality of vectors within a very close angle range.

Therefore, it is preferable that a plurality of vectors different by more than a set angle are extracted. For example, the extractor 330 may divide the entire angular range that the object can take into a plurality of angular ranges. The extraction unit 330 may extract a plurality of vectors having a similarity higher than a reference value corresponding to each of the planned angles with respect to the target area t. For example, when the entire angular range is 360 degrees based on the x-axis and the angular range is 60 degrees, the extractor 330 may extract one or more vectors satisfying the reference value in the 0-60 degree section. In addition, the extraction unit 330 is one or more that satisfy the reference value in each of a 60 degree to 120 degree section, a 120 degree to 180 degree section, a 180 degree to 240 degree section, a 240 degree to 300 degree section, and a 300 degree to 360 degree section. vector can be extracted. If there is no vector satisfying the reference value in a specific section, the vector may not be extracted in the corresponding section.

The extractor 330 may detect the object region t in the two-dimensional image using a deep learning-based object detection algorithm.

For example, the extractor 330 may limit or detect the object region t in the 2D image i using an object segmentation technique as shown in FIG. 2B .

The extractor 330 may localize the target area t as shown in FIG. 2(c) and remove a surrounding background (background deletion).

Only the target area t may be extracted from among various 2D images included in the 2D image i by the object segmentation, localization, and background subtraction performed by the extraction unit 330 .

The extractor 330 may apply the extracted object region t as an input of the pre-learned deep learning generation model.

According to an embodiment, the extractor 330 may measure the similarity between the embedded source vector and the storage vector stored in the codebook by using the target region t as the source in the encoder part of the deep learning generation model. The extractor 330 may extract a plurality of vectors having a high similarity to the target region from among the stored vectors according to the measured similarity.

According to an embodiment, the extractor 330 is a posture whose cosine similarity is higher than a reference value compared to a source vector among vectors (stored vectors) for each posture stored in the training stage of deep learning. It is possible to extract a plurality of vectors of .

According to an embodiment, a feature of a two-dimensional image may be extracted by the pre-learned encoder part of the deep learning generation model. The extracted 2D image features are embedded in a vector into a DB (database, database), and the vector may correspond to a source vector used as a first factor of cosine similarity.

A two-dimensional rendering image for each posture rendered using a three-dimensional model of the object may be provided. A feature of the 2D rendered image may be extracted using a pre-learned encoder. Features of the extracted 2D rendered image are embedded in a vector (DB), and the vector may correspond to a storage vector used as a second factor of cosine similarity. The '2D rendering image' may refer to a 2D image viewed from one side of the 3D model. For example, a top view, a side view, a front view, etc. of the 3D model may correspond to the 2D rendering image.

The cosine similarity may be calculated through the product of the first factor and the second factor.

Using a single vector product, the similarity of the entire DB can be calculated. A plurality of vectors having a similarity greater than a reference value may be arranged in an order of increasing similarity. In this case, a vector having the highest similarity among the plurality of vectors extracted by the extraction unit 330 may be defined as an 'initial vector'. In general, it can be expected that the initial vector v0 is most similar to the real object region t, but this is not the case. Experimentally, it was found that a vector other than the initial vector v0 was most similar to the target region t with a probability of about 30% or more.

As an example, (d) of FIG. 2 shows the initial vector v0, and it can be seen that the posture is significantly different from that of the target area t.

In order to accurately estimate the posture of the object, the estimation apparatus of the present invention may further perform an additional process without directly estimating the initial vector as the posture of the object.

The additional process may be performed by the determiner 350 and the estimator 370 .

Returning to FIG. 1 again, the determination unit 350 may determine whether to use the depth information of the object.

The depth information may include a distance value between a sensor such as a time-of-flight (ToF) camera, a distance measurer, and a vision and an object. The depth information may be acquired through the acquisition unit 310 . The acquisition unit 310 may include a sensor such as a Time-Of-flight (ToF) camera, a distance measurer, or a vision generating depth information, or a communication means for receiving depth information from the corresponding sensor.

The estimator 370 may estimate the posture of the object by using the two-dimensional image.

The estimator 370 may operate in a first mode for extracting a plurality of first vectors according to a similarity ranking among a plurality of vectors having a similarity to an object region higher than a reference value according to the determination result of the determining unit 350 . . Alternatively, the estimator 370 may operate in a second mode for extracting a plurality of second vectors according to an inverse similarity order among a plurality of vectors.

3 is a schematic diagram illustrating an operation of an estimation apparatus operating in a first mode.

In the first mode, the extractor 330 may extract a plurality of first vectors having a similarity higher than the first reference value with respect to the object region t representing the object from the 2D image i.

For example, the extractor 330 may extract n vectors (where n is a natural number equal to or greater than 3) having a similarity to the object region t representing the object from the 2D image i higher than the reference value.

The extraction unit 330 may extract m first vectors (where m is a natural number smaller than n and greater than or equal to 2) from among the n vectors in an order of high similarity.

As an example, the extraction unit 330 may omit the process of extracting the n vectors, and may immediately extract the m first vectors.

For example, the extractor 330 may extract m first vectors according to a similarity ranking among vectors other than the initial vector v0. In this case, the extraction unit 330 may additionally extract the initial vector v0. As a result, the extractor 330 may extract one initial vector v0 and m first vectors.

For example, the extractor 330 may extract m+1 first vectors without discriminating the initial vector v0. In this case, one initial vector v0 may also be naturally extracted by the extraction unit 330 .

The estimator 370 may apply the postures of the plurality of first vectors (including the initial vectors) extracted by the extraction unit 330 to the 3D model of the object.

The estimator 370 may obtain a plurality of 2D rendering images from the 3D model to which the postures of the plurality of first vectors v0, v1, v2, v3, and v4 are applied. The corresponding 2D rendered image may be as shown in (b) and (c) of FIG. 3 .

The estimator 370 may compare the plurality of 2D rendered images with the object region t illustrated in FIG. 3A .

The estimator 370 may estimate a specific posture of a specific first vector used to extract a specific 2D rendered image having the highest similarity with respect to the object region t as the posture of the object. In FIG. 3 , a specific 2D rendered image having the highest similarity to the object region t may be a 4th 2D rendered image from the left of (c). In this case, the estimator 370 may estimate the posture of the first vector v4 used to extract the fourth 2D rendered image of (c) as the posture of the object.

Specifically, the estimator 370 may 2D render the 3D model to which the postures of the plurality of first vectors are applied by the number of the plurality of first vectors.

The estimator 370 applies the BBox (Bounding Box) method to limit the object area t within the bounding box area, or applies the masking method to limit the object area t to the segmentation area. can

4 is a schematic diagram illustrating an object area t treated with a bounding box according to an embodiment of the present invention.

The estimator 370 may represent the object area t as a bounding box area BB or a segmentation area corresponding to a simple polygonal shape by applying a BBox method or a masking method. BBox is an abbreviation of Bounding Box and is called BBox for short. It may refer to a box having a minimum size that can include both the shapes of 2D or 3D objects. The segmentation area may refer to an area obtained through image segmentation of the object 90 .

In FIG. 4, three bounding boxes 0, 1, and 2 are formed.

The estimator 370 may two-dimensionally match the object area t limited to the bounding box area or the segmentation area with the 2D rendering image.

The estimator 370 considers the specific 2D rendered image having the highest score of 2D matching as the correct answer, and sets the specific posture of the specific first vector used to extract the specific 2D rendered image as the posture of the object. can be estimated

5 is a schematic diagram illustrating an operation of an estimation apparatus operating in a second mode.

The second mode may be an operation mode of the estimator for estimating the posture of the object using depth information of the object. If the accuracy of the depth information is high, the posture of the object may be accurately analyzed using only the depth information. However, if the depth information is inaccurate, the posture of the object must be analyzed using another method, and the second mode may be used as part of the analysis.

In the second mode, the extractor 330 may extract a plurality of vectors whose similarity with respect to the object region t included in the 2D image satisfies the second reference value.

For example, the extractor 330 may extract n vectors (where n is a natural number equal to or greater than 3) having a similarity to the object region t representing the object from the 2D image i higher than the reference value.

The extraction unit 330 may extract an initial vector v0 having the highest degree of similarity to the target region t from among a plurality of vectors as shown in FIG. 5B . In addition, the extractor 330 may extract a plurality of second vectors according to an order in which the degree of similarity to the object region t is low among the plurality of vectors.

For example, the extraction unit 330 may extract the number of m second vectors smaller than n and 2 or more.

Assume that n is 8 and m is 4. When a total of eight vectors v1, v2, v3, v4, v5, v6, v7, v8 are sorted in the order of high similarity, the extractor 330 performs 4 in the order of low similarity as shown in FIG. 5(c). Second vectors v8, v7, v6, and v5 can be extracted.

The estimator 370 may estimate the posture of the object using the 2D image i and depth information of the object. The estimator 370 may estimate the posture of the object by using at least one of a posture corresponding to the initial vector v0, postures of a plurality of second vectors, and depth information.

The estimator 370 may apply the postures of the initial vector v0 and the postures of the plurality of second vectors v8, v7, v6, and v5 to the 3D model of the object, respectively.

The estimator 370 uses the 3D model to which the posture of the initial vector v0 is applied and the 3D model to which the postures of the plurality of second vectors v8, v7, v6, and v5 are applied by the number of the initial vector and the plurality of second vectors. You can create a model point cloud.

A point cloud is a method for expressing three-dimensional data, and may refer to a format in which an object (outer surface) is represented by a plurality of points.

The estimator 370 may generate a target point cloud by using the depth information.

The estimator 370 may correct a plurality of model point clouds in a direction to follow a single target point cloud using a three-dimensional matching technique. Correction of the model point cloud using the three-dimensional matching technique may be generally performed incompletely.

For example, it is assumed that the target model point cloud is rotated by 30 degrees and the specific model point cloud is rotated by 15 degrees. In this case, if the 3D matching technique is used, the specific model point cloud rotated by 15 degrees may be rotated in a direction to be the same as (to follow) the target model point cloud. Through the rotation, the specific model point cloud in the existing 15 degree rotation state cannot be rotated up to 30 degrees, but may be corrected to be rotated by a specific angle between 30 degrees and 15 degrees, for example, 20 degrees. 3D matching techniques are very diverse, and a 3D matching algorithm may be determined according to a keypoint descriptor. For example, an Iterative Closest Point (ICP) series, RANdom SAmple Consensus (RANSAC) series, and Gaussian Mixture Model (GMM) series algorithms may be used as the 3D matching technique.

The estimator 370 may generate a plurality of 2D rendered images by applying the corrected postures of the plurality of model point clouds to the 3D model.

The estimator 370 may estimate the posture vt of a specific 2D rendered image having the highest similarity to the object region t among the plurality of 2D rendered images as the posture of the object.

According to the second mode, an initial vector v0 having the highest similarity to the object region t may be applied and a 2D rendering image of a 3D model corrected by depth information may be included in the posture candidate group of the object. In general, the two-dimensional rendered image derived from the initial vector v0 may have a tendency to follow the object region t.

However, there is a case in which the initial vector v0 is selected as a vector having no relation to the target area t at all experimentally with a probability of 10 to 30%. The two-dimensional rendered image derived from the initial vector v0 may be completely different from the object region t. In this case, there is a tendency that a vector similar to the actual target region t exists among vectors having a low similarity, and in order to cover the tendency, the estimator 370 uses a second vector having a low similarity rank to estimate the posture of the object. can

However, in the case of the second vector having a low similarity, posture correction is essential, and depth information may be used for the correction. Therefore, in the second mode, depth information is always required. Therefore, when using the depth information, it is preferable that the estimation apparatus operates in the second mode in which the initial vector v0 is different from the object area t. If depth information is not used, the estimation apparatus may inevitably operate in the first mode in which depth information is not used.

Here, even when a sensor capable of measuring depth information (the above-described TOF sensor, stereo camera, range sensor, etc.) exists and the depth information exists, when the reliability of the depth information is low, the estimator 370 performs the depth information can be decided not to use. In some cases, an option not to manually use depth information may be selected by the user.

In the second mode, the above-mentioned BBox (Bounding Box) method is applied to limit the object area t within the bounding box area, or the masking method is applied to restrict the object area t to within the segmentation area. can do. If the object area is limited within the bounding box area or the segmentation area, the amount of computation may be reduced.

6 is a flowchart illustrating an estimation method according to an embodiment of the present invention.

The estimation method shown in FIG. 6 may be performed by the estimation apparatus of FIG. 1 .

The estimation method of the present invention may include an acquisition step (S710), an extraction step (S720), a determination step (S730), and an execution step (S740, S750).

In the acquiring step ( S710 ), a two-dimensional image i of the object may be acquired. In some cases, depth information of the object and a three-dimensional model of the object may be additionally obtained in the obtaining step ( S710 ). The acquiring step ( S710 ) may be performed by the acquiring unit 310 .

In the extraction step ( S720 ), a plurality of vectors having a similarity higher than a reference value with respect to the object region t representing the object included in the 2D image i may be extracted. Through the extraction step ( S720 ), n vectors having a similarity higher than the reference value may be extracted. The n vectors may include an initial vector v0 having the highest similarity. The extraction step S720 may be performed by the extraction unit 330 .

Through the extraction step (S720), a vector similar to the target area t in a plan view may be mechanically grasped. However, due to errors in mechanical judgment, etc., there are cases where a vector and a posture different from the actual object are selected as the posture of the object, and the determination step (S 730) and the execution step (S 740, S 740, S750) may be further performed.

In the determination step S730, it may be determined whether the depth information of the object is used. The determination step S730 may be performed by the determination unit 350 .

In the execution steps S 740 and S 750 , if it is determined that the depth information is not used, the first mode S 740 may be executed, and if it is determined that the depth information is used, the second mode S 750 may be executed. The execution steps S740 and S750 may be performed by the estimator 370 . In the execution step, the extraction of the first vector and the second vector may be performed by the extraction unit 330 .

7 is a flowchart illustrating a first mode according to an embodiment of the present invention.

The first mode S 740 may include a plurality of steps S 741 , S 742 , S 743 , S 744 , S 745 , and S 746 .

In the first mode, a plurality of first vectors having a similarity to the object region t higher than the first reference value may be extracted from among the plurality of vectors (S 741). For example, the extractor 330 operating in the first mode may extract m first vectors from among n vectors.

In the first mode, the posture of each of the plurality of first vectors may be applied to the three-dimensional model of the object (S 742). The corresponding operation may be performed by the estimator 370 .

In the first mode, a plurality of 2D rendering images may be acquired from a plurality of 3D models to which the postures of the plurality of first vectors are applied ( S743 , S744 , and S745 ). The corresponding operation may be performed by the estimator 370 .

In order to reduce the processing load, the estimator 370 may reduce the processing load of the object area t by applying a BBox method (BBOX) or a masking method (S S 743 ).

When the BBox method is applied, the object area t may be limited within the bounding box area as shown in FIG. 4 by the estimator 370 (S 744).

When the masking method is applied, the object area t may be limited within the segmentation area by the estimator 370 (S 745).

In the first mode, a plurality of 2D rendered images may be compared with the object region t. If the object area t is restricted within the bounding box area or within the segmentation area, the first mode may use the bounding box or segmentation area instead of the object area t as a comparison target of the 2D rendering image. In the first mode, a specific posture of a specific first vector used to extract a specific 2D rendered image having the highest similarity to the object region may be estimated as the posture of the object (S 746). The corresponding operation may be performed by the estimator 370 .

8 is a flowchart illustrating a second mode according to an embodiment of the present invention.

The second mode S 750 may include a plurality of steps S 751 , S 752 , S 753 , S 754 , S 755 , S 756 , S 757 , and S 758 .

In the second mode, an initial vector v0 having the highest similarity to the target region t may be extracted from among the plurality of vectors. In addition, in the second mode, a plurality of second vectors may be extracted in an order in which the degree of similarity to the object region t is lower than the second reference value among the plurality of vectors ( S751 ). In an environment operating in the second mode, the extraction unit 330 may extract n vectors in an order of high similarity, and may extract m second vectors in an order of low similarity within the n range.

In the second mode, the posture of the initial vector v0 and the posture of the plurality of second vectors may be respectively applied to the 3D model of the object (S 752). In addition, in the second mode, the model point cloud may be generated by the number of the initial vector and the plurality of second vectors by using the 3D model to which the posture of the initial vector is applied and the 3D model to which the posture of the plurality of second vectors is applied. (S 752). The corresponding operation may be performed by the estimator 370 .

In the second mode, a target point cloud may be generated using depth information. The corresponding operation may be performed by the estimator 370 .

To reduce the processing load of the model point cloud and the target point cloud, the estimator 370 may selectively apply a BBox method (BBOX) or a masking method (S753).

When the BBox method is selected, the estimator 370 may limit at least one of the model point cloud and the target point cloud within the bounding box area (S 754).

When the masking method is selected, the estimator 370 may limit at least one of the model point cloud and the target point cloud within the segmentation area (S 755).

In the second mode, a plurality of model point clouds may be corrected in a direction that follows a single target point cloud by using a three-dimensional matching technique (S 756). The corresponding operation may be performed by the estimator 370 .

In the second mode, a plurality of two-dimensional rendering images may be generated by applying the corrected postures of the plurality of model point clouds to the three-dimensional model (S 757). The corresponding operation may be performed by the estimator 370 .

In the second mode, the posture vt of a specific 2D rendered image having the highest similarity to the target region among the plurality of 2D rendered images may be estimated as the posture of the object (S 758). The corresponding operation may be performed by the estimator 370 .

9 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 9 may be a device (eg, an estimation device, etc.) described herein. In the embodiment of FIG. 9 , the computing device TN100 may include at least one processor TN110 , a transceiver device TN120 , and a memory TN130 . Also, the computing device TN100 may further include a storage device TN140 , an input interface device TN150 , an output interface device TN160 , and the like. Components included in the computing device TN100 may be connected by a bus TN170 to communicate with each other.

The processor TN110 may execute a program command stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to an embodiment of the present invention are performed. The processor TN110 may be configured to implement procedures, functions, methods, and the like described in connection with an embodiment of the present invention. The processor TN110 may control each component of the computing device TN100 .

Each of the memory TN130 and the storage device TN140 may store various information related to the operation of the processor TN110 . Each of the memory TN130 and the storage device TN140 may be configured as at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory TN130 may include at least one of a read only memory (ROM) and a random access memory (RAM).

The transceiver TN120 may transmit or receive a wired signal or a wireless signal. The transceiver TN120 may be connected to a network to perform communication.

On the other hand, the various methods according to the embodiment of the present invention described above may be implemented in the form of a program readable by various computer means and recorded in a computer readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks ( magneto-optical media), and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language wires such as those created by a compiler, but also high-level language wires that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Above, an embodiment of the present invention has been described, but those of ordinary skill in the art can add, change, delete or add components within the scope that does not depart from the spirit of the present invention described in the claims. It will be said that various modifications and changes of the present invention can be made by, and this is also included within the scope of the present invention.

90...Object 310...Acquisition Department
330...extraction unit 350...judgment unit
370...estimate

Claims (11)

an acquisition unit for acquiring a two-dimensional image of an object;
an extractor configured to extract a plurality of vectors having a similarity with respect to an object region representing the object included in the two-dimensional image higher than a reference value;
a determination unit determining whether to use the depth information of the object; and
Including; an estimator for estimating the posture of the object by using the two-dimensional image,
When it is determined that the depth information of the object is used according to the determination result of the determination unit, the estimator operates in a second mode of extracting a plurality of second vectors according to an inverse order of similarity among the plurality of vectors,
When the estimator operates in the second mode, the estimator extracts the plurality of second vectors from among the plurality of vectors extracted by the extraction unit in an order of lower similarity to the target region, and the plurality of second vectors Estimate the posture of the object using the posture of
The plurality of vectors is n (where n is a natural number of 3 or more),
The plurality of second vectors are smaller than n and m numbers greater than or equal to 2
estimation device.
According to claim 1,
The extraction unit,
Using at least one of a deep learning technique and a similarity determination technique to generate the plurality of vectors in a similar order to the target area, or extract the plurality of vectors in a similar order to the target area from the three-dimensional model of the target, extracting the plurality of vectors in a similar order to the target area from the database in which the plurality of vectors are stored in advance
estimation device.
According to claim 1,
The extraction unit
dividing the entire angular range that the object can take into a plurality of angular ranges, and extracting the plurality of vectors having a similarity higher than the reference value corresponding to each of the partitioned angles in comparison with the object area
estimation device.
According to claim 1,
The extraction unit detects the target area in the two-dimensional image using a deep learning-based target detection algorithm,
The extraction unit localizes the target area,
The extraction unit applies the target area as an input of a pre-learned deep learning generation model,
The extraction unit measures the similarity between the embedded source vector and the storage vector stored in the codebook using the target region as a source in the encoder part of the deep learning generation model,
The extraction unit extracts the plurality of vectors having a high degree of similarity to the target region from among the stored vectors according to the measured similarity.
estimation device.
According to claim 1,
When it is determined that the depth information of the object is not used according to the determination result of the determination unit, it operates in a first mode for extracting a plurality of first vectors according to a ranking having a high degree of similarity among the plurality of vectors
estimation device.
delete an extracting unit for extracting a plurality of vectors having similarities with respect to an object region included in the two-dimensional image satisfying a second reference value; and
Including; an estimator for estimating the posture of the object by using the two-dimensional image and the depth information of the object;
The extraction unit extracts a plurality of second vectors in an order of lower similarity to the target region from among the plurality of vectors,
The estimator estimates the posture of the object using the postures of the plurality of second vectors,
The plurality of vectors is n (where n is a natural number of 3 or more),
The plurality of second vectors are smaller than n and m numbers greater than or equal to 2
estimation device.
8. The method of claim 7,
The estimator applies the postures of the plurality of second vectors to the three-dimensional model of the object, respectively,
The estimator generates a model point cloud by the number of the plurality of second vectors using a three-dimensional model to which the postures of the plurality of second vectors are applied,
The estimator generates a target point cloud by using the depth information,
The estimator corrects a plurality of the model point clouds in a direction to follow the single target point cloud using a three-dimensional matching technique,
The estimator generates a plurality of two-dimensional rendering images by applying the corrected postures of the plurality of model point clouds to the three-dimensional model,
wherein the estimator estimates a posture of a specific 2D rendered image having the highest similarity with the target region among a plurality of the 2D rendered images as the posture of the object
estimation device.
An estimation method performed by an estimation device, comprising:
acquiring a two-dimensional image of an object;
extracting a plurality of vectors having a similarity with respect to an object region representing the object included in the two-dimensional image higher than a reference value;
determining whether to use depth information of the object; and
Comprising the step of executing a second mode when it is determined that the depth information is used,
The second mode is
extracting a plurality of second vectors according to an order in which the degree of similarity to the target region from among the plurality of vectors is lower than a second reference value;
estimating the posture of the object using the plurality of second vectors,
The plurality of vectors is n (where n is a natural number of 3 or more),
The plurality of second vectors are smaller than n and are 2 or more m
Estimation method.
10. The method of claim 9,
When it is determined that the depth information is not used, executing the first mode;
The first mode is
extracting a plurality of first vectors having a similarity to the target region higher than a first reference value from among the plurality of vectors;
applying a posture of each of the plurality of first vectors to the three-dimensional model of the object;
obtaining a plurality of two-dimensional rendering images from a plurality of three-dimensional models to which the plurality of postures of the first vector are applied;
comparing the plurality of two-dimensional rendered images with the target area;
estimating, as the posture of the object, a specific posture of a specific first vector used to extract a specific two-dimensional rendering image having the highest similarity to the target region
Estimation method.
10. The method of claim 9,
The second mode is
applying the postures of the plurality of second vectors to the three-dimensional model of the object, respectively;
generating model point clouds as many as the number of the plurality of second vectors by using the three-dimensional model to which the postures of the plurality of second vectors are applied;
generating a target point cloud by using the depth information;
correcting a plurality of the model point clouds in a direction to follow the single target point cloud using a three-dimensional matching technique;
generating a plurality of two-dimensional rendering images by applying the corrected postures of the plurality of model point clouds to the three-dimensional model; and
estimating, as the posture of the object, a posture of a specific 2D rendered image having the highest similarity to the target region among a plurality of the 2D rendered images
Estimation method.

Images