WO2019235210A1

WO2019235210A1 - Two-dimensional position/attitude deduction apparatus and two-dimensional position/attitude deduction method

Info

Publication number: WO2019235210A1
Application number: PCT/JP2019/020071
Authority: WO
Inventors: 將暢山中
Original assignee: アズビル株式会社
Priority date: 2018-06-04
Filing date: 2019-05-21
Publication date: 2019-12-12
Also published as: JP7178802B2; JP2019211968A

Abstract

The present invention is provided with: an image acquisition unit (301) that acquires an image; a region extraction unit (302) that, from the image acquired by the image acquisition unit (301), extracts an object region that is a region indicating an object of interest; and a position/attitude deduction unit that, from a region based on the object region extracted by the region extraction unit (302), deducts the position of the object based on a reproducible unique point and the attitude of the object based on a reproducible unique direction.

Description

Two-dimensional position / orientation estimation apparatus and two-dimensional position / orientation estimation method

The present invention relates to a two-dimensional position / orientation estimation apparatus and a two-dimensional position / orientation estimation method for estimating the position and orientation of an object on an image.

2. Description of the Related Art Conventionally, there is known a two-dimensional position / orientation estimation apparatus that estimates a position / orientation of a target object from an image using a template matching technique (see, for example, Patent Document 1). The position and orientation of the object include the position (x, y) and orientation (θ) of the object. In this conventional two-dimensional position and orientation estimation apparatus, an object model (template) is registered in advance, and the degree of coincidence of image features between the image and the template is evaluated to estimate the position and orientation of the object on the image. To do.

JP 2016-207147 A

However, in the conventional two-dimensional position / orientation estimation apparatus, in order to estimate the position / orientation of an object by template matching, it is necessary to prepare an enormous number of templates for each object. In addition, since the conventional two-dimensional position / orientation estimation apparatus performs brute force matching, calculation cost and processing time increase.
In view of this, there has been disclosed a technique for performing template matching with high efficiency by hierarchically collecting templates having high similarity and reducing the templates. However, this technique does not change the point that a template needs to be prepared in advance. That is, even with this technique, the effort (labor and time) for preparing a template in advance is not reduced, and there is room for improvement.

The present invention has been made to solve the above-described problems, and an object thereof is to provide a two-dimensional position / orientation estimation apparatus that can detect the position / orientation of an object on an image without using a template. Yes.

The two-dimensional position and orientation estimation apparatus according to the present invention includes an image acquisition unit that acquires an image, an area extraction unit that extracts an object region that is a region indicating a target object from the image acquired by the image acquisition unit, A position / orientation estimation unit that estimates an object position based on a reproducible unique point and a posture of the object based on a reproducible unique direction from an area based on the object region extracted by the area extraction unit; It is characterized by that.

According to this invention, since it is configured as described above, the position and orientation of an object on an image can be detected without using a template.

It is a figure which shows the structural example of the robot picking system provided with the two-dimensional position and orientation estimation apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the structural example of the two-dimensional position and orientation estimation apparatus which concerns on Embodiment 1 of this invention. It is a flowchart which shows the operation example of the two-dimensional position and orientation estimation apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows an example of the image acquired by the image acquisition part in Embodiment 1 of this invention. 5A to 5C are diagrams for explaining the operation of the region extraction unit in the first embodiment of the present invention. 6A and 6B are diagrams for explaining operations by the position estimation unit and the posture estimation unit according to Embodiment 1 of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
Embodiment 1 FIG.
FIG. 1 is a diagram showing a configuration example of a robot picking system provided with a two-dimensional position / orientation estimation apparatus 3 according to Embodiment 1 of the present invention.
The two-dimensional position / orientation estimation apparatus 3 is applicable to, for example, a positioning guide for object picking operation or robot teaching work in FA (factory automation) or the like. Hereinafter, as an example, the two-dimensional position / orientation estimation device 3 is applied to a robot picking system, and the robot picking system is based on the position / orientation of the object 11 that is the target estimated by the two-dimensional position / orientation estimation device 3. The case where the said object 11 is picked is shown.

The robot picking system is, for example, arranged on a production line of a factory and picks the object 11 arranged on the work surface 12. Note that the number of objects 11 arranged on the work surface 12 is not limited to one, but a plurality of objects 11 may be provided, but when there are a plurality of objects 11, each object 11 is arranged on the work surface 12 without overlapping. Shall. As shown in FIG. 1, this robot picking system includes a robot 1, a camera (imaging device) 2, a two-dimensional position / orientation estimation device 3, and a robot controller 4.

The robot 1 has an end effector 101 (a hand in FIG. 1) at the tip of an arm, and picks the object 11 using the end effector 101.

The camera 2 is attached to the end effector 101 and images an imaging area to obtain an image. Data (image data) indicating an image obtained by the camera 2 is output to the two-dimensional position / orientation estimation device 3.
The image obtained by the camera 2 may be an image that can distinguish the region (object region) indicating the object 11 from the other region (background region) when the object 11 exists in the imaging region. Any format is acceptable. The camera 2 may obtain a two-dimensional image, for example. The two-dimensional image generally includes a color or monochrome visible image, but is not limited to this, and when the above-described determination is not easy with a visible image, a near-infrared image, a multispectral image, etc. A special two-dimensional image may be used. The camera 2 may obtain a distance image when the object 11 has a certain depth.

The two-dimensional position / orientation estimation apparatus 3 estimates the position / orientation of the object 11 on the image based on the image obtained by the camera 2. The position / orientation of the object 11 means the position (x, y) and orientation (θ) of the object 11. Data (estimation result data) indicating an estimation result by the two-dimensional position and orientation estimation device 3 is output to the robot controller 4. A configuration example of the two-dimensional position / orientation estimation apparatus 3 will be described later.

The robot controller 4 calculates the movement amount of the arm of the robot 1 based on the estimation result by the two-dimensional position / orientation estimation device 3 and controls the movement of the robot 1 so as to pick the object 11.

Next, a configuration example of the two-dimensional position / orientation estimation apparatus 3 will be described with reference to FIG.
As illustrated in FIG. 2, the two-dimensional position / orientation estimation apparatus 3 includes an image acquisition unit 301, a region extraction unit 302, a position estimation unit 303, an orientation estimation unit 304, and an output unit 305. The two-dimensional position / orientation estimation apparatus 3 is realized by a processing circuit such as a system LSI (Large Scale Integration), or a CPU (Central Processing Unit) that executes a program stored in a memory or the like.

The image acquisition unit 301 acquires an image obtained by the camera 2.

The region extraction unit 302 extracts an object region from the image acquired by the image acquisition unit 301 by image processing. That is, the area extraction unit 302 extracts an object area from the image based on the degree of contrast change. As a method for extracting an object region by the region extraction unit 302, for example, threshold processing or a background difference method is conceivable. However, there is no particular limitation as long as the method can extract the object region by distinguishing the object region and the background region. .

The position estimation unit 303 estimates the position of the object 11 based on a reproducible unique point from the region based on the object region extracted by the region extraction unit 302. The region based on the object region extracted by the region extraction unit 302 includes the object region, a region that includes the object region in a convex shape, a region that approximates the object region with a geometric shape, and the object region that has a geometric shape. One or more of the minimum area included or the minimum area circumscribed by the geometrical shape of the object area are included. Examples of the geometric shape include a rectangle or an ellipse. A reproducible unique point is a point that is uniquely determined without depending on the posture of the object 11.
The position estimation unit 303 uses, for example, a centroid of a region based on the object region extracted by the region extraction unit 302 as a reproducible unique point, and the centroid and an arbitrarily set reference point on the image. The position of the object 11 is estimated based on the relative relationship. An example of the reference point on the image is the center point of the image.

The posture estimation unit 304 estimates the posture of the object 11 based on a reproducible unique direction from the region based on the object region extracted by the region extraction unit 302.
The posture estimation unit 304 uses, for example, the major axis direction of the region based on the object region extracted by the region extraction unit 302 as a reproducible unique direction, and the relative direction between the major axis direction and an arbitrarily set reference direction Based on the relationship, the posture of the object 11 is estimated. Examples of the reference direction include the horizontal direction of the image.

The output unit 305 outputs data indicating the estimation result by the position estimation unit 303 and the estimation result by the posture estimation unit 304 to the robot controller 4.

Note that the position estimation unit 303 and the posture estimation unit 304 indicate that “from the region based on the object region extracted by the region extraction unit 302, the position of the object 11 based on a reproducible unique point and the reproducible unique direction. The position / orientation estimation unit for estimating the attitude of the object 11 based on “is configured.

Next, an operation example of the two-dimensional position / orientation estimation apparatus 3 shown in FIG. 2 will be described with reference to FIG.
In general, when expressing the posture of the object 11, as represented by Euler angles (α, β, γ) and quaternions (w, x, y, z), the amount of rotation (angle) ) In many cases. In this case, since the object 11 itself does not have the reference, the user needs to arbitrarily set the reference. There are no particular restrictions on the setting of standards. Therefore, the posture estimation unit 304 can express the posture of the object 11 without using a reference template by deriving a unique direction with reproducibility from the region based on the object region.
In addition, when the posture of the object 11 is expressed on the image, it is often expressed mainly by a rotation amount around the Z axis (the optical axis direction of the camera 2). Therefore, it is impossible (unnecessary) to express the posture of the object 11 having a rotationally symmetric shape around the Z axis. For this reason, the object 11 requiring posture estimation on the image is limited to one having some geometric feature in the shape. When the object 11 is assumed to have some geometric feature in the shape, the major axis direction of the region based on the object region is uniquely determined. Therefore, in the following, the posture estimation unit 304 represents the posture of the object 11 by using the long axis direction.
The same applies to the position of the object 11, and the position estimation unit 303 derives a reproducible unique point from the area based on the object area, thereby determining the position of the object 11 without using a reference template. It can be expressed. Since the centroid of the region based on the object region is a uniquely determined feature point, the position estimation unit 303 represents the position of the object 11 using the centroid below.

In the operation example of the two-dimensional position / orientation estimation apparatus 3 according to the first embodiment, as shown in FIG. 3, first, the image acquisition unit 301 acquires an image obtained by the camera 2 (step ST1).

Next, the region extraction unit 302 extracts an object region from the image acquired by the image acquisition unit 301 by image processing (step ST2). Here, it is assumed that the region extraction unit 302 extracts an object region from an image by threshold processing. Assume that the image acquisition unit 301 acquires an image as shown in FIG. This image includes the object 11, the work surface 12, and other backgrounds. In the two-dimensional position / orientation estimation apparatus 3 according to Embodiment 1, since no template is prepared, it is not easy to directly extract an object region from an image. Therefore, in the two-dimensional position / orientation estimation apparatus 3 according to the first embodiment, first, as shown in FIG. 5A, for example, an area indicating the work surface 12 (gray area shown in FIG. 5A) 501 is extracted from the image by threshold processing. It is desirable to do. Generally, in the production line, the object 11 is almost always disposed on the work surface 12. Therefore, in the area extraction unit 302 according to the first exemplary embodiment, an area 501 indicating the work surface 12 and an area (gray area illustrated in FIG. 5B) 502 in which the area is filled by closing processing or the like are included. By taking the difference, the object region (gray region shown in FIG. 5C) 503 can be extracted.

Next, the position estimating unit 303 estimates the position of the object 11 based on a reproducible unique point from the region based on the object region extracted by the region extracting unit 302 (step ST3). Here, the position estimation unit 303 uses the centroid of the region as a reproducible unique point, and determines the position of the object 11 based on the relative relationship between the centroid and an arbitrarily set reference point on the image. presume.

Specifically, the position estimation unit 303 first derives the centroid of the region based on the object region extracted by the region extraction unit 302 as a reproducible unique point. Here, when the coordinates of the points constituting the region are represented by (u _i , v _i ) (i, j = 1,..., N), the position estimation unit 303 performs the 0th-order moment of the region. And the centroid of the region is derived from the following equations (1) to (4) using the first moment. Note that N is the number of points constituting the region. In equations (1) to (4), m ₀₀ represents the 0th-order moment of the region, (m ₁₀ , m ₀₁ ) represents the first-order moment of the region, and (u (bar), v (bar )) Represents the centroid of the area.

Then, the position estimating unit 303 estimates the position of the object 11 from the following equation (5) based on the derived centroid and reference point. In Equation (5), (u ₀ , v ₀ ) represents a reference point, and (x, y) represents the position of the object 11.

Also, the posture estimation unit 304 estimates the posture of the object 11 based on a reproducible unique direction from the region based on the object region extracted by the region extraction unit 302 (step ST4). Here, the posture estimation unit 304 uses the long axis direction of the region as a reproducible unique direction, and estimates the posture of the object 11 based on the relative relationship between the long axis direction and an arbitrarily set reference direction. To do.

Specifically, when the reference direction is the horizontal direction on the image, the posture estimation unit 304 calculates from the following equations (6) to (9) based on the second moment of the region and the centroid of the region. The posture of the object 11 is estimated by deriving the rotation angle from the horizontal direction of the long axis direction. In equations (6) to (9), (m ₂₀ , m ₀₂ , m ₁₁ ) represents the second moment of the above region, and θ represents the rotation angle.

For example, assume that the region extraction unit 302 has extracted an object region 503 as shown in FIG. 5C. Further, it is assumed that the region used by the position estimation unit 303 and the posture estimation unit 304 is a minimum rectangular region 601 including the object region 503 as illustrated in FIG. 6A. In this case, the centroid 602 derived by the position estimation unit 303 and the long axis direction 603 derived by the posture estimation unit 304 are as shown in FIG. 6B.

Further, the position estimation unit 303 estimates the position of the object 11 with respect to each of a plurality of areas such as the object area 503 shown in FIG. 5C and the rectangular area 601 shown in FIG. 6A, and estimates them. The position of the object 11 may be estimated based on the result. Thereby, the position estimation unit 303 can improve the accuracy of position estimation. The same applies to the posture estimation unit 304.

In the above description, the position estimation unit 303 derives the centroid of the region. However, the present invention is not limited to this, and the position estimation unit 303 may estimate the position of the object 11 using, for example, feature points of the outline of the region (for example, a right-angle portion 504 of the object region shown in FIG. 5C).
In the above description, the posture estimation unit 304 has derived the major axis direction of the region. However, the present invention is not limited to this, and the posture estimation unit 304 may estimate the posture of the object 11 using, for example, a line segment that connects the centroid of the region and the feature points of the outline of the region.

Next, the output unit 305 outputs data indicating the estimation result by the position estimation unit 303 and the estimation result by the posture estimation unit 304 to the robot controller 4 (step ST5). Thereafter, the robot controller 4 controls the operation of the robot 1 to pick the object 11 based on the estimation result by the two-dimensional position / orientation estimation apparatus 3. As a result, the robot picking system can pick the object 11 even when the position and orientation of the object 11 are indefinite.

As described above, according to the first embodiment, the two-dimensional position / orientation estimation apparatus 3 acquires the target object 11 from the image acquisition unit 301 that acquires an image and the image acquired by the image acquisition unit 301. A region extracting unit 302 that extracts an object region that is a region to be shown, and a region based on a unique point having reproducibility and a reproducible unique point from a region based on the object region extracted by the region extracting unit 302 A position and orientation estimation unit that estimates the orientation of the object 11 based on the direction. Thereby, the two-dimensional position / orientation estimation apparatus 3 according to Embodiment 1 can detect the position / orientation on the image of various objects 11 without using a template.

In the present invention, any component of the embodiment can be modified or any component of the embodiment can be omitted within the scope of the invention.

The two-dimensional position / orientation estimation method and the two-dimensional position / orientation estimation method according to the present invention acquire an image, extract an object region that is a region indicating a target object from the acquired image, and extract the extracted object region. The position and orientation of the object on the image can be detected by estimating the position of the object based on a unique point with reproducibility and the posture of the object based on a unique direction with reproducibility Since it is configured, it is suitable for use in positioning guides for object picking operations or robot teaching operations in FA or the like.

1 Robot 2 Camera (imaging device)
3 Two-dimensional position and orientation estimation device 4 Robot controller 11 Object 12 Work surface 101 End effector 301 Image acquisition unit 302 Area extraction unit 303 Position estimation unit 304 Posture estimation unit 305 Output unit

Claims

An image acquisition unit for acquiring images;
A region extraction unit that extracts an object region that is a region indicating a target object from the image acquired by the image acquisition unit;
A position and orientation estimation unit that estimates the position of the object based on a reproducible unique point and the orientation of the object based on a reproducible unique direction from a region based on the object region extracted by the region extraction unit; A two-dimensional position and orientation estimation apparatus.
The two-dimensional position / orientation estimation apparatus according to claim 1, wherein the image acquired by the image acquisition unit is a two-dimensional image or a distance image.
The two-dimensional position / orientation estimation according to claim 1, wherein the region extraction unit extracts an object region from an image acquired by the image acquisition unit based on a degree of change in contrast. apparatus.
The position / orientation estimation unit includes, as the region based on the object region extracted by the region extraction unit, the object region, a region enveloping the object region, a region approximating the object region with a geometric shape, the object region One or more of a minimum region that includes a geometric shape and a minimum region that circumscribes the object region with a geometric shape is used. The two-dimensional position and orientation estimation apparatus according to claim 1.
The position and orientation estimation unit uses a centroid of a region based on the object region extracted by the region extraction unit as a reproducible unique point, and based on a relative relationship between the centroid and a reference point, The position is estimated. The two-dimensional position and orientation estimation apparatus according to any one of claims 1 to 4, wherein the position is estimated.
The position and orientation estimation unit uses the major axis direction of the region based on the object region extracted by the region extraction unit as a reproducible unique direction, and based on the relative relationship between the major axis direction and the reference direction, The two-dimensional position and orientation estimation apparatus according to any one of claims 1 to 4, wherein the posture of the object is estimated.
An image acquisition unit acquiring an image;
A region extracting unit extracting an object region which is a region indicating a target object from the image acquired by the image acquiring unit;
The position / orientation estimation unit uses the position of the object using a reproducible unique point and the reproducible unique direction from the region based on the object region extracted by the region extraction unit. A two-dimensional position / orientation estimation method comprising: