JP2003136465A - Three-dimensional position and posture decision method of detection target object and visual sensor of robot - Google Patents

Abstract

PROBLEM TO BE SOLVED: To eliminate needs to develop a detection algorithm for each target object when calculating a three-dimensional position and a posture 3 of the target object. SOLUTION: In a target object, a distinctive part of relatively simple shape, such as a cylinder, is captured and its central axis is determined by a three- dimension sensor. A camera mounted on a wrist of a manipulator is guided on the central axis to pickup a two-dimensional image of the target object. The target object in this two-dimensional image is compared with a model to detect the rotation angle, and the three-dimensional position of the detection target object is calculated from the rotation angle and an operation information of the manipulator.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for determining the position and orientation of a detection target in an automation system using an industrial robot.

[0002]

2. Description of the Related Art A conventional visual sensor used for recognizing the position and orientation of a work object in an industrial robot is a CCD.
A two-dimensional image processing device that processes an image obtained by a camera is often used. However, in two-dimensional image processing using a single CCD camera, when trying to realize picking (bin picking) of workpieces that are piled up in bulk, tilting due to overlapping of objects causes Since the appearance changes, there is a limit to the correspondence only with the two-dimensional image processing of the method of searching the two-dimensional image model. In order to realize this bin picking, International Patent Publication WO97-
2426 describes a multi-sensor robot system that combines a CCD camera and a laser sensor. Hereinafter, a brief description will be given with reference to the drawings.

FIG. 8 is a diagram for explaining this conventional technique.
This system projects three-dimensional position measurement by projecting light onto a first sensor means 830 having the ability to acquire a two-dimensional image in a relatively wide area 831 and an area of a relatively narrow area 811. Second having capability (irradiation from the laser emission unit 812 and reception by the light reception unit 813 for measurement)
This system is a system in which a robot is combined with a composite sensor equipped with the sensor means 810 of FIG.

This system also includes a first sensor means 8
Means for outputting an operation command for 30 (this means is not shown because it is a fixed camera in the figure), means for outputting an operation command for the second sensor means 810, and first sensor means Means for processing the image signal acquired by 830 (image processing device 802), means for processing the measurement output signal of the second sensor means 810 (laser sensor control unit 820), and robot control means for controlling the robot 840. (Robot controller 801).
The processing of the image signal acquired by the first sensor means 830 (processing in the image processing device 802) is performed in order to specify the position of the measuring object W on the work table 18 within a relatively wide area. The robot control means (robot controller 801) includes means for causing the robot to perform an approach operation on the measurement target object based on the specified position of the measurement target object. The processing of the measurement output signal of the second sensor unit 810 includes the processing of the measurement output signal (laser sensor control unit 820) output after the second sensor unit 810 moves to the approach position by the approach operation. There is.

In short, according to this conventional technique, the first two-dimensional visual sensor is used to see the rough position of the object,
The robot guides the second three-dimensional sensor to an appropriate position based on the obtained position information of the object. As a result, even if the position or orientation of the target object is unknown, the robot can quickly approach the target object and start the measurement by the sensor means having a three-dimensional position measuring capability such as a laser sensor. This leads to improved reliability of various work using robots and shortened cycle time.

Further, Japanese Unexamined Patent Publication No. 7-136959 also discloses 2
Another prior art combining a three-dimensional visual sensor and a three-dimensional visual sensor is shown. This conventional technique will be described with reference to FIG. The input image feature extraction unit 920 inputs the two-dimensional image of the target object captured by the fixed camera 910, and extracts the feature line segment and the feature point. Image / object association unit 9
40 associates the extracted feature line segment and feature point with the feature of the line segment description and point description of the object model prepared in advance in the object model description storage unit 930, and the object posture estimation unit 950 The orientation of the target object is estimated from this association. The robot primary guiding unit 960 guides the robot according to the estimated posture. The robot secondary guiding unit 980
The robot position is corrected based on the position information from the range sensor 970 on the robot hand and the position information from the camera 910 obtained at the time of guiding.

In short, the features of the two-dimensional image of the target object picked up by the image pickup means and the object model of the target object prepared in advance are associated with each other to estimate the posture of the target object,
After moving the robot according to the estimated posture obtained in this way, the accurate position of the object is estimated by integrating the object position observed by the range sensor and the object position obtained from the camera information. This conventional technique can be widely applied to realize an industrial visual sensor or a visual system of a robot, and can position an object even when an accurate shape model is not given in advance.

[0008]

However, in all of these conventional techniques, a two-dimensional sensor is used to obtain a rough position,
Based on the position, the three-dimensional sensor is guided to an appropriate measurement position to improve the detection accuracy, but there are the following problems. First, we have not dealt with the simplification of programming for each application, which is actually a problem in image processing. In addition, "first obtain a rough position with a two-dimensional sensor"
However, it is difficult for a two-dimensional sensor to detect an object whose posture is unclear, and it is not possible to cope with a large change in posture.

First, reference will be made to the simplification of programming for each application. The use of visual sensors for robot work has been performed for about 20 years, but has not been used in actual work as expected up to now. One of the main reasons for this is that it is often necessary to develop or change the recognition algorithm or parameter for each object to be detected.
For the two-dimensional visual sensor, just show the object,
Techniques for automating appropriate algorithm selection and parameter adjustment have been developed and are becoming popular. However, in the three-dimensional visual sensor, it is general to develop an algorithm by setting a detection strategy for each object, and in this respect, the simplicity of the two-dimensional vision is far short. In order to detect the three-dimensional position / orientation of an object with a sensor in this way, it is necessary to develop software for detection for each object to be detected, and the cost for the development is too high. Not introduced. This has hindered the spread of three-dimensional sensors.

Therefore, a first object of the present invention is to detect the three-dimensional position and orientation of an object without the need to develop a detection algorithm for each object. It is to provide a posture determination method and a visual sensor for a robot used for the method.

Next, regarding the recognition of an object whose posture is unclear, when the object is recognized by a two-dimensional sensor, rotation about an axis perpendicular to the two-dimensional image (rotation of the object in the screen) or two-dimensional The change in the size of the object in the image (change in the distance from the camera to the object) has already been technically established, but other changes in the posture are difficult with the two-dimensional image processing. Yes, that is the biggest reason why we need a three-dimensional sensor. In the above-mentioned two conventional techniques, the approximate position of the object is determined by the two-dimensional sensor, which means that these conventional techniques can be implemented only when the posture of the object does not change much. That is the implicit statement.

Therefore, a second object of the present invention is to provide a method for determining the three-dimensional position / orientation of an object to be detected which can flexibly cope with a large attitude change of the object and a robot vision sensor used therefor. It is to be.

[0013]

According to the present invention, the features of the three-dimensional visual sensor and the two-dimensional visual sensor are combined to achieve the above object. That is, the position and orientation of a relatively simple (eg, cylindrical) characteristic part of the detection target is obtained by the three-dimensional sensor, and the manipulator is operated with this as a reference to operate the camera to detect the characteristic part. To obtain a two-dimensional image of the object, and compare the image with the model image of the object to be detected to determine the deviation from the reference position of the object to be detected. A new position and orientation. As a result, the three-dimensional sensor only needs to have an algorithm for deriving a characteristic part having a relatively simple shape, and recognition of objects other than the characteristic part that changes for each object is already performed. The change is left to a two-dimensional visual sensor with established technology, and development of detection software for each object is avoided.

That is, a manipulator equipped with a camera capable of acquiring a two-dimensional image of a detection target and a three-dimensional sensor capable of acquiring a distance image of the detection target is operated to determine the three-dimensional position and orientation of the detection target. The method of the present invention is 3
The characteristic part of the detection target is detected from the distance image of the detection target acquired by the three-dimensional sensor, the position of the characteristic part is obtained, and the position of the characteristic part thus obtained is used as a reference for the camera. To the detection target, and the two-dimensional image of the detection target captured by the above-mentioned camera is
The displacement of the detection target with respect to the parallel movement and the rotation angle is obtained by comparing with the three-dimensional model, and the detection target's 3 It is characterized by determining the dimensional position and orientation.

The portion forming the characteristic portion of the object to be detected is a portion having the same shape when rotated by an angle within 360 degrees around the central axis of the portion. The position is given by the centerline vector and the center position, and the distance between the camera optical axis and the centerline vector of the characteristic part is set so that the camera can image the detection target from the center position of the characteristic part. The cameras are made to face the detection target by arranging them separately.

The characteristic portions are, for example, a cylindrical convex portion, a concave circular portion, a polygonal convex portion, a concave regular polygonal portion, a conical convex portion, and a concave portion. A conical portion, a regular polygonal pyramid-shaped convex portion, and a regular polygonal pyramid portion which are concave portions.

The robot visual sensor of the present invention is equipped with a manipulator equipped with a camera capable of acquiring a two-dimensional image of a detection target and a three-dimensional sensor capable of acquiring a distance image of the detection target, and acquired by the three-dimensional sensor. The distance image processing unit that detects a characteristic part of the detection target from the distance image of the detected target and obtains the position of the characteristic part, and the position of the characteristic part thus obtained as a reference A manipulator guiding unit that causes the camera to face the object to be detected,
A two-dimensional image processing unit that compares a two-dimensional image of the detection target imaged by the camera with a two-dimensional model of the detection target to obtain a displacement of the detection target object with respect to translation and rotation angle, and this two-dimensional image The three-dimensional position of the detection target is connected to the processing unit and the manipulator guiding unit, and based on the displacement of the detection target and the operation information of the manipulator when the camera faces the detection target. An object three-dimensional position deriving unit that determines a posture is provided.

[0018]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A visual sensor device of the present invention will be described with reference to FIGS. FIG. 1 shows the configuration of the present visual sensor. C is attached to the wrist flange 13 of the general 6-axis manipulator 12 whose operation is controlled by the manipulator guiding section 5.
A CD camera 14 is installed. Imaging information of the CCD camera 14 is input to the two-dimensional image input unit 6. The wrist flange and the CCD camera origin are calibrated. Further, a laser sensor 17 as a three-dimensional sensor is fixedly installed downwardly above the work table 18. The three-dimensional information acquired by the laser sensor 17 is the distance image input unit 1
Entered in. The origin of the laser sensor 17 and the manipulator origin 21 are calibrated. The detection target 15 is placed on the work table 18.

2 and 3 show the coordinate relationship of each component.
ing. 2 and 3 show the coordinate relationships of the same scene.
However, it is divided into two because the figure near the object becomes complicated.
It In the figure, the same items are indicated by the same numbers.
It Three-dimensional laser sensor seen from manipulator origin 21
Service origin 25^rT_l, The cylindrical part of the object detected by the laser sensor
Center position (detection position) 19 of^lP₀, Laser Sen
From the origin^lP₀Conversion of 26^lT₀, The flat top of the cylinder
Centerline vector 20 which is the direction perpendicular to the plane ^lV₀
(still,^lP₀and^lV₀Viewed from the laser sensor origin
It is expressed in coordinates. ), Seen from the origin of the manipulator
Camera origin conversion 22^rT_c, From the CCD camera origin
A point at a distance L to the target object (of the upper surface of the cylindrical portion of the detection target object)
Center position^lP₀The conversion 23 up to 19)^cT₀, CC
D camera image origin (for convenience of explanation, the origin is set in the center of the camera image.
Position) to convert the detection object origin 24⁰T_pAnd

The processing contents of the embodiment will be described below in time series.

1) A three-dimensional laser range sensor 17 (having a laser irradiation unit and a light receiving unit) irradiates the object 15 on the work table 18 with laser light, and the obtained data is used as the range image input unit 1 Is converted into a range image by. This range image data is position information of the object from the laser sensor origin.

2) Range image processing is performed on the range image data.
The object 15 is extracted by the characteristic part extraction unit 3 which is an element of the unit 2.
Of the cylindrical part 16 that is a part of the
^lV ₀And the center position of the top surface of the cylinder^lP₀Derive
It The specific method for three-dimensional detection in this case is "3rd order
Original image measurement (Co-authored by Shokodo, Seiji Iguchi, Kosuke Sato 199
6.1.1 Bin Pickin of "November 19th, 1st edition)"
Using the method described on page 146
It is possible now. Specifically, the edge part in the distance image is extracted.
Out, divide it into faces separated by its edges, and
It can be obtained by applying a circle equation to the surface.
You can

3) Using the derived center position ^l P ₀ 19 and center line vector ^l V ₀ 20, the optical axis of the camera 14 installed on the manipulator wrist flange 13 passes through the center line vector ^l V ₀ 20. , The position of the camera lens 21 is a preset distance L from the center point 19
The manipulator guiding unit 5 guides the manipulator so that the manipulator is moved to a position separated by a distance. It should be noted that this distance L is set to a distance at which the camera can capture an image of the entire object or an area within the object where rotation can be detected within the two-dimensional screen. The process of the manipulator guiding unit 5 will be described in detail later.

4) The camera 14 captures an image of the object, and the captured image is input to the two-dimensional image input unit 6.

5) The input two-dimensional image is processed by the two-dimensional image processing section 7 to derive the two-dimensional position and rotation angle of the object. The processing content is the model input unit 8 of the two-dimensional image processing unit 7.
The two-dimensional model previously set in
The model matching unit 9 searches for a three-dimensional image and detects a matching image. The two-dimensional model covers the entire object or a region in the two-dimensional screen where rotation can be detected in the object.

FIG. 5 is an image diagram in which an image corresponding to the two-dimensional model 33 is detected from the two-dimensional image 34 input from the model input unit 8. The cylindrical portion 35 is a CCD
Two-dimensional model 3 to guide the camera on its central axis
Both the 3D image and the 2D image 34 are located at the center of the image.
At the time of search, the image of the model is translated, rotated, or enlarged or reduced to perform the search. Since the CCD camera distance from the position of the cylinder obtained by the three-dimensional sensor in advance is fixed, the search amount other than rotation is actually small as shown in the figure. In the case of the figure, the image is detected by rotating clockwise by about 45 degrees.
Incidentally, the parallel movement, rotation, and enlargement / reduction search functions of the model image are provided as standard in a general commercially available two-dimensional visual sensor, and a description thereof will be omitted. In FIG. 5, the center of the cylindrical portion and the coordinate origin 36 of the object do not coincide with each other, and they are set as a fixed shift amount ( ⁰ X _p , ⁰ Yp, 0). Assuming that the detected rotation angle is α and the detected deviation is (Δx, Δy, 0), the position / rotation angle deriving unit 10 derives the transformation ⁰ T _p of the detection object origin from the camera image origin as follows.

[0027]

[Equation 1] 6) The object three-dimensional position deriving unit transforms the camera image origin obtained by the two-dimensional image processing unit from the origin of the detected object ⁰ T _p ,
And the camera coordinate position obtained from the manipulator guide
^{From r} T _c e, the three-dimensional position ^r T _p of the object is derived as follows.

The processing of the ^r T _p = ^r T _c e ^c T ₀ ⁰ T _p manipulator guiding unit 5 will be described with reference to FIG. The manipulator is located at an arbitrary position, and when the manipulator moves from the CCD camera position 29 to the target position 31, the processing contents are the following steps.

[0029]  ^lP₀and^lV₀The manipulator
Convert to coordinate system. The manipulator guide is
The laser sensor origin 25 viewed from the originator^rT_l
Is set. To convert this^rT_lUsing Manipi
Center position from the origin^rP₀,manipulator
Centerline vector position from origin^rV₀And     ^rP₀=^rT_l ^lP₀                                 Formula 1     ^rV₀=^rT_l ^lV₀                                 Formula 2 Becomes

At the target position 31 of the CCD camera origin, the direction of the camera (only the approach direction) is known, and the rotation angle around it is arbitrary, but it is necessary to determine that value in order to operate. Yes, there is a manipulator that asks for it as follows,
It will guide from that position to the target position. C from the manipulator origin at the operation start position of the manipulator
CD camera origin position / orientation start position / orientation 30 ^r
Let T _c s and target position / posture 32 be ^r T _c e, respectively.

[Equation 2] Can be expressed as

^Let the position element vector of ^r T _c s be ^r P _c s,
The orientation element matrix ^{r R} _c ^{s, r} _{T c} the position element vector ^r P of e _c e, the attitude element matrix ^r R _c e
And

Further, the ^r V ₀ is

[Equation 3] Can be expressed as

[0033] When the unit vector of ^r V ₀ and ^r v ₀

[Equation 4] Is.

The target position ^r P _c e is

[Equation 5] Required by.

The posture element matrix ^r R _c e of ^r T _c e, which is the target posture, is obtained as follows.

The approach direction posture element unit vector A _e of ^r R _c e is the inversion of the sign of the unit vector ^r v ₀ of the centerline vector.

[0037]

[Equation 6] ^r R _c e is ^r R _c s in the outer product kr around the approach direction orientation component unit vector _{A s} and _{A e} only _{A s} and _A angles obtained from the inner product of _{e θ} ^r T _c s orientation element matrix ^r R
Calculate by rotating _c s.

Kr = A _s × A _e Formula 9 θ = cos ⁻¹ (A _s · A _e ) Formula 10 ^r R _c e = ^r R _c sRot (kr, θ) Formula 11 Manipulator guidance target position and attitude ^r T _c e
Is derived.

The manipulator is operated to guide the CCD camera to the guidance target position ^r T _c e.

Next, an example of a shape other than the cylindrical portion is shown in FIG.
This will be described with reference to FIG. The above description has been given by taking the cylindrical part as the three-dimensional extraction target, but the three-dimensional extraction target may be a concave part of the cylindrical part, or a conical (convex part, concave part) regular polygonal column (convex part, concave part), regular polygonal cone. Any shape such as (convex portion or concave portion) may be used as long as it has the same angle when rotated about the center axis of the object without rotating 360 degrees.

A case where a work having a regular hexagonal prism (general shape of a bolt) portion as a regular polygonal prism is targeted will be described. FIG. 6 is a diagram showing the relationship of coordinate systems, and FIG. 7 is an image diagram in which a two-dimensional model image is searched from the acquired two-dimensional image. The processing content differs from the described embodiment in the processing content of the characteristic part extraction unit 3 in FIG. 1, and here, not a cylinder but a regular hexagonal prism is extracted. The specific method for three-dimensional detection is also "Three-dimensional image measurement (Shokodo, Seiji Iguchi, Kosuke Sato, November 19, 1990, first edition)".
This can be realized by using the method described in 6.1.1, Bin picking, page 146. Specifically, the edge portion in the distance image is extracted, divided into the surfaces divided by the edge, and the central axis of the regular hexagonal prism is obtained from the positional relationship of the planes. After that, as in the described embodiment, the CCD camera is guided to the central axis, and the image 39 acquired by the CCD camera is rotated so as to match the image image corresponding to the two-dimensional model 38 prepared in advance. The rotation angle is obtained, and the position and orientation of the object are derived from the manipulator position.

The three-dimensional sensor need not be a laser range sensor as long as it can acquire a range image. After all, in the present invention, the characteristic part extraction unit 3 of the three-dimensional image processing unit is provided with a relatively simple characteristic of the object. If a detection algorithm corresponding to the shape of a different part is prepared, it will suffice.

It is also possible to replace with a passive stereo camera system. Although the three-dimensional sensor is fixed in the embodiment, it may be attached to the wrist of the robot.

If the object to be detected has a shape, such as a cylinder, which has the same shape when rotated about the central axis of the object without rotating it by 360 degrees, first,
The two-dimensional image processing is made possible by obtaining the central axis of the two-dimensional sensor and guiding the CCD camera attached to the robot wrist on the central axis. Two-dimensional image processing has now reached a technical level that does not require algorithm development for each object. This eliminates the need to develop dedicated three-dimensional object detection software when detecting a three-dimensional object. For this reason, it becomes possible to easily apply it to the work that requires three-dimensional object detection, and the application of the robot to bin picking and the like is promoted.

[0045]

According to the present invention, the three-dimensional sensor only needs to have an algorithm for deriving a relatively simple shape part of the object, for example, a cylindrical part, and the cylindrical part that changes for each object. Since the recognition other than that is left to the two-dimensional visual sensor, which has established a generalized technique for changing the object,
It is not necessary to develop software for detecting each object as a system. Thereby, the applicable range of the three-dimensional sensor can be dramatically expanded.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of a visual sensor device of the present invention.

FIG. 2 is a diagram showing a relationship of coordinate systems related to a three-dimensional sensor of the visual sensor device of the present invention.

FIG. 3 is a diagram showing a relationship of a coordinate system related to a two-dimensional sensor of the visual sensor device of the present invention (a characteristic part is a cylinder).

FIG. 4 is a diagram illustrating a manipulator operation of the visual sensor device of the present invention.

FIG. 5 is an image diagram for searching a two-dimensional model image from a two-dimensional image (a characteristic part is a cylinder).

FIG. 6 is a diagram showing a relationship between coordinate systems of the visual sensor device of the present invention (a characteristic portion is a regular polygonal prism).

FIG. 7 is an image diagram in which a two-dimensional model image is searched from a two-dimensional image (a characteristic part is a regular polygonal prism).

FIG. 8 is a block diagram showing a configuration of a conventional technique. .

FIG. 9 is a block diagram showing the configuration of another conventional technique.

[Explanation of symbols] 1 Distance image input section 2 Distance image processing unit 3 Feature part extraction unit 4 Position / posture derivation unit 5 Manipulator guide 6 Two-dimensional image input section 7 Two-dimensional image processing unit 8 model input section 9 Model matching section 10 Position / rotation angle derivation unit 11 Object 3D position derivation unit 12 Manipulator 13 wrist flange 14 cameras 15 Target 16 characteristic parts 17 Laser sensor (three-dimensional sensor)

   ─────────────────────────────────────────────────── ─── Continued front page    F term (reference) 3C007 BS12 KT01 KT05 KV00 KX19                       MT00                 5B057 AA05 CA12 CA13 CA16 DC05                       DC08 DC36                 5L096 AA09 BA05 CA05 FA05 FA67                       FA69 JA11

Claims (4)

[Claims] 1. A manipulator equipped with a camera capable of acquiring a two-dimensional image of a detection target and a three-dimensional sensor capable of acquiring a range image of the detection target is operated to operate a three-dimensional detection target.
In the method of determining the three-dimensional position / orientation, a characteristic part of the detection target is detected from a distance image of the detection target acquired by a three-dimensional sensor, and the position of the characteristic part is calculated. The camera is made to face the detection target object based on the position of the target part, and the two-dimensional image of the detection target imaged by the camera is compared with the two-dimensional model of the detection target object to translate and rotate. Is obtained, and the three-dimensional position / orientation of the detection target is determined from this shift and the operation information of the manipulator when the camera is directly opposed to the detection target. 3D position / posture determination method of the detected object. 2. A portion forming a characteristic portion of the detection target is a portion having the same shape when rotated by an angle of 360 degrees or less around the central axis of the portion. The position of the part is given by the center line vector and the center position, and the optical axis of the camera matches the center line vector of the characteristic part, and the distance from the center position of the characteristic part to the extent that the camera can image the detection target. The method for determining the three-dimensional position / orientation of a detection target object according to claim 1, wherein the camera is made to face the detection target object by disposing the cameras apart from each other. 3. The characteristic part is a cylindrical convex part, a circular part which is a concave part, a polygonal convex part, a regular polygonal part which is a concave part, a conical convex part and a concave part. The three-dimensional position / posture determination method of the detection target according to claim 2, wherein the three-dimensional position / orientation of the detection object is a conical portion, a regular polygonal pyramidal convex portion, or a concave portion. 4. A manipulator equipped with a camera capable of acquiring a two-dimensional image of a detection target and a three-dimensional sensor capable of acquiring a range image of the detection target, and a range image of the detection target acquired by the three-dimensional sensor. A distance image processing unit that detects a characteristic part of the detection target from the position and finds the position of the characteristic part, and corrects the camera to the detection target based on the position of the characteristic part thus obtained as a reference. A manipulator guiding unit to be paired with, and a two-dimensional image processing for obtaining a displacement of the detection target with respect to translation and rotation angle by comparing a two-dimensional image of the detection target captured by the camera with a two-dimensional model of the detection target. Section, the two-dimensional image processing section, and the manipulator guiding section, and the displacement of the detection target and the manipulator when the camera is directly opposed to the detection target. Robot vision sensor, characterized in that a target 3-dimensional position deriving unit for determining the three-dimensional position and orientation of the detection object from the operation information.