JP2562047B2 - Position and orientation recognition method of target object - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to a method for recognizing the position or inclination of a target object in an image using an image processing device from an image captured by an ITV camera or the like, and more particularly, The present invention relates to a recognition method when a plurality of geometric points have geometrical features equal to each other.

(Prior Art) In automated production lines using robots and the like, cases in which robots perform assembly and transportation are increasing recently. However, the robot itself works on a certain target object at a predetermined position, and for this purpose, the target object must be positioned in advance. When always working on the same object, a mechanical positioning device may be provided, but when the shape of the target object changes, the positioning device becomes complicated and the cost increases. On the other hand, recently, a method in which a visual sensor is used to recognize the position and inclination of a target object using an image processing method, and a robot works at this position has begun to be adopted.

FIG. 6 is a schematic view of a robot position recognition device using a visual sensor. 41 is a robot, 42 is a control device for the robot, 43 is a camera for capturing the target object, 44 is a visual sensor for processing the image captured by the camera and measuring the position and inclination of the target object, 45 is a work table, 46 Is the target object.
A position recognition method using a visual sensor will be briefly described with reference to this drawing.

First, when the target object 46 is placed on the work table 45, the camera 43 photographs the target object and stores the image in the image memory of the visual sensor 44. FIG. 7 is an example of an image taken by the visual sensor. Reference numeral 51 is a target object, and 52 is a boundary between images captured by the camera. Normally, the input image is converted into a binary image of a bright area and a dark area and processed. The shaded area in FIG. 7 is a bright area in the image of the target object. The visual sensor calculates the position and inclination of the target object from this image. The position and inclination of the target object can be obtained by calculating geometrical feature quantities such as the area, center of gravity, and perimeter of the bright area of the image. For example, the position may be calculated by calculating the position of the center of gravity of the captured image. If the inclination is a rectangle as shown in the figure, the inclination of the principal axis of inertia may be calculated. In this way, after recognizing the position, inclination, etc. of the target object currently placed by the visual sensor, the robot is informed of the data. Generally, when the target object is located at the position of the broken line 53 shown in FIG. 7, the robot is taught to work in advance, so the data given to the robot from the visual sensor is the teaching position. It is the movement amount from. Therefore, the robot shifts the coordinates by the movement amount taught by the visual sensor to perform the work.

In this way, using the method in which the position of the target object is measured with the visual sensor and the robot performs work based on the data, if the target object is not placed at a position beyond the range captured by the camera, a positioning device etc. The system is simple because it is unnecessary. Furthermore, when the type or external shape of the target object changes, it can be handled by simply teaching the robot the visual sensor at the same time, so new equipment investment becomes unnecessary, and a manufacturing line suitable for FA can be configured. .

However, the method described above is effective for a target object having a relatively simple outer shape, but if the outer shape of the target object is complicated, there is a high possibility that the visual sensor will erroneously recognize it. was there.

Therefore, in Japanese Patent Laid-Open No. 63-44103, a plurality of feature points of the target object are extracted from the captured image of the target object, and the position and orientation of the target object are recognized from the positions and distances between the feature points. A position / orientation recognition device is disclosed. According to this device, based on the positions of the feature points of the extracted object image, the distances in all combinations of these feature points are calculated, and the feature points of the object image Of several candidate combinations (feature point candidates) that are considered to have a distance substantially equal to the mutual distance between the feature points of the master image, and select the feature point candidates of the selected object image to be measured. Only the position of the target image is sequentially compared with the position data of the feature points of the master image to determine the position and orientation of the target object, so it is possible to recognize with high accuracy even if the outer shape of the target object is complicated. There is.

(Problem to be Solved by the Invention) However, since this method calculates the distances in all combinations between feature points when extracting feature point candidates, the feature is calculated when the number of feature points is large. There is a problem in that the number of combinations of points is very large, resulting in a large amount of data processing when extracting feature point candidates and a long overall calculation time. Further, when the amount of data to be processed increases, the approximate position data and distance data also increase. Therefore, when the allowable error is set in the comparison process,
Although it is necessary to set the allowable error small, if the allowable error is set extremely small, the calculation time of the comparison process may become long, and in some cases, it may be erroneously determined that there is no corresponding data. On the contrary, if the number of extracted feature points is too small, high-precision recognition cannot be performed. Therefore, in this method, it is necessary to properly select the number of feature points to be extracted in order to perform recognition with high speed and high accuracy. It is not easy to select an appropriate number of feature points for each target object.

An object of the present invention is to reduce the calculation time required for extracting feature point candidates in a method of recognizing a target object when there are a large number of feature points whose geometric feature amounts are equal to each other, and to accurately calculate the target object. It is to provide a position / orientation recognition method of a target object for recognizing a position or an inclination.

(Means for Solving the Problems) Therefore, the present invention has solved the above-mentioned problems of the conventional art by providing a method for recognizing the position and orientation of a target object described in the claims.

(Operation) Hereinafter, the operation of the present invention will be described. The present invention, by comparing the geometrical characteristic amount and the positional relationship data of the respective feature points of the master image obtained by photographing the master target object and the measured object image obtained by photographing the target object becoming the measured object, It calculates how much the position of the target object to be measured is shifted with respect to the taught position of the target object taught to the robot so that the robot can accurately position with respect to the target object. is there.

The procedure firstly identifies which feature point of the master image each feature point of the DUT image corresponds to. First, a plurality of feature points are taught to the image-processed master image, and the area and the center of gravity of each taught point as a geometric feature amount are calculated for each of the plurality of taught feature points, that is, the taught points. The distance and inclination between each teaching point as the positional relationship data are calculated. In addition, a permissible error is set for the calculated geometrical characteristic amount and the positional relationship data. Next, an image of the object to be measured, which is an object to be measured, is subjected to the same image processing as that of the master image, and each of a plurality of extracted feature points is used as a geometrical feature amount. The area and the center of gravity of the feature points and the distance and the slope between the feature points as the positional relationship data are calculated. Furthermore, for all feature points of the DUT image,
The feature points, ie, feature point candidates included in the allowable error range of the geometric feature amount of the teaching point of the above-mentioned master image are extracted,
By comparing the extracted positional relationship data of the feature point candidates of the measured object image with the positional relationship data of the taught points of the master image, the characteristic points of the measured object image corresponding to the respective taught points of the master image are determined. It is specified from the feature point candidates.

Secondly, by comparing the geometric feature amounts and the positional relationship data of the specified feature points of the measured object image and the corresponding teaching points of the master image, the target object of the measured object image is compared. It is calculated how much the position is shifted with respect to the position of the target object in the master image. If the taught position of the target object taught to the robot is based on the position of the target object in the master image, the calculated shift amount is input to the robot controller as a correction amount. As a result, the robot can be accurately positioned with respect to the target object.

(Example) The present invention will be described with reference to the drawings. FIG. 1 is an example of a target object. The target object 1 has a complicated outer shape and has a plurality of holes for assembly therein. The shaded area is the surface of the target object 1 and is glossy so as to facilitate the binarization of the image described below. First, the target object 1 is photographed by the camera and stored in the image memory of the visual sensor. At this time, in the visual sensor, the image signal input from the camera is binarized and then stored in the image memory in order to increase the processing speed and save the image memory.
The binarization is a process in which a certain brightness of an input signal is used as a threshold value (binarization level) and a darker portion is converted into a black value and a brighter portion is converted into a white value (black and white). When the binarization level is set to the optimum value, the contents of the image memory are as shown in FIG. Here, 2 represents the range photographed by the camera, and this range is the target of image processing. Although a window for limiting the processing range of the input image is often used in image processing, 2 may be used as this window. Since the surface of the target object 1 in FIG. 1 is glossy, in the binarized image, the hatched portion is white and the background and the plurality of holes 11 to 19 are black. The image of the target object 1 is provided with nine holes 11 to 19 in total, and the holes 11 to 19 represent the characteristics of the target object 1. Therefore, the holes 11 to 19 are made to be targets for recognition by image processing, as portions having characteristic features in shape, that is, feature points.

Here, the method of extracting each hole as a feature point and the geometric feature amount and positional relationship data of each extracted hole will be described using the image of FIG.

To extract the nine holes 11 to 19 provided in the target object 1, a known part recognition method may be used. This is to classify a binarized image into a white pixel region and a black pixel region, and recognize a white pixel region surrounded by a black pixel region as one part. In FIG. 1, since each hole is a black pixel area, if a method of recognizing the white pixel area and the black pixel area in reverse is used,
It is possible to recognize each hole that is a white pixel area.

The geometric feature amount of each hole is the center of gravity and area of each hole.
The center of gravity represents the position of the hole, and the area represents the size of the hole. Further, if the major axis and the minor axis of each hole are calculated at the same time, only the one close to a perfect circle can be extracted from the ratio of the major axis and the minor axis, so that when the hole to be recognized is a perfect circle as in this embodiment. Can remove unnecessary things in advance. Here, the center of gravity and area of each hole are Gi and Si, and the set of this data is the geometrical feature amount Fi of each hole.
It should be noted that i is a temporary number corresponding to each hole and takes a value from 1 to 9. For example, the hole 11 in FIG. 2 has a geometrical feature amount F1, a center of gravity G1, and an area S1. Similarly, the geometric features F2 to F9 are assigned to each of the holes 12 to 19.

The positional relationship data between the holes is the distance between the holes and the direction between the holes. Here, the distance between the holes may be calculated, for example, as the distance between the centers of gravity of the holes. The direction is calculated, for example, as an angle with respect to the horizontal direction of the image processing screen. Assuming that the distance and the direction of the i-th hole and the j-th hole are Lij and θij, for example, the positional relationship data of the first hole 11 and the ninth hole 19 are as shown in FIG. L19, direction θ1
Will be 9. In the example described above, the range of i and j is from 1 to 9, so there are 81 (= 9 × 9) combinations, but for example, the distance Lij and the distance Lji are equivalent, so in practice 36 It becomes a set of data (Lij, θij). Here, the set of the distance Lij and the direction θij is the positional relationship data Pij.

Now, in order to recognize the position of the target object, the reference position of the target object must be calculated in advance. Therefore, usually, teaching or teaching is executed in the visual sensor. In this method, a target object serving as a master is photographed once, image processing is performed on the master image, and geometrical feature quantities and positional relationship data are calculated using a plurality of extracted holes as feature points. At this time, it is not necessary to calculate the geometrical characteristic amount and the positional relationship data for all the nine extracted holes. For example,
As shown in FIG. 4, five holes are selected as teaching points as teaching points from nine holes as feature points. Then, only the geometric feature amount and the positional relationship data for each of the five holes as the teaching points are calculated and stored. As shown in Fig. 4, hole 19 is teaching number 1 and hole 11 is teaching number 2.
22, hole 18 is teaching number 3 23, hole 13 is teaching number 4 24, hole
No. 17 is teaching number 5 to 25, and the remaining four holes shown in Fig. 2
12, 14, 15, and 16 are excluded from teaching targets. Note that the number of holes to be taught depends on the geometric feature amount of each hole as long as the target object can be specified, and is not limited to five. For example, in the target object 1 shown in FIG. 2, since the areas of the holes 11, 14, 18 are the same and the areas of the holes 12, 13, 15, 16, 17 are also the same, the geometric features of the holes are If the quantity is the area, at least 1 must be selected from each group of holes that make the area equal.
It suffices to select more than one hole as the teaching point. Further, although there is no limitation on the teaching order of the holes to be taught, it is possible to improve the calculation efficiency and shorten the recognition time by regularly selecting the teaching order. Here, the geometrical feature amount and the positional relationship data of the master target object are Fk ′ and Pk, respectively.
l '. Also, of the positional relationship data Pkl ′, the distance,
The directions are Lkl 'and θkl', respectively. k is the number of each hole taught by the target object to be the master. Since the number of holes to be taught is 5 in this example, the range of k and l is an integer from 1 to 5.

Next, an object to be measured that is to be actually recognized is photographed, image processing is performed on this measured object image, and geometrical feature amounts and positional relationship data for all the extracted holes are obtained. calculate. FIG. 5 is an example of a binarized image of the target object that is the measured object. As described above, the calculated geometrical feature amount and positional relationship data of each hole are F
Let i and Pij. In this example, the number of extracted holes is nine, so the range of i and j is 1 to 9. The fifth
As shown in the figure, since the target object to be measured is placed at an angle with respect to the master target object, the order of recognizing holes is different from that in the case of FIG. The number of the hole is set from the upper right of the screen to the lower left with symbols from to.

Next, the geometrical feature amount and positional relationship data of the hole as the teaching point of the target object to be the master, and the geometrical characteristic amount and positional relationship of the hole as the feature point of the target object to be the measured object. By comparing the data, the holes of the master target object (holes 21 to 25 shown in FIG. 4) correspond to which of the holes of the target object to be measured (holes shown in FIG. 5). I will identify in order. Although there are various methods of specifying, here, a method of specifying with high calculation efficiency will be described.

First, only the area Si as the geometric feature amount Fi of each hole of the target object to be measured is focused. That is,
The permissible error is set in advance for the area Si of the geometrical feature amount Fk ′ of each hole as the teaching point calculated in the master target object, and each hole as the feature amount of the target object to be measured is set. The geometrical feature Fi of is classified into the classes according to whether or not it falls within the range of this tolerance. In this way, the characteristic points of the target object, which is the object to be measured, which are divided into classes for each teaching point of the master target object, are called characteristic point candidates. As a result, a hole whose size is obviously different from that of the others is specified at this point. For example, the hole in FIG. 5 is obviously different in size from other holes, and is therefore classified as teaching number 1-21 taught in FIG. Next, the holes in FIG. 5 correspond to both the teaching numbers 2 22 and 3 23 in FIG. 4, so they are classified into both teaching numbers. Further, since the remaining holes in FIG. 5 correspond to both of the teaching numbers 4 24 and 5 25 in FIG. 4, they are classified into both teaching numbers. This is shown in the table below.

Teaching number Classification number 1 2, 3, 3, 4, 4, 5, 5 ... All holes in the target object to be measured are now classified as the holes taught by the target object to be the master. It will be.

Next, for each classified hole, the positional relationship data Pij
Each hole is specified by using.

First, since only the hole classified as the teaching number 1 is, this hole has been specified. That is, the hole in FIG. 5 was identified as the hole 21 in FIG.

Next, the hole classified into teaching number 2 is specified. Holes that have already been identified and classified into teaching number 2,
Distance L16, L46, L67 for each and teaching number 1 2
Compare the distance L12 'between the 1st hole and the 2nd 22nd hole and select the one with the closest value. In this case, teaching numbers 1-21 and 2-22
Since the distance L12 'from the hole is closest to the distance L46 from hole to hole, the hole in FIG. 5 is specified as the hole 22 in FIG. The holes in and are now identified.

Next, the hole classified into teaching number 3 is specified. Since the hole has already been specified, with respect to the hole, similarly, the respective distances L14 and L47 from the already specified hole are similarly set.
Is compared with the distance L23 'between the hole with teaching number 2-22 and the hole with number 3-23, and the one with the closest value is selected. In this case, the distance L23 'between the hole numbered 22 and the numbered hole 23 is closest to the distance L47 between the holes. Therefore, the hole in FIG. 5 is the hole 23 in FIG. Was identified by.

Next, 5 holes classified as No. 4 above,
,,, to identify one. As in the case of teaching number 3, the distances L27, L37, L57, L78, and L79 between the holes ,,, and are respectively the distance L34 between the holes of teaching number 3 23 and the number 4 24. ′ And select the one with the closest value. In this case, the distance L34 'between the hole number 3-23 and the hole number 4-24 is closest to the distance L27 between holes and the distance L79 between holes. Applicable In the case where there are two or more applicable candidates in this way, the hole corresponding to the hole of teaching number 5-25 is further investigated. That is, when the hole is identified as the hole of teaching number 4 to 24, the distances L29, L between the holes ,,, and, respectively.
39, L59, L89 are compared with the distance L45 'between the holes of teaching numbers 4-24 and 5-25 and the closest one is selected. However, in any case, since the corresponding one is not found, it can be understood that the hole specified as the teaching number 4 is wrong. Therefore, the same comparison is made with respect to the hole that was another candidate of the teaching number 4 earlier. That is, if the hole is identified as the hole with teaching number 4 to 24, the hole is ...
, And the distance between each hole L23, L25, L28, L29
Is compared with the distance L45 'between the hole number 4-24 and the hole number 5-25, and the one with the closest value is selected. In this case, the distance L45 'between the hole number 4-24 and the hole number 525 is closest to the distance L29 between the holes, so the hole in FIG. 5 is the hole 25 in FIG. Was identified by. Further, the hole corresponding to the teaching number 4 becomes a hole, and the identification of all holes is completed. This is shown in the table below.

Teaching number Classification number 1 2 3 4 5 In the process of specifying candidates, the one with the closest comparison value was selected, but the actual target object has the positional relationship data as taught due to processing errors. It may not be possible. Therefore, set an allowable range for each distance data, and select multiple candidates within this range,
Finally, the pattern with the smallest sum of all errors may be identified.

If the target object is a left-right object, the distance between the target object to be measured and the master target object is Li
In some cases, the hole cannot be uniquely identified only by j and Lkl ′. In this case, in addition to the distances Lij and Lkl ′, the directions θij and θkl ′ of the target object to be measured and the target object to be the master may be used together.

As described above, each hole selected as the teaching point of the master target object is a hole (feature point) of the target object which is the DUT.
Since it can be specified that the position and the inclination of the target object to be measured are shifted with respect to the position and the inclination of the target object to be the master, it is calculated.

First, the shift amount of the position of the target object to be measured with respect to the master target object is, for example, the geometrical feature amount F1 ′ of the hole of teaching number 1-21 of the master target object.
Is calculated by calculating the difference between the barycentric coordinate G1 ′ in G and the barycentric coordinate G6 of the geometrical feature F6 of the hole of the target object which is the object to be measured and is specified to correspond to this hole. That is, since the barycentric coordinates are usually given by X and Y coordinates, if the difference between the X and Y coordinates of both barycentric coordinates is calculated, the position of the target object to be measured is the position of the master target object. It can be seen that the position is shifted by ΔX in the X direction and by ΔY in the Y direction.

Next, the shift amount of the inclination of the target object to be measured with respect to the master target object is, for example, teaching number 1.
Positional relationship between the inclination θ12 'in the positional relationship data P12' between the 21st hole and the 2nd and 22nd holes, and the holes identified as corresponding to the 2nd and 22nd holes, respectively. Data P4
Calculated by calculating the difference from the slope θ46 in 6.
From this, it can be seen that the inclination of the target object to be measured is shifted by Δθ with respect to the inclination of the master target object.

Therefore, if the position of the target object to be measured is translated by (−ΔX, −ΔY) with respect to the position of the master target object and then rotated by −Δθ, the target object becomes The position of the object will return to the position of the target object that is the master position, which is the taught position.

By doing this, it is possible to know how much the target object placed has shifted with respect to the taught position taught by the robot, and by using this shift amount as the correction amount, the teaching data of the robot can be changed. , The robot can be positioned accurately with respect to the placed object.

In the above description, the feature point is the hole of the target object, but the feature point is not necessarily limited to this. For example, the position of the corner of the target object can be set as the feature point. Further, since the geometrical feature amount is intended for a circular hole, the description has been given by using the center of gravity and the area, but the geometrical feature amount is not limited to this.
Various geometric feature quantities can be used depending on the shape of the feature points. For example, in the case of a polygonal feature point, the number of corners can be an effective geometric feature amount.

(Effect of the Invention) According to the present invention, when recognizing how much the position of a target object photographed by a camera is shifted from a position taught in advance, a master image of a target object serving as a master and Since the geometrical feature quantities and the positional relationship data of the respective feature points of the image of the object to be measured, which is the object to be measured, are compared, the object to be measured even for the object having a complicated outer shape. It is possible to accurately calculate how much the position and the inclination of the target object to be measured are shifted with respect to the target object taught in advance.

In particular, the present invention selects at least one feature point as a teaching point from each group when there are a large number of groups of feature points whose geometric feature amounts are equal to each other in the master image, and for each teaching point. Then, by classifying the feature points of the measured object image having the same geometrical feature amount as these and using these as feature point candidates, the comparison process of the master image and the measured object image is carried out for each teaching point and a plurality of corresponding teaching points. Since it is only necessary to perform between the feature point candidates, compared with the conventional method of extracting the feature point candidates by calculating the distances between all the feature points of the master image and the measured object image. The calculation time was shortened.

Further, the number of teaching points to be set is determined depending on the geometric feature amount of the feature points of the target object, and if there are many feature points having the same geometric feature amount, the number of teach points is set. Can be set less. As a result, in the process of identifying the feature points of the target object to be measured, which correspond to the teach points of the target object to be the master, the number of taught points is appropriate according to the geometric feature amount of the feature points of the target object. It has become possible to set to, and recognition can be performed in a short time.

[Brief description of drawings]

FIG. 1 shows the shape of a target object used in the embodiment of the present invention. 1 is a target object. FIG. 2 shows a target object photographed by a camera. 2 is the boundary of the screen. Further, 11 to 19 are holes as feature points in the target object recognized by the visual sensor. FIG. 3 illustrates the positional relationship data between feature points. L19 and θ19 are the distances and directions of the two holes. FIG. 4 shows an example of teaching a target object. 21 to 25 are the feature points taught. FIG. 5 is an example in which a camera captures an object to be measured, which is an object to be recognized. ~ Are feature points (holes) recognized by the visual sensor. FIG. 6 is a schematic diagram of a robot work using a visual sensor. 41 is a robot, 42 is a robot controller, 43 is a camera, 44 is a visual sensor, 45 is a work table, and 46 is a target object. FIG. 7 is a diagram for explaining a conventional example, in which 51 is a target object being photographed and 52 is a target object at a teaching position.

Claims (3)

(57) [Claims] 1. A method of recognizing the position or inclination of a target object by photographing an object to be recognized by a camera and using an image processing method based on the photographed image. In a recognition method in the case where there are feature points having geometrical features equal to each other with respect to a plurality of regions that are parts and are geometrically characteristic, that is, in the recognition method, a captured image of the target object serving as a master, that is, A plurality of the characteristic points are extracted by performing image processing on the master image, and a plurality of groups are created for each of the extracted characteristic points whose geometrical feature amounts are equal to each other. , At least one or more of feature points in each of the plurality of groups is selected as a teaching point, and geometrical feature amounts of the selected plurality of teaching points and positional relationship data between the respective teaching points To The calculated geometrical feature amount and the allowable error of the positional relationship data are set, and then the captured image of the target object to be measured, that is, the measured object image is the master image. The same image processing is performed to extract the plurality of feature points, the geometrical feature amount of the feature points of the extracted object image and the positional relationship data between the feature points are calculated, and the object to be measured is calculated. With respect to all the feature points of the image, feature points, that is, a plurality of feature point candidates included in the permissible error range of the geometric feature amount of each teaching point of the master image are extracted, and the extracted object to be measured is extracted. By comparing the positional relationship data of the characteristic point candidates of the image and the positional relationship data of the teaching points of the master image, the characteristic points of the measured object image corresponding to the respective teaching points of the master image are identified as the characteristic point candidates. Identified from among By comparing the geometrical feature amount and the positional relationship data of each of the feature points of the measured object image and the corresponding teaching points of the master image, the position and the inclination of the target object of the master image are used as a reference. The position and orientation recognition method of the target object, characterized in that the position and the inclination of the target object in the image of the object under measurement are recognized. 2. The position / orientation recognition method for a target object according to claim 1, wherein the positional relationship data between the respective characteristic points comprises a distance and a slope between the characteristic points. 3. The position / orientation recognition method for a target object according to claim 1, wherein the geometrical feature amount comprises a center of gravity and an area of a feature point.