JP2000084876A - Robot arm visual chase method and system device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot arm visual chase method and system device which can chase the position.posture of a robot arm surely and properly without being affected to the background and luminosity of a robot work environment. SOLUTION: A template composition device 31 preserves plural texture patterns obtained by giving an affine conversion to various forms as a template to the texture of a marker 14. A marker position calculation device 32 can calculate respective marker positions for entire marker projection staying on the picture taken by a camera 2. An arm position.posture estimation calculation means 33 calculates the position.posture of the robot arm 12. A market/texture selection device 34 selects the marker projection of a higher rank prescribed number (at least four or more) most suitable for the observation at the present time and also for selecting the texture pattern with a prescribed number most suitable for coping with respectively the marker projection of the selected higher rank prescribed number from the templates.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and system for visually tracking a robot arm, and more particularly, to a robot for visually tracking the position and orientation of a robot arm during a robot operation online using a camera. The present invention relates to an arm visual tracking method and a robot arm visual tracking system device directly used for implementing the method.

[0002]

2. Description of the Related Art Conventionally, as a method of visually tracking the position and orientation of a robot arm while using a camera during robot work, for example, an edge of a robot arm main body projected on a camera image plane (a boundary with a background) is used. (A contour line of the robot arm main body).

As another technique, a plurality of light sources such as LEDs (light emitting diodes) are arranged in advance in the visible region of the surface of the robot arm as markers, and the projection of the markers such as these LEDs is performed on the camera image plane in two directions. In some cases, the position and orientation of the robot arm are tracked based on the result of the binarization recognition.

[0004]

However, in the technique of performing the required visual tracking by searching for the edge of the robot arm main body, the required robot work is performed in the field, such as the work of erection of a communication cable on a telephone pole. If it is assumed that the background of the robot arm projected on the camera image plane will naturally be different for each robot work environment, accurate edge extraction can be performed under particularly complicated situations. Sometimes it becomes impossible.

Similarly, if it is assumed that the robot work is performed outdoors, the required visual tracking using a marker projection such as an LED, for example, causes the luminous intensity of the robot work environment to suddenly become bright due to the influence of weather or the like. When, for example, the relative brightness of the marker projection on the camera screen also suddenly decreases, when falling into such a situation,
It is technically very difficult to instantly adjust the brightness of a camera image in order to properly perform the required binarization recognition.

Here, the main objects of the present invention are as follows.

That is, a first object of the present invention is to provide a robot arm visual tracking method and system apparatus capable of reliably tracking the position and orientation of a robot arm without being influenced by the background of the robot working environment. It is something you want to do.

A second object of the present invention is to provide a robot arm visual tracking method and system capable of properly tracking the position and orientation of a robot arm without being affected by the light intensity of the robot working environment. Is what you do.

A third object of the present invention is to provide a robot arm visual tracking method and a system device capable of extremely stably performing online tracking of a required position and orientation of a robot arm and improving reliability. Things.

[0010] Other objects of the present invention will become apparent from the description of the specification, drawings, and particularly from the claims.

[0011]

According to the method of the present invention,
To visually track the position and orientation of the robot arm online, markers consisting of a specific form of geometric pattern are placed in advance in the visible area of the surface of the robot arm, and the plurality of markers are captured with a camera. Various texture patterns that can be observed at the time of shooting are prepared in advance as templates, and the correlation between the camera images of the markers arranged on the robot arm and the various texture patterns prepared as templates is prepared. By calculating the position of the robot arm, the position of the robot arm is calculated based on each marker position and the known marker arrangement data indicating the arrangement positions of the plurality of markers on the robot arm. -It has the feature of taking measures to calculate the posture.

Also, in the system apparatus of the present invention, a marker formed of a geometric pattern of the above-described specific form, which is arranged in a visible region of the surface of the robot arm, a camera for photographing the plurality of markers, and an affine transformation to the texture of the marker A plurality of texture patterns obtained by performing the above in various forms, a template synthesizing apparatus that saves the plurality of texture patterns as the template, and a correlation operation between the camera images of the plurality of markers and the plurality of texture patterns in the template to perform camera correlation. A marker position calculating device capable of calculating each marker position for all marker projections present on the image plane, and calculating the position and orientation of the robot arm based on the calculated marker positions and the marker arrangement data. The arm position / posture estimation calculation device and the calculated robot Based on the position and orientation of the system, the top predetermined number of marker projections most suitable for observation at the present time are selected, and the correlation calculation is most suitable to correspond to the selected top predetermined number of marker projections, respectively. And a marker / texture selection device for selecting a predetermined number of texture patterns from the template.

More specifically, in solving the problem, the present invention achieves the above object by adopting a novel characteristic configuration technique ranging from a high-level concept to a low-level concept listed below. Is done.

That is, a first feature of the method of the present invention is that when the position and orientation of a robot arm during robot work are visually tracked online using a camera, a marker formed of a specific form of geometric pattern is used. Are placed in advance in the surface visible area of the robot arm, and various texture patterns corresponding to the appearance of the geometric pattern when the plurality of markers are observed from various angles are prepared as templates in advance. While performing a predetermined correlation operation in time series between the images of the cameras of the plurality of markers arranged on the robot arm and various texture patterns prepared in the template, Of the robot arm visual tracking method that calculates the position and orientation of the robot arm and sequentially tracks the robot.

A second feature of the method of the present invention is that when a robot arm is visually tracked online using a camera to determine the position and orientation of a robot arm, a marker having a specific form of geometric pattern is used. While pre-arranged in the surface visible area of the robot arm, various texture patterns that can be observed when a plurality of markers arranged on the robot arm are photographed with a camera are prepared in advance as templates, By performing a predetermined correlation operation between the camera images of a plurality of markers arranged on the robot arm and various texture patterns prepared as templates, the current position among all marker projections existing on the camera image plane is obtained. Calculate the marker positions for the top predetermined number of marker projections most suitable for observation The present invention is configured to adopt a robot arm visual tracking method that calculates the position and orientation of the robot arm based on the issued marker positions and known marker arrangement data indicating the arrangement positions of the plurality of markers on the robot arm. .

A third feature of the method of the present invention is that the marker according to the first or second feature of the method of the present invention has a form in which a part of the texture when enlarged at an arbitrary magnification matches the original texture. In addition to using the geometric pattern of the above, a texture pattern is applied to the texture of the marker in various forms by performing affine transformations other than scaling / transformation transformations and translation transformations to generate a robot arm visual tracking method. is there.

A fourth feature of the method of the present invention is that the predetermined correlation operation in the first, second or third feature of the above-described method of the present invention is most suitable for associating each with a predetermined number of marker projections. A robot arm vision system that selects a predetermined number of texture patterns from the template and calculates the sum of absolute values of the grayscale data differences between the predetermined number of texture patterns and the corresponding predetermined number of texture patterns in the vicinity of the coordinates of the upper predetermined number of marker projections It consists in adopting the configuration of the tracking method.

According to a fifth feature of the method of the present invention, the predetermined correlation operation in the above-mentioned fourth feature of the method of the present invention outputs, as marker positions, coordinates at which the sum of absolute values of the grayscale data differences gives the minimum value. In the configuration adoption of the robot arm visual tracking method.

According to a sixth feature of the method of the present invention, the calculation of the position / posture of the robot arm in the first, second, third, fourth or fifth feature of the method of the present invention is performed by using a predetermined number of upper markers. An object of the present invention is to adopt a configuration of a robot arm visual tracking method using an evaluation function defined for an error between a projection coordinate and each marker position.

According to a seventh feature of the method of the present invention, the calculation of the position and orientation of the robot arm in the sixth feature of the method of the present invention optimizes each marker position by applying the least square method to the evaluation function. Thus, a robot arm visual tracking method for estimating the position and orientation of the robot arm is employed.

An eighth feature of the method of the present invention is that the position / posture of the robot arm in the first, second, third, fourth, fifth, sixth or seventh feature of the method of the present invention is calculated. ,
A marker parallel perspective projection for the remaining marker projections when any one of the top predetermined number of marker projections is used as a reference marker projection is calculated based on the approximate position and posture of the robot arm. By adopting the configuration of a robot arm visual tracking method that calculates the position and orientation of the robot arm by solving a linear equation using the calculated marker positions of the parallel perspective projection and the marker positions of the reference marker projection as inputs. It is in.

A ninth feature of the method of the present invention is that, in the calculation of the position and orientation of the robot arm in the eighth feature of the method of the present invention, the process of re-inputting the once output position and orientation of the robot arm is repeated. When the marker positions for the marker parallel perspective projection calculated each time in the process of this iterative process converge to almost constant coordinates, the position and posture of the robot arm are output as final values. In other words, the configuration of the robot arm visual tracking method is adopted.

A tenth feature of the method of the present invention is that the predetermined predetermined number of the first, second, third, fourth, fifth, sixth, seventh, eighth or ninth features of the above-described method of the present invention. Is a configuration of a robot arm visual tracking method using at least four marker projections that are not on the same plane among all the marker projections present on the camera image plane.

An eleventh feature of the method of the present invention is the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, or tenth feature of the method of the present invention. An object of the present invention is to adopt a configuration of a robot arm visual tracking method using a marker having a geometric feature point which can indicate its own reference position in the center thereof.

A twelfth feature of the method of the present invention is that the marker according to the eleventh feature of the method of the present invention is configured such that two light-colored portions and two dark-colored portions are alternately arranged, and
An object of the present invention is to adopt a configuration of a robot arm visual tracking method using a symmetric point between two light-colored portions or two dark-colored portions generated by this arrangement as a geometric feature point.

According to a thirteenth feature of the method of the present invention, a marker in the twelfth feature of the above-described method of the present invention is a method for visually tracking a robot arm using a marker in which a bright portion is white and a dark portion is black. In the configuration adoption.

A fourteenth feature of the method of the present invention is that the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, and eleventh of the above-described method of the present invention. The twelfth or thirteenth feature is that the marker adopts a configuration of a robot arm visual tracking method using a marker having a square shape as a whole.

A fifteenth feature of the method of the present invention is that the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, and eleventh of the method of the present invention are described. Twelfth, thirteenth or fourteenth
The present invention resides in the adoption of a configuration of a robot arm visual tracking method in which markers having the above characteristics are intensively arranged in the surface visible region of the end effector forming the working point of the robot arm.

On the other hand, a first feature of the system apparatus of the present invention is that a geometric pattern arranged in the surface visible area of the robot arm and having a part of the texture when enlarged at an arbitrary magnification matches the original texture is provided. And a marker consisting of
A camera that captures a plurality of markers placed on the robot arm and a template that generates a plurality of texture patterns by applying affine transformations to the marker textures in various forms except for scaling / transformation transformations and translation transformations as templates A predetermined correlation between a template synthesizing device to be stored, a camera image obtained by photographing a plurality of markers arranged on the robot arm with a camera, and a plurality of texture patterns in the template stored in the template synthesizing device. By performing a calculation, a marker position calculation device capable of calculating each marker position for all marker projections present on the camera image plane, and each marker position actually calculated by the marker position calculation device, Known markers indicating the positions of multiple markers on the robot arm An arm position / posture estimation calculating device for calculating the position / posture of the robot arm based on the arrangement data; and a camera image plane based on the position / posture of the robot arm calculated by the arm position / posture calculation device. From among all the marker projections present, a top predetermined number of marker projections most suitable for observation at the present time are selected, and at the time of a predetermined correlation operation performed by the marker position calculation device, the selected top predetermined number of A configuration adoption of a robot arm visual tracking system device including a marker / texture selection device for selecting a predetermined number of texture patterns most suitable for each of marker projections from templates stored in the template synthesis device. It is in.

A second feature of the system device of the present invention is that the marker position calculating device in the first feature of the above-described system device of the present invention performs, as a predetermined correlation operation, a predetermined number of upper ranks selected by the marker / texture selecting device. In the vicinity of the coordinates of the marker projection, the marker / texture selecting device and the marker / texture selecting device are operable to perform a correlation operation to obtain the sum of absolute values of the grayscale data differences between the corresponding predetermined number of texture patterns also selected by the marker / texture selecting device. The present invention resides in adoption of a configuration of a connected robot arm visual tracking system device.

A third feature of the system device of the present invention is that, when the marker position calculating device in the second feature of the above-described system device of the present invention performs a correlation operation on the difference of gray data, the absolute value of the gray data difference is obtained. Of the robot arm visual tracking system device which is connected to the arm position / posture estimating arithmetic unit so as to freely output the coordinates at which the sum of the values gives the minimum value as each marker position.

A fourth feature of the system device of the present invention is that the arithmetic unit for estimating the arm position / posture in the first, second or third feature of the system device of the present invention has a higher rank selected by the marker / texture selecting device. A robot arm visual tracking system device having a function of defining an evaluation function representing an error between a predetermined number of marker projection coordinates and each marker position calculated by the marker position calculation device is adopted.

The fifth feature of the system device of the present invention is that the arm position and the arm position in the fourth feature of the system device of the present invention described above.
The posture estimation calculation device has a function of estimating the position and posture of the robot arm by optimizing each marker position calculated by the marker position calculation device by applying the least square method to the evaluation function. It is in the configuration adoption of the robot arm visual tracking system device.

A sixth feature of the system device of the present invention is that the first, second, third, fourth or fifth of the system device of the present invention described above.
The arm position / posture estimating calculation device in the feature of (1) above, when any one of the predetermined number of marker projections selected by the marker / texture selection device is used as the reference marker projection, the remaining marker projections A parallel perspective transformation device that calculates the marker parallel perspective projection based on the approximate position and posture of the robot arm calculated by the arm position / posture estimation calculation device, and a marker parallel perspective projection calculated by the parallel perspective transformation device By solving a linear equation with the input of each marker position and the marker position of the reference marker projection, the position of the robot arm
Another object of the present invention is to adopt a configuration of a robot arm visual tracking system device having a linear projection inverse operation device for calculating a posture.

A seventh feature of the system device of the present invention is that the arm position and the arm position in the sixth feature of the system device of the present invention described above.
The posture estimation arithmetic unit repeatedly performs a process of re-inputting the position and orientation of the robot arm once output from the linear projection inverse arithmetic unit to the parallel perspective transformation device. When the calculated marker parallel perspective projections converge to almost constant coordinates, the robot arm position / posture can be freely output as final values. is there.

An eighth feature of the system device of the present invention is that the first, second, third, fourth, fifth,
The marker / texture selection device according to the sixth or seventh aspect selects at least four marker projections that are not on the same plane from among all marker projections present on the camera image plane as the predetermined number of marker projections. The present invention resides in adopting a configuration of a robot arm visual tracking system device having a function.

A ninth feature of the system device of the present invention is that the first, second, third, fourth, fifth,
The sixth, seventh or eighth aspect of the present invention resides in the configuration adoption of a robot arm visual tracking system device having a geometrical feature point which can indicate its own reference position in the center of the marker.

A tenth feature of the system device of the present invention is that the marker according to the ninth feature of the system device of the present invention is configured such that two light-colored portions and two dark-colored portions are alternately arranged. An object of the present invention is to adopt a configuration of a robot arm visual tracking system device in which a symmetrical point between two light-colored portions or two dark-colored portions caused by an arrangement form is used as a geometric feature point.

An eleventh feature of the system device of the present invention resides in that the marker in the tenth feature of the above-described system device of the present invention adopts a configuration of a robot arm visual tracking system device in which a bright portion is white and a dark portion is black. It is in.

A twelfth feature of the system device of the present invention is that the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, or tenth of the above-described system device of the present invention. The eleventh feature resides in the adoption of a configuration of a robot arm visual tracking system device in which the marker is formed in a square shape as a whole.

A thirteenth feature of the system device of the present invention is that the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, and tenth aspects of the system device of the present invention are described above. Eleventh or first
The second feature resides in the adoption of a configuration of a robot arm visual tracking system device in which the markers are intensively arranged in the surface visible region of the end effector forming the working point of the robot arm.

[0042]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing an embodiment of the present invention;

(Example of Apparatus) FIG. 1 is an overall configuration diagram of a robot arm visual tracking system apparatus according to an example of the apparatus of the present invention. The robot work assumed in the present system device is the work of laying the communication cable on the electric pole described above, and the work robot shown in the following system device has a form adapted to the work. .

As shown in the figure, the robot arm visual tracking system device α according to this example of the device is a truck that can be driven and run on a road in order to perform a work of laying a communication cable (both not shown) to a utility pole. The present invention is applied to the work robot 1 mounted on the vehicle bogie unit 11.

Work robot 1 mounted on vehicle bogie unit 11
A robot arm 12 is mounted on the end of the robot arm 12. At the tip of the robot arm 12, a tip effector 13 for actually performing the work of laying a communication cable is attached. , A marker 14 composed of a geometric pattern in which a part of the texture when enlarged at an arbitrary magnification matches the original texture is arranged.

Although the plurality of markers 14 may be arranged on the main body of the robot arm 12, in order to obtain the required effect according to the present invention, they must be placed on the end effector 13 as shown. It is preferable to arrange them intensively.

A camera 2 (monaural camera) for photographing a plurality of markers 14 arranged on the end effector 13 of the robot arm 12 is mounted on a vehicle roof 15 such as a truck. , Work robot 1
Provided to the robot arm control device 3 mounted in the inside.

As shown in the figure, the robot arm control device 3 includes a template synthesizing device 31, a marker position calculating device 32, an arm position / posture estimating calculating device 33,
And a marker / texture selecting device 34. Of these, the template synthesizing device 31 performs a plurality of affine transformations on the texture of the marker 14 except for the enlargement / reduction transformation and the parallel movement transformation in various forms. The generated texture pattern is saved as a template. The specific texture of the marker 14 and the details of the template will be described in a method example.

The marker position calculating device 32 includes a camera image obtained by photographing a plurality of markers 14 arranged on the end effector 13 of the robot arm 12 with the camera 2 and a template image stored in the template synthesizing device 31. A predetermined correlation operation is performed between the plurality of texture patterns and the respective marker positions for all marker projections present on the camera image plane are calculated.

The arm position / posture estimating arithmetic unit 33 is a known marker that indicates each marker position actually calculated by the marker position arithmetic unit 32 and the arrangement position of the plurality of markers 14 in the end effector 13 of the robot arm 12. The position and orientation of the robot arm 12 (tip effector 13) are calculated based on the arrangement data. A further specific configuration of the arm position / posture estimation calculating device 33 will be described later.

The marker / texture selecting device 34 selects, based on the position / posture of the robot arm 12 calculated by the arm position / posture estimating operation device 33, from all marker projections present on the camera image plane at the present time. The uppermost predetermined number (at least four or more) of marker projections most suitable for observation are selected, and at the time of a predetermined correlation operation performed by the marker position calculation device 32, the selected uppermost predetermined number of marker projections are respectively corresponded. The most suitable number of texture patterns are selected from the templates stored in the template synthesizing device 31.

The marker position calculator 32 among the constituent elements of the robot arm controller 3 performs a marker / texture selector 34 as a predetermined correlation calculation.
Correlation calculation for calculating the sum of absolute values of differences in grayscale data between the predetermined number of texture patterns correspondingly selected by the marker / texture selection device 34 in the vicinity of the coordinates of the predetermined number of marker projections selected in the above. And outputs the coordinates at which the sum of the absolute values of the grayscale data differences gives the minimum value as each marker position.

The arm position / posture estimating arithmetic unit 33
Defines an evaluation function representing an error between the coordinates of the predetermined number of upper-ranked marker projections selected by the marker / texture selection device 34 and each marker position calculated by the marker position calculation device 32, and this evaluation function The position / posture of the robot arm is estimated by optimizing each marker position calculated by the marker position calculation device 32 by applying the least square method to the marker position.

Next, a further specific configuration of the above-described arm position / posture estimation calculating device 33 will be described.

FIG. 2 shows an arm position / posture estimation calculating device 3
3 is a functional configuration diagram of FIG.

As shown in the figure, the arm position / posture estimating arithmetic unit 33 sets an arbitrary one of the predetermined number of marker projections selected by the marker / texture selecting unit 34 as a reference marker projection. A parallel perspective transformation device 33a that calculates a marker parallel perspective projection for the remaining marker projections based on the approximate position / posture (marker position information) of the robot arm 12 calculated by the arm position calculation device 32; By solving a linear equation using the marker positions of the marker parallel perspective projection calculated by the parallel perspective transformation device 33a and the marker positions of the reference marker projection as inputs, the robot arm 12
And a linear projection inverse operation device 33b for calculating the position / posture of.

The arm position / posture estimating operation unit 33 repeats the process of inputting the position / posture of the robot arm 12 once output from the linear projection inverse operation unit 33b to the parallel perspective transformation unit 33a. In the course of the repetitive processing, when the marker positions for the marker parallel perspective projection calculated each time by the parallel perspective transformation device 33a converge to substantially constant coordinates (the absolute value of the difference from the previous output result falls below the threshold value). ), The position / posture of the robot arm 12 is output as a final value (final arm position / posture result).

(Example of Method) Subsequently, the robot arm visual tracking system device α according to the example of the device configured as described above.
An example of a method applied to the method will be described.

FIG. 3 is a diagram showing the form of the marker 14, its arrangement with respect to the tip effector 13, and the projection of the marker displayed on the camera image plane.

As shown in the figure, in this example of the method,
As a marker 14 made of a geometric pattern of a specific form, the whole is formed in a square shape, and a part of the texture when enlarged at an arbitrary magnification is made of a geometric pattern of a form that matches the original texture. Use something.

That is, the marker 14 used here is:
In order to satisfy the above texture conditions,
It is necessary to have a geometric feature point capable of indicating its own reference position. For this reason, in this example, two light-colored portions and two dark-colored portions are alternately arranged in a circle. Are used as the required geometric feature points between the two bright-colored parts or the two symmetrical points caused by the above. In general, it is preferable that the light-colored portion of the marker 14 be white and the dark-colored portion be black.

[0062]

[Outside 1]

[0063]

[Outside 2]

[0064]

[Outside 3]

[0065]

[Outside 4]

[0066]

[Outside 5]

As described above, the template synthesizing device 3
In the state where the texture pattern is prepared in 1, the respective units of the robot arm control device 3 sequentially perform the following processing in order to calculate the required position and orientation of the robot arm 12 and to track the position.

[0068]

[Outside 6]

[0069]

[Outside 7]

[0070]

[Outside 8]

[0071]

[Outside 9]

[0072]

[Outside 10]

[0073]

[Outside 11]

[0074]

[Outside 12]

The following is a description of further specific functions of the above-described arm position / posture estimation calculating device 33.

[0076]

[Outside 13]

[0077]

[Outside 14]

[0078]

[Outside 15]

[0079]

[Outside 16]

[0080]

[Outside 17]

[0081]

[Outside 18]

[0082]

[Outside 19]

[0083]

[Outside 20]

After all, the flow of a series of processes in the arm position / posture estimation calculation device 33 is summarized as follows.

[0085]

[Outside 21]

The embodiments of the present invention have been described above. However, the present invention is not necessarily limited to the above-described means and methods, but achieves the object of the present invention and has the following effects. Within this, it is possible to change and implement as appropriate.

[0087]

As described above, according to the present invention,
Various markers corresponding to the appearance of the geometric pattern when a plurality of markers are observed from various angles are placed in advance in the visible area of the surface of the robot arm, while markers composed of a specific form of the geometric pattern are placed in advance. A pattern is prepared in advance as a template, and further, camera images of a plurality of markers arranged on the robot arm,
Since correlation is made between various texture patterns prepared as templates, it is possible to perform online tracking of the required position and orientation of the robot arm in an extremely stable state.

Further, according to the present invention, the required visual tracking can be performed by the conventional edge search or L
Not performed by binarization recognition of marker projection such as ED,
Since this is done using markers consisting of specific forms of geometric patterns, online tracking of the position and orientation of the required robot arm is reliable and almost independent of the background and light intensity of the robot working environment. It can be performed properly.

[Brief description of the drawings]

FIG. 1 is an overall configuration diagram of a robot arm visual tracking system device according to a device example of the present invention.

FIG. 2 is a functional configuration diagram of an arm position / posture estimation calculation device.

FIG. 3 is a view showing a form of a marker, an arrangement form of the marker with respect to a tip effector, and marker projection displayed on a camera image plane.

FIG. 4 is a diagram showing templates including various texture patterns stored in the template synthesizing apparatus.

FIG. 5 is a diagram showing a perspective projection model for explaining a calculation performed by an arm position / estimation calculation device.

[Explanation of symbols]

 α: Robot arm visual tracking system device 1: Work robot 11: Vehicle trolley unit 12: Robot arm 13: Tip effector 14: Marker 15: Vehicle roof unit 2: Camera 3: Robot arm control device 31: Template synthesizing device 32 ... Marker position calculation device 33 ... Arm position / posture estimation calculation device 33a ... Parallel perspective conversion device 33b ... Linear projection inverse calculation device 34 ... Marker / texture selection device

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Mizuki Tanno 3-19-2 Nishi Shinjuku, Shinjuku-ku, Tokyo Japan Telegraph and Telephone Corporation (72) Inventor Michito Matsumoto 3-19 Nishishinjuku, Shinjuku-ku, Tokyo No. 2 Nippon Telegraph and Telephone Corporation F term (reference) 3F059 AA15 BB07 DA08 DB04 DB09 FB12 5H269 AB33 BB07 CC11 JJ09 QB15 5H303 AA10 BB03 FF13 GG14 HH04

Claims (28)

[Claims] When tracking the position / posture of a robot arm in a robot operation visually on-line using a camera, a marker made of a geometrical pattern of a specific form is previously visualized on the surface of the robot arm. While placing in the area, various texture patterns corresponding to the appearance of the geometric pattern when the plurality of markers are observed from various angles are prepared and placed in advance as templates, and further, the robot arm While performing a predetermined correlation operation in time series between the images of the cameras of the plurality of arranged markers and the various texture patterns prepared in the template, the position and the position of the robot arm at each time A robot arm visual tracking method, comprising: calculating a posture; and sequentially tracking the posture. 2. A method for visually tracking the position and orientation of a robot arm during a robot operation online using a camera, wherein a marker formed of a geometric pattern of a specific form is previously visualized on the surface of the robot arm. On the other hand, various texture patterns that can be observed when a plurality of markers arranged on the robot arm are photographed by the camera are prepared in advance as templates, and are arranged on the robot arm. By performing a predetermined correlation operation between the camera images of the plurality of markers and the various texture patterns prepared as the template, the most current projection of all marker projections present on the camera image plane is performed. Calculate the marker positions for a predetermined number of marker projections suitable for observation, Calculating a position and orientation of the robot arm based on the calculated marker positions and known marker arrangement data indicating arrangement positions of the plurality of markers on the robot arm. Visual tracking method. 3. The marker uses a geometric pattern in which a part of the texture when enlarged at an arbitrary magnification matches the original texture, and the texture pattern is a texture of the marker. The robot arm visual tracking method according to claim 1, wherein an affine transformation excluding an enlargement / reduction transformation and a translation transformation is performed in various forms. 4. The predetermined correlation operation includes: selecting a predetermined number of texture patterns most suitable for respectively corresponding to the upper predetermined number of marker projections from the template; and coordinates of the upper predetermined number of marker projections In the vicinity of
The robot arm visual tracking method according to claim 1, wherein a sum of absolute values of grayscale data differences between the predetermined number of texture patterns and the corresponding number of texture patterns is obtained. 5. The robot arm visual system according to claim 4, wherein in the predetermined correlation operation, coordinates at which the sum of absolute values of the grayscale data differences gives a minimum value are output as the marker positions. Tracking method. 6. The calculation of the position / posture of the robot arm, wherein an evaluation function defined for an error between the coordinates of the predetermined number of marker projections and each of the marker positions is used. Item 6. The robot arm visual tracking method according to any one of Items 1, 2, 3, 4, and 5. 7. The position / posture of the robot arm is calculated by optimizing each marker position by applying a least squares method to the evaluation function, thereby estimating the position / posture of the robot arm. The method of claim 6, wherein: 8. A method according to claim 8, wherein the position and orientation of the robot arm are calculated by: parallel perspective projection of the remaining marker projections when any one of the predetermined number of marker projections is used as a reference marker projection. Is calculated based on the approximate position / posture of the robot arm, and by solving a linear equation that receives the calculated marker positions of the marker parallel perspective projection and the marker positions of the reference marker projection as inputs. And calculating the position and orientation of the robot arm.
4. The visual tracking method of a robot arm according to item 1. 9. The calculation of the position / posture of the robot arm is repeated by repeatedly inputting the position / posture of the robot arm once output. 9. The robot arm according to claim 8, wherein when the marker positions for the perspective projection converge to substantially constant coordinates, the position and orientation of the robot arm are output as final values. 10. Visual tracking method. 10. The method according to claim 1, wherein the predetermined number of marker projections uses at least four marker projections that are not on the same plane among all the marker projections present on the camera image plane. 2, 3, 4, 5, 6, 7,
10. The visual tracking method for a robot arm according to 8 or 9. 11. The marker according to claim 1, wherein the marker has a geometric feature point at its central portion which can indicate its own reference position. , 6, 7,
11. The visual tracking method for a robot arm according to 8, 9, or 10. 12. The marker comprises two light-colored portions and two dark-colored portions which are alternately circulated around the marker, and wherein the two light-colored portions or the two dark-colored portions generated by this arrangement form. The robot arm visual tracking method according to claim 11, wherein a symmetric point is used as the geometric feature point. 13. The robot arm visual tracking method according to claim 12, wherein the marker uses a color in which the bright portion is white and the dark portion is black. 14. The marker according to claim 1, wherein the marker is formed in a square shape as a whole.
14. The robot arm visual tracking method according to 8, 9, 10, 11, 12, or 13. 15. The device according to claim 1, wherein the markers are arranged intensively in a surface visible region of a tip effector forming a working point of the robot arm. 7,
The robot arm visual tracking method according to 8, 9, 10, 11, 12, 13, or 14. 16. A marker which is arranged in a visible region of the surface of the robot arm and which is formed of a geometric pattern having a form in which a part of the texture when enlarged at an arbitrary magnification matches the original texture, is arranged on the robot arm. A camera that captures a plurality of markers, and a template composition that generates a plurality of texture patterns by applying affine transformations to the textures of the markers in various forms except for scaling / transformation and translation transformations, and saves them as templates. A predetermined number of images between the camera image obtained by photographing the plurality of markers arranged on the robot arm with the camera and the plurality of texture patterns in the template stored in the template synthesizing device; Of all marker projections existing in the camera image plane A marker position calculating device capable of calculating each of the marker positions, a known position indicating each of the marker positions actually calculated by the marker position calculating device, and an arrangement position of the plurality of markers on the robot arm. An arm position / posture estimation calculating device that calculates the position / posture of the robot arm based on the marker arrangement data; and, based on the position / posture of the robot arm calculated by the arm position / posture estimation calculation device, From among all the marker projections present on the camera image plane, and at the present time, selecting a top predetermined number of marker projections most suitable for observation, and at the time of the predetermined correlation calculation performed by the marker position calculation device, A predetermined number of texts most suitable to correspond to the selected predetermined number of marker projections, respectively. A marker / texture selecting device for selecting a cha pattern from the templates stored in the template synthesizing device. 17. The marker position calculating device according to claim 1, wherein the predetermined correlation operation is performed by the marker / texture selecting device in the vicinity of the coordinates of the upper predetermined number of marker projections selected by the marker / texture selecting device. 17. The marker / texture selection device according to claim 16, wherein the marker / texture selection device is connected so as to be capable of performing a correlation operation for obtaining a sum of absolute values of grayscale data differences between the selected corresponding predetermined number of texture patterns. Robot arm visual tracking system device. 18. The marker position calculating device, when performing the correlation operation on the grayscale data difference, outputs the coordinates at which the sum of absolute values of the grayscale data difference gives the minimum value as the respective marker positions. The robot arm visual tracking system device according to claim 17, wherein the robot arm visual tracking system device is connected to the arm position / posture estimation calculation device. 19. The arm position / posture estimating calculation device includes: coordinates of the predetermined number of marker projections selected by the marker / texture selection device; and each of the marker positions calculated by the marker position calculation device. The robot arm visual tracking system device according to claim 16, which has a function of defining an evaluation function representing an error between the robot arm and the robot arm. 20. The arm position / posture estimation calculating device, wherein the least square method is applied to the evaluation function to optimize each of the marker positions calculated by the marker position calculating device, so that the robot arm The robot arm visual tracking system device according to claim 19, further comprising a function of estimating the position and orientation of the robot arm. 21. The arm position / posture estimating arithmetic unit, wherein an arbitrary one of the upper predetermined number of marker projections selected by the marker / texture selector is used as a reference marker projection. A parallel perspective transformation device for calculating a marker parallel perspective projection for the marker projection based on the approximate position / posture of the robot arm calculated by the arm position / posture estimation calculation device; and A linear projection inverse operation device for calculating the position and orientation of the robot arm by solving a linear equation using the marker positions of the obtained marker parallel perspective projection and the marker positions of the reference marker projection as inputs. The method according to claim 16, 17, 18, 19 or 2.
0. The robot arm visual tracking system device according to 0. 22. The arm position / posture estimating operation device repeatedly performs a process of inputting the position / posture of the robot arm once output from the linear projection inverse operation device to the parallel perspective transformation device, and repeats this process. In the process, when the marker positions for the marker parallel perspective projection calculated each time by the parallel perspective transformation device converge to substantially constant coordinates, the position / posture of the robot arm is finally set. 22. The robot arm visual tracking system device according to claim 21, wherein the value is freely output as a value. 23. The marker / texture selection device, as the upper predetermined number of marker projections, selects at least four marker projections that are not on the same plane from among all the marker projections present on the camera image plane. 16. A function of performing the following:
23. The robot arm visual tracking system device according to 0, 21 or 22. 24. The marker according to claim 16, wherein the marker has a geometrical feature point at its center, which can indicate its own reference position.
24. The robot arm visual tracking system device according to 0, 21, 22 or 23. 25. The marker comprises two light-colored portions and two dark-colored portions arranged alternately around the marker, and the marker is provided between the two light-colored portions or the two dark-colored portions generated by this arrangement. The robot arm visual tracking system device according to claim 24, wherein a symmetry point is used as the geometric feature point. 26. The robot arm visual tracking system according to claim 25, wherein the marker has a white color in the light color portion and a black color in the dark color portion. 27. The marker according to claim 16, wherein the marker is entirely formed in a square shape.
The robot arm visual tracking system device according to 0, 21, 22, 23, 24, 25 or 26. 28. The marker according to claim 16, wherein the markers are arranged intensively in a surface visible region of a tip effector forming a working point of the robot arm.
28. The robot arm visual tracking system device according to 0, 21, 22, 23, 24, 25, 26 or 27.