JPS62222108A - Image processing method in robot system - Google Patents

Abstract

PURPOSE:To well perform confirmation even if the position of a camera changes, by using a reference article to calculate a converting parameter converting a picked up image of a TV camera into a reference coordinates system. CONSTITUTION:The image of parts on a table 10 is picked by TV cameras 1, 2, and the arm 11A grasping the parts and a robot main body 11 are controlled by a robot control apparatus 12 and an image processing unit 3 is connected to the cameras 1, 2 and the apparatus 12. A parameter for converting a scale into a reference coordinates system determined by the kind of a reference article and the relative distance between the cameras 2, 2 is preliminarily calculated. On the basis of this parameter, the scale correction of the picked-up image of the parts to be taught and confirmed is performed. The image after this correction is one under the reference coordinates system and comes to one from a reference position having no relation to the actual positions of the cameras 1, 2. Therefore, regardless of the positions of the cameras 1, 2, the pattern comparison and confirmation of the measured data from the parts to be confirmed and the reference teaching data from the parts to be taught can be performed.

Description

DETAILED DESCRIPTION OF THE INVENTION [Field of Industrial Application] The present invention relates to an image processing apparatus in a robot system, and particularly to an image processing method in a robot system having a fixed TV camera and a mobile TV camera.

[Conventional technology]

In a robot system, there may be a process of first recognizing an object to be handled. In order to recognize an article, a TV left camera is installed at a fixed position, this TV left camera takes an image of the article, and then the photographed image is recognized. In this recognition process, a coordinate system based on the TV left camera position is used to specify the size of the item. That is, image description is performed using a scale unique to the TV left camera.

On the other hand, since the fixed TV camera is insufficient in recognizing objects, there is a method of attaching a second TV left camera to the arm of the robot and switching between these two TV left cameras to take images of the object. A conventional example of attaching a TV left camera to a robot arm is described in Japanese Patent Publication No. 51-35988.

[Problem that the invention seeks to solve]

According to the content described in Japanese Patent Publication No. 51-35988, the position of the TV left camera changes with the arm of the robot. The size of the captured image changes depending on the position of the TV left camera. In image processing, pattern matching is performed between a captured image and a reference image, or predetermined features are extracted from the captured image, and pattern matching is performed between the extracted features and the reference features. Therefore, it is necessary to prepare reference images and reference features depending on the size of the captured image. However, this preparation is difficult and practically difficult to employ. Furthermore, since the shape of the target article varies, it is necessary to prepare a reference image and reference features for each article. From these points of view, it is difficult to recognize objects by attaching a TV left camera to the arm of a robot.The object of the present invention is to provide images even if the robot system has a fixed TV camera and a movable TV camera. An object of the present invention is to provide an image recognition method for a robot system that requires a small amount of various information for recognition and simplifies the recognition processing itself.

[Means for solving problems]

The present invention uses a reference article as a fixed TV camera.

Conversion parameters (correction amount) for converting to the reference coordinate system for each movable TV camera mounted on the robot arm
was determined, scale correction was performed on the captured image of the actual target article using the above parameters, and the image after this correction was recognized.

[For production]

According to the present invention, parameters for converting to the reference coordinate system are determined based on the reference article, and scale correction of the captured image of the actual target article is performed based on these parameters. The image after this scale correction is the original image of the reference coordinate system, and is an image from the reference position that is unrelated to the actual TV left camera position. In image recognition, this corrected image becomes the recognition target, and image recognition is performed regardless of the position of the TV left camera.

〔Example〕

FIG. 1 shows the robot system and control/processing system of the present invention.

The robot body 11 is controlled by a robot control device 12 . The image processing bag @3 recognizes images captured by the TV cameras 1 and 2. This device 3 is equipped with a CRT 4 and a keyboard 5.

The TV camera 1 is a fixed camera, and the TV camera 2 is a movable TV camera attached to a robot arm IIA.

Various parts 7, 8.9 are mounted on the workbench IO, and are handled by the robot hand IIB.

The light projector 6 projects light near the workbench.

The TV camera 1 does not move, and the TV camera 2 moves vertically and horizontally together with the arm IIA. It may also rotate. Therefore, the size of the captured image of one component 9 changes depending on the position of the TV camera 2.

In this robot system, parts 7.8 are used for hand IIB.
．． 9 to move or process. When gripping, grasp the shape and characteristics of the parts in advance to ensure accurate gripping with the hand 11B.

Alternatively, it can be used to specify the processing position, determine the moving direction, etc. Therefore, it is necessary to recognize the goods.

TV cameras 1 and 2 are used to recognize this part. Images of the parts are taken using the TV cameras 1 and 2.

Perform image recognition. This image recognition is performed by the image processing device 3. This image recognition result is sent to the control device 12.

It is used to control the robot 11, to control the movement of the arm 11A, and to control the grip of the hand 11B.

FIG. 2 is a system diagram centered on signal flow for the device disclosed in FIG. A switch 31 switches which image from the TV camera 1 or 2 is to be captured. The AD converter 32 performs AD conversion of the imaging signal from the switch 31. The image memory 33 stores captured images after AD conversion.

The CPU 34 controls the robot 11 and performs image processing and image recognition. Image processing includes binarization processing and feature extraction processing. The CRT 4 and keyboard 5 serve as input and display means for a man-machine system.

ROM 35 stores microprograms.

The RAM 36 stores various data. The RAM 37 stores teaching data. The communication unit 38 forms an interface between the host computer 13, the robot control device, and the CPU 34.

The host computer 13 serves as a computer above the CPU 34 and serves to manage the C:PU 34 and the like.

With the above configuration, each component 31.32.33.34.3
5°36, 37.38 constitute the image processing device 3 of FIG.

Now, with the above +q configuration, one image processing device 3 has three functions. The flow is shown in FIG. What are the three functions?
Reference teach mode, component teach mode 1''! This refers to the recognition mode. Which mode it is in is determined by the mode switch. The mode switch may be provided on the keyboard 5, or may be specified by the CRT display.

This can also be achieved through software.

(1), Reference teach mode.

This mode is specified first. The reference teaching mode refers to processing for obtaining conversion parameters (for correction) for scaling an actual measurement image to a reference coordinate system.

The reference teach mode is specified by any one of the following conditions.

■When the position of the fixed camera changes.

◎When the mounting position of camera 2 on the robot arm is changed.

◎When the height of table 10 is changed.

As is clear from this example, the reference teach mode is used to set the fixed TV camera 1. Movable TV camera 2. This is done when each position of the table 10 is determined.

The procedure of the reference teaching mode will be explained with reference to Fig. 4. 6 (a) The position of the fixed TV camera 1 is fixed, the mounting of the TV camera 2 on the arm is fixed, and the table 10 is fixed.
The reference part A1 is placed on the table 10 under the condition that the height of the reference part A1 is fixed (FIG. 5).

(b) The reference part AI is imaged by the fixed camera 1 and the camera 2. First, the switch 31 operates to capture an image captured by the camera 1. As a result, the vjl image of camera 1 is
The image is digitized through the AD converter 32 and stored in the image memory 33.

Next, the switch 31 selects the camera 2. Camera 2 moves together with arm IIA. Therefore, the arm 11A is moved to various specified positions PIIP2...pn, and the component A1 is imaged by the TV camera 2 at each specified position. As a result, the TV camera 2 obtains captured images with different scales for each designated position. These images are successively taken into the image memory 34 via the AD converter 32 and stored therein. Now, for the sake of simplicity, if we assume that the arm 11A moves only up and down, and that we give n specified positions, the 1-image memory 33 contains one image from the fixed TV camera 1 and one image from the TV camera 2. , n, and images with statistics (.+1) are stored.

(c) 9 Next, conversion parameters are created for each image in the image memory 33. This conversion parameter is created by CPU3.
4 will do it. The creation procedure is as follows.

(i) The surface area of the leveling part A1 is calculated from the keyboard 5 by the CPU.
34.

(]1) Calculate the measurement area of the reference component for each (n+1) image. This situation is shown in FIG. However, the position P is expressed as a vector Pi.

(U) When the reference area of the reference part is So, and the measured area for each of (.+1) images is 5IIS2...1sfl+1, the conversion parameter is calculated for each image using the following equation.

αn++ = J S n++ / J S.

α1 is a scale conversion parameter of the image taken by the fixed camera 1, α2 is a scale conversion parameter of the image taken at the position P1 of the fixed camera 2, and so on.

αn+1 is a scale conversion parameter of the captured image at the position pH of the fixed camera 2.

The thus obtained conversion parameter α1. α2. ....

αnu is the type and position of the TV camera (P + p P
2, etc.) and stored in the memory 36.

Here, the position of TV camera 1 is from keyboard 5 to CPU.
34, and each designated position of the TV camera 2 can be obtained by the CPU 34 taking in the designated movement position of the robot arm.

This conversion parameter α is correction data for converting the camera position to the reference coordinate system.

Therefore, if the feature quantity (for example, area) of an image obtained at a certain position P is scale-corrected using this conversion parameter, it becomes an image based on the reference coordinate system rather than a measured image at position P. In other words, the image is unrelated to the camera position.

This conversion parameter is the component teach mode, which will be described later.
Used for scale correction in recognition mode.

(d) Instead of the conversion parameter α, a conversion parameter as a higher level concept may be used.

The procedure is shown below.

The conversion parameter α is proportional to the distance between the camera and the surface to be imaged. Therefore, the following equation holds.

K-e ・(P 1Po) = a; ・
= (2) Po...Position of the center of gravity in the image of the reference part expressed in the robot coordinate system.

Pl...Camera position e expressed in robot coordinate system...
Camera optical axis direction unit vector K...proportionality constant Equation (2) can be transformed into the following equation if the direction of the camera does not change, that is, if the camera only moves up and down.

A-1', -B=αi...(3)
Here, A=-e (constant vector) and B=-e-Po (constant). Equation (3) holds true for all i. Therefore, when calculating the difference at i+i-1, we get A・ΔF,=Δα,...(5)
becomes. Here, ΔF+=P+ Pt-+, Δα1=
αI-α1-1 Therefore, from equation (5), A is Δ=ΔP, -1・Δα, ...(6)
Now, the difference ΔF at four different positions is ΔP + +Δ
P21ΔP3, and the difference in conversion parameters Δα1 is △
αi, Δα2. When Δα3, it can be expressed as equation (6). If the difference ΔF is expressed in three-dimensional coordinates (x, y+2), then B can be found as follows using equation (4).

B=A-Pt-α1...
(9) Thus, when A and B are found, α=A-P-8...(10)
From this, the conversion parameter α can be found.

In order to obtain α using this equation (10), A and B must be registered.

(2) 0 parts teach mode.

This mode is the next process after the standard teach mode. In the component teach mode, a component to be taught is placed on the table 10 instead of the reference component, and the camera 1. Or, it refers to the work of capturing an image in step 2 and importing it as teaching data.

Whether camera 1 or 2 is used depends on the type, size, and mounting position of the part to be taught.

FIG. 8 shows the processing flow of the component teach mode. First, camera 1 or 2 is selected depending on the part to be taught. This selection instruction is made using the keyboard 5. Next, input the camera position. Camera 1 is.

This is a fixed position, and for the camera 2, this is the arm position at that time.

After inputting the camera position, the part is imaged with the selected camera. This imaged part image (pattern) is taken into the image processing device, and the part number (code) is inputted from the keyboard 5. This part number is a part identification number.

The image processing device processes this image and extracts feature amounts. The feature quantity is the area of the image 2 the perimeter.

These include the second moment of inertia, the area of the internal hole in the image, etc.

Next, the position coordinates of the camera are converted to a reference coordinate system. When the fixed camera 1 is selected, scale conversion is performed using α0 shown by equation (1) in the reference teach mode as a conversion parameter. When the movable camera 2 is selected, scale conversion is performed using α shown in equation (1) or α shown in equation (10) in the reference teach mode as a conversion parameter.

For example, if the length Q is extracted as a feature, the length Qo after scale conversion is calculated using the following equation.

Ω0 Qo=□ (11) α For example, if one area Sr is extracted, the area S after scale conversion is calculated by the following formula.

As is clear from equations (11) and (12), the conversion is performed by α1 depending on the dimension of the feature amount.

Extraction of feature quantities is not limited to just one, but length and area.

It is often plural, such as perimeter. Therefore, each scale is converted to calculate the reference value in the Keitei coordinate system.

These calculated values are stored in memory separately for each part number.

The above processing is performed for the number of parts existing on the table 10. Therefore, various reference values identified by one part number are stored in the memory.

The method of inputting the part number is as follows. 6th on CRT
Component images 2 and 3 as shown in Figures (A) and (B) are displayed. Therefore, the user points to component image 2 with the cursor and inputs the number of the component using the keyboard. By pointing the cursor to part image 3 and keying in this part number on the keyboard, the part number of image 3 is input.

This completes the teaching of the parts teaching data.

What should be noted here is that regardless of the position of camera 1 or camera 2, a complete reference value can be obtained for the same part by teaching it once. That is, twice,
Teaching multiple times, such as three times, is unnecessary. The reason for this is
This is because the scale was converted to the reference coordinate system using the conversion parameter. This is a major feature of this embodiment.

(3) Recognition mode The recognition mode is a process in which a part to be recognized is actually placed on the table 10, and the part is imaged by the camera 1 or 2 to perform part recognition.

FIG. 9 shows the processing contents of the recognition mode.

First, as a premise, parts to be recognized are mounted on the table 10. Next, a selection is made as to which of cameras 1 and 2 should be used to image this part. In FIG. 2, switch 31 makes this selection. This selection is determined by the robot's instructions.

Enter the camera position. If you select camera 1,
Enter a fixed position. If camera 2 is selected, capture the position of the robot arm and obtain the camera position.

The selected camera images the part and obtains a part image (pattern). This part image is taken into the image processing device 3. The processing device 3 extracts feature quantities according to a predetermined method 6
If there are multiple parts, the feature amount is extracted for each part.

Next, the extracted feature amount is converted to the reference coordinate system for each part.

As a result, the feature amounts have the same scale as those measured using the reference coordinate system. The three methods for performing this scale conversion using the conversion parameter α are the same as those described above.

Comparison methods for comparing the teaching data (reference value) obtained in the teaching mode with the scale-converted feature amount include a pattern matching method and the like.

If the comparison results in a match, the part is assumed to match the teaching data and is subjected to post-processing. In post-processing, the recognition results are used to convert them into the data format required by the robot, or to be used to calculate the position of parts.

Output this result to the robot. It may also be displayed on a CRT.

If the result of the comparison is that they do not match, it is checked whether there is an unrecognized part on the screen, and if so, similar processing such as scale conversion of the feature amount of the part, comparison with teaching data, etc. is performed.

In addition to the surface 1iIS, length, concentration distribution, etc. can also be used as source data for the conversion scale.

According to this embodiment, there are the following effects.

(1) It is easy to teach.

There is no need to reteach parts for each camera.

(2) The amount of total teaching data is small.

There is no need to have data for each camera for one part.

(3) The teaching data is compatible.

If there are multiple units of this device, the data of the parts taught in one unit can be used as is in other devices because the scale is converted.

(4) It is possible to easily deal with misalignment of the camera.

Even if the camera is misaligned, all you need to do is teach the reference, and there is no need to reteach the parts.

(5)', can be easily combined with CA D and CAM.

Data that can be calculated using CAD or CAM can directly serve as teaching data for this processing device.

〔Effect of the invention〕

According to the present invention, even if the camera position changes, parts can be recognized by converting them into a general coordinate system (reference coordinate system).
Obtain the effect of performing good recognition.

[Brief explanation of drawings]

Fig. 1 is an embodiment diagram of the present invention, Fig. 2 is a processing system diagram, and Fig. 3 is a diagram of an embodiment of the present invention.
4 is a flowchart of processing in the standard teaching mode, FIG. 5 is a diagram showing the state of the table during standard teaching, and FIG. 6 (A). (b) is a side view of the captured image, FIG. 7 is a process diagram for obtaining reference data, FIG. 8 is a process flow diagram of component teaching mode, and FIG. 9 is a process flow diagram of recognition mode. 1.2... TV camera, 3... Image processing device, 11
...Robot body, 12...Robot control device. Agent Patent Attorney Tadashi Akimoto Figure 1, Figure 2, Figure 3, Figure 2, Figure 5? ”5 6 Figure (a) (b)

Claims (1)

[Claims] 1. A table on which a component is mounted, an arm that grips the component, a camera that images the component, a robot body that supports the arm, and a robot control device that controls the robot body and the arm. , in a robot system consisting of the robot control device and an image processing device connected to the camera, the image processing device calculates in advance a scale conversion parameter determined by the type of reference part and the relative distance between the reference part and the camera. In addition to being stored in the internal memory, the part to be taught mounted on the table is imaged with a camera.
The captured image is processed by an image processing device to extract the feature amount, and the extracted feature amount is scale-converted based on the conversion parameters determined in advance as described above to obtain reference teaching data. The standard teaching data is stored in the memory of the image processing device corresponding to each part, and then the recognition target part mounted on the table is imaged with a camera, and the captured image is processed by the image processing device to extract features. , the extracted feature quantity is scale-converted based on the conversion parameters obtained in advance, to obtain measurement data, and a pattern comparison is made between the measurement data and the reference teaching data in the memory, and if they match, An image processing method in a robot system, characterized in that: is recognized as a part that has provided the reference teaching data. 2. The processing method according to claim 1, wherein the camera comprises a fixed camera and a movable camera attached to an arm.