TWI776694B - Automatic robot arm system and method of coordinating robot arm and computer vision thereof - Google Patents

Abstract

An automatic robot arm system and a method of coordinating robot arm and computer vision thereof are disclosed. A beam-splitting mirror splits the incident light into the visible light and the ranging light and respectively guides to an image capturing device and optical ranging device in the different reference axes. In a calibration mode, a transformation relationship is computed based on the calibration postures and the corresponding calibration images. In a work mode, a mechanical space coordinate is computed based on the captured word image and the transformation relationship, and the robot arm is controlled to move based on the mechanical space coordinate. The present disclosure can make the capture optical axis overlap the flange axis of the robot arm, and improve the coordination between the robot arm and the computer vision.

Description

Coordination between the automated robotic arm system and the robotic arm and its computer vision Law

The present invention is related to a robotic arm, in particular to a method for coordinating between an automated robotic arm system and its robotic arm and computer vision.

In the existing robotic arm system, a camera is used to capture an image of the working object, the position of the working object is determined through image analysis, and the robotic arm is controlled to move to the determined position to act on the working object.

However, the disadvantage of the existing robotic arm system is that if the robotic arm and the camera are coaxially configured one above the other, the processing range of the robotic arm will be severely reduced, and the upper limit of the camera volume will be severely limited.

If the manipulator and the camera are set non-coaxially, there will be an offset between the flange axis of the manipulator and the shooting optical axis of the camera. The aforementioned offset will cause random errors between the visual space based on the captured image and the mechanical space of the robotic arm, making it impossible for computer vision to accurately control the robotic arm.

Therefore, the existing robotic arm systems have the above problems, and more effective solutions are urgently needed.

The main purpose of the present invention is to provide an automatic robotic arm system and a method for coordinating between its robotic arm and computer vision, so that the shooting optical axis and the flange axis can be superimposed, the target distance can be measured, and the robotic arm and computer vision can be coordinated. .

In one embodiment, a method for coordinating between a robotic arm and computer vision includes: in a calibration mode, controlling a robotic arm on an image capturing device based on a target distance measured by an optical ranging device. Move into a plurality of calibration postures within an effective shooting range, and capture a plurality of calibration images in the calibration postures through the image capture device, wherein a beam splitter guides visible light to a flange disposed on the robotic arm The image capture device is off-axis, and the ranging light is guided to the optical ranging device, the ranging axis of the optical ranging device is parallel to or overlapping the flange axis; based on the plurality of corrected attitudes and the plurality of A calibration image is used to calculate a conversion relationship between the visual space of the image capture device and the mechanical space of the robotic arm; in a working mode, a working image is captured by the image capture device, and based on the working image and the The conversion relationship determines a mechanical space coordinate for performing work; and, controls the robotic arm to move to the mechanical space coordinate.

In one embodiment, an automated robotic arm system includes a robotic arm, an image capturing device, an optical ranging device, an optical path structure and a control device. The mechanical arm is used to move in a three-dimensional space. The image capturing device is arranged outside the flange shaft of the robotic arm and is used for capturing images. The optical distance measuring device is arranged on the mechanical arm for measuring the distance of a target, and the distance measuring axis of the optical distance measuring device is parallel to or overlapping the flange axis. The optical path structure includes a beam splitter for guiding visible light to the image capturing device and guiding the ranging light to the optical ranging device. Should The control device is connected to the robot arm, the image capture device and the optical distance measuring device, and the control device is set to control an effective shooting distance of the robot arm to the image capture device based on the target distance in a calibration mode The internal movement is a plurality of calibration postures, and the image capture device is controlled to capture a plurality of calibration images respectively in the plurality of calibration postures, and the visual space and the visual space of the image capture device are calculated based on the plurality of calibration postures and the plurality of calibration images. A transformation relationship between the mechanical spaces of the robotic arm. The control device is set to control the image capture device to capture a working image in a working mode, determine a mechanical space coordinate for executing work based on the working image and the conversion relationship, and control the robotic arm to move to the mechanical space coordinate.

The invention can superimpose the shooting optical axis and the flange axis of the mechanical arm, and improve the coordination between the mechanical arm and computer vision.

1: Robotic arm system

10: Robotic arm

11: End effector

12: Camera

13: Goals

140: Flange shaft

141: Shooting Optical Axis

15: Target image

16: Range

17: Range after rotation

18: Axis

2: Automated robotic arm system

20: Control device

21: Image capture device

210: Photosensitive element

211: Lens

22: Optical ranging device

220: Light Emitter

221: Optical Receiver

23: Robotic Arm

230-233: Joints

24: Optical path structure

240: Beamsplitter

241: Reflector

25: Storage device

250: Computer programming

251: Effective shooting distance

252: Conversion relationship

26: Goals

30: Control the computer

31: Robotic arm controller

32: Peripherals

40: Shooting control module

41: Ranging control module

42: Arm control module

43: Calibration control module

44: Work Control Module

45: Conversion processing module

46: Image Analysis Module

50: light source

51: Goals

52: Jig

53: Flange shaft

54: Working device

55: Install the base

60: Video

600-601: Location

α 1: Rotation angle

d1: offset

h1: target distance

P1, P2: Attitude

V1: Variation

S10-S16: Coordination steps

S20-S25: Calibration steps

S30-S33: Working steps

FIG. 1 is a structural diagram of an automated robotic arm system according to an embodiment of the present invention.

FIG. 2 is a partial structural diagram of an automated robotic arm system according to an embodiment of the present invention.

FIG. 3 is a structural diagram of a control device according to an embodiment of the present invention.

FIG. 4 is a flowchart of a coordination method according to an embodiment of the present invention.

FIG. 5 is a partial flowchart of a coordination method according to an embodiment of the present invention.

FIG. 6 is a partial flowchart of a coordination method according to an embodiment of the present invention.

FIG. 7 is a schematic diagram of the arrangement of an automated robotic arm system according to an embodiment of the present invention.

FIG. 8 is a first schematic diagram of a calibration mode according to an embodiment of the present invention.

FIG. 9 is a second schematic diagram of a calibration mode according to an embodiment of the present invention.

FIG. 10 is a schematic diagram of the first corrected image captured in FIG. 8 .

FIG. 11 is a schematic diagram of the second corrected image captured in FIG. 9 .

FIG. 12 is a schematic diagram of the arrangement of a conventional robotic arm system.

FIG. 13 is a schematic diagram of the field of view of a conventional robotic arm system.

Hereinafter, a preferred embodiment of the present invention will be described in detail in conjunction with the drawings.

Please refer to FIG. 12 and FIG. 13 , FIG. 12 is a schematic diagram of the arrangement of a conventional robotic arm system, and FIG. 13 is a schematic diagram of a visual field of the conventional robotic arm system.

As shown in FIG. 12 , the camera 12 of the robot arm system 1 and the robot arm 10 are arranged on different axes. Due to the non-axial configuration, there is an offset d1 between the shooting optical axis 141 of the camera 12 and the flange axis 140 of the robot arm 10 . The offset d1 will cause random errors in positioning in the visual space and the mechanical space.

The end effector 11 is directly disposed at the end of the robotic arm 10 . When the robot arm 10 moves the end effector 11 (ie, changes the posture), the field of view of the camera 12 mounted on the robot arm also changes accordingly, and the target 13 can be photographed at different angles.

As shown in FIG. 13 , when the robot arm 10 rotates at an angle α 1 with the flange shaft 140 as the axis 18 , since the end effector 11 does not move in the horizontal direction, the target 13 can actually be processed.

However, the field of view of the rotated camera 12 will change from the range 16 to the rotated range 17 , so that the target image 15 of the target 13 is out of the field of view of the camera 12 , which will cause the robotic arm system 1 to fail to perform the visual space on the target 13 . position.

In order to solve the problems caused by the above-mentioned different axis configurations, the present invention proposes a coordination method between an automated robotic arm system and its robotic arm and computer vision, which can pass through a novel optical path structure. (especially the beam splitter) to make the incident end of the shooting optical axis fit to the flange axis to achieve the effect of hand-eye coaxial.

Moreover, since the present invention can realize the coaxiality of the hand and the eye, the offset problem caused by different axis configurations can be eliminated, and the target object can be prevented from exceeding the field of view.

In addition, because the present invention adopts different axis configuration, the processing range of the robot arm can be greatly increased, and the upper limit of the camera volume can be greatly increased.

In addition, the present invention can also improve the coordination between the mechanical arm and the computer vision by assisting the positioning correction between the visual space and the machine through optical ranging.

Please refer to FIG. 1 , which is a structural diagram of an automated robotic arm system according to an embodiment of the present invention. The automated robotic arm system 2 of the present invention mainly includes an image capturing device 21 , an optical ranging device 22 , a robotic arm 23 , a storage device 25 and a control device 20 connected to the above-mentioned devices.

The image capturing device 21 is, for example, a color camera such as an RGB camera, and is used for photographing objects in the work area to obtain color images including the objects (such as the correction image and the working image to be described later). The aforementioned color images are mainly used to perform computer vision analysis, and provide calculation results as a motion reference for the robotic arm 23 .

The robotic arm 23 is used to move the mounted device in the three-dimensional space, so as to perform measurement (mounting the optical ranging device 22 ), photographing (mounting the image capturing device 21 ), and processing (mounting the work) for different positions device 54) and so on.

The end of the robot arm 23 is set with a virtual flange axis (for example, the movement reference point of the robot arm 23 ), and the spatial position of the end can be calculated and determined based on the flange axis. The calculation of the aforementioned flange axis is a prior art in the field of control of the robot arm 23 , and details are not described herein again.

In the present invention, the image capturing device 21 is disposed outside the flange shaft of the robot arm 23, so as to increase the processing range of the robot arm 23 (determined by the movable range of the flange shaft), and improve the image. The upper limit of the allowable volume of the capture device 21 is that a camera with a larger volume and higher performance can be used, and the wiring restrictions are relatively loose.

In one embodiment, when the end of the robotic arm 23 is mounted with a working device 54 (as shown in FIGS. 8 and 9 ), the working device 54 can perform processing on different positions through the movement of the robotic arm 23 . By carrying different working devices 54, the present invention can realize different applications.

In one embodiment, the working device 54 can be connected to and controlled by the control device 20 to perform automated actions.

For example, when the working device 54 is a gripping end effector, welding heater, marking tool, grinding tool, assembling end effector, gluing tool and/or locking tool, the aforementioned automated actions may be corresponding gripping tools Actions (such as clamping or sucking electronic components), welding actions (such as controlling the heating of the laser welding head), marking actions (such as marking by branding, spraying, etc.), grinding actions (such as performing cutting, grinding, etc.) ), assembly actions (such as splicing, stacking, etc. of multiple objects according to the specified assembly method), gluing actions (such as gluing, dispensing, etc.) and/or locking actions (such as locking screws, nuts) .

The optical ranging device 22 , such as an infrared range finder, is used to measure the target distance between the optical ranging device 22 and the target through optical means.

In one embodiment, the aforementioned measurement is performed by placing the target on a virtual distance measuring axis, and obtaining the measurement in a direction parallel to the distance measuring axis through a triangulation method.

In one embodiment, the optical distance measuring device 22 is disposed at the end (or near the end) of the robotic arm 23, and can measure the distance between the end and the target.

In one embodiment, the distance measuring axis of the optical distance measuring device 22 can be parallel to or overlap the flange axis of the robot arm 23, so that the measured target distance corresponds to the end of the robot arm 23 and the target directly below the flange axis. depth value in between.

The optical path structure 24 is disposed at the end (or near the end) of the robotic arm 23 to receive incident light (light emitted or reflected from the target), divide the incident light into visible light and ranging light, and guide them to the image capturing device respectively 21 and the optical ranging device 22.

Specifically, the optical path structure 24 includes a beam splitter 240 (as shown in FIG. 7-9 , for example, an optical lens), and the beam splitter 240 can separate the incident light into rays of different wavelengths (the principle is that the rays of light with different wavelengths have different refractive indices) , for example, the incident light is divided into visible light and infrared light (ranging light). After separation, the visible light can be guided through the visible light path (reflectors or lenses can be provided or directly injected) to the lens 211 and the photosensitive element 210 of the image capture device 21 (as shown in FIG. 7 ). The distance optical path is guided (a mirror or lens can be provided or directly incident) to the light receiver 221 of the optical distance measuring device 22 . Thereby, the optical path structure 24 can realize the coaxial configuration of the flange axis, the distance measuring axis, and the photographing optical axis (for example, the center point of the photographing field or other reference points) at the incident end, and allows the image capturing device 21 to be arranged on the Outside the blue axis (and the distance measuring optical axis).

The storage device 25 , such as a hard disk, a solid-state drive, ROM, RAM, EEPROM, flash memory, or any combination of various storage media, is used to store data, such as the effective shooting distance 251 and the conversion relationship 252 .

The control device 20 is used to control the automated robotic arm system 2, for example, to control the calibration mode and the working mode.

Please refer to FIG. 2 , which is a partial structural diagram of an automated robotic arm system according to an embodiment of the present invention. In this embodiment, the control device 20 may include a control computer 30 and a robotic arm controller.

The robotic arm controller 31 is connected to the robotic arm 23 for controlling the movement of the robotic arm based on the received arm control commands.

In one embodiment, the robotic arm 23 includes a plurality of joints 230-233 (as shown in FIG. 8 to FIG. 9 ) for providing multiple degrees of freedom. The arm 23 can move in multiple degrees of freedom.

The arm control command can indicate the moving destination (mechanical space coordinate) of the robot arm 23, and the robot arm controller 31 can convert the arm control command into the corresponding posture coordinates (such as the rotation angle of each joint 230-233), and control each joint 230-233 turns to pose corresponding to arm control commands.

The control computer 30, such as an industrial computer or a personal computer, is connected (eg, through an industrial network or other local area network) to the robot arm controller 31, the image capture device 21, the optical distance measuring device 22 and the storage device 25, and controls the These devices are controlled. For example, the control computer 30 can control the robotic arm 23 by issuing the aforementioned arm control commands to the robotic arm controller 31 .

In one embodiment, the control computer 30 is also connected to peripheral devices 32 , such as a communication interface (used to connect to a network), a human-machine interface (used to interact with a user), a power supply device (used to provide power), and the like.

Please refer to FIG. 7 , which is a schematic diagram of the arrangement of an automated robotic arm system according to an embodiment of the present invention.

As shown in FIG. 7 , the automated robotic arm system 2 includes a mounting base 55 . The mounting base 55 is connected to the end of the robotic arm 23 and can be moved by the robotic arm 23 in a three-dimensional space.

In addition, the image capturing device 21 , the optical distance measuring device 22 and the optical path structure 24 are all disposed on the mounting base 55 .

In one embodiment, the mounting base 55 can be provided with one or more light sources 50 (eg, ring light sources), and the light sources 50 are used to illuminate the work area (especially the target 51 and the fixture 52 ), so that the image capture device 21 A target image with better brightness can be obtained, and the influence of changes in ambient brightness can be greatly reduced.

In one embodiment, the light path structure 24 may include a beam splitter 240 and a reflector 241 . The reflector 241 is used for reflecting the visible light separated by the beam splitter 240 to the lens 211 and the photosensitive element 210 of the image capturing device 21 . Through the beam splitter 240 and the reflecting mirror 241 , the photographing optical axis of the image capturing device 21 can fit the flange axis 53 of the robotic arm 23 . In addition, the distance measuring optical axis of the optical distance measuring device 22 can be parallel to or attached to the flange axis 53 .

In one embodiment, the beam splitter 240 can be a longpass dichroic mirror, and has a visible light reflectivity of more than 80% (eg, 97%) and an infrared transmittance of more than 75% (eg, 92%). , for example, allowing light with a wavelength above 730 nm (eg, 750 nm) to pass through, and reflecting light with a wavelength of 300 nm-730 nm (eg, 450 nm-490 nm).

In one embodiment, the optical ranging device 22 includes an optical transmitter 220, an optical receiver 221, and a ranging controller (not shown) connected to the above-mentioned devices. The vertical line between the midpoints of the light transmitter 220 and the light receiver 221 is the distance measuring optical axis (in FIG. 7 , the distance measuring optical axis is attached to the flange axis 53 ).

The light transmitter 220 is used for emitting ranging light (ranging infrared rays) toward the target 51 . After hitting the target 51 , the ranging light is reflected to the beam splitter 240 , and reaches the light receiver 221 after passing through the beam splitter 240 . The ranging controller (such as a microcontroller or SoC) is set to perform triangulation based on the transmit-receive time difference of the ranging light, the light propagation speed, and the distance between the light transmitter 220 and the light receiver 221 to calculate the target distance ( That is, the depth value of the target 51).

Please refer to FIG. 3 , which is a structural diagram of a control device according to an embodiment of the present invention. Control device 20 may include modules 40-46. Modules 40-46 are respectively configured to generate different functions for carrying out the present invention.

The shooting control module 40 is used for controlling the image capturing device 21, such as controlling the shooting action, controlling the focusing action, acquiring the image data, executing the set image processing, and the like.

The ranging control module 41 is used to control the optical ranging device 22 , such as controlling the execution of measurement, acquisition of measurement data (target distance), and execution of measurement correction.

The arm control module 42 is used to control the posture of the robotic arm 23 by sending arm control commands to the robotic arm controller 31 , and can obtain the current position of the robotic arm 23 .

The calibration control module 43 is used to execute the calibration mode.

The work control module 44 is used to execute the work mode.

The conversion processing module 45 is used for calculating the coordinate conversion from the visual space to the mechanical space and the coordinate conversion from the mechanical space to the visual space.

The image analysis module 46 is used for performing image analysis and processing on the target image.

The aforementioned modules 40-46 are connected to each other (which may be electrical connection and information connection), and may be hardware modules (such as electronic circuit modules, integrated circuit modules, SoCs, etc.), software modules ( For example, firmware, operating system or application) or a mix of software and hardware modules, which is not limited.

Furthermore, when the aforementioned modules 40-46 are software modules (such as firmware, operating systems or applications), the storage device 25 may include a non-transitory computer-readable recording medium (not shown), the aforementioned non-transitory The transient computer-readable recording medium stores a computer program 250, and the computer program 250 records a computer-executable program code. After the control device 20 executes the aforesaid program code, the functions of the corresponding modules 40-46 can be implemented.

In one embodiment, the aforementioned modules 40 - 46 may be disposed in the control computer 30 . For example, the storage device 25 may include a storage of the control computer 30, and the aforementioned storage stores a computer program 250. The processor of the control computer 30 may execute the computer program 250 to implement the functions of the corresponding modules 40-46.

Please refer to FIG. 4 , which is a flowchart of a coordination method according to an embodiment of the present invention. The method for coordinating between the robotic arm and computer vision in this embodiment includes calibration steps S10-S12 and working modes S13-S16.

Step S10: The control computer 30 enters the calibration mode through the calibration control module 43 to perform coordination and calibration between the mechanical space and the visual space.

For example, the control computer 30 may enter the calibration mode when it accepts the user's start of the calibration operation or receives the calibration command.

Step S11 : the control computer 30 controls the robotic arm 23 to move, obtains the current target distance, and determines whether (the end) of the robotic arm 23 enters the effective shooting range of the image capture device 21 according to the current target distance.

If it enters the effective shooting range, the robotic arm 23 is controlled to pose a plurality of calibration poses in sequence in the valid shooting range, and at least one calibration image is captured when posing each calibration pose, so as to obtain a plurality of calibration poses corresponding to the plurality of calibration poses respectively. Correct the image.

Step S12 : The control computer 30 calculates the conversion relationship between the visual space of the image capturing device 21 and the mechanical space of the robot arm 23 based on the plurality of calibration postures and the plurality of calibration images through the conversion processing module 45 .

In one embodiment, the control computer 30 can identify the visual space coordinates of the calibration target in each calibration image, calculate the changes of the visual space coordinates of the calibration target in the plurality of calibration images, and calculate a plurality of corresponding calibration poses. The change of the mechanical space coordinate, and the conversion relationship between the visual space and the mechanical space is calculated based on the change of the above-mentioned mechanical space coordinate and the change of the visual space coordinate.

、機械空間

與轉換關係

之間的數學關係為：

In one embodiment, the visual space

, mechanical space

relation to conversion

The mathematical relationship between is:

可透過以下方式計算獲得。 In one embodiment, the conversion relationship

It can be calculated in the following way.

，同時獲得多個當下的校正姿態的表示式

(W為機械空間座標，如世界座標)，由於特徵物在機械空間座標下固定為

，彼此的關係式可以表示為

,i=1~N。 The image capture device 21 shoots the feature f (eg, a checkerboard) of the calibration target multiple times to obtain the relationship between the image capture device 21 and the feature under a plurality of different calibration postures:

, and simultaneously obtain expressions of multiple current corrected poses

(W is the mechanical space coordinate, such as the world coordinate), because the feature is fixed in the mechanical space coordinate as

, the relationship between each other can be expressed as

, i =1~ N .

,i=1~N皆為已知，可透過取得多筆資料來最佳化方程式，以得到誤差項最小之最佳解

，即校正資料越多，轉換關係

越精準。 because

, i = 1~ N are all known, and the equation can be optimized by obtaining multiple pieces of data to obtain the best solution with the smallest error term

, that is, the more correction data, the conversion relationship

more precise.

Step S13 : the control computer 30 enters the work mode through the work control module 44 to perform work.

For example, the control computer 30 can enter the work mode when it accepts the user's start-to-work operation or receives a work command.

Step S14 : the control computer 30 controls the robot arm 23 to move, obtains the current target distance, and determines whether (the end) of the robot arm 23 enters the effective shooting range of the image capture device 21 according to the current target distance.

If it enters the effective shooting range, the control computer 30 controls the image capture device 21 to shoot the work target to obtain the work image, and executes the work-related image analysis processing through the image analysis module 46, and determines the position to be processed in the work image ( visual space coordinates). Next, the control computer 30 converts the visual space coordinates into mechanical space coordinates by using the conversion relationship through the conversion processing module 45 . In addition, the control computer 30 controls the robot arm 23 to move to the mechanical space coordinates.

Step S14 : the control computer 30 controls the robot arm 23 to move to the mechanical space coordinates.

In one embodiment, the control computer 30 can further control the working device 54 to control the robotic arm to perform automated actions in mechanical space coordinates, such as gripping actions, welding actions, marking actions, grinding actions, assembling actions, gluing actions and/or locking actions. solid action.

The invention can correct the mechanical arm and computer vision, and can improve the hand-eye coordination of the robot.

Please refer to FIG. 4 and FIG. 5 at the same time. FIG. 5 is a partial flowchart of a coordination method according to an embodiment of the present invention. Compared with the coordination method of FIG. 4 , step S11 of the coordination method of this embodiment further includes steps S20 - S24 .

Step S20 : the control computer 30 obtains the effective shooting distance of the image capturing device 21 (the effective shooting distance 251 shown in FIG. 1 ), and sets the effective shooting range based on the effective shooting distance.

The aforementioned effective shooting distance may be, for example, the maximum or minimum focusing distance of the image capturing device 21 , and the effective shooting range may be, for example, the focusing range of the image capturing device 21 .

In one embodiment, if the effective shooting distance is 50 cm, the control computer 30 can set 0-50 cm as the effective shooting range, or set 25-50 cm as the effective shooting range, or set 25-75 cm as the effective shooting range. The effective shooting range is not limited.

Furthermore, when the image capturing device 21 and the shooting target fall within the aforementioned effective shooting distance or effective shooting range, the image capturing device 21 can correctly focus on the shooting target, and can capture a clear target image; When the capturing device 21 and the shooting target are not within the effective shooting range, the image capturing device 21 cannot focus correctly, and a blurred target image will be generated.

Step S21 : the control computer 30 controls the movement of the robotic arm 23 , and continues to measure the target distance until it is judged based on the target distance to enter the effective shooting range.

Step S22 : Controlling the computer 30 to continuously measure the current target distance, and controlling the robotic arm 23 to move within the effective shooting range to assume different focus measurement postures, and to shoot focus measurement images of each focus measurement posture.

In an embodiment, the aforementioned plurality of focus measurement postures are posed at different target distances, that is, the control computer 30 continuously changes the height of the robotic arm 23 within the effective shooting range (eg, from far away from the target to close to the target) to Obtain focus images at different heights.

Step S23 : The control computer 30 determines the reference posture and the reference distance by performing focus analysis on the plurality of focus images and the corresponding target distances through the image analysis module 46 .

In one embodiment, the aforementioned focus analysis includes selecting one or more focus-focused images (ie, clear images) from a plurality of focus-measurement images, and determining a reference pose ( Such as the center or center of gravity of these focus measurement poses), and the reference distance (eg, average value) is determined based on the target distance for capturing these focus measurement images.

In one embodiment, the aforementioned focus analysis can determine the clearest focus measurement image by analyzing edge features, gradient magnitudes, etc. of multiple images, and obtain the focus measurement posture and target distance that can obtain the clearest focus measurement image, And as the reference attitude and reference distance.

Step S24 : The control computer 30 controls the robot arm 23 to move to a calibration posture based on the reference posture and the reference distance, and shoots a corresponding calibration image in the calibration posture.

In one embodiment, the target distance of each calibration posture is equal to or close to the reference distance, and is changed based on the reference posture, such as rotating or displacing the end of the robotic arm 23 on the same height plane.

Step S25: The control computer 30 determines whether the preset stop collection condition is satisfied through the calibration control module 43, so as to determine whether the collected calibration data is sufficient, for example, the preset number of records, such as 10 records, 50 records, or 100 records is satisfied. etc., without limitation.

If the stop collection condition is satisfied, the collection of calibration data is ended; otherwise, step S24 is performed next to obtain calibration images captured under different calibration postures.

In this way, the present invention can continuously change the rotation and displacement of the robotic arm to assume different calibration postures, and shoot calibration images of each calibration posture until sufficient calibration data are collected.

In one embodiment, among the collected calibration poses, at least two calibration poses are located on a plane parallel to the jig 52 .

In one embodiment, among the collected calibration poses, at least two calibration poses are at different target distances, ie, different heights.

In the present invention, since the photographing optical axis and the flange axis are fitted, the calculated conversion relationship can be more accurate.

Please refer to FIG. 4 and FIG. 6 at the same time. FIG. 6 is a partial flowchart of a coordination method according to an embodiment of the present invention. Compared with the coordination method of FIG. 4 , step S14 of the coordination method of this embodiment further includes steps S30 - S33 .

Step S30: The control computer 30 controls the robotic arm 23 to move (such as continuously approaching the work target), continuously obtains the target distance, and determines whether the robotic arm 23 enters the effective shooting range based on the target distance (such as whether the target distance is less than the effective shooting distance 251).

Step S31 : After the robotic arm 23 (including the image capture device 21 ) enters the effective shooting range, the control computer 30 shoots the work target to obtain the work image.

Step S32 : the control computer 30 performs image analysis on the working image through the image analysis module 46 .

In one embodiment, the aforementioned image analysis may include identifying the work target in the work image, and performing the work analysis based on the position of the work target in the visual space to determine the coordinates of the visual space where the work needs to be performed.

In one embodiment, the aforementioned work analysis may be defect detection processing (eg, detecting component defects), measurement processing (eg, measuring component area or length), sorting and screening processing (eg, identifying and classifying components), and component positioning. Processing (e.g. to determine component grab points, assembly points, solder points, etc.).

Step S33 : The control computer 30 converts the visual space coordinates of the execution work to the mechanical space coordinates of the execution work based on the conversion relationship 252 .

In one embodiment, the control computer 30 may further compensate the mechanical space coordinates according to the position difference between the working device 54 and the flange shaft, so as to obtain the compensated mechanical space coordinates. In addition, the control computer 30 can generate arm control commands based on the compensated mechanical space coordinates, and send the arm control commands to the robot arm controller 31 to control the robot arm to move the working device 54 to the mechanical space coordinates where the work is performed.

In this way, the present invention can automatically perform processing operations through computer vision.

Please refer to FIG. 8 to FIG. 11 , FIG. 8 is a first schematic diagram of a calibration mode according to an embodiment of the present invention, FIG. 9 is a second schematic diagram of a calibration mode according to an embodiment of the present invention, and FIG. 10 is a first schematic diagram taken in FIG. 8 A schematic diagram of the corrected image, FIG. 11 is a schematic diagram of the second corrected image captured in FIG. 9 .

In this embodiment, the optical path structure only includes the beam splitter 240 , the visible light separated by the beam splitter 240 is directly incident on the image capture device 21 , and the lens of the image capture device 21 is oriented perpendicular to the flange axis.

In addition, the working device 54 is disposed at the bottom of the mounting base 55 and is located outside the flange axis, so as to avoid disturbing the incident light.

Furthermore, in the above arrangement, the distance between the working device 54 and the flange shaft is fixed, which enables the control device 20 to quickly and accurately calculate the current spatial position of the working device 54 from the flange shaft.

After the end of the robotic arm 23 moves within the effective shooting distance h1, it can adjust the joints 230-233 to assume the first correction posture P1 as shown in FIG. the first corrected image shown.

Next, the robotic arm 23 can assume a different second calibration posture P2 as shown in FIG. 9 by adjusting the joints 232 and 233 , and use the image capturing device 21 to capture the second calibration image as shown in FIG. 11 . .

As shown in FIG. 10 , in the first calibration posture P1 , the feature of the target 60 in the first calibration image (here, the center point) is a position 600 in the visual space.

As shown in FIG. 111 , after the transformation to the second calibration pose P2 , the feature of the target 60 in the second calibration image moves to the position 601 in the visual space.

Next, calculate the mechanical space coordinate change between the first corrected posture P1 and the second corrected posture P2, and calculate the visual space change V1 from position 600 to position 601, and correlate the two sets of changes to obtain the visual space and The conversion relationship between mechanical spaces is completed while the calibration is completed.

The above description is only a preferred specific example of the present invention, and therefore does not limit the scope of the present invention. Therefore, all equivalent changes made by using the content of the present invention are all included in the scope of the present invention. Chen Ming.

S10-S16: Coordination steps

Claims (20)

A method for coordinating between a robotic arm and computer vision, comprising: a) in a calibration mode, controlling a robotic arm in an effective shooting range of an image capture device based on a target distance measured by an optical ranging device The inner movement is a plurality of calibration postures, and a plurality of calibration images are respectively captured in the plurality of calibration postures through the image capturing device, wherein a beam splitter guides the visible light to the outer flange shaft disposed on the mechanical arm. an image capturing device, and guiding the ranging light to the optical ranging device, the ranging axis of the optical ranging device is parallel to or overlapping the flange axis; b) based on the plurality of calibration postures and the plurality of calibration images Calculate a conversion relationship between the visual space of the image capture device and the mechanical space of the robotic arm; c) in a working mode, capture a working image through the image capture device, and based on the working image and the conversion The relationship determines a mechanical space coordinate for performing work; and d) controls the robotic arm to move to the mechanical space coordinate. The method for coordinating between a robotic arm and computer vision as claimed in claim 1, wherein the step a) comprises: a1) obtaining an effective shooting distance of the image capture device, and setting the effective shooting range based on the effective shooting distance A2) control this mechanical arm to be respectively a plurality of focus measuring attitudes of different this target distance in this effective shooting range, and shoot a plurality of focus measuring images respectively at this multiple focus measuring attitudes; and a3) to this multiple focus measuring attitude A focus analysis is performed on the focus measurement image and the corresponding target distances to determine a reference attitude and a reference distance. The method for coordinating between a robotic arm and computer vision as claimed in claim 2, wherein the focus analysis comprises: e1) selecting at least one focus-focused image among the plurality of focus-measurement images; and e2) based on shooting The focus measurement attitude and the target distance to the clear focus measurement image determine the reference attitude and the reference distance. The method for coordinating between a robotic arm and computer vision as claimed in claim 1, wherein the step a) comprises: a4) continuously controlling the robotic arm to move into different calibration postures based on a reference posture and a reference distance, and A plurality of the corrected images are taken separately until a stop collection condition is satisfied. The method for coordinating between a robotic arm and computer vision as claimed in claim 1, wherein the step b) comprises: b1) identifying a visual space coordinate of a calibration target in each of the calibration images; b2) calculating the calibration target in Changes in the plurality of visual space coordinates of the plurality of corrected images; b3) calculating the changes in the plurality of mechanical space coordinates; and b4) calculating the changes in the plurality of visual space coordinates based on the changes in the plurality of mechanical space coordinates and the plurality of visual space coordinates This transformation relationship between space and mechanical space. The method for coordinating between a robotic arm and computer vision according to claim 1, wherein the step c) comprises: c1) shooting the working image when the robotic arm enters the effective shooting range; c2) identifying the working image in the working image a job target, and performing a job analysis based on the position of the job target to determine a visual space coordinate for performing the job; and c3) Convert the visual space coordinates of the execution work to the mechanical space coordinates of the execution work based on the conversion relationship. The method for coordinating between a robotic arm and computer vision as claimed in claim 6, wherein the work analysis includes at least one of defect detection processing, measurement processing, sorting and screening processing, and component positioning processing. The method for coordinating between a robotic arm and computer vision according to claim 1, further comprising: f) controlling the robotic arm to execute an automated action on the mechanical space coordinates. The method for coordinating between a robotic arm and computer vision as claimed in claim 8, wherein the automated action includes at least one of a gripping action, a welding action, a marking action, a grinding action, an assembling action, a gluing action and a locking action one. The method for coordinating between a robotic arm and computer vision as claimed in claim 1, wherein the beam splitter has a visible light reflectivity of over 80% and an infrared transmittance of over 75%. An automated robotic arm system, comprising: a robotic arm used to move in a three-dimensional space; an image capture device arranged outside the flange shaft of the robotic arm and used for capturing images; an optical ranging device arranged On the robotic arm, for measuring a target distance, the distance measuring axis of the optical distance measuring device is parallel to or overlapping the flange axis; an optical path structure includes a beam splitter, and the beam splitter is used to guide visible light to the image capturing device and guiding the ranging light to the optical ranging device; and a control device connecting the robotic arm, the image capturing device and the optical ranging device; Wherein, the control device is set to be in a calibration mode, based on the target distance, to control the robotic arm to move into a plurality of calibration postures within an effective shooting range of the image capture device, and to control the image capture device in the image capture device. A plurality of calibration postures are respectively captured a plurality of calibration images, and a conversion relationship between the visual space of the image capture device and the mechanical space of the robotic arm is calculated based on the plurality of calibration postures and the plurality of calibration images; wherein, the control The device is set to control the image capturing device to capture a working image in a working mode, determine a mechanical space coordinate for executing work based on the working image and the conversion relationship, and control the robotic arm to move to the mechanical space coordinate. The automated robotic arm system of claim 11, wherein the control device comprises: a robotic arm controller connected to the robotic arm for controlling the robotic arm to move based on an arm control command received; and a control a computer, connected to the robotic arm controller, the image capture device and the optical distance measuring device, to issue the arm control command; wherein the control computer is configured to obtain an effective shooting distance of the image capture device, and The effective shooting range is set based on the effective shooting distance, the robotic arm is controlled to take a plurality of focus measurement poses with different target distances within the effective shooting range, and a plurality of focus measurement images are respectively captured in the multiple focus measurement positions ; wherein, the control computer is configured to perform a focus analysis on the plurality of focus measurement images and the corresponding plurality of target distances to determine a reference attitude and a reference distance; Wherein, the control computer is set to continuously control the robot arm to move to different calibration postures based on the reference posture and the reference distance, and shoot a plurality of the calibration images until a stop collection condition is satisfied. The automated robotic arm system of claim 12, wherein the control computer is configured to select at least one of the focus-measured images of the plurality of focus-measured images, and based on the focus-measured image that captures a clear focus-measured image The focus attitude and the target distance determine the reference attitude and the reference distance. The automated robotic arm system as claimed in claim 11, further comprising a mounting base for setting the image capturing device, the optical ranging device and the optical path structure; wherein the end of the robotic arm is connected to the mounting base , and is used to move the mounting base in the three-dimensional space; wherein, the control device is set to identify a visual space coordinate of a calibration target in each calibration image, and calculate the calibration target in the plurality of calibration images. For the changes of the plurality of visual space coordinates, the changes of the plurality of mechanical space coordinates are calculated, and the conversion relationship between the visual space and the mechanical space is calculated based on the changes of the plurality of mechanical space coordinates and the changes of the plurality of visual space coordinates. The automated robotic arm system as claimed in claim 11, wherein the robotic arm includes a plurality of joints for providing a plurality of degrees of freedom, and the mechanical space coordinates are based on the mechanical space coordinates to adjust the rotation angles of the plurality of joints to control the The robotic arm moves in the multiple degrees of freedom; wherein, the control device is set to capture the working image when the robotic arm enters the effective shooting range, identify a working target in the working image, and based on the working target perform a job analysis to determine a visual space in which to perform the job Coordinates, and based on the conversion relationship, the visual space coordinates of the execution work are converted into the mechanical space coordinates of the execution work; wherein, the work analysis includes at least one of defect detection processing, measurement processing, classification screening processing and component positioning processing one. The automated robotic arm system of claim 11, further comprising a working device connected to the control device; wherein the control device is set to control the working device to execute an automation when the robotic arm moves to the mechanical space coordinate action. The automated robotic arm system of claim 16, wherein the working device includes at least one of a gripping end effector, a welding heater, a marking tool, a grinding tool, an assembling end effector, a gluing tool and a locking tool wherein, the automatic action includes at least one of a clamping action, a welding action, a marking action, a grinding action, an assembling action, a gluing action and a locking action. The automated robotic arm system of claim 11, wherein the image capturing device comprises a color camera; wherein the optical ranging device comprises an infrared rangefinder; wherein the beam splitter is a longpass dichroic mirror ), and has a visible light reflectivity of more than 80% and an infrared transmittance of more than 75%. The automated robotic arm system of claim 11, wherein the optical ranging device comprises: an optical transmitter for emitting the ranging light toward a target; an optical receiver for receiving the reflected ranging light; and A ranging controller, connecting the optical transmitter and the optical receiver, is set based on a transmit-receive time difference of the ranging light, a light propagation speed, and the distance between the optical transmitter and the optical receiver Calculate the target distance. The automated robotic arm system as claimed in claim 11, wherein the optical path structure further comprises a reflector for reflecting the visible light reflected by the beam splitter to the image capturing device.

Images