JP7312663B2 - Linear object working position determining method, robot control method, linear object fixing method, linear object working position determining device, and linear object gripping system - Google Patents

Description

The present invention relates to a three-dimensional measurement method for a linear object having a glossy portion at its tip, a method for determining the working position of a linear object using the method, a method for controlling a robot, a method for fixing a linear object, a device for determining the working position of a linear object, and a linear object gripping system.

A robot system is known in which a linear object is measured by a three-dimensional measuring device and a robot arm grips the linear object based on the three-dimensional shape. For example, Patent Literature 1 discloses a linear object gripping method for selecting and gripping one target linear object from among a plurality of linear objects. As a three-dimensional measuring device for such a robot system, a stereo camera is used, in which two cameras fixed at different positions capture an image of an object, and the three-dimensional shape of the object is determined based on the principle of triangulation from the relative positional relationship between the two cameras.
In a robot system using such a stereo camera, it is required to hold a cable with a metal terminal, for example, connect the metal terminal to a connector, or solder the metal terminal by contacting it with a land of a substrate.

However, robot systems in practical use are based on the premise that objects to be gripped have matte surfaces. The reason for this is that reflected light from a glossy surface such as metal has a strong component in the direction of regular reflection, so when an image is taken with a general camera, the amount of exposure becomes excessive, and there are cases where the glossy part of the image, such as metal, is blown out. Specifically, the amount of light incident on the pixel exceeds a value that can be processed as image data, and, for example, all RGB components are recognized as upper limits and recognized as white. When such overexposure occurs, the shape of the object becomes unclear in the image. Furthermore, since the stereo camera uses two cameras with different positional relationships, depending on the positional relationship of the cameras, the lighting, and the object to be imaged, even if the glossy part of the object appears in the image of one camera, the glossy part of the object may not appear in the image of the other camera because it is overexposed. In this case, the pixel information of the object in the images of the two cameras cannot be matched, and an accurate three-dimensional shape of the object cannot be obtained.
In the case of a cable with metal terminals, the shape of the glossy part and the boundary between the glossy part and the cable itself may be unclear depending on the positional relationship between the camera, the lighting, and the object to be imaged, or the three-dimensional shape may not be obtained.

As a method of specifying the three-dimensional shape of such an object with a glossy surface and utilizing it in a robot system, for example, the image processing method disclosed in Patent Document 2 is known.
Patent Document 2 discloses an image processing method for extracting a comparison area corresponding to the target area in comparison two-dimensional image data obtained by capturing an image of the object by shortening the exposure time at a predetermined rate, and determining whether the target area in the two-dimensional image is a specular reflection area based on the length of the shortened exposure time when the area in the comparison two-dimensional image data is equal to or less than the brightness threshold for the specular reflection area when a target area indicating a brightness threshold corresponding to the specular reflection area is detected in the two-dimensional image data obtained by imaging the target. A method for generating three-dimensional shape image data based on an image without a specular reflection area is also disclosed.

WO2019/098074 JP 2011-75311 A

However, the image processing method described in Patent Document 2 acquires an image with a short exposure time, so unless a special sensor with high sensitivity is used, the entire image will be dark and the contrast between the object and the background will be low. Moreover, even if a special sensor is used, the exposure time must be gradually reduced to take multiple images, and it must be determined whether there is a specular reflection area for each image taken. Furthermore, even if the exposure time is reduced and an image with a predetermined luminance or less is finally obtained, the light source may be reflected in the image, and further processing is required to distinguish such a virtual image from the shape of the object and identify the shape of the object. Although this is theoretically possible, it is not realistic when using a general stereo camera.
An object of the present invention is to provide a linear object three-dimensional measurement method for accurately measuring the three-dimensional shape of a linear object having a glossy portion at its tip, a linear object working position determining method using the same, a robot control method, a linear object fixing method, a linear object working position determining device, and a linear object gripping system.

A three-dimensional measurement method for a linear object according to the present invention is a method for three-dimensional measurement of a linear object having at least two portions with different reflectances, and is characterized by comprising the steps of capturing a backlight image of the linear object with a stereo camera, and obtaining the three-dimensional coordinates of the linear object based on the backlight image.
Here, "a linear object having at least two or more parts with different reflectances" means a linear object having parts of different materials mixed on the surface and having a part that reflects light more strongly than other parts. Such a linear material includes a linear material whose surface has been changed in a manufacturing process or the like. For example, wires with defects on the surface, such as those with white powder called talc attached in the manufacturing process of electric wires, and wires with broken coating and exposed metal wires in the manufacturing process of electric wires, are included. When the talc is deposited in this way, or when the coating breaks and the metal lines are exposed, that area will be more reflective than other areas.
Here, "backlight" refers to light that is not blocked by a linear object to be measured, passes through a space free of linear objects, and reaches the camera. A "backlight image" is a silhouette image (transmitted light image) of a linear object captured by a camera in the backlight. Further, the "three-dimensional coordinates" are those that can be processed on a computer, and may represent a three-dimensional positional relationship between at least two points on an object.
The linear object three-dimensional measurement method of the present invention is suitable for a linear object partially having a glossy portion. In particular, it is suitable for a linear object having a glossy part at the tip of a non-glossy part.

Preferably, the three-dimensional measurement method of the linear object of the present invention further comprises the steps of: capturing a front-light image of the linear object with the stereo camera; obtaining linear object part information specifying a part of the linear object with different reflectance in the front-light image; and identifying a part with different reflectance in the three-dimensional coordinates of the linear object based on the linear object part information in the front-light image.
Here, the "follow-light image" is an image obtained by irradiating a linear object with light from the same side as the stereo camera and capturing the light reflected by the linear object with the camera.
"Linear object site information specifying sites with different reflectances" refers to information specifying the position of at least one of the sites with different reflectances, and may specify the positions of all the sites. For example, there is linear object site information specifying a non-glossy part and/or a glossy part in a linear object. Specifically, in an image of a linear object, the position of the non-glossy part and/or the glossy part of the linear object is specified. In addition to the position information on the image, the position information includes three-dimensional coordinate information and the like.

A second aspect of the three-dimensional measurement method for a linear object of the present invention is a three-dimensional measurement method for a linear object having at least two portions with different reflectances, comprising the steps of: capturing a backlight image of the linear object with a stereo camera; capturing a front-light image of the linear object with the stereo camera; obtaining linear object portion information specifying a portion of the linear object with a different reflectance in the front-light image; and a step of acquiring three-dimensional coordinates of the linear object identifying a portion having a different reflectance of the linear object based on the linear object portion information in the backlight image.
The second aspect of the linear object three-dimensional measurement method of the present invention is suitable for a linear object having a glossy part in part, and is particularly suitable for a linear object having a glossy part at the tip of a non-glossy part.

In the first or second aspect of the linear object three-dimensional measurement method of the present invention, it is preferable to acquire the linear object part information in the front-light image based on the color of the front-light image.
In the first or second aspect of the three-dimensional measurement method of the linear object of the present invention, the step of capturing the follow-light images of the linear object with the stereo camera is a step of capturing a plurality of follow-light images with different illumination conditions, and preferably, the linear object part information in the follow-light images is obtained based on the plurality of follow-light images. Here, the term “direct-light images with different illumination conditions” refers to at least two images captured by switching forward-light irradiation members at different positions, or at least two images captured by switching LEDs to emit light using a single forward-light irradiation member in which a plurality of LEDs are arranged in series, in a plane, or in a ring, or at least two images generated by simultaneously irradiating illumination with different wavelengths of light and generating the obtained images by wavelength decomposition.

A third aspect of the three-dimensional measurement method for a linear object of the present invention is a three-dimensional measurement method for a linear object having at least two portions with different reflectances, and is characterized by comprising the steps of capturing images of the linear object with a stereo camera under a plurality of lighting conditions, and obtaining the three-dimensional coordinates of the linear object based on the images captured under the plurality of lighting conditions.
Here, the “plurality of lighting conditions” include switching between a backlight irradiating member and a forward light irradiating member, switching between a plurality of front light irradiating members at different positions, switching LEDs to emit light using one forward light irradiating member in which a plurality of LEDs are arranged in series, in a plane, or in a ring, or irradiating light of different wavelengths.
The third aspect of the linear object three-dimensional measurement method of the present invention is suitable for a linear object partially having a glossy portion. In particular, it is suitable for a linear object having a glossy part at the tip of a non-glossy part.
In the third aspect of the method for three-dimensional measurement of a linear object of the present invention, it is preferable to further include a step of acquiring linear object part information specifying a part having a different reflectance of the linear object based on at least one image among the images captured under the plurality of illumination conditions.

A method for determining the working position of a linear object according to the present invention comprises the steps of obtaining the three-dimensional coordinates of the linear object by the three-dimensional measuring method of the linear object, and calculating the working position of the linear object based on at least one point of the three-dimensional coordinates of the linear object. In this case, it is preferable to further include a working direction determination step of determining a working direction for the linear object based on the three-dimensional coordinates of the linear object.

A robot control method of the present invention is characterized by comprising a step of calculating a working position of the linear object by the linear object working position determination method of the invention, and a step of causing a robot hand to work with respect to the working position.

A method for fixing a linear object of the present invention is characterized by comprising the steps of: calculating the working position of the linear object by the method of determining the working position of the linear object of the present invention; holding the working position with a robot hand; moving the linear object held by the robot hand to bring the linear object into contact with a mating member; and fixing the linear object and the mating member by soldering.

A working position determination device for a linear object according to the present invention is characterized by comprising a stereo camera, a backlight irradiating member for irradiating light on a linear object having at least two or more portions with different reflectances so as to be backlit with respect to the stereo camera, and an arithmetic unit for obtaining the three-dimensional coordinates of the linear object based on the backlight image of the linear object captured by the stereo camera, and calculating the working position of the linear object based on at least one point of the three-dimensional coordinates of the linear object.
In the working position determination device of the present invention, the linear object is suitable for a linear object having a glossy portion in part. In particular, it is suitable for a linear object having a glossy part at the tip of a non-glossy part.

A second aspect of the apparatus for determining the working position of a linear object according to the present invention is characterized by comprising a stereo camera, an irradiating member that irradiates a linear object having at least two portions with different reflectances with light under different illumination conditions, and an arithmetic unit that acquires the three-dimensional coordinates of the linear object based on a plurality of images captured by the stereo camera under different illumination conditions of the linear object, and calculates the working position of the linear object based on at least one point of the three-dimensional coordinates of the linear object.
In the second aspect of the working position determination device of the present invention, the linear object is suitable for a linear object partially having a glossy portion. In particular, it is suitable for a linear object having a glossy part at the tip of a non-glossy part.

A linear object gripping system according to the present invention includes the linear object working position determination device according to the present invention, and a robot having a robot hand for gripping the linear object, wherein the robot hand grips the working position calculated by the computing unit.

In the three-dimensional measurement method of the linear object of the present invention, the three-dimensional shape of the linear object is acquired using the backlit image, so even if the linear object has an area that strongly reflects light, the area can be specified accurately. For example, in a cable with shiny metal terminals, the shape of the metal terminals can be specified accurately. In particular, it can be measured accurately even with a general-purpose stereo camera. Since the working position determination method for a linear object of the present invention uses the three-dimensional measuring method for a linear object of the present invention, it is possible to accurately calculate the working position. Since the robot control method of the present invention uses the work position determining method of the present invention, stable operation is possible. Since the linear object fixing method of the present invention uses the working position determination method of the present invention, the linear object can be reliably fixed. The working position determination device for a linear object of the present invention is suitable for calculating the working position of a linear object having at least two portions with different reflectances. INDUSTRIAL APPLICABILITY The linear object gripping system of the present invention is suitable for controlling a robot that works with a linear object having at least two portions with different reflectances.

1 is a schematic diagram showing a robot system for implementing a first embodiment of a robot control method of the present invention; FIG. FIG. 2a is a flow chart showing the first embodiment of the robot control method of the present invention, and FIG. 2b is a flow chart showing the first embodiment of the linear object three-dimensional measurement method of the present invention. FIG. 3a is a schematic diagram showing the gripping posture of the robot arm, and FIG. 3b is a schematic diagram showing the robot arm in a position before gripping. FIG. 4a is a schematic diagram showing a robot system for implementing the second embodiment of the robot control method of the present invention, and FIG. 4b is a schematic diagram showing the robot system for implementing the third embodiment of the robot control method of the present invention. FIG. 5a is a flowchart showing the second embodiment of the robot control method of the present invention, and FIG. 5b is a flowchart showing the second embodiment of the linear object three-dimensional measurement method of the present invention. FIG. 6a is a flow chart showing the third embodiment of the linear object three-dimensional measurement method of the present invention, and FIG. 6b is a flow chart showing the fourth embodiment of the linear object three-dimensional measurement method of the present invention. FIG. 7a is a flow diagram showing a fifth embodiment of the linear object three-dimensional measurement method of the present invention, and FIG. 7b is a schematic diagram showing a robot system for carrying out the fourth embodiment of the robot control method of the present invention. FIG. 4 is a schematic diagram showing a state in which a cable gripped by the robot control method of the present invention is fixed to a substrate;

FIG. 1 shows a robot system used in a first embodiment of a three-dimensional measurement method for a linear object having metal terminals and a first embodiment of a robot control method using the method.
The robot system 1 of FIG. 1a includes a robot 10 having a robot hand 12, a stereo camera 20, a control device 30, a backlight irradiation member 40, and a cable support base 50.
The cable (linear object) C gripped by the robot system 1 has flexibility. The cable C, as shown in FIG. 1b, includes a flexible cable body C1 and a metal terminal C2 provided at its end. The cable main body C1 consists of a metal wire and a synthetic resin coating covering it. The metal terminal C2 is fixed to the end of the cable body C1 by covering the end of the cable body C1 with a metal tube and crimping it. Reference character G denotes a portion to be gripped by the robot hand 12 as described later.

The robot 10 has an articulated arm 11 and a robot hand 12 provided at the tip thereof, and the robot hand 12 is a known one having a pair of fingers (gripping portions) 12a. The finger 12a is not particularly limited, and may be a finger that grips the cable (linear object) C in planar or linear contact, or a finger that grips it at a point.

The stereo camera 20 has a first camera 21 , a second camera 22 and a camera control section 23 . The first camera 21 captures a first color two-dimensional image, and the second camera 22 captures a second color two-dimensional image, which are fixed at different positions. The camera control unit 23 controls the first camera 21 and the second camera 22 , calculates the three-dimensional shape of the object from the two images, and communicates with the control device 30 . Specifically, it receives an imaging instruction from the control device 30, instructs the first camera 21 and the second camera 22 to take an image, calculates the three-dimensional shape of the cable C from the first image and the second image, and transmits the three-dimensional shape to the control device 30.

The control device 30 communicates with the stereo camera 20 and instructs the robot 10 based on the three-dimensional shape of the cable C obtained from the stereo camera 20 . It also has a computing unit 31 that performs various computations and a storage unit 32 that stores various data. Note that the control device 30 may be a control device independent of the robot 10 and the stereo camera 20 , or may be either the camera control unit 23 provided in the stereo camera 20 or the robot control device provided in the robot 10 .

The backlight irradiation member 40 is a flat-type illumination unit including a frame 41 with an open top, a rectangular diffusion plate 42 provided in the opening, and a light source (not shown) fixed in the frame. The backlight irradiation member 40 irradiates the rear side of the cable C with respect to the stereo camera 20 in a state where the cable C is sandwiched between itself and the stereo camera 20 .
That is, the stereo camera 20 transmits and images only the light at the position where the cable C does not exist. In other words, the first camera 21 and the second camera 22 of the stereo camera 20 capture a silhouette image (outline) of the cable C. FIG.
Examples of the light source of the backlight irradiation member 40 include ordinary lighting that emits light by itself, and a reflector that indirectly emits light. Further, the backlight irradiation member 40 is a surface light source, but may be a point light source.
The light emitted from the backlight irradiation member 40 is not particularly limited, but white light is preferable. However, if the cable C is covered with white, there may be a case where the silhouette image of the cable C cannot be picked up well. In that case, the light of the color used for the cable C other than the coating is used.

The cable support base 50 supports the cable C in a substantially horizontal direction, and also supports the part of the cable C to be measured slightly above the backlight irradiation member 40 in a floating state. A reference numeral 51 is a jig for holding the cable C. FIG. The height of the cable support base 50 is not particularly limited.

Next, a first embodiment of a robot control method for gripping a linear object will be described with reference to FIG. 2a.
This robot control method for gripping a cable C (linear object) includes a step 110 of measuring the three-dimensional shape of the cable C, a step 120 of specifying a gripped portion G of the cable C, a step 130 of calculating the gripping posture and gripping position of the robot hand, a step 140 of calculating the approach direction of the robot hand to the cable C, and a step 150 of gripping the cable C with the robot hand. This gripping method is a method that can reliably grip a desired portion of either the body C1 of the cable C or the metal terminal C2, especially when the length L (see FIG. 1b) of the metal terminal C2 is known. In this case, the length L of the metal terminal C2 is registered in the storage unit 32 in advance. In this control method, the order of processing of steps 120, 130, and 140 is not particularly limited.

Measurement of the three-dimensional shape of the cable C (step 110) is the first embodiment of the method for three-dimensional measurement of linear objects. As shown in FIG. 2b, step 110 is performed by taking a backlight image of cable C with stereo camera 20 (step 111), extracting a silhouette image of cable C from the backlight image (step 112), confirming the shape of the silhouette image (step 113), and calculating the three-dimensional shape of cable C (step 114).

The backlit image of the cable C is picked up (step 111) by illuminating the cable C with the backlight irradiating member 40 and photographing the cable C with the stereo camera 20 in that state. Specifically, the first camera 21 of the stereo camera 20 captures a first backlight image including a first silhouette image showing the contour of the cable C, and the second camera 22 of the stereo camera 20 captures a second backlight image including a second silhouette image showing the contour of the cable C.

Next, a silhouette image of the cable C is extracted from the backlight image (step 112). The backlit image consists of a silhouette image where the light is blocked by the cable C and a transmitted light image where the cable C does not exist. Therefore, a silhouette image can be extracted by separating the transmitted light image. That is, the first silhouette image is extracted from the first backlight image, and the second silhouette image is extracted from the second backlight image.
A method of extracting a silhouette image from each backlight image utilizes the difference in color between the silhouette image and the transmitted light image in the backlight image. Specifically, before the measurement, an image is captured in the absence of the cable C, the luminance (RGB values) of the color of the transmitted light image is measured, and the color range is determined based on the color of the transmitted light image. Then, the brightness (RGB value) of the color of each pixel of the backlight image is checked, and the pixels outside the predetermined color range of the transmitted light image are extracted. For example, in a 24-bit color in which the brightness of red, green, and blue (RGB) is each represented by 8 bits (256 levels), white is represented by 255, which is the maximum brightness of all RGB. Therefore, when the light from the backlight irradiation member is white light, pixels other than pixels in which all of RGB indicate 255 are extracted. As a result, a silhouette image remains in the backlight image. Note that the backlight image may be binarized by the color of the transmitted light image and other colors.
After extracting the silhouette image, it is preferable to perform noise processing for removing isolated pixels and the like. In the silhouette image, for example, it is preferable to obtain the center line by selecting the pixels positioned at the center of the cable width.

Next, the shape of the silhouette image is confirmed (step 113). When the stereo camera 20 captures an image of the cable C with backlighting, depending on the direction of the metal terminal C2, reflected light close to the brightness of the transmitted light may be seen from the camera, and the shape of the metal terminal C2 may become unclear.
Therefore, if there is a part that is difficult to confirm in the shape of either the first silhouette image or the second silhouette image, for example, if the silhouette image is not continuous, or if the center line is not recognized when the center line of the silhouette image is obtained, the measurement of the three-dimensional shape of the cable C is stopped. Then, for example, as shown in FIG. 1a, the posture of the cable C is changed, such as by rotating the cable C about its axis, and the process returns to the step 111 of capturing the backlit image of the cable C again.

When the shapes of the first silhouette image and the second silhouette image can be confirmed, the three-dimensional shape of the cable C is calculated based on the first silhouette image and the second silhouette image (step 114).
A three-dimensional shape representation method is not particularly limited as long as it can be processed on a computer and represents a three-dimensional positional relationship between at least two points on the cable. For example, it may be represented by a set of three-dimensional coordinates (so-called point cloud data), a polygon mesh, a plane/curved surface formula or parameter representation, or a volume data representation (voxels, etc.), or a combination thereof.
In the first silhouette image and the second silhouette image of each cable C in the first backlight image and the second backlight image, the cable body C1 and the metal terminal C2 are not specified. Therefore, in the three-dimensional shape of the cable C, the cable main body C1 and the metal terminal C2 are not specified.

Returning to Figure 2a, the identification of the gripped portion G of the cable C (step 120) will now be described.
The gripped portion G (work portion) of the cable C is, in this embodiment, the portion of the cable body C1, as shown in FIG. For example, the distance is X mm from the end of the cable body C1 (the boundary between the cable body C1 and the metal terminal C2). This distance X mm is registered in the storage unit 32 in advance. However, the gripped portion G of the cable C is not particularly limited, and can be appropriately determined according to the shape and properties of the linear object and subsequent processing. For example, when a cable with a male member of a metal connector attached to the end of the cable body is inserted into the female member of the connector, the insertion operation can be stably performed by holding the male member rather than holding the cable body.
The gripped portion G (work portion) of the cable C is calculated based on one point that can be specified in the three-dimensional shape of the cable C. FIG. The three-dimensional shape of the cable C calculated in step 110 is calculated based on the tip E of the cable C because the cable body C1 and the metal terminal C2 are not specified as described above, so the boundary between the cable body C1 and the metal terminal C2 cannot be specified. That is, a predetermined length obtained by adding the length L of the metal terminal C2 and the distance X from the tip E of the cable C can be defined as the portion G of the cable C to be gripped.
As a specific operation, for example, a three-dimensional image of the cable C is expressed on three-dimensional coordinates, and a point (see FIG. 1b) having a predetermined length (L + X mm) registered in advance from the tip E of the three-dimensional image is specified, thereby obtaining the gripped portion G of the cable C. By registering the distance from the end of the cable body C1 to the metal terminal side or the distance to the cable body side instead of the distance (L + X mm) from the tip of the metal terminal, even when gripping a linear object for which the length of the metal terminal C2 is not registered in advance in the storage unit, it is possible to grip the metal terminal or the linear object main body as desired.

Next, calculation of the gripping posture and gripping position of the robot hand will be described (step 130).
The gripping posture of the robot hand 12 refers to the posture of the robot hand 12 when the cable C is gripped by the fingers 12a of the robot hand 12, as shown in FIG. 3a. In other words, it is the working direction of the robot hand 12 with respect to the cable C. FIG. Specifically, it is the angle between the orientation F1 of the finger 12a of the robot hand 12 and the orientation D1 of the gripped portion G of the cable C, and the angle of the gripped portion G of the cable C of the finger 12a around the axis. Although the angle between the orientation F1 of the finger 12a of the robot hand 12 and the orientation D1 of the gripped portion G of the cable C is not particularly limited, it is preferable to make the angle approximately perpendicular to facilitate handling after gripping the cable C. The angle around the axis of the portion G of the finger 12a where the cable C is gripped is also not particularly limited, but when the cable C is supported horizontally, the direction F1 of the finger 12a with respect to the cable C is preferably 90 degrees so that the direction F1 is directed downward in the vertical direction.
The gripping position of the robot hand is the midpoint coordinates of the gripping points of the fingers 12a when the cable C is clamped by narrowing the width of the pair of fingers 12a. In this embodiment, the coordinates of the gripped portion G of the cable C calculated in step 120 are used as the gripping position. It is calculated so that the robot hand does not come into contact with other parts of the cable C and adjacent instruments.

The approach direction of the robot hand to the cable C is calculated (step 140). Here, the gripping front position of the robot hand is calculated, and the approach direction toward the cable C from the gripping front position is calculated.
The gripping position of the robot hand is the position where the fingers 12a of the robot hand 12 approach the cable C when gripping the cable C, as shown in FIG. 3B. The front position of the grip is not particularly limited, but is preferably a position away from the gripping posture at the gripping position of the cable C by a predetermined distance in the direction opposite to the direction F1 of the fingers 12a. In other words, the approach direction is preferably the direction F1 of the fingers 12a in the gripping posture. By specifying the grasping front position and the approach direction in this way, the fingers 12a of the robot hand can be brought closer to the grasped portion G of the cable C without the fingers 12a interfering with other parts even if the cable C is bent, and without interfering with adjacent instruments.

Grasping of the cable C by the robot hand (step 150) moves the robot hand 12 from the standby position to the gripping position via the gripping position, and grips the gripped portion G of the cable C at the gripping position.
The standby position of the robot hand 12 is not particularly limited, and may be determined in advance according to the angle of each link of the robot and registered in the control device, or may be determined based on the three-dimensional shape of the cable C, such as a position away from the cable C by a predetermined position in the opposite direction to the gripping direction (position in front of the grip).
The trajectory of the robot hand 12 from the waiting position to the gripping position is not particularly limited, and the robot hand 12 may be brought closer so as not to come into contact with the surroundings. For example, it is preferable to gradually direct the gripping position by linear interpolation, circular interpolation, joint interpolation, or the like. The trajectory from the gripped position to the gripped position is preferably linearly interpolated in the approach direction (orientation F1 of the fingers 12a) as described above.

In this way, the three-dimensional measuring method for a linear object obtains the three-dimensional shape of the cable C using a backlit image, so it is possible to accurately confirm the shape of the shiny metal terminal C2. Further, since the gripped portion G is calculated based on the three-dimensional tip of the cable C, the gripped portion G of the cable C can also be specified accurately. Therefore, it can be preferably used for automatic control by a robot arm.

FIG. 4a shows a robot system 1A used in a second embodiment of a three-dimensional measurement method for a linear object having metal terminals and a second embodiment of a robot control method using the method. The robot system 1</b>A differs from the robot system 1 in that it includes a front light irradiation member 60 . The robot 10, the stereo camera 20, the control device 30, the backlight irradiation member 40, and the cable support base 50 are substantially the same as those of the robot system 1 of FIG.
The forward light irradiation member 60 irradiates the front side of the cable C with light to the stereo camera 20 . That is, the stereo camera 20 captures the light reflected on the surface of the cable C. FIG. The forward light irradiation member 60 is not particularly limited, but for example, an illumination device with a general surface light source can be used.

Next, a second embodiment of a robot control method for gripping a linear object will be described with reference to FIG. 5a.
This robot control method for gripping a cable C (linear object) includes a step 210 of acquiring a three-dimensional shape of the cable C by specifying a cable body C1 and a metal terminal C2 of the cable C, a step 220 of calculating the coordinates of the gripped portion G of the cable C, a step 230 of calculating the gripping posture and gripping position of the robot hand, a step 240 of calculating the approach direction of the robot hand to the cable C, and a step 250 of gripping the cable C with the robot hand. Unlike the first embodiment, this control method can grip cables of any standard. Steps 230, 240 and 250 are substantially the same as steps 130, 140 and 150 of the first embodiment, respectively. It should be noted that in this embodiment, the steps 220, 230, and 240 may be performed in any order.

The measurement of the three-dimensional shape of the cable C (step 210) in which the cable body C1 and the metal terminal C2 of the cable C are specified is the second embodiment of the three-dimensional measurement method for linear objects. As shown in FIG. 5B, the step 210 is to take a backlit image of the cable C with the stereo camera 20 (step 211), extract a silhouette image of the cable C from the backlit image (step 212), check the shape of the silhouette image (step 213), then take a frontlit image of the cable C with the stereo camera 20 (step 214), acquire linear object part information specifying the cable main body C1 in the frontlit image (step 215), and obtain linear object part information specifying the cable main body C1 in the frontlit image (step 215). This is performed by acquiring linear object part information specifying the cable main body C1 and the metal terminal C2 of the cable C in the backlight image based on the linear object part information (step 216), and calculating the three-dimensional shape specifying the cable main body C1 and the metal terminal C2 of the cable C based on the linear object part information in the backlight image (step 217). That is, the three-dimensional shape of the cable main body and the three-dimensional shape of the metal terminal are calculated. Steps 211 to 213 are substantially the same as steps 111 to 113 of the first embodiment, respectively. If the information required in the subsequent process is the three-dimensional shape of only one of the cable main body and the metal terminal, it is sufficient to calculate the three-dimensional shape of only the necessary portion, which speeds up the calculation processing speed.

Here, the steps after step 214 will be described in detail. Imaging of the front light image of the cable C (step 214) is performed by illuminating the cable C with the front light irradiation member 60 and photographing the cable C with the stereo camera 20 in that state. Specifically, the first camera 21 of the stereo camera 20 captures a first front-light image, and the second camera 22 of the stereo camera 20 captures a second front-light image. The backlit image in step S211 and the frontlit image in step S214 are captured by the first camera 21 and the second camera 22 of the stereo camera 20 fixed at the same position.

Next, linear object part information specifying the cable main body C1 in the front-light image is obtained (step 215). Specifically, the linear object part information is acquired by extracting the cable main body image from the front-light image.
The front-light image consists of a cable body image, a metal terminal image, and a background image. The cable body image and the metal terminal image are images of light reflected on the surface of the cable C, respectively. As a result, whiteout due to reflected light having a predetermined luminance or more is seen in the metal terminal image. On the other hand, no such overexposure is seen in the image of the main body of the cable. Therefore, the cable body image from which a clear shape can be obtained from the front-light image becomes the linear object part information specifying the cable body C1 in the front-light image. Here, whiteout refers to a state in which the amount of light incident on the image pickup device exceeds the upper limit value. For example, in addition to a state in which all of RGB are recognized as the upper limit and color densities cannot be represented and white areas are formed on the image, it also includes a state in which any one of RGB is recognized as the upper limit and color densities cannot be represented.
The cable body image extraction method utilizes the color of the front light image. Specifically, before the measurement, the luminance (RGB value) of the color of the cable main body C1 is registered in advance in the storage unit 32 as a color table. If two or more types of cables are targeted, register them all. For example, an RGB range that includes the required colors may be set and registered. Then, in the front-light image, only the corresponding specific color area is specified as the cable main body C1, and extracted as the cable main body image. Specifically, the brightness (RGB value) of each pixel in the frontlight image is compared with the brightness (RGB value) of the color in the color table, and if the color of the pixel is the same as the color in the color table, it is retained, and if it is different, it is deleted. Whether the colors are the same or different is determined by the difference between the two. For example, if the difference between the RGB value of each pixel and the RGB value of the color table is within a predetermined range, it is determined that they are the same, and if the difference is greater than or equal to the predetermined range, it is determined that they are different. After extracting the cable body image, noise is preferably removed. It should be noted that the image of the main body of the cable may be extracted by performing binarization processing.

Next, based on the linear object part information (cable body image) that identifies the cable body in the front-light image, the linear object part information that identifies the cable body C1 and the metal terminal C2 of the cable C in the backlit image is acquired (step 216).
As described above, since the backlit image and the direct light image are captured by the stereo camera 20 at the same position, the position and shape of the cable C in the first direct light image and the first backlit image, and the position and shape of the cable C in the second direct light image and the second backlit image match. Therefore, the silhouette image extracted from the backlit image and the cable main body image extracted from the frontlit image are compared, the cable main body C1 and the metal terminal C2 in the silhouette image are specified, and the linear object part information in the backlit image is acquired.
As a specific operation, the first silhouette image and the first cable body image are superimposed, and in the first silhouette image, the area overlapping the first cable body image can be determined to be the cable body C1, and the area not overlapping the first cable body image can be determined to be the metal terminal C2. Similarly, the second silhouette image and the second cable body image are superimposed to identify the cable body C1 and the metal terminal C2 in the second silhouette image.

Based on the linear object part information in the backlight image, the three-dimensional shape of the cable C is calculated by specifying the cable body C1 and the metal terminal C2 of the cable C (step 217).
Specifically, the three-dimensional shape of the cable C specifying the cable main body C1 and the metal terminal C2 is calculated based on the first silhouette image provided with the linear object part information specifying the cable main body C1 and the metal terminal C2 and the second silhouette image provided with the linear object part information specifying the cable main body C1 and the metal terminal C2.

Next, the calculation of the gripped portion G of the cable C will be described (step 220). Here, as in step 120, the calculation is based on one point that can be specified on the three-dimensional shape of the cable C. Specifically, the boundary between the cable body C1 and the metal terminal C2 (the end of the cable body C1) is found from the three-dimensional shape of the cable C in which the cable body C1 and the metal terminal C2 are specified, and the gripped portion G, which is X mm from this boundary, is directly calculated. Note that the distance X mm from the boundary is registered in advance.

By acquiring the identified three-dimensional shape of the cable C in which the cable main body C1 and the metal terminal C2 are identified using the front-light image in this way, cables with different standards such as cables with different colors and cables with different shapes of the metal terminals C2 can be handled using the same system. Then, these cables of different standards can be stably gripped by the robot arm.
In the second embodiment, steps 211 to 217 are described in order, but steps 214 to 217 (process of front-light image) may be performed prior to steps 211 to 215 (process of backlight image).

Next, a third embodiment of a three-dimensional measurement method for linear objects having metal terminals will be described with reference to FIG. 6a. This method, which uses the robot system 1A of FIG. 4, obtains a three-dimensional shape specifying the cable body C1 and the metal terminal C2 of the cable C in the same way as the step 210 of the second embodiment of the robot control method, and is a method that replaces the step 210 in the second embodiment of the robot control method.
Acquisition of the three-dimensional shape of the cable C specifying the cable main body C1 and the metal terminal C2 of the cable C (step 310) involves, as shown in FIG. A follow-light image of C is captured (step 315), linear object part information specifying the cable main body C1 in the follow-light image is obtained (step 316), and the cable main body C1 and the metal terminal C2 in the three-dimensional shape of the cable C are specified (step 317) based on the linear object part information specifying the cable main body C1 in the follow-light image.
Steps 311 to 314 are substantially the same as steps 111 to 114 of the first embodiment, respectively, and steps 315 and 316 are substantially the same as steps 214 and 215 of the second embodiment, respectively.

Here, details of the steps after step 317 will be described. The cable main body C1 and the metal terminal C2 in the three-dimensional shape of the cable C are specified based on the linear object part information in the front-light image (step 317).
Since the three-dimensional shape of the cable C calculated in step 314 is calculated based on the silhouette image of the backlight image, the cable main body C1 and the metal terminal C2 are not specified. However, since the stereo camera 20 captures each front light image and back light image from the same position, each point of the first cable main body image and second cable main body image extracted from each front light image and each point of the three-dimensional shape of the cable C correspond to each point. Therefore, the three-dimensional area of the cable C corresponding to each point of the first cable body image and the second cable body image is determined to be the cable body C1. Then, the remaining area is determined to be the metal terminal C2. As a result, it is possible to obtain a three-dimensional shape specifying the cable body C1 and the metal terminal C2 of the cable C similar to step 210 of the second embodiment.
The three-dimensional shape of the cable body C1 may be obtained from the first cable body image and the second cable body image, the three-dimensional shape of the cable C and the three-dimensional shape of the cable body C1 may be superimposed, and the overlapping portion may be determined to be the cable body C1 and the other region to be the metal terminal C2, thereby obtaining the three-dimensional shape of the cable C specifying the cable body C1 and the metal terminal C2.

Next, a fourth embodiment of a three-dimensional measurement method for linear objects having metal terminals will be described with reference to FIG. 6b. This method also determines an identified three-dimensional shape and can be used in place of step 210 of the second embodiment of the robot control method. However, it differs from the conventional one in that it uses the linear object part information specifying the glossy part in the front light image. Also, the robot system 1B of FIG. 4b is used. The robot system 1B includes a plurality of front light irradiation members 60a, 60b, and 60c, and is otherwise substantially the same as the robot system 1A.
Acquisition of the specified three-dimensional shape of the cable C by specifying the cable main body C1 and the metal terminal C2 of the cable C (step 410), as shown in FIG. The linear object portion information specifying the metal terminal C2 is obtained from a plurality of front-light images (step 415), the linear object portion information specifying the cable main body C1 and the metal terminal C2 of the cable C in the backlight image is obtained based on the linear object portion information specifying the metal terminal in the front-light image (step 416), and the three-dimensional shape specifying the cable main body C1 and the metal terminal C2 of the cable C is calculated based on the linear object portion information in the backlight image (step 417). .
Steps 411 to 413 are substantially the same as steps 111 to 113 of the first embodiment, respectively, and step 417 is substantially the same as step 217 of the second embodiment.

Details of the steps after step 414 will be described.
In capturing the front-light images of the cable C under different lighting conditions (step 414), a plurality of front-light images captured under illumination from the front-light irradiation members 60 arranged at different positions are captured. That is, the stereo camera 20 captures a front light image in a state in which the front light irradiation member 60a irradiates, the stereo camera 20 captures the front light image in a state in which the light source is switched and the front light irradiation member 60b irradiates, and the stereo camera 20 captures the front light image in a state in which the light source is switched and the front light irradiation member 60c irradiates.

Next, linear object part information identifying the metal terminal C2 in the front-light image is acquired from the plurality of front-light images (step 415).
Since the plurality of front-light irradiation members are arranged at different positions, the whiteout positions appearing on the surface of the metal terminal image of each front-light image are different. Therefore, the shape of the metal terminal can be confirmed by extracting the portion of the metal terminal image that is not overexposed from a plurality of front-light images. That is, the linear object part information specifying the metal terminal C2 of the cable C can be obtained. The metal terminal C2 in each front-light image is identified by registering the luminance (RGB value) of the color of the metal terminal C2 as a color table and comparing the luminance (RGB value) of the color of each pixel with the luminance (RGB value) of the color in the color table before measurement, as in step 215. Determination of blown-out highlights in each image is made based on whether the luminance (RGB value) of the color is equal to or greater than a threshold value. Although three front light irradiation members 60a, 60b, and 60c are provided here, the metal terminal image can be specified more accurately by increasing the number of front light irradiation members.

Next, linear object part information specifying the cable body C1 and the metal terminal C2 of the cable C in the backlight image is acquired based on the linear object part information specifying the metal terminal in the front-light image (step 416). In step 415, this step identifies the cable body C1 and the metal terminal C2 in the silhouette image by comparing the linear object part information identifying the metal terminal C2 of the cable C with the silhouette image of the backlight image.
As a specific operation, the cable body C1 and the metal terminal C2 in the silhouette image are specified by determining each position of the linear object part information and the position of the corresponding silhouette image as the metal terminal C2, and determining the other as the cable body C1.

As a modification of the fourth embodiment of the method for three-dimensionally measuring a linear object, the three-dimensional shape of the cable C may be calculated based on the silhouette image of the backlight image, and the three-dimensional cable main body C1 and the metal terminal C2 of the cable C may be specified based on the linear object part information specifying the metal terminal C2 in the front-light image.
As a modification of the fourth embodiment of the method for three-dimensionally measuring a linear object, instead of switching between the plurality of front-light irradiation members 60a, 60b, and 60c arranged at different positions for imaging, one front-light irradiation member in which a plurality of LEDs are arranged in series, in a plane, or in a ring may be used, and the LEDs to emit light may be switched for imaging. Furthermore, the light from the plurality of front light irradiation members 60a, 60b, and 60c may be set to different wavelengths, and images may be captured in a state of simultaneous irradiation, and a plurality of front light images may be obtained by wavelength-decomposing the obtained image.

Next, a fifth embodiment of a three-dimensional measurement method for linear objects having metal terminals will be described with reference to FIG. 7a. This method also obtains a three-dimensional shape specifying the cable body C1 and metal terminals C2 of the cable C, and can be used in place of step 210 of the second embodiment of the robot control method. However, this method is different in that it does not use backlight images. Then, the robot system 1C of FIG. 7b is used. The robot system 1C of FIG. 7b is substantially the same as the robot system 1B except that the backlighting member 40 is not provided.
Acquisition of the three-dimensional shape of the cable C by specifying the cable body C1 and the metal terminal C2 of the cable C (step 510) involves, as shown in FIG. 7a, capturing front-light images of the cable C under different lighting conditions with the stereo camera 20 (step 511), acquiring linear object part information specifying the cable main body C1 and the metal terminal C2 in the front-light images from a plurality of front-light images (step 512), and based on the linear object part information of the cable body C1 of the cable C in the front-light images. and calculating a three-dimensional shape specifying the metal terminal C2 (step 513). Here, step 511 is substantially the same as step 414 of the third embodiment, respectively.

Next, the process from step 512 will be described.
Linear object site information identifying both the cable body C1 and the metal terminal C2 in the front-light image is acquired from the plurality of front-light images (step 512).
As described in step 415, linear object site information identifying the metal terminal C2 of the cable C can be obtained from a plurality of front-light images. At the same time, linear object site information (cable body image) that identifies the cable body C1 of the cable C can be obtained from at least one front-light image, as described in step 215 . Here, linear object part information specifying the cable main body C1 and the metal terminal C2, which is a combination of both pieces of information, is acquired. For example, based on linear object part information specifying both, a complementary frontlight image excluding blown-out highlights may be created.

Calculation of the three-dimensional shape specifying the cable main body C1 and the metal terminal C2 of the cable C based on the linear object part information specifying the cable main body C1 and the metal terminal C2 in the front-light image (step 513) will be described.
That is, it is calculated based on the linear object part information specifying the cable main body C1 and the metal terminal C2 in the first front-light image and the linear object part information specifying the cable main body C1 and the metal terminal C2 in the second front-light image. For example, the three-dimensional shape of the cable C may be calculated based on the complementary first front-light image and the complementary second front-light image.

In this case, a three-dimensional shape specifying the cable body C1 and the metal terminal C2 can be calculated without using the backlight image.

Step 512 of obtaining linear object part information specifying the cable main body C1 and metal terminal C2 of the cable C can be substituted for step 415 as a modification of the fourth embodiment of the linear object three-dimensional measurement method.

FIG. 8 shows a state in which the cable C is grasped by the robot hand 12 by one of the robot control methods described above, the cable C is moved while being grasped, the metal terminal C2 of the cable C is brought into contact with the land L (connection portion) of the substrate P, the molten solder S is discharged from the solder discharge head H, and the metal terminal C2 and the land L (connection portion) are fixed by soldering.
By using the robot control method of the present invention, the cable C and the substrate P can be fixed using the robot 10 by aligning the metal terminals C2 of the cable C with the lands L provided on the substrate P at short intervals.
A robot hand is known that also serves as a solder ejection head that ejects melted solder to a gripped portion. This robot hand may be used in any of the linear object gripping methods described above. When using this robot hand, the portion of the cable C to be gripped is the metal terminal C2. Then, in a state in which the robot hand directly grips the metal terminal C2 of the cable C, the cable C is moved so that the metal terminal C2 contacts the land of the substrate P, the molten solder is discharged from the discharge hole, and the metal terminal C2 and the land are fixed.

In the present embodiment, an image (for example, silhouette image, cable body image) is extracted from each image based on the luminance (RGB value) of the color of each image, but the image may be extracted based on any of the hue, saturation, and brightness (HSV) of each image.
In this embodiment, a cable is used as a linear object, but it is not particularly limited. Examples include electric wires, wire harnesses, tubes, cords, threads, fibers, glass fibers, and optical fibers.
In this embodiment, the glossy portion at the tip is a metal terminal, but is not particularly limited. For example, a strip portion coated with an electric wire can be mentioned.
In the present embodiment, the three-dimensional shape of one cable (linear object) is measured and gripped by the robot arm, but one cable from a plurality of cables or one wire from a wire harness may be gripped. For example, as shown in Patent Document 1, a line-shaped object of interest may be determined and gripped.
In addition, it is preferable to display the recognition result on a display device such as a display so that the supervisor can monitor whether the recognition is performed correctly. In this case, it is preferable to display a silhouette image or a three-dimensional image specifying the cable main body C1 and the metal terminal C2, and mark these areas in a different color or the like so that these areas can be visually recognized.

Reference Signs List 1, 1A, 1B, 1C robot system 10 robot 11 articulated arm 12 robot hand 12a finger 20 stereo camera 21 first camera 22 second camera 23 camera control section 30 control device 31 calculation section 32 storage section 40 backlight irradiation member 41 frame 42 diffusion plate 50 cable support base 51 jig 60, 60a, 60 b, 60c forward light irradiation member C cable C1 cable main body C2 metal terminal E tip G part to be gripped H solder discharge head L substrate land P substrate S solder

Claims (9)

A method for determining a working position of a linear object having flexibility and having a glossy metal terminal at the tip thereof to be gripped by a robot hand, comprising :
a step of capturing a backlight image of the linear object with a stereo camera having two cameras ;
obtaining the three-dimensional coordinates of the linear object based on the backlight image ;
a step of capturing a front-light image of the linear object with the stereo camera;
a step of acquiring linear object part information specifying the main body of the linear object and the metal terminal in the front-light image;
a step of specifying the main body and the metal terminals in the three-dimensional coordinates of the linear object based on the linear object part information in the front-light image;
calculating the working position of the linear object based on at least one point of the three-dimensional coordinates of the linear object;
A work position determination method for a linear object . A method for determining a working position of a linear object having flexibility and having a glossy metal terminal at the tip thereof to be gripped by a robot hand, comprising :
a step of capturing a backlight image of the linear object with a stereo camera having two cameras ;
a step of capturing a front-light image of the linear object with the stereo camera;
a step of acquiring linear object part information specifying the main body of the linear object and the metal terminal in the front-light image;
a step of acquiring linear object part information specifying the main body and the metal terminals of the linear object in the backlight image based on the linear object part information in the front-light image;
a step of acquiring three-dimensional coordinates of the linear object specifying the main body and the metal terminals of the linear object based on the linear object part information in the backlight image ;
calculating the working position of the linear object based on at least one point of the three-dimensional coordinates of the linear object;
A work position determination method for a linear object . Acquiring linear object part information in the front-light image based on the color of the front-light image;
3. The working position determination method for a linear object according to claim 1 or 2 . further comprising a working direction determination step of determining a working direction for the linear object based on the three-dimensional coordinates of the linear object;
A working position determination method for a linear object according to any one of claims 1 to 3 . A step of calculating the working position of the linear object by the method for determining the working position of the linear object according to claim 4 ;
and causing the robot hand to grasp the working position .
How to control the robot. A step of calculating the working position of the linear object by the method for determining the working position of the linear object according to claim 4 ;
a step of causing the robot hand to grip the working position, moving the linear object gripped by the robot hand to bring the linear object into contact with a mating member, and fixing the linear object and the mating member by soldering;
A method for fixing linear objects. a stereo camera with two cameras ,
a backlight irradiating member that irradiates a linear object having flexibility and having a glossy metal terminal at its tip so as to be backlight for the stereo camera;
a front light irradiation member that irradiates the linear object with light so as to be front light with respect to the stereo camera;
Acquiring the three-dimensional coordinates of the linear object based on the backlight image of the linear object captured by the stereo camera;
Acquiring linear object part information specifying the main body of the linear object and the metal terminal in the front-light image of the linear object captured by the stereo camera;
identifying the main body and the metal terminals in the three-dimensional coordinates of the linear object based on the linear object part information in the front-light image;
a calculation unit that calculates a working position of the linear object to be gripped by the robot hand based on at least one point of the three-dimensional coordinates of the linear object;
Work positioning device for linear objects. a stereo camera with two cameras,
a backlight irradiating member that irradiates a linear object having flexibility and having a glossy metal terminal at its tip so as to be backlight for the stereo camera;
a front light irradiation member that irradiates the linear object with light so as to be front light with respect to the stereo camera;
Acquiring linear object part information specifying the main body of the linear object and the metal terminal in the front-light image of the linear object captured by the stereo camera;
Acquiring linear object part information specifying the main body and the metal terminals of the linear object in the backlight image of the linear object captured by the stereo camera, based on the linear object part information in the front-light image;
Acquiring three-dimensional coordinates of the linear object specifying the main body and the metal terminal of the linear object based on the linear object part information in the backlight image;
a calculation unit that calculates a working position of the linear object to be gripped by the robot hand based on at least one point of the three-dimensional coordinates of the linear object;
Work positioning device for linear objects. A working position determination device for a linear object according to claim 7 or 8 ;
a robot comprising the robot hand that grips the linear object,
the robot hand grips the working position calculated by the computing unit;
Linear object grasping system.

Images