DE102016200386B4 - Method for controlling a manipulator system - Google Patents

Abstract

Method for controlling a manipulator system (100) which comprises at least one manipulator (110) which carries an end effector (120) which is set up for setting components (150), in particular for setting plugs, and wherein the manipulator system (100 ) further comprising at least one light source (131) and a digital camera (135a, 135b), the method comprising the steps of: projecting at least one line of light (132) by means of the light source (131) onto a set component (150) a light line (132) intersects the set component (150); capturing an image (435a-c, 535a-e) by means of the digital camera (135a, 135b) showing the projected light line (432, 532) and determining the set quality of the set one Component (150) by means of the image (435a-c, 535a-e), and defining a first region (436a-c, 536) of the image (435a-c, 535a-e) which exposes the set component (451a-c, 551 ) and a second area (438a-c , 538) of the image (435a-c, 535a-e) which is not intended to show a set component; defining image lines which are parallel to the projected light line (432, 532) and calculating the gray value average, the gray value average for each of the first and calculating the second range (536, 538) separately for each image line to monitor the set quality in setting the component.

Description

The invention relates to a method for controlling a manipulator system, which is set up for setting components and in particular for setting plugs, as well as a manipulator system for setting components.

In many assembly processes, for example in the automotive industry, it is desirable to automatically set components, such as rivets, screws, bolts, seals, plugs and / or the like, by means of manipulators. In order to be attached to a workpiece, for example, the components must typically be introduced into an opening provided for this purpose and set and fixed in the latter. The placement of a component requires an exact positioning of the setting tool relative to the opening into which the component is to be placed.

To set components in automated assembly processes manipulators are used, which lead an end effector. Manipulators are devices that enable physical interaction with the environment. Such manipulators are, for example, industrial robots which have at least three freely programmable axes of movement. Such industrial robots are arranged mobile or stationary. The end effector is the last link in the kinematic chain and, for example, a tool. Such a tool can be set up, for example, for setting a component.

To comply with the necessary high position accuracy when setting a component, known industrial robots and / or end effectors require additional sensors, by means of which the fine positioning of the industrial robot and / or end effector can be performed. The additional sensors are required because typical industrial robots have a long kinematic chain of at least three kinematic links (axes of motion), each having an unavoidable positional tolerance. The positional tolerances of the individual kinematic members add up, so that the positional accuracy at the end effector may ultimately be insufficient.

Furthermore, in the assembly processes, for example in the automotive industry, the alignment of a workpiece (eg an automobile body), in which the components, such as rivets, screws, bolts, gaskets, plugs and / or the like to be set automatically by means of manipulators, often inaccurate , For correct placement of these components in the workpiece, the position and / or position of the workpiece relative to the manipulator and in particular relative to the end effector of the manipulator must be known.

Therefore, additional sensors are currently used, such as tactile sensors, which is provided at the end effector. Alternatively, industrial robots can be used, which have internal force sensors, such as the LBR-iiwa KUKA AG. However, the use of tactile and / or force sensors requires the approaching and touching of openings and / or reference structures on which the component is to be placed. The start-up requires a relatively long time, resulting in long cycle times.

Moreover, automated assembly lines often require monitoring of the set quality of the set component. The setting quality is typically determined by the position and position of a set component relative to the workpiece. Methods are known in the prior art which scan a surface of a workpiece on which a component has been placed by means of a point cloud or guide a line for scanning over the surface. Such methods are basically suitable for quality control in the placement of components, but require a high computational intensity, resulting in long cycle times Moreover, already scanning the surface requires long cycle times. This is particularly undesirable in automated assembly lines. In addition, in addition to the sensors, which is necessary for positioning the end effector additional sensors must be provided, creating a lot of space for the additional sensors.

Alternative concepts use the tactile or force sensors, which is used for positioning, to close from the force curves when setting the component on the setting quality. However, these methods are susceptible to errors and tend to misjudge the setting quality.

An exemplary application for the placement of components is the placement of plugs. Plugs are placed, for example, in automotive bodies to seal openings that are necessary due to previous assembly steps. For example, such openings can be used to set a spot weld in the body. The plugs for sealing such openings are typically made of plastic. In order to achieve a correct seal, a plug with high setting quality must be set, i. a desired position and position of the plug relative to the body must be achieved. In addition, it must be ensured that a stopper was set at all and the stopper was not damaged when setting. This also applies analogously to other components to be set, as described above.

From the publication EP 2 159 538 A1 is a method for optical inspection of Assembly steps, in particular for location and shape determination of objects in a surveillance area known.

The DE 20 2015 105 541 U1 shows a robot measuring cell, which is equipped with a device for detecting object contours.

The object of the present invention is to at least partially overcome the disadvantages described above and to provide a corresponding method for controlling a manipulator system and a corresponding manipulator system.

• • Projizieren zumindest einer Lichtlinie mittels der Lichtquelle auf ein gesetztes Bauteil, wobei die zumindest eine Lichtlinie das gesetzte Bauteil schneidet;
• • Aufnehmen eines Bildes mittels der Digitalkamera, welches die projizierte Lichtlinie zeigt, und
• • Bestimmen der Setzqualität des gesetzten Bauteils mittels des Bildes, und
• • Definieren eines ersten Bereichs des Bildes, welcher das gesetzte Bauteil zeigen soll, und eines zweiten Bereichs des Bildes, der kein gesetztes Bauteil zeigen soll;
• • Definieren von Bildzeilen, welche parallel zu der projizierten verlaufen, und
• • Berechnen des Grauwertdurchschnitts, wobei der Grauwertdurchschnitt jeweils für den ersten und den zweiten Bereich (536, 538) separat für jede Bildzeile berechnet wird, um die Setzqualität beim Setzen des Bauteils zu überwachen.

The object is achieved by a method according to claim 1 and a manipulator system according to claim 12. In particular, the object is achieved by a method for controlling a manipulator system, which manipulator system comprises at least one manipulator, which performs an end effector, which is set for setting components, in particular for setting plugs, and wherein the manipulator system further comprises at least one light source and a Digital camera, the method comprising the steps of:

• Projecting at least one light line by means of the light source onto a set component, wherein the at least one light line intersects the set component;
• • Taking a picture using the digital camera showing the projected line of light, and
• • Determining the setting quality of the set component by means of the image, and
• Defining a first area of the image which is to show the set component and a second area of the image which should not show a set component;
• • Defining image lines that run parallel to the projected image, and
• Calculating the gray scale average, the gray scale average for each of the first and second regions ( 536 . 538 ) is calculated separately for each image line to monitor the setting quality when setting the component.

The light source is, for example, a laser light source which generates at least one laser light line. Alternatively, the light source may generate the line of light, for example, by suitable lenses, apertures, screens, and / or the like.

The digital camera is preferably a digital camera that captures two-dimensional images, preferably in the form of raster graphics. Such digital cameras are inexpensive to procure and may have very small dimensions, allowing the digital camera to be attached to the end effector, preferably without restricting the working area of the end effector. The digital camera takes an image according to the method showing the projected light line. Besides the light line, a part of the environment (i.e., a part of the workpiece into which the component has been set) is also picked up. Since the projected light line is directed to intersect the set component, typically also the set component is picked up.

The component can furthermore have a component edge in the image plane recorded by the digital camera. If, for example, the component is a substantially rotationally symmetrical rivet or a plug, the component edge in the recorded image plane can be represented as a circle or ellipse. The component edge may also include any circumferential shapes, such as polygons and / or the like. Likewise, a pronounced on the component projection and / or recess or imprint define a suitable component edge. In order to cut the component by means of the at least one light line, the at least one light line is preferably projected onto the component in such a way that the at least one light line forms at least two points of intersection with the component edge of the component in the recorded image plane, i. Preferably, the projected light line intersects the set component such that at least two intersection points can be determined.

The setting quality of the set component can be determined by means of the image. For this purpose, the image of the projected light line in the image is evaluated and preferably compared with quality criteria which describe the setting quality of the component. Thus, with only one image to be evaluated, a quality determination can be performed. In particular, the method makes it possible to determine the setting quality without having to move the light line over the set component, as described in detail below. This allows reduced cycle times in quality control. The visual evaluation of the setting quality is faster and less susceptible to interference than tactile systems. Compared with conventional touch systems, as described above, the method according to claim 1 provides a more accurate determination of the setting quality.

• • Definieren eines ersten Bereichs des Bildes, welcher das gezeigte Bauteil zeigen soll und eines zweiten Bereichs des Bildes, der kein gesetztes Bauteil zeigen soll, wobei die Digitalkamera derart angeordnet ist, dass die projizierte Lichtlinie im ersten Bereich zu dem zweiten Bereich um einen Versatz versetzt erscheint, und
• • Bestimmen des Betrags des Versatzes, um die Setzqualität des gesetzten Bauteils zu überwachen.

Preferably, the method further comprises the following steps:

• Defining a first region of the image which is to show the component shown and a second region of the image which is not intended to show a set component, wherein the digital camera is arranged such that the projected image Light line in the first region appears offset from the second region by an offset, and
• • Determining the amount of offset to monitor the set quality of the set component.

The first and the second region are preferably chosen so that they show no edge region of the set component. For example, this can be ensured by not taking into account a part of the image in which the edge region of the component is securely shown in the evaluation of the image, d. H. is masked out. Thus, it is ensured that the first area is an area, in which preferably only the set part is shown, and the second area is an area where no set part is shown.

If the digital camera is arranged such that the projected light line appears offset from the first to the second area in the image, then the amount of the offset can be used to determine the placement quality of the component. This offset occurs when the set component and the workpiece, in which the component was set, have different heights and the digital camera is offset by an angle to the projection direction of the light source.

The offset between the projected line in the first area and the projected line in the second area is a measure of the height difference between the set component and the workpiece or the surface in which the component was placed. For example, to determine the setting quality, an offset can be determined which corresponds to a desired height difference between the component and the workpiece. This difference in height can be greater than zero (i.e., the component will jump), less than zero (i.e., the component will rebound), or equal to zero (i.e., the component and workpiece are planar). If a different offset is determined, such as the desired offset, then a higher offset corresponds to an overhanging component. If a lower offset is determined, the component jumps back too far (or not far enough). Too low an offset may also indicate that no component has been set at all and the light line is e.g. projected onto the bottom of an opening into which the component was to be placed. The offset can thus be determined as a first quality criterion and allows statements about the presence of components and their set height and thus setting quality.

• • Definieren eines ersten Bereichs des Bildes, welcher das gesetzte Bauteil zeigen soll und eines zweiten Bereichs des Bildes, der kein gesetztes Bauteil zeigen soll;
• • Definieren von Bildzeilen, welche parallel zu der projizierten Lichtlinie verlaufen, und
• • Berechnen des Grauwertdurchschnitts, wobei der Grauwertdurchschnitt jeweils für den ersten und den zweiten Bereich separat für jede Bildzeile berechnet wird, um die Setzqualität beim Setzen des Bauteils zu überwachen.

According to the invention, the method further comprises the following steps:

• • defining a first area of the image which is to show the set component and a second area of the image which should not show a set component;
• • Defining image lines that are parallel to the projected light line, and
• Calculating the gray scale average, wherein the gray scale average for each of the first and the second area is calculated separately for each image line in order to monitor the setting quality when setting the component.

The first and second regions of the image that are defined are preferably defined analogously to the first and second regions of the image for determining the offset. Thus, the first area of the image preferably shows only the set component and the second area of the image only an area in which no set component is present.

In particular, the image shows at least one projected light line passing through the image. Parallel to the projected light line, image lines are determined which have a defined line height. The line height can be predetermined, for example, by a number of pixels. Preferably, one row is at least one pixel, preferably at least three pixels, and most preferably at least five pixels high.

The gray value average is then determined separately for each line and for each area. This allows an accurate evaluation of the image even with the appearance of pixel errors, reflections and / or the like and under the influence of stray light. The gray value of each image line in the first or second area indicates where brightness maxima are located. Each brightness maximum corresponds to a projected light line. The offset of the light lines in the first and in the second range can then be determined from the position of the brightness maxima.

In addition to the improved offset determination, the formation of the gray value average also enables the determination of whether a set component was set obliquely and / or deformed. For this purpose, the image lines are preferably determined such that they run parallel to the projected light line in the second region. If the component is set obliquely or deformed, then the projected light line in the first region is not parallel to the projected light line in the second region.

If the gray scale average is formed line by line for the first area, then the projected light line intersects at least one image line in the first area. Consequently, it comes to training broad, blurred brightness maxima in the evaluation of the gray value average. If such blurred brightness maxima occur, then it can be assumed that the component was set obliquely or deformed. Thus, a second quality criterion can be defined, which is characteristic of the angular position and the deformation of the set component.

Preferably, the image is a two-dimensional image, which is perspective-corrected before determining the set quality, wherein the perspective equalization is preferably calculated by means of the pose of the digital camera. Pose is the location (position) and orientation of the digital camera in the room.

A perspective equalized image simplifies the subsequent quality control when setting the component, as it can be assumed that uniform, rectified image material and no perspective distortion effects must be considered. If the pose of the digital camera is known (e.g., via the position of the axes of motion of the manipulator), perspective equalization is a simple arithmetic operation that can be performed by computers or processors as used in remote computing units.

Preferably, the image is a two-dimensional image, wherein the image is rotated prior to determining the set quality such that the projected light line is substantially parallel to a main direction of the two-dimensional image, wherein the rotation is preferably by a partial displacement of the image.

Such two-dimensional images are typically raster graphics. Raster graphics are characterized by a matrix arrangement of pixels comprising rows and columns. Rows and columns are called main directions. If the image is rotated so that the projected light line is substantially parallel to a main direction, i. H. to a column or a row, for example, the previously described determination of the gray value average along a pixel row of the two-dimensional image can be performed. Similarly, the determination of the gray value average along a pixel column can be performed. The determination of the gray value average along a main direction of the image can be carried out quickly and requires only low computing capacity.

Furthermore, it is advantageous to perform the image rotation by means of an area-wise displacement of the image. This shift in regions is referred to as SLANT operation in image processing. For this, the image is divided into areas that intersect the projected light line. By shifting the areas of the image, the projected light line can be aligned parallel to a main direction. In the peripheral areas of the image, this results in a stepped border of the image. By properly cropping or widening the image, a completely filled image plane can be achieved.

• • Definieren einer ersten und einer zweiten Hilfslinie in dem Bild, in der Art, dass die erste und die zweite Hilfslinie die projizierte Lichtlinie schneiden;
• • Bestimmen der Helligkeitsmaxima auf der ersten und zweiten Lichtlinie, und
• • Bestimmen des Versatzes der Helligkeitsmaxima zwischen der ersten und der zweiten Hilfslinie, um den Grad der Drehung des Bildes zu ermitteln.

Preferably, the method further comprises the following steps:

• Defining first and second guides in the image such that the first and second guides intersect the projected line of light;
• • Determining the brightness maxima on the first and second light line, and
• Determining the offset of the brightness maxima between the first and second auxiliary lines to determine the degree of rotation of the image.

Preferably, the first and second auxiliary lines are defined so that they come to rest on two different sides of the component. The auxiliary lines are imaginary lines, on which the brightness information of the image is evaluated and on which the brightness maxima are determined. The brightness maxima correspond to the intersection between the auxiliary line and the projected line. If brightness maxima occur at the same position on the first and second auxiliary lines, the image is not twisted. However, if the brightness maxima occur at different points of the auxiliary lines, then the degree of rotation of the image in the form of a twist angle can be determined via the offset of the brightness maxima and the spacing of the auxiliary lines. The twist angle is the angle enclosed by a line of light with a main direction of the image. If the rotation angle of the image is known, the image can be rotated as desired by image rotation, SLANT or other methods.

• • Projizieren von zumindest drei Lichtlinien auf die Öffnung, mittels der Lichtquelle, wobei die Öffnung von einem Rand begrenzt wird, und wobei jede der Lichtlinien den Rand der Öffnung zumindest zweimal schneidet, und
• • Bestimmen der Position der Öffnung mittels der Schnittpunkte der zumindest drei Lichtlinien mit dem Rand der Öffnung.

Preferably, the component is placed in an opening, the method comprising the following steps before the component is set:

• Projecting at least three lines of light onto the aperture, by means of the light source, wherein the aperture is bounded by an edge, and wherein each of the light lines intersects the edge of the aperture at least twice, and
• • Determining the position of the opening by means of the intersections of the at least three Light lines with the edge of the opening.

If the shape of the edge of the opening is known, the shape of the edge of the opening can be reconstructed from the points of intersection of the lines of light with the edge of the opening. For example, by calculating the geometric center of gravity of the shape, the position of the opening in space can be determined. Depending on the shape of the opening, it may be necessary to use more than three lines of light to allow a reliable determination of the position of the opening. Furthermore, the distance of the lines of light must be adapted to the opening so that the light lines intersect the edge of the opening. The determination of the positions of the opening makes it possible to finely position a manipulator which guides an end effector for setting a component and to align it exactly relative to the opening, without using tactile sensors or other additional sensors. Determining the position of the opening by means of the intersections has a high accuracy and low susceptibility to errors.

Preferably, the opening is round, in particular circular, wherein an ellipse is determined from the intersections of the at least three lines with the edge of the round opening, which corresponds to a perspective distorted image of the round opening, wherein the center of the ellipse is calculated. The center of the ellipse corresponds to the geometric center of gravity of the ellipse. If the center of the ellipse is known, the position of the opening can be determined.

Preferably, the digital camera is a stereo camera. Stereo cameras typically have at least two digital cameras, which have different viewing directions. The viewing directions of the digital cameras of a stereo camera are typically arranged at an angle to one another, resulting in two different perspective representations of an object. From these different perspective representations, the position and location of points in space can be determined.

Preferably, the center of the ellipse for the images of both digital cameras of the stereo camera is calculated separately and calculated by means of the disparity of the images, the pose of the end effector relative to the opening. Preferably, the end effector is then aligned relative to the opening in a setting position for setting a component. This corresponds to a fine positioning, so that the end effector can be aligned exactly to the opening without further sensors. In particular, this cycle time is saved. By eliminating additional sensors, the end effector can be made lighter, allowing for greater dynamics.

Furthermore, the working space of the end effector is not limited by additional sensors.

The at least one projected light line is preferably projected immovably relative to the opening and / or to the component. According to the described method, the light line for determining the position of the opening and / or the setting quality of the component need not be moved over the opening / the component. This means that the line itself remains unmoved. For example, if the line is generated by a type of laser scanner that uses a laser light source and rotating mirrors, a spot of light will travel back and forth to create the line. The line itself, however, is not moved. Such a stationary line, which always cuts the opening and / or the component at the same points at the time of the determination, is sufficient. Thus, by means of a projection and an image, the position of the opening and / or the setting quality of the component can be determined. A scanning of the opening and / or the component and the recording of multiple images and their evaluation are not required, whereby calculation time and cycle time is saved.

The object is further achieved by a manipulator system comprising: a manipulator which carries an end effector, which is set up for setting components, in particular for setting plugs; a light source configured to project at least one line of light, wherein the at least one line of light intersects an aperture and / or a set component; a digital camera adapted to take an image showing the projected light line and a means to determine the set quality of the set component and / or or the pose of the end effector relative to the aperture by means of the image, the means being adapted to define a first region of the image which is to show the placed component and a second region of the image which is not intended to show a set component; Defining image lines running parallel to the projected light line and calculating the gray scale average, the gray value average for each of the first and second regions being calculated separately for each image line to monitor the set quality when setting the device.

In particular, such a manipulator system for implementing the method described above is set up. The manipulator is preferably an industrial robot. The end effector is, for example, a plug setting tool, a riveting tool, a bolt setting tool, a screwdriver or the like, and adapted to set suitable components such as plugs, rivets, screws, bolts, etc. In particular, the end effector is suitable for placement of plugs. Stoppers are widely used, for example, in automobile production and are preferably made of a plastic. The means for determining the placement quality of the component and / or the pose of the end effector relative to the aperture is preferably a computing unit that analyzes the images captured by the digital camera. The analysis includes, for example, the determination of the offset of the projected line as described above, or the determination of a gray value average, as also described above. Likewise, the means may be integrated in a control device of the manipulator system, wherein the control device of the manipulator system is arranged for controlling the manipulator and / or the end effector. Preferably, the means for determining the set quality of the set component and / or the pose of the end effector is mounted relative to the opening on the manipulator and / or on the end effector.

Preferably, the digital camera is a stereo camera, which comprises at least two digital cameras, wherein the two digital cameras are each aligned in a viewing direction and wherein the two viewing directions are preferably at an angle α of at least 10 °, preferably of at least 17 °, and most preferably of at least 25 °.

The viewing direction is the direction which is vertical to the image recording unit of the digital camera. An image acquisition unit may be, for example, a CCD chip, a CMOS chip or the like. The orientation of the directions of view at a certain angle α to each other ensures a different perspective shot in the pictures. Thus, the position of certain points in space can be determined. In particular, large angles allow a very accurate determination of the position. However, the angle must be chosen sufficiently small in order to achieve a space-saving design.

Preferably, the means for determining the set quality of the set component and / or the pose of the end effector is arranged on the manipulator and preferably on the end effector, wherein the means is a decentralized arithmetic unit which is adapted to the pose of the end effector and / or the set quality when setting of a component by means of the digital camera.

The use of decentralized computing units allows the evaluation of the images of the digital camera directly on the end effector and / or on the manipulator, so that short data transmission distances are achieved.

Preferably, the light source of the manipulator system is a laser light source which is adapted to project at least one light line. These projected lines of light are used, for example, to carry out the method described above. Laser light is advantageous because it can be transmitted over long distances and is less prone to interference. In addition, the known wavelength of the laser light simplifies the determination of brightness maxima, the offset and the gray value average in the images.

Preferably, the light source has a projection direction, wherein the projection direction with the viewing direction of the digital camera forms an angle of at least 10 °, preferably of at least 15 ° and most preferably of at least 26 °. The angle between projection direction and viewing direction of the digital camera makes it possible to detect an offset. The larger the angle, the larger the offset appears. However, the angle must be chosen sufficiently small in order to achieve a space-saving design.

Preferably, the component setting direction of the end effector with the viewing direction of the digital camera includes an angle of at least 10 °, preferably at least 15 °, and most preferably at least 29 °. The component setting direction of the end effector is the direction in which the component is displaced when being set. Typically, the component setting direction is perpendicular to the opening into which the component is to be placed. A corresponding angle of the viewing direction of the digital camera allows an optimal view of the digital camera on the opening and the component to determine the position of the opening and / or the setting quality of the set component.

• 1 eine schematische Darstellung eines Manipulatorsystems;
• 2 eine schematische Darstellung einer Stereokamera sowie von der Stereokamera erzeugte Bilder;
• 3 eine schematische Darstellung eines Verfahrens zur Bestimmung der Position einer Öffnung;
• 4A ein gesetztes Bauteil und ein Bild des gesetzten Bauteils mit hoher Setzqualität;
• 4B, 4C ein gesetztes Bauteil und ein Bild des gesetzten Bauteils mit schlechter Setzqualität;
• 5A eine schematische Darstellung eines perspektivisch verzerrten Bildes des Bauteils;
• 5B eine schematische Darstellung zur Bestimmung der Helligkeitsmaxima;
• 5C eine schematische Darstellung eines gedrehten Bildes des gesetzten Bauteils;
• 5D eine schematische Darstellung eines ersten und eines zweiten Bereichs des Bildes;
• 5E eine schematische Darstellung nach der Berechnung des Grauwertdurchschnitts des Bildes, und
• 5F ein Detail aus 5E.

In the following the invention will be explained with reference to the attached figures in certain details. Showing:

• 1 a schematic representation of a manipulator system;
• 2 a schematic representation of a stereo camera and generated by the stereo camera images;
• 3 a schematic representation of a method for determining the position of an opening;
• 4A a set component and an image of the set component with high setting quality;
• 4B . 4C a set component and an image of the set component with poor set quality;
• 5A a schematic representation of a perspective distorted image of the component;
• 5B a schematic representation for determining the brightness maxima;
• 5C a schematic representation of a rotated image of the set component;
• 5D a schematic representation of a first and a second region of the image;
• 5E a schematic representation of the calculation of the gray scale average of the image, and
• 5F a detail from 5E ,

In particular shows 1 a manipulator system 100 , which is set up to carry out the described method. The manipulator system 100 includes a manipulator 110 the one end effector 120 leads. The end effector 120 is designed for setting components, in particular plugs 15 °. The manipulator system 100 further comprises a control unit 160 for controlling the manipulator and / or the end effector 120 , At the end effector 120 is a stereo camera, comprising two digital cameras 135a . 135b arranged. Furthermore, the end effector is a light source 131 , which is preferably a laser light source, arranged. The stereo camera 135a . 135b and the light source 131 are so at the end effector 120 arranged that the setting of the component is not impaired, in particular they are not in the front region of the end effector 120 arranged.

The pictures of the stereo camera 135a . 135b be in the arithmetic unit 133 processed and determines the setting quality of the set component and / or the position of an opening into which a component is to be set. In the example shown, the component 150 into a workpiece 140 set. This workpiece 140 For example, it may be a vehicle body and the component 150 a plastic stopper.

2 shows a schematic representation of the stereo camera, and the two digital cameras 135a . 135b the stereo camera. The two digital cameras 135a . 135b have viewing directions, which are shown by dashed thin lines. These views of the digital cameras 135a . 135b close an angle α one.

The digital cameras 135a . 135b are on a circular opening 145 directed, in which a component is to be set. The opening 145 has an edge 146 on. On the opening 145 become light lines 132 projected. In particular, at least three lines of light 132 projected. In the opening become the light lines 132 preferably not reflected, so that the light lines within the opening 145 non-appearance. The position of the opening 145 is about its center 147 certainly.

In a first picture 235a the first digital camera 135a the opening appears 145 as perspectively distorted ellipse 245a , From the intersections of the light lines shown in the picture 132a with the edge of the ellipse 245a , first the shape of the ellipse and thus the perspective distortion can be determined. By determining the geometric center 247a the ellipse 245a the position of the opening can be determined. Analog becomes a second picture 235b the second digital camera 135b evaluated. The second picture 235b shows projected light lines 232b , a second ellipse 245b that due to the different arrangement of the digital camera 135b appears under another perspective distortion. The middle-point 247b the ellipse 245b can be determined analogously. From the disparity of the two pictures 235a . 235b can then change the position of the opening 145 which are about the center 147 is determined to be calculated. Subsequently, the end effector 120 relative to the opening 145 be aligned to put a component.

3 shows a schematic representation of a method 300 , which for determining the position of an opening 145 is being used. In a first process step 310 the points of intersection of the lines of light with the edge of the opening are determined. In step 320 an ellipse is determined that corresponds to the intersections. This is done for each of the first and second images of the digital cameras of a stereo camera. In a third step 330 become the geometric centers of the ellipses 245a . 245b certainly. Subsequently, in a fourth step 340 the disparity between the first picture 235a and the second picture 235b certainly. The disparity then becomes the position of the opening 145 determined in the room (step 350 ).

The 4A-4C each show on the left side a sectional view of a set component 451a . 451b . 451c , The components 451a . 451b . 451c For example, plugs that are in an apertured plate 441a . 441b . 441c were set.

In 4A is a stopper 451a to see which was set with good setting quality. The good setting quality is characterized by the fact that the plug 451a was introduced evenly into the opening and a desired height difference to the plate 441a having. In 4B is a stopper 451b to see which was introduced evenly into the opening, but is set too low. The height difference between plugs 151b and the tin 441b is less than the desired height difference 4A , 4C shows a stopper 451c , which obliquely into an opening of a sheet 441c was introduced.

On the right side of the 4A-4C are schematic representations of pictures 435a - 435c a digital camera showing the projected light lines 432a . 432b . 432c as well as the set plugs 451a . 451b and 451c show while are in The figures each have three lines of light to see, but preferably the setting quality can be determined with only one light line. In 4A which is a stopper 451a with good set quality shows are the light lines 432a in a second area 436a of the picture 435a to the light lines over the put stopper 451a added. The area above the put plug 451a is a first area 438a of the picture 435a , The offset between the light lines in the first area 438a and the second area 436a is a measure of the height difference between the plug 451a and the tin 441a ,

As in 4B to see is the offset of the light lines 432b from a first area 436b to a second area 438b lower than the one in 4A shown. This corresponds to the set too low plug 451b , The offset is a quality criterion for assessing the height difference between plug and sheet. If the plug were set too high (not shown), the offset between the first and second regions would be greater than in 4A shown.

4C shows an obliquely placed plug 451C , In picture 435c can the obliquely placed plug through the on the plug 451c in the first area 438c oblique projected light lines are detected. The oblique light lines in the first area 438c in particular are not parallel to the light lines 432c in the second area 436c , The orientation of the light lines in the first and in the second region is another quality criterion that describes the angular position of the plug.

The 5A-5F schematically show the steps of image processing for determining the setting quality of a set plug 551 , 5A shows a picture 535a a set stopper 551 on which lines of light 532 be projected. The light lines 532 cut the stopper 551 in each case completely, ie at least in two places. In a first method step, the perspective distorted representation of the image 535a equalized.

5B shows a perspective equalized image 535b of the plug 551 and the projected light lines 532 , image 535b is twisted by a twist angle. The twist angle is the angle that a light line 532 with a main direction x . y of the picture 535b includes. In picture 535b become guides 560 . 562 defines which the projected light lines 532 to cut. On these guides 560 . 562 which preferably on two different sides of the set plug 551 are defined, the brightness maxima 561 . 563 certainly. The brightness maxima correspond to an intersection of the projected light lines 532 with the guides 560 . 562 , From the position of the brightness maxima 561 . 563 on the guides 560 . 562 can the offset of the brightness maxima 561 . 563 be determined. By means of the offset and the distance of the auxiliary lines 560 . 562 then can the degree of rotation of the picture 535b be determined.

In 5C is a shot picture 535c shown the plug 551 and projected light lines 532 shows. The picture 535c was opposite picture 535b turned so that the projected light lines 532 essentially parallel to a main direction x of the picture 535c run. The picture 535c is preferably a raster graphic having two main directions. These are the row direction x and the column direction y ,

5D shows a picture 535d which is essentially picture 535c equivalent. In picture 535d became a first area 538 and a second area 536 Are defined. The first area 538 defines an area in which only the set plug 551 you can see. The area 536 defines the area in which no set stopper 551 you can see. To the transition area of the plug 551 to safely exclude to the environment and to hide interference effects at the transition in determining the setting quality becomes an area 537 unmasked and not considered below.

In a following step, image lines of the image 535d Are defined. In the schematic representation of 5D The image lines run in the X direction parallel to the projected light lines 532 , An image line is preferably defined by a certain pixel height in the Y direction. For example, an image line may be at least one pixel high, at least three pixels, or at least 5 pixels high.

In the first area 538 For each line a gray value average is calculated. Likewise, in the second area 536 a greyscale average is calculated for each line. The gray value average is calculated over the entire width of the first and / or second region.

In 5E a schematic representation of a gray value average for the first and the second area is shown. The image areas where no projected line 532 can be seen in the presentation of the 5E White. The areas where a projected light line 532 can be seen, are in 5E shown in black. By determining the gray value average interference can be hidden. Also, the offset between the projected lines 532 be determined. The gray scale averages of the projected lines are in 5E denoted by 532 '. The offset of the light line 532 ' from a first area 536 to a second area 538 is in detail in 5F represented and with reference numerals 570 characterized. Do not give sharp lines of light 532 ' after determining the gray value average, the plug is set obliquely or deformed.

LIST OF REFERENCE NUMBERS

100

manipulator system

110

manipulator

120

end effector

131

light source

132

light line

133

decentralized computing unit, means

135a, 135b

digital cameras

140

workpiece

145

opening

146

edge

147

Focus

150

component

160

control unit

235a, 235b

image

232a, 232b

light line

245a, 245b

perspective distorted representation of the opening 145 (Ellipses)

247a, 247b

geometric center of gravity of the ellipses

300

method

310

Determination of the intersections

320

Determine the shape of the opening

330

Determination of the geometric center of gravity

340

Calculation of disparity

350

Calculation of the position of the opening

441a-c

workpiece

432a-c

projected light lines

435a-c

image

436a-c

second area

438a-c

first area

451a-c

Plug

532

projected light line

532 '

Gray value average of the light lines

551

Plug

535a-e

image

560, 562

ledger line

561, 563

brightness maxima

536

second area

538

first area

537

unmasked area

570

offset

F

detail view

α

angle

x, y

main direction

Claims (17)

Method for controlling a manipulator system (100) which comprises at least one manipulator (110) which carries an end effector (120) which is set up for setting components (150), in particular for setting plugs, and wherein the manipulator system (100 ) further comprises at least one light source (131) and a digital camera (135a, 135b), the method comprising the steps of: Projecting at least one light line (132) onto a set component (150) by means of the light source (131), wherein the at least one light line (132) intersects the set component (150); Taking an image (435a-c, 535a-e) by means of the digital camera (135a, 135b) showing the projected light line (432, 532), and Determining the set quality of the set component (150) by means of the image (435a-c, 535a-e), and Defining a first area (436a-c, 536) of the image (435a-c, 535a-e) which is to show the set component (451a-c, 551) and a second area (438a-c, 538) of the image (435a-c, 535a-e), which should not show a set component; Defining image lines which are parallel to the projected light line (432, 532), and Calculating the gray scale average, wherein the gray scale average for each of the first and second regions (536, 538) is calculated separately for each image line to monitor the set quality in setting the device. Method according to Claim 1 the method comprising the steps of: defining a first area (436a-c, 536) of the image (435a-c, 535a-e) to show the set component (451a-c, 551) and a second area (438a-c, 538) of the image (435a-c, 535a-e), which is not intended to show a set component, wherein the digital camera (135a, 135b) is arranged such that the projected light line (432, 532) in the first region (436a-c, 536) appears offset from the second area by an offset (570), and Determining the amount of offset (570) to monitor the set quality of the set component (451a-c, 551). The method of any one of the preceding claims, wherein the image (435a-c, 535a-e) is a two-dimensional image that is perspective-skewed prior to determining the set quality, wherein the perspective equalization is preferably calculated by the pose of the digital camera (135a, 135b) becomes. The method of any one of the preceding claims, wherein the image (435a-c, 535a-e) is a two-dimensional image, and wherein the image (435a-c, 535a-e) is rotated prior to determining the set quality so that the projected light line (432, 532) is substantially parallel to a main direction (x, y) of the two-dimensional image (435a-c, 535a-e), wherein the rotation is preferably by a partial displacement of the image (435a-c, 535a-e) , Method according to Claim 4 the method further comprising the steps of: defining first and second auxiliary lines (560, 562) in the image (435a-c, 535a-e) such that the first and second auxiliary lines (560, 562) intersecting the projected light line (532), determining the brightness maxima (561, 563) on the first and second guides, and determining the offset of the brightness maxima (561, 563) between the first and second guides (560, 562) to determine the degree of Determine the rotation of the image. Method according to one of the preceding claims, wherein the component (150) is to be placed in an opening (145), and wherein the method comprises the following steps before setting the component (150): Projecting at least three lines of light (132) onto the aperture (145) by means of the light source (131), wherein the aperture (145) is bounded by an edge (146), and wherein each of the lines of light (132) is the edge (146) of the Opening (145) intersects at least twice, and Determining the position of the opening (145) by means of the intersections of the at least three lines of light (132) with the edge (146) of the opening (145). Method according to Claim 6 , wherein the opening (146) is round and from the intersections of the at least three lines of light (132) with the edge (146) of the round opening (145) an ellipse (245a, 245b) is determined, which is a perspective distorted image of the round opening (145), and calculating the center point (247a, 247b) of the ellipse (245a, 245b). Method according to one of the preceding claims, wherein the digital camera (135a, 135b) is a stereo camera. Method according to Claim 8 in which the center point (247a, 247b) of the ellipse (245a, 245b) for the images (235a, 235b) of both digital cameras (135a, 135b) of the stereo camera is calculated separately, and wherein by means of the disparity of the images (235a, 235b) Pose of the end effector (120) relative to the opening (145) is calculated. A method as claimed in any one of the preceding claims, wherein the end effector (120) is aligned after positioning relative to the aperture (145) in a set position for seating the component (150). Method according to one of the preceding claims, wherein the at least one projected light line (132) is immobile relative to the opening (145) and / or to the component (150) is projected. Manipulator system (100) comprising: a manipulator (110) carrying an end effector (120) adapted to set components (150), in particular to set plugs; a light source (131) configured to project at least one light line (132), the at least one light line (132) intersecting an aperture (145) and / or a set component (150); a digital camera (135a, 135b) adapted to capture an image (435a-c, 535a-e) showing the projected light line (432, 532), and a means (133) for determining the setting quality of the set component (150) and / or the pose of the end effector (120) relative to the opening (145) by means of the image (435a-c, 535a-e), the means comprising 133) is set up a first area (436a-c, 536) of the image (435a-c, 535a-e) which is to show the set component (451a-c, 551), and a second area (438a-c, 538) of the image ( 435a-c, 535a-e) which should not show a set component; Define image lines which are parallel to the projected light line (432, 532), and calculating the gray value average, wherein the gray value average is calculated for the first and the second area (536, 538) separately for each image line in order to monitor the setting quality when setting the component. Manipulator system (100) according to Claim 12 wherein the digital camera (135a, 135b) is a stereo camera comprising at least two digital cameras (135a, 135b) and wherein the two digital cameras (135a, 135b) are each aligned in a viewing direction, and wherein the two viewing directions are preferably at an angle ( α) of at least 10 °, preferably at least 17 °, and most preferably at least 25 °. Manipulator system (100) according to one of Claims 12 or 13 wherein the means (133) for determining the set quality of the set component and / or the pose of the end effector is arranged on the manipulator (110) and preferably on the end effector (120), the means (133) being a decentralized computing unit (133), which is adapted to determine the pose of the end effector (120) and / or the setting quality when setting a component (150) by means of the digital camera (135a, 135b). Manipulator system (100) according to one of the preceding Claims 12 to 14 wherein the light source (131) is a laser light source configured to project at least one light line (132). Manipulator system (100) according to one of the preceding Claims 12 to 15 wherein the light source (131) has a projection direction, and wherein the projection direction with the viewing direction of the digital camera (135a, 135b) forms an angle of at least 10 °, preferably at least 15 ° and most preferably at least 26 °. Manipulator system (100) according to one of the preceding Claims 12 to 16 wherein the component setting direction of the end effector (120) with the viewing direction of the digital camera (135a, 135b) includes an angle of at least 10 °, preferably at least 15 °, and most preferably at least 29 °.

Images