JP2015079414A - Image processing apparatus, robot, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To facilitate teaching in an inspection of an assembly.SOLUTION: An image acquisition section 42b acquires first image data of an assembly obtained by assembling a first part and a second part. A collation section 42c collates the first image data acquired by the image acquisition section 42b with second image data of the assembly stored in advance in a storage section 43. Further, the collation section 42c collates third image data obtained by collating the first image data with the second image data, with the second image data. A calculation section 42d calculates a relative position between the first part and the second part on the basis of collation results of the collation section 42c. An inspection section 42e inspects the assembly on the basis of the relative position calculated by the calculation section 42d.

Description

  The present invention relates to an image processing apparatus, a robot, and an image processing method.

  Patent Document 1 proposes an inspection method and apparatus for an assembly that can efficiently and accurately inspect the shape or assembly position of each component that constitutes the assembly.

  Patent Document 2 proposes an image collation apparatus, an image collation method, a program, and a recording medium that can improve collation accuracy even when a local distortion exists in an inspection image.

JP-A-8-5334 JP 2002-334331 A

  By the way, for example, when the robot is caused to assemble a plurality of parts and the robot is inspected whether or not the assembly is properly assembled, it is necessary to teach the robot image data of each part of the assembly. For example, the robot compares the image data of the assembled assembly with the image data of each part of the taught assembly, and checks whether or not the assembly has been properly assembled.

  That is, a plurality of image data corresponding to the number of parts must be taught to the robot, which takes time.

  Accordingly, an object of the present invention is to facilitate teaching in inspection of an assembly.

  According to a first aspect of the present invention for solving the above-described problem, an image acquisition unit that acquires first image data including an assembly in which a first part and a second part are assembled, and the image acquisition Obtained by collation of the first collation unit that collates the first image data acquired by the unit with the second image data including the assembly stored in advance in a storage device. Based on the second collation unit that collates the third image data and the second image data, the collation result of the first collation unit and the collation result of the second collation unit, A calculation unit that calculates a relative position between the first component and the second component, and an inspection unit that inspects the assembly based on the relative position calculated by the calculation unit. The image processing apparatus. According to the first aspect, it is only necessary to teach the image data of the assembly in which the first part and the second part are assembled, the assembly can be inspected with one teaching data, and teaching in the inspection of the assembly is possible. Can be made easier. In addition, the third image data obtained by collating the first image data and the second image data is collated again with the second image data, so that the first component and the second component can be compared with each other. The position can be calculated and the assembly can be inspected based on the calculated relative position. Thus, for example, if the relative position deviation between the second part and the second part is within the allowable range, it can be determined that the inspection has passed. On the other hand, if the relative position deviation is outside the allowable range, the inspection cannot be performed. It can be judged as passing.

  The first collating unit collates the first image data and the second image data, and the assembly included in the first image data includes the assembly included in the second image data. A portion corresponding to the assembly may be erased to obtain the third image data. Thereby, it is not necessary to teach the respective image data of the first part and the second part.

  The calculation unit includes the position and orientation of the assembly included in the first image data obtained by collating the first image data and the second image data in the first collation unit, Based on the position and orientation of the assembly included in the third image data obtained by collating the third image data and the second image data in the second collation unit, the relative position is calculated. It is good also as calculating. Thereby, the relative position of the first component and the second component can be calculated based on the position and orientation of the assembly.

  The inspection unit may inspect whether or not the assembly is a non-defective product based on the relative position calculated by the calculation unit. Thereby, it is possible to appropriately inspect whether or not the assembly is a non-defective product.

  The inspection unit may determine that the assembly is a non-defective product when the outline of the assembly is erased by a predetermined number of pixels or more based on a collation result of the first collation unit. As a result, it is possible to speed up the determination of whether the assembly is a non-defective product.

  The relative position may include a shift due to an angle between the first component and the second component. Accordingly, the assembly can be inspected even by a deviation due to the angle between the first component and the second component.

  According to a second aspect of the present invention for solving the above-described problem, a manipulator that assembles a first part and a second part, and an image that acquires first image data including an assembly that is assembled by the manipulator. An acquisition unit; a first verification unit that collates the first image data acquired by the image acquisition unit with second image data including the assembly stored in advance in a storage device; A second collation unit that collates the third image data obtained by the collation of the second collation unit and the second image data, a collation result of the first collation unit, and a collation of the second collation unit And a calculation unit that calculates a relative position between the first component and the second component, and an inspection unit that inspects the assembly based on the relative position calculated by the calculation unit. And a robot characterized by comprising: According to the second aspect, it is only necessary to teach image data of an assembly in which the first part and the second part are assembled, and teaching in inspection of the assembly can be facilitated. In addition, the third image data obtained by collating the first image data and the second image data is collated again with the second image data, so that the first component and the second component can be compared with each other. The position can be calculated and the assembly can be inspected based on the calculated relative position. Thus, for example, if the relative position deviation between the second part and the second part is within the allowable range, it can be determined that the inspection has passed. On the other hand, if the relative position deviation is outside the allowable range, the inspection cannot be performed. It can be judged as passing.

  According to a third aspect of the present invention for solving the above-described problem, an image acquisition step of acquiring first image data including an assembly in which a first part and a second part are assembled, and the image acquisition Obtained by collating the first image data acquired in the step with the first collation step that collates the second image data including the assembly stored in the storage device in advance, and the collation in the first collation step. Based on the second collation step for collating the third image data and the second image data, the collation result of the first collation step and the collation result of the second collation step, A calculation step for calculating a relative position between the first component and the second component; and an inspection step for inspecting the assembly based on the relative position calculated by the calculation step. Image It is a management method. According to the third aspect, it is only necessary to teach image data of an assembly in which the first part and the second part are assembled, and teaching in the inspection of the assembly can be facilitated. In addition, the third image data obtained by collating the first image data and the second image data is collated again with the second image data, so that the first component and the second component can be compared with each other. The position can be calculated and the assembly can be inspected based on the calculated relative position. Thus, for example, if the relative position deviation between the second part and the second part is within the allowable range, it can be determined that the inspection has passed. On the other hand, if the relative position deviation is outside the allowable range, the inspection cannot be performed. It can be judged as passing.

It is the figure which showed an example of schematic structure of the robot 11 which concerns on one Embodiment of this invention. It is the figure which showed an example of the function structure of the robot control apparatus. 6 is a diagram illustrating an example of image data stored in a storage unit 43. FIG. It is a figure which shows an example of the hardware constitutions which implement | achieve the function of the robot control apparatus. 7 is a flowchart showing an example of the operation of the image processing unit 42. FIG. FIG. 6 is a first diagram illustrating an image processing example of an image processing unit. It is FIG. (2) explaining an image process. It is FIG. (3) explaining an image process. It is a figure explaining the detection of the shift | offset | difference by rotation. It is a figure explaining the example of teaching data. It is a figure explaining the example in the case of giving teaching data about each component of the conventional assembly to a robot controller. It is a flowchart which shows an example of operation | movement of the robot control apparatus when teaching data is given about each component of the conventional assembly. It is a figure explaining the example of the conventional teaching data.

  FIG. 1 is a diagram showing an example of a schematic configuration of a robot 11 according to an embodiment of the present invention. In FIG. 1, in addition to the robot 11, parts 21 and 22 to be assembled by the robot 11 and a work table 31 are shown.

  The robot 11 includes a robot control device 12 that controls the entire robot 11, an arm 14 including one or more joints 15 and one or more links 16, a hand 18 provided at the tip of the arm 14, A force sensor 17 provided between the distal end portion and the hand 18 (also referred to as a wrist portion) and an imaging device 19 provided on the hand 18 are included.

  The force sensor 17 detects, for example, a force or moment acting on the hand 18. The output of the force sensor 17 is sent to the robot control device 12 and used for controlling the operation of the robot 11 by the robot control device 12. As the force sensor 17, for example, a 6-axis force sensor capable of simultaneously detecting 6 components of a force component in the translational 3 axis direction and a moment component around the 3 rotation axes can be used. Of course, the number of axes of the force sensor 17 is not particularly limited, and may be three axes, for example. The force sensor 17 can also be called a force detection unit.

  The hand 18 includes, for example, a plurality of fingers, and can hold each of the component 21 (first component) and the component 22 (second component) with at least two fingers. Further, the hand 18 can grip an assembly in which the component 21 and the component 22 are combined. The hand 18 may be detachable from the tip of the arm 14. The hand 18 can be said to be a kind of end effector. The end effector is a member for gripping, lifting, lifting, adsorbing, or processing a workpiece. The end effector can take various forms such as a hand, a hook, and a suction cup. Further, a plurality of end effectors may be provided for one arm.

  A unit including the arm 14, the force sensor 17, and the hand 18 can also be referred to as a movable part or a manipulator (hereinafter referred to as a movable part). In the example of FIG. 1, the robot 11 has two movable parts 13. For example, an actuator (not shown) is provided in the movable part 13 in order to operate each part such as the joint 15 and the hand 18. The actuator includes, for example, a servo motor and an encoder. The encoder value output by the encoder is used, for example, for feedback control of the robot 11 by the robot controller 12.

  The robot 11 drives the joints 15 in conjunction with each other according to a control command given from the robot controller 12, thereby setting a target position (hereinafter referred to as an end point) set at the tip of the arm 14 to a predetermined value. It can be moved freely within the movable range or directed in any direction. Further, an object or the like can be grasped or released by the hand 18. Note that the position of the end point is not limited to the tip of the arm, and may be set to the tip of the end effector, for example.

  The imaging device 19 images a work area of the robot 11 (for example, a three-dimensional space that is a workable range by the movable unit 13 on the worktable 31), and generates image data. As the imaging device 19, for example, a visible light camera or an infrared camera can be employed. An image captured by the imaging device 19 is input to the robot control device 12.

  The configuration of the robot 11 described above is not limited to the configuration example described above because the main configuration has been described in describing the features of the present embodiment. Further, the configuration of a general robot is not excluded.

  For example, FIG. 1 shows an example in which the number of joints is six (six axes), but the number of joints (also referred to as “number of axes”) and the number of links may be increased or decreased. In addition, the shape, size, arrangement, structure, and the like of various members such as joints, links, and hands may be changed as appropriate.

  Moreover, the installation position of the imaging device 19 is not particularly limited, and may be installed on the work table 31 or the ceiling or wall. Further, instead of or in addition to the imaging device 19, an imaging device may be provided at the tip portion, the body portion, or the head portion of the arm 14 of the robot 11.

  The robot control device 12 is included in the robot 11, but may be provided outside the robot 11 independently, for example. For example, a robot control device 12 provided independently outside the robot 11 may be connected to the robot 11 by, for example, wiring to control the robot 11.

  FIG. 2 is a diagram illustrating an example of a functional configuration of the robot control device 12. In addition to the robot control device 12 shown in FIG. 1, FIG. 2 shows the movable unit 13, the arm 14, the hand 18, and the imaging device 19 described in FIG. 1. The robot control device 12 includes a control unit 41, an image processing unit 42, and a storage unit 43.

  The control unit 41 controls the operation of the arm 14 and the hand 18 of the movable unit 13. For example, the control unit 41 controls the operations of the arm 14 and the hand 18 so as to assemble an assembly in which the component 21 is incorporated into the component 22.

  The image processing unit 42 includes a reception unit 42a, an image acquisition unit 42b, a collation unit 42c (a first collation unit and a second collation unit), a calculation unit 42d, and an inspection unit 42e. . The receiving unit 42a receives teaching data (image data) of the assembly from the user via an input device such as a keyboard or a touch panel. The receiving unit 42 a stores the teaching data received from the user in the storage unit 43. When the robot 11 is connected to a terminal device such as a personal computer, for example, the receiving unit 42a may receive teaching data from the user via a keyboard or the like provided in the terminal device.

  FIG. 3 is a diagram illustrating an example of teaching data stored in the storage unit 43. FIG. 3 shows an example of teaching data including an assembly 51 in which the part 52 is incorporated into the part 53. The component 52 and the component 53 are the same types of components as the component 21 and the component 22 shown in FIG.

  The receiving unit 42 a receives teaching data of the assembly 51 in which the component 52 is incorporated into the component 53 from the user, and stores the teaching data in the storage unit 43. That is, the user only needs to store one teaching data of the assembly 51 in the storage unit 43, and does not have to store teaching data for each of the parts 52 and 53 of the assembly 51 in the storage unit 43.

  Returning to the description of FIG. The imaging device 19 images the assembly assembled by the arm 14 and the hand 18. For example, the imaging device 19 images the assembly of the component 21 and the component 22 assembled by the arm 14 and the hand 18. The image acquisition unit 42b acquires image data (first image data) including the assembly imaged by the imaging device 19.

  The collation unit 42c collates the image data including the assembly acquired by the image acquisition unit 42b with the teaching data (second image data) stored in the storage unit 43, and is included in the image data ( Get the position and orientation of the assembly (shown). In addition, the collation unit 42c outputs an image in which a portion overlapping the teaching data of the assembly whose position and orientation are acquired is deleted (third image data). The collation unit 42c collates the image data including the assembly from which the portion overlapping the teaching data is erased with the teaching data stored in the storage unit 43, and determines the portion overlapping the teaching data included in the image data. Acquire the position and orientation of the erased assembly.

  The calculation unit 42d calculates the relative position of each component constituting the assembly based on the collation result of the collation unit 42c. For example, the calculation unit 42d calculates the deviation of the component 21 with respect to the component 22 in the assembly assembled by the arm 14 and the hand 18 based on the verification result of the verification unit 42c. The relative positions include, for example, horizontal and vertical shifts of components and rotation shifts.

  The inspection unit 42e inspects the assembly based on the relative position calculated by the calculation unit 42d. For example, the inspection unit 42e inspects whether the assembly is a non-defective product based on the deviation of the component 21 with respect to the component 22 calculated by the calculation unit 42d.

  FIG. 4 is a diagram illustrating an example of a hardware configuration that implements the functions of the robot control device 12.

  The robot controller 12 includes, for example, an arithmetic device 61 such as a CPU (Central Processing Unit), a main storage device 62 such as a RAM (Random Access Memory), an HDD (Hard Disk Drive), and the like as shown in FIG. An auxiliary storage device 63; a communication interface (I / F) 64 for connecting to a communication network by wire or wireless; an input device 65 such as a mouse, keyboard, touch sensor, or touch panel; and a display device 66 such as a liquid crystal display. It can be realized by a computer 60 including a read / write device 67 that reads / writes information from / to a portable storage medium such as a DVD (Digital Versatile Disk).

  For example, the functions of the control unit 41 and the image processing unit 42 are realized by the arithmetic device 61 executing a predetermined program loaded from the auxiliary storage device 63 or the like to the main storage device 62. The reception unit 42a is realized, for example, when the arithmetic device 61 uses the input device 65. The storage unit 43 is realized, for example, when the arithmetic device 61 uses the main storage device 62 or the auxiliary storage device 63. Communication between the robot 11, the robot control device 12, and other devices is realized, for example, when the arithmetic device 61 uses the communication I / F 64.

  The predetermined program may be installed from, for example, a storage medium read by the read / write device 67, or may be installed from a network via the communication I / F 64.

  Further, some or all of the functions of the control unit 41 or the image processing unit 42 may be realized by, for example, a controller board including an ASIC (Application Specific Integrated Circuit) including an arithmetic device, a storage device, a drive circuit, and the like. .

  The functional configuration of the robot 11 described above is classified according to the main processing contents in order to facilitate understanding of the configuration of the robot 11. The present invention is not limited by the way of classification and names of the constituent elements. The configuration of the robot 11 can be classified into more components depending on the processing content. Moreover, it can also classify | categorize so that one component may perform more processes. Further, the processing of each component may be executed by one hardware or may be executed by a plurality of hardware. For example, the processes of the control unit 41 and the image processing unit 42 may be executed by separate hardware, and the image processing unit 42 may be realized as an image processing apparatus.

  Next, the operation of the robot control device 12 will be described. The control unit 41 controls the arm 14 and the hand 18 to assemble an assembly composed of the parts 21 and 22, and places the assembly at a predetermined position on the work table 31.

  When the assembled assembly is placed at a predetermined position on the work table 31, the control unit 41 images the assembly with the imaging device 19. The control unit 41 controls the arm 14 and the hand 18 so that the assembled assembly is imaged under the same conditions as the conditions (for example, the direction and distance in which the assembly 51 is imaged). And the position of the imaging device 19 provided in the hand 18 is controlled.

  For example, when the imaging device 19 is installed on the work table 31 or the ceiling or wall instead of the hand 18, the control unit 41 sets the assembled assembly under the same condition as the condition where the teaching data assembly 51 is imaged. The assembly is placed at a predetermined position on the work table 31 so that the solid is imaged by the imaging device 19. When the assembly is placed at a predetermined position on the work table 31, the control unit 41 images the assembly with the imaging device 19.

  FIG. 5 is a flowchart showing an image processing method which is an example of the operation of the image processing unit 42. 6, 7, and 8 are diagrams for describing an image processing example of the image processing unit 42. Hereinafter, the operation of the image processing unit 42 will be described using the flowchart of FIG. 5, and the operation of the image processing unit 42 will be described using the image processing examples shown in FIGS. 6, 7, and 8 as appropriate. .

  As described above, when the assembly is imaged by the imaging device 19, the image processing unit 42 starts processing based on the following steps. The storage unit 43 stores teaching data of the assembly 51 in advance.

  The image acquisition unit 42b acquires image data of the assembly of the component 21 and the component 22 captured by the imaging device 19 (step S1). Hereinafter, the assembly of the component 21 and the component 22 assembled by the robot 11 may be referred to as an inspection object.

  6 (a), 7 (a), and 8 (a), an inspection object 71 (an assembly of the component 21 and the component 22 assembled by the robot 11) acquired by the image acquisition unit 42b is shown. An example image is shown. The inspection object 71 shown in FIG. 6A shows an example in which the component 21 is appropriately incorporated in the component 22. The inspection object 71 shown in FIG. 7A and FIG. 8A shows an example in which the component 21 is incorporated in the component 22 while being shifted to the right in the drawing. In the inspection object 71 shown in FIG. 8A, the part 21 is largely displaced from the part 22 compared to the inspection object 71 shown in FIG.

  Returning to the description of FIG. Next, the collation unit 42c collates the image data including the inspection target acquired by the image acquisition unit 42b with the teaching data stored in the storage unit 43, and is included in the image data acquired by the image acquisition unit 42b. The position and posture of the inspection object being acquired are acquired (step S2). For example, the collation unit 42c detects the contour of the inspection target included in the image data acquired by the image acquisition unit 42b and the contour of the assembly 51 included in the teaching data based on the search property described in FIG. To do. The collation unit 42c collates the detected contour of the image data with the contour of the teaching data based on the search property described in FIG. 10, and specifies the contour of the image data that best matches the contour of the teaching data (for example, The contour of the image data is specified by checking the degree of coincidence between the two contours). That is, the collation unit 42c specifies where the inspection object exists on the image data. Then, the collation unit 42c matches the position and orientation of the teaching data when it is determined that the inspection object 71 exists by collating the image data and the teaching data with respect to the inspection object included in the image data. Get as position and orientation.

  FIGS. 6B, 7B, and 8B show examples of acquisition results of the position and posture (the posture will be described later) of the inspection object 71 by the matching unit 42c. 6 (b), FIG. 7 (b), and FIG. 8 (b) indicate the outline of the teaching data (the outline of the assembly 51 shown in FIG. 3). Moreover, the dotted line 81 shown in FIG.6 (b), FIG.7 (b), and FIG.8 (b) has shown the position of the test target object 71 which the collation part 42c acquired. A line thinner than the dotted line 81 shown in FIGS. 6B, 7B, and 8B indicates the inspection object 71 included in the image data.

  As described above, the collation unit 42c collates the image data acquired by the image acquisition unit 42b with the teaching data, and specifies the contour of the image data that best matches the contour of the teaching data. In the case of FIG. 6B, the outline of the teaching data matches all the outlines of the inspection object 71 included in the image data as indicated by a dotted line 81. Accordingly, in the case of FIG. 6B, the collation unit 42c specifies that the inspection object 71 exists at the position indicated by the dotted line 81 on the image data. And the collation part 42c acquires the position (position of the dotted line 81 shown in FIG.6 (b)) of the teaching data when it specifies that the test target object 71 exists as a position of the test target object 71. FIG.

  In the case of FIG. 7B, the outline of the teaching data most closely matches the part 22 of the inspection object 71 included in the image data, as indicated by a dotted line 81. This is because the contour of the part 22 of the teaching data is larger than the contour of the part 21 and the contour of the part 52 of the teaching data shown in FIG. This is because the degree of coincidence of the contour is higher when it is superimposed on the contour of the component 22. Therefore, in the case of FIG. 7B, the collation unit 42c specifies that the inspection object 71 exists at the position indicated by the dotted line 81 on the image data. And the collation part 42c acquires the position (position of the dotted line 81 shown in FIG.7 (b)) of the teaching data when it specifies that the test target object 71 exists as a position of the test target object 71. FIG.

  In the case of FIG. 8B, the outline of the teaching data most closely matches the part 22 of the inspection object 71 included in the image data, as indicated by a dotted line 81. This is because the contour of the part 22 of the teaching data is larger than the contour of the part 21 and the contour of the part 52 of the teaching data shown in FIG. This is because the degree of coincidence of the contour is higher when it is superimposed on the contour of the component 22. Therefore, in the case of FIG. 8B, the collation unit 42c specifies that the inspection object 71 exists at the position indicated by the dotted line 81 on the image data. And the collation part 42c acquires the position (position of the dotted line 81 shown in FIG.8 (b)) of the teaching data when it specifies that the test target object 71 exists as a position of the test target object 71. FIG.

  In addition, the collation part 42c may acquire the gravity center position of the teaching data at the time of specifying that the inspection target object 71 exists as the position of the inspection target object 71, for example. In addition, a general method can be adopted for image processing for detecting (extracting) a contour.

  Returning to the description of FIG. Next, the collation part 42c erase | eliminates the part corresponding to teaching data of the test target object which acquired the position and attitude | position in step S2 (step S3). For example, the collation unit 42c erases a portion of the inspection object included in the image data that overlaps the teaching data when the inspection object 71 is identified as being present by collating the image data with the teaching data. .

  For example, in the case of FIG. 6B, the position of the inspection object 71 is acquired at the position of the dotted line 81. In FIG. 6B, the contour of the inspection object 71 included in the image data and the contour of the teaching data (dotted line 81) all overlap. Therefore, all the outlines of the inspection object 71 included in the image data are erased.

  In the case of FIG. 7B, the position of the inspection object 71 is acquired at the position of the dotted line 81. In FIG. 7B, the contour of the part 22 of the inspection object 71 included in the image data is entirely overlapped with the contour of the teaching data (dotted line 81), and the contour of the part 21 is partially taught. It overlaps with the outline of the data. Accordingly, as shown in FIG. 7C, the inspection object 71 included in the image data has the entire outline of the component 22 and the partial outline of the component 21 erased.

  In the case of FIG. 8B, the position of the inspection object 71 is acquired at the position of the dotted line 81. In FIG. 8B, the contour of the part 22 of the inspection object 71 included in the image data is entirely overlapped with the contour of the teaching data (dotted line 81), and the contour of the part 21 is partially taught. It overlaps with the outline of the data. Accordingly, in the inspection object 71 included in the image data, as shown in FIG. 8C, the entire contour of the component 22 and the partial contour of the component 21 are erased.

  In addition, the collation part 42c may erase the part corresponding to the teaching data of the image data of the inspection object with a width. For example, the collation unit 42c determines that the inspection object and the teaching data are overlapped if the inspection object and the teaching data are overlapped if the respective contours are deviated within a predetermined number of pixels. The image of the contour of the inspection object is erased.

  Returning to the description of FIG. Next, the collation unit 42c determines whether or not the image data of the inspection object has been erased by a predetermined number of pixels or more in Step S3 (Step S4). When it is determined that the image data of the inspection target has been erased by a predetermined number of pixels or more, the collation unit 42c proceeds to the process of step S8. When it is determined that the image data of the inspection target has not been erased by a predetermined amount or more, the collation unit 42c proceeds to the process of step S5.

  For example, it is assumed that the collation unit 42c determines whether 90% or more of the contour image data of the inspection target has been deleted. Further, the outline of the inspection object 71 shown in FIG. 6A is 100% erased, and the outline of the inspection object 71 shown in FIGS. 7A and 8A is the same as that shown in FIGS. As shown in FIG. 8C, it is assumed that 90% or more of the contour of the component 21 is not erased because a part of the contour of the component 21 is not erased. When the result of erasing the contour of the inspection object 71 is completely erased as shown in FIG. 6, the collation unit 42c proceeds to the process of step S8, as shown in FIGS. 7 (c) and 8 (c). If 90% or more is not erased, the process proceeds to step S5.

  Returning to the description of FIG. Next, when the collation unit 42c determines in step S4 that the image data of the inspection object has not been erased by a predetermined number of pixels or more, the image including the inspection object partially erased obtained in step S3. The data and the teaching data stored in the storage unit 43 are collated, and the position and orientation of the inspection object partially erased are acquired (step S5). For example, the collation unit 42c, based on the search property described in FIG. 10, the contour of the inspection object included in the image data that has been erased in step S3, and the contour of the assembly 51 included in the teaching data. Is detected. The collation unit 42c collates the detected contour of the image data with the contour of the teaching data based on the search property described in FIG. 10, and the image data that has been subjected to the erasing process that best matches the contour of the teaching data. Identify the outline of In other words, the collation unit 42c specifies where on the image data the inspection object with a partially erased contour exists. Then, the collation unit 42c includes the position and orientation of the teaching data when it is determined by the collation between the image data and the teaching data that the inspection object 71 whose contour is partially erased exists. Acquired as the position and orientation of the inspection object.

  FIG. 7D and FIG. 8D show examples of acquisition results of the position and posture (the posture will be described later) of the inspection target 71 from which the partial outline has been deleted by the collating unit 42c. . The dotted line 82 shown in FIG. 7D and FIG. 8D shows the outline of the teaching data (the outline of the assembly 51 shown in FIG. 3). Moreover, the dotted line 82 shown in FIG.7 (d) and FIG.8 (d) has shown the position of the test target object 71 which the collation part 42c acquired and the one part outline was deleted. The lines thinner than the dotted line 82 shown in FIG. 7D and FIG. 8D indicate the inspection object 71 (part 21 shown in FIG. 7C and FIG. 8C) from which the outline is partially erased. Show.

  In the case of FIG. 7D, the outline of the teaching data most closely matches the part 21 of the inspection object 71 included in the image data, as indicated by a dotted line 82. This is because the contour of the part 22 is erased and a part of the contour of the part 21 remains in the image data. Therefore, in the case of FIG. 7D, the collation unit 42c specifies that the inspection object 71 from which a part of the contour is erased exists at the position indicated by the dotted line 82 on the image data. Then, the collation unit 42c determines the position of the teaching data (the position of the dotted line 82 shown in FIG. 7D) when it is determined that the inspection object 71 whose partial outline has been deleted exists. Get as the position. In FIG. 8D as well, the collating unit 42c acquires the position of the inspection object 71 from which the partial contour has been deleted.

  In addition, the collation part 42c may acquire the gravity center position of the teaching data at the time of specifying that the inspection target object 71 exists as the position of the inspection target object 71, for example.

  Returning to the description of FIG. Next, the calculation unit 42d calculates the relative position of the part of the inspection target based on the position and orientation of the inspection target acquired by the collation unit 42c (step S6). For example, the calculation unit 42d erases the position and orientation of the inspection target and the partial contour acquired by the collation unit 42c for the first time (acquired in step S2) and acquired for the second time (acquired in step S5). ) Calculate the relative positions of the parts of the assembly based on the position and orientation of the inspection object.

  For example, a dotted line 81 shown in FIG. 7 (e) corresponds to the dotted line 81 shown in FIG. 7 (b) and indicates the position of the inspection object 71 acquired for the first time. A dotted line 82 shown in FIG. 7 (e) corresponds to the dotted line 82 shown in FIG. 7 (d) and indicates the position of the inspection object 71 acquired for the second time. The calculation unit 42d calculates the position of the inspection object 71 acquired at the first time indicated by the dotted line 81 in FIG. 7E and the inspection object 71 acquired at the second time indicated by the dotted line 82 in FIG. Based on the position, the relative position of the component 21 with respect to the component 22 is calculated. For example, the calculation unit 42d calculates the relative position of the component 21 with respect to the component 22 based on the centroid position 81a of the inspection target 71 acquired first time and the centroid position 82a of the inspection target 71 acquired second time. To do. In the case of FIG. 7E, the relative position of the component 21 with respect to the component 22 is “t1”. In the case of FIG. 8D, the relative position of the component 21 with respect to the component 22 is “t2”.

  In addition, it can be said that the collation part 42c is performing the process which pinpoints the position and attitude | position of a large component of a test target object, and specifies the position and attitude | position of a next small component. For example, when the parts 21 and 22 of the inspection target 71 are incorporated as shifted as shown in FIGS. 7A and 8A, the collation unit 42c configures the inspection target 71. The position and orientation of 22 are acquired, and then the position and orientation of the component 21 are acquired. Therefore, as described above, the calculation unit 42d can calculate the relative position of each component constituting the inspection target based on the position and orientation of the inspection target acquired by the matching unit 42c.

  Returning to the description of FIG. Next, the inspection unit 42e determines whether or not the relative position calculated by the calculation unit 42d in step S6 is smaller than the threshold Th (step S7). When the relative position calculated by the calculation unit 42d is smaller than the threshold Th, the inspection unit 42e proceeds to the process of step S8. When the relative position calculated by the calculation unit 42d is not smaller than the threshold Th, the inspection unit 42e proceeds to the process of step S9.

  Next, when it is determined in step S4 that the image data of the inspection target has been erased by a predetermined amount or in step S7, the inspection unit 42e determines that the relative position is smaller than the threshold value Th. Is determined to be a proper product (step S8).

  Next, when it is determined in step S7 that the relative position is not smaller than the threshold value Th, the inspection unit 42e determines that the inspection target is not a proper product (step S9).

  For example, the relative position “t1” shown in FIG. 7D is assumed to be smaller than the threshold Th (t1 <Th). That is, it is assumed that the deviation “t1” of the component 21 with respect to the component 22 of the inspection target 71 shown in FIG. 7A captured by the imaging device 19 is smaller than the threshold Th. In this case, the inspection object 71 shown in FIG. 7A imaged by the imaging device 19 is determined to be a proper product. Further, it is assumed that the relative position “t2” shown in FIG. 8D is larger than the threshold Th (t2> Th). That is, it is assumed that the deviation “t2” of the component 21 with respect to the component 22 of the inspection object 71 shown in FIG. 8A captured by the imaging device 19 is larger than the threshold Th. In this case, it is determined that the inspection object 71 shown in FIG. 8A imaged by the imaging device 19 is not an appropriate product. As described above, the inspection unit 42e inspects the inspection object 71 based on the relative position calculated by the calculation unit 42d.

  Each processing unit in the flowchart described above is divided according to the main processing contents in order to facilitate understanding of the processing of the image processing unit 42. The present invention is not limited by the way of dividing the processing unit or the name. The processing of the image processing unit 42 can be divided into more processing units according to the processing content. Moreover, it can also divide | segment so that one process unit may contain many processes.

  6 to 8, the operation of the image processing unit 42 has been described by taking the one-dimensional displacement (horizontal displacement in the drawing) of the components of the assembly as an example. The assembly can be inspected by detecting displacement (horizontal direction, vertical direction in the drawing, and displacement due to rotation (posture)). Hereinafter, the deviation of the postures of the components of the assembly will be described. Note that detection and inspection of component displacement in the vertical direction is the same as the horizontal displacement processing (similar to the processing described with reference to FIGS. 6 to 8), and thus description thereof is omitted.

  FIG. 9 is a diagram for explaining detection of deviation due to rotation. The image acquisition unit 42b acquires the image data of the assembly of the component 21 and the component 22 captured by the imaging device 19 by the same process as step S1 in FIG.

  FIG. 9A shows an image of the inspection object 71 (an assembly of the component 21 and the component 22) acquired by the image acquisition unit 42b. An inspection object 71 shown in FIG. 9A shows an example in which the component 21 is shifted and rotated counterclockwise by an angle θ with respect to the component 22 about the direction perpendicular to the paper surface.

  Next, the collation unit 42c acquires the position and orientation of the inspection object included in the image data acquired by the image acquisition unit 42b by the same process as step S2 in FIG.

  FIG. 9B shows an example of the result of obtaining the position and orientation of the inspection object 71 by the collation unit 42c. A dotted line 81 shown in FIG. 9B indicates the outline of the teaching data (the outline of the assembly 51 shown in FIG. 3). A dotted line 81 shown in FIG. 9B indicates the position and orientation of the inspection object 71 acquired by the collation unit 42c. A line thinner than the dotted line 81 shown in FIG. 9B indicates the inspection object 71.

  In the case of FIG. 9B, the outline of the teaching data most closely matches the part 22 of the inspection object 71 included in the image data, as indicated by a dotted line 81. This is because the contour of the part 22 of the teaching data is larger than the contour of the part 21 and the contour of the part 52 of the teaching data shown in FIG. This is because the degree of coincidence of the contour is higher when it is superimposed on the contour of the component 22. Therefore, in the case of FIG. 9B, the collation unit 42c specifies that the inspection object 71 exists at the position and posture indicated by the dotted line 81 on the image data. Then, the collation unit 42c acquires the position and orientation of the teaching data (the position and orientation of the dotted line 81) when it is determined that the inspection object 71 exists as the position and orientation of the inspection object 71.

  Next, the collation unit 42c erases the portion corresponding to the teaching data of the inspection target whose position and orientation have been acquired by the same processing as step S3 in FIG. For example, the collation unit 42c erases a portion of the inspection object included in the image data that overlaps the teaching data when the inspection object 71 is identified as being present by collating the image data with the teaching data. .

  For example, in the case of FIG. 9B, the position and posture of the inspection object 71 are acquired at the position and posture of the dotted line 81. In FIG. 9B, the contour of the part 22 of the inspection object 71 included in the image data is entirely overlapped with the contour of the teaching data (dotted line 81), and the contour of the part 21 is partially taught. It overlaps with the outline of the data. Therefore, the inspection object 71 included in the image data is partially erased as shown in FIG.

  When the inspection object is erased by a predetermined number of pixels or more by the above erasing process, the inspection unit 42e determines that the inspection object is a proper product and ends the process. This processing corresponds to the processing in steps S4 and S8 in FIG. On the other hand, when the inspection object is not erased by a predetermined number of pixels or more, the collation unit 42c collates the image data of the inspection object that has been partially erased with the teaching data, and the inspection object that has been partially erased. Get position and orientation. This processing corresponds to the processing in steps S4 and S5 in FIG.

  FIG. 9D shows an example of the result of acquiring the position and orientation of the inspection object 71 from which the partial contour has been deleted by the collating unit 42c. A dotted line 82 shown in FIG. 9D indicates the outline of the teaching data (the outline of the assembly 51 shown in FIG. 3). Further, a dotted line 82 shown in FIG. 9D indicates the position and orientation of the inspection target 71 obtained by the collating unit 42c and from which the partial outline has been deleted. A line thinner than the dotted line 82 shown in FIG. 9B indicates the inspection object 71 (part 21 shown in FIG. 9C) from which a part of the outline has been erased.

  In the case of FIG. 9D, the outline of the teaching data most closely matches the part 21 of the inspection object 71 included in the image data, as indicated by a dotted line 82. Therefore, in the case of FIG. 9D, the collation unit 42c specifies that the inspection object 71 from which a part of the contour is erased exists at the position and orientation (rotation) indicated by the dotted line 82 on the image data. . Then, the collation unit 42c collates the image data with the teaching data, and determines the position and orientation of the teaching data when it is determined that the inspection object 71 whose partial outline has been erased exists (FIG. 9D). Are obtained as the position and orientation of the inspection object 71.

  The collation unit 42c acquires, for example, the position of the center of gravity of the teaching data when it is specified that the inspection object 71 exists, and the rotation of the center of gravity of the teaching data is obtained as the position of the inspection object 71. You may acquire as an attitude | position of the test target object 71. FIG.

  Next, the calculation unit 42d calculates the relative positions of the parts of the inspection target based on the position and orientation of the inspection target acquired by the matching unit 42c by the same process as step S6 in FIG. For example, the calculation unit 42d sets the combination based on the position and orientation of the inspection object acquired by the collation unit 42c for the first time and the position and orientation of the inspection object acquired for the second time by erasing a partial outline. The relative position of the three-dimensional component is calculated.

  For example, a dotted line 81 shown in FIG. 9 (e) corresponds to the dotted line 81 shown in FIG. 9 (b) and indicates the position and orientation of the inspection object 71 acquired for the first time. A dotted line 82 shown in FIG. 9 (e) corresponds to the dotted line 82 shown in FIG. 9 (d) and indicates the position and orientation of the inspection object 71 acquired for the second time. The calculation unit 42d calculates the position and orientation of the inspection object 71 acquired at the first time indicated by the dotted line 81 in FIG. 9E and the inspection object acquired at the second time indicated by the dotted line 82 in FIG. 9E. Based on the position and orientation of 71, the relative position of the component 21 with respect to the component 22 is calculated. For example, the calculation unit 42d calculates the part 22 based on the center-of-gravity position 81a and the center-of-gravity rotation 81b of the inspection target 71 acquired at the first time and the center-of-gravity position 82a and the center-of-gravity rotation 82b of the inspection target 71 acquired at the second time. The relative position of the component 21 with respect to is calculated. In the case of FIG. 9D, the relative position of the component 21 with respect to the component 22 is “angle θ”.

  Next, the inspection unit 42e determines whether or not the relative position calculated by the calculation unit 42d is smaller than the threshold Th. For example, when the relative position “angle θ” calculated by the calculation unit 42d is smaller than a threshold Th (Th is indicated by an angle), the inspection unit 42e determines that the inspection target 71 is a proper product. This processing corresponds to the processing in steps S7 and S8 in FIG. On the other hand, when the relative position calculated by the calculation unit 42d is not smaller than the threshold Th, the inspection unit 42e determines that the inspection target 71 is not an appropriate product. This processing corresponds to the processing in steps S7 and S9 in FIG. Through the above processing, the image processing unit 42 can detect the deviation due to the posture of the parts of the assembly and inspect the assembly.

If the two axes of the two-dimensional orthogonal coordinate system are, for example, “x” and “y”, respectively, and the rotation in the coordinate system is represented by “θ”, the position and orientation of the inspection object is 3 It can be represented by two parameters “x, y, θ”. Here, assuming that the position and orientation of the inspection object based on the first search result is “T _A ”, and the position and orientation of the inspection object based on the second search result is “T _B ”, “T _A ” The position “ _B ” and the posture “T” (relative position) are expressed by the following equation (1).

T = T _B · T _A ⁻¹ (1)

Here, “T”, “T _A ”, and “T _B ” are indicated by a matrix including parameters “x, y, θ” as elements. “T _A ⁻¹ ” indicates an inverse matrix of “T _A ”.

That is, the collation unit 42c acquires the position and orientation “T _A ” of the inspection object by the first collation process, and the position of the inspection object from which the portion overlapping the teaching data is erased by the second collation process. And the posture “T _B ” is acquired. Then, the calculation unit 42d calculates the relative position T between the parts constituting the inspection object by the equation (1), and the inspection unit 42e calculates the relative position T and the threshold Th calculated by the calculation unit 42d. Are compared to determine whether the inspection object is a proper product.

  In the above description, the case where there are two parts in the assembly has been described. However, even if there are three or more parts, the same process as described above may be executed. For example, even when there are three or more parts, the image processing unit 42 compares the image data of the inspection object with the teaching data, and determines the contour that most closely matches the image data of the teaching data for the first inspection. The position and orientation of the object. The image processing unit 42 erases the contour of the inspection object that overlaps the teaching data, compares the image data from which the contour is partially erased with the teaching data, and determines the contour that most closely matches the image data of the teaching data, The position and posture of the second inspection object are assumed. Then, the image processing unit 42 shifts the components constituting the inspection object based on the position and orientation of the inspection object acquired first time and the position and orientation of the inspection object acquired second time. Inspect.

  FIG. 10 is a diagram illustrating an example of teaching data. Table A1 shown in FIG. 10 shows an example of data taught to the robot controller 12 when image processing is performed according to the flow of FIG. When image processing is performed in the flow of FIG. 5, the image data taught to the robot controller 12 is one image of the assembly after assembling the parts, as shown in Table A1.

  The search property shown in Table A1 is a parameter group for setting the operation of the image processing unit 42. For example, the collation unit 42c of the image processing unit 42 performs image outline detection or collation processing based on the search property.

  Search properties include, for example, a search range, an angle search, an edge detail level, and the like as shown in Table A1. The search range is a parameter that specifies whether to search the inspection object in the entire range of the image data or search the inspection object in a partial range of the image data. The angle search is a parameter that specifies, for example, how many times the teaching data is rotated to detect the posture of the assembly or part. The edge detail level is, for example, a parameter that specifies the density of the edge (contour) of the image data.

  Next, processing in the case where teaching data is given to the robot for each part of the conventional assembly will be described.

  FIG. 11 is a diagram for explaining an example in which teaching data is given to the robot controller for each component of the conventional assembly. FIG. 11A and FIG. 11B show a part 91 and a part 92 of the assembly. It is assumed that the image data of the component 91 shown in FIG. 11A and the image data of the component 92 shown in FIG. 11B are stored in advance as teaching data in the storage unit of the robot control device.

  FIG. 12 is a flowchart showing an example of the operation of the robot controller when teaching data is given for each component of the conventional assembly. Hereinafter, the flowchart of FIG. 12 will be described using FIG. 11 as appropriate.

  The robot control apparatus acquires image data of the assembly (inspection target) imaged by the imaging apparatus (step S11). FIG. 11C shows an image 93 of the inspection object acquired by the robot control device.

  Next, the robot control apparatus acquires the position and orientation of each part of the inspection object (step S12). For example, the robot controller collates the teaching data of the component 91 shown in FIG. 11A with the image 93 of the inspection object shown in FIG. 11C, and the position of the component 91 shown in FIG. And get posture. Further, the robot controller collates the teaching data of the part 92 shown in FIG. 11B with the image 93 of the inspection object shown in FIG. 11C, and the position of the part 92 shown in FIG. And search for posture. A dotted line 101 shown in FIG. 11D indicates the position and orientation of the component 91 searched by the robot control device. A dotted line 102 shown in FIG. 10D indicates the position and orientation of the component 92 searched by the robot control device. In addition, a line thinner than the dotted lines 101 and 102 indicates an image 93 of the inspection object.

Next, the robot control device calculates the relative positions of the parts constituting the inspection object (step S13). For example, the robot controller calculates the relative position of the part 91 with respect to the part 92 of the inspection object. For example, assuming that the position and orientation of the part 91 searched by the robot controller is “T _A ” and the position and attitude of the part 92 searched by the robot controller is “T _B ”, the relative position of the part 91 with respect to the part 92 T is represented by the above formula (1).

  Next, the robot controller determines whether or not the relative position calculated in step S13 is smaller than the threshold Th (step S14). For example, the robot controller determines whether the angle θ shown in FIG. 11D is smaller than the threshold Th. If the relative position calculated in step S13 is smaller than the threshold value Th, the robot control apparatus proceeds to the process of step S15. If the relative position calculated in step S13 is not smaller than the threshold Th, the robot control apparatus proceeds to the process of step S16.

  Next, when it is determined in step S14 that the relative position is smaller than the threshold value Th, the robot control apparatus determines that the inspection object is a proper product (step S15).

  Next, when it is determined in step S14 that the relative position is not smaller than the threshold Th, the robot control apparatus determines that the inspection target is not a proper product (step S16).

  FIG. 13 is a diagram for explaining an example of conventional teaching data. Table A2 shown in FIG. 13 shows an example of data taught to the robot controller when image processing is performed according to the flow of FIG. When image processing is performed in the flow of FIG. 12, the image data taught to the robot control device is an image of each of the parts A and B constituting the assembly, as shown in Table A2. That is, in the conventional image processing, the teaching data of each part must be given, but in the image processing of this case, for example, as described in Table A1 of FIG. . The search properties are the same as those in Table A1 shown in FIG.

  According to the above embodiment, the teaching data of the assembly after assembling a plurality of parts may be stored in the storage unit 43, so that the teaching in the inspection of the assembly having a plurality of parts can be facilitated. it can.

  In addition, by comparing the threshold and the deviation due to the position and orientation of each component and inspecting the assembly, it is possible to determine whether the assembly state of the component is good or not with a certain tolerance.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiment. In addition, it is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention. The present invention may be provided as a robot system having a robot and a robot control device or the like separately, or may be provided as a robot including the robot control device or the like in the robot, or provided as a robot control device. May be. The present invention can also be provided as a method for controlling a robot or the like, a program for controlling a robot or the like, and a storage medium storing the program.

  DESCRIPTION OF SYMBOLS 11 Robot, 12 Robot control apparatus, 13 Movable part, 14 Arm, 15 Joint, 16 Link, 17 Force sensor, 18 Hand, 19 Imaging device, 21, 22 Parts, 31 Worktable, 41 Control part, 42 Image processing part , 42a reception unit, 42b image acquisition unit, 42c collation unit, 42d calculation unit, 42e inspection unit, 43 storage unit, 51 assembly, 52, 53 parts, 60 computer, 61 arithmetic unit, 62 main storage unit, 63 auxiliary storage Device, 64 communication I / F, 65 input device, 66 display device, 67 read / write device, 71 inspection object, 81, 82 dotted line, 91, 92 parts, 93 image, 101, 102 dotted line.

Claims (8)

An image acquisition unit for acquiring first image data including an assembly in which the first component and the second component are assembled;
A first collation unit that collates the first image data acquired by the image acquisition unit with second image data including the assembly stored in advance in a storage device;
A second collation unit that collates the third image data obtained by the collation of the first collation unit and the second image data;
A calculation unit that calculates a relative position between the first component and the second component based on a verification result of the first verification unit and a verification result of the second verification unit;
An inspection unit that inspects the assembly based on the relative position calculated by the calculation unit;
An image processing apparatus comprising: The image processing apparatus according to claim 1,
The first collating unit collates the first image data and the second image data, and the assembly included in the first image data includes the assembly included in the second image data. An image processing apparatus, wherein the third image data is obtained by deleting a portion corresponding to an assembly. The image processing apparatus according to claim 2,
The calculation unit includes the position and orientation of the assembly included in the first image data obtained by collating the first image data and the second image data in the first collation unit, Based on the position and orientation of the assembly included in the third image data obtained by collating the third image data and the second image data in the second collation unit, the relative position is calculated. An image processing apparatus characterized by calculating. The image processing apparatus according to any one of claims 1 to 3,
The image processing apparatus, wherein the inspection unit inspects whether the assembly is a non-defective product based on the relative position calculated by the calculation unit. The image processing apparatus according to claim 2,
In the third image data, the inspection unit determines that the assembly is a non-defective product when the outline of the assembly is erased by a predetermined number of pixels or more. An image processing apparatus according to any one of claims 1 to 5,
The image processing apparatus according to claim 1, wherein the relative position includes a shift due to an angle between the first component and the second component. A manipulator for assembling the first part and the second part;
An image acquisition unit for acquiring first image data including an assembly assembled by the manipulator;
A first collation unit that collates the first image data acquired by the image acquisition unit with second image data including the assembly stored in advance in a storage device;
A second collation unit that collates the third image data obtained by the collation of the first collation unit and the second image data;
A calculation unit that calculates a relative position between the first component and the second component based on a verification result of the first verification unit and a verification result of the second verification unit;
An inspection unit that inspects the assembly based on the relative position calculated by the calculation unit;
A robot characterized by comprising: An image acquisition step of acquiring first image data including an assembly in which the first component and the second component are assembled;
A first collation step of collating the first image data acquired by the image acquisition step with second image data including the assembly stored in advance in a storage device;
A second collation step for collating the third image data obtained by the collation in the first collation step with the second image data;
A calculating step for calculating a relative position between the first component and the second component based on the matching result of the first matching step and the matching result of the second matching step;
An inspection step for inspecting the assembly based on the relative position calculated by the calculation step;
An image processing method comprising:

Images