JP7409199B2 - Visual inspection route search method, inspection route search device for visual inspection robot, inspection route search program, and visual inspection robot - Google Patents

Description

The present invention relates to a visual inspection route searching method, an inspection route searching device for a visual inspection robot, an inspection route searching program, and a visual inspection robot.

In visual inspection equipment where the robot has a camera and lighting, and/or the robot has an object, it is useful when illuminating the object from multiple irradiation angles and acquiring images of the object from multiple viewing angles. It is necessary to create a sequence of inspection points for photographing inspection images and to design an inspection work plan in which the camera follows the generated sequence of inspection points. When designing this inspection work plan, it is desirable to shorten the inspection time and efficiently acquire inspection images.

Japanese Patent Application Publication No. 2018-194443

2. Description of the Related Art An invention related to an appearance inspection apparatus that illuminates an object from a plurality of irradiation angles and acquires images of the object from a plurality of viewing angles is disclosed (Patent Document 1). However, this method does not provide an inspection work plan that achieves both inspection work efficiency and coverage of the appearance of the object.

Therefore, it is an object of the present invention to provide an inspection work plan that achieves both inspection work efficiency and coverage of the appearance of an object in an appearance inspection robot.

In order to solve the above-described problems, the visual inspection route searching method of the present invention changes the posture point of gripping at least one of the inspection target object and the camera with a hand to determine the route of the visual inspection robot that inspects the inspection target object. A visual inspection route search method generated in a simulation environment, wherein the determination unit determines, for each of the posture points, polygons that constitute a three-dimensional shape model of the surface of the inspection object that can be inspected by the camera. and a step in which a search unit searches for an inspection route based on a logical sum of the polygon set that can be inspected determined by the determination unit and a cost due to a change in the attitude point. .
The inspection route search device for a visual inspection robot according to the present invention is configured to change the posture point at which at least one of the inspected object and the camera is gripped by a hand and inspect the inspected object for each posture point of the visual inspection robot. a determination unit that determines polygons that can be inspected by the camera among the polygons constituting a three-dimensional shape model of the surface of the inspection object; a logical sum of the set of inspectable polygons determined by the determination unit; and a logical sum of the posture points. The present invention is characterized by comprising a search unit that searches for an inspection route based on the cost due to the change.

The inspection route search program of the present invention has a computer detect the surface of the inspection target for each posture point of a visual inspection robot that inspects the inspection target by changing the posture point at which the inspection target or the camera is gripped by the hand. a step of determining which polygons can be inspected by the camera among the polygons constituting the three-dimensional shape model; a step of searching for an inspection route based on a logical sum of a set of inspectable polygons and a cost due to a change in the posture point; It is for executing.

The appearance inspection robot of the present invention includes a camera, a hand that grips an object to be inspected or the camera, and a control unit that controls the hand so that the relative position of the camera and the object to be inspected is a desired attitude point. a determination unit that determines, at each of the attitude points, polygons that can be inspected by the camera among polygons that constitute a three-dimensional shape model of the surface of the inspection object; and a set of polygons that can be inspected determined by the determination unit. and a search unit that searches for an inspection route based on the logical sum of and the cost due to the change in the attitude point.
Other means will be explained in the detailed description.

According to the present invention, it is possible to provide an inspection work plan that achieves both inspection work efficiency and coverage of the appearance of a target object.

1 is a diagram showing an example of the overall configuration of a visual inspection robot. FIG. 2 is a block diagram showing an example of a hardware configuration of a control device. FIG. 2 is a block diagram showing an example of a functional configuration of a control device. It is a figure which shows the optical specification data of an inspection part. FIG. 3 is a diagram showing testable condition data. FIG. 3 is a diagram showing the relationship between light incident on a polygon and a camera optical axis. FIG. 3 is a diagram showing the relationship between an object to be inspected, a polyhedron, and a camera. 3 is a flowchart (Part 1) of robot motion determination processing. 12 is a flowchart (Part 2) of robot motion determination processing. It is a flowchart of optimal inspection route search processing. FIG. 3 is a diagram showing each inspection node located on the normal line of each face of a polyhedron that includes an object to be inspected. It is a figure which shows the polygon which can be inspected among the polygons which constitute an inspection object. FIG. 2 is a diagram schematically showing a complete graph in which all test nodes are connected. FIG. 7 is a diagram schematically showing a route that traces nodes at which the number of polygons that can be inspected increases at a minimum cost in a complete graph. FIG. 3 is a diagram showing an example of a set of polygons that can be inspected when an object to be inspected is photographed from a certain node. FIG. 7 is a diagram showing an example of a change in a set of testable polygons when an object to be inspected is photographed from another node. 2 is a flowchart (part 1) of robot motion determination processing that takes into account changes in hand gripping posture; 12 is a flowchart (part 2) of robot motion determination processing that takes into account changes in hand gripping posture; It is a flowchart of optimal inspection route search processing. FIG. 7 is a diagram showing line-of-sight incident angle permissible range data and illumination incident angle permissible range data in the case of dark field and bright field. FIG. 3 is a diagram showing the line-of-sight incident angle and the illumination incident angle in the case of dark field and bright field. FIG. 3 is a diagram showing an example of an appearance inspection robot in which a robot arm grips a camera.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the respective figures.
In this embodiment, the simulator divides a shape model of the surface of the inspection object into a plurality of triangular polygons under a simulation environment, and converts the model into a model represented by a set of a plurality of polygons. The simulator further determines polygons that can be inspected from the conditions of the incident angle of illumination and camera viewing angle for each polygon. This allows the simulator to plan the trajectory (inspection point sequence) of the camera and robot arm so as to maximize the number of polygons that can be inspected and minimize the time required for the robot to move each inspection point.

《First embodiment》
FIG. 1 is a diagram showing an example of the overall configuration of a visual inspection robot 1. As shown in FIG.
The appearance inspection robot 1 is a robot that performs an appearance inspection of an object to be inspected 6, and is mainly composed of a gripping means 2, a measuring means 3, a conveyor 41, a vision sensor 42, a conveyor 51, and a control device 7.

The conveyor 41 is a supply means for supplying the inspection object 6. The vision sensor 42 is arranged above the conveyor 41 and recognizes the position and orientation of the inspection object 6 on the conveyor 41. Based on the recognized position and orientation of the test object 6, the test object 6 is gripped by the hand 22 at the tip of the robot arm 21. Note that the supply means is not limited to the example shown in the figure, and may include a container (not shown) in which the objects 6 to be inspected are piled up, a tray (not shown) in which the objects 6 to be inspected are lined up, and a parts feeder (not shown). It may be composed of.

The gripping means 2 is composed of multi-jointed (for example, 6-axis) robot arms 21 and 23 arranged in a double-arm shape on both sides of the column 11. A hand 22 is provided at the tip of the robot arm 21. A hand 24 is provided at the tip of the robot arm 23.

The hands 22 and 24 may have any form as long as they have a structure that allows them to grip the object 6 to be inspected. It is desirable that the structure be exposed. If the hands 22 and 24 have such a structure, the entire surface of the object to be inspected 6 can be measured by changing the grip between the hands 22 and 24 once. The hands 22 and 24 may hold the object 6 to be inspected or may suck the object 6 to be inspected.

The measuring means 3 includes a multi-joint (for example, 6-axis) measuring arm 31 arranged on the upper part of the support column 11, a camera 32, and a light 34. The camera 32 is, for example, fixed to the tip of the measurement arm 31 directly or via another member. Illumination 34 is arranged around the optical axis of camera 32 and is interlocked with camera 32 . However, the present invention is not limited to this, and the camera 32 and the illumination 34 may be fixed to different robot arms.

The camera 32 is equipped with an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The lighting 34 is, for example, a white LED (Light Emitting Diode). When imaging the inspection object 6, the lighting of the illumination 34 is switched, and two-dimensional images with different illumination angles can be acquired.

This appearance inspection robot 1 grips the inspection object 6 with hands 22 and 24 and uses a camera 32 to inspect the inspection object 6 at each posture point while changing the angle of each joint of the robot arms 21 and 23. Photograph and inspect the surface.

The conveyor 51 is a discharge means for discharging the inspection object 6. Note that a plurality of conveyors 51 may be provided so that the inspection object 6 can be classified according to pass/fail inspection or type of defect.

The control device 7 is a computer that controls the operations of the conveyors 41 and 51 and the measuring means 3.
The hardware configuration (FIG. 2) and functional configuration (FIG. 3) of the control device 7 will be described below.

FIG. 2 is a block diagram showing an example of the hardware configuration of the control device 7. As shown in FIG.
As shown in FIG. 2, the control device 7 is a general-purpose computer to which a control section 71, a storage section 72, a communication section 73, an input section 74, a monitor 75, an interface section 76, a UPS 77, etc. are connected via a bus 79. It is realized by However, the configuration is not limited to this, and various configurations may be used depending on the use and purpose.

The control unit 71 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The CPU calls programs stored in the storage unit 72, ROM, recording medium, etc. to a work memory area on the RAM and executes them, and drives and controls each unit and each device connected via the bus 79.

ROM is a non-volatile memory that permanently stores programs and data such as a boot program and BIOS (Basic I/O System). The RAM is a volatile memory that temporarily stores programs, data, etc. loaded from the storage unit 72, ROM, recording medium, etc., and also functions as a work area used by the control unit 71 to perform various processes. .

The storage unit 72 is a HDD (Hard Disk Drive) or the like, and stores programs executed by the control unit 71, data necessary for program execution, an OS (Operating System), and the like. Regarding programs, a control program corresponding to an OS and an application program for causing a computer to execute processes to be described later are stored. Each of these program codes is read by the control unit 71 as needed, transferred to the RAM, and read by the CPU to execute various processes.

In this embodiment, the storage unit 72 stores shape data of the inspection object 6. The storage unit 72 stores an optical system simulator program. The control unit 71 executes the optical system simulator program and inputs the shape data, optical specifications, and testability conditions of the inspection object 6, thereby making it possible to calculate testable polygons in the shape data. There is.

The storage unit 72 further stores a ROS (Robot Operating System) program. Each function of the ROS is realized by the control unit 71 executing the ROS program. By using the simulator function of this ROS, it is possible to automatically generate the paths (inspection point sequence) of the robot arms 21 and 23 and the measurement arm 31. A method for generating an optimal route using these simulator functions will be described later.

The communication unit 73 has a communication control device, a communication port, etc., and is a communication interface that mediates communication between a computer and a network, and controls communication with other computers via the network. The network can be wired or wireless.

The input unit 74 inputs data and includes, for example, an input device such as a keyboard, a pointing device such as a mouse, and a numeric keypad. Via the input unit 74, operation instructions, operation instructions, data input, etc. can be given to the computer. The monitor 75 includes a display device such as a liquid crystal panel, and a logic circuit (such as a video adapter) for realizing the video function of the computer in cooperation with the display device. Note that the input unit 74 and the monitor 75 may be integrated like a touch panel display.

The interface section 76 is a port for connecting peripheral devices to the computer, and the computer sends and receives data to and from the peripheral devices via the interface section 76. The interface unit 76 is configured with a USB (Universal Serial Bus), a LAN (Local Area Network), IEEE1394, RS-232C, etc., and usually has interfaces for a plurality of peripheral devices. Connection with peripheral devices can be wired or wireless. The control device 7 is connected to the conveyor 41, the conveyor 51, the vision sensor 42, the robot arms 21, 23, the measuring arm 31, the camera 32, etc. via the interface section 76.

A UPS (Uninterruptible Power Supply) 77 is an uninterruptible power supply that continues to supply power even when power is cut off due to a power outage or the like.
The bus 79 is a path for transmitting and receiving control signals, data signals, etc. between each device.

The control device 7 may be configured with one computer, or may be configured such that a plurality of computers cooperate to execute the operations of the visual inspection robot 1. In the following description, an example in which the control device 7 is configured with one computer will be described as a simple configuration example.

FIG. 3 is a block diagram showing an example of the functional configuration of the control device 7. As shown in FIG.
A CAD (Computer Aided Design) system 91 is a unit that stores a three-dimensional shape model 60 (see FIG. 5) of the inspection object 6, converts this shape model 60 into one or more output formats, and outputs it. . This shape model 60 is composed of, for example, a plurality of polygons.

The inspection condition setting section 92 includes an optical specification setting section 921 and an inspection possible condition setting section 922. The optical specification setting section 921 is a section where the user sets optical specifications. The testable condition setting section 922 is a part for inputting testable conditions.

The testable polygon determining unit 93 determines whether all polygons forming the shape model 60 satisfy testable conditions when viewpoints (inspection positions) are provided at each posture point in the simulation environment. The inspectable polygon determining unit 93 determines whether all polygons on the geometric model 60 photographed by the cameras 32 placed at multiple viewpoints around the geometric model 60 of the object to be inspected satisfy the inspection enable condition. Determine. Here, each viewpoint (inspection position) of the camera 32 is realized by changing the angle of each joint of the robot arms 21 and 23 of the visual inspection robot 1 to obtain each posture point.

The inspection route processing unit 94 includes an inspection route search graph generation unit 941 and an optimal inspection route search unit 942. The inspection route search graph generation unit 941 takes each posture point (angle of each joint) of the robot arms 21 and 23 at which the camera 32 is at each inspection position as a node, and generates a complete graph that connects all nodes with edges. generate. The edges of this complete graph constitute the paths in the test.

In other words, the inspection route search graph generation unit 941 uses the combination of the viewpoint of the camera that photographs the three-dimensional shape model 60 of the inspection object 6, the illumination model, the polygon set that can be inspected, and the joint angle of the robot as nodes of the graph. , connect any two nodes with an edge to generate a complete graph.

The optimal inspection route search unit 942 searches on the complete graph for the optimal inspection route that maximizes the logical sum of the set of inspectable polygons determined by the inspectable polygon determination unit 93 and minimizes the cost due to changes in posture points. do. Here, the cost due to a change in attitude point is, for example, the time required to change the attitude point (inspection time), but is not limited to this, and may be other cost indicators such as the energy required to change the attitude point. .

However, the present invention is not limited to this, and the optimum inspection route search unit 942 determines whether the logical sum of the set of inspectable polygons determined by the inspectable polygon determination unit 93 is equal to or greater than a predetermined value, and the cost due to changes in posture points is less than a predetermined value. You may also search on the complete graph for the optimal test path that will satisfy. In other words, the optimal inspection route search unit 942 may search for an optimal inspection route on the complete graph based on the logical sum of a set of polygons that can be inspected and the cost due to changes in posture points.

For example, if the user specifies that the ratio of all polygon sets to the logical sum of polygon sets that can be inspected is 70% or more and that the inspection time is less than 20 minutes, the optimal inspection route search unit 942 completely searches for an optimal route that satisfies these specified conditions. Explore on the graph. In this way, it is desirable that the ratio of the total polygon set to the logical sum of polygon sets that can be inspected and the inspection time can be arbitrarily specified by the user.

The simulator 95 displays the robot moving along the optimal inspection route obtained by the optimal inspection route search unit 942 and an inspection image on the monitor 75 in which polygons determined to be inspectable are highlighted on the images taken at each inspection point. (display section).
The inspection condition setting section 92, the inspectable polygon determining section 93, the inspection route processing section 94, and the simulator 95 function as an inspection route search device for the visual inspection robot 1.

The robot control section 96 includes a parameter setting section 961 and a hand setting section 962.
The parameter setting section 961 is a section for setting the dimensions of the links of the robot and the arrangement relationship of each joint for generating the coordinate transformation matrix of the robot.

The hand setting section 962 is a part that sets the relative positional relationship between the coordinate system of the inspection object 6 when viewed from the hand coordinate system, and how the hands 22 and 24 grip the inspection object 6. The hand setting unit 962 also sets the relative positional relationship of the hand coordinate system to the end effector coordinate system of the robot.

FIG. 4A is a diagram showing optical specification data of the inspection section.
The optical specification data includes field of view items, working distance, and depth of field.
The field of view item indicates the field of view of the camera 32 at a position away from the camera 32 by the working distance, by an area obtained by multiplying the vertical and horizontal lengths of the field of view.

The working distance refers to the distance from the tip of the objective lens to the object when the camera 32 focuses on the object, and in the case of an objective lens that uses a cover glass, the distance from the top surface of the cover glass to the object. means.

The depth of field refers to the distance range on the subject side in which the image captured by the camera 32 appears to be in focus.

FIG. 4B is a diagram showing testable condition data. FIG. 4C is a diagram showing the relationship between the incident angle Θi of the illumination 34 on the surface of the inspection object 6 and the line-of-sight incident angle γi by the camera 32.

The testable condition data includes a line-of-sight incident angle allowable range and an illumination incident angle allowable range. Here, the line-of-sight incident angle permissible range refers to the range in which the line-of-sight incident angle of the optical axis of the camera 32 to the normal line of the surface of the inspection object 6 is allowed. In the simulation environment, this is the angle of incidence of the line of sight to the normal line of the polygon 601.

The allowable range of illumination incident angle refers to the angle of incidence of the illumination light from the illumination 34 to the normal line of the surface forming the inspection object 6, and in the simulation environment, the angle of incidence of the illumination light to the normal line of the polygon 601 forming the inspection object 6. This is the angle of incidence of illumination light. As shown in this inspection possible condition, when the angle of incidence of the illumination light to the normal line of the surface of the inspection object 6 is within a predetermined angle, and the angle of incidence of the line of sight from each camera 32 is within a predetermined angle, , this surface of the object to be inspected 6 can be inspected.

In this embodiment, a simulation is performed in a simulation environment to determine whether each surface of the inspection object 6 satisfies this inspection enable condition data.

FIG. 5 is a diagram showing the relationship between the shape model 60 of the inspection object 6, the polyhedron 61, and the viewpoint 33 in a simulation environment.
The shape model 60 of the inspection object 6 is placed so as to be held between the shape model 25 of the hand 22 . A polyhedron 61 is set so as to surround the shape model 60 of the inspection object 6 and the shape model 25 of the hand 22.

The polyhedron 61 is a truncated icosahedron. The viewpoint 33 is installed on one of the normal lines extending from the center of each surface of the polyhedron 61, and is oriented in the direction in which the center of each surface of the polyhedron 61 is viewed. Note that the polyhedron including the shape model 60 may be any point-symmetric polyhedron, such as a regular tetrahedron, a regular hexahedron, a regular octahedron, a regular dodecahedron, a regular icosahedron,
1585630711021_0
, truncated hexahedron, truncated octahedron, truncated dodecahedron, cuboctahedron, icosahedron, rhombohedral cuboctahedron, rhombic icosahedron dodecahedron, rhombic truncated cuboctahedron, Any of the rhombic truncated dodecahedron, modified cube, and modified dodecahedron may be used.

As a method of arranging the viewpoint 33 around the shape model 60, for example, there is a method of arranging a sphere around the shape model 60 and arranging the viewpoint 33 in the direction represented by the latitude and longitude of the sphere. In this method, the viewpoints 33 are arranged closely in the vicinity of the sphere, resulting in a decrease in inspection efficiency. In addition, near the equator of the sphere, the viewpoints 33 are arranged sparsely, so the inspection accuracy decreases. In this embodiment, since the viewpoints 33 are arranged on the normal line of each face of the point-symmetric polyhedron, the arrangement density is uniform over the entire circumference, and there is no deviation in inspection efficiency or inspection accuracy depending on the direction.

6A and 6B are flowcharts of the robot motion determination process.
In step S10, the CAD system 91 outputs a three-dimensional model of the inspection object 6 in an STL (Stereolithography) format model. As a result, the inspection object 6 is converted into a model represented by a set of a plurality of polygons.

In step S11, the optical specification setting unit 921 of the inspection condition setting unit 92 inputs the optical conditions of the camera 32 that acquires images in the inspection, and the inspection enable condition setting unit 922 inputs the inspection enable conditions under which the object can be inspected. Enter. The optical specifications inputted by the optical specification setting unit 921 include the field of view size, working distance, depth of field, etc., as shown in FIG. 4A.

As shown in FIG. 4B, the testable conditions input by the testable condition setting unit 922 are specifications of a line-of-sight incident angle allowable range and an illumination incident angle allowable range. The testable condition setting section 922 also inputs the arrangement information of the lighting 34.

In step S12, the testable polygon determining unit 93 forms a polyhedron 61 in a simulation environment so as to include the shape model 60 of the inspection object 6 held by the shape model 25 of the hand 22 (see FIG. 8). The shape model 60 held by the shape model 25 and the polyhedron 61 that includes it are shown in FIG. 5 described above.

In step S13, the inspectable polygon determination unit 93 generates each viewpoint 33 on the normal line of each surface of the polyhedron 61 in a simulation environment so as to satisfy the optical specifications of the camera 32 in FIG. 4A. The inspectable polygon determination unit 93 directs each viewpoint 33 to the center of each face of the polyhedron 61.

In step S14, the inspection route search graph generation unit 941 assigns an individual viewpoint number e to each viewpoint 33. The inspection route search graph generation unit 941 arranges the lighting models 35 in the simulation environment according to the designed arrangement information, and assigns an individual lighting number l to each lighting model 35. This is shown in FIG.
The inspection route search graph generation unit 941 further assigns individual numbers p to all polygons forming the shape model 60.

In step S15, the inspectable polygon determining unit 93 determines that all polygons 601 satisfy the inspectable condition and have a shape for each combination of (e, l) where the viewpoint number is e and the illumination number is l. It is determined whether the image can be captured without being hidden behind the model 25.

The testable polygon determining unit 93 stores the result of determining whether or not this polygon satisfies testable conditions as (e, l, p), assuming that the polygon number is p. The testable polygon determining unit 93 collects testable polygon numbers p having the same (e, l), forms a polygon set P, and stores (e, l, P).

In step S16, the inspectable polygon determination unit 93 determines that when the visual inspection robot 1 grips the inspection object 6 with the hands 22 and 24, the camera 32 detects the inspection object 6 for all (e, l, P). The joint angles a of the robot arms 21 and 23 are calculated so that the illumination 34 takes an angle corresponding to the viewpoint number e with respect to the inspection object 6, and the illumination 34 takes an angle corresponding to the illumination number l with respect to the inspection object 6. As a result, the combination (e, l, P, a) is obtained.

In step S17, the inspectable polygon determining unit 93 stores the calculated combination of (e, l, P, a).
In step S18, the inspection route search graph generation unit 941 sets the saved (e, l, P, a) as a graph node and creates a complete graph 8 (see FIG. 10) in which any two nodes are connected by an edge. Configure. The details of this complete graph 8 will be explained in detail with reference to FIG.

In step S19, the optimal test route search unit 942 searches for the optimal test route for the complete graph 8. The optimal test route search unit 942 starts searching from the given starting point node, and ends the route search after evaluating all nodes. Here, evaluating a node means arranging the viewpoint 33 at this node and determining whether or not to inspect the shape model 60. This search process for the optimal inspection route will be explained with reference to FIGS. 7 and 11, which will be described later.

In step S20, the optimal test route search unit 942 traces the route from the last reached node to the starting point node, obtains the optimal test route by reversing the searched route, and stores it.

In step S21, the simulator 95 simulates the motions of the robot arms 21, 23 and hands 22, 24 corresponding to each node of the optimal inspection path in the simulation environment on the monitor 75, and creates inspectable polygons according to this motion. Highlight. 12 and 13, which will be described later, are examples of screens in which testable polygons are highlighted. When the process in step S21 ends, the process in FIG. 6B ends.

FIG. 7 is a flowchart of the optimal inspection route search process.
The optimal inspection route search unit 942 sets the node determined as the starting point node among the nodes constituting the complete graph 8 as shown in FIG. Add to the possible polygon union (S31).

Then, the optimal inspection route search unit 942 calculates the cost of each edge constituting the route by dividing the number of inspection steps at the edge of each route by the increase in the union of inspectable polygons (S32). The inspection man-hour at the edge of each path is, for example, the movement time when the robot arms 21, 23 and hands 22, 24 of the visual inspection robot 1 move along the edge.
The increase in the union of inspectable polygons is the number of polygons that can be newly inspected by moving the robot arms 21, 23 and hands 22, 24 along the edges. Note that even if the edge transitions to an unprocessed node, the optimal inspection route search unit 942 does not use the edge as a route if the union of inspectable polygons does not increase at the node. Thereby, the optimal inspection route search unit 942 can maximize the set of polygons that can be inspected.

The optimal test route search unit 942 searches for an optimal route while calculating the cost of each route by adding up the costs of each edge that constitutes the route from the src node to the unprocessed dest node (S33). This route search ends when all nodes have been processed or when the inspectable polygon union no longer increases no matter which unprocessed node is moved to.
The optimal test route search unit 942 identifies the optimal route that minimizes the total cost of the routes (S34), and ends the optimal test route search process of FIG. 7. This optimal route is the route with the lowest cost and the most efficient inspection route.

FIG. 8 is a diagram showing each inspection node located on the normal line of each face of the polyhedron 61 including the shape model 60 in the simulation environment.
The shape model 60 of the inspection object 6 is placed so as to be held between the shape model 25 of the hand 22 .

Polyhedron 61 includes shape model 60 and shape model 25. The viewpoints 33 are installed on each normal line extending from the center of each surface of the polyhedron 61, and are oriented in a direction in which the center of each surface of the polyhedron 61 is viewed. Furthermore, each lighting model 35 is installed around the polyhedron 61.

FIG. 9 is a diagram showing polygons that can be inspected among the polygons forming the shape model 60 of the inspection object 6 in the simulation environment.
In the simulation environment, a shape model 60 held by the shape model 25, a viewpoint 33, and an illumination model 35 are arranged.

The shape model 60 is configured to include testable polygons 62 that are highlighted (dashed line hatching) and non-testable polygons 63 that are displayed in a dark color. The relationship between the normal of each polygon constituting the shape model 60 and the incident angle of the illumination light of the illumination model 35, and the relationship between the normal of each polygon and the incident angle of the viewpoint 33 are within the tolerance range shown in FIG. 4B. If so, the polygon is determined to be an inspectable polygon.

Either or both of the relationship between the normal of each polygon constituting the shape model 60 and the angle of incidence of the illumination light of the illumination model 35, and the relationship between the normal of each polygon and the angle of incidence of the viewpoint 33 are shown in FIG. 4B. If the polygon is outside the allowable range shown in , the polygon is determined to be an uninspectable polygon.

Note that even if a polygon satisfies the tolerance shown in FIG. 4B, if it is in the shadow of another model, the polygon is determined to be an uninspectable polygon.

FIG. 10 is a diagram schematically showing a complete graph 8 in which all test nodes are connected.
The complete graph 8 includes nodes 81 to 87 and edges that interconnect the nodes 81 to 87.

FIG. 11 is a diagram schematically showing an optimal exploration route that traces nodes in the complete graph 8 where the number of testable polygons increases at the minimum cost.
The optimal search route shown in FIG. 11 starts from node 82 and follows in this order node 87 → node 86 → node 84 → node 83.

The set at node 82 is equal to the set of inspectable polygons at node 82.
The checkable polygon union set when tracing from node 82 to node 87 is the sum of the checkable polygon union set at node 82 and the set of checkable polygons at node 87, and the sum of the checkable polygon union set at node 82 and the set of checkable polygons at node 87. It is the sum of sets.

Furthermore, the union of checkable polygons when tracing from node 87 to node 86 is the sum of the union of checkable polygons at node 87 and the set of checkable polygons at node 86, and the checkable polygon union at nodes 82, 87, and 86 is It is the sum of the set of possible polygons.

Furthermore, the union of checkable polygons when tracing from node 86 to node 84 is the sum of the union of checkable polygons at node 86 and the set of checkable polygons at node 84, and nodes 82, 87, 86, 84 is the sum of the set of testable polygons in .

Furthermore, the union of checkable polygons when tracing from node 84 to node 83 is the sum of the union of checkable polygons at node 84 and the set of checkable polygons at node 83, and nodes 82, 87, 86, 84 , 83 is the sum of the set of testable polygons in .

After tracing from node 82 to node 83, even if you move to either node 81 or 85, which is an unprocessed node, the union of polygons that can be inspected will not increase because the camera 32 can only photograph polygons that have already been inspected. . Therefore, this node 83 becomes the end point.

FIG. 12 shows a screen 68 that highlights inspectable polygons when the geometric model 60 of the inspection object 6 is photographed from a predetermined node.
Displayed on the screen 68 are a shape model 25 of the hands 22 and 24 in a simulation environment, a shape model 60 of the object to be inspected 6, a testable polygon 62, and a non-testable polygon 63.

A viewpoint 33 is installed at a predetermined node, and an illumination model 35 is also installed. Based on the angle between the line of sight from the viewpoint 33 and each polygon constituting the shape model 60, it is possible to simulate whether or not this polygon can be inspected. Further, based on the angle between the illumination light of the illumination model 35 and each polygon constituting the shape model 60, it is possible to simulate whether or not this polygon can be inspected. Here, the polygons 62 that can be inspected are highlighted, and the polygons that cannot be inspected 63 are displayed in a dark color.

FIG. 13 shows a screen 69 that highlights polygons that can be inspected when the inspection object 6 is photographed from another node.
The viewpoint 33 on the screen 69 is set at a different node from the predetermined node shown in FIG. 12, and the lighting model 35 is set at another location. The testable polygon 62 in FIG. 13 is different from the testable polygon 62 in FIG. This allows the user to know the movements of the robot arms 21, 23 and hands 22, 24 on the optimum inspection path, and the polygons on the inspection object 6 that can be inspected due to these movements. The optimal route here is a route that allows inspection to be performed most efficiently. In other words, the optimal inspection route search unit 942 can provide an inspection work plan that achieves both inspection work efficiency and coverage of the appearance of the object.

Even in a real environment, the surface of the inspection object 6 can be comprehensively inspected by operating the camera 32 and/or the inspection object 6 and photographing the inspection object 6 from different nodes as seen from the camera 32. can.

《Second embodiment》
FIGS. 14A and 14B are flowcharts of robot motion determination processing that takes into account changes in the posture of the hand that grips the inspection object.
In step S40, the CAD system 91 outputs a three-dimensional model of the inspection object 6 as an STL (Stereolithography) format model. As a result, the inspection object 6 is converted into a model represented by a set of a plurality of polygons.

In step S41, the optical specification setting unit 921 of the inspection condition setting unit 92 inputs the optical conditions of the camera 32 that acquires images in the inspection, and the inspection enable condition setting unit 922 inputs the inspection enable conditions under which the object can be inspected. Enter. The optical specifications inputted by the optical specification setting unit 921 include the field of view size, working distance, depth of field, etc., as shown in FIG. 4A.

As shown in FIG. 4B, the testable conditions input by the testable condition setting unit 922 are specifications of a line-of-sight incident angle allowable range and an illumination incident angle allowable range. The testable condition setting section 922 also inputs the arrangement information of the lighting 34.

In step S42, the testable polygon determining unit 93 forms a polyhedron 61 in the simulation environment so as to include the shape model 60 of the inspection object 6 held by the shape model 25 of the hand 22. The shape model 60 held by the shape model 25 and the polyhedron 61 that includes it are shown in FIG. 5 described above.

In step S43, the inspectable polygon determination unit 93 generates each viewpoint 33 on the normal line of each surface of the polyhedron 61 for each hand gripping posture in the simulation environment so as to satisfy the optical specifications of the camera 32 in FIG. 4A. . Here, the hand gripping posture refers to, for example, which of the two hands 22 and 24 is used to grip the inspection object 6. The inspectable polygon determination unit 93 directs each viewpoint 33 to the center of each face of the polyhedron 61.

In step S44, the inspection route search graph generation unit 941 assigns an individual viewpoint number e to each viewpoint 33. The inspection route search graph generation unit 941 arranges the lighting models 35 in the simulation environment according to the designed arrangement information, and assigns an individual lighting number l to each lighting model 35. The inspection route search graph generation unit 941 further assigns an individual number h to each hand gripping posture.
The inspection route search graph generation unit 941 further assigns each polygon 601 an individual number p.

In step S45, the inspectable polygon determining unit 93 inspects all polygons 601 for each combination of (e, l, h) where e is the viewpoint number, l is the illumination number, and h is the hand gripping posture. It is determined whether the possible conditions are satisfied and whether the image can be captured without being hidden behind the shape model 25.

The testable polygon determining unit 93 stores the determination result in association with (e, l, h, p), where the polygon number is p. Collect testable polygon numbers p with the same (e, l, h) as a polygon set P, and save (e, l, h, P).

In step S46, the inspectable polygon determining unit 93 determines that for all (e, l, h, P), the robot holding the inspection object 6 with the hands 22, 24 has the camera 32 corresponding to the viewpoint 33 and the illumination The joint angle a of the robot is calculated so that the robot takes a posture of 34. As a result, the combination (e, l, h, P, a) is obtained.

In step S47, the inspectable polygon determining unit 93 stores the calculated (e, l, h, P, a).
In step S48, the inspection route search graph generation unit 941 associates the saved (e, l, h, P, a) with the nodes of the graph to construct a complete graph.

In step S49, the optimal test route search unit 942 searches for the optimal test route for the complete graph 8. The optimal test route search unit 942 starts searching from the given starting point node, and ends the route search after evaluating all nodes. This optimum inspection route search process will be explained with reference to FIG. 15, which will be described later.

In step S50, the optimal inspection route search unit 942 traces the path from the last reached node to the starting point node, and obtains the optimal inspection route in reverse order.

In step S51, the simulator 95 simulates the motion along the optimal inspection path, including the hand-switching motion of the hands 22 and 24, in the simulation environment on the monitor 75, and highlights (emphasizes) polygons that can be inspected according to this motion. When the process in step S51 ends, the process in FIG. 14B ends.

FIG. 15 is a flowchart of the optimal inspection route search process.
The optimal inspection route search unit 942 sets the node determined as the starting point node among the nodes constituting the complete graph as the src node (S60), and adds the polygons that can be inspected at this src node to the union of inspectable polygons (S61). ).

Then, the optimal inspection route search unit 942 calculates a value obtained by dividing the inspection man-hours at the edges of each route by the increase in the union of inspectable polygons, and a penalty value that is added when the hand gripping posture is different from the node passed last time. (cost) is calculated as the cost of each edge (S62). The inspection man-hour at the edge of each path is, for example, the travel time when the robot arms 21, 23 and hands 22, 24 of the visual inspection robot 1 move along the edge. If the hand grasping posture is different between the src node and the dest node, the time required for the joints of the robot arms 21 and 23 to change from the posture point of the src node to the posture point of the dest node, as well as the time between the hands 22 and 24 Additional changeover time will be added. Here, the time required for changing hands between the hand 22 and the hand 24 is the above-mentioned penalty value (cost).
In this way, when there are costs for multiple indicators, their sum or linear weight sum may be used.

The increase in the union of inspectable polygons is the number of polygons that can be newly inspected by moving the robot arms 21, 23 and hands 22, 24 along the edges. Note that even if the edge transitions to an unprocessed node, the optimal inspection route search unit 942 does not select the edge as a route if the union of inspectable polygons does not increase at the node.

The optimal test route search unit 942 adds up the costs of each edge forming the route from the src node to the unprocessed dest node, and searches for the optimal route while calculating the cost of each route (S63). This route search ends when all nodes have been processed or when the inspectable polygon union no longer increases no matter which unprocessed node is moved to.
The optimal test route search unit 942 identifies the optimal route that minimizes the total cost of the routes (S64), and ends the optimal test route search process of FIG. 15. Thereby, the optimal inspection route search unit 942 can provide an inspection work plan that achieves both inspection work efficiency and coverage of the appearance of the object.

《Third embodiment》
In the third embodiment, the line-of-sight incident angle allowable range and the illumination incident angle allowable range are processed separately for bright-field observation and dark-field observation.

FIG. 16 is a diagram showing line-of-sight incident angle permissible range data and illumination incident angle permissible range data for dark field and bright field. FIG. 17 is a diagram showing the line-of-sight incident angle and illumination incident angle in the case of dark field and bright field.

Bright-field observation is a method of observing a sample by illuminating it from near the normal line with uniform light. In FIG. 16, when the illumination incident angle is from 0 degrees to Θ ₁ degree, it is defined as the allowable range for bright field observation. This is shown by the hatched area in FIG.

Dark-field observation is a method in which a sample is obliquely illuminated and observed using scattered light or reflected light from the sample. This method is suitable for transparent samples because defective parts of the sample appear to shine in the pitch-black field of view. In FIG. 16, when the illumination incident angle is from Θ ₁ degree to Θ ₂ degree, it is defined as the allowable range for dark field observation. This is also shown in FIG.

Here, the processes of the first and second embodiments are performed for bright-field observation and dark-field observation, respectively. Thereby, the area of the sample surface that can be observed from the outside can be made as wide as possible by bright field observation and dark field observation.

《Fourth embodiment》
In the first to third embodiments, the robot arm 21 and hand 22, or the robot arm 23 and hand 24 gripped the inspection object 6. In contrast, in the fourth embodiment, a robot arm and a hand grip the camera.

FIG. 18 is a diagram showing an example of a visual inspection robot in which a robot arm 26 and a hand 27 grip a camera 32.
The appearance inspection robot 1A includes a robot arm 26 and a hand 27. The appearance inspection robot 1A inspects the inspection object 6 for defects by having a robot arm 26 and a hand 27 grip the camera 32 and photographing the surroundings of the inspection object 6 placed on an inspection table.

The testable polygon determining unit 93 sets a polyhedron so as to surround the shape model 60 of the test object 6, and sets each viewpoint on a normal line extending from the center of each face of the polyhedron. The inspection route search graph generation unit 941 constructs a complete graph with each viewpoint as a node. The optimum inspection route search unit 942 then searches for an optimum inspection route.

The robot arm 26 and the hand 27 move the camera 32 to positions and directions corresponding to the respective viewpoints of the optimal inspection path, so that the robot arm 26 and the hand 27 move the camera 32 to the positions and directions corresponding to the respective viewpoints of the optimum inspection path, thereby detecting the inspection target in the same manner as in each of the visual inspection robots 1 of the first to third embodiments. The surface of the object 6 can be suitably inspected.

However, the present invention is not limited to this, and the robot arm and hand may hold the camera, and another robot arm and hand may hold the inspection object, and the inspection object may be inspected while changing the posture points of both. Not done. In other words, the visual inspection robot may be one that inspects the object by changing the posture point at which the hand holds at least one of the camera and the object.

(Modified example)
The present invention is not limited to the embodiments described above, and includes various modifications. For example, the embodiments described above are described in detail to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. It is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is also possible to add, delete, or replace some of the configurations of each embodiment with other configurations.

Part or all of the above configurations, functions, processing units, processing means, etc. may be realized by hardware such as an integrated circuit. Each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be stored in a storage device such as memory, hard disk, SSD (Solid State Drive), or recording medium such as a flash memory card or DVD (Digital Versatile Disk). can.

In each embodiment, the control lines and information lines are those considered necessary for explanation, and not all control lines and information lines are necessarily shown in the product. In reality, almost all components may be considered interconnected.

1 Appearance inspection robot 2 Grasping means 21 Robot arm 22 Hand 23 Robot arm 24 Hand 25 Shape model 26 Robot arm 27 Hand 3 Measuring means 31 Measuring arm 32 Camera 34 Lighting 41 Conveyor 42 Vision sensor 51 Conveyor 6 Inspection object 601 Polygon 60 Shape model 61 Polyhedron 62 Inspectable polygon 63 Uninspectable polygon 7 Control device 71 Control section 72 Storage section 73 Communication section 74 Input section 75 Monitor 76 Interface section 77 UPS
8 Complete graphs 81 to 87, 81a to 87a, 81b to 87b Node 91 CAD system 92 Inspection condition setting section 921 Optical specification setting section 922 Inspectability condition setting section 93 Inspectability polygon determination section (determination section)
94 Inspection route processing unit 941 Inspection route search graph generation unit 942 Optimal inspection route search unit (search unit)
95 Simulator 96 Robot control section 961 Parameter setting section 962 Hand setting section

Claims (14)

A visual inspection route search method for generating a route for a visual inspection robot that inspects an inspection target by changing a posture point of holding at least one of the inspection target and a camera with a hand in a simulation environment , the method comprising:
a determination unit determining, for each attitude point, polygons that can be inspected by the camera among polygons that constitute a three-dimensional shape model of the surface of the inspection object;
a step in which a search unit searches for an inspection route based on a logical sum of the polygon set that can be inspected determined by the determination unit and a cost due to a change in the attitude point;
A visual inspection route searching method characterized by performing the following steps. further performing a step in which the search unit searches for an inspection route in which a logical sum of the polygon sets that can be inspected determined by the determination unit is equal to or greater than a predetermined value, and a cost due to a change in the attitude point is less than a predetermined value;
The visual inspection route searching method according to claim 1, characterized in that: further performing a step in which the search unit searches for an inspection route in which a logical sum of the polygon sets that can be inspected determined by the determination unit is maximum and a cost due to a change in the attitude point is minimized;
The visual inspection route searching method according to claim 1, characterized in that: The appearance inspection robot grips the inspection object with a plurality of hands,
The searching unit further executes a step of searching for an inspection route based on a logical OR of the polygon set that can be inspected determined by the determining unit, and a cost due to a change in the posture point and a cost due to changing the hand.
The visual inspection route searching method according to claim 1, characterized in that: The search unit determines an inspection route in which the logical sum of the polygon sets that can be inspected determined by the determination unit is equal to or greater than a predetermined value, and the sum of the cost due to the change in the posture point and the cost due to the hand change is less than a predetermined value. perform further steps to explore,
5. The visual inspection route searching method according to claim 4. The searching unit searches for an inspection route in which the logical sum of the polygon sets that can be inspected determined by the determining unit is maximum and the sum of the cost due to the change in the posture point and the cost due to changing the hand is minimized. further execute ,
5. The visual inspection route searching method according to claim 4. The searching unit further executes a step of determining whether or not the polygon can be inspected based on the angle between the normal of each polygon and the optical axis of the camera.
The visual inspection route searching method according to any one of claims 1 to 6. The searching unit further executes a step of determining whether or not the polygon can be inspected based on the angle of the illumination light incident on each of the polygons.
The visual inspection route searching method according to any one of claims 1 to 7. Each of the posture points is such that the camera is located on a normal line of each face of a polyhedron that includes the three-dimensional shape model,
The visual inspection route searching method according to any one of claims 1 to 8. Each of the attitude points is such that the camera is located at an intersection of a latitude line and a longitude line of a sphere that includes the three-dimensional shape model,
The visual inspection route searching method according to any one of claims 1 to 8. highlighting polygons that the determination unit has determined can be inspected;
The visual inspection route searching method according to claim 1, characterized in that: An inspection route searching device that generates, in a simulation environment, an inspection point sequence of a visual inspection robot that inspects an inspection object by changing the posture point of gripping at least one of the inspection object and a camera with a hand,
a determining unit that determines, for each of the posture points, polygons that can be inspected by the camera among polygons that constitute a three-dimensional shape model of the surface of the inspection object;
a search unit that searches for an inspection route based on a logical sum of the polygon set that can be inspected determined by the determination unit and a cost due to a change in the attitude point;
An inspection route search device for an appearance inspection robot, comprising: to the computer,
A three-dimensional shape model of the surface of the object to be inspected is constructed for each attitude point of an appearance inspection robot that inspects the object by changing the attitude point at which at least one of the object to be inspected and a camera is gripped by a hand. determining which polygons can be inspected by the camera;
searching for an inspection path based on the logical sum of a set of polygons that can be inspected and the cost due to the change in the attitude point;
An inspection route search program for executing. camera and
a hand that grips an object to be inspected or the camera;
a control unit that controls the hand so that the relative position of the camera and the inspection object is at a desired attitude point;
a determination unit that determines, at each of the posture points, polygons that can be inspected by the camera among polygons that constitute a three-dimensional shape model of the surface of the inspection object;
a search unit that searches for an inspection route based on a logical sum of the polygon set that can be inspected determined by the determination unit and a cost due to a change in the attitude point;
Appearance inspection robot equipped with

Images