JP6973233B2 - Image processing system, image processing device and image processing program - Google Patents

Description

In the FA (Factory Automation) field, robots that visually inspect the workpiece to see if the parts are assembled correctly or if they are scratched are widespread. In such a robot, the position data of the measurement points indicating the position and posture of the robot at the time of inspection are taught and stored in advance.

As a method of teaching and storing the position data of the measurement point to the robot, for example, Japanese Patent Application Laid-Open No. 2005-052926 (Patent Document 1) states that "the position of the virtual camera is obtained from the virtual work on the three-dimensional CAD system". The technology is disclosed (see paragraph [0009]).

Further, regarding a technique for performing an accurate visual inspection, for example, Japanese Patent Application Laid-Open No. 8-313225 (Patent Document 2) states that "the camera is moved around the inspection location to change the feature amount such as the brightness and shape of the image. A visual inspection device that "determines a camera position suitable for image processing by measuring" is disclosed (see paragraph [0014]).

Japanese Unexamined Patent Publication No. 2005-0522926 Japanese Unexamined Patent Publication No. 8-313225

However, in both the devices of Patent Document 1 and Patent Document 2, it is not taken into consideration that the position shift when installing the work or the control error of the robot occurs.

The relative positional relationship between the robot and the work when the robot is actually moved to the measurement point is stored in the robot due to a positional shift when installing the work or an error when controlling the robot. There is a risk that the relative positional relationship between the robot and the work will be different. As a result, the camera, which is the imaging unit, may not be in focus at the inspection site. If the image pickup unit is out of focus at the inspection site, the accuracy of the inspection may drop.

Although there is a method in which the user adjusts the position of the robot by using the teaching pendant so that each inspection point is in focus, there is a problem that the user's adjustment man-hours increase as the number of inspection points increases.

Further, although there is a method of focusing on the image pickup unit by the autofocus function, there is a problem that when the autofocus function is used, the optical size changes due to the change of the focal length of the image pickup unit.

Therefore, there is a demand for a technique for arranging an imaging unit at an appropriate position with respect to a measurement point set on the work to perform appropriate imaging.

According to a certain aspect, an image processing system that uses an external image of a work to measure an image of one or a plurality of measurement points preset on the work includes a first image pickup unit that captures an external image and a first image processing unit. Based on the robot that changes the relative positional relationship between the image pickup unit and the work, and the information obtained by imaging the work, the work placement status is specified, and the work placement status is determined according to the specified work placement status. A reference line specifying means for specifying a reference line for each measurement point set in, and a reference line direction in which the first image pickup unit is positioned on the specified reference line and the optical axis of the first image pickup unit is specified. Based on the distance adjusting means that gives a command to the robot to change the distance between the first image pickup unit and the work in the matched state, and the appearance image captured by the first image pickup unit, the first image is obtained. A change in the first feature amount according to a change in the distance between the first image pickup unit and the work and the first feature amount calculation means for calculating the first feature amount regarding the degree of focus of the image pickup unit 1. A distance determining means for determining the distance between the first imaging unit and the work for performing image measurement on the measurement point is included.

Preferably, the image processing system is arranged so as to include at least a part of the work and the robot in the visual field range, and one or a plurality of image pickup units for capturing a three-dimensional image of a subject existing in the visual field range. It further includes a storage means for storing information defining the reference line direction associated with each of the measurement points. The reference line specifying means includes a first search means for searching a portion of the three-dimensional image that matches the shape information of the work, information on the portion searched by the first search means, and a reference stored in the storage means. It includes a reference line determining means for determining a reference line for each measurement point based on the information defining the line direction.

Preferably, the reference line specifying means is a reference of a measurement point corresponding to an image captured by the first image pickup unit in a state where the first image pickup unit is positioned on the reference line determined by the reference line determination means. It further includes a second search means for searching for a portion matching the image, and a correction means for correcting the reference line determined by the reference line determining means based on the information of the portion searched by the second search means. ..

Preferably, the image processing system predetermines the angle formed by the specified reference line and the optical axis of the first image pickup unit from 0 while maintaining the distance between the first image pickup unit and the work. A position adjusting means that gives a command to the robot so as to change the value, and a first image pickup unit for performing image measurement on the measurement point based on the change of the first feature amount according to the change of the angle. Includes a positioning means for determining the relative positional relationship with the work.

Preferably, the image processing system has a second feature amount calculation means for calculating a second feature amount for a preset detection target portion based on an external image imaged by the first image pickup unit, and a first. Presence or absence of a target measurement point or a detection target site in the vicinity of the target measurement point based on the second feature amount calculated in a state where the distance between the image pickup unit and the work is positioned at a distance determined by the distance determination means. Further includes an image measuring means for determining.

Preferably, the reference line specifying means gives a command to the robot to position the first image pickup unit at the next measurement point when the determination of the presence / absence of the detection target portion by the image measuring means is completed.

Preferably, the image processing system further includes a measurement point setting receiving means for displaying the design information of the work and setting one or a plurality of measurement points according to the user operation for the displayed design information.

Preferably, the image processing system calculates the surface shape of each measurement point based on the design information of the work for each of the measurement points set by the measurement point setting receiving means, and each of them is based on the calculated surface shape. It further includes a reference line calculation means for calculating the reference line of the measurement point.

According to another aspect, the image processing device for performing image measurement on one or a plurality of measurement points preset in the work by using the appearance image of the work is a first image pickup unit that captures the appearance image. An interface for receiving information about an external image from the work, an interface for communicating with a robot that changes the relative positional relationship between the work and the first image pickup unit, and an interface for communicating with a robot that changes the relative positional relationship between the work and the first image pickup unit. A reference line specifying means for specifying the placement status and specifying a reference line for each measurement point set on the work according to the placement status of the specified work, and positioning the first image pickup unit on the specified reference line. At the same time, the distance adjusting means for giving a command to the robot to change the distance between the first image pickup unit and the work while the optical axis of the first image pickup unit is aligned with the specified reference line direction. A first feature amount calculating means for calculating a first feature amount regarding the degree of focus of the first image pickup unit based on an external image imaged by the first image pickup unit, and a first image pickup unit. A distance determining means for determining the distance between the first imaging unit and the work for performing image measurement on the measurement point based on the change in the first feature amount according to the change in the distance between the work and the work. And include.

According to another aspect, the image processing program for performing image measurement on one or a plurality of measurement points preset in the work by using the appearance image of the work uses the first image pickup unit in the computer. About the step of imaging the work, the step of specifying the arrangement status of the work based on the information obtained by imaging the work, and each measurement point set in the work according to the arrangement status of the specified work. The first image pickup unit with the step of specifying the reference line, the step of positioning the first image pickup unit on the specified reference line, and the optical axis of the first image pickup unit aligned with the specified reference line direction. The step of giving a command to the robot that changes the relative positional relationship between the first image pickup unit and the work so as to change the distance between the image and the work, and the appearance image captured by the first image pickup unit. Based on the step of calculating the first feature amount regarding the degree of focus of the first image pickup unit and the change of the first feature amount according to the change of the distance between the first image pickup unit and the work. Then, the step of determining the distance between the first image pickup unit and the work for performing image measurement on the measurement point is executed.

In a certain aspect, the imaging unit can be arranged at an appropriate position with respect to the measurement point set on the work to perform appropriate imaging.

The above and other objects, features, aspects and advantages of the present disclosure will become apparent from the following detailed description of the invention as understood in connection with the accompanying drawings.

It is a schematic diagram which shows the basic structure of the image processing system which concerns on 1st Embodiment. It is a schematic diagram which shows the hardware configuration of an image processing system. It is a figure which shows an example of the movement of a robot at the time of performing the process of adjusting the focus of a 2D camera to the measurement point of a work. It is a flowchart of an inspection process. It is a figure which shows an example of the functional structure of an image processing apparatus. It is a flowchart of inspection route generation processing. It is a figure which shows an example of the functional structure of the image processing apparatus which functions in the execution of an inspection path generation process. It is a flowchart of a normal correction process. It is a schematic diagram which shows the functional structure of a normal determination system. It is a flowchart of a focus adjustment process. It is a schematic diagram which shows the functional structure of a focus adjustment system. It is a flowchart of an inspection execution process. It is a schematic diagram which shows the functional structure of an inspection execution system. It is a figure which shows an example of the arc trajectory of a camera. It is a figure which shows an example of the processing when a user inputs an inspection path using an input device. It is a flowchart of the normal determination process which concerns on 2nd Embodiment. It is a schematic diagram which shows the structure of the image processing system which concerns on 3rd Embodiment. It is a schematic diagram which shows the deformation example of a reference line.

Hereinafter, embodiments according to the present invention will be described with reference to the drawings. In the following description, the same parts and components are designated by the same reference numerals. Their names and functions are the same. Therefore, the detailed description of these will not be repeated. In addition, each embodiment and each modification described below may be selectively combined as appropriate.

<First Embodiment>
[A. Image processing system configuration]
FIG. 1 is a schematic diagram showing a basic configuration of an image processing system SYS according to the first embodiment. The image processing system SYS shown in FIG. 1 is a system for inspecting the appearance of the work W by using the appearance image of the work W provided by the line L. In the image processing system SYS, a visual inspection is performed for each measurement point set to one or a plurality of predetermined work Ws.

The image processing system SYS includes an image processing device 2, a robot 210, a two-dimensional camera 310, and a three-dimensional camera 410. Typically, the robot 210 is an articulated robot and may be a SCARA type robot or the like. The two-dimensional camera 310 is attached to the tip of the robot 210. As the robot 210 moves, the position and posture of the two-dimensional camera 310 are changed. Therefore, the robot 210 can change the relative positional relationship between the two-dimensional camera 310 and the work W.

The three-dimensional camera 410 is provided at a predetermined position on the line L, and is arranged so that the imaging field of view includes at least a part of the line L. When the work W is provided by the line L so as to be located within the imaging field of view of the 3D camera 410, the 3D camera 410 can image the work W.

The image processing device 2 includes a management device 100, a robot control unit 200, a two-dimensional image processing device 300, and a three-dimensional image processing device 400. In the first embodiment, the image processing device 2 is composed of four devices, but it may be composed of one device, or may be composed of two or more devices or five or more devices. ..

The management device 100 is connected to each of the robot control unit 200, the two-dimensional image processing device 300, and the three-dimensional image processing device 400 via the network NW. The network NW is, for example, a field network. As an example, EtherCAT (registered trademark), EtherNet / IP (registered trademark), and the like are adopted for the network NW.

Further, the management device 100 is electrically connected to the display device 110 and the input device 120. The robot control unit 200 is electrically connected to the robot 210. The two-dimensional image processing device 300 is electrically connected to the two-dimensional camera 310. The 3D image processing device 400 is electrically connected to the 3D camera 410. It should be noted that each electrically connected device may be integrally configured. That is, the robot 210 and the robot control unit 200 may be integrally configured. The two-dimensional image processing device 300 and the two-dimensional camera 310 may be integrally configured. The three-dimensional image processing device 400 and the three-dimensional camera 410 may be integrally configured.

[B. Image processing system SYS hardware configuration]
FIG. 2 is a schematic diagram showing a hardware configuration of the image processing system SYS. The management device 100 includes a CPU (Central Processing Unit) 130 which is an arithmetic processing unit, a main memory 140 and a hard disk 150 as a storage unit, a display controller 160, a data reader / writer 170, and an input interface (I / F). 181 includes a robot interface (I / F) 182, a two-dimensional camera interface (I / F) 183, and a three-dimensional camera interface (I / F) 184. Each of these parts is connected to each other via a bus 185 so as to be capable of data communication.

The CPU 130 expands a program (code) including an image processing program 151 installed in the hard disk 150 into the main memory 140, and executes these in a predetermined order to perform various calculations.

The main memory 140 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). In addition to the program read from the hard disk 150, the main memory 140 is used for inspection information 1411 for inspecting the appearance of the work W, and image processing sent from the two-dimensional image processing device 300 or the three-dimensional image processing device 400. As a result, information on the processing result of the CPU 130, and the position and orientation of the robot 210, the two-dimensional camera 310, and the three-dimensional camera 410 (coordinate values (X, Y, Z) and angles (θx, θy, θz) on the three-dimensional coordinates). And so on.

The inspection information 1411 includes, for example, three-dimensional CAD information of the work W, the coordinate position of the measurement point of the work W, a normal vector indicating the direction of the reference line for the measurement point, the inspection content to be inspected at the measurement point, and the measurement point. When there are a plurality of them, the inspection order, the movement path of the two-dimensional camera 310 at the time of inspection, and the like are included. Further, the inspection information 1411 is provided for each type of work W. Examples of the type of work W include an engine block and a PET bottle.

Various setting values and the like may be stored in the hard disk 150. In addition to the hard disk 150, or instead of the hard disk 150, a semiconductor storage device such as a flash memory may be adopted.

The display controller 160 is connected to a display, which is a display device 110, and notifies the user of the processing result of the CPU 130 and the like. That is, the display controller 160 is connected to the display device 110 and controls the display on the display device 110.

The data reader / writer 170 mediates data transmission between the CPU 130 and the memory card 171 which is a recording medium. That is, the memory card 171 is distributed in a state in which a program or the like executed by the management device 100 is stored, and the data reader / writer 170 reads the program from the memory card 171. Further, the data reader / writer 170 writes the image processing result sent from the two-dimensional image processing device 300 or the three-dimensional image processing device 400, the processing result of the CPU 130, and the like to the memory card 171 in response to the internal command of the CPU 130. .. The memory card 171 is a general-purpose semiconductor storage device such as CF (Compact Flash (registered trademark)) or SD (Secure Digital), a magnetic storage medium such as a flexible disk, or a CD-ROM (Compact). It consists of an optical storage medium such as Disk Read Only Memory).

The input I / F181 mediates data transmission between the CPU 130 and an input device 120 such as a mouse, keyboard, and touch panel. That is, the input I / F181 receives an operation command given by the user operating the input device 120. The operation command includes, for example, a command for designating a measurement point of the work W, a command for designating the work W, and a command for starting an inspection.

The robot I / F182 mediates data transmission between the CPU 130 and the robot control unit 200. That is, the robot I / F182 is connected to the robot control unit 200. Further, the robot I / F182 gives a command to instruct the robot control unit 200 to move according to an internal command from the CPU 130.

The two-dimensional camera I / F183 mediates data transmission between the CPU 130 and the two-dimensional image processing device 300. That is, the two-dimensional camera I / F183 is connected to the two-dimensional image processing device 300 for processing the two-dimensional image obtained by imaging the work W. Further, the two-dimensional camera I / F183 gives a command to instruct the two-dimensional image processing device 300 to execute various image processing according to an internal command from the CPU 130.

The 3D camera I / F 184 mediates data transmission between the CPU 130 and the 3D image processing device 400. Further, the 3D camera I / F 184 gives a command to instruct the 3D image processing device 400 to execute various image processing according to an internal command from the CPU 130.

The robot control unit 200 controls the operation of the robot 210 according to a command given from the robot I / F 182. The operation of the robot 210 is an operation of changing the position and posture of the two-dimensional camera 310 attached to the robot 210. That is, it can be said that the robot control unit 200 controls the position and posture of the two-dimensional camera 310.

The two-dimensional image processing device 300 includes an image pickup control unit 320, an image processing engine 330, and an input / output I / F 340. Each of these parts is connected to each other via a bus 350 so as to be capable of data communication. The image pickup control unit 320 controls the image pickup operation in the connected two-dimensional camera 310 according to the instruction from the image processing engine 330. The image processing engine 330 executes various image processing instructed by the CPU 130 via the input / output I / F 340. The input / output I / F 340 transmits the results of various image processing executed by the image processing engine 330 to the management device 100 via the two-dimensional camera I / F 183. Further, the input / output I / F 340 receives various instructions from the CPU 130 sent via the two-dimensional camera I / F 183.

The three-dimensional image processing apparatus 400 includes an image pickup control unit 420, an image processing engine 430, and input / output I / F 440. Each of these parts is connected to each other via a bus 450 so as to be capable of data communication. Of the hardware included in the three-dimensional image processing apparatus 400, the hardware having the same name as the hardware included in the two-dimensional image processing apparatus 300 shall have the same function as the hardware having the same name, and the description thereof is omitted. do.

Each of the two-dimensional camera 310 and the three-dimensional camera 410 is an imaging unit that images a subject existing in the imaging field of view. The two-dimensional camera 310 includes an optical system such as a lens and an aperture, and a light receiving element such as a CCD (Charge Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor as main components. The 3D camera 410 is typically an active 3D camera, which is composed of a projector that projects a pattern for measurement and a camera, and the main components of the camera are a lens, an aperture, and the like. It includes an optical system and a light receiving element such as a CCD image sensor or a CMOS image sensor. It may be a passive stereo camera.

The two-dimensional camera 310 takes an image according to a command from the two-dimensional image processing device 300, and outputs the two-dimensional image data obtained by the imaging to the two-dimensional image processing device 300.

The three-dimensional camera 410 takes an image according to a command from the three-dimensional image processing device 400, and outputs the three-dimensional image data obtained by the imaging to the three-dimensional image processing device 400.

[C. Image processing system overview]
Next, the outline of the image processing system SYS will be described. The image processing system SYS is a system for inspecting the appearance of the work W for each one or a plurality of measurement points Wp set in the work W. The processing related to the appearance inspection of the work W executed in the image processing system SYS is the processing of focusing the 2D camera 310 on the measurement point Wp of the work W and the processing of focusing the 2D camera 310 on the measurement point Wp. Includes processing to perform the inspection. The appearance inspection of the work W for each measurement point Wp means that the appearance inspection for the measurement area including the measurement point Wp is performed for each measurement point Wp set on the work W. The measurement point Wp is a point representing the measurement area and corresponds to a reference point for aligning the optical axis of the two-dimensional camera 310.

In the image processing system SYS, the process of adjusting the focus of the 2D camera 310 to the measurement point Wp of the work W and the process of executing the inspection while the focus of the 2D camera 310 is adjusted to the measurement point Wp are repeated. A visual inspection of the entire work W is performed.

The process of adjusting the focus of the 2D camera 310 to the measurement point Wp of the work W includes the process of specifying the reference line for the measurement point Wp and the process of adjusting the focus of the 2D camera 310 to the measurement point Wp on the specified reference line. including. In the image processing system SYS, the two-dimensional camera 310 can be positioned on the specified normal V. Here, the "reference line" is a reference line for setting the orientation of the two-dimensional camera 310, and is a normal line substantially perpendicular to the minute plane of the work W where the measurement point Wp is located. Includes lines and lines arbitrarily set by the user. In the present embodiment, the reference line will be described as the normal line V.

With reference to FIG. 3, a specific movement of the robot 210 when the process of adjusting the focus of the two-dimensional camera 310 to the measurement point Wp of the work W is executed will be described. FIG. 3 is a diagram showing an example of the movement of the robot 210 when the process of adjusting the focus of the two-dimensional camera 310 to the measurement point Wp of the work W is executed.

In FIG. 3, the movement of the robot 210 when the process of positioning the two-dimensional camera 310 on the normal V (step S1) is executed and the process of focusing the two-dimensional camera 310 on the normal V (step). The movement of the robot 210 when S1) is executed is shown.

In step S1, the management device 100 identifies the arrangement status of the work W based on the information obtained by imaging the work W. Here, the arrangement state is a concept including not only the position of the work W but also the direction of the work W. The management device 100 specifies the normal V for each measurement point Wp set in the work W according to the arrangement status of the specified work W. Here, when a plurality of measurement points Wp are set as shown in FIG. 3, the inspection path WL is generated by specifying the normal V. That is, in order to generate the inspection path WL, the management device 100 specifies the arrangement status of the work W, and the normal line for each measurement point Wp set in the work W according to the arrangement status of the specified work W. It is necessary to specify V.

The management device 100 determines the position and orientation of the two-dimensional camera 310 so that the specified normal V and the optical axis O of the two-dimensional camera 310 coincide with each other, and robot controls the information regarding the determined position and orientation of the two-dimensional camera 310. It is transmitted to the unit 200. The robot control unit 200 sends an operation command to the robot 210 based on the received information regarding the position and orientation of the two-dimensional camera 310. As a result, the posture of the robot 210 changes to the posture of the robot 210'shown by the broken line in FIG. 3 (S1), so that the position and posture of the two-dimensional camera 310 is shown by the broken line in FIG. 3 (S1). Change to the position and orientation of the 2D camera 310'.

In step S2, the management device 100 generates a normal trajectory for moving the two-dimensional camera 310 in a state where the determined normal V and the optical axis O of the two-dimensional camera 310 coincide with each other. The management device 100 transmits information regarding the generated normal trajectory and a command instructing an operation to the robot control unit 200. As a result, the robot 210 moves the two-dimensional camera 310 along the direction of the double-headed arrow in FIG. 3 (S2) to change the distance between the two-dimensional camera 310 and the measurement point Wp.

Further, in step S2, the management device 100 instructs the two-dimensional image processing device 300 to calculate the feature amount related to the in-focus degree a of the two-dimensional camera 310 based on the two-dimensional image captured by the two-dimensional camera 310. do. Here, the "focus degree a" is an index indicating how much the two-dimensional camera 310 is in focus. In the following, the feature amount relating to the in-focus degree a is also simply referred to as the in-focus degree a. The management device 100 obtains the in-focus degree a of the two-dimensional camera 310 at each position where the distance between the measurement point Wp and the two-dimensional camera 310 is different. The management device 100 is a two-dimensional camera in which the focus of the two-dimensional camera 310 is aligned with the measurement point Wp based on the change in the in-focus degree a according to the change in the distance between the two-dimensional camera 310 and the measurement point Wp. The distance between the 310 and the measurement point Wp is determined.

In this way, the image processing system SYS can specify the normal V for the set measurement point Wp of the work W according to the arrangement state of the work W provided by the line L. Further, the image processing system SYS can adjust the focus of the two-dimensional camera 310 to the measurement point Wp by changing the distance between the two-dimensional camera 310 and the measurement point Wp. As a result, the focus of the two-dimensional camera 310 can be adjusted to the measurement point Wp without changing the optical system of the two-dimensional camera 310. Therefore, in the image processing system SYS, accurate visual inspection can be performed. Further, in the image processing system SYS, the management device 100 determines the normal V of the measurement point Wp based on the external image, and the two-dimensional camera 310 is moved along the normal V. Based on the degree of focus a, the position where the focus of the two-dimensional camera 310 matches the measurement point Wp is determined. Therefore, in the image processing system SYS, the positional relationship between the position of the work W provided by the line L and the two-dimensional camera 310 can be adjusted.

[D. Inspection process]
The processing related to the appearance inspection of the work W executed in the image processing system SYS is the processing of focusing the 2D camera 310 on the measurement point Wp of the work W and the processing of focusing the 2D camera 310 on the measurement point Wp. Includes processing to perform the inspection. The process of adjusting the focus of the 2D camera 310 to the measurement point Wp of the work W is the process of specifying the normal V of the measurement point and the process of adjusting the focus of the 2D camera 310 to the measurement point Wp on the specified normal V. Including processing.

More specifically, the process of specifying the normal V for the measurement point Wp is a process of specifying the arrangement state of the work W based on the information obtained by imaging the work W, and a process of specifying the arrangement state of the work W according to the specified arrangement state. It includes a process of specifying a normal V for each measurement point Wp set in the work W.

In the first embodiment, the process of specifying the normal V for the measurement point Wp is performed by using the three-dimensional camera 410 and the two-dimensional camera 310. Specifically, the process of specifying the normal V for the measurement point Wp determines the normal V for each measurement point Wp set in the work W based on the three-dimensional image captured by the three-dimensional camera 410. The process includes a process and a process of correcting the determined normal V in a state where the two-dimensional camera 310 is positioned on the determined normal V based on the two-dimensional image captured by the two-dimensional camera 310.

That is, in the first embodiment, the processes for specifying the normal V for the measurement point Wp are (1) a process for specifying the arrangement status of the work W based on the three-dimensional image, and (2) a three-dimensional image. The process of determining the normal V for each measurement point Wp set in the work W according to the arrangement situation specified based on (3) the state in which the two-dimensional camera 310 is positioned on the determined normal V. The process of specifying the arrangement status of the work W based on the two-dimensional image captured by the two-dimensional camera 310, and (4) correcting the determined normal V according to the arrangement status specified based on the two-dimensional image. It consists of processing to be done.

With reference to FIG. 4, the inspection process executed by the CPU 130 of the management device 100 will be described in order to realize the process related to the appearance inspection of the work W executed in the image processing system SYS. FIG. 4 is a flowchart of the inspection process. The CPU 130 executes the inspection process based on the reception of the command for executing the inspection of the work W. The command for executing the inspection of the work W is input by the user from, for example, the input device 120. Further, the command for executing the inspection of the work W is a device for managing the 3D image processing device 400 or the line L according to the work W being provided within the imaging field of view of the 3D camera 410 (not shown). It may be input from. In the first embodiment, the CPU 130 receives a work number that can specify the type of the work W together with a command to execute the inspection of the work W.

In step S100, the CPU 130 identifies the inspection information 1411 corresponding to the work number from the plurality of inspection information 1411 stored in the main memory 140.

In step S200, the CPU 130 identifies the arrangement status of the work W based on the three-dimensional image of the work W captured by the three-dimensional camera 410, and executes the inspection route generation process for generating the inspection route WL. By executing the inspection route generation process by the CPU 130, (1) the process of specifying the arrangement status of the work W based on the three-dimensional image, and (2) the work W according to the arrangement status specified based on the three-dimensional image. The process of determining the normal V for each measurement point Wp set in is performed.

In step S300, the CPU 130 determines whether or not there is an inspection point. Specifically, the CPU 130 determines whether or not there is a measurement point Wp that has not been inspected. If all the measurement points Wp have been inspected (NO in step S300), the CPU 130 ends the inspection process. If there is a measurement point Wp that has not been inspected (YES in step S300), the CPU 130 switches the control to step S310.

In step S310, the CPU 130 determines one measurement point Wp that has not been inspected.

In step S400, the CPU 130 executes the normal correction process. The normal correction process is a process for correcting the normal V of one measurement point Wp that has not been inspected. By executing the normal correction process by the CPU 130, (3) the work W is arranged based on the two-dimensional image captured by the two-dimensional camera 310 in a state where the two-dimensional camera 310 is positioned on the determined normal V. The process of specifying the situation and (4) the process of correcting the determined normal V according to the arrangement status of the specified work W based on the two-dimensional image are performed.

In step S500, the CPU 130 executes the focus adjustment process. The focus adjustment process is a process of adjusting the focus of the two-dimensional camera 310 to the measurement point Wp. When the CPU 130 executes the focus adjustment process, the process of adjusting the focus of the two-dimensional camera 310 to the measurement point Wp on the determined normal V is performed.

In step S600, the CPU 130 executes the inspection execution process. The inspection execution process is a process of executing an inspection at the measurement point Wp. When the CPU 130 executes the inspection execution process, the process of executing the inspection in a state where the focus of the two-dimensional camera 310 is adjusted to the measurement point Wp is performed.

[E. Functional configuration of image processing device]
The entire function of the image processing apparatus 2 will be described with reference to FIG. FIG. 5 is a diagram showing an example of the functional configuration of the image processing device 2. As described above, the image processing device 2 includes a management device 100, a robot control unit 200, a two-dimensional image processing device 300, and a three-dimensional image processing device 400.

The management device 100 includes a main memory 140 which is a storage unit. The main memory 140 includes a storage unit 141 as a functional configuration for storing inspection information 1411. Further, the management device 100 has other functional configurations such as a position determination unit 131, a route generation unit 132, a normal correction unit 133, a normal trajectory generation unit 134, a focus position determination unit 135, an arc trajectory generation unit 136, and an inspection result. It includes a determination unit 137 and an inspection result registration unit 138. The position determination unit 131, the route generation unit 132, the normal correction unit 133, the normal trajectory generation unit 134, the focus position determination unit 135, the arc trajectory generation unit 136, the inspection result determination unit 137, and the inspection result registration unit 138 are typical. Specifically, it is realized by the CPU of the management device 100 executing the program.

As a functional configuration, the robot control unit 200 includes a conversion unit 220 that converts information on the position and orientation of the two-dimensional camera 310 in the world coordinate system into information on the position and orientation of the two-dimensional camera 310 in the robot coordinate system. Specifically, the conversion unit converts the position coordinates of the world coordinate system (Xw, Yw, Zw) into the position coordinates of the robot coordinate system (Xr, Yr, Zr), and also information on the orientation of the world coordinate system (Nw). Is converted into information (Nr) regarding the orientation of the robot coordinate system. Here, the information about the orientation is shown, for example, by a vector. It may be indicated by an angle.

The two-dimensional image processing apparatus 300 includes an image processing engine 330. The image processing engine 330 includes a specific unit 331, a focus degree calculation unit 332, and a feature amount calculation unit 333 as functional configurations.

The three-dimensional image processing apparatus 400 includes an image processing engine 430. The image processing engine 430 includes a specific unit 431 as a functional configuration.

Even if some or all of the functional configurations of the image processing apparatus 2 shown in FIG. 5 are realized by using a hard-wired circuit such as ASL (Application Sub-Layer) or FPGA (Field Programmable Gate Array). good. Note that some or all of the functions of the management device 100, the robot control unit 200, the two-dimensional image processing device 300, and the three-dimensional image processing device 400 may be mounted on another device.

The storage unit 141 stores information about the inspection path WL input from the input device 120. When the information about the inspection path WL is input from the input device 120, the CPU 130 records the information in the storage unit 141. The information stored in the storage unit 141 is not limited to the information input from the input device 120, but includes the information read from the memory card 171 and / and the hard disk 150. The information regarding the inspection path WL includes, for example, the coordinate positions (Xm, Ym, Zm) on the 3D CAD of each measurement point Wp and the normal vector (Nm) for each measurement point Wp.

The position determination unit 131 is information on the position and orientation of the two-dimensional camera 310 sent from each of the route generation unit 132, the normal correction unit 133, the normal trajectory generation unit 134, the focus position determination unit 135, and the arc trajectory generation unit 136. To determine the position and orientation of the 2D camera 310 in the world coordinate system. The position determination unit 131 transmits information regarding the position and orientation of the two-dimensional camera 310 in the world coordinate system to the robot control unit 200.

The specifying unit 431 of the three-dimensional image processing apparatus 400 specifies the arrangement state of the work W provided by the line L from the three-dimensional image. As an example, the specific unit 431 calculates the arrangement status of the work W by three-dimensional image processing such as three-dimensional pattern matching and ICP (Iterative Closest Point). The position of the work W measured by the specific unit 431 is typically represented by the coordinate values (Xc3, Yc3, Zc3) [pixcel] of the camera coordinate system in the three-dimensional camera 410. Further, the orientation of the work W measured by the specific unit 431 is typically the rotation angle (θxc3, θyc3, θzc3) around the x-axis, y-axis, and z-axis of the camera coordinate system in the three-dimensional camera 410 [radian. ] Is represented by. The position and inclination of the work W measured by the specific unit 431 are sent to the management device 100.

The route generation unit 132 inspects based on the position and orientation of the work W in the camera coordinate system of the three-dimensional camera 410 sent from the specific unit 431, the inspection information 1411 of the work W stored in the storage unit 141, and the like. Generate a path WL. The inspection path WL is typically represented by a coordinate position (Xw, Yw, Zw) [mm] and a normal vector (Nw) in the world coordinate system. That is, the path generation unit 132 determines the normal V for each measurement point Wp set in the work W according to the arrangement state specified based on the three-dimensional image.

In addition, the route generation unit 132 manages the start and end of the inspection. Specifically, the inspection path WL is generated based on the instruction to start the inspection. The route generation unit 132 determines the position and orientation (Xw, Yw, Zw, Nw) of the two-dimensional camera 310 based on the generated inspection path WL, and issues an instruction to the position determination unit 131. Further, the path generation unit 132 receives the end signal indicating that the inspection is completed for one measurement point, and the position and orientation of the two-dimensional camera 310 in order to position the two-dimensional camera 310 at the next measurement point Wp ( Xw, Yw, Zw) is determined, and an instruction is given to the position determination unit 131. The route generation unit 132 ends the inspection based on the fact that the end signals are received for all the measurement points Wp.

As described above, in the first embodiment, the route generation unit 132 receives the end signal indicating the end of the inspection and sequentially instructs the robot control unit 200 to move the two-dimensional camera 310. Therefore, it is not necessary for the user to confirm that the inspection at the measurement point is completed and instruct the management device 100 to start the inspection at the next measurement point Wp after confirming. That is, the image processing system SYS according to the first embodiment is provided with the route generation unit 132, so that the user's operability is improved.

The identification unit 331 of the two-dimensional image processing device 300 specifies information for correcting the normal V of the measurement point Wp determined by the path generation unit 132 from the two-dimensional image. As an example, the specific unit 331 calculates information for correcting the normal V of the measurement point Wp by image processing such as pattern matching. Here, the information for correcting the normal V of the measurement point Wp is the information regarding the position of the measurement point Wp. The position of the measurement point Wp calculated by the specific unit 431 is typically the coordinate value (Xc2, Yc2) [pixcel] of the camera coordinate system in the two-dimensional camera 310 and the axis rotation when the normal is the axis. It is represented by the rotation angle θc2 [radian] of. Information regarding the position of the measurement point Wp measured by the specific unit 331 is sent to the management device 100.

The normal correction unit 133 corrects the normal V of the measurement point Wp determined by the route generation unit 132. Specifically, the normal correction unit 133 specifies the normal V of the measurement point Wp based on the information regarding the position of the measurement point Wp measured by the specific unit 331. That is, the normal correction unit 133 indirectly specifies the normal V of the measurement point Wp based on the two-dimensional image. Hereinafter, the normal V of the measurement point Wp specified based on the two-dimensional image is also referred to as “the normal V of the measured measurement point Wp” (actual measurement value). On the other hand, the normal V of the measurement point Wp referred to when positioning the two-dimensional camera 310 is also referred to as "the normal V of the determined measurement point Wp" (designated value).

The normal correction unit 133 calculates the deviation value between the normal V of the measured measurement point Wp and the normal V of the determined measurement point Wp, and the position of the measurement point Wp determined based on the deviation value. Is corrected, and the position of the measurement point Wp is determined again. The normal correction unit 133 determines the position and orientation (Xw, Yw, Zw, Nw) of the two-dimensional camera 310 based on the position of the determined measurement point Wp, and transmits the position and orientation (Xw, Yw, Zw, Nw) to the position determination unit 131. The normal correction unit 133 determines the position of the measurement point Wp and the normal vector based on the fact that the deviation value becomes equal to or less than a predetermined value. The normal correction unit 133 transmits the position of the determined measurement point Wp and the normal vector (Xw, Yw, Zw, Nw) to the normal trajectory generation unit 134.

The normal trajectory generation unit 134 is two-dimensional in a state where the optical axis O of the two-dimensional camera 310 and the normal V for the measurement point Wp are matched based on the position of the received measurement point Wp and the normal vector. A normal trajectory (Xw (t), Yw (t), Zw (t)) of the two-dimensional camera 310 that changes the distance between the camera 310 and the measurement point Wp is generated. The normal trajectory generation unit 134 transmits the generated normal trajectory, the range to be moved, and the like to the position determination unit 131 and the focus position determination unit 135.

The in-focus degree calculation unit 332 calculates the in-focus degree a from the two-dimensional image, and transmits the relationship a (t) with the time t when the two-dimensional image is captured by the two-dimensional camera 310 to the focus position determination unit 135. do.

The focus position determination unit 135 is based on information such as the in-focus degree a (t) of the two-dimensional image transmitted from the in-focus degree calculation unit 332 and the normal orbit transmitted from the normal orbit generation unit 134, and the in-focus degree. Specify the position where a is the highest. The focus position determination unit 135 gives an instruction to generate a normal trajectory by designating a movement range from the specified position to the normal trajectory generation unit 134 as needed. The focus position determination unit 135 determines the focus position (Xw, Yw, Zw) based on the fact that sufficient information for determining the focus position is obtained from the focus degree calculation unit 332, and the arc trajectory generation unit. Send to 136.

The arc orbit generation unit 136 has a normal V for the measurement point and an optical axis of the two-dimensional camera 310 while maintaining the distance between the two-dimensional camera 310 and the measurement point Wp at the focus position sent from the focus position determination unit 135. An arc orbit is generated so that the angle formed with O changes. The arc orbit generation unit 136 describes the arc orbit (Xw (t), Yw (t), Zw (t), with respect to the position and orientation of the two-dimensional camera 310 so that the optical axis O of the two-dimensional camera 310 and the measurement point Wp intersect. Nw (t)) is generated. The arc orbit generation unit 136 transmits the generated arc orbit, the range to be moved, and the like to the position determination unit 131 and the inspection result determination unit 137.

The feature amount calculation unit 333 calculates the feature amount b for executing the inspection determined for each measurement point. The feature amount calculation unit 333 calculates the feature amount b corresponding to the inspection content from the two-dimensional image. The method for calculating the feature amount b may be determined from, for example, the type of the work W and the inspection content. The feature amount calculation unit 333 transmits the relationship b (t) between the calculated feature amount b and the time t when the two-dimensional image is captured by the two-dimensional camera 310 to the inspection result determination unit 137.

The inspection result determination unit 137 specifies a direction in which the feature amount b increases from information such as the feature amount b (t) of the two-dimensional image and the arc trajectory transmitted from the arc trajectory generation unit 136. The direction in which the feature amount b increases is transmitted to the arc orbit generation unit 136. The arc orbit generation unit 136 generates an arc orbit or the like when the two-dimensional camera 310 can be moved in the direction in which the feature amount b increases within the range in which the two-dimensional camera 310 can be moved. When the inspection result determination unit 137 obtains the feature amount b within the range in which the two-dimensional camera 310 can be moved, the inspection result determination unit 137 specifies the maximum value of the feature amount b, and the maximum value is equal to or higher than a predetermined predetermined value. The test result is judged based on whether or not it is. The inspection result determination unit 137 transmits the obtained inspection result to the inspection result registration unit 138.

The inspection result registration unit 138 outputs the inspection result sent from the inspection result determination unit 137 and also transmits the end signal to the route generation unit 132. The output inspection result is stored in a predetermined area of the main memory 140. The inspection result registration unit 138 may store the inspection result in the main memory 140 and display the inspection result on the display device 110.

[F. Inspection route generation process]
The inspection route generation process (step S200, see FIG. 4) executed by the CPU 130 of the management device 100 will be described in detail with reference to FIGS. 6 and 7. FIG. 6 is a flowchart of the inspection route generation process. FIG. 7 is a diagram showing an example of the functional configuration of the image processing apparatus 2 that functions in the execution of the inspection route generation process. In FIG. 7, the shape of the work W is shown in a simplified manner as compared with FIGS. 1 and 3.

When the CPU 130 executes the inspection route generation process, the arrangement status of the work W is specified, and the normal V for each measurement point Wp set in the work W according to the arrangement status of the specified work W. Is identified.

As shown in FIG. 4, the inspection route generation process (step S200) is executed after the inspection information 1411 corresponding to the work number is specified in step S100. The inspection information 1411 is stored, for example, in the storage unit 141 provided in the main memory 140 for each work number. For example, as the inspection information 1411, the recognition CAD model, the coordinate values (Xmi, Ymi, Zmi) and the size of the measurement points Wp on the 3D CAD for each measurement point number (No. i) are set as predetermined values. The normal vector (Nmi), the inspection item, and the reference model image (Ii) are stored in the storage unit 141 (see FIG. 7). Measurement point No. i (i = 1, 2, ...) Indicates the measurement order, and the route generation unit 132 is No. Next to the measurement point of i, No. The measurement point Wp at (i + 1) is controlled to be inspected.

Here, the reference model image (Ii) is a reference image of the measurement point, and the optical axis O of the two-dimensional camera 310 and No. Any image may be used as a reference image included in the imaging region of the two-dimensional camera 310 when the normal V for the measurement point Wp of i coincides with the normal line V. Further, the reference model image (Ii) is typically generated from a recognition CAD model by CG (Computer Graphics) rendering. The reference model image (Ii) may be generated from an image actually captured by the work W.

After the inspection information 1411 of the work W is specified, in step S201, the CPU 130 sends a recognition CAD model of the specified inspection information 1411 to the three-dimensional image processing device 400. The three-dimensional image processing device 400 temporarily stores the recognition CAD model in the storage unit 432 of the image processing engine 430. The three-dimensional image processing apparatus 400 may store the recognition CAD model for each work number in advance. In this case, in step S201, the CPU 130 transmits only the work number to the three-dimensional image processing device 400.

In step S202, the CPU 130 instructs the three-dimensional image processing device 400 to specify the arrangement status of the work W.

A method of specifying the arrangement status of the work W executed by the three-dimensional image processing apparatus 400 will be described with reference to FIG. 7. The specifying unit 431 of the three-dimensional image processing device 400 receives an instruction to specify the arrangement status of the work W, and instructs the imaging control unit 420 to acquire a three-dimensional image. The image pickup control unit 420 stores the acquired three-dimensional image in the storage unit 433 of the image processing engine 430. The specific unit 431 searches for an image area similar to the recognition CAD model stored in the storage unit 432 in the three-dimensional image stored in the storage unit 433. The specific unit 431 calculates the position of the work W in the three-dimensional image (Xc3, Yc3, Zc3) and the orientation of the work W in the three-dimensional image (θxc3, θyc3, θzc3) from the searched image area. The identification unit 431 transmits the position and orientation of the work W in the calculated three-dimensional image to the management device 100.

Returning to FIG. 6, the inspection route generation process will be described. Upon receiving information regarding the position of the work W and the orientation of the work W from the three-dimensional image processing device 400, in step S203, the CPU 130 specifies the arrangement state of the work W in the global coordinate system. Specifically, the conversion unit 139 (see FIG. 7) included in the functional configuration of the CPU 130 changes the values of the camera coordinate system (Xc3, Yc3, Zc3, θxc3, θyc3, θzc3) to the coordinate values (Xw0) of the three-dimensional camera 410. , Yw0, Zw0) and the values of the global coordinate system (Xw, Yw, Zw, θxw, θyw, θzw) based on the predetermined coordinate conversion formula. As a result, as shown in FIG. 7, the inspection path WL for the work W provided by the line L is generated.

In step S204, the CPU 130 generates the inspection path WL by determining the normal V, and returns to the inspection process. Specifically, the route generation unit 132 (see FIG. 7) has coordinates (Xmi, Ymi, Zmi) and normal vectors (Nmi) for each measurement point Wpi among the inspection information 1411 stored in the storage unit 141. The position (Xw, Yw, Zw) and orientation (θxw, θyw, θzw) of the work W in the global coordinate system, and the coordinate values (Xwi, Ywi, Zwi) of the global work coordinate system based on a predetermined conversion formula. Convert to normal vector (Nwi). The normal V is determined by calculating the coordinate value of the measurement point Wp and the normal vector in the global coordinate system. Further, the inspection path WL is also generated by calculating the coordinate value and the normal vector of the measurement point Wp in the global coordinate system. The route generation unit 132 updates the inspection information 1411 of the storage unit 141 with the information of the generated inspection route. The route generation unit 132 stores the updated information in the first storage unit 142 (see FIG. 9) of the main memory 140.

In this way, the CPU 130 executes the inspection route generation process, and the three-dimensional image processing device 400 specifies the arrangement status of the work W based on the three-dimensional image. Further, when the CPU 130 executes the inspection route generation process, the position and normal vector of each measurement point Wp in the world coordinate system are calculated for each measurement point Wp set in the work W. That is, when the CPU 130 executes the inspection route generation process, the normal V for each measurement point Wp set in the work W is determined according to the arrangement status of the specified work W.

[G. Normal correction processing]
The normal correction process (step S400, see FIG. 4) executed by the CPU 130 of the management device 100 will be described in detail with reference to FIGS. 8 and 9. FIG. 8 is a flowchart of the normal correction process. FIG. 9 is a diagram showing an example of the functional configuration of the image processing apparatus 2 that functions in the execution of the normal correction process. In FIG. 9, the shape of the work W is shown in a simplified manner as compared with FIGS. 1 and 3.

In step S401, the CPU 130 sends a reference model image to the two-dimensional image processing device 300. Specifically, the route generation unit 132 identifies the reference model image for the measurement point Wp determined in step S310 from the first storage unit 142 and transmits it to the two-dimensional image processing device 300 via the two-dimensional camera I / F183. do. The two-dimensional image processing device 300 temporarily stores the reference model image in the storage unit 334 of the image processing engine 330. The two-dimensional image processing device 300 may store the reference model image for each measurement point in advance. In this case, in step S401, the CPU 130 transmits only the measurement point number to the two-dimensional image processing device 300.

In step S402, the CPU 130 determines the position and orientation of the two-dimensional camera 310 from the position and normal vector of the measurement point Wp stored in the first storage unit 142. Specifically, the route generation unit 132 specifies the coordinate position and the normal vector of the world coordinate system for the measurement point Wp determined in step S310 from the first storage unit 142. The path generation unit 132 determines the orientation of the two-dimensional camera 310 so that the specified normal vector and the optical axis O of the two-dimensional camera 310 coincide with each other. Further, in the path generation unit 132, the two-dimensional camera 310 is located on the normal V with respect to the measurement point Wp, and the distance between the two-dimensional camera 310 and the measurement point Wp is a predetermined distance. The position of the dimensional camera 310 is determined. Here, the predetermined distance is a distance calculated from the focal length of the two-dimensional camera 310, and is theoretically a distance at which the measurement point is in focus of the two-dimensional camera 310.

In step S403, the CPU 130 instructs the robot control unit 200 to operate. Specifically, the route generation unit 132 transmits the determined position (Xw, Yw, Zw) and orientation (Nw) of the two-dimensional camera 310 to the position determination unit 131. The position determination unit 131 transmits the position and orientation of the two-dimensional camera 310 to the robot control unit 200 via the robot I / F 182 (see FIG. 9).

In step S404, the CPU 130 instructs the two-dimensional image processing device 300 to specify the normal of the measurement point Wp.

A method of specifying the normal of the measurement point Wp executed by the two-dimensional image processing apparatus 300 will be described with reference to FIG. 9. The specifying unit 331 of the two-dimensional image processing device 300 receives an instruction to specify the normal of the measurement point Wp, and instructs the imaging control unit 320 to acquire a two-dimensional image. The image pickup control unit 320 stores the acquired two-dimensional image in the storage unit 335 of the image processing engine 330.

The specific unit 331 searches for an image area similar to the reference model image stored in the storage unit 334 in the two-dimensional image stored in the storage unit 335. The specific unit 331 calculates the position of the measurement point Wp (Xc2, Yc2) in the two-dimensional image and the direction (θc2) of the measurement point Wp in the two-dimensional image from the searched image area. The identification unit 331 transmits the position and orientation of the measurement point Wp in the calculated two-dimensional image to the management device 100.

Returning to FIG. 8, the normal correction process will be described. Upon receiving information regarding the position of the measurement point Wp and the orientation of the measurement point Wp from the two-dimensional image processing apparatus 300, in step S405, the CPU 130 specifies the position of the measurement point Wp and the normal vector of the measurement point Wp in the global coordinate system. .. Specifically, the conversion unit 139 (see FIG. 9) included in the functional configuration of the CPU 130 is a two-dimensional camera in which the position determination unit 131 sends the values (Xc2, Yc2, θc2) of the camera coordinate system to the robot control unit 200. The position (Xw, Yw, Zw) and orientation (Nw) of 310, and the value of the global coordinate system (Xw, Yw, Zw, Nw) are converted based on a predetermined coordinate conversion formula. As a result, the normal V (measured value) of the measured measurement point Wp is specified.

In step S406, the CPU 130 determines the deviation value. The deviation value is a value indicating a deviation between the normal V (measured value) of the measured measurement point Wp and the normal line (designated value) of the determined measurement point. Here, the position and orientation of the two-dimensional camera 310 are determined so that the normal V (designated value) of the determined measurement point Wp and the optical axis O of the two-dimensional camera 310 coincide with each other. Therefore, the deviation value is also a value indicating the deviation between the normal V of the measured measurement point Wp and the optical axis O of the two-dimensional camera 310. The deviation value is represented by, for example, a vector (ΔN). Specifically, the deviation value calculation unit 1330 (see FIG. 9) included in the functional configuration of the normal correction unit 133 controls the position and orientation of the two-dimensional camera 310 and the normal Wp of the measured measurement point Wp. The deviation value (ΔN) is calculated from the line V (actual measurement value). Here, in the first embodiment, the direction (θc2) of the measurement point Wp in the two-dimensional image is the rotation angle around the axis about the normal line. Therefore, the normal vector of the measurement point Wp in the global coordinate system converted by the conversion unit 139 and the normal vector of the determined measurement point match. That is, the deviation between the normal V (measured value) of the measurement point Wp and the normal (specified value) of the determined measurement point is not the deviation caused by the non-matching of the normal vectors, but the deviation of the measurement point Wp. This is a deviation caused by the position coordinates not matching.

In step S407, the CPU 130 determines whether the deviation value is equal to or less than a predetermined first threshold value. When the deviation value is not equal to or less than the first threshold value (NO in step S407), the CPU 130 switches the control to step S408.

In step S408, the CPU 130 calculates the correction amount from the deviation value. Specifically, the correction amount calculation unit 1332 (see FIG. 9) included in the functional configuration of the normal correction unit 133 moves the two-dimensional camera 310 based on the deviation value (ΔN) (ΔX, ΔY, ΔZ, Δθx, Δθy, Δθz) is determined. As a result, the correction amount required to match the optical axis O of the two-dimensional camera 310 with the normal V (measured value) of the measured measurement point Wp is calculated.

In step S409, the CPU 130 determines the position and orientation of the two-dimensional camera 310 from the correction amount. Specifically, the correction amount calculation unit 1332 determines the position and orientation (Xw,) of the two-dimensional camera 310 from the calculated correction amount and the position and orientation of the two-dimensional camera 310 instructed by the position determination unit 131 in step S403. Yw, Zw, Nw) is determined. At this time, the update unit 1331 included in the functional configuration of the normal correction unit 133 corrects the position of the measurement point Wp stored in the first storage unit 142 based on the correction amount, and sets the corrected measurement point Wp to the corrected position. The information in the first storage unit 142 is updated. As a result, the measured normal line V (measured value) of the measured measurement point Wp is stored in the first storage unit 142 as the determined normal line V (indicated value) of the measured point Wp.

The CPU 130 repeats steps S403 to S409 until the deviation value becomes equal to or less than the first threshold value (YES in step S407). When the deviation value is equal to or less than the first threshold value (YES in step S407), the CPU 130 switches the control to step S410.

In step S410, the CPU 130 determines the normal V of the corrected measurement point Wp. Specifically, the information of the first storage unit 142 updated by the update unit 1331 in step S409 is determined as the position and normal vector of the corrected measurement point Wp.

As described above, the specific unit 431 included in the image processing device 2 of the image processing system SYS is an image similar to the recognition CAD model stored in the storage unit 432 in the three-dimensional image captured by the three-dimensional camera 410. Explore the area. The route generation unit 132 included in the image processing device 2 of the image processing system SYS has the information obtained from the image area searched by the specific unit 431 and the coordinates for each measurement point Wp on the 3D CAD stored in the storage unit 141. (Xmi, Ymi, Zmi) and the normal vector (Nmi) are converted into the coordinate values (Xwi, Ywi, Zwi) and the normal vector (Nwi) of the global work coordinate system. As a result, the specific unit 431 determines the normal V for each measurement point Wp set in the work W.

Further, in the image processing system SYS, in the two-dimensional image captured by the two-dimensional camera 310 in a state where the two-dimensional camera 310 is positioned on the normal line V determined by the specific unit 431, the specific unit 331 is a storage unit. An image area similar to the reference model image stored in 334 is searched. The normal correction unit 133 included in the image processing device 2 of the image processing system SYS corrects the normal V determined by the specific unit 431 based on the information obtained from the image area searched by the specific unit 331. Determine the line V.

That is, in the image processing system SYS according to the first embodiment, the normal V for each measurement point Wp set in the work W is determined based on the information obtained from the three-dimensional image, and then further. Correction is made based on the information obtained from the two-dimensional image. That is, the deviation between the optical axis O of the two-dimensional camera 310 and the normal V about the measurement point Wp caused by the control error of the robot 210 can be corrected based on the information obtained from the two-dimensional image. Further, the deviation between the optical axis O of the two-dimensional camera 310 and the normal V about the measurement point Wp caused by the error between the CAD model of the work W and the work W provided by the line L can be obtained from the two-dimensional image. It can be corrected based on the information. Therefore, by using the image processing system SYS according to the first embodiment, the positional relationship between the two-dimensional camera 310 and the work W can be adjusted more accurately.

In the first embodiment, the direction (θc2) of the measurement point Wp in the two-dimensional image is the rotation angle around the axis with the normal as the axis. However, the two-dimensional image processing device 300 searches for an image area similar to the reference model image, and from the distortion of the image in the image area obtained as a result of the search, the plane on which the measurement point Wp is located and the work W are installed. You may find the angle with the surface. In such a case, the CPU 130 can also detect the deviation of the normal vector.

[H. Focus adjustment process]
The focus position adjustment process (step S500, see FIG. 4) executed by the CPU 130 of the management device 100 will be described in detail with reference to FIGS. 10 and 11. FIG. 10 is a flowchart of the focus position adjustment process. FIG. 11 is a diagram showing an example of the functional configuration of the image processing apparatus 2 that functions in the execution of the focus position adjustment process. In FIG. 11, the shape of the work W is shown in a simplified manner as compared with FIGS. 1 and 3.

In step S501, the CPU 130 generates a trajectory for moving the two-dimensional camera 310. Specifically, the normal orbit generation unit 134 is two-dimensional in a state where the corrected normal V determined in the normal correction process (step S400) and the optical axis O of the two-dimensional camera 310 are matched with each other. A normal trajectory (Xw (t), Yw (t), Zw (t), Nw) of the two-dimensional camera 310 that changes the distance between the camera 310 and the measurement point Wp is generated.

In step S502, the CPU 130 determines the moving range of the two-dimensional camera 310. The initial value of the movement range is determined, and the movement range gradually changes by repeating steps S502 to S506. Specifically, from the relationship between the in-focus degree a obtained in step S505 and the position of the two-dimensional camera 310, the moving range is determined so that the two-dimensional camera 310 moves in the direction in which the in-focus degree a increases.

In step S503, the CPU 130 instructs the robot control unit 200 to move the two-dimensional camera 310. Specifically, the CPU 130 instructs the robot control unit 200 to move the two-dimensional camera 310 at a constant speed within the movement range determined in step S502 and on the orbit of step S501. Here, the speed at which the two-dimensional camera 310 is moved is such that the moving distance in the time required for one image capture is equal to or less than the depth of focus of the two-dimensional camera 310.

In step S504, the CPU 130 instructs the two-dimensional image processing device 300 to calculate the focus degree a. Specifically, the CPU 130 is two-dimensional from the two-dimensional image captured by the two-dimensional camera 310 during the first period in which the two-dimensional camera 310 is moving within the movement range instructed by the CPU 130 to the robot control unit 200 in step S503. The two-dimensional image processing apparatus 300 is instructed to calculate the in-focus degree a of the camera 310.

A method of calculating the degree of focus a executed by the two-dimensional image processing apparatus 300 will be described with reference to FIG. The in-focus degree calculation unit 332 of the two-dimensional image processing device 300 receives an instruction to calculate the in-focus degree a, and instructs the image pickup control unit 320 to acquire the two-dimensional image captured during the first period. do. The image pickup control unit 320 stores the acquired two-dimensional image in the storage unit 335 of the image processing engine 330. The in-focus degree calculation unit 332 executes predetermined image processing for calculating the in-focus degree a of the two-dimensional image stored in the storage unit 335 on the two-dimensional image. For example, the density deviation of the two-dimensional image may be used as the feature amount related to the focusing degree a, and the calculated density deviation may be used as the focusing degree a. The in-focus degree calculation unit 332 transmits to the focus position determination unit 135 the relationship a (t) between the calculated in-focus degree a and the time t in which the two-dimensional image is captured by the two-dimensional camera 310.

Returning to FIG. 10, the focus adjustment process will be described. Upon receiving the relationship a (t) between the in-focus degree a and the time t in which the two-dimensional image is captured by the two-dimensional camera 310 from the two-dimensional image processing apparatus 300, in step S505, the CPU 130 has the in-focus degree a and two-dimensional. The relationship with the position of the camera 310 is stored. Specifically, the CPU 130 specifies the position of the two-dimensional camera 310 at the time t of imaging from the movement range of the two-dimensional camera 310 instructed in step S503. The relationship between the position (Xw (t), Yw (t), Zw (t)) of the specified two-dimensional camera 310 and the in-focus degree a (t) is stored in the second storage unit 143.

In step S506, the CPU 130 determines whether or not the in-focus degree a takes an extreme value within the movement range. If the in-focus degree a does not take an extreme value within the moving range (NO in step S506), the CPU 130 repeats steps S502 to S506 until the in-focus degree a takes an extreme value within the moving range. Here, the CPU 130 sets the start point position of the two-dimensional camera 310 to a position away from or near the work W when determining the movement range in step S502, so that the in-focus degree a becomes an extreme value. Find a position to take. When the starting point position of the two-dimensional camera 310 is set to a position close to the work W, the position is set so that the work W does not come into contact with the two-dimensional camera 310 and the robot 210.

When the degree of focus a takes an extreme value within the movement range (YES in step S506), the CPU 130 switches the control to step S507.

In step S507, the position where the in-focus degree a takes an extreme value is determined as the focus position, and the process returns to the inspection process. Specifically, the focus position determining unit 135 focuses on the position of the two-dimensional camera 310 when an image having the highest focusing degree amax among the focusing degrees a stored in the storage unit is captured. (Xw, Yw, Zw) is determined (see FIG. 11).

In this way, when the CPU 130 executes the focus adjustment process, the process of adjusting the focus of the two-dimensional camera 310 to the measurement point Wp on the determined normal V is performed.

[I. Inspection execution process]
The inspection execution process (step S600, see FIG. 4) executed by the CPU 130 of the management device 100 will be described in detail with reference to FIGS. 12 to 14. FIG. 12 is a flowchart of the inspection execution process. FIG. 13 is a diagram showing an example of the functional configuration of the image processing apparatus 2 that functions in the execution of the inspection execution process. FIG. 14 is a diagram showing an example of an arc trajectory of the two-dimensional camera 310. In FIG. 13, the shape of the work W is shown in a simplified manner as compared with FIGS. 1 and 3.

In step S601, the CPU 130 specifies the inspection item of the measurement point. Specifically, the inspection item is specified based on the correspondence between the measurement point Wp stored in the first storage unit 142 and the inspection item.

In step S602, the CPU 130 determines whether or not the inspection item is inspection A. Here, the inspection A refers to an inspection in which the inspection result may change depending on the direction in which the two-dimensional camera 310 takes an image, such as whether or not the surface of the work W is accurately printed. In the case of inspection A (YES in step S602), the CPU 130 switches control to step S615.

In step S615, the CPU 130 executes the inspection at the focus position. Specifically, the inspection result determination unit 137 executes the inspection A at the focus position determined in the focus position adjustment process (step S500).

If it is not the inspection A (NO in step S602), the CPU 130 generates an arc trajectory centered on the focus position in step S603. Specifically, the arc orbit generation unit 136 maintains the distance between the position of the two-dimensional camera 310 determined by the focus position determination unit 135 and the measurement point Wp, and the normal V and the two-dimensional camera for the measurement point Wp. An arc orbit (Xw (t), Yw (t), Zw (t), Nw (t)) is generated so that the angle formed by the optical axis O of 310 changes from 0 to a predetermined value. Here, the arc orbit will be described with reference to FIG. As shown in FIG. 14A, the arc orbit generation unit 136 is located on the surface of a sphere having a radius r of the distance between the position C of the two-dimensional camera 310 determined by the focus position determination unit 135 and the measurement point Wp. Generate an arc trajectory so that the dimensional camera 310 is located. As shown in FIG. 14 (b), the arc orbit is made to go around the measurement point Wp with the angle between the optical axis O and the normal V of the two-dimensional camera 310 constant, and then to go around again by changing the angle. It may be an orbit that repeats the above. Further, as shown in FIG. 14 (c), the trajectory may be a spiral moving around the measurement point Wp while gradually changing the angle between the optical axis O and the normal line V of the two-dimensional camera 310.

Returning to FIG. 12, the inspection execution process will be described. In step S604, the CPU 130 determines the moving range of the two-dimensional camera 310. The movement range is, for example, a range that goes around at least around the measurement point Wp.

In step S605, the CPU 130 instructs the robot control unit 200 to move the two-dimensional camera 310. Specifically, the CPU 130 instructs the robot control unit 200 to move the two-dimensional camera 310 at a constant speed within the movement range determined in step S604 and on the orbit of step S601.

In step S606, the CPU 130 instructs the two-dimensional image processing apparatus 300 to calculate the feature amount b corresponding to the inspection content. Specifically, the CPU 130 inspects the inspection contents from the two-dimensional image captured by the two-dimensional camera 310 during the second period in which the two-dimensional camera 310 is moving within the movement range instructed by the CPU 130 to the robot control unit 200 in step S605. The two-dimensional image processing apparatus 300 is instructed to calculate a predetermined feature amount b for each.

A method of calculating the feature amount b executed by the two-dimensional image processing apparatus 300 will be described with reference to FIG. The feature amount calculation unit 333 of the two-dimensional image processing device 300 receives an instruction to calculate the feature amount b, and instructs the image pickup control unit 320 to acquire the two-dimensional image captured during the second period. The image pickup control unit 320 stores the acquired two-dimensional image in the storage unit 335 of the image processing engine 330. The feature amount calculation unit 333 executes predetermined image processing for calculating the feature amount b of the two-dimensional image stored in the storage unit 335 on the two-dimensional image. The feature amount calculation unit 333 transmits the relationship b (t) between the calculated feature amount b and the time t when the two-dimensional image is captured by the two-dimensional camera 310 to the inspection result determination unit 137.

The feature amount b is an amount that serves as an index for determining whether or not there is an abnormality in the appearance in the appearance inspection. For example, when a scratch inspection is performed on a work W having a specific pattern such as a striped pattern on the appearance, the feature amount calculation unit 333 is a process for removing the specific pattern on a two-dimensional image. After executing, the labeling process for identifying the area showing the defect is executed. As a result, the feature amount calculation unit 333 calculates the defect area or the defect length as the feature amount b. The feature amount b may be calculated by another method.

For example, the feature amount calculation unit 333 divides the two-dimensional image into a plurality of areas, calculates the degree of defect from the color difference between adjacent areas, and executes the labeling process to specify the area where the degree of defect is equal to or higher than the specified value. Then, the defect area or the defect length may be calculated from the region as the feature amount b. This method is typically used for the purpose of detecting fine scratches.

The feature amount calculation unit 333 may calculate the feature amount b by using the following method. The feature amount calculation unit 333 performs a known edge extraction process on the two-dimensional image, and then performs a known expansion process. The feature amount calculation unit 333 calculates the defect area or the defect length as the feature amount b by executing the labeling process for specifying the region showing the defect after the expansion process. This method is typically used for the purpose of detecting dents, dents, and the like.

The method for calculating the feature amount b may be any method as long as it is a method for extracting an amount that is an index for determining whether or not there is an abnormality in the appearance in the appearance inspection, and is not limited to the disclosed method.

Returning to FIG. 12, the inspection execution process will be described. Upon receiving the relationship b (t) between the feature amount b and the time t when the two-dimensional image is captured by the two-dimensional camera 310 from the two-dimensional image processing device 300, in step S607, the CPU 130 receives the feature amount b and the two-dimensional camera 310. Memorize the relationship with the position of. Specifically, the CPU 130 specifies the position and orientation of the two-dimensional camera 310 at the time t of imaging from the movement range of the two-dimensional camera 310 instructed in step S505. The relationship between the position and orientation (Xw (t), Yw (t), Zw (t), Nw (t)) of the specified two-dimensional camera 310 and the in-focus degree a (t) is stored in the third storage unit 144. do.

In step S608, the CPU 130 determines whether or not there is a direction in which the feature amount b increases. Specifically, the CPU 130 specifies the maximum value and the minimum value of the feature amount b in the movement range, and determines based on whether or not the difference between the maximum value and the minimum value is equal to or greater than the third threshold value. When the difference between the maximum value and the minimum value is equal to or greater than the third threshold value, the CPU 130 states that there is a direction in which the feature amount b increases as a direction for increasing the direction from the position where the minimum value becomes to the position where the maximum value becomes. to decide.

If there is no direction in which the feature amount b increases (NO in step S608), the CPU 130 switches control to step S616.

If there is a direction in which the feature amount b increases (YES in step S609), the CPU 130 generates a trajectory in the direction in which the feature amount b increases in step S609. Specifically, the CPU 130 determines the direction in which the feature amount b increases from the position and orientation information of the two-dimensional camera 310 stored in the third storage unit 144 and the feature amount b, and heads in that direction. Generate an orbit.

In step S610, the CPU 130 determines the moving range of the two-dimensional camera 310. Specifically, the CPU 130 determines the movement range in the direction in which the feature amount b increases.

In step S611, the CPU 130 instructs the robot control unit 200 to move the two-dimensional camera 310. Specifically, the CPU 130 instructs the robot control unit 200 to move the two-dimensional camera 310 at a constant speed in the orbit of step S609 within the movement range determined in step S610.

In step S612, the CPU 130 instructs the two-dimensional image processing apparatus 300 to calculate the feature amount b corresponding to the inspection content.

In step S613, the CPU 130 stores the relationship between the feature amount b and the position of the two-dimensional camera 310.

In step S614, the CPU 130 determines whether or not the movement range is the limit value. Specifically, the CPU 130 determines whether or not the two-dimensional camera 310 can be moved in the increasing direction. Whether it is possible to move the 2D camera 310 in the increasing direction is determined by whether the 2D camera 310 and / and the robot 210 and the work W do not hit each other, or whether the robot 210 has an operating limit. The unit 131 is determined. If the movement range is not the limit value (NO in step S614), the CPU 130 repeats steps S609 to S614 until the movement range reaches the limit value.

When the movement range is the limit value (YES in step S614), when there is no direction in which the feature amount b increases (NO in step S608), after executing the inspection at the focus position (after executing step S615), the CPU 130 , In step S616, the inspection data is transmitted. Specifically, the CPU 130 transmits the inspection result to the inspection result registration unit 138 and registers the inspection result. When there is no direction in which the feature amount b increases (NO in step S608), the CPU 130 calculates the maximum value of the feature amount b stored in the third storage unit 144, and whether the calculated maximum value is equal to or less than the specified value. Judge whether or not. If it is equal to or less than the specified value, the CPU 130 determines that there is no abnormality, and registers no abnormality (circle in FIG. 13) as an inspection result. If it is higher than the specified value, the CPU 130 determines that there is an abnormality and registers an abnormality as an inspection result ((cross mark in FIG. 13). When the moving range is the limit value (YES in step S614), the CPU 130 registers. The highest feature amount bmax among the feature amounts b stored in the third storage unit 144 is specified (see FIG. 13), and it is determined whether or not the feature amount bmax is equal to or less than the specified value. In this case, the CPU 130 determines that there is no abnormality and registers the inspection result. If the value is higher than the specified value, the CPU 130 determines that there is an abnormality and registers the inspection result. After executing the inspection at the focus position (after executing step S615). ), The CPU 130 registers the inspection result at the focus position. For example, the inspection result determination unit 137 registers the inspection result determination unit 137 in the two-dimensional image processing device 300 with the two-dimensional image obtained at the focus position and the reference two-dimensional image. It is instructed to calculate the matching degree with the image, and it is determined whether or not there is an abnormality according to the obtained matching degree.

As described above, in the image processing system SYS according to the first embodiment, the arc orbit generation unit 136 is a method for measuring a measurement point Wp while maintaining a distance between the two-dimensional camera 310 and the measuring point Wp. It generates an arc orbit in which the angle formed by the line V and the optical axis O of the two-dimensional camera 310 changes from 0 to a predetermined value. The CPU 130 instructs the robot control unit 200 to move the two-dimensional camera 310 on the arc orbit generated by the arc orbit generation unit 136. Further, the inspection result determination unit 137 determines the moving range of the two-dimensional camera 310 based on the change of the feature amount b according to the change of the angle formed by the normal line V and the optical axis O. That is, the inspection result determination unit 137 determines the relative positional relationship between the two-dimensional camera 310 and the work W based on the change in the feature amount b according to the change in the angle formed by the normal V and the optical axis O. do.

Since the inspection result determination unit 137 determines the positional relationship between the two-dimensional camera 310 and the work W based on the feature amount b, the inspection can be executed at the optimum position for calculating the feature amount b. That is, by using the image processing system SYS according to the present embodiment, it is possible to execute the inspection at a position where halation is unlikely to occur due to the degree of light hitting the work W. The feature amount b in the present embodiment may be a concentration deviation calculated by the focusing degree calculation unit 332 as the focusing degree a. That is, the feature amount calculation unit 33 3 and the focusing calculation unit 332 may be integrated.

[J. How to register an inspection route using an input device]
A method of registering the inspection path WL using the input device will be described with reference to FIG. FIG. 15 is a diagram showing an example of processing when the user inputs the inspection path WL using the input device 120. FIG. 15A is an example of a screen displayed on the display device 110 when the user inputs a route using the input device 120. FIG. 15B is a diagram for explaining a method of creating a normal vector.

As shown in FIG. 15A, the user places the cursor on the 3D CAD model of the work displayed on the display device 110, and selects the measurement point and the item to be inspected at the inspection point. When a measurement point is selected, a normal for the measurement point is created by the method shown in FIG. 15 (b). The CPU 130 of the management device 100 creates a normal vector based on the 3D CAD data of the work. Specifically, the CPU 130 specifies a microplane surrounded by P1 (x1, y1, z1), P2 (x2, y2, z2) and P3 (x3, y3, z3) where the measurement point Wp2 is located. Based on the specified microplane, the CPU 130 calculates a vector perpendicular to the microplane as a normal vector Nm2.

The CPU 130 registers the information registered by the user by the input device 120 in the storage unit 141. As a result, the inspection information 1411 is stored in the storage unit 141.

As described above, the image processing device 2 of the image processing system SYS in the first embodiment displays the 3D CAD model of the work, which is the design information of the work W, on the display device 110, and the measurement point Wp according to the user operation. To set. Therefore, in the image processing system SYS in the first embodiment, there is a high degree of freedom in setting the measurement point Wp.

Further, the image processing device 2 of the image processing system SYS in the first embodiment can calculate the normal V for each of the measurement points Wp set according to the user operation. Therefore, the user can execute an accurate inspection only by setting the measurement point Wp and instructing the start of the inspection, and the convenience of the user who uses the image processing system SYS is improved.
<Second embodiment>
In the first embodiment, when the normal for each measurement point Wp of the work W is determined, the management device 100 determines when generating a path based on the three-dimensional image, and then is further based on the two-dimensional image. The normal V determined for each measurement point Wp was corrected. However, it is not necessary to use the two-dimensional camera 310. Specifically, by the route generation process described with reference to FIGS. 6 and 7, the coordinate position of each measurement point Wp on the 3D CAD of each measurement point and the normal vector for each measurement point Wp are converted into the global coordinate system. The normal V of the measurement point Wp may be determined by the conversion and the calculation of the coordinate position of the measurement point Wp and the normal vector for each measurement point Wp in the global coordinate system. In this case, the management device 100 executes the focus position adjustment process (step S500) without executing the normal correction process (step S400) shown in FIG.

For example, instead of the route generation process (step S200) and the normal correction process (step S400) in the first embodiment, the normal determination process (step S400A) shown in FIG. 16 may be executed. FIG. 16 is a flowchart of the normal determination process according to the second embodiment.

In step S401A, the CPU 130 sends the recognition CAD model of the inspection information 1411 specified to the three-dimensional image processing device 400.

In step S402A, the CPU 130 instructs the three-dimensional image processing device 400 to specify the arrangement status of the work W.

In step S403A, the CPU 130 specifies the arrangement status of the work W in the global coordinate system.

In step S404A, the CPU 130 determines the normal V of each measurement point Wp. Specifically, among the inspection information stored in the storage unit 141, the coordinates (Xmi, Ymi, Zmi) and the normal vector (Nmi) for each measurement point Wp are set to the position (Xw, Yw) of the work W in the global coordinate system. , Zw) and orientation (θxw, θyw, θzw) and converted to the coordinate values (Xwi, Ywi, Zwi) and normal vector (Nwi) of the global work coordinate system based on the predetermined conversion formula, and each measurement. The normal V of the point Wp is determined. The route generation unit 132 updates the inspection information 1411 of the storage unit 141 to the coordinate values and normal vectors in the global coordinate system, and stores them in the first storage unit 142 (see FIG. 9).

Here, in step S501 of the focus adjustment process in the second embodiment, the trajectory of the two-dimensional camera 310 is generated based on the normal V determined by step S404A.

As described above, the specific unit 431 included in the image processing device 2 of the image processing system SYS according to the second embodiment is stored in the storage unit 432 in the three-dimensional image captured by the three-dimensional camera 410. Search for an image area similar to the recognition CAD model. The route generation unit 132 included in the image processing device 2 of the image processing system SYS has the information obtained from the image area searched by the specific unit 431 and the coordinates for each measurement point on the 3D CAD stored in the storage unit 141. Xmi, Ymi, Zmi) and normal vector (Nmi) are converted into coordinate values (Xwi, Ywi, Zwi) and normal vector (Nwi) of the global work coordinate system. As a result, the normal V for each measurement point Wp set in the work W is determined. That is, in the image processing system SYS, the normal V for each measurement point Wp set in the work W can be determined by one imaging, and the normal V can be determined quickly.
<Third embodiment>
In the first and second embodiments, the robot 210 changes the distance between the two-dimensional camera 310 and the work W by moving the position of the two-dimensional camera 310. However, as shown in FIG. 17, the robot 210 may change the distance between the two-dimensional camera 310 and the work W by moving the position of the work W. FIG. 17 is a schematic diagram showing the configuration of the image processing system according to the third embodiment. The robot 210 is typically a SCARA type robot, and can move the work W in the X, Y, and Z directions (coordinate axes shown in FIG. 17) in the world coordinate system. As a result, the two-dimensional camera 310 and the normal of the measurement point of the work W can be aligned with each other, and then the two-dimensional camera 310 can be focused.

Although the SCARA type robot is taken as an example in FIG. 17, if the work W is sufficiently small, the arm type robot may be configured to change not only the position of the work W but also the posture. .. Further, when the work W is sufficiently small, the operating range of the robot 210 is smaller than that in the case of moving the two-dimensional camera 310, so that the robot 210 can be moved faster.

<Modification example>
[Camera for inspection]
In the first to third embodiments, the camera used for the visual inspection is a two-dimensional camera 310. The displacement sensor may be used instead of the two-dimensional camera 310. Displacement sensors include optical disconnection sensors, ultrasonic displacement sensors and the like. By using the displacement sensor, it is possible to detect minute defects as compared with the case of using a two-dimensional camera.

[Lighting device]
In the first to third embodiments, the image processing system SYS is supposed to perform the inspection at a position where halation is unlikely to occur by moving the two-dimensional camera 310 along an arc trajectory. By changing the position or / or direction of the illumination (not shown) that irradiates the work W with light instead of the two-dimensional camera 310, the inspection may be performed in a situation where halation is unlikely to occur. That is, the arc orbit generation unit 136 may generate an orbit for changing the position and / or direction of the illumination, instead of generating an orbit for changing the position and direction of the two-dimensional camera 310.

[Positioning method for 2D camera]
In the first to third embodiments, as a method of determining the position of the two-dimensional camera 310 when performing the inspection, after focusing, the normal V and the optical axis O are formed according to the inspection contents. By changing the angle, it was decided to determine the optimum position as the position of the two-dimensional camera 310 when performing the inspection. However, the optimum position as the position of the two-dimensional camera 310 may be determined by changing the angle formed by the normal V and the optical axis O according to the inspection content while focusing. Specifically, the two-dimensional camera 310 may be slightly moved along the normal orbit, and the angle formed by the normal V and the optical axis O may be changed at the moved position. The image processing device 2 calculates a feature amount indicating the degree of matching between the two-dimensional image and the reference model image, and of the angle formed by the normal V and the optical axis O, the feature amount of the two-dimensional camera 310 is based on the feature amount. The angle of orientation may be determined. After that, the image processing device 2 changes the distance between the work W and the two-dimensional camera 310 while maintaining the determined orientation of the two-dimensional camera 310, and determines the degree of matching between the two-dimensional image and the reference model image. The distance may be determined from the indicated feature amount.

Further, in the first embodiment, the positioning of the two-dimensional camera is performed by using the two-dimensional camera and the three-dimensional camera. Further, in the second embodiment, the positioning of the two-dimensional camera used for the inspection is performed using only the three-dimensional camera. Only the two-dimensional camera may be used to determine each normal V for each measurement point Wp set in the work W.

[About the reference line]
In the present embodiment, the reference line is a perpendicular line perpendicular to the microplane in which the measurement point Wp is located. However, the reference line may be user-configurable. For example, the reference line is a reference line for setting the orientation of the two-dimensional camera 310 so that the workpiece located in the imaging region is located within the depth of field of the two-dimensional camera 310 as much as possible. You may.

FIG. 18 is a schematic diagram showing a modified example of the reference line. The reference line may be set based on, for example, the material of the work W and the direction in which the irradiation light is incident. For example, when the reference line V ₁ of the FIG. 18 (a), the light of the illumination is irradiated along the optical axis of the two-dimensional camera 310, and a material made of the surface of the workpiece W is a specular reflection, It is set to place the camera in a position where specularly reflected light is not incident. Specifically, the reference line V ₁ is a line that passes through the measurement point Wp ₁ and is set so that the perpendicular line is inclined. By doing so, the two-dimensional camera 310 can be arranged in consideration of the direction of the light incident on the measurement region including the measurement point Wp _{1 and reflected from the measurement region.}

Further, the reference line may be set based on the shape of the surface of the work W in the measurement region including the measurement point Wp. For example, the reference line V ₂ in FIG. 18 (a), the reference line V _{3 , the reference line V 5} in FIG. 18 (b), and the reference line V _{6 in} FIG. 18 (c) are each measured points Wp (that is, measured). When the _{vicinity of the point Wp 2} , the measurement point Wp ₃ , the measurement point Wp ₅ , and the measurement point Wp ₆ ) is composed of a plurality of surfaces, each of the plurality of surfaces is included in the imaging field of view of the two-dimensional camera 310. Set. By doing so, even if the surface shape of the work W is complicated, the two-dimensional camera 310 can be arranged in consideration of the shape.

_{Further, the reference line V 4} in FIG. 18 (b) is set to include the region in which the measurement point Wp _{4 is set in the imaging field of view.} Specifically, reference line V ₅ is set to avoid irregularities on the surface of the workpiece W which is an obstacle. By doing so, even if the surface of the work W has irregularities, it is possible to prevent the view from being obstructed by the irregularities in advance.

These reference lines may be calculated by the image processing apparatus 2 in a series of processes for registering the inspection path WL described with reference to FIG. 15, and may be set by the user with reference to the measurement point Wp. You may. Further, the configuration may be such that the user can make fine adjustments after the image processing device 2 calculates.

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present invention is shown by the scope of claims rather than the above description, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

SYS image processing system, 2 image processing device, 100 management device, 110 display device, 120 input device, 131 position determination unit, 132 route generation unit, 133 normal correction unit, 1330 deviation value calculation unit, 1331 update unit, 1332 correction unit. Quantity calculation unit, 134 normal trajectory generation unit, 135 focus position determination unit, 136 arc trajectory generation unit, 137 inspection result determination unit, 138 inspection result registration unit, 139,220 conversion unit, 140 main memory, 141,335,433 Storage unit, 142 1st storage unit, 143 2nd storage unit, 144 3rd storage unit, 150 hard disk, 160 display controller, 170 data reader / writer, 171 memory card, 181 input I / F, 182 robot I / F, 183 2D camera I / F, 184 3D camera I / F, 185,350,450 Bus, 200 Robot control unit, 210 robot, 300 2 2D image processing device, 310 2D camera, 320,420 Imaging control unit , 330,430 Image processing engine, 331,431 Specific unit, 332 Focus calculation unit, 333 Feature amount calculation unit, 334,432 Storage unit, 340,440 Input / output I / F, 400 3D image processing device, 410 3D camera, NW network, Wp measurement point, O optical axis, V normal, W work, WL inspection path, a focus degree, b feature quantity.

Claims (10)

An image processing system that measures an image of one or more measurement points preset for the work using an external image of the work.
The first image pickup unit that captures the appearance image and
A robot that changes the relative positional relationship between the first image pickup unit and the work,
Based on the information obtained by imaging the work, the arrangement status of the work is specified, and the reference line for each measurement point set on the work is specified according to the arrangement status of the specified work. Reference line identification means and
The first image pickup unit and the work are placed in a state where the first image pickup unit is positioned on the specified reference line and the optical axis of the first image pickup unit is aligned with the specified reference line. A distance adjusting means that gives a command to the robot to change the distance between them,
A first feature amount calculating means for calculating a first feature amount regarding the degree of focus of the first image pickup unit based on an appearance image captured by the first image pickup unit.
The first image pickup unit and the first image pickup unit for performing image measurement at the measurement point based on the change in the first feature amount according to the change in the distance between the first image pickup unit and the work. An image processing system including a distance determining means for determining a distance to a work. A second imaging unit that is arranged so as to include at least a part of the work and the robot in the visual field range and captures a three-dimensional image of a subject existing in the visual field range.
Further comprising a storage means for storing information defining a reference line direction associated with each of the one or more measurement points.
The reference line specifying means is
A first search means for searching a portion of the three-dimensional image that matches the shape information of the work,
A reference line determining means for determining a reference line for each measurement point based on the information of the portion searched by the first search means and the information for defining the reference line direction stored in the storage means. The image processing system according to claim 1, which includes. The reference line specifying means is
A portion matching the reference image of the corresponding measurement point with respect to the image captured by the first imaging unit in a state where the first imaging unit is positioned on the reference line determined by the reference line determining means. A second search method to search,
The image processing system according to claim 2, further comprising a correction means for correcting a reference line determined by the reference line determining means based on the information of the portion searched by the second search means. While maintaining the distance between the first image pickup unit and the work, the angle between the specified reference line and the optical axis of the first image pickup unit is changed from 0 to a predetermined value. As described above, the position adjusting means for giving a command to the robot and
A position that determines the relative positional relationship between the first imaging unit and the work for performing image measurement on the measurement point based on the change in the first feature amount in response to the change in the angle. The image processing system according to any one of claims 1 to 3, further comprising a determination means. A second feature amount calculation means for calculating a second feature amount for a preset detection target portion based on an external appearance image captured by the first image pickup unit, and a second feature amount calculation means.
A target measurement point or a measurement point based on the second feature amount calculated in a state where the distance between the first image pickup unit and the work is positioned at a distance determined by the distance determining means. The image processing system according to any one of claims 1 to 4, further comprising an image measuring means for determining the presence or absence of the detection target portion in the vicinity. 5. The reference line specifying means gives a command to the robot to position the first image pickup unit at the next measurement point when the determination of the presence / absence of the detection target portion by the image measurement means is completed. The image processing system described in. One of claims 1 to 6, further comprising a measurement point setting receiving means for displaying the design information of the work and setting the one or a plurality of measurement points according to a user operation for the displayed design information. The image processing system according to item 1. For each of the measurement points set by the measurement point setting receiving means, the surface shape of each measurement point is calculated based on the design information of the work, and the reference line of each measurement point is calculated based on the calculated surface shape. The image processing system according to claim 7, further comprising a reference line calculation means for calculation. An image processing device for measuring an image of one or a plurality of measurement points preset in the work by using an external image of the work.
An interface that receives information about the appearance image from the first image pickup unit that captures the appearance image, and
An interface for communicating with a robot that changes the relative positional relationship between the work and the first image pickup unit, and
Based on the information obtained by imaging the work, the arrangement status of the work is specified, and the reference line for each measurement point set on the work is specified according to the arrangement status of the specified work. Reference line identification means and
The first image pickup unit and the work are placed in a state where the first image pickup unit is positioned on the specified reference line and the optical axis of the first image pickup unit is aligned with the specified reference line. A distance adjusting means that gives a command to the robot to change the distance between them,
A first feature amount calculating means for calculating a first feature amount regarding the degree of focus of the first image pickup unit based on an appearance image captured by the first image pickup unit.
The first image pickup unit and the first image pickup unit for performing image measurement at the measurement point based on the change in the first feature amount according to the change in the distance between the first image pickup unit and the work. An image processing device including a distance determining means for determining a distance to a work. An image processing program for performing image measurement on one or a plurality of measurement points preset in the work using an external image of the work.
The image processing program is applied to a computer.
A step of imaging the work using the first imaging unit, and
A step of specifying the arrangement status of the work based on the information obtained by imaging the work, and
A step of specifying a reference line for each measurement point set on the work according to the arrangement status of the specified work, and a step of specifying the reference line.
The step of positioning the first image pickup unit on the specified reference line and
The first image pickup unit and the work so as to change the distance between the first image pickup unit and the work while the optical axis of the first image pickup unit is aligned with the specified reference line. The step of giving a command to the robot that changes the relative positional relationship with the work,
A step of calculating a first feature amount relating to the degree of focus of the first image pickup unit based on an appearance image captured by the first image pickup unit, and a step of calculating the first feature amount.
The first image pickup unit and the first image pickup unit for performing image measurement at the measurement point based on the change in the first feature amount according to the change in the distance between the first image pickup unit and the work. An image processing program that executes steps to determine the distance to the work.

Images