JP7131127B2 - APPEARANCE INSPECTION SYSTEM, APPEARANCE INSPECTION RESULT DISPLAY METHOD AND APPEARANCE INSPECTION RESULT DISPLAY PROGRAM - Google Patents

Description

The present disclosure relates to a visual inspection system that inspects an inspection target using a captured image, a visual inspection result display method, and a visual inspection result display program.

Many visual inspection systems have been proposed for inspecting inspection objects such as resins, metals, and substrates using image processing technology.

For example, Japanese Patent Laying-Open No. 2013-211323 (Patent Document 1) discloses a visual inspection system for inspecting the quality of a board. The visual inspection system displays the inspection results of the board on a two-dimensional map image. In the two-dimensional map image, the serial number of the board is arranged along the horizontal axis, and the component information within the board is arranged along the vertical axis. Also, in the two-dimensional map image, a cell is provided for each combination of the board serial number and the component information in the board, and the inspection result of each component of each board is displayed on the corresponding cell.

JP 2013-211323 A

The appearance inspection system disclosed in Patent Literature 1 inspects the appearance of a substrate by capturing an image of a substrate being conveyed in a production line by a fixed imaging device. If the imaging device is fixed, the area of the board that can be imaged is also limited, so the number of images used for inspection is relatively small.

On the other hand, in a visual inspection system that captures an image of an inspection object while moving an imaging device, one inspection object can be imaged from all directions, so the number of images used for inspection tends to be enormous. The number of images used for visual inspection of one inspection object may be several hundred. As a result, the number of test results to be output becomes enormous, and it becomes difficult for the user to check the test results one by one.

The present disclosure has been made to solve the problems as described above, and an object in a certain aspect is to provide a method for visual inspection performed by imaging an inspection object from a plurality of viewpoints while moving an imaging device. To provide a visual inspection system capable of assisting the confirmation work of the results of Another object of the present invention is to provide a display method capable of assisting confirmation work of the result of visual inspection performed by imaging an inspection object from a plurality of viewpoints while moving an imaging device. . Another object of the present invention is to provide a display program capable of assisting confirmation work of the result of visual inspection performed by imaging an inspection object from a plurality of viewpoints while moving an imaging device. .

In one example of the present disclosure, a visual inspection system for performing a visual inspection of an object to be inspected includes a display device, a robot for moving an imaging device, and the imaging device while the robot moves the imaging device. an inspection unit for inspecting each of the plurality of inspection portions of the inspection object for defects based on each image obtained by imaging each of the plurality of inspection portions of the inspection object; and a display control unit for displaying, on the display device, an inspection result matrix representing inspection results by the inspection unit for each of the inspected portions of (1) for each inspection object.

According to this disclosure, the inspection result of each inspection portion of each inspection object is displayed two-dimensionally, so that the user can immediately grasp which inspection portion of which inspection object has a defect. can. The two-dimensional display of inspection results becomes more effective as the number of inspection results increases.

In one example of the present disclosure, each row or each column of the test result matrix is grouped by row or by column according to a predetermined classification rule. The display control unit aggregates and displays the inspection results of the group of rows or the group of columns in one row or one column when an instruction to aggregate the group of rows or the group of columns is received, and When an expansion instruction is received for the test results displayed as a group, the display of the grouped and displayed test results is returned to the state before the grouping.

According to this disclosure, even if the number of rows or columns of the test result matrix is too large to be displayed on the display device, the user can efficiently check the test results by utilizing the aggregate display and expanded display. can do.

In one example of the present disclosure, the display control unit displays an inspection result indicating a defect among the inspection results included in the inspection result matrix in a display mode different from other inspection results.

According to this disclosure, the inspection result indicating the defect is displayed in a display mode different from other inspection results, so that the user can immediately determine the inspection result indicating the defect. It is possible to more easily grasp which inspection portion has a defect.

In one example of the present disclosure, the visual inspection system accepts a selection operation for selecting one or more inspection results from among the inspection results included in the inspection result matrix, thereby obtaining one or more inspection objects and one The operation unit capable of selecting the inspection part, and the inspection unit or the display control unit select the selected inspection object based on the selection operation received by the operation unit. and/or perform processing related to the inspected portion.

According to this disclosure, it is possible to provide an interface for easily designating a combination of an inspection object and an inspection portion.

In an example of the present disclosure, the display control unit displays at least one of an image used for inspection of the selected inspection portion and imaging conditions of the image on the display device.

According to this disclosure, the user can easily confirm the imaging conditions of any inspection portion indicated in the inspection result matrix.

In one example of the present disclosure, the visual inspection system further includes a storage device for storing a three-dimensional model representing the shape of the inspection object. The plurality of inspection portions are preset for the three-dimensional model. The display control unit displays the three-dimensional model on the display device, and displays the inspection result of the selected inspection portion on the corresponding portion on the three-dimensional model.

According to this disclosure, the user can confirm the inspection result of any inspection portion on the three-dimensional model, and can easily confirm which portion of the inspection object has a defect.

In an example of the present disclosure, when a plurality of test results are selected by the selection operation, the display control unit displays statistical results of the plurality of test results on the display device.

According to this disclosure, a user can easily analyze inspection results for any inspection portion.

In one example of the present disclosure, the inspection section re-inspects the selected inspection portion based on the image used for inspection of the selected inspection portion.

According to this disclosure, the user can easily re-inspect any inspected portion.

In one example of the present disclosure, the display control unit displays a comparison result between an expected value matrix indicating a true correct value for each inspection result included in the inspection result matrix and the inspection result matrix on the display device. .

According to this disclosure, a user can easily determine whether each test result shown in the test result matrix is as expected. Such comparison processing becomes more effective as the number of inspection results shown in the inspection result matrix 25 increases.

In another example of the present disclosure, a method for displaying results of a visual inspection performed by imaging a plurality of inspection portions of an object to be inspected by an imaging device comprises: while a robot is moving the imaging device; obtaining each image obtained by imaging each of the plurality of inspection portions; and inspecting each of the plurality of inspection portions for defects based on each image obtained in the obtaining step. and displaying, on a display device, an inspection result matrix showing inspection results for each of the plurality of inspection portions for each inspection object.

According to this disclosure, the inspection result of each inspection portion of each inspection object is displayed two-dimensionally, so that the user can immediately grasp which inspection portion of which inspection object has a defect. can. The two-dimensional display of inspection results becomes more effective as the number of inspection results increases.

In another example of the present disclosure, a program for displaying appearance inspection results obtained by imaging a plurality of inspection portions of an object to be inspected by an imaging device is stored in a computer while the robot moves the imaging device. obtaining each image obtained by imaging each of the plurality of inspection portions by the imaging device; The step of inspecting the presence/absence and the step of displaying, on a display device, an inspection result matrix showing inspection results for each inspection object for each of the plurality of inspected portions are executed.

According to this disclosure, the inspection result of each inspection portion of each inspection object is displayed two-dimensionally, so that the user can immediately grasp which inspection portion of which inspection object has a defect. can. The two-dimensional display of inspection results becomes more effective as the number of inspection results increases.

In a certain aspect, it is possible to assist the confirmation work of the result of the visual inspection performed by imaging the inspection target from a plurality of viewpoints while moving the imaging device.

BRIEF DESCRIPTION OF THE DRAWINGS It is a schematic diagram which shows the outline|summary of the visual inspection system according to embodiment. FIG. 4 is a diagram showing an example of inspection results displayed on the display device of the image processing apparatus according to the embodiment; 3 is a diagram showing a modified example of the inspection result matrix shown in FIG. 2; FIG. FIG. 4 is a diagram showing an example of a data structure of classification rules; FIG. 4 is a diagram showing an example of a data structure of classification rules; FIG. 11 is a diagram showing screen transition of an inspection result matrix when an expand/aggregate button is pressed; FIG. 11 is a diagram showing screen transition of an inspection result matrix when an expand/aggregate button is pressed; 1 is a schematic diagram showing an example of a hardware configuration of an image processing apparatus; FIG. It is a schematic diagram which shows an example of the hardware constitutions of PLC(Programmable Logic Controller). It is a schematic diagram which shows an example of the hardware constitutions of a setting device. It is a figure which shows an example of the data structure of a project file. It is a figure which shows an example of the data structure of an inspection result file. 4 is a flow chart showing an example of the flow of processing in the setting device; FIG. 10 is a diagram showing an example of a screen on which a schematic diagram showing the designed appearance of a workpiece is displayed; It is a figure which shows an example of the screen in which the inspection target part was displayed. FIG. 4 is a diagram showing an example of points on an inspection target area; FIG. 4 is a diagram showing an example of an inspection object position determined from an inspection object area and an effective field of view corresponding thereto; FIG. 4 is a diagram showing all inspected parts and effective fields of view determined from inspected areas; FIG. 10 is a diagram showing an example of a method of determining a photographing position of a work by a setting device; FIG. 4 is a diagram showing an example of an imaging route determined with respect to an imaging position; 4 is a flow chart showing an example of the flow of processing in a PLC; 4 is a flow chart showing an example of the flow of inspection processing by the image processing apparatus; FIG. 11 is a flowchart showing an example of the flow of display processing of an inspection result matrix; FIG. FIG. 10 is a diagram showing a specific example 1 of processing executed in response to a cell selection operation; FIG. 10 is a diagram showing a specific example 2 of processing executed in response to a cell selection operation; FIG. 10 is a diagram showing a specific example 3 of processing executed in response to a cell selection operation; FIG. 10 is a diagram showing a specific example 4 of processing executed in response to a cell selection operation; FIG. 13 is a diagram showing a specific example 5 of processing executed in response to a cell selection operation; FIG. 4 is a diagram showing a modified example of the inspection result matrix shown in FIG. 3; FIG. 4 is a diagram showing an example of a data structure of classification rules; FIG. 11 is a diagram showing screen transition of an inspection result matrix when an expand/aggregate button is pressed; FIG. 11 is a diagram showing screen transition of an inspection result matrix when an expand/aggregate button is pressed; FIG. 4 is a diagram schematically showing comparison processing between an inspection result matrix and an expected value matrix; It is a figure which shows an example of the preparation process of an expectation value matrix. 10 is a flowchart showing an example of the flow of three-dimensional display processing for inspection results; FIG. 10 is a diagram showing a specific example 1 of processing executed in response to a selection operation of a portion to be inspected shown in a three-dimensional model; FIG. 10 is a diagram showing a specific example 2 of processing executed in response to a selection operation of an inspection target portion shown in a three-dimensional model; FIG. 10 is a diagram showing a process of expanding/consolidating a portion to be inspected shown in a three-dimensional model; It is a figure which shows the appearance inspection system which concerns on a modification. FIG. 10 is a diagram showing another form of changing the relative position between the workpiece and the imaging device; FIG. 10 is a diagram showing still another form of changing the relative position between the workpiece and the imaging device;

Hereinafter, each embodiment according to the present invention will be described with reference to the drawings. In the following description, identical parts and components are given identical reference numerals. Their names and functions are also the same. Therefore, detailed description of these will not be repeated.

<A. Application example>
First, an example of a scene to which the present invention is applied will be described with reference to FIGS. 1 and 2. FIG. FIG. 1 is a schematic diagram showing an overview of an appearance inspection system 1 according to this embodiment.

Appearance inspection system 1 according to the present embodiment, for example, in a production line for industrial products, inspects a plurality of portions to be inspected on an inspection object (hereinafter also referred to as "workpiece W") placed on stage 90. is imaged, and the workpiece W is visually inspected using the obtained image. In the appearance inspection, the workpiece W is inspected for flaws, stains, presence of foreign matter, dimensions, and the like.

After completing the visual inspection of the work W placed on the stage 90 , the next work W is transferred onto the stage 90 . At this time, the workpiece W is placed at a predetermined position on the stage 90 in a predetermined posture.

As shown in FIG. 1 , the visual inspection system 1 includes an imaging device 10 , an image processing device 20 , a robot 30 , a robot controller 40 , a PLC 50 and a setting device 60 .

The image capturing device 10 captures an image of a subject existing in an image capturing field according to a command from the image processing device 20 to generate image data.

The image processing device 20 outputs an imaging command to the imaging device 10 according to the command from the PLC 50 . The image processing device 20 includes an inspection section 21 , a display control section 22 and a display device 23 . The inspection unit 21 and the display control unit 22 are functional modules executed by the processor 110 (see FIG. 8) of the image processing device 20 . The inspection unit 21 determines whether the appearance of the workpiece W is good or bad by executing predetermined processing on the image data generated by the imaging device 10 . The display control unit 22 causes the display device 23 to display the determination result by the inspection unit 21 . The inspection result may be output to the display device 23 provided in the image processing device 20, or may be output to a display device (for example, a display 366 described later) provided in the setting device 60. FIG.

The robot 30 is, for example, a vertically articulated robot in which a plurality of arms 32 are connected on a base 31 . Each connecting portion of the plurality of arms 32 includes a rotating shaft. The imaging device 10 is attached to the tip of the tip arm 32a. The robot controller 40 controls the robot 30 according to commands from the PLC 50 to change the relative position between the work W and the imaging device 10 and the posture of the imaging device 10 with respect to the work W.

As described above, the workpiece W is placed at a predetermined position on the stage 90 in a predetermined attitude. Therefore, by changing the relative position and orientation of the imaging device 10 with respect to the stage 90, the robot 30 can change the relative position between the work W and the imaging device 10 and the orientation of the imaging device 10 with respect to the work W. . In other words, the robot 30 moves the imaging device 10 using a coordinate system having a point on the stage 90 as an origin, thereby determining the relative position between the workpiece W and the imaging device 10 and the attitude of the imaging device 10 with respect to the workpiece W. can be changed.

The PLC 50 controls the robot controller 40 and the image processing device 20 so that the imaging device 10 sequentially images a plurality of inspected portions on the workpiece W. FIG. The PLC 50 controls the robot controller 40 according to the route that satisfies the imaging conditions set by the setting device 60 . Further, the PLC 50 controls the image processing device 20 so as to output an imaging command at the timing when the imaging device 10 satisfies the specified imaging conditions.

The setting device 60 sets a path that satisfies an imaging condition including the relative position between the work W and the imaging device 10 for sequentially imaging a plurality of portions to be inspected on the work W. The setting device 60 sets a path that satisfies imaging conditions suitable for the work W when a new product or a new kind of work W needs to be visually inspected.

With reference to FIG. 2, a display mode of inspection results by the appearance inspection system 1 shown in FIG. 1 will be described. FIG. 2 is a diagram showing an example of inspection results displayed on the display device 23 of the image processing device 20. As shown in FIG.

The inspection unit 21 of the image processing device 20 performs predetermined image processing on each image obtained by photographing the workpiece W from a plurality of directions, thereby detecting defects in each inspected portion of the workpiece W. Check for presence. The inspection result is represented by, for example, either "defective" or "defective". The inspection process by the inspection unit 21 is performed each time the stage 90 conveys the workpiece W to be inspected to a predetermined position. The inspection result by the inspection unit 21 is output to the display control unit 22 after being associated with the identification information of the work W and the portion of the work W to be inspected. The display control unit 22 displays, on the display device 23, an inspection result matrix 25 showing inspection results for each inspection target portion of the work W for each work W. FIG.

As shown in FIG. 2, on the horizontal axis of the inspection result matrix 25, parts to be inspected of the workpiece are arranged according to part names. Each region name may be registered in advance or may be set by the user. The vertical axis of the inspection result matrix 25 is arranged with identification information of inspected workpieces. The work identification information is represented by, for example, a work serial number (hereinafter also referred to as "work No.") or a work name. In the example of FIG. 2, the identification information of the work is represented by the work number.

In addition, cells are arranged on the inspection result matrix 25 in association with each combination of the inspection target portion of the workpiece and the workpiece number. The display control unit 22 displays the inspection result of each inspected portion of each workpiece W on the corresponding cell. In this way, the inspection result of each inspected portion of each work is two-dimensionally displayed, so that the user can immediately grasp which portion of which work has a defect. The two-dimensional display of inspection results becomes more effective as the number of inspection results increases.

Typically, the display control unit 22 highlights inspection results indicating defects among the inspection results included in the inspection result matrix 25 in a display mode different from other inspection results. As an example, an inspection result cell that indicates a defect may be represented in a particular color (eg, red) or may be represented by a blinking display. In the example of FIG. 2, the cells indicating "defective" are hatched, and the cells indicating "no defect" are not hatched. By highlighting the inspection result indicating the defect in a display mode different from that of other inspection results, the user can easily determine the inspection result indicating the defect, and can identify which portion of which workpiece has the defect. can be grasped more easily.

Note that, in the above description, an example has been described in which the parts to be inspected of the workpiece are arranged along the horizontal axis of the inspection result matrix 25, and the workpiece numbers are arranged along the vertical axis of the inspection result matrix 25. 25 may be arranged on the horizontal axis, and the parts to be inspected of the workpiece may be arranged on the vertical axis of the inspection result matrix 25 .

<B. Inspection result matrix development/aggregation function>
3 to 7, the expansion/consolidation function of the cells shown in the inspection result matrix 25 will be described. FIG. 3 is a diagram showing a modification of the inspection result matrix 25 shown in FIG. Each row or each column of the inspection result matrix 25 shown in FIG. 3 differs from the inspection result matrix 25 shown in FIG. 2 in that each row or column is grouped according to a predetermined classification rule.

In the example of FIG. 3, in the horizontal direction of the inspection result matrix 25, the parts "A1" to "A5" of the work are grouped as a group GV1, and the parts "B1" to "B3" of the work are grouped into the group GV1. Grouped as GV2. Grouping of inspection target portions of the workpiece is performed according to classification rules 134A shown in FIG.

FIG. 4 is a diagram showing an example of the data structure of the classification rule 134A. As shown in FIG. 4, the inspection target portion of the workpiece W is hierarchically associated with the classification rule 134A. The correspondence between each part may be registered in advance, or may be arbitrarily set by the user. As an example, a portion to be inspected defined in a higher hierarchy has a relationship of including a portion to be inspected defined in a lower hierarchy. In the example of FIG. 4, the part to be inspected "A" is included in the parts "A1" to "A5". That is, the parts "A1" to "A5" are regions within the inspection target portion "A".

Referring to FIG. 3 again, in the vertical direction of the inspection result matrix 25, work Nos. "001A" to "001E" are grouped as group GH1, and work Nos. "002A" to "002J" are grouped into group GH1. Grouped as GH2. The work numbers are grouped according to the classification rule 134B shown in FIG.

FIG. 5 is a diagram showing an example of the data structure of the classification rule 134B. As shown in FIG. 5, each workpiece to be inspected is hierarchically associated with the classification rule 134B. The hierarchical relationship of each work may be registered in advance, or may be arbitrarily set by the user. As an example, each work is associated with a production line lot number. A lot number defined in a higher hierarchy has a relationship of including works defined in a lower hierarchy. In the example of FIG. 5, lot number "001" is included in work numbers "001A" to "001E". That is, it indicates that work numbers "001A" to "001E" were produced in lot number "001".

Referring to FIG. 3 again, each group in the vertical and horizontal directions of the inspection result matrix 25 is assigned an expand/aggregate button. In the example of FIG. 3, an expand/aggregate button BH1 is assigned to the group GH1. An expand/aggregate button BH2 is assigned to the group GH2. An expand/aggregate button BV1 is assigned to the group GV1. An expand/consolidate button BV2 is assigned to GV2.

FIG. 6 is a diagram showing screen transition of the inspection result matrix 25 when the expand/consolidate button BV2 is pressed. The expand/aggregate button BV2 alternately accepts an aggregate instruction and an expand instruction for the cells of the group GV2 each time it is pressed. That is, when the expand/aggregate button BV2 is pressed while the cells of the group GV2 are expanded, the display control unit 22 of the image processing device 20 aggregates the cells of the group GV2 in a row. On the other hand, when the expand/aggregate button BV2 is pressed while the cells of the group GV2 are aggregated, the display control unit 22 restores the display of the cells of the group GV2 to the state before aggregation.

When the inspection results to be aggregated include the inspection results indicating “defective” and the inspection results indicating “no defect”, the display control unit 22 changes the display of the “defective” inspection result to “no defect”. ” is prioritized over the inspection result display. As an example, the inspection results within the dashed line AR1 include one inspection result indicating "defective" and two inspection results indicating "no defect". By the aggregation process, the three inspection results within the dashed line AR1 are aggregated into one inspection result shown within the dashed line AR2. At this time, since the inspection results before aggregation include "defective", the display control unit 22 displays the inspection results after aggregation as "defective".

In this way, if even one inspection result indicating "defective" is included in the inspection results before aggregation, the display control unit 22 displays the inspection results after aggregation as "defective". Yes. On the other hand, if the inspection results before aggregation do not include even one inspection result indicating "defective", the display control unit 22 determines that the inspection results after aggregation are "no defects". ”.

FIG. 7 is a diagram showing screen transition of the inspection result matrix 25 when the expand/consolidate button BH2 is pressed. The expand/aggregate button BH2 alternately accepts an aggregate instruction and an expand instruction for the cells of the group GH2 each time it is pressed. That is, when the expand/aggregate button BH2 is pressed while the cells of the group GH2 are expanded, the display control unit 22 aggregates the cells of the group GH2 into one row. On the other hand, when the expand/aggregate button BH2 is pressed while the cells of the group GH2 are aggregated, the display control unit 22 restores the display of the cells of the group GH2 to the state before aggregation.

As an example, the inspection results within the dashed line AR3 include one inspection result indicating "defective" and nine inspection results indicating "no defect". By the aggregation process, the ten inspection results within the dashed line AR3 are aggregated into one inspection result shown within the dashed line AR4. At this time, since the inspection results before aggregation include "defective", the display control unit 22 displays the inspection results after aggregation as "defective".

As described above, when the display control unit 22 of the image processing apparatus 20 receives an instruction to aggregate a group of rows or a group of columns, the display control unit 22 displays the inspection results of the group of rows or the group of columns in one row. Or aggregate and display in one line. Further, when the display control unit 22 receives an expansion instruction for the inspection results that are aggregated and displayed, the display control unit 22 returns the display of the inspection results that are aggregated and displayed to the state before the aggregation. As a result, even if the number of rows or columns of the test result matrix 25 is so large that it cannot be displayed on the display device 23, the user can efficiently check the test results by utilizing the combined display and expanded display. be able to.

In the above example, an example in which aggregated cells are represented by binary values of "defective" and "defective" has been described, but aggregated cells do not necessarily need to be represented by binary values. no. As an example, the display control unit 22 may display cells after aggregation in a darker color as more inspection results indicating “defective” are included in cells to be aggregated.

<C. Hardware Configuration>
Next, hardware configurations of the image processing device 20, the PLC 50, and the setting device 60 shown in FIG. 1 will be described in order with reference to FIGS. 8 to 10. FIG.

(C1. Hardware Configuration of Image Processing Apparatus 20)
FIG. 8 is a schematic diagram showing an example of the hardware configuration of the image processing device 20. As shown in FIG. Referring to FIG. 8, image processing apparatus 20 typically has a structure that conforms to a general-purpose computer architecture. image processing.

More specifically, the image processing device 20 includes a processor 110 such as a CPU (Central Processing Unit) or MPU (Micro-Processing Unit), a RAM (Random Access Memory) 112, a display controller 114, and a system controller 116. , an I/O (Input Output) controller 118 , a storage device 120 such as a hard disk, a camera interface 122 , an input interface 124 , a controller interface 126 , a communication interface 128 and a memory card interface 130 . These units are connected to each other centering on the system controller 116 so as to be able to communicate with each other.

The processor 110 exchanges programs (codes) and the like with the system controller 116 and executes them in a predetermined order to achieve the desired arithmetic processing.

The system controller 116 is connected to the processor 110, the RAM 112, the display controller 114, and the I/O controller 118 via buses, and exchanges data with each unit. preside over

The RAM 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory), and stores programs read from the storage device 120, camera images (image data) acquired by the imaging device 10, Stores processing results for camera images and work data.

The display controller 114 is connected to the display device 23 and outputs signals for displaying various information to the display device 23 according to internal commands from the system controller 116 .

The I/O controller 118 controls data exchange with recording media and external devices connected to the image processing apparatus 20 . More specifically, I/O controller 118 is connected to storage device 120 , camera interface 122 , input interface 124 , controller interface 126 , communication interface 128 and memory card interface 130 .

The storage device 120 stores various data such as a project file 144 in addition to an image processing program 142 and a display program 143 executed by the processor 110 . Details of the project file 144 will be described later. Typically, the storage device 120 may be a nonvolatile magnetic storage device such as a hard disk, a semiconductor storage device such as a flash memory, or a DVD-RAM (Digital Versatile Disk Random Access Memory). ) or other optical storage device.

The image processing program 142 and the display program 143 may be provided as part of another program. In that case, the image processing program 142 itself and the display program 143 themselves cooperate with other programs to perform predetermined processing. That is, the image processing program 142 and the display program 143 may be incorporated in such other programs. Alternatively, part or all of the functions provided by executing the image processing program 142 or the display program 143 may be implemented as a dedicated hardware circuit.

The camera interface 122 corresponds to an input unit that receives image data generated by photographing the work W, and mediates data transmission between the imaging device 10 and the processor 110 . More specifically, the camera interface 122 can be connected to one or more imaging devices 10 , and a photographing instruction is output from the processor 110 to the imaging device 10 via the camera interface 122 . As a result, the imaging device 10 captures a subject and outputs the generated image to the processor 110 via the camera interface 122 .

Input interface 124 mediates data transmission between processor 110 and input devices such as keyboard 134, mouse, touch panel, and dedicated console.

Controller interface 126 mediates data transmission between PLC 50 and processor 110 . More specifically, the controller interface 126 transmits information regarding the state of the production line controlled by the PLC 50, information regarding the workpiece W, and the like to the processor 110. FIG.

Communication interface 128 mediates data transmission between processor 110 and other personal computers and server devices (not shown). The communication interface 128 typically consists of Ethernet (registered trademark), USB (Universal Serial Bus), or the like.

Memory card interface 130 mediates data transmission between processor 110 and memory card 136, which is a recording medium. The memory card 136 stores an image processing program 142 and a display program 143 to be executed by the image processing apparatus 20 , and the memory card interface 130 reads these programs from the memory card 136 . The memory card 136 is a general-purpose semiconductor storage device such as an SD (Secure Digital), a magnetic recording medium such as a flexible disk, or an optical recording medium such as a CD-ROM (Compact Disk Read Only Memory). Become. Alternatively, a program downloaded from a distribution server or the like may be installed in the image processing apparatus 20 via the communication interface 128 .

(Hardware configuration of C2.PLC50)
Next, the hardware configuration of the PLC 50 will be described with reference to FIG. FIG. 9 is a schematic diagram showing an example of the hardware configuration of the PLC 50. As shown in FIG.

PLC 50 includes chipset 212 , processor 214 , nonvolatile memory 216 , main memory 218 , system clock 220 , memory card interface 222 , communication interface 228 , internal bus controller 230 , fieldbus controller 238 . including. Chipset 212 and other components are each coupled via various buses.

Processor 214 and chipset 212 are typically configured according to general computer architecture. That is, the processor 214 interprets and executes instruction codes sequentially supplied from the chipset 212 according to the internal clock. The chipset 212 exchanges internal data with various connected components and generates instruction codes necessary for the processor 214 . The system clock 220 generates a system clock with a predetermined period and outputs it to the processor 214 . The chipset 212 has a function of caching data obtained as a result of arithmetic processing executed by the processor 214 .

The PLC 50 has a nonvolatile memory 216 and a main memory 218 as storage means. The nonvolatile memory 216 nonvolatilely holds the OS, system programs, user programs, log information, and the like. The main memory 218 is a volatile storage area, holds various programs to be executed by the processor 214, and is also used as a working memory when executing various programs.

The PLC 50 has a communication interface 228, an internal bus controller 230, and a fieldbus controller 238 as communication means. These communication circuits transmit and receive data.

A communication interface 228 exchanges data with the image processing device 20 and the setting device 60 . As an example, the PLC 50 outputs imaging instructions to the image processing device 20 via the communication interface 228 . Alternatively, the PLC 50 receives the visual inspection result of the work W from the image processing device 20 via the communication interface 228 .

Internal bus controller 230 controls the exchange of data via internal bus 226 . More specifically, internal bus controller 230 includes a DMA (Dynamic Memory Access) control circuit 232 , an internal bus control circuit 234 and a buffer memory 236 .

A memory card interface 222 connects a memory card 224 removable from the PLC 50 and the processor 214 .

Fieldbus controller 238 is a communication interface for connecting to a field network. PLC 50 is connected to robot controller 40 via fieldbus controller 238 . For example, EtherCAT (registered trademark), EtherNet/IP (registered trademark), CompoNet (registered trademark), etc. are adopted for the field network.

(C3. Hardware configuration of setting device 60)
Next, the hardware configuration of the setting device 60 will be described with reference to FIG. FIG. 10 is a schematic diagram showing an example of the hardware configuration of the setting device 60. As shown in FIG.

Setting device 60 includes processor 362 , main memory 363 , storage device 364 , display 366 , input device 367 and communication interface 368 . These units are connected to each other via a bus 361 so as to be able to communicate with each other.

The processor 362 expands programs (codes) including the setting program 365 installed in the storage device 364 into the main memory 363 and executes them in a predetermined order to perform various operations. The main memory 363 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory).

The storage device 364 is an internal memory provided in the setting device 60 and is a non-volatile storage device, and stores various programs such as the setting program 365 . Note that the storage device 364 may be a hard disk or a semiconductor storage device such as a flash memory.

The setting program 365 is a program that indicates the procedure for setting the imaging condition change path by the setting device 60 . Various programs such as the setting program 365 do not need to be stored in the storage device 364 , and may be stored in a server that can communicate with the setting device 60 or an external memory that can be directly connected to the setting device 60 . For example, various programs executed by the setting device 60 and various parameters used by the various programs are stored in the external memory and distributed, and the setting device 60 reads the various programs and various parameters from the external memory. External memory stores information such as programs by electrical, magnetic, optical, mechanical or chemical action so that computers, other devices, machines, etc. can read information such as programs. is a medium. Alternatively, programs and parameters downloaded from a server communicatively connected to the setting device 60 may be installed in the setting device 60 .

Display 366 is, for example, a liquid crystal display. The input device 367 is composed of, for example, a mouse, keyboard, touch pad, and the like.

Communication interface 368 exchanges various data between PLC 50 and processor 362 . Note that communication interface 368 may transfer data between the server and processor 362 . The communication interface 368 includes network compatible hardware for exchanging various data with the PLC 50 .

<D. Project file 144>
The project file 144 (see FIG. 8) described above will be described with reference to FIG. FIG. 11 is a diagram showing an example of the data structure of the project file 144. As shown in FIG.

By associating the various data related to the inspection process of the work W with the project file 144, the various data related to the inspection process of the work W are centrally managed.

As an example, the project file 144 includes an imaging condition file 146A, an image file group 146B, an inspection condition file 146C, an inspection result file 146D, a work information file 146E, a production information file 146F, and a classification rule file 146G. , and the expected value file 146H.

The imaging condition file 146A is data that defines imaging conditions for each portion of the work W to be inspected. In other words, the image processing apparatus 20 can uniquely specify the imaging condition by referring to the imaging condition file 146A, using the inspection target portion as a key. Alternatively, the imaging conditions may be uniquely specified using a combination of the work No. and the portion to be inspected as a key. The imaging conditions include, for example, the relative position between the workpiece W and the imaging device 10, the lighting position when the workpiece W is imaged, and the like.

The image file group 146B is a data group obtained from the imaging device 10 by imaging the work W according to prescribed imaging conditions. Each image file included in the image file group 146B is associated with a workpiece number and an inspection target portion. That is, the image processing apparatus 20 can uniquely identify the image file used for the inspection using the combination of the workpiece number and the inspection target portion as a key. Information about the combination of the work No. and the portion to be inspected is specified, for example, in the file name of the image file or the header of the image file.

The inspection condition file 146C is data that defines inspection conditions for each inspection target portion of the workpiece. That is, the image processing apparatus 20 can uniquely identify the inspection condition as the inspection target portion key by referring to the inspection condition file 146C. The inspection conditions include, for example, the portion to be inspected in the image, the image processing flowchart to be executed, the measurement parameters to be read when image processing is executed, the threshold value for determining the presence or absence of defects, and the like.

The inspection result file 146D is data that defines measurement values, inspection results, and the like for each inspection target portion of each workpiece. Details of the data structure of the inspection result file 146D will be described later.

The work information file 146E includes a three-dimensional model representing the shape of the work to be inspected, type information of the work to be inspected, and the like.

The production information file 146F is data that defines the lot number, serial number, etc. of the work to be inspected.

The classification rule file 146G is data containing at least one of the classification rule 134A described above (see FIG. 4), the classification rule 134B described above (see FIG. 5), and the classification rule 134C described later (see FIG. 30). .

The expected value file 146H is data that defines the expected value of the inspection result for each inspection target portion of each workpiece to be inspected. Details of the expected value file 146H will be described later.

<E. Inspection result file 146D>
The inspection result file 146D included in the project file 144 (see FIG. 11) will be described with reference to FIG.

FIG. 12 is a diagram showing an example of the data structure of the inspection result file 146D. The inspection result file 146D contains workpiece identification information (for example, workpiece No.), part names indicating inspection target portions of the workpiece, measurement values obtained as output results of inspection processing, and inspection results indicating the presence or absence of defects. include.

By referring to the inspection result file 146D, the image processing apparatus 20 can uniquely identify the measurement value and the inspection result using the combination of the workpiece number and the inspection target portion as a key.

<F. Flow of Processing in Setting Device>
FIG. 13 is a flowchart showing an example of the flow of processing in the setting device 60. As shown in FIG. When the appearance inspection system 1 needs to inspect the appearance of a new product or a new type of work W, the setting device 60 performs processing according to the flowchart shown in FIG. A change route of the imaging conditions suitable for is set. The three-dimensional design data representing the designed surface of the new product or new type of work W is stored in advance in the storage device 364 of the setting device 60 .

In the example shown in FIG. 13, the processor 362 of the setting device 60 first reads the three-dimensional design data from the storage device 364 in step S1. Next, in step S2, the processor 362 displays on the display 366 of the setting device 60 a schematic diagram showing the designed appearance of the work W indicated by the three-dimensional design data, and selects an inspection target area on the work W according to user input. decide. At this time, on the premise that the workpiece W is placed at a predetermined position on the stage 90 in a predetermined posture, the processor 362 converts the coordinate system of the three-dimensional design data into points on the stage 90. Convert to the XYZ coordinate system with the origin. Therefore, the inspection target area is indicated by an XYZ coordinate system with a point on the stage 90 as the origin.

Next, in step S3, the processor 362 determines a plurality of inspection target positions from within the inspection target area so as to satisfy inspection requirements corresponding to the inspection target area. That is, the processor 362 determines each of the plurality of inspection target positions determined within the inspection target area as the inspection target portion.

Next, in step S4, the processor 362 determines imaging conditions including the relative position between the workpiece W and the imaging device 10 during inspection for each of the plurality of inspection target positions.

Next, in step S5, the processor 362 determines an imaging path so as to pass through the relative position between the workpiece W and the imaging device 10 determined in step S4.

The inspection target area determined in step S2, the inspection target position determined in step S3, the imaging conditions determined in step S4, and the imaging route determined in step S5 are transmitted to the image processing device 20 and the PLC 50. .

In the above example, it is assumed that a plurality of inspection target positions are determined from the inspection target portion in step S3. However, in step S2, a plurality of inspected portions may be determined, and in step S3, at least one inspected location from each of the plurality of inspected portions may be determined. Also by this, a plurality of positions to be inspected are determined in step S3.

<G. Method for Determining Parts to be Inspection>
An example of a method for determining a portion to be inspected will be described with reference to FIGS. 14 and 15. FIG. FIG. 14 is a diagram showing an example of a screen on which a schematic diagram showing the designed appearance of the workpiece W is displayed. FIG. 15 is a diagram showing an example of a screen displaying a portion to be inspected.

As shown in FIG. 14, the setting device 60 causes the display 366 to display a screen 61a including a three-dimensional model ML representing the shape of the workpiece W. As shown in FIG. The screen 61a includes a tool button 71 for rotating the three-dimensional model ML about the vertical direction, and a tool button 72 for rotating the three-dimensional model ML about the horizontal direction. The user can appropriately rotate the three-dimensional model ML by operating the tool buttons 71 and 72 .

The setting device 60 accepts designation of a place to be inspected from the user. Specifically, the user uses the input device 367 to click a plurality of points to be inspected on the three-dimensional model ML of the workpiece W. On the screen 61a shown in FIG. 14, a plurality of points clicked by the user are indicated by circular marks 74. In the screen 61a shown in FIG.

The setting device 60 cuts out an area including a plurality of circle marks 74 designated on the three-dimensional model ML as an inspection target area. Specifically, the setting device 60 obtains a range within a predetermined distance along the surface of the workpiece W from a point on the surface of the work W corresponding to each designated circle 74, and determines the union of the ranges as the inspection target area. Cut out as

Furthermore, the setting device 60 adjusts the inspection target area so that the outline is a straight line or a geometric figure such as a circle. A screen 61b of the display 366 shown in FIG. 15 shows an inspection target area 75 adjusted so that the outline is a straight line parallel to one of the ridgelines of the workpiece W. As shown in FIG.

Further, the setting device 60 receives an instruction for fine adjustment of the inspection target area 75 from the user, and finely adjusts the inspection target area 75 according to the instruction. The screen 61b includes tool buttons 76 and 77 for enlarging or reducing the inspection target area 75. FIG. The user uses the input device 367 to select one side forming the outline of the inspection target area 75 and operates the tool buttons 76 and 77 to input an instruction to enlarge or reduce the inspection target area 75 . Alternatively, the user uses the mouse included in the input device 367 to drag a point 78 on one side that forms the outline of the inspection target area 75 to input an instruction to enlarge or reduce the inspection target area 75 . may Thereby, the setting device 60 enlarges or reduces the area 75 to be inspected. In this manner, the setting device 60 determines the inspection target area 75 . In the example shown in FIG. 15, an area obtained by collecting partial areas of four of the six surfaces of the rectangular parallelepiped workpiece W is determined as the inspection target area 75 .

<H. Method for determining position to be inspected>
An example of a method for determining the position to be inspected by the setting device 60 will be described with reference to FIGS. 16 to 18. FIG. FIG. 16 is a diagram showing an example of points on an inspection target area. FIG. 17 is a diagram showing an example of an inspection object position determined from an inspection object area and an effective visual field corresponding thereto. FIG. 18 is a diagram showing all inspection target portions determined from the inspection target area and the effective field of view.

When the camera resolution is R (pix) and the required minimum defect size is D (mm), the maximum imaging field of view by the imaging device 10 that can satisfy the inspection requirement that the defect of the minimum defect size can be recognized. A FOV is set as the portion to be inspected. The diameter of the imaging field of view FOV is generally represented by a×D×R using a proportionality constant a.

The setting device 60 regards the inspection target area 75 as a set of points, and can investigate the three-dimensional shape near the points in the inspection target area 75 using the distribution of the normal vectors. For example, based on the three-dimensional design data of the workpiece W, the setting device 60 obtains the distribution of normal vectors in a range within a distance L along the surface from a point within the inspection target area 75 . The distance L is, for example, a value (=b×a×D×R) obtained by multiplying the diameter of the imaging field FOV (=a×D×R) by a proportionality constant b.

In the example shown in FIG. 16, the vicinity of point P1 is flat. Therefore, all the normal vectors in the range within the distance L along the surface of the work W from the point P1 become the vector n2. The point P2 is located near the ridge where the two surfaces intersect. Therefore, normal vectors within a range within a distance L along the surface of the work W from the point P2 include two vectors n2 and n4. The point P2 is located near the vertex where the three faces intersect. Therefore, the normal vectors within the distance L along the surface of the work W from the point P3 include three vectors n1, n2 and n4. Therefore, the setting device 60 determines whether the vicinity of the point is flat or whether a ridge exists in the vicinity of the point, based on the distribution of the normal vectors in the range within the distance L along the surface from the point in the inspection target area 75. It can be determined whether a ridge line exists in the vicinity of the point.

As shown in FIG. 17, the setting device 60 selects one point randomly selected from the inspection target area 75 as the inspection target position Bi. The setting device 60 obtains an effective field of view FOV2i by the imaging device 10 as an inspection target portion for the inspection target position Bi. The effective field of view FOV2i includes the inspection target position Bi, and is a field of view that can be imaged and inspected by the imaging device 10 using one imaging condition. The setting device 60 determines the effective field of view FOV2i according to the three-dimensional shape in the vicinity of the inspection target position Bi. The maximum diameter that the effective field of view FOV2i can take is set to the diameter of the imaging field of view FOV (=a×D×R).

For example, the setting device 60 determines the effective field of view FOV2i such that the variation in normal vector distribution in the effective field of view FOV2i falls within a predetermined range. If the vicinity of the inspection target position Bi is flat, the setting device 60 determines a range within the distance a×D×R along the surface from the inspection target position Bi (that is, the imaging field of view FOV) as the effective field of view FOV2i. On the other hand, if there is a ridgeline or vertex in the vicinity of the position to be inspected Bi, as shown in FIG. The effective field of view FOV2i is the range excluding the range of the part. The excluded range is the range of the surface that has normal vectors other than the normal vector that exhibits the largest amount of distribution in the normal vector distribution.

Next, the setting device 60 removes the set of points belonging to the determined effective field of view FOV2i from the set of points belonging to the inspection target region 75, and selects one point randomly from the set of remaining points as the next point. is selected as the inspection target position B(i+1). The setting device 60 also determines the effective field of view FOV2(i+1) for the selected inspection target position B(i+1). The setting device 60 repeats this process until the set of points belonging to the inspection target area 75 becomes zero. As a result, a plurality of inspection target positions Bi (i=1, 2, . . . ) are determined from the inspection target area 75, as shown in FIG. All points in the inspection target area 75 are included in the effective field of view FOV2i of any of the plurality of inspection target positions Bi (i=1, 2, . . . ). As described above, the maximum diameter that the effective field of view FOV2i can take is set to the diameter of the imaging field of view FOV. Therefore, the inspection target position is determined so that the defect of the minimum defect size can be recognized over the entire inspection target area 75 .

In the above description, the setting device 60 randomly extracts the inspection target position Bi from the inspection target area 75 . However, the setting device 60 may extract the inspection target position Bi from the inspection target area 75 according to predetermined geometric conditions. Alternatively, the setting device 60 may extract the inspection target positions Bi from the inspection target area 75 so that the extracted plurality of inspection target positions Bi are regularly aligned.

For example, if defects are likely to occur in the vicinity of edges or vertices in the workpiece W, the inspection target position may be determined so as to satisfy the inspection requirement of preferentially inspecting the vicinity of the edges or vertices. For example, the inspection target position Bi may be preferentially extracted from a set of points having edges or vertices within a predetermined distance within the inspection target region 75 .

<I. Method for Determining Imaging Conditions>
FIG. 19 is a diagram showing an example of how the setting device 60 determines the photographing position Ci of the workpiece W. As shown in FIG. The setting device 60 determines the photographing position Ci on the normal line of the designed external surface of the workpiece W at the position Bi to be inspected. Specifically, the imaging position Ci is located at a position away from the inspection target position Bi by an optimum subject distance that enables imaging of the effective field of view FOV2i corresponding to the inspection target position Bi and that the inspection target position Bi is in focus. It is determined.

Next, the setting device 60 determines imaging conditions based on the determined imaging position Ci. Typically, the setting device 60 includes the inspection target position Bi in its field of view, and determines the optimum imaging conditions for bringing the inspection target position Bi into focus.

The imaging conditions are, for example, the X, Y, and Z coordinates of imaging device 10 in an XYZ coordinate system with a point on stage 90 as the origin, and θx, θy, and θz specifying the direction of the optical axis of imaging device 10. contains six parameters of θx is the angle between the X-axis and the line obtained by projecting the optical axis of the imaging device 10 onto the XY plane, and θy is the angle formed between the Y-axis and the line obtained by projecting the optical axis of the imaging device 10 onto the YZ plane. , and θz is the angle between the Z axis and a line obtained by projecting the optical axis of the imaging device 10 onto the ZX plane. The XYZ coordinates are parameters specifying the relative position between the work W and the imaging device 10 , and θx, θy, θz are parameters specifying the posture of the imaging device 10 with respect to the work W.

<J. How to determine the shooting route>
FIG. 20 is a diagram showing an example of an imaging route determined for an imaging position Ci.

The setting device 60 determines the imaging route so as to pass through the determined imaging positions C1 to C7. At this time, the setting device 60 determines the imaging route so as to satisfy predetermined requirements. For example, if the predetermined requirement is to minimize the travel time, the setting device 60 selects the route candidate with the shortest travel time among the route candidates passing through the plurality of imaging positions in order as the imaging route. set. For example, the setting device 60 may calculate an evaluation value for each route candidate for evaluating an item (for example, travel time) indicated by a predetermined requirement, and set a shooting route based on the calculated evaluation value. . Furthermore, even when there are a plurality of imaging position candidates for each of the plurality of inspection target positions, the setting device 60 selects one of the plurality of imaging position candidates as the imaging position based on the evaluation value. good. In this way, an optimized imaging route is set to meet predetermined requirements.

FIG. 20 shows the determined imaging route PS. In the example of FIG. 20, the camera passes through "shooting position C1" → "shooting position C4" → "shooting position C5" → "shooting position C6" → "shooting position C7" → "shooting position C3" → "shooting position C2" in order. The photographing path PS is determined so as to

<K. Control processing by PLC 50>
FIG. 21 is a flow chart showing an example of the flow of processing in the PLC 50. As shown in FIG. The processing shown in FIG. 21 is implemented by processor 214 of PLC 50 executing a program. In other aspects, part or all of the processing may be performed by circuit elements or other hardware.

The PLC 50 controls the robot controller 40 so that the imaging device 10 passes through the determined imaging path PS (see FIG. 20), and the imaging device 10 performs imaging at the determined imaging position Ci (see FIG. 20). The image processing device 20 is controlled so as to

First, in step S10, the processor 214 determines whether or not the workpiece W placed on the stage 90 is placed at a predetermined position. When processor 214 determines that workpiece W is installed at a predetermined position (YES in step S10), processor 214 switches control to step S12. Otherwise (NO in step S10), processor 214 executes the process of step S10 again.

In step S12, the processor 214 sequentially outputs command values to the robot controller 40 according to the preset photographing path PS. The robot controller 40 drives each axis of the robot 30 according to command values from the PLC 50 . As a result, the imaging device 10 attached to the tip of the robot 30 sequentially moves along the imaging path PS.

In step S20, the processor 214 determines whether or not the imaging device 10 has reached the preset imaging position Ci (see FIG. 20). Whether or not the imaging device 10 has reached the preset imaging position Ci is determined, for example, based on the command value output to the robot controller 40 in step S12. When the processor 214 determines that the imaging device 10 has reached the preset shooting position Ci (YES in step S20), the control is switched to step S22. Otherwise (NO in step S20), processor 214 switches control to step S30.

In step S<b>22 , the processor 214 outputs a photographing instruction to the image processing device 20 . Thereby, the image processing device 20 executes the photographing process.

In step S30, the processor 214 determines whether or not the imaging device 10 has reached the end point of the preset imaging route PS. Whether or not the imaging device 10 has reached the end point of the preset imaging path PS is determined, for example, based on the command value output to the robot controller 40 in step S12. When the processor 214 determines that the imaging device 10 has reached the end point of the preset photographing path PS (NO in step S30), the processor 214 returns the control to step S10. Otherwise (NO in step S30), processor 214 returns control to step S12.

<L. Inspection Processing by Image Processing Apparatus 20>
FIG. 22 is a flowchart showing an example of the flow of inspection processing by the image processing device 20. As shown in FIG. The processing shown in FIG. 22 is implemented by the processor 110 of the image processing device 20 executing the image processing program 142 (see FIG. 8). Note that part or all of the processing may be performed by circuit elements or other hardware.

In step S<b>50 , processor 110 determines whether or not a photographing instruction has been received from PLC 50 . As described above with reference to FIG. 21, the PLC 50 outputs a shooting instruction to the image processing device 20 when the imaging device 10 reaches the preset shooting position Ci. When processor 110 determines that the shooting instruction has been received from PLC 50 (YES in step S50), processor 110 switches control to step S52. Otherwise (NO in step S50), processor 110 switches control to step S60.

In step S<b>52 , the processor 110 outputs a photographing instruction to the imaging device 10 as the inspection unit 21 (see FIG. 1 ) described above. Accordingly, the imaging device 10 executes the imaging process based on receiving the imaging instruction from the image processing device 20 . Thereby, the image processing device 20 can cause the image capturing device 10 to perform the image capturing process at the predetermined image capturing position Ci.

In step S<b>54 , the processor 110 performs inspection processing by performing predetermined image processing on the image obtained from the imaging device 10 as the inspection unit 21 described above. The image processing to be executed is specified, for example, in the above inspection condition file 146C (see FIG. 11). As described above, the inspection condition file 146C is data that defines inspection conditions for each portion of the work W to be inspected. That is, by referring to the inspection condition file 146C, the image processing apparatus 20 can uniquely identify the inspection condition using the inspection target portion as a key. The processor 110 performs image processing on the image obtained according to the inspection conditions acquired from the inspection condition file 146C, and obtains inspection results.

In step S56, the processor 110, acting as the inspection unit 21, writes the inspection result obtained in step S54 to the inspection result file 146D (see FIG. 12). As an example, the processor 110 includes identification information of the work W (for example, work No.), a part name indicating an inspection target portion of the work W, a measurement value obtained as an output result of the inspection process, and an inspection indicating whether or not there is a defect. After associating with the result, it is written in the inspection result file 146D.

In step S60, processor 110 determines whether or not an instruction to end the inspection process has been received. When processor 110 determines that an instruction to end the inspection process has been received (YES in step S60), processor 110 ends the inspection process shown in FIG. Otherwise (NO in step S60), processor 110 returns control to step S50.

In the example of FIG. 22, an example in which the inspection process is executed each time the photographing process of the workpiece W is executed has been described. may be That is, the image processing apparatus 20 may first store images and then inspect the stored images all at once.

<M. Display Flow of Inspection Result Matrix 25>
FIG. 23 is a flow chart showing an example of the flow of display processing for the inspection result matrix 25 described above. The processing shown in FIG. 23 is implemented by the processor 110 of the image processing device 20 executing the display program 143 (see FIG. 8). Note that part or all of the processing may be performed by circuit elements or other hardware.

In step S70, the processor 110 determines whether or not an operation to display the inspection result matrix 25 has been accepted. The display operation is performed on the operation unit provided in the image processing device 20 . The operating unit includes, for example, a keyboard 134 (see FIG. 8), a mouse, a touch panel, and the like. When processor 110 determines that an operation to display inspection result matrix 25 has been accepted (YES in step S70), processor 110 switches control to step S72. Otherwise (NO in step S70), processor 110 executes the process of step S70 again.

In step S72, the processor 110 refers to the inspection result file 146D (see FIG. 12) to determine the work No. defined in the inspection result file 146D and the inspection target portion defined in the inspection result file 146D. to get

In step S74, the processor 110 configures the vertical axis of the inspection result matrix 25 according to the work No. acquired in step S72 as the above-described display control unit 22 (see FIG. 1). As a result, the work numbers are arranged on the vertical axis of the inspection result matrix 25 .

In step S76, the processor 110, as the display control unit 22 described above, configures the horizontal axis of the inspection result matrix 25 according to the inspection target portion of the workpiece acquired in step S74. As a result, the parts to be inspected are arranged on the horizontal axis of the inspection result matrix 25 .

In step S78, the processor 110, as the display control unit 22 described above, displays a blank cell on the inspection result matrix 25 for each combination of the work No. acquired in step S72 and the inspection target portion of the work acquired in step S74. Deploy. Each cell is associated with a work No. and a part to be inspected.

In step S<b>80 , the processor 110 , acting as the display control unit 22 described above, refers to the inspection result file 146</b>D and reflects the inspection result in each cell of the inspection result matrix 25 . More specifically, the processor 110 acquires the inspection result for each combination of the work No. and the inspection target portion, and reflects the acquired inspection result in the cell corresponding to each combination. Typically, the processor 110 reflects the inspection results on each cell in a manner that allows distinguishing between the "defective" inspection results and the "no-defective" inspection results. As an example, if the inspection result indicates "defective", processor 110 displays the corresponding cell in a particular color (eg, red). On the other hand, if the inspection result indicates "no defects", processor 110 displays the corresponding cell in another color (eg, white or green).

At step S82, processor 110 determines whether any cell in test result matrix 25 has been selected. The selection operation is performed on the operation unit provided in the image processing device 20 . The operating unit includes, for example, a keyboard 134 (see FIG. 8), a mouse, a touch panel, and the like. When processor 110 determines that any cell in inspection result matrix 25 has been selected (YES in step S82), processor 110 switches control to step S84. Otherwise (NO in step S82), processor 110 switches control to step S90.

Since the inspection result matrix 25 is associated with the work No. and the portion to be inspected, selecting each cell means specifying the combination of the work No. and the portion to be inspected.

In step S84, the processor 110 performs processing corresponding to the selected cell as the inspection unit 21 (see FIG. 1) or the display control unit 22 described above. That is, the processor 110 executes processing according to the combination of the designated work No. and inspection target portion. Details of the processing executed in response to the cell selection operation will be described later.

In step S90, the processor 110 determines whether or not an operation to close the inspection result matrix 25 has been accepted. When processor 110 determines that an operation to close inspection result matrix 25 has been accepted (YES in step S90), processor 110 ends the process shown in FIG. Otherwise (NO in step S90), processor 110 returns control to step S82.

<N. Cell Selection Function in Inspection Result Matrix>
As shown in steps S82 and S84 in FIG. 24, the image processing device 20 executes processing according to the selected cell based on the selection of the cell in the inspection result matrix 25. FIG. The processing that can be executed includes, for example, the processing of specific examples 1 to 5 described below. These processes will be described in order below.

Note that typically, the image processing apparatus 20 executes at least one of the processes shown in specific examples 1 to 5 below based on reception of a cell selection operation. Alternatively, the image processing device 20 may display a process selection screen shown in specific examples 1 to 5 below and execute the process selected on the selection screen based on the acceptance of the cell selection operation. good.

(N1. Specific example 1)
FIG. 24 is a diagram showing a specific example 1 of processing executed in response to a cell selection operation. In the example of FIG. 24, cell CE1 in inspection result matrix 25 is selected.

Based on the selection of the cell CE1, the display control unit 22 of the image processing device 20 displays the image used for the inspection of the inspection target portion corresponding to the cell CE1 and the imaging conditions of the image on the display device 23. indicate. As a result, the user can visually confirm the image showing the defect and confirm the validity of the imaging conditions. In addition, the user can easily confirm which part of the work is indicated by the inspection target part shown in the inspection result matrix 25 .

More specifically, the image processing device 20 identifies the combination of the work No. and inspection target portion associated with the selected cell CE1. Next, the image processing apparatus 20 acquires the corresponding image file from the image file group 146B (see FIG. 11) using the specified combination of the workpiece number and the inspection target portion as a key. Next, the image processing apparatus 20 acquires the corresponding imaging conditions from the above-described imaging condition file 146A (see FIG. 11) using the specified inspection target portion as a key. The display control unit 22 displays the acquired image file and imaging conditions on the popup screen PU1 associated with the cell CE1. The displayed imaging conditions are configured to be changeable to arbitrary values.

In the example of FIG. 24, the example in which both the captured image and the imaging conditions are displayed in accordance with the cell selection operation has been described. and imaging conditions may be displayed.

(N2. Specific example 2)
FIG. 25 is a diagram showing a specific example 2 of processing executed in response to a cell selection operation. In the example of FIG. 25, the cell group CE2 within the inspection result matrix 25 is selected.

Based on the selection of the cell group CE2, the display control unit 22 of the image processing device 20 displays the three-dimensional model ML representing the shape of the workpiece to be inspected on the display device 23, and also displays the three-dimensional model ML corresponding to the cell group CE2. The inspection result of the inspected portion is represented on the corresponding portion on the three-dimensional model ML. This allows the user to easily confirm which part of the work has a defect.

More specifically, the display control unit 22 of the image processing device 20 selects the three-dimensional model of the workpiece to be inspected from the workpiece information file 146E (see FIG. 11) based on the selection of the cell group CE2. ML is obtained, and the three-dimensional model ML is displayed on the pop-up screen PU2 associated with the cell group CE2.

Next, the combination of the work No. and inspection target portion associated with each selected cell is specified. Next, the image processing apparatus 20 acquires the inspection result corresponding to each combination of the identified work No. and inspection target portion from the inspection result file 146D (see FIG. 12). Next, the image processing device 20 displays the inspection result of each inspection target portion in the corresponding portion of the three-dimensional model ML. As described above with reference to FIGS. 13 to 18, the position to be inspected Bi is set with respect to the three-dimensional model ML, so the relationship between the three-dimensional model ML and the position to be inspected Bi is known. Therefore, the display control unit 22 of the image processing device 20 can reflect the inspection result of each inspection target portion on the three-dimensional model ML based on this known information.

In the example of FIG. 25, inspection target portions A3 to A5 corresponding to the selected cell group CE2 are reflected on the three-dimensional model ML. At this time, the display control unit 22 displays the inspection target portion indicating "defective" on the three-dimensional model ML in a display mode different from "defective". A portion to be inspected that indicates "defective" is represented in a specific color (for example, red) on the three-dimensional model ML, and a portion to be inspected that indicates "no defect" is represented in another color on the three-dimensional model ML. (eg green). Alternatively, the portion to be inspected indicating "defective" may be indicated by blinking display.

Preferably, the display control unit 22 further displays an inspection target area 75A including the inspection target portion A3 and an inspection target area 75B including the inspection target portions A4 and A5 on the three-dimensional model ML. The area to be inspected is as described with reference to FIG. 15, so description thereof will not be repeated. The inspection target area 75B including the inspection target portion indicating "defective" is displayed in a manner different from the inspection target area 75A not including the inspection target portion indicating "defective". As an example, the inspection target region 75B may be indicated by a specific color (for example, red), or may be indicated by a blinking display.

Further, the display control unit 22 displays a tool button 71 for rotating the three-dimensional model ML about the vertical direction and a tool button 72 for rotating the three-dimensional model ML about the horizontal direction on the popup screen PU2. show more in The user can appropriately rotate the three-dimensional model ML by operating the tool buttons 71 and 72 .

(N3. Specific example 3)
FIG. 26 is a diagram showing a specific example 3 of processing executed in response to a cell selection operation. In the example of FIG. 26, the cell group CE3 within the inspection result matrix 25 is selected.

The display control unit 22 of the image processing device 20 displays the statistical results of the inspection results corresponding to the cell group CE3 on the display device 23 based on the selection of the cell group CE3. This allows the user to easily analyze the inspection results of any portion to be inspected.

More specifically, the image processing device 20 identifies the combination of the workpiece number and the inspection target portion associated with each cell of the selected cell group CE3. Next, the image processing apparatus 20 acquires the inspection results and measurement values corresponding to each combination of the identified work No. and inspection target portion from the inspection result file 146D. Next, the image processing device 20 executes predetermined statistical processing on the acquired measurement values.

As an example, the image processing device 20 generates a frequency distribution (that is, a histogram) by executing predetermined statistical processing. The horizontal axis of the histogram represents the divisions of the measured values, and the horizontal axis of the histogram represents the frequency of the measured values included in each division. The display control unit 22 of the image processing device 20 displays the generated histogram on the popup screen PU3 associated with the cell group CE3.

Preferably, a frequency distribution of the measurements is generated for each selected inspected portion. In the example of FIG. 26, the frequency distribution is shown for each of the selected inspection target portions A2 and A3.

As another example, the image processing device 20 generates the measured value transition graph by executing predetermined statistical processing. The horizontal axis of the measured value transition graph represents the workpiece number, and the vertical axis of the measured value transition graph represents the measured value. The display control unit 22 of the image processing device 20 displays the generated measured value transition graph on the popup screen PU3 associated with the cell group CE3.

Preferably, the measured value transition graph is generated for each inspection target portion corresponding to the selected cell group CE3. In the example of FIG. 26, measurement value transition graphs are shown for each of the selected inspection target portions A2 and A3.

In the example of FIG. 26, an example has been described in which two statistical results, the histogram and the measured value transition graph, are displayed in accordance with the selection operation of the cell group. At least one statistical result may be displayed according to the operation.

(N4. Specific example 4)
FIG. 27 is a diagram showing a specific example 4 of processing executed in response to a cell selection operation. In the example of FIG. 27, the cell group CE4 within the inspection result matrix 25 is selected.

Based on the selection of the cell group CE4, the inspection unit 21 of the image processing device 20 re-inspects the inspection target portion based on the image used for the inspection of the inspection target portion corresponding to the cell group CE4. . This allows the user to easily re-inspect any portion to be inspected.

More specifically, the user resets the inspection conditions and then selects the cell group CE4 of the inspection result matrix 25 . The inspection unit 21 identifies a combination of the work No. and inspection target portion associated with each cell of the selected cell group CE4. Next, the inspection unit 21 acquires the image data corresponding to each combination of the specified work No. and inspection target portion from the image file group 146B described above. Next, the inspection unit 21 acquires inspection conditions corresponding to each of the specified inspection target portions from the inspection condition file 146C described above. Next, the inspection unit 21 executes image processing according to the corresponding inspection conditions for each of the acquired image data. Next, the display control unit 22 reflects the results of the re-inspection on the inspection result matrix 25 .

In the example of FIG. 27, an example in which a plurality of cells are selected in the inspection result matrix 25 has been described, but in this example, the number of cells to be selected may be one or more.

(N5. Specific example 5)
FIG. 28 is a diagram showing a specific example 5 of processing executed in response to a cell selection operation. In the example of FIG. 28, cell group CE5 in inspection result matrix 25 is selected.

Based on the selection of the cell group CE5, the display control unit 22 of the image processing device 20 displays the three-dimensional model ML of the workpiece to be inspected on the display device 23, and displays the inspection target portion corresponding to the cell group CE5. is represented on the corresponding portion on the three-dimensional model ML. This allows the user to easily confirm the imaging conditions of the selected portion.

More specifically, the display control unit 22 of the image processing device 20 stores the three-dimensional model ML of the work to be inspected in the work information file 146E (see FIG. 11) based on the selection of the cell group CE5. , and displays the three-dimensional model ML on the pop-up screen PU5 associated with the cell group CE5.

Next, the combination of the work No. and inspection target portion associated with each selected cell is specified. Next, the image processing apparatus 20 acquires the imaging conditions corresponding to each combination of the identified work No. and inspection target portion from the imaging condition file 146A (see FIG. 11).

Next, the display control unit 22 of the image processing device 20 determines one work number (for example, the smallest work number) from among the work numbers corresponding to the cell group CE5, and corresponds to the determined work number. Each imaging condition to be used is represented in the corresponding part of the three-dimensional model ML. A return button 81 and a forward button 82 are displayed on the pop-up screen PU5, and the user can switch the imaging conditions to be displayed in order of the work number by pressing the return button 81 or the forward button 82. can.

The displayed imaging conditions include the imaging position of the imaging device 10 with respect to the three-dimensional model ML. As described above with reference to FIG. 19, the imaging position Ci of the imaging device 10 with respect to the three-dimensional model ML is determined by the setting process by the setting device 60, so the imaging position Ci of the imaging device 10 with respect to the three-dimensional model ML is Known. Therefore, the display control unit 22 of the image processing device 20 can display a schematic diagram representing the imaging device 10 on the shooting position Ci based on this known information. In the example of FIG. 28, schematic diagrams representing the imaging device 10 are displayed at the imaging positions C5 and C7.

As another example, the displayed imaging conditions include inspection target portions on the workpiece. As described above with reference to FIGS. 13 to 18, the position to be inspected Bi is set with respect to the three-dimensional model ML, so the relationship between the three-dimensional model ML and the position to be inspected Bi is known. Therefore, the display control unit 22 of the image processing device 20 can reflect the inspection result of each inspection target portion on the three-dimensional model ML based on this known information. In the example of FIG. 28, inspection target portions A5 and B1 are represented on the three-dimensional model ML.

In the above example, an example in which the imaging position and the portion to be inspected are displayed as the imaging conditions has been described, but the displayed imaging conditions are not limited to the imaging position and the portion to be inspected. For example, the optical conditions (imaging field of view, etc.) of the imaging device 10 at the time of imaging, the illumination conditions at the time of imaging, and the like may be displayed.

<O. Modified Example of Inspection Result Matrix 25>
FIG. 29 is a diagram showing a modification of the inspection result matrix 25 shown in FIG. Each row of the inspection result matrix 25 shown in FIG. 3 is grouped by work lot number, while each row of the inspection result matrix 25 shown in FIG. 29 is grouped by work type.

Workpieces of different types may have common parts to be inspected, or may have unique parts to be inspected. Therefore, when the inspection results of each work of different types are represented on one inspection result matrix 25, some ingenuity is required.

As an example, it is assumed that the work type α has unique inspection target portions "A1" and "A3" that are not found in the work type β. In this case, as shown in FIG. 29, the inspection results are reflected in the cells corresponding to the inspection target portions "A1" and "A3" of the workpiece type α, but the inspection results are reflected in the cells corresponding to the inspection target portions "A1" and "A3" of the workpiece type β. The cell corresponding to "A3" is displayed blank as "not applicable".

In addition, it is assumed that the work type β has unique inspection target portions "A5" and "B3" that are not found in the work type α. In this case, as shown in FIG. 29, the inspection results are reflected in the cells corresponding to the inspection target portions "A5" and "B3" of the workpiece type β, but the inspection results are reflected in the cells corresponding to the inspection target portions "A5" and "B3" of the workpiece type α. The cell corresponding to "B3" is displayed blank as "not applicable".

It is also assumed that the workpiece types α and β have common inspected parts "A2", "A4", "B1" and "B2". In this case, as shown in FIG. 29, cells corresponding to inspection target portions "A2", "A4", "B1" and "B2" of work type α and inspection target portions "A2", "B2" of work type β The inspection result is reflected in both the cells corresponding to "A4", "B1" and "B2".

The parts to be inspected for each type of workpiece are defined, for example, by classification rules 134C shown in FIG. FIG. 30 is a diagram showing an example of the data structure of the classification rule 134C.

As shown in FIG. 30, the inspection target portion of the workpiece W is hierarchically associated with the classification rule 134C. In the example of FIG. 30, "A1" to "A4" are associated with the inspection target portion "A" of the workpiece type α. "B1" and "B2" are associated with the inspection target portion "B" of the workpiece type α. "A2", "A4", and "A5" are associated with the inspection target portion "A" of the workpiece type β. "B1" to "B3" are associated with the inspection target portion "B" of the workpiece type β.

Referring to FIG. 29 again, a button is assigned to each group in the vertical and horizontal directions of inspection result matrix 25 . In the example of FIG. 29, an expand/aggregate button BH1 is assigned to the group GH1. An expand/aggregate button BH2 is assigned to the group GH2. An expand/aggregate button BV1 is assigned to the group GV1. An expand/aggregate button BV2 is assigned to the group GV2.

FIG. 31 is a diagram showing screen transition of the inspection result matrix 25 when the expand/consolidate button BV2 is pressed. The expand/aggregate button BV2 alternately accepts an aggregate instruction and an expand instruction for the cells of the group GV2 each time it is pressed. That is, when the expand/aggregate button BV2 is pressed while the cells of the group GV2 are expanded, the display control unit 22 of the image processing device 20 aggregates the cells of the group GV2 in a row. On the other hand, when the expand/aggregate button BV2 is pressed while the cells of the group GV2 are aggregated, the display control unit 22 restores the display of the cells of the group GV2 to the state before aggregation.

When the inspection results of "defective", "no defect", and "not applicable" are included in the aggregation target cell, the display control unit 22 changes the display of "defective" to "no defect", "not applicable", and "not applicable". Aggregation processing is performed with priority over the display of "None". Further, when the inspection results of “no defect” and “not applicable” are included in the cell to be aggregated, the display control unit 22 gives priority to the display of “no defect” over the display of “not applicable”. Aggregation processing is performed by

As an example, the dashed line AR10 includes one "defective" cell and two "non-applicable" cells. The aggregation process aggregates the three cells within the dashed line AR10 into one cell indicated within the dashed line AR12. At this time, since the cells before aggregation include cells with "defective" cells, the display control unit 22 displays the inspection results after aggregation as "defective".

As another example, the dashed line AR11 includes two "defect-free" cells and one "non-defective" cell. By the aggregation process, the three cells within the dashed line AR11 are aggregated into one cell indicated within the dashed line AR13. At this time, the cells before aggregation do not include cells with "defective" cells, but include cells with "no defects". represented as “no”.

FIG. 32 is a diagram showing screen transition of the inspection result matrix 25 when the expand/consolidate button BH2 is pressed. The expand/aggregate button BH2 alternately accepts an aggregate instruction and an expand instruction for the cells of the group GH2 each time it is pressed. That is, when the expand/aggregate button BH2 is pressed while the cells of the group GH2 are expanded, the display control unit 22 aggregates the cells of the group GH2 into one line. On the other hand, when the expand/aggregate button BH2 is pressed while the cells of the group GH2 are aggregated, the display control unit 22 restores the display of the cells of the group GH2 to the state before aggregation.

Ten "non-applicable" cells are included within the dashed line AR14. 10 cells within the dashed line AR14 are aggregated into one cell indicated within the dashed line AR15 by the aggregation processing of the cells of the group GH2. At this time, since the cells before aggregation do not include cells with "defective" and "non-defective" cells, the display control unit 22 displays the inspection results after aggregation as "not applicable".

<P. Comparison function between inspection result matrix and expected value matrix>
The image processing device 20 displays a result of comparison between an expected value matrix indicating true correct values (hereinafter also referred to as "expected values") for each inspection result included in the inspection result matrix 25 and the inspection result matrix 25. Displayed on device 23 . This allows the user to easily determine whether each test result shown in the test result matrix 25 is as expected. Such comparison processing becomes more effective as the number of inspection results shown in the inspection result matrix 25 increases.

FIG. 33 is a diagram schematically showing comparison processing between the inspection result matrix 25 and the expected value matrix 27. As shown in FIG.

The cells in the expected value matrix 27 define expected values for at least some of the inspection results included in the inspection result matrix 25 . The expected value of each cell of the expected value matrix 27 is set by user input, for example. Expected values that can be entered are, for example, one of "defective", "no defect", and "invalid". The expected value matrix 27 set by the user is stored in the image processing apparatus 20 as the expected value file 146H (see FIG. 11).

The image processing device 20 compares the cells in the same row and the same column between each cell in the inspection result matrix 25 and each cell in the expected value matrix 27 .

More specifically, when the inspection result of the inspection result matrix 25 is "no defect" and the expected value of the expected value matrix 27 is "no defect", "correct answer OK" is output as the comparison result. That is, "correct answer OK" means that the test result is as expected.

When the inspection result of the inspection result matrix 25 is "defective" and the expected value of the expected value matrix 27 is "defective", "correct answer NG" is output as the comparison result. That is, "correct answer NG" means that the test result is as expected.

When the inspection result of the inspection result matrix 25 is "no defect" and the expected value of the expected value matrix 27 is "defective", "missed" is output as the comparison result. "missed" means that a defect to be detected has been overlooked. That is, "missed" indicates that the inspection result is not as expected. One of the reasons why the comparison result is "missed" is that the threshold for judging the presence/absence of defects is too loose.

When the inspection result of the inspection result matrix 25 is "defective" and the expected value of the expected value matrix 27 is "no defect", "overdetection" is output as the comparison result. "Overdetection" means that a normal part has been detected as a defect. That is, "over-detection" indicates that the test result is not as expected. One of the reasons why the comparison result is "overdetected" is that the threshold for judging whether there is a defect or not is too strict.

If at least one of the inspection result of the inspection result matrix 25 and the expected value of the expected value matrix 27 is "invalid", "invalid" is output as the comparison result. "Invalid" means that at least one of the test result and the expected value does not exist.

A comparison result matrix 29 is output as a comparison result between the inspection result matrix 25 and the expected value matrix 27 . The display control unit 22 displays the comparison results “correct answer OK”, “correct answer NG”, “oversight”, “overdetection”, and “invalid” in a distinguishable manner. As an example, these comparison results may be distinguished by color or the type of hatching.

Note that the expected comparison results of “correct answer OK” and “correct answer NG” may be displayed in the same color (for example, white).

As with the inspection result matrix 25, the comparison result matrix 29 may also be implemented with a cell aggregation/development function. As an example, if even one "overlooked" or "overdetected" cell is included in the cells to be aggregated, the display control unit 22 sets the aggregated cell as "overlooked" or "overdetected." If the cells to be aggregated include both "missed" and "overdetected", the display control unit 22 displays the cells to be aggregated in a display manner indicating that both "missed" and "overdetected" are included. represents

<Q. Expected value matrix creation support function>
FIG. 34 is a diagram showing an example of the process of creating the expected value matrix 27. As shown in FIG. The image processing device 20 has a function of supporting creation of the expected value matrix 27 .

More specifically, the user selects each cell of test result matrix 25 . A cell selection operation is performed on an operation unit provided in the image processing apparatus 20 . The operating unit includes, for example, a keyboard 134 (see FIG. 8), a mouse, a touch panel, and the like.

As an example, assume that the user has selected cell group CE10. After that, the user copies the cell group CE10 of the inspection result matrix 25 to the expected value matrix 27 being edited. This eliminates the need for the user to input each cell of the expected value matrix 27 one by one, thereby reducing the labor involved in creating the expected value matrix 27 .

The expected value matrix 27 created by the user is stored in the image processing apparatus 20 as the expected value file 146H (see FIG. 11).

<R. Flow of three-dimensional display of inspection results>
The image processing device 20 has a function of displaying the inspection result in three dimensions by reflecting the inspection result of the workpiece on the three-dimensional model ML. The user can easily determine the location of the defect by checking the inspection result on the three-dimensional model ML.

Three-dimensional display processing for inspection results will be described below with reference to FIG. 35 . FIG. 35 is a flowchart showing an example of the flow of three-dimensional display processing for inspection results. The processing shown in FIG. 35 is implemented by the processor 110 of the image processing device 20 executing the display program 143 (see FIG. 8). Note that part or all of the processing may be performed by circuit elements or other hardware.

In step S<b>90 , processor 110 determines whether or not an operation to execute three-dimensional display processing of inspection results has been received. The display operation is performed on the operation unit provided in the image processing device 20 . The operating unit includes, for example, a keyboard 134 (see FIG. 8), a mouse, a touch panel, and the like. When processor 110 determines that an operation to execute the inspection result three-dimensional display process has been received (YES in step S90), processor 110 switches control to step S92. Otherwise (NO in step S90), processor 110 executes the process of step S90 again.

In step S92, the processor 110, as the above-described display control unit 22 (see FIG. 1), acquires the three-dimensional model ML of the workpiece from the above-described workpiece information file 146E (see FIG. 11), and stores the acquired three-dimensional model ML as It is displayed on the display device 23 .

In step S100, processor 110 determines whether or not an instruction to update three-dimensional display of inspection results has been received. The update instruction is issued based on, for example, new test results being obtained from the test unit 21 . Alternatively, the update instruction is issued based on detection of an inspection result indicating a defect by the inspection unit 21 . Alternatively, the update instruction is issued based on a user operation. When processor 110 determines that an instruction to update the three-dimensional display of the inspection results has been received (YES in step S100), processor 110 switches control to step S102. Otherwise (NO in step S100), processor 110 switches control to step S120.

In step S102, the processor 110 refers to the inspection result file 146D (see FIG. 12) to acquire the inspection result for each inspection target portion of the work No. of the workpiece to be inspected.

In step S104, processor 110, as above-described display control unit 22, displays each inspection result obtained in step S102 in the corresponding location on the three-dimensional model ML displayed in step S92. As described above with reference to FIGS. 13 to 18, the position to be inspected Bi is set with respect to the three-dimensional model ML, so the relationship between the three-dimensional model ML and the position to be inspected Bi is known. Therefore, the display control unit 22 of the image processing device 20 can reflect each inspection result obtained in step S102 on the three-dimensional model ML based on this known information.

Typically, processor 110 displays the defect-indicating portion in a different display manner from other portions when displaying the inspection result of each inspection target portion on the corresponding portion on the three-dimensional model ML. As an example, the portion indicating the defect may be indicated in a specific color (eg, red) or may be indicated by blinking. This makes it easier for the user to determine the portion showing the defect.

In step S110, processor 110 determines whether any of the parts to be inspected shown on the three-dimensional model have been selected. The selection operation is performed on the operation unit provided in the image processing device 20 . The operating unit includes, for example, a keyboard 134 (see FIG. 8), a mouse, a touch panel, and the like. If processor 110 determines that any of the parts to be inspected represented on the three-dimensional model has been selected (YES in step S110), processor 110 switches control to step S112. Otherwise (NO in step S110), processor 110 switches control to step S120.

In step S112, the processor 110, as the inspection unit 21 or the display control unit 22 described above, executes processing according to the inspection target portion selected in step S110. Details of the processing executed according to the selected inspection target portion will be described later.

In step S120, processor 110 determines whether or not an operation to close the three-dimensional display screen of inspection results has been received. When processor 110 determines that an operation to close the three-dimensional display screen of the inspection results has been accepted (YES in step S120), processor 110 ends the process shown in FIG. Otherwise (NO in step S120), processor 110 returns control to step S100.

As described above, the three-dimensional display of the inspection results is updated when the instruction to update the three-dimensional display of the inspection results is issued in step S100. As described above, the update instruction is issued, for example, based on detection of an inspection result indicating a defect by inspection unit 21 . That is, the processor 110 updates the inspection result represented in the three-dimensional model ML with the inspection result of each inspection target portion of the workpiece when a defect is detected in the sequentially inspected workpiece. As a result, the inspection results displayed on the three-dimensional model ML are not updated while inspection results indicating defects are not detected. As a result, the latest inspection results indicating defects are always displayed on the three-dimensional model ML. Therefore, the user is less likely to overlook inspection results that indicate defects.

<S. Selection Function of Inspection Target Portion for 3D Model>
As shown in steps S110 and S112 in FIG. 35, the image processing apparatus 20 selects one of the inspection target portions shown in the three-dimensional model ML, and adjusts the selected inspection target portion. Execute the process. Examples of the processing that can be executed include the processing of specific examples 1 and 2 described below. These processes will be described in order below.

Note that typically, the image processing apparatus 20 executes at least one of the processes shown in specific examples 1 and 2 below, based on receiving an operation to select a portion to be inspected on the three-dimensional model ML. . Alternatively, the image processing apparatus 20 displays a selection screen for processing shown in the following specific examples 1 and 2 based on receiving an operation for selecting a portion to be inspected on the three-dimensional model ML, and selects on the selection screen. process may be executed.

(S1. Specific example 1)
FIG. 36 is a diagram showing a specific example 1 of processing executed in response to a selection operation of a portion to be inspected shown in the three-dimensional model ML.

As shown in FIG. 36, the display device 23 has a three-dimensional model ML of a workpiece to be inspected, a tool button 71 for rotating the three-dimensional model ML about the vertical axis, and three buttons about the horizontal axis. A tool button 72 for rotating the dimensional model ML is displayed. The user can appropriately rotate the three-dimensional model ML by operating the tool buttons 71 and 72 .

As described above with reference to FIGS. 13 to 18, the position to be inspected Bi is set with respect to the three-dimensional model ML, so the relationship between the three-dimensional model ML and the position to be inspected Bi is known. Therefore, the display control unit 22 of the image processing device 20 can represent the inspection target portion on the three-dimensional model ML based on this known information. In the example of FIG. 36, inspected portions A1 to A4 are represented in the three-dimensional model ML.

In addition, the display control unit 22 displays the inspection results in inspection target portions A1 to A4 on the three-dimensional model ML. As an example, the portion indicating "defective" may be indicated in a specific color (eg, red) or may be indicated by blinking. The part indicating "no defect" is represented in a different color (eg, green) from the part indicating "defective".

In the example of FIG. 36, the inspection target portion A1 indicates "defective", and the inspection target portions A2 to A4 indicate "no defect". Preferably, the display control unit 22 highlights the defective portion A1_1 indicating the defect in the inspection target portion A1 more than other portions.

The user can select any one of the parts A1 to A4 to be inspected. In the example of FIG. 36, the inspection target portion A1 is selected. When the inspection target portion A1 is selected, the image processing device 20 displays the image used for the inspection of the inspection target portion A1 and the imaging conditions of the image on the display device 23 . As a result, the user can visually confirm the image showing the defect and confirm the validity of the imaging conditions.

More specifically, based on the selection of the inspection target portion A1, the image processing apparatus 20 generates the corresponding image file using the combination of the work No. of the inspection target work and the selected inspection target portion A1 as a key. is obtained from the image file group 146B (see FIG. 11). Next, the image processing apparatus 20 acquires the corresponding imaging conditions from the above-described imaging condition file 146A (see FIG. 11) using the inspection target portion A1 as a key. The display control unit 22 displays the acquired image file and imaging conditions on the pop-up screen PU6 associated with the selected inspection target portion A1. The displayed imaging conditions are configured to be changeable to arbitrary values.

In the example of FIG. 36, the example in which both the captured image and the imaging conditions are displayed in accordance with the selection operation of the inspection target portion has been described. Either one of the captured image and the imaging conditions may be displayed accordingly.

(S2. Specific example 2)
FIG. 37 is a diagram showing a specific example 2 of processing executed in response to a selection operation of a portion to be inspected shown in the three-dimensional model ML.

The three-dimensional model ML includes inspected parts A1 to A4. The user can select any one of the parts A1 to A4 to be inspected. In the example of FIG. 37, the inspection target portion A1 is selected. Based on the selection of the portion A1 to be inspected, the image processing apparatus 20 sets the corresponding imaging conditions using the combination of the work No. of the work to be inspected and the selected portion A1 to be inspected as a key to the above-described imaging condition file. 146A (see FIG. 11). Next, the display control unit 22 of the image processing device 20 displays the acquired imaging conditions in the corresponding locations of the three-dimensional model ML.

As an example, the displayed imaging conditions include the imaging position of the imaging device 10 with respect to the three-dimensional model ML. As described above with reference to FIG. 19, the imaging position Ci of the imaging device 10 with respect to the three-dimensional model ML is determined by the setting process by the setting device 60, so the imaging position Ci of the imaging device 10 with respect to the three-dimensional model ML is Known. Therefore, the display control unit 22 of the image processing device 20 can display the schematic diagram 10A representing the imaging device 10 on the imaging position Ci based on this known information. In the example of FIG. 37, a schematic diagram 10A of the imaging device 10 is displayed at the imaging position C1.

In the above example, an example in which the imaging position is displayed as the imaging condition has been described, but the displayed imaging condition is not limited to the imaging position. For example, the optical conditions of the imaging device 10 at the time of imaging, the illumination conditions at the time of imaging, and the like may be displayed.

<T. Expansion/aggregation function of inspection target parts>
FIG. 38 is a diagram showing the process of expanding/consolidating the portion to be inspected shown in the three-dimensional model ML.

The parts to be inspected shown in the three-dimensional model ML are hierarchically associated with the above-described classification rules 134A (see FIG. 4). A portion to be inspected defined in a higher hierarchy has a relationship of including a portion to be inspected defined in a lower hierarchy. A lower inspection target portion included in an inspection target portion defined in a higher hierarchy is regarded as belonging to the same group.

In the example of FIG. 38, parts A to C to be inspected are shown on the three-dimensional model ML. As an example, it is assumed that a high-level inspection target portion A is associated with low-level inspection target portions A1 to A3. The inspection target portions A1 to A3 included in the upper inspection target portion A are regarded as belonging to the same group.

It is assumed that lower inspection target portions A1_1 to A1_4 are associated with the upper inspection target portion A1. Inspection target portions A1_1 to A1_4 included in the upper inspection target portion A1 are regarded as belonging to the same group.

The inspection results are displayed on inspection target portions A to C on the three-dimensional model ML. The inspection results of inspection target portions A to C are obtained from the inspection result file 146D (see FIG. 12). As an example, inspection target portion A indicates "defective", and inspection target portions B and C indicate "no defect".

An expansion/consolidation button BT1 is assigned to the inspection target portion A. As shown in FIG. An expansion/consolidation button BT2 is assigned to the portion B to be inspected. An expansion/consolidation button BT3 is assigned to the inspection target portion C. FIG. "+" indicates an expand button, and "-" indicates an aggregate button.

The display control unit 22 aggregates the inspection results of the grouped inspection target portions when receiving an aggregation instruction for the grouped inspection target portions, and displays the inspection results after the aggregation as the grouped inspection target portions. Each portion on the three-dimensional model ML corresponding to the portion to be inspected is represented. Further, when receiving an expansion instruction for the aggregated test results, the display control unit 22 restores the display of the aggregated test results to the state before the aggregation.

For example, when the expand/consolidate button BT1 is pressed, the display control unit 22 expands the inspection target portion A into lower inspection target portions A1 to A3. Next, the display control unit 22 reflects the inspection results of the inspection target portions A1 to A3 on the inspection target portions A1 to A3. The inspection results of the inspection target portions A1 to A3 are obtained from the above inspection result file 146D (see FIG. 12). As an example, the inspection target portion A1 indicates "defective", and the inspection target portions A2 and A3 indicate "no defect".

Next, the display control unit 22 assigns the expand/consolidate button BT1_1 to the inspection target portion A1. Similarly, the display control unit 22 assigns the expand/consolidate button BT1_2 to the inspection target portion A2. Similarly, the display control unit 22 assigns the expand/consolidate button BT1_3 to the inspection target portion A3.

When the "+" of the expand/consolidate button BT1_1 is pressed, the display control unit 22 expands the inspection target portion A1 into lower inspection target portions A1_1 to A1_4. Next, the display control unit 22 reflects the inspection results of the inspection target portions A1_1 to A1_4 on the inspection target portions A1_1 to A1_4. The inspection results of inspection target portions A1_1 to A1_4 are obtained from the inspection result file 146D (see FIG. 12) described above. As an example, inspection target portions A1_1 to A1_3 indicate “no defect”, and inspection target portion A1_4 indicates “defective”.

When the expand/aggregate button BT1_1A is pressed, the display control unit 22 aggregates the inspection target portions A1_1 to A1_4 into the upper inspection target portion A1. At this time, if even one inspection result indicating "defective" is included in the inspection results of the inspection target portion to be aggregated, the display control unit 22 sets the inspection result of the inspection target portion after aggregation to "defective". ”. On the other hand, if the inspection results indicating "defective" are not included in the inspection results of the inspection target portion to be aggregated, the display control unit 22 sets the inspection result of the inspection target portion after aggregation to "no defect". ”. Since the inspection target portion A1_4 indicates “defective”, the display control unit 22 sets the inspection result of the inspection target portion A1, which is a collection of the inspection target portions A1_1 to A1_4, to “defective”.

When the "-" of the expand/aggregate button BT1 is pressed, the display control unit 22 aggregates the inspection target portions A1 to A3 into the inspection target portion A. FIG. Since the inspection target portion A1 indicates "defective", the display control unit 22 sets the inspection result of the inspection target portion A, which is an aggregation of the inspection target portions A1 to A3, as "defective".

<U. Modified example of appearance inspection system>
FIG. 39 is a diagram showing a visual inspection system according to a modification. The appearance inspection system shown in FIG. 39 differs from the appearance inspection system 1 shown in FIG. The image processing device 20a has both the configuration of the image processing device 20 and the configuration of the PLC 50 described above.

FIG. 40 is a diagram showing another form of changing the relative position between the work W and the imaging device 10. FIG. As shown in FIG. 40 , the robot 30 may move the workpiece W instead of the imaging device 10 . In the example shown in FIG. 40, the imaging device 10 is fixed. By moving the work W in this manner, the relative position between the work W and the imaging device 10 may be changed.

FIG. 41 is a diagram showing still another form of changing the relative position between the work W and the imaging device 10. FIG. As shown in FIG. 41, the work W may be placed on a rotary table 91. As shown in FIG. The rotary table 91 rotates according to instructions from the robot controller 40 . Thereby, the relative position between the workpiece W and the imaging device 10 can be easily changed.

Note that the robot 30 may be a robot (for example, a horizontal articulated robot, an orthogonal robot, etc.) other than the vertical articulated robot.

Although the imaging field of view FOV and the effective field of view FOV2 are circular in the above description, the shapes of the imaging field of view FOV and the effective field of view FOV2 are not limited to circular, and may be rectangular (rectangular, square), for example.

<V. Note>
As described above, the present embodiment includes the following disclosures.

[Configuration 1]
A visual inspection system (1) for performing a visual inspection of an inspection object (W),
a display device (23);
a robot (30) for moving the imaging device (10);
While the robot (30) is moving the imaging device (10), the imaging device (10) images each of the plurality of inspection portions of the inspection object (W) to the imaging device (10). an inspection unit (21) for inspecting the presence or absence of defects in each of the plurality of inspection portions based on each image obtained from
a display control unit for displaying, on the display device (23), an inspection result matrix (25) representing inspection results by the inspection unit (21) for each of the plurality of inspection parts for each inspection object (W); (22), and a visual inspection system.

[Configuration 2]
Each row or each column of the test result matrix (25) is grouped by row or column according to a predetermined classification rule,
The display control unit (22)
When receiving an instruction to aggregate a group of rows or a group of columns, aggregate and display the inspection results of the group of rows or the group of columns in one row or one column,
The appearance inspection system according to configuration 1, wherein when an expansion instruction for the inspection results that are aggregated and displayed is received, the display of the inspection results that are aggregated and displayed is returned to the state before aggregation.

[Configuration 3]
In configuration 1 or 2, the display control unit (22) displays an inspection result indicating a defect among the inspection results included in the inspection result matrix (25) in a display mode different from other inspection results. Appearance inspection system as described.

[Configuration 4]
The visual inspection system (1) accepts a selection operation for selecting one or more inspection results from among the inspection results contained in the inspection result matrix (25), thereby obtaining one or more inspection objects (W) and one or more test portions; and
The inspection unit (21) or the display control unit (22) selects an inspection object (W) and a selected inspection portion based on the acceptance of the selection operation by the operation unit (134). 4. The visual inspection system according to any one of configurations 1 to 3, which performs processing related to at least one of

[Configuration 5]
The appearance according to configuration 4, wherein the display control unit (22) displays at least one of an image used for inspection of the selected inspection portion and imaging conditions of the image on the display device (23). inspection system.

[Configuration 6]
The visual inspection system (1) further comprises a storage device for storing a three-dimensional model (ML) representing the shape of the inspection object (W), and the plurality of inspected parts are the three-dimensional model (ML ) is preset for
The display control unit (22) displays the three-dimensional model (ML) on the display device (23), and displays the inspection result of the selected inspection part on the corresponding part on the three-dimensional model (ML). 6. A visual inspection system according to configuration 4 or 5, wherein:

[Configuration 7]
Any one of configurations 4 to 6, wherein the display control unit (22) displays statistical results of the plurality of test results on the display device (23) when a plurality of test results are selected by the selection operation. Appearance inspection system described in the item.

[Configuration 8]
Appearance inspection according to any one of configurations 4 to 7, wherein the inspection unit (21) reinspects the selected inspection portion based on an image used for inspection of the selected inspection portion. system.

[Configuration 9]
The display control unit (22) displays a comparison result between an expected value matrix indicating a true correct value for each inspection result contained in the inspection result matrix (25) and the inspection result matrix (25) on the display device. The visual inspection system according to any one of configurations 1 to 8, displayed in (23).

[Configuration 10]
A method for displaying results of visual inspection performed by imaging a plurality of inspection portions of an inspection object (W) with an imaging device (10),
obtaining each image obtained by imaging each of the plurality of inspected parts with the imaging device (10) while the robot (30) is moving the imaging device (10);
inspecting each of the plurality of inspected parts for defects based on each image obtained in the obtaining step;
A method for displaying an appearance inspection result, comprising the step of displaying, on a display device (23), an inspection result matrix (25) representing an inspection result for each of the plurality of inspection portions for each inspection object (W).

[Configuration 11]
A program for displaying results of appearance inspection performed by imaging a plurality of inspection portions of an inspection object (W) with an imaging device (10),
The display program, in a computer,
obtaining each image obtained by imaging each of the plurality of inspected parts with the imaging device (10) while the robot (30) is moving the imaging device (10);
inspecting each of the plurality of inspected parts for defects based on each image obtained in the obtaining step;
A visual inspection result display program for displaying, on a display device (23), an inspection result matrix (25) representing inspection results for each of the plurality of inspection parts (W) for each inspection object (W).

It should be considered that the embodiments disclosed this time are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the above description, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

1 appearance inspection system 10 imaging device 10A schematic diagram 20, 20a image processing device 21 inspection unit 22 display control unit 23 display device 25 inspection result matrix 27 expected value matrix 29 comparison result matrix 30 robot , 31 base, 32 arm, 32a tip arm, 40 robot controller, 50 PLC, 60 setting device, 61a, 61b screen, 71, 72, 76, 77 tool button, 74 circle, 75, 75A, 75B inspection target area , 81 back button, 82 forward button, 90 stage, 91 rotary table, 110, 214, 362 processor, 112 RAM, 114 display controller, 116 system controller, 118 I/O controller, 120, 364 storage device, 122 camera interface, 124 input interface, 126 controller interface, 128, 228, 368 communication interface, 130, 222 memory card interface, 134 keyboard, 134A, 134B, 134C classification rule, 136, 224 memory card, 142 image processing program, 143 display program, 144 Project file, 146A Imaging condition file, 146B Image file group, 146C Inspection condition file, 146D Inspection result file, 146E Work information file, 146F Production information file, 146G Classification rule file, 146H Expected value file, 212 Chipset, 216 Nonvolatile memory, 218 main memory, 220 system clock, 226 internal bus, 230 internal bus controller, 232 control circuit, 234 internal bus control circuit, 236 buffer memory, 238 field bus controller, 361 bus, 363 main memory, 365 setting program, 366 Display, 367 input device.

Claims (10)

A visual inspection system for performing a visual inspection of an object to be inspected,
a display device;
a robot for moving the imaging device;
While the robot is moving the imaging device, the imaging device images each of the plurality of inspection portions of the inspection object, and based on each image obtained from the imaging device, the plurality of inspection portions. An inspection unit for inspecting the presence or absence of defects for each of
a display control unit for displaying, on the display device, an inspection result matrix showing inspection results by the inspection unit for each of the plurality of inspection parts for each inspection object;
Each row or each column of the inspection result matrix is grouped into a plurality of row groups or a plurality of column groups in units of rows or columns according to a predetermined classification rule,
The display control unit
displaying a button operated to accept an instruction to aggregate or expand the group in association with each of the plurality of grouped rows or each of the plurality of grouped columns;
When the button is operated, the test results displayed in expanded rows or columns associated with the button are aggregated and displayed in one row or one column, and the display mode of the button is changed. ,
An appearance inspection system that, when the button is operated, restores the display of the aggregated and displayed inspection results associated with the button to the state before aggregation, and returns the button to the display mode before the aggregation. 2. The appearance inspection system according to claim 1, wherein said display control unit displays inspection results indicating defects among the inspection results included in said inspection result matrix in a display mode different from other inspection results. The visual inspection system receives a selection operation for selecting one or more inspection results from among the inspection results contained in the inspection result matrix, thereby selecting one or more inspection objects and one or more inspection portions. an operation unit that can be selected;
2. The inspection unit or the display control unit executes processing related to at least one of the selected inspection object and the selected inspection portion based on the selection operation received by the operation unit. Or the appearance inspection system according to 2. 4. The appearance inspection system according to claim 3, wherein said display control unit displays at least one of an image used for inspection of said selected inspection portion and imaging conditions of said image on said display device. The visual inspection system further comprises a storage device for storing a three-dimensional model representing the shape of the inspection object, wherein the plurality of inspection parts are set in advance for the three-dimensional model,
5. The appearance inspection according to claim 3, wherein said display control unit displays said three-dimensional model on said display device, and expresses the inspection result of said selected inspection portion on a corresponding portion on said three-dimensional model. system. The appearance according to any one of claims 3 to 5, wherein, when a plurality of inspection results are selected by the selection operation, the display control unit displays statistical results of the plurality of inspection results on the display device. inspection system. The appearance inspection system according to any one of claims 3 to 6, wherein the inspection unit re-inspects the selected inspection portion based on the image used for inspection of the selected inspection portion. 8. The display control unit displays, on the display device, a comparison result between an expected value matrix indicating a true correct value for each inspection result included in the inspection result matrix and the inspection result matrix. Appearance inspection system according to any one of the. A method for displaying a result of an appearance inspection performed by imaging a plurality of inspection portions of an object to be inspected by an imaging device,
obtaining each image obtained by imaging each of the plurality of inspected parts with the imaging device while the robot is moving the imaging device;
inspecting each of the plurality of inspected parts for defects based on each image obtained in the obtaining step;
displaying, on a display device, an inspection result matrix representing inspection results for each of the plurality of inspection portions for each inspection object;
Each row or each column of the inspection result matrix is grouped into a plurality of row groups or a plurality of column groups in units of rows or columns according to a predetermined classification rule,
In the displaying step,
A button operated to accept an instruction to aggregate or expand the group is displayed in association with each of the plurality of grouped row groups or each of the grouped plurality of column groups,
When the button is operated, the test results displayed in expanded rows or columns associated with the button are aggregated and displayed in one row or one column, and the display mode of the button is changed. is,
Appearance inspection results, wherein when the button is operated, the display of the aggregated inspection results associated with the button is returned to the state before aggregation, and the button is returned to the display mode before aggregation. display method. A program for displaying results of appearance inspection performed by imaging a plurality of inspection portions of an inspection object with an imaging device,
The display program, in a computer,
obtaining each image obtained by imaging each of the plurality of inspected parts with the imaging device while the robot is moving the imaging device;
inspecting each of the plurality of inspected parts for defects based on each image obtained in the obtaining step;
displaying, on a display device, an inspection result matrix showing inspection results for each of the plurality of inspection portions for each inspection object;
Each row or each column of the inspection result matrix is grouped into a plurality of row groups or a plurality of column groups in units of rows or columns according to a predetermined classification rule,
In the displaying step,
A button is displayed in association with each of the plurality of grouped row groups or each of the grouped plurality of column groups, and operated to accept an instruction to aggregate or expand the group,
When the button is operated, the test results displayed in expanded rows or columns associated with the button are aggregated and displayed in one row or one column, and the display mode of the button is changed. is,
Appearance inspection results, wherein when the button is operated, the display of the aggregated inspection results associated with the button is returned to the state before aggregation, and the button is returned to the display mode before aggregation. display program.

Images