JP6729081B2 - Image processing system, image processing method, and image processing program - Google Patents

Description

The present disclosure relates to a technique for generating a model image that serves as an inspection reference for an inspection target.

In the field of FA (Factory Automation), a technique for automatically inspecting an inspection object such as a work is prevalent. The inspection of the work is realized by comparing the inspection image obtained by photographing the inspection target work with the model image serving as the inspection reference.

Regarding a technique for generating a model image (template), Japanese Patent Laying-Open No. 2010-276487 (Patent Document 1) discloses “a template matching template creation method capable of stably ensuring high matching accuracy”. doing.

JP, 2010-276487, A

The model image is generated from an image obtained by photographing an object of the same type as the inspection object. Below, the image from which the model image is generated is also referred to as a “learning image”. When a peculiar learning image is included in the learning target, a low quality model image is generated as an inspection standard, and the accuracy of the inspection decreases. Therefore, there is a demand for a technique capable of selecting a learning image used for generating a model image in order to exclude a unique learning image from learning targets.

The present disclosure has been made to solve the above problems, and an object of an aspect is to provide an image processing system capable of selecting a learning image used for generating a model image. is there. An object in another aspect is to provide an image processing method capable of selecting a learning image used for generating a model image. An object in still another aspect is to provide an image processing program capable of selecting a learning image used for generating a model image.

According to a certain aspect, the image processing system for inspecting the inspection object based on the image obtained by imaging the inspection object, a plurality of learning images representing an object of the same type as the inspection object, A storage unit for storing a first model image generated from the plurality of learning images as an inspection reference of the inspection object, and a display for displaying a selection item for accepting selection of each of the plurality of learning images. And a section. The display unit displays, for each of the selection items, identification information of a learning image selectable by the selection item and support information for supporting whether or not the learning image can be selected. The image processing system further generates a second model image as a comparison target with the image from the learning image selected in the selection item, and updates the first model image to the second model image. Section.

Preferably, the display unit sorts each of the plurality of pieces of support information together with the corresponding selection item and the identification information according to a predetermined sorting rule for the support information and then displays the sorted pieces of support information.

Preferably, the display unit has a first similarity group between the first model image and each of the plurality of learning images, and a second similarity group between the second model image and each of the plurality of learning images. Is displayed as the support information.

Preferably, the updating unit generates the second model image from the learning image selected in the selection item each time the selection for the selection item is updated. The display unit updates the display of the second similarity group each time the second model image is generated.

Preferably, the display unit displays a learning image selected as a display target from the plurality of learning images as the support information.

Preferably, the display unit displays the learning image selected in the selection item as the support information.

Preferably, the display unit displays the first model image and the second model image side by side.

Preferably, the updating unit updates the first model image to the second model image on the basis of receiving an instruction to update the first model image.

According to another aspect, an image processing method for inspecting the inspection object based on an image obtained by imaging the inspection object includes a plurality of learning images representing an object of the same type as the inspection object. A step of preparing a first model image generated from the plurality of learning images as an inspection reference of the inspection object, and a selection item for accepting selection of each of the plurality of learning images is displayed and the selection item. For each, the step of displaying the identification information of the learning image selectable in the selection item and the support information for supporting the selection availability of the learning image side by side, the image from the learning image selected in the selection item And a step of generating a second model image as a comparison target with and updating the first model image to the second model image.

According to another aspect, the image processing program for inspecting the inspection object based on the image obtained by imaging the inspection object causes the computer to display a plurality of objects representing the same type of object as the inspection object. A step of preparing a learning image and a first model image generated from the plurality of learning images as an inspection reference of the inspection object, and displaying selection items for accepting selection of each of the plurality of learning images, For each of the selection items, a step of displaying the identification information of the learning image selectable in the selection item and the support information for supporting the selection of the learning image side by side, and the learning image selected in the selection item To generate a second model image as a comparison target with the image and update the first model image to the second model image.

In an aspect, a learning image used for generating the model image can be selected.
The above as well as other objects, features, aspects and advantages of the present disclosure will become apparent from the following detailed description of the invention which is understood in conjunction with the accompanying drawings.

It is a schematic diagram which shows the basic composition of the image processing system according to an embodiment. It is a figure which shows the learning process of a model image in time series. It is a figure which shows the process in which a user selects a learning image. It is a figure which shows the model registration screen currently displayed on the display part. It is a figure which shows an example of a model registration screen. It is a conceptual diagram which shows the inspection process of a workpiece|work schematically. It is a figure which shows an example of the dimension inspection between the pins of a workpiece|work. It is a figure which shows the dimension inspection with respect to the length of a pin. It is a figure which shows the result of the visual inspection of a workpiece|work. It is a figure showing an example of functional composition of a controller according to an embodiment. 7 is a flowchart showing a learning process executed by the image processing system according to the embodiment. 7 is a flowchart showing an inspection process executed by the image processing system according to the embodiment. It is a schematic diagram which shows the hardware constitutions of the controller according to an embodiment.

Each embodiment according to the present invention will be described below with reference to the drawings. In the following description, the same parts and components are designated by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

<A. Image processing system 1>
A basic configuration of image processing system 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a schematic diagram showing the basic configuration of an image processing system 1 according to the present embodiment.

As shown in FIG. 1, the image processing system 1 includes a controller 100, a display unit 120, an operation unit 122, an image sensor 134, and an imaging unit 135.

The controller 100 controls the entire image processing system 1. A display unit 120 and an operation unit 122 can be connected to the controller 100. The display unit 120 is composed of a liquid crystal panel or the like capable of displaying an image. The operation unit 122 is typically composed of a touch panel, a keyboard, a mouse, and the like.

The controller 100 and the image sensor 134 are communicatively connected via a network. Communication between the controller 100 and the image sensor 134 may be realized by wireless communication or wire communication. The image sensor 135 is communicatively connected to the image sensor 134. The communication between the image sensor 134 and the imaging unit 135 may be realized by wireless communication or wire communication.

In the image processing system 1, a predetermined work is performed on the works W continuously conveyed on the conveyor 123. The work W is, for example, a connector having a plurality of pins. FIG. 1 shows a work process of inspecting the work W and eliminating the abnormal work W based on the inspection result.

The workpiece W to be inspected is conveyed onto the stage S by the conveyor 123. When the work W is placed on the stage S, the work W is imaged by the imaging unit 135. At this time, preferably, the work W is imaged after being aligned on the stage S by a jig (not shown) or the like. The image sensor 134 inspects the external appearance and dimensions of the work W by performing image processing on the image obtained by the imaging. As an example, the image sensor 134 inspects the work W by comparing a model image serving as an inspection reference of the work W with an inspection image obtained by imaging the work W to be inspected. The inspection result by the image sensor 134 is sent to the controller 100 and displayed on the display unit 120 or the like.

After the inspection of the work W is completed, the controller 100 moves the stage S so as to be adjacent to the conveyor 127 with the work W placed thereon. After that, the controller 100 drives the jig 124 and pushes the work W onto the conveyor 127. As a result, the work W is conveyed on the conveyor 127.

The controller 100 drives the exclusion mechanism 125 based on the inspection result of the work W. More specifically, when the inspection result indicates that the work W is abnormal, the controller 100 drives the exclusion mechanism 125 so as to guide the work W from the conveyor 127 to the conveyor 128. On the other hand, when the inspection result indicates that the work W is normal, the controller 100 drives the exclusion mechanism 125 so that the work W is not removed by the conveyor 128. As a result, the normal work W is carried on the conveyor 127, and the abnormal work W is carried on the conveyor 128.

Note that in the example of FIG. 1, a connector is shown as the work W, but the work W is not limited to the connector. The work W is a product or a semi-finished product, and may be, for example, a housing or a food product.

Further, in the above, the example in which the image sensor 134 inspects the work W has been described, but the inspection of the work W may be performed by another device. For example, the inspection of the work W may be performed by the controller 100.

<B. Main processing of image processing system 1>
The image processing system 1 executes a learning process and an inspection process as main processes. The learning process is executed in advance before the inspection process.

In the learning process, the image processing system 1 generates a model image that serves as an inspection standard for the work. The model image is generated from a plurality of learning images obtained by previously photographing a work of the same type as the work of the inspection object.

In the inspection process, the image processing system 1 compares the model image generated in the learning process with the inspection image obtained by photographing the workpiece of the inspection target, and determines the degree of similarity between the model image and the inspection image. To calculate. The degree of similarity is represented by, for example, a correlation value between the image information (for example, pixel value) of the inspection image and the image information of the model image. The image processing system 1 determines that the workpiece to be inspected has a defect when the similarity is lower than a predetermined value. The image processing system 1 determines that the workpiece to be inspected has no defect when the similarity is equal to or more than a predetermined value.

The learning process and the inspection process will be described in detail below.
<C. Learning process>
[C1. Learning process flow]
The flow of learning processing by the image processing system 1 will be described with reference to FIG. FIG. 2 is a diagram showing the learning process of the model image in time series.

In the learning process, the image 29 that is the generation source of the model image is prepared in advance. The image 29 to be prepared may be a single image or a plurality of images. The image 29 is obtained by imaging the work W of the same type as the inspection object. The image 29 is generated by the above-described image capturing unit 135 (see FIG. 1) capturing the image of the work W. The image 29 is output from the imaging unit 135 to the image sensor 134 (see FIG. 1).

The image sensor 134 cuts out the pin portions 29A to 29E of the work W from the image 29. The pin portions 29A to 29E may be designated by the user or may be automatically detected by image processing or the like. As an example, the image sensor 134 detects an edge by superimposing a differential filter on the image 29. The image sensor 134 detects a predetermined rectangular area from the detected edge portion, and detects a rectangular area having an aspect ratio (that is, an aspect ratio) within a predetermined range as the pin portions 29A to 29E. When the model image of the pin portion has been generated in the past, the image sensor 134 detects the pin portions 29A to 29E by a predetermined image processing (for example, template matching) using the model image. Good.

In FIG. 2, pin portions 29A to 29E cut out from the image 29 are shown as a learning image 30. The image sensor 134 generates the model image 31 from the learning image 30. As an example, the image sensor 134 generates the average image of the learning image 30 as the model image 31. More specifically, the image sensor 134 averages the pixel values of the same coordinates between the learning images. As a result, the model image 31 is generated. Since each of the learning images 30 includes various environmental changes such as a difference in disturbance, the learning images 30 are averaged to generate a model image 31 having high robustness.

In the above example, the example in which the model image 31 is generated by averaging the learning images 30 has been described, but the model image 31 may be generated by another method. For example, even if the model image 31 is generated by a weighted average of averaging the learning images after weighting the learning images or an adjusting average of averaging the learning images after excluding the learning images of low reliability, Good.

[C2. Select learning image]
If the work to be learned has a defect, or if the shooting environment of the work to be learned changes, the intended learning image may not be obtained. If the learning process is executed in a state in which such a peculiar learning image is included, a low-quality model image is generated as an inspection standard. Therefore, the image processing system 1 according to the present embodiment receives a selection of a learning image used for the learning process and generates a model image from the selected learning image. Thereby, a peculiar learning image can be excluded and a highly accurate model image is generated as an inspection standard.

FIG. 3 is a diagram showing a process in which a user selects a learning image. FIG. 3 shows learning images 30A to 30N as learning candidates and selection items 31A to 31N that accept selection of each of the learning images 30A to 30N. The selection items 31A to 31N are, for example, check boxes or buttons that accept image selection. In the example of FIG. 3A, all the learning images 30A to 30N are selected.

As shown in FIG. 3A, the image processing system 1 generates a model image 32 from the learning images 30A to 30N. After that, the image processing system 1 calculates the degree of similarity with the model image 32 for each of the learning images 30A to 30N. The degree of similarity is represented by, for example, a correlation value between the image information (for example, pixel value) of the model image and the image information (for example, pixel value) of the learning image. As a method of calculating the correlation value, for example, SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), NCC (Normalized Cross-Correlation), or ZNCC (Zero-mean Normalized Cross-Correlation) is adopted. ..

The correlation value is displayed as information (hereinafter, also referred to as “support information”) for supporting whether or not the learning image from which the model image is generated can be selected. The user can identify a unique learning image by confirming the learning image having a low degree of similarity. In the example of FIG. 3A, the learning image 30B having a defect has a lower correlation value than the other learning images, and thus the user can identify the learning image 30B as a unique learning image.

After confirming the correlation value, the user selects a learning target from the learning images 30A to 30N. In the example of FIG. 3B, the user removes the check in the selection item 31B to remove the learning image 30B from the learning target. When the selection for the selection items 31A to 31N is updated, the image processing system 1 generates a new model image 33 from the learning image selected in the selection items 31A to 31N. Then, the image processing system 1 calculates the correlation value with the model image 33 for each of the learning images 30A to 30N.

As described above, the image processing system 1 receives the selection of a learning image, the process of generating a model image from the selected learning image, and the correlation value between the generated model image and each learning image as support information. The process of displaying as is repeated. This allows the user to register the model image after confirming the quality of the model image.

[C3. Model registration screen]
A model registration screen 50 for registering a model image will be described with reference to FIG. FIG. 4 is a diagram showing the model registration screen 50 displayed on the display unit 120.

The model registration screen 50 is a user interface for registering an appropriate model image. The selection operation and the input operation on the model registration screen 50 are realized by the operation unit 122 (see FIG. 1) described above.

The model registration screen 50 includes an add button 51 for reading a learning image. When the add button 51 is pressed, the pin portions 29A to 29E of the work W cut out from the image 29 are displayed in the list 60 as learning candidates. A screen for designating an image file for the button operation may be displayed, and the image file designated on the screen may be displayed in the list 60 as a learning candidate.

The display unit 120 displays a selection item 61A in the list 60 that accepts selection of each learning image. In the example of FIG. 4, a check box is displayed as the selection item 61A. A check is displayed in the selected check box. No check appears in the unselected check box. When the select all button 52 on the model registration screen 50 is pressed, the selection items 61A are collectively selected. When the cancel button 53 on the model registration screen 50 is pressed, the selections in the selection items 61A are collectively canceled. When the delete button 54 of the model registration screen 50 is pressed while any row of the list 60 is selected, the learning image corresponding to the selected row is deleted from the list 60.

The display unit 120 further displays the identification information 61B and 61C of the learning image that can be selected by each of the selection items 61A in the list 60. The identification information of the learning image includes, for example, a file ID (Identification) attached to each learning image, a file name of the learning image, a storage destination (for example, a file path) of the learning image, a registration date and time of the learning image, and the like. In the example of FIG. 4, the file ID of the learning image is shown as the identification information 61B, and the registration date and time of the learning image is shown as the identification information 61C. Each of the selection items 61A, each of the identification information 61B, and each of the identification information 61C are displayed side by side for each corresponding learning image. Thereby, the correspondence between the selection item 61A and the identification information 61B and 61C becomes clear.

The display unit 120 further displays, on the list 60, support information 61D to 61F for supporting whether or not a learning image can be selected. Each of the support information 61D, each of the support information 61E, and each of the support information 61F are displayed side by side in the selection item 61A for each corresponding learning image. Thereby, the correspondence between the selection item 61A and the support information 61D to 61F becomes clear.

The support information 61D indicates a correlation value (first similarity group) between the model image before update and each learning image. The support information 61E indicates a correlation value (second similarity group) between the model image (second model image) provisionally generated from the learning image selected by the selection item 61A and each learning image. The support information 61F represents the difference between the first similarity group and the second similarity group.

Hereinafter, the existing model image that is currently registered is also referred to as a “pre-update model image” (first model image). The model image provisionally generated from the learning image selected by the selection item 61A is also referred to as “provisional model image”. The model image finally generated from the learning image selected by the selection item 61A by the user performing the main registration operation of the model image is also referred to as “updated model image”.

Preferably, the image processing system 1 generates a temporary model image from the learning image selected in the selection item 61A each time the selection for the selection item 61A is updated. The image processing system 1 updates the correlation value (that is, the support information 61E) of each learning image with respect to the temporary model image every time the temporary model image is generated. More specifically, when the selection for the selection item 61A is updated, the image processing system 1 generates a temporary model image from the learning image selected in the selection item 61A. After that, the image processing system 1 calculates the correlation value between the temporary model image and each learning image, and updates the display column of the support information 61E with the correlation value. At the same time, the image processing system 1 calculates the amount of change in the correlation value before and after the provisional model image is generated, and updates the support information 61F with the amount of change. Since the display of the support information 61E and 61F is linked to the selection operation for the selection item 61A, the user can immediately determine the validity of each learning image.

Preferably, among the rows in the list 60, the row of the learning image whose correlation value is lower than the predetermined value is displayed in a mode different from the other rows. As an example, in a row of a learning image whose correlation value is lower than a predetermined value, the background color is changed to a color that calls attention such as red or yellow. As a result, the user can discover at a glance a unique learning image having a low correlation value with the model image.

Each information in the list 60 is rearranged according to a predetermined sort rule. As an example, when the item column 62B is pressed, each information in the list 60 is sorted in a line unit according to a predetermined sorting rule for the identification information 61B. For example, when the item column 62B is pressed, each row of the list 60 is sorted in the order of the ID given to the learning image. Each time the item column 62B is pressed, the descending sort and the ascending sort are switched.

Similarly, when the item column 62C is pressed, each information in the list 60 is sorted in a row unit according to a predetermined sorting rule for the identification information 61C. For example, when the item column 62C is pressed, each row of the list 60 is sorted in the order of registration dates and times of learning images. Each time the item column 62C is pressed, the descending sort and the ascending sort are switched.

Similarly, when the item column 62D is pressed, each information in the list 60 is sorted in a line unit according to a predetermined sorting rule for the support information 61D. That is, the display unit 120 sorts each of the support information 61D together with the corresponding selection item 61A and the identification information 61B and 61C according to a predetermined sorting rule for the support information 61D, and then displays the sorted information. As an example, when the item column 62D is pressed, each row of the list 60 is sorted in the order of the correlation values shown in the support information 61D. Each time the item column 62D is pressed, the descending sort and the ascending sort are switched. By displaying the support information 61D in the order of the correlation value, the user can easily find a unique learning image having a low correlation value.

Similarly, when the item column 62E is pressed, each information in the list 60 is sorted in a row unit according to a predetermined sorting rule for the support information 61E. For example, when the item column 62E is pressed, each row of the list 60 is sorted in the order of the correlation values shown in the support information 61E. Each time the item column 62E is pressed, the descending sort and the ascending sort are switched. By displaying the support information 61E in the order of the correlation value, the user can easily find a unique learning image with a low correlation value.

Similarly, when the item column 62F is pressed, each information in the list 60 is sorted in a line unit according to a predetermined sorting rule for the support information 61F. For example, when the item column 62F is pressed, each row of the list 60 is sorted in the order of the amount of change in the correlation value shown in the support information 61E. Each time the item column 62F is pressed, the descending sort and the ascending sort are switched. By displaying the support information 61F in the order of the amount of change in the correlation value, the user can easily understand the quality of the model image before and after the update.

The display unit 120 further displays the learning image 70 as support information for supporting whether or not the learning image can be selected. More specifically, the model registration screen 50 receives an operation of selecting one learning image in the list 60. The learning image to be displayed is selected by selecting any row in the list 60. In the example of FIG. 4, the row 68 of the list 60 is selected, and the learning image 70 corresponding to the row 68 is displayed as support information. By displaying the selected learning image, the user can visually check the quality of the learning image. Preferably, the row selected in list 60 is displayed differently than the other rows.

The display unit 120 further displays the pre-update model image 71 and the temporary model image 72 side by side. This allows the user to compare the model image 71 before the update with the temporary model image 72. Preferably, each time the selection for the selection item 61A is updated, a temporary model image 72 is generated from the learning image selected by the selection item 61A, and the model image 72 is generated each time the temporary model image 72 is generated. The display is updated. In this way, the display of the model image 72 is linked to the selection operation for the selection item 61A, so that the user can easily recognize the quality of the model image 72.

The display unit 120 further displays the list 80 as support information for determining the quality of the model image. The list 80 includes an evaluation result 81A for the model image before updating, an evaluation result 81B for the temporary model image, and an evaluation result 81C for the model image before updating and the temporary model image. The evaluation result 81A includes, for example, the maximum value, the minimum value, and the average value of the correlation values shown in the support information 61D. The evaluation result 81B includes, for example, the maximum value, the minimum value, and the average value of the correlation values shown in the support information 61E. The evaluation result 81C includes, for example, the maximum value, the minimum value, and the average value of the correlation values shown in the support information 61F. The items displayed as the evaluation results 81A to 81C are not limited to the maximum value, the minimum value, and the average value of the correlation values. For example, as the evaluation results 81A to 81C, the standard deviation of the correlation value and other indexes may be displayed.

When the registration button 91 on the model registration screen 50 is pressed, the image processing system 1 executes the main registration of the model image. That is, the image processing system 1 main-registers the temporarily generated model image as a new model image. When the OK button 92 on the model registration screen 50 is pressed, the image processing system 1 reflects the update of the model image and then closes the model registration screen 50. When the cancel button 93 of the model registration screen 50 is pressed, the image processing system 1 closes the model registration screen 50 without reflecting the update of the model image.

[C4. Modification of model registration screen]
A model registration screen 50A, which is a modification of the model registration screen 50 described above, will be described with reference to FIG. FIG. 5 is a diagram showing an example of the model registration screen 50A.

The model registration screen 50 shown in FIG. 4 described above displays one learning image 70 selected in the list 60 as support information. On the other hand, the model registration screen 50A shown in FIG. 5 displays all or a part of the learning image selected in the selection item 61A as support information. These learning images are displayed in the list 95. By displaying the selected learning images as a list, the user can immediately understand the contents of the learning images.

In the example of FIG. 5, the learning image 70A of “No1” selected by the selection item 61A is displayed in the list 95. The learning image 70B of "No3" selected by the selection item 61A is displayed in the list 95. The learning image 70C of "No4" selected by the selection item 61A is displayed in the list 95. The “No5” learning image 70D selected by the selection item 61A is displayed in the list 95. The learning image 70E of "No6" selected by the selection item 61A is displayed in the list 95. The learning image 70F of "No7" selected by the selection item 61A is displayed in the list 95. The “No9” learning image 70G selected in the selection item 61A is displayed in the list 95. The “No10” learning image 70H selected by the selection item 61A is displayed in the list 95.

Preferably, every time the selection of the learning image for the selection item 61A is updated, the learning image displayed in the list 95 is updated.

The list 95 includes a combo box 96 for designating the display size of the learning image. When the combo box 96 is selected, a plurality of display sizes are expanded in the list. When a specific display size is selected from the expanded display sizes, the display unit 120 changes the display size of the learning images 70A to 70H according to the selected display size. By displaying the selected learning image in an arbitrary size, the user can efficiently confirm the learning image.

In the above description, the example in which the learning image selected by the selection item 61A is displayed in the list 95 has been described, but all the learning images read in the list 60 may be displayed in the list 95.

<D. Inspection processing>
[D1. Inspection process flow]
The inspection processing by the image processing system 1 will be described with reference to FIG. FIG. 6 is a conceptual diagram schematically showing a workpiece inspection process.

In the inspection process, the imaging unit 135 (see FIG. 1) described above captures the workpiece W to be inspected flowing on the conveyor 123 (see FIG. 1). As a result, it is assumed that the inspection image 40 is obtained. The inspection image 40 is sequentially output to the image sensor 134 (see FIG. 1) every time the work W is imaged.

The image sensor 134 searches the inspection image 40 for the pin portions 40A to 40E of the work W using the model image 34 generated by the above-described learning process. More specifically, the image sensor 134 repeats scanning of the model image 34 in the inspection image 40 and enlargement or reduction of the model image 34, and repeats scanning of the model image 34 with the model image 34 for each region in the inspection image 40. Calculate the correlation value. The image sensor 134 detects a region having a correlation value higher than a predetermined value as the pin portions 40A to 40E. The image sensor 134 inspects the work W based on the detection result of the pin portions 40A to 40E.

[D2. Dimension inspection between pins]
Various inspections of the work W are performed based on the detection results of the pin portions 40A to 40E. As an example, the dimensions between the pins of the connector are inspected.

FIG. 7 is a diagram illustrating an example of a dimension inspection between pins of the work W. As shown in FIG. 7, the image sensor 134 calculates the distances X1 to X4 between the pin portions 40A to 40E from the position information of the pin portions 40A to 40E in the inspection image 40. Distance X1 represents a distance between a predetermined position (for example, the center) of pin portion 40A and a predetermined position (for example, the center) of pin portion 40B. Distance X2 represents a distance between a predetermined position (for example, the center) of pin portion 40B and a predetermined position (for example, the center) of pin portion 40C. Distance X3 represents a distance between a predetermined position (for example, the center) of pin portion 40C and a predetermined position (for example, the center) of pin portion 40D. Distance X4 represents a distance between a predetermined position (for example, the center) of pin portion 40D and a predetermined position (for example, the center) of pin portion 40E.

In one aspect, the image sensor 134 determines that there is no defect in the dimension of the workpiece W when each of the distances X1 to X4 is within a predetermined allowable range. On the other hand, the image sensor 134 determines that the dimension of the work W is defective when any of the distances X1 to X4 is not within the predetermined allowable range.

In another aspect, the image sensor 134 determines that there is no defect in the dimension of the work W when the variation degree (for example, dispersion) of the distances X1 to X4 is within a predetermined allowable range. On the other hand, the image sensor 134 determines that the dimension of the work W is defective when the variation degree of the distances X1 to X4 is not within the predetermined allowable range.

[D3. Dimensional inspection for pin length]
In addition to the above-described dimensional inspection between pins, various inspections are performed. As an example, the length of each pin of the connector is inspected.

FIG. 8 is a diagram showing a dimensional inspection for the length of the pin. As shown in FIG. 8, the image sensor 134 calculates the lengths Y1 to Y5 of the pins of the work W from the detection results of the pin portions 40A to 40E in the inspection image 40. The length Y1 corresponds to the length of the pin portion 40A in the longitudinal direction. The length Y2 corresponds to the length of the pin portion 40B in the longitudinal direction. The length Y3 corresponds to the length of the pin portion 40C in the longitudinal direction. The length Y4 corresponds to the length of the pin portion 40D in the longitudinal direction. The length Y5 corresponds to the length of the pin portion 40E in the longitudinal direction. The lengths Y1 to Y5 may be represented by the distance from the reference position to the pin tip with any of the pin tips of the work W as the reference position.

In one aspect, the image sensor 134 determines that there is no defect in the dimension of the work W when the lengths Y1 to Y5 are all within the predetermined allowable range. On the other hand, the image sensor 134 determines that the dimension of the work W is defective when any of the lengths Y1 to Y5 is not within the predetermined allowable range.

In another aspect, the image sensor 134 determines that there is no defect in the dimension of the work W when the variation degree (for example, dispersion) of the lengths Y1 to Y5 falls within a predetermined allowable range. On the other hand, the image sensor 134 determines that the dimension of the workpiece W is defective when the variation degree of the lengths Y1 to Y5 is not within the predetermined allowable range.

[D4. Pin appearance inspection]
In addition to the above-described dimensional inspection, various inspections are performed. As an example, the appearance of the work W is inspected.

FIG. 9 is a diagram showing the results of the visual inspection of the work W. As shown in FIG. 9, the result of the visual inspection is displayed on the display unit 120, for example. The image sensor 134 determines whether or not the work W has a defect based on the correlation value of each of the pin portions 40A to 40E with respect to the model image. As an example, the image sensor 134 determines that there is no defect in the appearance of the work W when all the correlation values with respect to the pin portions 40A to 40E are equal to or more than a predetermined value. On the other hand, the image sensor 134 determines that the appearance of the work W is defective when any of the correlation values for the pin portions 40A to 40E is smaller than a predetermined value. In the example of FIG. 9, it is determined that the pin portion 40C has a defect.

The index used for the visual inspection is not limited to the correlation value of the pin portions 40A to 40E with respect to the model image. For example, as an index used for appearance inspection, the hue of a specific area in an inspection image, the area of a specific color, the position or size (width) of a specific part, the result of comparison of pixel values with the vicinity of a specific position, etc. Good.

Preferably, the defective pin portion 40C is displayed differently than the other pin portions 40A, 40B, 40D, 40E.

<E. Functional configuration>
The functions of the controller 100 that constitutes the image processing system 1 will be described with reference to FIG. 10. FIG. 10 is a diagram showing an example of the functional configuration of the controller 100.

As shown in FIG. 10, the controller 100 includes a selection receiving unit 152, a generation unit 154, a calculation unit 156, a display control unit 158, and an updating unit 160.

The selection receiving unit 152 receives a selection operation of a learning image used for generating a model image from the operation unit 122 (see FIG. 1) described above. The selection operation is input, for example, in the selection item 61A (see FIG. 4) of the model registration screen 50 (see FIG. 4) described above. The selection receiving unit 152 outputs the selected learning image to the generation unit 154.

The generation unit 154 specifies the learning image selected from the learning image 30 and generates the temporary model image 33 from the learning image. Preferably, the generation unit 154 generates the tentative model image 33 each time the selection reception unit 152 receives a learning image selection operation. The method of generating the model image is as described above.

The calculating unit 156 calculates a first correlation value group (first similarity group) between the currently registered model image 32 before update and each of the learning images 30, and also calculates the temporary model image 33 and the learning image 30. A second correlation value group (second similarity group) with each of the above is calculated. Preferably, the second correlation value group is calculated every time the temporary model image 33 is generated. The first correlation value group and the second correlation value group are output to the display control unit 158.

The display control unit 158 controls the display of the model registration screen 50 described above. As an example, the display control unit 158 displays the calculation result (that is, the first correlation value group and the second correlation value group) output by the calculation unit 156 on the model registration screen 50. Further, the display control unit 158 displays the pre-update model image 32 and the temporary model image 33 side by side on the model registration screen 50.

The updating unit 160 receives an operation for main registration of the temporary model image 33. The update unit 160 updates the pre-update model image 32 with the model image 33 based on receiving the registration operation for updating the pre-update model image 32. As a result, the model image 33 is main-registered.

Note that all or a part of the selection receiving unit 152, the generation unit 154, the calculation unit 156, the display control unit 158, and the update unit 160 do not necessarily have to be mounted on the controller 100. For example, these functions may be implemented in the above-described image sensor 134 (see FIG. 1) or the like.

<F. Flowchart>
The control structure of the image processing system 1 will be described with reference to FIGS. 11 and 12. FIG. 11 is a flowchart showing the learning process executed by the image processing system 1. FIG. 12 is a flowchart showing the inspection process executed by the image processing system 1.

Below, the control flow of the learning process and the control flow of the inspection process will be described in order.
[F1. Learning process control flow]
First, the control flow of the learning process by the image processing system 1 will be described with reference to FIG. 11.

The process shown in FIG. 11 is realized, for example, by a control device 101 (see FIG. 13), which will be described later, included in the controller 100 (see FIG. 1) executing a program. In other aspects, some or all of the processing may be performed by image sensor 134 or other hardware.

In step S10, the control device 101 determines whether or not the operation for opening the model registration screen 50 (see FIG. 4) described above has been accepted. When determining that the operation for opening the model registration screen 50 has been received (YES in step S10), the control device 101 switches control to step S12. Otherwise (NO in step S10), the control device 101 executes the process of step S10 again.

In step S12, the control device 101 displays the model registration screen 50 on the display unit 120 (see FIG. 1) described above.

In step S20, the control device 101 determines whether or not the selection accepting unit 152 (see FIG. 10) has accepted a learning image selection operation. As described above, the learning image selecting operation is performed on the selection item 61A (see FIG. 4) on the model registration screen 50. When determining that the learning image selection operation has been accepted (YES in step S12), control device 101 switches control to step S22. Otherwise (NO in step S12), the control device 101 switches control to step S30.

In step S22, the control device 101, as the generation unit 154 (see FIG. 10) described above, generates the temporary model image 33 from the learning image selected in the selection item 61A of the model registration screen 50.

In step S24, the control device 101, as the calculation unit 156 (see FIG. 10) described above, calculates the correlation value between each of the currently registered model image before update and the learning image. Further, the control device 101, as the calculation unit 156, calculates the correlation value between each of the temporary model image generated in step S22 and the learning image.

In step S26, the control device 101 updates the display of the model registration screen 50 as the display control unit 158 (see FIG. 10) described above. As an example, the control device 101 displays the calculation result in step S24 on the model registration screen 50. Further, the control device 101 displays the temporary model image generated in step S22 side by side with the currently registered model image before update.

In step S30, the control device 101 determines whether or not the main registration operation of the model image has been accepted. As an example, the control device 101 accepts the main registration operation of the model image when the registration button 91 (see FIG. 4) of the model registration screen 50 is pressed. When determining that the main registration operation of the model image has been received (YES in step S30), the control device 101 switches the control to step S32. Otherwise (NO in step S30), the control device 101 returns the control to step S20.

In step S32, the control device 101, as the updating unit 160 (see FIG. 10), updates the currently registered model image before update to the model image generated in step S22. At this time, preferably, the control device 101 writes the identification information of the learning image used to generate the model image in step S22 in the history information. The identification information of the learning image of the generation source is written in the history information every time the model image is updated, and thus the control device 101 can return the current model image to the past model image.

[F2. Inspection processing control flow]
The control flow of the inspection process by the image processing system 1 will be described with reference to FIG. 12. The process shown in FIG. 12 is realized by the image sensor 134 (see FIG. 1) executing a program, for example. In another aspect, some or all of the processing may be executed by the controller 100 or other hardware. The inspection flow will be described below by taking a connector having a plurality of pins as an example of the work.

In step S110, the image sensor 134 determines whether a work inspection start operation is received. When the image sensor 134 determines that the work inspection start operation is received (YES in step S110), the control is switched to step S112. Otherwise (NO in step S110), the image sensor 134 causes the control device 101 to execute the process of step S110 again.

In step S112, the image sensor 134 images the connector to be inspected and acquires an image for inspection. The inspection image is generated by the image capturing unit 135 (see FIG. 1) described above.

In step S114, the image sensor 134 searches the inspection image for the pin portion of the connector using the model image generated by the learning process described above. More specifically, the image sensor 134 calculates the correlation value between each region in the inspection image and the model image while repeating scanning of the model image in the inspection image and enlargement or reduction of the model image. .. The image sensor 134 detects an area having a correlation value higher than a predetermined value as a pin portion to be inspected. As a result, the position of the pin portion in the inspection image and the correlation value between the pin portion and the model image at each position are obtained as the detection result.

In step S116, the image sensor 134 executes the dimension inspection between the pins of the connector based on the detection result in step S114. The dimension inspection is as described in FIG. 7.

In step S118, the image sensor 134 performs a dimensional inspection for the length of the pin of the connector based on the detection result of step S114. The dimension inspection is as described in FIG.

In step S120, the image sensor 134 performs a visual inspection of the connector based on the detection result of step S114. The dimension inspection is as described in FIG.

In step S130, the image sensor 134 determines whether to finish the work inspection. As an example, the image sensor 134 determines to end the work inspection when the operation of ending the work inspection is received. When the image sensor 134 determines to end the work inspection (YES in step S130), the work inspection process ends. If not (NO in step S130), the image sensor 134 returns the control to step S112.

<G. Hardware configuration of controller 100>
With reference to FIG. 13, a hardware configuration of controller 100 according to the present embodiment will be described. FIG. 13 is a schematic diagram showing a hardware configuration of controller 100 according to the present embodiment.

The controller 100 is, for example, a computer configured according to a general-purpose computer architecture. The controller 100 includes a control device 101, a main memory 102, a communication interface 103, a sensor interface 104, an operation interface 105, a display interface 106, an optical drive 107, and a storage device 110 (storage unit). These components are communicatively connected to each other via an internal bus 119.

The control device 101 is composed of, for example, at least one integrated circuit. The integrated circuit is configured by, for example, at least one CPU (Central Processing Unit), at least one ASIC (Application Specific Integrated Circuit), at least one FPGA (Field Programmable Gate Array), or a combination thereof. The control device 101 expands the image processing program 111 stored in the storage device 110 into the main memory 102 and executes the image processing program 111 to implement the above-described learning processing and inspection processing. The main memory 102 is composed of a volatile memory, and functions as a work memory required for the program execution by the control device 101.

The communication interface 103 exchanges data with an external device via a network. The external device includes, for example, the image sensor 134 (see FIG. 1) described above, a server, and other communication devices. Controller 100 may be configured to be able to download image processing program 111 according to the present embodiment via communication interface 103.

The sensor interface 104 is connected to the image sensor 134 described above. The above-described image pickup unit 135 is connected to the image sensor 134, and the sensor interface 104 takes in an image signal obtained by the image pickup of the image pickup unit 135 and issues a command such as an image pickup timing to the image pickup unit 135 via the image sensor 134. send.

The operation interface 105 is connected to the operation unit 122 and captures a signal indicating a user operation from the operation unit 122. The operation unit 122 typically includes a keyboard, a mouse, a touch panel, a touch pad, and the like, and receives an operation from a user.

The display interface 106 is connected to the display unit 120, and sends an image signal for displaying an image to the display unit 120 according to a command from the control device 101 or the like. The display unit 120 includes a display, an indicator and the like, and presents various kinds of information to the user.

The optical drive 107 reads various programs stored therein from the optical disc 107A or the like and installs them in the storage device 110. The storage device 110 stores, for example, the image processing program 111 and the like.

FIG. 13 shows a configuration example in which a necessary program is installed in the controller 100 via the optical drive 107, but the present invention is not limited to this, and may be downloaded from a server device or the like on the network. Alternatively, the program on the controller 100 may be rewritten by a program written in a storage medium such as a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, or a CF (Compact Flash).

The storage device 110 (storage unit) is, for example, a hard disk or an external storage medium. As an example, the storage device 110 stores the learning image 30 used to generate the model image, the pre-update model image 32 generated from the learning image 30, and the temporary model generated from a part or all of the learning image 30. An image 33 and an image processing program 111 for implementing various processes according to the present embodiment are stored.

The image processing program 111 may be provided as a part of an arbitrary program instead of as a standalone program. In this case, the process according to the present embodiment is realized in cooperation with any program. Even a program that does not include some of such modules does not depart from the spirit of the controller 100 according to the present embodiment. Furthermore, some or all of the functions provided by image processing program 111 according to the present embodiment may be realized by dedicated hardware. Furthermore, controller 100 and the server may cooperate to realize the process according to the present embodiment. Furthermore, the controller 100 may be configured in the form of a so-called cloud service in which at least one server realizes the processing according to the present embodiment.

<H. Advantage>
As described above, the image processing system 1 according to the present embodiment displays the selection items for accepting the selection of each learning image, and, for each of the selection items, the identification information of the learning images selectable by the selection item. , And support information for supporting whether or not the learning image is selectable are displayed side by side. The image processing system 1 generates a new model image from the learning image selected in the selection item.

The user can omit the unique learning image from the learning target by confirming the support information for supporting whether or not the learning image can be selected. As a result, a highly accurate model image is generated, so that the accuracy of inspection is improved. In addition, by generating a highly accurate model image, it is possible to suppress a positional shift that may occur during the matching process between the inspection image and the model image.

As another advantage, the image processing system 1 according to the present embodiment can generate a highly robust model image. More specifically, in order to generate a model image with high robustness, learning of various works with different shooting environments (for example, illumination unevenness) and manufacturing environments (for example, processing method, mold, temperature, etc.) You need to create a model image from the image. At this time, in image processing system 1 according to the present embodiment, the user can select a learning image effective for generating a model image from a wide variety of learning images. This makes it possible to generate a more robust model image.

As still another advantage, the image processing system 1 according to the present embodiment displays the history of learning images used for generating the model image (for example, the shooting date and time and the registration date and time), so that the user can store the history. The learning image can be selected by referring to it.

As another advantage, the user can confirm the validity of the generated model image. More specifically, the image processing system 1 according to the present embodiment displays the judgment information indicating the validity of the model image every time the model image is generated by selecting the learning image. As the determination factor, for example, the correlation value of the learning image with respect to the generated model image, or the generated model image itself is displayed. The user can confirm the validity of the model image by confirming these judgment factors.

The embodiments disclosed this time are to be considered as illustrative in all points and not restrictive. The scope of the present invention is shown not by the above description but by the claims, and is intended to include meanings equivalent to the claims and all modifications within the scope.

1 image processing system, 29 images, 29A to 29E, 40A to 40E pin portion, 30, 30A to 30N, 70, 70A to 70H learning image, 31 to 34, 71, 72 model image, 31A to 31N, 61A selection item, 40 inspection image, 50, 50A model registration screen, 51 add button, 52 all selection button, 53 release button, 54 delete button, 60, 80 list, 61B, 61C identification information, 61D-61F support information, 62B-62F items Column, 68 lines, 81A to 81C evaluation result, 91 registration button, 92 OK button, 93 cancel button, 95 list, 96 combo box, 100 controller, 101 control device, 102 main memory, 103 communication interface, 104 sensor interface, 105 Operation interface, 106 display interface, 107 optical drive, 107A optical disk, 110 storage device, 111 image processing program, 119 internal bus, 120 display unit, 122 operation unit, 123, 127, 128 conveyor, 124 jig, 125 exclusion mechanism , 134 image sensor, 135 image pickup unit, 152 selection acceptance unit, 154 generation unit, 156 calculation unit, 158 display control unit, 160 update unit.

Claims (10)

An image processing system for inspecting the inspection object based on an image obtained by imaging the inspection object,
A storage unit for storing a plurality of learning images representing an object of the same type as the inspection object, and a first model image generated from the plurality of learning images as an inspection reference of the inspection object;
A display unit for displaying a selection item for accepting selection of each of the plurality of learning images, the display unit, for each of the selection items, identification information of learning images selectable by the selection item, Display side by side with support information for supporting whether or not learning images can be selected,
An update unit is provided for generating a second model image as a comparison target with the image from the learning image selected in the selection item, and updating the first model image with the second model image ,
The support information includes a first similarity group between the first model image and each of the plurality of learning images, and a second similarity group between the second model image and each of the plurality of learning images, Image processing system. The image processing system according to claim 1, wherein the display unit sorts each of the plurality of pieces of support information together with the corresponding selection item and the identification information according to a predetermined sorting rule for the support information, and then displays the sorted information. .. The support information further includes a change amount between each first similarity included in the first similarity group and each second similarity included in the second similarity group. Or the image processing system according to 2. The update unit generates the second model image from the learning image selected in the selection item each time the selection for the selection item is updated,
The image processing system according to claim 1, wherein the display unit updates the display of the second similarity degree group each time the second model image is generated. The image processing system according to claim 1, wherein the display unit displays, as the support information, a learning image selected as a display target from the plurality of learning images. The image processing system according to claim 1, wherein the display unit displays the learning image selected in the selection item as the support information. The image processing system according to claim 1, wherein the display unit displays the first model image and the second model image side by side. The image according to any one of claims 1 to 7, wherein the updating unit updates the first model image to the second model image based on receiving an update instruction of the first model image. Processing system. An image processing method for inspecting the inspection object based on the image obtained by imaging the inspection object,
Preparing a plurality of learning images representing an object of the same type as the inspection object, and a first model image generated from the plurality of learning images as an inspection reference of the inspection object,
A selection item for accepting selection of each of the plurality of learning images is displayed, and for each selection item, identification information of a learning image selectable by the selection item and support for supporting whether or not the learning image is selectable The step of displaying the information side by side,
Generating a second model image as a comparison target with the image from the learning image selected in the selection item, and updating the first model image to the second model image,
The support information includes a first similarity group between the first model image and each of the plurality of learning images, and a second similarity group between the second model image and each of the plurality of learning images, Image processing method. An image processing program for inspecting the inspection object based on an image obtained by imaging the inspection object,
The image processing program causes a computer to
Preparing a plurality of learning images representing an object of the same type as the inspection object, and a first model image generated from the plurality of learning images as an inspection reference of the inspection object,
A selection item for accepting selection of each of the plurality of learning images is displayed, and for each selection item, identification information of a learning image selectable by the selection item and support for supporting whether or not the learning image is selectable The step of displaying the information side by side,
Generating a second model image as a comparison target with the image from the learning image selected in the selection item, and updating the first model image to the second model image,
The support information includes a first similarity group between the first model image and each of the plurality of learning images, and a second similarity group between the second model image and each of the plurality of learning images, Image processing program.