JP2020107098A - Image determination device, image determination method, and image determination program - Google Patents

Abstract

To provide an image determination device and the like that include a learning model for referring to the knowledge accumulated in a memory to determine an inspection result.SOLUTION: An image determination device comprises: a memory that stores associated data associated with a test object in association with feature data representing one or more features of an image of the test object or derived data calculated based on the feature data; a feature extractor that outputs the feature data representing one or more features of the image of the test object based on the image; and a determiner that outputs output data including a result of determination on the image based on the feature data output from the feature extractor and reference data output from the memory based on the feature data.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image determination device, an image determination method, and an image determination program.

Conventionally, in the FA (Factory Automation) field, automatic control using image processing has been widely put into practical use. For example, a process is realized in which an inspection target such as a work is imaged and the quality of the work is inspected based on the feature amount extracted from the imaged image.

For example, Patent Document 1 discloses a device that determines the presence or absence of a defect in an inspection target based on the output result of a discriminator learned by using images of the inspection target captured under a plurality of imaging conditions as learning data. There is. Specifically, the apparatus described in Patent Document 1 uses an image of an object whose appearance is known from each of at least two images based on images taken under at least two different imaging conditions. A learning extraction unit that extracts a feature amount, and a selection unit that selects a feature amount for determining the quality of an object from the feature amounts that extend over the feature amounts extracted from the at least two images by the learning extraction unit. And a generation unit that generates a discriminator that determines pass/fail of an object based on the feature amount selected by the selection unit.

JP, 2017-49974, A

Jason Weston, Sumit Chopra, Antoine Bordes, "Memory Networks," November 29, 2015, Internet <URL: https://arxiv.org/abs/1410.3916>

When determining the presence/absence of a defect, the type of work, or the like in a learning model for an image to be inspected, the conventional apparatus such as Patent Document 1 does not utilize the knowledge accumulated in the memory. On the other hand, Non-Patent Document 1 proposes a learning model that utilizes knowledge accumulated in a memory for natural language processing. The inventors of the present invention may obtain an image determination device that performs more flexible determination processing by utilizing the knowledge accumulated in the memory in the appearance inspection or the like by the image determination device equipped with the learning model. I found that there is.

Therefore, the present invention provides an image determination device, an image determination method, and a determination program that include a learning model that determines an inspection result by referring to knowledge stored in a memory for an image to be inspected.

An image determination apparatus according to an aspect of the present disclosure associates related data related to an inspection target with feature data representing one or a plurality of features of an image of an inspection target or derived data calculated based on the feature data. Based on the image to be inspected, a feature extractor that outputs feature data representing one or more features of the image, feature data output from the feature extractor, and the feature data based on the feature data. And a determiner that outputs output data including a determination result regarding the image based on the reference data output from the memory.

According to this aspect, the determination device is provided that outputs the output data including the determination result regarding the image based on the feature data output from the feature extractor and the reference data output from the memory based on the feature data. A learning model is provided. This allows the learning model to utilize the knowledge accumulated in the memory to perform the determination process of the image to be inspected.

In the above aspect, the related data may include information that assists the determination. The information that assists the determination may include any one of the determination result regarding the image to be inspected, the score indicating the reliability of the determination result, and the time stamp indicating the time when the related data is added to the memory.

According to this aspect, it is possible to realize an image determination device that can obtain a more flexible determination result by utilizing the information stored in the memory that assists the determination.

In the above aspect, the related data may include information that assists visualization. The information that aids visualization may include an image of the inspection target.

According to this aspect, it is possible to obtain more useful output data in addition to the determination result by utilizing the information that is stored in the memory and that assists visualization.

A determination method according to another aspect of the present disclosure is related data related to an inspection target in association with feature data representing one or a plurality of features of an image of the inspection target or derived data calculated based on the feature data. In a learning model including a memory that stores, output feature data representing one or more features of an image based on an image to be inspected, output feature data, and a memory based on the feature data from the memory. Outputting output data including a determination result regarding the image based on the output reference data.

According to this aspect, in the learning model, the output data including the determination result regarding the image is output based on the feature data output from the feature extractor and the reference data output from the memory based on the feature data. A determination method is provided. This allows the learning model to utilize the knowledge accumulated in the memory to perform the determination process of the image to be inspected.

A determination program according to another aspect of the present disclosure is related data related to an inspection target in association with feature data representing one or more features of an image of the inspection target or derived data calculated based on the feature data. A computer equipped with a memory for storing a feature extractor for outputting feature data representing one or more features of an image based on an image to be inspected, feature data output from the feature extractor, and the feature Based on the reference data output from the memory based on the data, it functions as a determiner that outputs the output data including the determination result regarding the image.

According to this aspect, the determination device is provided that outputs the output data including the determination result regarding the image based on the feature data output from the feature extractor and the reference data output from the memory based on the feature data. A learning model is provided. This allows the learning model to utilize the knowledge accumulated in the memory to perform the determination process of the image to be inspected.

According to the present invention, it is possible to provide an image determination device, an image determination method, and a determination program that include a learning model that determines an inspection result by referring to knowledge stored in a memory for an image to be inspected. ..

It is a figure which shows the structural example of the image determination system which concerns on embodiment of this invention. It is a figure which shows the hardware constitutions of the image determination apparatus which concerns on this embodiment. It is a figure which shows the functional block of the image determination apparatus which concerns on this embodiment. It is a figure which shows the functional block of the learning model which concerns on this embodiment. 6 is a flowchart of an image determination process executed by the image determination device according to the present embodiment.

Hereinafter, an embodiment according to one aspect of the present invention (hereinafter also referred to as “the present embodiment”) will be described with reference to the drawings. However, the embodiments described below are merely examples of the present invention in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. That is, in implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted. It should be noted that although the data that appears in this embodiment is described in natural language, more specifically, it is specified by a computer-recognizable pseudo language, command, parameter, machine language, or the like.

§1 Application Example FIG. 1 is a diagram showing a configuration example of an image determination system 1 according to an embodiment of the present invention. First, an example of a scene to which the present invention is applied will be described. The image determination system 1 according to the present embodiment analyzes an input image generated by capturing an image of an inspection target with a learning model included in the image determination device 100, and performs defect inspection and classification of the target. The learning model according to the present embodiment refers to the knowledge accumulated in the memory and determines the inspection result. For example, the memory stores reference data related to the inspection target in association with the characteristic data representing one or a plurality of characteristics of the inspection target image. In addition to the characteristic data or in place of the characteristic data, a hash value or the like calculated from the characteristic data may be stored in the memory. The learning model may be learned in advance using the learning data so as to classify the presence or absence of a defect regarding the inspection target and the type of the inspection target. In addition, the learning model may be additionally learned when new learning data is obtained. The inspection target includes, for example, a work piece such as a part or a product in the manufacturing process.

As shown in FIG. 1, the image determination system 1 performs image measurement processing on an input image obtained by capturing an image of a work 4 that is an inspection target that is conveyed on a belt conveyor 2, thereby 4. Appearance inspection or appearance measurement of 4 is realized. In the following description, as a typical example of the image measurement process, an example applied to inspection of the presence or absence of a defect on the surface of the work 4 will be described. It can be applied to measurement, specification of the type of work 4, and the like.

A camera 102, which is an imaging unit, is arranged above the belt conveyor 2, and the imaging field of view 6 of the camera 102 is configured to include a predetermined area of the belt conveyor 2. The image data (hereinafter, also referred to as “input image”) generated by the image pickup by the camera 102 is transmitted to the image determination device 100. Imaging by the camera 102 is performed periodically or when an event occurs.

The image determination device 100 is connected to a PLC (programmable controller) 10, a database device 12, and the like via a higher-level network 8. The measurement result of the image determination device 100 may be transmitted to the PLC 10 and/or the database device 12. In addition to the PLC 10 and the database device 12, any device may be connected to the upper network 8.

The image determination apparatus 100 may be connected with a display 104 for displaying a processing status, a measurement result, and the like, and a keyboard 106 and a mouse 108 as an input unit that receives a user operation.

The image determination device 100 includes a learning model that refers to the knowledge accumulated in the memory. The learning model accepts an image as an input and outputs a feature extractor that outputs feature data representing one or more features of the image, feature data output from the feature extractor, and reference data output from the memory. And a determiner that outputs output data including a determination result regarding the image based on the determination result. Here, the feature extractor may include, for example, a feature extractor that extracts an edge included in the image, a plane area included in the image, a background pattern of the image, a binarized area included in the image, and the like. The feature extractor may be configured by one or a plurality of feature extractors. At this time, each feature extractor may be a model that extracts a single feature that is easy for a person to understand, a machine learning model, or a rule-based model. The determiner receives one or a plurality of feature vectors and the reference data output from the memory as inputs, and outputs output data including the determination result regarding the image. The output data may be, for example, data representing the presence or absence of a defect to be inspected, the type of defect, the size of the external shape of the defect, the type of inspection target, and the like. The determiner may be composed of, for example, a neural network, and may be learned in advance so as to output desired output data using the learning data.

§2 Configuration example [hardware configuration]
FIG. 2 is a diagram showing a hardware configuration of the image determination device 100 according to the present embodiment. The image determination apparatus 100 may be implemented by using a general-purpose computer configured according to a general-purpose computer architecture, for example. The image determination apparatus 100 includes a processor 110, a main memory (main storage unit) 112, a camera interface 114, an input interface 116, a display interface 118, a communication interface 120, and a storage (auxiliary storage unit) 130. .. These components are typically communicatively connected to each other via an internal bus 122.

The processor 110 expands the program stored in the storage 130 into the main memory 112 and executes the program to realize the functions and processes as described in detail with reference to the figures below. The main memory 112 is composed of a volatile memory and functions as a work memory necessary for the program execution by the processor 110.

The camera interface is an example of an acquisition unit. The camera interface 114 is connected to the camera 102 and acquires an input image captured by the camera 102. The camera interface 114 may instruct the camera 102 about image capturing timing and the like.

The input interface 116 is connected to an input unit such as a keyboard 106 and a mouse 108, which is operated by a user. The input interface 116 acquires a command indicating an operation performed by the user on the input unit.

The display interface 118 is connected to the display 104 as a display unit. The display interface 118 outputs various processing results generated by the execution of the program by the processor 110 to the display 104.

The communication interface 120 is in charge of processing for communicating with the PLC 10, the database device 12, and the like via the upper network 8.

The storage 130 stores a program for causing a computer to function as the image determination apparatus 100, such as an image processing program 132 for realizing the learning model 136 and an OS (operating system) 134. The storage 130 may further store a learning model 136, an input image 138 acquired from the camera 102, and learning data 140 used for learning of the learning model 136. The learning data 140 may be acquired from an external device such as the database device 12 via the host network 8 and may be temporarily stored in the storage 130.

The image processing program 132 stored in the storage 130 may be installed in the image determination apparatus 100 via an optical recording medium such as a DVD (digital versatile disc) or a semiconductor recording medium such as a USB (universal serial bus) memory. Good. Alternatively, the image processing program 132 may be downloaded from a server device or the like on the network.

The image processing program 132 according to the present embodiment does not include all the software modules for realizing the functions according to the present embodiment, and the necessary functions are provided by cooperating with the OS 134. You may

The image processing program 132 according to the present embodiment may be provided by being incorporated in a part of another program. Even in that case, the image processing program 132 itself does not include a module included in another program combined as described above, and the process is executed in cooperation with the other program. As described above, the image processing program 132 according to the present embodiment may be incorporated in another program.

FIG. 2 shows an example in which the image determination apparatus 100 is realized by using a general-purpose computer, but the invention is not limited to this, and all or a part of the functions may be a dedicated circuit (for example, ASIC (application specific integrated circuit)). Or FPGA (field-programmable gate array) or the like. Further, a part of the processing may be assigned to an external device connected to the network.

[Function configuration]
FIG. 3 is a diagram showing functional blocks of the image determination device 100 according to the present embodiment. FIG. 4 is a diagram showing functional blocks of the learning model 136 according to this embodiment. The image determination apparatus 100 includes a learning model 136 stored in the storage 130 and a learning unit 150.

The learning model 136 includes a memory 1361, a feature extractor 1362, and a determiner 1363. In the memory 1361, characteristic data indicating one or a plurality of characteristics of the inspection target and related data related to the inspection target are accumulated as knowledge about the inspection target. The characteristic data is also called a characteristic amount. The related data may include either information that assists the determination or information that assists the visualization. The information that assists the determination may be information used in the determination processing directly or indirectly during the determination processing of the inspection target image. For example, the label indicating the determination result of the inspection, the reliability of the determination result, It may be any of the score shown, the time stamp indicating the time when each related data is added to the memory, and the like. The information that assists the visualization may be information used when presenting the determination result to the user, and may be, for example, an image to be inspected. For example, the memory 1361 may store images, labels, scores, time stamps, and the like in association with the characteristic data of the inspection target that has been inspected in the past. Further, the knowledge obtained for the inspection target that has been newly inspected may be added to the memory 1361. In addition to the characteristic data or in place of the characteristic data, derivative data such as a hash value calculated from the characteristic data may be stored in the memory 1361.

The feature extractor 1362 outputs feature data representing one or more features of the image based on the image of the inspection target. The feature extractor 1362 may be composed of a plurality of feature extractors that output feature data representing specific features of the image. The output feature data is passed to the memory 1361 and the determiner 1363. The learning model 136 outputs the reference data from the memory 1361 by referring to the memory 1361 based on the feature data output from the feature extractor 1362. At this time, from the data stored in the memory 1361, feature data having a distance close to that of the feature data output from the feature extractor 1362 is extracted as a feature of the inspection target image, and the relation associated with the feature data is extracted. The data may be output as reference data. The distance may be measured using any distance function. Also, from the data stored in the memory 1361, a predetermined number of feature data are extracted in order from the data having the closest distance to the feature data output from the feature extractor 1362, and are associated with these feature data. The reference data may be output based on the predetermined number of related data. At this time, the memory 1361 may output a predetermined number of related data as reference data, or may output the reference data calculated by a majority decision, an average, or other predetermined calculation of the predetermined number of related data.

The determiner 1363 outputs the output data including the determination result regarding the image based on the feature data output from the feature extractor 1362 and the reference data output from the memory 1361. The reference data may include either data based on information that assists determination or data based on information that assists visualization. When the reference data is data based on information that assists the determination, the reference data is used for the determination process in the determiner. Further, when the reference data is data based on information that assists visualization, it is used when presenting the determination result to the user.

For example, if a label indicating the determination result of the inspection is used as the reference data, the determiner 1363 can perform the determination considering the past inspection result. When the score indicating the reliability of the determination result is used as the reference data, the determiner 1363 can perform the determination considering the reliability of the past determination result. For example, as the output data, the appearance of the inspection target is good. / It is possible to output the degree of reliability of the determination result in addition to the result of determining the defect. When a time stamp indicating the time when the related data is added to the memory is used as the reference data, the determiner 1363 can make a determination in consideration of the time element. For example, by dampening the weighting of the data as the time the relevant data was added to the memory 1361, the old data added to the memory may not be of much interest and the most recently added data may be added. On the other hand, more attention can be paid to the determination.

The learning unit 150 uses the learning data 140 including the learning image and the label data to represent the label data associated with the learning image based on the feature data output when the learning image is input to the feature extractor 1362. The determiner 1363 is trained so that the output data is output, that is, the correct answer is predicted. When learning the determiner 1363, learning may be performed using both the input from the feature extractor 1362 and the input from the memory 1361. For example, a part of the learning data is put into the feature extractor 1362 and its output is recorded in the memory 1361. Then, with respect to learning data other than the above, it is put into the feature extractor 1362, the output thereof is inputted to the determiner 1363, and the data similar to the output of the feature extractor 1362 is detected from the memory 1361, and the detected data ( (Reference data) to the determiner 1363. Then, the decision unit 1363 is trained using both the input from the feature extractor 1362 and the input from the memory 1361. Here, the learning image is an image of an inspection target that has been captured in advance. The label data is data indicating the state of the inspection target, and may be data indicating the presence or absence of a defect related to the inspection target, the type of the defect, the external shape size of the defect, the type of the inspection target, and the like. The learning process of the learning model 136 may be executed before the image determination device 100 is installed in the production line.

The learning unit 150 uses the new learning data, and based on the feature data output when the image to be inspected included in the new learning data is input to the one or more feature extractors 1362, the determiner 1363. The determiner 1363 may be additionally learned so that the output data representing the label data associated with the image is output. Here, the additional learning may be performed on the determiner 1363 and not on the feature extractor 1362. Further, the characteristic data of the new learning data image, the image of the learning data, the label data and the like may be written in the memory 1361 as new knowledge.

The learning unit 150 may perform additional learning of the determiner 1363 by performing at least one of updating the parameter of the determiner 1363 and updating the weight of the feature data input to the determiner 1363. .. Here, the parameter of the determiner 1363 may be the weight or bias of the node, for example, when the determiner 1363 is composed of a neural network. When there are N types of feature data input to the determiner 1363, the feature data weights w _i (i=1 to N) may be positive real numbers that satisfy Σ _{i =1} ^N w _i =1. .. When the determiner 1363 is configured by a neural network, the learning unit 150 may update the parameters of the determiner 1363 and the weights of the feature data input to the determiner 1363 by a back propagation method. By such additional learning, the determiner can be updated so that the determination result can be obtained with higher accuracy.

The learning unit 150 may update the parameters of the feature extractor 1362 together with the preliminary learning and/or the additional learning of the determiner 1363. For example, when the feature extractor 1362 is configured by a neural network and the determiner 1363 is configured by a neural network, the learning unit 150 uses the parameters of the determiner 1363 and the features input to the determiner 1363 by the back propagation method. The data weights and the parameters of the feature extractor 1362 may be updated. In this way, the feature extractor 1362 is also updated by the pre-learning and/or the additional learning, so that the feature extraction can be performed with higher accuracy and a more accurate determination result can be obtained. Can be updated.

§3 Operation Example FIG. 5 is a flowchart of the image determination processing executed by the image determination device 100 according to the present embodiment. First, the image determination apparatus 100 associates the relevant data related to the inspection target in the memory 1361 in association with the feature data representing one or a plurality of features of the image of the inspection target or the derived data calculated based on the feature data. It is stored (S10).

Then, the learning image included in the learning data is input to the feature extractor 1362, and the output feature data is input to the determiner 1363 so that the output data corresponding to the label data included in the learning data is reproduced. The learning of the determiner 1363 is performed (S11). Thus, the pre-learning process is completed.

Then, after the image determination device 100 is installed on the production line, the image of the inspection target is captured by the camera 102 (S12). Then, the newly captured image is input to the feature extractor 1362, and reference data is output from the memory 1361 based on the output feature data. Then, the feature data output from the feature extractor 1362 and the reference data output from the memory 1361 are input to the determiner 1363, and the image is determined based on the output data including the determination result regarding the image (S13). Of course, the image capturing (S12) and the determination process (S13) may be repeatedly performed. With the above, the image determination processing ends.

The embodiments described above are for facilitating the understanding of the present invention, and are not for limiting the interpretation of the present invention. Each element included in the embodiment and its arrangement, material, condition, shape, size and the like are not limited to those illustrated, but can be appropriately changed. Further, the configurations shown in different embodiments can be partially replaced or combined.

The whole or part of the exemplary embodiments disclosed above can be described as, but not limited to, the following supplementary notes.

[Appendix 1]
A memory (1361) for storing related data relating to the inspection target in association with characteristic data representing one or a plurality of characteristics of the image of the inspection target or derived data calculated based on the characteristic data;
A feature extractor (1362) for outputting feature data representing one or more features of the image based on the image to be inspected;
A determination that outputs output data including a determination result regarding the image based on the feature data output from the feature extractor (1362) and reference data output from the memory (1361) based on the feature data Vessel (1363),
An image determination device (100) including the.

[Appendix 2]
The related data includes information to assist the determination,
The image determination device (100) according to attachment 1.

[Appendix 3]
The information that assists the determination includes at least one of a determination result regarding an image to be inspected, a score indicating reliability with respect to the determination result, and a time stamp indicating a time at which related data is added to the memory,
The image determination device (100) according to attachment 1.

[Appendix 4]
The related data includes information to assist visualization.
The image determination device (100) according to any one of appendices 1 to 3.

[Appendix 5]
The information that assists the visualization includes an image of an inspection target,
The image determination device (100) according to attachment 4.

[Appendix 6]
A learning model including a memory (1361) that stores associated data relating to the inspection target in association with feature data representing one or a plurality of features of the image of the inspection target or derived data calculated based on the feature data. At (136),
Outputting feature data representing one or more features of the image based on the image to be inspected;
Outputting output data including a determination result regarding the image based on the output feature data and reference data output from the memory (1361) based on the feature data;
The determination method including.

[Appendix 7]
Image determination that includes a memory (1361) that stores related data related to the inspection target in association with feature data representing one or more features of the image of the inspection target or derived data calculated based on the feature data The calculation unit provided in the device (100) is
A feature extractor (1362) that outputs feature data representing one or a plurality of features of the image based on an image to be inspected, feature data output from the feature extractor, and the feature data based on the feature data. A determiner (1363) for outputting output data including a determination result regarding the image based on the reference data output from the memory,
Judgment program to function as.

DESCRIPTION OF SYMBOLS 1... Image determination system, 2... Belt conveyor, 4... Work, 6... Imaging field, 8... Upper network, 10... PLC, 12... Database device, 100... Image determination device, 102... Camera, 104... Display, 106... Keyboard, 108... Mouse, 110... Processor, 112... Main memory, 114... Camera interface, 116... Input interface, 118... Display interface, 120... Communication interface, 122... Internal bus, 130... Storage, 132... Image processing program, 134... OS, 136... Learning model, 1361... Memory, 1362... Feature extractor, 1363... Judgment device, 138... Input image, 140... Learning data, 150... Learning unit

Claims (7)

A memory that stores related data related to the inspection target in association with feature data representing one or more features of the image of the inspection target or derived data calculated based on the feature data.
A feature extractor for outputting feature data representing one or more features of the image based on the image to be inspected;
Based on the feature data output from the feature extractor and the reference data output from the memory based on the feature data, a determiner that outputs output data including a determination result regarding the image,
An image determination device including. The related data includes information to assist the determination,
The image determination device according to claim 1. The information that assists the determination includes at least one of a determination result regarding an image to be inspected, a score indicating reliability with respect to the determination result, and a time stamp indicating a time at which related data is added to the memory,
The image determination device according to claim 2. The related data includes information to assist visualization.
The image determination device according to claim 1. The information that assists the visualization includes an image of an inspection target,
The image determination device according to claim 4. In association with the feature data representing one or a plurality of features of the image of the inspection target or derived data calculated based on the characteristic data, in a learning model including a memory for storing related data related to the inspection target,
Outputting feature data representing one or more features of the image based on the image to be inspected;
Based on the output feature data and the reference data output from the memory based on the feature data, outputting output data including a determination result regarding the image,
The determination method including. A computer including a memory that stores related data relating to the inspection target in association with feature data representing one or more features of the image of the inspection target or derived data calculated based on the feature data.
A feature extractor that outputs feature data representing one or more features of the image based on an image to be inspected, feature data output from the feature extractor, and output from the memory based on the feature data A determiner that outputs output data including a determination result regarding the image based on the reference data,
Judgment program to function as.