JP2019109563A - Data generation device, data generation method, and data generation program - Google Patents

Abstract

To provide a data generation device, a data generation method, and a data generation program that generate data for learning with a composite image including a portion to be detected that may actually occur in an object to be inspected.SOLUTION: The data generation device is a data generation device that generates data for learning for allowing a classifier to learn, the classifier identifying the presence/absence of a portion to be detected included in an object to be inspected, and comprises: an acquisition unit that acquires an image of the object to be inspected; an input unit that receives designation of a partial image being part of the image and including the portion to be detected; a correction unit that corrects the partial image on the basis of the feature quantity of the portion to be detected; and a generation unit that generates a composite image obtained by compositing the partial image corrected by the correction unit and an image different from the image including the partial image, and associates information indicating that the portion to be detected is included with the composite image to generate new data for learning for allowing the classifier to learn.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a data generation device, a data generation method, and a data generation program.

  In recent years, techniques for identifying images using a classifier such as a neural network have been studied. The classifier may be learned using data (teacher data) including an image for learning and a label of the image.

  In order to efficiently generate learning data, in Patent Document 1 below, difference data between a non-defective product and a non-defective product is extracted from a defective image of a subject, and a non-defective product image and difference data are combined to create a pseudo-defective image. A pseudo bad image automatic creation device is described.

JP, 2005-156334, A

  For example, when it is determined by the discriminator whether or not a detection target site such as a defect is included in the image of the inspection target, an erroneous determination may occur in which the detection target site is missed. In order to reduce such false judgments, data augmentation (Data Augmentation) is performed to increase variation of the image by rotating or translating the image including the missed detection target site, and data for learning In some cases, the discriminator may be trained to make it more difficult for missing detection sites to occur. For example, in Patent Document 1, data augmentation is performed by extracting difference data between a non-defective product image and a non-defective product image, setting synthesis conditions using random number values, and combining the difference data with the non-defective product image.

  However, in the case of setting the synthesis condition by the random number value, the synthesized image including the detection target portion which is imaged at the site is not necessarily generated. Therefore, even if learning data is increased by data augmentation to learn a classifier, when the classifier is used in the field, good discrimination performance can not always be obtained.

  Thus, the present invention provides a data generation apparatus, data generation method, and data generation program for generating learning data from a composite image that includes a detection target portion that can actually occur for an inspection object.

  A data generation device according to an aspect of the present disclosure is a data generation device that generates learning data for learning a discriminator that identifies the presence or absence of a detection target portion included in an inspection object. An acquisition unit that acquires an image, an input unit that is a part of the image and receives specification of a partial image including a detection target region, and a correction unit that corrects the partial image based on feature amounts of the detection target region; A composite image is generated by combining the partial image corrected by the correction unit and an image different from the image including the partial image, and information indicating that the detection target portion is included is associated with the composite image to generate a classifier. And a generation unit that generates new learning data for causing the learning. Here, the learning data includes image data input to the classifier, and data indicating at least a correct determination result regarding the image data and data that can be compared with the output of the classifier. Further, the correction method by the correction unit may be, for example, a deformation method of the partial image, that is, a change of the shape of the detection target portion included in the partial image or a change method of the brightness of the partial image. In addition, the method of generating a composite image by the generation unit may be, for example, a method of determining a portion to which a partial image is to be combined, or a method of selecting a background image to which a partial image is to be combined. Further, the detection target site may be any site included in the test subject, and may be, for example, a site of a defect included in the test subject.

  According to this aspect, by correcting the partial image based on the feature amount of the detection target site, it is possible to create variations of various detection target sites that may actually occur for the inspection target object, and the detection target that may actually occur. A composite image can be generated that includes the site. In addition, by using the generated data for learning, a discriminator capable of obtaining a good discrimination result when used in the field can be obtained. In addition to the information on the presence or absence of the detection target site, information indicating that the detection target site is included is the correct determination result regarding the image data, such as information on the position at which the detection target site is included. Information related to the presence or absence of the detection target site may be included.

  In the above aspect, the input unit may receive designation of at least one of the correction method by the correction unit and the generation method of the composite image by the generation unit.

  According to this aspect, by receiving at least one of the correction method and the generation method by the input unit, it is possible to generate the learning data so as to include the detection target portion that may actually be generated for the inspection object. . As a result, it is possible to create variations of various detection target sites that may actually occur for the test object, and it is possible to generate a composite image that includes the detection target sites that may actually occur.

  In the above aspect, the image processing apparatus further includes a determination unit that determines whether the image acquired by the acquisition unit includes the detection target portion by the classifier learned using the learning data, the input unit including the determination unit If the determination unit determines that the image does not include the detection target portion, and the input unit determines that the determination result by the determination unit is not correct. The partial image may be corrected.

  According to this aspect, it is possible to generate a composite image based on an image erroneously determined by the discriminator that the detection target site is not included, and to increase an image including the detection target site that is difficult to discriminate by the discriminator. Training data is generated that can further improve the discrimination performance of the classifier.

  In the above aspect, the discriminator learns the feature amount of the image corresponding to the detection target portion by learning using data for learning, and the correction unit determines the feature amount different from the feature amount learned by the discriminator The partial image may be corrected as it has. Here, the discriminator is, for example, a feature amount related to the shape of the detection target site such as the length of the detection target site, the area of the region including the detection target site, and the curvature of the detection target site; A feature amount relating to the appearance of the detection target site such as a change in brightness of the image along with the contrast of the detection target site and the periphery of the detection target site may be learned.

  According to this aspect, it is possible to generate a partial image having feature amounts that the discriminator could not learn by learning using learning data, and to increase variations of the feature amounts that can be learned by the discriminator New training data can be generated.

  In the above embodiment, the determination unit determines whether the synthesized image includes the detection target region, and the determination unit determines whether the image including the partial image includes the detection target region. The generation unit may further register the composite image as new learning data when the verification unit obtains a similar determination result. Here, with the determination result of similarity, for example, when the determination unit determines whether or not the detection target portion is included in the composite image, an image from which the partial image is extracted (that is, a measurement image of the inspection object) Similarly to the above, it is a determination result that an erroneous determination that the detection target site is not included occurs. That is, similar determination results include meanings such as similar determination results or corresponding determination results.

  According to this aspect, it is possible to verify whether or not a composite image is generated that is erroneously determined by the discriminator not to include the detection target site, and the image may include more detection target sites that are difficult to discriminate by the discriminator. Training data is generated that can further improve the discrimination performance of the classifier.

  In the above aspect, the generation unit generates a plurality of composite images in which the partial image corrected by the correction unit according to the plurality of correction methods and an image different from the image including the partial image are generated, and the detection target portion is included. The information indicating the information may be associated with a plurality of composite images to generate a plurality of learning data.

  According to this aspect, the partial image including the detection target site is corrected by a plurality of correction methods, and a plurality of composite images are generated to detect the detection target sites of various aspects from a small number of images including the detection target site. A plurality of combined images can be generated, and learning data that can further improve the generalization performance of the classifier can be generated.

  In the above aspect, the generation unit combines the partial image corrected by the correction unit with a plurality of images different from the image including the partial image to generate a plurality of composite images, and the detection target portion is included. The information indicating “1” may be associated with a plurality of composite images to generate a plurality of learning data.

  According to this aspect, partial images including the detection target site are synthesized with respect to a plurality of images of the inspection target that do not include the detection target site, and the detection target site is generated by generating a plurality of composite images. From a small number of images included, it is possible to generate a composite image including a detection target site in various backgrounds, and it is possible to generate training data that can further improve the generalization performance of the classifier.

  In the above aspect, each of the plurality of learning data is applied to the learning of the different classifiers, and the plurality of classifiers are used to identify whether the inspection object includes the detection target portion based on the input image. And a selection unit for selecting one or a plurality of learning data from a plurality of learning data based on the discrimination performance of the plurality of learned classifiers learned by the learning unit. You may have.

  According to this aspect, it is possible to train the classifier by training the classifier using the plurality of generated training data independently, and selecting one or more training data based on the classification performance. When used, it is possible to select learning data that improves the discrimination performance of the classifier better, and it is possible to improve the discrimination performance and learning efficiency of the classifier.

  In the above aspect, the generation unit may select the synthesis position of the partial image so as to generate a synthetic image so that the background pattern around the detection target portion is similar between the image including the partial image and the synthetic image. .

  According to this aspect, it is possible to perform data augmentation so that the partial image is synthesized so as to look natural at the synthesis destination, and includes a detection target site that can actually occur for the inspection object.

  In the above aspect, the correction unit may correct the brightness of the partial image.

  According to this aspect, it is possible to obtain learning data capable of causing the discriminator to learn so that the discriminator does not miss the part to be detected due to the difference in the illumination condition or the occurrence part of the part to be detected.

  In the above aspect, the correction unit may correct the shape of the partial image.

  According to this aspect, it is possible to obtain learning data capable of causing the discriminator to learn so that the discriminator does not miss the part to be detected due to the difference in the shape of the part to be detected. Here, it is preferable that the shape of the detection target portion included in the partial image is changed by correcting the shape of the partial image.

  In the above aspect, the correction unit may correct the shape of the partial image so that the shape of the partial image approaches any of the plurality of shape templates.

  According to this aspect, based on the prepared plurality of shape templates stored in a freely changeable state, the detection target portion that can actually occur for the inspection object can be easily synthesized, and it is unfamiliar with the designation of the shape. Even an operator can generate learning data that further improves the discrimination performance when using a discriminator in the field.

  In the above aspect, the generation unit combines the partial images corrected using the plurality of shape templates into an image different from the image including the partial image to generate a plurality of composite images, and the determination unit generates a plurality of combined images. The correction unit determines whether the detection target portion is included in the composite image, and the correction unit selects one of the plurality of shape templates based on the reliability of the determination by the determination unit, and corrects the shape of the partial image You may

  According to this aspect, correction of the shape of the partial image is performed using a shape template suitable for improving the discrimination performance of the classifier, and the discrimination performance and learning efficiency of the classifier can be improved. Here, the degree of reliability indicates the certainty of the detection target portion likeness of the region (composite region) of the corrected partial image included in the composite image, and for example, the judgment result of “detection target region present” is from 0 It may be an index represented by a numerical value of 1, and the closer the numerical value is to 1, the higher the certainty of the determination result may be.

  In the above aspect, the generation unit may generate a combined image such that the partial image is combined with an image of a portion of the inspection object where a detection target site is likely to be generated statistically.

  According to this aspect, it is possible to synthesize a partial image at a site that is likely to actually occur at the site by combining the partial image with the site at which the detection target site is likely to be generated statistically among the inspection objects. Data augmentation can be performed to include sites of detection that may actually occur for an object.

  In the above aspect, the detection target portion is a portion of the defect included in the inspection object, and the feature amount includes the shape of the defect, the brightness of the image, the change in the brightness of the image along the defect, and the defect and the defect It may include at least one of the contrast with the surroundings.

  According to this aspect, the discriminator identifies the defect by capturing the feature amount of the defect that is likely to miss the defect by the discriminator, correct the partial image based on the feature amount, and generate a composite image. It is possible to increase the number of images including the hard defects, and to generate training data that can further improve the discrimination performance of the classifier.

  A data generation method according to another aspect of the present disclosure is a data generation method for generating learning data for learning a discriminator that identifies the presence or absence of a detection target portion included in an inspection object, which is an inspection object Acquiring an image of the image, accepting a designation of a partial image including a detection target portion that is a part of the image, correcting the partial image based on the feature amount of the detection target portion, and correcting For generating a composite image obtained by combining the partial image and an image different from the image including the partial image, and associating the information indicating that the detection target portion is included with the composite image to train the classifier. Generating new learning data.

  According to this aspect, by correcting the partial image based on the feature amount of the detection target site, it is possible to create variations of various detection target sites that may actually occur for the inspection target object, and the detection target that may actually occur. A composite image can be generated that includes the site. In addition, by using the generated data for learning, a discriminator capable of obtaining a good discrimination result when used in the field can be obtained.

  A data generation program according to another aspect of the present disclosure includes an arithmetic device provided in a data generation device that generates learning data for learning a discriminator that identifies the presence or absence of a detection target portion included in an inspection object. An acquisition unit that acquires an image of an inspection object, an input unit that is a part of the image and receives specification of a partial image including a detection target region, and a correction that corrects the partial image based on feature amounts of the detection target region A composite image is generated by combining the partial image corrected by the unit and the correction unit with an image different from the image including the partial image, and information indicating that the detection target portion is included is associated with the composite image. The controller operates as a generation unit that generates new learning data to make the classifier learn.

  According to this aspect, by correcting the partial image based on the feature amount of the detection target site, it is possible to create variations of various detection target sites that may actually occur for the inspection target object, and the detection target that may actually occur. A composite image can be generated that includes the site. In addition, by using the generated data for learning, a discriminator capable of obtaining a good discrimination result when used in the field can be obtained.

  Provided are a data generation device, a data generation method, and a data generation program for generating learning data from a composite image that includes a detection target portion that can actually occur for an inspection object.

It is a functional block diagram of the data generation device concerning the embodiment of the present invention. It is a conceptual diagram which shows the content of the process in the 1st learning part of the data generation apparatus concerning this embodiment, and a 2nd learning part. It is a conceptual diagram which shows the content of the process in the measurement part of the data generation apparatus which concerns on this embodiment. It is a block diagram showing the hardware constitutions of the data generation device concerning this embodiment. It is a flowchart which shows the content of the 1st process performed by the data generation apparatus which concerns on this embodiment. It is a flowchart which shows the content of the correction process performed by the data generation apparatus which concerns on this embodiment. It is a flowchart which shows the content of the synthetic | combination process performed by the data generation apparatus which concerns on this embodiment. It is a figure which shows the outline | summary of the correction | amendment process performed by the data generation apparatus which concerns on this embodiment, and a synthetic | combination process. It is a figure which shows the outline | summary of the correction process performed by the data generation apparatus which concerns on this embodiment. It is a figure which shows the outline | summary of the mask process performed by the data generation apparatus which concerns on this embodiment. It is a flowchart which shows the content of the 2nd process performed by the data generation apparatus which concerns on this embodiment.

  Hereinafter, an embodiment according to one aspect of the present invention (hereinafter referred to as “the present embodiment”) will be described based on the drawings. In addition, what attached the same code | symbol in each figure has the same or same structure.

First, an example of a scene to which the present invention is applied will be described with reference to FIG. FIG. 1 is a functional block diagram schematically exemplifying an application scene of the data generation apparatus 100 according to the present embodiment. The data generation apparatus 100 according to the present embodiment is a correction method of a partial image including a detection target portion by the user interface 170 and a generation method of a composite image when the detection target portion is included in the image of the inspection object. A device that receives an input, corrects a partial image based on the feature amount of a detection target site, combines the corrected partial image with an image different from the image including the partial image, and generates new learning data is there.

  As shown in FIG. 1, the data generation device 100 corrects the partial image including the detection target portion in the image based on the feature amount of the detection target portion, and the imaging device 153 that acquires the image of the inspection target. A composite image is generated by combining the correction unit 121, the corrected partial image, and an image different from the image including the partial image, and information indicating that the detection target portion is included is associated with the composite image, A generation unit 122 that generates new learning data for causing the classifiers (a, b, c, A, B,...) To learn, a correction method of a partial image by the correction unit 121, and a composite image by the generation unit 122 A user interface 170 for receiving at least one of the generation methods. Here, the imaging device 153 is an example of the “acquisition unit” in the present invention, and the user interface 170 is an example of the “input unit” in the present invention. In this specification, the image of the inspection object newly captured by the imaging device 153 is referred to as a “measurement image”, and is collected in advance to learn the classifiers (a, b, c, A, B,...) The image of the inspection object is called a "sample image". The data generation device 100 may not necessarily include the imaging device 153, and may acquire a measurement image from a separately provided imaging device. In the present embodiment, the case where the detection target site is a site of a defect included in the inspection target will be described. However, the detection target site is not limited to the site of the defect included in the test target, and may be any site included in the test target.

  The measurement image of the inspection object captured by the imaging device 153 is displayed on the user interface 170 of the second operator. The second operator designates, using the user interface 170, a partial image including a defect in the measurement image. In addition, the second operator designates by the user interface 170 how to correct the partial image by the correction unit 121 (correction method), and the generation unit 122 converts the corrected partial image into another image. It is specified by the user interface 170 whether to synthesize (generation method).

  The correction unit 121 changes the length of the defect, the area of the region including the defect, the shape of the defect such as the curvature of the defect, the brightness of the image, the change in the brightness of the image along the defect, and the defect and the periphery of the defect. The partial image may be corrected based on the feature amount of the defect such as the contrast of the image. The correction unit 121 may correct a partial image based on the feature amount by capturing a feature amount of a defect which is likely to miss a defect by a classifier (a, b, c, A, B,...) And an erroneous determination is likely to occur. . Specifically, the discriminator (a, b, c, A, B,...) Can discriminate if it is a relatively long defect, but if it misses a relatively short defect, the correction unit 121 The partial images may be corrected so as to be short to increase variations of defects that cause the classifiers (a, b, c, A, B,...) To be difficult to discriminate. As the correction method by the correction unit 121, a detailed method may be designated by the user interface 170, or a rough method is designated by the user interface 170 or the details of the correction method by the correction unit 121 are not accepted. It may be determined automatically.

  The generation unit 122 combines the partial image corrected by the correction unit 121 with an image different from the image including the partial image to generate a composite image, and associates information indicating that a defect is included with the composite image. And generate new learning data for causing the classifiers (a, b, c, A, B,...) To learn. Here, an image different from an image including a partial image may be an image in which a defect is hidden by a mask, in addition to an image not including any defect. As a method of generating a composite image by the generation unit 122, a detailed method may be designated by the user interface 170, or a rough method is designated by the user interface 170 or the generation is performed by the generation unit 122 without accepting the designation. The details of the method may be determined automatically.

  In this way, by correcting the partial image based on the feature amount of the defect, it is possible to create variations of various defects that may actually occur in the inspection object, and a composite image that includes the defects that may actually occur. Can be generated. In addition, by receiving at least one of the correction method of the partial image by the correction unit 121 and the generation method of the composite image by the generation unit 122 by the user interface 170, it is possible to include defects that may actually occur in the inspection object. It is possible to generate learning data. Then, by using the generated data for learning, a discriminator capable of obtaining good discrimination performance when used in the field can be obtained. According to the data generation apparatus 100 according to the present embodiment, it is possible to generate a composite image including a defect that actually occurs in the field. It is possible to generate a discriminator with higher discrimination performance in the field. In addition, the discriminator includes a defect that is difficult to be identified by a discriminator by capturing a feature amount of a defect that is likely to miss a defect and causing an erroneous determination, correct the partial image based on the feature amount, and generate a composite image. In addition, learning data can be generated that can increase the number of images and can further improve the discrimination performance of the classifier.

22 Configuration example [Function configuration]
Next, an example of a functional configuration of the data generation apparatus 100 according to the present embodiment will be described using FIG. 1. The data generation apparatus 100 uses the learning data, and attribute information including the presence or absence of a defect of the inspection object, and the position of the defect and the type of the defect when the inspection object includes a defect It has a function of generating a discriminator that outputs (label). Here, when the inspection object includes a plurality of defects, the attribute information may include the presence or absence of the defects, the positions of the plurality of defects, and the types of the defects. In addition, the data generation apparatus 100 has a function of acquiring a measurement image obtained by measuring an inspection object, and outputting attribute information of the inspection object using the generated discriminator. Therefore, the data generation apparatus 100 functions as an inspection apparatus or an inspection system which acquires the measurement image of the inspection object and outputs the measurement result including the information on the presence or absence of the defect of the inspection object based on the measurement image.

  As shown in FIG. 1, the data generation apparatus 100 corrects the measured image (measurement image) based on the image acquisition unit 110 which generates and stores learning data, and the designated correction method and generation method. When the image correction unit 120, the learning data generated by at least one of the image collection unit 110 and the image correction unit 120 are used, and there is a defect in the inspection object and the inspection object includes a defect, A first learning unit 130 and a second learning unit 140 for generating a classifier that outputs attribute information including the position of the defect and the type of the defect, and a measurement image obtained by measuring the inspection object; A measurement unit 150 that outputs attribute information of the inspection object using the classifier generated by at least one of the first learning unit 130 and the second learning unit 140, the measurement image and the attribute information thereof are associated Te comprises a measurement result management unit 160 for storing a measurement result. Here, the measurement result includes the measurement image, the attribute information of the inspection object output by the classifier, and the determination result by the determination unit 152. The learning data may be not only a single learning data but also a data set having a plurality of learning data.

  Here, the defect is an abnormality that the inspection object has, and includes, for example, unevenness such as flaws, colors, stains, chips, burrs, foreign matter, printing blur, positional deviation such as printing, and the like.

  The attribute information includes at least information on the presence or absence of a defect included in each image, information indicating the position of the defect when the image includes a defect, and information indicating the type of the defect. The attribute information is not limited to the presence or absence of the defect, the information on the position of the defect, and the information on the type of the defect, for example, the reliability of the information on the presence or absence of the defect output by the identifier, the information on the position of the defect and the type of the defect It may contain information on the degree.

  The data generation apparatus 100 includes a storage unit (for example, the auxiliary storage units 102 and 103 shown in FIG. 4) for storing a software program and data used to execute the software program, and an operation unit (for example, for calling and executing the software program) , And the processor 104) shown in FIG. 4. That is, each of the functional blocks of the image collection unit 110, the image correction unit 120, the first learning unit 130, the second learning unit 140, the measuring unit 150, and the measurement result managing unit 160 stores the auxiliary storage units 102 and 103, etc. The predetermined software program stored in the unit is realized by being executed by a hardware processor such as the processor 104. That is, a software program is executed by the hardware processor for each of the functions of the image collection unit 110, the image correction unit 120, the first learning unit 130, the second learning unit 140, the measurement unit 150, and the measurement result management unit 160. Shows a predetermined process implemented by In addition, each of the image collection unit 110, the image correction unit 120, the first learning unit 130, the second learning unit 140, the measurement unit 150, and the measurement result management unit 160 may be used alone or in combination. It is configured as a device. Note that the data generation device 100 may be configured by a single information processing device. The hardware configuration of the information processing apparatus in the present embodiment will be described later.

<Configuration of Image Collection Unit>
The image acquisition unit 110 acquires a sample image for generating learning data. The sample image may be an image obtained by imaging an inspection object, or may be a CG image generated by a known image synthesis technique. The attribute adding unit 111 adds attribute information to each image or to a plurality of images in the sample image acquired by the image collection unit 110. That is, the attribute assignment unit 111 assigns the attribute information to the sample image, thereby making the sample image into learning data.

  In the present embodiment, assignment (labeling) of attribute information is performed by the first operator via an arbitrary user interface 180. For example, when a defect is included in the sample image, the first operator may use a display device (such as a liquid crystal display device) and an input device (such as a touch panel, a keyboard, or a mouse) as the user interface 180 to perform predetermined operations. By enclosing the defect by the frame of the shape, attribute information including the presence or absence of the defect and the position of the defect can be given. In addition, attribute information including the type of the defect can be added to the sample image using the user interface 180. In addition, the method of provision of attribute information is not specifically limited. For example, the attribute information may be automatically added to the image using a classifier that has already learned the relationship between the image and the attribute information to be added to the image, or a known statistical method may be used. A plurality of images may be clustered by a method and attribute information may be attached to the cluster.

  The image (learning data) to which the attribute information is added by the attribute adding unit 111 is stored in the image database (hereinafter, referred to as "image DB") 112 and stored.

  The image collection unit 110 includes an image correction unit 113 that expands learning data according to, for example, some of the sample images to be acquired and some of the learning data stored in the image DB 112. The image correction unit 113 may extend the sample image based on the correction operation of the first operator performed via the user interface 180. The image correction unit 113 is not necessarily an essential element of the image collection unit 110. Also, the extension of learning data refers to data augmentation in which data sets used for learning are increased.

  For example, learning data may be expanded (or augmented) by translating, rotating, changing colors, enlarging, or reducing a sample image. Note that the image correction unit 113 is not a component essential to the image collection unit 110, and may be appropriately provided according to the necessary amount of learning data required for the image measurement process realized using the data generation apparatus 100. Further, the augmentation may be realized by the worker himself instructing each processing such as extraction of a partial image, composition with a background image, and the like through an arbitrary user interface 180. The learning data expanded by the image correction unit 113 is stored in the image DB 112 and stored.

  Here, the background image is an image to be combined with the partial image, and includes an image different from the measurement image from which the partial image is extracted. The background image may be a measurement image, but when the measurement image is used as a background image, the synthesis location is set so that the partial image is synthesized in an area other than the partial image extracted from the measurement image. It can be used limited. In addition, the background image may be generated based on an image other than the measurement image, and may or may not have the same pattern as the image of the area other than the defect of the measurement image. . Also, the background image may be an image having a uniform background pattern, an image having a background pattern such as a hairline, or an image on which characters and the like are printed. The image may be

  The image DB 112 acquires learning data from the attribute assignment unit 111 and the image correction unit 113. The image DB 112 may store learning data in a state of being classified based on the assigned attribute information. When attribute information is already added to the sample image acquired by the image collection unit 110, the image DB 112 acquires the sample image without passing through the attribute addition unit 111 and the image correction unit 113, May be stored. In addition, the image DB 112 determines whether or not learning data for learning processing used in learning processing for generating a classifier to be described later, and the generated classifier outputs desired attribute information. Test data for evaluation processing for evaluating whether or not may be stored separately. However, when the learning data and the test data are stored separately in the image DB 112 without distinction between the data for testing and the data for testing, the data set stored in the image DB 112 is divided into one for learning and one for testing by an arbitrary method. You may do it.

<Configuration of First Learning Unit>
FIG. 2 is a conceptual diagram showing the contents of processing in the first learning unit 130 and the second learning unit 140 of the data generation device according to the present embodiment. The first learning unit 130 acquires learning data from the image collection unit 110. The first learning unit 130 may acquire learning data from an image correction unit 120 described later. The first learning unit 130 includes a first learning processing unit 131 that executes machine learning using the acquired learning data and generates classifiers (a, b, c,...). The classifier (a, b, c,...) Takes image data as an input, and is an attribute including the presence or absence of a defect included in the image, and the position and the type of one or more defects if any. It outputs information. Data other than image data may be input to the classifiers (a, b, c,...). For example, in the classifier (a, b, c,...), A condition (a light amount, an exposure time of an imaging device, etc.) under which the image of the inspection object is captured is input or identification information of the inspection object is input It may be done.

  The classifiers (a, b, c,...) Are generated by performing learning processing and evaluation processing on an arbitrary machine learning model. The machine learning model has a predetermined model structure and processing parameters that change due to learning processing, and the processing parameters are optimized based on the experience obtained from the learning data, thereby improving the identification accuracy. It is a model. That is, the machine learning model is a model that learns optimal processing parameters by learning processing. The algorithm of the machine learning model may be, for example, support vector machine, logistic regression, neural network or the like, but the type is not particularly limited. In this embodiment, an example using a convolutional neural network suitable for image recognition among neural networks, in particular, deep neural networks having three or more layers will be described. The first learning unit 130 may include a model generation unit (not shown) for generating a machine learning model. In addition, the first learning unit 130 may obtain a machine learning model generated in advance from the outside.

  The discriminator according to the present embodiment learns a feature amount of an image corresponding to a defect, which has a predetermined width, by prior learning using learning data. Here, the feature amount having a predetermined width may be a feature amount having a predetermined numerical range. For example, if the feature amount represents the length of a defect, the feature amount is a numerical range of the length. You may have The type of feature to be learned can be set unique to the classifier. Further, the width of the feature value to be learned depends on the content of the learning data. Here, the feature amount refers to, for example, the feature length relating to the shape of the defect such as the length of the defect, the area of the region including the defect, the curvature of the defect, the brightness of the image, and the change of the image brightness along the defect. And / or a contrast between the defect and the periphery of the defect. In addition, the feature amount may be classified into, for example, four points of white point, black point, white line, and black line according to the type of defect. The measuring unit 150, which will be described later, uses the discriminator to discriminate whether or not there is a portion in the measurement image that matches the feature of the predetermined width learned by the discriminator, and based on the discrimination result And output the determination result including the information on the presence or absence of the defect.

  The classifier may be expressed by processing parameters. The processing parameter is information representing a discriminator, and the number of layers of the neural network and the number of nodes relating to each layer in the case of a neural network (hereinafter sometimes simply referred to as "network") as a discriminator. , Weight parameters of links connecting nodes, bias parameters for each node, and information on a function form of an activation function for each node. The internal parameters define the width of the feature to be detected as a defect.

  The first learning processing unit 131 learns the machine learning model so as to output an output value corresponding to the attribute of the measurement image represented by the attribute information when the data of the acquired image is input using the data for learning Perform learning processing. The first learning processing unit 131 further uses the test data for the classifiers (a, b, c,...) Generated by the learning process to discriminate the classifiers (a, b, c,. • Perform evaluation processing to compare the output of) with the attribute information given to the test data. If the output of the discriminator (a, b, c, ...) matches the attribute information given to the test data, the discriminator (a, b, c, ...) It is determined that the generation condition is satisfied, and the generation processing of the classifiers (a, b, c,...) Is completed.

  The first learning unit 130 uses the information including the generated model structure and processing parameters of the classifiers (a, b, c,...) As the learning result data to the measuring unit 150 and the second learning unit 140 described later. Output.

<Configuration of Second Learning Unit>
Subsequently, the content of the process in the second learning unit 140 will be described using FIG. 2. The second learning unit 140 performs additional learning (relearning) on the classifier generated by the predetermined generation processing. The description of the elements in common with the first learning unit 130 in the second learning unit 140 is omitted.

  The second learning unit 140 acquires learning result data of the learned classifiers (a, b, c,...) Generated by the first learning unit 130. Note that the second learning unit 140 may obtain learning result data related to a classifier generated by a predetermined generation process from the outside. The second learning unit 140 acquires learning data from the image collection unit 110. The second learning unit 140 may acquire learning data from the image correction unit 120 described later.

  The second learning unit 140 has a second learning processing unit 141 that executes additional learning (relearning) by machine learning using learning data, and generates classifiers (A, B, C,...) . The classifiers (A, B, C,...) Are generated by performing learning processing and evaluation processing on the classifiers (a, b, c,...) Generated by predetermined learning processing. Be done. In other words, the second learning processing unit 141 uses the learning data to perform learning processing and evaluation processing on learning result data including the model structure and processing parameters of the classifier (a, b, c,...) To generate classifiers (A, B, C,...). The discriminators (A, B, C,...) Receive image data, and the presence or absence of a defect included in the image, the position and the type of the defect if one or more defects are included, To output attribute information including

  When the second learning processing unit 141 receives data of the acquired image using the data for learning, the second learning processing unit 141 outputs the output value corresponding to the attribute of the measurement image represented by the attribute information. ..)) is subjected to learning processing (learning processing of additional learning) for learning the machine learning model represented by the learning result data of b),. Here, the learning process of the additional learning performed by the second learning processing unit 141 may be performed using the learning data acquired from the image correction unit 120 described later. As a result, it is possible to use learning data that is newly generated based on an image obtained by imaging the inspection object, and it is possible to improve the discrimination accuracy of the classifier.

  The second learning processing unit 141 further uses the test data for the classifiers (A, B, C,...) Generated by the learning process to discriminate the classifiers (A, B, C,. • Perform evaluation processing to compare the output of) with the attribute information given to the test data. If the output of the discriminator (A, B, C,...) Matches the attribute information given to the test data, the discriminator (A, B, C,...) It is determined that the generation condition is satisfied, and the generation process of the classifiers (A, B, C,...) Is completed. Here, the evaluation process performed by the second learning processing unit 141 may be performed using test data acquired from the image correction unit 120 described later. As a result, evaluation can be performed using learning data newly generated based on an image obtained by imaging the inspection object, and the recognition accuracy of the classifier can be improved.

  The second learning unit 140 outputs information including the generated model structure of the classifiers (A, B, C,...) And processing parameters as learning result data to the measurement unit 150 described later.

  The first learning unit 130 and the second learning unit 140 may be integrally configured, and the first learning processing unit 131 may perform the learning process and the evaluation process of the additional learning.

<Configuration of Measurement Unit>
FIG. 3 is a conceptual diagram showing the contents of processing in the measurement unit 150 of the data generation apparatus 100 according to the present embodiment. The measurement unit 150 causes the determination unit 152 to output attribute information of the measured image of the inspection object captured by the imaging device 153 using the classifier generated by the learning process and evaluation process using the learning data. It is determined whether the inspection object includes a defect, and a measurement result including a measurement image, attribute information, and a determination result is output. The attribute information includes the presence or absence of a defect, and the position and type of the defect if one or more defects are included.

  The measurement unit 150 acquires, as learning result data, information including the model structure of the classifier and the processing parameters from the first learning unit 130 and the second learning unit 140. The measurement unit 150 has a learning result DB 151 that stores the acquired learning result data. The measurement unit 150 also includes an imaging device 153 that images an inspection object. The imaging device 153 is installed, for example, to image a factory production line, and constitutes a part of the production line. The imaging device 153 captures an image of a part and a product (hereinafter, the part and the product may be collectively referred to as a “work”) as an inspection object flowing in the manufacturing line as a measurement image.

  The measurement unit 150 includes a determination unit 152 that outputs attribute information of the measurement image using a classifier configured based on the learning result data stored in the learning result DB 151. The determination unit 152 determines whether or not the measurement image captured by the imaging device 153 includes a defect by the classifiers (a, b, c, A, B,...) Learned using the learning data. judge.

  The measurement unit 150 outputs the measurement result including the measurement image, the attribute information of the measurement image output by the discriminator, and the determination result by the determination unit 152 to the measurement result management unit 160. The measuring unit 150 may output identification information for identifying the individual inspection object together with the measurement result to the measurement result management unit 160. At this time, the identification information of the inspection object may be acquired from the measurement image, or may be acquired from the outside via a predetermined interface. As the identification information, for example, a serial number or a graphic obtained by encoding the serial number can be used. Therefore, the identification information is preferably information that can identify an individual of the test object. The measurement unit 150 may output the measurement result of the inspection object to other than the measurement result management unit 160. For example, the measurement result and the identification information may be output to a user interface 170 described later. Also, for example, measurement results may be output directly or indirectly to a sorting device (not shown). The sorting apparatus sorts the inspection object based on the acquired measurement result.

<Configuration of Measurement Result Management Unit>
The measurement result management unit 160 includes a measurement result DB 161 that associates and stores the measurement image included in the measurement result, the attribute information output by the classifier and the determination result by the determination unit 152. The measurement result management unit 160 may obtain identification information for identifying the inspection object together with the measurement result. At this time, the identification information of the inspection object may be acquired from the measuring unit 150, or may be acquired from the outside through a predetermined interface. In addition, the measurement result DB 161 may classify and store the measurement image in association with the attribute of the type of defect, for example, according to the type of attribute included in the attribute information.

  The measurement result management unit 160 outputs the measurement result to the user interface 170. At this time, the measurement result management unit 160 may output identification information of the inspection object to the user interface 170. Also, the measurement result management unit 160 outputs the measurement result to the image correction unit 120 described later. At this time, the measurement result management unit 160 may output the identification information of the inspection object to the image correction unit 120 at the same time.

<User interface>
In the present embodiment, the user interface 170 is used by the second operator to confirm the determination result by the determination unit 152, or to specify a partial image including a defect included in the measurement image, It is used to receive an input as to whether the determination result by the determination unit 152 is correct, to be used to specify a correction method by the correction unit 121, or to specify a method to generate a composite image by the generation unit 122 . The user interface 170 is at least a display unit (such as a liquid crystal display device) for displaying the measurement image and the measurement result by the measurement unit 150, and designation of a partial image by the second operator, and determination result by the determination unit 152 is correct It has an input part (a touch panel, a keyboard, a mouse, etc.) for performing the input of no, the specification of the correction method, and the specification of the generation method. The second worker determines the correctness of the measurement result based on the measurement image displayed on the display unit and the measurement result of the measurement image, inputs the correctness of the measurement result using the input unit, and includes the part including the defect The area of the image may be input. Then, based on the measurement image displayed on the display unit and the measurement result of the measurement image, the second worker how to correct the designated partial image, the corrected partial image to another image It may be specified using the input unit whether to combine as described above.

<Configuration of Image Correction Unit>
The image correction unit 120 corrects the acquired partial image to generate learning data. The image correction unit 120 includes a correction unit 121, a generation unit 122, a correction image database (hereinafter referred to as "correction image DB") 123, a verification unit 124, and a selection unit 125, and the partial image received by the user interface 170 Are corrected, and the corrected partial image is synthesized with another image to generate learning data. In the following, the image correction unit 120 corrects the partial image received by the user interface 170 to generate new learning data, but corrects the partial image received by the user interface 180. , New training data may be generated.

  The correction unit 121 corrects a partial image including a defect in the image based on the feature amount of the defect. The correction unit 121 may correct the partial image when the determination unit 152 determines that the image does not contain a defect and the user interface 170 inputs that the determination result of the determination unit 152 is not correct. For example, the correction unit 121 compares the feature amount of the partial image of the defect with the feature amount learned by the classifier used by the determination unit 152, and has a feature amount different from the feature amount learned by the classifier. Thus, partial images may be corrected. More specifically, when the discriminator learns the range of the feature amount of the image corresponding to the defect by the prior learning using the data for learning, the correction unit 121 detects the feature of the defect as the discriminator The partial image may be corrected so that it does not fall within the range of learned feature amounts. For example, when the feature amount of the image corresponding to the defect learned by the discriminator represents the range of the length of the defect, the correction unit 121 deviates from the range of the length of the defect learned by the discriminator In addition, partial images including defects may be corrected by scaling. As a result, a composite image can be generated based on an image that is misjudged by the discriminator as not containing a defect, and images including defects that are difficult to discriminate by the discriminator can be increased. It is possible to generate learning data that can be further improved. In addition, it is possible to generate a partial image having a feature quantity that the discriminator could not learn by learning using data for learning, and for new learning that increases variations of the feature quantity that can be learned by the discriminator Data can be generated.

  The correction unit 121 may perform a plurality of different corrections on the partial image specified by the user interface 170. More specifically, the correction unit 121 may correct the brightness of the partial image or correct the shape of the partial image. By correcting the brightness of the partial image, it is possible to obtain learning data that can learn the classifier so that the classifier does not miss the defect due to the difference in the illumination condition or the location of the defect. . Further, by correcting the shape of the partial image, it is possible to obtain learning data that can make the classifier learn so that the classifier does not miss the defect due to the difference in the shape of the defect.

  Further, the correction unit 121 may correct the shape of the partial image so that the shape of the partial image approaches any of the plurality of shape templates. An example of such correction will be described more specifically using FIG. By preparing a plurality of shape templates, selecting a shape template, and correcting the shape of the partial image so as to approach the selected shape template, various deformation variations can be easily specified. It is easy to specify the method. As a result, it is possible to easily synthesize shape defects that may actually occur in the inspection object, and even for an unskilled worker for specifying a shape, the identification performance in the case where the identifier is used in the field is further improved. Data for learning can be generated.

  The correction unit 121 may select any one of the plurality of shape templates based on the reliability of the determination by the determination unit 152 and correct the shape of the partial image. Here, the degree of reliability indicates the probability of defect likeness of the region (composite region) of the corrected partial image included in the composite image, and for example, the judgment result with “defective” is a numerical value of 0 to 1 It is an expressed index, and the closer the numerical value is to 1, the higher the certainty of the judgment result. In this case, the determination unit 152 determines whether a defect is included in the plurality of composite images generated by combining the partial images corrected using the plurality of shape templates with an image different from the image including the partial image. It is judged whether or not it is not, and the reliability of each judgment is outputted. For example, the correction unit 121 may correct the shape of the partial image by selecting a shape template in which the reliability of the determination by the determination unit 152 is relatively low. In this manner, the shape of the partial image is corrected using the shape template suitable for improving the discrimination performance of the classifier, and the discrimination performance and learning efficiency of the classifier can be improved.

  The generation unit 122 generates a composite image obtained by combining the partial image corrected by the correction unit 121 and an image different from the image including the partial image, and associates information indicating that a defect is included with the composite image. And generate new learning data to make the classifier learn. The generation unit 122 generates a plurality of composite images obtained by combining the partial images corrected by the correction unit 121 according to the plurality of correction methods with the image of the inspection object different from the image including the partial images, and includes defects. The information indicating the information may be associated with a plurality of composite images to generate a plurality of learning data candidates. By correcting partial images including defects with multiple correction methods and generating multiple composite images, multiple composite images including defects of various aspects can be generated from a small number of images including defects. It is possible to generate training data candidates that can further improve the generalization performance of the classifier.

  Further, the generation unit 122 combines the partial image corrected by the correction unit 121 with a plurality of images different from the image including the partial image to generate a plurality of combined images, and indicates that a defect is included. Information may be associated with a plurality of composite images to generate a plurality of learning data candidates. By combining partial images containing defects with multiple images of the inspection object that does not contain defects, and generating multiple composite images, defects from a small number of images containing defects in various backgrounds A plurality of combined images can be generated, and candidates for training data that can further improve the generalization performance of the classifier can be generated.

  The generation unit 122 may generate a composite image by selecting a combination position of the partial images such that the background pattern around the defect is similar between the image including the partial image and the composite image. More specifically, for example, in the image including the partial image and the composite image, the partial image such that the variation of the lightness / darkness of the pixel value of each image independent of the texture of the imaging object falls within a predetermined range. The composite position of may be selected to generate a composite image. As a result, partial images can be synthesized so as to look natural at the synthesis destination, and data augmentation can be performed so as to include defects that may actually occur in the inspection object. In addition, the generation unit 122 may generate a composite image so that the partial image is composited at a position where defects are likely to be generated statistically among the inspection objects. The portion of the inspection object that is likely to cause a defect may be identified based on the history of the portions of the inspection object that have occurred in the past. By combining partial images with locations that are likely to cause defects statistically among the inspection objects, partial images can be combined with locations that are likely to actually occur in the field, and defects that may actually occur with respect to the inspection objects Data augmentation can be done to include.

  The correction image DB 123 stores and stores the new learning data generated by the generation unit 122. The correction image DB 123 may temporarily store the measurement image and the partial image received from the user interface 170, or may temporarily store the sample image and the partial image received from the user interface 180.

  The verification unit 124 causes the determination unit 152 to determine whether a defect is included in the composite image and causes the determination unit 152 to determine whether a defect is included in the image that is the composition source of the composite image. Verify that a similar judgment result can be obtained. For example, when the determination unit 152 erroneously determines that a defect is not included in the measurement image which is the composition source of the composite image, the verification unit 124 determines whether the composite image includes a defect. The determination is made by 152, and the determination unit 152 verifies whether or not it is erroneously determined that no defect is included. Thus, it is possible to verify whether a composite image is generated that is misjudged by the discriminator as not containing a defect, and it is possible to increase images containing defects that are difficult to discriminate by the discriminator, and the discriminator's discrimination performance Training data is generated that can be enhanced.

  The image correction unit 120 applies each of the plurality of learning data candidates generated by the generation unit 122 to the learning of the different classifiers, and identifies a plurality of input image images including defects. The classifiers (a, b, c, A, B,...) May be learned by the first learning unit 130 or the second learning unit 140. The image correction unit 120 generates, for example, an N × M combination of a partial image corrected by the N correction methods by the correction unit 121 and M images of the inspection object not including a defect. To generate a plurality of candidates for learning data and generate N classifiers (a, b, c, A, B,...) Using N × M learning data. The learning may be performed by the first learning unit 130 or the second learning unit 140. At this time, a plurality of classifiers (a, b, c, A, B,...) Are learned by the first learning unit 130 or the second learning unit 140 using N × M learning data independently. You may Thereby, it is possible to try which correction method and generation method are optimal for improving the discrimination performance of a plurality of classifiers (a, b, c, A, B,...).

  The selection unit 125 selects one or more learning data from a plurality of learning data candidates based on the discrimination performance of the plurality of learned classifiers learned by the first learning unit 130 or the second learning unit 140. Choose The selection unit 125 selects one or a plurality of learning data based on the discrimination performance when the plurality of classifiers are trained using the plurality of generated training data candidates, thereby performing learning of the classifiers. When used in the above, it is possible to select learning data that improves the discrimination performance of the discriminator better, and the discrimination performance and the learning efficiency of the discriminator can be improved.

[Hardware configuration]
Next, an example of the hardware configuration of the data generation apparatus 100 according to the present embodiment will be described using FIG. 4. The data generation device 100 according to the present embodiment may include the arithmetic device 101, the output device 191, the input device 192, and the imaging device 153. Here, the output device 191 and the input device 192 constitute a user interface 170 and a user interface 180.

  The arithmetic device 101 may include auxiliary storage units 102 and 103, a processor 104, a main storage unit 105, and an external interface (hereinafter referred to as "external I / F").

  The arithmetic device 101 includes a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), and the like, and controls each component according to information processing. The processor 104 may include a CPU, and the auxiliary storage units 102 and 103 and the main storage unit 105 may include a RAM and a ROM. The auxiliary storage unit 102 may be, for example, a hard disk drive, a solid state drive, etc., and stores the data generation program 102a executed by the processor 104, the measurement image 102b captured by the imaging device 153, learning data 102c, etc. You may Similar to the auxiliary storage unit 102, the auxiliary storage unit 103 and the main storage unit 105 may be, for example, a hard disk drive or a solid state drive. The data generation program 102 a performs processing for realizing the functions of the image collection unit 110, the image correction unit 120, the first learning unit 130, the second learning unit 140, the measurement unit 150, and the measurement result management unit 160 described above. Each function is realized by the program being executed by the processor 104. The data generation program 102 a is for realizing some of the functions of the image collection unit 110, the image correction unit 120, the first learning unit 130, the second learning unit 140, the measurement unit 150, and the measurement result management unit 160. It may be a program for performing processing, or may include a program for performing processing for realizing functions other than these.

  The external I / F 106 is a USB (Universal Serial Bus) port or the like, and is an interface for connecting to an external device such as the output device 191, the input device 192, and the imaging device 153. The data generation apparatus may include an interface for performing wired or wireless communication via a network, such as a wired LAN (Local Area Network) module or a wireless LAN module.

  A storage medium is a medium that stores information such as a program by electrical, magnetic, optical, mechanical or chemical action so that information such as a stored program can be read by a device such as a computer It is. The storage medium is, for example, a CD (Compact Disk), a DVD (Digital Versatile Disk), or the like, and may store the data generation program 102a. The data generation program 102 a stored in the storage medium may be copied to the auxiliary storage unit 103 and copied to the auxiliary storage unit 102 or the main storage unit 105. FIG. 4 exemplifies a disk-type storage medium such as a CD and a DVD as an example of the storage medium. However, the type of storage medium is not limited to the disc type, and may be other than the disc type. As a storage medium other than the disk type, for example, a semiconductor memory such as a flash memory can be mentioned. In addition, the data generation apparatus 100 may acquire data such as the data generation program 102 a via a network by wired or wireless communication.

  The specific hardware configuration of the data generation apparatus 100 may omit, replace, and add components as appropriate according to the embodiment. For example, the computing device 101 may include a plurality of processors, or may include a GPU (Graphical Processing Unit) or an FPGA (Field Programmable Gate Array). Further, the data generation device 100 may be configured by a plurality of information processing devices. The data generation apparatus 100 may be configured by a general-purpose desktop PC (Personal Computer), a tablet PC, or the like, in addition to an information processing apparatus designed specifically for the service to be provided.

Section 3 Operation Example FIG. 5 is a flowchart showing the contents of the first process executed by the data generation apparatus 100 according to the present embodiment. The first process is a process of generating new learning data by the generation unit 122 when the user interface 170 inputs that the determination result by the determination unit 152 is not correct. The data generation apparatus 100 first acquires an image of an inspection object (S10). Here, the image may be a sample image stored in advance in the image DB 112, or may be a measurement image newly captured by the imaging device 153.

  The determination unit 152 of the data generation apparatus 100 determines whether a defect is included in the acquired image by the classifier learned using the learning data (S11). The second worker confirms the image to be judged and the judgment result by the user interface 170, and the user interface 170 receives an input regarding whether or not the judgment result by the judgment unit 152 is correct (S12).

  When the determination unit 152 determines that the image does not contain a defect, and the user interface 170 inputs that the determination result by the determination unit 152 is not correct (S13: NO), that is, the defect is missed by the classifier The user interface 170 receives specification of a partial image including a defect in the image (S14). If the determination unit 152 can correctly determine the presence or absence of a defect (S13: YES), the first process ends.

  The correction unit 121 of the data generation device 100 corrects the designated partial image (S15). The details of the correction processing will be described in detail using the following figures. Further, the generation unit 122 of the data generation device 100 generates a composite image in which the partial image corrected by the correction unit 121 and an image different from the image including the partial image are synthesized (S16). Details of the synthesis process will be described in detail with reference to FIG.

  The verification unit 124 of the data generation apparatus 100 causes the determination unit 152 to determine whether a defect is included in the generated composite image (S17). As a result of the determination, it is verified whether a determination result that no defect is included is obtained for a defect included in the synthesized partial image, similar to the case where the determination unit 152 determines an image including a partial image S18). Note that the verification unit 124 may verify the determination result by the determination unit 125 for the area of the combined image in which the partial image is combined, and the determination unit related to the area other than the area of the combined image in which the partial image is combined. Verification of the determination result by 125 may not necessarily be performed. If a determination result similar to the image of the inspection object (hereinafter referred to as “composition source image”) which is the combination source can not be obtained (S18: NO), the data generation apparatus 100 corrects the partial image (S15) and The combining process (S16) is performed again by a different method, and the determining unit 152 determines whether the generated combined image includes a defect (S17), and verifies whether a determination result similar to the combining source image can be obtained Do. When a determination result similar to the combination source image is obtained (S18: YES), the generation unit 122 associates the information indicating that a defect is included with the combined image to learn a classifier. Learning data is generated and stored in the corrected image DB 123 (S19). Thus, the first process ends.

  FIG. 6 is a flowchart showing the contents of the correction process (S15) executed by the data generation apparatus 100 according to the present embodiment. First, the data generation apparatus 100 receives the selection of the correction method through the user interface 170 (S151). The selection of the correction method may include at least one of the selection of the feature amount of the defect used for the correction, the selection of the correction of the shape of the partial image, and the selection of the correction of the brightness of the partial image. Below, the case where correction | amendment of the shape of a partial image is performed and correction | amendment of the brightness of a partial image is performed according to selection is demonstrated.

  When the data generation apparatus 100 does not automatically select the shape template in the correction of the shape of the partial image (S152: NO), the user interface 170 receives the selection of the shape template (S153). On the other hand, when a shape template is automatically selected by the data generation apparatus 100 (S152: YES), the history in which it is determined whether a defect is included in the composite image generated using the plurality of shape templates by the determination unit 152 is used. Referring to (S154), one of the plurality of shape templates is selected based on the reliability of the determination by the determination unit 152 (S155). For example, a shape template that can generate a composite image with a relatively low degree of reliability of the determination by the determination unit 152 may be selected. Such selection makes it possible to generate a composite image that is difficult to distinguish by the classifier, and to generate a composite image that can further improve the discrimination performance of the classifier.

  The correction unit 121 corrects the shape of the partial image so that the shape of the partial image approaches the selected shape template (S156). After that, when the brightness of the partial image is corrected (S157: YES), the correction unit 121 corrects the partial image so that the brightness of the partial image changes (S158). Here, the change rate or change mode of the brightness may be specified by the user interface 170 or may be automatically adjusted by the correction unit 121. When the brightness correction method is automatically adjusted by the correction unit 121, as in the case of shape correction, the partial images corrected by a plurality of methods are respectively combined with the image of the inspection object containing no defect. The plurality of composite images are generated, and the determination unit 152 determines whether or not a defect is included in the generated plurality of composite images, based on the reliability of the determination by the determination unit. One of them may be selected. Thus, the correction process (S15) ends. In addition, each correction method such as selection of feature amount of defect used for correction, selection of presence / absence of correction of shape of partial image, selection of presence / absence of correction of brightness of partial image, etc. may be performed alone or in combination. It can be used.

  FIG. 7 is a flowchart showing the contents of the combining process (S16) executed by the data generation apparatus 100 according to the present embodiment. First, the data generation apparatus 100 receives a selection of a background image (hereinafter referred to as a synthesis destination image) to be a synthesis destination through the user interface 170 (S161). The synthesis destination image may be obtained by imaging the same inspection object as the inspection object captured in the image of the inspection object that is the synthesis source (synthesis source image), or the inspection that is captured in the synthesis source image It may be obtained by imaging an inspection object different from the object. Furthermore, the user interface 170 receives a selection of a method of generating a composite image (S162). The selection of the generation method is at least one of selection of whether to set a mask in the combination destination image, selection of whether to give priority to combination to a position where defects are likely to occur, selection of a combination position of partial images, etc. May be included.

  When the setting of the mask is selected by the user interface 170 (S163: YES), the data generating apparatus 100 sets the mask on the combining destination image, and restricts the part where the partial image is to be combined (S164). When setting the mask, the shape of the mask may be received by the user interface 170.

  After that, the data generation apparatus 100 determines whether to give priority to combining the partial image to the part where defects are likely to occur (S165). In the case where priority is given to combining with a location where defects are likely to occur (S165: YES), the generation unit 122 statistically tends to cause defects among inspection objects based on the history of defect locations detected for the inspection object. The location is identified (S166). Then, the generation unit 122 synthesizes a partial image at a position where a defect is likely to occur statistically among the inspection objects, and generates a synthesized image (S167).

  On the other hand, when priority is not given to combining with a location where defects are likely to occur (S165: NO), locations where the background pattern around the defect is similar between the combining source image and the combining destination image are specified (S168). For example, if hairline processing is performed around defects in the composite source image and hairline processing is also performed on the composite destination image, specify a composite location where the hairline direction matches between the composite source and composite destination You may In addition, when a defect included in the compositing source image is superimposed on the edge or printing of the inspection object, the compositing part of the partial image may be specified so as to be superimposed on the edge or printing also in the compositing destination image. The generation unit 122 synthesizes the partial image with the specified synthesis portion, and generates a synthesized image (S169). Thus, the combining process (S16) is completed. In addition, each generation method such as selection of whether or not to set a mask in the combination destination image, selection of whether or not to give priority to combination to a location where defects are likely to occur, and selection of a combination location of partial images are independent. Or can be used in combination.

  FIG. 8 is a diagram showing an outline of correction processing and combination processing executed by the data generation apparatus 100 according to the present embodiment. The data generation apparatus 100 acquires the compositing source image P1 and extracts the partial image P2 based on the designation from the user interface 170. In the case of this example, the composite original image P1 is an image of the inspection object that has been subjected to hairline processing and has a thin scratch on its surface. The partial image P2 is an image obtained by cutting out a scratched part.

  The correction unit 121 may correct the partial image P2 by a plurality of correction methods. In this example, the partial image P2 is corrected to the first corrected image P3 by the first correction, and is corrected to the second corrected image P4 by the second correction. The first correction and the second correction are corrections that are performed using different shape templates, as will be described using the following figures.

  The generation unit 122 generates the first composite image P11 by combining the first corrected image P3 with the first composite destination image P10, which is an image of the inspection object whose surface is textured. The generation unit 122 also generates the second composite image P21 by combining the second corrected image P4 with the second composite destination image P20, which is an image of the inspection object whose surface has been roughened and polished.

  As described above, according to the data generation apparatus 100 according to the present embodiment, a plurality of composite images such as the first composite image P11 and the second composite image P21 can be generated from one composite source image P1 including a defect. . Then, a plurality of composite images are generated so as to include defects that may actually occur. The data generation apparatus 100 can generate various composite images including defects that may actually occur by correcting the partial images by a plurality of correction methods and combining the partial images into a plurality of composite destination images.

  FIG. 9 is a diagram showing an outline of the correction process performed by the data generation apparatus 100 according to the present embodiment. The figure shows an example in which a plurality of composite images are generated by correcting the shape of one composite source image P1 so as to approach each of a plurality of shape templates.

  In this example, the shape of the compositing source image P1 is corrected so as to approach the first shape template to generate the first corrected image P3, and the shape of the compositing source image P1 is approached to the second shape template The case of correcting to generate the second corrected image P4, and the case of correcting the shape of the combining source image P1 so as to approach the third shape template to generate the third corrected image P5, and the shape of the combining source image P1 Is corrected to be closer to the fourth shape template to generate the fourth corrected image P6. As shown in the figure, the correction based on the first shape template is a correction that applies a non-linear transformation to the shape of the partial image, and the correction based on the first shape template is the thickness of the shape of the partial image A correction that changes and adds distortion, a correction based on the third shape template is a correction that duplicates and aligns the shapes of the partial images, and a correction based on the fourth shape template changes the shapes of the partial images It is a correction that applies a linear transformation to it.

  FIG. 10 is a diagram showing an outline of the mask process performed by the data generation apparatus 100 according to the present embodiment. The mask process is a process of restricting the synthesis position of the partial image in the synthesis destination image, and is a process of preventing the synthesis of the partial image in the area hidden by the mask.

  The image P30 before the mask in this example includes the defect D, and the edge of the inspection object is captured on the right side of the image. The black area on the right side of the image is an area where no inspection object exists. The data generation apparatus 100 receives the setting of the mask M through the user interface 170, and generates a third combined destination image P31. The area hidden by the black mask M in the third combining target image P31 is an area excluded from the combining part of the partial image. The area hidden by the black mask M covers the defect D and the area where no inspection object exists. By setting the mask M in this manner, it is possible to prevent the partial image from being superimposed on the defect D and the partial image to be combined in the area where the inspection object is not present, and the defect in a more realistic manner. Can be generated.

  FIG. 11 is a flowchart showing the contents of the second process executed by the data generation apparatus 100 according to the present embodiment. The second process is a process of generating new learning data by the generation unit 122 regardless of whether or not the determination result by the determination unit 152 is not correct by the user interface 170. The data generation apparatus 100 first acquires an image of an inspection object (S20). Here, the image may be a sample image stored in advance in the image DB 112, or may be a measurement image newly captured by the imaging device 153.

  Next, the user interface 170 receives specification of a partial image including a defect in the image (S21). Thereafter, the correction unit 121 of the data generation apparatus 100 corrects the designated partial image (S22). The details of the correction process may be the same as the contents shown in FIG.

  The generation unit 122 of the data generation device 100 generates a composite image obtained by combining the partial image corrected by the correction unit 121 and an image different from the image including the partial image (S23). The details of the combining process may be the same as the contents shown in FIG.

  The verification unit 124 of the data generation apparatus 100 causes the determination unit 152 to determine whether the generated composite image includes a defect (S24). As a result of the determination, it is verified whether a determination result that no defect is included is obtained for defects included in the combined partial image, similar to the case where the determination unit 152 determines the image including the partial image S25). Note that the verification unit 124 may verify the determination result by the determination unit 125 for the area of the combined image in which the partial image is combined, and the determination unit related to the area other than the area of the combined image in which the partial image is combined. Verification of the determination result by 125 may not necessarily be performed. When the determination result similar to the image of the inspection object (composition source image) which is the composition source is not obtained (S25: NO), the data generation apparatus 100 performs the partial image correction process (S22) and the composition process (S23) Is again performed by a different method, and the determination unit 152 determines whether or not a defect is included in the generated composite image (S24), and verifies whether a determination result similar to the composite source image can be obtained. When a determination result similar to that of the original image is obtained (S25: YES), the generation unit 122 associates the information indicating that a defect is included with the composite image, and performs new learning for learning the classifier. Data is generated and stored in the corrected image DB 123 (S26). Thus, the second process ends.

  The embodiments described above are for the purpose of facilitating the understanding of the present invention, and are not for the purpose of limiting the present invention. The elements included in the embodiment and the arrangement, the material, the conditions, the shape, the size, and the like of the elements are not limited to those illustrated, and can be changed as appropriate. In addition, configurations shown in different embodiments can be partially substituted or combined with each other.

[Supplementary Note 1]
A data generation apparatus (100) for generating learning data for learning classifiers (a, b, c, A, B) for identifying the presence or absence of a detection target part included in an inspection object,
An acquisition unit (153) for acquiring an image of the inspection object;
An input unit (170, 180) which is a part of the image and receives specification of a partial image including a detection target site;
A correction unit (121) for correcting the partial image based on the feature amount of the detection target portion;
A composite image is generated by combining the partial image corrected by the correction unit (121) and an image different from the image including the partial image, and information indicating that the detection target portion is included is A generation unit (122) that generates new learning data for causing the classifiers (a, b, c, A, B) to be associated with the composite image;
A data generator (100) comprising:

[Supplementary Note 2]
The input unit (170, 180) receives specification of at least one of a correction method by the correction unit (121) and a generation method of the composite image by the generation unit (122).
Appendix A. The data generation apparatus (100) according to appendix 1.

[Supplementary Note 3]
It is determined by the classifiers (a, b, c, A, B) learned using data for learning whether the image acquired by the acquisition unit (153) includes a detection target part. Further comprising a determination unit (152);
The input unit (170, 180) receives an input regarding whether the determination result by the determination unit (152) is correct or not,
In the correction unit (121), the determination unit (152) determines that the detection target portion is not included in the image, and the input unit (170, 180) correctly determines the determination result by the determination unit (152). Correct the partial image if it is not input,
The data generation device (100) according to appendix 1 or 2.

[Supplementary Note 4]
The classifiers (a, b, c, A, and B) learn feature amounts of an image corresponding to a detection target portion by learning using learning data.
The correction unit (121) corrects the partial image so as to have a feature amount different from the feature amount learned by the classifiers (a, b, c, A, B).
Appendix 3. The data generator (100) according to appendix 3.

[Supplementary Note 5]
The determination unit (152) determines whether the composite image includes the detection target region, and the determination unit (152) determines whether the detection target region is included in the image including the partial image. And a verification unit (124) for verifying whether or not a determination result similar to that determined by
The generation unit (122) registers the composite image as new learning data when the verification unit (124) obtains a similar determination result.
Appendix 3. The data generator (100) according to appendix 3 or 4.

[Supplementary Note 6]
The generation unit (122) generates a plurality of combined images obtained by combining the partial image corrected by the correction unit (121) by a plurality of correction methods and an image different from the image including the partial image. And generating information for learning by associating information indicating that a detection target site is included with the plurality of composite images.
The data generation device according to any one of appendices 1 to 5.

[Supplementary Note 7]
The generation unit (122) combines the partial image corrected by the correction unit (121) into a plurality of images different from the image including the partial image to generate a plurality of composite images, and is a detection target. Information indicating that a part is included is associated with the plurality of composite images to generate a plurality of learning data.
The data generation device (100) according to any one of appendices 1 to 6.

[Supplementary Note 8]
Whether each of the plurality of learning data is applied to the learning of different classifiers (a, b, c, A, B), and the inspection object includes the detection target portion based on the input image A learning unit (130, 140) for learning a plurality of classifiers (a, b, c, A, B) for discriminating whether or not
Based on the discrimination performance of the plurality of learned classifiers (a, b, c, A, B) learned by the learning unit (130, 140), one or more learnings from the plurality of learning data And a selection unit (125) for selecting the
The data generation device (100) according to appendix 6 or 7.

[Supplementary Note 9]
The generation unit (122) selects a synthesis position of the partial image such that a background pattern around the detection target portion is similar to the image including the partial image and the synthetic image, and the synthetic image To generate
The data generation device (100) according to any one of appendices 1 to 8.

[Supplementary Note 10]
The correction unit (121) corrects the brightness of the partial image.
The data generation device (100) according to any one of appendices 1-9.

[Supplementary Note 11]
The correction unit (121) corrects the shape of the partial image.
The data generating device (100) according to any one of appendices 3 to 10.

[Supplementary Note 12]
The correction unit (121) corrects the shape of the partial image such that the shape of the partial image approaches any of a plurality of shape templates.
The data generation device according to appendix 11.

[Supplementary Note 13]
The generation unit (122) generates a plurality of combined images by respectively combining the partial images corrected using the plurality of shape templates into an image different from the image including the partial image;
The determination unit (152) determines whether the plurality of composite images include a detection target portion,
The correction unit (121) selects one of the plurality of shape templates based on the reliability of the determination by the determination unit (152), and corrects the shape of the partial image.
The data generation device according to appendix 12.

[Supplementary Note 14]
The generation unit (122) generates the combined image such that the partial image is combined with an image of a portion where the detection target site is likely to be generated statistically among the inspection objects.
The data generation device (100) according to any one of appendices 1 to 13.

[Supplementary Note 15]
The detection target site is a site of a defect included in the inspection target,
The feature amount includes at least one of a shape of the defect, a brightness of the image, a change in brightness of the image along the defect, and a contrast between the defect and the periphery of the defect.
The data generating device (100) according to any one of appendices 1-14.

[Supplementary Note 16]
A data generation method for generating learning data for causing a classifier (a, b, c, A, B) for identifying the presence or absence of a detection target portion included in an inspection object to be learned,
Obtaining an image of an inspection object;
Accepting designation of a partial image that is part of the image and includes a detection target site;
Correcting the partial image based on the feature amount of the detection target site;
A composite image is generated by combining the corrected partial image and an image different from the image including the partial image, and information indicating that a detection target portion is included is associated with the composite image, Generating new learning data for causing the classifiers (a, b, c, A, B) to learn;
Data generation method including:

[Supplementary Note 17]
Arithmetic unit provided in data generation apparatus (100) for generating learning data for learning discriminators (a, b, c, A, B) for identifying presence / absence of detection target part included in inspection object ,
An acquisition unit (153) for acquiring an image of the inspection object;
An input unit (170, 180) that receives specification of a partial image that is a part of the image and includes a detection target site;
A correction unit (121) that corrects the partial image based on the feature amount of the detection target portion, and the partial image corrected by the correction unit (121) and an image different from the image including the partial image And a new image for causing the classifiers (a, b, c, A, B) to learn by associating information indicating that the detection target portion is included with the composite image. A generation unit (122) that generates learning data;
Data generator to operate as.

  100: data generation device, 101: arithmetic device, 102: auxiliary storage unit, 102a: data generation program, 102b: measurement image, 102c: learning data, 103: auxiliary storage unit, 104: processor, 105: main storage unit, 106: external I / F, 110: image collection unit, 111: attribute assignment unit, 112: image DB, 113: image correction unit, 120: image correction unit, 121: correction unit, 122: generation unit, 123: corrected image DB 124 124 verification unit 125 selection unit 130 first learning unit 131 first learning processing unit 140 second learning unit 141 second learning processing unit 150 measuring unit 151 learning Result DB, 152: determination unit, 153: imaging device, 160: measurement result management unit, 161: measurement result DB, 170: user interface, 180: user interface, 91: output device, 192: input device, D: defect, M: mask, P1: composite original image, P2: partial image, P3: first correction image, P4: second correction image, P5: third correction image, P6: fourth correction image, P10: first composite destination image, P11: first composite image, P20: second composite destination image, P21: second composite image, P30: image before masking, P31: third composite destination image

Claims (17)

A data generation apparatus that generates learning data for causing a classifier to identify the presence or absence of a detection target portion included in an inspection object.
An acquisition unit that acquires an image of the inspection object;
An input unit which is a part of the image and receives specification of a partial image including a detection target site;
A correction unit configured to correct the partial image based on the feature amount of the detection target portion;
A composite image is generated by combining the partial image corrected by the correction unit and an image different from the image including the partial image, and information indicating that a detection target portion is included is included in the composite image. A generation unit that generates new learning data for causing the classifier to be associated in association with each other;
A data generator comprising: The input unit receives specification of at least one of a correction method by the correction unit and a generation method of the composite image by the generation unit.
The data generator according to claim 1. The image processing apparatus further includes a determination unit that determines whether the image acquired by the acquisition unit includes a detection target part by the classifier learned using the data for learning.
The input unit receives an input regarding whether or not the determination result by the determination unit is correct;
The correction unit corrects the partial image when the determination unit determines that the detection target portion is not included in the image and the input unit determines that the determination result by the determination unit is not correct. ,
The data generation device according to claim 1. The classifier learns feature amounts of an image corresponding to a detection target portion by learning using learning data.
The correction unit corrects the partial image so as to have a feature amount different from the feature amount learned by the discriminator.
The data generation device according to claim 3. The case where the determination unit determines whether or not the detection target portion is included in the composite image, and the determination unit determines whether the detection target portion is included in the image including the partial image And a verification unit that verifies whether similar determination results can be obtained,
The generation unit registers the composite image as new learning data when the verification unit obtains a similar determination result.
The data generator according to claim 3 or 4. The generation unit generates a plurality of composite images by combining the partial image corrected by the correction unit according to a plurality of correction methods and an image different from the image including the partial image, and includes a detection target portion. Information indicating that the information is displayed is associated with the plurality of composite images to generate a plurality of learning data,
The data generator according to any one of claims 1 to 5. The generation unit generates a plurality of composite images by combining the partial image corrected by the correction unit into a plurality of images different from the image including the partial image, and includes a detection target portion. Information indicating that information is associated with the plurality of composite images to generate a plurality of learning data,
The data generation apparatus according to any one of claims 1 to 6. Each of the plurality of learning data is applied to the learning of a different classifier, and a plurality of classifiers for identifying whether or not the detection target portion is included in the inspection object based on the input image are learned And a learning unit
And a selection unit that selects one or more learning data from the plurality of learning data based on the discrimination performance of the plurality of already-trained classifiers learned by the learning unit.
The data generation device according to claim 6 or 7. The generation unit selects a combination position of the partial image such that a background pattern around the detection target portion is similar between the image including the partial image and the composite image, and generates the composite image. ,
The data generation apparatus according to any one of claims 1 to 8. The correction unit corrects the brightness of the partial image.
The data generation device according to any one of claims 1 to 9. The correction unit corrects the shape of the partial image.
The data generation device according to any one of claims 3 to 10. The correction unit corrects the shape of the partial image so that the shape of the partial image approaches any of a plurality of shape templates.
The data generation device according to claim 11. The generation unit generates a plurality of composite images by respectively combining the partial images corrected using the plurality of shape templates into an image different from the image including the partial image,
The determination unit determines whether or not a detection target part is included in the plurality of composite images.
The correction unit selects one of the plurality of shape templates based on the reliability of the determination by the determination unit, and corrects the shape of the partial image.
The data generation device according to claim 12. The generation unit generates the combined image such that the partial image is combined with an image of a portion where the detection target site is likely to be generated statistically among the inspection objects.
The data generation device according to any one of claims 1 to 13. The detection target site is a site of a defect included in the inspection target,
The feature amount includes at least one of a shape of the defect, a brightness of the image, a change in brightness of the image along the defect, and a contrast between the defect and the periphery of the defect.
The data generation device according to any one of claims 1 to 14. A data generation method for generating learning data for learning a discriminator that identifies the presence or absence of a detection target portion included in an inspection object, comprising:
Obtaining an image of an inspection object;
Accepting designation of a partial image that is part of the image and includes a detection target site;
Correcting the partial image based on the feature amount of the detection target site;
A composite image is generated by combining the corrected partial image and an image different from the image including the partial image, and information indicating that a detection target portion is included is associated with the composite image, Generating new training data for training the classifier;
Data generation method including: An arithmetic device provided in a data generation device for generating learning data for causing a classifier to identify presence / absence of a detection target portion included in an inspection object;
Acquisition unit for acquiring an image of an inspection object
An input unit that receives a specification of a partial image that is a part of the image and includes a detection target site;
A correction unit that corrects the partial image based on the feature amount of the detection target region, and a composite image obtained by combining the partial image corrected by the correction unit and an image different from the image including the partial image A generation unit for generating new learning data for causing the discriminator to learn by generating information, associating information indicating that the detection target portion is included with the composite image, and
Data generator to operate as.