KR20210003661A - Systems and methods for detecting flaws on panels using images of the panels - Google Patents

Abstract

One embodiment of the present disclosure relates to a computer program stored in a computer readable medium, wherein when the computer program is executed by one or more processors of a computing device, the computer program performs operations to provide the method for detecting defects, and the operations may include: extracting a defect patch from a defect image including a defect; preprocessing at least one of the defect image or non-defect image not including a defect; extracting a non-defect patch from at least one of the preprocessed defect image or non-defect image; and training a neural network model for classifying a defect or a non-defect of a patch using a training data set including the defect patch and the non-defect patch.

Description

Method and system for detecting panel defects using the image of the panel {SYSTEMS AND METHODS FOR DETECTING FLAWS ON PANELS USING IMAGES OF THE PANELS}

The present invention relates to the field of artificial intelligence technology, and more specifically, to defect detection using artificial intelligence technology.

Currently, in most manufacturing industries, the human eye is inspecting products for defects. For example, in a process for manufacturing a vehicle, a person can determine whether a panel obtained in the middle of the process (e.g., a panel made of steel, aluminum, plastic, etc.) or panels obtained after completion of the process is defective. Can be visually inspected. If there is a crack or necking on the panel, it can be determined that the panel is defective.

However, the conventional method of inspecting the products obtained in the process by a human eye is time and cost consuming. Moreover, the accuracy of such a conventional inspection method for a product may vary depending on a person's skill level, and a criterion for determining whether a defect may be changed according to the individual subjectivity of the inspector, and it may be difficult to keep an accurate record.

Accordingly, there is a need in the manufacturing industry for a convenient and highly accurate method for inspecting defects in a product.

Korean Patent Publication No. 2016-0012537 discloses a neural network learning method and apparatus, and a data processing apparatus.

The present disclosure has been devised in response to the above-described background technology, and an object thereof is to provide a defect detection method.

A computer program stored in a computer-readable storage medium according to an embodiment of the present disclosure for realizing the above-described problems, wherein the computer program is an operation for providing a method for detecting a defect when executed in one or more processors of a computing device. The operations include: extracting a defective patch from a defect image containing a defect; Pre-processing at least one of the defective image or a normal image containing no defects; Extracting a normal patch from at least one of the preprocessed defective image or normal image; And training a neural network model for classifying defects or normals of a patch by using a training data set including the defective patch and the normal patch.

In an alternative embodiment of computer program operations for performing the following operations for defect detection, the defect image may be an image labeled with one or more defect areas in advance.

In an alternative embodiment of computer program operations to perform the following operations for defect detection, the defective patch is a partial area of an image of a predetermined size extracted from the defective image, and the normal patch is the defective image or It may be a partial area of an image of a predetermined size extracted from at least one of the normal images.

In an alternative embodiment of computer program operations for performing the following operations for defect detection, the defective patch includes at least one defective area included in the defective image, and the normal patch includes in the defective image It may not include the one or more defective areas.

In an alternative embodiment of computer program operations for performing the following operations for defect detection, the operation of extracting a defective patch from the defect image containing the defect comprises: so that at least one defect area is included in the center of the patch, Extracting the defective patch; Extracting the defective patch so that the at least one defective area is included in a boundary of the patch; Extracting the defective patch so that the defective area having a predetermined number of areas or more is included in the patch; Or extracting the defective patch so that the defect regions arranged in parallel with a predetermined continuous number or more are included in the patch; It may include at least one of the operations.

In an alternative embodiment of computer program operations that cause the following operations to be performed for defect detection, preprocessing the defective patch; And the training data set may include the preprocessed defect patch.

In an alternative embodiment of computer program operations for performing the following operations for defect detection, the preprocessing of at least one of the defective image or a normal image without defects comprises: at least one of the defective image or the normal image One, an operation of removing noise included in an image to be preprocessed; Removing features of less than a predetermined size included in the pre-processing target image; Removing a gap between the two or more features when the distance between the two or more features included in the pre-processing target image is less than a predetermined length; Alternatively, at least one of an operation of smoothing a boundary of a feature included in the preprocessing target image may be included.

In an alternative embodiment of computer program operations for performing the following operations for defect detection, the preprocessing of at least one of the defective image or a normal image without defects comprises: at least one of the defective image or the normal image One, the operation of converting an image to be preprocessed into a binary image; Obtaining an inverted image by inverse of a binary value included in the binary image; Opening the inverse image to obtain an opened image; And obtaining a preprocessed image including a difference value between the inverse image and the opened image.

In an alternative embodiment of computer program operations for performing the following operations for defect detection, the operation of opening the inverse image to obtain an opened image comprises: a feature less than a predetermined size included in the inverse image. It may include an operation of removing.

In an alternative embodiment of computer program operations for performing the following operations for defect detection, the preprocessing of at least one of the defective image or a normal image without defects comprises: at least one of the defective image or the normal image One, an operation of converting an image to be preprocessed into a binary image; An operation of obtaining a closed image by closing the binary image; Obtaining a differential image including a difference value between the binary image and the closed image; And obtaining a preprocessed image by inverting a binary value included in the differential image.

In an alternative embodiment of computer program operations for performing the following operations for defect detection, the operation of closing the binary image to obtain a closed image includes: a distance between two or more features included in the binary image If less than a predetermined length, removing the gap between the two or more features; Or smoothing a boundary of a feature included in the binary image; It may include at least one of the operations.

In an alternative embodiment of computer program operations for performing the following operations for defect detection, the ratio of the number of defective patches and the number of normal patches included in the training data set may be determined by a predetermined balance ratio.

In an alternative embodiment of computer program operations for performing the following operations for defect detection, a neural network model for classifying defects or normals of a patch using a training data set including the defective patch and the normal patch is constructed. The training operation may include training different neural network models for each patch size.

In an alternative embodiment of computer program operations that cause the following operations to be performed for defect detection, the method includes: extracting one or more patches from an image to be classified; Calculating each of the one or more patches using the learned neural network model and classifying them as defective or normal patches; And when a predetermined number or more of the one or more patches are classified as defective patches, determining that a defect is included in the classification target image.

In an alternative embodiment of computer program operations for performing the following operations for defect detection, the defect image and the normal image include: an image photographing a product after completion of a manufacturing process; Alternatively, it may be at least one of images taken of the product during the manufacturing process.

In an alternative embodiment of computer program operations that cause the following operations to be performed for defect detection, the learned neural network model calculates an image acquired during a manufacturing process; If it is determined that the image acquired during the manufacturing process contains a defect, extracting an additional defect patch based on the image acquired during the manufacturing process; And adding the additional defective patch to the training data set.

A defect detection method according to an embodiment of the present disclosure for realizing the above-described problem, the method comprising: extracting a defect patch from a defect image including the defect; Pre-processing at least one of the defective image or a normal image containing no defects; Extracting a normal patch from at least one of the preprocessed defect image or normal image; And training a neural network model for classifying defects or normals of a patch by using a training data set including the defective patch and the normal patch.

A defect detection server according to an embodiment of the present disclosure for realizing the above-described problem, the server comprising: a processor including one or more cores; And memory; Including, the processor, extracting a defective patch from the defective image containing the defect, pre-processing at least one of the defective image or the normal image not containing the defect, and at least one of the pre-processed defective image or the normal image A neural network model for classifying defects or normals of the patch may be trained by extracting a normal patch from and using a training data set including the defective patch and the normal patch.

As a computer-readable recording medium storing a data structure corresponding to a parameter of a neural network that is at least partially updated in a learning process according to an embodiment of the present disclosure for realizing the above-described task, the operation of the neural network is Based at least in part on the parameter, the learning process comprises: extracting a defective patch from a defect image containing the defect; Pre-processing at least one of the defective image or a normal image containing no defects; Extracting a normal patch from at least one of the preprocessed defect image or normal image; And training a neural network model for classifying defects or normals of a patch by using a training data set including the defective patch and the normal patch.

According to an embodiment of the present disclosure, a method for detecting a defect may be provided.

1 is a diagram illustrating a block diagram of a computing device that performs an operation for providing a defect detection method according to an embodiment of the present disclosure.
2 is a flowchart illustrating a method of generating a model for defect detection according to an embodiment of the present disclosure.
3 is a flowchart illustrating a method of calculating a model for defect detection according to an embodiment of the present disclosure.
4 is a diagram illustrating a method of extracting a patch from an image for defect detection according to an embodiment of the present disclosure.
5 is a flowchart illustrating a preprocessing method for defect detection according to an embodiment of the present disclosure.
6 is a diagram illustrating a preprocessing method for defect detection according to an exemplary embodiment of the present disclosure.
7 is a diagram illustrating an image conversion method according to an embodiment of the present disclosure.
8 is a diagram illustrating a method of learning one or more models for defect detection according to an embodiment of the present disclosure.
9 is a diagram illustrating a method of detecting a defect using one or more models according to an embodiment of the present disclosure.
10 is a flowchart illustrating a method of generating additional learning data according to an embodiment of the present disclosure.
11 is a block diagram of a computing device according to an embodiment of the present disclosure.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is clear that these embodiments may be implemented without this specific description.

The terms "component", "module", "system" and the like as used herein refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or execution of software. For example, a component may be, but is not limited to, a process executed on a processor, a processor, an object, an execution thread, a program, and/or a computer. For example, both an application running on a computing device and a computing device may be components. One or more components may reside within a processor and/or thread of execution. A component can be localized on a single computer. A component can be distributed between two or more computers. In addition, these components can execute from a variety of computer readable media having various data structures stored therein. Components can be, for example, via a signal with one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and/or a signal through another system and a network such as the Internet. Depending on the data being transmitted), it may communicate via local and/or remote processes.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise or is not clear from the context, "X employs A or B" is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or, when X uses both A and B, “X uses A or B” can be applied to either of these cases. In addition, the term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the listed related items.

In addition, the terms "comprising" and/or "comprising" are to be understood as meaning that the corresponding features and/or components are present. However, it is to be understood that the terms "comprising" and/or "comprising" do not exclude the presence or addition of one or more other features, elements, and/or groups thereof. In addition, unless otherwise specified or when the context is not clear as indicating a singular form, the singular in the specification and claims should be interpreted as meaning "one or more" in general.

Those of skill in the art would further describe the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein, including electronic hardware, computer software, or a combination of both. It should be recognized that it can be implemented as To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or as software depends on the specific application and design limitations imposed on the overall system. Skilled technicians can implement the described functionality in various ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

A description of the presented embodiments is provided so that a person of ordinary skill in the art of the present disclosure can use or implement the present invention. Various modifications to these embodiments will be apparent to those of ordinary skill in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention is to be accorded the widest scope consistent with the principles and novel features presented herein.

In an embodiment of the present disclosure, the server may include any type of computing device. The server is a digital device, and may be a digital device equipped with a processor, such as a laptop computer, a notebook computer, a desktop computer, a web pad, and a mobile phone, and equipped with a computing power having a memory. The server may be a web server that processes services. The types of servers described above are only examples, and the present disclosure is not limited thereto.

1 is a diagram illustrating a block diagram of a computing device that performs an operation for providing a defect detection method according to an embodiment of the present disclosure.

With the development of the manufacturing industry, defects in products detected in the manufacturing process have significantly decreased. As a result, defective images (ie, defective images) are more difficult to obtain than images without defects (ie, normal images). In a general training data set, the number of normal data and abnormal data is proportional to maintain a balance. However, in the case of the manufacturing industry, it is difficult to acquire defect images in recent years, and there are cases in which the number of normal images is greater than that of defect images in the training data set. When a neural network is trained with a training data set including an unbalanced number of normal and abnormal data without refining images obtained in the manufacturing industry, the learning ability may be distorted. Moreover, even in the case of a defective image with a defect, many parts of the image may be parts that are not related to the defect, and only a small part may be the part related to the defect. That is, since the size of the defect is small, a pixel corresponding to the defect in the defect image may be a very small part. Thus, a data set (eg, an image set) collected in a manufacturing environment may have different properties when compared to a data set generally used for neural network training. Training of a deep learning algorithm/neural network model may become more difficult due to an imbalance problem between classified images (eg, classification into a defective image and a normal image).

Building good quality training data sets has to do with manufacturing productivity. To build a good quality training data set, it is necessary to collect data while the manufacturing environment is controlled, but this can affect the manufacturing process. Manufacturers may not want to increase or delay the time spent on the process. Manufacturers can avoid collecting data throughout the process while the manufacturing environment is controlled due to issues such as process time. Manufacturers may prefer to collect or capture data to create a data set after the process is complete, with the manufacturing environment uncontrolled. However, when a training data set is generated based on data (or images) collected in an uncontrolled environment, the quality of the data set may be greatly affected.

A description of the method for detecting defects of the present disclosure is also discussed in US Provisional Patent Application No. 62/869,919, which is incorporated herein by reference in its entirety.

2 is a flowchart illustrating a method of generating a model for defect detection according to an embodiment of the present disclosure. The flowchart of FIG. 2 may show a process of training a neural network model to detect defects.

The processor 120 may acquire 210 images. Images may include a normal image 230 and a defective image 220. The processor 120 may acquire a normal image and a defect image, which are manufacturing images. The processor 120 may classify the acquired images into a normal image 230 and a defective image 220, respectively. Each of the normal image and the defective image may be an image obtained in a manufacturing process. Each of the normal image and the defective image may be an image photographing a product after completion of the manufacturing process. Alternatively, each of the normal image and the defective image may be an image photographing a product during a manufacturing process.

The defect image may be an image including a defect. At least a portion of the area of the defect image may be an area related to the defect. At least some of the pixels in the defect image may be pixels associated with the defect. The defect image may be an image of a defective product.

The defect image may have one or more defect areas labeled in advance. The defect area may be an area of at least a portion of the image that is associated with the defect. For example, the defective area may be a pixel related to a defect and a set of one or more pixels related to a defect. For example, the defect area may be a pixel related to the defect in an image photographing a defective product. That is, the region may mean a pixel unit or a set of one or more pixels. Labeling of the defective area may indicate the defective area. The labeling of the defective area may be, for example, masking (eg, highlighted or selected) of a partial area included in the image. Each of the pixels included in the defective image may be labeled as defective or normal. Labeling may include information on which of the pixels included in the image is related to a defect. Labeling may include information about which of the pixels included in the image is not related to the defect. The detailed description of the above-described defect area is only an example, and the present disclosure is not limited thereto.

The normal image may be an image containing no defects. The normal image may be an image of a product without defects.

Defects can include any abnormal conditions that may occur in the manufacturing environment. Defects can include any abnormal condition that may occur in the product. For example, defects can include cracks, necking, and the like. Crack may mean a phenomenon in which the surface of the product is cracked. Necking may be a phenomenon that appears in non-uniform products. The necking may mean that when a part of the product is stretched and the part is not stretched, a constricted part is formed at the boundary between the stretched and non-stretched parts. Cracks may be partially detectable when a person inspects the product with the naked eye. However, it is impossible to detect all cracks from a product or an image photographed by a human eye, and minute cracks are often not detected by the human eye. Necking has a very low detection probability when a person inspects a product or an image of a product with the naked eye. Therefore, there is a need in the art for a method capable of accurately detecting even defects that are difficult to detect by humans. The detailed description of the above-described defect is only an example, and the present disclosure is not limited thereto.

The article of manufacture may be obtained during the manufacturing process or may be obtained after completion of the manufacturing process. For example, in a process for manufacturing a vehicle, the article of manufacture may be a material for producing the vehicle, a part, or a finished vehicle. For example, the product may be a material that is a panel composed of steel, aluminum, or the like. The specific description of the above-described product is merely an example, and the present disclosure is not limited thereto.

The processor 120 may generate/extract a patch from the normal image and the defective image. The processor 120 may extract 214 a defective patch from the defective image. The processor 120 may extract a normal patch from at least one of a normal image or a defective image.

The patch may be at least a portion of the image extracted from the image. A patch may be a set of one or more pixels extracted from an image. The processor 120 may extract a patch from the image in a predetermined size. One or more patches included in the same training data set may have the same size. One or more patches for training the same neural network model may have the same size.

The defect patch may be a patch extracted from the defect image. The defect patch may be a patch of a predetermined size extracted from the defect image. The defect patch may include at least one defect area included in the defect image. The defect image may include one or more defect areas. For example, a defective image may include one or more defective pixels. The defective patch may include at least one defective pixel. The defective patch may include a defective pixel and pixels around the defective pixel.

The processor 120 may extract the defective patch so that at least one defective area included in the defective image is included in the center of the patch. The processor 120 may extract the defective patch so that the defective area is located at the center pixel of the patch. The processor 120 may extract the defective patch so that some of the pixels included in the defective area are located within a predetermined distance from the center pixel of the patch. The processor 120 may extract the defective patch so that the pixel related to the defect included in the defective image becomes the center pixel of the patch. The processor 120 may extract the defective patch such that at least one defective pixel, or a set of one or more defective pixels, among one or more defective pixels labeled on the defective image is located at the center of the defective patch. The processor 120 may extract the defective patch so that the center of the defective area included in the defective image is located at the center of the defective patch. For example, the center of the defective area may be a center of gravity, an inner center of the defective area, etc., but the present disclosure is not limited thereto. In the present disclosure, the center may mean a pixel area within a predetermined distance from a pixel serving as the center of a certain area.

The processor 120 may extract the defective patch so that at least one defective area included in the defective image is included in the boundary of the patch. The processor 120 may extract the defective patch so that at least one defective pixel among one or more defective pixels included in the defective image is included in the boundary of the patch. The processor 120 may extract the defective patch such that at least one defective pixel or a set of one or more defective pixels among the one or more defective pixels labeled on the defective image is located at the boundary of the defective patch. For example, the processor 120 may allow the center or boundary of the defective area to be included in the boundary of the patch. In the present disclosure, the boundary may mean a pixel area within a predetermined distance from the edge of the image.

The processor 120 may extract the defective patch so that the defective area having a predetermined number of areas or more included in the defective image is included in the patch. The processor 120 may extract the defective patch so that a predetermined number or more of defective pixels among one or more defective pixels included in the defective image are included in the patch. For example, the processor 120 may extract the defective patch so that pixels related to five or more defects are included in the patch. The detailed description of the above-described defective patch is merely an example, and the present disclosure is not limited thereto.

The processor 120 may extract the defective patch so that the defect regions arranged in parallel with a predetermined continuous number or more included in the defect image are included in the patch. The processor 120 may extract the defective patch so that the defective pixels that are consecutively arranged in parallel with a predetermined number or more included in the defective image are included in the patch. The processor 120 may extract the defective patch so that a set of three or more pixels that are sequentially arranged in parallel among one or more pixels included in the defective image is included in the patch. The detailed description of the above-described defective patch is merely an example, and the present disclosure is not limited thereto.

The processor 120 may preprocess 240 each of the defective image and the normal image. The processor 120 may pre-process the defective image after extracting the defective patch. The processor 120 may extract the defective patch from the defect image and pre-process the remaining areas. The pretreatment method will be described later with reference to FIGS. 5 to 7. Alternatively, the processor 120 may perform pre-processing even on the defective patch.

The processor 120 may extract 250 a normal patch from at least one of a preprocessed defect image or a normal image. The processor 120 may extract the defective patch and extract the normal patch from the image obtained by preprocessing the remaining areas. The processor 120 may extract the normal patch after preprocessing the normal image. A normal patch may not include one or more defective regions included in the defective image. For example, the normal patch may be a region that does not include defective pixels included in the defective image. The normal patch may be a patch extracted from the normal image. The normal patch may include normal pixels that are not defective pixels. Alternatively, the normal patch may include a normal pixel at the center, and may include a normal pixel at the center and pixels around it. Specific description of the above-described patch is only an example, and the present disclosure is not limited thereto.

Each of the normal image and the defective image can be separated into one or more patches.

The processor 120 may configure 260 a training data set using at least some of the one or more patches extracted from the images. The processor 120 may configure the training data set to include the defect patch extracted from the defect image. The processor 120 may configure the training data set to include the preprocessed defective patch. The preprocessing method for the defective patch may be the same as the preprocessing method for the image. The processor 120 may configure the training data set to include a normal patch extracted from at least one of a defective image or a normal image.

The training data set may be a set of data used for training a neural network model. The training data set may include one or more training data. The training data may include an image that is an input of a neural network and a result of classification of the image. Alternatively, the training data may include a patch that is an input of a neural network and a result of classification of the patch. A neural network model trained with a training data set can classify images. The neural network model trained with the training data set can classify the image as a normal image or a defective image. The training data set may include defect images or defect patches so that the neural network can detect defects. The training data set may include normal images or normal patches so that the neural network can detect defects. The training data set may contain tens of thousands of images or patches, for example. The training data set may be, for example, a Modified National Institute of Standards and Technology database (MNIST) data set, a Canadian Institute For Advanced Research-10 (CIFAR-10) data set, or an ImageNet data set. The training data set may include a well-balanced number of training data for each class. Detailed description of the above-described training data set is merely an example, and the present disclosure is not limited thereto.

The training data set may include a balanced number of training data for each classification class to classify images. The ratio of the number of defective patches and the number of normal patches included in the training data set may be determined by a predetermined balance ratio. The predetermined balance ratio may be an optimized ratio for improving the classification performance of the neural network model. For example, the predetermined ratio may be 1 to 1 for each of a normal patch and an abnormal patch. For example, the training data set may each include the same number of normal patches and abnormal patches. Detailed description of the above-described training data set is merely an example, and the present disclosure is not limited thereto.

As described above, when the number of learning data corresponding to each of two or more classes is unbalanced, there may be a serious problem in learning efficiency. Instead of constructing the training data set using the normal image and the defective image, a plurality of defective patches can be obtained by constructing the training data set using the normal patch and the defective patch. That is, when a training data set is constructed using defect patches, more training data related to a defect class may be included than when a training data set is constructed using defect images. Therefore, when a training data set is constructed using patches instead of images, the imbalance problem between classes can be solved.

The processor 120 may train 270 a neural network model using the training data set.

The neural network model of the present disclosure may be configured as a teacher learning model. In an embodiment of the present disclosure configured as a teacher learning model, the teacher learning model may receive a normal patch or an abnormal patch as an input and output whether a normal patch or an abnormal patch is defective. The teacher learning model may be trained using a learning normal patch, a learning abnormal patch, and whether or not there is a defect labeled thereon.

The neural network model of the present disclosure may be configured as a comparative history learning model. The comparative history learning model may include a model capable of clustering input data. The comparative history learning model may be, for example, a model composed of a neural network that restores input data. The comparative history learning model may include, for example, an autoencoder. The non-satellite learning model may be trained using a training normal patch or an abnormal patch, and whether a defect matched with the training input data.

A description of the specific configuration of the neural network model of the present disclosure is discussed in more detail in US Patent No. US 9870768B2, which is incorporated by reference in its entirety in this application.

In order to determine whether the image is defective, a threshold number of defects may be determined 280. The threshold number of defects may be a threshold number of defect patches included in one image for determining the image as a defect image. In the inferencing step, when a patch having a defect threshold number or more among a plurality of patches included in one image is a defective patch, the processor 120 may determine the corresponding image as a defective image. For example, when the threshold number of defects is 5 and the number of defective patches is 5 or more among the plurality of patches separated from the image, the processor 120 may determine the corresponding image as the defect image. Alternatively, the processor 120 may determine the image as a normal image when a patch with less than the defect threshold number among the plurality of patches included in one image is a defective patch. The detailed description of the above-described defect threshold number is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the number of defect thresholds may be determined by a learned neural network model. The processor 120 may determine the number of defect thresholds based on the training data set and the learned neural network model. The processor 120 may calculate one or more defect images using each learned neural network model. The processor 120 may check the number of patches determined as defective patches for each of one or more defect images in the learned neural network model. The processor 120 may determine the defect threshold number based on the number of defective patches included in the defective images. The processor 120 may determine, for example, the minimum number of defective patches included in the defective image as the defect threshold number. For example, the processor 120 may determine 5, 7, 10, 6, and 13 defect patches included in the plurality of defect images by calculating using the learned neural network model. The processor 120 may determine an image including at least five defective patches as a defective image based on the result. The processor 120 may determine the number of defect thresholds as five. The detailed description of the above-described defect threshold number is only an example, and the present disclosure is not limited thereto.

Hereinafter, a method of inferring the learned neural network model will be described. 3 is a flowchart illustrating a method of calculating a model for defect detection according to an embodiment of the present disclosure.

The processor 120 may receive 810 an image to be classified. The image to be classified may be an image without information on normal or defect. The image to be classified may be an image requiring determination of normality or defect using a neural network model. The image to be classified may be an image obtained in connection with a manufacturing process. The classification target image may be an image photographing a product in connection with a manufacturing process. The image to be classified may be an image photographed after completion of the manufacturing process or during the manufacturing process.

The processor 120 may pre-process 820 the image to be classified. The pre-processing for the image to be classified in the inference process and the pre-processing for the image included in the training data can be performed in a similar manner.

The processor 120 may extract 830 one or more patches from the image to be classified. The processor 120 may extract patches of a predetermined size from the image to be classified.

The processor 120 may calculate 840 one or more patches using the learned neural network model. The processor 120 may classify each of the one or more patches as normal or defective using the learned neural network model. When the patch includes at least one defective area, the processor 120 may determine the patch as the defective patch. When the central region of the patch is the defective area, the processor 120 may determine the patch as the defective patch. When the boundary area of the patch is a defective area, the processor 120 may determine the patch as a defective patch. The processor 120 may determine the patch as a defective patch when a predetermined number or more of the areas included in the patch are defective areas. The processor 120 may determine the patch as the defective patch when the areas arranged in parallel with a predetermined number of consecutive or more included in the patch are defective areas.

The processor 120 may determine 850 whether or not the classification target image is defective using the normal or defect classification results of a plurality of patches extracted from one classification target image. The processor 120 may determine whether one classification target image is defective by synthesizing the normal or defective classification results of the plurality of patches. The processor 120 may determine the classification target image as a defect image including a defect when it is determined that patches having a threshold number of defects or more among the patches extracted from the classification target image are defective patches. Alternatively, the processor 120 may determine the classification target image as a normal image that does not contain any defects when it is determined that among the patches extracted from the classification target image, which have less than the defect threshold number, are defective patches. For example, if the threshold number of defects is 5 and the neural network model determines 6 of the patches extracted from the classification target image as defective patches, the classification target image may be determined as the defect image. Detailed description of the above-described reasoning method is only an example, and the present disclosure is not limited thereto.

4 is a diagram illustrating a method of extracting a patch from an image for defect detection according to an embodiment of the present disclosure. FIG. 4 exemplarily shows an original image 300, a partial region 310 for extracting a patch from the original image, and a patch 320 extracted from the original image. Patches can be of various sizes. According to an embodiment of the present disclosure, the size of the patch may be determined by a person. Alternatively, the size of the patch may be determined by an operation of the computing device. The size of the patch may be associated with a hyper parameter (or free parameter) of the neural network model. In general, if the size of the patch is too small, it may not be sufficient to contain information about the defect. In addition, if the size of the patch is too large, time, cost, or computing power may be consumed too much for training and inference of a neural network model. In addition, if the size of the patch is too large, the number of patches that can be extracted from one image is too small, and it may be difficult to obtain sufficient training data.

Hereinafter, a method of preprocessing an image or patch will be described with reference to FIGS. 5, 6 and 7. 5 to 7 are views exemplarily illustrating a preprocessing method for detecting a defect according to an embodiment of the present disclosure.

The pre-processing 240 can be composed of four steps. The preprocessing 240 according to an embodiment of the present disclosure may include a thresholding step, an inverse step, an opening step, and a differential operation step for an image.

The processor 120 may perform critical processing on the original image X242. The processor 120 may generate a binary image X'244 by thresholding the original image X 242. Thresholding on an image may be a method of binaryizing an image. The binary image may be an image obtained by classifying an image into two classes. For example, the binary image may be an image in which each pixel of the image is classified according to a certain criterion. For example, the binary image may be an image in which the image is displayed in black or white. For example, the binary image may be an image in which a pixel value included in the image is set to 0 or 1. The processor 120 may compare the pixel value of the image with a predetermined threshold value. The processor 120 may classify pixels having a pixel value equal to or greater than a predetermined threshold value into a first class. The processor 120 may classify pixels having a pixel value less than a predetermined threshold value into a second class. The first class and the second class may have a value of 0 or 1, or a color of black or white, for example. Thresholding for an image may include, for example, global fixed thresholding, locally adaptive thresholding, and hysteresis thresholding. For example, the processor 120 may generate a black and white binary image by performing thresholding on a grayscale image. For example, the original image X242 may be a grayscale image. The detailed description of the above-described image thresholding is only an example, and the present disclosure is not limited thereto.

Image X'244 may be a binary image. The image X'244 may be an image displayed with two values. For example, pixels included in the image X'244 may be displayed as black (indicated as 0 in 8-bit color representation) or white (indicated as 255 in 8-bit color representation). As described above, the image X'244 may also be expressed in two classes other than black and white.

The processor 120 may inverse the binary image X'244. The processor 120 may inverse the binary image X'244 to generate the inverted image X'' 246. The inverse of the binary image may be to reverse the binary value included in the binary image. Inverse to a binary image may be to reverse the classes of pixels of the binary image. For example, the processor 120 may convert a pixel value included in a binary image from 0 to 1, and convert from 1 to 0. For example, the processor 120 may convert a color of a pixel included in a binary image from white to black, and convert from black to white.

The processor 120 may open the inverse image X'' 246. The processor 120 may generate the opened image X''' 248 by opening the inverse image X'' 246. In order to describe the opening operation, first, erosion and dilation will be described.

Corrosion and expansion can be operations performed on the image. The corrosion and expansion calculation will be described below with reference to FIG. 7. Corrosion and expansion operations can be performed on binary images. Corrosion and expansion operations can be associated with binary images and kernels. The kernel can be the size of the matrix. The kernel can be square or circular, for example. The binary image and kernel may be expressed as a pixel matrix. The binary image may be, for example, an 800*600 pixel matrix, a 200*300 pixel matrix, or the like. The kernel may be, for example, a 5*5 pixel matrix, a 15*15 pixel matrix, or the like. The kernel can slide on a binary image, similar to a convolution operation to perform a corrosion or expansion operation. Referring to FIG. 7, a binary image 701 of an 8 * 8 pixel matrix and a kernel 710 of a 3 * 3 pixel matrix are shown. 7 is a diagram showing an embodiment of a corrosion operation. Corrosion may be an operation performed within the range of the kernel 710. The corrosion operation may be, for example, an operation of writing the pixel values of 9 cells as 1 to the image as they are if the pixel values of 9 cells of the kernel 710 are all 1s. For example, if at least one of the pixel values of 9 spaces of the kernel 710 is 0, the processor 120 may write all of the pixel values of 9 spaces as 0 in the image. Alternatively, for example, if at least one of the pixel values of 9 cells of the kernel 710 is 0, the center pixel value of the 9 cells may be recorded as 0 in the image. For example, since 6 of the pixel values of 9 columns of the kernel 710 of the first image 701 of FIG. 7 are included, the processor 120 records all 9 pixel values as 0 The image 702 can be created. Since the pixel values of 9 columns of the kernel 710 of the third image 703 are all 1, the processor 120 may generate the fourth image 704 by recording the pixel values of 9 columns as 1 as it is. The processor 120 may generate the sixth image 706 by displaying all pixel values of 9 spaces of the kernel 710 of the fifth image 705 as zeros. The processor 120 may output the final image 707 by repeating the corrosion operation on the image. The final image 707 is an image in which the range of 1 is decreased by repeating the corrosion operation on the first image 701. The expansion operation may be an operation opposite to the corrosion operation. The dilation operation may be, for example, an operation of writing all 9 pixel values to 1 in an image if at least one of the pixel values of 9 columns of the kernel 710 is 1. Alternatively, for example, if at least one of the pixel values of 9 columns of the kernel 710 is 1, it may be an operation of writing a center pixel value of 1 of the 9 pixels to the image. When the expansion operation is repeated for the first image 701, the range of 1 may increase. That is, the corrosion operation can increase the white area in the binary image. Corrosion operations can increase the size or thickness of a foreground object. The dilation operation can increase the black area in the binary image. The dilation operation can reduce the size or thickness of the foreground object. The dilation operation can increase the size of the background. The specific description of the above-described corrosion and expansion is only an example, and the present disclosure is not limited thereto.

The opening can be a combination of corrosion and expansion operations. The processor 120 may apply a corrosion operation to the image for an opening operation, and then apply an expansion operation thereafter. The opening may be an operation for removing noise included in the image. The opening may be an operation for removing noise included in the background of the image. The processor 120 may remove features of a small size by reducing the size of features of a small size included in the image through a corrosion operation. Thereafter, the processor 120 increases the size of the unremoved features included in the image through the expansion operation, so that the features can preserve the original size. Opening can be an operation to remove features of a small size and leave only features of a large size. That is, through the opening operation, the processor 120 may remove features less than or equal to a predetermined threshold size included in the image. Since the opening performs the expansion operation after the corrosion operation, the processor 120 may remove noise, small size, or narrow width features included in the image. The specific description of the above-described opening is only an example, and the present disclosure is not limited thereto.

Closing can be a combination of expansion and corrosion operations. Closing can be the opposite of opening. The processor 120 may apply an expansion operation to the image for a closing operation, and then apply a corrosion operation thereafter. Closing may be an operation for removing small holes included in the foreground. Closing may be an operation for removing small black dots included in an object. The processor 120 may increase the size of features through an expansion operation. The processor 120 may remove a narrow gap between features included in the image through the expansion operation. The processor 120 may smooth unbalanced boundaries of objects or features included in an image through a dilation operation. Thereafter, the processor 120 reduces the size of the features included in the image through a corrosion operation, so that the features can preserve the original size. That is, through the closing operation, the processor 120 may remove a narrow gap, in which a distance between two or more features is less than a predetermined length, or smooth a boundary of a feature. The detailed description of the above-described closing is only an example, and the present disclosure is not limited thereto.

Descriptions of the above-described corrosion, expansion, opening and closing operations are discussed in more detail in Erosion, dilation and related operators: Mariusz Jankowski, 2006, which is incorporated by reference in its entirety in this application.

The processor 120 may perform a difference calculation between the inverse image X" 246 and the opened image X"" 248. The processor 120 may perform a pixel-wise subtraction operation between the inverted image X" 246 and the opened image X"" 248. The processor 120 may generate a preprocessed image X'''' 249 by subtracting the opened image X''' 248 from the inverse image X'' 246 (i.e., X'' '' = X''-X'''). The processor 120 subtracts the pixel value of the opened image X''' (248) from the pixel value of the inverse image X'' (246), and the pixel value of the preprocessed image X'''' (249). Can be determined by value. The processor 120 calculates the difference value between the pixels included in the same location of the inverse image X'' 246 and the opened image X'' 248, respectively, of the preprocessed image X'' 249 It can be determined by the value of the pixel at the same location.

The processor 120 may extract the normal patch from the preprocessed image X'''' 249.

That is, the preprocessing operation may be as follows. X'= critical processing (X), X'' = inverse (X'), X''' = opening (X''), X'''' = X''-X'''

Through a pre-processing operation, non-trivial normal pixels included in the image may be displayed as first class values, and the remaining pixels may be displayed as second class values. For example, important normal pixels included in the image may be displayed in white, and the remaining pixels may be displayed in black. The processor 120 may extract a normal patch based on pixels indicated by a first class value in order to extract normal patches effective for learning. For example, the processor 120 may extract a normal patch to include white pixels included in the image. For example, if the background and defects (e.g. cracks) of the original image are displayed in black/dark color, and the product (e.g., steel plate panel) is displayed in white/light color, then pretreatment as above. Operations can be performed. For example, a normal patch containing more white pixels may improve the performance of neural network training. The processor 120 may extract, for example, a normal patch including pixels having a first class value or more than a predetermined ratio from the preprocessed image. That is, instead of configuring all the patches extracted from the image as training data, only normal patches having the first class value from the preprocessed image may be extracted and configured as training data. Specific description of the above-described patch is only an example, and the present disclosure is not limited thereto.

The preprocessing according to an embodiment of the present disclosure may include a threshold processing step, a closing step difference calculation step, and an inverse step for an image.

The processor 120 may generate a binary image X'by performing threshold processing on the original image X. The processor 120 may generate a closed image X'' by closing the binary image X'. The processor 120 may generate a differential image X'''(X''' = X'-X'') by performing a difference operation subtracting the closed image X'' from the binary image X'. The processor 120 may generate a preprocessed image X'''' by inverting a binary value included in the differential image X"". The processor 120 may extract the normal patch from the preprocessed image X''''.

6 is an image showing an example of an image preprocessing method. 6 is an image illustrating an image of each step illustrated in the flowchart of FIG. 5 by way of example. Through pre-processing, it is possible to extract a normal patch for constructing an effective training data set from the image.

According to an embodiment of the present disclosure, a normal patch may be randomly extracted from a normal image. However, when a normal patch is randomly extracted from a normal image, it may be difficult to select a good normal pixel due to the characteristics of the product (eg, panel) image. This is because, for example, the background of the product image displayed in black occupies most of the area of the product image, and the product displayed in white on the image by reflection of light occupies a small part of the product image. In addition, only a small portion of the product area included in the product image can contain meaningful information in the training of the neural network model. Thus, most of the randomly selected normal patches may be meaningless to learning. Further, training data sets including randomly selected normal patches may include a plurality of meaningless training data. As a result, the number of meaningful training data among training data included in the training data set may be much smaller than the number of meaningless training data. When extracting the normal patch from the preprocessed image as described above, a training data set including a balanced number of patches for each of the normal and defective classes may be configured. In addition, a neural network model trained with a training data set including a normal patch extracted from a preprocessed image can output effective performance. Accordingly, the pre-processing operation on the image can remove unnecessary data and improve the performance of neural network training.

In the process of inference, the image to be classified can be preprocessed. Through pre-processing, an opening operation on the image to be classified may be performed. Good candidate patches without noise can be extracted through an opening operation on an image to be classified. The black pixel included in the opened classification target image may be a pixel corresponding to a defect in the original classification target image. The white pixel included in the opened classification target image may be a pixel corresponding to a background in the original classification target image. Accordingly, when preprocessing is performed on an image to be classified, it is not necessary for the neural network model to perform calculations on all patches included in the image to be classified. In the preprocessed image to be classified, some patches including white pixels are selected to calculate whether or not the neural network model contains a defect, thereby reducing the inference time of the image to be classified.

8 is a diagram illustrating a method of learning one or more models for defect detection according to an embodiment of the present disclosure. The size of the defect area included in the defect image may be different. For example, the defective area may correspond to one defective pixel, or may correspond to a set of a plurality of defective pixels. Therefore, when a neural network model is trained with a training data set including only patches of the same size, it may be difficult to detect defects for images including defect regions of different sizes. For example, a neural network model trained with a training data set including a 19*19 pixel patch may have poor defect detection performance for an image including a defect region larger than 19*19 pixel size.

According to an embodiment of the present disclosure, the processor 120 may train a plurality of neural network models 400 by using a training data set including patches of a plurality of different sizes. The processor 120 may train different neural network models for each patch size. For example, the first neural network model 410 may be trained with patches having a size of 19 * 19 pixels. The K-th neural network model 420 may be trained with patches of n * n pixels.

According to an embodiment of the present disclosure, the patches may have a square (eg, m * n pixel size), a circular shape, or another shape or size. A different neural network model may be used according to each type or size of an image or patch.

9 is a diagram illustrating a method of detecting a defect using one or more models according to an embodiment of the present disclosure. The processor 120 may detect whether an image to be classified is defective using a plurality of neural network models 500. The processor 120 may extract patches of a plurality of sizes from one classification target image. The processor 120 may calculate patches of a plurality of sizes, respectively, using a model for each patch size. For example, when the size of the patch extracted from the classification target image is 19 * 19 pixels, the processor 120 may calculate the patch using the first neural network model 510. In addition, when the size of the patch extracted from the image to be classified is n * n pixels, the processor 120 may calculate the patch using the Kth neural network model 520. The processor 120 may determine whether the image to be classified is defective based on whether patches output from each of the plurality of neural network models 500 are defective. The processor 120 may determine an image to be classified as a defect image including a defect when patches having a defect threshold number or more among patches output from each of the plurality of neural network models 500 are output as defective patches. When determining whether an image is defective by using a plurality of models, it is possible to improve the accuracy and efficiency of defect detection, and/or convenience of selecting an image patch size.

10 is a flowchart illustrating a method of generating additional learning data according to an embodiment of the present disclosure. The processor 120 may add the image acquired during the manufacturing process or patches extracted from the image to the training data set. For example, the normal image and the defect image included in the training data set may be images of the product obtained after the manufacturing process is completed. The processor 120 may calculate an image of the product obtained during the manufacturing process using the learned neural network model. As a result of the calculation using the neural network model, if the image acquired during the manufacturing process contains a defect, the processor 120 may determine to add the image acquired during the manufacturing process to the training data set. Even before the completion of the manufacturing process, that is, when it is determined that the image acquired during the manufacturing process is defective, the processor 120 may include the corresponding image in the training data set. The processor 120 may include the image acquired during the manufacturing process or patches extracted from the image in the training data set. The image acquired during the manufacturing process may be an image captured 610 by the device in the middle of the process. For example, the image acquired during the manufacturing process may be an image captured by a handheld device such as a smart phone. The processor 120 may adjust the size of the image acquired during the manufacturing process to the same size as the normal image or the defective image of the training data set (620). The processor 120 may label 630 the defective area of the image acquired during the manufacturing process. The processor 120 may generate 640 defective patches based on the labeled defective area. In this case, only the defective patch may be extracted from the image acquired during the manufacturing process, and the normal patch may not be extracted. The processor 120 may add (650) the defective patch extracted from the image acquired during the manufacturing process to the training data set. The processor 120 may update the neural network model by using the reinforced training data set. When reinforcing the training data set using images acquired during the manufacturing process, it is possible to balance the normal and defect classes of the training data by adding more defect patches, and improve the neural network performance for defect determination. I can make it.

The computing device 100 for providing a defect detection method or a learning method of a neural network model for defect detection according to an embodiment of the present disclosure includes a network unit 110, a processor 120, and a memory 130. I can.

The network unit 110 may transmit and receive a normal image, a defective image, and the like according to an exemplary embodiment of the present disclosure. Further, the network unit 110 may enable communication between a plurality of computing devices so that learning of a network function is distributedly performed in each of the plurality of computing devices. The network unit 110 enables communication between a plurality of computing devices to perform distributed processing of an operation for detecting a defect using a network function.

The network unit 110 according to an embodiment of the present disclosure may use various wired/wireless communication systems.

The processor 120 may be composed of one or more cores, and a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. unit) and the like may include a processor for providing a neural network model. The processor 120 may provide a neural network model according to an embodiment of the present disclosure by reading a computer program stored in the memory 130. According to an embodiment of the present disclosure, the processor 120 may perform pre-processing of an image or patch for generating a training data set.

According to an embodiment of the present disclosure, the processor 120 may perform an operation for learning a neural network. The processor 120 processes input data for training in deep learning (DN), extracts features from the input data, calculates errors, and uses backpropagation to update the weights of the neural network. Can perform calculations. At least one of the CPU, GPGPU, and TPU of the processor 120 may process learning of the network function. For example, a CPU and a GPGPU can work together to provide learning of network functions and operations for fault detection using network functions. In addition, in an embodiment of the present disclosure, it is possible to provide an operation for learning a network function and detecting a defect using a network function by using a processor of a plurality of computing devices together. In addition, a computer program executed in the computing device according to an embodiment of the present disclosure may be a CPU, a GPGPU, or a TPU executable program.

According to an embodiment of the present disclosure, the memory 130 may store information in any form generated or determined by the processor 120 and information in any form received by the network unit 110.

According to an embodiment of the present disclosure, the memory 130 is a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (for example, For example, SD or XD memory), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read) -Only Memory), a magnetic memory, a magnetic disk, and an optical disk. The computing device 100 may operate in connection with a web storage that performs a storage function of the memory 130 on the Internet. The description of the above-described memory is only an example, and the present disclosure is not limited thereto.

The data structure may store data corresponding to the defect detection algorithm according to an embodiment of the present disclosure.

A computer-readable medium storing a data structure according to an embodiment of the present disclosure is disclosed.

Data structure can refer to the organization, management, and storage of data that enable efficient access and modification of data. The data structure may refer to an organization of data for solving a specific problem (eg, data search, data storage, data modification in the shortest time). Data structures may also be defined as physical or logical relationships between data elements designed to support specific data processing functions. The logical relationship between data elements may include a connection relationship between data elements that a user thinks. The physical relationship between the data elements may include an actual relationship between the data elements physically stored on a computer-readable storage medium (eg, hard disk). The data structure may specifically include a set of data, a relationship between data, and a function or instruction applicable to the data. Through an effectively designed data structure, the computing device can perform calculations while using the resources of the computing device to a minimum. Specifically, computing devices can increase the efficiency of computation, reading, insertion, deletion, comparison, exchange, and search through an effectively designed data structure.

The data structure can be classified into a linear data structure and a non-linear data structure according to the shape of the data structure. The linear data structure may be a structure in which only one data is connected after one data. The linear data structure may include a list, a stack, a queue, and a deck. The list may mean a series of data sets in which order exists internally. The list may include a linked list. The linked list may be a data structure in which data is linked in a way that each data has a pointer and is linked in one line. In the linked list, the pointer may include connection information with the next or previous data. The linked list can be expressed as a single linked list, a double linked list, or a circular linked list. The stack can be a data listing structure that allows limited access to data. The stack can be a linear data structure that can process (eg, insert or delete) data only at one end of the data structure. The data stored in the stack may be a data structure (LIFO-Last in First Out) that comes out sooner as it enters the stack. A queue is a data arrangement structure that allows limited access to data, and unlike a stack, a queue may be a data structure (FIFO-First in First Out) that appears later as data stored later. A deck can be a data structure that can process data at both ends of the data structure.

The nonlinear data structure may be a structure in which a plurality of data is connected behind one data. The nonlinear data structure may include a graph data structure. The graph data structure may be defined as a vertex and an edge, and the edge may include a line connecting two different vertices. The graph data structure may include a tree data structure. The tree data structure may be a data structure in which a path connecting two different vertices among a plurality of vertices included in the tree is one. That is, it may be a data structure that does not form a loop in the graph data structure.

Throughout this specification, a computational model, a neural network, a network function, and a neural network may be used with the same meaning. (Hereinafter, it will be unified as a neural network.) The data structure may include a neural network. In addition, the data structure including the neural network may be stored in a computer-readable medium. The data structure including the neural network may also include data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, active functions associated with each node or layer of the neural network, and a loss function for learning the neural network. have. The data structure including the neural network may include any of the components disclosed above. That is, the data structure including the neural network is the data input to the neural network, the weight of the neural network, the hyper parameter of the neural network, the data obtained from the neural network, the active function associated with each node or layer of the neural network, the loss function for training the neural network, etc. It may be configured to include any combination of. In addition to the above-described configurations, the data structure including the neural network may include any other information that determines the characteristics of the neural network. In addition, the data structure may include all types of data used or generated in a neural network operation process, and is not limited to the above. Computer-readable media may include computer-readable recording media and/or computer-readable transmission media. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. The neural network includes at least one or more nodes.

The data structure may include data input to the neural network. A data structure including data input to the neural network may be stored in a computer-readable medium. Data input to the neural network may include training data input during a neural network training process and/or input data input to a neural network on which training has been completed. Data input to the neural network may include data that has undergone pre-processing and/or data to be pre-processed. The pre-processing may include a data processing process for inputting data into the neural network. Accordingly, the data structure may include data to be preprocessed and data generated by preprocessing. The above-described data structure is only an example, and the present disclosure is not limited thereto.

The data structure may include data input to or output from the neural network. A data structure including data input to or output from the neural network may be stored in a computer-readable medium. The data structure stored in the computer-readable medium may include data input during an inference process of a neural network or output data output as a result of inference of a neural network. In addition, the data structure may include data processed by a specific data processing method, and thus may include data before and after processing. Accordingly, the data structure may include data to be processed and data processed through a data processing method.

The data structure may include weights of the neural network. (In the present specification, weights and parameters may have the same meaning.) And the data structure including the weights of the neural network may be stored in a computer-readable medium. The neural network may include a plurality of weights. The weight may be variable and may be changed by a user or an algorithm in order for the neural network to perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node includes values input to input nodes connected to the output node and a link corresponding to each input node. The output node value may be determined based on the parameter. The above-described data structure is only an example, and the present disclosure is not limited thereto.

As an example and not a limitation, the weight may include a weight variable in a neural network training process and/or a weight on which neural network training has been completed. The variable weight in the neural network learning process may include a weight at a point in time at which the learning cycle starts and/or a weight variable during the learning cycle. The weight at which the neural network training is completed may include a weight at which the learning cycle is completed. Accordingly, the data structure including the weight of the neural network may include a data structure including a weight variable during the neural network training process and/or a weight on which the neural network training has been completed. Therefore, it is assumed that the above-described weights and/or combinations of weights are included in the data structure including the weights of the neural network. The above-described data structure is only an example, and the present disclosure is not limited thereto.

The data structure including the weight of the neural network may be serialized and then stored in a computer-readable storage medium (eg, memory, hard disk). Serialization may be a process of storing data structures on the same or different computing devices and converting them into a form that can be reconstructed and used later. The computing device serializes the data structure to transmit and receive data through a network. The data structure including the weights of the serialized neural network can be reconstructed in the same computing device or in another computing device through deserialization. The data structure including the weights of the neural network is not limited to serialization. Furthermore, the data structure including the weight of the neural network is a data structure to increase the efficiency of computation while using the resources of the computing device to a minimum (e.g., B-Tree, Trie, m-way search tree, AVL tree, in nonlinear data structure, Red-Black Tree). The foregoing is only an example, and the present disclosure is not limited thereto.

The data structure may include a hyper-parameter of a neural network. In addition, the data structure including the hyperparameter of the neural network may be stored in a computer-readable medium. The hyper parameter may be a variable that is changed by a user. Hyper parameters include, for example, learning rate, cost function, number of iterations of the learning cycle, weight initialization (e.g., setting the range of weight values to be weighted to be initialized), Hidden Unit It may include the number (eg, the number of hidden layers, the number of nodes of the hidden layer). The above-described data structure is only an example, and the present disclosure is not limited thereto.

11 is a block diagram of a computing device according to an embodiment of the present disclosure.

11 shows a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

While the present disclosure has generally been described above with respect to computer-executable instructions that can be executed on one or more computers, those skilled in the art will appreciate that the present disclosure may be implemented in combination with other program modules and/or as a combination of hardware and software. will be.

Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Further, to those skilled in the art, the method of the present disclosure is not limited to single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable household appliances, etc. It will be appreciated that it may be implemented with other computer system configurations, including one or more associated devices).

The described embodiments of the present disclosure may also be practiced in a distributed computing environment where certain tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Any medium accessible by the computer may be a computer-readable medium. Computer-readable media includes volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Includes the medium. Computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage device, magnetic cassette, magnetic tape, magnetic disk storage device, or other magnetic storage device, Or any other medium that can be accessed by a computer and used to store desired information.

Computer-readable transmission media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as other transport mechanism, and includes all information delivery media. do. The term modulated data signal refers to a signal in which one or more of the characteristics of the signal is set or changed to encode information in the signal. By way of example, and not limitation, computer-readable transmission media include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above-described media are also intended to be included within the scope of computer-readable transmission media.

An exemplary environment 1100 is shown that implements various aspects of the present disclosure including a computer 1102, which includes a processing device 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 couples system components, including but not limited to, system memory 1106 to processing device 1104. The processing unit 1104 may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be used as processing unit 1104.

The system bus 1108 may be any of several types of bus structures that may be additionally interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 1106 includes read-only memory (ROM) 1110 and random access memory (RAM) 1112. The basic input/output system (BIOS) is stored in non-volatile memory 1110 such as ROM, EPROM, EEPROM, etc. This BIOS is a basic input/output system that helps transfer information between components in the computer 1102, such as during startup. Includes routines. RAM 1112 may also include high speed RAM such as static RAM for caching data.

The computer 1102 also includes an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA)-this internal hard disk drive 1114 can also be configured for external use within a suitable chassis (not shown). Yes-, magnetic floppy disk drive (FDD) 1116 (for example, to read from or write to removable diskette 1118), and optical disk drive 1120 (e.g., CD-ROM For reading the disk 1122 or reading from or writing to other high-capacity optical media such as DVD). The hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 are each connected to the system bus 1108 by a hard disk drive interface 1124, a magnetic disk drive interface 1126, and an optical drive interface 1128. ) Can be connected. The interface 1124 for implementing an external drive includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

These drives and their associated computer readable media provide non-volatile storage of data, data structures, computer executable instructions, and the like. In the case of computer 1102, drives and media correspond to storing any data in a suitable digital format. Although the description of the computer-readable medium above refers to a removable optical medium such as a HDD, a removable magnetic disk, and a CD or DVD, those skilled in the art may use a zip drive, a magnetic cassette, a flash memory card, a cartridge, etc. It will be appreciated that other types of media readable by a computer, such as the like, may also be used in the exemplary operating environment and that any such media may contain computer executable instructions for performing the methods of the present disclosure.

A number of program modules, including the operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136, may be stored in the drive and RAM 1112. All or part of the operating system, applications, modules, and/or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented on a number of commercially available operating systems or combinations of operating systems.

A user may input commands and information to the computer 1102 through one or more wired/wireless input devices, for example, a pointing device such as a keyboard 1138 and a mouse 1140. Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. These and other input devices are often connected to the processing unit 1104 through the input device interface 1142, which is connected to the system bus 1108, but the parallel port, IEEE 1394 serial port, game port, USB port, IR interface, It can be connected by other interfaces such as etc.

A monitor 1144 or other type of display device is also connected to the system bus 1108 through an interface such as a video adapter 1146. In addition to the monitor 1144, the computer generally includes other peripheral output devices (not shown) such as speakers, printers, etc.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148 via wired and/or wireless communication. The remote computer(s) 1148 may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other common network node, and is generally connected to the computer 1102. Although it includes many or all of the components described for simplicity, only memory storage device 1150 is shown. The logical connections shown include wired/wireless connections to a local area network (LAN) 1152 and/or to a larger network, eg, a wide area network (WAN) 1154. Such LAN and WAN networking environments are common in offices and companies, and facilitate an enterprise-wide computer network such as an intranet, all of which can be connected to a worldwide computer network, for example the Internet.

When used in a LAN networking environment, the computer 1102 is connected to the local network 1152 via a wired and/or wireless communication network interface or adapter 1156. Adapter 1156 may facilitate wired or wireless communication to LAN 1152, which also includes a wireless access point installed therein to communicate with wireless adapter 1156. When used in a WAN networking environment, the computer 1102 may include a modem 1158, connected to a communications computing device on the WAN 1154, or through the Internet, to establish communications over the WAN 1154. Have other means. The modem 1158, which may be an internal or external and a wired or wireless device, is connected to the system bus 1108 through a serial port interface 1142. In a networked environment, program modules described for the computer 1102 or portions thereof may be stored in the remote memory/storage device 1150. It will be appreciated that the network connections shown are exemplary and other means of establishing communication links between computers may be used.

Computer 1102 is associated with any wireless device or entity deployed and operated in wireless communication, e.g., a printer, scanner, desktop and/or portable computer, portable data assistant (PDA), communication satellite, wireless detectable tag. It operates to communicate with any device or place and phone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or may simply be ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) allows you to connect to the Internet, etc. without wires. Wi-Fi is a wireless technology such as a cell phone that allows such devices, for example computers, to transmit and receive data indoors and outdoors, ie anywhere within the coverage area of a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide a secure, reliable and high-speed wireless connection. Wi-Fi can be used to connect computers to each other, to the Internet and to a wired network (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example at 11 Mbps (802.11a) or 54 Mbps (802.11b) data rates, or in products that include both bands (dual band). have.

Those of ordinary skill in the art of this disclosure will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols and chips that may be referenced in the above description are voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields. Or particles, or any combination thereof.

Those of ordinary skill in the art of the present disclosure include various exemplary logical blocks, modules, processors, means, circuits and algorithm steps described in connection with the embodiments disclosed herein, electronic hardware, For the sake of clarity, it will be appreciated that it may be implemented by various forms of program or design code or a combination of both (referred to herein as "software"). To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. A person of ordinary skill in the art of the present disclosure may implement the described functions in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term “article of manufacture” includes a computer program or media accessible from any computer-readable device. For example, computer-readable media include magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CD, DVD, etc.), smart cards, and flash memory. Devices (eg, EEPROM, card, stick, key drive, etc.), but is not limited to these. In addition, the various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. Based on the design priorities, it is to be understood that within the scope of the present disclosure a specific order or hierarchy of steps in processes may be rearranged. The appended method claims provide elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those of ordinary skill in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but is to be interpreted in the widest scope consistent with the principles and novel features presented herein.

Claims (19)

A computer program stored on a computer-readable storage medium, wherein the computer program, when executed on one or more processors of a computing device, causes operations to be performed to provide a method for detecting a defect, the operations comprising:
Extracting a defective patch from a defect image containing a defect;
Pre-processing at least one of the defective image or a normal image containing no defects;
Extracting a normal patch from at least one of the preprocessed defective image or normal image; And
Training a neural network model for classifying defects or normals of a patch using a training data set including the defective patch and the normal patch;
Containing,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The defect image is previously labeled with one or more defect areas,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The defect patch is a partial area of an image of a predetermined size extracted from the defect image, and
The normal patch is a partial area of an image of a predetermined size extracted from at least one of the defective image or the normal image,
A computer program stored on a computer-readable storage medium.
The method of claim 2,
The defective patch includes at least one defective area included in the defective image, and
The normal patch does not include the one or more defective regions included in the defective image,
A computer program stored on a computer-readable storage medium.
The method of claim 2,
The operation of extracting the defect patch from the defect image including the defect is:
Extracting the defective patch so that at least one defective area is included in the center of the patch;
Extracting the defective patch so that the at least one defective area is included in a boundary of the patch;
Extracting the defective patch so that the defective area having a predetermined number of areas or more is included in the patch; or
Extracting the defective patch so that the defect regions arranged in parallel with a predetermined continuous number or more are included in the patch;
Including at least one operation of,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
Pre-processing the defective patch;
Contains more, and
The training data set includes the preprocessed defect patch,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The operation of preprocessing at least one of the defective image or the normal image containing no defect is:
Removing noise included in the preprocessing target image, which is at least one of the defective image or the normal image;
Removing features of less than a predetermined size included in the preprocessing target image;
Removing a gap between the two or more features when the distance between the two or more features included in the pre-processing target image is less than a predetermined length; or
Smoothing a boundary of a feature included in the preprocessing target image;
Including at least one operation of,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The operation of preprocessing at least one of the defective image or the normal image containing no defect is:
Converting at least one of the defective image or the normal image, which is a pre-processing target image, into a binary image;
Obtaining an inverted image by inverse of a binary value included in the binary image;
Opening the inverse image to obtain an opened image; And
Obtaining a preprocessed image including a difference value between the inverse image and the opened image;
Containing,
A computer program stored on a computer-readable storage medium.
The method of claim 8,
The operation of obtaining the opened image by opening the inverse image,
Removing features less than a predetermined size included in the inverse image;
Containing,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The operation of preprocessing at least one of the defective image or the normal image containing no defect is:
Converting at least one of the defective image or the normal image to a binary image;
Obtaining a closed image by closing the binary image;
Obtaining a differential image including a difference value between the binary image and the closed image; And
Obtaining a preprocessed image by inverting a binary value included in the differential image;
Containing,
A computer program stored on a computer-readable storage medium.
The method of claim 10,
Closing the binary image to obtain a closed image is:
If the distance between the two or more features included in the binary image is less than a predetermined length, removing a gap between the two or more features; or
Smoothing a boundary of a feature included in the binary image;
Including at least one operation of,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The ratio of the number of defective patches and the number of normal patches included in the training data set is determined by a predetermined balance ratio,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The operation of learning a neural network model for classifying defects or normals of a patch using a training data set including the defective patch and the normal patch,
Training different neural network models for each patch size;
Containing,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
Extracting one or more patches from the image to be classified;
Calculating each of the one or more patches using the learned neural network model and classifying them as defective or normal patches; And
Determining that a defect is included in the classification target image when a predetermined number or more of the one or more patches are classified as defective patches;
Further comprising,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The defective image and the normal image are:
An image of the product after completion of the manufacturing process; or
An image of a product during the manufacturing process;
At least one of,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
Calculating, by the learned neural network model, the image acquired during the manufacturing process;
If it is determined that the image acquired during the manufacturing process contains a defect, extracting an additional defect patch based on the image acquired during the manufacturing process; And
Adding the additional defective patch to the training data set;
Further comprising,
A computer program stored on a computer-readable storage medium.
As a defect detection method,
Extracting a defect patch from the defect image including the defect;
Pre-processing at least one of the defective image or a normal image containing no defects;
Extracting a normal patch from at least one of the preprocessed defect image or normal image; And
Training a neural network model for classifying defects or normals of a patch using a training data set including the defective patch and the normal patch;
Containing,
Defect detection method.
As a server for providing a defect detection method,
A processor including one or more cores; And
Memory;
Including,
The processor,
Extracting the defect patch from the defect image containing the defect,
Pre-processing at least one of the defective image or a normal image containing no defect,
Extracting a normal patch from at least one of the preprocessed defective image or normal image, and
Learning a neural network model for classifying defects or normals of a patch using a training data set including the defective patch and the normal patch,
Server for providing a method for detecting defects.
A computer-readable recording medium storing a data structure corresponding to a parameter of a neural network whose at least part is updated during a learning process, wherein an operation of the neural network is based at least in part on the parameter, and the learning process comprises:
Extracting a defect patch from the defect image including the defect;
Pre-processing at least one of the defective image or a normal image containing no defects;
Extracting a normal patch from at least one of the preprocessed defect image or normal image; And
Training a neural network model for classifying defects or normals of a patch using a training data set including the defective patch and the normal patch;
Containing,
A computer-readable recording medium having a data structure stored thereon.

Images