KR102653930B1 - System for providing circular knitting machine optical inspection service using anomaly detection - Google Patents

Abstract

A circular knitting machine vision inspection service providing system using an anomaly detection model is provided, and a user terminal and unsupervised device that detects, classifies, stores, and outputs defective images of fabric discharged from a circular knitting machine that knits fabric using a pre-built anomaly detection model. A modeling unit that models a learning-based anomaly detection model by learning and verifying it using only normal images, an anomaly detection unit that applies the anomaly detection model to the user terminal to detect defect images from images input from the vision inspection of the user terminal, and a defect image When detected, the inspection includes an analysis unit that analyzes and stores the fabric production coordinates of the defect image and defect type data of the defect image, and a database unit that outputs and stores the defect image, defect type data, and fabric production coordinates on the user terminal. Includes service provision servers.

Description

Circular knitting machine vision inspection service provision system using anomaly detection model {SYSTEM FOR PROVIDING CIRCULAR KNITTING MACHINE OPTICAL INSPECTION SERVICE USING ANOMALY DETECTION}

The present invention relates to a circular knitting machine vision inspection service providing system using an anomaly detection model, and provides a solution that can detect, classify, and store fabric defects occurring in a circular knitting machine using an anomaly detection model.

The manufacturing industry is developing rapidly in modern society, but the development of outlier detection and defect detection, which are one of the core processes of the manufacturing industry, has developed into a form that can only be applied to specific fields, so in most industries, workers still perform visual inspection directly. . Deep learning has shown outstanding performance in detecting outliers and defects and has become an essential element. However, feature learning or expression learning requires a lot of both normal and defective data, making it difficult to be used in manufacturing sites where defective data is relatively scarce. difficult. Accordingly, models created solely for outlier detection and defect detection based on unsupervised learning, which learn the characteristics of normal data using only normal data, have been researched and developed. These models learn the characteristics of normal data when new data is input. The similarity is measured each time, and if defective data is input, the similarity does not meet a certain level and is judged as defective.

At this time, a method of extracting defects in fabric based on big data or identifying defects using deep learning was researched and developed. In relation to this, a prior art, Korea Registered Patent No. 10-2445162 (announced on September 20, 2022) ) and International Publication Patent No. 2019-107614 (published on June 6, 2019), data on defects in the fabric manufacturing area are collected, big data is secured based on the collected information, and vision inspection (Automated Optical Inspection) is performed. , AOI) is added to big data, and defects in the fabric are detected and analyzed using machine learning or deep learning, and deep learning is used to classify normal and defective images for machine vision-based quality inspection. Configurations for learning to do this and determining normal and defective using the learned classifier are disclosed.

However, in the case of the former, machine vision inspection is a design method in which engineers set rules one by one according to the process characteristics and target items. Performance depends on how well the engineer creates the rules, and if the analysis target is complex, The detection accuracy for defects is low. In the latter case, a configuration using deep learning has been disclosed, but since defect images are rarely produced in precision processes, new facilities, or high-product industries, there is a lack of big data to build, and as a result, deep learning learning and verification is required due to the lack of defect images. This will not happen. In other words, deep learning is not a model created for detecting outliers and defects, and requires a lot of both normal and defective images to learn features. Accordingly, research and development of a solution that detects defective images using only normal images is required.

In one embodiment of the present invention, when using a deep learning-based anomaly detection model that detects outliers using only normal images, images with characteristics different from normal images learned using the twin structure of the Teacher-Student model are classified into step-by-step characteristics. By comparing and detecting, defects are marked on the fabric production coordinates detected as outliers, the type of defect is analyzed to generate defect type data, and an image of the part where the defect is detected is output to the user terminal, and the defect type data and It is possible to provide a circular knitting machine vision inspection service provision system using an anomaly detection model that allows fabric production coordinates to be stored in a database. However, the technical challenge that this embodiment aims to achieve is not limited to the technical challenges described above, and other technical challenges may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention detects, classifies, stores, and outputs defective images of fabric discharged from a circular knitting machine that knits fabric using a pre-built abnormality detection model. A modeling unit that models the user terminal and an unsupervised learning-based anomaly detection model by learning and verifying it using only normal images, and an anomaly detection model that applies the anomaly detection model to the user terminal to detect defective images from images input from the vision inspection of the user terminal. Detection unit, when a defect image is detected, an analysis unit that analyzes and stores the fabric production coordinates of the defect image and defect type data of the defect image, and a database that outputs and stores the defect image, defect type data, and fabric production coordinates on the user terminal. It includes an inspection service providing server including a stoker.

According to one of the means for solving the problem of the present invention described above, when using a deep learning-based anomaly detection model that detects outliers using only normal images, characteristics that are different from the normal image learned using the twin structure of the Teacher-Student model By detecting images with step-by-step characteristics by comparing them, defects are marked on the fabric production coordinates detected as outliers, the type of defect is analyzed to generate defect type data, and the image of the part where the defect is detected is sent to the user terminal. While printing, defect type data and fabric production coordinates can be saved in the database.

Figure 1 is a diagram illustrating a circular knitting machine vision inspection service providing system using an abnormality detection model according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating the inspection service providing server included in the system of FIG. 1.
Figures 3 and 4 are diagrams for explaining an embodiment in which a circular knitting machine vision inspection service using an anomaly detection model according to an embodiment of the present invention is implemented.
Figure 5 is an operation flowchart illustrating a method of providing a circular knitting machine vision inspection service using an anomaly detection model according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, this does not mean excluding other components unless specifically stated to the contrary, but may further include other components, and one or more other features. It should be understood that it does not exclude in advance the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

The terms "about", "substantially", etc. used throughout the specification are used to mean at or close to that value when manufacturing and material tolerances inherent in the stated meaning are presented, and are used to enhance the understanding of the present invention. Precise or absolute figures are used to assist in preventing unscrupulous infringers from taking unfair advantage of stated disclosures. The term “step of” or “step of” as used throughout the specification of the present invention does not mean “step for.”

In this specification, 'part' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized using one piece of hardware. Meanwhile, '~ part' is not limited to software or hardware, and '~ part' may be configured to reside in an addressable storage medium or may be configured to reproduce one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card.

In this specification, some of the operations or functions described as being performed by a terminal, apparatus, or device may instead be performed on a server connected to the terminal, apparatus, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed in a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal mean mapping or matching the terminal's unique number or personal identification information, which is identifying data of the terminal. It can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a diagram illustrating a circular knitting machine vision inspection service providing system using an abnormality detection model according to an embodiment of the present invention. Referring to Figure 1, the circular knitting machine vision inspection service providing system 1 using an anomaly detection model includes at least one user terminal 100, an inspection service providing server 300, and at least one information providing server 400. can do. However, since the circular knitting machine vision inspection service providing system 1 using the abnormality detection model of FIG. 1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1.

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1, at least one user terminal 100 may be connected to the test service providing server 300 through the network 200. In addition, the test service providing server 300 may be connected to at least one user terminal 100 and at least one information providing server 400 through the network 200. Additionally, at least one information providing server 400 may be connected to the inspection service providing server 300 through the network 200.

Here, the network refers to a connection structure that allows information exchange between each node, such as a plurality of terminals and servers. Examples of such networks include a local area network (LAN) and a wide area network (WAN). Wide Area Network, Internet (WWW: World Wide Web), wired and wireless data communication network, telephone network, wired and wireless television communication network, etc. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), and Wi-Fi. , Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth network, NFC ( It includes, but is not limited to, Near-Field Communication (Near-Field Communication) network, satellite broadcasting network, analog broadcasting network, and DMB (Digital Multimedia Broadcasting) network.

In the following, the term at least one is defined as a term including singular and plural, and even if the term at least one does not exist, each component may exist in singular or plural, and may mean singular or plural. This should be self-explanatory. In addition, whether each component is provided in singular or plural form may be changed depending on the embodiment.

At least one user terminal 100 is a user who detects, extracts, classifies and stores defects in the fabric discharged from the circular knitting machine using a web page, app page, program or application related to the circular knitting machine vision inspection service using an abnormality detection model. It may be a terminal of

Here, at least one user terminal 100 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc. At this time, at least one user terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one user terminal 100 is, for example, a wireless communication device that guarantees portability and mobility, and includes navigation, personal communication system (PCS), global system for mobile communications (GSM), personal digital cellular (PDC), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) ) It may include all types of handheld-based wireless communication devices such as terminals, smartphones, smartpads, and tablet PCs.

The inspection service providing server 300 may be a server that provides a circular knitting machine vision inspection service web page, app page, program, or application using an anomaly detection model. Additionally, the inspection service providing server 300 may be a server that builds an anomaly detection model by learning and verifying normal images using a twin-structured Teacher-Student model. In addition, the inspection service providing server 300 connects the user terminal 100 and a camera to detect and output defect images detected and classified by the circular knitting machine, and at the same time displays the fabric production coordinates to find defective parts in the fabric. It may be a server that allows this. And the inspection service providing server 300 builds a deep learning model to classify the defect image so that defect type data can be labeled in the defect image, and based on this, defect images are also used with normal images to prevent data shortage. It may be a server that builds data. When the amount of normal images and defective images are similar and accumulated on a large scale, various deep learning can be applied in addition to the Teacher-Student model described above, so the inspection service providing server 300 provides a dataset and a foundation for this. It could be a server.

Here, the inspection service providing server 300 may be implemented as a computer that can connect to a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc.

At least one information providing server 400 labels the defect image with defect type data using a web page, app page, program, or application related to the circular knitting machine vision inspection service using an abnormality detection model, and then labels the inspection service providing server 300 with defect type data. It may be a server that uploads to . The applicant's company (D-World Co., Ltd.) is a company that supplies machine vision inspection equipment, hardware, and software. Many of the applicant's companies have built infrastructure including machine vision computing resources and networking resources. Before setting up the solution of the present invention, the user may be a user of the user terminal 100, and after the infrastructure is built and the anomaly detection model is set, the user may be a user of the source that uploads the image, such as the information provision server 400. They also perform their roles together. Of course, the user terminal 100 can also serve as the information provision server 400 only if consent is given to image collection, and if consent is not given to image collection, the user terminal 100 can serve as the information provision server 400. It will no longer play the role of . In addition, the user terminal 100 may be the information providing server 400, but in one embodiment of the present invention, for convenience of explanation, the user terminal 100 is a user who builds and uses the solution of the present invention. The information provider of the information provision server 400 will be explained by dividing the role into an image provider.

Here, at least one information providing server 400 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc. At this time, at least one information providing server 400 may be implemented as a terminal that can access a remote server or terminal through a network. At least one information provision server 400 is, for example, a wireless communication device that guarantees portability and mobility, and includes navigation, personal communication system (PCS), global system for mobile communications (GSM), and personal digital cellular (PDC). , PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband) It may include all types of handheld-based wireless communication devices such as Internet) terminals, smartphones, smartpads, and tablet PCs.

Figure 2 is a block diagram for explaining the inspection service providing server included in the system of Figure 1, and Figures 3 and 4 are implementations of a circular knitting machine vision inspection service using an abnormality detection model according to an embodiment of the present invention. This drawing is for explaining an embodiment.

Referring to Figure 2, the inspection service providing server 300 includes a modeling unit 310, an abnormality detection unit 320, an analysis unit 330, a database unit 340, a model setting unit 350, and a learning unit ( 360), a classification unit 370, a construction unit 380, a big data conversion unit 390, a preprocessing unit 391, and a real-time detection unit 393.

The inspection service providing server 300 or another server (not shown) operating in conjunction with one embodiment of the present invention creates an anomaly detection model with at least one user terminal 100 and at least one information providing server 400. When transmitting a circular knitting machine vision inspection service application, program, app page, web page, etc., at least one user terminal 100 and at least one information provision server 400, a circular knitting machine vision inspection service application using an anomaly detection model , you can install or open programs, app pages, web pages, etc. Additionally, a service program may be run on at least one user terminal 100 and at least one information provision server 400 using a script executed in a web browser. Here, a web browser is a program that allows the use of web (WWW: World Wide Web) services and refers to a program that receives and displays hypertext written in HTML (Hyper Text Mark-up Language), for example, Netscape. , Explorer, Chrome, etc. Additionally, an application refers to an application on a terminal and includes, for example, an app running on a mobile terminal (smartphone).

Referring to FIG. 2, the modeling unit 310 can model an unsupervised learning-based anomaly detection model by learning and verifying it using only normal images. Recently, deep learning technology has shown excellent performance in computer vision fields such as images and videos, and its scope of application to various fields is currently expanding. However, high-specification deep learning models generally learn tens of millions or more. Because it has a complex network structure made up of variables, efficient knowledge distillation and knowledge transfer technologies are required to activate the application of the complex network structure that has already been learned to various application fields. Research has been conducted to extract useful information from a complex network that has already been trained and transfer that information to a relatively low-specification deep learning model. Transferring information from a teacher model to a complex network that has already been trained The low-specification network that receives and performs learning is defined as the Student model, and the value output from the probability-based Softmax Output Layer of the Teacher model is relaxed with the Softmax Factor to learn the Student model. A learning method, namely Knowledge Distillation (KD), was developed. Weight optimization using the hint layer of the teacher model can be included in general KD learning to improve learning performance by performing hint learning and KD learning sequentially. At this time, as will be described later, in one embodiment of the present invention, the Teacher-Student model is configured as a twin structure, but both the Teacher model and the Student model are ResNet, and the width is the same, and the Teacher model is configured to have a deeper network than the Student model. You may.

<Hint learning algorithm>

The hint learning algorithm for application to ResNet is first of all, using the output value from the hint layer, which is the middle layer of the already trained Teacher model, and the output value from the guide layer, which is the middle layer of the Student model to be learned, Calculate the L2 cost together.

Here, fh, fg, and fr are the outputs that pass through activation functions from Wh, which corresponds to the weight from the input layer to the hint layer, and the output that passes through the activation functions from Wg, which corresponds to the weight from the input layer to the guide layer. It represents one output and the output of a regressor function with weight Wr, respectively. At this time, the regression function refers to a regression operator that is inserted to match the size and number of feature maps of the hint layer and the guide layer when they are different. In order to reduce the complexity of calculating the regression function, we consider a ResNet structure where the number of layers is different but the feature map size is the same. Therefore, by excluding the regression function from Equation 1, the L2 cost for hint learning can be calculated as in Equation 2.

Hint-KD learning can be performed using (hat)Wg from the Student model extracted from Equation 2 to the guide layer.

<Hint-KD learning algorithm>

First, we explain the hint-KD learning method that uses both the weight information extracted from hint learning in Equation 2 and the KD information based on the relaxation constant of the teacher model. First, before learning Hint-KD, initialization of all weights of the Student model is performed as shown in Equation 3 below.

Here, W-1s refers to a weight with a random value from the guide layer to the final output layer in the Student model. That is, in the Student model, the weight from the input layer to the guide layer uses (hat)Wg obtained from the previous hint learning in Equation 2, and the remaining weight, W-1s, from the guide layer to the final output layer is a random value. Configure Ws by initializing it with . Next, the softmax output value in the final output layer of the Teacher model is relaxed to τ, the final output value of the Student model to be trained is also relaxed to τ, and KD learning is performed by constructing a cost function as shown in Equation 4 below.

Here, E is cross-entropy, λ is a multi-cost function control constant, Ytrue is GT (ground truth) with a true label, PT=softmax(aT/τ), Ps= softmax(aS/τ), aT represents the input vector before softmax output in the Teacher model, aS represents the input vector before softmax output in the Student model, and τ represents the relaxation constant. Therefore, using Equation 2, hint learning is first performed to obtain the (hat)Wg of the Student model to resemble Wh, the weight from the input layer to the hint layer in the Teacher model, and (hat)Wg is calculated using Equation 3. After constructing the initial weight Ws from , hint-KD learning using both hint information and KD information can be finally performed using Equation 4. Of course, it is obvious that various Teacher-Student models can be used, such as the twin structure Teacher-Student model, STPM (Student-Teacher Feature Pyramid Matching) model, and RD (Reverse Distillation) model, which will be described later.

The anomaly detection unit 320 may apply an anomaly detection model to the user terminal 100 to detect a defect image from an image input from a vision inspection of the user terminal 100. The user terminal 100 can detect, classify, store, and output defective images of fabric discharged from a circular knitting machine that knits fabric using a pre-built abnormality detection model. At this time, since the Teacher-Student model only learns normal images, it may be able to detect the defective image because it has a different pattern from the normal image, but it cannot detect the type of defect in the defective image due to another problem. At this time, in order to detect and classify the defect image, a dataset of [defect image-defect type data] is required. In a separate application filed by the present applicant, the defect image and defect type data are stored in a database to correspond to the codes of the Korean Standard Industrial Classification. and further build big data. For this purpose, a separate deep learning model is required, which will be explained in detail in the construction unit 380 described later.

When a defect image is detected, the analysis unit 330 may analyze and store the fabric production coordinates of the defect image and the defect type data of the defect image. At this time, since the defect type data is explained in the type classification unit 370 described later, detailed description will be omitted.

The database unit 340 can allow the user terminal 100 to output and store defect images, defect type data, and fabric production coordinates.

The model setting unit 350 may use a Teacher-Student model with an unsupervised learning-based twin structure as the anomaly detection model. When learning the Teacher-Student model, the learning unit 360 inputs normal images to both the Teacher encoder and the Student encoder, and calculates the difference between the output matrix of the result output from the Teacher layer and the output matrix of the result output from the Student layer. After calculating the difference and updating it as a weight in the Student layer, the Teacher layer and Student layer can be trained to output similar output matrices. The teacher layer is a pretrained layer that cannot be learned and is a layer that extracts features, and the student layer is a non-pretrained layer that can be learned and can be a layer that extracts features. At this time, before explaining the twin-structured Teacher-Student model, the STPM (Student-Teacher Feature Pyramid Matching) model and RD (Reverse Distillation) are explained with reference to FIGS. 3A and 3B.

Due to the limitation of lack of defective images, the model created solely for outlier detection and defect detection based on unsupervised learning, which learns the characteristics of normal images using only normal images, learns the characteristics of normal imagers and increases the similarity level every time new data comes in. is measured, and if a defective image is received and the similarity does not meet a certain level, it is generally judged as defective. Among these unsupervised learning-based models, when each class is learned one by one, it is classified into a One Class Anomaly Detection model. A technique frequently used in this field is the KD (Knowledge Distillation) method using an encoder. KD-based models include the STPM model that uses the Teacher-Student (or Student-Teacher) model. Additionally, there is a model built using the Teacher-Student encoder and decoder, which is called RD. These two models are similar in that the architecture learns only one class during training and allows visualization of defect areas, but the difference is that the STPM model uses only an encoder, while the RD model uses an encoder, decoder, and feature compression module. There is.

<STPM model>

Referring to Figure 3a, STPM has the characteristic of inputting normal data into both the Teacher encoder and the Student encoder during learning. Both Teacher and Student are made of the same layer based on ResNet, and have three major parts depending on the scale. In the case of the teacher, it is a pretrained layer, so it cannot be learned, and its role is to extract features. Student is a layer that can be learned and is composed of the same layers as Teacher, but has not been learned in advance. The feature maps extracted according to the scale are paired and the similarity is measured between T1 and S1, T2 and S2, and T3 and S3. This similarity becomes the loss, and learning is performed so that the teacher and student produce similar feature maps. When defective data comes in at the later inference stage, the pre-trained Teacher extracts the features well, but the Student who has only experienced normal images does not extract the features well. This difference can be collected and visualized, and the threshold at which the F1-Score appears high Classification is also possible through the threshold value.

<RD model>

Referring to Figure 3b, the RD model is similar to STPM, but the difference is that Student uses a decoder rather than an encoder and uses a feature compression module to compress the features from the encoder and then restore them using the decoder. Like STPM, the Teacher is a ResNet that has been trained in advance and cannot be trained, and the Student is a decoder that constructs a ResNet that can be trained using a Deconvolution Layer. The feature compression module is also part of ResNet and can be trained. Like STPM, the similarity between feature maps is measured and used for learning. In addition, it is possible to collect and visualize parts that have failed to be restored, and classification is also possible by setting a threshold for similarity.

<Teacher-Student model>

As shown in Figure 3c, the values output by twin-structure convolution and max pooling are output as output matrices as they pass through each layer, and the weights are updated to the Student based on the Teacher's output matrix so that the Student learns the output matrix output by the Teacher. and repeat learning. As shown in Figure 3d, in the verification (testing) process, it is checked whether a defective (defective) image is detected.

When the image is classified as a defect image in the anomaly detection model, the type classification unit 370 allows a classifier to classify the defect type of the defect image, and labels and stores defect type data of the defect type in the defect image. You can. At this time, it may be necessary to hire a labeler to initially label the defect type data of the defect image, but if imitation learning of reinforcement learning is used, automatic labeling similar to human labeling can gradually become possible. In video, an image is a way to express visual information on a two-dimensional plane. In terms of how to process information using images, deep neural networks have recently made rapid progress in object detection and pattern recognition in the image recognition field using convolution, sometimes showing higher reading rates than humans under limited conditions. A convolutional neural network is a type of artificial neural network consisting of several hidden layers between the input layer and the output layer, and learns convolution patterns on its own. Many convolutional neural networks are trained using supervised learning, and are given images as input and labels, which are words or images, as output. For an effective convolutional neural network, high accuracy results can be obtained while effectively solving the overfitting problem when there are many datasets.

For this purpose, the dataset is labeled with the labor of many experts, that is, labelers, and processed in parallel to effectively process many datasets. However, recently, image data is accumulating beyond the capacity that only experts can process. Accordingly, multiple experts need a collaborative method of acquiring datasets. Reinforcement learning is a neural network in which a model defined within an environment recognizes the current state and selects an action or sequence of actions that gives the maximum reward among selectable actions. In one embodiment of the present invention, a group of experts consists of deep learning-based learning image classification and reinforcement learning that imitates experts, performs labeling by experts, and simultaneously performs automated classification through reinforcement learning and records low-accuracy images from experts. A complementary system can be used as labeling.

A deep convolutional neural network is a deep learning neural network dedicated to image processing. An image is given as input and image label data is given as output. Several models show very high recognition rates in image classification problems using Imagenet Large Scale Visual Recognition Callenge data. R-CNN (Region-Convolutional Neural Network) is a deep convolutional neural network for use in real-time images such as videos. R-CNN is built around processing time and can compare many images per unit time. Reinforcement learning is a neural network that recognizes states rather than inputs and outputs and selects an action sequence that provides maximum reward from selectable actions. In one embodiment of the present invention, the labeling behavior of the imitation learner can be set to imitate the labeling behavior of an expert.

<Image labeling>

It can be configured as an imitation learner that collects the expert's labeling and imitates the expert's labeling pattern using reinforcement learning. The storage can be composed of an image and a labeling database. The image database stores images needed for classification, and the labeling database stores data labeled by experts or imitation learners. The type classification unit 370 according to an embodiment of the present invention provides an imitation learner to experts. The imitation learner performs labeling after object exploration using an R-CNN-based neural network based on data labeled by experts. The imitation learner allows reinforcement learning to be performed through reclassification by experts when confidence in the previously classified data for an object is low or there is no data in the classification category.

This imitation learner allows a group of experts to perform mass image labeling through more intensive evaluation and supplementation of the imitation learning results rather than continuous classification. The expert who receives the image obtains the coordinates of the object in the image and assigns a label. Labeled objects are sent to the server and stored in the labeling database. In addition, an expert additionally inputs a new image dataset and performs labeling directly, or specifies that an imitation learner can perform labeling. The imitation learner performs object search on objects obtained based on labeling using R-CNN. If the detected object obtains trust level data, the same label is added. If it is acquired with low accuracy, the object in the image is returned to the expert so that new labeling data for the object can be obtained. Data labeled by experts serves as a reward for reinforcement learning and updates the inference model when updated with new labeling.

The construction unit 380 may model the defect image by labeling the defect type data, constructing the data set, and learning and verifying at least one deep learning model. The deep learning model uses a two-stage detection algorithm, and among the two-stage algorithms, R-CNN (Regions with Convolutional Neuron Networks) can be used. Object detection is a technology that finds classification and location information using computer vision and image processing. Localization involves regression to find the Bounding Box, and Classification refers to the problem of classifying the objects within the Bounding Box. Object detection refers to a detection method that contains information about classification and localization, and is divided into single-stage detection (One-Stage Detector) and two-stage detection (Two-Stage Detector). Single-step detection is fast because it processes classification and localization simultaneously, but its accuracy is low. YOLO and SSD are representative examples. Two-stage detection divides classification and localization tasks and processes them sequentially, and representative examples include R-FCN, Faster R-CNN, and Mask R-CNN. In one embodiment of the present invention, since even the smallest defects or defects must be found, a two-step detection method is used instead of a single-step detection that is fast but less accurate. However, single-step detection is not completely excluded.

<R-CNN series object detection>

As described above, deep learning-based object detection models can be divided into the R-CNN series and the YOLO series. The R-CNN series includes Fast/Faster R-CNN, which is a higher version of R-CNN, and the YOLO series includes YOLO and SSD (Single Shot Detection), which complements YOLO's scale vulnerabilities. The biggest difference between the R-CNN series and the YOLO series is in the region proposal and classifier structure. R-CNN, an early object model, has a dual region proposal and classifier learning process. In other words, it is structured to generate a candidate bounding box through SS and learn a classifier. R-CNN requires candidate bounding boxes to be adjusted to the same size through lasizing, and then individually passed through an image recognition model such as VGG. Therefore, a considerable amount of calculation time is required. To compensate for these shortcomings, a Fast R-CNN model has emerged that creates a ROI (Region of Interest) layer, maps the candidate bounding box to the coordinates of the VGG feature map, and then extracts the same size. However, because the domain proposals are still separate, it cannot be considered end-to-end learning. Faster R-CNN proposes a subnetwork that can perform region proposals on VGG's feature maps, thereby simultaneously reducing computation time and improving detection accuracy through end-to-end learning. Accordingly, in one embodiment of the present invention, in addition to R-CNN, Fast/Faster R-CNN or Mask R-CNN can be used.

<Zone division subnetwork>

Techniques for segmenting regions in an input image include K-Means Clustering and graph partitioning techniques. However, because the performance of recent deep learning-based image segmentation networks is excellent, one embodiment of the present invention seeks to introduce a deep learning model. Representative image segmentation networks include FCN (Fully Convolutional Networks) and U-Net models. FCN largely consists of an encoder and a decoder. The encoder extracts features from the input image and reduces the size of the feature map through stride and pooling processes. Then, in the decoder, the feature map compressed in the encoder is enlarged through upsampling, and then the final region division map is estimated. The upsampling process is implemented using interpolation or transposed convolution. Of course, it is obvious that features can be extracted and compared using various methods other than those described above.

The big data unit 390 can receive the code of the Korean Standard Industrial Classification of the manufacturer equipped with the circular knitting machine from the user terminal 100 and construct big data as a dataset in which the defect image is labeled with defect type data. Since this is scheduled to be described in a separate application of the present applicant, detailed description will be omitted.

The preprocessor 391 can generate an image patch of a preset size from the image data input from the camera that captured the circular knitting machine to generate an image for input to the abnormality detection model, and record the fabric production coordinates. there is.

The real-time detection unit 393 connects a camera that photographs the fabric knitted by the circular knitting machine to the user terminal 100, as shown in FIGS. 4E and 4F, and transmits the results of photographing the fabric knitted along the rotation axis of the circular knitting machine from the camera to the user terminal. By receiving input as (100), defect images can be detected in real time using an anomaly detection model. At this time, lighting can be arranged on the back of the camera to face the circular knitting machine.

Hereinafter, the operation process according to the configuration of the inspection service providing server of FIG. 2 described above will be described in detail using FIGS. 3 and 4 as an example. However, it will be apparent that the embodiment is only one of various embodiments of the present invention and is not limited thereto.

3A to 3D are the same as described above in the Teacher-Student model, so detailed description is omitted. Figures 4a and 4b are photographs of a prototype of a vision inspection model installed on a circular knitting machine according to an embodiment of the present invention, and the specifications of the vision system are shown in Table 1 below.

division detail camera Area Camera 5M, 16mm Lens light 400k lux high brightness LED (326mm) optical resolution X: 40um, Y: 40um speed Max: 40rpm (about 120mpm)

Figure 4c is a learning screen of an anomaly detection model according to an embodiment of the present invention. For learning, a learning dataset is constructed by securing normal images of sample fabrics produced by a circular knitting machine, and the size is 256 × 256 × 3, 500 pieces were secured. Figure 4d is a solution (S/W) installed on the user terminal 100, and is a screen that receives images taken of the knit fabric being produced and detects defect images. Create an image patch with the size to be input to the anomaly detection model, record the length coordinates of the fabric, input the image patch into the anomaly detection model, receive the judgment result, and if it is a defective image, mark the defect in the fabric production coordinates on the screen. and displays a defect image. Fabric production coordinates and defect type data are stored in the database.

Figure 4e is a configuration diagram of a vision inspection model according to an embodiment of the present invention, and Figure 4f is an installation drawing. Figure 4g is a screen taken by a prototype in which the vision inspection model of the present invention is installed on a circular knitting machine, and Figure 4h is a prototype of the user terminal 100. Through the process of Figure 4i, defects can be detected, classified, and stored, and ultimately, deep learning is used to resolve the imbalance between normal and defective images and to store them as big data according to each industrial classification, thereby eliminating defects and defects between manufacturing companies. We want to build a dataset for detection. Figure 4j is a screen where a defect is detected, and the type of defect is the same as Figures 4k to 4n.

Matters that are not explained regarding the method of providing circular knitting machine vision inspection service using the abnormality detection model of FIGS. 2 to 4 are the same as those previously described regarding the method of providing circular knitting machine vision inspection service using the abnormality detection model through FIG. 1. Since it can be easily inferred from the explained contents, the description below will be omitted.

FIG. 5 is a diagram illustrating a process in which data is transmitted and received between each component included in the circular knitting machine vision inspection service providing system using the abnormality detection model of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of the process of transmitting and receiving data between each component will be described with reference to FIG. 5, but the present application is not limited to this embodiment, and the process shown in FIG. 5 according to the various embodiments described above It is obvious to those skilled in the art that the process of transmitting and receiving data can be changed.

Referring to FIG. 5, the inspection service providing server models an unsupervised learning-based anomaly detection model by learning and verifying it using only normal images (S5100), and applies the anomaly detection model to the user terminal to obtain information from the vision inspection of the user terminal. A defective image is detected from the input image (S5200).

In addition, when a defect image is detected, the inspection service providing server analyzes and stores the fabric production coordinates of the defect image and defect type data of the defect image (S5300), and the defect image, defect type data, and fabric production coordinates are stored in the user terminal. Print and save (S5400).

The sequence between the above-described steps (S5100 to S5400) is only an example and is not limited thereto. That is, the order between the above-described steps (S5100 to S5400) may change, and some of the steps may be executed simultaneously or deleted.

Matters not explained regarding the method of providing circular knitting machine vision inspection service using the abnormality detection model of FIG. 5 are the same as those explained previously regarding the method of providing circular knitting machine vision inspection service using the abnormality detection model through FIGS. 1 to 4 or Since it can be easily inferred from the explained contents, the description below will be omitted.

The method of providing a circular knitting machine vision inspection service using an abnormality detection model according to an embodiment described with reference to FIG. 5 is also available in the form of a recording medium containing instructions executable by a computer, such as an application or program module executed by a computer. It can be implemented. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include all computer storage media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

The method of providing a circular knitting machine vision inspection service using an abnormality detection model according to an embodiment of the present invention described above includes an application installed by default on a terminal (this may include programs included in the platform or operating system, etc., which are installed by default on the terminal). It may be executed by, and may also be executed by an application (i.e. program) installed by the user directly on the master terminal through an application providing server such as an application store server, an application, or a web server related to the service. In this sense, the method of providing a circular knitting machine vision inspection service using an abnormality detection model according to an embodiment of the present invention described above is implemented as an application (i.e., program) installed by default on the terminal or directly installed by the user, and is implemented as an application (i.e., program) installed by default on the terminal or directly installed by the user. It may be recorded on a computer-readable recording medium.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

Claims (10)

A user terminal that detects, classifies, stores, and outputs defective images of fabric discharged from a circular knitting machine that knits fabric using a pre-established abnormality detection model; and
A modeling unit that models an unsupervised learning-based anomaly detection model by learning and verifying it using only normal images, and applies the anomaly detection model to the user terminal to detect defective images from images input from the vision inspection of the user terminal. A detection unit, when the defect image is detected, an analysis unit that analyzes and stores the fabric production coordinates of the defect image and defect type data of the defect image, and outputs the defect image, defect type data, and fabric production coordinates from the user terminal. and a database storage unit for storing the abnormality detection model, a model setting unit that uses a Teacher-Student model of an unsupervised learning-based twin structure, and a normal image is input to both the Teacher encoder and the Student encoder when learning the Teacher-Student model, Calculate the difference between the output matrix of the results output from the Teacher layer and the output matrix of the results output from the Student layer, update the difference as a weight for the Student layer, and then output an output matrix where the Teacher layer and the Student layer are similar. It includes a test service providing server including a learning unit that teaches the user to do so,
The teacher layer is a pretrained layer that cannot be learned and is a layer that extracts features.
The Student layer is a non-pretrained layer that can be learned and is a layer that extracts features. A circular knitting machine vision inspection service providing system using an anomaly detection model.
delete delete delete According to claim 1,
The inspection service providing server,
When the anomaly detection model classifies the image as a defect image, a classifier classifies the defect type of the defect image, and the defect type data of the defect type is labeled and stored in the defect image. wealth;
A circular knitting machine vision inspection service providing system using an anomaly detection model further comprising:
According to claim 5,
The inspection service providing server,
a construction unit that constructs the defect image as a dataset after labeling the defect type data and models it by learning and verifying at least one deep learning model;
A circular knitting machine vision inspection service providing system using an anomaly detection model further comprising:
According to claim 6,
The inspection service providing server,
A big data converter that receives the code of the Korean Standard Industrial Classification of the manufacturer equipped with the circular knitting machine from the user terminal and constructs big data as a dataset in which the defect image is labeled with the defect type data;
A circular knitting machine vision inspection service providing system using an anomaly detection model further comprising:
According to claim 1,
The inspection service providing server,
A preprocessor that generates an image patch of a preset size from image data input from a camera that captures a circular knitting machine to generate an image for input to the abnormality detection model and records fabric production coordinates;
A circular knitting machine vision inspection service providing system using an anomaly detection model further comprising:
According to claim 1,
The inspection service providing server,
A camera that photographs the fabric knitted by a circular knitting machine is connected to the user terminal, and the results of photographing the fabric knitted along the rotation axis of the circular knitting machine are input from the camera to the user terminal, and a defect image is generated by the abnormality detection model in real time. A real-time detection unit that detects;
A circular knitting machine vision inspection service providing system using an anomaly detection model further comprising:
According to clause 9,
A circular knitting machine vision inspection service providing system using an anomaly detection model, characterized in that lighting is arranged on the rear of the camera to face the circular knitting machine.

Images