KR102658868B1 - Deep Learning System and its Control Method Applicable for Quality Inspection by Learning Only Non-Defective Manufactured Product Data - Google Patents

Abstract

Another aspect of the present invention is a deep learning-based quality inspection system for learning non-defective product data, which includes an input unit that receives a non-defective product image data set; For each of the plurality of images included in the image data set, resizing (adjusting each image to a desired size without applying a cropping operation that cuts out and processes only an area at a specific location within each image) A preprocessor that preprocesses images of the same size so that the model can learn them by applying a resizing operation and a padding operation that adjusts the image size while maintaining the ratio of each image; a control unit that extracts good product features that serve as a standard for good products from the preprocessed image and adds Gaussian noise features to the extracted good product features to generate a plurality of fake defective product features; and a discriminator that learns based on at least some of the non-defective product image data set received from the control unit, the extracted non-defective product features, and the plurality of fake defective product features.

Description

Deep Learning System and its Control Method Applicable for Quality Inspection by Learning Only Non-Defective Manufactured Product Data}

The present invention relates to a deep learning system and its control method that can be applied to quality inspection by learning only accurate and reliable manufacturing good data in product quality inspection, which is a core task in the injection process among various manufacturing fields.

In general, the process environment is complex and the collected images are not clear, so clear images are required for model learning, and image quality analysis and filtering must precede model learning.

At this time, the image classification AI (artificial intelligence) method requires a large amount of good and defective data for learning, but since the defect rate is low in the actual process, it is difficult to collect a large amount of defective data, so there are significantly fewer cases of surface inspection AI application in the manufacturing industry. There is a problem with it being small.

In addition, according to the conventional method, AI accuracy was unstable as the classification criteria at the manufacturing site were applied incorrectly, making field application difficult.

Furthermore, according to the prior art, the size, length, color, etc. of each detected defect differed depending on the type, so there was a problem that it was difficult to set a single standard for all facilities and train the network.

Therefore, there is a growing need for systems and methods that can be easily used by anyone with AI that does not require prior learning of the various types of defects that appear on the product surface, and that can be quickly applied to a rapidly changing manufacturing environment because labeling is not required.

Republic of Korea Intellectual Property Office Registration No. 10-2343272 Republic of Korea Intellectual Property Office Application No. 10-2021-0072282

In order to solve the problems of the prior art, the present invention is a deep learning system and a control method applicable to quality inspection by learning only accurate and reliable manufacturing good data in product quality inspection, which is a core task in the injection process among various manufacturing fields. I would like to propose.

Since the actual manufacturing process has a low defect rate and it is difficult to collect a large amount of defect data, we would like to propose a system and a control method for inspecting defects using only images of good products that can be easily collected developed in the present invention.

In addition, in consideration of actual manufacturing image characteristics and to avoid failure to detect defects located at the edges, the present invention proposes a system and a control method that performs preprocessing using the latest techniques to enhance defective samples.

In addition, the present invention proposes a method of augmenting fake defect features by adding Gaussian standard deviation as noise to non-defective samples, and a system that applies model architecture and loss function to modify the system to stabilize field application of deep learning-based quality inspection technology. and its control method.

Unlike most injection process surface inspection AIs developed to date, it uses only images of good products to conduct the entire product quality inspection pipeline through a method and system that takes into account the characteristics of the factory environment and the actual product production process. /We aim to provide users with superior performance than defect classification methodologies.

In addition, the present invention includes quality analysis of images, learning of non-defective images, and design of measurement methods for injection process products. This deep learning-based injection process product quality inspection technology is capable of detecting defects in an actual injection process environment. Problems such as failure to detect can be minimized, and we plan to expand its application to other manufacturing fields.

In addition, the present invention seeks to modify the model architecture and loss function to stabilize field application of deep learning-based quality inspection technology.

In addition, the present invention is difficult to apply in the field because each factory has different standards for determining good quality, so it is intended to use defect inspection as the final judgment without adding AI technology or training to the defect measurement stage.

In addition, the deep learning-based quality inspection technology applicable to the injection process proposed by the present invention is artificial intelligence that has already been learned from images of good products, so users do not need to build separate learning data, and it can be quickly applied at manufacturing sites such as small quantity production of multiple products. We want to make it possible.

Meanwhile, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly apparent to those skilled in the art from the description below. It will be understandable.

A deep learning-based quality inspection method for learning non-defective product data, which is an aspect of the present invention to achieve the above technical problem, includes a first step in which an input unit receives a non-defective product image data set; The pre-processing unit adjusts each image to a desired size without applying a cropping operation to each of the plurality of images included in the image data set by cutting and processing only an area at a specific location within each image. A second step of preprocessing images of the same size so that the model can learn them by applying a resizing operation and a padding operation to adjust the image size while maintaining the ratio of each image; A third step in which the control unit extracts features of good products that serve as standards for good products from the preprocessed image; A fourth step in which the control unit generates a plurality of fake defective product features by adding Gaussian noise features to the extracted good product features; A fifth step in which a discriminator learns based on at least some of the non-defective product image data set received from the control unit, the extracted non-defective product features, and the plurality of fake defective product features; A sixth step in which the input unit receives an actual image; A seventh step in which the preprocessing unit preprocesses the actual image by applying at least one of the resizing and padding operations to extract features; An eighth step in which the control unit extracts actual features from the preprocessed actual image; and a ninth step in which the discriminator determines whether the object on the actual image is a good product or a defective product through the extracted actual features, based on the content learned in the fifth step.

In addition, in the third step, the control unit inputs the preprocessed image to a backbone network capable of extracting features at multiple scales from multiple data, and extracts the features of the good product by combining the results of the outputs of a plurality of predetermined steps. can do.

In addition, in the third step, the control unit applies an average pooling operation to the second output result and the third output result of the backbone network and extracts the good product characteristics by combining the plurality of calculated results. can do.

Additionally, in the fourth step, the control unit may augment the plurality of fake defective product features by randomly selecting and adding the Gaussian noise feature from the standard deviation to the extracted non-defective product features.

Additionally, in the fifth step, the discriminator can be learned using hardness-aware soft cross entropy, a loss function that adjusts weights so that the model can focus and learn on a small number of samples less than a preset number.

In addition, the characteristics of the plurality of fake defective products include short shots, flash, sink marks, silver streak defects, cloudy surface, and weld line defects. It may include Weldline, Void, Surface Crazing/Cracking and Delamination.

Meanwhile, another aspect of the present invention for achieving the above-described technical problem is a quality inspection system based on deep learning for learning from good product data, including an input unit that receives a good product image data set; For each of the plurality of images included in the image data set, resizing (adjusting each image to a desired size without applying a cropping operation that cuts out and processes only an area at a specific location within each image) A preprocessor that preprocesses images of the same size so that the model can learn them by applying a resizing operation and a padding operation that adjusts the image size while maintaining the ratio of each image; a control unit that extracts good product features that serve as a standard for good products from the preprocessed image and adds Gaussian noise features to the extracted good product features to generate a plurality of fake defective product features; And a discriminator that learns based on at least some of the non-defective product image data set received from the control unit, the extracted non-defective product features, and the plurality of fake defective product features, wherein the input unit When receiving a real image, the preprocessor preprocesses the real image by applying at least one of the resizing and padding operations to extract features, and the control unit extracts real features from the preprocessed real image, and Based on the content learned in the fifth step, the discriminator can determine whether the object on the actual image is a good product or a defective product through the extracted actual features.

In addition, the control unit inputs the preprocessed image to a backbone network capable of extracting features at multiple scales from multiple data, and performs average pooling on the second and third output results of the backbone network. By applying the calculation, the characteristics of the good product can be extracted by combining the plurality of calculated results.

Additionally, the control unit may augment the plurality of fake defective product features by randomly selecting and adding the Gaussian noise feature from the standard deviation to the extracted non-defective product features.

Additionally, the discriminator can be learned using hardness-aware soft cross entropy, a loss function that adjusts weights so that the model can focus and learn on a small number of samples less than a preset number.

“Hardness-aware Soft Cross Entropy” according to the present invention is one of the loss functions of a deep learning model and is a method mainly used when performing classification tasks in datasets with unbalanced class distribution. This method can be designed and used to improve overall performance by encouraging the model to focus more on more difficult samples.

In order to solve the problems of the prior art, the present invention is a deep learning system and a control method applicable to quality inspection by learning only accurate and reliable manufacturing good data in product quality inspection, which is a core task in the injection process among various manufacturing fields. can be provided.

Since the actual manufacturing process has a low defect rate and it is difficult to collect a large amount of defect data, it is possible to provide a system and a control method for inspecting defects using only images of good products that can be easily collected developed by the present invention.

In addition, in consideration of actual manufacturing image characteristics, to avoid failing to detect defects located at the edges, the present invention can provide a system and a control method for performing preprocessing using the latest techniques to enhance defective samples.

In addition, the present invention proposes a method of augmenting fake defect features by adding Gaussian standard deviation as noise to non-defective samples, and a system that applies model architecture and loss function to modify the system to stabilize field application of deep learning-based quality inspection technology. and its control method can be provided.

Unlike most injection process surface inspection AIs developed to date, the present invention uses only images of good products to conduct the entire product quality inspection pipeline through methods and systems that take into account the characteristics of the factory environment and the actual product production process. It can provide users with superior performance than the good/defective classification method.

In addition, the present invention includes quality analysis of images, learning of non-defective images, and design of measurement methods for injection process products. This deep learning-based injection process product quality inspection technology is capable of detecting defects in an actual injection process environment. Problems such as failure to detect can be minimized, and can be expanded to other manufacturing fields.

Additionally, the present invention can modify the model architecture and loss function to stabilize field application of deep learning-based quality inspection technology.

In addition, the present invention is difficult to apply in the field because each factory has different standards for determining good quality, so defect inspection can be used as the final judgment without adding AI technology or training to the defect measurement stage.

In addition, the deep learning-based quality inspection technology applicable to the injection process proposed by the present invention is artificial intelligence that has already been learned from images of good products, so users do not need to build separate learning data, and it can be quickly applied at manufacturing sites such as small quantity production of multiple products. You can.

Meanwhile, the effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

Figure 1 shows a block diagram of the deep learning-based quality inspection system proposed by the present invention.
Figure 2 is a diagram for explaining an image pre-processing process related to the present invention.
Figure 3 shows an example of an AI model procedure for detecting defective areas in the injection process related to the present invention.
Figure 4 shows an example flow chart of the injection process quality inspection pipeline in relation to the present invention.
Figure 5 shows an example of a defect type related to the present invention.

Hereinafter, a preferred embodiment of the present invention will be described with reference to the drawings. In addition, the embodiments described below do not unduly limit the content of the present invention described in the claims, and it cannot be said that all of the configurations described in the present embodiments are essential as a solution to the present invention.

Deep learning-based quality inspection system

Figure 1 shows a block diagram of a deep learning-based quality inspection system applicable to the injection process proposed by the present invention.

Referring to Figure 1, the quality inspection system 100 according to the present invention includes a wireless communication unit 110, an A/V (Audio/Video) input unit 120, a user input unit 130, a sensing unit 140, and an output unit. It may include 150, memory 160, interface unit 170, control unit 180, and power supply unit 190.

However, since the components shown in FIG. 1 are not essential, a quality inspection system may be implemented with more or fewer components.

Below, we look at the above components in turn.

The wireless communication unit 110 may include one or more modules that enable wireless communication between a quality inspection system and a wireless communication system or between a quality inspection system and a network where the quality inspection system is located. For example, the wireless communication unit 110 may include a broadcast reception module 111, a mobile communication module 112, a wireless Internet module 113, a short-range communication module 114, and a location information module 115. .

The broadcast reception module 111 receives broadcast signals and/or broadcast-related information from an external broadcast management server through a broadcast channel.

The broadcast channels may include satellite channels and terrestrial channels. The broadcast management server may refer to a server that generates and transmits broadcast signals and/or broadcast-related information, or a server that receives previously generated broadcast signals and/or broadcast-related information and transmits them as a quality inspection system. The broadcast signal not only includes a TV broadcast signal, a radio broadcast signal, and a data broadcast signal, but may also include a broadcast signal in the form of a TV broadcast signal or a radio broadcast signal combined with a data broadcast signal.

The broadcast-related information may mean information related to a broadcast channel, broadcast program, or broadcast service provider. The broadcast-related information may also be provided through a mobile communication network. In this case, it can be received by the mobile communication module 112.

The broadcast-related information may exist in various forms. For example, it may exist in the form of an Electronic Program Guide (EPG) for Digital Multimedia Broadcasting (DMB) or an Electronic Service Guide (ESG) for Digital Video Broadcast-Handheld (DVB-H).

Broadcast signals and/or broadcast-related information received through the broadcast reception module 111 may be stored in the memory 160.

The mobile communication module 112 transmits and receives wireless signals with at least one of a base station, an external quality inspection system, and a server on a mobile communication network.

The wireless Internet module 113 refers to a module for wireless Internet access and may be built into or external to the quality inspection system. Wireless Internet technologies include Wireless LAN (WLAN) (Wi-Fi), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), and High Speed Downlink Packet Access (HSDPA).

The short-range communication module 114 refers to a module for short-range communication. Short range communication technologies include Bluetooth, Radio Frequency Identification (RFID), infrared data association (IrDA), Ultra-Wideband (UWB), and ZigBee.

The location information module 115 is a module for acquiring the location of the quality inspection system, and a representative example thereof is the Global Position System (GPS) module.

Referring to FIG. 1, the A/V (Audio/Video) input unit 120 is for inputting audio or video signals, and may include a camera 121, a microphone 122, etc.

The camera 121 processes image frames, such as still images or moving images, obtained by an image sensor in shooting mode. The processed image frame may be displayed on the display unit 151.

Image frames processed by the camera 121 may be stored in the memory 160 or transmitted externally through the wireless communication unit 110.

The cameras 121 are used in multiple numbers depending on the usage environment.

The microphone 122 receives external sound signals through a microphone in recording mode, voice recognition mode, etc., and processes them into electrical voice data. The processed voice data can be converted into a form that can be transmitted to a mobile communication base station through the mobile communication module 112 and output. Various noise removal algorithms may be implemented in the microphone 122 to remove noise generated in the process of receiving an external acoustic signal.

The user input unit 130 generates input data for the user to control the operation of the quality inspection system. The user input unit 130 may be composed of a key pad, a dome switch, a touch pad (static pressure/electrostatic), a jog wheel, a jog switch, etc.

In the present invention, a user can input a non-defective image data set through the user input unit 130.

Of course, in addition to receiving images of good products from the user, it is also possible to automatically input information about the result of images of good products that have been automatically machine-learned from another system.

The sensing unit 140 detects the current state of the quality inspection system, such as the open/closed state of the quality inspection system, the location of the quality inspection system, the presence or absence of user contact, the direction of the quality inspection system, and the acceleration/deceleration of the quality inspection system, and monitors the quality inspection system. Generates a sensing signal to control the operation of. Additionally, it is possible to sense whether the power supply unit 190 is supplying power and whether the interface unit 170 is connected to an external device. Meanwhile, the sensing unit 140 may include a proximity sensor 141.

The output unit 150 is for generating output related to vision, hearing, or tactile sensation, and includes a display unit 151, an audio output module 152, an alarm unit 153, a haptic module 154, and a projector module ( 155) etc. may be included.

The display unit 151 displays (outputs) information processed in the quality inspection system.

The display unit 151 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display (flexible display). It may include at least one of a display and a 3D display.

Some of these displays may be transparent or light-transmissive so that the outside can be viewed through them. This may be called a transparent display, and representative examples of the transparent display include TOLED (Transparant OLED). The rear structure of the display unit 151 may also be configured as a light-transmissive structure. Due to this structure, the user can view objects located behind the quality inspection system body through the area occupied by the display unit 151 of the quality inspection system body.

When the display unit 151 and a sensor that detects a touch operation (hereinafter referred to as a 'touch sensor') form a mutual layer structure (hereinafter referred to as a 'touch screen'), the display unit 151 is used in addition to the output device. It can also be used as an input device. The touch sensor may have the form of, for example, a touch film, a touch sheet, or a touch pad.

The touch sensor may be configured to convert a change in pressure applied to a specific part of the display unit 151 or a change in capacitance occurring in a specific part of the display unit 151 into an electrical input signal. The touch sensor can be configured to detect not only the location and area of the touch, but also the pressure at the time of touch.

When there is a touch input to the touch sensor, the corresponding signal(s) are sent to the touch controller. The touch controller processes the signal(s) and then transmits the corresponding data to the control unit 180. As a result, the control unit 180 can determine which area of the display unit 151 has been touched.

The proximity sensor 141 may be placed in an inner area of the quality inspection system surrounded by the touch screen or near the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing nearby without mechanical contact using the power of an electromagnetic field or infrared rays. Proximity sensors have a longer lifespan and are more useful than contact sensors.

Examples of the proximity sensor include a transmissive photoelectric sensor, a direct reflection photoelectric sensor, a mirror reflection photoelectric sensor, a high-frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, it is configured to detect the proximity of the pointer through a change in electric field according to the proximity of the pointer. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

The audio output module 152 may output audio data received from the wireless communication unit 110 or stored in the memory 160 in a recording mode, voice recognition mode, broadcast reception mode, etc.

The sound output module 152 also outputs sound signals related to functions performed in the quality inspection system. This sound output module 152 may include a receiver, speaker, buzzer, etc.

The alarm unit 153 outputs a signal to notify the occurrence of an event in the quality inspection system. Examples of events that occur in the quality inspection system include message reception, key signal input, and touch input. The alarm unit 153 may output a signal for notifying the occurrence of an event in another form, for example, vibration, in addition to a video signal or an audio signal. The video signal or audio signal can also be output through the display unit 151 or the audio output module 152, so they 151 and 152 may be classified as part of the alarm unit 153.

The haptic module 154 generates various tactile effects that the user can feel. A representative example of a tactile effect generated by the haptic module 154 is vibration. The intensity and pattern of vibration generated by the haptack module 154 can be controlled.

In addition to vibration, the haptic module 154 responds to stimulation such as pin arrays moving perpendicular to the contact skin surface, blowing force or suction force of air through a nozzle or inlet, grazing the skin surface, contact with an electrode, and electrostatic force. A variety of tactile effects can be generated, such as effects by reproducing hot and cold sensations using elements that can absorb heat or heat.

The haptic module 154 can not only deliver a tactile effect through direct contact, but can also be implemented so that the user can feel the tactile effect through muscle senses such as fingers or arms. Two or more haptic modules 154 may be provided depending on the configuration of the portable quality inspection system.

The projector module 155 is a component for performing an image project function using a quality inspection system, and is identical to or at least the image displayed on the display unit 151 according to a control signal from the control unit 180. Some can display different images on an external screen or wall.

Specifically, the projector module 155 generates an image to be output to the outside using a light source (not shown) that generates light (for example, laser light) for outputting an image to the outside, and the light generated by the light source. It may include an image generating means (not shown) for doing so, and a lens (not shown) for enlarging and outputting the image to the outside at a certain focal distance. Additionally, the projector module 155 may include a device (not shown) that can adjust the image projection direction by mechanically moving the lens or the entire module.

The projector module 155 can be divided into a CRT (Cathode Ray Tube) module, an LCD (Liquid Crystal Display) module, and a DLP (Digital Light Processing) module depending on the type of display device. In particular, the DLP module can be advantageous in miniaturizing the projector module 151 by enlarging and projecting an image generated by reflecting light generated from a light source on a DMD (Digital Micromirror Device) chip.

The memory unit 160 may store programs for processing and controlling the control unit 180, and has a function for temporarily storing input/output data (e.g., messages, audio, still images, videos, etc.) You can also perform . The frequency of use for each of the data may also be stored in the memory unit 160. Additionally, the memory unit 160 can store data on various patterns of vibration and sound output when a touch is input on the touch screen.

The memory 160 is a flash memory type, hard disk type, multimedia card micro type, card type memory (for example, SD or XD memory, etc.), RAM. (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic It may include at least one type of storage medium among disks and optical disks.

The interface unit 170 serves as a passageway to all external devices connected to the quality inspection system. The interface unit 170 receives data from an external device, receives power and transmits it to each component within the quality inspection system, or transmits data within the quality inspection system to an external device. For example, wired/wireless headset ports, external charger ports, wired/wireless data ports, memory card ports, ports for connecting devices equipped with identification modules, audio I/O (Input/Output) ports, A video I/O (Input/Output) port, an earphone port, etc. may be included in the interface unit 170.

The identification module is a chip that stores various information to authenticate the right to use the quality inspection system, and includes the user identification module (UIM), subscriber identity module (SIM), and universal user authentication module (Universal Subscriber). Identity Module, USIM), etc. A device equipped with an identification module (hereinafter referred to as an 'identification device') may be manufactured in a smart card format. Therefore, the identification device can be connected to the quality inspection system through the port.

When the mobile quality inspection system is connected to an external cradle, the interface unit becomes a path through which power from the cradle is supplied to the mobile quality inspection system, or various command signals input from the cradle by a user are used to control the mobile quality. Because it is an inspection system, it can be a conduit for transmission. Various command signals or the power source input from the cradle may be operated as signals for recognizing that the mobile quality inspection system is correctly mounted on the cradle.

The controller (controller, 180) typically controls the overall operation of the quality inspection system.

The control unit 180 may include a preprocessor 200.

The pre-processing unit 200 does not apply a crop operation to each of the plurality of images included in the image data set by cutting out and processing only the area at a specific location within each image, but cuts each image to a desired size. A resizing operation to adjust and a padding operation to adjust the image size while maintaining the ratio of each image can be applied.

The purpose of the operation of the preprocessor 200 is to preprocess images of the same size so that the model can learn.

Meanwhile, the control unit 180 can extract features of good products that serve as standards for good products from the preprocessed image.

Additionally, the control unit 180 may generate a plurality of fake defective product features by adding Gaussian noise features to the extracted good product features.

Meanwhile, a separate discriminator (not shown) can learn based on at least some of the good product image data set received from the control unit 180, the extracted good product features, and the plurality of fake defective product features. .

Additionally, when the input unit receives an actual image, the pre-processing unit 200 may apply at least one of the resizing and padding operations to the actual image for feature extraction.

In addition, the control unit 180 extracts actual features from the pre-processed actual image, and can determine whether the object in the actual image is a good product or a defective product through the extracted actual features based on the content learned by the discriminator.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 and supplies power necessary for the operation of each component.

Various embodiments described herein may be implemented, for example, in a recording medium readable by a computer or similar device using software, hardware, or a combination thereof.

According to hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs). In some cases, it may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions. The described embodiments may be implemented by the control unit 180 itself.

According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. Software code can be implemented as a software application written in an appropriate programming language. The software code may be stored in the memory 160 and executed by the control unit 180.

Deep learning-based quality inspection method

The quality inspection method proposed by the present invention will be described in detail based on the configuration of a deep learning-based quality inspection system applicable to the above-described injection process.

The present invention relates to an accurate and reliable deep learning/AI-based product surface quality inspection method in product quality inspection, which is a core task in the injection process among various manufacturing fields.

Unlike most injection process surface inspection AIs developed to date, the present invention uses only images of good products and developed the entire product quality inspection pipeline in consideration of the factory environment and characteristics of the actual product production process. This invention shows superior performance over the good/defective classification methodology, which is a commonly used method.

The present invention includes quality analysis of images, learning of non-defective images, and design of measurement methods for injection process products. This deep learning-based injection process product quality inspection technology does not detect defects well in the actual injection process environment. It is expected that problems such as failure to operate can be minimized, and that it can be applied to other manufacturing fields as well.

In the past, the process environment was complex, the collected images were not clear, and clear images were needed for model learning, so there was a problem that image quality analysis and filtering had to be done before model learning.

In addition, image classification AI requires a large amount of good and defective data for training, but in the actual process, it is difficult to collect a large amount of defective data because the defect rate is low, so there is a problem that there are significantly fewer cases of surface inspection AI application in the manufacturing industry. .

In contrast, the present invention has the advantage of being easy to apply to manufacturing sites by inspecting defects using only images of good products.

Additionally, when the classification criteria at the manufacturing site were incorrect, AI accuracy was unstable, making field application difficult.

However, the present invention can modify the model architecture and loss function to stabilize field application of deep learning-based quality inspection technology.

In the past, the size, length, color, etc. of each detected defect differed depending on the type, making it difficult to set a single standard for all facilities and train the network.

The present invention solves these problems, and is expected to be quickly applied to the rapidly changing manufacturing environment because it is easy for anyone to use as an AI that does not need to learn in advance the various types of defects that appear on the surface of the product, and does not require labeling.

Image data collected in the manufacturing industry is often not clear (clear), but clear data is essential for subsequent training.

Image quality analysis is performed based on the inclusion of objects in the image, brightness, contrast, etc., and only clear images are selected by removing low-quality images.

Considering the characteristics of actual manufacturing images, in order to avoid failing to detect defects located at the edges, the present invention performs preprocessing using the latest techniques to enhance defective samples.

The present invention proposes a method to enhance spurious defect features by adding Gaussian standard deviation as noise to non-defective samples.

Additionally, in the present invention, the model architecture and loss function are modified to stabilize field application of deep learning-based quality inspection technology.

Previously, although the defects were the same, the characteristics of the defects appeared slightly differently, making it difficult to train an AI model.

The present invention can determine whether a product is good or defective by learning only images of good products, without using various defective images.

In addition, since each factory has different criteria for determining good quality products, field application is difficult, so AI technology or training is not added to the defect measurement stage and defect inspection is used as the final judgment.

In addition, the deep learning-based quality inspection technology applicable to the injection process is artificial intelligence that has already been learned from images of good products, so users do not need to build separate learning data, and it is expected to be quickly applied at manufacturing sites such as small quantity production of multiple products. do.

The control unit 180 according to the present invention may include a preprocessing unit 200.

The pre-processing unit 200 does not apply a crop operation to each of the plurality of images included in the image data set by cutting out and processing only the area at a specific location within each image, but cuts each image to a desired size. A resizing operation to adjust and a padding operation to adjust the image size while maintaining the ratio of each image can be applied.

The purpose of the operation of the preprocessor 200 is to preprocess images of the same size so that the model can learn, and the control unit 180 can extract features of good products that serve as standards for good products from the preprocessed images.

Figure 2 is a diagram for explaining an image pre-processing process related to the present invention.

Referring to (a) of FIG. 2, an example is shown in which an object (1) is shown and the preprocessor 200 performs an operation of cropping only the center after resizing the object.

In this case, since the inspection image (4) is derived with the actual defective part (2) missing, the information on the defect type is ignored and an error occurs in which the product is judged to be good.

On the other hand, referring to (b) of FIG. 2, object 1 is shown, and the preprocessor 200 cuts out and processes only the area at a specific location within each image for each of the plurality of images included in the image data set. Instead of applying a crop operation, a resizing operation is applied to adjust each image to a desired size, and a padding operation is applied to adjust the image size while maintaining the ratio of each image. can do.

In this case, since the defect type (2) is preprocessed, the defective image remains included in the inspection image (5), making it possible to accurately determine good products.

Meanwhile, Figure 3 shows an example of an AI model procedure for detecting defective areas in the injection process related to the present invention.

Referring to FIG. 3, a step (S1) in which the input unit 120 receives a non-defective image data set is shown.

In addition, the preprocessor 200 does not apply a cropping operation to each of the plurality of images included in the image data set by cutting out and processing only an area at a specific location within each image, but instead cuts each image to a desired size. A preprocessing step to enable the model to learn images of the same size by applying a resizing operation to adjust the image size and a padding operation to adjust the image size while maintaining the ratio of each image (S2) is shown.

In addition, a step (S3) is shown in which the control unit 180 extracts features of good products that serve as standards for good products from the preprocessed image.

In addition, a step (S4) of generating a plurality of fake defective product features by adding Gaussian noise features to the good product features extracted by the control unit 180 is shown.

In addition, a step (S5) in which the discriminator learns based on at least some of the good product image data set received from the control unit 180, the extracted good product features, and the plurality of fake defective product features is shown. .

Thereafter, the control unit 180 extracts actual features from the pre-processed actual image and determines whether the object in the actual image is a good product or a defective product through the extracted actual features based on the contents learned by the discriminator (S8). It is shown.

Hereinafter, the method according to the present invention will be described in more detail with reference to FIG. 4.

Figure 4 shows an example flowchart of the injection process quality inspection pipeline in relation to the present invention.

Referring to FIG. 4, the first step (S1) in which the input unit 120 receives a non-defective image data set is performed.

Afterwards, the pre-processing unit 200 does not apply a cropping operation to each of the plurality of images included in the image data set by cutting out and processing only the area at a specific location within each image, but cuts each image to a desired size. A preprocessing step to enable the model to learn images of the same size by applying a resizing operation to adjust the image size and a padding operation to adjust the image size while maintaining the ratio of each image (S2) Perform.

In addition, a step (S3) is performed in which the control unit 180 extracts features of good products that serve as standards for good products from the preprocessed image.

In step S3, the control unit 180 inputs the preprocessed image to a backbone network capable of extracting features at multiple scales from multiple data, and extracts the features of the good product by combining the results of the outputs of a plurality of pre-designated steps. You can.

Additionally, in step S3, the control unit 180 applies an average pooling operation to the second and third output results of the backbone network and combines the plurality of calculated results to extract the characteristics of the good product. .

Here, the backbone network represents the core structure of the deep learning model and is an important element used in various computer vision tasks such as image classification, object detection, and segmentation. The backbone network helps transform input data into a high-dimensional feature space, extract abstracted information, and perform the final task.

Backbone networks are mainly used to process high-dimensional data such as image data, and are deep neural networks composed of multiple layers to learn complex patterns and features of such data. These neural networks extract high-level information by gradually converting input data into abstract features.

The choice of backbone network may vary depending on the given task. For example, in computer vision tasks, various backbone network architectures are used, such as VGG, ResNet, Inception, and MobileNet. These network architectures are designed for tasks such as image classification, object detection, and segmentation, and each has the ability to extract different features.

In summary, backbone networks are the core structure of deep learning models and are useful in various computer vision tasks by converting input data into abstract features.

Thereafter, the control unit 180 performs a step (S4) of generating a plurality of fake defective product features by adding Gaussian noise features to the extracted defective product features.

In step S4, the control unit 180 may augment the plurality of fake defective product features by randomly selecting and adding the Gaussian noise feature from the standard deviation to the extracted non-defective product features.

In addition, a discriminator (not shown) performs a learning step (S5) based on at least some of the good product image data set received from the control unit, the extracted good product features, and a plurality of fake defective product features. .

In step S5, the discriminator can be trained using hardness-aware soft cross entropy, a loss function that adjusts the weights so that the model can learn by focusing on a small number of samples below a preset number.

“Hardness-aware Soft Cross Entropy” is one of the loss functions of deep learning models and is a method mainly used when performing classification tasks in datasets with unbalanced class distribution. This method is designed to improve overall performance by encouraging the model to focus more on more difficult samples.

A typical cross-entropy loss function tries to minimize prediction error by treating all classes with equal weight. However, when the imbalance between classes is severe, classes with a large number of samples may be considered relatively less important. This may result in poor performance for minority classes.

“Hardness-aware Soft Cross Entropy” was designed to solve this problem. This method calculates the loss by applying soft weights considering the 'difficulty' of the sample. In other words, the model is trying to improve the performance of that class by giving greater weight to uncertain or difficult samples. By doing this, the model can achieve more balanced results even with an unbalanced class distribution.

This approach helps improve the model's ability and is one of the important techniques that can improve classification performance in imbalanced datasets.

“Hardness-aware Soft Cross Entropy” according to the present invention is one of the loss functions of a deep learning model and is a method mainly used when performing classification tasks in datasets with unbalanced class distribution. This method can be designed and used to improve overall performance by encouraging the model to focus more on more difficult samples.

In addition, the characteristics of the plurality of fake defective products include short shots, flash, sink marks, silver streak defects, cloudy surface, and weld line defects. It may include Weldline, Void, Surface Crazing/Cracking and Delamination.

Figure 5 shows an example of a defect type related to the present invention.

Referring to Figure 5, (a) Short shot, (b) Flash, (c) Sink mark, (d) Silver streak, (e) Cloudy Specific examples of Cloudy surface, (f) Weldline, (g) Void, (h) Crazing / Cracking, and (i) Delamination are shown. .

Various types of defects appear on the surface of the product, and their characteristics can be divided into 9 types as shown in Figure 5.

Most defect types are short shot, flash, sink mark, silver streak defect, cloudy surface, and weld line defect (as shown in Figure 5). It is divided into Weldline, Void, Surface Crazing/Cracking, and Delamination, and the characteristics of each defect type are distinctly different from the background, color, shape, and appearance.

Meanwhile, after setting the above contents, a step (S6) in which the input unit 120 receives the actual image proceeds, and the preprocessor 200 applies at least one of resizing and padding operations to the actual image for feature extraction. It is preprocessed (S7).

In addition, when the control unit 180 extracts actual features from the preprocessed actual image (S8), the discriminator determines whether the object in the actual image is a good product or a defective product through the extracted actual features based on the learned content. Step (S9) is performed.

That is, in the present invention, in the image preprocessing process, resizing and padding of the input image are performed so that the model can learn images of the same size.

Additionally, in feature extraction from good product images, good product data can be inserted into the backbone network to extract features in the second and third layers.

In addition, the characteristics of a good product image can be obtained by combining the results of the average pooling operation of the outputs of the second and third stages of the backbone network.

Additionally, in Fake Defect Feature Augmentation, fake defect features can be created by adding Gaussian noise to the characteristics of a good product image.

Additionally, for each sample, Gaussian noise can be randomly selected from the standard deviation to augment virtual defect characteristic data.

Additionally, in discriminator learning, real images and fake defective images are delivered to the discriminator.

At this time, the discriminator can be learned using hardness-aware soft cross entropy as the loss function.

In the actual process, it is difficult to collect a large amount of defect data because the defect rate is low, and the characteristics of each defect type appear slightly differently, making it difficult to train an AI model. However, in the present invention, it learns only from images of good products that can be easily collected to determine whether the product is good or defective. can be determined.

In addition, by modifying the model architecture of preprocessing, learning strategy, and loss function to stabilize the field application of deep learning-based quality inspection technology, workers can create a network with optimal and consistent performance even if learning is stopped at any time.

The present invention is an artificial intelligence model learned only from images of good products and does not need to learn in advance the various types of defects that appear on the surface of the product. Users do not need to build separate learning data, and it can be used quickly at various manufacturing sites such as small quantity production of multiple products. I hope it can be applied.

Effects according to the present invention

In order to solve the problems of the prior art, the present invention is a deep learning system and a control method applicable to quality inspection by learning only accurate and reliable manufacturing good data in product quality inspection, which is a core task in the injection process among various manufacturing fields. can be provided.

Since the actual manufacturing process has a low defect rate and it is difficult to collect a large amount of defect data, it is possible to provide a system and a control method for inspecting defects using only images of good products that can be easily collected developed by the present invention.

In addition, in consideration of actual manufacturing image characteristics, to avoid failing to detect defects located at the edges, the present invention can provide a system and a control method for performing preprocessing using the latest techniques to enhance defective samples.

In addition, the present invention proposes a method of augmenting fake defect characteristics by adding Gaussian standard deviation as noise to non-defective samples, and a system that applies model architecture and loss function to modify the system to stabilize field application of deep learning-based quality inspection technology. and its control method can be provided.

Unlike most injection process surface inspection AIs developed to date, the present invention uses only images of good products to conduct the entire product quality inspection pipeline through methods and systems that take into account the characteristics of the factory environment and the actual product production process. It can provide users with superior performance than the good/defective classification method.

In addition, the present invention includes quality analysis of images, learning of non-defective images, and design of measurement methods for injection process products. This deep learning-based injection process product quality inspection technology is capable of detecting defects in an actual injection process environment. Problems such as failure to detect can be minimized, and can be expanded to other manufacturing fields.

Additionally, the present invention can modify the model architecture and loss function to stabilize field application of deep learning-based quality inspection technology.

In addition, the present invention is difficult to apply in the field because each factory has different standards for determining good quality, so defect inspection can be used as the final judgment without adding AI technology or training to the defect measurement stage.

In addition, the deep learning-based quality inspection technology applicable to the injection process proposed by the present invention is artificial intelligence that has already been learned from images of good products, so users do not need to build separate learning data, and it can be quickly applied at manufacturing sites such as small quantity production of multiple products. You can.

In this specification, we identify the differences between academic research and applications in the manufacturing industry and introduce an accurate and reliable anomaly detection and detection system to respond to the differences.

Compared to previous research, the product according to the present invention was designed to intensively solve the difficulties of quality inspection in actual industry, users do not need to build separate learning data, and it can be quickly applied at various manufacturing sites such as small quantity production of multiple products. It is expected that there will be.

Additionally, the system according to the present invention is expected to increase the efficiency of the manufacturing process by more effectively detecting defects in the actual manufacturing industry.

Meanwhile, the effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

The embodiments of the present invention described above can be implemented through various means. For example, embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

In the case of hardware implementation, the method according to embodiments of the present invention uses one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), and Programmable Logic Devices (PLDs). , can be implemented by FPGAs (Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

In the case of implementation by firmware or software, the method according to embodiments of the present invention may be implemented in the form of a module, procedure, or function that performs the functions or operations described above. Software code can be stored in a memory unit and run by a processor. The memory unit is located inside or outside the processor and can exchange data with the processor through various known means.

A detailed description of preferred embodiments of the invention disclosed above is provided to enable any person skilled in the art to make or practice the invention.

Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the scope of the present invention.

For example, a person skilled in the art may use each configuration described in the above-described embodiments by combining them with each other.

Accordingly, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The present invention may be embodied in other specific forms without departing from the spirit and essential features of the present invention.

Accordingly, the above detailed description should not be construed as restrictive in all respects and should be considered illustrative.

The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

The present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

In addition, claims that do not have an explicit reference relationship in the patent claims can be combined to form an embodiment or included as a new claim through amendment after filing.

Claims (10)

A first step in which the input unit receives a good product image data set;
The pre-processing unit adjusts each image to a desired size without applying a cropping operation to each of the plurality of images included in the image data set by cutting and processing only an area at a specific location within each image. A second step of preprocessing images of the same size so that the model can learn them by applying a resizing operation and a padding operation to adjust the image size while maintaining the ratio of each image;
A third step in which the control unit extracts features of good products that serve as standards for good products from the preprocessed image;
A fourth step in which the control unit generates a plurality of fake defective product features by adding Gaussian noise features to the extracted good product features;
A fifth step in which a discriminator learns based on at least some of the non-defective product image data set received from the control unit, the extracted non-defective product features, and the plurality of fake defective product features;
A sixth step in which the input unit receives an actual image;
A seventh step in which the preprocessing unit preprocesses the actual image by applying at least one of the resizing and padding operations to extract features;
An eighth step in which the control unit extracts actual features from the preprocessed actual image; and
A ninth step in which the discriminator determines whether the object in the actual image is a good product or a defective product through the extracted actual features based on the content learned in the fifth step,

In the third step, the control unit,
The preprocessed image is input to a backbone network capable of extracting features at multiple scales from multiple data, and the results of the outputs of a plurality of pre-designated steps are combined to extract the good product features,

In the third step, the control unit,
For each of the second and third output results of the backbone network, an average pooling operation is applied to calculate the average value of a plurality of points and replace it with one representative value, and the average pooling operation is applied to the average pooling operation to which the average pooling operation is applied. Combining the second and third output results to extract the features of the good product,

In the fourth step, the control unit,
Augmenting the plurality of fake defective product features by adding the Gaussian noise features generated using a randomly selected standard deviation to the extracted good product features,

In the fifth step, the discriminator is,
A quality inspection method based on deep learning for learning good quality data, which is learned using hardness-aware soft cross entropy, a loss function that adjusts the weights so that the model can focus and learn on a small number of samples below a preset number.
delete delete delete delete ◈Claim 6 was abandoned upon payment of the setup registration fee.◈ According to clause 1,
The characteristics of the plurality of fake defective products are:
Short shot, Flash, Sink mark, Silver streak, Cloudy surface, Weldline, Void, Surface crack/cracking Quality inspection method based on deep learning learning from good product data, including (crazing / cracking) and layer separation (Delamination).
◈Claim 7 was abandoned upon payment of the setup registration fee.◈ An input unit that receives a good product image data set;
For each of the plurality of images included in the image data set, resizing (adjusting each image to a desired size without applying a cropping operation that cuts out and processes only an area at a specific location within each image) A preprocessor that preprocesses images of the same size so that the model can learn them by applying a resizing operation and a padding operation that adjusts the image size while maintaining the ratio of each image;
Extract features of good products that serve as standards for good products from the preprocessed image,
a control unit that generates a plurality of fake defective product features by adding Gaussian noise features to the extracted good product features; and
A discriminator that learns based on at least some of the non-defective product image data set received from the control unit, the extracted non-defective product features, and the plurality of fake defective product features,

When the input unit receives an actual image,
The preprocessor preprocesses the actual image by applying at least one of the resizing and padding operations to extract features,
The control unit extracts actual features from the preprocessed actual image,
Based on the learned content, the discriminator determines whether the object in the actual image is a good product or a defective product through the extracted actual features,

The control unit,
Input the preprocessed image into a backbone network capable of extracting features at multiple scales from multiple data,
For each of the second and third output results of the backbone network, an average pooling operation is applied to calculate the average value of a plurality of points and replace it with one representative value,
Extracting the good product features by combining the second and third output results to which the average pooling operation is applied,

The control unit,
Augmenting the plurality of fake defective product features by adding the Gaussian noise features generated using a randomly selected standard deviation to the extracted good product features,

The discriminator is,
A deep learning-based quality inspection system that learns from good quality data that learns using hardness-aware soft cross entropy, a loss function that adjusts the weights so that the model can focus and learn on a small number of samples below a preset number.
delete delete ◈Claim 10 was abandoned upon payment of the setup registration fee.◈ According to clause 7,
The characteristics of the plurality of fake defective products are:
Short shot, Flash, Sink mark, Silver streak, Cloudy surface, Weldline, Void, Surface crack/cracking Deep learning-based quality inspection system that learns from good product data, including (crazing / cracking) and layer separation (Delamination).