KR102545144B1 - Apparatus and method for providing defective product inspection solution based on deep learning algorithm - Google Patents

Abstract

An apparatus for providing a defective product inspection solution, including a communication unit that communicates with at least one client server, and a processor, wherein the processor, when a signal requesting data collection is received from the at least one client server through the communication unit, at least one transmits a signal for requesting image capture of the test target to the client server of the image capture request, and when an image of the test target captured from at least one client server is received through the communication unit in response to the image capture request, the received image of the test target Performs data pre-processing according to a specified method for the pre-processed image data, generates a deep learning artificial intelligence model based on a deep learning algorithm according to a defect type based on the pre-processed image data, and transmits the generated artificial intelligence model through the communication unit to at least one client It is characterized in that it is set to provide to the server.

Description

DEFECTIVE PRODUCT INSPECTION SOLUTION BASED ON DEEP LEARNING ALGORITHM}

The present disclosure relates to a method for providing a defective product inspection solution and a system implementing the same. More specifically, the present disclosure relates to a method for providing a defective product inspection solution based on a deep learning algorithm and a system implementing the same.

In the conventional production field, inspection of defects in products has been dependent on judgment using the eyes of inspectors. Existing visual inspection is expensive and it is difficult to maintain a constant inspection quality.

In order to solve the problem of visual inspection, defective product inspection solutions using machine learning techniques have been proposed. Machine learning techniques may be based on rule-based algorithms. In the case of a rule-based algorithm, a manager must set clear standards such as dimensions, and there are significant limitations in inspecting atypical defects.

Therefore, a method of providing a defective product inspection solution showing optimized inspection performance even for atypical defects has been devised.

Korean Registered Patent Publication No. 10-1237998 (registration date: February 21, 2013)

An embodiment disclosed in the present disclosure aims to provide an artificial intelligence model based on a deep learning algorithm in response to a data collection request of a client.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

An apparatus for providing a defective product inspection solution according to an aspect of the present disclosure for achieving the above technical problem includes a communication unit that communicates with at least one client server, and a processor, wherein the processor includes at least one client server through the communication unit. When a signal requesting data collection is received from the at least one client server, a signal requesting capturing an image of the inspection target is transmitted, and in response to the image capturing request, the captured image is captured from the at least one client server through the communication unit. When the image of the inspection target is received, data pre-processing is performed on the received inspection target image according to a designated method, and based on the pre-processed image data, a deep learning artificial intelligence model based on a deep learning algorithm according to the type of defect is created. , It can be set to provide the artificial intelligence model generated through the communication unit to at least one client server.

In addition to this, a computer program stored in a computer readable recording medium for execution to implement the present disclosure may be further provided.

In addition to this, a computer readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

According to the above-described problem solving means of the present disclosure, an artificial intelligence model generated using an image captured by a client server is provided, thereby providing an effect of providing a defective product inspection solution optimized for a client.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 illustrates a system for providing a defective product inspection solution according to an embodiment of the present disclosure.
2 is a block diagram illustrating a master server and a client server according to an embodiment of the present disclosure.
3 is a flowchart illustrating a method of providing a defective product inspection solution according to an embodiment of the present disclosure.
4 illustrates types of deep learning algorithms according to an embodiment of the present disclosure.
5 is a flowchart illustrating a method of providing a defective product inspection solution including hardware, according to an embodiment of the present disclosure.
6A is a block diagram illustrating a cloud server according to an embodiment of the present disclosure.
6B illustrates a defective product inspection solution method using a database of a cloud server according to an embodiment of the present disclosure.
7 illustrates an artificial intelligence model composed of a plurality of groups according to an embodiment of the present disclosure.
8 illustrates production environment monitoring through CCTV according to an embodiment of the present disclosure.

Like reference numbers designate like elements throughout this disclosure. The present disclosure does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the present disclosure belongs is omitted. The term 'unit, module, member, or block' used in the specification may be implemented as software or hardware, and according to embodiments, a plurality of 'units, modules, members, or blocks' may be implemented as one component, It is also possible that one 'part, module, member, block' includes a plurality of components.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case of being directly connected but also the case of being indirectly connected, and indirect connection includes being connected through a wireless communication network. do.

In addition, when a certain component is said to "include", this means that it may further include other components without excluding other components unless otherwise stated.

Throughout the specification, when a member is said to be located “on” another member, this includes not only a case where a member is in contact with another member, but also a case where another member exists between the two members.

Terms such as first and second are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

Expressions in the singular number include plural expressions unless the context clearly dictates otherwise.

In each step, the identification code is used for convenience of description, and the identification code does not explain the order of each step, and each step may be performed in a different order from the specified order unless a specific order is clearly described in context. there is.

Hereinafter, the working principle and embodiments of the present disclosure will be described with reference to the accompanying drawings.

In this specification, the 'apparatus according to the present disclosure' includes all various devices capable of providing results to users by performing calculation processing. For example, a device according to the present disclosure may include a computer, a server device, and a portable terminal, or may be in any one form.

Here, the computer may include, for example, a laptop computer, a desktop computer, a laptop computer, a tablet PC, a slate PC, and the like equipped with a web browser.

The server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

The portable terminal is, for example, a wireless communication device that ensures portability and mobility, and includes a Personal Communication System (PCS), a Global System for Mobile communications (GSM), a Personal Digital Cellular (PDC), a Personal Handyphone System (PHS), and a PDA. (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, smart phone ) and wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-devices (HMDs). can include

1 illustrates a system for providing a defective product inspection solution according to an embodiment of the present disclosure.

In one embodiment, the defective product inspection solution providing system 10 may include a master server 100 , a client server 110 , and/or a network 120 . Master server 100 may communicate with client server 110 via network 120 . In one embodiment, the defective product inspection solution providing system 10 may include a plurality of client servers (not shown).

In one embodiment, the master server 100 may create a defective product inspection solution according to the request of the client server 110 and provide it to the client server 110 .

In one embodiment, the client server 110 may be a server located at a production site of a client (eg, a factory or a customer company) performing defective product inspection. In another embodiment, the client server 110 may be a server that manages inspection of defective products at a production site at a location physically different from the production site.

In one embodiment, the client server 110 may request data collection from the master server 100 . The master server 100 may request image capture from the client server 110 . The master server 100 may request image capture of the examination target to the client server 110 . For example, the inspected object may be a product (e.g. bolts, nuts, chassis) produced at the client's production site. Products may include good and defective products.

In one embodiment, the client server 110 may capture an image of an examination target and transmit the captured image to the master server 100 . The master server 100 may perform data pre-processing on the inspection target image according to a designated method. For example, the designated method may be a supervised method or an unsupervised method.

In one embodiment, the master server 100 may generate a deep learning artificial intelligence model based on a deep learning algorithm according to a defect type based on preprocessed image data. The type of defect may be based on the material of the inspection target, the surface pattern, and the like. The defect types of this disclosure can be formal or atypical.

In one embodiment, the master server 100 may provide the generated artificial intelligence model to the client server 110 .

In one embodiment, the defective product inspection solution providing system 10 may further include a cloud server 130 . The artificial intelligence model generated by the master server 100 may be provided through the cloud server 130 in the form of software as a service (SAAS). At least one client server (eg, the client server 110) may receive a defective product inspection solution through the cloud server 130.

2 is a block diagram illustrating a master server and a client server according to an embodiment of the present disclosure. The components shown in FIG. 2 are not essential to implement the master server 100 and / or client server 110 according to the present disclosure, so the master server 100 and / or client server ( 110) may have more or fewer components than those listed above.

In one embodiment, the master server 100 may include a processor 200 and a communication unit 210 . The processor 200 may include a data preprocessor 220 and/or a deep learning artificial intelligence model generator 230 .

In one embodiment, the processor 200 may perform the above operations using a memory for storing data for an algorithm or a program for reproducing the algorithm for controlling the operation of components of the master server 100, and the data stored in the memory. It may be implemented with at least one processor (not shown) that performs. In this case, the memory and the processor may be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

In one embodiment, among the components, the communication unit 210 may include one or more components enabling communication with an external device, for example, a broadcast receiving module, a wired communication module, a wireless communication module, and a short-distance communication module. It may include at least one of a communication module and a location information module.

Wired communication modules include not only various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, but also USB (Universal Serial Bus) ), high definition multimedia interface (HDMI), digital visual interface (DVI), recommended standard 232 (RS-232), power line communication, or plain old telephone service (POTS).

In addition to the WiFi module and the WiBro module, wireless communication modules include global system for mobile communication (GSM), code division multiple access (CDMA), wideband code division multiple access (WCDMA), and universal mobile telecommunications system (UMTS). ), time division multiple access (TDMA), long term evolution (LTE), and a wireless communication module supporting various wireless communication schemes such as 4G, 5G, and 6G.

The short-range communication module is for short-range communication, and includes Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and Near Field Communication (NFC). Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and wireless USB (Wireless Universal Serial Bus) technologies may be used to support short-distance communication.

In one embodiment, the data pre-processing unit 220 may pre-process the image of the inspection target according to a predetermined method. For example, the predetermined method may be a supervised learning method or an unsupervised learning method. The predetermined method may be based on a deep learning algorithm according to a defect type. The deep learning algorithm will be described later.

In one embodiment, the deep learning artificial intelligence model generating unit 230 may generate a deep learning artificial intelligence model using a preprocessed image. The deep learning artificial intelligence model may use at least one deep learning algorithm. In one embodiment, the deep learning artificial intelligence model may include a first group and a second group. Each of the first group and the second group may include a plurality of artificial intelligence models. Configurations of the first group and the second group may be changed according to the performance of the artificial intelligence model in the client server 110 .

In addition, the processor may control any one or a combination of the components described above in order to implement various embodiments according to the present disclosure described in FIGS. 2 through N below on the present device.

In one embodiment, the client server 110 may include a processor 250, a communication unit 260, and/or an image acquisition unit 270. Descriptions of the processor 250 and the communication unit 260 may be referenced by descriptions of the processor 200 and the communication unit 210, respectively.

In one embodiment, the image acquisition unit 270 may include a camera for photographing the inspection target. A camera processes an image frame such as a still image or a moving image obtained by an image sensor in a photographing mode. The processed image frame may be displayed on a display unit or stored in a memory.

On the other hand, when there are a plurality of cameras, they can be arranged to form a matrix structure, and a plurality of image information having various angles or focal points can be input through the cameras forming the matrix structure, and the cameras form a three-dimensional image. It may be arranged in a stereo structure so as to obtain left and right images for realizing a stereoscopic image.

In one embodiment, when a signal requesting data collection is received from the client server 110 through the communication unit 210, the processor 200 of the master server 100 captures an image of the test target with the client server 110. transmits a signal requesting an image capture, and when an image of an examination target photographed from the client server 110 is received through the communication unit 210 in response to the image capture request, according to a method specified for the received examination target image It is set to perform data preprocessing, generate a deep learning artificial intelligence model based on a deep learning algorithm according to the type of defect based on the preprocessed image data, and provide the generated artificial intelligence model to the client server through the communication unit 210. can For example, the captured image may include an image of a product being inspected by a client of the client server 110, an image being used by the client in a rule-based algorithm, and an augmented image. For example, the designated method may include a supervised learning method and an unsupervised learning method.

At least one component may be added or deleted corresponding to the performance of the components shown in FIG. 2 . In addition, it will be easily understood by those skilled in the art that the mutual positions of the components may be changed corresponding to the performance or structure of the system.

Meanwhile, each component shown in FIG. 2 means software and/or hardware components such as a Field Programmable Gate Array (FPGA) and Application Specific Integrated Circuit (ASIC).

3 is a flowchart illustrating a method of providing a defective product inspection solution according to an embodiment of the present disclosure.

In step S300, the client server 110 may request data collection from the master server 100. Hereinafter, data requested by the client server 110 may be referred to as reference data.

In step S305, the master server 100 may request image capture to the client server 110.

In step S310, the client server 110 may capture an image of the examination target. In one embodiment, the inspection target may be a product produced at a client's production site. Products may include good and defective products.

In step S315, the client server 110 may transmit the captured image to the master server 100. In one embodiment, the captured image may include an image of the product being inspected by the client, an image the client is using in a rule-based algorithm, and an augmented image.

In step S320, the master server 100 may perform data pre-processing according to a predetermined method using the received image. In one embodiment, the designated method may be a supervised learning method or an unsupervised learning method. For example, in the case of a map method, a person in charge of the client server 130 may perform a labeling task for a photographed image.

In step S325, the master server 100 may generate a deep learning artificial intelligence model according to the defect type based on the preprocessed image. The defect type may be determined based on the material and surface pattern of the inspection target. The defect types of this disclosure can be formal or atypical. The deep learning artificial intelligence model may be based on at least one deep learning algorithm according to FIG. 4 .

In step S330, the master server 100 may provide the generated artificial intelligence model to the client server 110. The client server 110 may perform defective product inspection based on the provided artificial intelligence model.

4 illustrates types of deep learning algorithms according to an embodiment of the present disclosure.

In one embodiment, the deep learning algorithm may include supervised learning (400) and unsupervised learning (410).

In one embodiment, supervised learning 400 can be categorized into a first algorithm 402 , a second algorithm 404 , and a third algorithm 406 .

In one embodiment, the first algorithm 402 may be a deep learning algorithm. For example, deep learning algorithms in the field of image processing may include segmentation, object detection, and classification algorithms. The segmentation algorithm can be used to recognize the boundary and inner shape of an image. An object detection algorithm can be used to recognize the center value of an image and the bounding box of a rectangle. Classification algorithms may be used to identify categories of images.

In one embodiment, the second algorithm 404 may be a learning algorithm for improving detectability. For example, the second algorithm 404 may include multi-channel image, image comparison, and GAN-based synthetic image generation algorithms. A multi-channel image algorithm can be understood as an algorithm that improves detection performance by fusing image data to amplify a learning amount. The image comparison algorithm can be understood as an algorithm that learns by focusing on differences between good and bad images. A GAN-based synthetic image generation algorithm may be understood as an algorithm that randomly places a defective region on a good sample image to generate and learn a similar defective image.

In one embodiment, the third algorithm 406 may be an algorithm for improving learning productivity. For example, the third algorithm 406 may include auto-labeling and labeling assistant algorithms. An auto-labeling algorithm may be understood as an algorithm that pre-labels a region that is considered to be defective by using a pre-processing technique. The labeling assistant algorithm can be understood as an algorithm that can automatically generate labels for objects by manually specifying only a small number of points during labeling.

In one embodiment, unsupervised learning 410 may include an unsupervised anomaly detection algorithm. The unsupervised anomaly detection algorithm may be an algorithm that detects defects with a model learned using only good samples.

5 is a flowchart illustrating a method of providing a defective product inspection solution including hardware, according to an embodiment of the present disclosure.

In step S500, the client server 110 may request data collection and hardware from the master server 100.

In step S510, the master server 100 may determine hardware for detecting defective products based on the defective type.

In step S520, the master server 100 may provide the artificial intelligence model and hardware to the client server 110. The artificial intelligence model may be created according to the embodiment of FIG. 3 . For example, the hardware may include optics loaded with deep learning artificial intelligence models. For example, hardware can be designed to be optimized for artificial intelligence models.

In one embodiment, the master server 100 maximizes the performance of the artificial intelligence model by supplying the deep learning artificial intelligence model and hardware to the client server 110 in a turn-key manner, as in step S520. We can provide a solution optimized for the client.

6A is a block diagram illustrating a cloud server according to an embodiment of the present disclosure.

In one embodiment, the cloud server 130 may include a processor 600, a communication unit 610, and/or a database 620. Descriptions of the processor 600 and the communication unit 610 may be referenced by the description of the processor 200 and the communication unit 210 of FIG. 2 , respectively.

In one embodiment, the cloud server 130 may communicate with the master server 100 and a plurality of client servers (eg, the client server 110) through the communication unit 610. In one embodiment, the master server 100 may provide the artificial intelligence model to a plurality of client servers in the form of software as a service (Saas) through the cloud server 130 . The master server 100 builds the database 620 in the cloud server 130, and connects the artificial intelligence model provided to the plurality of client servers with the plurality of reference data requested by the plurality of client servers to form the database 620. ) can be stored. For example, the plurality of reference data may refer to data related to defective products requested by the plurality of client servers to the master server 100 to inspect defective products. For example, the plurality of reference data may include product images and/or defect type data captured by a plurality of client servers.

6B illustrates a defective product inspection solution method using a database of a cloud server according to an embodiment of the present disclosure.

In one embodiment, a plurality of reference data may be stored in the database 620 in association with an artificial intelligence model optimized for each of the plurality of reference data. In one embodiment, the first artificial intelligence model 655 is the first reference data 650, the second artificial intelligence model 665 is the second reference data 660, the third artificial intelligence model 675 is It may be an artificial intelligence model optimized for each of the third reference data 670 .

In one embodiment, the master server 100 may receive a data collection request from the new client server 110 . The master server 100 may calculate a similarity between the plurality of reference data 650 , 660 , and 670 stored in the database 620 of the cloud server 130 and data requested by the new client server 110 . Similarity can be calculated based on Equation 1.

The master server 100 may provide the new client server 110 with an artificial intelligence model optimized for reference data having a high similarity to data requested by the new client server 110 .

In another embodiment, the master server 100 may receive a data collection request from the new client server 110 . The master server 100 uses at least one artificial intelligence model (eg, the first artificial intelligence model 655, the second artificial intelligence model 655) stored in the database 620 using the image data of the examination target received from the new client server 110. The artificial intelligence model 665 and the third artificial intelligence model 675 may be trained respectively.

The master server 100 may identify an artificial intelligence model having the highest performance among at least one learned artificial intelligence model, and additionally train the identified artificial intelligence model using preprocessed data. The master server 100 may provide the artificial intelligence model to the new client server 110 if the performance of the artificial intelligence model that has been trained is equal to or greater than a reference value.

7 illustrates an artificial intelligence model composed of a plurality of groups according to an embodiment of the present disclosure.

In one embodiment, the artificial intelligence model provided by the master server 100 to the client server 110 may include a plurality of groups (first group and second group). In one embodiment, the first group may include a specified number of artificial intelligence models 700 , 710 , and 720 . The second group may include a specified number of AI models 750, 760, and 770.

In one embodiment, the client server 110 measures the performance of the artificial intelligence models 700, 710, and 720 included in the first group and demotes the artificial intelligence model with low performance to the second group based on a predetermined criterion. can make it In one embodiment, the client server 110 measures the performance of the artificial intelligence models 750, 760, and 770 included in the second group and promotes the artificial intelligence model with high performance to the first group based on predetermined criteria. can make it For example, the predetermined criterion may be a performance indicator set by a manager of the client server 110 .

8 illustrates production environment monitoring through CCTV according to an embodiment of the present disclosure.

In one embodiment, the client server 110 may be electrically connected to a closed circuit television (CCTV) 800. The CCTV may capture images of the client's production environment 810 .

In one embodiment, the master server 100 may perform at least one of object detection, action recognition, and posture measurement for workers in the production environment 810 based on the CCTV image. In one embodiment, the master server 100 may track the working and resting time of the workers to calculate the optimal number of people data required for production.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing instructions executable by a computer. Instructions may be stored in the form of program codes, and when executed by a processor, create program modules to perform operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media in which instructions that can be decoded by a computer are stored. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device, and the like.

As above, the disclosed embodiments have been described with reference to the accompanying drawings. Those skilled in the art to which the present disclosure pertains will understand that the present disclosure may be implemented in a form different from the disclosed embodiments without changing the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims (10)

a communication unit that communicates with at least one client server; and
When a signal requesting data collection is received from the at least one client server through the communication unit, a signal requesting capturing an image of an examination target is transmitted to the at least one client server, and in response to the image capturing request , When an image of an inspection target photographed from the at least one client server is received through the communication unit, data pre-processing is performed on the received inspection target image according to a designated method, and a defect is obtained based on the pre-processed image data. A processor configured to generate a deep learning artificial intelligence model based on a deep learning algorithm according to a type and provide the generated artificial intelligence model to the at least one client server through the communication unit;
the processor is further configured to provide hardware to the client based on the defect type;
The artificial intelligence model is provided through a cloud server based on Saas (Software as a Service),
The processor is further configured to build a database in the cloud server, wherein data requested by the at least one client server is stored as a plurality of reference data, and an artificial intelligence model optimized for the plurality of reference data. This is stored in association with the plurality of reference data,
The processor, when data collection is requested from the new client server, calculates the similarity of the requested data of the new client server and the plurality of reference data stored in the database, and the calculated similarity among the plurality of reference data Further configured to provide the new client server with an artificial intelligence model optimized for high reference data,
Defective product inspection solution provider. According to claim 1,
The captured image includes an image of a product being inspected by a client of the client server, an image being used by the client in a rule-based algorithm, and an augmented image.
Defective product inspection solution provider. According to claim 2,
The designated method includes a supervised learning method and an unsupervised learning method,
Defective product inspection solution provider. delete delete delete delete According to claim 1,
The artificial intelligence model provided by the processor includes a first group and a second group,
The first group and the second group each include a specified number of artificial intelligence models,
Among the artificial intelligence models of the first group, an artificial intelligence model having low performance based on a predetermined criterion is demoted to the second group by the client server;
Among the artificial intelligence models of the second group, an artificial intelligence model having high performance based on the preset criteria is promoted to the first group by the client server.
Defective product inspection solution provider. According to claim 1,
The client server is electrically connected to a closed circuit television (CCTV),
the processor,
Further configured to perform at least one of object detection, action recognition, and posture measurement for workers in the production environment of the client server based on the image taken by the CCTV,
Defective product inspection solution provider. A method for providing a defective product inspection solution performed by a master server communicating with at least one client server,
requesting capturing of an image of an examination target to the at least one client server when data collection is requested from the at least one client server;
In response to the request, when an image of an examination target photographed from the at least one client server is received, performing data pre-processing according to a designated method on the received examination target image;
generating a deep learning artificial intelligence model based on a deep learning algorithm according to a defect type based on data of the preprocessed image; and
Including; providing the generated artificial intelligence model to the at least one client server,
The master server provides hardware to the client based on the defect type;
The artificial intelligence model is provided through a cloud server based on Saas (Software as a Service),
The master server builds a database in the cloud server, data requested by the at least one client server is stored as a plurality of reference data, and an artificial intelligence model optimized for the plurality of reference data is stored in the database. It is stored in conjunction with a plurality of reference data,
The master server, when data collection is requested from the new client server, calculates the similarity between the requested data of the new client server and the plurality of reference data stored in the database, and the calculated similarity among the plurality of reference data. Providing an artificial intelligence model optimized for high reference data to the new client server,
How to provide a reject inspection solution.

Images