KR20220167826A - Defect detection integrated control system based on vision inspection using artificial intelligence-based visual intelligence - Google Patents

Abstract

The present invention relates to a defect detection integrated control system based on visual inspection and, more specifically, to defect detection integrated control system based on visual inspection using artificial intelligence-based visual intelligence, which can remove noise by utilizing artificial intelligence-based visual intelligence based on an image for a filmed object and provide an environment, in which determination on defect detection is accurately performed and the defect detection is integrally controlled. According to one embodiment of the present invention, the defect detection integrated control system based on vision inspection using artificial intelligence-based visual intelligence includes: an object image filming tool for filming an image of an object; a defect detection determination unit for determining object defect detection by receiving the filmed image to remove noise included in the image and removing a defective element in the noise-removed image to construct an image only consisting of normal elements; a worker terminal for receiving and checking information on the determined defect detection; and a data collection unit for collecting and managing the information on the determined defect detection.

Description

Defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence

The present invention relates to a defect detection integrated control system according to vision inspection, and more specifically, based on an image of a photographed object, noise is removed by using artificial intelligence-based visual intelligence, and defect detection is performed through the removed image. It is about a defect detection integrated control system based on vision inspection using artificial intelligence-based visual intelligence that can provide an environment in which the discrimination is made accurately and the defect detection is integratedly controlled.

An optical system (eg, vision) refers to a system that collects image data for a specific purpose in a factory. For example, this is to detect defects, abnormalities, and the like based on images to be photographed. The optical system is composed of a camera module, a lens, and the like, and lighting is installed around the optical system to generate brightness for capturing a desired image.

However, defect detection systems using existing optical systems clearly define the shape, size, and degree of defects to be detected, such as surface flaw detection, and are suitable for inspection environments limited to the definition, and are suitable for processes such as assembly. Since there is a great diversity of assembly results in , it is not suitable in an environment where judgment is required based on a set rule.

In addition, the conventional object recognition technology focuses on the fact that the recognition performance of each algorithm varies depending on the properties of an object, and clusters various object image data and then selectively uses the most effective algorithm to improve recognition performance. Since the multiclass classification method is mainly used, there is a limit to practical and diverse utilization after object recognition.

Republic of Korea Patent Registration No. 10-0459893 Republic of Korea Patent Publication No. 10-2014-0095333

The present invention is to solve the above-mentioned problems, by utilizing artificial intelligence-based visual intelligence, accurately and quickly detects defects for objects in any environment, and provides an environment in which the defect detection is integrated and controlled. It is intended to provide a defect detection integrated control system according to vision inspection.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problems described above, and other technical problems may exist.

According to an embodiment of the present invention, a defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence includes: object image capturing means for capturing an image of an object; a defect detection and judging unit for receiving the captured image, removing noise included in the image, removing defective elements from the noise-removed image, and constructing an image composed only of normal elements to determine object defect detection; a worker terminal for receiving and confirming the determined defect detection information; and a data collection unit for collecting and managing the determined defect detection information.

The defect detection and determination unit may classify the collected defect detection information according to target objects and calculate a defect detection rate corresponding to each object.

In addition, the defect detection and discrimination unit is characterized in that, as classification according to the target object, the heterogeneous material assembly part is classified according to the object corresponding to each part.

In addition, when the received captured image is a video, the defect detection and discrimination unit configures the corresponding video as a continuous image.

In addition, the defect detection and discrimination unit may include an artificial intelligence-based learning model for removing noise from an image and constructing an image composed of only normal elements.

In addition, the defect detection and discrimination unit is characterized in that the artificial intelligence-based learning model is configured based on deep learning.

In addition, the defect detection and discrimination unit is characterized in that it additionally performs image acquisition composed of a consistent form of color based on the noise-removed image.

In addition, the defect detection and discrimination unit includes a noise elimination model among artificial intelligence-based learning models, and the denoising model learns features of the captured image and determines a part excluding the learned feature as noise to perform noise elimination. characterized by

In addition, the defect detection and discrimination unit includes a defect recognition model among artificial intelligence-based learning models, and the defect recognition model finds and removes defective elements based on the denoised image, builds an image with only normal elements, and removes only noise. It is characterized in that defect detection and determination is performed through comparison with the image.

In addition, the defect recognition model removes a certain area from the entire image based on the denoised image, builds an image with only normal elements, and performs defect detection and discrimination through comparison with the image from which only the noise has been removed. do.

According to the present invention, the defect detection integrated control system according to the vision inspection using artificial intelligence-based visual intelligence utilizes artificial intelligence-based visual intelligence to accurately and quickly detect defects for objects in any environment and determine the defects. An environment in which detection is integrally controlled can be provided.

In addition, it is possible to improve defect detection accuracy compared to the existing environment.

In addition, it is possible to provide a defect detection environment for assembly parts made of different materials compared to the existing environment.

However, the effects of the present invention are not limited to the above-mentioned effects, and effects not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings.

1 is a conceptual diagram of a defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence as an embodiment of the present invention.
2 is a view for explaining noise removal of a defect detection integrated control system according to a vision inspection using artificial intelligence-based visual intelligence as an embodiment of the present invention.
3 is an exemplary view of a defect recognition model of a defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence as an embodiment of the present invention.
4 is a diagram for explaining defect detection of a defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence as an embodiment of the present invention.
5 is a flowchart illustrating an integrated control method for detecting defects according to vision inspection using artificial intelligence-based visual intelligence as an embodiment of the present invention.

The terms or words used in this specification and claims should not be construed as being limited to ordinary or dictionary meanings, and the inventors may appropriately define the concept of terms in order to explain their invention in the best way. It should be interpreted as a meaning and concept consistent with the technical idea of the present invention based on the principle that there is.

Therefore, the embodiments described in this specification and the configurations shown in the drawings are only one of the most preferred embodiments of the present invention, and do not represent all the technical ideas of the present invention, so that they can be substituted at the time of this application. It should be understood that there may be many equivalents and variations.

Hereinafter, a defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence according to the present invention will be described with reference to the drawings.

1 is a conceptual diagram of a defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence as an embodiment of the present invention.

The defect detection integrated control system according to the vision inspection using artificial intelligence-based visual intelligence according to the present invention is basically composed of an object image capturing unit, a defect detection determining unit, a worker terminal, and a data collection unit.

Referring to FIG. 1 , more specifically, the present invention provides an object image photographing means for photographing an image of an object; a defect detection and judging unit for receiving the captured image, removing noise included in the image, removing defective elements from the noise-removed image, and constructing an image composed only of normal elements to determine object defect detection; a worker terminal for receiving and verifying the determined defect detection information; and a data collection unit for collecting and managing the determined defect detection information.

In addition, the defect detection integrated control system according to the vision inspection may further include a receiver for receiving a photographed image generated by photographing an object and a storage unit for storing information necessary to implement detection and discrimination, wherein the receiver Information may be received from the object image photographing means.

Also, as an example, the object image capture means may be composed of devices for image capture, and may be a vision camera (including image sensors, lenses, shutters, and control devices) or a group composed of a plurality of vision cameras. However, it is not limited thereto, and can be variously modified at a level obvious to those skilled in the art.

In addition, the image received by the defect detection integrated control system according to the vision inspection may be generated by capturing an object image, or may be an image created by an external server or a computer program.

In addition, the worker terminal refers to a smartphone, a tablet PC (tablet personal computer), a mobile phone, a video phone, a desktop PC (desktoppersonal computer), a laptop PC (laptop personal computer), a netbook computer ( netbook computer), personal digital assistant (PDA), portable multimedia plater (PMP), wearable device (e.g. smart glasses, head-mounted-device (HMD), etc.), smart mirror (smart mirror) It may include at least one of mirrors or a smart watch, and it means a device capable of transmitting and receiving a value changed to an electrical signal in various forms without being limited thereto.

In addition, as an example of the worker terminal, it may include an input module, and the module may include a plurality of input keys and function keys for receiving numeric or character information and setting various functions. It includes direction keys, side keys, and shortcuts set to perform functions, and the input module may generate key signals related to function control of the operator terminal and transmit them to the operator terminal.

In addition, when the worker terminal supports a full touch screen, it may include only a volume key for volume control formed on the side of the case of the terminal, a power key for screen on/off and portable terminal on/off, in particular, Various input signals such as an input signal instructing control and confirmation of detected defect detection information, an input signal corresponding to authentication information input, an input signal instructing selection of a specific sandbox, an input signal instructing changing the color and transparency of content, etc. may be generated and transmitted to the defect detection and determination unit and the data collection unit.

In addition, as an example of the worker terminal, it may include a terminal control module, the overall operation of the defect detection integrated control system according to the vision inspection using artificial intelligence-based visual intelligence to be transmitted by the worker terminal and the interior of the worker terminal It may also perform a data processing function of controlling signal flow between blocks and processing data.

In addition, for example, the terminal control module includes a global positioning system (GPS)-based measurement sensor for generating real-time location information of a central processing unit (CPU), an application processor, and the like, and worker terminals. can also be provided.

In addition, as an example, the data collection unit may be built to serve as a kind of web server, database server, mobile server, and cloud server. It can be transmitted through a page. Here, the web page may include software for performing a specific task, such as a web application, in addition to simple text, image, sound, and video.

In addition, the online network referred to in the present invention may be a core network integrated with a wired public network, a wireless mobile communication network, or a mobile Internet, etc. Protocol), HTTPS (Hyper Text Transfer Protocol Secure), Telnet, FTP (File Transfer Protocol), DNS (Domain Name System), SMTP (Simple Mail Transfer Protocol), etc. , It may mean comprehensively a data communication network capable of transmitting and receiving data in various forms without being limited to these examples.

In addition, the above-described server may exchange and manage data through a transmission medium such as light, metal wire, or waveguide including a carrier wave that transmits a signal specifying program commands, data structures, etc. Examples include not only machine code produced by a compiler, but also a device that processes information electronically using an interpreter, for example, high-level language code that can be executed by a computer. It may be configured to act as one or more software modules to perform the operations of the invention and vice versa.

2 is a diagram for explaining noise removal of a defect detection integrated control system according to a vision inspection using artificial intelligence-based visual intelligence as an embodiment of the present invention, and FIG. 3 is an artificial intelligence as an embodiment of the present invention. Figure 4 is an example of a defect recognition model of a defect detection integrated control system according to vision inspection using visual intelligence based visual intelligence, and FIG. It is a diagram for explaining the defect detection of the system.

2 to 5, the defect detection determination unit classifies the collected defect detection information according to target objects, calculates a defect detection rate corresponding to each object, and classifies according to the target object. It is characterized in that the material assembly parts are classified according to objects corresponding to each part.

In addition, as an example, heterogeneous material assembly parts are products assembled by mixing various materials (eg plastic, rubber, paper, metal, etc.), It includes various elements such as fastening form (eg, alignment form, etc.), and for heterogeneous material assembly parts, it can be classified according to the object corresponding to each part.

In addition, when the received captured image is a video, the defect detection and discrimination unit may configure the corresponding video as a continuous image, and as a continuous image, overlapping images in temporal order at a predetermined time interval (eg, 0.1s, 1s) etc.), the images are extracted and arranged into a plurality of images in temporal order.

In addition, the defect detection and discrimination unit includes an artificial intelligence-based learning model for removing noise from the image and constructing an image composed of only normal elements, and the artificial intelligence-based learning model is configured based on deep learning. In addition, the defect detection and discriminating unit may additionally acquire an image composed of a consistent form of color based on the noise-removed image.

In addition, the defect detection and discrimination unit includes a noise elimination model among artificial intelligence-based learning models, and the denoising model learns characteristics of the captured image and determines a portion excluding the learned characteristics as noise to perform noise elimination. may be

In addition, the denoising model includes an encoder configured with a structure in which a deep learning-based convolutional neural network (CNN) is continuously connected and has a final output value as all connected layers (Dense Layer), and deep learning-based data extension It includes a decoder configured with a structure in which this possible convolutional neural network (Dilated CNN) is continuously connected and does not have a final output value as all connected layers (Dense Layer).

In addition, as an example of the deep learning-based convolutional neural network (CNN) used in the denoising model, a mixed network structure is constructed based on ResNet and the Inception concept of GoogLeNet is added, which is an image due to down-sampling In order to avoid loss of detail information of , based on the concept of Dilated Convolution, receptive fields of various sizes may be applied to the input image, and values of each receptive field may be mixed by performing a concatenated operation.

In addition, by applying ResNet's Skip Connection, small changes (fluctuations) can be easily detected, which can be easily optimized for deep networks while applying the structure of ResNet, and accuracy can be improved due to the increased network depth.

In addition, the size of the receptive field can be expanded without loss of resolution by using the concept of Dilated Convolution. For example, only the coefficient of the position of an arbitrary rectangle for an image divided into pixels exists in the entire reactive field, and all others are 0. , and the dilated convolution has a parameter called the dilation rate, which defines the interval between kernel values for the basic convolutional layer.

Also, as an example, since down-sampling causes loss of information in convolution operation, dilated convolution may be applied instead of applying the pooling layer.

In addition, as an example, in defining a network of structures, inputs go into three different dilated convolutional layers, respectively, concatenate the result after each dilated convolution operation, and then perform a 1 x 1 convolution operation to utilize ResNet's Skip Connection to obtain an initial The input value and Add operation are performed, and the kernels of all convolutional layers of the network defined above are all defined as 3 x 3 in size, and the padding size is 1, 2, 3 in conjunction with the dilation rate of 1, 2, 3. , and since the output of the network block is the same as the input size, the output size of the entire network may also be the same. Accordingly, the output size of the dilated convolution layer may be calculated as in Equation 2 below.

로 표현되며, i는 input size, k는 커널(필터) size, p는 zero-padding, s는 stride, d는 dilation rate를 의미하고, 이에 있어서, d = p, k = 3, s = 1로 설정하고 앞서 설명한 바와 같이 input 사이즈는 output 사이즈와 동일할 수도 있다.Also, Equation 2 is

, where i is the input size, k is the kernel (filter) size, p is zero-padding, s is the stride, and d is the dilation rate, where d = p, k = 3, s = 1 As set up and described earlier, the input size may be the same as the output size.

In addition, as an example of the network structure, the network is a convolutional network configured without a pooling layer, and it is difficult to learn to use large-sized data as it is without a pooling process, but a batch normalization layer is also applied to maintain the color information of the image. It does not, and the data size and convolution layer size of the learnable settings can be experimentally and repeatedly performed to determine.

In addition, the Initial Convolution Layer is set to 64 channels, and in the case of the final convolution layer, the color scale is set to 3 channels and the gray scale is set to 1 channel, and all ReLU layers have the same positive and negative values of the Final Residual output. It is defined as a parametric ReLU (pReLU) in order to do so, and the network described as an example above may be learned to estimate noise (residual image) through Global Skip Connection.

In addition, as an example, in order to perform noise removal more efficiently using a defined network, the concept of U-Net is applied and the detail map (mask area) obtained through U-Net is used as outline extension information to match the original image and After concatenating, noise removal (denoising) can be performed by passing it to the input of the previously defined network.

In addition, U-Net can be used as a backbone network to extract detail maps, and through the encoding/decoding process of U-Net, the size of the feature is reduced by half and the number of dimensions is doubled as it passes through the Conv block in the encoding process. , and in the decoding process, as it passes through the Conv block opposite to the encoding process, the size of the feature doubles and the number of dimensions decreases by half. A 3 x 3 kernel is applied to sampling and up-sampling, and batch normalization is applied after conv operation. U-Net can also minimize feature information that can be lost in sub-sampling and up-sampling by using Symmetric Shortcut Connection. there is.

In addition, the defect detection and discrimination unit includes a defect recognition model among artificial intelligence-based learning models, and the defect recognition model finds and removes defective elements based on the denoised image, builds an image with only normal elements, and removes only noise. Defective detection and discrimination may be performed through comparison with images.

In addition, the defect recognition model removes a certain area (eg, after converting to a 128 x 128 pixel size image, each of 3 x 3 areas that are not identical) from the entire image based on the denoised image Defect detection and discrimination can also be performed by constructing an image with only normal elements and comparing it with an image from which only noise has been removed.

In addition, the faulty cognitive model includes a layer (reduction layer) in which deep learning-based convolutional neural networks (CNNs) are continuously connected and input data dimensions are maintained as they are, and deep learning-based convolutional neural networks (CNNs) It includes a feature extract layer that is continuously connected and extracts only features without changing the data dimension, and a convolutional neural network (Dilated CNN) that can expand data based on deep learning is continuously connected and has the same size as the input. It includes a layer that outputs data (Upscaling Layer).

In addition, as an example, the poor recognition model is trained for image construction (completion) through clear normal images from which noise has been removed (denoising), and an image completion network (deep learning-based convolutional neural network (CNN) is continuously ), and when the inspection target (object) image comes in, it is converted into a 128 x 128 pixel size image, then the patch is removed from each area in the 3 x 3 area 9 times and the process of restoring it is performed You may.

In addition, the difference between each patch area and the restored image corresponding area is calculated, the difference is calculated as an abnormality score, the area with the large difference is determined to contain defects, and the matching difference in pixel units is calculated to obtain the pixel with the large difference. By extracting the regions and masking them on the original image, the corresponding region may be marked as a defective region.

In addition, as an example of the network structure for image construction (completion), the 32 x 32 central area of the input image patch (128 x 128) is removed (set to 0) before entering the input of DCNN (Deep Convolutional Neural Network), and the image After reconstructing the image in the network for completion (construction), the generated patch area is subtracted to obtain the difference from the corresponding area of the original image, and the absolute value of the image difference can be used as an anomaly score. there is.

In addition, the network for image completion may be composed of 17 layers as shown below, Conv(5, 1, 1, 32) - Conv(3, 1, 1, 64) - Conv(3, 1, 1, 64 ) - Conv(3, 1, 2, 128) - Conv(3, 1, 1, 128) - Conv(3, 1, 1, 128) - Conv(3, 2, 1, 128) - Conv(3, 4, 1, 128) - Conv(3, 8, 1, 128) - Conv(3, 16, 1, 128) - Conv(3, 1, 1, 128) - Conv(3, 1, 1, 128) - Bilinear Upscaling(2x) - Conv(3, 1, 1, 64) - Conv(3, 1, 1, 64) - Conv(3, 1, 1, 32) - Conv(3, 1, 1, 16) - Conv (3, 1, 1, 1) - Clip (-1, 1), where Conv (k, d, s, c) represents a convolutional layer with a kernel size of k x k, and d is the dilation rate, s may mean a stride and c may mean an output channel.

In addition, as an example of the network structure, the network as a whole consists of 17 layers, the resolution of the feature map after the third layer is reduced by half by stride convolution, and layer 7 to increase the receptive field of the output neuron. In layer 10, dilated convolution can be continuously applied, and in layer 13, bilinear rescaling is performed through convolution operation for upscaling to the input size, mirror padding is used for all convolutional layers, and activation function As an ELU (Exponential Linear Unit) activation function may be applied.

In addition, the network is trained based on the L1 reconstruction loss, and the 32 x 32 central region defined by the binary mask M is weighted differently from the rest of the region, and when X is the image patch to be inspected, the network is It may also be trained through Equation 1.

로 표현되며, 여기서

는 element-wise 행렬곱을 의미하며,

은 M의 complement mask를 의미하고,

는 중심 영역과 나머지 영역 사이의 가중치 파라미터이며, N은 X의 픽셀 수,

는 L1 행렬정규화(matrix normalization)을 의미한다.In addition, Equation 1 shows a loss function,

is expressed as, where

means element-wise matrix multiplication,

Means the complement mask of M,

is the weighting parameter between the center area and the rest area, N is the number of pixels in X,

Means L1 matrix normalization.

In addition, the image completion (construction) network performs learning from scratch with a batch size of 128 using the ADAM optimizer, and all weights have a Gaussian omitted distribution with a mean of '0' and a standard deviation of '1' ( truncated Gaussian distribution).

5 is a flowchart illustrating an integrated control method for detecting defects according to vision inspection using artificial intelligence-based visual intelligence as an embodiment of the present invention.

On the other hand, as a method for the present invention, the defect detection integrated control method according to vision inspection using artificial intelligence-based visual intelligence is basically an object image photographing step, a noise removal step, an image construction step, a defect detection determination step, and a defect detection step. It is composed of a detection confirmation step and a data management step.

More specifically, referring to FIG. 5 , an object image capturing step of capturing an image of an object by the object image capturing unit, a noise removing step of removing noise included in a photographed image by the defect detection discriminator, and An image construction step of constructing an image composed of only normal elements after removing defective elements, a defect detection and discrimination step of performing defect detection and discrimination through comparison with an image from which only noise has been removed based on the constructed image, the worker terminal A defect detection confirmation step of receiving and confirming the determined defect detection information and a data management step of collecting and managing the determined defect detection information by the data collection unit.

In the above description of the present invention with reference to the accompanying drawings, the specific shape and direction have been mainly described, but the present invention can be variously modified and changed by a person having ordinary knowledge in the technical field belonging to the invention, and these Modifications and changes should be construed as being included in the scope of the present invention.

Claims (10)

object image capturing means for capturing an image of the object;
a defect detection and judging unit for receiving the captured image, removing noise included in the image, removing defective elements from the noise-removed image, and constructing an image composed only of normal elements to determine object defect detection;
a worker terminal for receiving and verifying the determined defect detection information; and
Characterized in that it comprises a; data collection unit for collecting and managing the determined defect detection information,
Defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence.
The method of claim 1,
The defect detection and discrimination unit,
Characterized in that the collected defect detection information is classified according to the target object and a defect detection rate corresponding to each object is calculated.
Defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence.
The method of claim 2,
The defect detection and discrimination unit,
As a classification according to the target object, characterized in that the heterogeneous material assembly parts are classified according to the object corresponding to each part,
Defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence.
The method of claim 1,
The defect detection and discrimination unit,
Characterized in that when the received image is a video, the video is composed of continuous images.
Defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence.
The method of claim 1,
The defect detection and discrimination unit,
Characterized in that it includes an artificial intelligence-based learning model for removing noise from the image and constructing an image composed of only normal elements.
Defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence.
The method of claim 5,
The defect detection and discrimination unit,
Characterized in that the artificial intelligence-based learning model is configured based on deep learning,
Defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence.
The method of claim 1,
The defect detection and discrimination unit,
Characterized in that additionally performing image acquisition composed of a consistent form of color based on the denoised image,
Defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence.
The method of claim 5,
The defect detection and discrimination unit,
Among the artificial intelligence-based learning models, including a noise removal model,
The denoising model is characterized in that it learns the features of the captured image and performs denoising by determining a part excluding the learned features as noise.
Defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence.
The method of claim 1,
The defect detection and discrimination unit,
Among the artificial intelligence-based learning models, including a poor cognitive model,
The defect recognition model is characterized in that it finds and removes defective elements based on the denoised image, builds an image with only normal elements, and performs defect detection and discrimination through comparison with an image from which only noise has been removed.
Defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence.
The method of claim 9,
The defect recognition model,
Characterized in that, based on the denoised image, a certain area of the entire image is removed, and an image having only normal elements is constructed to perform defect detection and discrimination through comparison with an image from which only noise is removed.
Defect detection integrated control system according to vision inspection using artificial intelligence-based visual intelligence.

Images