KR102670083B1 - Vision inspection system for applying environmental weight to output of plurality of artificial intelligence models - Google Patents

Abstract

A vision inspection system is launched. This vision inspection system includes a camera that photographs the inspection object, a first artificial intelligence model that detects failures by extracting semantic information related to the failure, and a second artificial intelligence model that detects failures by extracting feature information. Includes memory and processor. The processor inputs the image obtained through the camera into the first artificial intelligence model to obtain first output information, inputs the image into the second artificial intelligence model to obtain second output information, and obtains the first output information and the first output information. 2 Detect faults in objects included in the image based on the output information.

Description

A vision inspection system that applies weights according to the environment to the output of multiple artificial intelligence models { VISION INSPECTION SYSTEM FOR APPLYING ENVIRONMENTAL WEIGHT TO OUTPUT OF PLURALITY OF ARTIFICIAL INTELLIGENCE MODELS }

This disclosure relates to a vision inspection system, and more specifically, to a vision inspection system that detects failures by combining outputs independently obtained from a plurality of artificial intelligence models.

In vision inspection technology for fault detection, artificial intelligence models have begun to be used for automation of fault detection and precision of judgment.

However, in order to ensure the judgment accuracy of the artificial intelligence model, a large amount of training data for various situations is required, so a process of classifying and preprocessing image data from various sources is necessary.

In addition, since the pros and cons are clear depending on the specific network configuration or training technique of the artificial intelligence model, the technology level has not been reached to rely 100% on the judgment accuracy of the artificial intelligence model.

Public Patent Publication No. 10-2020-0046137

This disclosure provides a vision inspection system that performs failure detection based on an ensemble technique that combines the output of two independently trained and operated artificial intelligence models.

*This disclosure provides a vision inspection system that utilizes different artificial intelligence models with the same input and output format, minimizes training, and provides an efficient failure detection environment.

The objects of the present disclosure are not limited to the purposes mentioned above, and other objects and advantages of the present disclosure that are not mentioned can be understood by the following description and will be more clearly understood by the examples of the present disclosure. Additionally, it will be readily apparent that the objects and advantages of the present disclosure can be realized by the means and combinations thereof indicated in the patent claims.

A vision inspection system according to an embodiment of the present disclosure includes a camera that photographs an inspection object, a first artificial intelligence model that detects a failure by extracting semantic information related to the failure, and a failure that is detected by extracting feature information. It includes a memory including a second artificial intelligence model, the camera, and a processor connected to the memory. The processor acquires first output information by inputting the image acquired through the camera into the first artificial intelligence model, and obtains second output information by inputting the image into the second artificial intelligence model, A failure of an object included in the image is detected based on the first output information and the second output information.

The first artificial intelligence model may be a model trained based on a first image including a normal object and a second image including a faulty object. And, the second artificial intelligence model can extract characteristic information of a normal object from the first image.

The first output information may include a first mask indicating a failure area of the object included in the image, and the second output information may include a second mask indicating a failure area of the object included in the image. there is.

In this case, the first artificial intelligence model may acquire semantic information related to a failure of an object included in the image and output the first mask according to the obtained semantic information. In addition, the second artificial intelligence model extracts feature information of the object included in the image, identifies a classification of the object included in the image based on the extracted feature information, and identifies a normal object matching the classification. Failure information may be obtained by comparing the characteristic information of with the extracted characteristic information, and the second mask may be output according to the obtained failure information.

Meanwhile, the processor obtains a first value by applying a first weight to the first output information, obtains a second value by applying a second weight to the second output information, and obtains the first value and the Based on the second value, the failure area of the object included in the image can be identified.

In this case, the processor may set the first weight and the second weight, respectively, based on information about the environment in which the image was captured.

When an image of a size that does not match the input of each of the first artificial intelligence model and the second artificial intelligence model is input, the processor inputs the input image into at least one artificial intelligence model for object recognition An object may be identified, an area including the identified object may be extracted from the input image, and the extracted area may be input into each of the first artificial intelligence model and the second artificial intelligence model.

A vision inspection system according to an embodiment of the present disclosure includes a first artificial intelligence model that detects a failure by extracting semantic information related to the failure, and a second artificial intelligence model that detects the failure by extracting feature information. It includes a memory, and a processor connected to the memory. The processor acquires first output information by inputting an image including an object into the first artificial intelligence model, obtains second output information by inputting the image into the second artificial intelligence model, and obtains second output information by inputting the image including the object into the first artificial intelligence model. A failure of an object included in the image is detected based on the output information and the second output information.

At this time, the image may be pre-stored in the memory 120 or may be received from at least one external electronic device.

The vision inspection system according to the present disclosure has the effect of enabling precise failure detection for objects of various classes by utilizing the output of two different artificial intelligence models.

The vision inspection system according to the present disclosure can utilize common training data (images) for training/building two artificial intelligence models, and it is sufficient if training to update weights between nodes is performed for only one artificial intelligence model, The advantage is that the load required to establish a precise failure detection environment is relatively small.

1 is a block diagram for explaining the configuration of a vision inspection system according to an embodiment of the present disclosure;
2 is a flowchart for explaining the operation of a vision inspection system according to an embodiment of the present disclosure;
3 is a diagram illustrating the operation of the vision inspection system inputting images into two different artificial intelligence models according to an embodiment of the present disclosure;
4 is a flowchart illustrating the process of training the first artificial intelligence model of the vision inspection system according to an embodiment of the present disclosure;
5 is a flowchart illustrating the process of building a second artificial intelligence model of a vision inspection system according to an embodiment of the present disclosure;
6 is a flowchart illustrating a process in which a vision inspection system detects a failure using a second artificial intelligence model according to an embodiment of the present disclosure;
7 is a block diagram for explaining the detailed configuration of a vision inspection system according to various embodiments of the present disclosure; and
FIG. 8 is a diagram for explaining the configuration of a camera including an FPGA in a vision inspection system according to an embodiment of the present disclosure.

Before explaining the present disclosure in detail, the description method of the present specification and drawings will be explained.

First, the terms used in the specification and claims are general terms selected in consideration of their functions in various embodiments of the present disclosure. However, these terms may vary depending on the intention of technicians working in the relevant technical field, legal or technical interpretation, and the emergence of new technologies. Additionally, some terms are arbitrarily selected by the applicant. These terms may be interpreted as defined in this specification, and if there is no specific term definition, they may be interpreted based on the overall content of this specification and common technical knowledge in the relevant technical field.

In addition, the same reference numbers or symbols in each drawing attached to this specification indicate parts or components that perform substantially the same function. For convenience of explanation and understanding, the same reference numerals or symbols are used in different embodiments. That is, even if all components having the same reference number are shown in multiple drawings, the multiple drawings do not represent one embodiment.

Additionally, in this specification and claims, terms including ordinal numbers such as “first”, “second”, etc. may be used to distinguish between components. These ordinal numbers are used to distinguish identical or similar components from each other, and the meaning of the term should not be interpreted limitedly due to the use of these ordinal numbers. For example, the order of use or arrangement of components combined with these ordinal numbers should not be limited by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, singular expressions include plural expressions, unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “consist of” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are intended to indicate the presence of one or more other It should be understood that this does not exclude in advance the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof.

In embodiments of the present disclosure, terms such as “module”, “unit”, “part”, etc. are terms to refer to components that perform at least one function or operation, and these components are either hardware or software. It may be implemented or may be implemented through a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. are integrated into at least one module or chip, except in cases where each needs to be implemented with individual specific hardware, and is integrated into at least one processor. It can be implemented as:

Additionally, in an embodiment of the present disclosure, when a part is connected to another part, this includes not only direct connection but also indirect connection through other media. In addition, the meaning that a part includes a certain component does not mean that other components are excluded, but that other components can be further included, unless specifically stated to the contrary.

1 is a block diagram for explaining the configuration of a vision inspection system according to an embodiment of the present disclosure.

Referring to FIG. 1, the vision inspection system 100 may include a camera 110, memory 120, processor 130, etc.

The vision inspection system 100 may correspond to a system/device for detecting failures in various parts/products, etc. All of the above-described components within the vision inspection system 100 may be included in one device, or may be provided in a plurality of devices/modules capable of wired and wireless communication with each other.

The camera 110 is configured to capture an inspection object and generate at least one image. The camera 110 may be implemented as an RGB camera, a time of flight (TOF) camera, etc., but is not limited thereto.

The camera 110 may include at least one optical sensor, a lens that adjusts the path of light to photograph a specific point or a specific range, etc.

The camera 110 can photograph a fixed or moving inspection target. In this case, the inspection object may be arranged to move along a path passing through the capturing range of the camera 110. Additionally, photography may be performed while the camera 110 is attached to at least one moving means (e.g. wheels, rollers, etc.) and moves. Specifically, an area scan method or a line scan method for the inspection object may be used, but the method is not limited thereto.

The memory 120 stores an operating system (OS) for controlling the overall operation of at least one electronic device included in the vision inspection system 100 and at least one instruction or data related to components of the electronic device. This is a configuration for

The memory 120 may include non-volatile memory, such as ROM or flash memory, and may include volatile memory, such as DRAM. Additionally, the memory 120 may include a hard disk, solid state drive (SSD), etc.

Referring to FIG. 1, the memory 120 may include a first artificial intelligence model 121 and a second artificial intelligence model 122 that independently perform failure detection.

Each of these artificial intelligence models may correspond to a network model (neural network model) based on a neural network. A network model may include a plurality of network nodes with weights. Multiple network nodes can form a connection relationship based on weights between nodes of different layers.

In one embodiment, the first artificial intelligence model 121 may correspond to a Semantic Segmentation model that extracts semantic information related to a failure of an object.

Semantic information may include information about object classification (e.g. part type, product type, etc.), failure status, failure degree, failure area, etc.

When an image is input to the first artificial intelligence model 121, the first artificial intelligence model 121 can recognize the object in the image, determine whether the object is broken, and create a mask indicating the broken area of the object in the image. can be output.

As an example, the second artificial intelligence model 122 may correspond to an Anomaly Detection model that detects outliers through comparison with normal objects.

The second artificial intelligence model 122 can extract feature information of an image through at least one layer. Here, the second artificial intelligence model 122 can identify the classification of the object using feature information.

In addition, the second artificial intelligence model 122 can compare the characteristic information of a normal object (: an object without a failure) matching the identified classification with the above-described characteristic information to determine whether there is a failure, a failure area, etc. As a result, a mask representing the faulty area of the object within the image may be output.

The processor 130 is configured to overall control the vision inspection system 100. Specifically, the processor 130 may perform operations according to various embodiments of the present disclosure by being connected to the memory 110 and executing at least one instruction stored in the memory 110.

Processor 130 may be comprised of one or more processors. At this time, the one or more processors may include general-purpose processors such as CPU, AP, and DSP (Digital Signal Processor), graphics-specific processors such as GPU and VPU (Vision Processing Unit), or artificial intelligence-specific processors such as NPU.

The processor 130 can drive the above-described artificial intelligence models 121 and 122 stored in the memory 110, respectively, and can also perform training of each model.

Figure 2 is a flowchart for explaining the operation of a vision inspection system according to an embodiment of the present disclosure.

Referring to FIG. 2, the processor 130 may input the image acquired through the camera 110 into the first artificial intelligence model 121 to obtain first output information (S210). At this time, the first artificial intelligence model 121 may be the Semantic Segmentation model described above.

Here, the first output information may correspond to a mask indicating a faulty area of an object in the image.

Additionally, the processor 130 may input the image acquired through the camera 110 to the second artificial intelligence model 122 to obtain second output information (S220). At this time, the second artificial intelligence model 122 may be the Anomaly Detection model described above.

Here, the second output information may also correspond to a mask representing the faulty area of the object in the image.

And, the processor 130 may detect a failure of an object included in the image based on the first output information and the second output information (S230).

For example, the processor 130 may obtain a first value by applying a first weight to first output information and obtain a second value by applying a second weight to second output information. Additionally, the processor 130 may identify a faulty area of an object included in the image based on the first value and the second value. Specifically, the first value and the second value may be simply added, but the present invention is not limited to this.

Here, the first weight is applied to each pixel of the first mask matching the first output information, and the second weight is applied to each pixel of the second mask matching the second output information, indicating the fault area of the object. The (final) mask can be obtained. At this time, an area in the final mask where the pixel value is above a certain value may be identified as a failure area.

*Relatedly, FIG. 3 is a diagram illustrating the operation of a vision inspection system inputting images to two different artificial intelligence models according to an embodiment of the present disclosure.

Referring to FIG. 3, the processor 130 may input an image 310 in which an object to be inspected is photographed into each artificial intelligence model 121 and 122.

At this time, the first artificial intelligence model 121 and the second artificial intelligence model 122 may output masks 320 and 330 representing the failure area, respectively.

Additionally, the processor 130 may obtain the final mask 340 by assigning weights (first weight, second weight) to each of the masks 320 and 330 and adding them together.

As a specific example, when the first weight and the second weight are the same, the average value of the first output information (mask 320) and the second output information (mask 330) may be obtained.

In this case, the processor 130 can obtain the mask 340 by calculating the average for each pixel position of the masks 320 and 330 output from each artificial intelligence model 121 and 122. An area within the mask 340 having an R/G/B value greater than a certain value may be identified as a fault area.

Meanwhile, the first weight and the second weight may be set differently for each pixel position of the masks 320 and 330 output from each model 121 and 122.

That is, for some areas of the mask 340, the first weight for the mask 320 is applied larger than the second weight for the mask 330, and for other areas of the mask 340, the first weight for the mask 320 is applied larger than the second weight for the mask 330. The second weight for the mask 320 may be applied larger than the first weight for the mask 320 .

Additionally, in one embodiment, the processor 130 may set the first weight and the second weight, respectively, based on information about the environment in which the image was captured.

Information about the environment may include, but is not limited to, information about the brightness, humidity, air pollution level, distance between the camera and the object, etc. of the place where the shooting was performed.

In this case, information about the accuracy of each artificial intelligence model 121 and 122 depending on the environment may be stored in the memory 120 for each pixel location.

For example, for images taken at a brightness above the threshold for a specific pixel position, the accuracy of the mask output by the first artificial intelligence model 121 is higher than the mask output by the second artificial intelligence model 122. , for images taken at brightness below the threshold, the accuracy of the mask output by the second artificial intelligence model 122 may be higher than the accuracy of the mask output by the first artificial intelligence model 121.

In relation to this, the processor 130 can identify the surrounding brightness of the camera 110 through the illuminance sensor and acquire an image by performing photography through the camera 110.

Here, for at least one pixel position in the mask, the processor 130 sets the first weight higher than the second weight if the surrounding brightness is greater than the threshold, and sets the second weight higher than the first weight if the surrounding brightness is less than the threshold. can be set higher.

Meanwhile, an artificial intelligence model trained to determine the above-described first and second weights for each pixel in the mask may be stored in the memory 120.

This artificial intelligence model may be a neural network model, and for example, inputs information about various environments (ex. brightness, humidity, air pollution, camera distance, etc.), and uses the first and second weights described above. It may be an artificial intelligence model (weight judgment model) with as output.

Specifically, the processor 130 provides an image captured through the camera 110, information about the environment in which the image was captured, and masks output from each model 121 and 122 as a result of image input (by pixel position). ) The above-described artificial intelligence model (weight judgment model) can be trained by utilizing the accuracy of the final mask derived according to various weight combinations (first weight, second weight) (compared to the actual fault area in the image), etc. . Here, the weights between nodes constituting the weight determination model can be updated in a way that improves the accuracy of each pixel position of the final mask.

Thereafter, the processor 130 may flexibly set the first weight and the second weight according to the result of inputting information about the environment related to shooting by the camera 110 into the weight determination model. Then, the processor 130 reflects the first weight and the second weight on the masks obtained as a result of inputting the image captured through the camera 110 into each model 121 and 122 to create a final mask (failure detection result). can be obtained.

Meanwhile, FIG. 4 is a flowchart illustrating the process of training the first artificial intelligence model of the vision inspection system according to an embodiment of the present disclosure.

Training of the first artificial intelligence model 121 may be performed by the processor 130 or by at least one external electronic device. 4, the description will be made assuming that training of the first artificial intelligence model 121 is performed through the processor 130.

Referring to FIG. 4, the processor 130 may obtain one or more images of a normal object and one or more images of a malfunctioning object (S410). Each image may be an image of an object of various classes (classification of objects), may be captured through the camera 110, or may be an image received from an external electronic device.

At this time, a label related to the fault area may be matched and registered in each image. For example, the label may include, but is not limited to, information on normal/failure status, fault area, fault level, etc.

The processor 130 may perform preprocessing on the acquired image (S420).

In one embodiment, the processor 130 may reduce/enlarge each image to fit the input size of the first artificial intelligence model 121.

Alternatively, when an image of a size that does not match the input of the first artificial intelligence model 121 is input, the processor 130 inputs the image into at least one artificial intelligence model for object recognition to determine the area containing the object. Only can be extracted to fit the input size.

Additionally, to secure training data, the processor 130 may additionally generate one or more images using images of normal objects and images of malfunctioning objects, respectively.

For example, the processor 130 may change the brightness, contrast, etc. of the acquired image, or perform operations such as rotating, warping, and cropping the image to create a new image.

As another example, the processor 130 may utilize at least one Generative Adversarial Networks (GAN) to generate an image.

Specifically, as a result of being trained using at least one image of a normal object and an image of at least one faulty object, GAN generates an image of a faulty object by utilizing an image of a normal object, or generates an image of a faulty object by utilizing an image of a faulty object. An image can be created. In this case, GAN can be trained for each classification of the object being inspected.

Additionally, the GAN may be trained using at least one image of a normal object and a mask image representing a faulty area of a faulty object.

When the preprocessing process is performed in this way, the processor 130 inputs images for training into the first artificial intelligence model 121 (S430) and compares the output of the first artificial intelligence model 121 with the above-described label (S440) )can do. Additionally, the processor 130 may update the weights between nodes in the first artificial intelligence model 121 according to the comparison result (S450).

Meanwhile, Figure 5 is a flowchart illustrating the process of constructing a second artificial intelligence model of a vision inspection system according to an embodiment of the present disclosure.

Construction of the second artificial intelligence model 122 may be performed by the processor 130 or by at least one external electronic device. 5, the description will be made assuming that construction of the second artificial intelligence model 122 is performed through the processor 130.

Referring to FIG. 5, the processor 130 may acquire one or more images in which a normal object is photographed (S510). Each image may be an image of an object corresponding to various classes (classification of objects), may be captured through the camera 110, or may be an image received from an external electronic device.

The second artificial intelligence model 122 is an Anomaly Detection model that extracts feature information from images of normal objects and can then be used to detect faulty objects.

Referring to FIG. 5, the processor 130 may perform preprocessing on the acquired image of the normal object (S520). Similar to S420 described above, a preprocessing process according to various embodiments may be performed on an image of a normal object.

And, the processor 130 may input images (of normal objects) into the second artificial intelligence model 122 (S530).

In this case, since feature information can be output through multiple layers, only feature information of some layers can be randomly selected (S540).

Additionally, for each class (classification of an object), the processor 130 may extract and store the average and covariance of feature information in each layer (S550).

Thereafter, when the image of the malfunctioning object is input to the second artificial intelligence model 122, the second artificial intelligence model 122 may detect the malfunction using the average/covariance stored for the normal object.

In relation to this, FIG. 6 is a flowchart illustrating a process in which a vision inspection system detects a failure using a second artificial intelligence model according to an embodiment of the present disclosure.

Referring to FIG. 6, the processor 130 may obtain an image in which an object to be inspected is captured (S610). At this time, the processor 130 may capture an image through the camera 110 or receive an image from an external electronic device.

And, the processor 130 may input an image into the second artificial intelligence model 122 (S620). Of course, while an image is input to the second artificial intelligence model 122, an image is also input to the first artificial intelligence model 121.

At this time, the processor 130 may obtain a large number of layer-wise feature information through the layers of the second artificial intelligence model 122.

Here, the processor 130 may classify the class of the object in the image using feature information (S640).

Additionally, the processor 130 may extract feature information by selecting some layers among the layers that output feature information (S630). For example, feature information (e.g. Feature Map) of a certain number of layers may be extracted in the form of an encoder, but the method is not limited to this.

At this time, the processor 130 may compare the average and covariance of the feature information of the normal object corresponding to the classified class with the extracted feature information (S650). For example, the deviation can be calculated according to the Mahalanobis distance, but in addition, the PatchCore technique and the Semi-orthogonal technique, in which outliers are extracted by comparing the KNN distance of the pooled feature map, can also be used.

In addition, the processor 130 may perform failure detection by obtaining failure information (e.g., failure status, failure degree, failure area, etc.) according to the comparison result (S660). Specifically, a result mask (ex. 330) with pixel values set differentially for areas within the image that have a large difference from the normal object as a result of comparison may be output.

Figure 7 is a block diagram for explaining the detailed configuration of a vision inspection system according to various embodiments of the present disclosure.

Referring to FIG. 7, the vision inspection system 100 may include a user input unit 140, a communication unit 150, an output unit 160, etc. in addition to a camera 110, a memory 120, and a processor 130. .

The user input unit 140 is configured to receive input of various commands or information from the user.

For example, the user input unit 140 may be comprised of a touch sensor, button, camera, microphone, keyboard, etc. of a device included in the vision inspection system 100, but is not limited thereto.

In one embodiment, the vision inspection system 100 controls shooting of the camera 110 and/or detects failures (ex. the first artificial intelligence model and the second artificial intelligence model) according to the user input received through the user input unit 140. Intelligent model operation) can be performed.

The communication unit 150 is a component that allows the vision inspection system 100 to transmit and receive data with various external devices, and may include at least one circuit for communication.

The communication unit 150 includes TCP/IP (Transmission Control Protocol/Internet Protocol), UDP (User Datagram Protocol), HTTP (Hyper Text Transfer Protocol), HTTPS (Secure Hyper Text Transfer Protocol), FTP (File Transfer Protocol), and SFTP ( Various information can be transmitted and received with one or more external electronic devices using communication protocols (protocols) such as Secure File Transfer Protocol (Secure File Transfer Protocol) and Message Queuing Telemetry Transport (MQTT).

To this end, the communication unit 150 may be connected to an external device based on a network implemented through wired communication and/or wireless communication. At this time, the communication unit 150 may be connected directly to an external device, but may also be connected to an external electronic device through one or more external servers (ex. Internet Service Provider (ISP)) that provide a network.

Depending on the area or size, the network may be a personal area network (PAN), a local area network (LAN), or a wide area network (WAN). Depending on the openness of the network, it may be an intranet, It may be an extranet, or the Internet.

Wireless communications include LTE (long-term evolution), LTE-A (LTE Advance), 5G (5th Generation) mobile communications, CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), and WiBro. At least one of the following communication methods: (Wireless Broadband), GSM (Global System for Mobile Communications), DMA (Time Division Multiple Access), WiFi (Wi-Fi), WiFi Direct, Bluetooth, NFC (near field communication), and Zigbee. It can be included.

Wired communication may include at least one of communication methods such as Ethernet, optical network, USB (Universal Serial Bus), and Thunderbolt.

Here, the communication unit 130 may include a network interface or a network chip according to the wired or wireless communication method described above. Meanwhile, the communication method is not limited to the examples described above and may include newly emerging communication methods as technology develops.

In one embodiment, the vision inspection system 100 may receive an image from an external electronic device through the communication unit 150. Here, the image may be an image of a normal object or a malfunctioning object.

In this way, when the communication unit 150 is provided, an embodiment in which the vision inspection system 100 does not include the camera 110 is also possible.

The output unit 160 is configured to output various information and provide it to the user. The output unit 160 may be implemented as a speaker, display, earphone/headset terminal, etc., but is not limited thereto.

In one embodiment, when a failure of an object subject to inspection is detected, the vision inspection system 100 may display the failure area in the image through the display of the output unit 160. Here, the fault area may be selected according to the final mask in which the masks output from each model 121 and 122 are combined according to the weight.

Meanwhile, the camera 110 may simply include an image sensor, but may also include at least one processing module for the image sensor.

In relation to this, FIG. 8 is a diagram for explaining the configuration of a camera including field programmable gate arrays (FPGA) in a vision inspection system according to an embodiment of the present disclosure.

A typical smart camera has a structure in which a CMOS image sensor is directly connected to a 64-bit RISC Processor Module through a MIPI Interface or LVDS Interface. In other words, image processing varies depending on the resolution and type of image sensor used. In this case, when the image sensor is changed, the program of the module that processes the inspection algorithm also needs to be changed. Because users must separately purchase a camera with a resolution appropriate for each type of inspection, it can be a burden in terms of cost.

On the other hand, referring to FIG. 8, the camera 110 according to an embodiment of the present disclosure includes a separate module (FPGA) that receives images from an image sensor and processes the images, and processes the images with a processor 130 for inspection algorithm processing. Data can be transmitted quickly (ex. USB 3.0).

In this case, the processor 130 can easily apply images of various resolutions simply by setting the buffer appropriate for the connected resolution. Since the user can connect and use a variety of image sensors of the required resolution to one processor 130, there are many advantages in terms of cost and it is possible to respond easily and quickly to each inspection type.

In particular, while data received from a CMOS image sensor is difficult to use directly for inspection due to noise, etc., the FPGA module of the camera 110 in FIG. 8 processes the data as desired by the user after ISP (Image Signal Processing) for noise reduction and defect correction. The image size can be set and the corrected image data can be transmitted to the processor 130.

Thereafter, the processor 130 may perform failure detection on the corrected image using the pre-learned dataset (ex. FIG. 2). Here, rule-based inspection and deep learning inspection can be used together. In this way, because all inspections can be performed within the vision inspection system 100 without a separate computing unit (separate server, etc.), the cost consumed in configuring the system can be dramatically reduced.

Meanwhile, the various embodiments described above may be implemented by combining two or more of them as long as they do not conflict with each other.

Meanwhile, the various embodiments described above may be implemented in a recording medium that can be read by a computer or similar device using software, hardware, or a combination thereof.

According to hardware implementation, embodiments described in the present disclosure include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs). ), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

In some cases, embodiments described herein may be implemented in the processor itself. According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules described above may perform one or more functions and operations described herein.

Meanwhile, computer instructions or computer programs for performing processing operations in the vision inspection system 100 according to various embodiments of the present disclosure described above are non-transitory computer-readable medium. ) can be stored in . Computer instructions or computer programs stored in such non-transitory computer-readable media, when executed by a processor of a specific device, cause the specific device to perform processing operations in the vision inspection system according to the various embodiments described above.

A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specific examples of non-transitory computer-readable media may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc.

In the above, preferred embodiments of the present disclosure have been shown and described, but the present disclosure is not limited to the specific embodiments described above, and may be used in the technical field pertaining to the disclosure without departing from the gist of the disclosure as claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical ideas or perspectives of the present disclosure.

100: vision inspection system 110: camera
120: memory 130: processor

Claims (1)

In the vision inspection system,
A camera that photographs the inspection object;
A memory including a first artificial intelligence model for detecting a failure by extracting semantic information related to the failure and a second artificial intelligence model for detecting the failure by extracting feature information; and
Including a processor connected to the camera and the memory,
The processor,
Input the image acquired through the camera into the first artificial intelligence model to obtain first output information,
Input the image into the second artificial intelligence model to obtain second output information,
Detecting a failure of an object included in the image based on the first output information and the second output information,
The first output information is,
Includes a first mask representing a faulty area of an object included in the image,
The second output information is,
A second mask representing a faulty area of an object included in the image,
The first artificial intelligence model is,
Obtaining semantic information related to failure of objects included in the image,
Output the first mask according to the obtained semantic information,
The second artificial intelligence model is,
Extract feature information of objects included in the image,
Identify the classification of objects included in the image based on the extracted feature information,
Obtaining failure information by comparing feature information of a normal object matching the classification with the extracted feature information,
Outputting the second mask according to the obtained failure information,
The processor,
Obtaining a first value by applying a first weight to the first output information,
Obtaining a second value by applying a second weight to the second output information,
Identifying a faulty area of an object included in the image based on the sum of the first value and the second value,
The processor,
Inputting information about the environment in which the image was captured into a weight determination model, setting the first weight and the second weight respectively according to the output of the weight determination model,
The weight judgment model is,
An artificial intelligence model trained to determine the first weight and the second weight for each pixel in the mask,
A vision inspection system that is a model that inputs information about the environment including at least one of brightness, humidity, air pollution, and camera distance, and outputs the first weight and the second weight.