WO2022241784A1 - 缺陷检测方法及装置、存储介质及电子设备 - Google Patents

缺陷检测方法及装置、存储介质及电子设备 Download PDF

Info

Publication number
WO2022241784A1
WO2022241784A1 PCT/CN2021/095306 CN2021095306W WO2022241784A1 WO 2022241784 A1 WO2022241784 A1 WO 2022241784A1 CN 2021095306 W CN2021095306 W CN 2021095306W WO 2022241784 A1 WO2022241784 A1 WO 2022241784A1
Authority
WO
WIPO (PCT)
Prior art keywords
images
defect detection
image
defect
types
Prior art date
Application number
PCT/CN2021/095306
Other languages
English (en)
French (fr)
Inventor
王耀平
张美娟
贺王强
柴栋
王洪
Original Assignee
京东方科技集团股份有限公司
北京中祥英科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司, 北京中祥英科技有限公司 filed Critical 京东方科技集团股份有限公司
Priority to US17/772,562 priority Critical patent/US20240202893A1/en
Priority to CN202180001233.1A priority patent/CN115699082A/zh
Priority to PCT/CN2021/095306 priority patent/WO2022241784A1/zh
Publication of WO2022241784A1 publication Critical patent/WO2022241784A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776—Validation; Performance evaluation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20092—Interactive image processing based on input by user

Definitions

  • the present disclosure relates to the technical field of defect detection, and in particular, to a defect detection method and device, a computer-readable storage medium, and electronic equipment.
  • the purpose of the present disclosure is to provide a defect detection method, a defect detection device, a computer-readable medium, and an electronic device, thereby improving detection efficiency at least to a certain extent and reducing development costs.
  • a defect detection method including:
  • Defect detection is performed on various types of images by using the defect detection model corresponding to each type of the image to obtain a defect detection result.
  • a defect detection device including:
  • An image acquisition module configured to acquire a detection task, and acquire multiple types of images corresponding to the detection task
  • a model acquisition module configured to acquire defect detection models trained by the same initial model corresponding to the types of the images
  • the defect detection module is configured to use the defect detection model corresponding to each type of the image to perform defect detection on various types of images to obtain defect detection results.
  • a computer-readable medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.
  • an electronic device characterized in that it includes:
  • the memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.
  • FIG. 1 shows a schematic diagram of an exemplary system architecture to which embodiments of the present disclosure can be applied
  • FIG. 2 shows a schematic diagram of an electronic device to which an embodiment of the present disclosure can be applied
  • FIG. 3 schematically shows a flowchart of a defect detection method in an exemplary embodiment of the present disclosure
  • FIG. 4 schematically shows a flow chart of an algorithm in a defect detection method in an exemplary embodiment of the present disclosure
  • Fig. 5 schematically shows the framework diagram of the defect gold policy model in the exemplary embodiment of the present disclosure
  • Fig. 6 schematically shows an operator interface diagram in an exemplary embodiment of the present disclosure
  • Fig. 7 schematically shows a composition diagram of a defect detection method device in an exemplary embodiment of the present disclosure.
  • Example embodiments will now be described more fully with reference to the accompanying drawings.
  • Example embodiments may, however, be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concept of example embodiments to those skilled in the art.
  • the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
  • Fig. 1 shows a schematic diagram of a system architecture of an exemplary application environment in which a defect detection method and device according to an embodiment of the present disclosure can be applied.
  • the system architecture 100 may include one or more of terminal devices 101 , 102 , 103 , a network 104 and a server 105 .
  • the network 104 is used as a medium for providing communication links between the terminal devices 101 , 102 , 103 and the server 105 .
  • Network 104 may include various connection types, such as wires, wireless communication links, or fiber optic cables, among others.
  • the terminal devices 101, 102, 103 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, tablet computers and so on. It should be understood that the numbers of terminal devices, networks and servers in Fig. 1 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.
  • the server 105 may be a server cluster composed of multiple servers.
  • the defect detection methods provided by the embodiments of the present disclosure are generally executed by the terminal devices 101 , 102 , and 103 , and correspondingly, the defect detection devices are generally disposed in the terminal devices 101 , 102 , and 103 .
  • the defect detection method provided by the embodiment of the present disclosure can also be executed by the server 105, and correspondingly, the defect detection device can also be set in the server 105, which is not mentioned in this exemplary embodiment. Make a special limit.
  • the user can obtain various types of images corresponding to the product through the terminal equipment 101, 102, 103 according to the product information and upload them to the server 105, and the server implements
  • the defect detection method provided by the example obtains the defect detection result, and transmits the defect detection result to the terminal devices 101, 102, 103 and so on.
  • An exemplary embodiment of the present disclosure provides an electronic device for implementing a defect detection method, which may be the terminal devices 101 , 102 , 103 or the server 105 in FIG. 1 .
  • the electronic device includes at least a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the defect detection method by executing the executable instructions.
  • the mobile terminal 200 in FIG. 2 As an example below, the structure of the electronic device will be exemplarily described. Those skilled in the art will appreciate that, in addition to components specifically intended for mobile purposes, the configuration in Fig. 2 can also be applied to equipment of a stationary type.
  • the mobile terminal 200 may include more or fewer components than shown, or combine some components, or separate some components, or arrange different components.
  • the illustrated components may be realized in hardware, software, or a combination of software and hardware.
  • the interface connection relationship among the various components is only schematically shown, and does not constitute a structural limitation on the mobile terminal 200 . In some other implementation manners, the mobile terminal 200 may also adopt an interface connection manner different from that in FIG. 2 , or a combination of multiple interface connection manners.
  • the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (Universal Serial Bus, USB) interface 230, a charging management module 240, a power management module 241, battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, speaker 271, receiver 272, microphone 273, earphone interface 274, sensor module 280, display screen 290, camera module 291, indication device 292, motor 293, button 294, subscriber identification module (subscriber identification module, SIM) card interface 295, etc.
  • the sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyro sensor 2803, and the like.
  • the processor 210 may include one or more processing units, for example: the processor 210 may include an application processor (Application Processor, AP), a modem processor, a graphics processor (Graphics Processing Unit, GPU), an image signal processor (Image Signal Processor, ISP), controller, video codec, digital signal processor (Digital Signal Processor, DSP), baseband processor and/or neural network processor (Neural-Network Processing Unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
  • an application processor Application Processor, AP
  • modem processor a graphics processor
  • ISP image signal processor
  • ISP image Signal Processor
  • controller video codec
  • digital signal processor Digital Signal Processor
  • DSP Digital Signal Processor
  • NPU neural network Processing Unit
  • different processing units may be independent devices, or may be integrated in one or more processors.
  • NPU is a neural network (Neural-Network, NN) computing processor.
  • NN neural network
  • Applications such as intelligent cognition of the mobile terminal 200 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.
  • a memory is provided in the processor 210 .
  • the memory can store instructions for realizing six modular functions: detection instruction, connection instruction, information management instruction, analysis instruction, data transmission instruction and notification instruction, and the execution is controlled by the processor 210 .
  • the charging management module 240 is configured to receive charging input from the charger.
  • the power management module 241 is used for connecting the battery 242 , the charging management module 240 and the processor 210 .
  • the power management module 241 receives the input of the battery 242 and/or the charging management module 240 to supply power for the processor 210 , the internal memory 221 , the display screen 290 , the camera module 291 and the wireless communication module 260 .
  • the wireless communication function of the mobile terminal 200 can be realized by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, a modem processor, a baseband processor, and the like.
  • the antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals;
  • the mobile communication module 250 can provide solutions for wireless communication including 2G/3G/4G/5G applied on the mobile terminal 200;
  • the modem processor can include Modulator and demodulator;
  • the wireless communication module 260 can provide wireless local area network (Wireless Local Area Networks, WLAN) (such as wireless fidelity (Wireless Fidelity, Wi-Fi) network), Bluetooth (Bluetooth) applied on the mobile terminal 200 , BT) and other wireless communication solutions.
  • the antenna 1 of the mobile terminal 200 is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, so that the mobile terminal 200 can communicate with the network and other devices through wireless communication technology.
  • the mobile terminal 200 realizes the display function through the GPU, the display screen 290 and the application processor.
  • the GPU is a microprocessor for image processing, and is connected to the display screen 290 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 210 may include one or more GPUs that execute program instructions to generate or change display information.
  • the mobile terminal 200 can realize the shooting function through the ISP, the camera module 291 , the video codec, the GPU, the display screen 290 and the application processor.
  • the ISP is used to process the data fed back by the camera module 291; the camera module 291 is used to capture still images or videos; the digital signal processor is used to process digital signals, and can process other digital signals in addition to digital image signals;
  • a codec is used to compress or decompress digital video, and the mobile terminal 200 may also support one or more video codecs.
  • the external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the mobile terminal 200.
  • the external memory card communicates with the processor 210 through the external memory interface 222 to realize the data storage function. Such as saving music, video and other files in the external memory card.
  • the internal memory 221 may be used to store computer-executable program codes including instructions.
  • the internal memory 221 may include an area for storing programs and an area for storing data.
  • the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like.
  • the data storage area can store data created during the use of the mobile terminal 200 (such as audio data, phonebook, etc.) and the like.
  • the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, Universal Flash Storage (UFS) and the like.
  • the processor 210 executes various functional applications and data processing of the mobile terminal 200 by executing instructions stored in the internal memory 221 and/or instructions stored in a memory provided in the processor.
  • the mobile terminal 200 can implement audio functions through an audio module 270 , a speaker 271 , a receiver 272 , a microphone 273 , an earphone interface 274 , and an application processor. Such as music playback, recording, etc.
  • the depth sensor 2801 is used to obtain the depth information of the scene.
  • a depth sensor can be disposed on the camera module 291 .
  • the pressure sensor 2802 is used to sense the pressure signal and convert the pressure signal into an electrical signal.
  • a pressure sensor 2802 may be disposed on the display screen 290 .
  • the gyro sensor 2803 can be used to determine the motion posture of the mobile terminal 200 .
  • the angular velocity of the mobile terminal 200 around three axes ie, x, y and z axes
  • the gyroscope sensor 2803 can be used for shooting anti-shake, navigation, somatosensory game scenes, etc.
  • sensors with other functions can also be set in the sensor module 280 according to actual needs, such as air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor, bone conduction sensor, etc. sensors etc.
  • the mobile terminal 200 may also include other devices providing auxiliary functions.
  • the key 294 includes a power key, a volume key, etc.
  • the user may input key signals related to user settings and function control of the mobile terminal 200 through key input.
  • AOI Automatic Optical Inspection, automatic optical inspection
  • TDI Time Delayed and Integration, time delay integration
  • DM digital micro, digital microcomputer
  • optical AOI equipment will be used to take pictures of the circuits on the glass substrate.
  • the industrial camera CCD will take pictures of each glass substrate (covered with circuits) to generate the entire glass plate.
  • the grayscale image this is the DM image. Only one DM image will be generated for each glass substrate, and the DM image is generally a single-channel grayscale image above 1500x1500. If there are bad suspicious points, use a microscope to capture the TDI image and AOI color image of the point.
  • the TDI image is generally a single-channel 64x64 grayscale image, which is a "fuzzy image" at the defect point. Small, easy to generate and transfer, each glass substrate will generate approximately 500 TDI images.
  • the AOI color image is generally a three-channel 1360x1020 RGB image, which is a high-definition display image of the defect point. Because it is not easy to generate and transmit, generally about 150 AOI color images are captured for each glass substrate.
  • defect detection method and defect detection device according to the exemplary embodiments of the present disclosure will be described in detail below.
  • Fig. 3 shows the process flow of a defect detection method in this exemplary embodiment, including the following steps:
  • Step S310 obtaining a detection task, and obtaining multiple types of images corresponding to the detection task
  • Step S320 acquiring the defect detection models trained by the same initial model respectively corresponding to the types of the images
  • Step S330 using the defect detection model corresponding to each type of the image to perform defect detection on various types of images to obtain defect detection results.
  • the present disclosure adopts a defect detection model to complete the detection of product defects, adopts various types of images of products, and adopts different defect detection models for different types of images. Due to the different All defect detection models are trained by the same initial model, that is, the same algorithm can be used to obtain the defect detection model, which saves development resources and the cost of defect detection.
  • step S310 a detection task is obtained, and multiple types of images corresponding to the detection task are obtained.
  • a detection task may be obtained first, and then product information corresponding to the detection task may be obtained according to the detection task, and then multiple types of images of the product may be obtained according to the product information.
  • the different types of images may be represented by different resolutions, different numbers of channels, or different resolutions and different numbers of channels.
  • the above product information includes product name, product site and other information, and the product information can also be customized according to user needs, which is not specifically limited in this exemplary embodiment.
  • the above-mentioned various types of images can be obtained by taking photos of the above-mentioned products with cameras using different configuration parameters.
  • the above-mentioned various types of images can include DM images, TDI images, and AOI color images corresponding to the products. There is no specific limitation in this example implementation.
  • the above product information may be obtained according to the above product information field, and the server receives the detection task sent by the training system and analyzes the task. In order to obtain the product name/site name through the detection task, the product information word is included in the detection task. By parsing the fields, the server can clarify the site name and product name corresponding to this detection task.
  • to obtain multiple types of images corresponding to products according to the above product information may first obtain the path information corresponding to various types of images in the product information, and then obtain the product in the storage library according to the above path information corresponding to various types of images.
  • the defect detection method of the present disclosure may further include preprocessing the acquired images of different types, specifically, the above-mentioned The number of channels for both types of images is set to be the same. You can first determine the maximum number of channels in each type of image, and set the number of channels of all images to the above-mentioned maximum number of channels. Then the size of each type of image can be adjusted to the preset size corresponding to each type.
  • DM images and TDI images are single-channel images
  • AOI color images are three-channel RGB images.
  • channel expansion processing is performed on the DM image and the TDI image, that is, the single channel is copied three times and converted into a three-channel image for processing.
  • Experimental verification shows that this processing method does not affect the accuracy of the algorithm.
  • a resolution threshold can be set according to the defect detection model, and the threshold can be determined by the hardware system where the model resides, or can be customized according to user requirements, which is not specifically limited in this example embodiment.
  • the above-mentioned resolution threshold may be that the shortest side is 1000-1200, for example, 1020, 1036, etc., which is not specifically limited in this example implementation.
  • Each type of image will be scaled to a different preset resolution, and will be processed into a three-channel image input to the defect detection model.
  • the DM map will be scaled to a three-channel image input defect detection model with the shortest side of 1200. Since the resolution of the TDI map is low, excessive zooming will cause image distortion.
  • the TDI map can be scaled to 256x256
  • the three-channel image input defect detection model, the AOI color map can be changed, for example, keep 1360x1020, or you can input the three-channel image whose shortest side is greater than or equal to 1000 to the defect detection model.
  • each type of image will be normalized accordingly.
  • the preset resolutions corresponding to different types of images can also be customized according to user requirements, and are not specifically limited in this example embodiment.
  • a preprocessing parameter modification interface may also be provided for the preprocessing operation, so that the user can adjust parameter information of the preprocessing operation through the preprocessing parameter modification interface.
  • the preprocessing parameter modification interface may include a resolution modification interface and a channel number modification interface, and other modification interfaces may also be added according to user needs, which are not specifically limited in this example embodiment, and the user may have passed the above preprocessing parameter modification interface Modify the above parameters such as resolution and number of channels.
  • the server may first obtain multiple product information corresponding to the detection task; obtain images corresponding to each product information with the same resolution and the same number of channels. That is, the detection task includes the defect detection task of different products, where the different types of images acquired can be different products or different sites in the images, and the resolution and number of channels of the images are the same, that is, at least one of the sites and products is different , but the same image in image form.
  • step S320 defect detection models respectively corresponding to the types of the images and obtained by training with the same initial model are acquired.
  • the person then acquires the defect detection model corresponding to the above-mentioned image type from a storage library according to the above-mentioned image type.
  • the above-mentioned multiple types of defect detection models may be obtained by training from the same initial model.
  • an initial model may be obtained first, and then training data corresponding to various types of images may be obtained, wherein the training The data may include normal images and image normal information; defective images and image defect information.
  • the server may respectively use training data corresponding to various types of images to train the initial model to obtain defect detection models corresponding to various types of images.
  • the training process is the process of using the training data to modify the model parameters. Therefore, after training, due to the different training data, the internal configuration parameters of the initial model are modified differently, and defect detection models corresponding to various types of images can be obtained.
  • the DM map corresponds to the DM model xx
  • the TDI map corresponds to the TDI model xx
  • the AOI color map corresponds to the AOI model xx, where xx represents the version information of the model.
  • the commonly used method is the target detection algorithm.
  • the defect detection scene normal images often occupy 60% or more of all images. These normal images have no "target” for the target detection algorithm. Yes, how to correctly classify normal images and defect images is a big problem in the defect detection scene.
  • the common method is to classify the images first, and divide them into normal images and defect images, and then input the defect images into the defect detection algorithm to perform The subdivision and positioning of defect categories is complicated.
  • defects only exist in a small number of samples, and 50% or more of the input images are normal and defect-free images, while traditional target detection algorithms need to mark the target category and target position in each image.
  • target For normal images in defect detection, there is no "target” required. Therefore, in this example embodiment, the entire picture of a normal image is input into the network as a target, and a class of normal images can be added in addition to the defect category.
  • the "defect" position of this type of image is the entire picture, that is, in Normal images and image normal information are set in the training data.
  • the accuracy rate of the normal images is basically above 95%, and the recall rate is above 85%.
  • the above-mentioned defect detection model is mainly a neural network model based on deep learning.
  • a defect detection model may be based on a feed-forward neural network.
  • Feedforward networks can be implemented as acyclic graphs where nodes are arranged in layers.
  • a feedforward network topology includes an input layer and an output layer, separated by at least one hidden layer.
  • the hidden layer transforms the input received by the input layer into a representation useful for generating the output in the output layer.
  • Network nodes are fully connected to nodes in adjacent layers via edges, but no edges exist between nodes within each layer.
  • the output of the defect detection model can take various forms, which are not limited by the present disclosure.
  • the defect detection model can also include other neural network models, for example, a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a generative confrontation network (GAN) model, but not limited thereto, and techniques in the art can also be used Other neural network models known to personnel.
  • CNN convolutional neural network
  • RNN recurrent neural network
  • GAN generative confrontation network
  • Defect detection models usually need to be obtained through training.
  • Using the above training algorithm to train the initial model may include the following steps: selecting a network topology; using a set of training data representing the problem modeled by the network; and adjusting the weights until the network model behaves with respect to all instances of the training data set minimum error. For example, during a supervised learning training process for a neural network, the output produced by the network in response to an input representing an instance in the training dataset is compared to the "correct" labeled output for that instance; computing the output representing the an error signal of the difference from the labeled output; and adjusting the weights associated with the connections to minimize the error when propagating the error signal back through the layers of the network. When the error for each output generated from instances of the training dataset is minimized, this initial model is considered "trained” and defined as a defect detection model, and can be used for AI inference tasks.
  • step S330 defect detection is performed on various types of images using the defect detection models corresponding to the types of images to obtain defect detection results.
  • the defect influence weights can be set for various types of images according to the configuration parameters; use the defect detection model corresponding to each type of image to perform defect detection on various types of images to obtain the references corresponding to each type of image Defect detection result; determine the defect detection result by fusion according to the defect impact weight and the reference defect detection result.
  • the defect impact weights of DM maps, TDI maps, and AOI color maps can be set to 1:2:7, or 1:3:6, and can also be customized according to user needs.
  • the real-time mode is not specifically limited.
  • the defect detection model corresponding to each type of image is used to perform defect detection on various types of images to obtain defect detection results, and fusion processing may not be performed after obtaining the defect detection results of various types of images.
  • the defect detection models obtained by training different neural network models can be used for separate detection to obtain defect categories. There are differences.
  • the above-mentioned preprocessing process has been described in detail above, which may specifically include setting the number of channels of the above-mentioned various types of images to be the same, each type of image will be scaled to a different preset resolution, etc.
  • the above-mentioned post-processing The processing operation may include setting defect influence weights for various types of images according to configuration parameters; performing defect detection on various types of images using defect detection models corresponding to each type of image to obtain reference defect detection results corresponding to each type of image; According to the defect impact weight and the reference defect detection result, the defect detection result is determined.
  • the above-mentioned defect detection model may belong to a two-stage target detection algorithm, which may specifically include a feature extraction network and a defect recognition network.
  • the feature extraction network may be used to analyze various types of Feature extraction is performed on the image to obtain the feature image; and the feature image is standardized.
  • the feature extraction network is used to perform feature extraction on the image to be detected 510, which is the feature extraction layer 520, which is composed of deep learning basic units such as convolutional layers and pooling layers, specifically similar to the classic VGG model, and the residual structure ResNet model, lightweight MobileNet model, depending on specific project requirements.
  • the feature extraction layer 520 which is composed of deep learning basic units such as convolutional layers and pooling layers, specifically similar to the classic VGG model, and the residual structure ResNet model, lightweight MobileNet model, depending on specific project requirements.
  • a convolutional network composed of blocks with a residual structure is used to extract features, and the rest of the structures are exposed to external interfaces, which can be selected in the configuration file.
  • the feature image obtained above can be sent to the candidate frame extraction network, and the candidate frames in the image are initially screened into foreground frames (defective) and background frames (no defect), and the coordinates of the candidate frames are adjusted by regression, that is, the feature image Perform screening to obtain the target feature image, and determine the defect category and coordinates of the target candidate feature image; obtain the defect detection result according to the preset screening strategy and the defect category and coordinates of the target feature image.
  • the standardization processing layer 530 is used to perform normalization processing on the above-mentioned target feature image, and ROI Pooling or ROI Align can be used, which is not limited according to the figure in this exemplary embodiment.
  • the candidate frame is processed into a uniform size and input to the defect recognition network.
  • ROI Align is used for standardization processing.
  • a normalization processing parameter adjustment interface can be configured, so that the user can adjust the parameter information of the normalization processing through the standardization processing parameter adjustment interface.
  • defect classification is performed on the standard feature images by using the defect recognition network to obtain the defect category of each feature image, and the coordinates of each feature image are determined; defect detection results are obtained according to the defect category and coordinates.
  • the above-mentioned standardized feature images can be input into the defect recognition network, which is mainly composed of a fully connected layer and softmax540.
  • the defect recognition network does not need to set up a convolutional layer, and replacing the fully connected layer with a convolutional layer will greatly improve the performance of the defect recognition network. Reduce model size.
  • the defect categories can be classified, and the coordinates can be adjusted by regression.
  • the NMS algorithm is a general algorithm in face recognition and defect detection algorithms, and is mainly used to filter a large number of feature images processed by the algorithm according to whether the coordinates of the confidence score and the overlap.
  • the specific process is: the NMS algorithm receives the coordinates of thousands of feature images obtained by the previous algorithm and the confidence score of each feature image, first takes out the one with the highest confidence in the feature image, and calculates the cross-merge comparison with the rest of the feature images in turn. , delete the feature image that exceeds the threshold (over the threshold, indicating that the two feature images have a high degree of overlap, and the feature image is the same object), and then save the feature image with the highest score. Then, take out the feature image with the highest confidence from the rest, and repeat the previous step until the loop ends. This results in non-overlapping feature images on each map.
  • the number of target feature images obtained through screening may be 10, 20, etc., which is not specifically limited in this example implementation.
  • the defect detection result can also be obtained according to the set screening strategy and the defect category and coordinates of the target feature image.
  • a preset screening strategy is used to judge and select the final defect category and coordinates.
  • the preset screening strategy includes that if the category appears in the final multiple target feature images, then it is judged as the category; if two or more specific categories appear in the candidate frame at the same time, then select one of the specific categories; press The ranking of the target feature image selects the largest category; if all of the target feature images are normal categories, then select the normal category, and the "defect" coordinates are the size of the original image.
  • the above preset screening strategy can also be customized according to user needs, that is, a customized result post-processing method to cope with various image types and scenarios, which is not specifically limited in this example embodiment.
  • a defect category and the coordinates of this category can be given to the DM diagram, TDI diagram and AOI color diagram in each of the above tasks, and then the obtained results can be packaged and pushed to the business The staff operates the system for manual review.
  • step S410 can be performed to input AOI color image, TDI image, and DM image.
  • step S420 preprocessing operation, including AOI color image preprocessing, TDI image preprocessing, DM image preprocessing, the specific process of preprocessing operation has been described in detail above, so it will not be repeated here Repeat; after that, step S430 can be executed to input the image into the defect detection model, and then step S440 and step S450 can be executed to post-process and obtain the AOI color map defect detection result, TDI map defect detection result, DM map defect detection result, post-processing Including AOI color map post-processing, TDI map post-processing, DM map post-processing, that is, the output of the model is post-processed to obtain the above defect detection results.
  • the largest display area on the interface operated by the salesman is the AOI color map 610, and the display area 620 in the upper right corner can switch the AOI color map to AI recognized images and AI unrecognized images, as well as TDI picture.
  • the defect category can be given under the thumbnails of AI-recognized images and TDI images, and the operator only needs to roughly browse the thumbnails, which greatly increases the operator's image judgment speed.
  • the operator needs to judge the image in the middle display area and give the corresponding defect category of the image.
  • the DM chart can be displayed in the display area 630 in the lower right corner, and the defect is shown above the picture, and the operator can modify the judgment function.
  • the defect detection method in the present disclosure provides an efficient and convenient processing method for DM diagrams, TDI diagrams, and AOI diagrams of screen production inspection based on specific business scenarios.
  • the joint processing of multiple different types of images greatly improves the detection efficiency of images and the efficiency of subsequent manual work.
  • the classification of normal images, defect images, and defect segmentation and location of defect images are integrated into the same end-to-end task process, which simplifies the task difficulty and improves the task processing speed.
  • the embodiment of this example also provides a defect detection device 700 , including an image acquisition module 710 , a model acquisition module 720 and a defect detection module 730 . in:
  • the image acquisition module 710 may be used to acquire a detection task, and acquire various types of images corresponding to the detection task.
  • the image acquisition module 710 can extract the product information field in the detection task to obtain product information, obtain product information corresponding to the detection task; obtain various types of images according to the product information, specifically obtain product information Various types of images of the corresponding product; among them, various types of images are obtained by using cameras with different configuration parameters to take pictures of the product, among which, various types of images include the AOI color map, TDI map and DM map of the product one or more of .
  • the image acquisition module 710 can also perform preprocessing operations on various types of images at the same time, so that the number of channels of each type of image is the same, specifically, the maximum number of channels in each type of image can be determined; for each type of image The image is pre-processed, the number of channels of each type of image is set to the maximum number of channels, and the resolution of each type of image is adjusted to a preset resolution corresponding to each type.
  • a preprocessing parameter modification interface can also be provided for the preprocessing operation, so that the user can adjust the parameter information of the preprocessing operation through the preprocessing parameter modification interface.
  • the image acquisition module 710 may acquire multiple pieces of product information corresponding to the detection task; and acquire images corresponding to each of the product information with the same resolution and the same number of channels.
  • the model obtaining module 720 may obtain defect detection models respectively corresponding to the types of the images and trained from the same initial model.
  • the defect detection module 730 is configured to use the defect detection model corresponding to each type of the image to perform defect detection on various types of images to obtain defect detection results;
  • the defect detection model corresponding to each image type is trained from the same initial model.
  • the defect detection model includes a feature extraction network and a defect recognition network
  • the defect detection module 730 can use the feature extraction network to perform feature extraction on various types of images to obtain feature images; Carry out standardization processing; use the defect recognition network to classify defects according to the standard characteristic images to obtain the defect categories of each characteristic image, and determine the coordinates of each characteristic image; obtain defect detection results according to the defect categories and coordinates.
  • the defect detection module 730 may also configure a standardization processing parameter adjustment interface, so that the user can adjust the parameter information of the standardization processing through the standardization processing parameter adjustment interface.
  • the defect detection module 730 can filter the feature image to obtain the target feature image, and determine the defect category and coordinates of the target candidate feature image; Defect categories and coordinates of policy and target feature images are used to obtain defect detection results.
  • the defect detection module 730 can calculate the confidence of each feature image; according to the confidence, use the NMS algorithm to filter the feature images to obtain the target feature image.
  • the defect detection module 730 can also set defect influence weights for various types of images according to the configuration parameters; use the defect detection models corresponding to the types of images to perform defect detection on various types of images to obtain various types of The reference defect detection results corresponding to the images respectively; the defect detection results are determined according to the defect influence weight and the reference defect detection results.
  • Exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method in this specification is stored.
  • various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code.
  • the program product runs on the terminal device, the program code is used to make the terminal device execute the above-mentioned Steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Methods" section.
  • the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two.
  • a computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
  • Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
  • program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc., as well as conventional procedural Programming language - such as "C" or a similar programming language.
  • the program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server to execute.
  • the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider). business to connect via the Internet).
  • LAN local area network
  • WAN wide area network
  • Internet service provider an Internet service provider

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

提出一种缺陷检测方法及装置、计算机可读存储介质及电子设备，方法包括：获取检测任务，并获取检测任务对应的多种类型的图像；获取与图像的类型分别对应的由同一初始模型训练所得的缺陷检测模型；利用各图像的类型对应的缺陷检测模型对各种类型的图像进行缺陷检测得到缺陷检测结果。本技术方案提高了检测效率，降低开发成本。

Description

缺陷检测方法及装置、存储介质及电子设备 技术领域
本公开涉及缺陷检测技术领域，具体而言，涉及一种缺陷检测方法及装置、计算机可读存储介质及电子设备。
背景技术
在生产加工技术领域，由于设备、参数、操作、环境干扰等环节存在的问题，会使产出的产品产生不良，随着以深度学习为代表的人工智能算法的兴起，利用深度算法学习模型来进行缺陷检测也应用的越来越广泛。
但是现有技术中，对于深度算法学习模型中的往往需要人为的筛选正常图像，且不能够对不同种类的图像采用相同的算法就进行检测，浪费人力资源，且需要开发不同的模型来实现对不同类型图像进行缺陷检测，检测效率较低，开发成本较高。
需要说明的是，在上述背景技术部分公开的信息仅用于加强对本公开的背景的理解，因此可以包括不构成对本领域普通技术人员已知的现有技术的信息。
发明内容
本公开的目的在于提供一种缺陷检测方法、缺陷检测装置、计算机可读介质和电子设备，进而至少在一定程度上提高检测效率，降低开发成本。
根据本公开的第一方面，提供一种缺陷检测方法，包括：
获取检测任务，并获取所述检测任务对应的多种类型的图像；
获取与所述图像的类型分别对应的由同一初始模型训练所得的缺陷检测模型；
利用各所述图像的类型对应的所述缺陷检测模型对各种类型的图像进行缺陷检测得到缺陷检测结果。
根据本公开的第二方面，提供一种缺陷检测装置，包括：
图像获取模块，用于获取检测任务，并获取所述检测任务对应的多种类型的图像；
模型获取模块，用于获取与所述图像的类型分别对应的由同一初始模型训练所得的缺陷检测模型；
缺陷检测模块，用于利用各所述图像的类型对应的所述缺陷检测模型对各种类型的图像进行缺陷检测得到缺陷检测结果。
根据本公开的第三方面，提供一种计算机可读介质，其上存储有计算机程序，计算机程序被处理器执行时实现上述的方法。
根据本公开的第四方面，提供一种电子设备，其特征在于，包括：
处理器；以及
存储器，用于存储一个或多个程序，当一个或多个程序被一个或多个处理器执行时，使得一个或多个处理器实现上述的方法。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本公开。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。在附图中：
图1示出了可以应用本公开实施例的一种示例性系统架构的示意图；
图2示出了可以应用本公开实施例的一种电子设备的示意图；
图3示意性示出本公开示例性实施例中一种缺陷检测方法的流程图；
图4示意性示出本公开示例性实施例中一种缺陷检测方法的中算法的流程图；
图5示意性示出本公开示例性实施例中缺陷金策模型的架构图；
图6示意性示出本公开示例性实施例中操作员界面图；
图7示意性示出本公开示例性实施例中缺陷检测方法装置的组成示意图。
具体实施方式
现在将参考附图更全面地描述示例实施方式。然而，示例实施方式能够以多种形式实施，且不应被理解为限于在此阐述的范例；相反，提供这些实施方式使得本公开将更加全面和完整，并将示例实施方式的构思全面地传达给本领域的技术人员。所描述的特征、结构或特性可以以任何合适的方式结合在一个或更多实施方式中。
此外，附图仅为本公开的示意性图解，并非一定是按比例绘制。图中相同的附图标记表示相同或类似的部分，因而将省略对它们的重复描述。附图中所示的一些方框图是功能实体，不一定必须与物理或逻辑上独立的实体相对应。可以采用软件形式来实现这些功能实体，或在一个或多个硬件模块或集成电路中实现这些功能实 体，或在不同网络和/或处理器装置和/或微控制器装置中实现这些功能实体。
图1示出了可以应用本公开实施例的一种缺陷检测方法及装置的示例性应用环境的系统架构的示意图。
如图1所示，系统架构100可以包括终端设备101、102、103中的一个或多个，网络104和服务器105。网络104用以在终端设备101、102、103和服务器105之间提供通信链路的介质。网络104可以包括各种连接类型，例如有线、无线通信链路或者光纤电缆等等。终端设备101、102、103可以是各种具有图像处理功能的电子设备，包括但不限于台式计算机、便携式计算机、智能手机和平板电脑等等。应该理解，图1中的终端设备、网络和服务器的数目仅仅是示意性的。根据实现需要，可以具有任意数目的终端设备、网络和服务器。比如服务器105可以是多个服务器组成的服务器集群等。
本公开实施例所提供的缺陷检测方法一般由终端设备101、102、103中执行，相应地，缺陷检测装置一般设置于终端设备101、102、103中。但本领域技术人员容易理解的是，本公开实施例所提供的缺陷检测方法也可以由服务器105执行，相应的，缺陷检测装置也可以设置于服务器105中，本示例性实施例中对此不做特殊限定。举例而言，在一种示例性实施例中，可以是用户通过终端设备101、102、103根据所述产品信息获取所述产品对应的多种类型的图像上传至服务器105，服务器通过本公开实施例所提供的缺陷检测方法得到缺陷检测结果，将缺陷检测结果传输给终端设备101、102、103等。
本公开的示例性实施方式提供一种用于实现缺陷检测方法的电子设备，其可以是图1中的终端设备101、102、103或服务器105。该电子设备至少包括处理器和存储器，存储器用于存储处理器的可执行指令，处理器配置为经由执行可执行指令来执行缺陷检测方法。
下面以图2中的移动终端200为例，对电子设备的构造进行示例性说明。本领域技术人员应当理解，除了特别用于移动目的的部件之外，图2中的构造也能够应用于固定类型的设备。在另一些实施方式中，移动终端200可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件、软件或软件和硬件的组合实现。各部件间的接口连接关系只是示意性示出，并不构成对移动终端200的结构限定。在另一些实施方式中，移动终端200也可以采用与图2不同的接口连接方式，或多种接口连接方式的组合。
如图2所示，移动终端200具体可以包括：处理器210、内部存储器221、外部存储器接口222、通用串行总线(Universal Serial Bus，USB)接口230、充电管理模块240、电源管理模块241、电池242、天线1、天线2、移动通信模块250、无线通信模块260、音频模块270、扬声器271、受话器272、麦克风273、耳机接口274、传感器模块280、显示屏290、摄像模组291、指示器292、马达293、按键294以及用户标识模块(subscriber identification module，SIM)卡接口295等。其中传感器模块280可以包括深度传感器2801、压力传感器2802、陀螺仪传感器2803等。
处理器210可以包括一个或多个处理单元，例如：处理器210可以包括应用处理器(Application Processor，AP)、调制解调处理器、图形处理器(Graphics Processing Unit，GPU)、图像信号处理器(Image Signal Processor，ISP)、控制器、视频编解码器、数字信号处理器(Digital Signal Processor，DSP)、基带处理器和/或神经网络处理器(Neural-Network Processing Unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
NPU为神经网络(Neural-Network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现移动终端200的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。
处理器210中设置有存储器。存储器可以存储用于实现六个模块化功能的指令：检测指令、连接指令、信息管理指令、分析指令、数据传输指令和通知指令，并由处理器210来控制执行。
充电管理模块240用于从充电器接收充电输入。电源管理模块241用于连接电池242、充电管理模块240与处理器210。电源管理模块241接收电池242和/或充电管理模块240的输入，为处理器210、内部存储器221、显示屏290、摄像模组291和无线通信模块260等供电。
移动终端200的无线通信功能可以通过天线1、天线2、移动通信模块250、无线通信模块260、调制解调处理器以及基带处理器等实现。其中，天线1和天线2用于发射和接收电磁波信号；移动通信模块250可以提供应用在移动终端200上的包括2G/3G/4G/5G等无线通信的解决方案；调制解调处理器可以包括调制器和解调器；无线通信模块260可以提供应用在移动终端200上的包括无线局域网(Wireless Local Area Networks，WLAN)(如无线保真(Wireless Fidelity，Wi-Fi)网络)、蓝牙 (Bluetooth，BT)等无线通信的解决方案。在一些实施例中，移动终端200的天线1和移动通信模块250耦合，天线2和无线通信模块260耦合，使得移动终端200可以通过无线通信技术与网络以及其他设备通信。
移动终端200通过GPU、显示屏290及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏290和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器210可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
移动终端200可以通过ISP、摄像模组291、视频编解码器、GPU、显示屏290及应用处理器等实现拍摄功能。其中，ISP用于处理摄像模组291反馈的数据；摄像模组291用于捕获静态图像或视频；数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号；视频编解码器用于对数字视频压缩或解压缩，移动终端200还可以支持一种或多种视频编解码器。
外部存储器接口222可以用于连接外部存储卡，例如Micro SD卡，实现扩展移动终端200的存储能力。外部存储卡通过外部存储器接口222与处理器210通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。
内部存储器221可以用于存储计算机可执行程序代码，可执行程序代码包括指令。内部存储器221可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储移动终端200使用过程中所创建的数据(比如音频数据，电话本等)等。此外，内部存储器221可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(Universal Flash Storage，UFS)等。处理器210通过运行存储在内部存储器221的指令和/或存储在设置于处理器中的存储器的指令，执行移动终端200的各种功能应用以及数据处理。
移动终端200可以通过音频模块270、扬声器271、受话器272、麦克风273、耳机接口274及应用处理器等实现音频功能。例如音乐播放、录音等。
深度传感器2801用于获取景物的深度信息。在一些实施例中，深度传感器可以设置于摄像模组291。
压力传感器2802用于感受压力信号，可以将压力信号转换成电信号。在一些实施例中，压力传感器2802可以设置于显示屏290。压力传感器2802的种类很多，如电阻式压力传感器，电感式压力传感器，电容式压力传感器等。
陀螺仪传感器2803可以用于确定移动终端200的运动姿态。在一些实施方式中，可以通过陀螺仪传感器2803确定移动终端200围绕三个轴(即，x，y和z轴)的角速度。陀螺仪传感器2803可以用于拍摄防抖、导航、体感游戏场景等。
此外，还可以根据实际需要在传感器模块280中设置其他功能的传感器，例如气压传感器、磁传感器、加速度传感器、距离传感器、接近光传感器、指纹传感器、温度传感器、触摸传感器、环境光传感器、骨传导传感器等。
移动终端200中还可包括其它提供辅助功能的设备。例如，按键294包括开机键，音量键等，用户可以通过按键输入，产生与移动终端200的用户设置以及功能控制有关的键信号输入。再如，指示器292、马达293、SIM卡接口295等。
在相关技术中，对于一个产品，会从不同维度(整体/局部、高/低分辨率、灰度/彩图)进行产品拍照检测，在生产屏幕过程中，由于设备、参数、操作、环境干扰等环节存在的问题，会使产出的产品产生不良，每段工艺后利用光学(AOI(Automated Optical Inspection，自动光学检测))检测后，都有许多不同类型(AOI(Automated Optical Inspection，自动光学检测)彩图、TDI(Time Delayed and Integration，时间延迟积分)图、DM(digital micro，数字微型计算机)图)图像数据产生，需要专业的操作员对这些图像进行不良判级，随着以深度学习为代表的人工智能算法的兴起，将AI(Artificial Intelligence，人工智能)算法引入到不良图像判级过程中，进行不良图像自动检测的系统(ADC(Automatic defect detection and classification system，自动缺陷检测与分类系统))应运而生。
在屏幕生产过程中，在质检环节，会使用光学AOI设备对玻璃基板上的电路进行拍照检测，首先会由工业相机CCD对每个玻璃基板(上面覆有电路)进行拍照，生成整个玻璃板的灰度图，这就是DM图。每个玻璃基板只会生成一张DM图，DM图一般为单通道的1500x1500以上的灰度图。若有不良可疑点位，则使用显微镜进行该点位的TDI图和AOI彩图抓取，TDI图一般为单通道的64x64的灰度图像，是缺陷点处的“模糊图像”，由于图像较小，易于生成和传输，每个玻璃基板会生成大约500张的TDI图。AOI彩图一般为三通道的1360x1020的RGB图像，是缺陷点出的高清显示图，由于不易生成和传输，一般为每个玻璃基板抓取150张左右AOI彩图。
在生产过程中，往往由AOI设备抓取每个玻璃基板的以上三种图像后，由人工对以上三种类型的图像进行判别，确定每张图像是否具有真实的不良，并给出不良 的具体类别。在一个工厂的生产中，每天会产生上百万张的图像，需要投入大量人力进行不良的判级检测，而且由于人的精力有限，往往会出现错判和漏判的情况，最终影响产品的良率。
下面对本公开示例性实施方式的缺陷检测方法和缺陷检测装置进行具体说明。
图3示出了本示例性实施方式中一种缺陷检测方法的流程，包括以下步骤：
步骤S310，获取检测任务，并获取所述检测任务对应的多种类型的图像；
步骤S320，获取与所述图像的类型分别对应的由同一初始模型训练所得的缺陷检测模型；
步骤S330，利用各所述图像的类型对应的所述缺陷检测模型对各种类型的图像进行缺陷检测得到缺陷检测结果。
相较于现有技术，本公开采用了缺陷检测模型来完成对产品缺陷的检测，采用了产品的多种类型的图像，对不同类型的图像采用了不同的缺陷检测模型，由于本公开的不同的缺陷检测模型均由同一个初始模型训练得到，即在获取缺陷检测模型可以采用相同的算法来获取，节约了开发资源，同时节约了缺陷检测的成本。
在步骤S310中，获取检测任务，并获取所述检测任务对应的多种类型的图像。
在本公开的一种示例实施方式中，可以首先获取检测任务，然后根据检测任务获取与检测任务对应的产品信息，然后根据产品信息获取该产品的多种类型的图像。其中图像的类型不同可以表现为分辨率不同、通道数不同、或者分辨率与通道数均不相同。
在本示例实施方式中，上述产品信息包括产品名称、产品的站点等信息，产品信息还可以根据用户需求进行自定义，在本示例实施方式中不做具体限定。其中上述各种类型的图像可以是有采用不同配置参数的相机对上述产品进行拍照获取，具体而言，上述多种类型的图像可以包括产品对应的DM图、TDI图、AOI彩图等，在本示例实施方式中不做具体限定。
在本公开的一种示例实施方式中，可以根据上述产品信息字段获取上述产品信息，服务端接收训练系统发送的检测任务，并对任务进行解析。为了通过检测任务来产品名/站点名，检测任务中包含了产品信息字。通过对字段的解析，服务端能明确此次检测任务对应的站点名和产品名。
在本示例实施方式中，具体而言，根据上述产品信息获取产品对应的多种类型的图像可以首先获取产品信息中各种类型图像对应的路径信息，然后根据上述路径 信息在存储库中获取产品对应的多种类型的图像。
在本示例实施方式中，在获取到检测任务对应的不同类型的图像之后，本公开的缺陷检测方法还可以包括对上述获取到的不同类型的图像及进行预处理，具体而言，将上述各种类型的图像的通道数设置为相同。可以首先确定各中类型图像中的最大通道数，将所有图像的通道数设置为上述最大通道数。然后可以将各类型的图像的尺寸调整为各类型分别对应的预设尺寸。
举例而言，多种类型的图像包括DM图、TDI图、AOI彩图为例进行详细说明，其中，DM图和TDI图是单通道图像，而AOI彩图是三通道的RGB图像，为兼容以上三种图像，在预处理操作中，对DM图和TDI图进行通道扩充处理，即将单通道复制三份，转换为三通道图像进行处理，经实验验证，该处理方式不影响算法精度。
进一步的，可以根据缺陷检测模型设置一分辨率阈值，该阈值可以由模型所在的硬件系统所决定，也可以根据用户的需求进行自定义，在本示例实施方式中不做具体限定。
举例而言，其中上述分辨率阈值可以为最短边为1000～1200，例如，1020、1036等，在本示例实施方式中不做具体限定。对每种类型的图片都会被缩放位不同的预设分辨率，并且都会被处理成三通道图像输入至缺陷检测模型。举例而言，DM图会被缩放至最短边为1200的三通道图像输入缺陷检测模型，由于TDI图的分辨率较低，过分放大会使得图像失真，因此，可以将TDI图会缩放至256x256的三通道图像输入缺陷检测模型，AOI彩图可以可做改变，如，保持1360x1020，也可以将其所放置最短边大于等于1000的三通道图像输入缺陷检测模型。同时，依据每个数据集的均值和方差，每种类型的图像会进行相应的归一化处理。其中不同类型图像对应的预设分辨率还可以根据用户需求进行自定义，在本示例实施方式中不做具体限定。
在本示例实施方式中，还可以为预处理操作配合一预处理参数修改界面，以使得用户能够通过预处理参数修改界面调整预处理操作的参数信息。其中预处理参数修改界面中可以包括分辨率修改界面以及通道数修改界面，还可以根据用户需求增加其他修改界面，在本示例实施方式中不做具体限定，用户可以已通过上述预处理参数修改界面修改上述上述分辨率、通道数等参数。
在本公开的另一种示例实施方式中，服务器可以首先获取检测任务对应的多个 产品信息；获取各产品信息对应的分辨率和通道数均相同的图像。即检测任务包括了对不同产品的缺陷检测任务，其中，获取的不同类型的图像可以为图像中的产品不同或者站点不同，图像的分辨率和通道数均相同，即站点和产品中至少一个不同，但是图像形式相同的图像。
在步骤S320中，获取与所述图像的类型分别对应的由同一初始模型训练所得的缺陷检测模型。
在本公开的一种示例实施方式中，根据上述产品和站点获取模型获取路径，人后根据上述图像的类型从存储库中获取与上述图像类型对应的缺陷检测模型。
在本示例实施方式中，上述多种类型的缺陷检测模型可以是由同一个初始模型训练得到的，具体可以为，首先获取一个初始模型，然后获取各种类型的图像对应的训练数据，其中训练数据中可以包括正常图像和图像正常信息；缺陷图像和图像缺陷信息。服务器可以分别利用各种类型的图像对应的训练数据对初始模型进行训练得到各种类型图像对应的缺陷检测模型。
在训练之前，各种类型图像对应的初始模型的参数完全相同，无需对多种类型的图像分别设计不通的初始模型，节约了设计成本。训练过程即利用训练数据修改模型参数的过程，因此，在训练后，由于训练数据不同，对初始模型内部的配置参数修改不同，能够得到各种类型图像分别对应的缺陷检测模型。
举例而言，DM图对应DM模型xx，TDI图对应TDI模型xx，AOI彩图对应AOI模型xx，其中xx表示模型的版本信息，该版本在算法调用系统传入模型路径消息时确定，即针对每一个特定的算法任务，都会有一组特定版本的三个模型信息传入。
在进行缺陷检测时，常用的方法为目标检测算法，在缺陷检测场景中，正常图像往往占据了所有图像的60％甚至以上的比例，这些正常图像对于目标检测算法而言，是没有“目标”的，如何正确的分类正常图像与缺陷图像，是缺陷检测场景中的一大难题，常用的方法为先对图像进行分类，分为正常图像和缺陷图像，缺陷图像再输入至缺陷检测算法，进行缺陷类别的细分和定位，处理流程复杂。
在缺陷检测的场景中，缺陷只存在于少部分的样本中，50％甚至以上的输入图像都是正常无缺陷的图像，而传统的目标检测算法需要在每张图中标注目标类别与目标位置，对于缺陷检测中的正常图像而言，并不存在所需要的“目标”。因此，在本示例实施方式中将正常图像的整张图作为一个目标输入至网络，除了缺陷类别之 外，可以增加一类正常图像，这类图像的“缺陷”位置为整张图，即在训练数据中设置了正常图像与图像正常信息。以使得上述缺陷检测模型能够解决正常图与缺陷图的分类问题，其中，正常图的准确率基本在95％以上，召回率在85％以上。
在本示例实施方式中，上述缺陷检测模型主要是基于深度学习的神经网络模型。例如，缺陷检测模型可以是基于前馈神经网络的。前馈网络可以被实现为无环图，其中节点布置在层中。通常，前馈网络拓扑包括输入层和输出层，输入层和输出层通过至少一个隐藏层分开。隐藏层将由输入层接收到的输入变换为对在输出层中生成输出有用的表示。网络节点经由边缘全连接至相邻层中的节点，但每个层内的节点之间不存在边缘。在前馈网络的输入层的节点处接收的数据经由激活函数被传播(即，“前馈”)至输出层的节点，所述激活函数基于系数(“权重”)来计算网络中的每个连续层的节点的状态，所述系数分别与连接这些层的边缘中的每一个相关联。缺陷检测模型的输出可以采用各种形式，本公开对此不作限制。缺陷检测模型还可以包括其他神经网络模型，例如，卷积神经网络(CNN)模型、循环神经网络(RNN)模型、生成式对抗网络(GAN)模型，但不限于此，也可以采用本领域技术人员公知的其他神经网络模型。
缺陷检测模型通常需要通过训练获得。在上述利用训练算法对初始模型进行训练可以包括如下步骤：选择网络拓扑；使用表示被网络建模的问题的一组训练数据；以及调节权重，直到网络模型针对训练数据集的所有实例表现为具有最小误差。例如，在用于神经网络的监督式学习训练过程期间，将由网络响应于表示训练数据集中的实例的输入所产生的输出与该实例的“正确”的已标记输出相比较；计算表示所述输出与已标记输出之间的差异的误差信号；以及当将误差信号向后传播穿过网络的层时，调节与所述连接相关联的权重以最小化该误差。当从训练数据集的实例中生成的每个输出的误差被最小化时，该初始模型被视为“已经过训练”并定义为缺陷检测模型，并可以用于人工智能推理任务。
在步骤S330中，利用各图像的类型对应的缺陷检测模型对各种类型的图像进行缺陷检测得到缺陷检测结果。
在本示例实施方式中，可以根据配置参数对各种类型的图像设置缺陷影响权重；利用各图像的类型对应的缺陷检测模型对各种类型的图像进行缺陷检测得到各类型的图像分别对应的参考缺陷检测结果；根据缺陷影响权重和参考缺陷检测结果进行融合确定缺陷检测结果。
举例而言，可以将DM图、TDI图、AOI彩图的缺陷影响权重设置为1:2:7，也可以设置为1:3:6，还可以根据用户的需求进行自定义，在本示例实时方式中不做具体限定。
在一种示例实施方式中，利用各图像的类型对应的缺陷检测模型对各种类型的图像进行缺陷检测得到缺陷检测结果，可以在获取各种类型的图像的缺陷检测结果之后不做融合处理。
在针对上述DM图、TDI图、AOI彩图进行缺陷检测时，可以采取不同的神经网络模型进行训练得到的缺陷检测模型进行分别检测，从而获得缺陷类别，但从缺陷识别准确度和一致性上均有差异。
本公开将同一产品不同类型的图片通过同一初始神经网络模型进行分别训练，并进行特定的预处理和后处理等设计，大大提高了缺陷检测的准确性。其中，上述预处理过程上述已经进行了详细说明，具体可以包括将上述各种类型的图像的通道数设置为相同，对每种类型的图片都会被缩放位不同的预设分辨率等，上述后处理操作可以包括根据配置参数对各种类型的图像设置缺陷影响权重；利用各图像的类型对应的缺陷检测模型对各种类型的图像进行缺陷检测得到各类型的图像分别对应的参考缺陷检测结果；根据缺陷影响权重和参考缺陷检测结果进行融合确定缺陷检测结果。
需要说明的是，上述预处理和后处理上述已经进行了详细说明，因此，此处不再赘述。
在本示例实施方式中，参照图5所示，上述缺陷检测模型可以属于两阶段的目标检测算法，具体可以包括特征提取网络和缺陷识别网路，其中，首先可以利用特征提取网络对各种类型的图像进行特征提取得到特征图像；并对特征图像进行标准化处理。
具体而言，特征提取网络用于对待检测图像510进行特征提取，为特征提取层520，由卷积层、池化层等深度学习基本单元组成，具体的类似经典的VGG模型，残差结构的ResNet模型，轻量化的MobileNet模型，依据具体的项目需求而定。本实例中使用由残差结构的block组成的卷积网络提取特征，其余结构均对外暴露接口，可在配置文件中进行选择。可以将上述得到的特征图像送入候选框提取网络，将图像中的候检框初步筛选为前景框(有缺陷)和背景框(无缺陷)，并且回归调整候选框的坐标，即对特征图像进行筛选得到目标特征图像，并确定目标候特征图 像的缺陷类别和坐标；根据预设筛选策略和目标特征图像的缺陷类别和坐标得到缺陷检测结果。
在本示例实施方式中，利用标准化处理层530在对上述目标特征图像进行标准化处理，可以使用ROI Pooling或ROI Align，在本示例实施方式中不做据图限定。将候选框处理为统一大小，输入至缺陷识别的网络。本实例中使用ROI Align进行标准化处理，在本示例实施方式中，可以配置一标准化处理参数调整界面，以使得用户能够通过准化处理参数调整界面调整标准化处理的参数信息。
在本示例实施方式中，利用缺陷识别网络根据标准后的特征图像进行缺陷分类得到个特征图像的缺陷类别，并确定各特征图像的坐标；根据缺陷类别和坐标得到缺陷检测结果。
具体而言，可以将上述标准化后的特征图像输入至缺陷识别网络中，该网络主要由全连接层和softmax540组成，缺陷识别网络无需设置卷积层，利用全连接层替换为卷积层会大幅减小模型大小。在缺陷识别网络结构中可以对缺陷类别进行分类，并且可以对坐标进行回归调整。
可以得到约2000个带有缺陷类别和坐标的特征图像，先通过NMS算法进行候特征图像的过滤，然后进行阈值调整策略。依据每种类型的图像多次实验的结果，提出一种特殊的阈值筛选策略，对每种图像中的每个类别选取合适的阈值对缺陷类别筛选得到目标特征图像。上述阈值可以根据用户的需求进行自定义，在本示例实施方式中不做具体修改。
在本示例实施方式中，NMS算法为人脸识别和缺陷检测算法中的一种通用算法，主要用于将算法处理的大量特征图像按照置信度得分和的坐标是否重叠进行过滤。具体流程为：NMS算法接收前文算法处理得到的数千个特征图像坐标和每个特征图像的置信度得分，首先取出特征图像中置信度最大的那个，将他依次同其余特征图像计算交并比，删除掉超过阈值的特征图像(超过阈值，说明两个特征图像重合度很高，特征图像中的为同一物体)，然后，将得分最高的特征图像保存。接着，再从剩余的中取出置信度最大的那个特征图像，重复上一步，直到循环结束。这样可以获得每张图上不重叠的特征图像。
在本示例实施方式中，筛选得到的目标特征图像的数量可以是10个、20个等，在本示例实施方式中不做具体限定。
在本示例实施方式中，还可以根据设筛选策略和目标特征图像的缺陷类别和坐 标得到缺陷检测结果。具体而言，在最终得到的多个目标特征图像中，根据缺陷对业务的重要程度，缺陷出现的频率，采用预设筛选策略判断选取最后的缺陷类别和坐标。其中，预设筛选策略包括若最终的多个目标特征图像中出现该类别，则判断为该类别；若候选框中同时出现某2个或多个特定类别，则选取为其中一个特定类别；按目标特征图像的排名选取最大的类别；若目标特征图像中全部为正常类别，则选取为正常类，“缺陷”坐标为原始图片尺寸。上述预设筛选策略还可以根据用户需求及进行自定义，即为定制化的结果后处理方式，以应对多样化的图片类型和场景，在本示例实施方式中不做具体限定。
在本公开的一种示例实施方式中，可以对上述每个任务中的DM图、TDI图和AOI彩图都给出一个缺陷类别和这个类别的坐标，然后可以将得到的结果打包推送至业务员操作系统，进行人工复核。
在本示例实施方式中，参照图4所示，以多种类型的图像包括AOI彩图、TDI图、DM图为例对上述缺陷检测方法进行介绍，首先可以执行步骤S410，输入AOI彩图、TDI图、DM图，然后执行步骤S420，预处理操作，包括AOI彩图预处理、TDI图预处理、DM图预处理，预处理操作的具体过程上述已经进行了详细说明，因此此处不再赘述；之后则可以执行步骤S430，将图像输入至缺陷检测模型，然后执行步骤S440和步骤S450，后处理并得到AOI彩图缺陷检测结果、TDI图缺陷检测结果、DM图缺陷检测结果，后处理包括AOI彩图后处理、TDI图后处理、DM图后处理，即利用模型的输出经过后处理得到上述缺陷检测结果。
参照图6所示，业务员操作的界面上最大展示区域为AOI彩图610，右上角显示区域620可以通过切换标识640切换AOI彩图为AI识别过的图像和AI未识别的图像，以及TDI图。其中，AI识别过的图像和TDI图像的缩略图下方可以给出缺陷类别，操作员只需大致浏览缩略图即可，大大增加作业员的判图速度。AI未识别的图像需要作业员在中间展示区进行图片判级，给出该图片的对应缺陷类别。DM图可以在在右下角显示区域630显示，缺陷在图片上方给出，并提供作业员改判功能。
综上，本公开中的缺陷检测方法，以具体的业务场景为依托，为屏幕生产检测的DM图、TDI图、AOI图提供了一种高效方便的处理方法。将多种不同类型的图像联合处理，大大提高了图像的检测效率和后续人工的工作效率。同时，合理的复用同一种算法网络并提取关键参数，增强了该算法的可扩展性。最后，将正常图像、缺陷图像的分类和缺陷图像的缺陷细分与定位融合于同一个端到端的任务流程，简 化了任务难度，提高了任务处理速度。
需要注意的是，上述附图仅是根据本公开示例性实施例的方法所包括的处理的示意性说明，而不是限制目的。易于理解，上述附图所示的处理并不表明或限制这些处理的时间顺序。另外，也易于理解，这些处理可以是例如在多个模块中同步或异步执行的。
进一步的，参考图7所示，本示例的实施方式中还提供一种缺陷检测装置700，包括图像获取模块710、模型获取模块720和缺陷检测模块730。其中：
图像获取模块710可以用于获取检测任务，并获取检测任务对应的多种类型的图像。
在一种示例实施方式中，图像获取模块710可以提取检测任务中的产品信息字段以获取产品信息，获取检测任务对应的产品信息；根据产品信息获取多种类型的图像，具体而言获取产品信息对应的产品的多种类型的图像；其中，各种类型的图像采用不同配置参数的相机对产品及进行拍照获得的，其中，各种类型的图像包括产品的AOI彩图、TDI图和DM图的一个或多个。
图像获取模块710同时还可以对各类型的图像进行预处理操作，以使得各类型的图像的通道数相同，具体而言，可以确定各类型的图像中的最大通道数；对各所述类型的图像进行预处理操作，将各所述类型的图像的通道数设置为所述最大通道数，将各类型的图像的分辨率调整为各类型分别对应的预设分辨率。还可以为预处理操作配合一预处理参数修改界面，以使得用户能够通过预处理参数修改界面调整预处理操作的参数信息。
在本公开的另一种示例实施方式中，图像获取模块710可以获取所述检测任务对应的多个产品信息；获取各所述产品信息对应的分辨率和通道数均相同的图像。
模型获取模块720可以获取与所述图像的类型分别对应的由同一初始模型训练所得的缺陷检测模型。
缺陷检测模块730，用于利用各所述图像的类型对应的所述缺陷检测模型对各种类型的图像进行缺陷检测得到缺陷检测结果；
其中，各图像的类型对应的缺陷检测模型由同一初始模型训练所得。
在本公开的一种示例实施方式中，缺陷检测模型包括特征提取网络和缺陷识别网路，缺陷检测模块730可以利用特征提取网络对各种类型的图像进行特征提取得到特征图像；并对特征图像进行标准化处理；利用缺陷识别网络根据标准后的特征 图像进行缺陷分类得到个特征图像的缺陷类别，并确定各特征图像的坐标；根据缺陷类别和坐标得到缺陷检测结果。
在本示例实施方式中，缺陷检测模块730还可以配置一标准化处理参数调整界面，以使得用户能够通过准化处理参数调整界面调整标准化处理的参数信息。
在本示例实施方式中，在根据缺陷类别和坐标得到缺陷检测结果时，缺陷检测模块730可以对特征图像进行筛选得到目标特征图像，并确定目标候特征图像的缺陷类别和坐标；根据预设筛选策略和目标特征图像的缺陷类别和坐标得到缺陷检测结果。其中，缺陷检测模块730可以计算各特征图像的置信度；根据置信度利用NMS算法对特征图像进行筛选得到目标特征图像。
在本示例实施方式中，缺陷检测模块730还可以根据配置参数对各种类型的图像设置缺陷影响权重；利用各图像的类型对应的缺陷检测模型对各种类型的图像进行缺陷检测得到各类型的图像分别对应的参考缺陷检测结果；根据缺陷影响权重和参考缺陷检测结果确定缺陷检测结果。
上述装置中各模块的具体细节在方法部分实施方式中已经详细说明，未披露的细节内容可以参见方法部分的实施方式内容，因而不再赘述。
所属技术领域的技术人员能够理解，本公开的各个方面可以实现为系统、方法或程序产品。因此，本公开的各个方面可以具体实现为以下形式，即：完全的硬件实施方式、完全的软件实施方式(包括固件、微代码等)，或硬件和软件方面结合的实施方式，这里可以统称为“电路”、“模块”或“系统”。
本公开的示例性实施方式还提供了一种计算机可读存储介质，其上存储有能够实现本说明书上述方法的程序产品。在一些可能的实施方式中，本公开的各个方面还可以实现为一种程序产品的形式，其包括程序代码，当程序产品在终端设备上运行时，程序代码用于使终端设备执行本说明书上述“示例性方法”部分中描述的根据本公开各种示例性实施方式的步骤。
需要说明的是，本公开所示的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。
在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF等等，或者上述的任意合适的组合。
此外，可以以一种或多种程序设计语言的任意组合来编写用于执行本公开操作的程序代码，程序设计语言包括面向对象的程序设计语言—诸如Java、C++等，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算设备上执行、部分地在用户设备上执行、作为一个独立的软件包执行、部分在用户计算设备上部分在远程计算设备上执行、或者完全在远程计算设备或服务器上执行。在涉及远程计算设备的情形中，远程计算设备可以通过任意种类的网络，包括局域网(LAN)或广域网(WAN)，连接到用户计算设备，或者，可以连接到外部计算设备(例如利用因特网服务提供商来通过因特网连接)。
本领域技术人员在考虑说明书及实践这里公开的发明后，将容易想到本公开的其他实施例。本申请旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性的，本公开的真正范围和精神由权利要求指出。
应当理解的是，本公开并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本公开的范围仅由所附的权利要求来限。

Claims (17)

  1. 一种缺陷检测方法，其特征在于，包括：
    获取检测任务，并获取所述检测任务对应的多种类型的图像；
    获取与所述图像的类型分别对应的由同一初始模型训练所得的缺陷检测模型；
    利用各所述图像的类型对应的所述缺陷检测模型对各种类型的图像进行缺陷检测得到缺陷检测结果。
  2. 根据权利要求1所述的方法，其特征在于，获取所述检测任务对应的多种类型的图像，包括：
    获取所述检测任务对应的产品信息；
    根据所述产品信息获取多种类型的图像。
  3. 根据权利要求2所述的方法，其特征在于，根据所述产品信息获取多种类型的图像，包括：
    获取所述产品信息对应的同一产品的多种类型的图像；
    其中，各种所述类型的图像采用不同配置参数的相机对所述产品及进行拍照获得的；
    其中，所述配置参数包括分辨率、色彩和放大倍数中的一个或多个。
  4. 根据权利要求3所述的方法，其特征在于，所述方法还包括：
    确定各所述类型的图像中的最大通道数；
    对各所述类型的图像进行预处理操作，将各所述类型的图像的通道数设置为所述最大通道数。
  5. 根据权利要求3所述的方法，其特征在于，各种所述类型的图像包括所述产品的AOI彩图、TDI图和DM图中的一个或多个。
  6. 根据权利要求3所述的方法，其特征在于，所述利用各所述图像的类型对应的所述缺陷检测模型对各种类型的图像进行缺陷检测得到缺陷检测结果，包括：
    根据所述配置参数对所述各种类型的图像设置缺陷影响权重；
    利用各所述图像的类型对应的所述缺陷检测模型对各种类型的图像进行缺陷检测得到各所述类型的图像分别对应的参考缺陷检测结果；
    根据所述缺陷影响权重和所述参考缺陷检测结果确定所述缺陷检测结果。
  7. 根据权利要求2所述的方法，其特征在于，获取所述检测任务对应的产品信息，包括：
    提取所述检测任务中的产品信息字段以获取所述产品信息。
  8. 根据权利要求7所述的方法，其特征在于，对各所述类型的图像进行预处理操作，还包括：
    根据所述缺陷检测模型确定一分辨率阈值；
    根据所述分辨率阈值调整各所述类型图像的分辨率。
  9. 根据权利要求8所述的方法，其特征在于，所述方法还包括：
    为所述预处理操作配合一预处理参数修改界面，以使得用户能够通过所述预处理参数修改界面调整所述预处理操作的参数信息。
  10. 根据权利要求1所述的方法，其特征在于，所述缺陷检测结果包括图像正常信息或图像缺陷信息。
  11. 根据权利要求1所述的方法，其特征在于，所述缺陷检测模型包括特征提取网络和缺陷识别网路，所述利用各所述图像的类型对应的所述缺陷检测模型对各种类型的图像进行缺陷检测得到缺陷检测结果，包括：
    利用所述特征提取网络对各种类型的图像进行特征提取得到特征图像；并对所述特征图像进行标准化处理；
    利用所述缺陷识别网络根据所述标准后的特征图像进行缺陷分类得到个特征图像的缺陷类别，并确定各特征图像的坐标；
    根据所述缺陷类别和所述坐标得到所述缺陷检测结果。
  12. 根据权利要求11所述的方法，其特征在于，所述方法还包括：
    配置一标准化处理参数调整界面，以使得用户能够通过所述准化处理参数调整界面调整所述标准化处理的参数信息。
  13. 根据权利要求11所述的方法，其特征在于，根据所述缺陷类别和所述坐标得到所述缺陷检测结果，包括：
    对所述特征图像进行筛选得到目标特征图像，并确定所述目标候特征图像的所述缺陷类别和所述坐标；
    根据预设筛选策略和所述目标特征图像的所述缺陷类别和所述坐标得到所述缺陷检测结果。
  14. 根据权利要求13所述的方法，其特征在于，所述对所述特征图像进行筛选得到目标特征图像，包括：
    计算各所述特征图像的置信度；
    根据所述置信度利用NMS算法对所述特征图像进行筛选得到目标特征图像。
  15. 一种缺陷检测装置，其特征在于，包括：
    图像获取模块，用于获取检测任务，并获取所述检测任务对应的多种类型的图像；
    模型获取模块，用于获取与所述图像的类型分别对应的由同一初始模型训练所得的缺陷检测模型；
    缺陷检测模块，用于利用各所述图像的类型对应的所述缺陷检测模型对各种类型的图像进行缺陷检测得到缺陷检测结果。
  16. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述程序被处理器执行时实现如权利要求1至14中任一项所述的缺陷检测方法。
  17. 一种电子设备，其特征在于，包括：
    处理器；以及
    存储器，用于存储一个或多个程序，当所述一个或多个程序被所述一个或多个处理器执行时，使得所述一个或多个处理器实现如权利要求1至14中任一项所述的缺陷检测方法。
PCT/CN2021/095306 2021-05-21 2021-05-21 缺陷检测方法及装置、存储介质及电子设备 WO2022241784A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US17/772,562 US20240202893A1 (en) 2021-05-21 2021-05-21 Method and device for detecting defect, storage medium and electronic device
CN202180001233.1A CN115699082A (zh) 2021-05-21 2021-05-21 缺陷检测方法及装置、存储介质及电子设备
PCT/CN2021/095306 WO2022241784A1 (zh) 2021-05-21 2021-05-21 缺陷检测方法及装置、存储介质及电子设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/095306 WO2022241784A1 (zh) 2021-05-21 2021-05-21 缺陷检测方法及装置、存储介质及电子设备

Publications (1)

Publication Number Publication Date
WO2022241784A1 true WO2022241784A1 (zh) 2022-11-24

Family

ID=84140160

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/095306 WO2022241784A1 (zh) 2021-05-21 2021-05-21 缺陷检测方法及装置、存储介质及电子设备

Country Status (3)

Country Link
US (1) US20240202893A1 (zh)
CN (1) CN115699082A (zh)
WO (1) WO2022241784A1 (zh)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115931905A (zh) * 2023-02-02 2023-04-07 北京百度网讯科技有限公司 Poy产品的检测方法、装置、电子设备和存储介质
CN116433661A (zh) * 2023-06-12 2023-07-14 锋睿领创(珠海)科技有限公司 多任务学习的半导体晶圆检测方法、装置、设备及介质
CN116843625A (zh) * 2023-06-05 2023-10-03 广东粤桨产业科技有限公司 工业质检场景的缺陷检测模型部署方法、系统及设备
CN117079182A (zh) * 2023-07-31 2023-11-17 上海启呈信息科技有限公司 一种基于大数据分析的管网管理方法及系统
CN117274257A (zh) * 2023-11-21 2023-12-22 山西集智数据服务有限公司 基于机器视觉的图书品相自动分级系统
CN117934462A (zh) * 2024-03-21 2024-04-26 高视科技(苏州)股份有限公司 用于调整检测参数的方法、电子设备及存储介质
CN118037741A (zh) * 2024-04-15 2024-05-14 中国水利水电第十工程局有限公司 基于机器视觉的人字门焊接质量检测方法及系统
CN118038151A (zh) * 2024-02-19 2024-05-14 广州里工实业有限公司 晶圆缺陷分类检测方法、系统、电子设备及存储介质

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116503412B (zh) * 2023-06-29 2023-12-08 宁德时代新能源科技股份有限公司 外观缺陷检测方法、装置、计算机设备和存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109711474A (zh) * 2018-12-24 2019-05-03 中山大学 一种基于深度学习的铝材表面缺陷检测算法
US10607331B1 (en) * 2019-06-28 2020-03-31 Corning Incorporated Image segmentation into overlapping tiles
CN111353413A (zh) * 2020-02-25 2020-06-30 武汉大学 一种输电设备低漏报率缺陷识别方法
CN111489339A (zh) * 2020-04-08 2020-08-04 北京交通大学 高速铁路定位器螺栓备母缺陷的检测方法
CN111951249A (zh) * 2020-08-13 2020-11-17 浙江理工大学 基于多任务学习网络的手机导光板缺陷可视化检测方法
CN112053318A (zh) * 2020-07-20 2020-12-08 清华大学 基于深度学习的二维pcb缺陷实时自动检测与分类装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109711474A (zh) * 2018-12-24 2019-05-03 中山大学 一种基于深度学习的铝材表面缺陷检测算法
US10607331B1 (en) * 2019-06-28 2020-03-31 Corning Incorporated Image segmentation into overlapping tiles
CN111353413A (zh) * 2020-02-25 2020-06-30 武汉大学 一种输电设备低漏报率缺陷识别方法
CN111489339A (zh) * 2020-04-08 2020-08-04 北京交通大学 高速铁路定位器螺栓备母缺陷的检测方法
CN112053318A (zh) * 2020-07-20 2020-12-08 清华大学 基于深度学习的二维pcb缺陷实时自动检测与分类装置
CN111951249A (zh) * 2020-08-13 2020-11-17 浙江理工大学 基于多任务学习网络的手机导光板缺陷可视化检测方法

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115931905B (zh) * 2023-02-02 2023-09-26 北京百度网讯科技有限公司 Poy产品的检测方法、装置、电子设备和存储介质
CN115931905A (zh) * 2023-02-02 2023-04-07 北京百度网讯科技有限公司 Poy产品的检测方法、装置、电子设备和存储介质
CN116843625A (zh) * 2023-06-05 2023-10-03 广东粤桨产业科技有限公司 工业质检场景的缺陷检测模型部署方法、系统及设备
CN116433661A (zh) * 2023-06-12 2023-07-14 锋睿领创(珠海)科技有限公司 多任务学习的半导体晶圆检测方法、装置、设备及介质
CN116433661B (zh) * 2023-06-12 2023-08-18 锋睿领创(珠海)科技有限公司 多任务学习的半导体晶圆检测方法、装置、设备及介质
CN117079182B (zh) * 2023-07-31 2024-02-23 上海启呈信息科技有限公司 一种基于大数据分析的管网管理方法及系统
CN117079182A (zh) * 2023-07-31 2023-11-17 上海启呈信息科技有限公司 一种基于大数据分析的管网管理方法及系统
CN117274257A (zh) * 2023-11-21 2023-12-22 山西集智数据服务有限公司 基于机器视觉的图书品相自动分级系统
CN117274257B (zh) * 2023-11-21 2024-01-23 山西集智数据服务有限公司 基于机器视觉的图书品相自动分级系统
CN118038151A (zh) * 2024-02-19 2024-05-14 广州里工实业有限公司 晶圆缺陷分类检测方法、系统、电子设备及存储介质
CN117934462A (zh) * 2024-03-21 2024-04-26 高视科技(苏州)股份有限公司 用于调整检测参数的方法、电子设备及存储介质
CN118037741A (zh) * 2024-04-15 2024-05-14 中国水利水电第十工程局有限公司 基于机器视觉的人字门焊接质量检测方法及系统
CN118037741B (zh) * 2024-04-15 2024-06-11 中国水利水电第十工程局有限公司 基于机器视觉的人字门焊接质量检测方法及系统

Also Published As

Publication number Publication date
US20240202893A1 (en) 2024-06-20
CN115699082A (zh) 2023-02-03

Similar Documents

Publication Publication Date Title
WO2022241784A1 (zh) 缺陷检测方法及装置、存储介质及电子设备
WO2021036059A1 (zh) 图像转换模型训练方法、异质人脸识别方法、装置及设备
WO2019128507A1 (zh) 图像处理方法、装置、存储介质及电子设备
WO2021217775A1 (zh) 一种基于神经网络模型融合的废钢评级方法及装置
CN110866471A (zh) 人脸图像质量评价方法及装置、计算机可读介质、通信终端
AU2021201933B2 (en) Hierarchical multiclass exposure defects classification in images
CN111914812B (zh) 图像处理模型训练方法、装置、设备及存储介质
CN112668696A (zh) 一种基于嵌入式深度学习的无人机电网巡检方法和系统
WO2022160222A1 (zh) 缺陷检测方法及装置、模型训练方法及装置及电子设备
KR102322773B1 (ko) 전극편의 버를 검출하기 위한 방법 및 장치
CN111930964B (zh) 内容处理方法、装置、设备及存储介质
CN111950570B (zh) 目标图像提取方法、神经网络训练方法及装置
US20230048386A1 (en) Method for detecting defect and method for training model
CN112651358A (zh) 一种目标检测方法、装置、网络摄像机及存储介质
CN114882437A (zh) 一种识别模型的训练方法、装置、电子设备和存储介质
US20240221126A1 (en) Image splicing method and apparatus, and device and medium
CN114581994A (zh) 一种课堂考勤管理方法和系统
WO2022033306A1 (zh) 目标跟踪方法和装置
Nagaraj et al. Edge-based street object detection
WO2018211602A1 (ja) 学習装置、推定装置、学習方法及びプログラム
CN116977195A (zh) 复原模型的调整方法、装置、设备及存储介质
CN116188846A (zh) 一种基于振动图像的设备故障检测方法及装置
CN114972893A (zh) 一种数据标注方法、装置、电子设备和存储介质
CN117992898B (zh) 异常检测模型的训练方法、物体异常检测方法和装置
CN118212231B (zh) 图像处理方法及相关设备

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 17772562

Country of ref document: US

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21940237

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 25.03.2024)