WO2021120179A1 - 产品制造消息处理方法、设备和计算机存储介质 - Google Patents

产品制造消息处理方法、设备和计算机存储介质 Download PDF

Info

Publication number
WO2021120179A1
WO2021120179A1 PCT/CN2019/127040 CN2019127040W WO2021120179A1 WO 2021120179 A1 WO2021120179 A1 WO 2021120179A1 CN 2019127040 W CN2019127040 W CN 2019127040W WO 2021120179 A1 WO2021120179 A1 WO 2021120179A1
Authority
WO
WIPO (PCT)
Prior art keywords
product
defect
product manufacturing
manufacturing
message
Prior art date
Application number
PCT/CN2019/127040
Other languages
English (en)
French (fr)
Inventor
张美娟
王耀平
李昭月
路元元
柴栋
王洪
Original Assignee
京东方科技集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司 filed Critical 京东方科技集团股份有限公司
Priority to CN201980003143.9A priority Critical patent/CN113557425B/zh
Priority to US17/044,160 priority patent/US12020516B2/en
Priority to PCT/CN2019/127040 priority patent/WO2021120179A1/zh
Publication of WO2021120179A1 publication Critical patent/WO2021120179A1/zh
Priority to US18/666,299 priority patent/US20240304034A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C3/00—Registering or indicating the condition or the working of machines or other apparatus, other than vehicles
    • G07C3/14—Quality control systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • G06Q10/06395—Quality analysis or management
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B29—WORKING OF PLASTICS; WORKING OF SUBSTANCES IN A PLASTIC STATE IN GENERAL
    • B29C—SHAPING OR JOINING OF PLASTICS; SHAPING OF MATERIAL IN A PLASTIC STATE, NOT OTHERWISE PROVIDED FOR; AFTER-TREATMENT OF THE SHAPED PRODUCTS, e.g. REPAIRING
    • B29C45/00—Injection moulding, i.e. forcing the required volume of moulding material through a nozzle into a closed mould; Apparatus therefor
    • B29C45/17—Component parts, details or accessories; Auxiliary operations
    • B29C45/76—Measuring, controlling or regulating
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3688—Test management for test execution, e.g. scheduling of test suites
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/245—Classification techniques relating to the decision surface
    • G06F18/2451—Classification techniques relating to the decision surface linear, e.g. hyperplane
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G06T7/001—Industrial image inspection using an image reference approach
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
    • H01L21/00—Processes or apparatus adapted for the manufacture or treatment of semiconductor or solid state devices or of parts thereof
    • H01L21/67—Apparatus specially adapted for handling semiconductor or electric solid state devices during manufacture or treatment thereof; Apparatus specially adapted for handling wafers during manufacture or treatment of semiconductor or electric solid state devices or components ; Apparatus not specifically provided for elsewhere
    • H01L21/67005—Apparatus not specifically provided for elsewhere
    • H01L21/67242—Apparatus for monitoring, sorting or marking
    • H01L21/67288—Monitoring of warpage, curvature, damage, defects or the like
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8887—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges based on image processing techniques
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Definitions

  • the present disclosure relates to a method, equipment and electronic equipment for processing product manufacturing messages.
  • the present disclosure also relates to the field of artificial intelligence and big data, in particular to product manufacturing assistance systems, methods, and computer-readable storage media.
  • the output products will not meet the process requirements or even lead to defects, so it needs to be in each process.
  • the traditional identification method mainly relies on manual detection. This requires professional training for inspectors.
  • semiconductor products have a wide variety of defects, which may include particles, residues, defective wires, holes, splashes, electrostatic breakdown, wrinkles, film discoloration, bubbles, etc., which need to be tested
  • defects may include particles, residues, defective wires, holes, splashes, electrostatic breakdown, wrinkles, film discoloration, bubbles, etc., which need to be tested
  • the personnel devote a long time and focus to defect finding and related judgments.
  • the existing technical means to solve the above problems have the problems of low efficiency and low accuracy.
  • product manufacturing messages In the intelligent product manufacturing process, a large number of product manufacturing messages will be generated. These product manufacturing messages can be used to prompt the production process of the product, or prompt the product's possible defects in the manufacturing process. For example, in the manufacturing process of semiconductor products, due to problems in equipment, parameters, operation, environmental interference and other links, the products produced do not meet the process requirements or even lead to defects. Therefore, it is necessary to promptly check the defects after each process.
  • the defect type, defect size, location and other information of the defective products that meet the requirements are calculated and identified for timely correction and improvement.
  • a method for processing product manufacturing messages including: monitoring multiple product manufacturing messages; establishing a product defect analysis task queue according to the multiple product manufacturing messages; Auxiliary equipment distributes product defect analysis tasks, where the product defect analysis tasks include product defect content identification tasks based on a defect recognition model, and the product defect content includes any of the defect type, defect location, and defect size of the product Or multiple.
  • an electronic device including: a processor; a memory, where computer instructions are stored in the memory, and the computer instructions implement the above-mentioned method when executed by the processor.
  • a computer-readable storage medium having computer instructions stored thereon, and the computer instructions implement the above-mentioned method when executed by a processor.
  • FIG. 1 is a schematic diagram showing an example of a scenario for message processing of product manufacturing.
  • Fig. 2 is a flowchart illustrating a method for processing a product manufacturing message according to at least one embodiment of the present disclosure.
  • Fig. 3 is another flowchart showing a method for processing a product manufacturing message according to at least one embodiment of the present disclosure.
  • Fig. 4 is a schematic diagram illustrating a method for processing a product manufacturing message according to at least one embodiment of the present disclosure.
  • Fig. 5 is another flowchart illustrating a method for processing a product manufacturing message according to at least one embodiment of the present disclosure.
  • Fig. 6 is another schematic diagram illustrating a method for processing a product manufacturing message according to at least one embodiment of the present disclosure.
  • Fig. 7 is another schematic diagram illustrating a method for processing a product manufacturing message according to at least one embodiment of the present disclosure.
  • Fig. 8 shows a schematic diagram of a product manufacturing message processing device according to at least one embodiment of the present disclosure.
  • Fig. 9 shows a structural diagram of an electronic device according to at least one embodiment of the present disclosure.
  • the products mentioned below include the raw materials in the actual production process, as well as the semi-finished or finished products after each processing process (product manufacturing equipment processing).
  • products include glass that enters the production line from the very beginning , Array substrate that has gone through the exposure process, the screen that has gone through the box-forming process, etc.
  • Product images include product images directly acquired by image acquisition equipment (such as cameras, AOI equipment, etc.), as well as product images that contain defective content annotations (that is, product images that have undergone defective content recognition).
  • FIG. 1 is a schematic diagram showing an example of a scenario 100 for processing a product manufacturing message.
  • scenario 100 multiple products pass through site 101 in sequence.
  • the station 101 represents a point through which the product may pass in the entire production line flow.
  • the site 101 may be a certain physical device that completes a process in standardized production on a product assembly line, or a system composed of multiple physical devices.
  • the site corresponding to the process can be a system consisting of cleaning equipment, pre-baking equipment, cooling equipment, gluing equipment, exposure equipment, developing equipment, post-baking equipment, cooling equipment, etc. .
  • the station 101 may also be a single device (exposure device) corresponding to the exposure process or an AOI device corresponding to image detection.
  • the site 101 may also be a virtual node in the product manufacturing process, which represents a step of processing the product in a non-physical form.
  • the site 101 may perform a defect detection process on a product, which acquires and analyzes all process information used for product defect detection to determine product defects.
  • the site 101 will capture the entry message (trackin message).
  • the site will capture the leaving site message (trackout message).
  • the product information/product data in the trackin message and the trackout message must meet the requirements of product manufacturing.
  • the site 101 may include a product manufacturing message service device 102, a product manufacturing message processing device 103, and a product manufacturing auxiliary device 104.
  • the product manufacturing message service device 102 may not be included in the site 101.
  • the product manufacturing message service device 102, the product manufacturing message processing device 103, and the product manufacturing auxiliary device 104 may be computing devices including a processor and a memory. These devices can be connected via the network.
  • the above-mentioned devices can directly or indirectly communicate with each other, for example, send and receive data and/or signals to and from each other through a network.
  • the network can be the Internet of Things (Internet of Things) based on the Internet and/or telecommunication network. It can be a wired network or a wireless network.
  • LAN local area network
  • MAN metropolitan area network
  • WAN wide area network
  • Each device can use one or more communication protocols to communicate with each other, for example, FTP, TCP/IP, HSMS, Tibco, and so on.
  • the site 101 is mainly used for product defect detection, analysis, and processing. Those skilled in the art should understand that the site 101 may also be applied to other processes of product manufacturing.
  • the product manufacturing message service device 102 can capture all or part of the product manufacturing messages in the product manufacturing process (for example, the aforementioned trackin message), and broadcast or send these product manufacturing messages to the product manufacturing message processing device 103.
  • the product manufacturing message processing device 103 may further process the product manufacturing message, and send a task message for detecting and analyzing product defects to the product manufacturing auxiliary device 104.
  • the product manufacturing auxiliary equipment 104 may include one or more of the following: equipment used by inspectors to detect defects in products, equipment used to detect defects in products using artificial intelligence defect recognition models, equipment used to deploy artificial intelligence defect recognition models, Equipment for training artificial intelligence defect recognition models, product defect alarm equipment, etc.
  • the product manufacturing auxiliary device 104 may return the analysis result to the product manufacturing message processing device 103 after completing the product defect analysis and detection.
  • FIG. 2 is a flowchart illustrating a method 200 for processing a product manufacturing message according to at least one embodiment of the present disclosure.
  • the product manufacturing message processing method 200 may include some or all of the operations as shown in FIG. 2 (for example, some or all of operations 210 to 230). Of course, the product manufacturing message processing method 200 may also include other operations not shown in FIG. 2.
  • the product manufacturing message processing method 200 can also be executed by any other electronic device with communication function and calculation function. Hereinafter, description will be made by taking the product manufacturing message processing device 103 as an example.
  • the product manufacturing message processing device 103 may listen to multiple product manufacturing messages.
  • product manufacturing messages In the intelligent product manufacturing process, a large number of product manufacturing messages will be generated. These product manufacturing messages can be used to prompt the production process of the product, or prompt the product's possible defects in the manufacturing process.
  • the product manufacturing message may include record information generated by any product manufacturing equipment that the product experiences.
  • the product manufacturing message can be used to know that the product has been processed by the product manufacturing equipment and other related processing results.
  • the product manufacturing message includes a message for instructing product image generation.
  • the product manufacturing equipment may be the aforementioned AOI (Automated Optical Inspection) equipment.
  • the product manufacturing equipment can perform optical inspection and photographing during the screen production process, and determine the difference between the product image taken by it and the standard product image. Based on these differences, product manufacturing equipment can determine that there are product defects in the product being tested.
  • the product manufacturing equipment can also be other cameras or cameras with image acquisition functions. Then, the product manufacturing equipment can send the captured product image and corresponding files to the product image database.
  • the product image database may be the DFS (Distributed File System, distributed file system) described above or other data storage devices.
  • the corresponding message indicating that the product image is generated can be a message (such as trackin message) that enters the product manufacturing equipment (such as AOI equipment) and/or a message (such as trackout message) that leaves the product manufacturing equipment (such as AOI equipment), or it can be Product manufacturing equipment (such as AOI equipment) generates a message for product image files or sends a message for product images to a product image database.
  • the product manufacturing equipment may also be other equipment that can be used for product manufacturing, which is not limited in the present disclosure.
  • the product manufacturing message service device 102 can capture all or part of the product manufacturing messages in the product manufacturing process, and broadcast or send these product manufacturing messages to the product manufacturing message processing device 103.
  • the product manufacturing device may also directly send the product manufacturing message to the product manufacturing message processing device 103.
  • the product manufacturing message acquired by the product manufacturing message processing device 103 is a plurality of product manufacturing messages monitored.
  • the product manufacturing information service equipment 102 includes a manufacturing execution system (Manufacturing Execution System, MES), and may also include a supervisory information system (Executive Information System, EIS).
  • the product manufacturing information service equipment 10 may also be other equipment used to monitor product manufacturing. This disclosure does not limit this. Therefore, the product manufacturing message may be generated by the product manufacturing device, or captured by the product message service device 102.
  • the product manufacturing message also includes product manufacturing site information and product information.
  • the product manufacturing site information includes the site’s identity, the physical location of the site (for example, the physical location of the AOI equipment), and the site’s node information in the product manufacturing process (for example, defect identification/detection in the exposure process, and in the cleaning process).
  • This information can assist the product manufacturing message service device 102 to identify or locate a specific site.
  • the product information may be a product type, a product name, a product identification, a product priority, etc. This information can assist the product manufacturing message service device 102 to identify or locate a specific product.
  • Those skilled in the art should understand that the content of product manufacturing site information and product information is not limited to the above examples, as long as they are associated with defect identification/detection in the product manufacturing process.
  • the foregoing multiple product manufacturing messages include at least one single product manufacturing message and at least one batch product manufacturing message.
  • the AOI can send a single product manufacturing message (for example, GlassTrackOut message) as a product after a screen (or a large glass substrate screen, also known as Glass) is inspected.
  • the manufacturing message is sent to the product manufacturing message service device 102, and the picture file (.jpg/.gls) is sent to the image database (DFS system), or it can also send a single product manufacturing message to all activated devices in the current factory.
  • AOI equipment may also send mass product manufacturing messages in units of LOT (1 Lot contains 20 Glass, and each Glass is a single large glass substrate screen).
  • the AOI will send a mass product manufacturing message (for example, LotTrackOut message) as a product manufacturing message to the product manufacturing message service device 102 or any other related devices.
  • monitoring multiple product manufacturing messages further includes: Monitor batch product manufacturing messages in interrupt mode; and monitor single product manufacturing messages in polling mode.
  • Interrupted monitoring of mass product manufacturing messages means that the monitoring is interrupted after the first mass product manufacturing message is monitored, and the monitoring is continued until the next mass product manufacturing message is generated.
  • Monitoring a single product manufacturing message in a polling manner refers to continuously monitoring a single product manufacturing message device at a preset frequency.
  • manufacturing messages are usually recorded and transmitted in units of batches (Lot), which can improve the efficiency of message processing.
  • Lot batches
  • the product manufacturing message processing equipment 103 and the equipment for product defect detection and analysis will often be in an idle state.
  • the product manufacturing message processing device 103 can monitor batch product manufacturing messages in an interrupted manner, and monitor single product manufacturing messages in a polling manner during the interruption interval, and process single product manufacturing messages in a timely manner.
  • the single product may be one product in the batch product or not in the batch product. In this way, the processing of most of the single product manufacturing messages of the batch has been completed when the batch product manufacturing message is received. After the processing of all products in the batch (for example, one LOT) is completed, the next batch of product manufacturing messages is processed, thereby improving the message processing efficiency of the product manufacturing message processing device 103.
  • the product manufacturing message processing device 103 may also register relevant information about the mass product manufacturing message that it wants to monitor to the product manufacturing message service device 102.
  • the present disclosure will describe the registration process in detail in the following embodiments. Therefore, the product manufacturing message service device 102 broadcasts to the product manufacturing message processing device 103 after the mass product manufacturing is completed. The product manufacturing message processing device 103 then receives the mass product manufacturing message, and interrupts after receiving it. The product manufacturing message processing device 103 can start polling and monitoring the single product manufacturing message. When there is a mass product manufacturing message again, the product manufacturing message service device 102 broadcasts to the product manufacturing message processing device 103.
  • the product manufacturing message processing device 103 may establish a product defect analysis task queue according to a plurality of product manufacturing messages.
  • the station 101 can be used as a testing station in the entire product manufacturing process. Products of different assembly lines produced by the factory may flow into the station 101 during the inspection process. At present, in factories, due to the abundance of products and complex processes, there are many product manufacturing sites and complex product defects. In this case, the frequency and quantity of products flowing into the station 101 are uncertain. The incoming products that need to be tested sometimes suddenly become a lot, and sometimes they are few. Therefore, it is necessary to perform reasonable scheduling and allocation of inspection tasks for various products.
  • the product manufacturing message processing equipment 103 will establish a product defect analysis task queue based on the product manufacturing messages it receives, so as to follow the product defect analysis task queue. Distribute tasks sequentially.
  • the product defect analysis task is distributed to the product manufacturing auxiliary device according to the product defect analysis task queue.
  • the product defect analysis task includes the product defect content identification task based on the defect recognition model.
  • the reasons for product defects in the manufacturing process of products are different, such as insufficient cleaning in the semiconductor production process, insufficient corrosion, excessive corrosion, inaccurate raw material matching, excessive dust in the cleaning environment, and exposure intensity Insufficient, high exposure intensity, adulteration, etc.
  • the product defect classification and defect location for example, the circuit board where the product defect is located and the level, mask layer, etc., and the specific coordinate position on the circuit board (for example, the outer rectangular shape)
  • the vertex coordinates can also be expressed as the coordinates of a vertex plus the length and width), the relationship between the defect and the shape of the background circuit model (for example, between the two lines on the gate island, the number of gate islands covered by the defect area, and whether the defect is complete It falls within the Gate island, intersects with it, or near the outside, etc.), defect size (for example, the length of the defect or the area of the defect (for example, the pixel area)).
  • the product manufacturing auxiliary equipment 104 includes: a first product manufacturing auxiliary device 104-1 that performs configuration management on artificial intelligence defect recognition models, a second product manufacturing auxiliary device 104-2 that performs artificial intelligence detection and analysis on product defects, and a product The third product manufacturing auxiliary equipment 104-3 for manual defect detection and so on.
  • the first product manufacturing auxiliary device 104-1 may be one or more devices (for example, a model management cluster) that sets the parameters of the artificial intelligence defect recognition model and manages the artificial intelligence training process of other product manufacturing auxiliary devices.
  • the second product manufacturing auxiliary device 104-2 may be one or more devices (for example, a product defect analysis cluster) that can use GPU computing resources to perform inference and training tasks of the artificial intelligence defect recognition model, and to schedule and allocate hardware resources.
  • the third product manufacturing auxiliary device 104-3 may be a terminal (for example, a product manufacturing client device) that displays product defects to related workers and allows them to judge the product defects. Take the product manufacturing client device in the factory as an example. It can display product defect images to relevant staff, let relevant staff judge product defects, set relevant information, analyze relevant data, or let relevant staff conduct defect judgment exams .
  • the task of product defect analysis also includes the task of identifying product defect content based on the artificial intelligence defect recognition model.
  • the product defect content includes any one or more of the defect type, defect location, and defect size of the product.
  • Product defect analysis tasks may also include artificial intelligence defect model training tasks.
  • the artificial intelligence defect recognition model includes one or more of the feedforward neural network artificial intelligence defect recognition model, the convolutional neural network model, the recurrent neural network model, and the generative confrontation network model.
  • the realization of the product defect content recognition task based on the artificial intelligence defect recognition model is as follows. First, the product image is scaled to a fixed pixel size MxN (or may not be scaled). Then send the MxN image to the deep convolutional neural network (VGG/Resnet/MobileNet, etc.). After the MxN image passes through the multi-layer convolutional layer/activation layer/pooling layer of the deep convolutional neural network, feature maps of the entire image can be obtained. The feature map will be input to the screening area network (ZF/SSD/RPN, etc.), and after calculation, a proposal region (proposal region) will be obtained.
  • VCG/Resnet/MobileNet deep convolutional neural network
  • the method for identifying the content of product defects based on the artificial intelligence defect model can adopt similar variants of the above methods or other methods known to those skilled in the art, and the present disclosure is not limited herein.
  • the second product manufacturing auxiliary device 104-2 can be used to process a product defect analysis task that uses an artificial intelligence defect recognition model to identify product defect content from a product image.
  • the second product manufacturing auxiliary device 104-2 may be one or more devices that can use GPU (Graphics Processing Unit, image processing unit) computing resources to perform inference and training tasks of the artificial intelligence defect recognition model.
  • GPU Graphics Processing Unit, image processing unit
  • the artificial intelligence defect recognition model is mainly based on neural network.
  • the artificial intelligence defect recognition model may be based on a feedforward neural network, that is, a feedforward neural network model.
  • the feedforward network can be implemented as an acyclic graph, where nodes are arranged in layers.
  • a feedforward network topology includes an input layer and an output layer, and the input layer and the output layer are separated by at least one hidden layer.
  • the hidden layer transforms the input received by the input layer into a useful representation for generating output in the output layer.
  • Network nodes are fully connected to nodes in adjacent layers via edges, but there are no edges between nodes in each layer.
  • the data received at the nodes of the input layer of the feedforward network is propagated (ie, "feedforward") to the nodes of the output layer via an activation function, which is calculated based on coefficients ("weights") for each successive layer in the network The status of the node.
  • the coefficients are respectively associated with each of the edges connecting these layers.
  • the output of the artificial intelligence defect recognition model can take various forms, which are not limited in the present disclosure.
  • Artificial intelligence defect recognition models can also include other neural network models, such as convolutional neural network (CNN) models, recurrent neural network (RNN) models, and generative adversarial network (GAN) models, but they are not limited to this, and can also be used. Other neural network models known to those skilled in the art.
  • the second product manufacturing auxiliary device 104-2 may also involve training of an artificial intelligence defect recognition model. For example, it may involve: selecting the network topology; using a set of training data representing the problem modeled by the network; and adjusting the weights until the network artificial intelligence defect recognition model appears to have the smallest error for all instances of the training data set. For example, during a supervised learning training process for a neural network, the output produced by the network in response to an input representing an example in the training data set is compared with the "correct" labeled output of the example; the calculation represents the output The error signal of the difference from the marked output; and when the error signal is propagated backwards through the layers of the network, the weights associated with the connection are adjusted to minimize the error. When the error of each output generated from the instance of the training data set is minimized, the artificial intelligence defect recognition model is considered "trained.”
  • the accuracy of the artificial intelligence defect recognition model can be greatly affected by the quality of the data set used to train the algorithm.
  • the training process can be computationally intensive, so it is beneficial to use GPUs to train many types of artificial intelligence defect recognition models.
  • the calculations performed when adjusting the coefficients in the neural network are naturally suitable for parallel implementations. Specifically, many machine learning algorithms and software applications have been adapted to use parallel processing hardware within general graphics processing devices. It is efficient in processing the calculations associated with training deep neural networks. Therefore, the use of multiple GPU integrated GPU clusters can effectively improve the training and inference speed of artificial intelligence defect recognition models.
  • the second product manufacturing auxiliary device 104-2 can also schedule and allocate hardware resources.
  • the product manufacturing message processing device 103 can determine whether artificial intelligence defect recognition model detection is required according to the product type, and whether it is necessary to train related models, and report to the first product manufacturing auxiliary device 104-1 to the third product manufacturing auxiliary device based on the judgment result.
  • 104-3 distributes different product defect analysis tasks. For example, for a known product type (for example, a product that has been trained), a defect content recognition task based on an artificial intelligence defect recognition model can be generated. For unknown product types (such as new products that have not been trained), it is possible to generate defect content recognition tasks based on manual (such as operators) recognition.
  • the product manufacturing message processing device 103 can also generate a defect content recognition task based on manual recognition.
  • the product manufacturing message processing device 103 can also perform the tasks of the first product manufacturing auxiliary device 104-1, the second product manufacturing auxiliary device 104-2, and the third product manufacturing auxiliary device 104-3 according to the quantity in the product defect analysis task queue. Classification allows computer resources and human resources to operate efficiently.
  • the method 200 for processing product manufacturing messages can improve the processing efficiency of product manufacturing messages in the entire product manufacturing process, so that each device participating in product defect detection and analysis can run efficiently, which is convenient for subsequent defect cause search and analysis. , Improve the efficiency of product manufacturing.
  • FIG. 3 is another flowchart showing a product manufacturing message processing method 200 according to at least one embodiment of the present disclosure, which shows a process of obtaining a product manufacturing message in the product manufacturing message processing method 200, for example, the above-mentioned operation 210 Some or all of the operations in.
  • the product manufacturing message processing device 103 sends registration information to the product manufacturing message service device 102.
  • the registration information includes product manufacturing site information and/or first product information.
  • the product manufacturing message service device 102 can learn the product manufacturing information related to the registration information that the product manufacturing message processing device 103 wants to know through the registration information.
  • the product manufacturing message related to the registration information includes a product manufacturing message including product manufacturing site information or first product information. Therefore, when the product manufacturing message service device 102 collects the product manufacturing message related to the registration information, the product manufacturing message can be broadcast first. In this way, the product manufacturing message processing device 103 can monitor the product manufacturing message related to the registration information in an interrupted manner.
  • Product manufacturing site information includes site identification, physical location of the site (for example, the physical location of the AOI equipment), node information of the site in the product manufacturing process (for example, defect identification/detection in the exposure process, Identification/defects, etc.), etc., which can assist the product manufacturing message service device 102 to identify or locate a specific site.
  • the first product information can be one or more of product type, product name, product identification, product priority, etc., and this information can assist the product manufacturing message service device 102 to identify or locate a specific product.
  • the content of the product manufacturing site information and the first product information is not limited to the above examples, as long as they are associated with defect identification/detection in the product manufacturing process.
  • the product manufacturing message processing device 103 listens to the first product manufacturing message sent from the product manufacturing message service device 102.
  • the first product manufacturing message includes product manufacturing messages related to registration information.
  • the first product manufacturing message may be a batch product manufacturing message or a single product manufacturing message, as long as it is associated with the product manufacturing site information or the first product information in the registration information.
  • the first product manufacturing message may include change information, status information, etc. of the site identified by the product manufacturing message service device 102.
  • the first product manufacturing message may include the address where the image of the product is stored, the number of images taken for the image, the current manufacturing process of the product, and so on.
  • the product manufacturing message processing device 103 listens to the second product manufacturing message sent from the product manufacturing message service device 102.
  • the second product manufacturing message has nothing to do with the registration information.
  • the product manufacturing message service device 102 may use the same port to broadcast the first product manufacturing message and the second product manufacturing message, and it may also use different ports to broadcast the first product manufacturing message and the second product manufacturing message, which this disclosure does not do. limit.
  • the second product manufacturing message may include information irrelevant to the content of the registration information, for example, the temperature and humidity of the current factory environment, and so on.
  • the second product manufacturing message may be a batch product manufacturing message or a single product manufacturing message.
  • the product manufacturing message processing device 103 determines whether the product manufacturing keyword list includes the product manufacturing keyword.
  • the product manufacturing message processing device 103 retains the second product manufacturing information.
  • the product manufacturing message processing device 103 discards the second product manufacturing information.
  • the registration of the product manufacturing message processing device 103 with the product manufacturing message service device 102 may require more processes, for example, relevant personnel are required to configure and verify this.
  • relevant personnel are required to configure and verify this.
  • products may be constantly adjusted and updated, and there may be situations in which it is necessary to obtain information about new products in time but it is too late to register. Therefore, the product manufacturing message processing device 103 can also monitor messages unrelated to the registration information.
  • the product manufacturing keyword list stores product manufacturing keywords that are irrelevant to the registration information but related to product defect identification, detection, and analysis.
  • the product manufacturing message processing device 103 can analyze the various fields in the second product manufacturing message, compare these fields with the product manufacturing keyword list, and retain the product manufacturing key The word's second product manufacturing message. For example, suppose that the relevant personnel find that product defects in the screens produced in the last few batches are likely to be caused by excessive environmental humidity. At this time, relevant personnel can add a keyword: environmental humidity to the product manufacturing keyword list. Then, when the product manufacturing message processing device 103 receives a message related to environmental humidity, it can retain such information for analysis by relevant personnel.
  • the environmental humidity keyword in the product manufacturing keyword list can be removed to reduce the redundant information stored in the product manufacturing message processing device 103 and improve the product manufacturing message processing device 103 The efficiency of message processing. In the entire process of obtaining environmental humidity information, the product manufacturing message processing device 103 does not need to register with the product manufacturing message service device 102, which reduces the process of product manufacturing message processing.
  • the product manufacturing message processing device 103 can preferentially monitor the registered information, and at the same time filter the non-registered information according to the product manufacturing keywords.
  • the product manufacturing message processing device 103 not only monitors the registered messages, but also can monitor all messages sent by the product manufacturing message service device 102, thereby improving the scalability of the system.
  • the factory manufacturing system has higher requirements for processing product manufacturing messages.
  • the product manufacturing message processing device 103 may also have message buffering and message queuing functions.
  • the product manufacturing message processing device 103 can cache product manufacturing messages in RabbitMQ for message queue management to prevent message loss due to delays or other abnormalities.
  • FIG. 4 is a schematic diagram showing a method 200 for processing a product manufacturing message according to at least one embodiment of the present disclosure, which shows a process of establishing a product defect analysis task queue in the method 200 for processing a product manufacturing message.
  • the product defect analysis task queue is located in the product manufacturing message processing equipment, which provides a buffer between the product image generated by the manufacturing equipment and the product defect analysis task performed by the product defect auxiliary equipment.
  • the product defect analysis tasks in the product defect analysis task queue include the tasks to be performed for product defect analysis.
  • the product manufacturing message processing equipment can control the distribution speed of the tasks in the product defect analysis task queue according to the load pressure of the product manufacturing auxiliary equipment.
  • the task distribution speed is reduced.
  • the task distribution speed is increased.
  • the product manufacturing message processing device 103 establishing a product defect analysis task queue based on multiple product manufacturing messages further includes: based on any one or more pairs of the order of receiving product manufacturing messages, the priority of the product, and the product scheduling plan.
  • the product defect analysis tasks are sorted to construct a product defect analysis task queue.
  • the order of the product defect analysis tasks based on the order in which the product manufacturing messages are received is to facilitate scheduling in chronological order. That is, the corresponding hardware resource matching is performed for processing according to the sequence in which the product manufacturing message is monitored and entered into the queue to be processed.
  • the corresponding hardware resource matching is performed for processing according to the sequence in which the product manufacturing message is monitored and entered into the queue to be processed.
  • 9 product manufacturing messages are received in sequence, and 9 product defect analysis tasks Job1 value Job9 are established accordingly, so these 9 tasks can be arranged in the order of receipt.
  • the method of sequential scheduling is simple to set, and can be better matched with the production plan in terms of time.
  • Sorting the product defect analysis tasks based on the priority of the product includes scheduling according to the set product priority. That is, the products are sorted according to their priority in the production plan.
  • the entire scheduling message queue is a dynamic scheduling process, and high-priority products can be inserted into the front position of the product defect analysis task queue. As shown in Figure 4, assuming that the priority of Job2 and Job3 is higher than Job1, so Job2 and Job3 can be arranged before Job1 for processing. Priority-based scheduling can ensure the smooth completion of inspection tasks for high-priority products.
  • Sorting the product defect analysis tasks based on the product scheduling plan allows the dispatcher to specify the corresponding scheduling plan, and supports temporary insertion and adjustment of the scheduling plan. For example, as shown in Figure 4, Job9 is temporarily inserted between Job2 and Job3. As a result, dispatchers can focus on monitoring and verification of certain defects, and at the same time can synthesize information on other assembly lines to temporarily intervene in the work of the entire product manufacturing assembly line.
  • Some embodiments of the present disclosure may integrate any one or more of the above three methods to sort the product defect analysis tasks to construct a product defect analysis task queue.
  • the changes in the number of outflow tasks over time after scheduling basically maintain a balance (as shown in the number of outflow tasks graph).
  • the embodiments of the present disclosure can perform reasonable scheduling and allocation of inspection tasks for various products, thereby reasonably allocating computing resources and tasks, and at the same time, meet the needs of actual production with the greatest efficiency.
  • FIG. 5 is another flowchart showing a method 200 for processing a product manufacturing message according to at least one embodiment of the present disclosure, which shows an example of operation 220.
  • operation 220 may include all or part of operations 221 to 227.
  • the product manufacturing message processing device 103 acquires a plurality of product images based on the plurality of product manufacturing messages.
  • AOI equipment will shoot a large number of high-definition product images (AOI equipment may take multiple pictures on a single large glass substrate screen), so AOI equipment will send these images to the product image database (such as DFS system) ), and the product manufacturing message sent by the product manufacturing message includes a field in which the image is stored in a specific location of the product image database, so that the product manufacturing message processing device 103 can obtain these product images from the product image database.
  • the product image database such as DFS system
  • the product manufacturing message processing device 103 acquires product defects from a plurality of product images.
  • the product manufacturing message processing device 103 will perform a preliminary analysis on these product images to obtain product defects.
  • the product manufacturing message processing device 103 can locate the location of product defects, the number of product defects, and so on.
  • the image to be analyzed is encoded, and then the encoded image is detected to find out abnormal parts that do not conform to specific rules, so as to eliminate and correct the abnormal parts, and perform reverse encoding on the eliminated and corrected images.
  • the reverse-encoded image is compared with the image to be annotated to obtain the defect location, defect size, and number of defects.
  • the product manufacturing message processing device 103 can generate a product defect analysis task that identifies the product defect or infers the cause of the product defect (inferring the product defect includes identifying the type and location of the product defect). If there is no artificial intelligence defect recognition model corresponding to the product defect, the product manufacturing message processing device 103 can generate a product defect analysis task for training the artificial intelligence defect recognition model.
  • the product manufacturing message processing device 103 may send to the aforementioned model management cluster to query whether the artificial intelligence defect recognition model exists.
  • the product manufacturing message processing device 103 can also query its internally stored list of artificial intelligence defect recognition models to query whether the artificial intelligence defect recognition model exists. The present disclosure does not limit the manner in which the product manufacturing message processing device 103 queries whether the artificial intelligence defect recognition model exists.
  • the product manufacturing message processing device 103 will preprocess these product images, so that the product manufacturing auxiliary device 104 can obtain better analysis results or better artificial intelligence defect recognition models.
  • the number distribution of product defects for large glass substrate screens may be uneven. For example, the number of defects located in the middle of the glass substrate may far exceed the number of defects at the edge of the glass substrate. If preprocessing is not performed, the data in the training set of the artificial intelligence defect recognition model may deviate too much, and the recognition effect of the model may be poor.
  • Product defects may also be inconspicuous. For example, on a large glass substrate screen, there may be defects in only a small pixel block area. If such an image is directly input to the artificial intelligence defect recognition model, the recognition effect may also be poor.
  • the characteristics among multiple product defects may also be inconspicuous, and product defects caused by different reasons may have similar characteristics, which may also lead to poor recognition results.
  • operation 222 further includes any one or more of operations 223 to 226.
  • the product manufacturing message processing device 103 performs any one or more of rotation, scaling, color transformation, and clipping on the product image. Under normal circumstances, you can use data such as data skewness and variance to measure whether product defects are evenly distributed. For example, if the product manufacturing message processing device 103 recognizes that the data skewness of the quantity distribution of product defects in the product image is greater than a predetermined data skewness, it can be determined that the quantity distribution of the product is uneven. In order to identify whether there is an uneven distribution of the number of product defects, the product manufacturing message processing device 103 can periodically perform sample statistics according to products and sites to check the distribution changes.
  • the product manufacturing message processing device 103 can expand these product images.
  • the product manufacturing message processing device 103 may perform the following processing on these product images: rotation, scaling, color transformation, interception, and so on. These processing methods can expand the number of relevant samples, so that the artificial intelligence defect recognition model can more effectively identify product defects.
  • the product manufacturing message processing device 103 enlarges the image area where the product defect is located.
  • the first predetermined threshold may be the maximum area of the product defect area, the maximum ratio of the product defect area to the image area, and the like.
  • the product manufacturing message processing device 103 can perform simple image processing. For example, the product manufacturing message processing device 103 may preliminarily screen out similar images and perform image cutting. Then, the product manufacturing message processing device 103 can process the cut image, for example, enlarge the area where the defect is located.
  • the product manufacturing message processing device 103 merges the product images of the any two product defects.
  • the first predetermined threshold may be the maximum similarity of any two product defects.
  • the similarity of product defect images can be characterized by cosine similarity, Euclidean distance, and Manhattan distance, which is not limited in the present disclosure.
  • the product manufacturing message processing device 103 can merge similar product defects during rough classification, and then perform refined processing later.
  • the product manufacturing message processing device 103 can merge different product defects with similar characteristics to increase the overall number of samples. Then the increased number of sample sets are used as samples for training the artificial intelligence defect recognition model.
  • the product manufacturing message processing device 103 acquires a frequency domain image of the product image, and based on the frequency domain The image is adjusted for positive or negative samples.
  • the third predetermined threshold may be the minimum value of the similarity between the positive sample and the negative sample.
  • the product manufacturing message processing device 103 can adjust the images of the positive sample and the negative sample through image processing to reduce the similarity between the two.
  • the product manufacturing message processing device 103 can obtain the frequency domain image through Fourier transform or wavelet.
  • the product manufacturing message processing device 103 can perform preliminary processing on these frequency domain images (for example, pass these images through any one or more of a high-pass filter, a low-pass filter, and a band-pass filter, In order to enhance the frequency domain image), the frequency domain image is converted into a time domain image through inverse transformation. After that, the inspector or reviewer can review the marks of the positive and negative samples and determine whether the converted image is distorted. This greatly reduces the manual workload and at the same time increases the speed of sample collection.
  • the product manufacturing message processing device 103 generates a product defect analysis task based on the product defect.
  • the product manufacturing message processing device 103 After the product manufacturing message processing device 103 preprocesses the product defect, it can generate a product defect analysis task based on its analysis result and the adjusted product image.
  • Product defect analysis tasks can include product defect analysis tasks that train artificial intelligence defect recognition models, product defect analysis tasks that perform artificial intelligence inferences on the causes of product defects, and product defect analysis tasks that perform artificial intelligence inference and recognition of product defects (that is, based on The product defect content identification task of the artificial intelligence defect recognition model), the product defect analysis task that sets the artificial intelligence defect recognition model, the product defect analysis task that allows relevant personnel to review samples of the product defect model, and allows relevant personnel to check product defects Perform product defect analysis tasks for judgment and so on.
  • the present disclosure does not limit the type of the product defect analysis task, as long as it is associated with the product defect analysis.
  • the product manufacturing message processing device 103 will establish a product defect analysis task queue, and distribute the product defect analysis task to the product manufacturing auxiliary device 104 based on the product defect analysis task queue.
  • the product manufacturing message processing device 103 may also obtain the product type and product defect analysis task type from multiple product manufacturing messages, and generate a product defect analysis request message based on the product type and the product defect analysis task type, and pass Distribute product defect analysis tasks by sending product defect analysis request messages.
  • the product defect analysis request message may also include the product level (such as task priority).
  • the product manufacturing message sent by the AOI device and the product manufacturing message service device 102 may include statistical information on the entire batch of products (for example, a LOT product) or all production messages of a single product. There are a large number of fields in these messages that have nothing to do with product defect identification. Therefore, the product manufacturing message processing device 103 needs to decompose the product manufacturing message, and at the same time encapsulate it into a product defect analysis request message that can be recognized by the product manufacturing auxiliary device 104 according to the requirements of the product manufacturing auxiliary device 104.
  • the product manufacturing message processing device 103 can obtain the number of AOI color pictures contained in the batch of products, the product type of the AOI color pictures, the size of the color pictures, and so on.
  • the AOI color image can be an image taken in any step of the semiconductor screen from the substrate, to deposition, etching, to the final box formation.
  • the product manufacturing message processing device 103 needs to determine the type of product to be analyzed in the product defect analysis task, and the type of product defect analysis task to be performed (for example, reasoning, training, status query, etc.).
  • the product manufacturing message processing device 103 can also verify the message format after the product defect analysis request message, and after determining that the format of the product defect analysis request message is qualified, distribute it to the product manufacturing auxiliary device 104 by sending the product defect analysis request message.
  • Product defect analysis task The product defect analysis task type is used to indicate different product defect analysis tasks.
  • FIG. 6 is another schematic diagram illustrating a method 200 for processing a product manufacturing message according to at least one embodiment of the present disclosure.
  • the distribution of the product defect analysis task by the product manufacturing message processing device 103 to the product manufacturing auxiliary device according to the product defect analysis task queue further includes operations 231 to 235.
  • the product manufacturing message processing device 103 determines whether there is an artificial intelligence defect recognition model corresponding to the product type.
  • the product manufacturing message processing device 103 determining whether there is an artificial intelligence defect recognition model corresponding to the product type includes determining based on the product type in the product manufacturing message.
  • the defect task analysis can be performed based on the artificial intelligence defect recognition model corresponding to the product type.
  • the product type is an unknown product (ie, an untrained product)
  • the recognition score represents the probability that the artificial intelligence recognizes the defect category.
  • the performance of the artificial intelligence defect recognition model is insufficient to meet the defect analysis tasks of the corresponding product type (such as accuracy, precision, and recall rate below the preset threshold), it can also be determined that there is no artificial intelligence defect corresponding to the product type Identify the model.
  • the product manufacturing message processing device 103 can also be determined by other methods such as manual judgment, artificial intelligence defect recognition model pre-judgment, etc.
  • the present disclosure does not deal with the manner in which the product manufacturing message processing device 103 determines whether there is an artificial intelligence defect recognition model corresponding to the product type. limit.
  • the product manufacturing message processing device 103 sends a first product defect analysis request message to the first product manufacturing auxiliary device 104-1.
  • the first product defect analysis request message where the first product defect analysis request message includes the product type, the storage address of the product image, the number of product images, and the task ID for training the artificial intelligence defect recognition model (the task ID indicates the use of The storage address of the product image and the number of product images are trained for the product of this product type).
  • the first product defect analysis request message corresponds to the distribution of the training model task in the product defect analysis task.
  • the first product manufacturing auxiliary device 104-1 may be the aforementioned model management cluster, which may implement, for example, functions such as configuration or management of artificial intelligence defect recognition model training.
  • the first product manufacturing auxiliary device 104-1 can realize the function of training task management for the artificial intelligence defect recognition model.
  • the first product manufacturing auxiliary equipment 104-1 can manage these artificial intelligence defect recognition models according to sites, products, time nodes, etc., and configure training and testing of new models, and so on. Since each artificial intelligence defect recognition model needs to be trained, verified, tested and evaluated before it can be used, when the artificial intelligence defect recognition model is trained and computing resources are limited, the first product manufacturing auxiliary equipment 104-1 can detect artificial intelligence defects According to the priority, training quantity, and the status of hardware resources, the tasks of the recognition model training are scheduled and allocated reasonably for the training tasks of the artificial intelligence defect recognition model.
  • the first product manufacturing auxiliary equipment 104-1 can also realize the function of visualizing the training of the artificial intelligence defect recognition model, so that relevant personnel can observe the status of the artificial intelligence defect recognition model training in real time (for example, whether the training of the artificial intelligence defect recognition model converges, etc. ), to terminate or adjust the training process in time.
  • the first product manufacturing auxiliary equipment 104-1 can also be used in the model library.
  • the artificial intelligence defect recognition model is updated.
  • the first product manufacturing auxiliary equipment 104-1 can also initiate training tasks and evaluate the artificial intelligence defect recognition model in real time according to the process information (such as Loss, verification accuracy, etc.) fed back during the training process to determine the process of model training Is it normal?
  • the product manufacturing auxiliary equipment 104-1 can also add, delete, modify, edit, etc., the training set, verification set, and test set required for training.
  • the product manufacturing auxiliary equipment 104-1 can also deploy artificial intelligence defect recognition models according to product requirements, and perform work such as passing evaluation and testing before deployment.
  • the foregoing description of the functions of the first product manufacturing auxiliary device 104-1 is only an example, and those skilled in the art should understand that the first product manufacturing auxiliary device 104-1 can also implement more unmentioned functions. No restrictions.
  • the first product manufacturing auxiliary device 104-1 may send the product manufacturing message processing device 103 sends a first product defect analysis response message.
  • the product manufacturing message processing device 103 receives the first product defect analysis response message sent by the first product manufacturing auxiliary device 104-1.
  • the first product defect analysis response message includes one or more of the following: identification, accuracy, and recall rate of the artificial intelligence defect recognition model.
  • the artificial intelligence defect recognition model is determined based on the product type, the storage address of the product image, and the number of product images in the first product defect analysis request message.
  • the product manufacturing message processing device 103 sends a second product defect analysis request message to the second product manufacturing auxiliary device.
  • the second product defect analysis request message includes the product type, the storage address of the product image, and the quantity of the product image.
  • the second product defect analysis request message corresponds to the product defect content recognition task based on the artificial intelligence defect recognition model in the product defect analysis task.
  • the second product manufacturing auxiliary device 104-2 may be the aforementioned GPU cluster, which may implement, for example, functions such as performing inference and training tasks of artificial intelligence defect recognition models using GPU computing resources, and scheduling and allocating hardware resources.
  • the second product manufacturing auxiliary device 104-2 may implement the function of loading the artificial intelligence defect recognition model. Since there are many products and sites, and the loading time of the artificial intelligence defect recognition model is long, the second product manufacturing auxiliary device 104-2 can load the artificial intelligence defect recognition model in advance according to preset settings. After the settings are changed, the second product manufacturing auxiliary device 104-2 can also use an independent background server to complete the loading of the model, which avoids the process of loading the model when the product is changed, thereby improving the overall efficiency.
  • the second product manufacturing auxiliary device 104-2 can also perform reasonable scheduling and allocation of GPU resources to improve the use efficiency of hardware resources.
  • the second product manufacturing auxiliary device 104-2 may also test the artificial intelligence defect recognition model to determine the recognition effect of the artificial intelligence defect recognition model.
  • the foregoing description of the functions of the second product manufacturing auxiliary device 104-2 is only an example, and those skilled in the art should understand that the second product manufacturing auxiliary device 104-2 can also implement more unmentioned functions. No restrictions.
  • the second product manufacturing auxiliary device 104-2 may send the second product defect to the product manufacturing message processing device 103 Analyze the response message.
  • the product manufacturing message processing device 103 receives a second product defect analysis response message sent by the second product manufacturing auxiliary device.
  • the second product defect analysis response message includes one or more of the following items: product image identification, product defect location, product defect identification, and repair identification. Wherein, the product defect location, product defect identification and repair identification are determined based on the product type, the storage address of the product image, and the number of product images in the second product defect analysis request.
  • the product manufacturing message processing device 103 can also monitor one or more of the accuracy, precision, recall, F score, and speed of the product manufacturing auxiliary device 104 for processing product defect analysis tasks.
  • the monitoring product manufacturing auxiliary device 104 processes one or more of the accuracy, precision, recall, F score, and speed of the product defect content recognition task based on the artificial intelligence defect recognition model.
  • the speed can refer to the speed of processing product images by the second product manufacturing auxiliary equipment 104-2 (the unit can be the number of product images/second, etc.), or the speed of training a single artificial intelligence defect recognition model (the unit can be Number of training models/hour, etc.).
  • the product manufacturing message processing device 103 can monitor the performance of the artificial intelligence defect recognition model regularly or in real time. If the performance of the product defect model cannot satisfy the production, the product manufacturing message processing device 103 makes adjustments in time. For example, the product manufacturing message processing device 103 can alert in time when a problem occurs, and perform the deployment and confirmation of a new artificial intelligence defect recognition model.
  • the product manufacturing message processing device 103 can also monitor the inference performance of the product defect recognition model, that is, the product manufacturing message processing device 103 handles the accuracy and accuracy of the first product defect recognition task based on the product defect recognition model.
  • the speed may refer to the speed at which the product defect recognition cluster 202 recognizes product images (the unit may be the number of product images per second, etc.).
  • the accuracy rate can be calculated by the following formula (1), the accuracy rate can be calculated by the following formula (2), the recall rate can be calculated by the following formula (3), and then the F score can be calculated by the following formula (4):
  • F Score [(1+ ⁇ 2 ) ⁇ Precision ⁇ Recall]/( ⁇ 2 ⁇ Precision+Recall) (4).
  • TP Ture Positive, true
  • A A represents a result
  • the actual result is also A, and the calculated result is consistent with the actual result at this time.
  • FP False Positive
  • FN False Negative
  • TN Ture Negative
  • the product manufacturing message processing device 103 can monitor the performance of the defect recognition model regularly or in real time. If the performance of the product defect recognition model cannot meet the production requirements, it will be adjusted in time. For example, the product manufacturing message processing device 103 can alert in time when a problem occurs, and perform the deployment and confirmation of a new defect recognition model.
  • the product manufacturing message processing device 103 can monitor the accuracy, precision, recall, F score, or speed of the product manufacturing auxiliary device 104 in processing product defect analysis tasks in the following three ways. Those skilled in the art should understand that the following three methods are only examples. The product manufacturing message processing device 103 can also monitor product defect analysis tasks in other ways, as long as the accuracy, precision, and recall rate of processing product defect analysis tasks can be obtained. Any one of, F score and speed is sufficient.
  • Standard data samples may be stored in the product manufacturing message processing device 103 in advance. Standard data samples can be reviewed by senior factory operators and senior inspectors to determine that the selected range of the sample (for example, defect type, defect number, defect distribution, etc.) is consistent with the artificial intelligence defect recognition model training sample. Then, the product manufacturing message processing device 103 can compare the inference result through the artificial intelligence defect recognition model with the manually set standard result, and then calculate the accuracy, precision, recall, F value and processing of the artificial intelligence defect recognition model. Speed etc. At the same time, the product manufacturing message processing device 103 can also update the standard sample data according to the time period, production status, manual adjustment mode, and so on.
  • the selected range of the sample for example, defect type, defect number, defect distribution, etc.
  • the product manufacturing message processing device 103 can compare the inference result through the artificial intelligence defect recognition model with the manually set standard result, and then calculate the accuracy, precision, recall, F value and processing of the artificial intelligence defect recognition model. Speed etc.
  • the product manufacturing message processing device 103 can also update the standard
  • multiple (for example, 3) senior inspectors can conduct random inspections on product defects. For example, these senior inspectors can extract multiple product defect images (for example, 100 product defects) for a certain product defect. Image), then, manually judge and mark the product defect. After that, multiple inspectors can use the above-mentioned marked product defects to independently review the artificial intelligence defect recognition model against the inference results of the same sample. After the review is over, multiple inspectors can vote on the inference results of the same product defect, and the result with the highest vote can be used as the standard result. Then the standard result is compared with the inference result of the artificial intelligence defect recognition model to monitor the accuracy, precision, recall, F value and processing speed of the artificial intelligence defect recognition model.
  • the product manufacturing message processing device 103 obtains the review result, and determines the accuracy, precision, recall, F value, and processing speed of the artificial intelligence defect recognition model based on the review result.
  • the product manufacturing message processing device 103 can also monitor the status of the human resources for defect identification. For example, in the production process of a factory, there may be cases where the work efficiency of the inspectors is not high and the tasks to be processed cannot be processed in time, or some important alarm information is not processed in time by the operators. These situations will bring greater losses to production. Therefore, the product manufacturing message processing device 103 can also monitor the status of the product manufacturing auxiliary device 104 for manual processing of product defects. For example, the product manufacturing message processing device 103 can monitor the speed of manual processing product defects that it pushes to the product manufacturing auxiliary device 104 (for example, monitor whether the operator is performing manual review of pictures at a normal speed), and judge the operator's work efficiency And working status. Thus, combined with the scheduling plan, the task assignment of product defect identification/detection is optimized.
  • the product manufacturing message processing device 103 can also monitor the efficiency of use of computer resources for the artificial intelligence defect recognition model. If almost all computer resources are used and the reasoning work of defect identification cannot be completed, hardware resources may need to be dynamically increased at this time.
  • the product manufacturing message processing device 103 can monitor the usage efficiency of computing resources in the GPU cluster, including usage of memory, video memory, etc., to determine whether the GPU resources are abnormal.
  • the product manufacturing message processing device 103 can also monitor the alarm status in the factory system. In the process of product defect detection, large-scale defect aggregation may occur.
  • the product manufacturing message processing device 103 can match the product defect information with the department or process department in the factory that handles the product defect, and send alarm information to these departments and departments in time. If the product defect requires the corresponding department/department to make process or production adjustments for the defect, the product manufacturing message processing device 103 can monitor the alarm information and the state of the alarm being processed in real time.
  • the product manufacturing message processing equipment 103 will automatically upgrade the alarm information, and at the same time, send the upgraded alarm information to the upper-level department leader of the department to urge relevant departments/departments to pay attention to and timely Processing alarm information.
  • the product manufacturing message processing device 103 may also acquire analysis result data of multiple product defect analysis tasks (for example, data acquired through the aforementioned analysis unit 103).
  • the product manufacturing message processing device 103 may integrate the aforementioned analysis result data based on one or more of the type of product defect, the format of the result data, and the way of solving the product defect.
  • the product manufacturing message processing device 103 can integrate the analysis result data, so as to provide the various analysis result data to other devices in time, so as to facilitate the provision of alarm information and guidance of subsequent process operations.
  • the product manufacturing message processing device 103 can integrate the result of product defect model judgment and the result of manual review.
  • the product manufacturing message processing device 103 can also integrate the judgment of the product defect model with the process rules, and so on.
  • the product manufacturing message processing device 103 may integrate the aforementioned analysis result data according to the type of product defect.
  • the types of defects can include particles, residues, lines, holes, splashes, electrostatic breakdowns (ESD), wrinkles, and film discoloration. (film color), bubbles (bubble), etc., these belong to the major categories.
  • each major category is further divided into multiple sub-categories. For example, particles are divided into P0 to P9, respectively Refers to different forms of dust defects.
  • Product manufacturing The message processing device 103 may first integrate the product defects of the major categories according to the above-mentioned correspondence between the small categories and the major categories (the correspondence may be many-to-many), so as to facilitate subsequent repairs and treatments.
  • the product manufacturing message processing device 103 can also integrate and count the type information of product defects to determine which department or department can complete the subsequent processing work. If a critical type of product defect occurs on a large scale suddenly, the product manufacturing message processing device 103 will send a large amount of alarm information to the corresponding department.
  • the product manufacturing message processing device 103 can also continue to integrate small categories of product defects. For example, there may be differences in the appearance of the same type of defects, and different departments (Sputter/PECVD) may be required for processing. The product manufacturing message processing device 103 can integrate these different information, classify them, and push them to different departments. .
  • the product defect analysis task further includes analyzing the cause of the product defect.
  • the product manufacturing message processing device 103 obtains the defect content of the product defect (defect type, defect location, defect size, etc.), as well as manufacturing process information related to the product defect content, such as the time of manufacturing the current product and environmental information (for example, including Temperature, humidity, pressure information, etc.), operator information, equipment parameter information, material information, configuration information, etc., the product manufacturing auxiliary equipment 104 responds to the above information (product defect content, and manufacturing process information related to the product defect content) )
  • the product manufacturing auxiliary equipment 104 receives the data generated during the product manufacturing process, and can obtain the current defect cause based on the above-mentioned defect cause analysis model, such as insufficient cleaning link in the semiconductor production process, insufficient corrosion, excessive corrosion, inaccurate raw material matching, Too much dust in the cleaning environment, insufficient exposure intensity, too high exposure intensity, adulteration, etc.
  • the product manufacturing message processing device 103 can give corresponding process and/or equipment adjustment suggestions according to the reason of the defect.
  • FIG. 7 is another schematic diagram illustrating a method 200 for processing a product manufacturing message according to at least one embodiment of the present disclosure.
  • the product manufacturing message processing method 200 may further include all or part of the operations in operation 240.
  • the product manufacturing message processing device 103 may update the artificial intelligence defect recognition model of the product manufacturing auxiliary device 104.
  • operation 240 includes operation 241 to operation 243.
  • the product manufacturing message processing device 103 distributes a product defect analysis task for offline testing of the first artificial intelligence defect recognition model to obtain the first product defect analysis result.
  • the fourth product manufacturing auxiliary equipment 104-4 is currently using the second artificial intelligence defect recognition model to analyze this type of product defect.
  • the product manufacturing message processing device 103 can use the data inferred by the second artificial intelligence defect recognition model as a test set to perform an offline test on the first artificial intelligence defect recognition model to compare the first artificial intelligence defect recognition model and the second artificial intelligence defect recognition The pros and cons of the model.
  • the inspectors in the factory can also review and mark the data generated by the second artificial intelligence defect recognition model inference, and the product manufacturing message processing device 103 then uses the reviewed and marked data as a test set for testing.
  • the product manufacturing message processing device 103 initiates an offline test on the newly trained model, and the product manufacturing message processing device 103 distributes the fourth product defect analysis request message by sending the fourth product defect analysis request message to the fifth product manufacturing auxiliary device 104-5
  • the fourth product defect analysis request message includes the storage path of the first product defect model and the storage path of the test set required for offline testing.
  • the fifth product manufacturing auxiliary device 104-5 may schedule a product defect model testing algorithm to test the first artificial intelligence defect recognition model.
  • the product manufacturing message processing device 103 will obtain the first product defect analysis result, which indicates the accuracy rate, analysis speed, etc. of the product defect analysis of the first product defect model.
  • the inspector can confirm to enter the next test process, or You can go directly to operation 242.
  • the above-mentioned preset standard may also include any one of indicators such as the recall rate of the first product defect analysis result, F1-Score, etc. Because the second artificial intelligence defect recognition model is generated for the same data set According to the analysis results, this disclosure does not limit the preset standards, as long as the pros and cons of the two artificial intelligence defect recognition models can be measured.
  • the product manufacturing message processing device 103 may distribute a product defect analysis task for online testing of the first artificial intelligence defect recognition model to obtain the second product defect analysis result.
  • the online test of the first artificial intelligence defect recognition model may also be referred to as the gray deployment process of the first artificial intelligence defect recognition model.
  • the product manufacturing message processing device 103 may use real-time data generated on the factory assembly line to perform online testing on the first artificial intelligence defect recognition model.
  • the first artificial intelligence defect recognition model and the second artificial intelligence defect recognition model both use the same real-time data, and respectively reason to produce different product defect reasoning results.
  • the product defect reasoning result generated by the reasoning of the first artificial intelligence defect recognition model is the above-mentioned second product defect analysis result.
  • the product manufacturing message processing device 103 may further analyze whether the second product defect analysis result meets the preset standard (for example, whether it is better than the product defect reasoning result generated by the second artificial intelligence defect recognition model inferring the same data).
  • the above-mentioned preset standards may also include the recall rate of the first product defect analysis result, F1-Score and other indicators due to the analysis of the second artificial intelligence defect recognition model for real-time data.
  • the present disclosure does not limit the preset standards, as long as the pros and cons of the two artificial intelligence defect recognition models can be measured.
  • the intelligent defect recognition model performs the fourth product manufacturing auxiliary device 104-4 (for example, sending a fifth product defect request message to the fourth product manufacturing auxiliary device 104-4) that performs the task of product defect recognition reasoning, and performs the first artificial intelligence defect recognition
  • the fifth product manufacturing auxiliary device 104-5 for online model testing for example, sending a sixth product defect request message to the fifth product manufacturing auxiliary device 104-5.
  • the fourth product manufacturing auxiliary device 104-4 and the fifth product manufacturing auxiliary device 104-5 may be the aforementioned GPU cluster, or may be other devices capable of performing analysis or testing of the artificial intelligence defect recognition model.
  • the fourth product manufacturing auxiliary device 104-4 and the fifth product manufacturing auxiliary device 104-5 may be the same device or different devices, which is not limited in the present disclosure.
  • the fifth product manufacturing auxiliary device 104-5 After the fourth product manufacturing auxiliary device 104-4 and the fifth product manufacturing auxiliary device 104-5 receive the product defect analysis request message for online testing sent by the product manufacturing message processing device 103, the fifth product manufacturing auxiliary device 104-5
  • the artificial intelligence online testing algorithm can be scheduled to perform online testing on the first product defect model.
  • the online test of the first artificial intelligence defect recognition model and the reasoning of the second artificial intelligence defect recognition model are carried out simultaneously, and the data sources of the two are the same.
  • the product manufacturing message processing device 103 sends the test task of the first artificial intelligence defect recognition model to the fifth product manufacturing auxiliary device 104-5 in the form of a product defect request message, and the fifth product manufacturing auxiliary device 104-5 can download from The product defect request message obtains the data set storage address required for the online test of the first artificial intelligence defect recognition model and the storage address of the first artificial intelligence defect recognition model. Then, the fifth product manufacturing auxiliary device 104-5 may schedule an artificial intelligence algorithm to execute the online test task of the first artificial intelligence defect recognition model. The fifth product manufacturing auxiliary device 104-5 may return the second product defect analysis result to the product manufacturing message processing device 103 through the sixth product defect analysis response message.
  • the product manufacturing message processing device 103 sends a model replacement request (for example, as shown in FIG. The seventh product defect analysis request message shown) to replace the second artificial intelligence defect recognition model in the fourth product manufacturing auxiliary device 104-4 with the first artificial intelligence defect recognition model.
  • the first artificial intelligence defect recognition model trained by the product manufacturing auxiliary device 104 can pass the entire process of convenient offline testing, efficient and visualized grayscale deployment (online testing), and update release.
  • the gray-scale deployment and online process does not affect the real-time reasoning service of the old model of the production line (the second artificial intelligence defect recognition model), so it has no impact on the normal production of the production line, and will not cause the traditional manual test to deploy the new model (the first artificial intelligence) Defect recognition model) caused by system downtime and other huge losses.
  • the test data used in the gray-scale deployment process is the result of re-judgment by the inspector in the product manufacturing message processing device 103 and the real-time inference data generated by the old model. No other test data is used, which can save a lot of testing. Data collection costs.
  • an embodiment of the present disclosure reasonably utilizes the data generated by the entire automatic bad classification system, realizes the recycling of data, and reduces the cost of training and testing data collection.
  • the product manufacturing message processing device 103 completes the offline test and gray-scale deployment functions of the new model without affecting the production line, avoiding the loss of the artificial intelligence defect recognition model due to the need to stop the test and update.
  • the process described above can also be implemented as a computer software program.
  • an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly embodied on a machine-readable medium, and the computer program includes program code for executing the method of the above-mentioned process.
  • FIG. 8 shows a schematic diagram of a product manufacturing message processing device 800 according to at least one embodiment of the present disclosure.
  • the product manufacturing message processing device 800 may include: a monitoring unit 801 configured to monitor multiple product manufacturing messages; a scheduling unit 802 configured to establish a product defect analysis task queue according to multiple product manufacturing messages; and a distribution unit 803 configured To distribute product defect analysis tasks to product manufacturing auxiliary equipment according to the product defect analysis task queue.
  • FIG. 9 shows a structural diagram of an electronic device 900 according to at least one embodiment of the present disclosure.
  • an electronic device 900 may include a processor 901 and a memory 902. Both the processor 901 and the memory 902 can be connected through the bus 903.
  • the virtual resource transfer device 900 may be a tower server, a rack server (Rack), a blade server (Blade Server), a cabinet server, and the like.
  • the processor 901 can perform various actions and processing according to a program stored in the memory 902.
  • the processor 901 may be an integrated circuit chip with signal processing capability.
  • the above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components.
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA off-the-shelf programmable gate array
  • the methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc., and may be of an X86 architecture or an ARM architecture.
  • the memory 902 stores computer instructions, and the above method 200 is implemented when the computer instructions are executed by the processor 901.
  • the memory 1402 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory.
  • the non-volatile memory may be read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), or flash memory.
  • Volatile memory may be random access memory (RAM), which acts as an external cache.
  • RAM random access memory
  • DRAM dynamic random access memory
  • SDRAM synchronous dynamic random access memory
  • DDRSDRAM double data rate synchronous dynamic Random access memory
  • ESDRAM enhanced synchronous dynamic random access memory
  • SLDRAM synchronous connection dynamic random access memory
  • DRRAM direct memory bus random access memory
  • the product manufacturing message processing method, device, and electronic device of at least one embodiment of the present disclosure can improve the processing efficiency of product manufacturing messages in the entire product manufacturing process, so that each device participating in product defect detection and analysis can run efficiently, which is convenient for follow-up Finding and analyzing the causes of defects improves the efficiency of product manufacturing.
  • each block in the flowchart or block diagram may represent a module, program segment, or part of code, and the module, program segment, or part of code contains one or more logic for implementing the specified Executable instructions for the function.
  • the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two blocks shown one after another can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved.
  • each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart can be implemented by a dedicated hardware-based system that performs the specified functions or operations Or it can be realized by a combination of dedicated hardware and computer instructions.
  • the various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.
  • firmware or software that may be executed by a controller, microprocessor, or other computing device.
  • the processor in at least one embodiment of the present disclosure may be an integrated circuit chip with signal processing capability.
  • the above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components.
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA off-the-shelf programmable gate array
  • the methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc., and may be of an X86 architecture or an ARM architecture.
  • the computer-readable storage medium in at least one embodiment of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory.
  • the non-volatile memory may be read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), or flash memory.
  • Volatile memory may be random access memory (RAM), which acts as an external cache.
  • RAM synchronous dynamic random access memory
  • DRAM dynamic random access memory
  • SDRAM synchronous dynamic random access memory
  • DDRSDRAM double data rate synchronous dynamic Random access memory
  • ESDRAM enhanced synchronous dynamic random access memory
  • SLDRAM synchronous connection dynamic random access memory
  • DR RAM direct memory bus random access memory
  • each block in the flowchart or block diagram can represent a module, program segment, or part of code, and the module, program segment, or part of code contains one or more for realizing the specified logic function.
  • Executable instructions can also occur in a different order from the order noted in the drawings. For example, two blocks shown one after another can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved.
  • each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart can be implemented by a dedicated hardware-based system that performs the specified functions or operations Or it can be realized by a combination of dedicated hardware and computer instructions.
  • the various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.
  • firmware or software that may be executed by a controller, microprocessor, or other computing device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Human Resources & Organizations (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Strategic Management (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Economics (AREA)
  • Educational Administration (AREA)
  • Development Economics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Manufacturing & Machinery (AREA)
  • Operations Research (AREA)
  • General Business, Economics & Management (AREA)
  • Tourism & Hospitality (AREA)
  • Marketing (AREA)
  • Game Theory and Decision Science (AREA)
  • Signal Processing (AREA)
  • Pathology (AREA)
  • Immunology (AREA)
  • General Health & Medical Sciences (AREA)
  • Biochemistry (AREA)
  • Analytical Chemistry (AREA)
  • Chemical & Material Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Power Engineering (AREA)
  • Microelectronics & Electronic Packaging (AREA)
  • Condensed Matter Physics & Semiconductors (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Mechanical Engineering (AREA)
  • Artificial Intelligence (AREA)

Abstract

一种产品制造消息处理方法、设备和电子设备。该产品制造消息处理方法，包括：监听多个产品制造消息（210）；根据所述多个产品制造消息建立产品缺陷分析任务队列（220）；根据产品缺陷分析任务队列向产品制造辅助设备分发产品缺陷分析任务（230），其中，所述产品缺陷分析任务包括基于缺陷识别模型的产品缺陷内容识别任务，其中所述产品缺陷内容包括产品的缺陷类型、缺陷位置和缺陷大小中的任意一项或多项。

Description

产品制造消息处理方法、设备和计算机存储介质 技术领域
本公开涉及一种产品制造消息处理方法、设备和电子设备。本公开还涉及一种人工智能领域和大数据领域，具体地涉及产品制造辅助系统、方法及计算机可读存储介质。
背景技术
在产品制造过程中，例如半导体产品的制造过程中，由于设备、参数、操作、环境干扰等环节存在的问题，会导致产出的产品不符合工艺要求甚至导致不良出现，所以需要在每道工序后及时把不符合要求的不良缺陷产品的不良种类、不良大小、位置等信息计算识别出来，进行及时的修正和改善，避免不良的继续产生。目前传统的识别方法主要是依靠人工检测。这要求对检测人员进行专业培训。尤其在生产的产品型号多、问题复杂的情况下，例如半导体产品的缺陷种类繁多，可能包括微粒、残留、线不良、孔洞、溅落、静电击穿、褶皱、膜层变色、气泡等，需要检测人员投入较长时间和专注力去进行缺陷查找和有关判断。综上，现有技术手段解决上述问题存在效率较低，准确度较低的问题。
在智能化的产品制造过程中，会产生大量的产品制造消息。这些产品制造消息可以用于提示产品的生产进程，或者提示产品在制造过程中可能出现的缺陷。例如，在半导体产品的制造过程中，由于设备、参数、操作、环境干扰等环节存在的问题，会导致产出的产品不符合工艺要求甚至导致不良出现，所以需要在每道工序后及时把不符合要求的缺陷产品的缺陷种类、缺陷大小、位置等信息计算识别出来，以进行及时的修正和改善。
目前，对产品制造消息的处理，尤其是对关于产品缺陷的产品制造消息的处理仍存在处理效率较低的问题。并且，目前对产品制造消息的处理仍无法与产品的生产进程很好的配合，从而给产品制造带来不便。
发明内容
根据本公开至少一实施例提供了一种产品制造消息处理方法，包括：监听多个产品制造消息；根据所述多个产品制造消息建立产品缺陷分析任务队列；根据产品缺陷分析任务队列向产品制造辅助设备分发产品缺陷分析任务，其中，所述产品缺陷分析任务包括基于缺陷识别模型的产品缺陷内容识别任务，其中所述产品缺陷内容包括产品的缺陷类型、缺陷位置和缺陷大小中的任意一项或多项。
根据本公开至少一实施例提供了一种电子设备，包括：处理器；存储器，存储器存储有计算机指令，该计算机指令被处理器执行时实现上述的方法。
根据本公开至少一实施例提供了一种计算机可读存储介质，其上存储有计算机指令，所述计算机指令被处理器执行时实现上述的方法。
附图说明
为了更清楚地说明本公开至少一实施例的技术方案，下面将对实施例的描述中所需要使用的附图作简单的介绍。下面描述中的附图仅仅是本公开的示例性实施例。
图1是示出用于产品制造消息处理的场景的示例示意图。
图2是示出根据本公开至少一实施例的产品制造消息处理方法的流程图。
图3是示出根据本公开至少一实施例的产品制造消息处理方法的另一流程图。
图4是示出根据本公开至少一实施例的产品制造消息处理方法的示意图。
图5是示出根据本公开至少一实施例的产品制造消息处理方法的另一流程图。
图6是示出根据本公开至少一实施例的产品制造消息处理方法的另一示意图。
图7是示出根据本公开至少一实施例的产品制造消息处理方法的另一示意图。
图8示出了根据本公开至少一实施例的产品制造消息处理设备的示意图。
图9示出了根据本公开至少一实施例的电子设备的结构图。
具体实施方式
为了使得本公开的目的、技术方案和优点更为明显，下面将参照附图详细描述根据本公开的示例实施例。显然，所描述的实施例仅仅是本公开的一部分实施例，而不是本公开的全部实施例，应理解，本公开不受这里描述的示例实施例的限制。
在本说明书和附图中，具有基本上相同或相似步骤和元素用相同或相似的附图标记来表示，且对这些步骤和元素的重复描述将被省略。同时，在本公开的描述中，术语“第一”、“第二”等仅用于区分描述，而不能理解为指示或暗示相对重要性或排序。
一方面，在相关的产品制造质量检测的过程中，人工检测存在诸多不稳定因素，会导致检测质量的下降，从而给产品质量带来隐患。另一方面，检测过程中，所有数据均是手动录入，效率低下，同时人工在有限的时间内在待检测产品的图像上获取的信息颗粒度较粗，对后续的不良原因查找和分析带来不便。基于上述全部或部分的原因，本公开提出以下实施例。
下文中提及的产品，包括了实际生产过程中的原材料，以及经过每一道加工工艺(产品制造设备处理)后的半成品或成品，例如，在半导体行业中，产品包括从最开始进入生产线的玻璃、经历曝光工艺的阵列基板、经历过成盒工艺的屏幕等。产品图像，包括了由图像获取设备(如照相机、AOI设备等)直接获取的产品图像，也包括了包含缺陷内容标注的产品图像(即经历过缺陷内容识别的产品图像)。
图1是示出处理产品制造消息的场景100的示例示意图。
参考图1，在场景100中多个产品依次经过站点101。站点101表示在整个生产线流程中产品可能会经过的一个点。
站点101可以是产品流水线上完成标准化生产中一道工序的某个实体设备，或者由多个实体设备组成的系统。例如，半导体行业中阵列基板的光刻工序，对应该工序的站点可以是包括清洗设备、前烘设备、冷却设备、涂胶设备、曝光设备、显影设备、后烘设备、冷却设备等组成的系统。站点101也可以是对应曝光工序的单一设备(曝光设备)或者对应图像检测的AOI设备。站点101还可以是产品制造过程中的虚拟节点，其表示对产品进行非实体形式的加工的步骤。例如，站点101可以对产品进行缺陷检测的过程，其获取并分析用于产品缺陷检测的所有过程信息，进而确定产品缺陷。当产品进入站点101时，站点101会捕获到进入站点消息(trackin消息)。而当产品离开站点101时， 站点会捕获到离开站点消息(trackout消息)。为了保证产品的品质，trackin消息和trackout消息中的产品信息/产品数据均需满足产品制造的要求。
站点101可以包括产品制造消息服务设备102、产品制造消息处理设备103和产品制造辅助设备104。产品制造消息服务设备102也可以不包括在站点101中。产品制造消息服务设备102、产品制造消息处理设备103和产品制造辅助设备104可以是包括处理器和存储器的计算设备。这些设备可以通过网络连接。上述各设备之间可以直接或间接地互相通信，例如，通过网络互相发送和接收数据和/或信号。网络可以是基于互联网和/或电信网的物联网(Internet of Things)，其可以是有线网也可以是无线网，例如，其可以是局域网(LAN)、城域网(MAN)、广域网(WAN)、蜂窝数据通信网络等能实现信息交换功能的电子网络。各个设备之间可以采用一种或多种通信协议来相互通信，例如，FTP、TCP/IP、HSMS和Tibco等。
在本公开中，站点101主要应用于产品的缺陷检测、分析和处理。本领域技术人员应当理解，站点101还可以应用于产品制造生产的其他流程中。
产品制造消息服务设备102可以捕获产品制造流程中的全部或部分产品制造消息(例如上述的trackin消息)，并将这些产品制造消息广播或发送至产品制造消息处理设备103。产品制造消息处理设备103可以对产品制造消息进行进一步的处理，并向产品制造辅助设备104发送对产品缺陷进行检测分析的任务消息。产品制造辅助设备104可以包括以下各项中的一个或多个：检测人员对产品进行缺陷检测的设备、利用人工智能缺陷识别模型对产品进行缺陷检测的设备、部署人工智能缺陷识别模型的设备、训练人工智能缺陷识别模型的设备、产品缺陷报警设备等。产品制造辅助设备104在完成产品缺陷分析检测之后，可以将分析结果返回至产品制造消息处理设备103。
图2是示出根据本公开至少一实施例的产品制造消息处理方法200的流程图。
产品制造消息处理方法200可以包括如图2中所示的一些或全部的操作(例如，操作210至操作230中的部分或全部)。当然，产品制造消息处理方法200也可以包括其他未在图2中示出的操作。产品制造消息处理方法200还可以由其它任何具有通信功能和计算功能的电子设备执行。以下，以产品制造消息处理设备103为例进行说明。
参见图2，在操作210中，产品制造消息处理设备103可以监听多个产品 制造消息。
在智能化的产品制造过程中，会产生大量的产品制造消息。这些产品制造消息可以用于提示产品的生产进程，或者提示产品在制造过程中可能出现的缺陷。产品制造消息可以包括由产品经历任意一个产品制造设备产生的记录信息。通过产品制造消息可以知晓产品已经经过了该产品制造设备的处理以及其他相关处理结果。对于站点101，产品制造消息包括用于指示产品图像产生的消息。
例如，在半导体行业中的屏幕缺陷检测过程中，产品制造设备可以是上文中所述的AOI(Automated Optical Inspection，光学自动检测)设备。产品制造设备可以对屏幕生产过程中进行光学检查和拍照，并确定其拍摄的产品图像与标准产品图像之间的不同。基于这些不同，产品制造设备可以确定正在检测的产品中存在产品缺陷。产品制造设备还可以是其他具有图像获取功能的摄像头或者照相机。然后，产品制造设备可以把所拍摄的产品图像和对应文件发送到产品图像数据库。产品图像数据库可以是上文中所述的DFS(Distributed File System，分布式文件系统)或者其他数据存储设备。对应的指示产品图像产生的消息可以是进入产品制造设备(如AOI设备)的消息(如trackin消息)和/或离开该产品制造设备(如AOI设备)的消息(如trackout消息)，还可以是产品制造设备(如AOI设备)生成产品图片文件的消息或将产品图像发送到产品图像数据库的消息。产品制造设备还可以是其他可以用于产品制造的设备，本公开对此不做限制。
产品制造消息服务设备102可以捕获产品制造流程中的全部或部分产品制造消息，并将这些产品制造消息广播或发送至产品制造消息处理设备103。产品制造设备也可以直接将产品制造消息发送至产品制造消息处理设备103。产品制造消息处理设备103获取的产品制造消息即为监听到的多个产品制造消息。
产品制造消息服务设备102包括制造执行系统(Manufacturing Execution System，MES)，也可以包括主管信息系统(Executive Information System，EIS)，产品制造消息服务设备10还可以是其他用于监控产品制造的设备，本公开对此不作限制。因此，产品制造消息可以是产品制造设备产生的，也可以是产品消息服务设备102捕获的。产品制造消息还包括产品制造站点信息和产品信息。其中，产品制造站点信息包括站点的标识、站点的物理位置(例如， AOI设备的物理位置)、站点在产品制造流程中的节点信息(例如，在曝光流程中的缺陷识别/检测、在清洗流程中的识别/缺陷等等)等，这些信息可以辅助产品制造消息服务设备102识别或定位到具体的站点。产品信息可以是产品类型、产品名称、产品标识、产品优先级等等，这些信息可以辅助产品制造消息服务设备102识别或定位到具体的产品。本领域技术人员应当理解，产品制造站点信息和产品信息的内容并不局限于上述示例，只要其与产品制造流程中的缺陷识别/检测相关联即可。
上述的多个产品制造消息包括至少一个单一产品制造消息和至少一个批量产品制造消息。例如，在AOI设备对产品流水线的屏幕产品进行检查时，AOI可以在一张屏幕(或大玻璃基板屏，又称为Glass)检测完成后发送单一产品制造消息(例如，GlassTrackOut消息)作为一个产品制造消息到产品制造消息服务设备102，并发送图片文件(.jpg/.gls)到图像数据库(DFS系统)，或者其也可以向当前工厂中所有激活的设备发送单一产品制造消息。AOI设备还可能以LOT为单位(1个Lot中包含20个Glass，每张Glass为单张大玻璃基板屏幕)来发送批量产品制造消息。例如，当一个Lot检测完成时，AOI会发送批量产品制造消息(例如，LotTrackOut消息)作为一个产品制造消息到产品制造消息服务设备102或其他任何相关的设备。
工业生产中，将多件产品组成一个批次(LOT)，同一批次进行相同的加工工艺流程，便于产品制造消息的记录和整理。一个批量产品制造消息是指同一批次(如，LOT)的多件产品的产品制造消息总和，一个单一产品制造消息是指该一件单一产品(如，GLASS)的产品制造消息。由于生成批量产品制造消息的周期较长，而生成单一产品制造消息的周期较短，为了提高产品制造消息的处理效率，可选地，在操作210中，监听多个产品制造消息还包括：以中断的方式监听批量产品制造消息；以及以轮询的方式监听单一产品制造消息。中断的方式监听批量产品制造消息是指，监听完第一个批量产品制造消息后中断监听，直至下一个批量产品制造消息产生，再继续监听。轮询的方式监听单一产品制造消息是指以预设频率不断地监听单一产品制造消息设备。在产品制造过程中，通常以批量产品为单位(Lot)进行制造消息的记录和传递，这样能够提高消息处理的效率。但在产品缺陷分析任务过程中，如果仅监听批量产品制造消息，则会导致进行产品制造消息处理设备103和产品缺陷检测分析的设备经常处于空闲状态。因此，产品制造消息处理设备103可以以中 断的方式监听批量产品制造消息，并在中断的间隙中以轮询的方式监听单一产品制造消息，并且及时处理单一产品制造消息。该单一产品可以是该批量产品中的一个产品，也可以不是该批量产品中的。这样可以使得在接收到批量产品制造消息时已经完成了该批次大部分的单一产品制造消息的处理。在完成该批次(例如一个LOT)中的所有产品的处理之后，再进行下一批次的产品制造消息的处理，从而提高了产品制造消息处理设备103的消息处理效率。
可选地，产品制造消息处理设备103为了能以中断的方式监听批量产品制造消息，其还可以将其想要监听的批量产品制造消息的相关信息注册至产品制造消息服务设备102。本公开将在之后的实施例中详细描述注册的过程。由此，批量产品制造完成后由产品制造消息服务设备102广播给产品制造消息处理设备103。产品制造消息处理设备103接着接收批量产品制造消息，接收完后中断，产品制造消息处理设备103可以开始轮询监听单一产品制造消息。当再次有批量产品制造消息时，产品制造消息服务设备102广播给产品制造消息处理设备103。
在操作220中，产品制造消息处理设备103可以根据多个产品制造消息建立产品缺陷分析任务队列。
如上所述，站点101可以作为整个产品制造流程中的检测站点。工厂生产的不同流水线作业的产品在检测流程中都可能流入站点101。目前，在工厂中，由于产品丰富和工艺复杂，从而导致产品制造站点繁多以及产品缺陷复杂。在这样的情况下，产品的流入站点101的频率和数量是不确定的。流入的需要检测的产品有时会突然变得很多，有时又很少。因此，需要对各种产品的检测任务进行合理的调度和分配。为使多个产品制造辅助设备104的检测分析工作有序高效地运行，产品制造消息处理设备103将基于其接收到的产品制造消息建立产品缺陷分析任务队列，以按照该产品缺陷分析任务队列的顺序分发任务。
在操作230中，根据产品缺陷分析任务队列向产品制造辅助设备分发产品缺陷分析任务。其中，产品缺陷分析任务包括基于缺陷识别模型的产品缺陷内容识别任务。
以产品缺陷为例，产品在制造过程中产生产品缺陷的原因各异，例如半导体生成过程中的清洗环节力度不够、腐蚀不够、腐蚀过度、原料匹配不精确、清洗环境微尘过多、曝光强度不够、曝光强度过高、掺杂异物等等。因此，需 要对产品缺陷进行进一步的分析以获取产品的缺陷分类、缺陷位置(例如，产品缺陷所处的电路板及所在层级、掩膜层等、电路板上的具体坐标位置(例如外围矩形的顶点坐标，也可以表示成一个顶点的坐标外加长和宽)、缺陷与背景电路模型形状的关系(例如Gate岛上的两根线之间、缺陷区域所覆盖的Gate岛的数量、缺陷是否完全落在Gate岛内、与之相交、还是其外附近等))、缺陷大小(例如，缺陷的长度或者缺陷的区域面积(例如像素面积))。
不同的产品缺陷分析任务可能需要不同的产品制造辅助设备104来处理。产品制造辅助设备104包括：对人工智能缺陷识别模型进行配置管理的第一产品制造辅助设备104-1、对产品缺陷进行人工智能进行检测分析的第二产品制造辅助设备104-2、和对产品缺陷进行人工检测的第三产品制造辅助设备104-3等等。第一产品制造辅助设备104-1可以是设定人工智能缺陷识别模型的参数并且管理其他产品制造辅助设备的人工智能训练进程的一个或多个设备(例如，模型管理集群)。第二产品制造辅助设备104-2可以是能够利用GPU计算资源进行人工智能缺陷识别模型的推理和训练任务以及对硬件资源进行调度和分配的一个或多个设备(例如，产品缺陷分析集群)。第三产品制造辅助设备104-3可以是向相关的工作人员展示产品缺陷并让其对产品缺陷进行判断的终端(例如，产品制造客户端设备)。以工厂中的产品制造客户端设备为例，其可以向相关工作人员展示产品缺陷图像，让相关工作人员对产品缺陷进行判断、设置相关信息、分析相关数据或者让相关工作人员进行缺陷判断的考试。
产品缺陷分析任务还包括基于人工智能缺陷识别模型的产品缺陷内容识别任务。其中，产品缺陷内容包括产品的缺陷类型、缺陷位置和缺陷大小中的任意一项或多项。产品缺陷分析任务还可以包括人工智能缺陷模型训练任务。人工智能缺陷识别模型包括前馈神经网络人工智能缺陷识别模型、卷积神经网络模型、循环神经网络模型、生成式对抗网络模型中的一项或多项。
在本公开的实施例中，基于人工智能缺陷识别模型的产品缺陷内容识别任务的实现方式如下，首先将产品图像缩放至固定像素大小MxN(也可不进行缩放)。然后将MxN图像送入深度卷积神经网络(VGG/Resnet/MobileNet等)。MxN图像经过深度卷积神经网络的多层卷积层/激活层/池化层之后可获得整个图像的特征图(feature maps)。特征图将被输入到筛选区域网络(ZF/SSD/RPN等)，经过计算，获得建议区域(proposal region)。然后，针 对建议区域进行卷积池化等操作，获得建议区域的区域特征图(proposal feature)，再将区域特征图(proposal feature)送入后续全连接和softmax网络作分类(classification即分类proposal到底是什么缺陷)，获得最大概率的缺陷类别作为最后分类结果，记录类别和概率。另外建议区域(proposal region)的坐标和尺寸代表了缺陷的位置和大小。基于人工智能缺陷模型识别产品缺陷内容的方法可以采用上述方法的类似变形或者其他本领域技术人员公知的方法，本公开在此不做限定。
第二产品制造辅助设备104-2可以用于处理利用人工智能缺陷识别模型从产品图像中识别产品缺陷内容的产品缺陷分析任务。第二产品制造辅助设备104-2可以是能够利用GPU(Graphics Processing Unit,图像处理单元)计算资源进行人工智能缺陷识别模型的推理和训练任务的一个或多个设备。
人工智能缺陷识别模型主要是基于神经网络的。例如，人工智能缺陷识别模型可以是基于前馈神经网络的，即前馈神经网络模型。前馈网络可以被实现为无环图，其中节点布置在层中。通常，前馈网络拓扑包括输入层和输出层，输入层和输出层通过至少一个隐藏层分开。隐藏层将由输入层接收到的输入变换为对在输出层中生成输出有用的表示。网络节点经由边缘全连接至相邻层中的节点，但每个层内的节点之间不存在边缘。在前馈网络的输入层的节点处接收的数据经由激活函数被传播(即，“前馈”)至输出层的节点，激活函数基于系数(“权重”)来计算网络中的每个连续层的节点的状态。系数分别与连接这些层的边缘中的每一个相关联。人工智能缺陷识别模型的输出可以采用各种形式，本公开对此不作限制。人工智能缺陷识别模型还可以包括其他神经网络模型，例如，卷积神经网络(CNN)模型、循环神经网络(RNN)模型、生成式对抗网络(GAN)模型，但不限于此，也可以采用本领域技术人员公知的其他神经网络模型。
第二产品制造辅助设备104-2还可以涉及对人工智能缺陷识别模型的训练。例如，其可以涉及：选择网络拓扑；使用表示被网络建模的问题的一组训练数据；以及调节权重，直到网络人工智能缺陷识别模型针对训练数据集的所有实例表现为具有最小误差。例如，在用于神经网络的监督式学习训练过程期间，将由网络响应于表示训练数据集中的实例的输入所产生的输出与该实例的“正确”的已标记输出相比较；计算表示所述输出与已标记输出之间的差异的误差信号；以及当将误差信号向后传播穿过网络的层时，调节与所述连接相 关联的权重以最小化该误差。当从训练数据集的实例中生成的每个输出的误差被最小化时，人工智能缺陷识别模型被视为“已经过训练”。
人工智能缺陷识别模型的准确度会受到用于训练所述算法的数据集的质量的很大影响。训练过程可以是计算密集型的，因此，使用GPU来训练许多类型的人工智能缺陷识别模型是有益的。在调节神经网络中的系数时执行的计算本身自然地适于并行实现方式。具体地，许多机器学习算法和软件应用已被适配成在通用图形处理设备内使用并行处理硬件。在处理与训练深度神经网络相关联的计算时是高效的。因此，使用多个GPU集成的GPU集群可以有效地提高人工智能缺陷识别模型的训练和推理速度。第二产品制造辅助设备104-2还可以对硬件资源进行调度和分配。
由此，产品制造消息处理设备103可以按照产品类型判断是否需要人工智能缺陷识别模型检测，是否需要训练相关模型，并基于判断结果向第一产品制造辅助设备104-1至第三产品制造辅助设备104-3分发不同的产品缺陷分析任务。例如，对于已知产品类型(如，已训练过的产品)可以生成基于人工智能缺陷识别模型的缺陷内容识别任务。而对于未知产品类型(如，未训练过的新产品)可以生成基于人工(如操作人员)识别的缺陷内容识别任务，此外对于人工智能缺陷识别模型不能识别的产品图像(如，人工智能识别概率低于预设阈值)，产品制造消息处理设备103也可生成基于人工识别的缺陷内容识别任务。产品制造消息处理设备103还可以根据产品缺陷分析任务队列中的数量多少进行第一产品制造辅助设备104-1、第二产品制造辅助设备104-2和第三产品制造辅助设备104-3的任务分类，让计算机资源和人工资源有效率地运行。
根据本公开至少一实施例的产品制造消息处理方法200可以提高整个产品制造过程中的产品制造消息的处理效率，使得参与产品缺陷检测分析的各个设备均高效地运行，便于后续缺陷原因查找和分析，提高了产品制造的效率。
图3是示出根据本公开至少一实施例的产品制造消息处理方法200的另一流程图，其示出了产品制造消息处理方法200中的获取产品制造消息的过程，例如，上述的操作210中的部分或全部操作。
在操作211中，产品制造消息处理设备103向产品制造消息服务设备102发送注册信息。其中，注册信息包括产品制造站点信息和/或第一产品信息。
产品制造消息服务设备102可以通过注册信息获知产品制造消息处理设备103希望知晓的与注册信息相关的产品制造消息。与注册信息相关的产品制造消息包括包含产品制造站点信息或第一产品信息的产品制造消息。从而，在产品制造消息服务设备102收集到与注册信息相关的产品制造消息时，可以优先广播该产品制造消息。这样产品制造消息处理设备103可以以中断的方式监听与注册信息相关的产品制造消息。
产品制造站点信息包括站点的标识、站点的物理位置(例如，AOI设备的物理位置)、站点在产品制造流程中的节点信息(例如，在曝光流程中的缺陷识别/检测、在清洗流程中的识别/缺陷等等)等中的一项或多项，这些信息可以辅助产品制造消息服务设备102识别或定位到具体的站点。第一产品信息可以是产品类型、产品名称、产品标识、产品优先级等种的一项或多项，这些信息可以辅助产品制造消息服务设备102识别或定位到具体的产品。本领域技术人员应当理解，产品制造站点信息和第一产品信息的内容并不局限于上述示例，只要其与产品制造流程中的缺陷识别/检测相关联即可。
在操作212中，基于注册信息，产品制造消息处理设备103监听从产品制造消息服务设备102发送的第一产品制造消息。其中，第一产品制造消息包括与注册信息相关的产品制造消息。第一产品制造消息可以是批量产品制造消息也可以是单一产品制造消息，只要其与注册信息中的产品制造站点信息或第一产品信息相关联即可。当第一产品制造消息与产品制造站点信息相关联时，第一产品制造消息可以包括产品制造消息服务设备102所识别出的站点的变化信息、状态信息等等。当第一产品制造消息与第一产品相关联时，第一产品制造消息可以包括该产品的图像所存储的地址、为该图像拍摄的图像的数量、当前产品的制造进程等等。
在操作213中，产品制造消息处理设备103监听从产品制造消息服务设备102发送的第二产品制造消息。其中，第二产品制造消息与注册信息无关。产品制造消息服务设备102可以使用相同的端口广播第一产品制造消息和第二产品制造消息，其也可以使用不同的端口广播第一产品制造消息和第二产品制造消息，本公开对此不做限制。可选地，第二产品制造消息中可能包括与注册信息的内容不相关的信息，例如，当前工厂环境的温度和湿度等等。当然，第二产品制造消息可以是批量产品制造消息也可以是单一产品制造消息。
在操作214中，产品制造消息处理设备103确定产品制造关键字列表是 否包括所述产品制造关键字。
在产品制造关键字列表中包括产品制造关键字的情况下，在操作215中，产品制造消息处理设备103保留所述第二产品制造信息。
在产品制造关键字列表中不包括产品制造关键字的情况下，在操作216中，产品制造消息处理设备103丢弃第二产品制造信息。
产品制造消息处理设备103向产品制造消息服务设备102进行的注册可能需要较多的流程，例如需要相关人员对此进行配置、核验等等。但是，在工厂中，产品可能是不断调整和更新的，可能存在需要及时获知新产品的相关信息但是来不及注册的情况。因此，产品制造消息处理设备103还可以监听与注册信息无关的消息。
例如，可以通过轮询的方式监听与注册信息无关的消息。产品制造关键字列表中存储着与注册信息无关的但是对产品缺陷识别、检测、分析有关的产品制造关键字。当产品制造消息处理设备103接收到了第二产品制造消息时，其可以分析第二产品制造消息中的各个字段，并将这些字段与产品制造关键字列表进行比对，并且保留包括了产品制造关键字的第二产品制造消息。例如，假设相关人员发现最近几个批次生产的屏幕中的产品缺陷很有可能是由于环境湿度过大导致的。此时，相关人员可以在产品制造关键字列表中增设关键字：环境湿度。然后，当产品制造消息处理设备103接收到与环境湿度有关的消息时，就可以保留这样的信息，以便于相关人员进行分析。当相关人员排除掉环境湿度这样的影响因素后，可以将产品制造关键字列表中的环境湿度关键字去掉，以减少产品制造消息处理设备103中存储的冗余信息，提高产品制造消息处理设备103的消息处理效率。在获取环境湿度信息的整个过程中，产品制造消息处理设备103并不需要向产品制造消息服务设备102注册，减少了产品制造消息处理的流程。
通过这样的机制，产品制造消息处理设备103可以优先监听注册过的信息，同时对非注册的信息进行按照产品制造关键字进行筛选。产品制造消息处理设备103不仅监听经注册的消息，还可以监听所有产品制造消息服务设备102发出的消息，进而提高了系统的可扩展性。
在产品制造消息的监听和广播过程中，工厂制造系统对处理产品制造消息的要求较高。可选地，为了防止消息丢失，产品制造消息处理设备103还可以具有消息缓存和消息队列功能。产品制造消息处理设备103可以将产品制 造消息缓存到RabbitMQ中来进行消息队列管理，以防止由于延时或其他异常造成消息的丢失问题。
图4是示出根据本公开至少一实施例的产品制造消息处理方法200的示意图，其示出了产品制造消息处理方法200中的建立产品缺陷分析任务队列的过程。
参见图4，由于产品的流入站点101的频率和数量是不确定的，从而会导致流入的需要检测的产品有时会突然变得很多，有时又很少，其流入任务数量随时间的变化可以如流入任务数量图所示。
如果不经调度，产品制造辅助设备104的检测分析工作将有时压力过大而有时又相对较空闲，导致产品缺陷分析任务的处理效率不高。产品缺陷分析任务队列位于产品制造消息处理设备中，为制造设备产生产品图像和产品缺陷辅助设备进行产品缺陷分析任务之间提供了缓冲。产品缺陷分析任务队列中的产品缺陷分析任务包括待执行产品缺陷分析的任务，产品制造消息处理设备可根据产品制造辅助设备的负载压力，控制产品缺陷分析任务队列中任务的分发速度。当产品制造消息数量大于预设阈值，以及/或者产品缺陷分析任务队列中的任务数量高于预设阈值，以及/或者产品制造辅助设备负载大于预设阈值时，降低任务分发速度，当产品制造消息数量小于预设阈值，以及/或者产品缺陷分析任务队列中的任务数量小于预设阈值，以及/或者产品制造辅助设备负载小于预设阈值时，增加任务分发速度。
可选地，产品制造消息处理设备103根据多个产品制造消息建立产品缺陷分析任务队列还包括：基于接收产品制造消息的顺序、产品的优先级和产品调度计划中的任意一项或多项对产品缺陷分析任务进行排序以构建产品缺陷分析任务队列。
其中，基于接收产品制造消息的顺序对产品缺陷分析任务进行排序是为了方便按照时间顺序进行调度。也即，按照监听到产品制造消息的进入待处理队列的先后顺序，进行对应硬件资源匹配进行处理。如图4所示，假设接收到依次接收到9条产品制造消息，并据此建立了9个产品缺陷分析任务Job1值Job9，于是可以将这9个任务按照接收顺序排列。顺序调度的方式设定简单，并且可以和生产计划在时间上进行较好的匹配。
基于产品的优先级对产品缺陷分析任务进行排序包括按照设定的产品优先级进行调度。也即，按照产品在生产计划中的优先级进行排序。整个调度消 息队列是一个动态的调度过程，可以将优先级高的产品插入产品缺陷分析任务队列靠前的位置。如图4所示，假设Job2和Job3的优先级高于Job1，可因此可以将Job2和Job3排列在Job1之前进行处理。基于优先级的调度能优先保证高优先级的产品的检测任务顺利完成。
基于产品调度计划对产品缺陷分析任务进行排序可以使得调度人员能够指定对应的调度计划，同时支持临时插入和调整调度计划。例如，如图4所示的将Job9临时插入到Job2和Job3之间。由此，调度人员可以对某些缺陷进行重点的监控和验证，同时可以综合其他流水线上的信息进行临时干预整个产品制造流水线上的工作。
本公开的一些实施例可以综合以上三种方式中的任意一项或多项对产品缺陷分析任务进行排序以构建产品缺陷分析任务队列。如图4所示，经调度之后流出任务数量随时间的变化基本保持了均衡(如流出任务数量图所示)。本公开的实施例可以对各种产品的检测任务进行合理的调度和分配，进而合理地分配了计算资源和任务，同时最大效率的满足实际生产的需要。
图5是示出根据本公开至少一实施例的产品制造消息处理方法200的另一流程图，其示出了操作220的一个示例。
参见图5，操作220可以包括操作221至操作227中的全部或部分的操作。
在操作221中，产品制造消息处理设备103基于多个产品制造消息，获取多个产品图像。由于在产品缺陷检测过程中，AOI设备将拍摄大量的高清产品图像(AOI设备可能对单张大玻璃基板屏幕拍摄多张图片)，因此AOI设备会将这些图像发送到产品图像数据库(如DFS系统)中进行存储，并在其发出的产品制造消息中包括图像存储在产品图像数据库的具体位置的字段，以便于产品制造消息处理设备103从产品图像数据库中获取这些产品图像。
在操作222中，产品制造消息处理设备103从多个产品图像中获取产品缺陷。产品制造消息处理设备103将对这些产品图像进行初步分析以获取产品缺陷。例如，产品制造消息处理设备103可以定位产品缺陷的位置、产品缺陷的数量等。在一些实施例中，对待分析图像进行编码，之后对编码的图像进行检测，找出不符合特定规律的异常部分，以剔除和补正其中的异常部分，对经剔除和补正的图像进行反编码，将经反编码的图像和待标注的图像进行对比，以获得缺陷位置、缺陷大小和缺陷数量等。
如果存在对应产品缺陷的人工智能缺陷识别模型，产品制造消息处理设备103可以生成识别产品缺陷或推理产品缺陷产生原因的产品缺陷分析任务(推理产品缺陷包括识别产品缺陷的类型、位置等信息)。如果不存在产品缺陷对应的人工智能缺陷识别模型，产品制造消息处理设备103可以生成训练该人工智能缺陷识别模型的产品缺陷分析任务。产品制造消息处理设备103可以向上述的模型管理集群发送查询人工智能缺陷识别模型是否存在。产品制造消息处理设备103还可以查询其内部存储的人工智能缺陷识别模型列表来查询人工智能缺陷识别模型是否存在。本公开对产品制造消息处理设备103查询人工智能缺陷识别模型是否存在的方式不作限制。
由于产品缺陷的种类繁多并且其分布各异，产品制造消息处理设备103将对这些产品图像进行预处理，以便于产品制造辅助设备104获取更好的分析结果或者更好的人工智能缺陷识别模型。
例如，大玻璃基板屏幕的产品缺陷的数量分布可能是不均匀的。例如，位于玻璃基板的中部的缺陷数量可能远远超出玻璃基板边缘的缺陷数量。如果未进行预处理，可能导致人工智能缺陷识别模型的训练集的数据偏离过大，进而导致模型的识别效果不好。产品缺陷还有可能是不明显的，例如在大玻璃基板屏幕上，可能只有很小的像素块区域中存在缺陷。如果直接将这样的图像输入至人工智能缺陷识别模型，也可能会导致识别效果不好。多个产品缺陷之间的特征还可能是不明显的，不同原因导致的产品缺陷可能具有相似的特征，这样也可能导致识别效果不好。为了识别产品缺陷，站点101中可能训练了大量的人工智能缺陷识别模型。然而在训练这些模型的时候，产品图像中的正样本和负样本之间可能对比不明显，导致训练处的人工智能缺陷识别模型识别效果不好。综上所述，如果不对产品图像进行预处理，可能会导致产品制造辅助设备104的识别难度增大并且识别效果不好。
因此，产品制造消息处理设备103可以对产品图像进行以下判断和处理以改善人工智能缺陷识别模型的识别效果。可选地，操作222还包括操作223至操作226中的任意一项或多项。
在操作223中，在产品缺陷的数量分布不均匀的情况下，产品制造消息处理设备103对所述产品图像进行旋转、缩放、色彩变换和截取中的任一项或多项处理。通常情况下，可以使用数据偏斜度、方差等数据来衡量产品缺陷是否是均匀分布的。例如，如果产品制造消息处理设备103识别到产品图像 中产品缺陷的数量分布的数据偏斜度大于预定的数据偏斜度，则可以确定该产品的数量分布不均匀。为了识别产品缺陷的数量分布是否存在不均匀的情况，产品制造消息处理设备103可以按照产品和站点定期进行样本统计，查看分布的变化。对于产品缺陷数量较少但是工艺要求很高的产品图像，产品制造消息处理设备103可以对这些产品图像进行扩充。例如，产品制造消息处理设备103可以对这些产品图像进行以下处理：旋转、缩放、色彩变换、截取等。这些处理手段可以扩充相关样本数量，从而使得人工智能缺陷识别模型能够更有效的识别产品缺陷。
在操作224中，在产品缺陷所在的图像区域小于第一预定阈值的情况下，产品制造消息处理设备103放大产品缺陷所在的图像区域。第一预定阈值可以是产品缺陷区域的最大面积、产品缺陷区域占图像区域的最大比率等。针对大产品图像中的产品缺陷区域很小的情况，产品制造消息处理设备103可以进行简单的图像处理。例如，产品制造消息处理设备103可以初筛出类似图像并进行图像切割。然后，产品制造消息处理设备103可以对切割后的图像进行处理，例如，对缺陷所在的区域进行放大处理。
在操作225中，在任意两个产品缺陷的相似度大于第二预定阈值的情况下，产品制造消息处理设备103合并所述任意两个产品缺陷的产品图像。第一预定阈值可以是任意两个产品缺陷的最大相似度。产品缺陷图像的相似度可以通过余弦相似度、欧几里得距离、曼哈顿距离来表征，本公开对此不作限定。针对产品缺陷之间特征不明显的情况，产品制造消息处理设备103在粗分类时可以对相似的产品缺陷进行合并处理，后续再进行精细化处理。而在训练人工智能缺陷识别模型的阶段，产品制造消息处理设备103可以对特征相似的不同产品缺陷做合并处理，以增加样本整体数量。然后再将增加了数量后的样本集作为训练人工智能缺陷识别模型的样本。
在操作226中，在产品图像中的正样本和负样本之间的相似度大于第三预定阈值的情况下，产品制造消息处理设备103获取所述产品图像的频域图像，并基于该频域图像调整正样本或负样本。第三预定阈值可以是正样本和负样本之间的相似度的最小值。针对上述提到的正负样本过于相似的情况，产品制造消息处理设备103可以通过图像处理的手段来调整正样本和负样本的图像以使二者的相似度降低。产品制造消息处理设备103可以通过傅里叶变换或小波获取频域图像。然后，基于所获取的频域图像来计算和分析产品图像中 的正样本和负样本的统计特征。根据这些统计特征，产品制造消息处理设备103可以对这些频域图像进行初步的处理(例如，将这些图像通过高通滤波器、低通滤波器、带通滤波器中的任意一项或多项，以对频域图像进行增强)，再通过反变换将频域图像转换为时域图像。之后，检测人员或复核任意可以复核正负样本的标记并确定转换后的图像是否失真。这样大大减轻了人工的工作量，同时提高了样本收集的速度。
在操作227中，产品制造消息处理设备103基于产品缺陷生成产品缺陷分析任务。
产品制造消息处理设备103在对产品缺陷的预处理之后，可以基于其的分析结果和调整后的产品图像来生成产品缺陷分析任务。
产品缺陷分析任务可以包括训练人工智能缺陷识别模型的产品缺陷分析任务、对产品缺陷产生原因进行人工智能推理的产品缺陷分析任务、对产品缺陷内容进行人工智能推理识别的产品缺陷分析任务(即基于人工智能缺陷识别模型的产品缺陷内容识别任务)、对人工智能缺陷识别模型进行设置的产品缺陷分析任务、让相关人员对产品缺陷模型的样本进行复核的产品缺陷分析任务、让相关人员对产品缺陷进行判断的产品缺陷分析任务等等。本公开不对产品缺陷分析任务的类型进行限定，只要其与产品缺陷分析相关联即可。
在产品缺陷分析任务生成之后，产品制造消息处理设备103将建立产品缺陷分析任务队列，并基于该产品缺陷分析任务队列向产品制造辅助设备104分发产品缺陷分析任务。
可选地，产品制造消息处理设备103还可以从多个产品制造消息中获取产品类型、产品缺陷分析任务类型，并且基于产品类型、和产品缺陷分析任务类型，生成产品缺陷分析请求消息，并通过发送产品缺陷分析请求消息的方式分发产品缺陷分析任务。产品缺陷分析请求消息还可以包括产品等级(如任务优先级)。
AOI设备和产品制造消息服务设备102中发送的产品制造消息可能包括对整个批次的产品(例如，一个LOT的产品)的统计信息或者单个产品的所有生产消息。这些消息中存在大量的字段与产品缺陷识别无关的内容。因此，产品制造消息处理设备103需要对产品制造消息进行分解，同时按照产品制造辅助设备104的要求进行封装成产品制造辅助设备104能识别的产品缺陷分析请求消息。例如在产品制造消息分析时，产品制造消息处理设备103可 以从中获取该批次的产品中包含的AOI彩图数量、AOI彩图的产品类型、彩图的尺寸等等。AOI彩图可以是半导体屏幕从基板、到沉积、刻蚀到最后成盒的任意一步中拍摄的图像。在生成产品缺陷分析请求消息时，产品制造消息处理设备103需要判断产品缺陷分析任务中要进行分析的产品类型、要执行的产品缺陷分析任务类型(例如，推理、训练、状态查询等等)。之后，产品制造消息处理设备103还可以对产品缺陷分析请求消息后进行消息格式验证，在确定产品缺陷分析请求消息的格式合格后再通过发送产品缺陷分析请求消息的方式向产品制造辅助设备104分发产品缺陷分析任务。产品缺陷分析任务类型用于指示不同的产品缺陷分析任务。
图6是示出根据本公开至少一实施例的产品制造消息处理方法200的另一示意图。
参见图6，可选地，产品制造消息处理设备103根据产品缺陷分析任务队列向产品制造辅助设备分发产品缺陷分析任务还包括操作231至操作235。
在操作231中，产品制造消息处理设备103确定是否存在与产品类型相对应的人工智能缺陷识别模型。
产品制造消息处理设备103确定是否存在产品类型相对应的人工智能缺陷识别模型包括基于产品制造消息中的产品类型进行确定。
如果产品类型为已知产品(即训练过的产品)并且确定存在与产品类型相对应的人工智能缺陷识别模型，则可基于产品类型对应的人工智能缺陷识别模型进行缺陷任务分析的执行。
如果产品类型为未知产品(即未训练过的产品)，则确定不存在与产品类型相对应的人工智能缺陷识别模型。
如果产品类型为已知产品但基于该产品类型对应的人工智能缺陷识别模型识别不出产品缺陷内容(如识别得分低于预设阈值)，则也可以确定为不存在与产品类型相对应的人工智能缺陷识别模型。可选地，识别得分代表了人工智能识别出该缺陷类别的概率。
如果人工智能缺陷识别模型的性能不足以满足对应产品类型的缺陷分析任务(如准确率、精确率、召回率低于预设阈值)，也可以确定为不存在与产品类型相对应的人工智能缺陷识别模型。
产品制造消息处理设备103还可以通过人工判断、人工智能缺陷识别模型预判断等其他方式进行确定，本公开对产品制造消息处理设备103确定是 否存在产品类型相对应的人工智能缺陷识别模型的方式不作限制。
在不存在与所述产品类型相对应的人工智能缺陷识别模型的情况下，在操作232中，产品制造消息处理设备103向第一产品制造辅助设备104-1发送第一产品缺陷分析请求消息。第一产品缺陷分析请求消其中，第一产品缺陷分析请求消息包括所述产品类型、产品图像的存储地址、产品图像的数量和对人工智能缺陷识别模型进行训练的任务标识(该任务标识指示利用产品图像的存储地址、产品图像的数量对该产品类型的产品进行训练)。第一产品缺陷分析请求消息对应于产品缺陷分析任务中训练模型任务的分发。第一产品制造辅助设备104-1可以是上述的模型管理集群，其可以实现，例如对人工智能缺陷识别模型训练进行配置或管理等的功能。例如，第一产品制造辅助设备104-1可以实现人工智能缺陷识别模型训练任务管理的功能。
由于产品和站点众多，利用通用的人工智能缺陷识别模型并不能解决所有问题，所以需要提供多种人工智能缺陷识别模型来对不同的产品类型的产品进行判断。在工厂生产的过程中，还可能存在对产品模型进行微调的需求。例如，当工厂流水线的生产工艺发生变化，或者生产设备进行变更之后，产生的AOI图像将会变化。这样的变化会导致工厂流水线的人工智能缺陷识别模型的准确率下降，需要重新训练模型。
第一产品制造辅助设备104-1可以对这些人工智能缺陷识别模型按照站点、产品、时间节点等进行管理，并配置新模型的训练和测试等等。由于每个人工智能缺陷识别模型都需要进行训练、验证、测试和评估才能使用，在训练人工智能缺陷识别模型并且计算资源有限的情况下，第一产品制造辅助设备104-1可以对人工智能缺陷识别模型训练的任务按照优先级、训练数量、硬件资源的状态，进行人工智能缺陷识别模型训练任务调度和合理分配。
第一产品制造辅助设备104-1还可以实现将人工智能缺陷识别模型训练可视化的功能，以便于相关人员实时观察人工智能缺陷识别模型训练的状态(例如，人工智能缺陷识别模型的训练是否收敛等)，以及时终止或者调整训练过程。随着数据集的不断丰富，训练参数的不断优化，针对同一种产品或缺陷检测，如果新训练的模型比旧模型效果要好，第一产品制造辅助设备104-1还可以在模型库中对该人工智能缺陷识别模型进行更新。第一产品制造辅助设备104-1还可以发起训练任务并且按照训练过程反馈的过程信息(例如Loss、验证的准确率等)，对人工智能缺陷识别模型实时地进行评估，以确定模型训 练的过程是否正常。产品制造辅助设备104-1还可以针对训练所需的训练集、验证集、测试集进行增加、删除、修改编辑等。产品制造辅助设备104-1还可以按照产品要求进行人工智能缺陷识别模型的部署，并在部署前进行通过评估与测试等工作。上述对第一产品制造辅助设备104-1的功能的描述仅是示例，本领域技术人员应当理解，第一产品制造辅助设备104-1还可以实现更多未提及的功能，本公开对此不作限制。
在第一产品制造辅助设备104-1完成产品缺陷分析的任务(例如，确定与该产品类型对应的人工智能缺陷识别模型)之后，第一产品制造辅助设备104-1可以向产品制造消息处理设备103发送第一产品缺陷分析响应消息。在操作233中，产品制造消息处理设备103接收由第一产品制造辅助设备104-1发送的第一产品缺陷分析响应消息。第一产品缺陷分析响应消息包括以下各项中的一项或多项：人工智能缺陷识别模型的标识、准确率和召回率。其中，人工智能缺陷识别模型是基于第一产品缺陷分析请求消息中的产品类型、产品图像的存储地址和产品图像的数量而确定的。
在存在与所述产品类型相对应的人工智能缺陷识别模型的情况下，在操作234中，产品制造消息处理设备103向第二产品制造辅助设备发送第二产品缺陷分析请求消息。其中，第二产品缺陷分析请求消息包括产品类型、产品图像的存储地址和产品图像的数量。第二产品缺陷分析请求消息对应于产品缺陷分析任务中的基于人工智能缺陷识别模型的产品缺陷内容识别任务。
第二产品制造辅助设备104-2可以是上述的GPU集群，其可以实现，例如对利用GPU计算资源进行人工智能缺陷识别模型的推理和训练任务、对硬件资源进行调度和分配等的功能。例如，第二产品制造辅助设备104-2可以实现人工智能缺陷识别模型的加载的功能。由于产品和站点较多，且人工智能缺陷识别模型的加载时间较长，第二产品制造辅助设备104-2可以按照预先设置进行提前加载人工智能缺陷识别模型。在设置变更后，第二产品制造辅助设备104-2还可以利用独立的后台服务器完成对模型的加载，避免了产品变更时，需要大量时间加载模型的过程，从而提高了整体效率。由于产线每天会产生大量的图像进行人工智能缺陷识别模型算法的检测工作，第二产品制造辅助设备104-2还可以对GPU资源进行合理的调度和分配，以提高硬件资源的使用效率。第二产品制造辅助设备104-2还可以对人工智能缺陷识别模型进行测试以确定人工智能缺陷识别模型的识别效果。上述对第二产品制造辅助 设备104-2的功能的描述仅是示例，本领域技术人员应当理解，第二产品制造辅助设备104-2还可以实现更多未提及的功能，本公开对此不作限制。
在第二产品制造辅助设备104-2完成产品缺陷分析的任务(例如，对产品缺陷进行推理分析)之后，第二产品制造辅助设备104-2可以向产品制造消息处理设备103发送第二产品缺陷分析响应消息。在操作235中，产品制造消息处理设备103接收由第二产品制造辅助设备发送的第二产品缺陷分析响应消息。第二产品缺陷分析响应消息包括以下各项中的一项或多项：产品图像标识、产品缺陷位置、产品缺陷标识和修复标识。其中，产品缺陷位置、产品缺陷标识和修复标识是基于第二产品缺陷分析请求中的产品类型、产品图像的存储地址和产品图像的数量而确定的。
可选地，产品制造消息处理设备103还可以监控产品制造辅助设备104处理产品缺陷分析任务的准确率、精确率、召回率、F分数和速度中的一项或多项。例如，监控产品制造辅助设备104处理基于人工智能缺陷识别模型的产品缺陷内容识别任务的准确率、精确率、召回率、F分数和速度中的一项或多项。其中，速度可以指第二产品制造辅助设备104-2处理产品图像的速度(其单位可以是产品图像的数量/秒等)，也可以指训练单个人工智能缺陷识别模型的速度(其单位可以是训练模型数量/小时等)。
在工厂生产过程中，产品电路的背景复杂多样，许多产品缺陷容易混淆，产品的生产计划也会随着订单进行调整，这些因素都给模型性能带来较大的挑战。产品的变更、工艺变化或者产品图片的调整都可能会导致人工智能缺陷识别模型的性能变差。产品制造消息处理设备103可以定期或实时对人工智能缺陷识别模型的性能进行监控。如果产品缺陷模型的性能不能满足生产时，产品制造消息处理设备103及时进行调整。例如，产品制造消息处理设备103可以在出现问题时及时报警，并进行新的人工智能缺陷识别模型的部署和确认。
可选地，产品制造消息处理设备103还可以监控产品缺陷识别模型的推理性能，即产品制造消息处理设备103基于该产品缺陷识别模型处理第一产品缺陷识别任务的准确率(Accuracy)、精确率(Precision)、召回率(Recall)、F分数(FScore)和速度中的一项或多项。其中，速度可以指产品缺陷识别集群202识别产品图像的速度(其单位可以是产品图像的数量/秒等)。可以通过如下公式(1)计算出准确率，通过如下公式(2)计算出精确率，通过如下 公式(3)计算出召回率，之后通过如下公式(4)计算出F分数：
Accuracy＝(TP+TN)/(TP+FP+TN+FN)     (1)；
Precision＝TP/(TP+FP)      (2)；
Recall＝TP/(TP+FN)     (3)；
F Score＝[(1+β 2)·Precision·Recall]/(β 2·Precision+Recall)     (4)。
其中，TP(Ture Positive，真正)表示计算结果为正，实际结果也为正，也就是说，缺陷识别模型经推理计算后，得到的计算结果为A(这里，A代表一种结果)，实际结果也为A，此时计算结果与实际结果一致。
FP(False Positive，假正)表示计算结果为正，实际结果为负，也就是说，缺陷识别模型经推理计算后，得到的计算结果A，但实际结果为非A，此时计算结果和实际结果不一致。
FN(False Negative，假负)表示计算结果为负，实际结果为正，也就是说，缺陷识别模型经推理计算后，得到的计算结果为非A，但实际结果是A，此时计算结果和实际结果不一致。
TN(Ture Negative，真负)表示推理计算结果为负，实际结果也为负，也就是说，缺陷识别模型经推理计算后，得到的计算结果为非A，实际结果也为非A，此时计算结果与实际结果一致。
在工厂生产过程中，产品电路的背景复杂多样，许多产品缺陷容易混淆，产品的生产计划也会随着订单进行调整，这些因素都给模型性能带来较大的挑战。产品的变更、工艺变化或者产品图片的调整都可能会导致人工智能缺陷识别模型的性能变差。产品制造消息处理设备103可以定期或实时对缺陷识别模型的性能进行监控。如果产品缺陷识别模型的性能不能满足生产时，将及时进行调整。例如，产品制造消息处理设备103可以在出现问题时及时报警，并进行新的缺陷识别模型的部署和确认。
产品制造消息处理设备103可以通过以下3种方式来监控产品制造辅助设备104处理产品缺陷分析任务的准确率、精确率、召回率、F分数或速度。本领域技术人员应当理解，以下3种方式仅是示例，产品制造消息处理设备103还可以通过其它方式来监控产品缺陷分析任务，只要能够获得处理产品缺陷分析任务的准确率、精确率、召回率、F分数和速度中的任意一项即可。
方式一：通过标准数据样本进行监控
在产品制造消息处理设备103中可以预先存储标准数据样本。标准数据样本可以由资深工厂操作人员和高级检测人员进行审核，以确定样本的选定 范围(例如，缺陷种类、缺陷数量和缺陷分布等)与人工智能缺陷识别模型训练样本保持一致。然后，产品制造消息处理设备103可以比对通过人工智能缺陷识别模型的推理结果与人工设定的标准结果，进而统计该人工智能缺陷识别模型的准确率、精确率、召回率、F值和处理速度等。同时，产品制造消息处理设备103还可以对标准样本数据按照时间周期、生产状况、人工调整模式等进行更新。
方式二：通过在线抽检模型进行监控
在工厂的生产过程中，多名(例如，3名)高级检测人员可以对产品缺陷进行随机抽检，例如，这些高级检测人员可以针对某个产品缺陷抽取多张产品缺陷图像(例如100张产品缺陷图像)，然后，人工判断和标记该产品缺陷。之后，多名检测人员可以利用上述标记后的产品缺陷将人工智能缺陷识别模型针对同一份样本的推理结果分别独立地进行复核。在复核结束后，可以将多名检测人员针对同一产品缺陷的推理结果进行投票，投票最高的结果可以作为标准结果。然后将该标准结果与人工智能缺陷识别模型的推理结果进行比较以监控人工智能缺陷识别模型的准确率、精确率、召回率、F值和处理速度等。
方式三：抽取产品批次进行监控
多名(例如，3名)高级检测人员可以抽取人工智能缺陷识别模型对整个批次(例如，一个LOT，其中，一个LOT可以包括20张Glass，而每张Glass中可能包括50至300个不等的产品缺陷)的产品缺陷的推理结果进行复核。产品制造消息处理设备103获取复核结果，并根据复核结果确定人工智能缺陷识别模型的准确率、精确率、召回率、F值和处理速度等。
产品制造消息处理设备103还可以监控进行缺陷识别的人工资源的状态。例如，在工厂生产过程中，可能存在检测人员的工作效率不高而导致待处理的任务不能及时处理的情况，或者，某些重要的报警信息没有被操作人员及时处理的情况。这些情况都会给生产带来较大的损失。因此，产品制造消息处理设备103还可以监控用于人工处理产品缺陷的产品制造辅助设备104的状态。例如，产品制造消息处理设备103可以监控其推送给产品制造辅助设备104的人工处理产品缺陷的速度(例如，监控操作员是否按照正常的速度在进行图片的人工复核)，判断操作员的工作效率和工作状态。从而结合调度计划，优化产品缺陷识别/检测的任务分配。
在工厂生产过程中，产品制造消息处理设备103还可以监控用于人工智能缺陷识别模型的计算机资源的使用效率。如果使用了几乎全部的计算机资源也无法完全完成缺陷识别的推理工作，这时候可能需要动态地增加硬件资源等。产品制造消息处理设备103可以监控GPU集群中计算资源的使用效率，包括内存、显存等使用情况，以确定GPU资源是否存在异常等。
产品制造消息处理设备103还可以监控工厂系统中的报警状态。在产品缺陷检测过程中，可能会发生大规模缺陷聚集情况。产品制造消息处理设备103可以把产品缺陷信息与工厂中处理该产品缺陷科室或工艺部门进行匹配，并向这些科室和部门及时发送报警信息。如果该产品缺陷需要对应的部门/科室针对缺陷进行工艺或生产调整，产品制造消息处理设备103可以实时监控报警信息和报警被处理的状态。并且在出现严重报警没有及时处理的情况，产品制造消息处理设备103会自动升级报警信息，同时将升级后的报警信息发送到该科室上一级的部门领导，以督促相关科室/部门及时重视和处理报警信息。
可选地，产品制造消息处理设备103还可以获取多个产品缺陷分析任务的分析结果数据(例如，通过上述分析部103获取的数据)。并且，产品制造消息处理设备103可以基于产品缺陷类型、结果数据的格式和解决产品缺陷的方式中的一项或多项整合上述分析结果数据。
在生产过程中，由于各种影响因素存在，导致各种产品缺陷多达上百种，但是这些产品缺陷并不是全部影响产品的最终品质，也不是所有影响产品的最终品质的产品缺陷都能被修复。产品制造消息处理设备103可以对分析结果数据进行整合，以将纷杂多样的分析结果数据及时提供给其他设备，从而便于提供警报信息和指导后面工艺操作等。产品制造消息处理设备103可以对产品缺陷模型判断的结果和人工复核的结果进行整合。产品制造消息处理设备103还可以将产品缺陷模型的判断与工艺规则进行整合等等。
例如，产品制造消息处理设备103可以按照产品缺陷类型来对上述的分析结果数据进行整合。对于半导体制造工艺，缺陷类型可以包括微粒(particle)、残留(remain)、线不良(line)、孔洞(hole)、溅落(splash)、静电击穿(ESD)、褶皱(wrinkle)、膜层变色(film color)、气泡(bubble)等，这些属于大类，除此之外，必要时，每个大类下面还进一步划分为多个小类，例如微粒(particle)分为P0至P9，分别指的是不同形态的微尘缺陷。产品制造 消息处理设备103可以根据上述的小类与大类的对应关系(该对应关系可能是多对多的)，先整合大类的产品缺陷，以便于后续如何进行维修和处理。例如对于Short类型的产品缺陷可能需要进行切割，而对于Open类型的产品缺陷可能需要进行连接等。产品制造消息处理设备103还可以对产品缺陷的类型信息进行整合和统计，以确定哪个部门或科室能够完成后续的处理工作。如果突然出现关键类型产品缺陷大规模的产生的情况，产品制造消息处理设备103会向对应科室发送大量的报警信息。产品制造消息处理设备103还可以继续整合小类的产品缺陷。例如，同一种小类缺陷展现的形态可能存在差异，并且需要不同的部门(Sputter/PECVD)来进行处理，产品制造消息处理设备103可以整合这些不同信息，进行分类后，分别推送给不同的部门。
在一些实施例中，产品缺陷分析任务还包括对产品缺陷产生原因分析。产品制造消息处理设备103获取到产品缺陷的缺陷内容(缺陷类型、缺陷位置、缺陷大小等)，以及与产生该产品缺陷内容相关的制造过程信息，例如制造当前产品的时间、环境信息(例如包括温度、湿度、压力信息等)、操作人员信息、设备参数信息、材料信息、配置信息等，产品制造辅助设备104对上述信息(产品缺陷内容，以及与产生所述产品缺陷内容相关的制造过程信息)进行清洗(去除无效和存在问题的数据)、整合(整合成需要的标准数据格式)、数据挖掘与分析(挖掘和分析例如包括对整合的数据进行分类、聚类、特征提取、降维以及相关性分析等无监督模式的数据处理)后，得到缺陷原因分析模型。产品制造辅助设备104接收产品制造过程中产生的数据，可基于上述缺陷原因分析模型，得到产生当前缺陷原因，例如半导体生成过程中的清洗环节力度不够、腐蚀不够、腐蚀过度、原料匹配不精确、清洗环境微尘过多、曝光强度不够、曝光强度过高、掺杂异物等等。产品制造消息处理设备103可以根据缺陷原因给出对应的工艺和/或设备调整建议。
图7是示出根据本公开至少一实施例的产品制造消息处理方法200的另一示意图。
参考图7，产品制造消息处理方法200还可以包括操作240中的全部或部分操作。在操作240中，产品制造消息处理设备103可以更新产品制造辅助设备104的人工智能缺陷识别模型。
可选地，操作240包括操作241至操作243。
在操作241中，产品制造消息处理设备103分发对第一人工智能缺陷识 别模型进行离线测试的产品缺陷分析任务以获取第一产品缺陷分析结果。假设此时第四产品制造辅助设备104-4中正在使用第二人工智能缺陷识别模型对该类型的产品缺陷进行分析。产品制造消息处理设备103可以使用第二人工智能缺陷识别模型推理产生的数据作为测试集对第一人工智能缺陷识别模型进行离线测试，以比较第一人工智能缺陷识别模型和第二人工智能缺陷识别模型的优劣。可选地，工厂中的检测人员还可以对第二人工智能缺陷识别模型推理产生的数据进行复核和标记，产品制造消息处理设备103再将复核和标记后的数据作为测试集进行测试。
如图7所示，产品制造消息处理设备103对新训练好的模型发起离线测试，产品制造消息处理设备103通过向第五产品制造辅助设备104-5发送第四产品缺陷分析请求消息来分发该产品缺陷分析任务，该第四产品缺陷分析请求消息包含第一产品缺陷模型的存储路径和离线测试所需的测试集的存储路径。第五产品制造辅助设备104-5在解析了该产品缺陷分析请求消息后，可以调度产品缺陷模型测试算法来进行第一人工智能缺陷识别模型的测试。第一产品缺陷模型测试完成后，产品制造消息处理设备103将获取第一产品缺陷分析结果，其指示了第一产品缺陷模型的分析产品缺陷的准确率、分析速度等。如果第一产品缺陷分析结果满足了预设标准(例如，第一产品缺陷分析结果的准确率大于第二人工智能缺陷识别模型的准确率)，可以通过检测人员来确认进入下一测试流程，或者可以直接进入操作242。本领域技术人员应当理解，上述的预设标准还可以包括第一产品缺陷分析结果的召回率、F1-Score等指标中的任意一项指标由于第二人工智能缺陷识别模型针对相同的数据集产生的分析结果，本公开不对预设标准做出限定，只要能衡量两个人工智能缺陷识别模型的优劣即可。
在操作242中，产品制造消息处理设备103可以分发对第一人工智能缺陷识别模型进行在线测试的产品缺陷分析任务以获取第二产品缺陷分析结果。
第一人工智能缺陷识别模型的在线测试也可以称为第一人工智能缺陷识别模型的灰度部署过程。产品制造消息处理设备103可以使用工厂流水线上产生的实时数据来对第一人工智能缺陷识别模型进行在线测试。第一人工智能缺陷识别模型与第二人工智能缺陷识别模型都使用相同的实时数据，并分别推理产生不同的产品缺陷推理结果。此时，第一人工智能缺陷识别模型推理产生的产品缺陷推理结果即是上述的第二产品缺陷分析结果。产品制造消息 处理设备103可以进一步分析第二产品缺陷分析结果是否满足预设标准(例如，是否优于第二人工智能缺陷识别模型对相同的数据推理产生的产品缺陷推理结果)。本领域技术人员应当理解，上述的预设标准还可以包括第一产品缺陷分析结果的召回率、F1-Score等指标中的任意一项指标由于第二人工智能缺陷识别模型针对实时数据产生的分析结果，本公开不对预设标准做出限定，只要能衡量两个人工智能缺陷识别模型的优劣即可。
如图7所示，如果第一人工智能缺陷识别模型通过离线测试后，产品制造消息处理设备103在读取到生产线实时产生数据后，将该数据复制成两份，分别分发给利用第二人工智能缺陷识别模型执行产品缺陷识别推理任务的第四产品制造辅助设备104-4(例如，向第四产品制造辅助设备104-4发送第五产品缺陷请求消息)、和执行第一人工智能缺陷识别模型在线测试的第五产品制造辅助设备104-5(例如，向第五产品制造辅助设备104-5发送第六产品缺陷请求消息)。第四产品制造辅助设备104-4和第五产品制造辅助设备104-5可以是上述的GPU集群，也可以是能够执行人工智能缺陷识别模型的分析或测试的其它设备。第四产品制造辅助设备104-4和第五产品制造辅助设备104-5可以为相同的设备也可以是不同的设备，本公开对此不作限制。
第四产品制造辅助设备104-4和第五产品制造辅助设备104-5接收到产品制造消息处理设备103发送的用于在线测试的产品缺陷分析请求消息后，第五产品制造辅助设备104-5可以调度人工智能在线测试算法来对第一产品缺陷模型进行在线测试。第一人工智能缺陷识别模型的在线测试与第二人工智能缺陷识别模型的推理是同步进行的，并且二者的数据来源一致。同样的，产品制造消息处理设备103将第一人工智能缺陷识别模型的测试任务以产品缺陷请求消息的形式发送给第五产品制造辅助设备104-5，第五产品制造辅助设备104-5可以从该产品缺陷请求消息中获取第一人工智能缺陷识别模型在线测试所需的数据集存储地址和第一人工智能缺陷识别模型的存储地址。然后，第五产品制造辅助设备104-5可以调度人工智能算法来执行第一人工智能缺陷识别模型的在线测试任务。第五产品制造辅助设备104-5可以通过第六产品缺陷分析响应消息来向产品制造消息处理设备103返回第二产品缺陷分析结果。
在第一产品缺陷分析结果和第二产品缺陷分析结果满足预设标准的情况下，在操作243中，产品制造消息处理设备103向第四产品制造辅助设备104- 4发送模型替换请求(例如图示的第七产品缺陷分析请求消息)，以将第四产品制造辅助设备104-4中的第二人工智能缺陷识别模型替换为第一人工智能缺陷识别模型。
经过以上三个操作，由产品制造辅助设备104训练好的第一人工智能缺陷识别模型可以通过方便的离线测试、高效可视化的灰度部署(在线测试)、更新发布的全过程。
该灰度部署与上线过程不影响生产线旧模型(第二人工智能缺陷识别模型)的实时推理服务，因此对生产线的正常生产没有影响，不会造成传统的手动测试部署新模型(第一人工智能缺陷识别模型)造成的系统停机等带来的巨大损失。并且，灰度部署过程中使用的测试数据都是产品制造消息处理设备103中检测人员复判后的结果和旧模型产生的实时推理数据，没有用到其他的测试数据，这样可以节约大量的测试数据收集成本。
由此，本公开的至少一实施例合理利用了整个自动不良分类系统产生的数据，实现了数据的循环利用并且减少了训练和测试数据收集的成本。产品制造消息处理设备103在不影响生产线生产的前提下，完成新模型的离线测试和灰度部署功能，避免了人工智能缺陷识别模型由于需要停机测试更新而造成的损失。根据本公开的实施例，上文描述的过程也可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括有形地包含在机器可读介质上的计算机程序，所述计算机程序包含用于执行上述过程的方法的程序代码。
图8示出了根据本公开至少一实施例的产品制造消息处理设备800的示意图。
产品制造消息处理设备800可以包括：监听单元801，其被配置为监听多个产品制造消息；调度单元802，被配置为根据多个产品制造消息建立产品缺陷分析任务队列；分发单元803，被配置为根据产品缺陷分析任务队列向产品制造辅助设备分发产品缺陷分析任务。
图9示出了根据本公开至少一实施例的电子设备900的结构图。
参考图9，根据本公开至少一实施例的电子设备900可以包括处理器901和存储器902。处理器901和存储器902都可以通过总线903相连。虚拟资源转移设备900可以是塔式服务器、机架服务器(Rack)、刀片服务器(Blade Server)、机柜式服务器等。
处理器901可以根据存储在存储器902中的程序执行各种动作和处理。具体地，处理器901可以是一种集成电路芯片，具有信号的处理能力。上述处理器可以是通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现成可编程门阵列(FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等，可以是X86架构或ARM架构的。
存储器902存储有计算机指令，在计算机指令被处理器901执行时实现上述方法200。存储器1402可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。非易失性存储器可以是只读存储器(ROM)、可编程只读存储器(PROM)、可擦除可编程只读存储器(EPROM)、电可擦除可编程只读存储器(EEPROM)或闪存。易失性存储器可以是随机存取存储器(RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、同步动态随机存取存储器(SDRAM)、双倍数据速率同步动态随机存取存储器DDRSDRAM)、增强型同步动态随机存取存储器(ESDRAM)、同步连接动态随机存取存储器(SLDRAM)和直接内存总线随机存取存储器(DRRAM)。应注意，本文描述的方法的存储器旨在包括但不限于这些和任意其它适合类型的存储器。
本公开至少一实施例的产品制造消息处理方法、设备和电子设备，可以提高整个产品制造过程中的产品制造消息的处理效率，使得参与产品缺陷检测分析的各个设备均高效率地运行，便于后续缺陷原因查找和分析，提高了产品制造的效率。
需要说明的是，附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，所述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图 中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
一般而言，本公开的各种示例实施例可以在硬件或专用电路、软件、固件、逻辑，或其任何组合中实施。某些方面可以在硬件中实施，而其他方面可以在可以由控制器、微处理器或其他计算设备执行的固件或软件中实施。当本公开的实施例的各方面被图示或描述为框图、流程图或使用某些其他图形表示时，将理解此处描述的方框、装置、系统、技术或方法可以作为非限制性的示例在硬件、软件、固件、专用电路或逻辑、通用硬件或控制器或其他计算设备，或其某些组合中实施。
本公开至少一实施例中的处理器可以是一种集成电路芯片，具有信号的处理能力。上述处理器可以是通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现成可编程门阵列(FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等，可以是X86架构或ARM架构的。
本公开至少一实施例中的计算机可读存储介质可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。非易失性存储器可以是只读存储器(ROM)、可编程只读存储器(PROM)、可擦除可编程只读存储器(EPROM)、电可擦除可编程只读存储器(EEPROM)或闪存。易失性存储器可以是随机存取存储器(RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、同步动态随机存取存储器(SDRAM)、双倍数据速率同步动态随机存取存储器DDRSDRAM)、增强型同步动态随机存取存储器(ESDRAM)、同步连接动态随机存取存储器(SLDRAM)和直接内存总线随机存取存储器(DR RAM)。应注意，本文描述的系统和方法的存储器旨在包括但不限于这些和任意其它适合类型的存储器。
需要说明的是，附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可 以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
一般而言，本公开的各种示例实施例可以在硬件或专用电路、软件、固件、逻辑，或其任何组合中实施。某些方面可以在硬件中实施，而其他方面可以在可以由控制器、微处理器或其他计算设备执行的固件或软件中实施。当本公开的实施例的各方面被图示或描述为框图、流程图或使用某些其他图形表示时，将理解此处描述的方框、装置、系统、技术或方法可以作为非限制性的示例在硬件、软件、固件、专用电路或逻辑、通用硬件或控制器或其他计算设备，或其某些组合中实施。
在上面详细描述的本发明的示例实施例仅仅是说明性的，而不是限制性的。本领域技术人员应该理解，在不脱离本发明的原理和精神的情况下，可对这些实施例或其特征进行各种修改和组合，这样的修改应落入本发明的范围内。

Claims (18)

  1. 一种产品制造消息处理方法，包括：
    监听多个产品制造消息；
    根据所述多个产品制造消息建立产品缺陷分析任务队列；
    根据产品缺陷分析任务队列向产品制造辅助设备分发产品缺陷分析任务，其中，所述产品缺陷分析任务包括基于缺陷识别模型的产品缺陷内容识别任务，
    其中所述产品缺陷内容包括产品的缺陷类型、缺陷位置和缺陷大小中的任意一项或多项。
  2. 如权利要求1所述的产品制造消息处理方法，其中，所述多个产品制造消息包括至少一个单一产品制造消息和至少一个批量产品制造消息，所述监听多个产品制造消息还包括：
    以中断的方式监听批量产品制造消息；以及
    以轮询的方式监听单一产品制造消息。
  3. 如权利要求1所述的产品制造消息处理方法，还包括：
    向产品制造消息服务设备发送注册信息，其中所述注册信息包括产品制造站点信息和/或第一产品信息；
    基于所述注册信息，监听从所述产品制造消息服务设备发送的第一产品制造消息，所述第一产品制造消息与所述产品制造站点信息和/或第一产品信息相关联。
  4. 如权利要求3所述的产品制造消息处理方法，还包括：
    监听从所述产品制造消息服务设备发送的第二产品制造消息，其中，所述第二产品制造消息与所述注册信息无关；
    确定产品制造关键字列表是否包括所述产品制造关键字；
    在产品制造关键字列表中包括所述产品制造关键字的情况下，保留所述第二产品制造信息；以及
    在所述产品制造关键字列表中不包括所述产品制造关键字的情况下，丢弃所述第二产品制造信息。
  5. 如权利要求1所述的产品制造消息处理方法，其中，所述根据所述多个产品制造消息建立产品缺陷分析任务队列还包括：基于接收产品制造消息的顺序、产品的优先级和产品调度计划中的任意一项或多项对产品缺陷分析任务进行排序以构建产品缺陷分析任务队列。
  6. 如权利要求1所述的产品制造消息处理方法，其中，所述缺陷识别模型包括：前馈神经网络模型、卷积神经网络模型、循环神经网络模型、生成式对抗网络模型中的一项或多项。
  7. 如权利要求1所述的产品制造消息处理方法，还包括：
    基于所述多个产品制造消息，获取多个产品图像；
    从所述多个产品图像中获取产品缺陷；
    其中，从所述多个产品图像中获取产品缺陷包括以下方法的任意一项或多项：
    在产品缺陷的数量分布不均匀的情况下，对所述产品图像进行旋转、缩放、色彩变换和截取中的任一项或多项处理；
    在产品缺陷所在的图像区域小于第一预定阈值的情况下，放大产品缺陷所在的图像区域；
    在任意两个产品缺陷的相似度大于第二预定阈值的情况下，合并所述任意两个产品缺陷的产品图像；以及
    在所述产品图像中的正样本和负样本之间的相似度大于第三预定阈值的情况下，获取所述产品图像的频域图像，并基于所述频域图像调整所述正样本或负样本；
    基于产品缺陷生成产品缺陷分析任务。
  8. 如权利要求1所述的产品制造消息处理方法，其中，所述向产品制造辅助设备分发产品缺陷分析任务还包括：
    从所述多个产品制造消息中获取产品类型和产品缺陷分析任务类型；
    基于所述产品类型和产品缺陷分析任务类型，生成产品缺陷分析请求消息。
  9. 如权利要求8所述的产品制造消息处理方法，其中，所述产品制造辅助设备包括第一产品制造辅助设备和第二产品制造辅助设备，
    其中，所述根据产品缺陷分析任务队列向产品制造辅助设备分发产品缺陷分析任务还包括：
    确定是否存在与产品类型相对应的人工智能缺陷识别模型；
    在不存在与所述产品类型相对应的人工智能缺陷识别模型的情况下，向所述第一产品制造辅助设备发送第一产品缺陷分析请求消息，所述第一产品缺陷分析请求消息包括所述产品类型、产品图像的存储地址、产品图像的数量和对人工智能缺陷识别模型进行训练的任务标识；
    在存在与所述产品类型相对应的人工智能缺陷识别模型的情况下，向所述第二产品制造辅助设备发送第二产品缺陷分析请求消息，所述第二产品缺陷分析请求消息包括产品类型、产品图像的存储地址和产品图像的数量。
  10. 如权利要求9所述的产品制造消息处理方法，还包括：
    接收由第一产品制造辅助设备发送的第一产品缺陷分析响应消息，所述第一产品缺陷分析响应消息包括以下各项中的一项或多项：人工智能缺陷识别模型的标识、准确率和召回率，
    其中，所述人工智能缺陷识别模型是基于产品类型、产品图像的存储地址和产品图像的数量而确定的。
  11. 如权利要求9所述的产品制造消息处理方法，还包括：
    接收由第二产品制造辅助设备发送的第二产品缺陷分析响应消息，所述第二产品缺陷分析响应消息包括以下各项中的一项或多项：产品图像标识、产品缺陷位置、产品缺陷标识和修复标识，
    其中，所述产品缺陷位置、产品缺陷标识和修复标识是基于产品类型、产品图像的存储地址和产品图像的数量而确定的。
  12. 如权利要求1所述的产品制造消息处理方法，还包括：
    监控所述产品制造辅助设备处理产品缺陷分析任务的准确率、精确率、召回率、F分数和速度中的一项或多项。
  13. 如权利要求1所述的产品制造消息处理方法，还包括：
    获取多个产品缺陷分析任务的分析结果数据；
    基于产品缺陷类型、结果数据的格式和解决产品缺陷的方式中的一项或多项整合所述分析结果数据。
  14. 如权利要求13所述的产品制造消息处理方法，还包括：
    基于所述分析结果数据，发送产品缺陷警报。
  15. 如权利要求1所述的产品制造消息处理方法，还包括：
    更新所述缺陷识别模型。
  16. 如权利要求15所述的产品制造消息处理方法，其中，所述更新所述缺陷识别模型还包括：
    分发对第一缺陷识别模型进行离线测试的产品缺陷分析任务以获取第一产品缺陷分析结果；
    分发对第一缺陷识别模型进行在线测试的产品缺陷分析任务以获取第二产品缺陷分析结果；
    在第一缺陷分析结果和第二缺陷分析结果满足预设标准的情况下，分发将产品制造辅助设备中的第二缺陷识别模型替换为第一缺陷识别模型的产品缺陷分析任务。
  17. 一种电子设备，包括：
    处理器；
    存储器，存储器存储有计算机指令，该计算机指令被处理器执行时实现如权利要求1-16中任一项所述的方法。
  18. 一种计算机可读存储介质，其上存储有计算机指令，所述计算机指令被处理器执行时实现如权利要求1-16中的任一项所述的方法。
PCT/CN2019/127040 2019-12-20 2019-12-20 产品制造消息处理方法、设备和计算机存储介质 WO2021120179A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN201980003143.9A CN113557425B (zh) 2019-12-20 2019-12-20 产品制造消息处理方法、设备和计算机存储介质
US17/044,160 US12020516B2 (en) 2019-12-20 2019-12-20 Method and device for processing product manufacturing messages, electronic device, and computer-readable storage medium
PCT/CN2019/127040 WO2021120179A1 (zh) 2019-12-20 2019-12-20 产品制造消息处理方法、设备和计算机存储介质
US18/666,299 US20240304034A1 (en) 2019-12-20 2024-05-16 Method and device for processing product manufacturing messages, electronic device, and computer-readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/127040 WO2021120179A1 (zh) 2019-12-20 2019-12-20 产品制造消息处理方法、设备和计算机存储介质

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US17/044,160 A-371-Of-International US12020516B2 (en) 2019-12-20 2019-12-20 Method and device for processing product manufacturing messages, electronic device, and computer-readable storage medium
US18/666,299 Continuation US20240304034A1 (en) 2019-12-20 2024-05-16 Method and device for processing product manufacturing messages, electronic device, and computer-readable storage medium

Publications (1)

Publication Number Publication Date
WO2021120179A1 true WO2021120179A1 (zh) 2021-06-24

Family

ID=76478155

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/127040 WO2021120179A1 (zh) 2019-12-20 2019-12-20 产品制造消息处理方法、设备和计算机存储介质

Country Status (3)

Country Link
US (2) US12020516B2 (zh)
CN (1) CN113557425B (zh)
WO (1) WO2021120179A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113610834A (zh) * 2021-08-23 2021-11-05 珠海格力电器股份有限公司 医用面罩的缺陷检测方法、装置、存储介质及电子设备
CN114723682A (zh) * 2022-03-22 2022-07-08 厦门微亚智能科技有限公司 产品缺陷管理方法、系统及计算机可读存储介质

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113632099B (zh) * 2019-12-20 2024-06-28 京东方科技集团股份有限公司 分布式产品缺陷分析系统、方法及计算机可读存储介质
US20220012707A1 (en) * 2020-07-10 2022-01-13 Paypal, Inc. Transaction type categorization for enhanced servicing of peer-to-peer transactions
JP7509659B2 (ja) * 2020-11-02 2024-07-02 株式会社日立製作所 生産情報管理システムおよび生産情報管理方法
CN117577759B (zh) * 2024-01-17 2024-04-09 深圳市极光光电有限公司 一种led封装设备智能控制方法及系统

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108268892A (zh) * 2017-12-29 2018-07-10 英特尔产品（成都）有限公司 生产故障管理分析方法
CN110376220A (zh) * 2019-08-02 2019-10-25 中电九天智能科技有限公司 实现实时判定玻璃面板缺陷的装置及方法
CN110554047A (zh) * 2019-09-06 2019-12-10 腾讯科技（深圳）有限公司 产品缺陷检测数据处理方法、装置、系统和设备

Family Cites Families (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5481707A (en) 1991-05-19 1996-01-02 Unisys Corporation Dedicated processor for task I/O and memory management
JP3111931B2 (ja) * 1997-06-11 2000-11-27 日本電気株式会社 検査結果解析装置及び解析方法並びに解析プログラムを記録した記録媒体
US7552016B2 (en) 2007-05-30 2009-06-23 Allegro Microsystems, Inc. Integrated circuit having a multi-purpose node configured to receive a threshold voltage and to provide a fault signal
US8331650B2 (en) * 2010-07-29 2012-12-11 Sharp Laboratories Of America, Inc. Methods, systems and apparatus for defect detection
FR2987533B1 (fr) * 2012-02-23 2014-11-28 Aspserveur Procede et systeme d'analyse de correlation de defauts pour un centre informatique
CN103458527B (zh) 2012-06-01 2017-02-08 中兴通讯股份有限公司 前导检测任务处理调度方法及装置
US20150324234A1 (en) 2013-11-14 2015-11-12 Mediatek Inc. Task scheduling method and related non-transitory computer readable medium for dispatching task in multi-core processor system based at least partly on distribution of tasks sharing same data and/or accessing same memory address(es)
US20160147883A1 (en) * 2015-12-29 2016-05-26 Carterpillar Inc. Method and system for determining defects per unit of a product
CN107025205B (zh) 2016-01-30 2021-06-22 华为技术有限公司 一种分布式系统中的训练模型的方法及设备
JP6424874B2 (ja) 2016-10-12 2018-11-21 オムロン株式会社 動作状態監視装置、学習データ生成装置、方法およびプログラム
US10402688B2 (en) 2016-12-07 2019-09-03 Kla-Tencor Corporation Data augmentation for convolutional neural network-based defect inspection
JP2018180875A (ja) * 2017-04-12 2018-11-15 富士通株式会社 判定装置、判定方法および判定プログラム
CN107274471B (zh) 2017-06-15 2020-10-09 深圳市彬讯科技有限公司 一种基于实时离线渲染并行的多优先级队列调度系统
US10604814B2 (en) 2017-09-27 2020-03-31 International Business Machines Coporation Manufacturing process control with deep learning-based predictive model for hot metal temperature of blast furnace
US10831519B2 (en) 2017-11-22 2020-11-10 Amazon Technologies, Inc. Packaging and deploying algorithms for flexible machine learning
CN108564104A (zh) * 2018-01-09 2018-09-21 北京百度网讯科技有限公司 产品缺陷检测方法、装置、系统、服务器及存储介质
CN109087280A (zh) * 2018-07-02 2018-12-25 北京百度网讯科技有限公司 连铸坯质量检测方法、装置、控制设备、服务器及介质
CN109800137A (zh) 2018-12-06 2019-05-24 珠海西山居互动娱乐科技有限公司 一种服务器性能测试方法和系统
US11263737B2 (en) * 2019-01-10 2022-03-01 Lam Research Corporation Defect classification and source analysis for semiconductor equipment
JP7147615B2 (ja) 2019-02-15 2022-10-05 株式会社デンソー タスク管理装置
CN110109821A (zh) * 2019-03-19 2019-08-09 深圳壹账通智能科技有限公司 软件程序质量评价方法、装置、计算机设备及存储介质
CN109903287A (zh) * 2019-03-29 2019-06-18 北京百度网讯科技有限公司 质量检测方法及装置
EP4020336A4 (en) * 2019-08-19 2023-06-07 LG Electronics Inc. AI-ASSISTED SYSTEM FOR DETERMINING EXERCISE MODELS AND AI-ASSISTED SYSTEM FOR ADMINISTRATION OF VISUAL INSPECTION THAT USES THIS SYSTEM FOR PRODUCT PRODUCTION LINES
CN110990132B (zh) 2019-11-01 2023-03-10 浙江大搜车软件技术有限公司 异步任务处理方法、装置、计算机设备和存储介质
US11144437B2 (en) * 2019-11-25 2021-10-12 International Business Machines Corporation Pre-populating continuous delivery test cases
EP3859587A1 (en) 2020-01-29 2021-08-04 Tata Consultancy Services Limited Robotic task planning for complex task instructions in natural language
CN111738156A (zh) 2020-06-23 2020-10-02 国网宁夏电力有限公司培训中心 一种高压开关设备状态智能巡检管理方法及系统
US11494417B2 (en) 2020-08-07 2022-11-08 Commvault Systems, Inc. Automated email classification in an information management system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108268892A (zh) * 2017-12-29 2018-07-10 英特尔产品（成都）有限公司 生产故障管理分析方法
CN110376220A (zh) * 2019-08-02 2019-10-25 中电九天智能科技有限公司 实现实时判定玻璃面板缺陷的装置及方法
CN110554047A (zh) * 2019-09-06 2019-12-10 腾讯科技（深圳）有限公司 产品缺陷检测数据处理方法、装置、系统和设备

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
ANONYMOUS: "Integrated Industrial Visual Inspection Scheme in Combination with Training and Inference", WORLD PEACE INDUSTRIAL GROUP, 30 October 2019 (2019-10-30), XP055821314, Retrieved from the Internet <URL:https://wpig-iotsolutionaggregator.wpgholdings.com/uploads/images/Article/EIS/WPI-Intel_EIS-EN.pdf> *
KARTHIKEYAN S.; VIMALA DEVI K.; VALARMATHI K.: "Design and implementation of CfoTS networks for industrial fault detection and correction mechanism", JOURNAL OF SUPERCOMPUTING., KLUWER ACADEMIC PUBLISHERS, DORDRECHT., NL, vol. 76, no. 8, 13 September 2019 (2019-09-13), NL, pages 5763 - 5779, XP037195483, ISSN: 0920-8542, DOI: 10.1007/s11227-019-02993-5 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113610834A (zh) * 2021-08-23 2021-11-05 珠海格力电器股份有限公司 医用面罩的缺陷检测方法、装置、存储介质及电子设备
CN114723682A (zh) * 2022-03-22 2022-07-08 厦门微亚智能科技有限公司 产品缺陷管理方法、系统及计算机可读存储介质

Also Published As

Publication number Publication date
CN113557425A (zh) 2021-10-26
US20230142383A1 (en) 2023-05-11
US20240304034A1 (en) 2024-09-12
CN113557425B (zh) 2024-08-30
US12020516B2 (en) 2024-06-25

Similar Documents

Publication Publication Date Title
WO2021120179A1 (zh) 产品制造消息处理方法、设备和计算机存储介质
WO2021120186A1 (zh) 分布式产品缺陷分析系统、方法及计算机可读存储介质
US20200357109A1 (en) Method for detecting display screen quality, apparatus, electronic device and storage medium
CN109118482B (zh) 一种面板缺陷分析方法、装置及存储介质
JP7442550B2 (ja) 推論演算装置、モデル訓練装置、及び推論演算システム
WO2022148109A1 (zh) 产品缺陷检测方法、装置、设备及计算机可读存储介质
CN111401418A (zh) 一种基于改进Faster r-cnn的员工着装规范检测方法
CN111566789B (zh) 使用电子显微法的半导体计量及缺陷分类
WO2022147966A1 (zh) 晶圆检测方法、装置、设备及存储介质
TW202209153A (zh) 檢查方案的自動最佳化
DE112019005951T5 (de) Zentralisierte Analyse mehrerer visueller Prüfvorrichtungen
CN114495498B (zh) 一种交通数据分布有效性判别方法及装置
CN117252851B (zh) 一种基于图像检测识别的标准质量检测管理平台
CN113269190B (zh) 基于人工智能的数据分类方法、装置、计算机设备及介质
US11928808B2 (en) Wafer detection method, device, apparatus, and storage medium
CN115756922A (zh) 一种故障预测诊断方法、装置、电子设备及存储介质
CN115512098A (zh) 一种桥梁电子化巡检系统及巡检方法
US20230419466A1 (en) Method and apparatus for identifying defect grade of bad picture, and storage medium
CN118154068B (zh) 三角胶物流配送优化方法及系统
CN118608526B (zh) 基于数字孪生的工业互联网检测方法、系统、介质及设备
Devi et al. Improving Train Safety: Computer Vision-Based Error Detection for Precise Upkeep
Hsu et al. An Improved Visual Recognition Model of Interference Noise Image
CN114723682A (zh) 产品缺陷管理方法、系统及计算机可读存储介质
CN116501944A (zh) 基于高质量模板资源库的数据主动抓取系统
CN115965813A (zh) 印版缺陷分级分类模型训练方法及印版缺陷分级分类方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19956337

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19956337

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 19956337

Country of ref document: EP

Kind code of ref document: A1