CN113689040A

CN113689040A - Control method and control system for measurement model and computer readable medium

Info

Publication number: CN113689040A
Application number: CN202110981298.9A
Authority: CN
Inventors: 法提·奥尔梅兹
Original assignee: Yangtze Memory Technologies Co Ltd
Current assignee: Yangtze Memory Technologies Co Ltd
Priority date: 2021-08-25
Filing date: 2021-08-25
Publication date: 2021-11-23
Anticipated expiration: 2041-08-25
Also published as: CN113689040B

Abstract

The disclosure relates to a control method and a control system of a measurement model, wherein the measurement model comprises a training model and a prediction model, the training model is used for updating the prediction model, and the prediction model is used for predicting products manufactured by one or more machines, and the control method comprises the following steps: detecting the application prediction error degree of the prediction model aiming at the products manufactured by one or more machines and the operation abnormal degree of the prediction model; and issuing an alarm and/or at least partially changing an operating state of the predictive model in response to at least one of the following conditions being met: the application prediction error degree is greater than the reference application prediction error degree, and the operational anomaly degree is greater than the reference operational anomaly degree.

Description

Control method and control system for measurement model and computer readable medium

Technical Field

The present application relates to virtual measurement, and more particularly, to a control method and a control system of a virtual measurement model.

Background

In semiconductor manufacturing, inspection of products at various stages of a production line is an important component of product yield management. In the conventional inspection method, a plurality of measurement stations are introduced into a production line to actually measure a product. Such actual measurements may affect the production of the production line and may not be possible at some stage. The virtual measurement is a measurement means for predicting or estimating the quality of a product based on relevant data collected from, for example, a production machine by using a virtual measurement model, and with this technique, the influence of the measurement on the actual production of a production line can be reduced.

In a semiconductor manufacturing process, when a virtual measurement technique is used to perform on-line or off-line product quality prediction or real-time batch control, small changes in the product manufacturing process, the material recipe, the machine operating condition, the connection condition between the prediction model and the database, and the like may have a large impact on the actual application of the prediction model. In addition, in practical applications, situations such as regular maintenance of the production equipment, sudden failure of the production equipment, and abnormal material state may occur, which also affect the prediction accuracy of the virtual model. Therefore, the prediction model of the virtual measurement needs to be monitored by a computer or directly by an engineer to timely find and eliminate or reduce such adverse effects.

On the other hand, problems of low prediction capability, old training set, communication failure with the database and the like may also occur in the training process of the prediction model. These problems can lead to poor training of the prediction model, which in turn affects the final prediction accuracy of the prediction model. Therefore, the training process of the predictive model also needs to be monitored, and the above problems are discovered and eliminated in time.

Therefore, a method and a system for monitoring the training and prediction of the virtual measurement model and taking corresponding measures when the virtual measurement model is abnormal are needed.

It should be appreciated that this background section is intended in part to provide a useful background for understanding the present technology and does not imply that such matter has necessarily been the relevant art known to those skilled in the art prior to the present application.

Disclosure of Invention

The present disclosure provides a control method of a measurement model, wherein the measurement model includes a training model and a prediction model, wherein the training model is used for updating the prediction model, and the prediction model is used for predicting products manufactured by one or more machines, and the method includes: detecting the application prediction error degree of the prediction model aiming at the products manufactured by one or more machines and the operation abnormal degree of the prediction model; and issuing an alarm and/or at least partially changing an operating state of the predictive model in response to at least one of the following conditions being met: the application prediction error degree is greater than the reference application prediction error degree, and the operational anomaly degree is greater than the reference operational anomaly degree.

The present disclosure also provides a method for controlling a measurement model, where the measurement model includes a training model and a prediction model, where the training model is used to update the prediction model, and the prediction model is used to predict a product manufactured by one or more machines, and the method includes: detecting the training prediction accuracy of a training model aiming at a training set and the association abnormality degree of a sample and a machine for manufacturing the sample in the training set, wherein the training set comprises products manufactured by one or more machines; and issuing an alarm and/or at least partially changing an operating state of the predictive model in response to at least one of the following conditions being met: the training prediction accuracy is less than the reference training prediction accuracy and the association anomaly is greater than the reference association anomaly.

The present disclosure also provides a control system of a measurement model, the control system including: a processor; and memory having one or more programs stored thereon that, when executed by the processor, cause the processor to perform the method of: detecting the application prediction error degree of the prediction model aiming at the products manufactured by one or more machines and the operation abnormal degree of the prediction model; and issuing an alarm and/or at least partially changing an operating state of the predictive model in response to at least one of the following conditions being met: the application prediction error degree is greater than the reference application prediction error degree, and the operational anomaly degree is greater than the reference operational anomaly degree.

The present disclosure also provides a control system of a measurement model, the control system including: a processor; and memory having one or more programs stored thereon that, when executed by the processor, cause the processor to perform the method of: detecting the training prediction accuracy of a training model aiming at a training set and the association abnormality degree of a sample and a machine for manufacturing the sample in the training set, wherein the training set comprises products manufactured by one or more machines; and issuing an alarm and/or at least partially changing an operating state of the predictive model in response to at least one of the following conditions being met: the training prediction accuracy is less than the reference training prediction accuracy and the association anomaly is greater than the reference association anomaly.

The present disclosure also provides a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of: detecting the application prediction error degree of the prediction model aiming at the products manufactured by one or more machines and the operation abnormal degree of the prediction model; and issuing an alarm and/or at least partially changing an operating state of the predictive model in response to at least one of the following conditions being met: the application prediction error degree is greater than the reference application prediction error degree, and the operational anomaly degree is greater than the reference operational anomaly degree.

The present disclosure also provides a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of: detecting the training prediction accuracy of a training model aiming at a training set and the association abnormality degree of a sample and a machine for manufacturing the sample in the training set, wherein the training set comprises products manufactured by one or more machines; and issuing an alarm and/or at least partially changing an operating state of the predictive model in response to at least one of the following conditions being met: the training prediction accuracy is less than the reference training prediction accuracy and the association anomaly is greater than the reference association anomaly.

The present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the method of: detecting the application prediction error degree of the prediction model aiming at the products manufactured by one or more machines and the operation abnormal degree of the prediction model; and issuing an alarm and/or at least partially changing an operating state of the predictive model in response to at least one of the following conditions being met: the application prediction error degree is greater than the reference application prediction error degree, and the operational anomaly degree is greater than the reference operational anomaly degree.

The present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the method of: detecting the training prediction accuracy of a training model aiming at a training set and the association abnormality degree of a sample and a machine for manufacturing the sample in the training set, wherein the training set comprises products manufactured by one or more machines; and issuing an alarm and/or at least partially changing an operating state of the predictive model in response to at least one of the following conditions being met: the training prediction accuracy is less than the reference training prediction accuracy and the association anomaly is greater than the reference association anomaly.

Drawings

The above and other advantages and features of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.

Fig. 1 is a schematic diagram illustrating an application environment of a control method and a control system of a measurement model according to an embodiment of the present application.

Fig. 2 shows a control method of a measurement model according to an embodiment of the present application.

Fig. 3 illustrates a control method of a measurement model according to another embodiment of the present application.

FIG. 4 shows a diagram illustrating real-time prediction of R according to an embodiment of the present application²Schematic diagram of the score calculation method.

Detailed Description

Exemplary embodiments of the present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which preferred embodiments of the invention are shown. This invention may, however, be embodied in different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

It will also be understood that when an element or layer is referred to as being "on," "connected to" or "coupled to" another element or layer, it can be directly on or connected to the other element or layer or intervening elements or layers may be present. When an element or layer is referred to as being "directly on," "directly connected to" or "directly coupled to" another element or layer, there are no intervening elements or layers present. To this end, the term "connected" may refer to physical, electrical, and/or fluid connections, with or without intervening elements.

Like reference numerals refer to like elements throughout the specification. In the drawings, the thickness of layers and regions are exaggerated for clarity.

Although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms may be used to distinguish one element from another. Thus, a first element discussed below could be termed a second element without departing from the teachings of one or more embodiments. The description of an element as a "first" element may not require or imply the presence of a second element or other elements. The terms "first," "second," and the like may also be used herein to distinguish between different classes or groups of elements. For the sake of simplicity, the terms "first", "second", etc. may denote "first class (or first group)", "second class (or second group)", etc. respectively.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, regions, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, regions, steps, operations, elements, components, and/or groups thereof.

Furthermore, relative terms, such as "lower" or "bottom" and "upper" or "top," may be used herein to describe one element's relationship to another element as illustrated in the figures. It will be understood that relative terms are intended to encompass different orientations of the device in addition to the orientation depicted in the figures. In an exemplary embodiment, when the device in one of the figures is turned over, elements described as being on the "lower" side of other elements would then be oriented on "upper" sides of the other elements. Thus, the exemplary term "lower" can encompass both an orientation of "lower" and "upper," depending on the particular orientation of the figure. Similarly, when the device in one of the figures is turned over, elements described as "below" or "beneath" other elements would then be oriented "above" the other elements. Thus, the exemplary terms "below" or "beneath" can encompass both an orientation of above and below.

As used herein, "about" or "approximately" includes the stated value as well as the average value within an acceptable range of deviation of the specified value as determined by one of ordinary skill in the art in view of the measurement in question and the error associated with the measurement of the specified quantity (i.e., the limitations of the measurement system). For example, "about" can mean within one or more standard deviations.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present invention and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Some example embodiments are described and illustrated in the accompanying drawings with respect to functional blocks, units and/or modules as is conventional in the art. Those skilled in the art will appreciate that the blocks, units, and/or modules are physically implemented with electrical (or optical) circuitry, such as logic, discrete components, microprocessors, hardwired circuitry, memory elements, wiring connectors, and so forth, which may be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. Where the blocks, units, and/or modules are implemented by a microprocessor or other similar hardware, they may be programmed and controlled by software (e.g., microcode) to perform the various functions discussed herein, and may optionally be driven by firmware and/or software. It is also contemplated that each block, unit, and/or module may be implemented by dedicated hardware for performing some functions or as a combination of dedicated hardware for performing some functions and a processor (e.g., one or more programmed microprocessors and associated circuitry) for performing other functions. In addition, each block, unit and/or module in some example embodiments may be physically separated into two or more interactive and discrete blocks, units and/or modules without departing from the scope of the inventive concept. Furthermore, the blocks, units and/or modules in some example embodiments may be physically combined into more complex blocks, units and/or modules without departing from the scope of the inventive concept.

As shown in fig. 1, an application environment of a control method and a control system of a metrology model (i.e., a virtual metrology model) according to an embodiment of the present disclosure may include a machine, a controller 120, a metrology model 130, a metrology model control unit 133, and a database 140, wherein the machine may include one or more machines, for example, a first machine 111 to an nth machine 112(N is a natural number), and the metrology model 130 may include a training model 131 and a prediction model 132. The measurement model 130, the measurement model control unit 133, and the database 140 may be implemented by instructions stored in a computer readable medium, for example, that when executed by a computer processor may implement the functionality of the measurement model 130, the measurement model control unit 133, and the database 140.

The first through nth tools 111 through 112 may be used to produce products, such as semiconductor devices, wafers, etc. The first through nth stations 111 through 112 may communicate with the database 140 to send data related to the stations (e.g., temperature, production speed, etc.) and data related to the product (e.g., such as product photographs, temperature, etc.) to the database 140.

The controller 120 may control the operations of the first through nth machines 111 through 112, for example, the operations of one or more of the first through nth machines 111 through 112 may be stopped. The controller 120 may communicate with the predictive model 132 to request the predictive model 132 to make a prediction of, for example, a product manufactured by the first tool 111, or to stop or resume operation of, for example, the first tool 111 based on the results of the prediction by the predictive model 132. The controller 120 may include a plurality of controllers, wherein each controller may correspond to one machine. Although the controller is shown in fig. 1 as being separate from the machine, the controller of the present application is not limited thereto, and may be integrated in the machine.

The training model 131 is a predictive model for training. The training model 131 may be used to update the predictive model 132 after being trained, e.g., the predictive model 132 may be replaced with the trained training model 131. The present application is not limited to the training method of the training model 131, for example, when the training model 131 and the prediction model 132 are constructed based on the convolutional neural network, a back propagation method may be used, the prediction result of the training model 131 is compared with the reference labels of the samples in the training set to obtain the difference between the two, the parameters of the convolutional neural network are adjusted according to the difference, and then the next prediction and comparison are performed until the difference between the prediction result and the reference labels is no longer reduced. During the training process, the training models 131 may communicate with the database 140 to obtain training set data.

The prediction model 132 is used for predicting the product produced by the corresponding machine in response to the prediction request of the controller 120 to determine whether the product is qualified or not, whether there is a defect or not, for example. The predictive models 132 may communicate with the database 140 to obtain data related to the respective tools and/or products to make the predictions. For example, the predictive model 132 may obtain various parameters of the tool (e.g., temperature, production speed, etc.) from the database 140, and may also obtain data related to the product (e.g., such as product photographs, temperature, etc.).

The measurement model control unit 133 is configured to execute a measurement model control method and may communicate with the training model 131 and the prediction model 132 to control the training model 131 and the prediction model 132 according to the training prediction accuracy of the training model on the training set, the degree of abnormality of the training process, the degree of abnormality of the association between the samples in the training set and the machine station for manufacturing the samples, the application prediction error degree of the prediction model on the products manufactured by the machine station, the operation abnormality degree of the prediction model, and the like, wherein the training prediction accuracy reflects the prediction accuracy of the training model on the samples in the training set, and since the training model is used to update the prediction model, the actual prediction accuracy of the prediction model is directly affected by too low training prediction accuracy; the degree of abnormality in the training process reflects the severity of abnormality occurring in the training process of the training model, for example, the time for completing one training, the time interval between two successful trainings, and the like, and the fact that the connection between the training model or the measurement model control unit and the database may have problems in the training process or the data itself may have problems due to the overlarge indexes; the association abnormality degree of the samples in the training set and the machine station for manufacturing the samples reflects the degree that the samples can represent products manufactured by the machine station, if the association abnormality degree is too low, the overall situation that the used samples cannot well represent the products manufactured by the machine station is reflected, and then the model trained by using the samples cannot accurately predict the products manufactured by the machine station; the application of the prediction error degree reflects the prediction accuracy of the prediction model for the actual product, and the application of the prediction error degree excessively reflects that the prediction model cannot make accurate prediction for the product; the degree of running abnormality reflects the severity of abnormality occurring in the prediction process, and the degree of running abnormality reflects that the server load is possibly too high, the data used for prediction is in a problem, the coverage rate of prediction for each machine is insufficient, and the like. The measurement model control unit 133 may also determine whether to issue an alarm, stop training, stop machine operation, or the like, based on the above information. The measurement model control unit 133 may communicate with the database 140, for example, to read history data about the training model 131 and the prediction model 132, or to store control information and the like to the database 140.

The database 140 is used to store data, for example, data about the tool and the product from the tool, a training set of prediction models, control information of measurement model control units, logs, and the like may be stored.

The control method of the measurement model will be described in detail below with reference to fig. 2.

As shown in fig. 2, a control method of a measurement model according to an embodiment of the present application may include: detecting the training prediction accuracy of the training model for the training set (S211) and the associated abnormality degree of the sample in the training set and the machine station for manufacturing the sample (S221); and issuing an alarm and/or at least partially changing an operating state of the predictive model in response to at least one of the following conditions being met (S250): the training prediction accuracy is lower than the reference training prediction accuracy (S212) and the associated degree of abnormality is greater than the reference associated degree of abnormality (S222).

At step S211, the control method of the measurement model according to the embodiment of the present application may detect the training prediction accuracy of the training model for the training set. Training prediction accuracy may use R²Is expressed by a fraction, wherein R²The score may be obtained using equation 1 below:

wherein the content of the first and second substances,

predicted value, y, for the training model for the ith sample in the training set_iIs the reference value of the ith sample, ∈_iRepresenting the difference between the predicted value and the reference value for the ith sample. R²The score reflects the fitting ability of the training model to the sample data, and the larger the value of the score is, the stronger the fitting ability is, and the higher the corresponding prediction accuracy is. By aiming at R²The score is set to a reference value (e.g., reference R)²Score) as a reference trainingAnd the prediction accuracy can be correspondingly operated when the training prediction accuracy of the training model is lower than the reference training prediction accuracy. For example, when training R of a model²Fraction less than reference R²When the score is given, the fitting ability of the training model is low, the prediction accuracy can be affected, and the user can be immediately informed of the abnormity and/or the prediction of all machines is suspended. For example, alarm information is displayed to the user through a display device and/or an instruction is issued to the prediction model 132 to suspend prediction for all machines through the measurement model control unit 133 shown in fig. 1.

In this embodiment, R of the model is trained²The score may include a simple R²Fraction (also called "first R²Fraction "), median R ″²Fraction (may also be referred to as "second R²Score ") and real-time prediction R²Fraction (also called "third R²Score ").

Simple R²The score may be R calculated for all samples in the training set²And (4) scoring.

Median value R²The score can be obtained by the following method: randomly dividing a training set into a first training set and a second training set; training the training model with a first training set; predicting the second training set using the trained training model; obtaining R of the trained training model for the second training set²A score; randomly dividing the training set into a new first training set and a new second training set again; repeating the above steps a predetermined number of times, and obtaining R²The median score of the scores is taken as the second R²And (4) scoring.

For example, 80% of samples in the training sample set may be randomly divided into a training sample set, the remaining 20% of samples may be divided into a test sample set, the training model may be trained using the training sample set, the samples in the test sample set may be predicted using the trained training model, and R of the trained training model may be obtained²And (4) scoring. Then, 80% of the samples in the training set are divided into a new training sample set again, and the rest 20% of the samples are divided intoAnd replacing the original training sample set and the original testing sample set with the new training sample set and the new testing sample set respectively for the new testing sample set. The above process may be repeated, for example, 1000 times and 1000 Rs are obtained²Fraction, and finally dividing the 1000R²The median of the scores is taken as the median R of the training model²Score of

Real-time prediction of R²The score can be obtained by the following method: sequencing the products in the training set in the order of the production completion time from first to last; selecting a predetermined number as a selection window; selecting a third training set using the selection window starting from the first sample in the training set; training the training model by using a third training set; predicting the sample which is positioned one bit behind the selection window in the training set by using the trained training model; moving the selection window backward by one sample step in the training set; updating the third training set by using the samples in the moved selection window; repeating the above steps until the last sample in the training set is predicted; and calculating R for all of the predictors²Score as real-time prediction R²And (4) scoring.

FIG. 3 shows a diagram illustrating real-time prediction of R according to an embodiment of the present application²Schematic diagram of the score calculation method.

In this embodiment, the samples in the training set may be products produced by the first machine 111 to the nth machine 112 shown in fig. 1 and labeled with the detection results. As shown in fig. 3, the samples in the training set may be arranged in the order of generation completion time from first to last, for example, the generation completion time of the sample 301 is earlier than the generation completion time of the sample 305, and the generation completion time of the sample 305 is earlier than the generation completion time of the sample 306. For example four samples may be selected as a selection window and a third training set 310 is selected using the selection window starting with the first sample 301 in the training set. The trained training model is then used to predict the samples 305 in the training set that are one bit after the selection window. After the prediction is completed, the selection window may be shifted backward by one sample step to select four samples after the sample 301, and a third training may be performed using the four samplesThe training set 310 is updated to a third training set 320, and then training a training model using the third training set 320, and the sample 306 is predicted using the trained training model. By analogy, the training process is ended when the last sample is predicted. Finally, R is calculated for all predicted results in the above process²Score as real-time prediction R²And (4) scoring.

In this embodiment, simple R²The fraction may be greater than the median R²Fraction, and median R²The score may be greater than the real-time prediction R²And (4) scoring.

In this embodiment, simple R's can be respectively assigned²Fractional, median R²Fractional and real-time prediction of R²The score is used as the training prediction accuracy of the training model and is respectively compared with the first R²Fraction, second R²Fraction and third R²Fraction-corresponding R²The fractional reference value is used as a reference to train the prediction accuracy. The measurement model control method may include: when simple R²Fractional, median R²Fractional and real-time prediction of R²When at least one of the scores is below the corresponding reference training prediction accuracy (i.e., S212, the training prediction accuracy is less than the reference training prediction accuracy), an alarm may be raised and/or training of the training models for all of the machines may be suspended (S250). The reference training prediction accuracy may be determined experimentally or based on historical data.

In some embodiments, the control method of the measurement model may further include: and detecting the training process abnormality degree of the training model aiming at the training set. The training process anomaly degree reflects the severity of the anomaly occurring during the training of the training model.

In the training process, the interval time between two successful trainings of the training model is increased due to abnormal data flow between the training model and the database or the occurrence of problems of the training data, the monitoring of the interval time between the two successful trainings can find the problems in time, and corresponding measures are taken. Thus, in some embodiments, a method of detecting training process outliers of a training model to a training set may comprise: detecting an interval time between successive successful trainings of the training model; and representing the training process abnormality degree by using the interval time, and using the corresponding interval time reference value as a reference training process abnormality degree. The measurement model control method may include: and issuing an alarm and/or suspending training of training models for all machines if the interval time is greater than the interval time reference value and lasts for more than a preset time.

For example, the timing may be started when the training model successfully completes one training, and stopped until the training is successfully completed next time, the timed time is the interval time, and when the interval time is greater than the interval time reference value and continues for more than a predetermined time, it may be determined that the training of the training model is abnormal, at which time an alarm may be issued and/or the training of the training models for all the machines may be suspended. The interval time reference value may be determined experimentally or based on historical data.

Problems with data connections between the training model and the database may cause the training time of the training model to be too long, for example, too long latency of the connections between the training model and the database may cause the training time to be too long, and monitoring of the training time may enable such problems to be discovered in time and appropriate measures to be taken. Thus, in some embodiments, a method of detecting training process anomalies of a training model may comprise: detecting the training time for finishing the training of the training model each time; and representing the training process abnormality degree by using the training time, and using the corresponding training time reference value as the reference training process abnormality degree. The measurement model control method may include: and issuing an alarm and/or suspending training of training models for all machines if the training time is greater than the training time reference value and lasts for more than a preset time.

For example, the timing may be started each time training of the training model is started, and stopped when the training is completed, and the timed time is the training time. When there is a training time exceeding the training time reference value in the obtained training time and lasting for more than a predetermined time, it may be determined that the training of the training model is abnormal, at which time an alarm may be issued and/or the training of the training model for all the machines may be suspended. The training time reference value may be determined experimentally or based on historical data

Referring again to fig. 2, in step S221, the associated abnormality degree of the sample in the training set and the machine station manufacturing the sample may be detected. The degree of association anomaly may represent the degree of anomaly in the correlation between the samples and the machines in the training set.

As described above, the samples in the training set may be products actually produced and tested by each machine, and in order to ensure that the trained model can be universally applied to each machine, it is generally desirable that the number of samples from each machine in the sample set is substantially the same, and the difference between the number of samples from each machine can reflect the degree of abnormality associated between the samples and the machines in the training set. Thus, in some embodiments, detecting the associated degree of abnormality of a sample in the training set with the machine that manufactured the sample may include: detecting a number of samples from each of a plurality of machines in a training set; calculating a difference between the number of samples from each pair of machines in the plurality of machines; the maximum value of the difference in the number of samples is used as the correlation abnormality degree, and the difference reference value of the corresponding number of samples is used as the reference correlation abnormality degree. The measurement model control method may include: and when the maximum value of the difference of the sample numbers of each pair of machines is larger than the difference reference value of the sample numbers, giving an alarm and/or suspending the training of the training model for the machine with the larger sample number in the two machines corresponding to the maximum value.

For example, the number of samples from each machine station in the training set may be counted, and the number of samples of each machine station may be compared with the number of samples of other machine stations to obtain the difference between the number of samples of each pair of machine stations, and then the maximum value thereof may be taken as the correlation abnormality degree, and the difference reference value of the corresponding number of samples may be used as the reference correlation abnormality degree. When the maximum value of the differences of the sample numbers exceeds the difference reference value of the sample numbers, the correlation degree of the training set to a certain machine is too large, which causes the dependence of the trained model to the certain machine to be too large, and further influences the prediction of other machines. Therefore, an alarm can be issued at this time, and training of the training model for the machine concerned can be suspended. For example, training of the training model for the first machine may be stopped when the number of samples from the first machine is greater than the number of samples from the second machine, and the difference between them constitutes the maximum of the difference in number of samples between all machines. The reference value for the difference between the sample numbers may be determined experimentally or based on historical data.

After a product is manufactured, the association between the manufacturing tool and the product recipe may decrease over time due to changes in the condition of the manufacturing tool or the product recipe, especially after regular maintenance of the tool or adjustments to the product recipe. Thus, if the samples in the training set have been manufactured for a long time, their effect on the training model may be weak. Thus, in some embodiments, detecting the associated anomaly of the sample in the training set with the tool that manufactured the sample may include: detecting the manufacturing completion time of the samples in the training set; the manufacturing completion time is used as the associated degree of abnormality, and the corresponding reference manufacturing completion time is used as the reference associated degree of abnormality. The measurement model control method may include: and when the manufacturing completion time is earlier than the reference manufacturing completion time, giving an alarm and/or suspending training of the training model for the corresponding machine, wherein the corresponding machine is a machine corresponding to the sample with the manufacturing completion time earlier than the reference manufacturing completion time.

For example, the manufacturing completion time of each sample in the training set may be obtained from the database and compared with the reference manufacturing completion time, and when the manufacturing completion time of a certain sample is greater than the reference manufacturing completion time, it indicates that the association degree of the sample with the corresponding machine station may be already weak, and may indicate that the association degree of the sample with the reference of the machine station is high, at which point, an alarm may be issued and/or training of the training model for the machine station manufacturing the sample may be suspended. The reference manufacturing completion time may be determined experimentally or based on historical data.

In the training process, if the training using a certain training set is over a large time span, the relevance between the sample set and the machine is also reduced. Thus, in some embodiments, detecting the associated anomaly of the sample in the training set with the tool that manufactured the sample may include: detecting a training time span of a training set; the training time span is used as the association anomaly and the corresponding reference training time span is used as the reference association anomaly. The measurement model control method may include: issuing an alert and/or suspending training of training models for all machines when the training time span is greater than the reference training time span.

For example, the training time span may be counted from the beginning of training using a certain sample set. When the training time span is larger than the reference training time span, the association degree of the instruction sample set and the machine table is reduced, and accordingly, the association abnormality degree is increased and exceeds the reference association abnormality degree. At this point, an alarm may be raised and/or training of the training models for all of the stations may be suspended.

Referring to fig. 4, a control method of a measurement model according to another embodiment of the present application may include: detecting an application prediction error degree (S231) of the prediction model for the products manufactured by one or more machines and a running abnormal degree (S241) of the prediction model; and issuing an alarm and/or at least partially changing an operating state of the predictive model in response to at least one of the following conditions being met (S250): the application prediction error degree is lower than the reference application prediction error degree (S232), and the operation abnormality degree is higher than the reference operation abnormality degree (S242).

It should be noted that the control method of the measurement model described with reference to fig. 4 and the control method of the measurement model described with reference to fig. 3 may be used alone or in combination. For example, the measurement model may be monitored only using the method shown in fig. 3 to take a corresponding measure when an abnormality occurs, or the measurement model may be monitored only using the method shown in fig. 4 to take a corresponding measure when an abnormality occurs, and further, the method shown in fig. 3 and the method shown in fig. 4 may be used simultaneously to monitor the training model and the measurement model simultaneously.

In step S231, an applied prediction error degree of the prediction model for the product manufactured by the machine may be detected. The prediction accuracy may reflect the accuracy of the results predicted by the prediction model.

In a production environment, machines often need to be maintained irregularly, products produced by the machines before maintenance and products produced by the machines after maintenance may have a large difference, and at this time, a prediction model may be inapplicable to the machines after maintenance. Errors that may appear as a result of prediction by the predictive model on the detected data are outliers, e.g., errors that are significantly too large. If the proportion of the abnormal value is larger, the prediction model is probably not suitable for most machines, and the prediction accuracy is reduced. Therefore, in some embodiments, detecting the prediction error degree of the prediction model applied to the product manufactured by the machine may include: detecting a prediction error abnormal value ratio of prediction performed by the prediction model, wherein the prediction error abnormal value ratio is represented by a ratio of occurrence of a prediction error abnormal value in prediction of a predetermined number of times most recently made by the prediction model; the prediction error outlier ratio is used as an applied prediction error degree, and the corresponding reference prediction error outlier ratio is used as a reference applied prediction error degree. The measurement model control method may include: and when the prediction error abnormal value ratio is larger than the reference prediction error abnormal value ratio and is maintained for more than a preset time, giving an alarm and/or stopping the prediction of the prediction model for all the machine stations.

In the generation process, actual detection can be carried out on part of products, and the prediction error of the prediction model can be obtained by comparing the actual detection result of the products with the prediction result of the prediction model. In the present embodiment, the rate at which the prediction error abnormal value occurs in the last 10 predictions of the model can be detected. The prediction error abnormal value may be a prediction error exceeding a predetermined threshold, for example, 10% of the predetermined threshold, and when the prediction error of a certain prediction is 15%, the prediction error may be regarded as a prediction error abnormal value. When the rate of occurrence of prediction error outliers in the last 10 predictions of the model is greater than the corresponding reference prediction error outlier rate (e.g., 20%), it can be considered that the prediction accuracy of the prediction model will be affected. The measurement model control unit may alarm and/or stop prediction of the prediction model for all the machines if the prediction error abnormal value ratio exceeds the reference prediction error abnormal value ratio for a predetermined time. The reference prediction error outlier ratio may be determined experimentally or based on historical data.

The median prediction error value can reflect an average prediction error of the prediction model, and therefore, in some embodiments, detecting the application prediction error degree of the prediction model for the product manufactured by the machine may include: detecting a median prediction error value of predictions performed by the prediction model; the prediction error degree is applied using the median of the prediction errors as an application prediction error degree, and using the corresponding median of the reference prediction errors as a reference. The measurement model control method may include: and when the prediction error median is larger than the reference prediction error median and is maintained for more than a preset time, giving an alarm and/or stopping the prediction of the prediction model for all the machine stations.

For example, the prediction errors of the last 15 predictions made by the prediction model may be sorted, and the prediction error (e.g., 20%) at the middle position (bit 8) may be taken as the median of the prediction errors of the prediction model. When the median prediction error is greater than the corresponding median reference prediction error (e.g., 15%), it is considered that the prediction accuracy of the prediction model will be affected. The measurement model control unit may alert and/or stop the prediction of the prediction model for all machines if the median prediction error exceeds the median reference prediction error for a predetermined time. The reference median prediction error may be determined experimentally or based on historical data.

The prediction error change rate can reflect the change trend of the prediction error of the prediction model, and is beneficial to timely finding out the abnormal state of the prediction model. Therefore, in some embodiments, detecting the prediction error degree of the prediction model applied to the product manufactured by the machine may include: detecting a prediction error change rate of the prediction performed by the prediction model; the prediction error degree is applied using the prediction error change rate as an application prediction error degree, and using the corresponding reference prediction error change rate as a reference. The measurement model control method may include: and when the prediction error change rate is larger than the reference prediction error change rate and is maintained for more than a preset time, giving an alarm and/or stopping the prediction of the prediction model for all the machines.

For example, the prediction error of the latest 10 predictions by the prediction model may be compared with the prediction error of its respective previous prediction to obtain 10 prediction error change rates, and the average of the 10 prediction error change rates may be taken as the prediction error change rate of the prediction model. In another embodiment, the prediction error of the latest prediction by the prediction model may be compared with the prediction error of the previous prediction to obtain the prediction error change rate, and the prediction error change rate may be used as the prediction model prediction error change rate. When the prediction error change rate of the prediction model exceeds the reference prediction error change rate and exceeds a predetermined time, which indicates that the prediction error of the prediction model is continuously increasing, it can be considered that the prediction accuracy of the prediction model will be affected, and at this time, the measurement model control unit may alarm and/or stop the prediction of the prediction model for all the machines. The reference prediction error rate of change may be determined experimentally or based on historical data.

Referring again to fig. 4, in step S241, the degree of operation abnormality of the prediction model may be detected, and the degree of operation abnormality may represent the degree to which the prediction model is abnormal during the prediction process.

When the prediction model runs, the prediction model may suffer from problems of database connection or excessive load of the prediction server, and the like, and these problems may cause that the time for completing one prediction of the prediction model is too long, thereby increasing the abnormal degree of the running of the prediction model. Thus, detecting a degree of operational anomaly of the predictive model may include: detecting a predicted time for the predictive model to complete prediction in response to a prediction request by a controller of one or more machines; the predicted time is used as the degree of operational abnormality, and the corresponding reference predicted time is used as the reference degree of operational abnormality. The measurement model control method may include: an alarm is issued when the predicted time is greater than the reference predicted time.

For example, the timing may be started when the controller of a certain machine issues a prediction request to the prediction model, and the timing may be ended when the prediction model obtains a prediction result, where the time to be measured is the prediction time. And comparing the prediction time with the corresponding reference prediction time, and when the prediction time is greater than the reference prediction time, indicating that the operation of the prediction model is abnormal, namely the operation abnormality degree of the prediction model is greater than the reference operation abnormality degree, and then sending an alarm. The reference predicted time may be determined experimentally or based on historical data.

When the prediction model runs, the prediction model may encounter data integrity problems in the database, and the loss of partial data may cause the prediction model to fail, thereby reducing the prediction success rate of the prediction model. Thus, in some embodiments, detecting a degree of operational anomaly of the predictive model may include: detecting a prediction success rate of the prediction model for successfully predicting in response to a prediction request of a controller of one or more machines; the prediction success rate is used as the degree of operational anomaly, and the corresponding reference prediction success rate is used as the reference degree of operational anomaly. The measurement model control method may include: an alarm is issued when the prediction success rate is less than the reference prediction success rate.

For example, when the reference prediction success rate is 70%, when the prediction model performs the nth prediction, the success rate from the N-10 th prediction to the nth prediction performed by the prediction model may be calculated, and if the success rate is greater than 70%, it may be considered that the current prediction model operates normally, and the prediction success rate after detection may be continued. When the prediction model carries out the next prediction (namely, the (N + 1) th prediction), the prediction success rate from the (N-9) th prediction to the (N + 1) th prediction can be detected, and if the prediction success rate is less than 70%, the prediction model is considered to be abnormal in operation, and an alarm can be sent out. In other words, in the present embodiment, it is possible to always detect the prediction success rate of the last 10 predictions of the prediction model, and issue an alarm when the prediction success rate is less than the reference prediction success rate. Although the reference prediction success rate is described as 70% in the present embodiment, and the prediction success rate of detecting the latest 10 predictions is described, the present application is not limited thereto, and the number of detections and the reference prediction success rate may be set to other values, wherein the reference prediction success rate may be determined by experiment or based on historical data.

In order to fully utilize the prediction model and to apply the virtual test to each machine as widely as possible, it is beneficial to monitor the number of prediction requests from each machine. Thus, in some embodiments, detecting a degree of operational anomaly of the predictive model may include: detecting a number of predicted requests from each of the one or more machines; calculating the difference between the predicted request times of each pair of machines in one or more machines; the maximum value of the difference in the number of predicted requests is used as the degree of operational abnormality, and the reference value of the difference in the corresponding number of predicted requests is used as the reference degree of operational abnormality. The measurement model control method may include: an alarm is issued when the maximum value among the differences of the predicted number of requests is greater than the difference reference value of the predicted number of requests.

For example, the prediction requests from each machine may be counted, and the prediction request times of each machine may be compared with the prediction request times of other machines to obtain the difference between the prediction request times of each pair of machines, and then the maximum value thereof may be taken as the operation abnormality degree, and the corresponding difference reference value between the prediction request times may be used as the reference operation abnormality degree. When the maximum value of the difference of the prediction request times exceeds the difference reference value of the prediction request times, it indicates that a certain machine occupies too large prediction model resources, which may cause that some machines cannot be fully detected. Thus, an alarm may be issued at this time. The difference reference value between the predicted request times may be determined experimentally or based on historical data.

In some embodiments, when at least one of the training prediction accuracy is less than the reference training prediction accuracy, the association abnormality degree is greater than the reference association abnormality degree, the application prediction error degree is greater than the reference application prediction error degree, and the operation abnormality degree is greater than the reference operation abnormality degree, it indicates that the state of the measurement model is abnormal, which may affect the prediction accuracy and thus the product quality detection of the production line, and at this time, the measures for operating one or more machines may be stopped. For example, the operation of the tool that may be affected by the metrology model anomaly may be stopped. The machine station which stops working can resume working after the measurement model abnormity is discharged.

The various embodiments described above may be implemented as software including instructions that may be stored in a machine-readable storage medium that may be read by a machine (e.g., a computer or control system). The machine (or control system) is a device that invokes instructions stored in a storage medium, and includes a processor and a memory that stores one or more programs that, when executed by the processor, may perform the method described above with reference to fig. 3 or the method described with reference to fig. 4, or the method described with reference to fig. 3 and the method described with reference to fig. 4.

If the instructions are executed by a processor, the processor may perform the functions corresponding to the instructions by itself or by using other elements under the control of the processor. The instructions may include code generated or executed by a compiler or interpreter. For example, the control method of the measurement model described above may be performed when instructions stored in the memory are executed by the processor.

The machine-readable storage medium may be provided in the form of a non-transitory storage medium.

According to an embodiment, the method according to the various embodiments described above may be provided as comprised in a computer program product. The computer program product may be used to conduct a transaction as a product between a seller and a consumer. The computer program product may be distributed in the form of a machine-readable storage medium, such as a compact disc ROM (CD-ROM), or online through an application store. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated in a manufacturer's server, a server of an application store, or a storage medium such as a memory of a relay server.

While the disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims and their equivalents.

Claims

1. A method for controlling a metrology model, wherein the metrology model comprises a training model and a prediction model, wherein the training model is used to update the prediction model, and the prediction model is used to predict a product manufactured by one or more tools, the method comprising:

detecting the application prediction error degree of the prediction model aiming at the products manufactured by the one or more machine stations and the operation abnormity degree of the prediction model; and

issuing an alarm and/or at least partially changing an operating state of the predictive model in response to at least one of the following conditions being met:

the applied prediction error degree is greater than the reference applied prediction error degree, an

The operational anomaly is greater than a reference operational anomaly.

2. The control method of claim 1, wherein detecting a degree of operational anomaly of the predictive model comprises:

detecting a predicted time for the predictive model to complete a prediction in response to a prediction request by the one or more machines;

using the predicted time as the degree of operational anomaly and using a corresponding reference predicted time as the reference degree of operational anomaly.

3. The control method according to claim 2, characterized by further comprising:

issuing an alert when the predicted time is greater than the reference predicted time.

4. The control method of claim 1, wherein detecting a degree of operational anomaly of the predictive model comprises:

detecting a prediction success rate of the prediction model for successfully predicting in response to the prediction request of the one or more machines, wherein the prediction success rate is represented by a latest success rate predicted for a predetermined number of times;

the operational anomaly is calculated based on the prediction success rate, and the reference operational anomaly is calculated based on a corresponding reference prediction success rate.

5. The control method according to claim 4, characterized by further comprising:

when the prediction success rate is less than the reference prediction success rate, determining that the operation abnormality degree is greater than the reference operation abnormality degree, and issuing an alarm.

6. The control method of claim 1, wherein detecting a degree of operational anomaly of the predictive model comprises:

detecting a predicted number of requests from each of the one or more machines;

calculating a difference between the predicted number of requests from each pair of machines in the one or more machines;

using a maximum value of the difference in the prediction request times as the operation abnormality degree, and using a reference value of the difference in the corresponding prediction request times as the reference operation abnormality degree.

7. The control method according to claim 6, characterized by further comprising:

and sending an alarm when the maximum value in the difference of the prediction request times is larger than the difference reference value of the prediction request times.

8. The control method as claimed in claim 1, wherein detecting the prediction error degree of the prediction model applied to the one or more machines for manufacturing the product comprises:

detecting a prediction error outlier ratio of a prediction performed by the prediction model, wherein the prediction error outlier ratio is represented by a ratio of occurrence of a prediction error outlier in a prediction made by the prediction model a predetermined number of times most recently, wherein the prediction error outlier is a prediction error exceeding a predetermined threshold;

using the prediction error outlier ratio as the application prediction error degree, and using a corresponding reference prediction error outlier ratio as the reference application prediction error degree.

9. The control method according to claim 8, characterized by further comprising:

and when the prediction error abnormal value ratio is larger than the reference prediction error abnormal value ratio and is maintained for more than a preset time, giving an alarm and/or stopping the prediction of the prediction model for all the machine stations.

10. The control method as claimed in claim 1, wherein detecting the prediction error degree of the prediction model applied to the one or more machines for manufacturing the product comprises:

detecting a median prediction error value of predictions performed by the prediction model, wherein the median prediction error value is represented by a median value of prediction errors of a predetermined number of predictions most recently made by the prediction model;

using the median prediction error value as the applied prediction error degree, and using a corresponding median reference prediction error value as the reference applied prediction error degree.

11. The control method according to claim 10, characterized by further comprising:

and when the prediction error median is larger than the reference prediction error median and is maintained for more than a preset time, giving an alarm and/or stopping the prediction of the prediction model for all the machines.

12. The control method as claimed in claim 1, wherein detecting the prediction error degree of the prediction model applied to the one or more machines for manufacturing the product comprises:

detecting a prediction error change rate of prediction performed by the prediction model, wherein the prediction error change rate is represented by a change degree of a prediction error predicted a predetermined number of times by the prediction model most recently;

using the prediction error change rate as the application prediction error degree, and using a corresponding reference prediction error change rate as the reference application prediction error degree.

13. The control method according to claim 12, characterized by further comprising:

and when the prediction error change rate is larger than the reference prediction error change rate and is maintained for more than a preset time, giving an alarm and/or stopping the prediction of the prediction model for all the machines.

14. The control method of claim 1, wherein the method further comprises:

detecting the training prediction accuracy of the training model for a training set, and the association abnormality degree of a sample and a machine station for manufacturing the sample in the training set, wherein the training set is a product manufactured by one or more machine stations; and

the training prediction accuracy is less than the reference training prediction accuracy, an

The association anomaly is greater than a reference association anomaly.

15. The control method of claim 14, wherein detecting the training prediction accuracy of the training model for the training set comprises:

by means of R²The score represents the training prediction accuracy, wherein the R²The score includes a first R²Fraction and second R²Fraction, the first R²The score is R calculated for all samples in the training set²Fraction, the second R²Scoring R calculated for samples in a plurality of subsets of the training set²Median fraction of fractions.

16. The control method of claim 15, wherein the second R is²The score is obtained by:

randomly dividing the training set into a first training set and a second training set;

training the training model using the first training set;

predicting the second training set using the trained training model;

obtaining R of the trained training model for the second training set²A score;

randomly re-dividing the training set into a new first training set and a new second training set; and

repeating the above steps for a predetermined number of times to obtain R²The median score of the scores is taken as the second R²And (4) scoring.

17. The control method according to claim 15, wherein R is the same as R²The score further comprises a third R²Fraction, the third R²The scores include: training model trained on temporally prior produced samples, R for temporally later produced samples²And (4) scoring.

18. The control method of claim 17, wherein the third R is²The score is obtained by:

ordering the samples in the training set in a production completion time from first to last order;

selecting a predetermined number as a selection window for determining a consecutive, predetermined number of data when selecting data from the training set;

determining the third R according to the training set and the selection window²And (4) scoring.

19. The control method of claim 18, wherein said determining said third R is based on said training set and said selection window²Points, including:

selecting a third training set using the selection window starting from a first sample in the training set;

training the training model using the third training set;

predicting the sample which is positioned one bit behind the selection window in the training set by using the training model trained by the third training set;

a step of moving the selection window one sample backward in the training set;

updating the third training set by using the samples in the moved selection window;

repeating the above steps until the last sample in the training set is predicted; and

calculating R for all predictors²Score as the third R²And (4) scoring.

20. The control method of claim 17, wherein the first R is²Fraction greater than the second R²Is a fraction, and the second R²Fraction greater than the third R²And (4) scoring.

21. The control method of claim 17, wherein detecting the training prediction accuracy of the training model for the training set further comprises:

respectively using the first R²Fraction, the second R²Score and the third R²Scores represent the training prediction accuracy and are separately associated with the first R²Fraction, the second R²Score and the third R²Fraction-corresponding R²And taking the fractional reference value as the reference training prediction accuracy.

22. The control method according to claim 21, characterized by further comprising:

when the first R is²Fraction, the second R²Score and the third R²At least one of the scores is lower than the corresponding R²When the score reference value is determined, the training prediction accuracy is determined to be lower thanThe reference training predicts accuracy and issues an alarm and/or suspends training of training models for all machines.

23. The control method of claim 14, wherein the method further comprises:

detecting the training process abnormality degree of the training model; and

at least partially changing an operating state of the predictive model in response to the training process anomaly being greater than a reference training process anomaly.

24. The control method of claim 23, wherein detecting the degree of training process abnormality of the training model comprises:

detecting an interval time between successive successful trainings of the training model;

and representing the training process abnormality degree by using the interval time, and using a corresponding interval time reference value as the reference training process abnormality degree.

25. The control method according to claim 24, characterized by further comprising:

issuing an alarm and/or suspending training of training models for all machines if the interval time is greater than the interval time reference value and continues for more than a predetermined time.

26. The control method of claim 23, wherein detecting the degree of training process abnormality of the training model comprises:

detecting a training time for each time of completing training of the training model;

and representing the training process abnormality degree by using the training time, and using a corresponding training time reference value as the reference training process abnormality degree.

27. The control method according to claim 26, characterized by further comprising:

issuing an alarm and/or suspending training of training models for all machines if the training time is greater than the training time reference value and continues for more than a predetermined time.

28. The method of claim 14, wherein detecting the degree of abnormality associated with the sample in the training set and the machine from which the sample was made when the sample in the training set came from multiple machines comprises:

detecting a number of samples from each of the plurality of machines in the training set;

calculating a difference in the number of samples from each pair of machines in the plurality of machines;

using a maximum value of the differences in the number of samples as the associated degree of abnormality, and using a reference value of the difference in the corresponding number of samples as the reference associated degree of abnormality.

29. The control method according to claim 28, characterized by further comprising:

when the maximum value of the difference of the sample numbers of each pair of machines is larger than the difference reference value of the sample numbers, giving an alarm and/or suspending the training of the training model of the machine with the larger sample number in the two machines corresponding to the maximum value.

30. The method of claim 14, wherein detecting the degree of abnormality associated with the sample in the training set and the machine from which the sample was made when the sample in the training set came from multiple machines comprises:

detecting a manufacturing completion time of the samples in the training set;

using the manufacturing completion time as the associated degree of abnormality and using a corresponding reference manufacturing completion time as the reference associated degree of abnormality.

31. The control method according to claim 30, characterized by further comprising:

when the manufacturing completion time is earlier than the reference manufacturing completion time, issuing an alarm and/or suspending training of a training model for a corresponding machine, wherein the corresponding machine is a machine that manufactures a sample whose manufacturing completion time is earlier than the reference manufacturing completion time.

32. The method of claim 14, wherein detecting the degree of abnormality associated with the sample in the training set and the machine from which the sample was made when the sample in the training set came from multiple machines comprises:

detecting a training time span of the training set, the training time span representing a time since when the training set was used to train the training model;

utilizing the training time span as the association degree of anomaly and utilizing a corresponding reference training time span as the reference association degree of anomaly.

33. The control method according to claim 32, characterized by further comprising:

issuing an alert and/or suspending training of training models for all machines when the training time span is greater than the reference training time span.

34. The control method of any one of claims 1-13, wherein issuing an alert comprises:

information relating to the degree of application prediction error and the degree of operational anomaly.

35. The control method of any one of claims 14-33, wherein issuing an alert comprises:

displaying information related to the training prediction accuracy and the degree of associated abnormality through a display device.

36. The control method according to claims 1 to 13, characterized in that the control method further comprises:

deactivating at least one of the one or more machines in response to at least one of the following conditions being met:

The operational anomaly is greater than a reference operational anomaly.

37. The control method according to claims 14 to 33, characterized in that the control method further comprises:

The association anomaly is greater than a reference association anomaly.

38. A method for controlling a metrology model, the metrology model comprising a training model and a prediction model, wherein the training model is used to update the prediction model, and the prediction model is used to predict a product manufactured by one or more tools, the method comprising:

detecting training prediction accuracy of the training model for a training set, and an association anomaly degree of a sample in the training set and a machine station for manufacturing the sample, wherein the training set comprises products manufactured by the one or more machine stations; and

The association anomaly is greater than a reference association anomaly.

39. A control system for a metrology model, comprising:

a processor; and

memory having one or more programs stored thereon that, when executed by the processor, cause the processor to implement the method of any of claims 1-37.

40. A control system for a metrology model, comprising:

a processor; and

memory having one or more programs stored thereon that, when executed by the processor, cause the processor to implement the method of claim 38.

41. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the control method according to any one of claims 1 to 37.

42. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the control method according to claim 38.

43. A computer program product comprising a computer program which, when executed by a processor, implements a control method according to any one of claims 1-37.

44. A computer program product comprising a computer program which, when executed by a processor, implements a control method according to claim 38.