JP5696354B2 - Reliability judgment device - Google Patents

Description

  The present invention relates to a technique for determining the reliability of an estimation model obtained by a statistical method.

  In recent years, efforts have been made to construct a model for estimating a process (hereinafter referred to as an “estimated model”) by performing a statistical method on various sensor data of a process apparatus being processed. Specifically, an estimation model is prepared in advance with teacher data (combination of objective variables and explanatory variables) consisting of actual measurement values using a large number of process evaluation lots. Multivariate analysis, multiple regression analysis, regression analysis, etc.). By using the estimation model, it is possible to judge the machining process result without actually inspecting with the inspection equipment, and also reduce the input of test products, so it will continue to be used in the future. .

  Conventionally, an estimation model constructed at a mass production site is constructed by collecting objective variables and explanatory variables in a certain short period of time. Therefore, during production of products using the estimation model, it is necessary to determine at an appropriate time whether or not the appropriate state of the estimation model can be maintained.

  For example, in an environment where a new product is introduced in one year, data of objective variables and explanatory variables for model generation was collected over half a year or one year. Opportunities should disappear. Therefore, for example, an explanatory variable having a high correlation with the objective variable is selected in a short period such as one month, and an estimation model must be constructed. Therefore, if there is an explanatory variable that fluctuates over a longer period than the period for which the data was collected for constructing the estimation model, or if the relationship of explanatory variables with high correlation changes due to maintenance, etc., the accuracy of the estimation model There is a possibility of lowering. Of course, it is sufficient if the estimation model can be updated to the latest data and reconstructed at any time, but in practice, teacher data is often not easily collected. Therefore, it was necessary to regularly check the reliability of the estimation model itself.

  In the general process estimation model, the reliability of the estimation model is not confirmed, but the difference between the predicted value obtained by the estimation model and the actual measurement value obtained by actual measurement is calculated. An alarm is issued when the threshold is exceeded. For example, Patent Document 1 describes that a model for predicting a polishing time is constructed in a semiconductor CMP (Chemical Mechanical Polishing) apparatus, and an error judgment is made when the difference between the predicted value and the actual measurement value exceeds a threshold value. Has been. Patent Document 2 discloses a method for extracting a correlation between various data.

JP 2008-258510 A JP 2006-86403 A

  However, in the technique disclosed in Patent Document 1, the difference between the predicted value and the actual measurement value is simply viewed, and the reliability of the estimation model is not determined. Therefore, it becomes a threshold setting problem and it is difficult to judge it as a false alarm. In addition, the technique disclosed in Patent Document 2 is intended to find a relationship between data from data collected from various conditions. The reliability of the model cannot be determined.

In view of the above circumstances, the present invention aims at providing a reliability determination equipment capable of determining the reliability of the estimation model.

  One aspect of the present invention is a reliability determination device that divides a range of values of explanatory variables in a plurality of actually measured values into a plurality of sections for an estimated model statistically obtained based on a plurality of actually measured values. A section dividing unit, a section determination unit for determining which section of the plurality of sections the value of the explanatory variable of the estimation target belongs to the estimation target for which an actual measurement value is obtained in advance, and the section A determination unit configured to determine the reliability of the estimation model based on the plurality of actual measurement values belonging to the section determined by the determination unit and the estimation result of the estimation model in the section;

  One aspect of the present invention is a reliability determination method, in which an information processing device calculates a range of values of explanatory variables in a plurality of actually measured values for an estimated model statistically obtained based on a plurality of actually measured values. A section dividing step for dividing into a plurality of sections, and for the estimation object for which the measured value is obtained in advance, the value of the explanatory variable in the estimation object is in which section of the plurality of sections. The estimation model based on the section determination step for determining whether to belong, the information processing apparatus based on the plurality of actually measured values belonging to the section determined by the section determination step, and the estimation result of the estimation model in the section And a determination step of determining the reliability of.

  One aspect of the present invention is a computer program for determining reliability, and for an estimation model statistically obtained based on a plurality of actually measured values, a range of values of explanatory variables in the plurality of actually measured values is divided into a plurality of sections. And a section determination step for determining which section of the plurality of sections the value of the explanatory variable in the estimation target object is for the estimation target object in which the actual measurement value is obtained in advance. A determination step of determining a reliability of the estimation model based on the plurality of actually measured values belonging to the section determined by the section determination step and an estimation result by the estimation model in the section; A computer program for execution.

  According to the present invention, it is possible to determine the reliability of the estimation model.

It is a schematic block diagram showing the functional structure of a reliability judgment apparatus. It is a figure showing the example of an objective variable, an explanatory variable, and an estimation model. It is a flowchart showing the flow of a process of a reliability judgment apparatus. It is a flowchart showing the flow of the distribution coefficient calculation process by a distribution coefficient calculation part.

  FIG. 1 is a schematic block diagram illustrating a functional configuration of the reliability determination apparatus 1. The reliability determination apparatus 1 includes a CPU (Central Processing Unit) connected via a bus, a memory, an auxiliary storage device, and the like, and executes a reliability determination computer program to thereby obtain an interval division unit 101 and a distribution coefficient calculation unit 102. , Section determination unit 103, determination unit 104, contribution rate calculation unit 105, variance ratio calculation unit 106, comparison unit 107, and output unit 108. Note that all or part of the functions of the reliability determination apparatus 1 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit) or a PLD (Programmable Logic Device).

  The estimation model prepares in advance a plurality of teacher data (combinations of objective variables and explanatory variables) consisting of actual measurement values using a large number of process evaluation lots, and statistical methods (multivariate analysis and This is a prediction formula obtained by performing multiple regression analysis or regression analysis. The estimation model obtained in this way is used, for example, to represent the characteristics of a process change state such as a non-defective product determination or a wear state while continuing production. After the estimation model is obtained, data on the test evaluation lots (hereinafter referred to as “test data”) is obtained periodically. The test data includes an explanatory variable value for the test evaluation lot and an actual measurement value of the objective variable obtained by actually performing the measurement for the test evaluation lot. The reliability determination apparatus 1 receives as input an estimation model constructed by a statistical method such as multivariate analysis or multiple regression analysis, and all the teacher data and test data used to construct the estimation model. The reliability of the estimation model is determined based on the test data, and the determination result is output. In order to perform such processing, each functional unit of the reliability determination apparatus 1 operates as follows.

  The section dividing unit 101 divides the range of the explanatory variable values used when the estimation model is generated to form a plurality of sections. The distribution coefficient calculation unit 102 calculates a distribution coefficient in each section by executing a distribution coefficient calculation process. The section determining unit 103 determines which section of the plurality of sections formed by the section dividing unit 101 the input test data belongs to. The determination unit 104 determines whether or not there is a sufficient data distribution when the estimated model is generated in the section to which the test data belongs. The contribution rate calculation unit 105 calculates the contribution rate of the section to which the test data belongs. The variance ratio calculation unit 106 calculates the variance ratio of the section to which the test data belongs. The comparison unit 107 determines the reliability by comparing the contribution rate and the variance ratio for the section to which the test data belongs. The output unit 108 outputs the determination result by the determination unit 104 or the determination result by the comparison unit 107 to the user of the reliability determination device 1 by the voice output unit or the display unit. Further, the output unit 108 may be configured to output data representing the determination result not to the user but to another device.

  FIG. 2 is a diagram illustrating an example of an objective variable, an explanatory variable, and an estimation model. The objective variable and the explanatory variable serving as the teacher data are black dots in FIG. 2 and are obtained, for example, when the semiconductor manufacturing apparatus processes the process evaluation lot. The estimation model is a straight line in FIG. 2, and is obtained from the objective variable and the explanatory variable by multivariate analysis, multiple regression analysis, regression analysis, or the like. In FIG. 2, the number of the objective variable and the explanatory variable is one, but there may be a plurality of types. In addition, the estimation model means an approximate straight line or curve for actual measurement including linear or non-linear.

  FIG. 3 is a flowchart showing a process flow of the reliability determination apparatus 1. First, the section dividing unit 101 divides the data range of the explanatory variable of the estimation model into a plurality of sections based on a predetermined method (step S101). The divided sections may have a constant width as shown in FIG. 3, or the width may be different for each section. In addition, each section may be set so as not to overlap each other as shown in FIG. 3, or may be set so that a part of the sections overlaps each other (so as to overlap).

  Next, the distribution coefficient calculation unit 102 calculates a distribution coefficient in each section by executing a dispersion coefficient calculation process (step S102). The distribution coefficient indicates whether or not the number of teacher data at the time of construction of the estimation model is sufficient, and whether or not the teacher data is distributed without being concentrated on one point.

  FIG. 4 is a flowchart showing the flow of distribution coefficient calculation processing by the distribution coefficient calculation unit 102. First, the distribution coefficient calculation unit 102 selects a section for which a distribution coefficient is to be calculated (step S201). This selection may be performed in any order. For example, the selection may be made in order from the smallest value of the explanatory variable.

  Next, the distribution coefficient calculation unit 102 selects data points in the section for which the distribution coefficient is to be calculated as a reference point in order from the end of the section, and further sets a data point in the next order of the reference point as an adjacent point. Select (step S202). A data point is a point representing one teacher data, and is a single black point shown in FIG. Also, the above order may be from the smallest to the largest along the explanatory variable axis, or from the smallest to the largest along the objective variable axis or both axes, The order may be from the largest to the smallest along any axis, or may be in any other order.

  Next, the distribution coefficient calculation unit 102 calculates the distance between the reference point and the adjacent point (hereinafter referred to as “inter-adjacent distance”) (step S203). Next, the distribution coefficient calculation unit 102 determines whether or not the calculated adjacent distance is greater than a preset threshold value (step S204). If the inter-adjacent distance is equal to or smaller than the threshold (step S204—NO), the distribution coefficient calculation unit 102 ignores the current adjacent point and further selects the next sequential data point as a new adjacent point (step S205). . Then, the distribution coefficient calculation unit 102 calculates an inter-adjacent distance based on the new adjacent point (step S203) and compares it with a threshold value (step S204).

  On the other hand, when the distance between adjacent areas is larger than the threshold in the process of step S204 (step S204-YES), the distribution coefficient calculating unit 102 updates the accumulated distance between adjacent areas in the section to be calculated (step s206). Next, the distribution coefficient calculation unit 102 increments the counter value (step S207). The value of this counter represents the number of times that the distance between adjacent points has been accumulated, in other words, the number of combinations of reference points and adjacent points used for the accumulation of distances between adjacent points.

  Next, the distribution coefficient calculation unit 102 determines whether or not the processing in steps S202 to S207 has been performed using all data points in the section for which the distribution coefficient is to be calculated as a reference point (step S208). The processes in steps S202 to S207 are repeatedly executed until there are no remaining data points (step S208—NO). When the process returns to step S202, the distribution coefficient calculation unit 102 sets a data point in the next order after the data point that has been used as the reference point as a new reference point, and further adds a data point next to the new reference point. Processing is performed with the sequential data point as a new adjacent point.

  On the other hand, when there is no data point that is not a reference point (step S208—YES), the distribution coefficient calculation unit 102 divides the latest cumulative value of adjacent distances by the latest counter value to calculate the distribution coefficient. Calculate (step S209).

Next, the distribution coefficient calculation unit 102 determines whether or not the processing in steps S201 to S209 has been completed for all the sections formed by the section dividing unit 101 (step S210), and distribution coefficients are calculated for all the sections. The processes of steps S201 to S209 are repeatedly executed until NO (step S210-NO). When the processing returns to step S201, the distribution coefficient calculation unit 102 resets the cumulative value of the adjacent distance and the counter value to zero, and executes the processing after step S201.
On the other hand, when distribution coefficients are calculated for all sections (step S210—YES), the distribution coefficient calculation unit 102 ends the distribution coefficient calculation process.

  When the distribution coefficient calculation process ends, the section determination unit 103 determines which section of the plurality of sections formed by the section division unit 101 the test data belongs to (step S103). Next, the determination unit 104 determines whether or not the distribution coefficient of the section determined in step S103 is greater than or equal to a threshold value (step S104). When the distribution coefficient is less than the threshold value (step S104-NO), that is, when there is not a sufficient number of teacher data at the time of generating the estimated model, the determination unit 104 uses the model because there is little teacher data at the time of generating the estimated model. It is determined that the reliability cannot be determined (step S105). In this case, the output unit 108 outputs the determination result “model reliability determination impossible” by the determination unit 104.

On the other hand, when the distribution coefficient is larger than the threshold (step S104-YES), that is, when there is a sufficient number of teacher data at the time of generation of the estimation model, the contribution rate calculation unit 105 estimates the section determined in step S103. A contribution ratio of the model is calculated (step S106). Specifically, the contribution rate calculation unit 105 calculates the contribution rate of the estimation model according to the following Equation 1. Note that the value of i in Equation 1 represents identification information of each teacher data. Yi _pre represents a predicted value calculated by the estimation model when the explanatory variable is Xi, and is equal to a + bXi. Yi represents an actual measurement value when the explanatory variable is Xi, and is obtained from the teacher data. Y _ave represents an average value of actually measured values. The explanatory variables Xi used in Equation 1 are all of the explanatory variables Xi existing as teacher data that belong to the section determined in step S103. Therefore, Y _ave represents the average value of the actual measurement values of the objective variables in all the teacher data in which the explanatory variable Xi belongs to the section determined in step S103.

  In Formula 1, a value obtained by dividing the squared cumulative value of the difference between the measured value and the average value of each teacher data belonging to the section determined in step S103 by the squared cumulative value of the difference between each predicted value and the average value Is obtained as a contribution rate. Here, the contribution rate is a value indicating the fitness of the estimation model. The method of calculating the contribution rate is not limited to the method according to Equation 1, and if a value representing the degree of fit of the predicted value with the actual measurement value can be calculated, for example, the difference value between the actual measurement value and the predicted value is used as the contribution rate. Other calculation methods may be used.

Next, the variance ratio calculation unit 106 calculates the ratio of variance between the predicted value and the actual measurement value of the test evaluation lot with respect to the center of the teacher data when the estimated model is generated in the section determined in step S103. . Specifically, the dispersion ratio calculation unit 106 calculates the dispersion ratio according to the following Equation 2. Yt _pre represents a predicted value calculated by the estimation model when the explanatory variable is the explanatory variable Xt of the test data, and is equal to a + bXt. Yt represents an actual measurement value when the explanatory variable is Xt, and is obtained from test data. Y _ave represents an average value of actually measured values. The explanatory variables Xt used in Equation 2 are all of the explanatory variables Xt existing as test data that belong to the section determined in step S103. Therefore, Y _ave represents the average value of the actual measurement values of the objective variable in all the test data in which the explanatory variable Xt belongs to the section determined in step S103.

  In Formula 2, the value obtained by dividing the squared cumulative value of the difference between the measured value and the average value of each test data belonging to the section determined in step S103 by the squared cumulative value of the difference between each predicted value and the average value Is obtained as a dispersion ratio. Here, the dispersion ratio is a value indicating the degree of fitness of the estimation model. The calculation method of the dispersion ratio is not limited to the method according to Equation 2, and if a value representing the degree of fit of the predicted value with the actual measurement value can be calculated, for example, the difference value between the actual measurement value and the predicted value is set as the dispersion ratio. Other calculation methods may be used.

  The comparison unit 107 compares the contribution rate calculated by the contribution rate calculation unit 105 with the dispersion ratio calculated by the dispersion ratio calculation unit 106 (step S108). When the variance ratio is larger than the contribution rate (step 108-YES), the comparison unit 107 determines that the reliability of the estimation model is high and determines that the estimation model does not need to be reconstructed (step S109). On the other hand, when the variance ratio is less than or equal to the contribution rate (step 108—NO), the comparison unit 107 determines that the reliability of the estimation model is low and determines that the estimation model needs to be reconstructed (step S110). After the process of step S109 or step S110, the output unit 108 outputs the determination result.

The reliability determination device 1 may be realized as an independent information processing device that is not related to an arithmetic device in an actual manufacturing device or individual manufacturing devices.
According to the reliability determination device 1 configured as described above, it is possible to easily determine the stability of the estimation model once constructed without newly preparing new teacher data based on a new process evaluation lot. This makes it possible to easily confirm the reliability of the estimation model.

  Moreover, according to the reliability determination apparatus 1, the reliability of the estimation model once constructed is determined for each section of the teacher data, and “determination” is performed without determining the reliability of the estimation model for the section with less teacher data. Since it is “impossible”, it is possible to prevent an erroneous report from being output in a section with a small amount of teacher data.

  In addition, according to the reliability determination apparatus 1, in determining the reliability of the estimation model, the determination is not performed using a lot of teacher data using a lot of process evaluation lots necessary to reconstruct the estimation model. Judgment is made with a small amount of test data using a small number of test evaluation lots. Therefore, it is possible to reduce the number of test evaluation lots to be actually measured, and it is possible to improve the processing efficiency related to the actual product instead of the test evaluation lot.

  Moreover, according to the reliability determination apparatus 1, the following can also be performed. That is, there are cases where an estimation model constructed by selecting a parameter having a high correlation in a short period is constructed without including a correlation parameter that is originally necessary when the parameter is examined in a longer period. However, since it is possible to immediately determine that the reconstruction is necessary when the reliability decreases, the estimation model is constructed by the teacher data obtained in such a short period of time. Can also be used in practice. Therefore, it is possible to avoid a situation in which the production of the corresponding product is finished when the estimation model is completed.

  In addition, as an application example of the reliability determination apparatus 1, a semiconductor apparatus that estimates process contents by using sensor data of the apparatus can be cited.

<Modification>
The comparison unit 107 may determine that it is necessary to reconstruct the estimation model when the number of times that the variance ratio of the evaluation lot in step S108 is smaller than the contribution rate is repeated a certain number of times.
The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 1 ... Reliability judgment apparatus, 101 ... Section division part, 102 ... Distribution coefficient calculation part, 103 ... Section determination part, 104 ... Determination part, 105 ... Contribution rate calculation part, 106 ... Dispersion ratio calculation part, 107 ... Comparison part ( Judgment part), 108 ... output part

Claims (2)

A section dividing unit that divides a range of values of explanatory variables in the plurality of actually measured values into a plurality of sections for the estimation model statistically obtained based on the plurality of actually measured values,
A section determination unit that determines which section of the plurality of sections the value of the explanatory variable in the estimation target belongs to the estimation target for which the actual measurement value is obtained in advance,
A determination unit that determines the reliability of the estimation model based on the plurality of actual measurement values belonging to the section determined by the section determination unit and the estimation result of the estimation model in the section;
A distribution coefficient calculation unit that calculates a distribution coefficient for evaluating the number or distribution of the actual measurement values used when obtaining the estimation model in the section determined by the section determination unit;
The reliability determination device , wherein the determination unit does not determine the reliability when the distribution coefficient does not satisfy a predetermined criterion . A section dividing unit that divides a range of values of explanatory variables in the plurality of actually measured values into a plurality of sections for the estimation model statistically obtained based on the plurality of actually measured values,
A section determination unit that determines which section of the plurality of sections the value of the explanatory variable in the estimation target belongs to the estimation target for which the actual measurement value is obtained in advance,
A contribution ratio and a dispersion ratio between the plurality of actual measurement values belonging to the section determined by the section determination unit and an estimation result of the estimation model in the section are calculated, and the larger the dispersion ratio is, the larger the contribution ratio is. A determination unit that determines that the reliability of the estimation model is high, and determines that the reliability of the estimation model is low as the variance ratio is smaller than the contribution rate;
The contribution rate is a value representing a degree of fit of a predicted value calculated by the estimated model with respect to an actual measurement value when the value of the explanatory variable is teacher data used for the construction of the estimated model,
The variance ratio is a reliability determination device that is a value representing a degree of fit of a predicted value calculated by the estimated model with respect to an actual measured value when the value of the explanatory variable is test data used after the estimated model is constructed .

Images