JP2011077287A - Device and method for determining reliability, and computer program for determining reliability - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To determine the reliability of an estimation model. <P>SOLUTION: The reliability decision device includes a section division unit for dividing a range of values of an explanatory variable in a plurality of actually measured values into a plurality of sections on the basis of a distance between the respective actually measured values for the estimation model statistically obtained on the basis of the plurality of actually measured values, a section decision unit for determining which of the plurality of sections the value of the explanatory variable in an estimation object belongs to for the estimation object for which the actually measured values are obtained beforehand, and a decision unit for determining the reliability of the estimation model on the basis of the plurality of actually measured values belonging to the section determined by the section decision unit and estimated results by the estimation model in the section. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for determining the reliability of an estimation model obtained by a statistical method.

  In recent years, efforts have been made to construct a model for estimating a process (hereinafter referred to as an “estimated model”) by performing a statistical method on various sensor data of a process apparatus being processed. Specifically, an estimation model is prepared in advance with teacher data (combination of objective variables and explanatory variables) consisting of actual measurement values using a large number of process evaluation lots. This is a prediction formula obtained by performing multiple regression analysis of multivariate analysis). By using the estimation model, it is possible to judge the machining process result without actually inspecting with the inspection equipment, and also reduce the input of test products, so it will continue to be used in the future. .

  Conventionally, an estimation model constructed at a mass production site is constructed by collecting objective variables and explanatory variables in a certain short period of time. Therefore, during production of products using the estimation model, it is necessary to determine at an appropriate time whether or not the appropriate state of the estimation model can be maintained.

  For example, in an environment where a new product is introduced in one year, data of objective variables and explanatory variables for model generation was collected over half a year or one year. Opportunities should disappear. Therefore, for example, an explanatory variable having a high correlation with the objective variable is selected in a short period such as one month, and an estimation model must be constructed. Therefore, if there is an explanatory variable that fluctuates over a longer period than the period for which the data was collected for constructing the estimation model, or if the relationship of explanatory variables with high correlation changes due to maintenance, etc., the accuracy of the estimation model There is a possibility of lowering. Of course, it is sufficient if the estimation model can be updated to the latest data and reconstructed at any time, but in practice, teacher data is often not easily collected. Therefore, it was necessary to regularly check the reliability of the estimation model itself.

  In the general process estimation model, the reliability of the estimation model is not confirmed, but the difference between the predicted value obtained by the estimation model and the actual measurement value obtained by actual measurement is calculated. An alarm is issued when the threshold is exceeded. For example, Patent Document 1 describes that a model for predicting a polishing time is constructed in a semiconductor CMP (Chemical Mechanical Polishing) apparatus, and an error judgment is made when the difference between the predicted value and the actual measurement value exceeds a threshold value. Has been. Patent Document 2 discloses a method for extracting a correlation between various data.

JP 2008-258510 A JP 2006-86403 A

  However, in the technique disclosed in Patent Document 1, the difference between the predicted value and the actual measurement value is simply viewed, and the reliability of the estimation model is not determined. Therefore, it becomes a threshold setting problem and it is difficult to judge it as a false alarm. In addition, the technique disclosed in Patent Document 2 is intended to find a relationship between data from data collected from various conditions. The reliability of the model cannot be determined.

  In view of the above circumstances, an object of the present invention is to provide a reliability determination apparatus, a reliability determination method, and a reliability determination computer program capable of determining the reliability of an estimation model.

  One aspect of the present invention is a reliability determination apparatus, and for an estimated model statistically obtained based on a plurality of actually measured values, a range of values of explanatory variables in the plurality of actually measured values is calculated. Based on the distance between the section dividing unit that divides into a plurality of sections and the estimation object for which the actual measurement value is obtained in advance, the value of the explanatory variable in the estimation object The reliability of the estimation model is determined based on the section determination unit that determines whether it belongs to the section, the plurality of actually measured values that belong to the section determined by the section determination unit, and the estimation result of the estimation model in the section. A determination unit for determining.

  One aspect of the present invention is a reliability determination method, in which an information processing device calculates a range of values of explanatory variables in a plurality of actually measured values for an estimated model statistically obtained based on a plurality of actually measured values. The section dividing step of dividing into the plurality of sections based on the distance between each measured value, and the information processing apparatus, with respect to the estimated object for which the measured value is obtained in advance, the explanatory variable in the estimated object A section determination step for determining which section of the plurality of sections belongs to the value, the plurality of measured values belonging to the section determined by the section determination step, and the information in the section And a determination step of determining the reliability of the estimation model based on the estimation result of the estimation model.

  One aspect of the present invention is a computer program for reliability determination, and for an estimation model statistically obtained based on a plurality of actual measurement values, ranges of values of explanatory variables in the plurality of actual measurement values are measured. Based on the distance between the values, the section dividing step for dividing into the plurality of sections, and the estimation object for which the actual measurement value is obtained in advance, the value of the explanatory variable in the estimation object is the value of the plurality of sections. The section determination step for determining which section of the sections belongs, the plurality of actual measurement values belonging to the section determined by the section determination step, and the estimation result of the estimation model in the section, the reliability of the estimation model And a determination step for determining the degree of the computer.

  According to the present invention, it is possible to determine the reliability of the estimation model.

It is a schematic block diagram showing the functional structure of a reliability judgment apparatus. It is a figure showing the example of an objective variable, an explanatory variable, and an estimation model. It is a flowchart showing the flow of a process of a reliability judgment apparatus. It is a flowchart showing the flow of the distribution coefficient calculation process by a distribution coefficient calculation part.

  FIG. 1 is a schematic block diagram illustrating a functional configuration of the reliability determination apparatus 1. The reliability determination apparatus 1 includes a CPU (Central Processing Unit) connected via a bus, a memory, an auxiliary storage device, and the like, and executes a reliability determination computer program to thereby obtain an interval division unit 100 and a distribution coefficient calculation unit 101. , Section determination unit 102, determination unit 103, contribution rate calculation unit 104, variance ratio calculation unit 105, comparison unit 106, and output unit 107. Note that all or part of the functions of the reliability determination apparatus 1 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit) or a PLD (Programmable Logic Device).

  The estimation model prepares a plurality of teacher data (combinations of objective variables and explanatory variables) consisting of actual measurement values using a large number of process evaluation lots, and uses a statistical method (multivariate analysis) for a plurality of teacher data. This is a prediction formula obtained by performing multiple regression analysis or the like. The estimation model obtained in this way is used, for example, to represent the characteristics of a process change state such as a non-defective product determination or a wear state while continuing production. Data (hereinafter referred to as “test data”) can be obtained by processing the test evaluation lots with the apparatus at a certain frequency. The test data includes an explanatory variable value for the test evaluation lot and an actual measurement value of the objective variable obtained by actually performing the measurement for the test evaluation lot. The reliability determination apparatus 1 receives an estimation model constructed by a statistical method such as multiple regression analysis of multivariate analysis and all the teacher data and test data used to construct the estimation model as inputs. The reliability of the estimation model is determined based on the test data, and the determination result is output. In order to perform such processing, each functional unit of the reliability determination apparatus 1 operates as follows.

  The section dividing unit 100 executes section dividing processing to divide the range of the explanatory variable values used when the estimation model is generated, thereby forming a plurality of sections. Details of the section division processing will be described later. The distribution coefficient calculation unit 101 calculates a distribution coefficient in each section by executing a distribution coefficient calculation process. Details of the distribution coefficient calculation process will be described later. The section determination unit 102 determines which section of the plurality of sections formed by the section dividing unit 100 the input test data belongs to. The determination unit 103 determines whether there is a sufficient data distribution at the time of generating the estimated model in the section to which the test data belongs. The contribution rate calculation unit 104 calculates the contribution rate of the section to which the test data belongs. The variance ratio calculation unit 105 calculates the variance ratio of the section to which the test data belongs. The comparison unit 106 compares the contribution rate and the variance ratio for the section to which the test data belongs, and determines the reliability. The output unit 107 outputs the determination result by the determination unit 103 or the determination result by the comparison unit 106 to the user of the reliability determination device 1 by the voice output unit or the display unit. Further, the output unit 107 may be configured to output data representing the determination result not to the user but to another device.

  FIG. 2 is a diagram illustrating an example of an objective variable, an explanatory variable, and an estimation model. The objective variable and the explanatory variable serving as the teacher data are black dots in FIG. 2 and are obtained, for example, when the semiconductor manufacturing apparatus processes the process evaluation lot. The estimation model is a straight line in FIG. 2, and is obtained from the objective variable and the explanatory variable by multiple regression analysis of multivariate analysis. In FIG. 2, the number of the objective variable and the explanatory variable is one, but there may be a plurality of types. In addition, the estimation model means an approximate straight line or curve for actual measurement including linear or non-linear.

  In FIG. 2, a plurality of vertically extending wavy lines represent the boundaries of each section. The boundary of each section is determined by the section dividing process of the section dividing unit 100. The width of each section, that is, the length of the section between the boundaries is not uniform and is determined for each section. Also, the number of teacher data included in each section is not uniform, but is determined for each section.

  FIG. 3 is a flowchart showing a process flow of the reliability determination apparatus 1. First, the section dividing unit 100 divides the data range of the explanatory variable of the estimation model into a plurality of sections by executing the section dividing process. Then, the distribution coefficient calculation unit 102 calculates a distribution coefficient by executing a distribution coefficient calculation process for each section (step S101). The sections after the division may be set so as not to overlap each other as shown in FIG. 3, or may be set so that a part of the adjacent sections overlap (overlapping). The distribution coefficient indicates whether or not the number of teacher data at the time of construction of the estimation model is sufficient, and whether or not the teacher data is distributed without being concentrated on one point.

  FIG. 4 is a flowchart showing the flow of the section dividing process and the distribution coefficient calculating process by the section dividing unit 100 and the distribution coefficient calculating unit 101. First, the rank is determined by the user (step S201). The rank is a value that represents an allowable value of the distance between data points included in the same section. As the rank is larger, data points that are far apart are included in the same section, and as the rank is smaller, only data that are closer to each other are included in the same section. For example, a plurality of ranks may be set in advance, and the user may be configured to select one rank from them. The rank value is an integer multiple of the average distance between adjacent data points, an integer multiple of the mode, an integer multiple of the median, the power of the average, the power of the mode, It may be set to a power of the median.

  The section dividing unit 100 selects one reference point and one adjacent point. Specifically, the section division unit 100 selects the reference point in order from the next data point at the end of the previous section (or the first data point if there is no previous section), and further selects the reference point. The next data point in the order of the points is selected as an adjacent point (step S202). A data point is a point representing one teacher data, and is a single black point shown in FIG. Also, the above order may be from the smallest to the largest along the explanatory variable axis, or from the smallest to the largest along the objective variable axis or both axes, The order may be from the largest to the smallest along any axis, or may be in any other order.

  Next, the section dividing unit 100 calculates a distance between the reference point and the adjacent point (hereinafter referred to as “inter-adjacent distance”) (step S203). Next, the section dividing unit 100 determines whether or not the calculated distance between adjacent areas is larger than the rank value selected in step S201 (step S204). If the inter-adjacent distance is equal to or less than the rank value (step S204—NO), the section dividing unit 100 determines that the current processing target reference point and the adjacent point are data points belonging to the same section. Then, each of the reference point and the adjacent point is moved to the next point (step S205). Specifically, the section dividing unit 100 selects a data point that has been an adjacent point as a new reference point, and selects a data point in the next order after the reference point as a new adjacent point. Then, the distribution coefficient calculation unit 101 calculates the inter-adjacent distance between the new reference point and the adjacent point (step S203), and compares it with the rank value (step S204).

  On the other hand, when the adjacent distance is larger than the rank value in the process of step S204 (step S204-YES), the section dividing unit 100 sets the data point at which the current processing target reference point is located at the end of the section. And the boundary line of the section is set between the adjacent points that are currently processed (step S206). Next, the distribution coefficient calculating unit 101 calculates the total distance between adjacent points for the section newly set by the section dividing unit 100 (step S207), and counts the number of data points existing in the section (step S207). S208).

Then, the distribution coefficient calculation unit 101 calculates the distribution coefficient of this section by dividing the total calculated in step S207 by the value obtained by subtracting 1 from the number of data counted in step S208 (step S209). . Specifically, when the cumulative total calculated in step S207 is ΣD and the number of data counted in step S208 is n, the distribution coefficient M in this section is expressed by the following equation.
M = ΣD / (n−1) (Formula 0)

  Next, the section dividing unit 100 determines whether or not the processing of steps S202 to S209 has been performed with all data points as reference points or adjacent points (step S210), and data points that are not selected as reference points or adjacent points. Steps S202 to S209 are repeatedly executed until no more remains (Step S210-NO). When the process returns to step S202, the section dividing unit 100 selects the last selected point as an adjacent point as a new reference point, and further selects the next data point next to the new reference point. Process as a new adjacent point.

  On the other hand, when there is no data point not selected as the reference point or the adjacent point (step S210—YES), the section dividing unit 100 and the distribution coefficient calculating unit 101 finish the section dividing process and the distribution coefficient calculating process. If there is no newly selected adjacent point in the process of step S202, the process jumps to the process of step S206 at that time, and the processes after step S206 are executed.

  When the distribution coefficient calculation process ends, the section determination unit 102 determines which section of the plurality of sections formed by the section division unit 100 the test data belongs to (step S102). Next, the determination unit 103 determines whether or not the distribution coefficient in the section determined in step S102 is greater than or equal to a threshold value (step S103). When the distribution coefficient is less than the threshold value (step S103-NO), that is, when there is not a sufficient number of teacher data at the time of generating the estimated model, the determination unit 103 uses the model because there is little teacher data at the time of generating the estimated model. It is determined that the reliability cannot be determined (step S104). In this case, the output unit 107 outputs the determination result “model reliability determination impossible” by the determination unit 103.

On the other hand, when the distribution coefficient is larger than the threshold value (step S103-YES), that is, when there is a sufficient number of teacher data at the time of generating the estimation model, the contribution rate calculation unit 104 estimates the section determined in step S102. The model contribution rate is calculated (step S105). Specifically, the contribution rate calculation unit 104 calculates the contribution rate of the estimation model according to the following Equation 1. Note that the value of i in Equation 1 represents identification information of each teacher data. Yi _pre represents a predicted value calculated by the estimation model when the explanatory variable is Xi, and is equal to a + bXi. Yi represents an actual measurement value when the explanatory variable is Xi, and is obtained from the teacher data. Y _ave represents an average value of actually measured values. The explanatory variables Xi used in Equation 1 are all of the explanatory variables Xi existing as teacher data that belong to the section determined in step S102. Therefore, Y _ave represents the average value of the actual measurement values of the objective variables in all the teacher data in which the explanatory variable Xi belongs to the section determined in step S102.

  In Formula 1, a value obtained by dividing the cumulative value of the square of the difference between the measured value and the average value of each teacher data belonging to the section determined in step S102 by the cumulative value of the square of the difference between each predicted value and the average value Is obtained as a contribution rate. Here, the contribution rate is a value indicating the fitness of the estimation model. The method of calculating the contribution rate is not limited to the method according to Equation 1, and if a value representing the degree of fit of the predicted value with the actual measurement value can be calculated, for example, the difference value between the actual measurement value and the predicted value is used as the contribution rate. Other calculation methods may be used.

Next, the variance ratio calculation unit 105 calculates the ratio of variance between the predicted value and the actual measurement value of the test evaluation lot with respect to the center of the teacher data when the estimated model is generated in the section determined in step S102. . Specifically, the dispersion ratio calculation unit 105 calculates the dispersion ratio according to the following Equation 2. Yt _pre represents a predicted value calculated by the estimation model when the explanatory variable is the explanatory variable Xt of the test data, and is equal to a + bXt. Yt represents an actual measurement value when the explanatory variable is Xt, and is obtained from test data. Y _ave represents an average value of actually measured values. The explanatory variables Xt used in Equation 2 are all of the explanatory variables Xt existing as test data that belong to the section determined in step S102. Therefore, Y _ave represents the average value of the actual measurement values of the objective variable in all the test data in which the explanatory variable Xt belongs to the section determined in step S102.

  In Expression 2, a value obtained by dividing the cumulative value of the square of the difference between the measured value and the average value of each test data belonging to the section determined in step S102 by the cumulative value of the square of the difference between each predicted value and the average value Is obtained as a dispersion ratio. Here, the dispersion ratio is a value indicating the degree of fitness of the estimation model. The calculation method of the dispersion ratio is not limited to the method according to Equation 2, and if a value representing the degree of fit of the predicted value with the actual measurement value can be calculated, for example, the difference value between the actual measurement value and the predicted value is set as the dispersion ratio. Other calculation methods may be used.

  The comparison unit 106 compares the contribution rate calculated by the contribution rate calculation unit 104 with the dispersion ratio calculated by the dispersion ratio calculation unit 105 (step S107). When the variance ratio is larger than the contribution ratio (step 107-YES), the comparison unit 106 determines that the reliability of the estimation model is high and determines that the estimation model does not need to be reconstructed (step S108). On the other hand, when the variance ratio is equal to or less than the contribution ratio (step 107—NO), the comparison unit 106 determines that the reliability of the estimation model is low and determines that the estimation model needs to be reconstructed (step S109). After the process of step S108 or step S109, the output unit 107 outputs the determination result.

The reliability determination device 1 may be realized as an independent information processing device that is not related to an arithmetic device in an actual manufacturing device or individual manufacturing devices.
According to the reliability determination device 1 configured as described above, it is possible to easily determine the stability of the estimation model once constructed without newly preparing new teacher data based on a new process evaluation lot. This makes it possible to easily confirm the reliability of the estimation model. In particular, since the section dividing unit 100 dynamically determines the section based on the rank value selected by the user, the user selects the accuracy of the reliability determination by appropriately selecting the rank according to the application or environment. It becomes possible to do.

  Moreover, according to the reliability determination apparatus 1, the reliability of the estimation model once constructed is determined for each section of the teacher data, and “determination” is performed without determining the reliability of the estimation model for the section with less teacher data. Since it is “impossible”, it is possible to prevent an erroneous report from being output in a section with a small amount of teacher data.

  In addition, according to the reliability determination apparatus 1, in determining the reliability of the estimation model, the determination is not performed using a lot of teacher data using a lot of process evaluation lots necessary to reconstruct the estimation model. Judgment is made with a small amount of test data using a small number of test evaluation lots. Therefore, it is possible to reduce the number of test evaluation lots to be actually measured, and it is possible to improve the processing efficiency related to the actual product instead of the test evaluation lot.

  Moreover, according to the reliability determination apparatus 1, the following can also be performed. That is, there are cases where an estimation model constructed by selecting a parameter having a high correlation in a short period is constructed without including a correlation parameter that is originally necessary when the parameter is examined in a longer period. However, since it is possible to immediately determine that the reconstruction is necessary when the reliability decreases, the estimation model is constructed by the teacher data obtained in such a short period of time. Can also be used in practice. Therefore, it is possible to avoid a situation in which the production of the corresponding product is finished when the estimation model is completed.

  In addition, as an application example of the reliability determination apparatus 1, a semiconductor apparatus that estimates process contents by using sensor data of the apparatus can be cited.

<Modification>
The comparison unit 106 may determine that it is necessary to reconstruct the estimation model when the number of times the variance ratio of the evaluation lot in step S107 is smaller than the contribution rate is repeated a certain number of times.

  The distribution coefficient calculation unit 101 calculates the inter-adjacent distance if the inter-adjacent distance between a certain reference point and the adjacent point is smaller than a predetermined threshold when calculating the inter-adjacent distance interval total in step S207. Without using it, calculate the inter-adjacent distance using the next point as a new adjacent point without changing the reference point until the inter-adjacent distance exceeding the predetermined threshold is obtained, and only the inter-adjacent distance exceeding the predetermined threshold May be used for the total. In this case, when the number of data is counted in step S208, the distribution coefficient calculation unit 101 may be configured to exclude the number of adjacent points not used for accumulation in step S207 from the count.

In the flowchart of FIG. 4, every time the section dividing unit 100 sets one section, the distribution coefficient calculating unit 101 calculates a distribution coefficient for the section, but the section dividing unit 100 performs steps S202 to S206. It is also possible to set all the sections by repeatedly executing the process, and then calculate the distribution coefficients for all the sections by the distribution coefficient calculating unit 101 repeatedly executing the processes of steps S207 to S209.
The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 1 ... Reliability judgment apparatus, 100 ... Section division part, 101 ... Distribution coefficient calculation part, 102 ... Section determination part, 103 ... Determination part, 104 ... Contribution rate calculation part, 105 ... Dispersion ratio calculation part, 106 ... Comparison part ( Judgment unit), 107 ... output unit

Claims (4)

A section that divides a range of explanatory variable values in the plurality of actual measurement values into the plurality of sections based on a distance between the actual measurement values for an estimation model that is statistically obtained based on the plurality of actual measurement values. A dividing section;
A section determination unit that determines which section of the plurality of sections the value of the explanatory variable in the estimation target belongs to the estimation target for which the actual measurement value is obtained in advance,
A determination unit that determines the reliability of the estimation model based on the plurality of actual measurement values belonging to the section determined by the section determination unit and the estimation result of the estimation model in the section;
A reliability determination apparatus comprising:   The section dividing unit calculates a distance between actually measured values continuously arranged along a predetermined axis, and when the distance is larger than a value selected by a user, a boundary of the section between the actually measured values The reliability determination device according to claim 1, wherein the plurality of sections are divided by setting. For the estimation model statistically obtained based on the plurality of actual measurement values, the information processing apparatus determines the range of the value of the explanatory variable in the plurality of actual measurement values based on the distance between the respective actual measurement values. An interval dividing step to divide into intervals;
A section determination step in which the information processing apparatus determines which section of the plurality of sections the value of the explanatory variable in the estimation object belongs to the estimation object for which an actual measurement value is obtained in advance.
A determination step in which the information processing apparatus determines the reliability of the estimation model based on the plurality of actually measured values belonging to the section determined by the section determination step and an estimation result by the estimation model in the section; ,
A reliability determination method comprising: A section that divides a range of explanatory variable values in the plurality of actual measurement values into the plurality of sections based on a distance between the actual measurement values for an estimation model that is statistically obtained based on the plurality of actual measurement values. Splitting step;
A section determination step for determining which section of the plurality of sections the value of the explanatory variable in the estimation target belongs to the estimation target for which the actual measurement value is obtained in advance;
A determination step of determining the reliability of the estimation model based on the plurality of actual measurement values belonging to the section determined by the section determination step and the estimation result of the estimation model in the section;
A computer program for determining the degree of reliability for causing a computer to execute.

Images