CN116432769A - Model drift detection method and device, electronic equipment and storage medium - Google Patents

Model drift detection method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN116432769A
CN116432769A CN202310120742.7A CN202310120742A CN116432769A CN 116432769 A CN116432769 A CN 116432769A CN 202310120742 A CN202310120742 A CN 202310120742A CN 116432769 A CN116432769 A CN 116432769A
Authority
CN
China
Prior art keywords
data
detection period
target
target model
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310120742.7A
Other languages
Chinese (zh)
Inventor
杨镇恺
吴文超
郑毅贤
王达一
张琪萱
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Siemens Ltd China
Original Assignee
Siemens Ltd China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Siemens Ltd China filed Critical Siemens Ltd China
Priority to CN202310120742.7A priority Critical patent/CN116432769A/en
Publication of CN116432769A publication Critical patent/CN116432769A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Complex Calculations (AREA)

Abstract

The embodiment of the application provides a model drift detection method, a device, electronic equipment and a storage medium, wherein the model drift detection method comprises the following steps: determining a reference detection period of the target model according to the target detection period of the target model, and determining a plurality of sampling data contained in each of the target detection period and the reference detection period, wherein each sampling data in the target detection period and the reference detection period comprises labeled data and unlabeled data; determining a target log-likelihood value of the target detection period and a reference log-likelihood value of the reference detection period according to the prediction results of the target model corresponding to the sampling data in the target detection period and the reference detection period and the label information of the labeled data in the sampling data; and obtaining a model drift detection result of the target model corresponding to the target detection period according to the target log likelihood value of the target detection period and the reference log likelihood value of the reference detection period. The present application may perform model drift detection based on semi-supervised mode.

Description

Model drift detection method and device, electronic equipment and storage medium
Technical Field
The embodiment of the application relates to the field of data processing, in particular to a model drift detection method and device, electronic equipment and a storage medium.
Background
With the wide application of machine learning models in predictive maintenance systems, performance monitoring of machine learning models is becoming increasingly important. The model drift is an important factor for detecting the performance quality of the model.
Model drift (also referred to as model decay) refers to a phenomenon in which the predictive power of a machine learning model is reduced due to changes in the production environment after the model has been deployed in the production environment. Specifically, since the machine learning model is highly dependent on its training data, when the prediction data of the machine learning model changes, for example, environmental changes, seasonal changes, etc., the prediction result of the model also changes accordingly. Therefore, when model drift occurs, the model prediction accuracy is reduced, and periodic optimization updating for the machine learning model is required to ensure the model prediction performance.
In view of this, in order to ensure high performance of the machine learning model, it is necessary to accurately recognize whether there is drift in the model.
Disclosure of Invention
In order to solve the above problems, embodiments of the present application provide a method, an apparatus, an electronic device, and a storage medium for detecting model drift, so as to at least partially solve the above problems.
According to a first aspect of an embodiment of the present application, there is provided a model drift detection method, including: determining a reference detection period of a target model according to the target detection period of the target model, and determining a plurality of sampling data contained in the target detection period and the reference detection period respectively, wherein each sampling data in the target detection period and the reference detection period comprises labeled data and unlabeled data; determining a target log-likelihood value of the target detection period and a reference log-likelihood value of the reference detection period according to the prediction results of the target model corresponding to the sampling data in the target detection period and the reference detection period and the label information of the labeled data in the sampling data; and obtaining a model drift detection result of the target model corresponding to the target detection period according to the target log likelihood value of the target detection period and the reference log likelihood value of the reference detection period.
Optionally, the method further comprises determining each detection period of the target model, comprising: according to each sampling time corresponding to each sampling data, arranging each sampling data in sequence to obtain a sampling data sequence; determining the number of sampling data covered by the sliding window according to the given detection period duration; repeatedly executing the sliding treatment meeting the sliding step length on each sampling data in the sampling data sequence based on a given sliding step length by utilizing the sliding window to obtain each detection period corresponding to each sliding treatment and each sampling data in each detection period; wherein each sample data includes one of device sensing data, environment sensing data.
Optionally, the reference detection period includes a plurality of target detection periods and each reference detection period is determined by: determining each detection period corresponding to each detection period according to each sampling time corresponding to each sampling data in each detection period, and arranging each detection period in sequence according to each detection period; determining a plurality of reference detection periods of the target detection period from among the detection periods according to one target detection period determined from among the detection periods;
Optionally, the number of the plurality of reference detection periods is between 5 reference detection periods and 20 reference detection periods; and each detection period corresponding to each reference detection period is earlier than the detection period of the target detection period, and each reference detection period and the target detection period are sequentially continuous.
Optionally, the target model comprises a classification prediction model; the label information at least comprises first label information and second label information, wherein the first label information is used for identifying the real type of each sampled data, and the first label information of any one labeled data is matched with one of the prediction types output by the target model; the second tag information includes a drift category and a non-drift category for identifying whether each sample data has drift.
Optionally, the determining the target log-likelihood value of the target detection period and the reference log-likelihood value of the reference detection period according to the prediction result of the target model corresponding to each sample data in the target detection period and the reference detection period and the label information of the labeled data in each sample data includes: determining a model parameter value and a prediction probability value of the target model according to the target model corresponding to each piece of tagged data, each prediction category of each piece of untagged data and each piece of tag information corresponding to each piece of tagged data; determining a target log likelihood value of the target detection period according to the target model corresponding to each labeled data in the target detection period, each predicted category of each unlabeled data, each label information corresponding to each labeled data in the target detection period, the model parameter value and the predicted probability value, and determining a reference log likelihood value of each reference detection period according to each labeled data in the target model corresponding to each reference detection period, each predicted category of each unlabeled data, each label information corresponding to each labeled data in each reference detection period, the model parameter value and the predicted probability value.
Optionally, the determining the model parameter value and the prediction probability value of the target model according to the target model corresponding to each tagged data, each prediction category of each untagged data, and each tag information corresponding to each tagged data includes: determining a given initial parameter value as a current parameter value of the target model, and determining a log-likelihood function expected value of the target model according to the initial parameter value, the target model corresponding to each piece of tagged data, each prediction category of each piece of untagged data and each piece of tag information corresponding to each piece of tagged data; a current probability value obtaining step, namely obtaining a current probability value of the target model according to the current parameter value of the target model, the corresponding labeled data of the target model, the corresponding prediction category of the unlabeled data and the corresponding label information of the labeled data; obtaining an updated parameter value of the target model according to the current probability value of the target model, the corresponding prediction category of the target model to each tagged data, each untagged data and each tag information corresponding to each tagged data; updating the current parameter value by using the updated parameter value, and continuing to execute the current probability value acquisition step until a given updating end condition is met; and respectively determining a model parameter value and a predicted probability value of the target model according to the current parameter value and the current probability value which meet the update ending condition.
Optionally, the current probability value obtaining step includes: for each tagged data, obtaining a current probability value of the target model according to the current parameter value of the target model, each prediction category of the target model corresponding to each tagged data, and tag information of each tagged data; and for each piece of unlabeled data, obtaining a current probability value of the target model according to the current parameter value of the target model and each prediction category of the target model corresponding to each piece of unlabeled data.
Optionally, for each tagged data, the obtaining the current probability value of the target model according to the current parameter value of the target model, each prediction category of the target model corresponding to each tagged data, and the tag information of each tagged data includes: obtaining a current probability value of the target model according to the current parameter value of the target model, each prediction category of the target model corresponding to each tagged data and tag information of each tagged data by using a first unit probability conversion formula;
the first unit probability conversion formula is expressed as:
Figure BDA0004081264250000031
wherein, p represents the current probability value of the target model, K represents the kth tag class, K represents the total number of the tag classes, and x j Representing the target model corresponding to the jth tagged dataMeasuring class, z j Tag information representing jth tagged data, the L representing a total number of the tagged data, the θ representing a current parameter value of the target model;
optionally, for each unlabeled data, the obtaining the current probability value of the target model according to the current parameter value of the target model and each prediction category of the target model corresponding to each unlabeled data includes: obtaining a current probability value of the target model according to the current parameter value of the target model and each prediction category of the target model corresponding to each unlabeled data by using a second unit probability conversion formula;
the second unit probability conversion formula is expressed as:
Figure BDA0004081264250000032
wherein, p represents the current probability value of the target model, K represents the kth tag class, K represents the total number of the tag classes, and x i Indicating that the target model corresponds to a predictive category of ith unlabeled data, the U indicating a total number of the unlabeled data, the θ indicating a current parameter value of the target model.
Optionally, the obtaining the updated parameter value of the target model according to the current probability value of the target model, the target model corresponding to each tagged data, each prediction category of each untagged data, and each tag information corresponding to each tagged data includes: obtaining an updated parameter value of the target model according to a current probability value of the target model, each piece of tagged data corresponding to the target model, each piece of prediction category of each piece of untagged data, and each piece of tag information corresponding to each piece of tagged data by using a first model parameter updating formula and a second model parameter updating formula determined by the expected value of the log likelihood function;
The first model parameter update formula is expressed as:
Figure BDA0004081264250000041
the second model parameter update formula is expressed as:
Figure BDA0004081264250000042
wherein p represents the current probability value of the target model, k represents the kth tag class, N represents the total number of tagged data and untagged data, and x i Representing a predicted class of the target model corresponding to the ith unlabeled data, the x j Representing a predicted class of the target model corresponding to the jth tagged data, the z j Tag information representing the j-th tagged data; the θ represents an updated parameter value of the object model.
Optionally, the target log-likelihood value or the reference log-likelihood value is determined by: determining a log likelihood value of the current detection period according to each labeled data, each prediction category of each unlabeled data, each label information corresponding to each labeled data in the current detection period, the model parameter value and the prediction probability value in the current detection period of the target model corresponding to each labeled data in the current detection period by using a log likelihood function conversion formula for any one of the current detection period and each reference detection period;
The log likelihood function conversion formula is expressed as:
Figure BDA0004081264250000043
wherein said l 1 Log likelihood values representing the current detection period, θ representing the model parameter values, X representing the prediction categories of the target model corresponding to the labeled data and unlabeled data in the current detection period, and Z representing the correspondence of the labeled data in the current detection periodThe p represents the predictive probability value, the x i Representing a predicted class of the target model corresponding to the ith unlabeled data in the current detection period, the x j Representing a prediction category of the target model corresponding to a j-th tagged data in the current detection period, the U representing a total number of untagged data in the current detection period, the L representing a total number of tagged data in the current detection period, the z j Tag information representing the j-th tagged data in the current detection period.
Optionally, the obtaining a model drift detection result of the target model corresponding to the target detection period according to the target log likelihood value of the target detection period and the reference log likelihood value of the reference period includes: determining a normal distribution value according to each reference log likelihood value corresponding to each reference detection period; determining a log-likelihood deviation value of the target model corresponding to the target detection period according to the target log-likelihood value of the target detection period and the normal distribution value; and according to the log-likelihood deviation value and a given normal threshold range, if the log-likelihood deviation value does not fall into the normal threshold range, obtaining a detection result that the target model has drift in the target detection period, and if the log-likelihood deviation value falls into the normal threshold range, obtaining a detection result that the target model has no drift in the target detection period.
According to a second aspect of embodiments of the present application, there is provided a model drift detection apparatus, including: the period determining module is used for determining a reference detection period of the target model according to the target detection period of the target model and determining a plurality of sampling data contained in the target detection period and the reference detection period respectively, wherein each sampling data in the target detection period and the reference detection period comprises labeled data and unlabeled data; the calculation module is used for determining a target log-likelihood value of the target detection period and a reference log-likelihood value of the reference detection period according to the prediction results of the target model corresponding to the sampling data in the target detection period and the reference detection period and the label information of the labeled data in the sampling data; and the detection module is used for obtaining a model drift detection result of the target model corresponding to the target detection period according to the target log likelihood value of the target detection period and the reference log likelihood value of the reference detection period.
According to a third aspect of embodiments of the present application, there is provided an electronic device, including: the device comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is configured to store at least one executable instruction, where the executable instruction causes the processor to execute an operation corresponding to the model drift detection method described in the first aspect.
According to a fourth aspect of embodiments of the present application, there is provided a computer storage medium having stored thereon a computer program which, when executed by a processor, implements the model drift detection method according to the first aspect.
According to the model drift detection scheme provided by the embodiment of the application, the accuracy of the model drift detection result can be effectively improved by comparing the log likelihood values of the target model corresponding to different detections. Moreover, the scheme combines and utilizes the labeled data and the unlabeled data, realizes the detection processing of the model drift based on a semi-supervision mode, and can reduce the label marking cost and the detection cost of the model drift.
According to the model drift detection scheme provided by the embodiment of the application, according to the sampling time of each sampling data, each detection period of the target model is determined by utilizing a sliding window technology, so that the number of the sampling data in each detection period is ensured to be kept the same, and the accuracy of a model drift detection result is improved.
According to the model drift detection scheme provided by the embodiment of the application, the plurality of reference detection periods which are positioned in the front of the target detection period and are mutually continuous in each detection period are determined according to the target detection period determined in each detection period, so that the objectivity of the model drift detection result is improved, and the accuracy of the detection result is improved.
According to the model drift detection scheme provided by the embodiment of the application, the second label information for identifying each sampling data as the drift type or the non-drift type is added to the label information of the labeled data, so that the model drift detection in the semi-supervision mode can be realized.
According to the model drift detection scheme provided by the embodiment of the application, the model parameter value and the prediction probability value of the target model are firstly determined, and then the log likelihood value of the target model corresponding to each detection period is calculated according to the determined model parameter value and the determined prediction probability value, so that the accuracy of the calculation result of the log likelihood value of each detection period is improved, and the accuracy of the model drift detection result is improved.
According to the model drift detection scheme provided by the embodiment of the application, the labeled data and the unlabeled data are simultaneously used by utilizing the semi-supervised maximum expected algorithm, and the model parameter value and the prediction probability value of the target model are iteratively optimized, so that the data labeling cost can be effectively reduced, and the accuracy of the model drift detection result can be improved.
According to the model drift detection scheme provided by the embodiment of the application, the data characteristics and the data labels of the target model corresponding to each detection period are observed, so that the log likelihood function of the target model corresponding to each detection period can be accurately determined based on the semi-supervision mode, and the accuracy of the model drift detection result is improved.
According to the model drift detection scheme provided by the embodiment of the application, whether the target model has a drift phenomenon in the current detection period can be accurately estimated by judging whether the log likelihood deviation value of the target detection period relative to each reference detection period falls into the normal threshold range.
Drawings
The following drawings are only for purposes of illustration and explanation of the present application and are not intended to limit the scope of the present application. Wherein,,
fig. 1 is a process flow diagram of a model drift detection method according to an exemplary embodiment of the present application.
Fig. 2 is a process flow diagram of a model drift detection method according to another exemplary embodiment of the present application.
Fig. 3 is a process flow diagram of a model drift detection method according to another exemplary embodiment of the present application.
Fig. 4 is a process flow diagram of a model drift detection method according to another exemplary embodiment of the present application.
Fig. 5 is a process flow diagram of a model drift detection method according to another exemplary embodiment of the present application.
Fig. 6 is a block diagram of a model drift detection device according to an exemplary embodiment of the present application.
Fig. 7 is a block diagram of an electronic device according to an exemplary embodiment of the present application.
Reference numerals illustrate:
600. model drift detection means; 602. a period determination module; 604. a computing module; 606. a detection module; 700. an electronic device; 702. a processor; 704. a communication interface; 706. a memory; 708. a communication bus; 710. computer program.
Detailed Description
For a clearer understanding of technical features, objects, and effects of embodiments of the present application, a specific implementation of embodiments of the present application will be described with reference to the accompanying drawings.
Model drift detection is critical in order to ensure high performance of machine learning models. Current model drift detection mainly includes two methods: a distribution-based detection method and a performance-based detection method. The distribution-based detection method is mainly implemented by identifying the distribution of detected data, for example, using statistics such as mean, variance, and classification imbalance to evaluate the change of the data distribution; performance-based detection methods are mainly implemented by using indexes (e.g., indexes of accuracy, error rate, etc.) that evaluate the performance of a model.
While most model-based drift detection methods can lead to higher accuracy, most of these detection methods rely on tag information of the data. However, in the conventional industrial predictive maintenance operation, unlabeled detection data still occupies a great majority, so that a great deal of manpower is required to perform the marking operation of the detection data, which results in the problem that the conventional model drift detection operation has high cost.
In view of this, the application provides a model drift detection scheme, which can realize the detection operation of model drift based on a semi-supervision mode, thereby greatly reducing the model drift detection cost.
Fig. 1 shows a process flow of a model drift detection method according to an exemplary embodiment of the present application. As shown in the figure, this embodiment mainly includes the following steps:
step S102, determining a reference detection period of the target model according to the target detection period of the target model, and determining a plurality of sampling data contained in each of the target detection period and the reference detection period.
Optionally, the target model comprises a classification prediction model. For example, the target model may be a PDM (product data management) model applied in an industrial predictive maintenance scenario.
Alternatively, the sampled data may include, but is not limited to, device sense data, environmental sense data, and the like.
In this embodiment, each sample data in the target detection period and the reference detection period includes labeled data and unlabeled data.
In this embodiment, the object model may perform classification prediction based on the sampled data, and output a classification prediction result. For example, the target model may obtain class prediction results of the device operating state based on sampled data of the device sensed data, the environment sensed data, and the like. For example, the classification prediction result output by the target model may only include two categories of "normal" and "abnormal", and for example, the classification prediction result output by the target model may also include multiple categories of "normal", "primary alarm", "secondary alarm", "tertiary alarm", and the like, which is not limited in this application.
Optionally, the present embodiment further includes determining each detection period of the target model, and determining the target detection period and the reference detection period of the target model according to the determined each detection period.
Alternatively, a sliding window technique may be utilized to determine each detection period of the target model.
Optionally, the reference detection period includes a plurality of reference detection periods.
Optionally, the target detection period and each reference detection period of the target model are determined by:
according to each sampling time corresponding to each sampling data in each detection period, each detection period corresponding to each detection period is determined, each detection period is arranged in sequence according to each detection period, and a plurality of reference detection periods in each detection period are determined according to one target detection period determined from each detection period.
In this embodiment, each sample data may be time series data acquired at intervals (e.g. acquired by a sensor), so that each sample data has a corresponding sample time.
Specifically, for any one current detection period in each detection period, determining the detection origin-destination time of the current detection period according to each sampling time corresponding to each sampling data in the current detection period, and determining the detection period of the current detection period according to the detection origin-destination time of the current detection period.
Alternatively, the target detection period in each detection period may be determined according to a given factor. For example, given factors for determining a target detection period may include, but are not limited to: production cycle, production lot, data trend (e.g., data change severity), etc., which is not limiting in this application.
Alternatively, the number of reference detection periods may be between 5 reference detection periods and 20 reference detection periods. That is, the ratio between the data amount of the target sample data covered in the target detection period and the data amount of all the reference sample data covered in all the reference periods may be between 1:5 to 1:20, but not limited thereto, and those skilled in the art can adjust the above ratio range based on actual detection requirements.
In this embodiment, each detection period of each reference detection period is earlier than the detection period of the target detection period, and each reference detection period and the target detection period are consecutive in sequence.
Step S104, determining the target log-likelihood value of the target detection period and the reference log-likelihood value of the reference detection period according to the prediction results of the target model corresponding to the sampling data in the target detection period and the reference detection period and the label information of the labeled data in the sampling data.
In this embodiment, the tag information of the tagged data includes at least a first tag information and a second tag information.
The first tag information is used for identifying the real type of each sampled data, and the first tag information of any one tagged data is matched with one of the prediction types output by the target model, that is, each real type contained in the first tag information corresponds to each prediction type output by the target model. For example, if the prediction result output by the target model includes two categories of "normal" and "abnormal", the real category of the first tag information includes two categories of "normal" and "abnormal"; if the predicted result output by the target model comprises three categories of normal, low-level alarm and high-level alarm, the real category of the first tag information comprises three categories of normal, low-level alarm and high-level alarm.
The second tag information is used for identifying whether each sampled data is of a drift type or a non-drift type, so that model drift detection processing in a semi-supervision mode is realized.
As can be seen from the above, the tag information of the tagged data in the present application adds the drift type tag based on the output type of the target model, that is, if the predicted output result of the target model includes M predicted types, the tag information of the tagged data includes m+1 predicted types.
Specifically, a model parameter value and a prediction probability value of the target model can be determined according to a class prediction result of the target model corresponding to each sample data and label information of labeled data in each sample data; determining a target log likelihood value of the target detection period according to the model parameter value, the prediction probability value, the prediction result of the target model corresponding to each sampling data in the target detection period and each label information corresponding to each labeled data in the target detection period; and determining the reference log likelihood value of each reference detection period according to the model parameter value, the prediction probability value, the prediction result of the target model corresponding to the sampling data in each reference detection period and the label information corresponding to the labeled data in each reference detection period.
Step S106, obtaining a model drift detection result of the target model corresponding to the target detection period according to the target log likelihood value of the target detection period and the reference log likelihood value of the reference detection period.
Alternatively, the normal distribution value of each reference detection period may be determined according to each reference log likelihood value corresponding to each reference detection period, and the model drift detection result of the target model corresponding to the target detection period with or without drift may be obtained according to the target log likelihood value of the target detection period and the normal distribution value of each reference detection period.
In summary, according to the model drift detection scheme provided by the embodiment, the accuracy of the model drift detection result can be effectively improved by comparing the log likelihood values of the target model corresponding to different detection periods.
Furthermore, the model drift detection scheme provided by the embodiment of the application combines the labeled data and the unlabeled data to realize the model drift detection processing in a semi-supervision mode, so that the labeling cost of the sampled data can be effectively reduced. Specifically, in the embodiment, by adding the second tag information for identifying whether the drift exists in each piece of sampled data in the tag information of the tagged data, the model drift detection in the semi-supervision mode can be realized, so that the detection cost of the model drift is reduced.
In addition, according to the model drift detection scheme provided by the embodiment of the application, a plurality of reference detection periods which are earlier than the target detection period and are mutually continuous are determined according to the target detection period determined in each detection period, so that the objectivity of a model drift detection result is improved, and the accuracy of the detection result is improved.
Fig. 2 shows a process flow of a model drift detection method according to another exemplary embodiment of the present application. The present embodiment mainly shows a determination implementation means of each detection period of the target model, which mainly includes the following steps:
Step S202, according to each sampling time corresponding to each sampling data, each sampling data is arranged in sequence to obtain a sampling data sequence.
Alternatively, each sample data may be time series data acquired at intervals (e.g., by a sensor) such that each sample data has a corresponding sample time.
In this embodiment, the sampling data may be sequentially arranged in order from early to late according to the sampling times, and a sampling data sequence including the sampling data may be generated.
Step S204, according to the given detection period duration, the number of sampling data covered by the sliding window is determined.
Alternatively, the detection period duration may be determined based on a given factor. In this embodiment, the given factors for determining the detection period duration may include, but are not limited to: production cycle, production lot, data trend (e.g., data change severity), etc., which is not limiting in this application.
In this embodiment, the window length of the sliding window may be determined according to the detection period duration, and the number of the sample data covered in each detection period may be determined according to the window length.
Step S206, repeatedly executing the sliding process meeting the sliding step length on each sample data in the sample data sequence based on the given sliding step length by utilizing the sliding window, and obtaining each detection period corresponding to each sliding process and each sample data in each detection period.
Specifically, the sliding window may repeatedly perform sliding processing on each sample data in the sample data sequence multiple times based on a given sliding step length, so as to determine each detection period corresponding to each sliding processing, and determine each sample data in each detection period according to each sample data covered in the sliding window after each sliding processing is performed.
The sliding step length of the sliding window can be adjusted at will according to actual requirements of detection precision, data sampling frequency and the like. Optionally, the sliding step length of the sliding window may be equal to the window length of the sliding window, so that sampling data contained in two adjacent detection periods do not overlap; alternatively, the sliding step size of the sliding window may be smaller than the window length of the sliding window, such that the sample data contained in adjacent two detection periods partially overlap.
In summary, according to the model drift detection scheme provided in the embodiment, according to the sampling time of each sampling data, each detection period of the target model is determined by using the sliding window technology, so as to ensure that the number of sampling data in each detection period remains the same, thereby improving the objectivity and accuracy of the model drift detection result.
Fig. 3 shows a process flow of a model drift detection method according to another exemplary embodiment of the present application. The embodiment mainly shows the implementation means of the step S104, which mainly includes the following steps:
step S302, determining model parameter values and prediction probability values of the target model according to the label information corresponding to the labeled data, the prediction categories of the unlabeled data and the labeled data of the target model.
Alternatively, the model parameter value (θ value) and the prediction probability value (p value) of the target model may be determined using a maximum expectation algorithm (EM algorithm) from the target model corresponding to each tagged data, each prediction category of each untagged data, each tag information corresponding to each tagged data.
Step S304, determining a target log-likelihood value of the target detection period according to the target model corresponding to the labeled data, the unlabeled data, the prediction category, the label information, the model parameter value and the prediction probability value of the labeled data in the target detection period, and determining a reference log-likelihood value of each reference detection period according to the target model corresponding to the labeled data, the unlabeled data, the prediction category, the label information, the model parameter value and the prediction probability value of the labeled data in each reference detection period.
Alternatively, the target log-likelihood value or the reference log-likelihood value may be determined by:
for any one current detection period of the target detection period and each reference detection period, a log likelihood function conversion formula is utilized to determine a log likelihood value of the current detection period according to each labeled data, each prediction category of each unlabeled data, each label information, a model parameter value and a prediction probability value corresponding to each labeled data in the current detection period, which correspond to the target model.
In this embodiment, the log likelihood function conversion formula can be expressed as the following formula 1:
Figure BDA0004081264250000111
in the above formula 1, l 1 The log likelihood value of the current detection period is represented, θ represents the model parameter value of the target model, X represents the prediction category of each labeled data and each unlabeled data in the current detection period, Z represents each label information corresponding to each labeled data in the current detection period, p represents the prediction probability value of the target model, X i Indicating the predictive category, x, of the target model corresponding to the ith unlabeled data in the current detection period j Representing the predictive category of the target model corresponding to the jth tagged data in the current detection period, U representing the total number of untagged data in the current detection period, z j Label information representing the j-th labeled data in the current detection period, and L represents the total number of labeled data in the current detection period.
Alternatively, the above formula 1 is also expressed as the following formula 2:
Figure BDA0004081264250000112
in the above formula 2, l 0 Representing the respective failures in the current detection periodLog likelihood value of tag data.
In summary, according to the model drift detection scheme provided in this embodiment, the model parameter value and the prediction probability value of the target model are determined first, and then the processing mechanism of the log likelihood value of the target model corresponding to each detection period (including the target detection period and each reference detection period) is calculated according to the determined model parameter value and the determined prediction probability value, so that the model drift detection result with higher accuracy can be obtained.
In addition, according to the model drift detection scheme provided by the embodiment, the log likelihood function of the target model corresponding to each detection period can be determined in the semi-supervision mode by observing the data characteristics and the data labels of the target model corresponding to each detection period, so that the model drift detection cost is reduced.
Fig. 4 shows a process flow of a model drift detection method according to another exemplary embodiment of the present application. The embodiment mainly shows the implementation means of the step S302, which mainly includes the following steps:
Step S402, determining the given initial parameter value as the current parameter value of the target model, and determining the expected value of the log-likelihood function of the target model according to the initial parameter value, the label information corresponding to the target model, the prediction category of the unlabeled data, and the labeled data.
Specifically, a person skilled in the art can set an initial value for the current parameter value of the target model according to the actual detection requirement.
In this embodiment, the expected value of the log likelihood function of the target model may be determined according to the initial parameter value, the target model corresponding to each labeled data, each prediction class of each unlabeled data, and each label information corresponding to each labeled data by using an expected value conversion formula.
Wherein the expected value conversion formula may be expressed as the following formula 3:
Figure BDA0004081264250000121
in the above equation 3, E represents the expected value of the log likelihood function, θ old For the initial parameter value of the target model, X represents the predicted category of each labeled data and each unlabeled data in the current detection period, Z represents each label information corresponding to each labeled data in the current detection period, p represents the predicted probability value of the target model, and X i Indicating the predictive category, x, of the target model corresponding to the ith unlabeled data in the current detection period j Representing the predictive category of the target model corresponding to the jth tagged data in the current detection period, U representing the total number of untagged data in the current detection period, z j Label information representing the j-th labeled data in the current detection period, wherein L represents the total number of labeled data in the current detection period, K represents the K-th label category, and K represents the total number of categories of label categories.
In the present embodiment, θ in the above formula 3 new Can be found from the hypothetical parameter distribution. For example, θ can be found assuming that the parameters X, Z, K in the above formula 3 (expected value conversion formula) conform to a gaussian distribution new 。
Step S404, obtaining the current probability value of the target model according to the current parameter value of the target model, the prediction categories of the target model corresponding to the labeled data and the unlabeled data, and the label information corresponding to the labeled data.
Optionally, for each tagged data, a current probability value of the target model may be obtained according to a current parameter value of the target model, each prediction category of the target model corresponding to each tagged data, and tag information of each tagged data; for each unlabeled data, a current probability value of the target model may be obtained based on the current parameter value of the target model, each prediction category of the target model corresponding to each unlabeled data.
In this embodiment, each labeled data and each unlabeled data used for calculating the current probability value may be selected from each sampling data in the target detection period and each reference detection period.
In this embodiment, the first unit probability conversion formula may be used to obtain the current probability value of the target model according to the current parameter value of the target model, each prediction category of the target model corresponding to each tagged data, and the tag information of each tagged data.
Wherein the first unit probability conversion formula may be expressed as the following formula 4:
Figure BDA0004081264250000131
in the above formula 4, p represents the current probability value of the object model (which may also be referred to as the current probability value of the object model corresponding to each tag class), K represents the kth tag class, K represents the total number of the tag classes, and x j Representing the predictive category, z, of the target model corresponding to the jth tagged data j Label information indicating the j-th labeled data, L indicating the total number of labeled data, and θ indicating the current parameter value of the target model.
In this embodiment, the second unit probability conversion formula may be used to obtain the current probability value of the target model according to the current parameter value of the target model and each prediction category of the target model corresponding to each unlabeled data.
Wherein the second unit probability conversion formula is expressed as the following formula 5:
Figure BDA0004081264250000132
in the above formula 5, p represents the current probability value of the object model (which may also be referred to as the current probability value of the object model corresponding to each tag class), K represents the kth tag class, K represents the total number of the tag classes, and x i Representing the predicted class of the object model corresponding to the ith unlabeled data, U representing the total number of unlabeled data, θ representing the current parameter value of the object model.
In summary, the present embodiment performs the step E in the EM algorithm to perform the optimization update for the current probability value (p value) of the target model based on the current parameter value (θ value) of the target model.
Step S406, the updated parameter value of the target model is obtained according to the current probability value of the target model, the prediction categories of the target model corresponding to the labeled data and the unlabeled data, and the label information corresponding to the labeled data.
In this embodiment, the updated parameter values of the target model may be obtained according to the current probability value of the target model, the corresponding label information of the target model corresponding to each labeled data, each prediction category of each unlabeled data, and each label information corresponding to each labeled data by using the first model parameter updating formula and the second model parameter updating formula determined by the expected value of the log likelihood function.
In this embodiment, the derivative operation may be performed by the expected value conversion formula (formula 3), and the value after the derivative is equal to 0, so as to obtain the first model parameter updating formula and the second model parameter updating formula.
The first model parameter update formula may be expressed as the following formula 6:
Figure BDA0004081264250000141
the second model parameter update formula may be expressed as formula 7:
Figure BDA0004081264250000142
in the above formulas 6 and 7, p represents the current probability value of the object model, k represents the kth tag class, N represents the total number of each tagged data and each untagged data, and x is the number of tag classes i Representing a predicted class of the target model corresponding to the ith unlabeled data, said x j Representing the predictive category, z, of the target model corresponding to the jth tagged data j Tag information representing the j-th tagged data; θ represents the updated parameter values of the target model.
In summary, the present embodiment performs the optimization update for the current parameter value (θ value) of the target model based on the current probability value (p value) of the target model by performing the M steps in the EM algorithm.
Step S408, determining whether the difference between the current parameter value and the updated parameter value of the target model satisfies the given update end condition, if not, executing step S410, and if so, executing step S412.
Alternatively, the determination result satisfying the update end condition may be obtained when the difference between the target model update parameter value (i.e., the model parameter value acquired after performing step S406) and the current parameter value (i.e., the model parameter value employed when performing step S404) is smaller than a given difference threshold.
In the present embodiment, when θ i+1 -θ i When the value is less than epsilon, a judgment result meeting the update end condition is obtained, if the value is less than theta i+1 -θ i And (5) the I is not less than epsilon, and a judgment result that the updating ending condition is not met is obtained.
Wherein θ i+1 Representing the updated parameter values of the object model (i.e., the iteratively updated parameter values), θ i Representing the current parameter value of the target model (i.e., the parameter value prior to the iterative update), epsilon may be set to a small positive number (e.g., indicating that the change in model parameters is very small after one iterative update.
In the present embodiment, the difference threshold (. Epsilon.) may be set to 1e -15 (i.e., 10 to the power of-15).
Step S410, update the current parameter value with the updated parameter value, and execute step S404.
Specifically, the current parameter value is updated with the updated parameter value, and the step S404 is executed back to continue to execute updating optimization with respect to the current probability value (p value) of the target model based on the optimized updated current parameter value.
Step S412, determining a model parameter value and a predicted probability value of the target model according to the current parameter value and the current probability value satisfying the update end condition, respectively.
Specifically, the model parameter value and the predicted probability value of the target model may be determined using the current parameter value and the current probability value that satisfy the update end condition, respectively, for performing the subsequent log likelihood function calculation step (refer to step S304).
In summary, according to the model drift detection scheme provided by the embodiment, the labeled data and the unlabeled data are combined to iteratively optimize the model parameter value and the prediction probability value of the target model by using the semi-supervised maximum expected algorithm, so that the data labeling cost can be effectively reduced, and the accuracy of the model drift detection result can be improved.
Fig. 5 shows a process flow of a model drift detection method according to another exemplary embodiment of the present application. The embodiment mainly shows the implementation means of the step S106, which mainly includes the following steps:
step S502, determining a normal distribution value of each reference detection period according to each reference log likelihood value corresponding to each reference detection period.
Specifically, the average value and standard deviation of each reference detection period may be determined according to each reference log-likelihood value corresponding to each reference detection period.
Step S504, according to the target log likelihood value and the normal distribution value of the target detection period, determining the log likelihood deviation value of the target model corresponding to the target detection period.
In this embodiment, a log-likelihood deviation value conversion formula may be used to calculate a log-likelihood deviation value of the target model corresponding to the target detection period according to the target log-likelihood value of the target detection period, the average value and the standard deviation of each reference detection period.
Wherein the log likelihood deviation value conversion formula can be expressed as the following formula 6:
Figure BDA0004081264250000151
in the above formula 6, e represents a log-likelihood deviation value of the target model corresponding to the target detection period, L1 represents a target log-likelihood value of the target detection period, L2 represents an average value of each reference detection period, and σ represents a standard deviation of each reference detection period.
Step S506, obtaining a model drift detection result of the target model corresponding to the target detection period according to the log-likelihood deviation value and the given normal threshold range.
Specifically, according to the log-likelihood deviation value and the normal threshold range, if the log-likelihood deviation value does not fall into the normal threshold range, a detection result that the target model has drift in the target detection period is obtained, and if the log-likelihood deviation value falls into the normal threshold range, a detection result that the target model has no drift in the target detection period is obtained.
In this embodiment, the normal threshold range may be set based on factors such as the prediction type of the target model, the data type of the sampled data, and the like, which is not limited in this application.
In summary, according to the model drift detection scheme provided in the embodiment, by judging whether the log likelihood deviation value of the target detection period relative to each reference detection period falls within the normal threshold range, whether the target model has a drift phenomenon in the current detection period can be accurately estimated.
Fig. 6 shows a block diagram of a model drift detection device according to an exemplary embodiment of the present application. As shown in the figure, the model drift detection apparatus 600 of the present embodiment mainly includes a period determination module 602, a calculation module 604, and a detection module 606.
The period determining module 602 is configured to determine a reference detection period of the target model according to a target detection period of the target model, and determine a plurality of sample data included in each of the target detection period and the reference detection period, where each sample data in the target detection period and the reference detection period includes labeled data and unlabeled data.
A calculation module 604 for determining a target log-likelihood value of the target detection period and a reference log-likelihood value of the reference detection period according to the prediction result of the target model corresponding to each sample data in the target detection period and the reference detection period and the label information of the labeled data in each sample data
And the detection module 606 is configured to obtain a model drift detection result of the target model corresponding to the target detection period according to the target log likelihood value of the target detection period and the reference log likelihood value of the reference detection period.
Optionally, the period determining module 602 is further configured to: according to each sampling time corresponding to each sampling data, arranging each sampling data in sequence to obtain a sampling data sequence; determining the number of sampling data covered by the sliding window according to the given detection period duration; and repeatedly executing the sliding process meeting the sliding step length on each sampling data in the sampling data sequence based on a given sliding step length by utilizing the sliding window to obtain each detection period corresponding to each sliding process and each sampling data in each detection period.
Optionally, each sample data includes one of device sensing data, environment sensing data.
Optionally, the period determining module 602 is further configured to: determining each detection period corresponding to each detection period according to each sampling time corresponding to each sampling data in each detection period, and arranging each detection period in sequence according to each detection period; a plurality of reference detection periods of the target detection period are determined from among the detection periods, based on one target detection period determined from among the detection periods.
Optionally, the number of the plurality of reference detection periods is between 5 reference detection periods and 20 reference detection periods.
Optionally, each detection period corresponding to each reference detection period is earlier than the detection period of the target detection period, and each reference detection period is sequentially continuous with the target detection period.
Optionally, the target model comprises a classification prediction model; the label information at least comprises first label information and second label information, wherein the first label information is used for identifying the real type of each sampled data, and the first label information of any one labeled data is matched with one of the prediction types output by the target model; the second tag information includes a drift category and a non-drift category for identifying whether each sample data has drift.
Optionally, the computing module 604 is further configured to: determining a model parameter value and a prediction probability value of the target model according to the target model corresponding to each piece of tagged data, each prediction category of each piece of untagged data and each piece of tag information corresponding to each piece of tagged data; determining a target log likelihood value of the target detection period according to the target model corresponding to each labeled data in the target detection period, each predicted category of each unlabeled data, each label information corresponding to each labeled data in the target detection period, the model parameter value and the predicted probability value, and determining a reference log likelihood value of each reference detection period according to each labeled data in the target model corresponding to each reference detection period, each predicted category of each unlabeled data, each label information corresponding to each labeled data in each reference detection period, the model parameter value and the predicted probability value.
Optionally, the computing module 604 is further configured to: determining a given initial parameter value as a current parameter value of the target model, and determining a log-likelihood function expected value of the target model according to the initial parameter value, the target model corresponding to each piece of tagged data, each prediction category of each piece of untagged data and each piece of tag information corresponding to each piece of tagged data; a current probability value obtaining step, namely obtaining a current probability value of the target model according to the current parameter value of the target model, the corresponding labeled data of the target model, the corresponding prediction category of the unlabeled data and the corresponding label information of the labeled data; obtaining an updated parameter value of the target model according to the current probability value of the target model, the corresponding prediction category of the target model to each tagged data, each untagged data and each tag information corresponding to each tagged data; updating the current parameter value by using the updated parameter value, and continuing to execute the current probability value acquisition step until the difference value between the updated parameter value and the current parameter meets a given update ending condition; and respectively determining a model parameter value and a predicted probability value of the target model according to the current parameter value and the current probability value which meet the update ending condition.
Optionally, the computing module 604 is further configured to: for each tagged data, obtaining a current probability value of the target model according to the current parameter value of the target model, each prediction category of the target model corresponding to each tagged data, and tag information of each tagged data; and for each piece of unlabeled data, obtaining a current probability value of the target model according to the current parameter value of the target model and each prediction category of the target model corresponding to each piece of unlabeled data.
Optionally, the computing module 604 is further configured to: obtaining a current probability value of the target model according to the current parameter value of the target model, each prediction category of the target model corresponding to each tagged data and tag information of each tagged data by using a first unit probability conversion formula;
the first unit probability conversion formula is expressed as:
Figure BDA0004081264250000181
wherein, p represents the current probability value of the target model, K represents the kth tag class, K represents the total number of the tag classes, and x j Representing a predicted class of the target model corresponding to the jth tagged data, the z j Label information representing jth labeled data, L representing a total number of the labeled data, and θ representing a current parameter value of the target model.
Optionally, the computing module 604 is further configured to: obtaining a current probability value of the target model according to the current parameter value of the target model and each prediction category of the target model corresponding to each unlabeled data by using a second unit probability conversion formula;
the second unit probability conversion formula is expressed as:
Figure BDA0004081264250000182
wherein the method comprises the steps ofThe p represents the current probability value of the target model, the K represents the kth tag class, the K represents the total number of the tag classes, and the x i Indicating that the target model corresponds to a predictive category of ith unlabeled data, the U indicating a total number of the unlabeled data, the θ indicating a current parameter value of the target model.
Optionally, the computing module 604 is further configured to: obtaining an updated parameter value of the target model according to a current probability value of the target model, each piece of tagged data corresponding to the target model, each piece of prediction category of each piece of untagged data, and each piece of tag information corresponding to each piece of tagged data by using a first model parameter updating formula and a second model parameter updating formula determined by the expected value of the log likelihood function;
the first model parameter update formula is expressed as:
Figure BDA0004081264250000183
The second model parameter update formula is expressed as:
Figure BDA0004081264250000184
wherein p represents the current probability value of the target model, k represents the kth tag class, N represents the total number of tagged data and untagged data, and x i Representing a predicted class of the target model corresponding to the ith unlabeled data, the x j Representing a predicted class of the target model corresponding to the jth tagged data, the z j Tag information representing the j-th tagged data; the θ represents an updated parameter value of the object model.
Optionally, the computing module 604 is further configured to: determining a log likelihood value of the current detection period according to each labeled data, each prediction category of each unlabeled data, each label information corresponding to each labeled data in the current detection period, the model parameter value and the prediction probability value in the current detection period of the target model corresponding to each labeled data in the current detection period by using a log likelihood function conversion formula for any one of the current detection period and each reference detection period;
the log likelihood function conversion formula is expressed as:
Figure BDA0004081264250000191
wherein said l 1 A log likelihood value representing the current detection period, θ representing the model parameter value, X representing the target model corresponding to each labeled data and each unlabeled data prediction category in the current detection period, Z representing each label information corresponding to each labeled data in the current detection period, p representing the prediction probability value, X i Representing a predicted class of the target model corresponding to the ith unlabeled data in the current detection period, the x j Representing a prediction category of the target model corresponding to a j-th tagged data in the current detection period, the U representing a total number of untagged data in the current detection period, the L representing a total number of tagged data in the current detection period, the z j Tag information representing the j-th tagged data in the current detection period.
Optionally, the detection module 606 is further configured to: determining a normal distribution value according to each reference log likelihood value corresponding to each reference detection period; determining a log-likelihood deviation value of the target model corresponding to the target detection period according to the target log-likelihood value of the target detection period and the normal distribution value; and according to the log-likelihood deviation value and a given normal threshold range, if the log-likelihood deviation value does not fall into the normal threshold range, obtaining a detection result that the target model has drift in the target detection period, and if the log-likelihood deviation value falls into the normal threshold range, obtaining a detection result that the target model has no drift in the target detection period.
The model drift detection device 600 provided in the embodiment of the present invention corresponds to the model drift detection method provided in the embodiment of the present invention, and other descriptions can refer to the description of the model drift detection method provided in the embodiment of the present invention, which is not repeated here.
Another embodiment of the present invention provides an electronic device, including: the device comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface are in communication with each other through the communication bus.
Fig. 7 is a block diagram of an electronic device according to an exemplary embodiment of the present invention, and as shown in fig. 7, the electronic device 700 of the present embodiment may include a processor (processor) 702, a communication interface (communication interface) 704, and a memory (memory) 706.
The processor 702, the communication interface 704, and the memory 706 may communicate with each other via a communication bus 708.
The communication interface 704 is used to communicate with other electronic devices, such as terminal devices or servers.
The processor 702 is configured to execute the computer program 710, and may specifically perform relevant steps in the above-described method embodiments, that is, perform steps in the model drift detection method described in the above-described embodiments or perform steps in the winder control method described in the above-described embodiments.
In particular, the computer program 710 may include program code including computer operating instructions.
The processor 702 may be a Central Processing Unit (CPU), or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the electronic device may be the same type of processor, such as one or more CPUs; but may also be different types of processors such as one or more CPUs and one or more ASICs.
Memory 706 for storing a computer program 710. The memory 706 may comprise high-speed RAM memory or may further comprise non-volatile memory (non-volatile memory), such as at least one disk memory.
Another embodiment of the present invention provides a computer storage medium having a computer program stored thereon, which when executed by a processor, implements the model drift detection method described in the foregoing embodiments.
It should be noted that, according to implementation requirements, each component/step described in the embodiments of the present invention may be split into more components/steps, or two or more components/steps or part of operations of the components/steps may be combined into new components/steps, so as to achieve the objects of the embodiments of the present invention.
The above-described methods according to embodiments of the present invention may be implemented in hardware, firmware, or as software or computer code storable in a recording medium such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk, or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium and to be stored in a local recording medium downloaded through a network, so that the methods described herein may be stored on such software processes on a recording medium using a general purpose computer, special purpose processor, or programmable or special purpose hardware such as an ASIC or FPGA. It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a memory component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the model drift detection methods described herein. Further, when the general-purpose computer accesses code for implementing the model drift detection method shown herein, execution of the code converts the general-purpose computer into a special-purpose computer for executing the model drift detection method shown herein.
Those of ordinary skill in the art will appreciate that the elements and method steps of the examples described in connection with the embodiments disclosed herein can be implemented as electronic hardware, or as a combination of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the embodiments of the present invention.
The above embodiments are only for illustrating the embodiments of the present invention, but not for limiting the embodiments of the present invention, and various changes and modifications may be made by one skilled in the relevant art without departing from the spirit and scope of the embodiments of the present invention, so that all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the scope of the embodiments of the present invention should be defined by the claims.

Claims (11)

1. A model drift detection method, comprising:
determining a reference detection period of a target model according to the target detection period of the target model, and determining a plurality of sampling data contained in the target detection period and the reference detection period respectively, wherein each sampling data in the target detection period and the reference detection period comprises labeled data and unlabeled data;
Determining a target log-likelihood value of the target detection period and a reference log-likelihood value of the reference detection period according to the prediction results of the target model corresponding to the sampling data in the target detection period and the reference detection period and the label information of the labeled data in the sampling data;
and obtaining a model drift detection result of the target model corresponding to the target detection period according to the target log likelihood value of the target detection period and the reference log likelihood value of the reference detection period.
2. The method of claim 1, further comprising determining each detection period of the object model, comprising:
according to each sampling time corresponding to each sampling data, arranging each sampling data in sequence to obtain a sampling data sequence;
determining the number of sampling data covered by the sliding window according to the given detection period duration;
repeatedly executing the sliding treatment meeting the sliding step length on each sampling data in the sampling data sequence based on a given sliding step length by utilizing the sliding window to obtain each detection period corresponding to each sliding treatment and each sampling data in each detection period;
Wherein each sample data includes one of device sensing data, environment sensing data.
3. The method of claim 2, wherein the reference detection period comprises a plurality of target detection periods and each reference detection period are determined by:
determining each detection period corresponding to each detection period according to each sampling time corresponding to each sampling data in each detection period, and arranging each detection period in sequence according to each detection period;
determining a plurality of reference detection periods of the target detection period from among the detection periods according to one target detection period determined from among the detection periods;
wherein the number of the plurality of reference detection periods is between 5 reference detection periods and 20 reference detection periods;
wherein, each detection period corresponding to each reference detection period is earlier than the detection period of the target detection period, and each reference detection period is sequentially continuous with the target detection period.
4. The method of claim 1, wherein the step of determining the position of the substrate comprises,
the target model comprises a classification prediction model;
the label information at least comprises first label information and second label information, wherein the first label information is used for identifying the real type of each sampled data, and the first label information of any one labeled data is matched with one of the prediction types output by the target model; the second tag information includes a drift category and a non-drift category for identifying whether each sample data has drift.
5. The method according to claim 1, 3 or 4, wherein the determining the target log likelihood value of the target detection period and the reference log likelihood value of the reference detection period based on the prediction results of the target model corresponding to the respective sample data in the target detection period and the reference detection period, the tag information of the tagged data in the respective sample data, comprises:
determining a model parameter value and a prediction probability value of the target model according to the target model corresponding to each piece of tagged data, each prediction category of each piece of untagged data and each piece of tag information corresponding to each piece of tagged data;
determining a target log likelihood value of the target detection period according to the target model corresponding to each labeled data in the target detection period, each predicted category of each unlabeled data, each label information corresponding to each labeled data in the target detection period, the model parameter value and the predicted probability value, and determining a reference log likelihood value of each reference detection period according to each labeled data in the target model corresponding to each reference detection period, each predicted category of each unlabeled data, each label information corresponding to each labeled data in each reference detection period, the model parameter value and the predicted probability value.
6. The method of claim 5, wherein determining model parameter values and predicted probability values for the target model based on the target model corresponding to each tagged data, each predicted category of each untagged data, each tagged information corresponding to each tagged data, comprises:
determining a given initial parameter value as a current parameter value of the target model, and determining a log-likelihood function expected value of the target model according to the initial parameter value, the target model corresponding to each piece of tagged data, each prediction category of each piece of untagged data and each piece of tag information corresponding to each piece of tagged data;
a current probability value obtaining step, namely obtaining a current probability value of the target model according to the current parameter value of the target model, the corresponding labeled data of the target model, the corresponding prediction category of the unlabeled data and the corresponding label information of the labeled data;
obtaining an updated parameter value of the target model according to the current probability value of the target model, the corresponding prediction category of the target model to each tagged data, each untagged data and each tag information corresponding to each tagged data;
Updating the current parameter value by using the updated parameter value, and continuing to execute the current probability value acquisition step until the difference value between the updated parameter value and the current parameter meets a given update ending condition;
and respectively determining a model parameter value and a predicted probability value of the target model according to the current parameter value and the current probability value which meet the update ending condition.
7. The method of claim 6, wherein the current probability value obtaining step comprises:
for each tagged data, obtaining a current probability value of the target model according to the current parameter value of the target model, each prediction category of the target model corresponding to each tagged data, and tag information of each tagged data;
and for each piece of unlabeled data, obtaining a current probability value of the target model according to the current parameter value of the target model and each prediction category of the target model corresponding to each piece of unlabeled data.
8. The method of claim 7, wherein the step of determining the position of the probe is performed,
for each tagged data, obtaining a current probability value of the target model according to a current parameter value of the target model, each prediction category of the target model corresponding to each tagged data, and tag information of each tagged data, including:
Obtaining a current probability value of the target model according to the current parameter value of the target model, each prediction category of the target model corresponding to each tagged data and tag information of each tagged data by using a first unit probability conversion formula;
the first unit probability conversion formula is expressed as:
Figure FDA0004081264230000031
wherein, p represents the current probability value of the target model, K represents the kth tag class, K represents the total number of the tag classes, and x j Representing a predicted class of the target model corresponding to the jth tagged data, the z j Tag information representing jth tagged data, the L representing a total number of the tagged data, the θ representing a current parameter value of the target model;
and wherein the obtaining, for each unlabeled data, a current probability value of the target model according to a current parameter value of the target model, each prediction category of the target model corresponding to each unlabeled data, comprises:
obtaining a current probability value of the target model according to the current parameter value of the target model and each prediction category of the target model corresponding to each unlabeled data by using a second unit probability conversion formula;
The second unit probability conversion formula is expressed as:
Figure FDA0004081264230000032
wherein, p represents the current probability value of the target model, K represents the kth tag class, K represents the total number of the tag classes, and x i Representing the object model pairThe U represents the total number of unlabeled data and θ represents the current parameter value of the target model, corresponding to the predictive category of the ith unlabeled data.
9. The method of claim 6, wherein the obtaining updated parameter values for the target model based on the current probability values for the target model, the target model corresponding to each tagged data, each prediction category of each untagged data, each tag information corresponding to each tagged data, comprises:
obtaining an updated parameter value of the target model according to a current probability value of the target model, each piece of tagged data corresponding to the target model, each piece of prediction category of each piece of untagged data, and each piece of tag information corresponding to each piece of tagged data by using a first model parameter updating formula and a second model parameter updating formula determined by the expected value of the log likelihood function;
the first model parameter update formula is expressed as:
Figure FDA0004081264230000041
The second model parameter update formula is expressed as:
Figure FDA0004081264230000042
wherein p represents the current probability value of the target model, k represents the kth tag class, N represents the total number of tagged data and untagged data, and x i Representing a predicted class of the target model corresponding to the ith unlabeled data, the x j Representing a predicted class of the target model corresponding to the jth tagged data, the z j Tag information representing the j-th tagged data; the θ represents an updated parameter value of the object model.
10. The method of claim 5, wherein the target log-likelihood value or the reference log-likelihood value is determined by:
for any one of the target detection period and each reference detection period,
determining a log likelihood value of the current detection period according to each piece of tagged data, each piece of untagged data, each piece of tag information, the model parameter value and the prediction probability value, which correspond to each piece of tagged data in the current detection period, of the target model, by using a log likelihood function conversion formula;
The log likelihood function conversion formula is expressed as:
Figure FDA0004081264230000043
wherein said l 1 A log likelihood value representing the current detection period, θ representing the model parameter value, X representing the target model corresponding to each labeled data and each unlabeled data prediction category in the current detection period, Z representing each label information corresponding to each labeled data in the current detection period, p representing the prediction probability value, X i Representing a predicted class of the target model corresponding to the ith unlabeled data in the current detection period, the x j Representing a prediction category of the target model corresponding to a j-th tagged data in the current detection period, the U representing a total number of untagged data in the current detection period, the L representing a total number of tagged data in the current detection period, the z j Tag information representing the j-th tagged data in the current detection period.
11. The method according to claim 1, wherein the obtaining the model drift detection result of the target model corresponding to the target detection period according to the target log likelihood value of the target detection period and the reference log likelihood value of the reference period includes:
Determining a normal distribution value according to each reference log likelihood value corresponding to each reference detection period;
determining a log-likelihood deviation value of the target model corresponding to the target detection period according to the target log-likelihood value of the target detection period and the normal distribution value;
and according to the log-likelihood deviation value and a given normal threshold range, if the log-likelihood deviation value does not fall into the normal threshold range, obtaining a detection result that the target model has drift in the target detection period, and if the log-likelihood deviation value falls into the normal threshold range, obtaining a detection result that the target model has no drift in the target detection period.
CN202310120742.7A 2023-02-14 2023-02-14 Model drift detection method and device, electronic equipment and storage medium Pending CN116432769A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310120742.7A CN116432769A (en) 2023-02-14 2023-02-14 Model drift detection method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310120742.7A CN116432769A (en) 2023-02-14 2023-02-14 Model drift detection method and device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116432769A true CN116432769A (en) 2023-07-14

Family

ID=87089651

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310120742.7A Pending CN116432769A (en) 2023-02-14 2023-02-14 Model drift detection method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116432769A (en)

Similar Documents

Publication Publication Date Title
US7617010B2 (en) Detecting instabilities in time series forecasting
CN113884961B (en) SOC calibration method, modeling device, computer equipment and medium
US20160299938A1 (en) Anomaly detection system and method
US20060074828A1 (en) Methods and apparatus for detecting temporal process variation and for managing and predicting performance of automatic classifiers
CN110263824B (en) Model training method, device, computing equipment and computer readable storage medium
US7373332B2 (en) Methods and apparatus for detecting temporal process variation and for managing and predicting performance of automatic classifiers
US11709474B2 (en) Method and apparatus for detecting abnormality of manufacturing facility
CN113705726A (en) Traffic classification method and device, electronic equipment and computer readable medium
CN113377484A (en) Popup window processing method and device
CN113240076A (en) Clock error abnormal data detection method based on clustering and neural network algorithm
CN115526241A (en) Real-time abnormity detection method, device, equipment and medium for aviation hydraulic pump station
CN114048546B (en) Method for predicting residual service life of aeroengine based on graph convolution network and unsupervised domain self-adaption
CN116451081A (en) Data drift detection method, device, terminal and storage medium
CN114611409A (en) Method and device for establishing power distribution terminal abnormity detection model
CN116432769A (en) Model drift detection method and device, electronic equipment and storage medium
CN116861214A (en) Health state identification method and system based on convolution long short-time memory network
CN115861809A (en) Rod detection and training method and device for model thereof, electronic equipment and medium
CN113469176B (en) Target detection model training method, target detection method and related equipment thereof
CN114358581A (en) Method and device for determining abnormal threshold of performance index, equipment and storage medium
CN113111729B (en) Training method, recognition method, system, device and medium for personnel recognition model
US20060074826A1 (en) Methods and apparatus for detecting temporal process variation and for managing and predicting performance of automatic classifiers
US20060074827A1 (en) Methods and apparatus for detecting temporal process variation and for managing and predicting performance of automatic classifiers
US20210064941A1 (en) Method for adaptive context length control for on-line edge learning
Zhou et al. Fault variables recognition using improved k-nearest neighbor reconstruction
CN114298083A (en) Inverter three-phase current imbalance identification method and system based on VAE

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination