WO2023155378A1 - 模型的监控方法和装置 - Google Patents

模型的监控方法和装置 Download PDF

Info

Publication number
WO2023155378A1
WO2023155378A1 PCT/CN2022/106948 CN2022106948W WO2023155378A1 WO 2023155378 A1 WO2023155378 A1 WO 2023155378A1 CN 2022106948 W CN2022106948 W CN 2022106948W WO 2023155378 A1 WO2023155378 A1 WO 2023155378A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
monitoring
prediction
model
parameters
Prior art date
Application number
PCT/CN2022/106948
Other languages
English (en)
French (fr)
Inventor
邱峰志
周恺
王倩
孙权
张彬琳
李亮
陈嘉乐
戴欣
董周杰
张衡
Original Assignee
北京百度网讯科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京百度网讯科技有限公司 filed Critical 北京百度网讯科技有限公司
Publication of WO2023155378A1 publication Critical patent/WO2023155378A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
    • G06F11/3093—Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
    • G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/906—Clustering; Classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/865—Monitoring of software

Definitions

  • the present disclosure relates to the field of artificial intelligence technology, specifically to the field of cloud platform technology and machine learning technology, and in particular to a model monitoring method and device.
  • Models refer to files that can realize prediction functions, such as face recognition models, object detection models, etc. As time goes by, the accuracy of the model may drift. Therefore, how to continuously monitor the performance of the model and ensure the accuracy of the prediction results is an urgent problem to be solved.
  • the commonly used model monitoring method is: monitor the model according to the operation information of the model, and obtain the monitoring results, such as the monitoring index parameters of the model calculated based on the operation information (such as the accuracy of the model, etc. ).
  • the present disclosure provides a model monitoring method and device for improving monitoring accuracy.
  • a method for monitoring a model including:
  • the target model is monitored to obtain a monitoring result of the target model.
  • a model monitoring device comprising:
  • An acquisition unit configured to acquire data to be monitored, wherein the data to be monitored is at least part of the operating data of the target model to be monitored;
  • a classification unit configured to classify each data in the data to be monitored, and obtain data characteristics corresponding to each data, wherein different data characteristics are data for determining prediction results of the target model from different prediction dimensions Features, a data feature corresponds to a prediction dimension;
  • the monitoring unit is configured to monitor the target model based on data of data characteristics corresponding to at least one prediction dimension, and obtain a monitoring result of the target model.
  • an electronic device including:
  • the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the method described in the first aspect.
  • a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect.
  • a computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device can read from the The computer program is read by reading the storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.
  • an embodiment of the present application provides a computer program, including program code, and when a computer runs the computer program, the program code executes the method described in any one of the above embodiments.
  • each data in the data to be monitored is classified and processed, and the data based on one or more data characteristics is used to monitor the target model, which can avoid the rough monitoring of the target model as a whole in the related art.
  • the disadvantage of lack of flexibility improves the diversity and flexibility of monitoring, and makes the monitoring of the target model more targeted, so that the monitoring results can be determined more accurately, thereby improving the effectiveness and reliability of monitoring .
  • FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure
  • FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure.
  • Fig. 3 is a schematic diagram according to a third embodiment of the present disclosure.
  • FIG. 4 is a schematic diagram according to a fourth embodiment of the present disclosure.
  • FIG. 5 is a schematic diagram according to a fifth embodiment of the present disclosure.
  • FIG. 6 is a schematic diagram according to a sixth embodiment of the present disclosure.
  • FIG. 7 is a schematic diagram according to a seventh embodiment of the present disclosure.
  • FIG. 8 is a block diagram of an electronic device for implementing the model monitoring method of the embodiment of the present disclosure.
  • a model has a type attribute, and the type attribute of the model is used to characterize the predictive function of the model. That is, different types of models have different prediction functions. For example, according to the corresponding prediction functions of each model, the models can be divided into: classification model (such as image classification model, etc.), regression model, object detection model, instance segmentation model, and human Face recognition models, etc., will not be listed here.
  • classification model such as image classification model, etc.
  • regression model object detection model
  • instance segmentation model instance segmentation model
  • human Face recognition models etc.
  • the model has high intelligence and can provide more convenient and efficient prediction functions, it is widely used in different fields.
  • the accuracy of the model may drift (ie model accuracy drift), and accordingly, the prediction effect of the model will become less and less reliable over time. expected. Therefore, how to continuously monitor the performance of the model and ensure the accuracy of the prediction results is an urgent problem to be solved.
  • the operation information can be the resource consumption information of the platform when the model is running on the platform, or the difference between the predicted result and the real result obtained based on the operation of the model information, etc.
  • the above method is relatively rough to monitor the model as a whole, and the accuracy and pertinence are low.
  • the inventors of the present disclosure obtained the inventive concept of the present disclosure through creative work: classify and process each data in the monitoring data from different prediction dimensions to obtain multiple data features, based on one or more The data of a data characteristic is used to monitor the model.
  • the present disclosure provides a model monitoring method and device, which are applied to cloud computing in the field of artificial intelligence technology, specifically related to platform applications, so as to improve the reliability of monitoring.
  • Fig. 1 is a schematic diagram according to the first embodiment of the present disclosure. As shown in Fig. 1 , the monitoring method of the model in the embodiment of the present disclosure includes:
  • the data to be monitored is at least part of the operating data of the target model to be monitored.
  • the execution subject of this embodiment may be a model monitoring device (hereinafter referred to as the monitoring device), and the monitoring device may be a server (such as a local server, or a cloud server, or a service platform, or a server cluster ), may also be a computer, may also be a terminal device, may also be a processor, or may be a chip, etc., which are not limited in this embodiment.
  • the monitoring device may be a server (such as a local server, or a cloud server, or a service platform, or a server cluster ), may also be a computer, may also be a terminal device, may also be a processor, or may be a chip, etc., which are not limited in this embodiment.
  • target model in the target model is only used to distinguish the monitored model from other models. That is, the target model refers to the model monitored by implementing the method of the present disclosure, and should not be understood as a limitation on the target model.
  • the target model will generate relevant data during the service prediction process, and the generated relevant data includes the data to be monitored. Since the data to be monitored is the data generated by the target model when providing prediction services, based on The data to be monitored realizes the monitoring of the target model, which can make the monitoring more appropriate to the characteristics of the prediction service of the target model, so that the monitoring has higher reliability.
  • the operating data of the target model can be characterized from at least the following dimensions:
  • One dimension is the data of resources and other aspects of the platform deployed with the target model when the target model is running; the other dimension is the data generated by the target model running, such as prediction results; the other dimension is the real result of the target model running, And the difference between the predicted result and the real result; another dimension is the data used to support the target model to realize the prediction service, such as face images, etc.
  • the data to be monitored may include one or more of parameters needed by the target model to provide forecasting services, parameters related to results obtained by providing forecasting services, and resource consumption parameters of the target model.
  • S102 Perform classification processing on each data in the data to be monitored, and obtain data characteristics corresponding to each data.
  • different data features are features of data that determine the prediction results of the target model from different prediction dimensions, and one data feature corresponds to one prediction dimension.
  • the target model can determine the prediction results through different prediction dimensions.
  • the face recognition model can determine the face recognition results through dimensions such as age and gender.
  • the data feature can be age or gender.
  • this embodiment it can be understood as: dividing and processing the data to be monitored from different prediction dimensions, so as to obtain data characteristics corresponding to each data of the data to be monitored, that is, respective prediction dimensions corresponding to each data.
  • S103 Based on the data of the data characteristics corresponding to at least one prediction dimension, monitor the target model, and obtain the monitoring result of the target model.
  • the data to be monitored includes a variety of different data characteristics, so that the target model can complete prediction services from different prediction dimensions, and in order to improve the pertinence and flexibility of monitoring, data based on one data characteristic
  • the target model can be monitored, and the target model can also be monitored based on data of various data characteristics.
  • the present embodiment provides a model monitoring method, the method includes: obtaining data to be monitored, wherein the data to be monitored is at least part of the data in the operating data of the target model to be monitored, and the data to be monitored Each data is classified and processed to obtain the corresponding data characteristics of each data.
  • different data characteristics are the characteristics of the data that determine the prediction results of the target model from different prediction dimensions.
  • One data characteristic corresponds to one prediction dimension, based on at least A kind of data corresponding to the data characteristics of the prediction dimension, monitor the target model, and obtain the monitoring result of the target model.
  • the target model based on characteristic data can avoid the disadvantages of inflexibility caused by relatively rough overall monitoring of the target model in related technologies, improve the diversity and flexibility of monitoring, and make the monitoring of the target model With strong pertinence, the monitoring results can be determined more accurately, thereby improving the effectiveness and reliability of monitoring.
  • Fig. 2 is a schematic diagram according to the second embodiment of the present disclosure. As shown in Fig. 2 , the monitoring method of the model in the embodiment of the present disclosure includes:
  • the predictive service request may be initiated to the monitoring device by the business system (referring to the system calling the predictive service in the monitoring device), or it may be monitored by the monitoring device when the business system initiates to the third-party platform, And this embodiment does not limit the way to obtain the prediction service request, for example:
  • the predictive service request may be initiated by the user to the monitoring device through the service system, for example, the user initiates the predictive service request to the monitoring device through the service system by means of touch screen or voice control.
  • it may also be a forecast service request initiated by the business system to the monitoring device based on a preset time interval.
  • S202 Obtain a prediction identifier corresponding to the prediction service request, and obtain data to be monitored from the operation data of the target model corresponding to the prediction service request according to the prediction identifier.
  • the prediction identifier refers to the Universally Unique Identifier (UUID) used to identify sequential prediction services, that is, the prediction identifier can be understood as an identifier used to distinguish different prediction services.
  • UUID Universally Unique Identifier
  • the acquired data to be monitored can be highly correlated with the target model by combining the predictive identifiers to obtain the data to be monitored, so as to achieve the technical effect of accuracy and reliability of target model monitoring.
  • the data to be monitored includes the operation data of the target model corresponding to the prediction service request, such as the prediction result.
  • the prediction result refers to the result generated by the target model providing the prediction service, such as the face recognition result generated by the face recognition model providing the prediction service of face recognition.
  • the data to be monitored includes the prediction identifier and the prediction result, so as to determine the monitoring result based on the prediction identifier and the prediction result.
  • Different monitoring can be distinguished by the prediction identifier, and the prediction result can represent the target model when providing prediction services.
  • the predictive ability so that the technical effect of monitoring has high accuracy and reliability.
  • the description of the monitoring device obtaining the prediction identifier according to the prediction service request is described as follows:
  • the monitoring device may acquire the prediction identifier in at least multiple ways.
  • the prediction service request may carry the prediction identifier, and in another example, the monitoring device may generate the prediction identifier.
  • the monitoring device may determine whether the prediction identifier is included in the prediction service request, if yes, extract the prediction identifier in the prediction service request, and if not, generate the prediction identifier.
  • This embodiment does not limit the method of generating the prediction identifier.
  • the monitoring device may generate the prediction identifier based on random generation, and the prediction identifier may be used to distinguish different prediction services.
  • the prediction service request includes a request header, and the request header includes a prediction identifier.
  • the disadvantage of the target model being invaded when the prediction identifier is written into the body area can be avoided, thereby improving the security protection of the target model during monitoring , the technical effect of avoiding information leakage related to the target model.
  • the prediction result can also be written into the response header.
  • the disadvantage of the target model being invaded when the prediction result is written into the body area can be avoided, so as to improve the security protection of the target model during monitoring and avoid conflicts with Technical effects of information leakage related to the target model.
  • the monitoring device may be a device that provides prediction services, for example, the target model may be deployed and run in the monitoring device.
  • the monitoring device receives the prediction service request, it can call and run the target model, so as to obtain the prediction result.
  • the monitoring device can establish a communication link with a third-party platform, wherein a target model is deployed in the third-party platform, that is, the third-party platform can provide prediction services based on the target model deployed in it, and the monitoring device can send to The third-party platform forwards the prediction service request.
  • the third-party platform receives a prediction service request, it can call and run the target model deployed in it to obtain the prediction result, and feed back the prediction result to the monitoring device, or, the third-party platform feeds back the prediction to the business system
  • the monitoring device can monitor the prediction result.
  • the data to be monitored may also include a target type parameter, which represents the type attribute of the target model used to provide forecasting services; the target type parameters include input parameters and output parameters for requesting forecasting services.
  • the monitoring device obtains the data to be monitored from the operating data of the target model according to the prediction identification, which may include the following steps:
  • the first step determine the model monitoring parameters corresponding to the target type parameters to be acquired according to the preset mapping relationship, wherein the mapping relationship is used to represent the corresponding relationship between the type parameters and the model monitoring parameters, and the model monitoring parameters are used for calculation The parameters of the monitoring metrics for the model.
  • Step 2 Obtain the model monitoring parameters corresponding to the target request parameters from the running data of the target model according to the prediction identifier.
  • this embodiment can be understood as: firstly determine the parameters to be obtained according to the mapping relationship (that is, the model monitoring parameters corresponding to the target type parameters to be obtained), and then obtain the determined required parameters from the operating data of the target model Get parameters.
  • models with the same type of attributes are analyzed to obtain parameters for calculating monitoring indicators corresponding to the models with the same type of attributes, and the analyzed parameters are determined as model monitoring parameters of the models with the same type of attributes.
  • model monitoring parameters of models with the same type of attributes include the model monitoring parameters corresponding to the target type parameters.
  • the model of the type attribute can be analyzed to determine the parameters used to calculate the monitoring indicators of the model of the type attribute, so as to determine the model of the type attribute The model monitoring parameters of .
  • the type parameter can be understood as a parameter input by the user for requesting a prediction service
  • the model monitoring parameter can be understood as a parameter corresponding to the type parameter that can be recognized by the model.
  • the target type parameter is specifically an input parameter
  • the input parameter is an image picture
  • the corresponding model monitoring parameter is an image image
  • this embodiment does not limit the way of mapping, for example, the way of mapping can be one-to-one mapping, as in the above example, the input parameter "image picture” is mapped to the model monitoring parameter "image image”; it can also be The methods of the four arithmetic operations, etc., will not be listed here one by one.
  • each input parameter is mapped to the corresponding model monitoring parameter by means of mapping, which can realize the unification of the parameters of the model, especially when the When the monitoring data is unstructured data, the monitoring of the model can be realized, thereby increasing the applicable scope of monitoring and improving the flexibility of monitoring.
  • unstructured data refers to the data whose data structure is irregular or incomplete, which is inconvenient to be represented by two-dimensional logical tables of the database.
  • unstructured data can include office documents, texts, pictures, various reports, images and audio/video information, etc.
  • mapping relationship is used to represent the corresponding relationship between type parameters and model monitoring parameters.
  • users can establish the corresponding relationship between type parameters and model monitoring parameters by means of visualization or writing codes (such as information extraction library (Jsonpath) syntax, etc.).
  • the type parameters (including: input parameters and output parameters) can be configured in advance. For example, configuration may be performed manually.
  • the prediction service is a microservice obtained by encapsulating the target model.
  • Microservices can be understood as loosely coupling applications including forecasting services and independently deploying them as multiple components or services.
  • the forecasting service can run in an independent process, and a lightweight communication mechanism can be used between the forecasting service and other services. Communication, such as Hyper Text Transfer Protocol (Hyper Text Transfer Protocol, HTTP) application design style and development method (RESTful) interface (Application Programming Interface, API).
  • HTTP Hyper Text Transfer Protocol
  • RESTful Application Programming Interface
  • the target model may be encapsulated into a microservice, and a prediction service may be provided by means of a hypertext transfer protocol interface and a full-duplex communication protocol (WebSocket) interface.
  • a prediction service may be provided by means of a hypertext transfer protocol interface and a full-duplex communication protocol (WebSocket) interface.
  • the parameters to be monitored also include real results corresponding to predicted service requests.
  • the real result is the ground truth (GroundTruth) value.
  • the real result is a relative concept to the predicted result, and refers to the result actually corresponding to the input parameters.
  • the input parameter is an image
  • the prediction result refers to the gender and/or age of the object in the image output by the image through the face recognition model
  • the real result refers to the actual The gender and/or age of the overall subject of the image.
  • this embodiment does not limit the way to obtain the real result, for example, the real result may be obtained by marking, or other methods may be used to obtain the real result.
  • the data to be monitored also includes the real result
  • the monitoring result can be determined based on the predicted identification, the predicted result, and the real result
  • the real result is the actual predicted result corresponding to the predicted result. Therefore, by Combining the predicted results with the real results, the technical effect of the reliability and accuracy of monitoring can be further improved from the predicted dimension and the actual dimension.
  • the model can also be monitored in conjunction with the resource consumption information of the system deployed with the target model when the target model is running, such as when the target model is running, the resource utilization of the central processing unit CPU of the system deployed with the target model Rate, etc., to monitor the target model more comprehensively, so as to further improve the technical effect of monitoring reliability and accuracy.
  • the real result can also be written into the response header.
  • Real results can be written to the body area to avoid interference between real results.
  • S203 Perform classification processing on each data in the data to be monitored, and obtain data characteristics corresponding to each data.
  • the data feature is the feature of the data that determines the prediction result of the target model from different prediction dimensions.
  • S204 Based on the data of the data characteristic corresponding to at least one prediction dimension, monitor the target model to obtain a monitoring result of the target model.
  • Fig. 3 is a schematic diagram according to a third embodiment of the present disclosure. As shown in Fig. 3 , the model monitoring method of the embodiment of the present disclosure includes:
  • S302 Perform feature extraction processing on each data in the to-be-monitored data to obtain respective prediction dimensions corresponding to each data.
  • the target model as a face recognition model as an example, among the data to be monitored, if the data feature of the first data is age, the data feature of the second data is gender, that is, the first data is The age-related data representing the object in the image, and the second data representing the gender-related data of the object in the highlight.
  • the determined data features can have a technical effect of higher accuracy and reliability.
  • S304 Based on the data of the data characteristic corresponding to at least one prediction dimension, monitor the target model to obtain a monitoring result of the target model.
  • the first data can be used as the monitoring object of the target model monitoring, that is, the monitoring of the target model can be realized based on the first data
  • the second data can also be used as the monitoring object of the target model monitoring, that is, based on the second
  • the data realizes the monitoring of the target model
  • both the first data and the second data can also be used as the monitoring objects of the target model monitoring, that is, the monitoring of the target model is realized based on the first data and the second data.
  • Fig. 4 is a schematic diagram according to a fourth embodiment of the present disclosure. As shown in Fig. 4 , the model monitoring method of the embodiment of the present disclosure includes:
  • S402 Perform classification processing on each data in the data to be monitored, and obtain data characteristics corresponding to each data.
  • the data feature is the feature of the data that determines the prediction result of the target model from different prediction dimensions.
  • the preset monitoring index may be determined by the monitoring device based on requirements, historical records, and experiments, which is not limited in this embodiment.
  • different models providing different prediction services may set different preset monitoring indicators, for example, the preset monitoring indicators of the classification model may be different from the preset monitoring indicators of the regression model. That is, for the type of prediction service that the model can provide, the corresponding model may be assigned a preset monitoring index for monitoring the model based on the type of prediction service.
  • the preset monitoring indicators may include: accuracy rate, precision rate, recall rate, false positive rate, F1 score (F1-score), receiver Area under the receiver operating characteristic (ROC) curve, area under the precision vs recall (P-R) curve, statistics (Kolmogorov-Smirnov, KS), logarithmic loss, etc.
  • the target model is a binary classification model
  • the preset monitoring index is accuracy rate
  • the data used to determine the monitoring result is age data
  • the preset monitoring indicators can include: accuracy rate, precision rate, recall rate, false positive rate, F1 score, weighted precision rate, weighted recall rate, weighted false positive rate, weighted F1 score, etc. .
  • the preset monitoring indicators may include: accuracy rate, precision rate, recall rate, F1 score, and the like.
  • the preset monitoring indicators can include: total square error, error sum of squares, regression sum of squares, mean absolute error, mean absolute percentage error, root mean square error, root mean square error, coefficient of determination, interpretable Variance, raw column mean, predicted result mean, etc.
  • the preset monitoring indicators may include: target detection evaluation indicators (mean average precision, mAP), precision rate, recall rate, etc.
  • the preset monitoring indicators may include: target detection evaluation indicators (mean average precision, mAP), precision rate, recall rate, etc.
  • S403 may include the following steps:
  • the first step Obtain the monitoring period corresponding to the data of the data feature corresponding to this type of prediction dimension.
  • the second step according to the monitoring period corresponding to the data of the data characteristic corresponding to the prediction dimension, calculate the monitoring value of the data of the data characteristic corresponding to the prediction dimension under the preset monitoring index.
  • the monitoring period refers to a pre-configured interval for monitoring the target model.
  • the monitoring period is one week, that is, the target model is monitored every other week to determine the accuracy and reliability of the target model when it runs for one week.
  • monitoring periods of data with different data characteristics may be the same or different, which is not limited in this embodiment.
  • the monitoring cycle of data whose data feature is age can be the same as the monitoring cycle of data whose data feature is gender; or, the monitoring cycle of data whose data feature is age can be the same as that of data whose data feature is gender The monitoring period is different.
  • the monitoring period can be set in advance for data with different data characteristics, so as to trigger the calculation of the monitoring value based on the monitoring period, so as to realize the monitoring of the target model based on the calculated monitoring value.
  • the target model is monitored by means of a monitoring cycle, which can achieve the technical effects of orderliness and flexibility of monitoring.
  • S404 Determine the monitoring result according to the monitoring values corresponding to the data of the data characteristics corresponding to various prediction dimensions.
  • the target model can be monitored from different dimensions (such as age, gender, etc.), so as to realize the comprehensiveness of monitoring and improve the accuracy and reliability of monitoring. technical effect.
  • early warning information can be output through one or more methods of email, text message, and calling a third-party interface, so that the target The model is updated to maintain a high-quality target model.
  • the monitoring result may be data drift, which refers to the consistency between prediction data (such as the input parameters in the above examples) and training data (referring to the data used for training to obtain the target model).
  • the data drift is greater than the pre-set early warning threshold, that is, the consistency between the predicted data and the training data decreases, that is, there is a large difference between the characteristics of the predicted data and the distribution of the predicted results. you can send emails, text messages, and call the first One or more methods in the three-party interface output early warning information in order to update the target model and maintain a high-quality target model.
  • the third-party interface may be an interface provided by a device other than the service platform that can be invoked by the service platform.
  • the service platform can call a third-party interface to output early warning information through the third-party interface.
  • the early warning threshold can be determined by the monitoring device based on requirements, historical records, and experiments, which is not limited in this embodiment, and the early warning thresholds corresponding to different models, or data with different data characteristics of the same model, may be the same , may also be different.
  • Fig. 5 is a schematic diagram according to a fifth embodiment of the present disclosure.
  • the monitoring device 500 of the model of the embodiment of the present disclosure includes:
  • the obtaining unit 501 is configured to obtain the data to be monitored, wherein the data to be monitored is at least part of the operating data of the target model to be monitored.
  • the classification unit 502 is used to classify and process each data in the data to be monitored, and obtain the corresponding data characteristics of each data, wherein different data characteristics are the characteristics of the data that determine the prediction results of the target model from different prediction dimensions, a kind of Data features correspond to a predictive dimension.
  • the monitoring unit 503 is configured to monitor the target model based on data of data characteristics corresponding to at least one prediction dimension, and obtain a monitoring result of the target model.
  • Fig. 6 is a schematic diagram according to the sixth embodiment of the present disclosure.
  • the monitoring device 600 of the model of the embodiment of the present disclosure includes:
  • the obtaining unit 601 is configured to obtain the data to be monitored, wherein the operation data of the target model to be monitored includes the data to be monitored.
  • the acquiring unit 601 includes:
  • the first obtaining subunit 6011 is configured to obtain the prediction identification obtained from the prediction service request in response to listening to the prediction service request.
  • the second acquiring subunit 6012 is configured to acquire the data to be monitored from the operating data of the target model according to the prediction identifier.
  • the running data of the target model includes a target type parameter
  • the target type parameter represents the type attribute of the target model used to provide the prediction service
  • the obtaining unit 601 further includes:
  • the second determining subunit 6013 is used to determine the model monitoring parameters corresponding to the target type parameters to be obtained according to the preset mapping relationship, wherein the mapping relationship is used to represent the corresponding relationship between the type parameters and the model monitoring parameters, and the model monitoring Parameters are the parameters used to calculate the monitoring metrics of the model.
  • the second obtaining subunit 6012 is configured to obtain the model monitoring parameters corresponding to the target model parameters from the running data of the target model according to the prediction identifier.
  • the acquiring unit 601 further includes:
  • the analysis subunit 6014 is configured to analyze models of the same type of attributes, obtain parameters for calculating monitoring indicators corresponding to models of the same type of attributes, and determine the analyzed parameters as model monitoring parameters of models of the same type of attributes, wherein, the model monitoring parameters of models with the same type of attributes include the model monitoring parameters corresponding to the target type parameters.
  • the prediction service request includes a request header, and the request header includes a prediction identifier.
  • the prediction service is a microservice obtained by encapsulating the target model.
  • the classification unit 602 is configured to classify and process each data in the data to be monitored, and obtain data characteristics corresponding to each data, wherein the data characteristics are characteristics of data that determine the prediction results of the target model from different prediction dimensions.
  • the classification unit 602 includes:
  • the processing subunit 6021 is configured to analyze and process each data in the data to be monitored, and obtain the prediction dimensions corresponding to each data.
  • the first determining subunit 6022 is configured to determine the data features corresponding to each data according to the prediction dimensions corresponding to each data.
  • the monitoring unit 603 is configured to monitor the target model based on data of data characteristics corresponding to at least one prediction dimension, and obtain a monitoring result of the target model.
  • the monitoring unit 603 includes:
  • the calculation subunit 6031 is configured to calculate the monitoring value of the data of the data feature corresponding to each type of prediction dimension under the preset monitoring index according to the data of the data feature corresponding to each type of prediction dimension.
  • the computing subunit 6031 includes:
  • the obtaining module is used to obtain the monitoring period corresponding to the data of the data characteristic corresponding to the prediction dimension.
  • the calculation module is used to calculate the monitoring value of the data of the data characteristic corresponding to the prediction dimension under the preset monitoring index according to the monitoring cycle corresponding to the data of the data characteristic corresponding to the prediction dimension.
  • the third determination subunit 6032 is configured to determine the monitoring results according to the monitoring values corresponding to the data of the data characteristics corresponding to various prediction dimensions.
  • FIG. 7 is a schematic diagram according to a seventh embodiment of the present disclosure.
  • an electronic device 700 in the present disclosure may include: a processor 701 and a memory 702 .
  • the memory 702 is used to store programs; the memory 702 may include a volatile memory (English: volatile memory), such as a random-access memory (English: random-access memory, abbreviation: RAM), such as a static random-access memory (English: RAM) : static random-access memory, abbreviation: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviation: DDR SDRAM), etc.; memory can also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory).
  • the memory 702 is used to store computer programs (such as application programs, functional modules, etc. for realizing the above methods), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories 702 . And the above-mentioned computer programs, computer instructions, data, etc. can be invoked by the processor 701 .
  • the above-mentioned computer programs, computer instructions, etc. may be partitioned and stored in one or more memories 702 . And the above-mentioned computer programs, computer instructions, etc. may be invoked by the processor 701 .
  • the processor 701 is configured to execute the computer program stored in the memory 702, so as to implement various steps in the methods involved in the foregoing embodiments.
  • the processor 701 and the memory 702 may be an independent structure, or may be an integrated structure integrated together. When the processor 701 and the memory 702 have independent structures, the memory 702 and the processor 701 may be coupled and connected through the bus 703 .
  • the electronic device in this embodiment can execute the technical solution in the above method, and its specific implementation process and technical principle are the same, and will not be repeated here.
  • the collection, storage, use, processing, transmission, provision, and disclosure of user personal information (such as images, etc.) involved are in compliance with relevant laws and regulations, and do not violate public order and good customs.
  • the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.
  • the present disclosure also provides a computer program product.
  • the computer program product includes: a computer program, the computer program is stored in a readable storage medium, and at least one processor of an electronic device can read from the readable storage medium. Taking a computer program, at least one processor executes the computer program so that the electronic device executes the solution provided by any one of the above embodiments.
  • FIG. 8 shows a schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present disclosure.
  • Electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers.
  • Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.
  • the components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.
  • the device 800 includes a computing unit 801 that can execute according to a computer program stored in a read-only memory (ROM) 802 or loaded from a storage unit 808 into a random access memory (RAM) 803. Various appropriate actions and treatments. In the RAM 803, various programs and data necessary for the operation of the device 800 can also be stored.
  • the computing unit 801, ROM 802, and RAM 803 are connected to each other through a bus 804.
  • An input/output (I/O) interface 805 is also connected to the bus 804 .
  • the I/O interface 805 includes: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc. ; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, and the like.
  • the communication unit 809 allows the device 800 to exchange information/data with other devices over a computer network such as the Internet and/or various telecommunication networks.
  • the computing unit 801 may be various general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of computing units 801 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc.
  • the calculation unit 801 executes various methods and processes described above, such as a model monitoring method. For example, in some embodiments, the method of monitoring a model may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 808 .
  • part or all of the computer program may be loaded and/or installed on the device 800 via the ROM 802 and/or the communication unit 809.
  • the computer program When the computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of the monitoring method of the model described above can be performed.
  • the computing unit 801 may be configured to execute the model monitoring method in any other suitable manner (for example, by means of firmware).
  • Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips Implemented in a system of systems (SOC), complex programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof.
  • FPGAs field programmable gate arrays
  • ASICs application specific integrated circuits
  • ASSPs application specific standard products
  • SOC system of systems
  • CPLD complex programmable logic device
  • computer hardware firmware, software, and/or combinations thereof.
  • programmable processor can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.
  • Program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special purpose computer, or other programmable data processing devices, so that the program codes, when executed by the processor or controller, make the functions/functions specified in the flow diagrams and/or block diagrams Action is implemented.
  • the program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.
  • a machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • a machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing.
  • machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.
  • RAM random access memory
  • ROM read only memory
  • EPROM or flash memory erasable programmable read only memory
  • CD-ROM compact disk read only memory
  • magnetic storage or any suitable combination of the foregoing.
  • the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user. ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer.
  • a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
  • a keyboard and pointing device eg, a mouse or a trackball
  • Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.
  • the systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system.
  • the components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN) and the Internet.
  • a computer system may include clients and servers.
  • Clients and servers are generally remote from each other and typically interact through a communication network.
  • the relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other.
  • the server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the problem of traditional physical host and VPS service ("Virtual Private Server", or "VPS”) Among them, there are defects such as difficult management and weak business scalability.
  • the server can also be a server of a distributed system, or a server combined with a blockchain.
  • the embodiments of the present application further provide a computer program, including program code, and when the computer runs the computer program, the program code executes the method described in any of the above embodiments.
  • steps may be reordered, added or deleted using the various forms of flow shown above.
  • each step described in the present disclosure may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in the present disclosure can be achieved, no limitation is imposed herein.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Testing And Monitoring For Control Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本公开提供了一种模型的监控方法和装置，涉及人工智能技术领域，具体涉及云平台技术和机器学习技术领域，包括：获取待监控数据，待监控数据为待监控的目标模型的运行数据中的至少部分数据，对待监控数据中的各数据进行分类处理，得到各数据各自对应的数据特征，不同的数据特征为从不同预测维度确定目标模型的预测结果的数据的特征，一种数据特征对应一种预测维度，基于至少一种预测维度对应的数据特征的数据，对目标模型进行监控，得到目标模型的监控结果，可以避免较为粗略的从整体上对目标模型进行监控造成的缺乏灵活性的弊端，提高了监控的多样性和灵活性，且使得对目标模型的监控具有较强的针对性，从而提高了监控的有效性和可靠性。

Description

模型的监控方法和装置
本公开要求于2022年02月16日提交中国专利局、申请号为CN 202210143047.8、申请名称为“模型的监控方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开涉及人工智能技术领域，具体涉及云平台技术和机器学习技术领域，尤其涉及一种模型的监控方法和装置。
背景技术
模型是指能实现预测功能的文件，如人脸识别模型、物体检测模型等。随着时间的推移，模型的准确性可能发生漂移，所以，如何持续监控模型的表现，确保预测结果的准确率，是亟待解决的问题。
在相关技术中，通常采用的模型的监控方法为：根据模型的运行信息对模型进行监控，得到监控结果，如监控结果为基于运行信息计算得到的模型的监控指标参数(如模型的精确率等)。
然而，上述方法可以粗略的从整体上进行监控，却相对缺乏针对性和准确性，
发明内容
本公开提供了一种用于提高监控的准确性的模型的监控方法和装置。
根据本公开的第一方面，提供了一种模型的监控方法，包括：
获取待监控数据，其中，所述待监控数据为待监控的目标模型的运行数据中的至少部分数据；
对所述待监控数据中的各数据进行分类处理，得到各数据各自对应的数据特征，其中，不同的数据特征为从不同预测维度确定所述目标模型的预测结果的数据的特征，一种数据特征对应一种预测维度；
基于至少一种预测维度对应的数据特征的数据，对所述目标模型进行监控，得到所述目标模型的监控结果。
根据本公开的第二方面，提供了一种模型的监控装置，包括：
获取单元，用于获取待监控数据，其中，所述待监控数据为待监控的目标模型的运行数据中的至少部分数据；
分类单元，用于对所述待监控数据中的各数据进行分类处理，得到各数据各自对应的数据特征，其中，不同的数据特征为从不同预测维度确定所述目标模型的预测结果的数据的特征，一种数据特征对应一种预测维度；
监控单元，用于基于至少一种预测维度对应的数据特征的数据，对所述目标模型进行监控，得到所述目标模型的监控结果。
根据本公开的第三方面，提供了一种电子设备，包括：
至少一个处理器；以及
与所述至少一个处理器通信连接的存储器；其中，
所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行第一方面所述的方法。
根据本公开的第四方面，提供了一种存储有计算机指令的非瞬时计算机可读存储介质，其中，所述计算机指令用于使所述计算机执行根据第一方面所述的方法。
根据本公开的第五方面，提供了一种计算机程序产品，所述计算机程序产品包括：计算机程序，所述计算机程序存储在可读存储介质中，电子设备的至少一个处理器可以从所述可读存储介质读取所述计算机程序，所述至少一个处理器执行所述计算机程序使得电子设备执行第一方面所述的方法。
根据第六方面，本申请实施例提供了一种计算机程序，包括程序代码，当计算机运行所述计算机程序时，所述程序代码执行如上任一实施例所述的方法。
根据本公开的对待监控数据中的各数据进行分类处理，以基于一种或多种数据特征的数据对目标模型进行监控，可以避免相关技术中较为粗略的从整体上对目标模型进行监控造成的缺乏灵活性的弊端，提高了监控的多样性和灵活性，且使得对目标模型的监控具有较强的针对性，从而可以更为准确地确定监控结果，从而提高了监控的有效性和可靠性。
应当理解，本部分所描述的内容并非旨在标识本公开的实施例的关键或重要特征，也不用于限制本公开的范围。本公开的其它特征将通过以下的说明书而变得容易理解。
附图说明
附图用于更好地理解本方案，不构成对本公开的限定。其中：
图1是根据本公开第一实施例的示意图；
图2是根据本公开第二实施例的示意图；
图3是根据本公开第三实施例的示意图；
图4是根据本公开第四实施例的示意图；
图5是根据本公开第五实施例的示意图；
图6是根据本公开第六实施例的示意图；
图7是根据本公开第七实施例的示意图；
图8是用来实现本公开实施例的模型的监控方法的电子设备的框图。
具体实施方式
以下结合附图对本公开的示范性实施例做出说明，其中包括本公开实施例的各种细节以助于理解，应当将它们认为仅仅是示范性的。因此，本领域普通技术人员应当认识到，可以对这里描述的实施例做出各种改变和修改，而不会背离本公开的范围和精神。同样，为了清楚和简明，以下的描述中省略了对公知功能和结构的描述。
模型具有类型属性，模型的类型属性用于表征模型的预测功能。即不同类型的模型具有不同的预测功能，如按照各模型各自对应的预测功能，可以将模型分为：分类模型(如图像分类模型等)、回归模型、物体检测模型、实例分割模型、以及人脸识别模型等，此处不再一一列举。
由于模型具有较高的智能化，能够提供较为便捷且高效的预测功能，因此被广泛地应用于不同的领域。然而，随着模型被使用的时间的推移，以及数据的不断更新等，模型的准确性可能发生漂移(即模型准确性漂移)，相应的，模型的预测效果会随着时间而越来越不可预期。所以，如何持续监控模型的表现，确保预测结果的准确率，是亟待解决的问题。
在相关技术中，通过对模型的运行信息的监控，运行信息可以为模型在平台中运行时，平台的资源消耗信息，也可以为基于模型的运行而得到的预测结果与真实结果之间的差异信息，等等。
然而，上述方法相对比较粗略的从整体上对模型进行监控，准确性和针对性偏低。
为了避免上述技术问题，本公开发明人经过创造性的劳动，得到了本公开的发明构思：从不同预测维度对待监控数据中的各数据进行分类处理，以得到多个数据特征，基于一种或多种数据特征的数据对模型进行监控。
基于上述发明构思，本公开提供一种模型的监控方法和装置，应用于人工智能技术领域中的云计算，具体涉及平台应用，以提高监控的可靠性。
图1是根据本公开第一实施例的示意图，如图1所示，本公开实施例的模型的监控方法，包括：
S101：获取待监控数据。
其中，待监控数据为待监控的目标模型的运行数据中的至少部分数据。
示例性的，本实施例的执行主体可以为模型的监控装置(下文简称为监控装置)，监控装置可以为服务器(如本地服务器，或者，云端服务器，也可以为服务平台，也可以为服务器集群)，也可以为计算机，也可以为终端设备，也可以为处理器，也可以为芯片等，本实施例不做限定。
应该理解的是，目标模型中的“目标”只是用于将被监控的模型与其他的模型进行区分。即目标模型是指被实施本公开的方法进行监控的模型，而不能理解为对目标模型的限定。
目标模型在运行过程中，即目标模型在提供服务预测过程中会产生相关数据，产生的相关数据中包括待监控数据，由于待监控数据为目标模型在提供预测服务时而产生的数据，因此，基于待监控数据实现对目标模型的监控，可以使得监控更加贴切于目标模型的预测服务的特性，从而使得监控具有较高的可靠性。
结合上述分析可知，目标模型的运行数据至少可以从以下维度进行表征：
一个维度为部署有目标模型的平台在目标模型运行时，平台在资源等方面的数据；另一个维度为目标模型运行而产生的数据，如预测结果；另一个维度为目标模型运行的真实结果，以及预测结果与真实结果之间的差异；再一个维度为用于支持目标模型实现预测服务的数据，如人脸图像等。
也就是说，待监控数据可以包括目标模型提供预测服务需要用到的参数、提供预测服务的而得到的结果相关的参数、以及目标模型的资源消耗的参数中的一种或多种。
S102：对待监控数据中的各数据进行分类处理，得到各数据各自对应的数据特征。
其中，不同的数据特征为从不同预测维度确定目标模型的预测结果的数据的特征，一种数据特征对应一种预测维度。
值得说明的是，目标模型可以通过不同的预测维度确定预测结果，以人脸识别模型为例，人脸识别模型可以通过年龄和性别等维度确定人脸识别结果。相应的，数据特征可以为年龄，也可以为性别。
在本实施例中，可以理解为：从不同的预测维度对待监控数据进行划分处理，以得到待监控数据的各数据各自对应的数据特征，即各数据各自对应的预测维度。
S103：基于至少一种预测维度对应的数据特征的数据，对目标模型进行监控，得 到目标模型的监控结果。
结合上述分析可知，待监控数据中包括多种不同的数据特征，以使得目标模型可以从不同的预测维度完成预测服务，而为了提高监控的针对性和灵活性，可以基于一种数据特征的数据对目标模型进行监控，也可以基于多种数据特征的数据对目标模型进行监控。
基于上述分析可知，本实施例提供了一种模型的监控方法，该方法包括：获取待监控数据，其中，待监控数据为待监控的目标模型的运行数据中的至少部分数据，对待监控数据中的各数据进行分类处理，得到各数据各自对应的数据特征，其中，不同的数据特征为从不同预测维度确定目标模型的预测结果的数据的特征，一种数据特征对应一种预测维度，基于至少一种预测维度对应的数据特征的数据，对目标模型进行监控，得到目标模型的监控结果，在本实施例中，通过对待监控数据中的各数据进行分类处理，以基于一种或多种数据特征的数据对目标模型进行监控，可以避免相关技术中较为粗略的从整体上对目标模型进行监控造成的缺乏灵活性的弊端，提高了监控的多样性和灵活性，且使得对目标模型的监控具有较强的针对性，从而可以更为准确地确定监控结果，从而提高了监控的有效性和可靠性。
图2是根据本公开第二实施例的示意图，如图2所示，本公开实施例的模型的监控方法，包括：
S201：监听预测服务请求。
为了避免冗余地陈述，关于本实施例与上述实施例相同的技术特征，本实施例不再赘述。
示例性的，预测服务请求可以为由业务系统(是指调用监控装置中的预测服务的系统)向监控装置发起的，也可以为业务系统向第三方平台发起时，被监控装置监听到的，且本实施例对获取预测服务请求的方式不做限定，例如：
一个示例中，预测服务请求可以为用户通过业务系统向监控装置发起的，如用户采用触屏或声控等方式，通过业务系统向监控装置发起预测服务请求。
另一个示例中，也可以为业务系统基于预设时间间隔向监控装置发起的预测服务请求。
S202：获取预测服务请求对应的预测标识，并根据预测标识从预测服务请求对应的目标模型的运行数据中获取待监控数据。
预测标识是指，用于标识依次预测服务的通用唯一识别码(Universally Unique Identifier，UUID)，即预测标识可以理解为用于对不同的预测服务进行区分的标识。
在本实施例中，通过结合预测标识获取待监控数据，可以使得获取待监控数据与目标模型高度关联，从而实现对目标模型监控的准确性和可靠性的技术效果。
一些实施例中，待监控数据包括预测服务请求对应的目标模型的运行数据，如预测结果。预测结果是指，目标模型提供预测服务而生成的结果，如人脸识别模型提供人脸识别的预测服务而生成的人脸识别结果等。
在本实施例中，待监控数据中包括预测标识和预测结果，以基于预测标识和预测结果确定监控结果，可以通过预测标识将不同的监控进行区分，预测结果可以表征目标模型的提供预测服务时的预测能力，从而使得监控具有较高的准确性和可靠性的技术效果。
示例性的，对监控装置根据预测服务请求获取预测标识的描述进行阐述如下：
监控装置至少可以通过多种方式获取预测标识，一个示例中，预测服务请求中可以携带预测标识，另一个示例中，监控装置可以生成预测标识。
例如，监控装置可以判断预测服务请求中是否包括预测标识，如果是，则提取预测服务请求中的预测标识，如果否，则可以生成预测标识。
本实施例对生成预测标识的方式不做限定，例如，监控装置可以基于随机生成的方式生成预测标识，预测标识可以用于对不同的预测服务进行区分即可。
在一些实施例中，预测服务请求包括请求头header，请求头中包括预测标识。
在本实施例中，通过将预测标识写入请求头中，可以避免当将预测标识写入主体(body)区域时造成目标模型被入侵的弊端，从而提高监控时对目标模型的安全性进行保护，避免与目标模型相关的信息泄露的技术效果。
同理，在一些实施例中，预测结果也可以被写入至响应头header中。
相应的，通过将预测结果写入请求头中，可以避免当将预测结果写入主体(body)区域时造成目标模型被入侵的弊端，从而提高监控时对目标模型的安全性进行保护，避免与目标模型相关的信息泄露的技术效果。
示例性的，对监控装置根据预测服务请求获取预测结果的描述进行阐述如下：
在一些实施例中，监控装置可以为提供预测服务的装置，如目标模型可以部署并运行于监控装置中。相应的，在监控装置接收到预测服务请求时，可以调用并运行目标模型，从而得到预测结果。
在另一些实施例中，监控装置可以与第三方平台建立通信链路，其中，第三方平台中部署有目标模型，即第三方平台可以基于其内部署的目标模型提供预测服务，监控装置可以向第三方平台转发预测服务请求。相应的，第三方平台在接收到预测服务 请求时，可以调用并运行其内部署的目标模型，从而得到预测结果，并将预测结果反馈给监控装置，或者，第三方平台在向业务系统反馈预测结果时，监控装置可以监听到预测结果。
在另一些实施例中，待监控数据中还可以包括目标类型参数，目标类型参数表征用于提供预测服务的目标模型的类型属性；目标类型参数包括请求预测服务的输入参数和输出参数等。
相应的，监控装置根据预测标识从目标模型的运行数据中，获取待监控数据，可以包括如下步骤：
第一步骤：根据预设的映射关系确定待获取的与目标类型参数对应的模型监控参数，其中，映射关系用于表征类型参数与模型监控参数之间的对应关系，模型监控参数为用于计算模型的监控指标的参数。
第二步骤：根据预测标识从目标模型的运行数据中，获取与目标请求参数对应的模型监控参数。
示例性的，该实施例可以理解为：先根据映射关系确定需要获取的参数(即待获取的与目标类型参数对应的模型监控参数)，而后从目标模型的运行数据中，获取确定出的需要获取的参数。
在一些实施例中，对相同类型属性的模型进行分析，得到用于计算相同类型属性的模型对应的监控指标的参数，并将分析得到的参数确定为相同类型属性的模型的模型监控参数。
其中，相同类型属性的模型的模型监控参数中包括目标类型参数对应的模型监控参数。
示例性的，结合上述分析，针对回归模型的类型属性的模型，可以对该类型属性的模型进行分析，以确定用于计算该类型属性的模型的监控指标的参数，从而确定该类型属性的模型的模型监控参数。
类型参数与模型监控参数存在对应关系，而模型监控参数与预测服务之间也存在对应的关系。且结合上述可知，目标类型参数可以理解为用户用于请求预测服务而输入的参数，而模型监控参数可以理解为模型可以识别的与类型参数对应的参数。
示例性的，以图像分类模型为例，目标类型参数具体为输入参数，且输入参数为图像picture，而与之对应的模型监控参数为图像image。
其中，本实施例对映射的方式不做限定，例如映射的方式可以为一一映射的方式，如上述示例中的将输入参数“图像picture”映射为模型监控参数“图像image”；也 可以为四则运算的方式，等等，此处不再一一列举。
由于不同的厂商对同一模型的参数命名可能不同，而在本实施例中，通过映射的方式将各输入参数映射至与之对应的模型监控参数，可以实现模型在参数上的统一，尤其当待监控数据为非结构化数据时，可以实现对模型的监控，从而提高了监控的可适用范围，提高了监控的灵活性。
其中，非结构化数据是指，数据结构不规则或不完整，不方便用数据库二维逻辑表来表现的数据。例如，非结构化数据可以包括办公文档、文本、图片、各类报表、图像和音频/视频信息等等。
结合上述分析可知，映射关系用于表征类型参数与模型监控参数之间的对应关系。其中，用户可以通过可视化或者编写代码(如信息抽取库(Jsonpath)语法等)等方式建立类型参数与模型监控参数之间的对应关系。
在建立类型参数与模型监控参数之间的映射关系之前，可以预先对类型参数(包括：输入参数和输出参数)进行配置。例如，可以通过人工的方式进行配置。
值得说明的是，预测服务为对目标模型进行封装得到的微服务。微服务可以理解为将包括预测服务的应用程序松散耦合且独立部署为多个组件或服务，预测服务可以运行在独立的进程中，预测服务与其他服务之间可以采用轻量级的通信机制相互沟通，如超文本传输协议(Hyper Text Transfer Protocol，HTTP)的应用程序的设计风格和开发方式(RESTful)的接口(Application Programming Interface，API)。
示例性的，可以将目标模型封装成微服务，对外提供超文本传输协议接口和全双工通信的协议(WebSocket)接口等方式提供预测服务。
在一些实施例中，待监控参数中还包括与预测服务请求对应的真实结果。
其中，真实结果为真实(GroundTruth)值。真实结果为与预测结果的相对概念，是指输入参数实际对应的结果。示例性的，以人脸图像识别模型为例，输入参数为图像，预测结果是指，该图像经过人脸识别模型而输出的图像中的对象的性别和/或年龄等，真实结果是指实际该图像总的对象的性别和/或年龄。
同理，本实施例对获取真实结果的方式不做限定，例如，可以通过标注的方式获取真实结果，也可以通过其他的方式获取真实结果。
在本实施例中，待监控数据中还包括真实结果，则可以基于预测标识、预测结果、以及真实结果确定监控结果，而真实结果是表征与预测结果对应的实际的预测的结果，因此，通过结合预测结果和真实结果，可以从预测维度和实际维度进一步提高监控的可靠性和准确性的技术效果。
在另一些实施例中，还可以结合目标模型运行时，部署有目标模型的系统的资源消耗信息对模型进行监控，如目标模型运行时，部署有目标模型的系统的中央处理器CPU的资源利用率等，以更为全面的对目标模型进行监控，从而进一步提高监控的可靠性和准确性的技术效果。
同理，在一些实施例中，真实结果也可以被写入至响应头header中。
相应的，在本实施例中，通过将真实结果写入请求头中，可以避免当将真实结果写入主体(body)区域时造成目标模型被入侵的弊端，从而提高监控时对目标模型的安全性进行保护，避免与目标模型相关的信息泄露的技术效果。
值得说明的是，若为单个真实结果的反馈，则可以将真实结果写入至响应头中；若为多个真实结果的反馈，如对多个模型监控，真实结果的数量为多个，则可以将真实结果写入至主体区域，以避免各真实结果之间的干扰。
S203：对待监控数据中的各数据进行分类处理，得到各数据各自对应的数据特征。
其中，数据特征为从不同预测维度确定目标模型的预测结果的数据的特征。
S204：基于至少一种预测维度对应的数据特征的数据，对目标模型进行监控，得到目标模型的监控结果。
图3是根据本公开第三实施例的示意图，如图3所示，本公开实施例的模型的监控方法，包括：
S301：获取待监控数据。
同理，为了避免冗余地陈述，关于本实施例与上述实施例相同的技术特征，本实施例不再赘述。
S302：对待监控数据中的各数据进行特征提取处理，得到各数据各自对应的预测维度。
S303：根据各数据各自对应的预测维度，确定各数据各自对应的数据特征。
示例性的，结合上述分析，以目标模型为人脸识别模型为例，待监控数据的各数据中，若第一数据的数据特征为年龄，第二数据的数据特征为性别，即第一数据为表征图像中对象的年龄相关的数据，第二数据为表征凸显中对象的性别相关的数据。
在本实施例中，通过对各数据进行特征提取处理，以得到各数据各自对应的数据特征，可以使得确定出的数据特征具有较高的准确性和可靠性的技术效果。
S304：基于至少一种预测维度对应的数据特征的数据，对目标模型进行监控，得到目标模型的监控结果。
结合上述分析可知，可以将第一数据作为对目标模型监控的监控对象，即基于第 一数据实现对目标模型的监控；也可以将第二数据作为对目标模型监控的监控对象，即基于第二数据实现对目标模型的监控；也可以将第一数据和第二数据均作为对目标模型监控的监控对象，即基于第一数据和第二数据实现对目标模型的监控。
图4是根据本公开第四实施例的示意图，如图4所示，本公开实施例的模型的监控方法，包括：
S401：获取待监控数据。
同理，为了避免冗余地陈述，关于本实施例与上述实施例相同的技术特征，本实施例不再赘述。
S402：对待监控数据中的各数据进行分类处理，得到各数据各自对应的数据特征。
其中，数据特征为从不同预测维度确定目标模型的预测结果的数据的特征。
S403：根据每一种预测维度对应的数据特征的数据，计算该种预测维度对应的数据特征的数据在预设监控指标下的监控值。
其中，预设监控指标可以由监控装置基于需求、历史记录、以及试验等方式进行确定，本实施例不做限定。
在一些实施例中，提供不同预测服务的不同模型可以设置不同的预设监控指标，例如，分类模型的预设监控指标与回归模型的预设监控指标可以不同。也即，针对模型所能提供的预测服务的类别，可以基于预测服务的类别为与之对应的模型分配用于监控该模型的预设监控指标。
示例性的，结合上述分析，若模型为分类模型，如二分类模型，则预设监控指标可以包括：准确率、精确率、召回率、伪阳性率、F1分数(F1-score)、接收者操作特征(receiver operating characteristic,ROC)曲线下面积、精确率-召回率(precision vs recall，P-R)曲线下面积、统计量(Kolmogorov-Smirnov，KS)、对数损失等。
例如，若目标模型为二分类模型，预设监控指标为准确率，用于确定监控结果的数据为年龄数据，则计算该种预测维度对应的数据特征的数据在预设监控指标下的监控值可以理解为：计算二分类模型基于年龄数据预测得到二分类结果的准确率的值(即为监控值)。
若分类模型具体为多分类模型，则预设监控指标可以包括：准确率、精确率、召回率、伪阳性率、F1分数、加权精准率、加权召回率、加权伪阳性率、加权F1分数等。
若分类模型具体为图像分类模型，则预设监控指标可以包括：准确率、精确率、召回率、F1分数等。
若模型为回归模型，则预设监控指标可以包括：总平方差、误差平方和、回归平方和、平均绝对误差、平均绝对百分误差、均方差根、均方根误差、判定系数、可解释方差、原始列均值、预测结果均值等。
若模型为物体检测模型，则预设监控指标可以包括：目标检测评价指标(mean average precision，mAP)、精确率、召回率等。
若模型为实例分割模型，则预设监控指标可以包括：目标检测评价指标(mean average precision，mAP)、精确率、召回率等。
值得说明的是，上述示例只是用于示范性的说明，可以基于相同的预设监控指标对不同的模型进行监控，也可以基于不同的预设监控指标对不同的模型进行监控，且预设监控指标可能包括上述列举的监控指标，而不能理解为对预设监控指标的限定。
在一些实施例中，S403可以包括如下步骤：
第一步骤：获取该种预测维度对应的数据特征的数据对应的监控周期。
第二步骤：根据该种预测维度对应的数据特征的数据对应的监控周期，计算该种预测维度对应的数据特征的数据在预设监控指标下的监控值。
其中，监控周期是指，预先配置用于对目标模型进行监控的间隔时长。如监控周期为一周，即每隔一周对目标模型进行监控一次，以确定目标模型运行一周时的准确性和可靠性。
值得说明的是，不同数据特征的数据的监控周期可能相同，也可能不同，本实施例不做限定。
示例性的，结合上述实施例，数据特征为年龄的数据的监控周期可以与数据特征为性别的数据的监控周期相同；或者，数据特征为年龄的数据的监控周期可以与数据特征为性别的数据的监控周期不同。
也就是说，可以预先为不同数据特征的数据设置监控周期，以基于监控周期触发监控值的计算，从而基于计算得到的监控值实现对目标模型的监控。
在本实施例中，通过监控周期的方式对目标模型进行监控，可以实现监控的有序性和灵活性的技术效果。
S404：根据各种预测维度对应的数据特征的数据各自对应的监控值确定监控结果。
在本实施例中，通过结合各监控值确定监控结果，可以实现从不同的维度(如年龄、性别等)对目标模型进行监控，从而实现监控的全面性，提高监控的准确性和可靠性的技术效果。
在一些实施例中，若基于监控结果确定目标模型提供预测服务的可靠性相对偏低， 则可以通过邮件、短信、以及调用第三方接口中的一种或多种方式输出预警信息，以便对目标模型进行更新，维护高质量的目标模型。
示例性的，监控结果可以为数据漂移，数据漂移是指预测数据(如上述示例中的输入参数)与训练数据(是指用于训练得到目标模型时的数据)的一致性。
例如，若数据漂移大于预先设置的预警阈值，即预测数据与训练数据的一致性下降，也即预测数据的特征与预测结果分布相比有较大差异，则可以通过邮件、短信、以及调用第三方接口中的一种或多种方式输出预警信息，以便对目标模型进行更新，维护高质量的目标模型。
示例性的，结合上述实施例，第三方接口可以为可被服务平台调用的、服务平台以外的设备提供的接口。相应的，服务平台可以调用第三方接口，以通过第三方接口输出预警信息。
同理，预警阈值可以由监控装置基于需求、历史记录、以及试验等方式进行确定，本实施例不做限定，且不同的模型，或者同一模型的不同数据特征的数据对应的预警阈值，可能相同，也可能不同。
值得说明的是，上述实施例之间可以相互组合，得到新的实施例，如将第一实施例和第四实施例组合，得到新的实施例等，此处不再一一列举。
图5是根据本公开第五实施例的示意图，如图5所示，本公开实施例的模型的监控装置500，包括：
获取单元501，用于获取待监控数据，其中，待监控数据为待监控的目标模型的运行数据中的至少部分数据。
分类单元502，用于对待监控数据中的各数据进行分类处理，得到各数据各自对应的数据特征，其中，不同的数据特征为从不同预测维度确定目标模型的预测结果的数据的特征，一种数据特征对应一种预测维度。
监控单元503，用于基于至少一种预测维度对应的数据特征的数据对目标模型进行监控，得到目标模型的监控结果。
图6是根据本公开第六实施例的示意图，如图6所示，本公开实施例的模型的监控装置600，包括：
获取单元601，用于获取待监控数据，其中，待监控的目标模型的运行数据中包括待监控数据。
结合图6可知，在一些实施例中，获取单元601，包括：
第一获取子单元6011，用于响应于监听到预测服务请求，获取与预测服务请求获 取预测标识。
第二获取子单元6012，用于根据预测标识从目标模型的运行数据中，获取待监控数据。
在一些实施例中，目标模型的运行数据中包括目标类型参数，目标类型参数表征用于提供预测服务的目标模型的类型属性；获取单元601，还包括：
第二确定子单元6013，用于根据预设的映射关系确定待获取的与目标类型参数对应的模型监控参数，其中，映射关系用于表征类型参数与模型监控参数之间的对应关系，模型监控参数为用于计算模型的监控指标的参数。
以及，第二获取子单元6012用于，根据预测标识从目标模型的运行数据中，获取与目标模型参数对应的模型监控参数。
在一些实施例中，获取单元601，还包括：
分析子单元6014，用于对相同类型属性的模型进行分析，得到用于计算相同类型属性的模型对应的监控指标的参数，并将分析得到的参数确定为相同类型属性的模型的模型监控参数，其中，相同类型属性的模型的模型监控参数中包括目标类型参数对应的模型监控参数。
在一些实施例中，预测服务请求包括请求头，请求头中包括预测标识。
在一些实施例中，预测服务为对目标模型进行封装得到的微服务。
分类单元602，用于对待监控数据中的各数据进行分类处理，得到各数据各自对应的数据特征，其中，数据特征为从不同预测维度确定目标模型的预测结果的数据的特征。
结合图6可知，在一些实施例中，分类单元602，包括：
处理子单元6021，用于对待监控数据中的各数据进行分析处理，得到各数据各自对应的预测维度。
第一确定子单元6022，用于根据各数据各自对应的预测维度，确定各数据各自对应的数据特征。
监控单元603，用于基于至少一种预测维度对应的数据特征的数据对目标模型进行监控，得到目标模型的监控结果。
结合图6可知，在一些实施例中，监控单元603，包括：
计算子单元6031，用于根据每一种预测维度对应的数据特征的数据，计算该种预测维度对应的数据特征的数据在预设监控指标下的监控值。
在一些实施例中，计算子单元6031，包括：
获取模块，用于获取该种预测维度对应的数据特征的数据对应的监控周期。
计算模块，用于根据该种预测维度对应的数据特征的数据对应的监控周期，计算该种预测维度对应的数据特征的数据在预设监控指标下的监控值。
第三确定子单元6032，用于根据各种预测维度对应的数据特征的数据各自对应的监控值确定监控结果。
图7是根据本公开第七实施例的示意图，如图7所示，本公开中的电子设备700可以包括：处理器701和存储器702。
存储器702，用于存储程序；存储器702，可以包括易失性存储器(英文：volatile memory)，例如随机存取存储器(英文：random-access memory，缩写：RAM)，如静态随机存取存储器(英文：static random-access memory，缩写：SRAM)，双倍数据率同步动态随机存取存储器(英文：Double Data Rate Synchronous Dynamic Random Access Memory，缩写：DDR SDRAM)等；存储器也可以包括非易失性存储器(英文：non-volatile memory)，例如快闪存储器(英文：flash memory)。存储器702用于存储计算机程序(如实现上述方法的应用程序、功能模块等)、计算机指令等，上述的计算机程序、计算机指令等可以分区存储在一个或多个存储器702中。并且上述的计算机程序、计算机指令、数据等可以被处理器701调用。
上述的计算机程序、计算机指令等可以分区存储在一个或多个存储器702中。并且上述的计算机程序、计算机指据等可以被处理器701调用。
处理器701，用于执行存储器702存储的计算机程序，以实现上述实施例涉及的方法中的各个步骤。
具体可以参见前面方法实施例中的相关描述。
处理器701和存储器702可以是独立结构，也可以是集成在一起的集成结构。当处理器701和存储器702是独立结构时，存储器702、处理器701可以通过总线703耦合连接。
本实施例的电子设备可以执行上述方法中的技术方案，其具体实现过程和技术原理相同，此处不再赘述。
本公开的技术方案中，所涉及的用户个人信息(如图像等)的收集、存储、使用、加工、传输、提供和公开等处理，均符合相关法律法规的规定，且不违背公序良俗。
根据本公开的实施例，本公开还提供了一种电子设备、一种可读存储介质和一种计算机程序产品。
根据本公开的实施例，本公开还提供了一种计算机程序产品，计算机程序产品包 括：计算机程序，计算机程序存储在可读存储介质中，电子设备的至少一个处理器可以从可读存储介质读取计算机程序，至少一个处理器执行计算机程序使得电子设备执行上述任一实施例提供的方案。
图8示出了可以用来实施本公开的实施例的示例电子设备800的示意性框图。电子设备旨在表示各种形式的数字计算机，诸如，膝上型计算机、台式计算机、工作台、个人数字助理、服务器、刀片式服务器、大型计算机、和其它适合的计算机。电子设备还可以表示各种形式的移动装置，诸如，个人数字助理、蜂窝电话、智能电话、可穿戴设备和其它类似的计算装置。本文所示的部件、它们的连接和关系、以及它们的功能仅仅作为示例，并且不意在限制本文中描述的和/或者要求的本公开的实现。
如图8所示，设备800包括计算单元801，其可以根据存储在只读存储器(ROM)802中的计算机程序或者从存储单元808加载到随机访问存储器(RAM)803中的计算机程序，来执行各种适当的动作和处理。在RAM 803中，还可存储设备800操作所需的各种程序和数据。计算单元801、ROM 802以及RAM 803通过总线804彼此相连。输入/输出(I/O)接口805也连接至总线804。
设备800中的多个部件连接至I/O接口805，包括：输入单元806，例如键盘、鼠标等；输出单元807，例如各种类型的显示器、扬声器等；存储单元808，例如磁盘、光盘等；以及通信单元809，例如网卡、调制解调器、无线通信收发机等。通信单元809允许设备800通过诸如因特网的计算机网络和/或各种电信网络与其他设备交换信息/数据。
计算单元801可以是各种具有处理和计算能力的通用和/或专用处理组件。计算单元801的一些示例包括但不限于中央处理单元(CPU)、图形处理单元(GPU)、各种专用的人工智能(AI)计算芯片、各种运行机器学习模型算法的计算单元、数字信号处理器(DSP)、以及任何适当的处理器、控制器、微控制器等。计算单元801执行上文所描述的各个方法和处理，例如模型的监控方法。例如，在一些实施例中，模型的监控方法可被实现为计算机软件程序，其被有形地包含于机器可读介质，例如存储单元808。在一些实施例中，计算机程序的部分或者全部可以经由ROM 802和/或通信单元809而被载入和/或安装到设备800上。当计算机程序加载到RAM 803并由计算单元801执行时，可以执行上文描述的模型的监控方法的一个或多个步骤。备选地，在其他实施例中，计算单元801可以通过其他任何适当的方式(例如，借助于固件)而被配置为执行模型的监控方法。
本文中以上描述的系统和技术的各种实施方式可以在数字电子电路系统、集成电 路系统、场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、芯片上系统的系统(SOC)、复杂可编程逻辑设备(CPLD)、计算机硬件、固件、软件、和/或它们的组合中实现。这些各种实施方式可以包括：实施在一个或者多个计算机程序中，该一个或者多个计算机程序可在包括至少一个可编程处理器的可编程系统上执行和/或解释，该可编程处理器可以是专用或者通用可编程处理器，可以从存储系统、至少一个输入装置、和至少一个输出装置接收数据和指令，并且将数据和指令传输至该存储系统、该至少一个输入装置、和该至少一个输出装置。
用于实施本公开的方法的程序代码可以采用一个或多个编程语言的任何组合来编写。这些程序代码可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理器或控制器，使得程序代码当由处理器或控制器执行时使流程图和/或框图中所规定的功能/操作被实施。程序代码可以完全在机器上执行、部分地在机器上执行，作为独立软件包部分地在机器上执行且部分地在远程机器上执行或完全在远程机器或服务器上执行。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
为了提供与用户的交互，可以在计算机上实施此处描述的系统和技术，该计算机具有：用于向用户显示信息的显示装置(例如，CRT(阴极射线管)或者LCD(液晶显示器)监视器)；以及键盘和指向装置(例如，鼠标或者轨迹球)，用户可以通过该键盘和该指向装置来将输入提供给计算机。其它种类的装置还可以用于提供与用户的交互；例如，提供给用户的反馈可以是任何形式的传感反馈(例如，视觉反馈、听觉反馈、或者触觉反馈)；并且可以用任何形式(包括声输入、语音输入或者、触觉输入)来接收来自用户的输入。
可以将此处描述的系统和技术实施在包括后台部件的计算系统(例如，作为数据服务器)、或者包括中间件部件的计算系统(例如，应用服务器)、或者包括前端部件的计算系统(例如，具有图形用户界面或者网络浏览器的用户计算机，用户可以通 过该图形用户界面或者该网络浏览器来与此处描述的系统和技术的实施方式交互)、或者包括这种后台部件、中间件部件、或者前端部件的任何组合的计算系统中。可以通过任何形式或者介质的数字数据通信(例如，通信网络)来将系统的部件相互连接。通信网络的示例包括：局域网(LAN)、广域网(WAN)和互联网。
计算机系统可以包括客户端和服务器。客户端和服务器一般远离彼此并且通常通过通信网络进行交互。通过在相应的计算机上运行并且彼此具有客户端-服务器关系的计算机程序来产生客户端和服务器的关系。服务器可以是云服务器，又称为云计算服务器或云主机，是云计算服务体系中的一项主机产品，以解决了传统物理主机与VPS服务("Virtual Private Server"，或简称"VPS")中，存在的管理难度大，业务扩展性弱的缺陷。服务器也可以为分布式系统的服务器，或者是结合了区块链的服务器。
根据本申请实施例的另一个方面，本申请实施例还提供了一种计算机程序，包括程序代码，当计算机运行所述计算机程序时，所述程序代码执行如上任一实施例所述的方法。
应该理解，可以使用上面所示的各种形式的流程，重新排序、增加或删除步骤。例如，本发公开中记载的各步骤可以并行地执行也可以顺序地执行也可以不同的次序执行，只要能够实现本公开公开的技术方案所期望的结果，本文在此不进行限制。
上述具体实施方式，并不构成对本公开保护范围的限制。本领域技术人员应该明白的是，根据设计要求和其他因素，可以进行各种修改、组合、子组合和替代。任何在本公开的精神和原则之内所作的修改、等同替换和改进等，均应包含在本公开保护范围之内。

Claims (22)

  1. 一种模型的监控方法，包括：
    获取待监控数据，其中，所述待监控数据为待监控的目标模型的运行数据中的至少部分数据；
    对所述待监控数据中的各数据进行分类处理，得到各数据各自对应的数据特征，其中，不同的数据特征为从不同预测维度确定所述目标模型的预测结果的数据的特征，一种数据特征对应一种预测维度；
    基于至少一种预测维度对应的数据特征的数据，对所述目标模型进行监控，得到所述目标模型的监控结果。
  2. 根据权利要求1所述的方法，其中，对所述待监控数据中的各数据进行分类处理，得到各数据各自对应的数据特征，包括：
    对所述待监控数据中的各数据进行特征分析处理，得到各数据各自对应的预测维度；
    根据各数据各自对应的预测维度，确定各数据各自对应的数据特征。
  3. 根据权利要求1或2所述的方法，其中，获取待监控数据，包括：
    响应于监听到预测服务请求，获取与所述预测服务请求对应的预测标识；
    根据所述预测标识从所述目标模型的运行数据中，获取所述待监控数据。
  4. 根据权利要求3所述的方法，所述目标模型的运行数据中包括目标类型参数，所述目标类型参数表征用于提供预测服务的所述目标模型的类型属性；所述方法还包括：
    根据预设的映射关系确定待获取的与所述目标类型参数对应的模型监控参数，其中，所述映射关系用于表征类型参数与模型监控参数之间的对应关系，模型监控参数为用于计算模型的监控指标的参数；
    以及，根据所述预测标识从所述目标模型的运行数据中，获取所述待监控数据，包括：根据所述预测标识从所述目标模型的运行数据中，获取与所述目标类型参数对应的模型监控参数。
  5. 根据权利要求4所述的方法，其中，在根据预设的映射关系确定待获取的与所述目标类型参数对应的模型监控参数之前，所述方法还包括：
    对相同类型属性的模型进行分析，得到用于计算所述相同类型属性的模型对应的监控指标的参数，并将分析得到的参数确定为所述相同类型属性的模型的模型监控参 数，其中，所述相同类型属性的模型的模型监控参数中包括所述目标类型参数对应的模型监控参数。
  6. 根据权利要求3所述的方法，其中，所述预测服务请求包括请求头，所述请求头中包括所述预测标识。
  7. 根据权利要求4所述的方法，其中，所述预测服务为对所述目标模型进行封装得到的微服务。
  8. 根据权利要求1-7任一项所述的方法，其中，基于至少一种预测维度对应的数据特征的数据对所述目标模型进行监控，得到所述目标模型的监控结果，包括：
    根据每一种预测维度对应的数据特征的数据，计算该种预测维度对应的数据特征的数据在预设监控指标下的监控值；
    根据各种预测维度对应的数据特征的数据各自对应的监控值确定所述监控结果。
  9. 根据权利要求8所述的方法，其中，根据每一种预测维度对应的数据特征的数据，计算该种预测维度对应的数据特征的数据在预设监控指标下的监控值，包括：
    获取该种预测维度对应的数据特征的数据对应的监控周期；
    根据该种预测维度对应的数据特征的数据对应的监控周期，计算该种预测维度对应的数据特征的数据在预设监控指标下的监控值。
  10. 一种模型的监控装置，包括：
    获取单元，用于获取待监控数据，其中，所述待监控数据为待监控的目标模型的运行数据中的至少部分数据；
    分类单元，用于对所述待监控数据中的各数据进行分类处理，得到各数据各自对应的数据特征，其中，不同的数据特征为从不同预测维度确定所述目标模型的预测结果的数据的特征，一种数据特征对应一种预测维度；
    监控单元，用于基于至少一种预测维度对应的数据特征的数据，对所述目标模型进行监控，得到所述目标模型的监控结果。
  11. 根据权利要求10所述的装置，其中，所述分类单元，包括：
    处理子单元，用于对所述待监控数据中的各数据进行分析处理，得到各数据各自对应的预测维度；
    第一确定子单元，用于根据各数据各自对应的预测维度，确定各数据各自对应的数据特征。
  12. 根据权利要求10或11所述的装置，其中，所述获取单元，包括：
    第一获取子单元，用于响应于监听到预测服务请求，获取与所述预测服务请求对 应的预测标识；
    第二获取子单元，用于根据所述预测标识从所述目标模型的运行数据中，获取所述待监控数据。
  13. 根据权利要求12所述的装置，其中，所述目标模型的运行数据中包括目标类型参数，所述目标类型参数表征用于提供预测服务的所述目标模型的类型属性；所述获取单元，还包括：
    第二确定子单元，用于根据预设的映射关系确定待获取的与所述目标类型参数对应的模型监控参数，其中，所述映射关系用于表征类型参数与模型监控参数之间的对应关系，模型监控参数为用于计算模型的监控指标的参数；
    以及，所述第二获取子单元用于，根据所述预测标识从所述目标模型的运行数据中，获取与所述目标模型参数对应的模型监控参数。
  14. 根据权利要求13所述的装置，其中，所述获取单元，还包括：
    分析子单元，用于对相同类型属性的模型进行分析，得到用于计算所述相同类型属性的模型对应的监控指标的参数，并将分析得到的参数确定为所述相同类型属性的模型的模型监控参数，其中，所述相同类型属性的模型的模型监控参数中包括所述目标类型参数对应的模型监控参数。
  15. 根据权利要求12所述的装置，其中，所述预测服务请求包括请求头，所述请求头中包括所述预测标识。
  16. 根据权利要求13所述的装置，其中，所述预测服务为对所述目标模型进行封装得到的微服务。
  17. 根据权利要求10-16任一项所述的装置，其中，所述监控单元，包括：
    计算子单元，用于根据每一种预测维度对应的数据特征的数据，计算该种预测维度对应的数据特征的数据在预设监控指标下的监控值；
    第三确定子单元，用于根据各种预测维度对应的数据特征的数据各自对应的监控值确定所述监控结果。
  18. 根据权利要求17所述的装置，其中，所述计算子单元，包括：
    获取模块，用于获取该种预测维度对应的数据特征的数据对应的监控周期；
    计算模块，用于根据该种预测维度对应的数据特征的数据对应的监控周期，计算该种预测维度对应的数据特征的数据在预设监控指标下的监控值。
  19. 一种电子设备，包括：
    至少一个处理器；以及
    与所述至少一个处理器通信连接的存储器；其中，
    所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行权利要求1-9中任一项所述的方法。
  20. 一种存储有计算机指令的非瞬时计算机可读存储介质，其中，所述计算机指令用于使所述计算机执行根据权利要求1-9中任一项所述的方法。
  21. 一种计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现权利要求1-9中任一项所述方法的步骤。
  22. 一种计算机程序，包括程序代码，当计算机运行所述计算机程序时，所述程序代码执行如权利要求1-9中任一项所述的方法。
PCT/CN2022/106948 2022-02-16 2022-07-21 模型的监控方法和装置 WO2023155378A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210143047.8 2022-02-16
CN202210143047.8A CN114461502B (zh) 2022-02-16 2022-02-16 模型的监控方法和装置

Publications (1)

Publication Number Publication Date
WO2023155378A1 true WO2023155378A1 (zh) 2023-08-24

Family

ID=81413730

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/106948 WO2023155378A1 (zh) 2022-02-16 2022-07-21 模型的监控方法和装置

Country Status (2)

Country Link
CN (1) CN114461502B (zh)
WO (1) WO2023155378A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150106512A1 (en) * 2013-10-16 2015-04-16 International Business Machines Corporation Predictive cloud provisioning based on human behaviors and heuristics
CN109885452A (zh) * 2019-01-23 2019-06-14 平安科技（深圳）有限公司 性能监控方法、装置及终端设备
CN110928859A (zh) * 2019-11-26 2020-03-27 太平金融科技服务（上海）有限公司 模型监控方法、装置、计算机设备和存储介质
CN113704063A (zh) * 2021-08-26 2021-11-26 北京百度网讯科技有限公司 一种云手机的性能监控方法、装置、设备及存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108090678B (zh) * 2017-12-19 2022-08-02 马上消费金融股份有限公司 一种数据模型监控方法、系统、设备及计算机存储介质
CN110688273B (zh) * 2018-07-05 2021-02-19 马上消费金融股份有限公司 分类模型的监控方法、装置、终端以及计算机存储介质
US11204847B2 (en) * 2018-12-21 2021-12-21 Microsoft Technology Licensing, Llc Machine learning model monitoring
CN111290924B (zh) * 2020-01-22 2022-05-10 北京字节跳动网络技术有限公司 监控方法、装置及电子设备
CN111881185A (zh) * 2020-07-30 2020-11-03 苏州浪潮智能科技有限公司 一种数据监控方法、装置、设备及存储介质
CN112529335B (zh) * 2020-12-25 2021-12-31 广州文远知行科技有限公司 一种模型检测方法、装置、设备和存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150106512A1 (en) * 2013-10-16 2015-04-16 International Business Machines Corporation Predictive cloud provisioning based on human behaviors and heuristics
CN109885452A (zh) * 2019-01-23 2019-06-14 平安科技（深圳）有限公司 性能监控方法、装置及终端设备
CN110928859A (zh) * 2019-11-26 2020-03-27 太平金融科技服务（上海）有限公司 模型监控方法、装置、计算机设备和存储介质
CN113704063A (zh) * 2021-08-26 2021-11-26 北京百度网讯科技有限公司 一种云手机的性能监控方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN114461502B (zh) 2023-11-14
CN114461502A (zh) 2022-05-10

Similar Documents

Publication Publication Date Title
JP7437351B2 (ja) 計測手段が組み込まれたソフトウェアを分析するためのデータストリーム処理言語
US10394693B2 (en) Quantization of data streams of instrumented software
US10922206B2 (en) Systems and methods for determining performance metrics of remote relational databases
WO2023103390A1 (zh) 任务处理方法、任务处理装置、电子设备以及存储介质
WO2022134428A1 (zh) 小程序页面渲染方法、装置、电子设备及存储介质
US9832259B2 (en) Method and apparatus for cell configuration
US20230134615A1 (en) Method of processing task, electronic device, and storage medium
WO2017041681A1 (zh) 一种获取和呈现信息的方法及终端
KR20220091441A (ko) 데이터 동기화 방법과 장치, 전자 기기, 저장 매체 및 컴퓨터 프로그램
US10212057B2 (en) Disparate monitoring for IoT
WO2022142048A1 (zh) 唤醒指标监测方法、装置及电子设备
WO2022068183A1 (zh) 配置生成方法、装置、电子设备及存储介质
KR20210125448A (ko) 데이터 어노테이션 방법, 장치, 전자 설비 및 저장매체
WO2023155378A1 (zh) 模型的监控方法和装置
US20160117786A1 (en) Residential pipeline platform methods, apparatuses, and media
US20220398244A1 (en) Query method and device and storage medium
WO2023011093A1 (zh) 任务模型的训练方法、装置、电子设备及存储介质
WO2023015942A1 (zh) 确定图像特征的方法、装置、电子设备和存储介质
US20220129418A1 (en) Method for determining blood relationship of data, electronic device and storage medium
US20230342369A1 (en) Data processing method and apparatus, and electronic device and storage medium
AU2016271116B2 (en) Elektron pulse methods, apparatuses and media
US11196766B2 (en) Detecting denial of service attacks in serverless computing
US10133997B2 (en) Object lifecycle analysis tool
US10992518B2 (en) Segregation of real time event stream
US20230087081A1 (en) Method of detecting event, electronic device, and storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22926684

Country of ref document: EP

Kind code of ref document: A1