CN117952446A

CN117952446A - Monitoring method of business processing model, related equipment and storage medium

Info

Publication number: CN117952446A
Application number: CN202311183275.9A
Authority: CN
Inventors: 李国冬; 李云彬
Original assignee: Mashang Consumer Finance Co Ltd
Current assignee: Mashang Consumer Finance Co Ltd
Priority date: 2023-09-13
Filing date: 2023-09-13
Publication date: 2024-04-30

Abstract

The application discloses a monitoring method of a business processing model, related equipment and a storage medium. The method comprises the following steps: after a first business process model in a development environment is online to a production environment, monitoring performance indexes related to the first business process model through a monitoring plug-in deployed in the production environment, wherein the performance indexes comprise system performance indexes of a business system deployed with the first business process model, data performance indexes of business data in the production environment and model performance indexes of the first business process model, and the business data comprises data required by the first business process model for business process; under the condition that at least one performance index is monitored to be abnormal, in the development environment, optimizing the first business processing model based on a performance optimization strategy matched with the abnormal performance index to obtain a second business processing model; and (3) uploading the second business processing model to the production environment.

Description

Monitoring method of business processing model, related equipment and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method for monitoring a service processing model, a related device, and a storage medium.

Background

With the rapid development of artificial intelligence in recent years, machine learning models are helping businesses make some important decisions. Thus, once these models are deployed into a production environment, dependencies must be maintained in the context of the most current data. At this time, model monitoring is particularly important, and problems existing in the model can be found in advance through model monitoring.

Disclosure of Invention

The embodiment of the application aims to provide a monitoring method, related equipment and storage medium for a business processing model, which are used for realizing multi-dimensional hybrid monitoring and optimization of the business processing model in an online-to-production environment and improving the monitoring comprehensiveness and accuracy.

In order to achieve the above object, the embodiment of the present application adopts the following technical scheme:

In a first aspect, an embodiment of the present application provides a method for monitoring a service processing model, including:

After a first business process model in a development environment is online to a production environment, monitoring performance indexes related to the first business process model through a monitoring plug-in deployed in the production environment, wherein the performance indexes comprise system performance indexes of a business system deployed with the first business process model, data performance indexes of business data in the production environment and model performance indexes of the first business process model, and the business data comprises data required by the first business process model for business process;

Under the condition that at least one performance index is monitored to be abnormal, in the development environment, optimizing the first business processing model based on a performance optimization strategy matched with the abnormal performance index to obtain a second business processing model;

And (3) uploading the second business processing model to the production environment.

In a second aspect, an embodiment of the present application provides a monitoring apparatus for a service processing model, including:

A monitoring unit, configured to monitor, after a first service processing model in a development environment is online to a production environment, performance indexes related to the first service processing model by a monitoring plug-in deployed in the production environment, where the performance indexes include a system performance index of a service system in which the first service processing model is deployed, a data performance index of service data in the production environment, and a model performance index of the first service processing model, and the service data includes data required by the first service processing model for performing service processing;

The optimizing unit is used for optimizing the first business processing model based on a performance optimizing strategy matched with abnormal performance indexes in the development environment under the condition that at least one performance index is monitored to be abnormal, so as to obtain a second business processing model;

And the online unit is used for online the second business processing model to the production environment.

In a third aspect, an embodiment of the present application provides an electronic device, including: a processor; a memory for storing the processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method of monitoring a business process model according to the first aspect.

In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, where instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform a method for monitoring a service processing model according to the first aspect.

The above at least one technical scheme adopted by the embodiment of the application can achieve the following beneficial effects:

Deploying a monitoring plug-in the production environment in advance to collect related data in the development environment; after a first business processing model in a development environment is online to a production environment, monitoring system performance indexes of a business system deployed with the first business processing model, data performance indexes of business data in the production environment, model performance indexes of the first business processing model and other performance indexes through a monitoring plug-in, so that multi-dimensional mixed monitoring of the real situation of the first business processing model in the production environment is realized, and the monitoring accuracy is improved; further, as the abnormality of different performance indexes is caused by different links in the development environment, under the condition that at least one performance index abnormality is monitored, the production environment flows back to the development environment, and the first business processing model can be subjected to targeted optimization based on the performance optimization strategy of abnormal performance index matching, so that the second business processing with better performance is obtained and is online to the production environment, and the business processing requirements in the production environment are better met.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

FIG. 1 is a schematic diagram of an implementation environment to which a method for monitoring a business process model according to an embodiment of the present application is applicable;

FIG. 2 is a schematic flow chart of model monitoring based on an implementation environment according to an embodiment of the present application;

FIG. 3 is a flowchart of a method for monitoring a business process model according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of a monitoring device of a service processing model according to an embodiment of the present application;

Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the application may be practiced otherwise than as specifically illustrated or described herein. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/" generally means that the associated object is an or relationship.

In the related art, a part of test data with labels is usually marked manually to simulate real data in a production environment, and whether the performance of a business processing model meets the requirement is evaluated by testing the business processing model to be online in the production environment through the test data. However, this approach is too ideal, and cannot accurately find the real situation of the business processing model in the production environment, and there are many limitations.

In view of the above, an embodiment of the present application is directed to providing a monitoring method for a service processing model, in which a monitoring plug-in is deployed in advance in a production environment to collect related data in an development environment; after a first business processing model in a development environment is online to a production environment, monitoring system performance indexes of a business system deployed with the first business processing model, data performance indexes of business data in the production environment, model performance indexes of the first business processing model and other performance indexes through a monitoring plug-in, so that multi-dimensional mixed monitoring of the real situation of the first business processing model in the production environment is realized, and the monitoring accuracy is improved; further, as the abnormality of different performance indexes is caused by different links in the development environment, under the condition that at least one performance index abnormality is monitored, the production environment flows back to the development environment, and the first business processing model can be subjected to targeted optimization based on the performance optimization strategy of abnormal performance index matching, so that the second business processing with better performance is obtained and is online to the production environment, and the business processing requirements in the production environment are better met.

It should be understood that the method for monitoring the service processing model provided by the embodiment of the application can be executed by the electronic device or software installed in the electronic device. The electronic devices referred to herein may include terminal devices such as smartphones, tablet computers, notebook computers, desktop computers, intelligent voice interaction devices, intelligent home appliances, smart watches, vehicle terminals, aircraft, etc.; or the electronic device may further include a server, such as an independent physical server, or may be a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server that provides a cloud computing service.

The following describes in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

An implementation environment to which the method for monitoring a service processing model according to the embodiment of the present application is applicable will be described with reference to fig. 1 and fig. 2. As shown in fig. 1, one implementation environment includes a machine learning platform and a monitoring plug-in. The machine learning platform provides a developer with a development environment for developing a machine learning model. The monitoring plug-in is a pre-developed plug-in with data acquisition and processing functions. The monitoring plug-in is deployed in the production environment, and required data can be acquired from the production environment so as to monitor the real situation of the business processing model in the production environment.

And in a development environment, the machine learning platform receives labeling data and a basic model related to the target service, which are provided by a developer, and obtains training data corresponding to the target service by performing data preprocessing, feature engineering and the like on the training data, wherein the training data comprises sample service data and service tags corresponding to the sample service data, the sample service data comprises sample feature values of a plurality of service features, and the service tags comprise reference service processing results corresponding to the sample service data. And then, training and experiment are carried out on the basic model based on the training data of the target service, so as to obtain a service processing model for processing the target service. Finally, judging whether the performance of the business processing model meets the requirement or not by evaluating and comparing the business processing model; if yes, deploying the business processing model into the production environment, and distributing a model service, so that the business processing model is online into the production environment.

After the monitoring plug-in is on line to the production environment, the system performance index of the service system, the data performance index of the service data in the production environment, the model performance index of the service processing model and other various performance indexes are monitored, so that the multi-dimensional mixed monitoring of the service processing model is realized, and the monitoring accuracy is improved. The system performance index may include, for example, a resource utilization rate of a service system, and the data performance index may include, for example, but not limited to, a data quality (such as an outlier, an integrity) of service data in a production environment, a data distribution difference between the production environment and a development environment, and the like, and the model performance index may include, for example, but not limited to, a prediction performance of a service processing model, an input drift, a prediction drift, and the like.

Further, under the condition that any performance index is monitored to be abnormal, the system can flow back to the development environment from the production environment, and in the development environment, the business processing model is optimized based on the abnormal performance index and then is re-online to the production environment. For example, the service features extracted by the feature engineering link are optimized to obtain new training data and train the basic model again, or the model training and experimental link are optimized, the new basic model is adopted for model training, and the like.

Based on the above implementation environment, the following describes a method for monitoring a service processing model according to an embodiment of the present application.

Referring to fig. 3, a flowchart of a method for monitoring a service processing model according to an embodiment of the present application may include the following steps:

S302, after a first business processing model in a development environment is online to a production environment, monitoring performance indexes related to the first business processing model through a monitoring plug-in deployed in the production environment.

The first business process model is a business process model trained in a development environment. The performance metrics include a system performance metric of a business system in which the first business process model is deployed, a data performance metric of business data in the production environment, and a model performance metric of the first business process model.

The system performance index is used to represent the performance of the business system. The system performance metrics may include resource utilization of the business system, interface delay, total number of calls made by the business system to the first business process model, requests per second, and error rate. The resource utilization may include, for example, but not limited to, a CPU (Central Processing Unit ) utilization, a GPU (Graphic Processing Unit, graphics processor) utilization, and a memory utilization, where the resource utilization reflects a response speed of the first business process model. The interface delay reflects the response performance of the service system, and the total number of calls, the number of requests per second, the error rate and the like of the service system to the first service processing model are used for reflecting the service condition of the first service processing model.

The data performance index is used to reflect the reality of the business data in the production environment. The data performance metrics may include data quality of the business data, data distribution differences between the development environment and the production environment, and the like.

The model performance index is used to reflect the performance of the first business process model itself. The model performance metrics may include predictive performance, input drift, predictive drift, etc. of the first business process model. The input drift is used for reflecting the time-dependent change condition of service data input to the first service processing model in the production environment, and the prediction drift is used for reflecting the time-dependent change condition of service processing results output by the first service processing model in the production environment.

In an embodiment, the step S302 may include the following steps: monitoring the service life cycle of the first service processing model in real time, and determining a target performance index corresponding to the service life cycle from the performance indexes; then, the target performance index is monitored through the monitoring plug-in.

The business lifecycle refers to the period between the model being brought up to the production environment and coming down from the model production environment. The service life cycle comprises a plurality of stages such as a first period, a second period and a third period. The first period is an initial period from the model line to the production environment, and specifically may be a first period from the model line to the production environment; the second period is an initial period of running of the model in the production environment, and specifically may refer to a second period from an end time point of an initial stage of online; the third period is a period in which the model is stably operated in the production environment, and may specifically be a period from the end time point of the initial stage of operation.

In order to purposefully monitor the real situation of the first business process in each period, the corresponding target performance index can be determined from the performance indexes for monitoring based on the current period of the first business process model, wherein the factors influencing the first business process model are different in different periods of the business life cycle.

Specifically, the target performance index corresponding to the first period includes a system performance index of the service system. Because the service data used for the first service processing model to process the service in the production environment is less in the first period, the first service processing model is mainly influenced by the performance of the service system, and the performance problem of the first service processing model in the first period can be found in a targeted manner by monitoring the system performance index of the service system, so that the first service processing model is fed back in the development environment in a targeted manner, and the optimized second service processing model can be better suitable for the service system in the first period.

The target performance index corresponding to the second time period includes a data performance index of the business data in the production environment. Because the service data in the production environment is increased in the second period, the processing result of the first service processing model can be directly influenced by the quality of the service data, and the influence of the service data on the performance of the first service processing model can be found in time by monitoring the data performance index of the service data in the production environment, so that the first service processing model is fed back in a targeted manner in the development environment, and the optimized second service processing model can be better suitable for the service data characteristics of the second period.

The target performance index corresponding to the third time period includes a model performance index of the first business process model. Because the business data and the corresponding business processing results in the production environment are comprehensive in the third period, the processing results of the first business processing model are mainly influenced by the model performance index, and the factors influencing the performance of the first business processing model are beneficial to timely finding out by monitoring the model performance index of the first business processing model, so that the first business processing model is fed back in a targeted manner in the development environment, and the optimized second business processing model can perform business processing more accurately in the third period.

In another embodiment, the monitoring plug-in monitors the performance metrics in real time at each stage of the service life of the first service processing model. Under the condition, the comprehensive monitoring of the real situation of the first business processing model can be realized at each stage of the business life cycle of the first business processing model, and the monitoring accuracy is further improved.

The embodiment of the present application herein shows a part of the implementation of S302 described above. Of course, it should be understood that S302 may be implemented in other manners, which are not limited by the embodiment of the present application.

It should be noted that, after S302, various performance indexes may be displayed, so that a developer may better understand the actual situation of the first business processing model in the production environment. Of course, under the condition of monitoring any abnormal performance index, alarm information can be output and pushed to the developer.

And S304, under the condition that at least one performance index is monitored to be abnormal, optimizing the first business processing model based on a performance optimization strategy matched with the abnormal performance index in a development environment to obtain a second business processing model.

Because different performance index anomalies are caused by different links in the development environment, under the condition that at least one performance index anomaly is monitored, the system flows back to the development environment from the production environment, and the first business processing model can be subjected to targeted optimization based on a performance optimization strategy matched by the abnormal performance index, so that second business processing with better performance is obtained and is online to the production environment, and the business processing requirements in the production environment are better met.

In the embodiment of the application, different monitoring modes can be adopted for monitoring aiming at different performance indexes so as to discover abnormal performance indexes in time and improve the monitoring accuracy. Correspondingly, aiming at different abnormal performance indexes, different performance optimization strategies can be adopted to optimize the first business processing model. The following describes the monitoring modes of the system performance index, the data performance index and the model performance index and the corresponding performance optimization strategies respectively.

(1) Regarding the system performance index:

In one embodiment, the system performance index includes resource utilization. In S302 above, the resource utilization is monitored as follows: a1, determining a server node where a service system is located based on service configuration information of the service system; a2, acquiring resource use information of the server node through a monitoring plug-in unit based on a preset sampling time interval; and step A3, determining a system performance index based on the resource use information.

The service configuration information is information for configuring the service system by operation and maintenance personnel, and specifically comprises an address of a server node where the service system is located. The address of the server node can be obtained by analyzing the service configuration information; based on the address of the server node, the monitoring plug-in sends a query request to the server node, and the response data returned by the server node is analyzed through prometheus and other open source tools, so that the resource use information of the server node can be obtained. The resource usage information may include, for example, but is not limited to, CPU utilization, GPU utilization, disk utilization, memory utilization, etc. of the server node at different points in time; and then, analyzing the resource utilization information of all the server nodes, and determining the CPU utilization rate, GPU utilization rate, disk utilization rate, memory utilization rate and the like of the service system as the resource utilization rate of the service system.

In this embodiment, by monitoring the resource utilization rate of the service system, it is helpful to discover the influence of the performance of the service system on the performance of the first service processing model, and further, it is helpful to optimize the corresponding development links in the development environment, so that the first service processing model can be better adapted to the service system.

It should be appreciated that the system performance metrics may also include interface delays for the business system, total number of calls to the first business process model, requests per second, error rates, and the like. Similar to the above-mentioned monitoring method of resource utilization, based on the address of the server node, the above-mentioned system performance ratio can be obtained by sending a corresponding query request to the server node through the monitoring plug-in.

Correspondingly, under the condition of abnormal system performance indexes, a performance optimization strategy matched with the system performance indexes can be adopted in a development environment to optimize the first business processing model.

In an embodiment, the performance optimization strategy for system performance index matching may include: model training is resumed using the new base model. Accordingly, the step S304 may include the following steps:

s341a, acquiring historical training data and a historical base model used for training the first business processing model in a development environment.

S342a, determining a target basic model based on the historical basic model and the system performance index.

For example, the first business processing model is a dialogue model, the history basic model is a large-scale language model, the system performance index indicates that the resource utilization rate of the business system is higher, and because the large-scale language model occupies more system resources, small-scale language models such as Bert and LSTM are adopted as target basic models, so that the resource occupation of the business system after online is reduced. As another example, the first service processing model is a text classification model, the history base model includes an encoder and a classifier with a plurality of full connection layers, the system performance index indicates that the resource utilization rate of the service system is higher, and since too many full connection layers in the classifier may cause more system resources to be occupied in the service processing process, the number of full connection layers in the classifier can be properly reduced, and the obtained target base model includes the encoder and the classifier with fewer full connection layers.

S343a, training the target basic model based on the historical training data to obtain a second business processing model.

In this embodiment, since the abnormal resource utilization rate of the service system is generally caused by the structure of the first service processing model, and the new basic model is adopted to perform model training again under the condition of abnormal resource utilization rate, occupation of the second service processing obtained by training on the service system resource can be reduced, so that the second service processing model can be better suitable for the service system.

(1) Regarding the data performance index:

in one embodiment, the data performance index includes data quality. In S302 above, the data quality is monitored by:

And step B1, acquiring business data in a first historical period from the production environment by monitoring the plug-in.

The service data is the data required by the first service processing model for service processing. The business data differs according to the first business process model. For example, if the first business processing model is a text classification model suitable for a text classification task, the business data is a text to be predicted; if the first business processing model is a prediction model suitable for predicting the consumption level of the user, the business data is historical consumption data of the user to be predicted, such as consumption amount, consumption time, consumption times and the like; if the first service processing model is a voice synthesis model suitable for voice synthesis service, the service data is a text to be processed and a phoneme sequence thereof; if the first service processing model is a mail classification model suitable for a spam classification service, the service data is mail content text, such as a theme, a text, and the like, of the mail to be processed.

The first history period may be set according to actual needs, which is not limited in the embodiment of the present application. For example, if it is monitored that the current first business process model is in the first period of the business lifecycle, the first historical period may be a period between the time the first business process model was online to the production environment to the current time.

And step B2, based on an abnormality detection strategy corresponding to the data type of the service data, performing abnormality detection on the service data to obtain abnormal service data in a first historical period.

The abnormal service data refers to service data with abnormal values. The data types may include an image type and a data type, and the data type includes both a numeric type and a classification type. For image type service data, various self-encoders with anomaly detection function commonly used in the art can be used to perform anomaly detection to determine whether the service data is abnormal.

For the data type business data, whether the abnormal business data is abnormal or not can be determined according to the value range of the business data. For example, the age value range is usually 0-100, and if the acquired age is less than 0, the service data can be determined to be abnormal service data; the value range of the direct administration city comprises Beijing, shanghai, tianjin and Chongqing, and if the obtained direct administration city is Guangzhou, the service data can be determined to be abnormal service data.

For the data type service data, various abnormality detection algorithms commonly used in the field such as an isolated forest algorithm can be adopted to identify the abnormal data type service data in the first history period.

And B3, based on an integrity detection strategy corresponding to the data type of the service data, carrying out integrity detection on the service data to obtain missing service data in the first historical period.

The missing service data refers to the service data with missing value. For the image type service data, various self encoders with missing value detection functions commonly used in the field can be adopted to detect the abnormality and determine the missing image type service data.

For data type service data, the missing data type service data can be determined according to the common identifier of the service data in the service scene. For example, in some service scenarios, an identifier such as Null, none, null, N/a is generally used to represent the missing value, and if it is monitored that the value of a certain service data includes any identifier, the service data is determined to be the missing service data.

And step B4, carrying out quality evaluation on the service data in the first historical period based on the missing service data and the abnormal service data to obtain data quality.

For example, the integrity score may be performed on the service data in the first historical period according to a correspondence between the number of missing service data and the integrity score to obtain a first quality score; according to the corresponding relation between the quantity of the abnormal service data and the abnormal score, carrying out abnormal scoring on the service data in the first historical period to obtain a second quality score; the first quality score and the second quality score are then weighted and summed, and the resulting total quality score can be used to represent the data quality of the production environment. If the total quality score is smaller than the preset score, determining that the data quality of the production environment is abnormal.

In this embodiment, by monitoring missing service data and abnormal service data in the production environment, it is helpful to find out the influence of the service data in the production environment on the first service processing model, and further, it is helpful to optimize the corresponding development links in the development environment, so that the first service processing model can be better adapted to the production environment.

In another embodiment, the data performance indicators may further include data distribution differences between the development environment and the production environment, which may be monitored by:

and step C1, carrying out statistical distribution analysis on the business data in the first historical period to obtain first data distribution information.

In practical applications, the step C1 may be implemented by using various statistical distribution analysis commonly used in the art.

The first data distribution information is used for representing the distribution situation of the service data in the first history period. The first data distribution information may include, for example, but is not limited to: the number of traffic data (count), mean (mean), standard deviation (std), minimum (min), 25% quantile, 50% quantile, 75% quantile, maximum (max), etc.

And C2, acquiring historical training data used for training the first business processing model, and carrying out statistical distribution analysis on the historical training data to obtain second data distribution information.

The embodiment of step C2 is similar to that of step C1, and will not be described again.

The second data distribution information is used for representing the distribution situation of the historical training data. The second data distribution information may include, for example, but is not limited to: the number of historical training data (count), mean (mean), standard deviation (std), minimum (min), 25% quantile, 50% quantile, 75% quantile, maximum (max), etc.

And C3, comparing and analyzing the first data distribution information and the second data distribution information to obtain data distribution differences.

The data distribution difference is used to represent a difference between the first data distribution information and the second data distribution information.

In this embodiment, by monitoring the data distribution difference between the development environment and the production environment, it is helpful to accurately determine whether the first business processing model trained in the development environment is separated from the production environment, and further, it is helpful to optimize the corresponding development link in the development environment, so that the first business processing model can be better adapted to the production environment.

Accordingly, under the condition that the data performance index is abnormal, a performance optimization strategy matched with the data performance index can be adopted in a development environment to optimize the first business processing model.

In an embodiment, the performance optimization strategy for data performance index matching may include: model training is performed again after the training data is updated. Accordingly, the step S304 may include the following steps:

And S341b, acquiring historical training data used for training the first business processing model in the development environment.

And S342b, updating the historical training data based on the data performance index, and optimally training the first business processing model based on the updated historical training data to obtain a second business processing model.

In the specific implementation, if the data quality in the data performance index is abnormal, updating the historical training data based on the missing service data and the abnormal service data in the first historical period, and optimizing and training the first service processing model based on the updated historical training data to obtain a second service processing model.

For example, taking the first service processing model as a junk mail classification model as an example, the service data comprises the text and the theme of the mail, and the history training data comprises the text and the theme of the sample mail and the class label corresponding to the sample mail. If the mail text in the production environment is abnormal and the mail subject is missing, modifying the text of the sample mail in the training data into an abnormal text, and deleting the subject of the sample mail to obtain updated historical training data; and then, optimizing and training the first business processing model by using the updated historical training data, wherein the obtained second business processing can accurately identify whether the mail is junk mail or not under the condition that the mail text is abnormal and the mail subject is missing.

If the data distribution difference in the data performance index is abnormal, updating the historical training data based on the first data distribution information, so that the difference between the data distribution information of the updated historical training data and the first data distribution information is smaller than the preset difference.

For example, the first business process model is taken as a clothes size prediction model, the business data comprises body data such as height and weight of the user, and the historical training data comprises body data such as height and weight of the sample user and clothes size of the sample user. If the difference between the user height distribution in the production environment and the user height distribution in the historical data is large, the height and the clothes size of the sample user are remarked based on the user height distribution in the production environment, so that the updated height distribution in the historical training data is close to the user height distribution in the production environment; and then, optimizing and training the first business processing model by using the updated historical training data, wherein the obtained second business processing model can be suitable for the height distribution characteristics of the user in the production environment, and the clothes size of the user can be predicted more accurately.

In the embodiment, under the condition that the data performance index is abnormal, the historical training data in the development environment is updated based on the data performance index, so that the consistency between the historical training data in the development environment and the business data in the production environment can be improved; further, the updated historical training data is utilized to optimally train the first business processing model, so that the second business processing model obtained through training can be ensured to better adapt to the data characteristics in the production environment, and the accuracy of business processing is improved.

(3) Regarding model performance metrics:

In one embodiment, the model performance metrics include predicted performance of the first business process model. In S302 above, the predicted performance is monitored as follows:

and D1, acquiring service data in a second historical period, service processing results corresponding to the service data and reference processing results from the production environment by monitoring the plug-in.

And the service processing result corresponding to the service data is obtained by performing service processing on the service data by the first service processing model. The second history period may be set according to actual needs, which is not limited by the embodiment of the present application. For example, if it is monitored that the first business process model is currently in the second period of the business lifecycle, the second historical period may be a period between the end time of the first period and the current time.

And D2, if the reference processing result corresponding to the service data in the second historical period meets the preset evaluation condition, determining the prediction performance of the first service processing model based on the service processing result corresponding to the service data and the reference processing result.

The preset evaluation conditions may be set according to actual needs, for example, the number of service data having the reference processing result in the second history period is more than a preset ratio, which is not limited in the embodiment of the present application.

The predictive performance of the first business process model may be represented using a variety of accuracy indicators. The adopted accuracy index is different for different types of first business processing models. In particular, for the first business process model of the classification, the accuracy index employed may include, for example, but is not limited to, at least one of the following: the prediction accuracy rate for single service data, the overall accuracy rate of the first service processing model, the recall rate for single service data, the overall recall rate of the first service processing model, the F1 value and the like; for the first business process model of regression, the accuracy index employed may include, for example, but is not limited to, at least one of the following: average absolute error, root mean square error, etc. The accuracy index corresponding to the first business processing model of the classification can be obtained by calculation according to the following formulas (1) to (5):

Wherein Precision _i represents the prediction accuracy for the ith service data, TP _i represents the number of service data whose reference processing result is positive and whose service processing result is positive, and FP _i represents the number of service data whose reference processing result is negative and whose service processing result is positive; precision _macro represents the overall accuracy of the first business process model, and L represents the total number of business data; recall _i represents the Recall rate for the ith service data, FN _i represents the number of service data with reference to the processing result being positive and the service processing result being negative; recall _macro represents the overall Recall of the first business process model; f1 _macro denotes the F1 value of the first business process model.

In this embodiment, by monitoring the prediction performance of the first service processing model, the prediction performance of the first service processing model in the production environment is facilitated to be accurately obtained, and further, the optimization of the corresponding development links in the development environment is facilitated, so that the first service processing model can be better adapted to the production environment.

In another embodiment, the model performance metrics may further include input drift and prediction drift of the first business process model, which are monitored by:

and E1, if the reference processing result corresponding to the service data in the second history period does not meet the preset evaluation condition, dividing the second history period into a plurality of sub-periods.

For example, the second history period is two weeks before the current time, then the second history period may be divided into two sub-periods, each sub-period being one week, with granularity. As another example, the second history period is one week before the current time, then the second history period may be divided into 7 sub-periods, each sub-period being one day, and so on, with the day granularity. The embodiment of the application does not limit the dividing manner of the second history period.

And E2, carrying out statistical distribution analysis on the service data in the subinterval aiming at each subinterval to obtain third data distribution information corresponding to the subinterval.

The statistical distribution analysis of the service data in each sub-period can be implemented by various statistical distribution analysis methods commonly used in the art, which is not limited in the embodiment of the present application. The third data distribution information corresponding to the subinterval is used for representing the distribution condition of the service data in the subinterval.

And E3, carrying out statistical distribution analysis on service processing results in each subinterval to obtain fourth data distribution information corresponding to the subinterval.

The embodiment of step E3 is similar to the embodiment of step E2 and will not be described again. The fourth data distribution information corresponding to the subinterval is used for representing the distribution condition of the service processing result corresponding to the service data in the subinterval.

And E4, determining input drift based on the difference between the third data distribution information corresponding to two adjacent sub-periods in the plurality of sub-periods.

The input drift is used to represent the distribution variation of the traffic data in the production environment over different subintervals. The input offset may include a distribution difference between service data of adjacent sub-periods.

In practical applications, PSI value, KS value, wasperstein distance, etc. between the service data of two adjacent sub-periods can be used to quantify the input drift. Illustratively, taking the standard deviation in the third data distribution information as an example, the PSI value between the standard deviations of the service data of the adjacent two sub-periods can be calculated by the following formula (6):

Wherein PSI represents a PSI value between standard deviations of service data of the first sub-period and the second sub-period, actual% represents a standard deviation of service data of the first sub-period, and Expected% represents a standard deviation of service data of the second sub-period. It can be understood that the larger the PSI value, the more severe the standard deviation of the traffic data varies between two sub-periods; otherwise, the smaller the standard deviation of the traffic data is changed between two sub-periods. If the PSI value between the service data of any two adjacent sub-periods is larger than a preset PSI value, the input drift abnormality is determined.

And E5, determining prediction drift based on the difference between fourth data distribution information corresponding to two adjacent subintervals in the multiple subintervals.

The prediction drift is used for representing the distribution change condition of service processing results corresponding to service data in the production environment in different subintervals. The input offset may include a distribution difference between service processing results corresponding to the service data in every two adjacent sub-periods.

The embodiment of step E5 is similar to the embodiment of step E4 described above, and will not be described again.

In this embodiment, by monitoring the input drift and the prediction drift of the first service processing model, it is helpful to find out the influence of the data distribution change in the production environment on the performance of the first service processing model, and further, it is helpful to optimize the corresponding development links in the development environment, so that the first service processing model can be better adapted to the production environment.

Accordingly, under the condition that the model performance index of the first business processing model is abnormal, a performance optimization strategy matched with the model performance index can be adopted in a development environment to optimize the first business processing model.

In an embodiment, the performance optimization strategy for model performance index matching may include: and re-acquiring training data to perform model training.

Accordingly, if the predicted performance of the first business process model is abnormal, the step S304 may include the following steps:

and S341c, acquiring a history basic model used for training the first business processing model in the development environment.

And S342c, taking the service data in the second history period as a training sample, taking a reference processing result corresponding to the service data as a label corresponding to the training sample, and training the history basic model based on the training sample and the label corresponding to the training sample to obtain a second service processing model.

For example, taking the first service processing model as a junk mail classification model, service data includes text and subject of the mail, service processing results corresponding to the service data include prediction classification results obtained by classifying the mail by the first service processing model, and reference processing results corresponding to the service data include real categories of the mail. Under the condition, the text and the subject of the mail can be used as training samples, the real type of the mail is used as a label corresponding to the training samples, and the historical basic model is trained by adopting a supervised training method to obtain a second business processing model.

In this embodiment, since the reference processing result obtained from the production environment satisfies the preset evaluation condition, the service data in the production environment and the corresponding reference processing result are used as the training data, so that the inclination of the training data in the development environment can be avoided; based on the training data, the historical basic model corresponding to the first business processing model is trained, so that the second business processing model obtained through training can be better adapted to the production environment, and the accuracy of business processing is improved.

If the input drift or the prediction drift of the first business process model is abnormal, the step S304 may include the following steps:

And S341d, determining a target subperiod from the plurality of subperiods of the second history period.

The difference between the target subinterval and other subintervals on the third data distribution information is greater than a preset difference. The other sub-period is a sub-period excluding the target sub-period from the above-described plurality of sub-periods. For example, the second history period includes a sub-period 1, a sub-period 2, and a sub-period 3, and the difference in the third data distribution information between the sub-period 2 and other sub-periods is large, so that the sub-period 2 is determined as a target sub-period, and the other sub-periods include the sub-period 1 and the sub-period 3.

S342d, acquiring the historical business event occurring in the target subperiod.

And S343d, if the historical service event does not belong to the abnormal service event, carrying out statistical distribution analysis on the service data in the second historical period to obtain fifth data distribution information.

The fifth data distribution information is used to represent the distribution of the service data in the second history period.

And S344d, acquiring target training data and a history base model used for training the first business processing model based on the fifth data distribution information.

The difference between the data distribution information of the target training data and the fifth data distribution information is smaller than or equal to a preset difference.

For example, the first business processing model is taken as a consumption flow prediction model, and the business data comprises historical consumption user quantity of the merchant and the like. Assuming that the second history period includes sub-period 1, sub-period 2, and sub-period 3, the difference in the number of history consumption users between the sub-period 2 and other sub-periods is large, and thus the sub-period 2 is determined as the target sub-period. If the marketing activity is carried out by the merchant in the subinterval 2 and belongs to an abnormal business event, determining that the performance of the first business processing model in the subinterval 2 is normal; if the commercial tenant does not conduct marketing activities in the subinterval 2, determining that the performance of the first business processing model in the subinterval 2 is normal, and further conducting statistical analysis on the number of historical consumption users in the second historical interval to obtain a distribution rule of historical consumption user data; then, the target training data having the same distribution rule is reacquired.

And S345d, training the historical basic model based on the target training data to obtain a second business processing model.

In this embodiment, by analyzing the data distribution information of each sub-period in the second history period, determining a target sub-period in which the data distribution information is abnormal, and re-acquiring the target training data by using the data distribution information of the service data in the second history period when no abnormal service event occurs in the target sub-period, the distribution consistency between the model training data and the service data in the production environment can be improved; further, the historical basic model corresponding to the first business processing model is trained by utilizing the training data, so that the second business processing model obtained through training can be ensured to be better adapted to the data distribution characteristics in the production environment, and the accuracy of business processing is improved.

S306, the second business processing model is online to the production environment.

And deploying the second business processing model into a business system, and distributing a model service, namely finishing the online of the second business processing model.

According to the monitoring method of the business processing model provided by one or more embodiments of the application, a monitoring plug-in is deployed in the production environment in advance so as to collect related data in the development environment; after a first business processing model in a development environment is online to a production environment, monitoring a plurality of performance indexes such as a system performance index of a business system deployed with the first business processing model, a data performance index of business data in the production environment and a model performance index of the first business processing model through a monitoring plug-in, so that multi-dimensional mixed monitoring of the real situation of the first business processing model in the production environment is realized, and the monitoring accuracy is improved; further, as the abnormality of different performance indexes is caused by different links in the development environment, under the condition that at least one performance index abnormality is monitored, the production environment flows back to the development environment, and the first business processing model can be subjected to targeted optimization based on the performance optimization strategy of abnormal performance index matching, so that the second business processing with better performance is obtained and is online to the production environment, and the business processing requirements in the production environment are better met.

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Based on the same inventive concept, the embodiment of the application also provides a monitoring device of the business processing model. As shown in fig. 4, a schematic structural diagram of a monitoring device 400 for a service processing model according to an embodiment of the present application includes a monitoring unit 410, an optimizing unit 420, and an on-line unit 430.

The monitoring unit 410 is configured to monitor, after a first service processing model in a development environment is online to a production environment, performance indexes related to the first service processing model by a monitoring plug-in deployed in the production environment, where the performance indexes include a system performance index of a service system in which the first service processing model is deployed, a data performance index of service data in the production environment, and a model performance index of the first service processing model, and the service data includes data required by the first service processing model for performing service processing.

And the optimizing unit 420 is configured to optimize the first business processing model based on a performance optimization policy matched with the abnormal performance index in the development environment under the condition that at least one performance index is monitored to be abnormal, so as to obtain a second business processing model.

And an online unit 430, configured to online the second business processing model into the production environment.

Optionally, the monitoring unit 410 is specifically configured to:

monitoring the service life cycle of the first service processing model in real time;

Determining a target performance index corresponding to the service life cycle from the performance indexes;

and monitoring the target performance index through the monitoring plug-in.

Optionally, the service life cycle includes a first period, a second period, and a third period, the target performance index corresponding to the first period includes the system performance index, the system performance index corresponding to the second period includes the data performance index, and the system performance index corresponding to the third period includes the model performance index.

Optionally, the system performance index includes a resource utilization, the resource utilization being monitored by:

Determining a server node where the service system is located based on service configuration information of the service system;

Acquiring resource use information of the server node through the monitoring plug-in based on a preset sampling time interval;

and determining the resource utilization rate of the service system based on the resource utilization information.

Optionally, the performance optimization strategy for matching the system performance index includes: model training is conducted again by using the new basic model;

the optimizing unit 420 is specifically configured to:

if the system performance index is abnormal, acquiring historical training data and a historical basic model used for training the first business processing model in the development environment;

determining a target base model based on the historical base model and the system performance index;

and training the target basic model based on the historical training data to obtain the second business processing model.

Optionally, the data performance indicator comprises a data quality, the data quality being monitored by:

Acquiring service data in a first historical period from the production environment through the monitoring plug-in, wherein the service data is data required by the first service processing model for service processing;

Based on an abnormality detection strategy corresponding to the data type of the service data, carrying out abnormality detection on the service data to obtain abnormal service data in the first historical period;

Based on an integrity detection strategy corresponding to the data type of the service data, carrying out integrity detection on the service data to obtain missing service data in the first historical period;

and carrying out quality evaluation on the service data in the first historical period based on the missing service data and the abnormal service data to obtain the data quality.

Optionally, the data performance index further comprises a data distribution difference between the development environment and the production environment, the data distribution difference being monitored by:

carrying out statistical distribution analysis on the business data in the first historical period to obtain first data distribution information;

acquiring historical training data used for training the first business processing model, and carrying out statistical distribution analysis on the historical training data to obtain second data distribution information;

And comparing and analyzing the first data distribution information and the second data distribution information to obtain the data distribution difference.

Optionally, the performance optimization strategy for matching the data performance indexes includes: model training is carried out again after the training data are updated;

the optimizing unit 420 is specifically configured to:

If the data performance index is abnormal, acquiring historical training data used for training the first business processing model in the development environment;

updating the historical training data based on the data performance index, and optimally training the first business processing model based on the updated historical training data to obtain the second business processing model.

Optionally, the model performance index includes a predicted performance of the first business process model, the predicted performance being monitored by:

Acquiring service data in a second historical period, a service processing result corresponding to the service data and a reference processing result from the production environment through the monitoring plug-in, wherein the service processing result is obtained by performing service processing on the service data by the first service processing model;

and if the reference processing result corresponding to the service data in the second historical period meets a preset evaluation condition, determining the prediction performance of the first service processing model based on the service processing result corresponding to the service data and the reference processing result.

Optionally, the performance optimization strategy for matching the model performance indexes includes: re-acquiring training data to perform model training;

the optimizing unit 420 is specifically configured to:

If the predicted performance is abnormal, acquiring a history basic model used for training the first business processing model in the development environment;

and taking the service data in the second history period as a training sample, taking a reference processing result corresponding to the service data as a label corresponding to the training sample, and training the history basic model based on the training sample and the label corresponding to the training sample to obtain the second service processing model.

Optionally, the model performance index further comprises an input drift and a prediction drift of the first business process model, the input drift and the prediction drift being monitored by:

if the reference processing result corresponding to the service data in the second history period does not meet the preset evaluation condition, dividing the second history period into a plurality of sub-periods;

for each subinterval, carrying out statistical distribution analysis on service data in the subinterval to obtain third data distribution information corresponding to the subinterval, and carrying out statistical distribution analysis on service processing results in the subinterval to obtain fourth data distribution information corresponding to the subinterval;

Determining the input drift based on a difference between third data distribution information corresponding to two adjacent sub-periods of the plurality of sub-periods;

And determining the prediction drift based on the difference between fourth data distribution information corresponding to two adjacent sub-periods in the plurality of sub-periods.

the optimizing unit 420 is specifically configured to:

If the input drift or the prediction drift is abnormal, determining a target subinterval from the plurality of subintervals, wherein the difference between the target subinterval and other subintervals on third data distribution information is larger than a preset difference, and the other subintervals comprise subintervals except the target subinterval in the plurality of subintervals;

acquiring a historical service event occurring in the target subperiod;

If the historical service event does not belong to the abnormal service event, carrying out statistical distribution analysis on the service data in the second historical period to obtain fifth data distribution information;

acquiring target training data and a history basic model used for training the first business processing model based on the fifth data distribution information, wherein the difference between the data distribution information of the target training data and the fifth data distribution information is smaller than or equal to the preset difference;

and training the historical basic model based on the target training data to obtain the second business processing model.

Obviously, the monitoring device for the service processing model provided by the embodiment of the application can be used as an execution main body of the monitoring method for the service processing model shown in fig. 3, so that the function of the monitoring method for the service processing model in fig. 3 can be realized. The principle is the same and will not be described again.

Fig. 5 is a schematic structural view of an electronic device according to an embodiment of the present application. Referring to fig. 5, at the hardware level, the electronic device includes a processor, and optionally an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.

The processor, network interface, and memory may be interconnected by an internal bus, which may be an ISA (Industry Standard Architecture ) bus, a PCI (PERIPHERAL COMPONENT INTERCONNECT, peripheral component interconnect standard) bus, or EISA (Extended Industry Standard Architecture ) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in FIG. 5, but not only one bus or type of bus.

And the memory is used for storing programs. In particular, the program may include program code including computer-operating instructions. The memory may include memory and non-volatile storage and provide instructions and data to the processor.

The processor reads the corresponding computer program from the nonvolatile memory to the memory and then runs the computer program to form a monitoring device of the business processing model on a logic level. The processor is used for executing the programs stored in the memory and is specifically used for executing the following operations:

The method performed by the monitoring device of the service processing model disclosed in the embodiment of fig. 3 of the present application may be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but may also be a digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be embodied directly in the execution of a hardware decoding processor, or in the execution of a combination of hardware and software modules in a decoding processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.

The electronic device may also execute the method of fig. 3 and implement the functions of the monitoring device of the service processing model in the embodiment shown in fig. 3, which is not described herein again.

Of course, other implementations, such as a logic device or a combination of hardware and software, are not excluded from the electronic device of the present application, that is, the execution subject of the following processing flows is not limited to each logic unit, but may be hardware or a logic device.

The embodiments of the present application also provide a computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a portable electronic device comprising a plurality of application programs, enable the portable electronic device to perform the method of the embodiment shown in fig. 3, and in particular to perform the operations of:

In summary, the foregoing description is only of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

Claims

1. A method for monitoring a business process model, comprising:

2. The method of claim 1, wherein monitoring performance metrics associated with the first business process model via a monitoring plug-in deployed in the production environment comprises:

and monitoring the target performance index through the monitoring plug-in.

3. The method of claim 2, wherein the service lifecycle comprises a first time period, a second time period, and a third time period, wherein the target performance index corresponding to the first time period comprises the system performance index, wherein the system performance index corresponding to the second time period comprises the data performance index, and wherein the system performance index corresponding to the third time period comprises the model performance index.

4. The method of claim 1, wherein the system performance indicator comprises a resource utilization, the resource utilization being monitored by:

5. The method of claim 4, wherein the performance optimization strategy for system performance index matching comprises: model training is conducted again by using the new basic model;

under the condition that at least one performance index is monitored to be abnormal, in the development environment, optimizing the first business processing model based on a performance optimization strategy matched with the abnormal performance index to obtain a second business processing model, wherein the method comprises the following steps:

6. The method of claim 1, wherein the data performance indicator comprises a data quality, the data quality being monitored by:

7. The method of claim 6, wherein the data performance metrics further comprise a data distribution difference between the development environment and the production environment, the data distribution difference being monitored by:

8. The method according to claim 6 or 7, wherein the performance optimization strategy for data performance index matching comprises: model training is carried out again after the training data are updated;

9. The method of claim 1, wherein the model performance metrics comprise a predicted performance of the first business process model, the predicted performance being monitored by:

10. The method of claim 9, wherein the performance optimization strategy for model performance index matching comprises: re-acquiring training data to perform model training;

11. The method of claim 9, wherein the model performance metrics further comprise an input drift and a prediction drift of the first business process model, the input drift and the prediction drift being monitored by:

12. The method of claim 11, wherein the performance optimization strategy for model performance index matching comprises: re-acquiring training data to perform model training;

If the input drift or the prediction drift is abnormal, determining a target subinterval from the plurality of subintervals, wherein the difference between the target subinterval and other subintervals on third data distribution information is larger than a preset difference, and the other subintervals are subintervals which do not comprise the target subinterval in the plurality of subintervals;

acquiring a historical service event occurring in the target subperiod;

13. A monitoring device for a business process model, comprising:

14. An electronic device, comprising:

A processor;

A memory for storing the processor-executable instructions;

Wherein the processor is configured to execute the instructions to implement the method of monitoring a business process model according to any one of claims 1 to 12.

15. A computer readable storage medium, characterized in that instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the method of monitoring a business process model according to any one of claims 1 to 12.