CN115658419A - Model data monitoring method, device, medium and equipment - Google Patents

Abstract

The embodiment of the application provides a method, a device, a medium and equipment for monitoring model data, wherein the method comprises the following steps: receiving a data acquisition request; acquiring a target predicted value and a target real value of data to be predicted in the data acquisition request; receiving a data calculation request; calling a target algorithm corresponding to a target model index in the data calculation request; substituting the target predicted value and the target true value into a target algorithm to calculate to obtain target result data; generating a performance report of the target prediction model according to target result data; and monitoring and early warning operation is carried out on the performance of the target prediction model based on the performance report. By using the model data monitoring method provided by the embodiment of the application, a user can automatically obtain a performance report of a target prediction model for predicting data to be predicted by selecting the specified data to be predicted and the specified model index on a platform, and automatically monitor and early warn the performance of the target prediction model according to the performance report.

Description

Model data monitoring method, device, medium and equipment

technical field

The present application relates to the technical field of electronic communication, in particular to the technical field of model data monitoring, in particular to a model data monitoring method, device, medium and equipment.

Background technique

Model monitoring is one of the frontier research fields of AI in recent years. As the application scenarios of models in actual production of enterprises grow exponentially, the effect and stability of models will also directly affect the production efficiency of enterprise products. For example, the bank's risk control model will directly affect core business scenarios such as user loans and account opening. If the model fails, it will not be able to accurately predict the user's risk level, which will directly affect the user's use process and cause loss of revenue.

Therefore, algorithm engineers need to update and iterate or offline models in a timely manner to avoid production accidents caused by model problems, which leads to the need for model monitoring.

Prior to this, algorithm engineers mainly performed offline index query by writing their own codes, which took a considerable amount of time to write codes. For example, there are 100 models, and each model takes 5 days to write monitoring codes, which will take 500 days To do model monitoring, as the model grows, more labor costs are required.

Contents of the invention

The embodiment of the present application provides a model data monitoring method, device, medium and equipment. By using the model data monitoring method provided in the embodiment of the present application, the user can select and specify the data to be predicted and the specified model index on the platform, and then automatically obtain The performance report of the target prediction model used to predict the data to be predicted, and automatically monitor and warn the performance of the target prediction model according to the performance report.

An embodiment of the present application provides a model data monitoring method on the one hand, and the model data monitoring method includes:

Receive data acquisition request;

Acquiring a target predicted value and a target real value of the data to be predicted in the data acquisition request, the target predicted value indicates the predicted result of the data to be predicted through a target prediction model, and the target real value indicates the real result of the data to be predicted;

Receive data calculation request;

calling the target algorithm corresponding to the target model index in the data calculation request;

Based on the target algorithm, substituting the target predicted value and target real value for calculation to obtain target result data;

generating a performance report of the target predictive model based on the target outcome data;

The performance of the target prediction model is monitored and pre-warned based on the performance report.

In the model data monitoring method described in the embodiment of the present application, the acquisition of the target predicted value and target actual value of the data to be predicted in the data acquisition request includes:

Generate a target SQL statement based on the preset identification of the data to be predicted and the pre-generated SQL statement template;

The target predicted value and target real value corresponding to the data to be predicted are obtained from the database according to the target SQL statement.

In the model data monitoring method described in the embodiment of the present application, the method further includes:

Receive data comment requests;

Obtain the remark information in the data remark request;

The remark information is inserted into the target model indicator, so that the target model indicator is displayed in a preset form in the performance report and the meaning of the target model indicator is explained.

In the model data monitoring method described in the embodiment of the present application, the monitoring and early warning operation on the performance of the target prediction model based on the performance report includes:

Comparing each target result data in the performance report with its corresponding preset threshold to obtain a comparison result;

If the comparison result is abnormal, generating prompt information including the comparison result.

In the model data monitoring method described in the embodiment of the present application, the method further includes:

According to the pre-configured weight value of each target model index, different early warning levels are set for different target model indicators, and the early warning levels of different target model indicators are written into the corresponding comparison results.

In the model data monitoring method described in the embodiment of the present application, the method further includes:

Receive data output request;

determining the target performance report and target receiving address in the data output request;

establishing a connection interface between the target performance report and the target receiving address;

Transmitting the target performance report to the target receiving address.

In the model data monitoring method described in the embodiment of the present application, the method further includes:

Receive data view request;

A monitoring list page including multiple performance reports is provided according to the data viewing request, and the multiple performance reports belong to multiple prediction models of the same type.

Correspondingly, another aspect of the embodiment of the present application provides a model data monitoring device, the model data monitoring device includes:

A first receiving module, configured to receive a data acquisition request;

A data acquisition module, configured to acquire a target predicted value and a target real value of the data to be predicted in the data acquisition request, the target predicted value indicates the forecast result obtained by the target forecast model for the data to be predicted, and the target real value indicates the target real value to be predicted Predict the true outcome of the data;

A second receiving module, configured to receive a data calculation request;

An algorithm calling module, configured to call the target algorithm corresponding to the target model index in the data calculation request;

The data calculation module is used to calculate the target result data by substituting the target predicted value and the target actual value based on the target algorithm;

A report generating module, configured to generate a performance report of the target prediction model according to the target result data;

A monitoring and early warning module, configured to perform monitoring and early warning operations on the performance of the target prediction model based on the performance report.

Correspondingly, another aspect of the embodiment of the present application provides a storage medium, where a plurality of instructions are stored in the storage medium, and the instructions are suitable for being loaded by a processor to execute the model data monitoring method as described above.

Correspondingly, another aspect of the embodiment of the present application provides a terminal device, including a processor and a memory, the memory stores a plurality of instructions, and the processor loads the instructions to perform the above-mentioned model data monitoring method.

The embodiment of the present application provides a model data monitoring method, device, medium, and equipment. The method receives a data acquisition request; acquires the target predicted value and target real value of the data to be predicted in the data acquisition request, and the target forecast The value indicates the prediction result of the data to be predicted through the target prediction model, and the true target value indicates the real result of the data to be predicted; receive the data calculation request; call the target algorithm corresponding to the target model index in the data calculation request; The target algorithm is substituted into the target predicted value and the target real value for calculation to obtain the target result data; according to the target result data, a performance report of the target prediction model is generated; based on the performance report, the performance of the target prediction model is Carry out monitoring and early warning operations. Using the model data monitoring method provided by the embodiment of the present application, the user can automatically obtain the performance report of the target prediction model used to predict the data to be predicted by selecting the specified data to be predicted and the specified model index on the platform, and according to The performance report automatically monitors and warns the performance of the target prediction model.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings that need to be used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative efforts.

FIG. 1 is a schematic flowchart of a model data monitoring method provided in an embodiment of the present application.

FIG. 2 is a schematic structural diagram of a model data monitoring device provided by an embodiment of the present application.

Fig. 3 is another schematic structural diagram of the model data monitoring device provided by the embodiment of the present application.

FIG. 4 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Apparently, the described embodiments are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts belong to the protection scope of this application.

It should be noted that the following content is a brief introduction to the background of this program:

This solution is mainly carried out around the technical problem of "the current model performance monitoring method is highly dependent on manual labor and takes a long time, resulting in rising labor costs and time costs". It is understandable that at present, in order for algorithm engineers to update and iterate models with degraded performance or to go offline in a timely manner, so as to avoid production accidents caused by model problems, algorithm engineers mainly query offline indicators by writing their own codes, which requires a lot of effort. It takes a lot of time to write code. For example, if there are 100 models, each model takes 5 days to write monitoring code, and it will take 500 days to do model monitoring. As the model grows, more labor and time costs are required .

In order to solve the above technical problem, an embodiment of the present application provides a method for monitoring model data. Using the model data monitoring method provided by the embodiment of the present application, the user can select and specify the data to be predicted and the specified model index on the platform, and then automatically obtain the performance report of the target prediction model used to predict the data to be predicted, and according to the performance report Automatic monitoring and early warning of the performance of the target prediction model.

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of a model data monitoring method provided in an embodiment of the present application. The model data monitoring method is applied to terminal equipment. Optionally, the terminal device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication , middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, etc., but is not limited thereto.

In one embodiment, the method may include the following steps:

Step 101, receiving a data acquisition request.

Wherein, the data acquisition request refers to a request sent to the terminal device when a user such as an algorithm engineer needs to check the performance of a specified prediction model when predicting specified result data. For example, when the algorithm engineer wants to know the excellent performance of the bank's risk control model in predicting the user's risk level, he can initiate a data acquisition request to the terminal device based on the bank's risk control model and user data. Specifically, the data acquisition request may be initiated by a user such as an algorithm engineer to the terminal device through an instruction input device such as a keyboard or a mouse.

Step 102: Obtain the target predicted value and target real value of the data to be predicted in the data acquisition request, the target predicted value indicates the prediction result of the data to be predicted through the target prediction model, and the target real value indicates the target value of the data to be predicted real results.

Wherein, the target prediction value indicates the prediction result of the data to be predicted through the target prediction model, and the target real value indicates the real result of the data to be predicted. In simple terms, the target prediction value is the prediction result obtained by the target prediction model through data processing based on the user data input by the user, and its accuracy depends on the performance of the prediction model. The target real value is a relatively objective real result obtained through manual calculation, and the default is absolute real.

In some embodiments, the target predicted value and the target actual value can be obtained in the following ways:

Based on the preset identification of the data to be predicted (such as the character string "xxx" representing the user's identity) and the pre-generated SQL statement template, the target SQL statement is generated, and the target prediction value corresponding to the data to be predicted is obtained from the database according to the target SQL statement and target true value.

What needs to be explained is that the SQL statement template contains multiple variables, including at least "preset identifier", "predicted value", and "true value". By modifying the variables in the SQL statement template to generate the target SQL statement, the target SQL statement can be targeted The target predicted value and target real value corresponding to the data to be predicted are obtained from the database. And by setting the SQL statement template, it is convenient for the user to quickly construct the SQL statement without writing it separately, saving time and cost and improving efficiency.

Step 103, receiving a data calculation request.

Among them, since there are usually multiple indicators that can reflect the excellent performance of the prediction model, the user can select a specified model indicator according to his own needs, and initiate a data calculation request to the terminal device. For example, the classification model generally needs to look at the ROC, KS, GINI, confusion matrix, etc. of the model, and the regression model is MSE, RMSE, MAE, R-Square, etc.

Step 104, call the target algorithm corresponding to the target model index in the data calculation request.

Among them, the target algorithm refers to the algorithm used to calculate the target model index specified by the user. Since the model indicators mentioned in this plan can be calculated using existing mature formula algorithms, and this plan does not improve the formula algorithm, Therefore, the formula algorithm will not be listed and explained.

Step 105, based on the target algorithm, substitute the target predicted value and the target real value for calculation to obtain target result data.

Wherein, based on the target algorithm corresponding to the user target model index, the target predicted value and target real value obtained above are substituted into the target algorithm for calculation to obtain the target result data. That is, ROC, KS, GINI, confusion matrix, etc. corresponding to the classification model, and MSE, RMSE, MAE, R-Square, etc. corresponding to the regression model.

Step 106, generating a performance report of the target prediction model according to the target result data.

Among them, the performance report specifically shows the target result data corresponding to each target model index of the target prediction model. In some embodiments, the performance report provides two types of online report and excel report, and the report records the status of each model monitoring operation ;Click to view the indicator trend, you can see the change trend of the indicator, which is convenient for the algorithm personnel to adjust the monitoring strategy in time; click on the log, you can see the execution process of each indicator in each run, and it is convenient to locate the error of the indicator operation failure.

In some embodiments, in order to make it easier for people viewing the performance report to understand and understand the contents of the performance report, this solution provides a data comment function. Algorithm engineers can initiate a data remark request containing remark information to the terminal device, and the terminal device will insert the remark information into the target model indicator after obtaining the remark information in the data remark request, so that the target model indicator can be displayed in a preset form in the performance report and Explain the meaning of the target model metrics. Exemplarily, the target model index can be displayed in the form of an embedded comment in the performance report, and the corresponding remark information can pop up when the user clicks on the corresponding target result parameter.

Step 107 , monitor and warn the performance of the target prediction model based on the performance report.

Among them, each target result data in the performance report is compared with its corresponding preset threshold to obtain the comparison result. If the comparison result is abnormal, a prompt message containing the comparison result is generated and the algorithm engineer is notified to realize the target prediction model. Performance monitoring and early warning.

In some embodiments, different warning levels can be set for different target model indicators according to the pre-configured weight values of each target model indicator, and the warning levels of different target model indicators can be written into the corresponding comparison results. It can also trigger different warning levels according to different weights, and automatically trigger warning emails or text messages according to the warning level.

In some embodiments, the user can actively send a data viewing request to the terminal device to view the target performance report in a directional manner. After receiving the data viewing request, the terminal device provides a monitoring list page containing multiple performance reports according to the data viewing request, and each performance report can be viewed separately. It should be noted that multiple performance reports belong to multiple prediction models of the same type on the same monitoring list page, which is convenient for centralized management.

In some embodiments, a performance report output interface is also provided, which can be connected with other systems to provide result data corresponding to various target model indicators, so as to facilitate other operations, such as further analysis of the performance report. Specifically, the terminal device receives the data output request sent by the user, determines the target performance report and the target receiving address in the data output request, establishes a connection interface between the target performance report and the target receiving address, and transmits the target performance report to the target receiving address .

All the above optional technical solutions may be combined in any way to form optional embodiments of the present application, which will not be repeated here.

During specific implementation, the present application is not limited by the execution order of the described steps, and some steps may be performed in other orders or simultaneously in the case of no conflict.

As can be seen from the above, the model data monitoring method provided by the embodiment of the present application receives a data acquisition request; acquires the target predicted value and target real value of the data to be predicted in the data acquisition request, and the target predicted value indicates that the data to be predicted passes through the target The prediction result obtained by the prediction model, the target real value indicates the real result of the data to be predicted; receiving the data calculation request; calling the target algorithm corresponding to the target model index in the data calculation request; based on the target algorithm, substituting the Calculate the target predicted value and target real value to obtain target result data; generate a performance report of the target prediction model according to the target result data; perform monitoring and early warning operations on the performance of the target prediction model based on the performance report. Using the model data monitoring method provided by the embodiment of the present application, the user can automatically obtain the performance report of the target prediction model used to predict the data to be predicted by selecting and specifying the data to be predicted and the specified model index on the platform, and according to the performance The report automatically monitors and warns the performance of the target prediction model.

The embodiment of the present application also provides a model data monitoring device, and the model data monitoring device can be integrated into a terminal device.

Please refer to FIG. 2 . FIG. 2 is a schematic structural diagram of a model data monitoring device provided in an embodiment of the present application. The model data monitoring device 30 may include:

The first receiving module 31 is configured to receive a data acquisition request;

The data acquisition module 32 is configured to acquire the target predicted value and the target real value of the data to be predicted in the data acquisition request, the target predicted value indicates the forecast result obtained by the target forecast model for the data to be predicted, and the target real value indicates The real result of the data to be predicted;

The second receiving module 33 is configured to receive a data calculation request;

An algorithm calling module 34, configured to call the target algorithm corresponding to the target model index in the data calculation request;

The data calculation module 35 is used to calculate the target result data by substituting the target predicted value and the target actual value based on the target algorithm;

A report generating module 36, configured to generate a performance report of the target prediction model according to the target result data;

The monitoring and early warning module 37 is configured to perform monitoring and early warning operations on the performance of the target prediction model based on the performance report.

In some embodiments, the data acquisition module 32 is configured to generate a target SQL statement based on the preset identification of the data to be predicted and the pre-generated SQL statement template; The target predicted value and target actual value corresponding to the data to be predicted.

In some embodiments, the device further includes a data remark module, configured to receive a data remark request; obtain remark information in the data remark request; insert the remark information into the target model indicator, so that the target The model index is displayed in a preset form in the performance report and explains the meaning of the target model index.

In some embodiments, the monitoring and early warning module 37 is configured to compare each target result data in the performance report with its corresponding preset threshold to obtain a comparison result; if the comparison result is abnormal, generate a The prompt information of the comparison result.

In some embodiments, the device further includes a level setting module, configured to set different early warning levels for different target model indicators according to the pre-configured weight values of each target model indicator, and set the early warning levels of different target model indicators Write their corresponding comparison results.

In some embodiments, the device further includes a data output module, configured to receive a data output request; determine a target performance report and a target receiving address in the data output request; establish the target performance report and the target receiving address connecting interface between; transmitting the target performance report to the target receiving address.

In some embodiments, the device further includes a data viewing module, configured to receive a data viewing request; according to the data viewing request, a monitoring list page including multiple performance reports is provided, and the multiple performance reports belong to multiple performance reports of the same type. a predictive model.

During specific implementation, each of the above modules may be implemented as an independent entity, or may be combined arbitrarily to be implemented as the same or several entities.

It can be seen from the above that the model data monitoring device 30 provided in the embodiment of the present application receives the data acquisition request through the first receiving module 31; the data acquisition module 32 acquires the target predicted value and target actual value of the data to be predicted in the data acquisition request, The target prediction value indicates the prediction result of the data to be predicted through the target prediction model, and the target real value indicates the real result of the data to be predicted; the second receiving module 33 receives a data calculation request; the algorithm calling module 34 calls the The target algorithm corresponding to the target model index in the data calculation request; the data calculation module 35, based on the target algorithm, substitutes the target predicted value and the target real value for calculation to obtain the target result data; the report generation module 36 generates according to the target result data The performance report of the target prediction model; the monitoring and early warning module 37 performs monitoring and early warning operations on the performance of the target prediction model based on the performance report.

Please refer to FIG. 3. FIG. 3 is another structural schematic diagram of the model data monitoring device provided by the embodiment of the present application. The model data monitoring device 30 includes a memory 120, one or more processors 180, and one or more application programs, wherein The one or more application programs are stored in the memory 120 and are configured to be executed by the processor 180; the processor 180 may include a first receiving module 31, a data acquisition module 32, a second receiving module 33, and an algorithm tuning module. fetching module 34, data calculation module 35, report generating module 36, and monitoring and early warning module 37. For example, the structure and connection relationship of the above components can be as follows:

The memory 120 can be used to store applications and data. The application programs stored in the memory 120 include executable codes. Applications can be composed of various functional modules. The processor 180 executes various functional applications and data processing by executing the application programs stored in the memory 120 . In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 120 may further include a memory controller to provide the processor 180 with access to the memory 120 .

The processor 180 is the control center of the device. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing the application program stored in the memory 120 and calling the data stored in the memory 120, various functions of the device are executed. function and process data for overall monitoring of the installation. Optionally, the processor 180 may include one or more processing cores; preferably, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces, and application programs, etc. .

Specifically, in this embodiment, the processor 180 loads the executable code corresponding to the process of one or more application programs into the memory 120 according to the following instructions, and the processor 180 runs the executable code stored in the memory 120. applications to achieve various functions:

The first receiving module 31 is configured to receive a data acquisition request;

The data acquisition module 32 is configured to acquire the target predicted value and the target real value of the data to be predicted in the data acquisition request, the target predicted value indicates the forecast result obtained by the target forecast model for the data to be predicted, and the target real value indicates The real result of the data to be predicted;

The second receiving module 33 is configured to receive a data calculation request;

An algorithm calling module 34, configured to call the target algorithm corresponding to the target model index in the data calculation request;

The data calculation module 35 is used to calculate the target result data by substituting the target predicted value and the target actual value based on the target algorithm;

A report generating module 36, configured to generate a performance report of the target prediction model according to the target result data;

The monitoring and early warning module 37 is configured to perform monitoring and early warning operations on the performance of the target prediction model based on the performance report.

In some embodiments, the data acquisition module 32 is configured to generate a target SQL statement based on the preset identification of the data to be predicted and the pre-generated SQL statement template; The target predicted value and target actual value corresponding to the data to be predicted.

In some embodiments, the device further includes a data remark module, configured to receive a data remark request; obtain remark information in the data remark request; insert the remark information into the target model indicator, so that the target The model index is displayed in a preset form in the performance report and explains the meaning of the target model index.

In some embodiments, the monitoring and early warning module 37 is configured to compare each target result data in the performance report with its corresponding preset threshold to obtain a comparison result; if the comparison result is abnormal, generate a The prompt information of the comparison result.

In some embodiments, the device further includes a level setting module, configured to set different early warning levels for different target model indicators according to the pre-configured weight values of each target model indicator, and set the early warning levels of different target model indicators Write their corresponding comparison results.

In some embodiments, the device further includes a data output module, configured to receive a data output request; determine a target performance report and a target receiving address in the data output request; establish the target performance report and the target receiving address connecting interface between; transmitting the target performance report to the target receiving address.

In some embodiments, the device further includes a data viewing module, configured to receive a data viewing request; according to the data viewing request, a monitoring list page including multiple performance reports is provided, and the multiple performance reports belong to multiple performance reports of the same type. a predictive model.

The embodiment of the present application also provides a terminal device. The terminal device may be a server, a smart phone, a computer, a tablet computer or the like.

Please refer to FIG. 4 . FIG. 4 shows a schematic structural diagram of a terminal device provided by an embodiment of the present application, and the terminal device may be used to implement the model data monitoring method provided in the foregoing embodiments. The terminal device 1200 may be a TV set, a smart phone, or a tablet computer.

As shown in FIG. 4, the terminal device 1200 may include an RF (Radio Frequency, radio frequency) circuit 110, a memory 120 including one or more (only one shown in the figure) computer-readable storage media, an input unit 130, and a display unit. 140, a sensor 150, an audio circuit 160, a transmission module 170, a processor 180 including one or more (only one is shown in the figure) processing cores, a power supply 190 and other components. Those skilled in the art can understand that the structure of the terminal device 1200 shown in FIG. 4 does not constitute a limitation on the terminal device 1200, and may include more or less components than those shown in the figure, or combine certain components, or different components. layout. in:

The RF circuit 110 is used to receive and send electromagnetic waves, realize mutual conversion between electromagnetic waves and electrical signals, and communicate with communication networks or other devices. The RF circuit 110 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption/decryption chips, Subscriber Identity Module (SIM) cards, memory, and the like. The RF circuit 110 can communicate with various networks such as the Internet, intranet, wireless network, or communicate with other devices through a wireless network.

The memory 120 can be used to store software programs and modules, such as the program instructions/modules corresponding to the model data monitoring method in the above-mentioned embodiments, and the processor 180 executes various functional applications and data by running the software programs and modules stored in the memory 120 Processing, the vibration reminder mode can be automatically selected according to the current scene of the terminal device to monitor the model data, which can not only ensure that scenes such as meetings are not disturbed, but also ensure that users can perceive incoming calls, which improves the intelligence of the terminal device. The memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 120 may further include a memory that is remotely located relative to the processor 180, and these remote memories may be connected to the terminal device 1200 through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input unit 130 can be used to receive input numbers or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 130 may include a touch-sensitive surface 131 and other input devices 132 . The touch-sensitive surface 131, also referred to as a touch screen or a touchpad, can collect user touch operations on or near it (for example, the user uses any suitable object or accessory such as a finger or a stylus on the touch-sensitive surface 131 or near the touch-sensitive surface 131), and drive the corresponding connection device according to the preset program. Optionally, the touch-sensitive surface 131 may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, and sends it It is converted into contact coordinates and then sent to the processor 180, and can receive and execute commands sent by the processor 180. In addition, the touch-sensitive surface 131 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 131 , the input unit 130 may also include other input devices 132 . Specifically, other input devices 132 may include, but are not limited to, one or more of physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, joysticks, and the like.

The display unit 140 can be used to display information input by or provided to the user and various graphical user interfaces of the terminal device 1200. These graphical user interfaces can be composed of graphics, text, icons, videos and any combination thereof. The display unit 140 may include a display panel 141. Optionally, the display panel 141 may be configured in the form of LCD (Liquid Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, organic light-emitting diode), and the like. Further, the touch-sensitive surface 131 may cover the display panel 141, and when the touch-sensitive surface 131 detects a touch operation on or near it, the touch operation is sent to the processor 180 to determine the type of the touch event, and then the processor 180 determines the type of the touch event according to the touch operation. According to the type of control event, a corresponding visual output is provided on the display panel 141. Although in FIG. 4 , the touch-sensitive surface 131 and the display panel 141 are used as two independent components to realize input and output functions, in some embodiments, the touch-sensitive surface 131 and the display panel 141 can be integrated to realize input. and output functions.

The terminal device 1200 may further include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 141 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 141 when the terminal device 1200 moves to the ear. and/or backlighting. As a kind of motion sensor, the gravitational acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when it is stationary, and can be used for applications that recognize the attitude of mobile phones (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tap), etc.; as for other sensors such as gyroscope, barometer, hygrometer, thermometer, and infrared sensor that can also be configured on terminal device 1200, here No longer.

The audio circuit 160 , the speaker 161 and the microphone 162 can provide an audio interface between the user and the terminal device 1200 . The audio circuit 160 can transmit the electrical signal converted from the received audio data to the loudspeaker 161, and the loudspeaker 161 converts it into an audio signal output; After being received, it is converted into audio data, and then the audio data is processed by the output processor 180, and then sent to another terminal through the RF circuit 110, or the audio data is output to the memory 120 for further processing. The audio circuit 160 may also include an earplug jack to provide communication between an external earphone and the terminal device 1200 .

The terminal device 1200 can help the user to send and receive e-mails, browse the web, access streaming media, etc. through the transmission module 170 (such as a Wi-Fi module), and it provides users with wireless broadband Internet access. Although FIG. 4 shows the transmission module 170, it can be understood that it is not an essential component of the terminal device 1200, and can be completely omitted as required without changing the essence of the invention.

The processor 180 is the control center of the terminal device 1200, and uses various interfaces and lines to connect various parts of the entire mobile phone, by running or executing software programs and/or modules stored in the memory 120, and calling data stored in the memory 120 , execute various functions and process data of the terminal device 1200, so as to monitor the mobile phone as a whole. Optionally, the processor 180 may include one or more processing cores; in some embodiments, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and applications, etc., and the modem processor mainly handles wireless communications. It can be understood that the foregoing modem processor may not be integrated into the processor 180 .

The terminal device 1200 also includes a power supply 190 for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 180 through the power management system, so as to implement functions such as discharge management and power consumption management through the power management system. The power supply 190 may also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

Although not shown, the terminal device 1200 may also include a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be repeated here. Specifically in this embodiment, the display unit 140 of the terminal device 1200 is a touch screen display, and the terminal device 1200 also includes a memory 120 and one or more programs, wherein one or more programs are stored in the memory 120, and One or more programs configured to be executed by one or more processors 180 include instructions for:

a first receiving instruction, configured to receive a data acquisition request;

The data acquisition instruction is used to acquire the target predicted value and the target real value of the data to be predicted in the data acquisition request, the target predicted value indicates the forecast result obtained by the target forecast model for the data to be predicted, and the target real value indicates the target real value to be predicted Predict the true outcome of the data;

The second receiving instruction is used to receive a data calculation request;

Algorithm call instruction, used to call the target algorithm corresponding to the target model index in the data calculation request;

A data calculation instruction, used to calculate the target result data by substituting the target predicted value and the target actual value based on the target algorithm;

report generating instructions, configured to generate a performance report of the target prediction model based on the target result data;

The monitoring and early warning instruction is used to perform monitoring and early warning operations on the performance of the target prediction model based on the performance report.

In some embodiments, the data acquisition instruction is used to generate a target SQL statement based on the preset identifier of the data to be predicted and the pre-generated SQL statement template; The target predicted value and target actual value corresponding to the forecast data.

In some embodiments, the program further includes a data remark instruction for receiving a data remark request; obtaining remark information in the data remark request; inserting the remark information into the target model index, so that the target The model index is displayed in a preset form in the performance report and explains the meaning of the target model index.

In some embodiments, the monitoring and early warning instruction is used to compare each target result data in the performance report with its corresponding preset threshold to obtain the comparison result; if the comparison result is abnormal, generate an A prompt message describing the comparison result.

In some embodiments, the program also includes a level setting instruction, which is used to set different warning levels for different target model indicators according to the pre-configured weight values of each target model indicator, and set the warning levels of different target model indicators Write the respective comparison results.

In some embodiments, the program further includes a data output instruction for receiving a data output request; determining a target performance report and a target receiving address in the data output request; establishing the target performance report and the target receiving address connecting interface between; transmitting the target performance report to the target receiving address.

In some embodiments, the program further includes a data viewing instruction, configured to receive a data viewing request; according to the data viewing request, a monitoring list page including multiple performance reports is provided, and the multiple performance reports belong to the same type of multiple a predictive model.

The embodiment of the present application also provides a terminal device. The terminal device may be a smart phone, a computer and other devices.

It can be seen from the above that the embodiment of the present application provides a terminal device 1200, and the terminal device 1200 performs the following steps:

Receive data acquisition request;

Acquiring a target predicted value and a target real value of the data to be predicted in the data acquisition request, the target predicted value indicates the predicted result of the data to be predicted through a target prediction model, and the target real value indicates the real result of the data to be predicted;

Receive data calculation request;

calling the target algorithm corresponding to the target model index in the data calculation request;

Based on the target algorithm, substituting the target predicted value and target real value for calculation to obtain target result data;

generating a performance report of the target predictive model based on the target outcome data;

The performance of the target prediction model is monitored and pre-warned based on the performance report.

An embodiment of the present application further provides a storage medium, in which a computer program is stored, and when the computer program is run on a computer, the computer executes the model data monitoring method described in any one of the above embodiments.

It should be noted that, for the model data monitoring method described in this application, ordinary testers in the field can understand that all or part of the process of implementing the model data monitoring method described in the embodiment of this application can be controlled by computer programs. To complete, the computer program can be stored in a computer-readable storage medium, such as stored in the memory of the terminal device, and executed by at least one processor in the terminal device, and the execution process can include such as the model Flow of an embodiment of a data monitoring method. Wherein, the storage medium may be a magnetic disk, an optical disk, a read only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory) and the like.

For the model data monitoring device in the embodiment of the present application, its functional modules may be integrated into one processing chip, or each module may exist separately physically, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium, such as read-only memory, magnetic disk or optical disk.

The model data monitoring method, device, medium and equipment provided in the embodiments of the present application have been introduced in detail above. In this paper, specific examples are used to illustrate the principle and implementation of the application. The description of the above embodiments is only used to help understand the method and core idea of the application; meanwhile, for those skilled in the art, according to the application Thoughts, specific implementation methods and application ranges all have changes. In summary, the content of this specification should not be construed as limiting the application.

Claims (10)

1. A model data monitoring method, characterized in that, comprising: Receive data acquisition request; Acquiring a target predicted value and a target real value of the data to be predicted in the data acquisition request, the target predicted value indicates the predicted result of the data to be predicted through a target prediction model, and the target real value indicates the real result of the data to be predicted; Receive data calculation request; calling the target algorithm corresponding to the target model index in the data calculation request; Based on the target algorithm, substituting the target predicted value and target real value for calculation to obtain target result data; generating a performance report of the target predictive model based on the target outcome data; The performance of the target prediction model is monitored and pre-warned based on the performance report. 2. The model data monitoring method according to claim 1, wherein said obtaining the target predicted value and target actual value of the data to be predicted in the data acquisition request comprises: Generate a target SQL statement based on the preset identification of the data to be predicted and the pre-generated SQL statement template; The target predicted value and target real value corresponding to the data to be predicted are obtained from the database according to the target SQL statement. 3. The model data monitoring method according to claim 1, wherein said method further comprises: Receive data comment requests; Obtain the remark information in the data remark request; The remark information is inserted into the target model indicator, so that the target model indicator is displayed in a preset form in the performance report and the meaning of the target model indicator is explained. 4. The model data monitoring method according to claim 1, wherein the monitoring and early warning operation of the performance of the target prediction model based on the performance report comprises: Comparing each target result data in the performance report with its corresponding preset threshold to obtain a comparison result; If the comparison result is abnormal, generating prompt information including the comparison result. 5. the model data monitoring method as claimed in claim 4, is characterized in that, described method also comprises: According to the pre-configured weight value of each target model index, different early warning levels are set for different target model indicators, and the early warning levels of different target model indicators are written into the corresponding comparison results. 6. The model data monitoring method according to claim 1, wherein said method further comprises: Receive data output request; determining the target performance report and target receiving address in the data output request; establishing a connection interface between the target performance report and the target receiving address; Transmitting the target performance report to the target receiving address. 7. The model data monitoring method according to claim 1, wherein said method further comprises: Receive data view request; A monitoring list page including multiple performance reports is provided according to the data viewing request, and the multiple performance reports belong to multiple prediction models of the same type. 8. A model data monitoring device, characterized in that the model data monitoring device comprises: A first receiving module, configured to receive a data acquisition request; A data acquisition module, configured to acquire a target predicted value and a target real value of the data to be predicted in the data acquisition request, the target predicted value indicates the forecast result obtained by the target forecast model for the data to be predicted, and the target real value indicates the target real value to be predicted Predict the true outcome of the data; A second receiving module, configured to receive a data calculation request; An algorithm calling module, configured to call the target algorithm corresponding to the target model index in the data calculation request; The data calculation module is used to calculate the target result data by substituting the target predicted value and the target actual value based on the target algorithm; A report generating module, configured to generate a performance report of the target prediction model according to the target result data; A monitoring and early warning module, configured to perform monitoring and early warning operations on the performance of the target prediction model based on the performance report. 9. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the method described in any one of claims 1 to 7. Model data monitoring methods. 10. A terminal device, characterized in that it includes a processor and a memory, the memory stores a plurality of instructions, and the processor loads the instructions to perform the model data monitoring described in any one of claims 1 to 7 method.

Images