CN115658419A

CN115658419A - Model data monitoring method, device, medium and equipment

Info

Publication number: CN115658419A
Application number: CN202211117959.4A
Authority: CN
Inventors: 李芳�
Original assignee: Ping An Technology Shenzhen Co Ltd
Current assignee: Ping An Technology Shenzhen Co Ltd
Priority date: 2022-09-14
Filing date: 2022-09-14
Publication date: 2023-01-31

Abstract

The embodiment of the application provides a method, a device, a medium and equipment for monitoring model data, wherein the method comprises the following steps: receiving a data acquisition request; acquiring a target predicted value and a target real value of data to be predicted in the data acquisition request; receiving a data calculation request; calling a target algorithm corresponding to a target model index in the data calculation request; substituting the target predicted value and the target true value into a target algorithm to calculate to obtain target result data; generating a performance report of the target prediction model according to target result data; and monitoring and early warning operation is carried out on the performance of the target prediction model based on the performance report. By using the model data monitoring method provided by the embodiment of the application, a user can automatically obtain a performance report of a target prediction model for predicting data to be predicted by selecting the specified data to be predicted and the specified model index on a platform, and automatically monitor and early warn the performance of the target prediction model according to the performance report.

Description

Model data monitoring method, device, medium and equipment

Technical Field

The present application relates to the field of electronic communications technologies, and in particular, to a method, an apparatus, a medium, and a device for monitoring model data.

Background

Model monitoring is one of AI leading edge research fields in recent years, and as the application scene of the model in actual production of an enterprise increases exponentially, the effect and stability of the model also directly influence the production benefit of enterprise products. For example, a bank wind control model directly affects core service scenes such as loan and account opening of a user, if the model is degraded in effect, a user risk level prediction result cannot be accurately given, and therefore the use process of the user is directly affected, and income loss is caused.

Therefore, the algorithm engineer needs to update and iterate or offline the model with reduced effect in time, so as to avoid production accidents caused by model problems, which brings forward the demand of model monitoring.

Before that, the algorithm engineer mainly performs offline index query by writing codes by himself, which requires a considerable amount of time to write codes, for example, 100 models are available, each model consumes 5 days to write monitoring codes, 500 days to monitor the models are consumed, and as the models grow, the labor cost is increased.

Disclosure of Invention

By using the model data monitoring method provided by the embodiment of the application, a user can select the appointed data to be predicted and appointed model indexes on a platform, so that a performance report of a target prediction model for predicting the data to be predicted can be automatically obtained, and the performance of the target prediction model is automatically monitored and early warned according to the performance report.

An embodiment of the present application provides a model data monitoring method, which includes:

receiving a data acquisition request;

acquiring a target predicted value and a target real value of the data to be predicted in the data acquisition request, wherein the target predicted value indicates a predicted result of the data to be predicted obtained through a target prediction model, and the target real value indicates a real result of the data to be predicted;

receiving a data calculation request;

calling a target algorithm corresponding to a target model index in the data calculation request;

substituting the target predicted value and the target real value into the target algorithm to calculate to obtain target result data;

generating a performance report of the target prediction model according to the target result data;

and performing monitoring and early warning operation on the performance of the target prediction model based on the performance report.

In the method for monitoring model data according to the embodiment of the present application, the obtaining a target predicted value and a target true value of data to be predicted in the data obtaining request includes:

generating a target SQL statement based on the preset identification of the data to be predicted and a pre-generated SQL statement template;

and acquiring a target predicted value and a target real value corresponding to the data to be predicted from a database according to the target SQL statement.

In the model data monitoring method according to the embodiment of the present application, the method further includes:

receiving a data remark request;

acquiring remark information in the data remark request;

and inserting the remark information into the target model index so that the target model index is displayed in a preset form in the performance report and the meaning of the target model index is explained.

In the method for monitoring model data according to the embodiment of the present application, the monitoring and early warning operation of the performance of the target prediction model based on the performance report includes:

comparing each target result data in the performance report with a corresponding preset threshold value to obtain a comparison result;

and if the comparison result is abnormal, generating prompt information containing the comparison result.

and setting different early warning levels for different target model indexes according to the weight values pre-configured for the target model indexes, and writing the early warning levels of the different target model indexes into respective corresponding comparison results.

receiving a data output request;

determining a target performance report and a target receiving address in the data output request;

establishing a connection interface between the target performance report and the target receiving address;

transmitting the target performance report to the target receiving address.

receiving a data viewing request;

and providing a monitoring list page containing a plurality of performance reports according to the data viewing request, wherein the performance reports belong to a plurality of prediction models of the same type.

Correspondingly, another aspect of the embodiments of the present application further provides a model data monitoring apparatus, where the model data monitoring apparatus includes:

the first receiving module is used for receiving a data acquisition request;

the data acquisition module is used for acquiring a target predicted value and a target real value of the data to be predicted in the data acquisition request, wherein the target predicted value indicates a predicted result of the data to be predicted obtained through a target prediction model, and the target real value indicates a real result of the data to be predicted;

the second receiving module is used for receiving a data calculation request;

the algorithm calling module is used for calling a target algorithm corresponding to a target model index in the data calculation request;

the data calculation module is used for substituting the target predicted value and the target true value to calculate to obtain target result data based on the target algorithm;

a report generation module for generating a performance report of the target prediction model according to the target result data;

and the monitoring and early warning module is used for carrying out monitoring and early warning operation on the performance of the target prediction model based on the performance report.

Accordingly, another aspect of the embodiments of the present application further provides a storage medium, where the storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to perform the model data monitoring method as described above.

Correspondingly, another aspect of the embodiments of the present application further provides a terminal device, which includes a processor and a memory, where the memory stores multiple instructions, and the processor loads the instructions to perform the model data monitoring method as described above.

The embodiment of the application provides a method, a device, a medium and equipment for monitoring model data, wherein the method comprises the steps of receiving a data acquisition request; acquiring a target predicted value and a target real value of the data to be predicted in the data acquisition request, wherein the target predicted value indicates a predicted result of the data to be predicted obtained through a target prediction model, and the target real value indicates a real result of the data to be predicted; receiving a data calculation request; calling a target algorithm corresponding to a target model index in the data calculation request; substituting the target predicted value and the target real value into the target algorithm to calculate to obtain target result data; generating a performance report of the target prediction model according to the target result data; and monitoring and early warning operation is carried out on the performance of the target prediction model based on the performance report. By using the model data monitoring method provided by the embodiment of the application, a user can automatically obtain a performance report of a target prediction model for predicting data to be predicted by selecting the specified data to be predicted and the specified model index on a platform, and automatically monitor and early warn the performance of the target prediction model according to the performance report.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the description of the embodiments will be briefly introduced below. It is obvious that the drawings in the following description are only some embodiments of the application, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.

Fig. 1 is a schematic flowchart of a model data monitoring method according to an embodiment of the present disclosure.

Fig. 2 is a schematic structural diagram of a model data monitoring apparatus according to an embodiment of the present application.

Fig. 3 is another schematic structural diagram of the model data monitoring apparatus according to the embodiment of the present application.

Fig. 4 is a schematic structural diagram of a terminal device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without inventive step, are within the scope of the present application.

It should be noted that the following is a brief introduction to the background of the present solution:

the scheme is mainly developed around the technical problem that the existing model performance monitoring mode is high in dependence on manpower and long in time consumption, and manpower cost and time cost are increased. It can be understood that, in order to update, iterate or offline a model with performance degradation in time and avoid a production accident caused by a model problem, an algorithm engineer mainly performs offline index query by writing a code by himself at present, which requires considerable time to write the code, for example, 100 models are provided, each model consumes 5 days to write a monitoring code and 500 days to monitor the model, and as the model increases, the more the manpower cost and the time cost are required.

In order to solve the foregoing technical problems, an embodiment of the present application provides a model data monitoring method. By using the model data monitoring method provided by the embodiment of the application, a user can select the appointed data to be predicted and the appointed model index on the platform, so that a performance report of a target prediction model for predicting the data to be predicted can be automatically obtained, and the performance of the target prediction model is automatically monitored and early warned according to the performance report.

Referring to fig. 1, fig. 1 is a schematic flow chart of a model data monitoring method according to an embodiment of the present disclosure. The model data monitoring method is applied to terminal equipment. Optionally, the terminal device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, web service, cloud communication, middleware service, domain name service, security service, CDN (Content Delivery Network), big data and artificial intelligence platform. Optionally, the terminal is a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, and the like, but is not limited thereto.

In an embodiment, the method may comprise the steps of:

step 101, a data acquisition request is received.

The data acquisition request refers to a request sent to the terminal device when a user, such as an algorithm engineer, needs to view the performance exhibited by a specified prediction model when predicting specified result data. For example, when an algorithm engineer wants to know the excellent performance of the bank wind control model in the process of predicting the risk level of the user, a data acquisition request can be initiated to the terminal device based on the bank wind control model and the user data. In particular, the data acquisition request may be initiated by a user, such as an algorithm engineer, to the terminal device through an instruction input device, such as a keyboard, mouse, or the like.

And 102, acquiring a target predicted value and a target real value of the data to be predicted in the data acquisition request, wherein the target predicted value indicates a predicted result of the data to be predicted obtained through a target prediction model, and the target real value indicates a real result of the data to be predicted.

The target predicted value indicates a predicted result of the data to be predicted, which is obtained through the target prediction model, and the target real value indicates a real result of the data to be predicted. In short, the target prediction value is a prediction result obtained by data processing of the target prediction model according to user data input by a user, and the accuracy of the target prediction value depends on the performance of the prediction model. The target true value is a true result which is obtained through manual calculation and is relatively objective, and the default is absolute true.

In some embodiments, the target predicted value and the target true value may be obtained by:

and generating a target SQL statement based on a preset identifier (such as a character string 'xxx' representing the identity of the user) of the data to be predicted and a pre-generated SQL statement template, and acquiring a target predicted value and a target real value corresponding to the data to be predicted from the database according to the target SQL statement.

It should be explained that the SQL statement template includes a plurality of variables, at least including a "preset identifier", "predicted value", and "true value", and the target SQL statement is generated by modifying the variables in the SQL statement template, so that the target predicted value and the target true value corresponding to the data to be predicted can be obtained from the database in a targeted manner. And the SQL sentence template is arranged, so that the user can conveniently and quickly construct the SQL sentence without independently compiling, the time cost is saved, and the efficiency is improved.

Step 103, receiving a data calculation request.

Since there are usually a plurality of indexes that can reflect the excellent performance of the prediction model, the user can select a designated model index according to the needs of the user, and initiate a data calculation request to the terminal device. For example, classification models generally need to look at the ROC, KS, GINI, confusion matrix, etc. of the model, and regression models are MSE, RMSE, MAE, R-Square, etc.

And 104, calling a target algorithm corresponding to the target model index in the data calculation request.

The target algorithm refers to an algorithm for calculating a target model index designated by a user, and the model indexes mentioned in the scheme can be calculated by adopting the existing mature formula algorithm, and meanwhile, the formula algorithm is not improved, so the formula algorithm is not listed and explained.

And 105, substituting the target predicted value and the target true value into the target algorithm to calculate to obtain target result data.

And substituting the obtained target predicted value and the target true value into a target algorithm to calculate based on a target algorithm corresponding to the user target model index to obtain target result data. Namely ROC, KS, GINI, confusion matrix and the like corresponding to the classification model, and MSE, RMSE, MAE, R-Square and the like corresponding to the regression model.

And 106, generating a performance report of the target prediction model according to the target result data.

The performance report specifically shows target result data corresponding to each target model index of the target prediction model, in some embodiments, the performance report provides an online report and an excel report, and the reports record the monitoring and operating conditions of each model; the index trend can be seen by clicking to look up the index trend, so that algorithm personnel can conveniently adjust the monitoring strategy in time; and clicking the log to see the execution process of each index operated each time, so that the error of index operation failure is conveniently positioned.

In some embodiments, the present solution provides a data remarking function in order for a person viewing the performance report to understand and understand the contents within the performance report. The algorithm engineer can initiate a data remark request containing remark information to the terminal device, and the terminal device inserts the remark information into the target model index after acquiring the remark information in the data remark request, so that the target model index is displayed in a preset form in the performance report and the meaning of the target model index is explained. Illustratively, the target model index can be shown in the performance report in the form of an embedded annotation, and when a user clicks the response target result parameter, the user can pop up remark information corresponding to the target result parameter.

And 107, monitoring and early warning the performance of the target prediction model based on the performance report.

And if the comparison result is abnormal, prompt information containing the comparison result is generated and informed to an algorithm engineer, so that the performance of the target prediction model is monitored and early warned.

In some embodiments, different early warning levels may be set for different target model indicators according to a weight value pre-configured for each target model indicator, and the early warning levels of the different target model indicators are written into respective corresponding comparison results. And different early warning grades can be triggered according to different weights, and an early warning mailbox or a short message can be automatically triggered according to the early warning grades.

In some embodiments, the user can actively send a data viewing request to the terminal device to directionally view the target performance report. After receiving the data viewing request, the terminal device provides a monitoring list page containing a plurality of performance reports according to the data viewing request, and each performance report can be viewed independently. It should be noted that multiple performance reports belong to multiple prediction models of the same type in the same monitoring list page, which is convenient for centralized management.

In some embodiments, a performance report output interface is also provided, which can interface with other systems to provide result data corresponding to each target model index, so as to facilitate other operations, such as further analysis of the performance report. Specifically, the terminal device receives a data output request sent by a user, determines a target performance report and a target receiving address in the data output request, establishes a connection interface between the target performance report and the target receiving address, and transmits the target performance report to the target receiving address.

All the above optional technical solutions may be combined arbitrarily to form optional embodiments of the present application, and are not described herein again.

In particular implementation, the present application is not limited by the execution sequence of the described steps, and some steps may be performed in other sequences or simultaneously without conflict.

As can be seen from the above, the model data monitoring method provided in the embodiment of the present application obtains a request by receiving data; acquiring a target predicted value and a target real value of the data to be predicted in the data acquisition request, wherein the target predicted value indicates a predicted result of the data to be predicted obtained through a target prediction model, and the target real value indicates a real result of the data to be predicted; receiving a data calculation request; calling a target algorithm corresponding to a target model index in the data calculation request; substituting the target predicted value and the target real value into the target algorithm to calculate to obtain target result data; generating a performance report of the target prediction model according to the target result data; and monitoring and early warning operation is carried out on the performance of the target prediction model based on the performance report. By using the model data monitoring method provided by the embodiment of the application, a user can automatically obtain a performance report of a target prediction model for predicting data to be predicted by selecting the specified data to be predicted and the specified model index on a platform, and automatically monitor and early warn the performance of the target prediction model according to the performance report.

The embodiment of the application also provides a model data monitoring device, and the model data monitoring device can be integrated in the terminal equipment.

Referring to fig. 2, fig. 2 is a schematic structural diagram of a model data monitoring apparatus according to an embodiment of the present disclosure. The model data monitoring apparatus 30 may include:

a first receiving module 31, configured to receive a data obtaining request;

a data obtaining module 32, configured to obtain a target predicted value and a target true value of the data to be predicted in the data obtaining request, where the target predicted value indicates a predicted result of the data to be predicted obtained through a target prediction model, and the target true value indicates a true result of the data to be predicted;

a second receiving module 33, configured to receive a data calculation request;

the algorithm calling module 34 is configured to call a target algorithm corresponding to a target model index in the data calculation request;

the data calculation module 35 is configured to substitute the target predicted value and the target true value to perform calculation based on the target algorithm to obtain target result data;

a report generation module 36, configured to generate a performance report of the target prediction model according to the target result data;

and the monitoring and early warning module 37 is configured to perform monitoring and early warning operation on the performance of the target prediction model based on the performance report.

In some embodiments, the data obtaining module 32 is configured to generate a target SQL statement based on a preset identifier of the data to be predicted and a pre-generated SQL statement template; and acquiring a target predicted value and a target real value corresponding to the data to be predicted from a database according to the target SQL statement.

In some embodiments, the apparatus further comprises a data remark module to receive a data remark request; acquiring remark information in the data remark request; and inserting the remark information into the target model index so that the target model index is displayed in a preset form in the performance report and the meaning of the target model index is explained.

In some embodiments, the monitoring and early warning module 37 is configured to compare each target result data in the performance report with a corresponding preset threshold to obtain a comparison result; and if the comparison result is abnormal, generating prompt information containing the comparison result.

In some embodiments, the apparatus further includes a level setting module, configured to set different early warning levels for different target model indicators according to a weight value pre-configured for each target model indicator, and write the early warning levels of the different target model indicators into respective corresponding comparison results.

In some embodiments, the apparatus further comprises a data output module to receive a data output request; determining a target performance report and a target receiving address in the data output request; establishing a connection interface between the target performance report and the target receiving address; transmitting the target performance report to the target recipient address.

In some embodiments, the apparatus further comprises a data view module to receive a data view request; and providing a monitoring list page containing a plurality of performance reports according to the data viewing request, wherein the performance reports belong to a plurality of prediction models of the same type.

In specific implementation, the modules may be implemented as independent entities, or may be combined arbitrarily and implemented as one or several entities.

As can be seen from the above, the model data monitoring apparatus 30 provided in the embodiment of the present application receives the data obtaining request through the first receiving module 31; the data obtaining module 32 obtains a target predicted value and a target true value of the data to be predicted in the data obtaining request, wherein the target predicted value indicates a predicted result of the data to be predicted obtained through a target prediction model, and the target true value indicates a true result of the data to be predicted; the second receiving module 33 receives the data calculation request; the algorithm calling module 34 calls a target algorithm corresponding to the target model index in the data calculation request; the data calculation module 35 substitutes the target predicted value and the target actual value to calculate based on the target algorithm to obtain target result data; the report generation module 36 generates a performance report of the target prediction model according to the target result data; the monitoring and early-warning module 37 performs monitoring and early-warning operation on the performance of the target prediction model based on the performance report.

Referring to fig. 3, fig. 3 is another schematic structural diagram of a model data monitoring apparatus according to an embodiment of the present disclosure, in which the model data monitoring apparatus 30 includes a memory 120, one or more processors 180, and one or more applications, where the one or more applications are stored in the memory 120 and configured to be executed by the processor 180; the processor 180 may include a first receiving module 31, a data obtaining module 32, a second receiving module 33, an algorithm calling module 34, a data calculating module 35, a report generating module 36, and a monitoring and pre-warning module 37. For example, the structures and connection relationships of the above components may be as follows:

the memory 120 may be used to store applications and data. The memory 120 stores applications containing executable code. The application programs may constitute various functional modules. The processor 180 executes various functional applications and data processing by running the application programs stored in the memory 120. Further, the memory 120 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 120 may also include a memory controller to provide the processor 180 with access to the memory 120.

The processor 180 is a control center of the device, connects various parts of the entire terminal using various interfaces and lines, performs various functions of the device and processes data by running or executing an application program stored in the memory 120 and calling data stored in the memory 120, thereby monitoring the entire device. Optionally, processor 180 may include one or more processing cores; preferably, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, an application program, and the like.

Specifically, in this embodiment, the processor 180 loads the executable code corresponding to the processes of one or more application programs into the memory 120 according to the following instructions, and the processor 180 runs the application programs stored in the memory 120, thereby implementing various functions:

a first receiving module 31, configured to receive a data obtaining request;

a second receiving module 33, configured to receive a data calculation request;

In some embodiments, the apparatus further includes a level setting module, configured to set different early warning levels for different target model indexes according to a weight value pre-configured for each target model index, and write the early warning levels of the different target model indexes into respective corresponding comparison results.

In some embodiments, the apparatus further comprises a data output module to receive a data output request; determining a target performance report and a target receiving address in the data output request; establishing a connection interface between the target performance report and the target receiving address; transmitting the target performance report to the target receiving address.

In some embodiments, the apparatus further comprises a data viewing module to receive a data viewing request; providing a monitoring list page containing a plurality of performance reports according to the data viewing request, wherein the performance reports belong to a plurality of prediction models of the same type.

The embodiment of the application further provides the terminal equipment. The terminal device can be a server, a smart phone, a computer, a tablet computer and the like.

Referring to fig. 4, fig. 4 is a schematic structural diagram of a terminal device provided in the embodiment of the present application, where the terminal device may be used to implement the model data monitoring method provided in the foregoing embodiment. The terminal device 1200 may be a television, a smart phone, or a tablet computer.

As shown in fig. 4, the terminal device 1200 may include components such as an RF (Radio Frequency) circuit 110, a memory 120 including one or more (only one shown) computer-readable storage media, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a transmission module 170, a processor 180 including one or more (only one shown) processing cores, and a power supply 190. Those skilled in the art will appreciate that the terminal device 1200 configuration shown in fig. 4 does not constitute a limitation of terminal device 1200, and may include more or fewer components than those shown, or some components in combination, or a different arrangement of components. Wherein:

the RF circuit 110 is used for receiving and transmitting electromagnetic waves, and performs interconversion between the electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices. The RF circuitry 110 may include various existing circuit elements for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption/decryption chip, a Subscriber Identity Module (SIM) card, memory, and so forth. The RF circuitry 110 may communicate with various networks such as the internet, an intranet, a wireless network, or with other devices over a wireless network.

The memory 120 may be configured to store a software program and a module, such as a program instruction/module corresponding to the model data monitoring method in the foregoing embodiment, and the processor 180 executes various functional applications and data processing by operating the software program and the module stored in the memory 120, and may automatically select a vibration alert mode according to a current scene where the terminal device is located to perform model data monitoring, so as to ensure that scenes such as a conference are not disturbed, ensure that a user can sense an incoming call, and improve intelligence of the terminal device. Memory 120 may include high speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 120 may further include memory located remotely from the processor 180, which may be connected to the terminal device 1200 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input unit 130 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. In particular, the input unit 130 may include a touch-sensitive surface 131 as well as other input devices 132. The touch-sensitive surface 131, also referred to as a touch-sensitive display screen or a touch pad, may collect touch operations by a user on or near the touch-sensitive surface 131 (e.g., operations by a user on or near the touch-sensitive surface 131 using any suitable object or attachment such as a finger, a stylus, etc.) and drive the corresponding connection device according to a predetermined program. Alternatively, the touch-sensitive surface 131 may comprise two parts, a touch detection device and a touch controller. The touch detection device detects a touch direction of a user, detects a signal brought by touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 180, and can receive and execute commands sent by the processor 180. Additionally, the touch sensitive surface 131 may be implemented using various types of resistive, capacitive, infrared, and surface acoustic waves. In addition to touch-sensitive surface 131, input unit 130 may include other input devices 132. In particular, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

The display unit 140 may be used to display information input by or provided to a user and various graphic user interfaces of the terminal apparatus 1200, which may be configured by graphics, text, icons, video, and any combination thereof. The Display unit 140 may include a Display panel 141, and optionally, the Display panel 141 may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like. Further, touch sensitive surface 131 may overlay display panel 141, and when touch sensitive surface 131 detects a touch operation on or near touch sensitive surface 131, it may be transmitted to processor 180 to determine the type of touch event, and then processor 180 may provide a corresponding visual output on display panel 141 according to the type of touch event. Although in FIG. 4, touch-sensitive surface 131 and display panel 141 are shown as two separate components to implement input and output functions, in some embodiments, touch-sensitive surface 131 may be integrated with display panel 141 to implement input and output functions.

The terminal device 1200 may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor that may adjust the brightness of the display panel 141 according to the brightness of ambient light, and a proximity sensor that may turn off the display panel 141 and/or the backlight when the terminal device 1200 is moved to the ear. As one of the motion sensors, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), can detect the magnitude and direction of gravity when stationary, and can be used for applications of identifying the gesture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer gesture calibration), vibration identification related functions (such as pedometer and tapping), and the like; as for other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which may be further configured in the terminal device 1200, detailed descriptions thereof are omitted.

The audio circuitry 160, speaker 161, microphone 162 may provide an audio interface between the user and the terminal device 1200. The audio circuit 160 may transmit the electrical signal converted from the received audio data to the speaker 161, and convert the electrical signal into a sound signal for output by the speaker 161; on the other hand, the microphone 162 converts the collected sound signal into an electric signal, converts the electric signal into audio data after being received by the audio circuit 160, and then outputs the audio data to the processor 180 for processing, and then to the RF circuit 110 to be transmitted to, for example, another terminal, or outputs the audio data to the memory 120 for further processing. The audio circuitry 160 may also include an earbud jack to provide communication of peripheral headphones with the terminal device 1200.

The terminal device 1200, through the transmission module 170 (e.g., wi-Fi module), may assist the user in e-mail, web browsing, streaming media access, etc., which provides the user with wireless broadband internet access. Although fig. 4 shows the transmission module 170, it is understood that it does not belong to the essential constitution of the terminal device 1200, and may be omitted entirely as needed within the scope not changing the essence of the invention.

The processor 180 is a control center of the terminal device 1200, connects various parts of the entire mobile phone by using various interfaces and lines, and performs various functions of the terminal device 1200 and processes data by running or executing software programs and/or modules stored in the memory 120 and calling data stored in the memory 120, thereby performing overall monitoring of the mobile phone. Optionally, processor 180 may include one or more processing cores; in some embodiments, the processor 180 may integrate an application processor, which primarily handles operating systems, user interfaces, applications, etc., and a modem processor, which primarily handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 180.

Terminal device 1200 also includes a power supply 190 for powering the various components, which in some embodiments may be logically coupled to processor 180 via a power management system to manage power discharge and power consumption via the power management system. The power supply 190 may also include any component including one or more of a dc or ac power source, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.

Although not shown, the terminal device 1200 may further include a camera (e.g., a front camera, a rear camera), a bluetooth module, and the like, which are not described in detail herein. Specifically, in this embodiment, the display unit 140 of the terminal device 1200 is a touch screen display, and the terminal device 1200 further includes a memory 120 and one or more programs, wherein the one or more programs are stored in the memory 120 and configured to be executed by the one or more processors 180, and the one or more programs include instructions for:

a first receiving instruction for receiving a data acquisition request;

a data obtaining instruction, configured to obtain a target predicted value and a target true value of the data to be predicted in the data obtaining request, where the target predicted value indicates a predicted result obtained by the data to be predicted through a target prediction model, and the target true value indicates a true result of the data to be predicted;

a second receiving instruction for receiving a data calculation request;

the algorithm calling instruction is used for calling a target algorithm corresponding to a target model index in the data calculation request;

the data calculation instruction is used for substituting the target predicted value and the target real value to calculate to obtain target result data based on the target algorithm;

report generating instructions for generating a performance report of the target prediction model from the target result data;

and the monitoring and early warning instruction is used for carrying out monitoring and early warning operation on the performance of the target prediction model based on the performance report.

In some embodiments, the data obtaining instruction is configured to generate a target SQL statement based on a preset identifier of the data to be predicted and a pre-generated SQL statement template; and acquiring a target predicted value and a target real value corresponding to the data to be predicted from a database according to the target SQL statement.

In some embodiments, the program further comprises data remark instructions for receiving a data remark request; obtaining remark information in the data remark request; and inserting the remark information into the target model index so that the target model index is displayed in a preset form in the performance report and the meaning of the target model index is explained.

In some embodiments, the monitoring and early warning instruction is configured to compare each target result data in the performance report with a corresponding preset threshold to obtain a comparison result; and if the comparison result is abnormal, generating prompt information containing the comparison result.

In some embodiments, the program further includes a level setting instruction, configured to set different early warning levels for different target model indicators according to a weight value pre-configured for each target model indicator, and write the early warning levels of the different target model indicators into respective corresponding comparison results.

In some embodiments, the program further comprises data output instructions for receiving a data output request; determining a target performance report and a target receiving address in the data output request; establishing a connection interface between the target performance report and the target receiving address; transmitting the target performance report to the target recipient address.

In some embodiments, the program further comprises data viewing instructions for receiving a data viewing request; and providing a monitoring list page containing a plurality of performance reports according to the data viewing request, wherein the performance reports belong to a plurality of prediction models of the same type.

The embodiment of the application further provides the terminal equipment. The terminal equipment can be equipment such as a smart phone and a computer.

As can be seen from the above, an embodiment of the present application provides a terminal device 1200, where the terminal device 1200 executes the following steps:

receiving a data acquisition request;

receiving a data calculation request;

and monitoring and early warning operation is carried out on the performance of the target prediction model based on the performance report.

An embodiment of the present application further provides a storage medium, where a computer program is stored in the storage medium, and when the computer program runs on a computer, the computer executes the model data monitoring method according to any of the above embodiments.

It should be noted that, for the model data monitoring method described in this application, a person skilled in the art may understand that all or part of the process of implementing the model data monitoring method described in this application may be implemented by controlling related hardware through a computer program, where the computer program may be stored in a computer readable storage medium, such as a memory of a terminal device, and executed by at least one processor in the terminal device, and during the execution process, the process of implementing the embodiment of the model data monitoring method may be included. The storage medium may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like.

For the model data monitoring device in the embodiment of the present application, each functional module may be integrated into one processing chip, or each module may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium, such as a read-only memory, a magnetic or optical disk, or the like.

The model data monitoring method, apparatus, medium, and device provided in the embodiments of the present application are described in detail above. The principle and the implementation of the present application are explained herein by applying specific examples, and the above description of the embodiments is only used to help understand the method and the core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. A method of monitoring model data, comprising:

receiving a data acquisition request;

receiving a data calculation request;

2. The method for monitoring model data according to claim 1, wherein the obtaining of the target predicted value and the target actual value of the data to be predicted in the data obtaining request comprises:

3. The method of model data monitoring of claim 1, the method further comprising:

receiving a data remarking request;

acquiring remark information in the data remark request;

4. The model data monitoring method of claim 1, wherein the performing a monitoring and forewarning operation on the performance of the target prediction model based on the performance report comprises:

5. The method of model data monitoring of claim 4, the method further comprising:

6. The model data monitoring method of claim 1, further comprising:

receiving a data output request;

transmitting the target performance report to the target receiving address.

7. The method of model data monitoring of claim 1, the method further comprising:

receiving a data viewing request;

providing a monitoring list page containing a plurality of performance reports according to the data viewing request, wherein the performance reports belong to a plurality of prediction models of the same type.

8. A model data monitoring apparatus, characterized in that the model data monitoring apparatus comprises:

the first receiving module is used for receiving a data acquisition request;

the second receiving module is used for receiving a data calculation request;

the data calculation module is used for substituting the target predicted value and the target real value to calculate based on the target algorithm to obtain target result data;

9. A computer-readable storage medium storing instructions adapted to be loaded by a processor to perform the model data monitoring method of any one of claims 1 to 7.

10. A terminal device comprising a processor and a memory, the memory storing a plurality of instructions, the processor loading the instructions to perform the model data monitoring method of any one of claims 1 to 7.