CN115292144A

CN115292144A - Credibility evaluation method, device and equipment for multi-party model and multi-party financial model

Info

Publication number: CN115292144A
Application number: CN202211230845.0A
Authority: CN
Inventors: 唐丹叶; 王帅; 王爽; 郑灏; 李帜
Original assignee: Beijing Nuowei Information Technology Co ltd
Current assignee: Beijing Nuowei Information Technology Co ltd
Priority date: 2022-10-10
Filing date: 2022-10-10
Publication date: 2022-11-04

Abstract

The embodiment of the application provides a credible assessment method, a credible assessment device and credible assessment equipment for a multi-party model and a multi-party financial model, wherein the credible assessment method for the multi-party model comprises the following steps: responding to a virtual machine configuration instruction of a target user, and distributing a corresponding virtual machine for the target user; obtaining an analysis result and an analysis duration corresponding to the model to be evaluated, which are provided by the target user; acquiring a monitoring log, and determining target evaluation information according to the monitoring log; and generating an evaluation result of the model to be evaluated according to the target evaluation information, the analysis result and the analysis duration so as to determine a comparison result of the multi-party model. The model automatic evaluation based on the virtual machines is realized, and the evaluation and comparison of multi-party models can be realized through a plurality of groups of virtual machines so as to select a better model, so that the model evaluation efficiency is high; and the evaluation equipment generates the evaluation result of the model based on the target evaluation information corresponding to the monitoring log, the analysis result and the analysis duration of the model, so that the evaluation accuracy is improved.

Description

Credibility evaluation method, device and equipment for multi-party model and multi-party financial model

Technical Field

The embodiment of the application relates to the technical field of information processing, in particular to a credibility assessment method, device and equipment for a multi-party model and a multi-party financial model.

Background

With the rapid development of artificial intelligence technology, artificial intelligence algorithms are widely applied in the data processing fields such as image processing and natural language processing. In many fields, model evaluation competitions are held through set application tasks such as behavior prediction fields, image recognition fields, natural language translation and the like, artificial intelligence models provided by competitors are evaluated, and therefore optimal models are screened out, and progress of artificial intelligence technology is promoted.

In the traditional evaluation process, participants upload respective models to a review terminal, and review personnel use the uploaded models to perform data analysis, complete corresponding tasks and perform manual scoring based on output results.

The evaluation mode consumes a large amount of labor and time cost, has low evaluation efficiency and cannot meet the requirement.

Disclosure of Invention

The embodiment of the application provides a credible assessment method, a credible assessment device and credible assessment equipment for a multi-party model and a multi-party financial model, and the method, the credible assessment device and the credible assessment equipment for the multi-party model and the multi-party financial model realize automatic assessment of the multi-party model by configuring a virtual machine and improve assessment efficiency.

In a first aspect, an embodiment of the present application provides a method for credible evaluation of a multi-party model, where the method includes:

responding to a virtual machine configuration instruction of a target user, and distributing a corresponding virtual machine for the target user; the virtual machine is provided with a monitoring component, the monitoring component is used for monitoring evaluation indexes of the virtual machine in a target time period, forming a monitoring log according to the detected evaluation indexes and transmitting the monitoring log to evaluation equipment according to preset interval duration, and the target time period comprises the following steps: analyzing the time period of the data to be analyzed stored in the virtual machine based on the code corresponding to the model to be evaluated, which is provided by the corresponding virtual machine and operated by the target user;

obtaining an analysis result and an analysis duration corresponding to the model to be evaluated, which are provided by the target user;

acquiring the monitoring log, and determining target evaluation information according to the monitoring log;

generating an evaluation result of the model to be evaluated according to the target evaluation information, the analysis result and the analysis duration so as to determine a comparison result of the multi-party model;

the evaluation indexes in the monitoring log comprise a first index related to a central processing unit, a second index related to a graphic processor, a third index related to a cache, a fourth index related to a bandwidth and a fifth index related to data to be analyzed.

Optionally, the determining target evaluation information according to the monitoring log includes at least one of the following steps:

determining process use information of a central processing unit according to the first index so as to determine a first score as target evaluation information;

determining algorithm use information of the graphics processor according to the second index to determine a second score as target evaluation information;

according to the third index, cache use information related to the cache is determined, and a third score is determined to serve as target evaluation information;

according to the fourth index, bandwidth use information related to the bandwidth is determined, and a fourth score is determined to serve as target evaluation information;

and determining data use information related to the data to be analyzed according to the fifth index, and determining a fifth score as target evaluation information.

Optionally, the virtual machine includes a child virtual machine, a first storage area in a locked state is set in the child virtual machine, and training data is stored in the first storage area, and the method further includes:

generating a key pair of a first public key and a first private key, and sending the first public key to the virtual machine;

receiving a first unlocking request encrypted by a first public key from a virtual machine, wherein the first unlocking request comprises first verification information;

the first private key is adopted to decrypt the first unlocking request, and a first unlocking instruction containing the first verification information is generated in response to the first unlocking request;

outputting the first unlocking instruction to a virtual machine, so that a sub-virtual machine of the virtual machine verifies the first unlocking instruction according to the first verification information, and after the verification is passed, the first storage area is switched from a locking state to an unlocking state according to the first unlocking instruction, so that the model to be evaluated is trained in the virtual machine according to the training data, and the state of the first storage area is switched to the locking state when the training is finished;

and monitoring the use amount of the training data to determine the evaluation result of the model to be evaluated by combining the use amount of the training data.

Optionally, the method further includes:

receiving augmentation information of the training data uploaded by the virtual machine, wherein the augmentation information comprises: amplification mode and amplification amount; the amplification mode comprises artificial amplification, amplification according to an amplification rule and amplification through an amplification model;

generating an evaluation result of the model to be evaluated according to the target evaluation information, the analysis result and the analysis duration, wherein the evaluation result comprises:

and determining the evaluation result of the model to be evaluated according to the amplification information, the target evaluation information, the analysis result and the analysis duration.

Optionally, the virtual machine is provided with a second storage area in a locked state, and the second storage area stores data to be analyzed, and the method further includes:

generating a key pair of a second public key and a second private key, and sending the second public key to the virtual machine;

receiving a second unlocking request encrypted by a second public key from the virtual machine, wherein the second unlocking request comprises second verification information;

the second unlocking request is decrypted by adopting the second private key, and a second unlocking instruction containing the second verification information is generated in response to the second unlocking request;

and outputting the second unlocking instruction to the virtual machine, so that the virtual machine verifies the second unlocking instruction according to the second verification information, switches the locking state of the second storage area to the unlocking state according to the second unlocking instruction after the verification is passed, analyzes the data to be analyzed and the model to be evaluated in the virtual machine, and switches the state of the second storage area to the locking state when the analysis is finished.

Optionally, the virtual machine configuration instruction includes the number of virtual machines, and in response to the virtual machine configuration instruction of the target user, configuring the virtual machine corresponding to the target user includes:

acquiring a virtual machine configuration instruction of each target user in a multi-party user;

and when the number of the virtual machines in the virtual machine configuration instruction is greater than or equal to two, allocating the virtual machines with the corresponding number of the virtual machines, and configuring the bandwidth between the virtual machines into a preset bandwidth.

Optionally, the method further includes:

storing an image file of the virtual machine into a block chain at a target time so as to call verification, wherein the target time comprises: the method comprises the steps of a first time before a model to be evaluated analyzes data to be analyzed, a second time after the model to be evaluated analyzes the data to be analyzed, a third time before the model to be evaluated is trained by training data and a fourth time after the model to be evaluated is trained by the training data.

In a second aspect, the present application provides a credible assessment method for a multi-party financial model, comprising:

responding to a virtual machine configuration instruction of a target user, and distributing a corresponding virtual machine for the target user; the virtual machine is provided with a monitoring component, the monitoring component is used for monitoring evaluation indexes of the virtual machine in a target time interval, forming a monitoring log according to the detected evaluation indexes according to preset interval duration, and transmitting the monitoring log to evaluation equipment, and the target time interval comprises: analyzing the time period of the data to be analyzed stored in the virtual machine based on the corresponding code of the financial model provided by the target user when the corresponding virtual machine runs;

obtaining an analysis result and an analysis duration corresponding to the financial model provided by the target user;

generating an evaluation result of the financial model according to the target evaluation information, the analysis result and the analysis duration so as to determine a comparison result of the multi-party financial model;

the evaluation indexes in the monitoring log comprise a first index related to a central processing unit, a second index related to a graphic processing unit, a third index related to a cache, a fourth index related to a bandwidth and a fifth index related to data to be analyzed.

In a third aspect, the present application further provides an apparatus for trusted evaluation of a multi-party model, where the apparatus includes:

the virtual machine configuration module is used for responding to a virtual machine configuration instruction of a target user and distributing a corresponding virtual machine to the target user; the virtual machine is provided with a monitoring component, the monitoring component is used for monitoring evaluation indexes of the virtual machine in a target time interval, forming a monitoring log according to the detected evaluation indexes according to preset interval duration, and transmitting the monitoring log to evaluation equipment, and the target time interval comprises: the time period of the data to be analyzed stored in the virtual machine is analyzed based on the corresponding code of the model to be evaluated, which is provided by the corresponding virtual machine and operated by the target user;

the analysis result acquisition module is used for acquiring an analysis result and analysis duration corresponding to the model to be evaluated, which are provided by the target user;

the evaluation information determining module is used for acquiring the monitoring logs and determining target evaluation information according to the monitoring logs;

the evaluation result generation module is used for generating an evaluation result of the model to be evaluated according to the target evaluation information, the analysis result and the analysis duration so as to determine a comparison result of the multi-party model;

In a fourth aspect, the present application further provides an electronic device, including: a memory and at least one processor; the memory stores computer-executable instructions; the at least one processor executes computer-executable instructions stored by the memory to cause the at least one processor to perform the method provided by the first or second aspect of the present application.

In a fifth aspect, the present application further provides a computer-readable storage medium having stored thereon computer-executable instructions for implementing the method as provided in the first or second aspect of the present application when executed by a processor.

In a sixth aspect, the present application also provides a computer program product comprising a computer program that, when executed by a processor, performs the method as provided in the first or second aspect of the present application.

According to the credible evaluation method, device and equipment of the multi-party model and the multi-party financial model, aiming at an application scene of multi-party model evaluation, a target user such as a model designer issues a virtual machine configuration instruction through a model end, evaluation equipment responds to the virtual machine configuration instruction, allocates a corresponding virtual machine for the target user, provides an operating environment through the virtual machine, and executes a code corresponding to a model to be evaluated, so that analysis of data to be analyzed stored in the virtual machine is performed, after the analysis is finished, analysis duration and an analysis result are obtained, meanwhile, based on a monitoring log which is output by a monitoring component in the virtual machine and comprises a processor, a bandwidth, a cache, the data to be analyzed and other multi-dimensional evaluation indexes, target evaluation information is determined, and through the target evaluation information, the analysis duration and the analysis result, evaluation results such as scores of the model to be evaluated are automatically generated, so that automatic evaluation of the model is realized. After determining the respective evaluation results of the multiple parties, the evaluation results of the multiple parties may be compared to obtain a comparison result (e.g., a ranking).

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

FIG. 1 is a schematic diagram of a model evaluation process provided in an embodiment of the present application;

FIG. 2 is a flowchart illustrating a method for credible evaluation of a multi-party model according to an embodiment of the present application;

FIG. 3 is a schematic flowchart of a credibility assessment method for a multi-party model according to another embodiment of the present application;

FIG. 4 is a schematic diagram of a model evaluation system provided herein;

FIG. 5 is a schematic flowchart of a credibility assessment method for a multi-party model according to another embodiment of the present application;

FIG. 6 is a flowchart illustrating a method for credible assessment of a multi-party financial model according to one embodiment of the present application;

fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

With the above figures, there are shown specific embodiments of the present application, which will be described in more detail below. The drawings and written description are not intended to limit the scope of the inventive concepts in any manner, but rather to illustrate the concepts of the application by those skilled in the art with reference to specific embodiments.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

Artificial intelligence models, such as deep learning models, are widely used in data analysis in various fields, such as automatic driving, visual recognition, natural language processing, natural disaster prediction, and the like. In order to promote the benign development of the artificial intelligence model, a model evaluation game is usually held, and a model with better performance for a corresponding task is selected through the game evaluation.

Fig. 1 is a schematic diagram of a model evaluation process provided in an embodiment of the present application, and as shown in fig. 1, competitors upload respective trained models to an evaluation platform, and fig. 1 takes N competitors as an example, that is, competitor 1 to competitor N correspond to model 1 to model N, respectively; the evaluation platform evenly issues the N models to M evaluation terminals, namely an evaluation terminal 1 to an evaluation terminal M; respectively carrying out data analysis on the evaluation terminal 1 to the evaluation terminal M by the evaluation personnel 1 to the evaluation personnel M by adopting the models uploaded by the contestants to obtain the analysis time length and the analysis accuracy of the models, and further judging the scores of the models by the evaluation personnel based on the analysis time length and the analysis accuracy; and obtaining a competition result based on the scores of the models.

In the evaluation process, the evaluation personnel needs to manually operate the model uploaded by the competition personnel, so that the time and the labor are consumed, and the evaluation cost is high and the efficiency is low; and when the evaluation terminal faces evaluation tasks of a plurality of models, only serial evaluation can be performed, so that the overall evaluation efficiency is further low, and the requirements cannot be met. Meanwhile, the operation of different reviewers cannot be consistent, so that the model evaluation accuracy is influenced.

In order to improve the efficiency of model evaluation and realize standardized and automatic model evaluation, the application provides a model evaluation method based on virtual machines, wherein competitors request one or more virtual machines at a model end, codes corresponding to the model to be evaluated are operated based on the virtual machines, so that the model to be evaluated analyzes data to be analyzed stored in the virtual machines, and evaluation results of the model to be evaluated are automatically generated based on analysis results and analysis duration. Wherein, the virtual machine can also be a container (Docker).

The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. These several specific embodiments may be combined with each other below, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

Fig. 2 is a flowchart illustrating an evaluation method of a multi-party model according to an embodiment of the present application. The method is performed by an evaluation device, which may be a server, a virtual machine, or other device, or a model evaluation system. As shown in fig. 2, the model evaluation method provided in this embodiment includes the following steps:

step S201, in response to a virtual machine configuration instruction of a target user, allocating a corresponding virtual machine to the target user.

Wherein the target user may be any of a plurality of parties to the competition.

The target user can be a competitor or a model designer, the target user can issue a virtual machine configuration instruction through the model terminal, the evaluation device analyzes the virtual machine configuration instruction after receiving the virtual machine configuration instruction to obtain virtual machine configuration information, and then configures the corresponding virtual machine for the target user based on the virtual machine configuration information, and allocates the configured virtual machine to the target user. The model end is a terminal (or physical machine) such as a computer used by a competitor or a model designer.

The competitor or the model designer can open the virtual machine application page through the model end, apply for one or more virtual machines in the virtual machine application page, and accordingly generate the virtual machine configuration instruction based on the configuration information of the virtual machine application page.

The virtual machine distributed for the target user provides an operating environment and data to be analyzed for the model to be evaluated, so that the data to be analyzed in the virtual machine is analyzed based on the model to be evaluated provided by the target user through the operating environment provided by the distributed virtual machine, an analysis result and analysis duration are obtained, and a monitoring log is obtained through a monitoring component in the virtual machine during the analysis of the model to be evaluated.

Further, after receiving the virtual configuration instruction, the evaluation device may further determine, according to the model parameters in the virtual configuration instruction and the number of virtual machines applied, the target number of virtual machines corresponding to the model to be evaluated, to allocate the target number of virtual machines to the target user, and bandwidths of the allocated target number of virtual machines are the same (other parameters allocated to the virtual machines of each user are kept consistent, such as memory size, processor, graphics processor, and memory are kept consistent).

The model parameters may include parameters such as a main architecture adopted by the model to be evaluated, the number of model layers, and the magnitude of the parameters included in the model to be evaluated.

The virtual machine configuration instructions may further include a secure computing mode adopted by the model to be evaluated, such as TEE (Trusted Execution Environment computing), MPC (Multi-party computing), face (federal AI Technology Enabler), and the like.

The number of virtual machines, such as 1, 2, or other numbers, may also be included in the virtual machine configuration instructions.

In one embodiment, the bandwidth between virtual machines is the same, such as 5MB, 10MB, 20MB or other values.

The virtual machine is provided with a monitoring component, the monitoring component is used for monitoring the evaluation index of the virtual machine in a target time interval, forming a monitoring log according to the detected evaluation index according to the preset interval duration, and transmitting the monitoring log to evaluation equipment, and the target time interval comprises: and analyzing the time period of the data to be analyzed stored in the virtual machine based on the code corresponding to the model to be evaluated, which is provided by the corresponding virtual machine operation target user. The evaluation indexes in the monitoring log comprise a first index related to the central processing unit, a second index related to the graphic processing unit, a third index related to the cache, a fourth index related to the bandwidth and a fifth index related to the data to be analyzed. The target time interval can be determined according to an instruction sent by a user, for example, the starting time of the target time interval can be determined according to an instruction sent by the user for starting model analysis; the end time of the target period may also be determined based on an end instruction issued by the user (or based on an action of the model analysis to end).

The preset interval duration may be 30s, 60s, 2min, or other durations.

The monitoring component is responsible for monitoring the target time period of the operation of the virtual machine, the central processing unit, the graphic processor, the cache, the bandwidth and the use condition or the occupation condition of the data to be analyzed. The target time period comprises a time period for running the code corresponding to the model to be evaluated to analyze the data to be analyzed stored in the virtual machine.

The number of the data to be analyzed may be plural, such as multiple frames of images, multiple pieces of text, and the like.

The first index related to the central processing unit may include parameters such as a usage rate, a process amount, a process duration, and the like of the central processing unit corresponding to each time node, and the second index related to the graphics processing unit may include a usage rate, a data input amount, a data output amount, a data processing time consumption, and the like of the graphics processing unit corresponding to each time node. The third index related to the cache may include the cache occupancy rate corresponding to each time node, and may further include parameters such as the input times, the output times, and the duration of the input data. The fourth index related to the bandwidth may include the bandwidth occupancy corresponding to each time node, and may further include parameters such as the input times, the output times, and the duration of the bandwidth occupancy. The fifth index related to the data to be analyzed may include input and output data to be analyzed, and may further include a data amount of the input and output data to be analyzed.

In some embodiments, the evaluation index may include only a part of the above-described five indexes (first to fifth indexes), such as only the first to fourth indexes.

The monitoring log comprises various evaluation indexes monitored by various time sections.

Step S202, obtaining the analysis result and the analysis duration corresponding to the model to be evaluated provided by the target user.

The analysis result comprises the output of the model to be evaluated relative to each data to be analyzed, and can also comprise the analysis accuracy of each data to be analyzed.

After distributing a corresponding number of virtual machines (stored in a virtual machine configuration instruction) for each target user, confirming to start analysis, thereby running an evaluation script, starting the deployed virtual machines on corresponding command lines, then automatically running codes corresponding to models to be evaluated, analyzing data to be analyzed, recording the analysis accuracy of each group of data to be analyzed by the virtual machines in the analysis process, and obtaining the analysis duration after the analysis is finished. And then the analysis accuracy and the analysis duration of each group of data to be analyzed are sent to the evaluation equipment.

In one embodiment, before the analysis is started and after the analysis result is confirmed, the image of the allocated virtual machine may also be written into the blockchain, so as to check whether the analysis result and the analysis duration are accurate. The hash function may employ a SHA256 function.

Step S203, acquiring the monitoring log, and determining target evaluation information according to the monitoring log.

Specifically, values of evaluation indexes (including the first index to the fifth index) in the monitoring log may be sequentially extracted, and the target evaluation information may be determined based on the values of the evaluation indexes.

The target rating information may be represented by a score, such as a percentile score.

determining process use information of a central processing unit according to the first index so as to determine a first score as target evaluation information; determining algorithm use information of the graphics processor according to the second index to determine a second score as target evaluation information; determining cache use information related to the cache according to the third index, and determining a third score as target evaluation information; according to the fourth index, bandwidth use information related to the bandwidth is determined, and a fourth score is determined to serve as target evaluation information; and determining data use information related to the data to be analyzed according to the fifth index, and determining a fifth score as target evaluation information.

The process usage information may include a maximum value, a minimum value, a mean value, a variance, and the like of the process amount corresponding to each time node in the first index, and may further include a duration of the process.

The algorithm usage information may include the data amount input to the graphics processor, the data amount output to the graphics processor, the data amount processed by the graphics processor, and the occupancy rate of the graphics processor corresponding to each time node in the second index, and may further include the average value, the maximum value, the minimum value, and the like of the foregoing parameters.

The cache usage information may include a maximum value, a minimum value, an average value, and the like of the cache occupancy rate corresponding to each time node in the third index, and may further include the input frequency and the output frequency of the data in the cache, and the duration from the output of the data to the output of the data.

The bandwidth usage information may include a maximum value, a minimum value, an average value, and the like of the bandwidth occupancy corresponding to each time node in the fourth index, and may further include the input times and the output times of the output in the bandwidth, and the duration from the output to the output of the data.

The data usage information may include the call amount, the number of calls, the call frequency, etc. of the data to be analyzed.

The target evaluation information may include at least one of the above-described process usage information, algorithm usage information, cache usage information, bandwidth usage information, and data usage information.

Evaluation information of each dimensionality for model evaluation is extracted through the monitoring log, and comprehensiveness and accuracy of model evaluation are improved.

Step S203 and step S202 may be executed in series or in parallel, for example, step S203 may be executed first and then step S202 is executed, and step S202 is executed first and then step S203 is executed in fig. 2 as an example.

And step S204, generating an evaluation result of the model to be evaluated according to the target evaluation information, the analysis result and the analysis duration so as to determine a comparison result of the multi-party model.

The evaluation result may include an evaluation score, and may further include parameters such as an analysis duration, an analysis accuracy of each data to be analyzed, and the like.

Specifically, the evaluation score of the model to be evaluated may be determined according to the weighted average of the target evaluation information, the analysis result, and the analysis duration. And then, obtaining a comparison result by comparing the evaluation scores of the models to be evaluated provided by each target user in the multi-party users participating in the competition.

Wherein, the evaluation score is in an inverse correlation with the analysis duration and in a positive correlation with the analysis accuracy. When data analysis is performed on the basis of a model to be evaluated through the running environment provided by the virtual machine, if the analysis time and the analysis accuracy are fixed, the more reasonable the central processing unit, the graphic processing unit, the cache and the bandwidth are used (the more reasonable the information such as the utilization rate and the extreme value is judged to be smaller is judged), the higher the score is, and the more reasonable the calling use condition of the data to be analyzed is, the higher the evaluation score or the rating of the model to be evaluated is.

Specifically, based on a pre-established model evaluation relational expression, evaluation scores of models to be evaluated provided by multiple users are generated according to target evaluation information, analysis results and analysis duration, and the model to be evaluated with the highest evaluation score is determined to be the optimal model.

In the trusted evaluation method of the multi-party model provided by this embodiment, for an application scenario of multi-party model evaluation, a target user, such as a model designer, issues a virtual machine configuration instruction through a model end, an evaluation device responds to the virtual machine configuration instruction, allocates a corresponding virtual machine to the target user, provides an operating environment through the virtual machine, and executes a code corresponding to a model to be evaluated, so as to analyze data to be analyzed stored in the virtual machine, and after the analysis is completed, obtain analysis time and an analysis result, and at the same time, determine target evaluation information based on a monitoring log including multidimensional evaluation indexes such as a processor, a bandwidth, a cache, and data to be analyzed, which is output by a monitoring component in the virtual machine, and automatically generate an evaluation result, such as a score, of the model to be evaluated through the target evaluation information, analysis duration, and the analysis result, thereby achieving automatic evaluation of the model, and improving evaluation efficiency and accuracy compared with an artificial evaluation mode.

It should be noted that, in this embodiment, in addition to determining the evaluation results of each party and then comparing, each item of information may also be compared to determine the difference, and then determine the comparison result of the models of different parties, for example, the index or the usage information of the CPU of the first party may be compared with the index or the usage information of the CPU of the second party to determine the difference, so as to determine the comparison result of multiple parties according to multiple differences (weights may also be set for the differences).

Fig. 3 is a schematic flow chart of a trusted evaluation method for a multi-party model according to another embodiment of the present application, and this implementation is directed to a case that a virtual machine is provided with a second storage area in a locked state, where data stored in the storage area cannot be read in the locked state. Based on the embodiment shown in fig. 2, the model evaluation method provided in this embodiment further refines step S201, and adds a step of unlocking a second storage area after step S201 to obtain data to be analyzed, as shown in fig. 3, the model evaluation method provided in this embodiment may include the following steps:

step S301, obtaining virtual machine configuration instructions sent by a plurality of target users.

The virtual machine configuration instruction includes the number of virtual machines, and the virtual machine configuration instruction may further include other information, which may be specifically configured according to a requirement.

Step S302, aiming at each target user in a plurality of target users, allocating corresponding virtual machines to the target user according to the virtual machine configuration instruction of the target user, and when the number of the virtual machines in the virtual machine configuration instruction is larger than or equal to two, allocating the virtual machines with the number corresponding to the number of the virtual machines to the target user, and configuring the bandwidth between the virtual machines to be a preset bandwidth.

The preset bandwidth may be 10MB, 20MB, or other values.

In some embodiments, the evaluation device may further set parameters such as cache, memory, and the like of the allocated virtual machines, so that configurations of the virtual machines allocated by different target users are the same, so as to ensure fairness of model evaluation. The same virtual machines are configured to provide the same running environment for each to-be-evaluated model of the competition, so that the consistency of the evaluation environments of the models is ensured, and the accuracy of the evaluation result is improved.

A virtual machine can be used as an analysis node and is responsible for executing corresponding processes and monitoring analysis processes. Analyzing the data to be analyzed based on the model to be evaluated uploaded by the participants, monitoring various operations executed in the analysis process and recording the output data. The analysis node can also encrypt and transmit data (such as the monitoring log) output in the analysis process to the evaluation device to obtain an evaluation result.

The virtual machine is provided with a second storage area in a locked state, and data to be analyzed is stored in the second storage area, wherein the virtual machine can identify an instruction for accessing the second storage area so as to allow or reject the instruction (a locking control can be set to complete instruction identification and processing), the second storage area has a locked state and an unlocked state, when in the locked state, the virtual machine rejects instructions (such as instructions of reading, writing and the like) except the unlocked instruction, and when in the unlocked state, the virtual machine allows the data in the second storage area to be processed in the virtual machine.

Since the data to be analyzed stored in the second storage area in the locked state cannot be directly read, the unlocking and re-locking of the second storage area need to be performed through the relevant steps provided in step S303 to step S306, so as to prevent the data to be analyzed from being tampered with.

Step S303, generating a key pair of the second public key and the second private key, and sending the second public key to the virtual machine.

And when the model evaluation is carried out, the virtual machine encrypts the second unlocking request based on the second public key and sends the encrypted second unlocking request to the evaluation equipment so as to unlock the second storage area. In order to improve the security of data interaction, the second unlocking request includes second verification information.

Step S304, receiving a second unlocking request encrypted by a second public key from the virtual machine, where the second unlocking request includes second verification information.

Step S305, decrypting the second unlocking request by using the second private key, and generating a second unlocking instruction including the second verification information in response to the second unlocking request.

The evaluation device decrypts the encrypted second unlocking request based on the second private key to obtain a second unlocking request, responds to the second unlocking request, and generates a corresponding second unlocking instruction, wherein the second unlocking instruction comprises the second verification information (the evaluation device adds the second verification information to the second unlocking instruction).

Step S306, outputting the second unlocking instruction to a virtual machine, so that the virtual machine verifies the second unlocking instruction according to the second verification information, and after the verification is passed, switching the second storage area from the locked state to the unlocked state according to the second unlocking instruction, so as to perform analysis in the virtual machine according to the data to be analyzed and the model to be evaluated, and switching the state of the second storage area to the locked state when the analysis is finished.

And the evaluation equipment sends the second unlocking instruction to the virtual machine, the virtual machine extracts second verification information in the second unlocking instruction, verifies the second unlocking instruction based on the second verification information, and responds to the second unlocking instruction to unlock the second storage area if the verification is passed, so that the state of the second storage area is switched from the locking state to the unlocking state, and analysis is performed based on the data to be analyzed stored in the second storage area.

In order to avoid tampering of the data to be analyzed, the virtual machine switches the state of the second storage area back to the locked state after the analysis is completed.

Step S307, a monitoring log formed by a monitoring component in the virtual machine is obtained, and target evaluation information is determined according to the monitoring log.

The monitoring control is used for monitoring various indexes in the running process of the virtual machine, periodically (according to a first period, such as 50 ms) forming a monitoring log, and uploading the formed monitoring log to the evaluation equipment.

The monitoring control may upload the monitoring log to the evaluation device periodically (according to a second period, for example, 1 s), or upload the newly added monitoring log to the evaluation device every time one monitoring log is generated. If the evaluation device does not receive the monitoring log for a long time (such as ten seconds), it can be determined that the virtual machine is out of control, and the processing can be performed through manual intervention or a virtual machine pause mode (sending a pause instruction).

In the virtual environment provided by the virtual machine, the competitor adopts the trained model to be evaluated to analyze the data to be analyzed stored in the second storage area, and obtains an analysis result. In the process of virtual machine analysis, the monitoring component monitors various indexes of the virtual machine in real time, such as memory, bandwidth and the like, and a plurality of monitoring logs are formed.

The virtual machine is also provided with a locking control, the second storage area is unlocked through the locking control in the process of analyzing the data to be analyzed, namely during the period of generating the first monitoring log and the last detection log, so as to read the data to be analyzed, and the second storage area is locked based on the locking control after the last monitoring log is output, so that the data to be analyzed is prevented from being tampered.

Taking data to be analyzed as natural language, such as an original text and a corresponding translation, performing original text translation through the model to be analyzed to obtain a to-be-evaluated translation output by the model, calculating an error between the to-be-evaluated translation and the translation corresponding to the original text, and obtaining an analysis accuracy or an analysis error corresponding to the model to be evaluated.

Taking data to be analyzed as an image as an example, such as an image and a label corresponding to the image, if the model to be evaluated needs to identify an object in the image, the image can be input into the model to be evaluated, so as to obtain the object (the label to be evaluated) in the image identified by the model to be evaluated and the probability corresponding to the object, and if the label to be evaluated is consistent with the label corresponding to the image, an analysis accuracy corresponding to the model to be evaluated is determined based on the probability corresponding to the label to be evaluated; and if the label to be evaluated is inconsistent with the label corresponding to the image, the corresponding analysis accuracy is 0.

Exemplarily, fig. 4 is a schematic structural diagram of the model evaluation system provided in the present application, and as shown in fig. 4, the model evaluation system includes a plurality of working nodes, management nodes, and servers, and 3 working nodes are taken as an example in fig. 4, and both the working nodes and the management nodes are virtual machines. The working node is a virtual machine allocated to a target user, and is used for executing a model analysis task (that is, a code corresponding to a model to be evaluated is run based on a running environment provided by the virtual machine so as to analyze data to be analyzed stored in the virtual machine based on the model to be evaluated), monitoring operations performed by competitors in the execution process of the model analysis task, various indexes of the virtual machine, the use condition of the data to be analyzed, and the like, forming a monitoring log, and encrypting and transmitting the monitoring log to the server.

The monitoring log can also comprise identification information such as contestant identifications, belonging team identifications, team names, team units and the like.

Recording the time point of the beginning, the time point of the end and the analysis duration of the model to be evaluated in the monitoring log, acquiring the occupied memory once according to a set period (such as 30s, 60s, 100s and the like), recording the data sending amount and the data receiving amount in the analysis process, and recording the output of the model to be evaluated, such as a matrix, a tree and the like.

The management node is provided with a service process module and a resource scheduling module, wherein the service process module is used for receiving the mirror image, the algorithm and the like of the virtual machine and storing the mirror image, the algorithm and the like to the block chain; and the resource scheduling module is used for allocating resources for the model end. The server is used for receiving the monitoring logs sent by the working nodes, determining the analysis accuracy and the target evaluation information of the models to be evaluated based on the output results in the monitoring logs, and calculating the evaluation score of the models to be evaluated based on the parameters such as the analysis duration, the analysis accuracy and the occupied memory. The server is also used for generating a competition result based on the evaluation scores of the models to be evaluated. The evaluation device may be the server, or include the server and a management node.

Taking the memory limit of the virtual machines as 6G as an example, the bandwidth between the virtual machines is limited to 10MB. The virtual machines 1 to n are virtual machines deployed at different model ends and are used for executing analysis tasks of corresponding models to be evaluated. In the analysis process, the virtual machines 1 to n respectively send (push) the generated monitoring logs to a log virtual machine, which may be the management node or the evaluation device.

Step S308, obtaining the analysis result and the analysis duration corresponding to each model to be evaluated.

Steps S307 and S308 may be executed serially or in parallel, for example, step S308 may be executed first and then step S307 is executed, and parallel execution is taken as an example in fig. 3.

Step S309, generating an evaluation result of each model to be evaluated according to the target evaluation information, the analysis result and the analysis duration, and determining a comparison result of the multi-party model according to the evaluation result of each model to be evaluated.

Specifically, the evaluation score of the model to be evaluated may be determined according to the target evaluation information, the analysis result, and the weighted average of the analysis duration.

In one embodiment, a model identifier may be set for each model to be evaluated, so as to obtain an analysis result, an analysis duration and a monitoring log corresponding to the model to be evaluated based on the model identifier, so as to calculate an evaluation score.

Optionally, the monitoring log includes at least one evaluation index, where the at least one evaluation index includes at least one of a central processing unit usage rate, a graphics processing unit usage rate, a memory usage rate, and a bandwidth usage rate.

The memory usage rate may include an operating memory usage rate and a storage space usage rate.

The lower each usage rate, the higher the evaluation score corresponding to the model to be evaluated.

Optionally, generating an evaluation result of the model to be evaluated according to the analysis result, the analysis duration and the monitoring log, including:

determining the value of an evaluation index corresponding to the model to be evaluated according to the monitoring log; and calculating the evaluation score of the model to be evaluated according to the analysis result, the analysis duration, the evaluation index value and a preset relational expression.

Specifically, each monitoring log corresponding to the model to be evaluated may be analyzed, and based on the analysis result, the value of each evaluation index in each monitoring log is obtained.

For example, if the graphics processor usage rate is 40%, the evaluation index may take a value of 40 or 0.4.

The preset relational expression can be a linear relational expression or a nonlinear relational expression, and the specific form of the preset relational expression is not limited in the application.

For example, the preset relationship may be:

wherein, w ₁ 、w ₂ 、w ₃ 、c _j (j = 1, 2, 3, 4) are all weight coefficients; s represents the evaluation score of the model to be evaluated; t represents the analysis duration of the model to be evaluated; a represents the analysis accuracy of the model to be evaluated; o _ij Representing the jth utilization rate in the ith monitoring log, wherein the jth utilization rate comprises four utilization rates of the central processing unit utilization rate, the graphics processing unit utilization rate, the memory utilization rate and the bandwidth utilization rate; and k is the number of the monitoring logs corresponding to the model to be evaluated.

Further, before and after the analysis of the model to be evaluated is completed, the mirror image of the virtual machine, including data of the model to be evaluated, the analysis result, the analysis time length, the monitoring log and the like, can be written into the block chain to store the data, so that when the evaluation result needs to be rechecked subsequently, the rechecking can be performed based on the intermediate state data stored in the block chain.

Based on the pre-designed relational expression, the evaluation score is calculated according to the evaluation index, the analysis duration and the analysis accuracy, the calculation efficiency is high, and the model evaluation efficiency is improved.

In the embodiment, the indexes in the model analysis process are introduced for model evaluation, so that the comprehensiveness of model evaluation is improved, and the accuracy of model evaluation is further improved; the multi-party model parallel evaluation is carried out through the virtual machines with the same configuration, evaluation deviation caused by the fact that competitors use equipment with different configurations to carry out model analysis is effectively avoided, accuracy of model evaluation is improved, and efficiency of model evaluation is greatly improved through the model parallel evaluation; by locking the second storage area for storing the data to be analyzed and unlocking the second storage area based on the secret key during model analysis, the safety of the data to be analyzed is improved, and the data to be analyzed is prevented from being tampered before analysis or after the analysis is finished.

Fig. 5 is a schematic flow chart of a trusted evaluation method for a multi-party model according to another embodiment of the present disclosure, in this embodiment, a virtual machine includes a sub-virtual machine, a first storage area in a locked state is disposed in the sub-virtual machine, and training data is stored in the first storage area, so that training of a model to be evaluated is performed based on the training data, and thus model evaluation is performed in combination with a training process and an analysis process, so as to further improve comprehensiveness and accuracy of model evaluation.

Based on the foregoing embodiment, the model evaluation method provided in this embodiment further refines step S204, and increases the steps of obtaining training data and monitoring usage amount of the model training process after step S201, as shown in fig. 5, the model evaluation method provided in this embodiment may include the following steps:

step S501, responding to a virtual machine configuration instruction of a target user, and allocating a corresponding virtual machine to the target user.

The virtual machine comprises a sub virtual machine, wherein a first storage area in a locked state is arranged in the sub virtual machine, and training data are stored in the first storage area.

In order to read the training data stored in the first memory area, the first memory area in the locked state needs to be unlocked.

Step S502, generating a key pair of the first public key and the first private key, and sending the first public key to the virtual machine.

Step S503 is to receive a first unlocking request encrypted by using a first public key from the virtual machine, where the first unlocking request includes first verification information.

Step S504, decrypting the first unlocking request by using the first private key, and generating a first unlocking instruction including the first verification information in response to the first unlocking request.

Step S505, outputting the first unlocking instruction to a virtual machine, so that a sub-virtual machine of the virtual machine verifies the first unlocking instruction according to the first verification information, and after the verification is passed, switching the first storage area from a locked state to an unlocked state according to the first unlocking instruction, so as to train the model to be evaluated in the virtual machine according to the training data, and switching the state of the first storage area to the locked state when the training is finished.

Step S506, the usage amount of the training data is monitored.

The usage amount of training data during the training of the model to be evaluated can be monitored based on the monitoring component or other monitoring components.

Step S507, obtaining an analysis result and an analysis duration corresponding to the model to be evaluated, which are provided by the target user.

Step S508, obtaining the monitoring log, and determining target evaluation information according to the monitoring log.

Steps S506, S507, and S508 may be executed in parallel, serially, or partially serially and partially in parallel, or based on the occurrence time sequence, and the execution sequence of the above three steps is not limited in the present application.

Step S509, generating an evaluation result of the model to be evaluated according to the usage amount of the training data, the target evaluation information, the analysis result, and the analysis duration, so as to determine a comparison result of the multi-party model.

The evaluation result may also include the identification of the contestant, such as a number.

Specifically, the evaluation score of the model to be evaluated may be determined according to the usage amount of the training data, the target evaluation information, the analysis result, and the weighted average of the analysis duration, and the comparison result of the multi-party model may be determined based on the evaluation score of each model to be evaluated.

In the case where other parameters are the same, the less the amount of use of the training data, the higher the evaluation score of the evaluation model.

In order to further improve the comprehensiveness of model evaluation, the embodiment implements evaluation on the model training process through the training data in the child virtual machine, that is, model evaluation is performed by combining the usage amount of the training data, so as to further improve the accuracy of model evaluation.

The training data is stored in a first memory area of the child virtual machine. And training the model to be evaluated based on the training data stored in the running environment provided by the sub virtual machine, and acquiring a training log based on a monitoring control in the sub virtual machine in the training process.

The training log comprises training indexes such as corresponding training errors, training duration, occupied memory and the like during training of each batch.

In one embodiment, the first storage area in the child virtual machine may be locked by a lock control to prevent training data from being tampered with during non-model training periods.

Further, training data before training and training data after training can be stored in the blockchain for subsequent checking.

In one embodiment, the competitor may optimize or amplify the training data, for example, manually amplify the training data, amplify the training data based on an amplification rule, or amplify the training data based on a model, and take an image as an example, the training data may be amplified through image flipping, stitching, translating, and the like, so as to obtain more training data to perform model training to be evaluated.

The augmented information of the training data can also be comprehensively considered in model evaluation, and the method further comprises the following steps: receiving augmentation information of the training data uploaded by the virtual machine, wherein the augmentation information comprises: amplification mode and amplification amount; the amplification mode comprises artificial amplification, amplification according to an amplification rule and amplification through an amplification model.

Correspondingly, the step S204 specifically includes: and determining the evaluation result of the model to be evaluated according to the amplification information, the target evaluation information, the analysis result and the analysis duration.

The corresponding step S509 specifically includes: and determining the evaluation result of the model to be evaluated according to the amplification information, the usage amount of the training data, the target evaluation information, the analysis result and the analysis duration.

If the analysis accuracy of the model to be evaluated is improved after the training data is amplified, the amplification operation can improve the evaluation score of the model to be evaluated, and the evaluation score is decreased progressively according to the modes of model amplification, regular amplification and manual amplification.

If the analysis accuracy of the model to be evaluated is not improved after the training data is amplified, the operation of amplification reduces the evaluation score of the model to be evaluated.

Illustratively, assuming that other indexes or parameters except amplification and analysis accuracy are the same when the model A to be evaluated and the model B to be evaluated are evaluated, the training data is amplified when the model A to be evaluated is trained, the training data is not amplified when the model B to be evaluated is trained, and if the analysis accuracy of the model A to be evaluated is higher than the analysis accuracy of the model B to be evaluated, the evaluation score of the model A to be evaluated is higher than that of the model B to be evaluated; and if the analysis accuracy of the model A to be evaluated is lower than or equal to that of the model B to be evaluated, the evaluation score of the model A to be evaluated is lower than that of the model B to be evaluated.

To facilitate the rechecking, the image file of the virtual machine may be stored in a block chain. Optionally, the method further includes:

The image files of the virtual machine at four moments before the model to be evaluated starts to be trained, after the model to be evaluated is trained, before the model to be evaluated is analyzed and after the model to be evaluated is analyzed are stored in the block chain for backup, so that when objections exist in the evaluation result, whether the evaluation result is accurate can be checked again by referring to the data stored in the block chain. Further, the evaluation result of the model to be evaluated can be generated according to the training log, the monitoring log, the analysis result and the analysis duration.

Optionally, the method further includes: and acquiring the sample number of training data used for training the model to be evaluated. Correspondingly, generating an evaluation result of the model to be evaluated according to the training log, the analysis result and the analysis duration, including:

and generating an evaluation result of the model to be evaluated according to the number of the samples, the training log, the analysis result and the analysis duration.

When the child virtual machine does not perform training data locking, a competitor can adjust training data through operations such as training data amplification, optimization and the like, the training data before training (training data provided by the child virtual machine) and training data used during training of a model to be evaluated, namely adjusted training data, need to be recorded, and if the number of samples of the adjusted training data is larger than that of the training data before training, it is determined that the training data is amplified.

And setting the other parameters of the two models to be evaluated to be the same, such as the analysis duration and the analysis accuracy rate, and the evaluation score of the model to be evaluated, which is amplified by the training data, to be lower than the evaluation score of the model to be evaluated, which is not amplified.

Specifically, the weight corresponding to the analysis accuracy may be adjusted according to the number of samples. If the number of samples of training data used for training is greater than the number of samples of training data provided in the child virtual machine, the weight of the analysis accuracy rate is reduced.

By allowing the training data to be adjusted, the flexibility of model training is improved, the model training process is closer to the actual situation, whether the training data are adjusted or not is integrated, the evaluation score is calculated, and the accuracy of the evaluation score calculation is improved.

Fig. 6 is a flowchart illustrating a trusted evaluation method for a multi-party financial model according to an embodiment of the present application, as shown in fig. 6, the method includes the following steps:

step S601, responding to a virtual machine configuration instruction of a target user, and allocating a corresponding virtual machine to the target user.

The virtual machine is provided with a monitoring component, the monitoring component is used for monitoring evaluation indexes of the virtual machine in a target time period, forming a monitoring log according to the detected evaluation indexes and transmitting the monitoring log to evaluation equipment according to preset interval duration, and the target time period comprises the following steps: and analyzing the time period of the data to be analyzed stored in the virtual machine based on the corresponding code of the financial model provided by the target user when the corresponding virtual machine runs.

The financial model is applied to the financial field, such as a stock market price prediction model, a bankruptcy prediction model, a customer classification model and the like.

Step S602, obtaining the analysis result and the analysis duration corresponding to the financial model provided by the target user.

Step S603, acquiring a monitoring log, and determining target evaluation information according to the monitoring log.

And step S604, generating an evaluation result of the financial model according to the target evaluation information, the analysis result and the analysis duration so as to determine a comparison result of the multi-party financial model.

The credible assessment method of the multi-party model provided by the foregoing embodiment can be adopted to achieve assessment of the financial model, and only the model to be assessed needs to be replaced by the financial model, which is not described herein again.

An embodiment of the present application provides a credible evaluation apparatus for a multi-party model, where the apparatus includes:

the virtual machine configuration module is used for responding to a virtual machine configuration instruction of a target user and distributing a corresponding virtual machine to the target user; the virtual machine is provided with a monitoring component, the monitoring component is used for monitoring evaluation indexes of the virtual machine in a target time period, forming a monitoring log according to the detected evaluation indexes and transmitting the monitoring log to evaluation equipment according to preset interval duration, and the target time period comprises the following steps: the time period of the data to be analyzed stored in the virtual machine is analyzed based on the corresponding code of the model to be evaluated, which is provided by the corresponding virtual machine and operated by the target user; the analysis result acquisition module is used for acquiring an analysis result and analysis duration corresponding to the model to be evaluated, which are provided by the target user; the evaluation information determining module is used for acquiring the monitoring logs and determining target evaluation information according to the monitoring logs; an evaluation result generation module, configured to generate an evaluation result of the model to be evaluated according to the target evaluation information, the analysis result, and the analysis duration, so as to determine a comparison result of the multi-party model; the evaluation indexes in the monitoring log comprise a first index related to a central processing unit, a second index related to a graphic processing unit, a third index related to a cache, a fourth index related to a bandwidth and a fifth index related to data to be analyzed.

Optionally, the evaluation information determining module is specifically configured to perform at least one of the following steps:

determining process use information of a central processing unit according to the first index so as to determine a first score as target evaluation information; determining algorithm use information of the graphic processor according to the second index to determine a second score as target evaluation information; according to the third index, cache use information related to the cache is determined, and a third score is determined to serve as target evaluation information; according to the fourth index, bandwidth using information related to the bandwidth is determined, and a fourth score is determined to serve as target evaluation information; and determining data use information related to the data to be analyzed according to the fifth index, and determining a fifth score as target evaluation information.

Optionally, the virtual machine includes a child virtual machine, a first storage area in a locked state is provided in the child virtual machine, and training data is stored in the first storage area, and the apparatus further includes:

the first storage area unlocking module is used for generating a key pair of a first public key and a first private key and sending the first public key to the virtual machine; receiving a first unlocking request encrypted by a first public key from a virtual machine, wherein the first unlocking request comprises first verification information; the first private key is adopted to decrypt the first unlocking request, and a first unlocking instruction containing the first verification information is generated in response to the first unlocking request; outputting the first unlocking instruction to a virtual machine, so that a sub-virtual machine of the virtual machine verifies the first unlocking instruction according to the first verification information, and after the verification is passed, the first storage area is switched from a locking state to an unlocking state according to the first unlocking instruction, so that the model to be evaluated is trained in the virtual machine according to the training data, and the state of the first storage area is switched to the locking state when the training is finished; and the training data usage monitoring module is used for monitoring the usage of the training data so as to determine the evaluation result of the model to be evaluated by combining the usage of the training data.

Optionally, the apparatus further comprises:

an augmented information receiving module, configured to receive augmented information of the training data uploaded by the virtual machine, where the augmented information includes: amplification mode and amplification amount; the amplification mode comprises artificial amplification, amplification according to an amplification rule and amplification through an amplification model;

correspondingly, the evaluation result generation module is specifically configured to:

Optionally, the virtual machine is provided with a second storage area in a locked state, and the second storage area stores data to be analyzed, and the apparatus further includes:

the second storage area unlocking module is used for generating a key pair of a second public key and a second private key and sending the second public key to the virtual machine; receiving a second unlocking request encrypted by a second public key from the virtual machine, wherein the second unlocking request comprises second verification information; the second unlocking request is decrypted by adopting the second private key, and a second unlocking instruction containing the second verification information is generated in response to the second unlocking request; and outputting the second unlocking instruction to the virtual machine, so that the virtual machine verifies the second unlocking instruction according to the second verification information, switches the locking state of the second storage area to the unlocking state according to the second unlocking instruction after the verification is passed, analyzes the data to be analyzed and the model to be evaluated in the virtual machine, and switches the state of the second storage area to the locking state when the analysis is finished.

Optionally, the virtual machine configuration instruction includes the number of virtual machines, and the virtual machine configuration module is specifically configured to:

acquiring a virtual machine configuration instruction of each target user in multi-party users; and when the number of the virtual machines in the virtual machine configuration instruction is greater than or equal to two, allocating the virtual machines with the corresponding number of the virtual machines, and configuring the bandwidth between the virtual machines into a preset bandwidth.

Optionally, the apparatus further comprises:

the uplink module is used for storing the image file of the virtual machine into the block chain at a target moment so as to call the verification, wherein the target moment comprises: the method comprises the steps of a first time before a model to be evaluated analyzes data to be analyzed, a second time after the model to be evaluated analyzes the data to be analyzed, a third time before the model to be evaluated is trained by training data and a fourth time after the model to be evaluated is trained by the training data.

The trusted evaluation device for the multi-party model provided by the embodiment can execute the trusted evaluation method for the multi-party model provided by any embodiment of the application, and has corresponding functional modules and beneficial effects of the execution method.

Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application, and as shown in fig. 7, the electronic device includes: a memory 710, and at least one processor 720.

Wherein the memory 710 stores computer-executable instructions, and the at least one processor 720 executes the computer-executable instructions stored by the memory 710, such that the at least one processor 720 performs a method as provided by any of the embodiments of the present application.

Wherein the memory 710 and the processor 720 are connected by a bus 730.

The related description may be understood according to the related description and effect corresponding to the steps provided in the above method embodiments, and will not be described in detail herein.

One embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when a processor executes the computer-executable instructions, a method as provided in any embodiment of the present application is implemented.

The computer readable storage medium may be, among others, ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

An embodiment of the present application provides a computer program product comprising a computer program which, when executed by a processor, performs the method as provided in any of the embodiments of the present application.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of modules is merely a logical division, and other divisions may be realized in practice, for example, a plurality of modules or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. A method for trusted evaluation of a multi-party model, the method comprising:

responding to a virtual machine configuration instruction of a target user, and distributing a corresponding virtual machine for the target user; the virtual machine is provided with a monitoring component, the monitoring component is used for monitoring evaluation indexes of the virtual machine in a target time interval, forming a monitoring log according to the detected evaluation indexes according to preset interval duration, and transmitting the monitoring log to evaluation equipment, and the target time interval comprises: the time period of the data to be analyzed stored in the virtual machine is analyzed based on the corresponding code of the model to be evaluated, which is provided by the corresponding virtual machine and operated by the target user;

2. The method according to claim 1, wherein determining target evaluation information from the monitoring log comprises at least one of:

determining algorithm use information of the graphic processor according to the second index to determine a second score as target evaluation information;

according to the fourth index, bandwidth using information related to the bandwidth is determined, and a fourth score is determined to serve as target evaluation information;

3. The method of claim 1, wherein the virtual machine comprises a child virtual machine, a first memory area in a locked state is disposed in the child virtual machine, and training data is stored in the first memory area, and the method further comprises:

4. The method of claim 3, further comprising:

5. The method of claim 1, wherein the virtual machine is provided with a second memory area in a locked state, the second memory area storing data to be analyzed, the method further comprising:

6. The method of claim 1, wherein the virtual machine configuration instructions comprise a virtual machine number, and wherein assigning a corresponding virtual machine to a target user in response to a virtual machine configuration instruction of the target user comprises:

7. The method according to any one of claims 1-6, further comprising:

8. A credible assessment method of a multi-party financial model is characterized by comprising the following steps:

responding to a virtual machine configuration instruction of a target user, and distributing a corresponding virtual machine for the target user; the virtual machine is provided with a monitoring component, the monitoring component is used for monitoring evaluation indexes of the virtual machine in a target time period, forming a monitoring log according to the detected evaluation indexes and transmitting the monitoring log to evaluation equipment according to preset interval duration, and the target time period comprises the following steps: analyzing the time period of the data to be analyzed stored in the virtual machine based on the corresponding code of the financial model provided by the target user when the corresponding virtual machine runs; obtaining an analysis result and an analysis duration corresponding to the financial model provided by the target user;

acquiring a monitoring log, and determining target evaluation information according to the monitoring log;

9. An apparatus for trusted evaluation of a multi-party model, the apparatus comprising:

10. An electronic device, comprising: a memory and at least one processor;

the memory is used for storing computer execution instructions;

the at least one processor is configured to execute computer-executable instructions stored in the memory to cause the at least one processor to perform the method of any of claims 1 to 8.