CN110334955B - Index evaluation processing method, device, equipment and storage medium - Google Patents

Index evaluation processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN110334955B
CN110334955B CN201910610384.1A CN201910610384A CN110334955B CN 110334955 B CN110334955 B CN 110334955B CN 201910610384 A CN201910610384 A CN 201910610384A CN 110334955 B CN110334955 B CN 110334955B
Authority
CN
China
Prior art keywords
data
evaluation
machine
identifier
index
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910610384.1A
Other languages
Chinese (zh)
Other versions
CN110334955A (en
Inventor
周兴
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing ByteDance Network Technology Co Ltd
Original Assignee
Beijing ByteDance Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing ByteDance Network Technology Co Ltd filed Critical Beijing ByteDance Network Technology Co Ltd
Priority to CN201910610384.1A priority Critical patent/CN110334955B/en
Publication of CN110334955A publication Critical patent/CN110334955A/en
Application granted granted Critical
Publication of CN110334955B publication Critical patent/CN110334955B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06QDATA PROCESSING SYSTEMS OR METHODS, SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/06Resources, workflows, human or project management, e.g. organising, planning, scheduling or allocating time, human or machine resources; Enterprise planning; Organisational models
    • G06Q10/063Operations research or analysis
    • G06Q10/0639Performance analysis
    • G06Q10/06393Score-carding, benchmarking or key performance indicator [KPI] analysis

Abstract

The embodiment of the disclosure provides a processing method, a device, equipment and a storage medium for index evaluation, wherein the method comprises the following steps: monitoring the data cached in the caching device in real time; acquiring identifiers of a plurality of evaluation machines corresponding to all data cached in the caching device within a preset period range of real-time monitoring; and aiming at the identifier of each of the plurality of the identifiers of the evaluation machines, if the number of the data corresponding to the identifier of the evaluation machine is less than or equal to the evaluation index processing upper limit of the evaluation machine, acquiring the data corresponding to the identifier of the evaluation machine, and sending the data corresponding to the identifier of the evaluation machine to the evaluation machine corresponding to the identifier of the evaluation machine, so that the evaluation machine performs index evaluation on the data corresponding to the identifier of the evaluation machine. The embodiment of the disclosure can solve the problem that the waiting time for receiving data by the evaluation machine cannot be effectively reduced in the prior art, and further the evaluation processing efficiency is reduced.

Description

Index evaluation processing method, device, equipment and storage medium
Technical Field
The embodiment of the disclosure relates to the technical field of data processing, and in particular, to a processing method, a processing device, processing equipment and a storage medium for index evaluation.
Background
In the present day of rapid development of digital information, the influence of data is increasingly strengthened, and particularly, the evaluation of each model is good or bad, so that the model index evaluation is required to be carried out on the model through each evaluation model or evaluation machine.
In the prior art, in a large-scale model index evaluation scene, data of each model is randomly distributed to each evaluation machine for model evaluation, so that each evaluation machine needs to wait for receiving a sufficient amount of model data to realize model evaluation, the time spent is long, and the efficiency is low.
Therefore, in the index evaluation process of one or more models, the waiting time for the evaluation machine to receive data cannot be effectively reduced, and the evaluation processing efficiency is further reduced.
Disclosure of Invention
The embodiment of the disclosure provides a processing method, a processing device, a processing apparatus and a storage medium for index evaluation, so as to overcome the problem that the prior art cannot effectively reduce the waiting time for an evaluation machine to receive data, thereby reducing the evaluation processing efficiency.
In a first aspect, an embodiment of the present disclosure provides a processing method for index evaluation, including:
monitoring data cached in a caching device in real time, wherein each piece of data comprises an identifier of an evaluation machine, the identifiers of the evaluation machines correspond to upper limit processing of evaluation indexes of the evaluation machines one by one, and the upper limit processing of the evaluation indexes is used for representing an upper limit value of the evaluation machines for receiving and processing data each time;
acquiring identifiers of a plurality of evaluation machines corresponding to all data cached in the caching device within a preset period range of real-time monitoring;
and aiming at the identifier of each of the plurality of the identifiers of the evaluation machines, if the number of the data corresponding to the identifier of the evaluation machine is less than or equal to the evaluation index processing upper limit of the evaluation machine, acquiring the data corresponding to the identifier of the evaluation machine, and sending the data corresponding to the identifier of the evaluation machine to the evaluation machine corresponding to the identifier of the evaluation machine, so that the evaluation machine performs index evaluation on the data corresponding to the identifier of the evaluation machine.
In a second aspect, an embodiment of the present disclosure provides an index evaluation processing apparatus, including:
the monitoring module is used for monitoring data cached in the caching device in real time, each piece of data comprises an identifier of an evaluation machine, the identifiers of the evaluation machines correspond to the upper limit of evaluation index processing of the evaluation machines one by one, and the upper limit of the evaluation index processing is used for representing the upper limit value of the evaluation machine for receiving and processing the data each time;
the identification determining module is used for acquiring the identifications of the plurality of evaluating machines corresponding to all the data cached in the caching device in a preset period range monitored in real time;
and the data processing module is used for acquiring the data corresponding to the identifier of the evaluation machine if the quantity of the data corresponding to the identifier of the evaluation machine is less than or equal to the evaluation index processing upper limit of the evaluation machine aiming at the identifier of each evaluation machine in the identifiers of the plurality of evaluation machines, and sending the data corresponding to the identifier of the evaluation machine to the evaluation machine corresponding to the identifier of the evaluation machine so that the evaluation machine can evaluate the indexes of the data corresponding to the identifier of the evaluation machine.
In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the method of index evaluation processing described above in the first aspect and in various possible designs of the first aspect.
In a fourth aspect, the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored, and when a processor executes the computer-executable instructions, the processing method for index evaluation is implemented as described in the first aspect and various possible designs of the first aspect.
The index evaluation processing method, apparatus, device, and storage medium provided by the embodiments of the present disclosure, first monitor data cached in a caching apparatus in real time, the caching apparatus continuously accumulates the cached data in time, where each piece of data includes an identifier of an evaluator, and the identifier of the evaluator corresponds to an upper limit of evaluation index processing of the evaluator one by one, the upper limit of evaluation index processing is used to represent an upper limit value of the evaluator for receiving and processing data each time, and then, within a preset period range of real-time monitoring, obtain identifiers of a plurality of evaluators corresponding to all data cached in the caching apparatus, so as to determine whether the number of data corresponding to the identifier of each evaluator exceeds the upper limit of evaluation index processing of the evaluator, and therefore, for the identifier of each evaluator in the identifiers of the plurality of evaluators, if the number of data corresponding to the identifier of the evaluator is less than or equal to the upper limit of evaluation index processing of the evaluator, and acquiring data corresponding to the identifier of the evaluation machine, and sending the data corresponding to the identifier of the evaluation machine to the evaluation machine corresponding to the identifier of the evaluation machine, so that the evaluation machine performs index evaluation on the data corresponding to the identifier of the evaluation machine, the waiting time for the evaluation machine to receive the data is effectively reduced, and the evaluation processing efficiency is further improved. According to the embodiment of the invention, the data cached in the caching device is monitored in real time, whether the cached data exceeds the processing upper limit of the evaluation index of the evaluation machine in the preset period range of real-time monitoring is judged, the phenomenon that the evaluation machine is overloaded due to the fact that a large amount of data is randomly distributed to the evaluation machine is avoided, the data which does not exceed the processing upper limit of the evaluation index of the evaluation machine is obtained for the identification of each evaluation machine, the obtained data corresponding to the identification of the evaluation machine is sent to the corresponding evaluation machine according to the identification of the evaluation machine, the purpose of directional concurrency can be achieved, the data waiting time of the evaluation machine is effectively reduced, and the evaluation processing efficiency is further improved.
Drawings
In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present disclosure, and for those skilled in the art, other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic flow chart of a processing method for index evaluation provided in an embodiment of the present disclosure;
fig. 2 is a schematic view of a scene of a processing method for index evaluation provided in the embodiment of the present disclosure;
fig. 3 is a schematic view of a scenario of a processing method for index evaluation according to another embodiment of the disclosure;
fig. 4 is a schematic view of a scenario of a processing method for index evaluation according to yet another embodiment of the disclosure;
FIG. 5 is a flowchart illustrating a processing method for index evaluation according to another embodiment of the disclosure;
FIG. 6 is a flowchart illustrating a processing method for index evaluation according to yet another embodiment of the disclosure;
fig. 7 is a block diagram of a processing apparatus for index evaluation provided in the embodiment of the present disclosure;
fig. 8 is a block diagram of a processing device for index evaluation according to another embodiment of the disclosure;
fig. 9 is a block diagram of a processing device for index evaluation according to still another embodiment of the disclosure;
FIG. 10 is an architecture diagram of a processing system for index evaluation provided by an embodiment of the present disclosure;
fig. 11 is a schematic diagram of a hardware structure of an electronic device according to the disclosed embodiment.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
In the prior art, in a large-scale model index evaluation scenario, data of each model is randomly allocated to an evaluator to perform model evaluation, for example, if there are 10 evaluators, model data sent by 1 model machine is randomly dispersed to 10 evaluators, each evaluator needs to wait for data to be able to perform model evaluation after enough model data is accumulated (assuming that each evaluator needs to accumulate 10 data to perform model index evaluation, if 10 data are sent per minute, each evaluator can only obtain 1 data in 1 minute and needs to wait for 10 minutes), that is, the evaluator needs to wait for receiving enough model data to perform model evaluation, which takes a long time and has low efficiency; if a large amount of data are sent to the same evaluation machine, the phenomenon of overload of the evaluation machine is easily caused. The embodiment of the present disclosure provides a processing method for index evaluation to solve the above problem.
Referring to fig. 1, fig. 1 is a schematic flow chart of a processing method for index evaluation according to an embodiment of the disclosure. The method of the embodiment of the present disclosure may be applied to a terminal device or a server, that is, the execution subject may be the terminal device or the server, which is not limited herein. The index evaluation processing method comprises the following steps:
s101, monitoring data cached in a caching device in real time, wherein each piece of data comprises an identifier of an evaluation machine, the identifier of the evaluation machine is in one-to-one correspondence with an evaluation index processing upper limit of the evaluation machine, and the evaluation index processing upper limit is used for representing an upper limit value of the evaluation machine for receiving and processing the data each time.
In this embodiment of the present disclosure, the data cached in the cache device is not limited, and the data cached in the cache device may be multiple pieces of data output by one model machine, and is used by the evaluation machine to evaluate one or more indexes of the model, or multiple pieces of data output by multiple models, and is used by the evaluation machine to evaluate the same performance index of all models, and determine which model is better or used by the evaluation machine to evaluate a certain performance index of a certain model, so that an implementation scenario of the present disclosure is: besides concurrent evaluation of a plurality of different models, the method can also be applied to a multi-index evaluation scene of one model.
In practical applications, in order to evaluate the quality of a model, the embodiment of the present disclosure evaluates the performance index of the model machine through the evaluation machine, and a local cache (local caching is performed through a cache device) is arranged between the model machine and the evaluation machine, and the local cache is provided with an upper limit, so that the local cache needs to be continuously monitored in real time.
The number of the model machines can be multiple, the number of the evaluation machines can be multiple, one evaluation machine can evaluate the quality of at least one index, and one evaluation machine is generally used for evaluating one index. The terminal device or the server monitors the data cached in the caching device in real time through the configured query rate per second, and the evaluator establishes a relationship with the model machine through the identifier, so that the data sent from the model machine to the caching device contains the identifier of the evaluator, that is, each piece of data contains the identifier of the evaluator, and the processing upper limit of one evaluator corresponding to one evaluation index is the upper limit value of the evaluator for receiving and processing the data each time.
S102, obtaining the identifications of the plurality of evaluation machines corresponding to all the data cached in the caching device in a preset period range of real-time monitoring.
In the embodiment of the disclosure, when monitoring cached data in real time, a terminal device or a server continuously queries identifiers of evaluation machines in each piece of data, and counts identifiers of multiple evaluation machines corresponding to all data cached in a caching device once within a preset period range of real-time monitoring, for example, within one minute of real-time monitoring, that is, within one minute as a period, the purpose of counting the identifiers of the evaluation machines is to determine whether an upper limit of processing of an evaluation index of a corresponding evaluation machine is reached in all data cached in the one minute, so as to avoid a breakdown of one or some evaluation machines when a waiting time of the evaluation machines is too long or cached data is respectively and concurrently sent to the corresponding evaluation machines.
S103, aiming at the identifier of each of the plurality of the identifiers of the evaluating machines, if the number of the data corresponding to the identifier of the evaluating machine is less than or equal to the evaluation index processing upper limit of the evaluating machine, the data corresponding to the identifier of the evaluating machine is obtained, and the data corresponding to the identifier of the evaluating machine is sent to the evaluating machine corresponding to the identifier of the evaluating machine, so that the evaluating machine carries out index evaluation on the data corresponding to the identifier of the evaluating machine.
In the embodiment of the disclosure, in order to avoid a phenomenon that overload occurs to an evaluation machine due to random distribution of a large amount of data to the evaluation machine, each evaluation machine is provided with an upper limit value for receiving and processing data each time, therefore, when monitoring data cached in a cache device, monitoring is performed within a preset period, whether data containing an identifier of the same evaluation machine has reached an evaluation index processing upper limit of a corresponding evaluation machine, if the amount of the data corresponding to the identifier of the evaluation machine is less than or equal to the evaluation index processing upper limit of the evaluation machine, data corresponding to the identifier of the evaluation machine is acquired, and the data corresponding to the identifier of the evaluation machine is directly sent to the evaluation machine corresponding to the identifier of the evaluation machine, so that the evaluation machine performs index evaluation on the data corresponding to the identifier of the evaluation machine.
For example, in the prior art, if there are 10 evaluation machines, model data sent by 1 model machine may be randomly dispersed to 10 evaluation machines, and each evaluation machine needs to wait for data, and then can realize model evaluation after enough model data is accumulated (assuming that each evaluation machine needs to accumulate 10 data to perform index evaluation of the model, and if 10 data are sent every minute, each evaluation machine can only obtain 1 data in 1 minute, and needs to wait for 10 minutes). In contrast, in the embodiment of the present disclosure, 10 evaluation machines correspond to 10 model machines, and then the model data sent by one model machine corresponds to one evaluation machine, it is still assumed that 10 data are needed to realize evaluation, and 10 data are sent every minute, and it only needs 1min to obtain enough model data and realize model evaluation.
Another implementation manner of acquiring, by a terminal device or a server, data corresponding to the identifier of the evaluation machine and sending the data corresponding to the identifier of the evaluation machine to the evaluation machine corresponding to the identifier of the evaluation machine may be: and the terminal equipment or the server controls the cache device to send the data corresponding to the identifier of the evaluation machine to the evaluation machine corresponding to the identifier of the evaluation machine.
In an embodiment of the present disclosure, each piece of data further includes: the method comprises the steps of obtaining a predicted value and a real value, wherein the predicted value is data output by a model machine, the real value is an actual value which is not processed by the model machine, the predicted value and the real value are in one-to-one correspondence, and the model machine corresponds to at least one piece of data; the data cached in the caching device is sent to the caching device for caching in real time through at least one model machine; wherein each evaluation machine is used for evaluating the performance index of the corresponding model machine.
In practical application, scenario one: scenario for realizing concurrency evaluation by multiple different models
Referring to fig. 2, fig. 2 is a scene schematic diagram of a processing method for index evaluation provided by the embodiment of the disclosure. For the same index evaluation scenario of a plurality of different model machines, a plurality of evaluation machines may be provided in the same number as the model machines, where each evaluation machine is configured identically, and is used to evaluate the model index of the same type of data output by the model machine, for example, there are n ═ 3 model machines (model machine 1, model machine 2, model machine 3) and m ═ 3 evaluation machines (evaluation machine 1, evaluation machine 2, evaluation machine 3), the model machines and the evaluation machines correspond one-to-one, that is, model machine 1 is evaluated by evaluation machine 1, model machine 2 is evaluated by evaluation machine 2, model machine 3 is evaluated by evaluation machine 3, and the evaluation index processing upper limit of each evaluation machine is the same.
Specifically, after some original data are input into a model machine, a predicted value can be obtained, the model machine sends the predicted value and a corresponding true value as a piece of data to a cache device, a terminal device or a server monitors the data cached in the cache device in real time, wherein each piece of data also carries an identifier of an evaluator corresponding to the model machine, for example, the data output by the model machine 1 carries the identifier 1 of the evaluator 1, then, through a query rate per second, when the terminal device or the server monitors the cache device in real time for a preset period (for example, one minute), identifiers of all evaluators (for example, the identifier 1 of the evaluator 1, the identifier 2 of the evaluator 2, and the identifier 3 of the evaluator 3) corresponding to all data cached in the cache device need to be queried, and then, for the identifier of each evaluator, it is determined that the number of the data corresponding to the identifier of the evaluator is less than or equal to an upper evaluation index processing limit of the evaluator (for example, at a minute point) of each evaluator How much data) is obtained, namely whether the quantity of the data corresponding to the identifier of the evaluator is within the upper limit of the evaluation index processing of the evaluator, if the quantity of the data corresponding to the identifier of the evaluator is less than or equal to the upper limit of the evaluation index processing of the evaluator, the data corresponding to the identifier of the evaluator is obtained, namely, if the quantity of the data corresponding to the identifiers of 3 evaluators is within the upper limit of the evaluation index processing of the corresponding evaluator, the data corresponding to the identifier 1 of the evaluator 1, the data corresponding to the identifier 2 of the evaluator 2, the data corresponding to the identifier 3 of the evaluator 3 are obtained, meanwhile, the data corresponding to the identifier 1 of the evaluator 1 is distributed to the evaluator 1 for the index evaluation of the model machine 1, the data corresponding to the identifier 2 of the evaluator 2 is distributed to the evaluator 2 for the index evaluation of the model machine 2, and the data corresponding to the identifier 3 of the evaluator 3 is distributed to the evaluator 3 for the index evaluation of the model machine 3, each evaluation machine directly carries out model evaluation after receiving the corresponding model data, so that the time for waiting for receiving the data of each evaluation machine is saved, the evaluation processing efficiency is improved, and meanwhile, the breakdown of the evaluation machine due to overload operation of the evaluation machine caused by concurrent exceeding of the processing upper limit of the evaluation machine is avoided.
Scene two: multi-index evaluation scene of model
Referring to fig. 3, fig. 3 is a scene schematic diagram of a processing method for index evaluation according to another embodiment of the present disclosure. When a plurality of indexes of a model machine are evaluated according to different index evaluation scenes of the model machine, in order to realize concurrent processing, a plurality of evaluation machines which are as many as the plurality of indexes of the model machine can be arranged, wherein each evaluation machine is configured according to the corresponding evaluation index, the configuration comprises an evaluation index processing upper limit of the evaluation machine, and the evaluation index processing upper limit of the evaluation machine can be configured in a cache device. Each evaluation machine is used for evaluating model indexes of the same type of data output by the model machine, for example, n is 1 model machine (model machine 1), three indexes (index 1, index 2 and index 3) of the model machine 1 are evaluated, there are provided 3 estimators (estimator 1, estimator 2, and estimator 3), each index corresponding to one estimator, that is, the index 1 of the model machine 1 is evaluated by the evaluation machine 1, that is, the data of the index 1 of the model machine 1 is distributed to the evaluation machine 1 (the data of the index 1 of the model machine 1 shown in fig. 2 is the data corresponding to the identifier 1 of the evaluation machine 1, and the same applies to the other things), the index 2 of the model machine 1 is evaluated by the evaluation machine 2, the index 3 of the model machine 1 is evaluated by the evaluation machine 3, the upper limit of the evaluation index processing of each evaluation machine may be different, and the upper limit of the evaluation index processing of each evaluation machine may be specifically configured according to the number of the data of the actual evaluation index.
Specifically, after some original data are input into the model machine, a predicted value can be obtained, the model machine sends the predicted value and a corresponding true value as one piece of data to the cache device, and the terminal device or the server monitors the data cached in the cache device in real time, wherein one model machine can output a plurality of pieces of data, the plurality of pieces of data can be used for providing a plurality of evaluators to evaluate a plurality of performance indexes of the model, each piece of data further carries an identifier of the evaluator corresponding to the model machine, that is, the piece of data is used for evaluating a certain index of the model by the evaluator, for example, the data output by the model machine 1 carries the identifier 1 of the evaluator 1, which indicates that the data needs to be distributed to the evaluator 1. Then, through the query rate per second, when the terminal device or the server monitors the cache device in real time for a preset period (e.g., one minute), the identifiers of all evaluators corresponding to all data cached in the cache device (e.g., identifier 1 of evaluator 1, identifier 2 of evaluator 2, identifier 3 of evaluator 3) need to be queried, then, for each identifier of evaluator, it is determined whether the number of data corresponding to the identifier of the evaluator is less than or equal to the upper limit of evaluation index processing of the evaluator (e.g., how many data are processed per minute), that is, whether the number of data corresponding to the identifier of the evaluator is within the upper limit of evaluation index processing of the evaluator, and if the number of data corresponding to the identifier of the evaluator is less than or equal to the upper limit of evaluation index processing of the evaluator, the data corresponding to the identifier of the evaluator is obtained, that is, if the number of data corresponding to the identifier of each of the 3 evaluating machines is within the upper limit of the corresponding evaluation index processing of the evaluating machine, the data corresponding to the identifier 1 of the evaluating machine 1 (i.e. the data of the identifier 1 of the model machine 1 shown in fig. 3, the same applies to the other), the data corresponding to the identifier 2 of the evaluating machine 2, and the data corresponding to the identifier 3 of the evaluating machine 3 are obtained, and meanwhile, the data corresponding to the identifier 1 of the evaluating machine 1 are distributed to the evaluating machine 1 for evaluation of the identifier 1 of the model machine 1, the data corresponding to the identifier 2 of the evaluating machine 2 are distributed to the evaluating machine 2 for evaluation of the identifier 2 of the model machine 1, the data corresponding to the identifier 3 of the evaluating machine 3 are distributed to the evaluating machine 3 for evaluation of the identifier 3 of the model machine 1, and each evaluating machine directly performs model evaluation after receiving the corresponding model data, so as to save the time length of data waiting for receiving of each evaluating machine, the evaluation processing efficiency is improved, and meanwhile, the fault that the evaluation machine is broken down due to overload operation of the evaluation machine caused by concurrent exceeding of the processing upper limit of the evaluation machine is avoided.
Scene three: scene for realizing concurrent evaluation of multiple indexes by multiple different models
Referring to fig. 4, fig. 4 is a scene schematic diagram of a processing method for index evaluation according to still another embodiment of the disclosure. For different index evaluation scenes of a plurality of model machines, when a plurality of indexes of the plurality of model machines are evaluated, in order to realize concurrent processing, a plurality of evaluation machines (the number of the plurality of model machines and the number of the plurality of indexes) with the same number as the plurality of indexes of the plurality of model machines can be arranged, so that one model machine corresponds to a plurality of evaluation machines (for example, three indexes of one model machine are evaluated, one model machine corresponds to three evaluation machines, and a model machine 1 corresponds to an evaluation machine 11, an evaluation machine 12 and an evaluation machine 13, wherein one evaluation machine is used for evaluating the quality of one index of one model or the quality of the same index of the plurality of models, and when the quality of the same index of the plurality of models is evaluated, the same batch is input into the corresponding evaluation machine according to the data of one model), each evaluation machine is configured according to the corresponding evaluation index, the configuration comprises an evaluation index processing upper limit of the evaluation machine, the evaluation index processing upper limit of the evaluation machine can be configured in the cache device, for the evaluation of the same index of different models, a plurality of evaluation machines which are configured in the same way are arranged, and data distribution is carried out according to the identification of the evaluation machine carried by each piece of data. In the embodiment of the present disclosure, each of the evaluation machines is configured to evaluate the model indexes of the same type of data output by the model machine, for example, n ═ 2 model machines (model machine 1, model machine 2), and evaluate three indexes (index 1, index 2, index 3, i.e., m ═ 3) of the model machine 1 and three indexes (index 1, index 2, index 3) of the model machine 2, n ═ m ═ 6 evaluation machines (evaluation machine 11, evaluation machine 12, evaluation machine 13, evaluation machine 21, evaluation machine 22, and evaluation machine 23) are provided, each index of each model is one-to-one corresponding to an evaluation machine, i.e., index 1 of model machine 1 is evaluated by evaluation machine 11, i.e., data corresponding to data of index 1 of model machine 1 shown in fig. 4 (i.e., data corresponding to identifier 11 of the evaluation machine, and the same other things) is sent to evaluation machine 11, index 2 of model machine 1 is evaluated by evaluation machine 12, i.e., data of index 2 of model machine 1 is sent to evaluation machine 12, the index 3 of the model machine 1 is evaluated by the evaluation machine 13, the index 1 of the model machine 2 is evaluated by the evaluation machine 21, the index 2 of the model machine 2 is evaluated by the evaluation machine 22, and the index 3 of the model machine 2 is evaluated by the evaluation machine 23. The evaluation index processing upper limit of each evaluation machine may be different, and the evaluation index processing upper limit of each evaluation machine may be configured specifically according to the number of the data of the actual evaluation index.
Specifically, after some original data are input into the model machine, a predicted value may be obtained, the model machine sends the predicted value and a corresponding true value as one piece of data to the cache device, and the terminal device or the server monitors the data cached in the cache device in real time, where one model machine may output a plurality of pieces of data, the plurality of pieces of data may be used to provide a plurality of evaluation machines to evaluate a plurality of performance indexes of the model, and each piece of data further carries an identifier of the evaluation machine corresponding to the model machine, that is, the piece of data is used by the evaluation machine to evaluate a certain index of which model, for example, the data output by the model machine 1 carries the identifier 11 of the evaluation machine 11, which indicates that the data needs to be distributed to the evaluation machine 11. Then, through the query rate per second, when the terminal device or the server monitors the cache device in real time for a preset period (e.g., one minute), the identifiers of all evaluators corresponding to all data cached in the cache device (e.g., the identifier 11 of the evaluator 11, the identifier 12 of the evaluator 12, and the identifier 23 of the evaluator 23) need to be queried, then, for each identifier of the evaluator, it is determined whether the number of data corresponding to the identifier of the evaluator is less than or equal to the upper limit of processing of the evaluation index of the evaluator (e.g., how many data are processed per minute), that is, whether the number of data corresponding to the identifier of the evaluator is within the upper limit of processing of the evaluation index of the evaluator, and if the number of data corresponding to the identifier of the evaluator is less than or equal to the upper limit of processing of the evaluation index of the evaluator, the data corresponding to the identifier of the evaluator is obtained, that is, if the number of data corresponding to the identifier of 6 evaluators is within the upper limit of the evaluation index processing of the corresponding evaluator, the data corresponding to the identifier 11 of the evaluator 11, the data corresponding to the identifier 12 of the evaluator 12, the data corresponding to the identifier 13 of the evaluator 13, the data corresponding to the identifier 21 of the evaluator 21, the data corresponding to the identifier 22 of the evaluator 22, and the data corresponding to the identifier 23 of the evaluator 23 are acquired, and simultaneously, the data corresponding to the identifier 11 of the evaluator 11 is distributed to the evaluator 11 for the index 1 evaluation of the model machine 1, the data corresponding to the identifier 12 of the evaluator 12 is distributed to the evaluator 12 for the index 2 evaluation of the model machine 1, the data corresponding to the identifier 13 of the evaluator 13 is distributed to the evaluator 13 for the index 3 evaluation of the model machine 1, and the data corresponding to the identifier 21 of the evaluator 21 is distributed to the evaluator 21 for the index 1 evaluation of the model machine 2, and distributing the data corresponding to the identifier 22 of the evaluator 22 to the evaluator 22 for evaluation of the index 2 of the model machine 2, and distributing the data corresponding to the identifier 23 of the evaluator 23 to the evaluator 23 for evaluation of the index 3 of the model machine 2. Each evaluation machine directly carries out model evaluation after receiving the corresponding model data, so that the time for waiting for receiving the data of each evaluation machine is saved, the evaluation processing efficiency is improved, and meanwhile, the breakdown of the evaluation machine due to overload operation of the evaluation machine caused by concurrent exceeding of the processing upper limit of the evaluation machine is avoided.
The index evaluation processing method provided by the embodiment of the disclosure monitors data cached in a caching device in real time, the caching device continuously accumulates the cached data in time, wherein each piece of data includes an identifier of an evaluator, the identifiers of the evaluators correspond to upper evaluation index processing limits of the evaluators one by one, the upper evaluation index processing limits are used for representing upper limit values of the evaluators receiving and processing data each time, then within a preset period range of real-time monitoring, identifiers of a plurality of evaluators corresponding to all data cached in the caching device are obtained for judging whether the number of data corresponding to the identifier of each evaluator exceeds the upper evaluation index processing limit of the evaluator, so that, for each identifier of the evaluators, if the number of data corresponding to the identifier of the evaluator is less than or equal to the upper evaluation index processing limit of the evaluator, and acquiring data corresponding to the identifier of the evaluation machine, and sending the data corresponding to the identifier of the evaluation machine to the evaluation machine corresponding to the identifier of the evaluation machine, so that the evaluation machine performs index evaluation on the data corresponding to the identifier of the evaluation machine, the waiting time for the evaluation machine to receive the data is effectively reduced, and the evaluation processing efficiency is further improved.
According to the embodiment of the invention, the data cached in the caching device is monitored in real time, whether the cached data exceeds the processing upper limit of the evaluation index of the evaluation machine in the preset period range of real-time monitoring is judged, the phenomenon that the evaluation machine is overloaded due to the fact that a large amount of data is randomly distributed to the evaluation machine is avoided, the data which does not exceed the processing upper limit of the evaluation index of the evaluation machine is obtained for the identification of each evaluation machine, the obtained data corresponding to the identification of the evaluation machine is sent to the corresponding evaluation machine according to the identification of the evaluation machine, the purpose of directional concurrency can be achieved, the data waiting time of the evaluation machine is effectively reduced, and the evaluation processing efficiency is further improved.
In order to determine whether the number of data corresponding to the identifier of the evaluation machine reaches the upper limit of the evaluation index processing of the evaluation machine, in an embodiment of the present disclosure, another embodiment of the present disclosure provides a schematic flow chart of a processing method for index evaluation, and on the basis of the embodiment of fig. 1, after obtaining the identifiers of a plurality of evaluation machines corresponding to all data cached in the cache device in step S102, the embodiment of the present disclosure describes in detail the processing method for index evaluation.
After the obtaining of the identifiers of the plurality of evaluation machines corresponding to all the data cached in the caching device, the method further includes: and respectively counting the data containing the identifier of the same evaluator in all the data cached in the cache device according to the identifiers of the evaluators to obtain the number of the data corresponding to the identifier of each evaluator.
As can be seen from the above description, in the embodiment of the present disclosure, after the identifiers of the multiple evaluation machines corresponding to all the data cached in the cache device are obtained, the number of the data corresponding to the identifier of each evaluation machine is counted first, so as to determine whether the distribution condition is met and how to implement the distribution specifically according to the number of the data corresponding to the identifier of the evaluation machine and the index evaluation processing upper limit of the corresponding evaluation machine.
In the embodiment of the present disclosure, firstly, for the identifier of each of the plurality of identifiers of the evaluators, statistics is performed on data that contains the identifier of the same evaluator in all data cached in the cache device, for example, if all data cached in the cache device contains identifier 1 of the evaluator and identifier 2 of the evaluator, the number of all data that contain identifier 1 of the evaluator and the number of all data that contain identifier 2 of the evaluator are respectively counted from all data cached in the cache device, so as to determine whether and how to distribute the data.
For the identifier of each of the multiple evaluators, whether the evaluation exceeds the upper limit of the index evaluation processing of the evaluator, and in order to ensure that the data can be normally distributed to the corresponding evaluator when the upper limit of the index evaluation processing of the evaluator is exceeded, in an embodiment of the present disclosure, another embodiment of the present disclosure provides a flowchart of a processing method for index evaluation, on the basis of the above-described embodiment of the present disclosure, for example, on the basis of the embodiment of fig. 1, after acquiring the identifiers of the multiple evaluators corresponding to all the data cached in the caching device in step S102, the embodiment of the present disclosure describes in detail the processing method for index evaluation.
In the embodiment of the present disclosure, for each identifier of the multiple evaluators, if the number of data corresponding to the identifier of the evaluator is greater than the evaluation index processing upper limit of the evaluator, the data corresponding to the identifier of the evaluator in the cache device needs to be cleaned, and if unnecessary data is not deleted each time, the subsequent backlog is more, which may cause the storage space of the local cache to be occupied and unable to work.
In a specific implementation process, if the number of data corresponding to the identifier of the evaluation machine is greater than the evaluation index processing upper limit of the evaluation machine, determining target data of the evaluation machine corresponding to the identifier sent to the evaluation machine from the data corresponding to the identifier of the evaluation machine, where the target data is used for index evaluation of the target data by the evaluation machine. The identifier (key) of the evaluator in each piece of data can be used for indicating the evaluator to which the model data is sent, the model data is the data cached in the caching device, and the upper processing limit of the evaluation index is used for deciding whether to send the cached model data to the evaluator.
With reference to the foregoing disclosure, the method for determining whether to send the data cached in the caching device to the evaluation machine may be:
judging whether the number (S is assumed) of data (data) corresponding to the key reaches the upper limit (P is assumed) of index evaluation processing of the corresponding evaluation machine according to the cached key of the model data; if not, directly sending the S data to an evaluation machine corresponding to the key; if yes, determining Y target data in the S data, and sending the target data to an evaluation machine; wherein Y is not more than P. The implementation mode avoids the condition that the number of the data received by the evaluation machine exceeds the self processing upper limit (the evaluation index processing upper limit of the evaluation machine), so that the evaluation machine processes abnormity.
In practical application, for the model data corresponding to each key, Y data are determined as evaluation data in the S model data, and the rest S-Y residual data are directly deleted. The (practical application scheme) considers that data is sent in real time, and if unnecessary data is not deleted each time, the more backlog is carried out subsequently, the storage space of the local cache can be occupied and the local cache cannot work. In order to determine the target data sent to the evaluation machine corresponding to the identifier of the evaluation machine from the data corresponding to the identifier of the evaluation machine, the index evaluation processing method provides two implementation modes.
The first method is as follows: referring to fig. 5, fig. 5 is a schematic flow chart of a processing method for index evaluation according to another embodiment of the present disclosure, where in the embodiment of the present disclosure, target data of an evaluation machine corresponding to an identifier sent to the evaluation machine is determined from data corresponding to an identifier of the evaluation machine and is described in detail. Wherein each piece of data further comprises the cache time of each piece of data; the determining, from the data corresponding to the identifier of the evaluation machine, the target data sent to the evaluation machine corresponding to the identifier of the evaluation machine includes:
s501, sequencing all data corresponding to the identifier of the evaluation machine in time sequence according to the cache time of each piece of data;
s502, selecting data with a number corresponding to the upper evaluation index processing limit of the evaluation machine from all the data corresponding to the sorted identification of the evaluation machine as the target data, and deleting the residual data except the target data from all the data corresponding to the identification of the evaluation machine from the cache device.
In this disclosure, Y data received first may be regarded as evaluation data as target data according to a receiving order of data received by a cache device, and specifically, a cache time may be recorded for each piece of data according to the receiving order of the data received by the cache device, where the cache time may be understood as a cache time corresponding to the data received by the cache device, or may be understood as a total cache time corresponding to the data received from the time of receiving the data until the data is counted before distribution, and all data corresponding to an identifier of an evaluation machine may be sorted in a time sequence according to the cache time of each piece of data, where the time sequence refers to a sequence in which a model machine caches the data in the cache device, and after all data corresponding to the identifier of the evaluation machine is sorted for each identifier of the evaluation machines, an evaluation index of the evaluation machine is selected from all data corresponding to the identifier of the evaluation machine after sorting to process an upper limit of processing And taking the corresponding number of data as the target data, for example, one S data, and if the upper limit of index evaluation processing of the evaluation machine is Y, selecting the first Y data from all the data corresponding to the sorted identifiers of the evaluation machine as evaluation data (target data), where Y is smaller than S. And then deleting the residual data except the Y data in the S from the cache device to ensure the normal cache of the cache device, simultaneously acquiring the target data and distributing the target data to the corresponding evaluation machine, so that the evaluation machine performs index evaluation on the corresponding model according to the target data, the waiting time for the evaluation machine to receive the data is effectively reduced, and the evaluation processing efficiency is further improved.
The second method comprises the following steps: referring to fig. 6, fig. 6 is a schematic flow chart of a processing method for index evaluation according to still another embodiment of the present disclosure, where in the embodiment of the present disclosure, target data of an evaluation machine corresponding to an identifier sent to the evaluation machine is determined from data corresponding to an identifier of the evaluation machine and is described in detail. The determining, from the data corresponding to the identifier of the evaluation machine, the target data sent to the evaluation machine corresponding to the identifier of the evaluation machine includes:
s601, carrying out predefined sampling on all data corresponding to the identifier of the evaluation machine to obtain data of a number corresponding to the upper limit of evaluation index processing of the evaluation machine;
s602, taking the data with the number corresponding to the upper limit of the evaluation index processing of the evaluation machine as the target data, and deleting the residual data except the target data in all the data corresponding to the identifier of the evaluation machine from the cache device.
In the embodiment of the disclosure, when the number of the data corresponding to the identifier of the evaluation machine is greater than the upper limit of the evaluation index processing of the evaluation machine, in order to ensure the accuracy of the evaluation machine in evaluating the model, the data distributed to the evaluation machine may be randomly selected. Specifically, predefined sampling or other self-defining modes are performed on all data corresponding to the identifier of the evaluation machine, to obtain data of a number corresponding to the upper limit of evaluation index processing of the evaluation machine, for example, data corresponding to the identifier of one evaluation machine in the cache device is S data, the upper limit of index evaluation processing of the evaluation machine is Y, sampling may be performed at intervals or in a self-defining mode in S model data, to obtain Y data (Y is smaller than S), then the Y data is used as target data, and the remaining data except the Y data in S is deleted from the cache device, so as to ensure normal caching of the cache device. And meanwhile, the target data is acquired and distributed to the corresponding evaluation machine, so that the evaluation machine performs index evaluation on the corresponding model according to the target data, the waiting time for the evaluation machine to receive data is effectively reduced, and the evaluation processing efficiency is improved.
Fig. 7 is a block diagram of a processing apparatus for index evaluation provided in the embodiment of the present disclosure, which corresponds to the processing method for index evaluation in the embodiment disclosed above. For ease of illustration, only portions that are relevant to embodiments of the present disclosure are shown. Referring to fig. 7, the processing device 70 for index evaluation includes: a monitoring module 701, an identification determination module 702 and a data processing module 703; the monitoring module 701 is configured to monitor data cached in the caching device in real time, where each piece of data includes an identifier of an evaluator, the identifier of the evaluator corresponds to an upper limit of evaluation index processing of the evaluator one to one, and the upper limit of evaluation index processing is used to represent an upper limit value at which the evaluator receives and processes data each time; the identifier determining module 702 is configured to obtain identifiers of multiple evaluation machines corresponding to all data cached in the caching device within a preset period range of real-time monitoring; the data processing module 703 is configured to, for an identifier of each of the multiple identifiers of the evaluation machine, if the number of data corresponding to the identifier of the evaluation machine is less than or equal to an evaluation index processing upper limit of the evaluation machine, obtain data corresponding to the identifier of the evaluation machine, and send the data corresponding to the identifier of the evaluation machine to the evaluation machine corresponding to the identifier of the evaluation machine, so that the evaluation machine performs index evaluation on the data corresponding to the identifier of the evaluation machine.
The apparatus provided in the embodiment of the present disclosure may be used to implement the technical solutions of the above method embodiments, and the implementation principles and technical effects are similar, which are not described herein again in the embodiment of the present disclosure.
Referring to fig. 8, fig. 8 is a block diagram illustrating a structure of a processing device for index evaluation according to another embodiment of the disclosure. The embodiment of the present disclosure details the device on the basis of the above-described embodiment, for example, on the basis of the embodiment of fig. 7. The device further comprises: a data statistics module 704; the data statistics module 704 is configured to, after obtaining the identifiers of the multiple evaluation machines corresponding to all the data cached in the cache device, respectively perform statistics on the data, which contains the identifier of the same evaluation machine, in all the data cached in the cache device according to the identifiers of the multiple evaluation machines to obtain the number of the data corresponding to the identifier of each evaluation machine.
Referring to fig. 9, fig. 9 is a block diagram of a processing device for index evaluation according to still another embodiment of the disclosure. The embodiment of the present disclosure details the device on the basis of the above-described embodiment, for example, on the basis of the embodiment of fig. 7. The device further comprises: target data determination module 705; the target data determining module 705 is configured to, after the identifiers of the multiple evaluation machines corresponding to all the data cached in the caching device are obtained, determine, for each identifier of the multiple evaluation machines, target data sent to the evaluation machine corresponding to the identifier of the evaluation machine from the data corresponding to the identifier of the evaluation machine if the number of the data corresponding to the identifier of the evaluation machine is greater than an evaluation index processing upper limit of the evaluation machine, so that the evaluation machine performs index evaluation on the target data.
In an embodiment of the present disclosure, on the basis of the above-described embodiment, for example, on the basis of the disclosed embodiment shown in fig. 9, each piece of data further includes a cache time of each piece of data; the target data determining module 705 is specifically configured to: sequencing all data corresponding to the identifier of the evaluation machine in time sequence according to the cache time of each piece of data; and selecting data with the number corresponding to the upper evaluation index processing limit of the evaluation machine from all the data corresponding to the sorted identification of the evaluation machine as the target data, and deleting the residual data except the target data from all the data corresponding to the identification of the evaluation machine from the cache device.
In an embodiment of the present disclosure, on the basis of the above-mentioned embodiment, for example, on the basis of the disclosed embodiment shown in fig. 9, the target data determining module 705 is specifically configured to:
carrying out predefined sampling on all data corresponding to the identifier of the evaluation machine to obtain data of a number corresponding to the upper processing limit of the evaluation index of the evaluation machine; and taking the data with the number corresponding to the upper processing limit of the evaluation index of the evaluation machine as the target data, and deleting the residual data except the target data in all the data corresponding to the identifier of the evaluation machine from the cache device.
In an embodiment of the present disclosure, on the basis of the above embodiment, each piece of data further includes: the method comprises the steps of obtaining a predicted value and a real value, wherein the predicted value is data output by a model machine, the real value is an actual value which is not processed by the model machine, the predicted value and the real value are in one-to-one correspondence, and the model machine corresponds to at least one piece of data; the data cached in the caching device is sent to the caching device for caching in real time through at least one model machine; wherein each evaluation machine is used for evaluating the performance index of the corresponding model machine.
In practical applications, referring to fig. 10, the embodiment of the present disclosure provides a processing system for index evaluation, which includes a processing device 70 for index evaluation, as described in the above disclosed embodiment, a user terminal 10, at least one model machine 20, at least one evaluation machine 30, and a caching device 40, where the processing device for index evaluation may be a terminal device or a server. The terminal may be a mobile terminal, a fixed terminal, an electronic device, or a test system, and the index evaluation processing system 100 may be configured to implement a processing procedure of index evaluation.
The embodiment of the present disclosure is not limited to the type of the model, the algorithm of the model, the evaluation algorithm of the evaluation machine, and the like. With regard to the above-mentioned "every time" or "times", a preset local cache (cache device) transmission cycle may be used in an actual scenario. For example, if the local cache is controlled to send data to the evaluator every 0.5 minute, the local cache receives the cached data in each period, and the above determination is performed before sending, and Y pieces of evaluation data are sent to the evaluator.
After a user inputs some original data into at least one model machine 20, the at least one model machine 20 transmits the output data to a cache device 40, a terminal device or a server performs global monitoring on the cache device, and once it is monitored that data corresponding to the identifier of each corresponding evaluation machine in the data cached in the cache device meets the upper limit of index evaluation processing of the evaluation machine within a preset period range of real-time monitoring, selected evaluation data is transmitted to the corresponding evaluation machine for index evaluation, wherein the implementation manner of transmitting the selected evaluation data to the corresponding evaluation machine for index evaluation may be: the terminal device or the server obtains the target data and sends the target data to the corresponding evaluation machine for index evaluation, and the method may also include: the terminal equipment or the server controls the cache device to send the target data to the corresponding evaluation machine for index evaluation, so that the waiting time for the evaluation machine to receive the data can be reduced, and high-efficiency evaluation processing is realized.
Specifically, after some raw data is input into at least one model machine 20, an estimated value can be obtained, and each model machine 20 sends the estimated value and the corresponding true value as one piece of data to a cache device, wherein each model machine 20 can output a plurality of pieces of data. A terminal device or a server (processing device 70 for index evaluation) monitors data cached in the caching device in real time, where a model machine may output multiple pieces of data, where the multiple pieces of data may be used to provide multiple evaluation machines to evaluate multiple performance indexes of the model, and each piece of data further carries an identifier of the evaluation machine corresponding to the model machine, that is, the piece of data is used by the evaluation machine to evaluate a certain index of which model, for example, the data output by the model machine 1 carries the identifier 11 of the evaluation machine 11, which indicates that the data needs to be distributed to the evaluation machine 11. Then, through the query rate per second, when the terminal device or the server monitors the cache device in real time for a preset period (for example, one minute), the identifiers of all the evaluation machines (for example, the identifier 11 of the evaluation machine 11, the identifier 12 of the evaluation machine 12, and the identifier 23 of the evaluation machine 23) corresponding to all the data cached in the cache device need to be queried, and then, for each identifier of the evaluation machine, it is determined whether the number of the data corresponding to the identifier of the evaluation machine is less than or equal to the evaluation index processing upper limit of the evaluation machine (for example, how many data are processed per minute), that is, whether the number of the data corresponding to the identifier of the evaluation machine is within the evaluation index processing upper limit of the evaluation machine.
If the number of the data corresponding to the identifier of the evaluator is less than or equal to the evaluation index processing upper limit of the evaluator, acquiring the data corresponding to the identifier of the evaluator, that is, if the number of the data corresponding to the identifiers of 6 evaluators is within the evaluation index processing upper limit of the corresponding evaluator, acquiring the data corresponding to the identifier 11 of the evaluator 11, the data corresponding to the identifier 12 of the evaluator 12, the data corresponding to the identifier 13 of the evaluator 13, the data corresponding to the identifier 21 of the evaluator 21, the data corresponding to the identifier 22 of the evaluator 22, the data corresponding to the identifier 23 of the evaluator 23, simultaneously distributing the data corresponding to the identifier 11 of the evaluator 11 to the evaluator 11 for evaluation of the indicator 1 of the model machine 1, distributing the data corresponding to the identifier 12 of the evaluator 12 to the evaluator 12 for evaluation of the indicator 2 of the model machine 1, and distributing the data corresponding to the identifier 13 of the evaluator 13 to the evaluator 13 for evaluation of the indicator 3 of the model machine 1, and distributing the data corresponding to the identifier 21 of the estimator 21 to the estimator 21 for the index 1 evaluation of the model machine 2, distributing the data corresponding to the identifier 22 of the estimator 22 to the estimator 22 for the index 2 evaluation of the model machine 2, and distributing the data corresponding to the identifier 23 of the estimator 23 to the estimator 23 for the index 3 evaluation of the model machine 2. Or the terminal equipment or the server controls the cache device to send the data corresponding to the identifier of the evaluation machine to the evaluation machine corresponding to the identifier of the evaluation machine. Each evaluation machine directly carries out model evaluation after receiving the corresponding model data, so that the time for waiting for receiving the data of each evaluation machine is saved, the evaluation processing efficiency is improved, and meanwhile, the breakdown of the evaluation machine due to overload operation of the evaluation machine caused by concurrent exceeding of the processing upper limit of the evaluation machine is avoided.
If the number of the data corresponding to the identifier of the evaluation machine is greater than the evaluation index processing upper limit of the evaluation machine, the terminal device or the server determines the target data sent to the evaluation machine corresponding to the identifier of the evaluation machine from the data corresponding to the identifier of the evaluation machine, so that the evaluation machine performs index evaluation on the target data, and the specific implementation manner may be: in a first manner, each piece of data further includes a caching time of each piece of data, for example, in the data cached in the caching device, the identifier of one evaluation machine corresponds to S pieces of data, the upper limit of index evaluation processing of the evaluation machine is Y, then the top Y pieces of data are selected as evaluation data (target data) from all the data corresponding to the identifier of the sorted evaluation machine, where Y is smaller than S. And then deleting the residual data except the Y data in the S from the cache device to ensure the normal cache of the cache device, simultaneously acquiring the target data and distributing the target data to the corresponding evaluation machine, so that the evaluation machine performs index evaluation on the corresponding model according to the target data, the waiting time for the evaluation machine to receive the data is effectively reduced, and the evaluation processing efficiency is further improved. In a second mode, predefined sampling or other self-defining modes are performed on all data corresponding to the identifier of the evaluator, to obtain data of a number corresponding to the upper limit of evaluation index processing of the evaluator, for example, data corresponding to the identifier of one evaluator in the cache device is S data, the upper limit of index evaluation processing of the evaluator is Y data, sampling can be performed at intervals or in a self-defining mode in S model data, to obtain Y data (Y is smaller than S), then the Y data is used as target data, and the remaining data in S except the Y data is deleted from the cache device, so as to ensure normal caching of the cache device. And meanwhile, the target data is acquired and distributed to the corresponding evaluation machine, so that the evaluation machine performs index evaluation on the corresponding model according to the target data, the waiting time for the evaluation machine to receive data is effectively reduced, and the evaluation processing efficiency is improved.
Referring to fig. 11, which shows a schematic structural diagram of an electronic device 1100 suitable for implementing the embodiment of the present disclosure, the electronic device 1100 may be a terminal device or a server. Among them, the terminal Device may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a Digital broadcast receiver, a Personal Digital Assistant (PDA), a tablet computer (PAD), a Portable Multimedia Player (PMP), a car terminal (e.g., car navigation terminal), etc., and a fixed terminal such as a Digital TV, a desktop computer, etc. The electronic device shown in fig. 11 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 11, the electronic device 1100 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 1101, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 1102 or a program loaded from a storage means 1108 into a Random Access Memory (RAM) 1103. In the RAM 1103, various programs and data necessary for the operation of the electronic device 1100 are also stored. The processing device 1101, the ROM 1102, and the RAM 1103 are connected to each other by a bus 1104. An input/output (I/O) interface 1105 is also connected to bus 1104.
Generally, the following devices may be connected to the I/O interface 1105: input devices 1106 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 1107 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage devices 1108, including, for example, magnetic tape, hard disk, etc.; and a communication device 1109. The communication means 1109 may allow the electronic device 1100 to communicate wirelessly or wiredly with other devices to exchange data. While fig. 11 illustrates an electronic device 1100 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via the communication device 1109, or installed from the storage device 1108, or installed from the ROM 1102. The computer program, when executed by the processing device 1101, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.
Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of Network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present disclosure, and not for limiting the same; while the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present disclosure.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present disclosure, and not for limiting the same; while the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present disclosure.

Claims (12)

1. A processing method for index evaluation is characterized by comprising the following steps:
monitoring data cached in a caching device in real time, wherein each piece of data comprises an identifier of an evaluation machine, the identifiers of the evaluation machines correspond to upper limit processing of evaluation indexes of the evaluation machines one by one, and the upper limit processing of the evaluation indexes is used for representing an upper limit value of the evaluation machines for receiving and processing data each time;
acquiring identifiers of a plurality of evaluation machines corresponding to all data cached in the caching device within a preset period range of real-time monitoring;
for the identifier of each of the plurality of evaluators, if the number of data corresponding to the identifier of the evaluator is less than or equal to an evaluation index processing upper limit of the evaluator, acquiring data corresponding to the identifier of the evaluator, and sending the data corresponding to the identifier of the evaluator to the evaluator corresponding to the identifier of the evaluator, so that the evaluator evaluates the index of the data corresponding to the identifier of the evaluator;
wherein each piece of data further comprises: the method comprises the steps of obtaining a predicted value and a real value, wherein the predicted value is data output by a model machine, the real value is an actual value which is not processed by the model machine, the predicted value and the real value are in one-to-one correspondence, and the model machine corresponds to at least one piece of data;
the data cached in the caching device is sent to the caching device for caching in real time through at least one model machine;
wherein each evaluation machine is used for evaluating the performance index of the corresponding model machine.
2. The method of claim 1, wherein after obtaining the identities of the plurality of evaluation machines corresponding to all of the data cached in the caching apparatus, the method further comprises:
and respectively counting the data containing the identifier of the same evaluator in all the data cached in the cache device according to the identifiers of the evaluators to obtain the number of the data corresponding to the identifier of each evaluator.
3. The method of claim 1, wherein after obtaining the identities of the plurality of evaluation machines corresponding to all of the data cached in the caching apparatus, the method further comprises:
and for each identifier of the plurality of evaluators, if the number of data corresponding to the identifier of the evaluator is greater than the evaluation index processing upper limit of the evaluator, determining target data sent to the evaluator corresponding to the identifier of the evaluator from the data corresponding to the identifier of the evaluator so that the evaluator evaluates the index of the target data.
4. The method of claim 3, wherein each piece of data further comprises a buffer time of each piece of data;
the determining, from the data corresponding to the identifier of the evaluation machine, the target data sent to the evaluation machine corresponding to the identifier of the evaluation machine includes:
sequencing all data corresponding to the identifier of the evaluation machine in time sequence according to the cache time of each piece of data;
and selecting data with the number corresponding to the upper evaluation index processing limit of the evaluation machine from all the data corresponding to the sorted identification of the evaluation machine as the target data, and deleting the residual data except the target data from all the data corresponding to the identification of the evaluation machine from the cache device.
5. The method of claim 3, wherein determining the data sent to the evaluator identifying the corresponding evaluator from the data identifying the corresponding evaluator comprises:
carrying out predefined sampling on all data corresponding to the identifier of the evaluation machine to obtain data of a number corresponding to the upper processing limit of the evaluation index of the evaluation machine;
and taking the data with the number corresponding to the upper processing limit of the evaluation index of the evaluation machine as the target data, and deleting the residual data except the target data in all the data corresponding to the identifier of the evaluation machine from the cache device.
6. A processing apparatus for index evaluation, comprising:
the monitoring module is used for monitoring data cached in the caching device in real time, each piece of data comprises an identifier of an evaluation machine, the identifiers of the evaluation machines correspond to the upper limit of evaluation index processing of the evaluation machines one by one, and the upper limit of the evaluation index processing is used for representing the upper limit value of the evaluation machine for receiving and processing the data each time;
the identification determining module is used for acquiring the identifications of the plurality of evaluating machines corresponding to all the data cached in the caching device in a preset period range monitored in real time;
the data processing module is used for acquiring data corresponding to the identifier of the evaluation machine if the number of the data corresponding to the identifier of the evaluation machine is smaller than or equal to an evaluation index processing upper limit of the evaluation machine aiming at the identifier of each evaluation machine in the identifiers of the plurality of evaluation machines, and sending the data corresponding to the identifier of the evaluation machine to the evaluation machine corresponding to the identifier of the evaluation machine so that the evaluation machine can evaluate the index of the data corresponding to the identifier of the evaluation machine;
each piece of data further includes: the method comprises the steps of obtaining a predicted value and a real value, wherein the predicted value is data output by a model machine, the real value is an actual value which is not processed by the model machine, the predicted value and the real value are in one-to-one correspondence, and the model machine corresponds to at least one piece of data;
the data cached in the caching device is sent to the caching device for caching in real time through at least one model machine;
wherein each evaluation machine is used for evaluating the performance index of the corresponding model machine.
7. The apparatus of claim 6, further comprising: a data statistics module;
and the data counting module is used for respectively counting the data containing the identifier of the same evaluator in all the data cached in the caching device according to the identifiers of the evaluators after the identifiers of the evaluators corresponding to all the data cached in the caching device are obtained, so as to obtain the number of the data corresponding to the identifier of each evaluator.
8. The apparatus of claim 6, further comprising: a target data determination module;
the target data determining module is configured to, after the identifiers of the multiple evaluation machines corresponding to all the data cached in the cache device are obtained, determine, for each identifier of the multiple evaluation machines, target data sent to the evaluation machine corresponding to the identifier of the evaluation machine from the data corresponding to the identifier of the evaluation machine if the number of the data corresponding to the identifier of the evaluation machine is greater than an evaluation index processing upper limit of the evaluation machine, so that the evaluation machine performs index evaluation on the target data.
9. The apparatus of claim 8, wherein each piece of data further comprises a buffer time of each piece of data;
the target data determination module is specifically configured to:
sequencing all data corresponding to the identifier of the evaluation machine in time sequence according to the cache time of each piece of data;
and selecting data with the number corresponding to the upper evaluation index processing limit of the evaluation machine from all the data corresponding to the sorted identification of the evaluation machine as the target data, and deleting the residual data except the target data from all the data corresponding to the identification of the evaluation machine from the cache device.
10. The apparatus of claim 8, wherein the target data determination module is specifically configured to:
carrying out predefined sampling on all data corresponding to the identifier of the evaluation machine to obtain data of a number corresponding to the upper processing limit of the evaluation index of the evaluation machine;
and taking the data with the number corresponding to the upper processing limit of the evaluation index of the evaluation machine as the target data, and deleting the residual data except the target data in all the data corresponding to the identifier of the evaluation machine from the cache device.
11. An electronic device, comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the method of index evaluation processing of any of claims 1 to 5.
12. A computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, implement the method of index evaluation according to any one of claims 1 to 5.
CN201910610384.1A 2019-07-08 2019-07-08 Index evaluation processing method, device, equipment and storage medium Active CN110334955B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910610384.1A CN110334955B (en) 2019-07-08 2019-07-08 Index evaluation processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910610384.1A CN110334955B (en) 2019-07-08 2019-07-08 Index evaluation processing method, device, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN110334955A CN110334955A (en) 2019-10-15
CN110334955B true CN110334955B (en) 2021-09-14

Family

ID=68143372

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910610384.1A Active CN110334955B (en) 2019-07-08 2019-07-08 Index evaluation processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN110334955B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109376844A (en) * 2018-10-30 2019-02-22 银河水滴科技(北京)有限公司 The automatic training method of neural network and device recommended based on cloud platform and model
CN109389143A (en) * 2018-06-19 2019-02-26 北京九章云极科技有限公司 A kind of Data Analysis Services system and method for automatic modeling
CN109670267A (en) * 2018-12-29 2019-04-23 北京航天数据股份有限公司 A kind of data processing method and device

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9888121B1 (en) * 2016-12-13 2018-02-06 Afiniti Europe Technologies Limited Techniques for behavioral pairing model evaluation in a contact center system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109389143A (en) * 2018-06-19 2019-02-26 北京九章云极科技有限公司 A kind of Data Analysis Services system and method for automatic modeling
CN109376844A (en) * 2018-10-30 2019-02-22 银河水滴科技(北京)有限公司 The automatic training method of neural network and device recommended based on cloud platform and model
CN109670267A (en) * 2018-12-29 2019-04-23 北京航天数据股份有限公司 A kind of data processing method and device

Also Published As

Publication number Publication date
CN110334955A (en) 2019-10-15

Similar Documents

Publication Publication Date Title
CN105468513B (en) Test method, device and system based on mobile terminal
US20160034504A1 (en) Efficient aggregation, storage and querying of large volume metrics
CN110781373A (en) List updating method and device, readable medium and electronic equipment
CN111246228A (en) Method, device, medium and electronic equipment for updating gift resources of live broadcast room
CN112019406B (en) Flow monitoring method and device, electronic equipment and computer readable medium
CN110334955B (en) Index evaluation processing method, device, equipment and storage medium
CN111405341A (en) Control display method and device and electronic equipment
CN110752958A (en) User behavior analysis method, device, equipment and storage medium
CN110727558A (en) Information prompting method and device, storage medium and electronic equipment
CN110636367A (en) Video loading method and device, terminal equipment and medium
CN110765354A (en) Information pushing method and device, electronic equipment and storage medium
CN109522133B (en) Data splicing method and device, electronic equipment and storage medium
CN108089823B (en) Information output method and device for storage system
CN111259174B (en) Multimedia transmission method, device, electronic equipment and medium
CN111291254A (en) Information processing method and device
US10281973B2 (en) Application power usage
CN112685271A (en) Pressure measurement data processing method and device, electronic equipment and readable storage medium
CN110856004A (en) Message processing method and device, readable storage medium and electronic equipment
CN112488412A (en) Duration information determination method and device, electronic equipment and computer storage medium
CN112543487A (en) Control method and device of electronic equipment, terminal and storage medium
CN111538717A (en) Data processing method and device, electronic equipment and computer readable medium
CN112379982A (en) Task processing method and device, electronic equipment and computer readable storage medium
CN111274104A (en) Data processing method and device, electronic equipment and computer readable storage medium
CN112712570A (en) Image processing method, image processing apparatus, electronic device, and medium
CN111343245A (en) Uploading line scheduling method and device, electronic equipment and readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant