CN111882347A - Model performance detection method, device, computer equipment and storage medium - Google Patents
Model performance detection method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111882347A CN111882347A CN202010597805.4A CN202010597805A CN111882347A CN 111882347 A CN111882347 A CN 111882347A CN 202010597805 A CN202010597805 A CN 202010597805A CN 111882347 A CN111882347 A CN 111882347A
- Authority
- CN
- China
- Prior art keywords
- information
- model
- information display
- display
- attribute value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0241—Advertisements
- G06Q30/0242—Determining effectiveness of advertisements
- G06Q30/0245—Surveys
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0241—Advertisements
- G06Q30/0277—Online advertisement
Landscapes
- Business, Economics & Management (AREA)
- Engineering & Computer Science (AREA)
- Accounting & Taxation (AREA)
- Development Economics (AREA)
- Strategic Management (AREA)
- Finance (AREA)
- Game Theory and Decision Science (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- Marketing (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The disclosure relates to a method and a device for detecting model performance, a computer device and a storage medium. The method comprises the following steps: acquiring various historical information display data stored in an information display system, wherein the historical information display data at least comprises information of information display positions; determining a plurality of sample data from each historical information display data; retrieving display information corresponding to the information display bits in each sample data through an information display system; judging whether an information combination consisting of the information display position and the retrieved display information exists in historical information display data or not; if the information combination exists in the historical information display data, acquiring interactive information corresponding to the information combination, wherein the interactive information is used for representing interactive operation between the account and the display information; and determining the performance detection result of the model to be detected based on the interaction information. The method and the device can improve the evaluation accuracy of the account feedback evaluation model.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method and an apparatus for detecting model performance, a computer device, and a storage medium.
Background
At present, programmed advertisement delivery has become a mainstream mode of internet advertisement delivery. An account feedback evaluation model based on machine learning technology is a core model for programmed advertising. The account feedback evaluation model can estimate the account feedback rate of the advertisement when the advertisement is delivered to an information display position, so as to determine the delivery of the advertisement according to the account feedback rate. In order to improve the accuracy of the account feedback evaluation model, the information presentation system will generally continuously train the account feedback evaluation model with new advertisement data to obtain a new account feedback evaluation model. Before the new account feedback evaluation model is deployed and brought online, whether the accuracy of the new account feedback evaluation model is higher than that of the old account feedback evaluation model needs to be detected, and if the performance is improved, the new model is brought online.
In a conventional evaluation scheme, historical information display data in an information display system is divided into two parts. One part is a training set used for training the account feedback evaluation model, and the other part is a testing set used for testing the account feedback evaluation model. After the training set and the test set are divided, the account feedback evaluation model can be trained through historical information display data in the training set. And after the training is finished, testing the account feedback evaluation model through the historical advertisements in the test set to obtain a test result. The accuracy of the account feedback evaluation model can then be evaluated based on the magnitude of the difference between the test results and the actual results.
However, in the conventional technology, a large amount of historical information display data is adopted to train and test the account feedback evaluation model, and the defect that the evaluation result is not accurate enough exists.
Disclosure of Invention
The present disclosure provides a method and an apparatus for detecting model performance, a computer device, and a storage medium, so as to solve the technical problem of low accuracy of model evaluation in the conventional technology. The technical scheme of the disclosure is as follows:
according to a first aspect of embodiments of the present disclosure, there is provided a method for detecting model performance, the method including:
acquiring various historical information display data stored in an information display system, wherein the historical information display data at least comprise information of information display positions;
determining a plurality of sample data from each historical information display data;
retrieving, by the information presentation system, presentation information corresponding to information presentation bits in each of the sample data, wherein a model to be tested of a performance to be tested is set in the information presentation system, and the model to be tested is used to perform one or more operations in the retrieval process;
judging whether an information combination consisting of the information display position and the retrieved display information exists in the historical information display data or not;
if the information combination exists in the historical information display data, acquiring interactive information corresponding to the information combination, wherein the interactive information is used for representing interactive operation between an account and the display information;
and determining the performance detection result of the model to be detected based on the interaction information.
As an optional implementation manner, the historical information display data further includes display information, and the retrieving, by the information display system, display information corresponding to the information display position in each sample data includes:
searching display information matched with the characteristics of each information display position from an information base through a plurality of search algorithms arranged in the information display system; and one or more retrieval algorithms are set in the model to be tested.
As an optional implementation manner, the obtaining of the interaction information corresponding to the information combination includes:
acquiring the performance detection dimension of the model to be detected;
reading attribute values of the performance detection dimensions corresponding to the information combinations;
and determining the average value of the attribute values of the information combinations as the attribute value of the interactive information.
As an optional implementation manner, the determining, based on the interaction information, a performance detection result of the model to be detected includes:
judging whether the attribute value of the interactive information exceeds a target attribute value or not;
if the attribute value of the interaction information exceeds the target attribute value, determining that the performance detection result of the model to be detected is qualified;
and if the attribute value of the interaction information does not exceed the target attribute value, determining that the performance detection result of the model to be detected is unqualified.
According to a second aspect of embodiments of the present disclosure, there is provided an apparatus for detecting performance of a model, the apparatus comprising:
the information display system comprises a first acquisition module, a second acquisition module and a display module, wherein the first acquisition module is configured to acquire various historical information display data stored in the information display system, and the historical information display data at least comprise information of information display bits;
the first determining module is configured to determine a plurality of sample data from each historical information display data;
a retrieval module configured to retrieve, by the information presentation system, presentation information corresponding to information presentation bits in each sample data, where a model to be tested of a performance to be tested is set in the information presentation system, and the model to be tested is used to perform one or more operations in the retrieval process;
a judging module configured to judge whether an information combination composed of the information display bit and the retrieved display information exists in the history information display data;
the second obtaining module is configured to obtain interactive information corresponding to the information combination if the information combination exists in the historical information display data, wherein the interactive information is used for representing interactive operation between an account and the display information;
and the second determination module is configured to determine a performance detection result of the model to be detected based on the interaction information.
As an optional implementation manner, the historical information display data further includes display information, and the retrieval module is specifically configured to:
searching display information matched with the characteristics of each information display position from an information base through a plurality of search algorithms arranged in the information display system; and one or more retrieval algorithms are set in the model to be tested.
As an optional implementation manner, the second obtaining module is specifically configured to:
acquiring the performance detection dimension of the model to be detected;
reading attribute values of the performance detection dimensions corresponding to the information combinations;
and determining the average value of the attribute values of the information combinations as the attribute value of the interactive information.
As an optional implementation manner, the second determining module is specifically configured to:
judging whether the attribute value of the interactive information exceeds a target attribute value or not;
if the attribute value of the interaction information exceeds the target attribute value, determining that the performance detection result of the model to be detected is qualified;
and if the attribute value of the interaction information does not exceed the target attribute value, determining that the performance detection result of the model to be detected is unqualified.
According to a third aspect of embodiments of the present disclosure, there is provided a computer device comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the method of detecting performance of a model according to any one of the first aspect.
According to a fourth aspect of embodiments of the present disclosure, there is provided a storage medium, wherein instructions that, when executed by a processor of a computer device, enable the computer device to perform the method of detecting model performance according to any one of the first aspect.
According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product comprising a computer program stored in a readable storage medium, from which at least one processor of an apparatus reads and executes the computer program, so that the apparatus performs the method of detecting model performance as described in any one of the embodiments of the first aspect.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:
the computer equipment acquires various historical information display data stored in the information display system. The historical information display data at least comprises information of information display bits. The computer equipment determines a plurality of sample data from each historical information display data, and retrieves the display information corresponding to the information display position in each sample data through the information display system. The information display system is provided with a model to be detected of the performance to be detected, and the model to be detected is used for executing one or more operations in the retrieval processing. Then, the computer device determines whether an information combination of the information display bit and the retrieved display information exists in the history information display data. And if the information combination exists in the historical information display data, the computer equipment acquires interactive information corresponding to the information combination, wherein the interactive information is used for representing interactive operation between the account and the display information, and the performance detection result of the model to be detected is determined based on the interactive information. Thus, the performance of the model is tested by on-line simulation of the model; meanwhile, in the process of testing the performance of the model, the historical data is screened and filtered through a system containing the model to be tested, the performance of the model to be tested is determined based on the test result brought by the data meeting the requirements, and the evaluation accuracy of the account feedback evaluation model can be improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.
FIG. 1 is a flow diagram illustrating a method of detecting performance of a model in accordance with an exemplary embodiment;
FIG. 2 is a flow diagram illustrating a method of detecting performance of a model in accordance with an exemplary embodiment;
FIG. 3 is a flow diagram illustrating a method of detecting performance of a model in accordance with an exemplary embodiment;
FIG. 4 is a block diagram illustrating a detection apparatus for model performance according to an exemplary embodiment;
FIG. 5 is a diagram illustrating an internal structure of a computer device, according to an example embodiment.
Detailed Description
In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
The embodiment of the disclosure provides a method for detecting model performance. The detection method of the model performance can be applied to an information display system to evaluate whether an account feedback evaluation model in the information display system meets a preset accuracy requirement.
FIG. 1 is a flow chart illustrating a method of detecting performance of a model, as shown in FIG. 1, according to an exemplary embodiment, the method including the following steps.
And step 110, acquiring various historical information display data stored in the information display system.
The historical information display data at least comprises information of information display bits.
In implementation, when the computer device needs to perform performance detection on a to-be-detected model of performance to be detected in the information presentation system, the computer device may obtain historical information presentation data within a preset time length (for example, 40 days) from the current time from the information presentation system. The historical information display data at least comprises information of the information display positions, and in the information display system, the display positions are advertisement display positions. Optionally, since the historical information display data is used for training the account feedback evaluation model, the historical information display data further includes account feedback accordingly. The account feedback may be account click feedback, account download feedback, or other types of account feedback, and the embodiment of the present disclosure is not limited.
Step 120, determining a plurality of sample data from the historical information display data.
In implementation, after the computer device obtains the historical information display data, a preset number of sample data (that is, a preset number of historical information display data) may be further selected from the historical information display data. Optionally, the computer device may randomly select a preset number of sample data from the historical information display data, may select a preset number of sample data from the historical information display data according to a preset selection rule, and may also select a preset number of sample data from the historical information display data by using other selection rules, which is not limited in the embodiment of the present disclosure.
For example, after the computer device obtains the history information display data, 3 sample data, which are sample data 1, sample data 2, and sample data 3, are further selected from the history information display data, where an information display bit in the sample data 1 is P1, an information display bit in the sample data 2 is P2, and an information display bit in the sample data 1 is P3.
Step 130, retrieving, by the information presentation system, presentation information corresponding to the information presentation bits in each sample data.
The information display system is provided with a model to be detected of the performance to be detected, and the model to be detected is used for executing one or more operations in the retrieval processing.
In implementation, after the computer device determines a plurality of sample data from each historical information display data, for each sample data, the computer device may retrieve, from each historical information display data, display information corresponding to an information display position in the sample data through a to-be-detected model of performance to be detected set in the information display system.
As an optional implementation manner, the historical information display data further includes display information. The specific processing process of the computer equipment for retrieving the display information corresponding to the information display position in each sample data through the information display system is as follows: and searching the display information matched with the characteristics of each information display position from the information base through a plurality of searching algorithms arranged in the information display system. Wherein, the model to be tested is provided with one or more retrieval algorithms.
In implementation, the historical information presentation data further includes presentation information. One or more retrieval algorithms are set in the model to be tested set in the information display system. After the computer equipment determines a plurality of sample data from each historical information display data, aiming at each sample data, the computer equipment can retrieve the display information corresponding to the information display position in the sample data from each historical information display data through one or more retrieval algorithms in the to-be-detected model with to-be-detected performance set in the information display system. Optionally, the historical information display data stored in the information display system may further include, in addition to the information display position and the display information, other types of information such as display time of the display information, account information, whether the bidding is successful, whether the display is performed, and the like, and the embodiment of the disclosure is not limited.
For example, the history information presentation data further includes presentation information a1, a2, and A3. The sample data selected by the computer device is sample data 1, sample data 2 and sample data 3, wherein the information display bit in the sample data 1 is P1, the information display bit in the sample data 2 is P2, and the information display bit in the sample data 3 is P3. For each sample data, the computer device may combine the presentation information included in the historical information presentation data (i.e., a1, a2, and A3) with the information presentation bits in the sample data, resulting in an information combination. For sample data 1, the obtained information sets are { P1, a1}, { P1, a2}, { P1, A3}, for sample data 2, the obtained information sets are { P2, a1}, { P2, a2}, { P2, A3}, and for sample data 3, the obtained information sets are { P3, a1}, { P3, a2}, { P3, A3 }. After the computer equipment obtains a plurality of information combinations corresponding to each information display position, aiming at each information display position, the computer equipment can input each information combination corresponding to the information display position into the to-be-detected model of the to-be-detected performance. Correspondingly, the model to be detected of the performance to be detected can output the account click probability of each information combination corresponding to the information display position. For example, the information display bit in the sample data 1 is the information display bit P1, the information combinations corresponding to the information display bit P1 are { P1, a1}, { P1, a2}, and { P1, A3}, respectively, and after the computer device inputs each information combination corresponding to the information display bit P1 (i.e., { P1, a1}, { P1, a2}, and { P1, A3}) into the to-be-tested model of the to-be-tested performance, the account hit probability of the information combination { P1, a1} corresponding to the information display bit P1, which is output by the to-be-tested model of the to-be-tested performance, is 0.6, the account hit probability of the information combination { P1, a2} is 0.8, and the account hit probability of the information combination { P1, A3} is 0.75. For each information display position, after obtaining the account click probability of each information combination corresponding to the information display position, the computer device may further determine the information combination with the highest account click probability among the information combinations corresponding to the information display position as the target information combination corresponding to the information display position. For example, the information presentation bits are information presentation bits P1, the information combinations corresponding to the information presentation bits are { P1, a1}, { P1, a2}, and { P1, A3}, the account click probabilities of the information combinations corresponding to the information presentation bits are 0.6, 0.8, and 0.75, respectively, and the target information combinations corresponding to the information presentation bits are { P1, a2 }.
Step 140, determine whether the information combination composed of the information display position and the retrieved display information exists in the historical information display data.
In implementation, after the computer device retrieves the display information corresponding to the information display bits in the sample data from each historical information display data, for each information display bit of the sample data, the computer device may further determine whether an information combination composed of the information display bit corresponding to the sample data and the retrieved display information exists in each historical information display data. If the information combination of the information display bit corresponding to the sample data and the retrieved display information exists in each history information display data, step 150 is executed.
And 150, if the information combination exists in the historical information display data, acquiring the interactive information corresponding to the information combination.
And the interactive information is used for representing the interactive operation of the account and the display information.
In implementation, if an information combination consisting of the information display bits corresponding to the sample data and the retrieved display information exists in each historical information display data, the computer device may further obtain the interactive information corresponding to the information combination. And the interactive information is used for representing the interactive operation of the account and the display information.
As an alternative implementation, as shown in fig. 2, in step 150, the process of acquiring the mutual information corresponding to the information combination by the computer device is as follows.
And 151, acquiring the performance detection dimension of the model to be detected.
In implementation, if an information combination composed of the information display bits corresponding to the sample data and the retrieved display information exists in each historical information display data, the computer device may further obtain the performance detection dimension of the model to be detected. The performance detection dimension may be an account click rate or an account download rate, or may be other detection dimensions, which is not limited in the embodiment of the present disclosure.
Step 152, reading the attribute values of the performance detection dimensions corresponding to each information combination.
In implementation, after the computer device obtains the performance detection dimension of the model to be detected, for each information combination, the computer device may further read the attribute value of the performance detection dimension corresponding to the information combination. For example, the detection dimension is an account click rate, the computer device may read the account click rate corresponding to the information combination, and the attribute value may also be an account download rate, which is not limited in the embodiment of the present disclosure.
In step 153, the average value of the attribute values of each information combination is determined as the attribute value of the interaction information.
In implementation, after the computer device reads the attribute values of the performance detection dimensions corresponding to the information combinations, the average value of the attribute values of the information combinations can be further determined as the attribute value of the interaction information.
For example, after the computer device obtains the target information combination corresponding to each information display bit corresponding to each sample data, it may further determine whether the target information combination corresponding to the information display bit exists in the historical information display data stored in the information display system.
If the historical information display data stored in the information display system has the target information combination corresponding to the information display position, the computer device may use the ratio of the number of the target information combinations clicked by the account to the total number of the target information combinations in the historical information display data as the account click probability of the target information combination corresponding to the information display position. For example, the information display bit is an information display bit P1, the target information combination corresponding to the information display bit is { P1, a2}, the history information display data stored in the information display system includes 1000 target information combinations { P1, a2}, wherein the account click type of 700 target information combinations { P1, a2} is account clicked, and the account click probability of the target information combination { P1, a2} is 700/1000 ═ 0.7. For another example, the information presentation bit is an information presentation bit P2, the target information combination corresponding to the information presentation bit is { P2, a1}, the history information presentation data stored in the information presentation system includes 2000 target information combinations { P2, a1}, where the account click type of 1000 target information combinations { P2, a1} is account clicked, and the account click probability of the target information combination { P2, a1} is 1000/2000 ═ 0.5.
If the target information combination corresponding to the information display position does not exist in the historical information display data stored in the information display system, the computer equipment can use the account click type in the historical information display data as the ratio of the number of the clicked historical information display data of the account to the total number of the historical information display data as the account click probability of the target information combination corresponding to the information display position. For example, the information display bit is an information display bit P1, the target information combination corresponding to the information display bit is { P1, a2}, 10000 pieces of history information display data are stored in the information display system, wherein the account click type of 3000 pieces of history information display data is that an account is clicked, and the account click probability of the target information combination { P1, a2} is 3000/10000 ═ 0.3. For another example, the information display bit is an information display bit P2, the target information combination corresponding to the information display bit is { P2, a1}, 10000 pieces of history information display data are stored in the information display system, wherein the account click type of 3000 pieces of history information display data is account clicked, and the account click probability of the target information combination { P2, a1} is also 3000/10000-0.3. After the computer device obtains the account click probability of each target information combination, the average value of the account click probability of each information combination can be determined as the attribute value of the interaction information. The computer device may also determine only an average value of the account click probabilities corresponding to the information presentation bits present in the history information presentation data stored in the information presentation system as the attribute value of the interaction information. The computer equipment can also determine the detection value of the model to be detected of the performance to be detected, namely the attribute value of the interactive information according to a preset model performance detection algorithm and the account click probability of each target information combination. The model performance detection algorithm may be a cross entropy algorithm, an auc (area Under cut) algorithm, a relative information gain algorithm, and the like, and the embodiment of the present disclosure is not limited.
And step 160, determining the performance detection result of the model to be detected based on the interaction information.
In implementation, after the computer device obtains the interaction information, the performance detection result of the model to be detected can be further determined based on the interaction information.
As an alternative implementation, as shown in fig. 3, in step 160, the computer device determines the performance test result of the model to be tested based on the interaction information as follows.
Step 161, determine whether the attribute value of the interactive information exceeds the target attribute value.
In implementation, a target attribute value may be stored in advance in the computer device, and the target attribute value may be set by a technician based on experience. After obtaining the attribute value of the interactive information, the computer device may further determine whether the attribute value of the interactive information exceeds the target attribute value. If the attribute value of the interactive information exceeds the target attribute value, step 162 is performed. If the attribute value of the interactive information does not exceed the target attribute value, step 163 is performed. The excess may be greater than or equal to, and the embodiment of the present disclosure is not limited. Correspondingly, the content of the non-excess may be equal to or less than the content of the non-excess, and the embodiment of the disclosure is not limited.
And step 162, if the attribute value of the interaction information exceeds the target attribute value, determining that the performance detection result of the model to be detected is qualified.
In implementation, if the attribute value of the interaction information exceeds the target attribute value, it indicates that the performance of the model to be tested meets the preset performance index requirement, that is, the computer device determines that the performance detection result of the model to be tested is qualified.
And 163, if the attribute value of the interaction information does not exceed the target attribute value, determining that the performance detection result of the model to be detected is unqualified.
In implementation, if the attribute value of the interaction information does not exceed the target attribute value, it indicates that the performance of the model to be tested does not meet the preset performance index requirement, that is, the computer device determines that the performance detection result of the model to be tested is unqualified.
The embodiment of the disclosure provides a method for detecting model performance, and computer equipment acquires historical information display data stored in an information display system. The historical information display data at least comprises information of information display bits. The computer equipment determines a plurality of sample data from each historical information display data, and retrieves the display information corresponding to the information display position in each sample data through the information display system. The information display system is provided with a model to be detected of the performance to be detected, and the model to be detected is used for executing one or more operations in the retrieval processing. Then, the computer device determines whether an information combination of the information display bit and the retrieved display information exists in the history information display data. And if the information combination exists in the historical information display data, the computer equipment acquires interactive information corresponding to the information combination, wherein the interactive information is used for representing interactive operation between the account and the display information, and the performance detection result of the model to be detected is determined based on the interactive information. Thus, the performance of the model is tested by on-line simulation of the model; meanwhile, in the process of testing the performance of the model, the historical data is screened and filtered through a system containing the model to be tested, the performance of the model to be tested is determined based on the test result brought by the data meeting the requirements, and the evaluation accuracy of the account feedback evaluation model can be improved.
It should be understood that although the various steps in the flow charts of fig. 1-3 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 1-3 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.
FIG. 4 is a block diagram illustrating a detection apparatus for model performance according to an exemplary embodiment. Referring to fig. 4, the apparatus includes a first obtaining module 410, a first determining module 420, a retrieving module 430, a judging module 440, a second obtaining module 450, and a second determining module 460, wherein,
a first obtaining module 410, configured to obtain each historical information display data stored in the information display system, where the historical information display data at least includes information of an information display bit;
a first determining module 420 configured to determine a plurality of sample data from the historical information display data;
a retrieval module 430 configured to retrieve, through an information presentation system, presentation information corresponding to the information presentation bits in each sample data, wherein a model to be tested with a performance to be tested is set in the information presentation system, and the model to be tested is used for performing one or more operations in the retrieval process;
a judging module 440 configured to judge whether an information combination composed of the information display bits and the retrieved display information exists in the history information display data;
the second obtaining module 450 is configured to obtain interaction information corresponding to the information combination if the information combination exists in the historical information display data, where the interaction information is used to represent an interaction operation between the account and the display information;
and a second determining module 460 configured to determine a performance test result of the model to be tested based on the interaction information.
As an optional implementation manner, the historical information display data further includes display information, and the retrieval module 430 is specifically configured to:
searching display information matched with the characteristics of each information display position from an information base through a plurality of searching algorithms arranged in the information display system; wherein, the model to be tested is provided with one or more retrieval algorithms.
As an optional implementation manner, the second obtaining module 450 is specifically configured to:
acquiring a performance detection dimension of a model to be detected;
reading attribute values of performance detection dimensions corresponding to all information combinations;
and determining the average value of the attribute values of all the information combinations as the attribute value of the interaction information.
As an optional implementation manner, the second determining module 460 is specifically configured to:
judging whether the attribute value of the interactive information exceeds a target attribute value or not;
if the attribute value of the interaction information exceeds the target attribute value, determining that the performance detection result of the model to be detected is qualified;
and if the attribute value of the interactive information exceeds the target attribute value, determining that the performance detection result of the model to be detected is unqualified.
The embodiment of the disclosure provides a device for detecting model performance, and computer equipment acquires historical information display data stored in an information display system. The historical information display data at least comprises information of information display bits. The computer equipment determines a plurality of sample data from each historical information display data, and retrieves the display information corresponding to the information display position in each sample data through the information display system. The information display system is provided with a model to be detected of the performance to be detected, and the model to be detected is used for executing one or more operations in the retrieval processing. Then, the computer device determines whether an information combination of the information display bit and the retrieved display information exists in the history information display data. And if the information combination exists in the historical information display data, the computer equipment acquires interactive information corresponding to the information combination, wherein the interactive information is used for representing interactive operation between the account and the display information, and the performance detection result of the model to be detected is determined based on the interactive information. Thus, the performance of the model is tested by on-line simulation of the model; meanwhile, in the process of testing the performance of the model, the historical data is screened and filtered through a system containing the model to be tested, the performance of the model to be tested is determined based on the test result brought by the data meeting the requirements, and the evaluation accuracy of the account feedback evaluation model can be improved.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
FIG. 5 is a block diagram illustrating a computer device 500 for model evaluation, according to an example embodiment. For example, the device 500 may be a server. Referring to fig. 5, device 500 includes a processing component 520 that further includes one or more processors and memory resources, represented by memory 522, for storing instructions, such as applications, that are executable by processing component 520. The application programs stored in memory 522 may include one or more modules that each correspond to a set of instructions. Further, the processing component 520 is configured to execute instructions to perform the above-described method of detecting performance of a model.
The device 500 may also include a power component 524 configured to perform power management for the device 500, a wired or wireless network interface 526 configured to connect the device 500 to a network, and an input/output (I/O) interface 528. The device 500 may operate based on an operating system stored in the memory 522, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.
In an exemplary embodiment, a storage medium comprising instructions, such as the memory 522 comprising instructions, executable by the processor of the apparatus 500 to perform the above-described method of detecting performance of a model is also provided. The storage medium may be a non-transitory computer readable storage medium, which may be, for example, a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
In an exemplary embodiment, there is also provided a computer program product comprising a computer program stored in a readable storage medium, from which at least one processor of a device reads and executes the computer program, causing the device to perform the method of detecting a performance of a model as described above.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (10)
1. A method for detecting model performance, the method comprising:
acquiring various historical information display data stored in an information display system, wherein the historical information display data at least comprise information of information display positions;
determining a plurality of sample data from each historical information display data;
retrieving, by the information presentation system, presentation information corresponding to information presentation bits in each of the sample data, wherein a model to be tested of a performance to be tested is set in the information presentation system, and the model to be tested is used to perform one or more operations in the retrieval process;
judging whether an information combination consisting of the information display position and the retrieved display information exists in the historical information display data or not;
if the information combination exists in the historical information display data, acquiring interactive information corresponding to the information combination, wherein the interactive information is used for representing interactive operation between an account and the display information;
and determining the performance detection result of the model to be detected based on the interaction information.
2. The method of claim 1, wherein the historical information presentation data further includes presentation information, and the retrieving, by the information presentation system, presentation information corresponding to information presentation bits in each of the sample data includes:
searching display information matched with the characteristics of each information display position from an information base through a plurality of search algorithms arranged in the information display system; and one or more retrieval algorithms are set in the model to be tested.
3. The method according to claim 1 or 2, wherein the obtaining of the interaction information corresponding to the information combination comprises:
acquiring the performance detection dimension of the model to be detected;
reading attribute values of the performance detection dimensions corresponding to the information combinations;
and determining the average value of the attribute values of the information combinations as the attribute value of the interactive information.
4. The method according to claim 1 or 2, wherein the determining the performance test result of the model to be tested based on the interaction information comprises:
judging whether the attribute value of the interactive information exceeds a target attribute value or not;
if the attribute value of the interaction information exceeds the target attribute value, determining that the performance detection result of the model to be detected is qualified;
and if the attribute value of the interaction information does not exceed the target attribute value, determining that the performance detection result of the model to be detected is unqualified.
5. An apparatus for detecting performance of a model, the apparatus comprising:
the information display system comprises a first acquisition module, a second acquisition module and a display module, wherein the first acquisition module is configured to acquire various historical information display data stored in the information display system, and the historical information display data at least comprise information of information display bits;
the first determining module is configured to determine a plurality of sample data from each historical information display data;
a retrieval module configured to retrieve, by the information presentation system, presentation information corresponding to information presentation bits in each sample data, where a model to be tested of a performance to be tested is set in the information presentation system, and the model to be tested is used to perform one or more operations in the retrieval process;
a judging module configured to judge whether an information combination composed of the information display bit and the retrieved display information exists in the history information display data;
the second obtaining module is configured to obtain interactive information corresponding to the information combination if the information combination exists in the historical information display data, wherein the interactive information is used for representing interactive operation between an account and the display information;
and the second determination module is configured to determine a performance detection result of the model to be detected based on the interaction information.
6. The apparatus according to claim 5, wherein the historical information presentation data further includes presentation information, and the retrieval module is specifically configured to:
searching display information matched with the characteristics of each information display position from an information base through a plurality of search algorithms arranged in the information display system; and one or more retrieval algorithms are set in the model to be tested.
7. The apparatus according to claim 5 or 6, wherein the second obtaining module is specifically configured to:
acquiring the performance detection dimension of the model to be detected;
reading attribute values of the performance detection dimensions corresponding to the information combinations;
and determining the average value of the attribute values of the information combinations as the attribute value of the interactive information.
8. The apparatus according to claim 5 or 6, wherein the second determining module is specifically configured to:
judging whether the attribute value of the interactive information exceeds a target attribute value or not;
if the attribute value of the interaction information exceeds the target attribute value, determining that the performance detection result of the model to be detected is qualified;
and if the attribute value of the interaction information does not exceed the target attribute value, determining that the performance detection result of the model to be detected is unqualified.
9. A computer device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the method of detecting performance of the model according to any one of claims 1 to 4.
10. A storage medium in which instructions, when executed by a processor of a computer device, enable the computer device to perform a method of detecting performance of a model as claimed in any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010597805.4A CN111882347A (en) | 2020-06-28 | 2020-06-28 | Model performance detection method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010597805.4A CN111882347A (en) | 2020-06-28 | 2020-06-28 | Model performance detection method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111882347A true CN111882347A (en) | 2020-11-03 |
Family
ID=73157122
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010597805.4A Pending CN111882347A (en) | 2020-06-28 | 2020-06-28 | Model performance detection method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111882347A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024087217A1 (en) * | 2022-10-28 | 2024-05-02 | 北京小米移动软件有限公司 | Model performance monitoring method and apparatus, device, and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104992348A (en) * | 2015-06-24 | 2015-10-21 | 深圳市腾讯计算机系统有限公司 | Method and device for displaying information |
| CN106803190A (en) * | 2017-01-03 | 2017-06-06 | 北京掌阔移动传媒科技有限公司 | A kind of ad personalization supplying system and method |
| CN109460513A (en) * | 2018-10-31 | 2019-03-12 | 北京字节跳动网络技术有限公司 | Method and apparatus for generating clicking rate prediction model |
| CN110796477A (en) * | 2019-09-23 | 2020-02-14 | 北京三快在线科技有限公司 | Advertisement display method and device, electronic equipment and readable storage medium |
| CN110874787A (en) * | 2019-11-08 | 2020-03-10 | 腾讯科技（深圳）有限公司 | Recommendation model effect evaluation method and related device |
-
2020
- 2020-06-28 CN CN202010597805.4A patent/CN111882347A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104992348A (en) * | 2015-06-24 | 2015-10-21 | 深圳市腾讯计算机系统有限公司 | Method and device for displaying information |
| CN106803190A (en) * | 2017-01-03 | 2017-06-06 | 北京掌阔移动传媒科技有限公司 | A kind of ad personalization supplying system and method |
| CN109460513A (en) * | 2018-10-31 | 2019-03-12 | 北京字节跳动网络技术有限公司 | Method and apparatus for generating clicking rate prediction model |
| CN110796477A (en) * | 2019-09-23 | 2020-02-14 | 北京三快在线科技有限公司 | Advertisement display method and device, electronic equipment and readable storage medium |
| CN110874787A (en) * | 2019-11-08 | 2020-03-10 | 腾讯科技（深圳）有限公司 | Recommendation model effect evaluation method and related device |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024087217A1 (en) * | 2022-10-28 | 2024-05-02 | 北京小米移动软件有限公司 | Model performance monitoring method and apparatus, device, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113688167A (en) | Deep interest capture model construction method and device based on deep interest network | |
| CN103440199B (en) | Test bootstrap technique and device | |
| CN112995690B (en) | Live content category identification method, device, electronic equipment and readable storage medium | |
| CN110096512A (en) | Question bank establishing method and device, learning equipment and storage medium | |
| EP4273750A1 (en) | Data processing method and apparatus, computing device, and test simplification device | |
| CN111444075A (en) | Method for automatically discovering key influence indexes | |
| US20100131497A1 (en) | Method for determining which of a number of test cases should be run during testing | |
| CN111882347A (en) | Model performance detection method, device, computer equipment and storage medium | |
| CN110955774B (en) | Word frequency distribution-based character classification method, device, equipment and medium | |
| US11360974B2 (en) | Ontology driven crowd sourced multi-dimensional question-answer processing for automated bid processing for rapid bid submission and win rate enhancement | |
| JP5640796B2 (en) | Name identification support processing apparatus, method and program | |
| CN113159537B (en) | Assessment method and device for new technical project of power grid and computer equipment | |
| CN115269932A (en) | Training scoring method and device for simulation training equipment, storage medium and equipment | |
| CN114944219A (en) | Mental scale recommendation method and device based on artificial intelligence and storage medium | |
| CN114048148A (en) | Crowdsourcing test report recommendation method and device and electronic equipment | |
| CN111338742B (en) | Point cloud data batch processing method and device | |
| CN113628077A (en) | Method for generating non-repeated examination questions, terminal and readable storage medium | |
| CN114501163A (en) | Video processing method, device and storage medium | |
| CN112396498A (en) | Commodity sales promotion method, device, equipment and storage medium | |
| CN113177023B (en) | Log retrieval method and device and electronic equipment | |
| CN114020643B (en) | Knowledge base testing method and device | |
| CN117540062B (en) | Retrieval model recommendation method and device based on knowledge graph | |
| CN112395455B (en) | Recommendation method and device for musical composition information and electronic equipment | |
| CN117992341A (en) | Defect supplementing method, device and storage medium for test report | |
| CN117828040A (en) | Intelligent question-answering method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |