CN106817296B - Information recommendation test method and device and electronic equipment - Google Patents
Information recommendation test method and device and electronic equipment Download PDFInfo
- Publication number
- CN106817296B CN106817296B CN201710024224.XA CN201710024224A CN106817296B CN 106817296 B CN106817296 B CN 106817296B CN 201710024224 A CN201710024224 A CN 201710024224A CN 106817296 B CN106817296 B CN 106817296B
- Authority
- CN
- China
- Prior art keywords
- recommendation information
- operation amount
- information
- recommendation
- initiator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/21—Monitoring or handling of messages
- H04L51/212—Monitoring or handling of messages using filtering or selective blocking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/50—Testing arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/21—Monitoring or handling of messages
- H04L51/214—Monitoring or handling of messages using selective forwarding
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The embodiment of the application discloses a method and a device for testing an information recommendation algorithm and electronic equipment, wherein the method comprises the following steps: receiving copy data from an online server, the copy data being obtained by copying access request data received from a terminal by the online server; receiving the operation amount of first recommendation information obtained by an online server by using the access request data based on an online first information recommendation algorithm at a terminal; obtaining second recommendation information based on an offline second information recommendation algorithm by using the copy data; inquiring a historical database according to the copied data, and determining the operation amount of an initiator of the access request data on the known recommendation information; determining the operation amount of the second recommendation information at the terminal according to the operation amount of the initiator of the access request data on the known recommendation information; and determining a test result according to the comparison result of the operation amount of the first recommendation information and the second recommendation information. The method not only ensures the test accuracy, but also solves the problem of resource waste of the service providing end.
Description
Technical Field
The present disclosure relates to the field of internet technologies, and in particular, to a method and an apparatus for testing information recommendation, and an electronic device.
Background
With the rapid development of internet technology, information recommendation services based on the internet are becoming more and more popular, and the common recommended information includes news, social information, advertisements, and the like.
The current information recommendation service is generally implemented by the following means: after receiving an access request sent by a user, a service provider firstly inputs pre-collected data such as user identity, operation habit and the like into a preset information recommendation algorithm, processes the data through the information recommendation algorithm to determine information to be recommended, and finally pushes the determined information to a terminal held by the user. Therefore, the performance of the information recommendation algorithm is particularly important to the effect of information recommendation.
In the prior art, before an information recommendation algorithm is applied, a recommendation effect of the algorithm is tested, so as to determine whether to apply the algorithm or not according to a test result. The common test method comprises an AB shunt test method, and the process is as follows: firstly, simultaneously setting a first information recommendation algorithm to be applied and an applied second information recommendation algorithm on a service line where an information recommendation service is located; subsequently, when an access request of a user is received, randomly importing the related flow of the access request data into a first or second information recommendation algorithm; and after a certain time, determining whether the first recommendation algorithm has better effect according to the operation amount of the user on the information recommended by the first and second information recommendation algorithms.
However, when the AB shunt test method is used to test the recommendation effect of the information recommendation algorithm, the first and second information recommendation algorithms need to be applied to the service line of the information recommendation service at the same time, if the recommendation effect of the first recommendation algorithm to be applied is poor, the operation amount of the user on the information recommended by the first recommendation algorithm will be very small, and the invalid information recommendations will cause resource waste at the service providing end, and also reduce user experience.
Disclosure of Invention
An object of the embodiments of the present application is to provide a method and an apparatus for testing information recommendation, and an electronic device, which are used to solve the above problems.
According to a first aspect of the embodiments of the present disclosure, there is provided a test method for information recommendation, including:
receiving copy data from an online server, wherein the copy data is obtained by copying access request data received by the online server from a terminal;
receiving the operation amount of first recommendation information obtained by the online server by using the access request data based on an online first information recommendation algorithm at the terminal;
obtaining second recommendation information based on an offline second information recommendation algorithm by using the copy data;
inquiring a historical database according to the copy data, and determining the operation amount of an initiator of the access request data on the known recommendation information;
determining the operation amount of the second recommendation information at the terminal according to the operation amount of the initiator of the access request data on the known recommendation information;
and determining a test result according to the comparison result of the operation amount of the first recommendation information and the second recommendation information.
In one embodiment, the copy data is obtained by copying the access request data received from the terminal in real time through a copy code located in the on-line server.
In one embodiment, the access request data includes: the system comprises initiator identity information, request environment information and object information to be requested.
In an embodiment, after obtaining the second recommendation information based on the offline second information recommendation algorithm by using the copy data, before querying the historical database according to the copy data, the method further includes:
confirming that the second recommendation information is different from the first recommendation information;
and, the method further comprises:
and if the second recommendation information is judged to be the same as the first recommendation information, taking the operation amount of the first recommendation information on the terminal as the operation amount of the second recommendation information on the terminal.
In an embodiment, querying a history database according to the copy data, and determining an operation amount of an initiator of the access request data on the known recommendation information includes:
inquiring a historical database according to the copied data, and judging whether the operation amount of an initiator of the access request data on target recommendation information exists in the historical database, wherein the target recommendation information is known recommendation information with the same type as the second recommendation information;
if so, taking the operation quantity of the initiator of the access request data on the target recommendation information as the operation quantity of the initiator on the known recommendation information;
if not, taking the default user operation amount of the initiator of the access request data as the operation amount of the initiator on the known recommendation information; the default user operation amount is the average value of the operation amounts of the user on all known recommendation information.
In an embodiment, after determining that the operation amount of the initiator on the target recommendation information does not exist in the history database, before taking the default user operation amount as the operation amount of the initiator on the known recommendation information, the test method further includes:
confirming that the default user operation amount of the initiator of the access request data is not null;
and, the method further comprises:
if the default user operation amount of the initiator of the access request data is judged to be empty, taking the default group operation amount of the initiator of the access request data as the operation amount of the initiator on the known recommendation information; the default group operation amount is the average value of the operation amounts of all known initiators on the known recommendation information.
In an embodiment, the first recommendation information and the second recommendation information are both advertisement information, and the advertisement information is an exposure advertisement or a display advertisement;
when the advertisement information is an exposure advertisement, the operation amount is an exposure amount;
and when the advertisement information is click advertisements, the operation amount is click amount.
In an embodiment, the first recommendation information and the second recommendation information both have resource consumption;
determining a test result according to a comparison result of the operation amount of the first recommendation information and the second recommendation information, specifically comprising:
determining the total resource consumption amount of the first recommendation information according to the operation amount of the first recommendation information and the resource consumption amount of the first recommendation information of unit operation amount;
determining the total resource consumption amount of the second recommendation information according to the operation amount of the second recommendation information and the resource consumption amount of the second recommendation information of unit operation amount;
judging whether the total resource consumption amount of the second recommendation information is larger than that of the first recommendation information;
if not, determining that the test result is as follows: the effect of the first recommendation information is better;
if yes, determining that the test result is: the effect of the second recommendation information is better.
According to a second aspect of the embodiments of the present disclosure, there is provided a test apparatus for information recommendation, including:
the data receiving module is used for receiving copy data from an online server, wherein the copy data is obtained by copying access request data received by the online server from a terminal;
a first operation amount receiving module, configured to receive an operation amount of first recommendation information obtained by the online server by using the access request data based on an online first information recommendation algorithm at the terminal;
the information recommendation module is used for obtaining second recommendation information based on an offline second information recommendation algorithm by using the copy data;
a known operation quantity query module, configured to query a historical database according to the copy data, and determine an operation quantity of an initiator of the access request data on known recommendation information;
the second operation amount determining module is used for determining the operation amount of the second recommendation information on the terminal according to the operation amount of the initiator of the access request data on the known recommendation information;
and the test result determining module is used for determining a test result according to a comparison result of the operation amount of the first recommendation information and the second recommendation information.
In one embodiment, the copy data is obtained by copying the access request data received from the terminal in real time through a copy code located in the on-line server.
In one embodiment, the access request data includes: the system comprises initiator identity information, request environment information and object information to be requested.
In one embodiment, the test apparatus further comprises:
the information comparison module is used for judging whether the second recommendation information is the same as the first recommendation information;
the operation amount equivalent module is used for taking the operation amount of the first recommendation information on the terminal as the operation amount of the second recommendation information on the terminal when the second recommendation information is the same as the first recommendation information;
the known operation quantity query module is specifically configured to:
and only when the second recommendation information is different from the first recommendation information, inquiring a historical database according to the copy data, and determining the operation amount of the initiator of the access request data on the known recommendation information.
In an embodiment, the known operation amount query module is specifically configured to:
inquiring a historical database according to the copied data, and judging whether the operation amount of an initiator of the access request data on target recommendation information exists in the historical database, wherein the target recommendation information is known recommendation information with the same type as the second recommendation information;
if so, taking the operation quantity of the initiator of the access request data on the target recommendation information as the operation quantity of the initiator on the known recommendation information;
if not, taking the default user operation amount of the initiator of the access request data as the operation amount of the initiator on the known recommendation information; the default user operation amount is the average value of the operation amounts of the user on all known recommendation information.
In an embodiment, the known operation amount query module is further configured to:
after determining that the operation quantity of the initiator of the access request data to the target recommendation information does not exist in the history database, judging whether the default user operation quantity of the initiator of the access request data is empty or not before taking the default user operation quantity as the operation quantity of the initiator to the known recommendation information;
if not, taking the default user operation amount as the operation amount of the initiator on the known recommendation information;
if so, taking the default group operation amount of the initiator of the access request data as the operation amount of the initiator on the known recommendation information; the default group operation amount is the average value of the operation amounts of all known initiators on the known recommendation information.
In an embodiment, the first recommendation information and the second recommendation information both have resource consumption;
the test result determining module is specifically configured to:
determining the total resource consumption amount of the first recommendation information according to the operation amount of the first recommendation information and the resource consumption amount of the first recommendation information of unit operation amount;
determining the total resource consumption amount of the second recommendation information according to the operation amount of the second recommendation information and the resource consumption amount of the second recommendation information of unit operation amount;
judging whether the total resource consumption amount of the second recommendation information is larger than that of the first recommendation information;
if not, determining that the test result is as follows: the effect of the first recommendation information is better;
if yes, determining that the test result is: the effect of the second recommendation information is better.
According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including:
a processor;
a memory for storing processor-executable instructions;
the processor is configured to:
receiving copy data from an online server, wherein the copy data is obtained by copying access request data received by the online server from a terminal;
receiving the operation amount of first recommendation information obtained by the online server by using the access request data based on an online first information recommendation algorithm at the terminal;
obtaining second recommendation information based on an offline second information recommendation algorithm by using the copy data;
inquiring a historical database according to the copy data, and determining the operation amount of an initiator of the access request data on the known recommendation information;
determining the operation amount of the second recommendation information at the terminal according to the operation amount of the initiator of the access request data on the known recommendation information;
and determining a test result according to the comparison result of the operation amount of the first recommendation information and the second recommendation information.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:
processing the access request data through a first information recommendation algorithm on a line to obtain first recommendation information, and then obtaining the operation amount of the first recommendation information on the terminal; and processing the copy data consistent with the access request data by a second information recommendation algorithm under the line to obtain second recommendation information, determining the operation quantity of the second recommendation information by the operation quantity of the initiator of the access request data on the known recommendation information, and finally determining a test result according to the operation quantities of the first recommendation information and the second recommendation information. The second information recommendation algorithm is not on the service line, so that the test accuracy is ensured, and the problem of resource waste of a service providing end possibly caused by poor recommendation effect of the second recommendation algorithm is solved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without any creative effort.
FIG. 1 is a schematic diagram of a system to which a test method is applicable, according to an example embodiment.
FIG. 2 is a flow chart illustrating a testing method according to an exemplary embodiment.
FIG. 3 is a flow chart illustrating another testing method according to an exemplary embodiment.
FIG. 4 is an architecture diagram illustrating a history database in accordance with an exemplary embodiment.
FIG. 5 is a flow chart illustrating yet another testing method according to an exemplary embodiment.
FIG. 6 is a block diagram illustrating a testing apparatus according to an exemplary embodiment.
FIG. 7 is a block diagram illustrating another testing apparatus in accordance with an exemplary embodiment.
FIG. 8 is a block diagram illustrating an electronic device suitable for algorithmic testing according to an example embodiment.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
FIG. 1 is a schematic diagram of a system to which a test method is applicable, according to an example embodiment. As shown in fig. 1, the system may include: a terminal 10 held by a user, an online server 20 that establishes communication with the terminal 10 through a network, and an offline server 30 that establishes communication with the online server 20 through a network. The terminal 10 may include, but is not limited to: mobile phones, tablet computers, PC devices, etc. The network for connecting the terminal 10 with the online server 20 and the network for connecting the online server 20 with the offline server 30 may each include: WIreless networks such as WI-FI (WIreless-Fidelity), bluetooth, etc., or wired networks that transmit information using cables. The online server 20 and the offline server 30 may be a server, a server cluster composed of a plurality of servers, or a cloud computing service center, and even the online server 20 and the offline server 30 may be integrated into the same device in terms of hardware, which is not described herein again.
The terminal 10 may be installed with a client App such as a browser, a social application, a video application, or a news information application, the online server 20 is a provider of an online service corresponding to the client, the online server 20 stores resources required for the corresponding client to operate, a user may open the corresponding client App on the terminal 10 and request the online server 20 to access a certain page, the online server 20 responds to the access request and sends data of the page to be requested to the terminal 10, and the terminal 10 parses the received data and displays the data on the client.
In the process that the online server 20 responds to the access request of the user and sends the data of the page to be accessed to the terminal 10, recommendation information is also added to the data of the page to be accessed, so that the page to be accessed displayed on the client where the terminal 10 is located carries the recommendation information.
The offline server 30 is connected to the online server 20, the offline server 30 stores the offline information recommendation algorithm and the test application for testing the information recommendation algorithm, and the offline server 30 can test whether the offline information recommendation algorithm has a better recommendation effect than the information recommendation algorithm in the online server 20 through the test application.
Embodiments of the offline server 30-based test method will be described below based on the system shown in fig. 1.
FIG. 2 is a flow chart illustrating a testing method according to an exemplary embodiment. The test method in this embodiment is applied to the offline server 30, and the test method can be started and terminated by installing a corresponding test application on the online server 20. As shown in FIG. 2, the testing method of the present embodiment may include the following steps 101 to 106. The sequence of steps in the test method is not limited to steps 101 to 106, for example, step 102 may be selected at any position before step 106, which is not described herein again.
In step 101, copy data from an online server is received, the copy data being obtained by copying access request data received by the online server from a terminal.
When a user operates a client on the terminal 10 to request the online server 20 to access a certain page, the client generates an access request according to an operation behavior of the user in the client on the terminal 10, where the access request carries access request data. Subsequently, the terminal 10 encapsulates the access request data and transmits the encapsulated access request data to the online server 20.
The access request includes various requests generated by the terminal 10 and transmitted to the online server 20 when the terminal 10 interacts with the online server 20, such as a login request for requesting a login account, an access request for requesting access to a page, such as a social information page, a news page, or a search request for requesting a search for a target object.
The access request data may include at least one of: identity data of the requesting party, request environment data and object data to be requested. From these types of data, the service content returned to the client can be determined.
The identity data of the requesting party is used to describe the identity of the user registering the client and may include, for example, a customer account number, a customer identification number, a driver license number, a bank card number, a customer location, a customer purchase record, a customer collection record, a customer interest record, and a customer blackout record. From this data, the unique requestor can be determined. The request environment data is used to describe the relevant parameters of the hardware device and the software program used when the user operates the client to initiate the request, the parameters of the hardware device may be the MAC address, the UMID code, the SIM card number, the IMEI code, the IP address, the VPN address, the device model, etc. of the terminal 10, and the parameters of the software program may be the operating system type, the browser type, the client version, etc. The data of the object to be requested is used to describe the information of the object to be requested, and may be, for example, the type, address, etc. of the object to be requested.
After receiving the access request data, the online server 20 copies the access request data in real time as copy data, and then sends the copy data to the offline server 30. In this embodiment, the traffic copy code may be stored in the online server 20, and after the online server 20 receives the access request data, the traffic copy code is executed to copy the access request data. Since the copy data and the access request data have the same content, the online server 20 and the online server 30 have the same input data when the information recommendation algorithm is aligned.
In step 102, receiving an operation amount of the terminal by the online server based on the first recommendation information obtained by the online first information recommendation algorithm by using the access request data.
The online server 20 stores a first information recommendation algorithm, the first information recommendation algorithm is located on a service line where the online service request is located, and the access request data is imported into the first information recommendation algorithm after entering the online server 20. And processing the access request data through a first information recommendation algorithm to obtain first recommendation information. The first recommendation information may be an advertisement or other type of information.
Following the course of the service line, the online server 20 also generates response data for return to the terminal 10 based on the access request data. For example, if the service request is an access request of a social information page, the response data is page data of the social information page to be accessed, and the terminal 10 renders the page data after receiving the page data, so as to display a corresponding page on the client. In this embodiment, the online server 20 packages the page data and the first recommendation information and transmits them to the terminal 10.
The first recommendation information may be click-type information or exposure-type information, and may be click-type advertisement information or exposure-type advertisement information, for example. When the first recommendation information is click information, a user can actively click the first recommendation information through finger touch, keyboard selection, mouse selection and other modes to check the specific content of the first recommendation information; when the first recommendation information is exposure information, the first recommendation information can be exposed on a page along with the display of the information to be accessed, so that a user can see the first recommendation information. The operation amount of the different types of recommendation information is defined differently, the number of times that the first recommendation information of the click class is actively operated by the user is the operation amount, and the number of times that the first recommendation information of the exposure class is exposed on the display interface is the operation amount.
In this embodiment, regardless of the type of the first recommendation information, the operation amount of the first recommendation information can be determined quickly by acquiring the operation log from the client on the terminal 10.
In step 103, second recommendation information is obtained based on an offline second information recommendation algorithm by using the copy data.
The offline server 30 stores a second information recommendation algorithm, where the second information recommendation algorithm is an algorithm to be online or applied, and the second information recommendation algorithm may be obtained by upgrading the first information recommendation algorithm, and a recommendation effect difference between the two is to be determined.
After receiving the copy data from the online server 20, the offline server 30 imports the copy data into a second information recommendation algorithm, and obtains second recommendation information after processing by the second information recommendation algorithm, where the second recommendation information may be the same as or different from the first recommendation information.
In step 104, according to the copy data, querying a history database, and determining the operation amount of the initiator of the access request data on the known recommendation information.
A plurality of sets of historical data are stored in the historical database, each set of historical data includes the identity information of the user and a historical operation record associated with the identity information, and the operation amount of the user on the known recommendation information is recorded in the historical operation record, and the operation amount of the known recommendation information can be obtained from the operation log of the online server 20 on the recommendation information and is prestored.
In this embodiment, the operation amount of the initiator on the known recommendation information can be screened out in the history database by copying the initiator identity information in the data.
In step 105, determining an operation amount of the second recommendation information at the terminal according to an operation amount of the initiator of the access request data on the known recommendation information.
And the operation quantity of the same initiator on the second recommendation information is presumed by the operation quantity of the initiator accessing the request data on the known recommendation information, so that the operation quantity of the second recommendation information is determined when no real service flow exists.
In step 106, a test result is determined according to a comparison result of the operation amounts of the first recommendation information and the second recommendation information.
The operation amount of the recommendation information can reflect the matching degree of the recommendation information to the user requirements, and the higher the operation amount of the determined number of recommendation information (namely, the higher the arrival rate of the recommendation information), the more the recommendation information meets the user requirements, and the better the recommendation effect of the recommendation information is.
As the traffic flow continuously enters the online server 20, the online server 20 forms a large amount of first recommendation information, and the offline server 30 also forms a same amount of second recommendation information, and by comparing the operation amounts of the first recommendation information and the second recommendation information, the arrival rates of the first recommendation information and the second recommendation information can be determined. In this embodiment, if the operation amount of the second recommendation information is lower, it is determined that the test result is: the effect of the first recommendation information is better; if the operation amount of the second recommendation information is higher, determining that the test result is as follows: the effect of the second recommendation information is better.
In the present embodiment, in the process of generating and recommending the first recommendation information and the second recommendation information, both the online server 20 and the offline server 30 generate resource consumption. This resource consumption is related to both the content of the two recommendations, the originator. Step 106 may be further adjusted to:
and determining the total resource consumption amount of the first recommendation information according to the operation amount of the first recommendation information and the resource consumption amount of the first recommendation information of unit operation amount.
And determining the total resource consumption amount of the second recommendation information according to the operation amount of the second recommendation information and the resource consumption amount of the second recommendation information of unit operation amount.
And judging whether the total resource consumption amount of the second recommendation information is larger than that of the first recommendation information.
If not, determining that the test result is as follows: the effect of the first recommendation information is better;
if yes, determining that the test result is: the effect of the second recommendation information is better.
The resource consumption by the recommendation information per unit operation amount is the resource consumption of the relevant server by the recommendation information being operated each time. By introducing the resource consumption brought by unit operation amount on the basis of the operation amount, the resource consumption total amount of each recommendation information can be obtained by multiplying the resource consumption amount and the unit operation amount, and the recommendation effect of the second recommendation information can be more completely evaluated.
In practical application, the first recommendation information and the second recommendation information may be advertisements, resource consumption of the advertisement recommendation information may be monetized and materialized, resource consumption of the recommendation information per unit operation amount may be an advertisement fee paid by an initiator to a service provider each time the advertisement is operated, and then an advertisement fee total amount after a large magnitude of recommendation information is operated may be obtained, and accordingly, an effect of advertisement information recommendation may be evaluated in a targeted manner.
In summary, according to the test method of the information recommendation algorithm provided by the embodiment of the application, after the access request data is processed through the first information recommendation algorithm on the line to obtain the first recommendation information, the operation amount of the first recommendation information on the terminal is obtained; and processing the copy data consistent with the access request data by a second information recommendation algorithm under the line to obtain second recommendation information, determining the operation quantity of the second recommendation information by the operation quantity of the initiator of the access request data on the known recommendation information, and finally determining a test result according to the operation quantities of the first recommendation information and the second recommendation information. The second information recommendation algorithm is not on the service line, so that the test accuracy is ensured, and the problem of resource waste of a service providing end possibly caused by poor recommendation effect of the second recommendation algorithm is solved.
FIG. 3 is a flow chart illustrating another testing method according to an exemplary embodiment. The test method may be applied to the off-line server 30. As shown in fig. 3, the test method includes steps 201 to 206.
In step 201, copy data is received from an online server, the copy data being obtained by copying access request data received by the online server from a terminal.
In step 202, receiving an operation amount of the terminal by the online server based on the first recommendation information obtained by the online first information recommendation algorithm by using the access request data.
In step 203, second recommendation information is obtained based on an offline second information recommendation algorithm by using the copy data.
Step 204 may include steps 241 through 243 as follows.
In step 241, querying a history database according to the copy data, and determining whether an operation amount of an initiator of the access request data on target recommendation information exists in the history database, where the target recommendation information is known recommendation information having the same type as the second recommendation information, if so, entering step 242, and if not, entering step 243.
In step 242, the operation amount of the initiator of the access request data for the target recommendation information is used as the operation amount of the initiator for the known recommendation information.
In step 243, it is determined whether the default user operation amount of the initiator of the access request data is empty, if not, the process proceeds to step 244, and if so, the process proceeds to step 245.
In step 244, the default user operation amount of the initiator of the access request data is used as the operation amount of the initiator on the known recommendation information.
In step 245, the default group operation amount of the initiator of the access request data is used as the operation amount of the initiator on the known recommendation information.
It is noted that in other embodiments of the present invention, either of steps 244 and 245 may be selected as a subsequent step after step 243 is entered. Not limited in this embodiment, step 243 must be followed by step 244 and step 245 at the same time, which is not described herein.
FIG. 4 is an architecture diagram illustrating a history database in accordance with an exemplary embodiment. As shown in fig. 4, in an exemplary scenario, the history database is in the form of a hierarchical tree, and the lowest layer is the operation amount of the initiator on different types of known recommendation information; the upper layer is the default user operation amount of the initiator, and the default user operation amount is the average value of the operation amounts of the user on all types of known recommendation information; the top layer is the default group operation amount of the initiator, and the default group operation amount is the average value of the operation amounts of all known initiators on the known recommendation information.
When the operation amount of the initiator on the known recommendation information is determined in step 204, whether the initiator operates on the target recommendation information is confirmed by using the identity information of the initiator from the lowest layer of the history database. And when the initiator is determined not to have operated the target recommendation information, continuing to the upper layer, determining whether the initiator has operated any type of known recommendation information, and when the initiator has operated any type of recommendation information, taking the average value of the operation amounts of the user on all types of known recommendation information as a default user operation amount and taking the default user operation amount as the operation amount of the initiator on the known recommendation information. And when the initiator is determined not to operate any type of known recommendation information, continuing to operate upwards, taking the average value of the operation amounts of all the initiators on the known recommendation information as a default group operation amount, and taking the default group operation amount as the operation amount of the initiators on the known recommendation information.
In step 205, the operation amount of the second recommendation information at the terminal is determined according to the operation amount of the initiator of the access request data on the known recommendation information.
In step 206, a test result is determined according to a comparison result of the operation amounts of the first recommendation information and the second recommendation information.
FIG. 5 is a flow chart illustrating yet another testing method according to an exemplary embodiment. The test method may be applied to the off-line server 30. As shown in fig. 5, the testing method includes steps 301 to 308:
in step 301, copy data is received from an online server, the copy data being obtained by copying access request data received by the online server from a terminal.
In step 302, an operation amount of the terminal by the first recommendation information obtained by the online server by using the access request data based on the online first information recommendation algorithm is received.
In step 303, second recommendation information is obtained based on an offline second information recommendation algorithm using the copy data.
In step 304, it is determined whether the second recommendation information is the same as the first recommendation information, if so, the process proceeds to step 305, and if not, the process proceeds to step 306.
When the second recommendation information is the same as the first recommendation information, the operation amount of the first recommendation information can be directly used without predicting the operation amount of the second recommendation information.
In step 305, the operation amount of the first recommendation information on the terminal is used as the operation amount of the second recommendation information on the terminal.
In step 306, according to the copy data, querying a history database, and determining the operation amount of the initiator of the access request data on the known recommendation information.
It is noted that in other embodiments of the present invention, either of step 305 and step 306 may be selected as a subsequent step after step 304 is entered. Not limited in this embodiment, step 305 and step 306 must be included after step 304, which is not described herein.
In step 307, the operation amount of the second recommendation information on the terminal is determined according to the operation amount of the initiator of the access request data on the known recommendation information.
In step 308, a test result is determined according to a comparison result of the operation amounts of the first recommendation information and the second recommendation information.
FIG. 6 is a block diagram illustrating a testing apparatus according to an exemplary embodiment. The testing apparatus may be used on the off-line server 30, and in this embodiment, the testing apparatus may include:
a data receiving module 401, configured to receive copy data from an online server, where the copy data is obtained by copying access request data received by the online server from a terminal;
a first operation amount receiving module 402, configured to receive an operation amount of first recommendation information obtained by the online server by using the access request data based on an online first information recommendation algorithm at the terminal;
an information recommendation module 403, configured to obtain second recommendation information based on an offline second information recommendation algorithm by using the copy data;
a known operand query module 404, configured to query a historical database according to the copy data, and determine an operand of an initiator of the access request data to the known recommendation information;
a second operation amount determining module 405, configured to determine, according to an operation amount of an initiator of the access request data on known recommendation information, an operation amount of the second recommendation information on the terminal;
the test result determining module 406 is configured to determine a test result according to a comparison result of the operation amounts of the first recommendation information and the second recommendation information.
In one embodiment, the copy data is obtained by copying the access request data received from the terminal in real time through a copy code located in the on-line server.
In one embodiment, the access request data includes: the system comprises initiator identity information, request environment information and object information to be requested.
In an embodiment, the first recommendation information and the second recommendation information are both advertisement information, and the advertisement information is an exposure advertisement or a display advertisement;
when the advertisement information is an exposure advertisement, the operation amount is an exposure amount;
and when the advertisement information is click advertisements, the operation amount is click amount.
In an embodiment, the first recommendation information and the second recommendation information both have resource consumption;
the test result determining module 406 is specifically configured to:
determining the total resource consumption amount of the first recommendation information according to the operation amount of the first recommendation information and the resource consumption amount of the first recommendation information of unit operation amount;
determining the total resource consumption amount of the second recommendation information according to the operation amount of the second recommendation information and the resource consumption amount of the second recommendation information of unit operation amount;
judging whether the total resource consumption amount of the second recommendation information is larger than that of the first recommendation information;
if not, determining that the test result is as follows: the effect of the first recommendation information is better;
if yes, determining that the test result is: the effect of the second recommendation information is better.
In an embodiment, the known operand query module 404 is specifically configured to:
inquiring a historical database according to the copied data, and judging whether the operation amount of an initiator of the access request data on target recommendation information exists in the historical database, wherein the target recommendation information is known recommendation information with the same type as the second recommendation information;
if so, taking the operation quantity of the initiator of the access request data on the target recommendation information as the operation quantity of the initiator on the known recommendation information;
if not, taking the default user operation amount of the initiator of the access request data as the operation amount of the initiator on the known recommendation information.
In an embodiment, the default user operation amount is an average value of operation amounts of the user on all known recommendation information.
In an embodiment, the known operand querying module 404 is further configured to:
after determining that the operation quantity of the initiator of the access request data to the target recommendation information does not exist in the history database, judging whether the default user operation quantity of the initiator of the access request data is empty or not before taking the default user operation quantity as the operation quantity of the initiator to the known recommendation information;
if not, taking the default user operation amount as the operation amount of the initiator on the known recommendation information;
and if so, taking the default group operation amount of the initiator of the access request data as the operation amount of the initiator on the known recommendation information.
In an embodiment, the default group operation amount is an average value of operation amounts of all known initiators on the known recommendation information.
In summary, the testing device for the information recommendation algorithm provided in the embodiment of the present application obtains the operation amount of the first recommendation information on the terminal after processing the access request data through the first information recommendation algorithm on the line to obtain the first recommendation information; and processing the copy data consistent with the access request data by a second information recommendation algorithm under the line to obtain second recommendation information, determining the operation quantity of the second recommendation information by the operation quantity of the initiator of the access request data on the known recommendation information, and finally determining a test result according to the operation quantities of the first recommendation information and the second recommendation information. The second information recommendation algorithm is not on the service line, so that the test accuracy is ensured, and the problem of resource waste of a service providing end possibly caused by poor recommendation effect of the second recommendation algorithm is solved.
FIG. 7 is a block diagram illustrating another testing apparatus in accordance with an exemplary embodiment. On the basis of the embodiment shown in fig. 6, the testing apparatus further includes an information comparing module 407 and an operation quantity equating module 408.
The information comparison module 407 is configured to determine whether the second recommendation information is the same as the first recommendation information.
An operation amount equivalent module 408, configured to use, when the second recommendation information is the same as the first recommendation information, an operation amount of the first recommendation information at the terminal as an operation amount of the second recommendation information at the terminal. When the operation amount equivalent module 408 is activated, the known operation amount query module 404 and the second operation amount determination module 405 are not activated. It is understood that, in this case, the operation amount of the first recommendation information received by the first operation amount receiving module 402 and the operation amount of the second recommendation information determined by the operation amount equating module 408 are sent to the test result determining module 406 for comparison.
The known operation amount query module 404 is specifically configured to:
and taking the operation amount of the first recommendation information on the terminal as the operation amount of the second recommendation information on the terminal only when the second recommendation information is different from the first recommendation information. The second operation amount determination module 405 is activated when the known operation amount query module 404 is activated. It can be understood that, in this case, the operation amount of the first recommendation information received by the first operation amount receiving module 402 and the operation amount of the second recommendation information determined by the second operation amount determining module 405 are sent to the test result determining module 406 for comparison.
Meanwhile, when the amount of the recommendation information targeted by the test apparatus is large, the test result determining module 406 may receive the operation amount of the second recommendation information from the second operation amount determining module 405 and the operation amount equating module 408, so as to compare the operation amount with the operation amount of the first recommendation information received from the first operation amount receiving module 402, which is not described herein again.
FIG. 8 is a block diagram illustrating an electronic device suitable for device testing in accordance with an exemplary embodiment. For example, the electronic device 500 may be provided as a server. Referring to fig. 8, electronic device 500 includes a processing component 510 that further includes one or more processors and memory resources, represented by memory 520, for storing instructions, such as application programs, that are executable by processing component 510. The application programs stored in memory 520 may include one or more modules that each correspond to a set of instructions. The processing component 510 is configured to:
receiving copy data from an online server, wherein the copy data is obtained by copying access request data received by the online server from a terminal;
receiving the operation amount of first recommendation information obtained by the online server by using the access request data based on an online first information recommendation algorithm at the terminal;
obtaining second recommendation information based on an offline second information recommendation algorithm by using the copy data;
inquiring a historical database according to the copy data, and determining the operation amount of an initiator of the access request data on the known recommendation information;
determining the operation amount of the second recommendation information at the terminal according to the operation amount of the initiator of the access request data on the known recommendation information;
and determining a test result according to the comparison result of the operation amount of the first recommendation information and the second recommendation information.
The electronic device 500 may also include a power component 530, the power component 530 configured to perform power management of the electronic device 500, a wired or wireless network interface 540 configured to connect the electronic device 500 to a network, and an input/output (I/O) interface 550. The electronic device 500 may operate based on an operating system stored in the memory 520, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like.
In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 520 comprising instructions, executable by the processing component 510 of the electronic device 500 to perform the above-described method is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (12)
1. A test method for information recommendation is characterized by comprising the following steps:
receiving copy data from an online server, wherein the copy data is obtained by copying access request data received by the online server from a terminal;
receiving the operation amount of first recommendation information obtained by the online server by using the access request data based on an online first information recommendation algorithm at the terminal;
obtaining second recommendation information based on an offline second information recommendation algorithm by using the copy data;
inquiring a historical database according to the copy data, and determining the operation amount of an initiator of the access request data on the known recommendation information;
determining the operation amount of the second recommendation information at the terminal according to the operation amount of the initiator of the access request data on the known recommendation information;
and determining a test result according to the comparison result of the operation amount of the first recommendation information and the second recommendation information.
2. The method of testing as claimed in claim 1, wherein after obtaining second recommendation information based on an offline second information recommendation algorithm using the copied data, before querying a historical database based on the copied data, the method further comprises:
confirming that the second recommendation information is different from the first recommendation information;
and, the method further comprises:
and if the second recommendation information is judged to be the same as the first recommendation information, taking the operation amount of the first recommendation information on the terminal as the operation amount of the second recommendation information on the terminal.
3. The testing method according to claim 1, wherein querying a history database according to the copy data to determine an operation amount of an initiator of the access request data on known recommendation information includes:
inquiring a historical database according to the copied data, and judging whether the operation amount of an initiator of the access request data on target recommendation information exists in the historical database, wherein the target recommendation information is known recommendation information with the same type as the second recommendation information;
if so, taking the operation quantity of the initiator of the access request data on the target recommendation information as the operation quantity of the initiator on the known recommendation information;
if not, taking the default user operation amount of the initiator of the access request data as the operation amount of the initiator on the known recommendation information; the default user operation amount is the average value of the operation amounts of the user on all known recommendation information.
4. The testing method of claim 3, wherein after determining that the amount of operations on the target recommendation information by the initiator for which the access request data does not exist in the history database is not determined, and before taking a default user operation amount as the amount of operations on the known recommendation information by the initiator, the testing method further comprises:
confirming that the default user operation amount of the initiator of the access request data is not null;
and, the method further comprises:
if the default user operation amount of the initiator of the access request data is judged to be empty, taking the default group operation amount of the initiator of the access request data as the operation amount of the initiator on the known recommendation information; the default group operation amount is the average value of the operation amounts of all known initiators on the known recommendation information.
5. The test method according to claim 1, wherein the first recommendation information and the second recommendation information are advertisement information, and the advertisement information is an exposure-type advertisement or a display-type advertisement;
when the advertisement information is an exposure advertisement, the operation amount is an exposure amount;
and when the advertisement information is click advertisements, the operation amount is click amount.
6. The test method of claim 5, wherein the first recommendation information and the second recommendation information each have a resource consumption amount;
determining a test result according to a comparison result of the operation amount of the first recommendation information and the second recommendation information, specifically comprising:
determining the total resource consumption amount of the first recommendation information according to the operation amount of the first recommendation information and the resource consumption amount of the first recommendation information of unit operation amount;
determining the total resource consumption amount of the second recommendation information according to the operation amount of the second recommendation information and the resource consumption amount of the second recommendation information of unit operation amount;
judging whether the total resource consumption amount of the second recommendation information is larger than that of the first recommendation information;
if not, determining that the test result is as follows: the effect of the first recommendation information is better;
if yes, determining that the test result is: the effect of the second recommendation information is better.
7. An information recommendation testing device, comprising:
the data receiving module is used for receiving copy data from an online server, wherein the copy data is obtained by copying access request data received by the online server from a terminal;
a first operation amount receiving module, configured to receive an operation amount of first recommendation information obtained by the online server by using the access request data based on an online first information recommendation algorithm at the terminal;
the information recommendation module is used for obtaining second recommendation information based on an offline second information recommendation algorithm by using the copy data;
a known operation quantity query module, configured to query a historical database according to the copy data, and determine an operation quantity of an initiator of the access request data on known recommendation information;
the second operation amount determining module is used for determining the operation amount of the second recommendation information on the terminal according to the operation amount of the initiator of the access request data on the known recommendation information;
and the test result determining module is used for determining a test result according to a comparison result of the operation amount of the first recommendation information and the second recommendation information.
8. The test apparatus of claim 7, wherein the test apparatus further comprises:
the information comparison module is used for judging whether the second recommendation information is the same as the first recommendation information;
the operation amount equivalent module is used for taking the operation amount of the first recommendation information on the terminal as the operation amount of the second recommendation information on the terminal when the second recommendation information is the same as the first recommendation information;
the known operation quantity query module is specifically configured to:
and only when the second recommendation information is different from the first recommendation information, inquiring a historical database according to the copy data, and determining the operation amount of the initiator of the access request data on the known recommendation information.
9. The test apparatus as claimed in claim 7, wherein the known operand query module is specifically configured to:
inquiring a historical database according to the copied data, and judging whether the operation amount of an initiator of the access request data on target recommendation information exists in the historical database, wherein the target recommendation information is known recommendation information with the same type as the second recommendation information;
if so, taking the operation quantity of the initiator of the access request data on the target recommendation information as the operation quantity of the initiator on the known recommendation information;
if not, taking the default user operation amount of the initiator of the access request data as the operation amount of the initiator on the known recommendation information; the default user operation amount is the average value of the operation amounts of the user on all known recommendation information.
10. The test apparatus of claim 9, wherein the known operand querying module is further configured to:
after determining that the operation quantity of the initiator of the access request data to the target recommendation information does not exist in the history database, judging whether the default user operation quantity of the initiator of the access request data is empty or not before taking the default user operation quantity as the operation quantity of the initiator to the known recommendation information;
if not, taking the default user operation amount as the operation amount of the initiator on the known recommendation information;
if so, taking the default group operation amount of the initiator of the access request data as the operation amount of the initiator on the known recommendation information; the default group operation amount is the average value of the operation amounts of all known initiators on the known recommendation information.
11. The testing apparatus of claim 7, wherein the first recommendation information and the second recommendation information each have a resource consumption amount;
the test result determining module is specifically configured to:
determining the total resource consumption amount of the first recommendation information according to the operation amount of the first recommendation information and the resource consumption amount of the first recommendation information of unit operation amount;
determining the total resource consumption amount of the second recommendation information according to the operation amount of the second recommendation information and the resource consumption amount of the second recommendation information of unit operation amount;
judging whether the total resource consumption amount of the second recommendation information is larger than that of the first recommendation information;
if not, determining that the test result is as follows: the effect of the first recommendation information is better;
if yes, determining that the test result is: the effect of the second recommendation information is better.
12. An electronic device, comprising:
a processor;
a memory for storing processor-executable instructions;
the processor is configured to:
receiving copy data from an online server, wherein the copy data is obtained by copying access request data received by the online server from a terminal;
receiving the operation amount of first recommendation information obtained by the online server by using the access request data based on an online first information recommendation algorithm at the terminal;
obtaining second recommendation information based on an offline second information recommendation algorithm by using the copy data;
inquiring a historical database according to the copy data, and determining the operation amount of an initiator of the access request data on the known recommendation information;
determining the operation amount of the second recommendation information at the terminal according to the operation amount of the initiator of the access request data on the known recommendation information;
and determining a test result according to the comparison result of the operation amount of the first recommendation information and the second recommendation information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710024224.XA CN106817296B (en) | 2017-01-12 | 2017-01-12 | Information recommendation test method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710024224.XA CN106817296B (en) | 2017-01-12 | 2017-01-12 | Information recommendation test method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106817296A CN106817296A (en) | 2017-06-09 |
| CN106817296B true CN106817296B (en) | 2020-04-14 |
Family
ID=59110849
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710024224.XA Active CN106817296B (en) | 2017-01-12 | 2017-01-12 | Information recommendation test method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106817296B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110610377A (en) * | 2019-08-09 | 2019-12-24 | 微梦创科网络科技（中国）有限公司 | Advertisement effect testing method and system |
| CN111190801A (en) * | 2019-12-19 | 2020-05-22 | 广州华多网络科技有限公司 | Recommendation system testing method and device and electronic equipment |
| CN113129061B (en) * | 2021-04-20 | 2023-07-18 | 微梦创科网络科技（中国）有限公司 | Advertisement index verification method and system |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150012852A1 (en) * | 2013-07-08 | 2015-01-08 | Kobo Incorporated | User interface tool for planning an ab type of test |
| CN103559303A (en) * | 2013-11-15 | 2014-02-05 | 南京大学 | Evaluation and selection method for data mining algorithm |
| CN105488107A (en) * | 2015-11-20 | 2016-04-13 | 天津大学 | Offline evaluation method for recommendation system |
| CN105608121B (en) * | 2015-12-14 | 2020-09-25 | 东软集团股份有限公司 | Personalized recommendation method and device |
| CN105630946B (en) * | 2015-12-23 | 2019-03-19 | 百度在线网络技术（北京）有限公司 | A kind of field intersection recommended method and device based on big data |
| CN106027329A (en) * | 2016-05-16 | 2016-10-12 | 乐视控股（北京）有限公司 | Push service testing method and device |
| CN106157151A (en) * | 2016-06-29 | 2016-11-23 | 厦门趣处网络科技有限公司 | A kind of information aggregation method based on user interest, system |
-
2017
- 2017-01-12 CN CN201710024224.XA patent/CN106817296B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN106817296A (en) | 2017-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11119794B2 (en) | Mobile application activity detector | |
| US8984151B1 (en) | Content developer abuse detection | |
| US11556955B2 (en) | Systems and methods for leveraging social queuing to identify and prevent ticket purchaser simulation | |
| CN107305611B (en) | Method and device for establishing model corresponding to malicious account and method and device for identifying malicious account | |
| CN109257321B (en) | Secure login method and device | |
| CN110609937A (en) | Crawler identification method and device | |
| CN106897905B (en) | Method and device for pushing information and electronic equipment | |
| CN111061956A (en) | Method and apparatus for generating information | |
| CN104580406A (en) | Method and device for synchronizing login status | |
| CN109510874A (en) | Electronic certificate method for pushing, device and electronic equipment based on LBS | |
| CN106817296B (en) | Information recommendation test method and device and electronic equipment | |
| CN111080374A (en) | Test method of advertisement delivery strategy, bidding server and advertisement delivery system | |
| CN108805332B (en) | Feature evaluation method and device | |
| CN111008059A (en) | Control method and device for popup window display, terminal and storage medium | |
| US10885565B1 (en) | Network-based data discovery and consumption coordination service | |
| US10755307B2 (en) | Systems and methods for leveraging social queuing to simulate ticket purchaser behavior | |
| CN107634942B (en) | Method and device for identifying malicious request | |
| CN109804349B (en) | System and method for reducing download requirements | |
| CN113609516B (en) | Information generation method and device based on abnormal user, electronic equipment and medium | |
| CN108229127B (en) | System and method for generating authentication data in advance to distinguish clients | |
| WO2017155589A1 (en) | Weighted reviews of applications based on usage history | |
| CN115221442A (en) | Webpage loading method and device, computer equipment and computer readable storage medium | |
| US20160337195A1 (en) | Automated multi-user system detection | |
| KR102005932B1 (en) | Server for managing social network friends and method for managing social network friends using the same | |
| CN109685561B (en) | Electronic certificate pushing method and device based on user behavior and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |