CN106817296A - The method of testing of information recommendation, device and electronic equipment - Google Patents
The method of testing of information recommendation, device and electronic equipment Download PDFInfo
- Publication number
- CN106817296A CN106817296A CN201710024224.XA CN201710024224A CN106817296A CN 106817296 A CN106817296 A CN 106817296A CN 201710024224 A CN201710024224 A CN 201710024224A CN 106817296 A CN106817296 A CN 106817296A
- Authority
- CN
- China
- Prior art keywords
- recommendation information
- operational ton
- information
- recommendation
- initiator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/21—Monitoring or handling of messages
- H04L51/212—Monitoring or handling of messages using filtering or selective blocking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/50—Testing arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/21—Monitoring or handling of messages
- H04L51/214—Monitoring or handling of messages using selective forwarding
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The embodiment of the present application discloses a kind of method of testing of information recommendation algorithm, device and electronic equipment, and the method includes：The copy data from aol server are received, copy data are as obtained by copying the access request data that aol server is received from terminal；Receive by aol server utilize the access request data, based on the first recommendation information obtained by the first information proposed algorithm on line terminal operational ton；Using data are copied, the second recommendation information is obtained based on the second information recommendation algorithm under line；According to copy data, enquiry of historical data storehouse determines the operational ton of the initiator to known recommendation information of access request data；Initiator according to access request data determines operational ton of second recommendation information in terminal to the operational ton of known recommendation information；According to the first recommendation information and the comparison result of the operational ton of the second recommendation information, test result is determined.Both ensure test accuracy, the problem of resource waste of service providing end is solved again.
Description
Technical field
The application is related to Internet technical field, more particularly to a kind of method of testing of information recommendation, device and electronics
Equipment.
Background technology
With the fast development of Internet technology, the information recommendation service based on internet is also more popularized, common quilt
The information of recommendation has news, social information, advertisement etc..
Current information recommendation service is realized usually in the following way：Service providing end is sent receiving user
Access request after, first will collection in advance the default information recommendation algorithm of data input such as user identity, operating habit, lead to
Cross these data of information recommendation algorithm process to determine information to be recommended, the information pushing that will finally determine to user is held
Terminal.As can be seen here, the performance of information recommendation algorithm is particularly important for the effect of information recommendation.
In the prior art, before information recommendation algorithm application, the recommendation effect of the algorithm can be tested, with basis
Test result determines whether to apply.Common method of testing has AB to shunt method of testing, and its process is as follows：First by be applied
One information recommendation algorithm is concurrently set on service line where information recommendation service with the second information recommendation algorithm applied；Afterwards
It is continuous, when receiving the access request of user, the correlative flow of the access request data is directed into the first or second information at random
Proposed algorithm；After maintaining certain hour, according to user to the operational ton of first, second information recommendation algorithm institute recommendation information come really
Whether effect is more excellent for fixed first proposed algorithm.
However, when shunting method of testing come the recommendation effect for testing information recommendation algorithm using AB, need to be by first, second information
Proposed algorithm apply simultaneously information recommendation service service line on, if the recommendation effect of the first proposed algorithm to be applied compared with
Difference, user can be little to the operational ton of the first proposed algorithm institute recommendation information, and these invalid information recommendations can cause service to carry
For the wasting of resources at end, Consumer's Experience is also reduced.
The content of the invention
The purpose of the embodiment of the present application is to provide a kind of method of testing of information recommendation, device and electronic equipment, for solving
Certainly above mentioned problem.
According to the first aspect of the embodiment of the present disclosure, there is provided a kind of method of testing of information recommendation, including：
The copy data from aol server are received, the copy data are by copying the aol server from end
The access request data gained that termination is received；
Receive and the access request data is utilized by the aol server, based on the first information proposed algorithm institute on line
Operational ton of the first recommendation information for obtaining in the terminal；
Using the copy data, the second recommendation information is obtained based on the second information recommendation algorithm under line；
According to the copy data, enquiry of historical data storehouse determines that the initiator of the access request data pushes away to known
Recommend the operational ton of information；
Initiator according to the access request data determines second recommendation to the operational ton of known recommendation information
Cease the operational ton in the terminal；
According to first recommendation information and the comparison result of the operational ton of the second recommendation information, test result is determined.
In one embodiment, the copy data are copied in real time from terminal by copying code in aol server
The access request data gained for receiving.
In one embodiment, the access request data includes：Initiator's identity information, request environmental information and treat please
Seek object information.
In one embodiment, with the copy data, the second recommendation is obtained based on the second information recommendation algorithm under line
After breath, according to the copy data, before enquiry of historical data storehouse, methods described also includes：
Confirm that second recommendation information is differed with the first recommendation information；
And, methods described also includes：
If it is judged that second recommendation information is identical with the first recommendation information, with first recommendation information described
The operational ton of terminal as second recommendation information the terminal operational ton.
In one embodiment, according to the copy data, enquiry of historical data storehouse determines the hair of the access request data
Operational ton of the side to known recommendation information is played, is specifically included：
According to the copy data, enquiry of historical data storehouse is judged in historical data base with the presence or absence of the access request
To the operational ton of target recommendation information, the target recommendation information is type and the second recommendation information identical to the initiator of data
Known recommendation information；
If so, using the initiator of the access request data to the operational ton of target recommendation information as the initiator to
Know the operational ton of recommendation information；
If it is not, using the default user operational ton of the initiator of the access request data as the initiator to known recommendation
The operational ton of information；The default user operational ton is average of the user to the operational ton of all known recommendation informations.
In one embodiment, determine that the initiator in historical data base in the absence of the access request data recommends target
After the operational ton of information, before the operational ton using default user operational ton as the initiator to known recommendation information, institute
Stating method of testing also includes：
The default user operational ton for confirming the initiator of the access request data is not sky；
And, methods described also includes：
If judging, the default user operational ton of the initiator of the access request data is sky, with the access request number
According to initiator default group body operational ton as the initiator to the operational ton of known recommendation information；The default group gymnastics is made
Measure the average of the operational ton for all known initiators to known recommendation information.
In one embodiment, first recommendation information and the second recommendation information are advertising message, the advertising message
It is exposure series advertisements or display series advertisements；
When the advertising message is for exposure series advertisements, the operational ton is light exposure；
When the advertising message is to click on series advertisements, the operational ton is click volume.
In one embodiment, first recommendation information and the second recommendation information are respectively provided with consumed resource；
According to first recommendation information and the comparison result of the operational ton of the second recommendation information, test result is determined, have
Body includes：
The consumed resource of the first recommendation information of operational ton and unit operation amount according to first recommendation information, really
The resource consumption total amount of fixed first recommendation information；
The consumed resource of the second recommendation information of operational ton and unit operation amount according to second recommendation information, really
The resource consumption total amount of fixed second recommendation information；
Judge whether the resource consumption total amount of second recommendation information is more than the resource consumption total amount of the first recommendation information；
If it is not, determining that test result is：The effect of first recommendation information is more excellent；
If so, determining that test result is：The effect of second recommendation information is more excellent.
According to the second aspect of the embodiment of the present disclosure, there is provided a kind of test device of information recommendation, including：
Data reception module, for receiving the copy data from aol server, the copy data are by copy
The access request data gained that the aol server is received from terminal；
First operational ton receiver module, for receiving by the aol server using the access request data, is based on
The operational ton of the first recommendation information obtained by first information proposed algorithm on line in the terminal；
Information recommendation module, for utilizing the copy data, second is obtained based on the second information recommendation algorithm under line
Recommendation information；
Known operation amount enquiry module, for according to the copy data, enquiry of historical data storehouse to determine that the access please
The initiator of data is asked to the operational ton of known recommendation information；
Second operational ton determining module, for the initiator according to the access request data to the behaviour of known recommendation information
Measure, determine operational ton of second recommendation information in the terminal；
Test result determining module, for the comparison according to first recommendation information and the operational ton of the second recommendation information
As a result, test result is determined.
In one embodiment, the copy data are copied in real time from terminal by copying code in aol server
The access request data gained for receiving.
In one embodiment, the access request data includes：Initiator's identity information, request environmental information and treat please
Seek object information.
In one embodiment, the test device also includes：
Information comparison module, for judging whether second recommendation information is identical with the first recommendation information；
Operational ton equivalent module, for when second recommendation information is identical with the first recommendation information, with described first
Recommendation information the terminal operational ton as second recommendation information the terminal operational ton；
The known operation amount enquiry module, specifically for：
Only when second recommendation information and the first recommendation information are differed, according to the copy data, query history
Database, determines the operational ton of the initiator to known recommendation information of the access request data.
In one embodiment, the known operation amount enquiry module, specifically for：
According to the copy data, enquiry of historical data storehouse is judged in historical data base with the presence or absence of the access request
To the operational ton of target recommendation information, the target recommendation information is type and the second recommendation information identical to the initiator of data
Known recommendation information；
If so, using the initiator of the access request data to the operational ton of target recommendation information as the initiator to
Know the operational ton of recommendation information；
If it is not, using the default user operational ton of the initiator of the access request data as the initiator to known recommendation
The operational ton of information；The default user operational ton is average of the user to the operational ton of all known recommendation informations.
In one embodiment, the known operation amount enquiry module, is additionally operable to：
Determine do not exist the operational ton of the initiator to target recommendation information of the access request data in historical data base
Afterwards, before the operational ton using default user operational ton as the initiator to known recommendation information, judge that the access please
Whether the default user operational ton for asking the initiator of data is empty；
If it is not, using default user operational ton as the initiator to the operational ton of known recommendation information；
If so, using the default group body operational ton of the initiator of the access request data as the initiator to known recommendation
The operational ton of information；The default group body operational ton is the average of all known operational tons of the initiator to known recommendation information.
In one embodiment, first recommendation information and the second recommendation information are respectively provided with consumed resource；
The test result determining module, specifically for：
The consumed resource of the first recommendation information of operational ton and unit operation amount according to first recommendation information, really
The resource consumption total amount of fixed first recommendation information；
The consumed resource of the second recommendation information of operational ton and unit operation amount according to second recommendation information, really
The resource consumption total amount of fixed second recommendation information；
Judge whether the resource consumption total amount of second recommendation information is more than the resource consumption total amount of the first recommendation information；
If it is not, determining that test result is：The effect of first recommendation information is more excellent；
If so, determining that test result is：The effect of second recommendation information is more excellent.
According to the third aspect of the embodiment of the present disclosure, there is provided a kind of electronic equipment, including：
Processor；
Memory for storing processor-executable instruction；
The processor is configured as：
The copy data from aol server are received, the copy data are by copying the aol server from end
The access request data gained that termination is received；
Receive and the access request data is utilized by the aol server, based on the first information proposed algorithm institute on line
Operational ton of the first recommendation information for obtaining in the terminal；
Using the copy data, the second recommendation information is obtained based on the second information recommendation algorithm under line；
According to the copy data, enquiry of historical data storehouse determines that the initiator of the access request data pushes away to known
Recommend the operational ton of information；
Initiator according to the access request data determines second recommendation to the operational ton of known recommendation information
Cease the operational ton in the terminal；
According to first recommendation information and the comparison result of the operational ton of the second recommendation information, test result is determined.
The technical scheme provided by this disclosed embodiment can include the following benefits：
After the first information proposed algorithm treatment access request data on line obtains the first recommendation information, first is obtained
Operational ton of the recommendation information in terminal；It is consistent with access request data to process by the second information recommendation algorithm under line again
Copy data obtain the second recommendation information, the then operation by the initiator of access request data to known recommendation information
Measure to determine the operational ton of the second recommendation information, the operational ton finally according to the first recommendation information and the second recommendation information determines to survey
Test result.Second information recommendation algorithm is not on service line, it is ensured that while test accuracy, also solves probably due to second
The recommendation effect of proposed algorithm is poor and causes the problem of the wasting of resources of service providing end.
It should be appreciated that the general description of the above and detailed description hereinafter are only exemplary and explanatory, not
The disclosure can be limited.
Brief description of the drawings
In order to illustrate more clearly of the embodiment of the present application or technical scheme of the prior art, below will be to embodiment or existing
The accompanying drawing to be used needed for having technology description is briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments described in application, for those of ordinary skill in the art, are not paying the premise of creative labor
Under, other accompanying drawings can also be obtained according to these accompanying drawings.
Fig. 1 is the schematic diagram of the system that the method for testing according to an exemplary embodiment is applicable.
Fig. 2 is a kind of flow chart of the method for testing according to an exemplary embodiment.
Fig. 3 is the flow chart of another method of testing according to an exemplary embodiment.
Fig. 4 is a kind of Organization Chart of the historical data base according to an exemplary embodiment.
Fig. 5 is the flow chart of another method of testing according to an exemplary embodiment.
Fig. 6 is a kind of module map of the test device according to an exemplary embodiment.
Fig. 7 is the module map of another test device according to an exemplary embodiment.
Fig. 8 is the block diagram of the electronic equipment suitable for test of heuristics according to an exemplary embodiment.
Specific embodiment
In order that those skilled in the art more fully understand the technical scheme in the application, below in conjunction with the application reality
The accompanying drawing in example is applied, the technical scheme in the embodiment of the present application is clearly and completely described, it is clear that described implementation
Example is only some embodiments of the present application, rather than whole embodiments.Based on the embodiment in the application, this area is common
The every other embodiment that technical staff is obtained under the premise of creative work is not made, should all belong to the application protection
Scope.
Fig. 1 is the schematic diagram of the system that the method for testing according to an exemplary embodiment is applicable.As shown in figure 1,
The system can include：The terminal 10 that user is held and terminal 10 set up by network the aol server 20 that communicates and
Server 30 under the line that communicates is set up by network with aol server 20.Terminal 10 can be included but is not limited to：Mobile phone, flat board
Computer, PC equipment etc..Under connecting terminal 10 with the network of aol server 20 and for being connected aol server 20 and line
The network of server 30 may each comprise：Wireless network or the utilizations such as WI-FI (WIreless-Fidelity, Wireless Fidelity), bluetooth
The cable network of cable transmission information.If under aol server 20 and line server 30 may each be a server or by
Dry platform server group into server cluster or be a cloud computing service center, or even say, under aol server 20 and line
Server 30 can be integrated among same equipment on hardware, will not be described here.
Terminal 10 can install such as customer end A pp such as browser, social networking application, Video Applications or Domestic News application,
Aol server 20 is the offer end of the online service corresponding to foregoing client, and aol server 20 stores corresponding client
Resource needed for the operation of end, user can open corresponding customer end A pp in terminal 10 and ask to access to aol server 20
Certain page, the response access request of aol server 20 simultaneously treats the data is activation of requests for page to terminal 10, and terminal 10 parses institute
The data that receive simultaneously show on the client.
The access request of the response user of aol server 20 and by the process of the data is activation of the page to be visited to terminal 10
In, can also add recommendation information in the data of the page to be visited so that the page to be visited that terminal 10 shows on the client
Recommendation information is carried in face.
The connection of server 30 aol server 20 under line, information recommendation algorithm under the memory storage of server 30 is wired under line with
And for testing the test application of the information recommendation algorithm, before server 30 can realize test by foregoing test application under line
State whether the information recommendation algorithm under line has preferably recommendation relative to the information recommendation algorithm in aol server 20
Effect.
Below by based on the system shown in above-mentioned Fig. 1, each embodiment to the method for testing based on server under line 30 is entered
Row narration.
Fig. 2 is a kind of flow chart of the method for testing according to an exemplary embodiment.Method of testing in the present embodiment
Using on server under line 30, can start and terminate by installing corresponding test application in the aol server 20
Aforementioned test method.As shown in Fig. 2 in the present embodiment, the method for testing can comprise the steps 101~106.The test side
The order of each step is not limited to step 101 to 106, such as step 102 any position before may be selected in step 106 in method
Put, will not be described here.
In a step 101, the copy data from aol server are received, the copy data are by copying the line
The access request data gained that upper server is received from terminal.
When client accesses certain page to the request of aol server 20 in user's operation terminal 10, client can basis
Operation behavior of the user in the client in terminal 10 carries access request number come the access request for generating in access request
According to.Subsequently, terminal 10 is sent to aol server 20 after can access request data be encapsulated.
Access request is produced when being interacted with aol server 20 including terminal 10 by terminal 10 and is sent to aol server
20 all kinds of requests, for example, can be used for asking the logging request of login account or for asking to access for example social information
The access request of the pages such as the page, news pages or the searching request for request search destination object etc..
Access request data can include it is following at least one：The identity data of requesting party, request environmental data and treat
Request object data.Can determine to return to the service content of client by the data of these types.
The identity data of requesting party is used for the identity that the user of the client is registered in description, for example, can include customer account
Number, customer's identity card number, licence number, bank card number, client location, client's purchaser record, client's collection record,
Client's concern record and client draw black recording.Unique requesting party can be determined by these data.Request environmental data is used
The relevant parameter of hardware device and software program, the ginseng of hardware device are used when user's operation client initiation request is described
Number can be the MAC Address of terminal 10, UMID yards, SIM number, IMEI code, IP address, VPN addresses and unit type etc., it is soft
The parameter of part program can be OS Type, browser type, client release etc..Object data to be asked is used to describe
The information of object to be asked, for example, can be type, address of object to be asked etc..
After server 20 receives access request data on line, the access request data is copied as copying in real time
Shellfish data, then server 30 under data is activation to line will be copied.In the present embodiment, can the memory storage stream of server 20 on line
Amount copy code, after server 20 receives access request data on line, operating flux copy code is realized access request
Data copy.Because copy data are identical with the content of access request data so that aol server 20 and online service
When information recommendation algorithm is compared in device 30, with identical input data.
In a step 102, receive and the access request data is utilized by the aol server, based on the first letter on line
Cease operational ton of the first recommendation information obtained by proposed algorithm in the terminal.
The internal memory of aol server 20 contains first information proposed algorithm, and first information proposed algorithm is asked positioned at online service
On the service line of place, access request data is into can be imported into first information proposed algorithm after aol server 20.By
Treatment of the one information recommendation algorithm to access request data, can obtain the first recommendation information.First recommendation information can be wide
Accuse or other kinds of information.
The trend of service line is followed, aol server 20 can be also generated for returning to terminal 10 according to access request data
Response data.Such as service request is the access request of the social information page, then response data is then social money to be visited
The page data of the page is interrogated, terminal 10 is rendered after receiving page data, to show the corresponding page on the client.At this
In embodiment, aol server 20 can be latter by page data and the encapsulation of the first recommendation information and be sent to terminal 10.
First recommendation information can be click on category information or exposure category information, for example, can be click on series advertisements information or exposure
Light series advertisements information.In the first recommendation information to click on category information, user can be touched by finger, keyboard is selected, mouse is selected
Medium mode actively clicks on the first recommendation information, checking the particular content of the first recommendation information；It is in the first recommendation information
Exposure category information, the first recommendation information can also be exposed on the page with the displaying of information to be visited so that user can
See first recommendation information.To different types of recommendation information, the definition of operational ton is also different, to click on the first recommendation of class
Information is then its operational ton by the number of times of user's active operation, for the first recommendation information for exposing class is exposed on display interface
Number of times then be its operational ton.
In the present embodiment, no matter how the type of the first recommendation information changes, can be by obtaining come on self terminal 10
The Operation Log of client, then can quickly determine the operational ton of the first recommendation information.
In step 103, using the copy data, the second recommendation is obtained based on the second information recommendation algorithm under line
Breath.
The internal memory of server 30 contains the second information recommendation algorithm under line, and the second information recommendation algorithm is to wait to reach the standard grade or to be applied
Algorithm, the second information recommendation algorithm can be done on the basis of first information proposed algorithm upgrade gained, the recommendation of the two
Difference on effect is to be determined.
After server 30 receives copy data from aol server 20 under the line, copy data are directed into the second information
Proposed algorithm, by obtaining the second recommendation information after the second information recommendation algorithm process, the second recommendation information can be pushed away with first
Recommend information identical or different.
At step 104, according to the copy data, enquiry of historical data storehouse determines the hair of the access request data
Play operational ton of the side to known recommendation information.
The multigroup historical data of historical data base memory storage, identity information including user of every group of historical data and with this
The historical operation record of identity information association, operational ton of the user to known recommendation information is record in historical operation record,
The operational ton of known recommendation information here can be simultaneously pre- to being obtained in the Operation Log of recommendation information from aol server 20
Deposit.
In the present embodiment, this can be filtered out in historical data base by copying initiator's identity information in data
Operational ton of the initiator to known recommendation information.
In step 105, the initiator according to the access request data determines institute to the operational ton of known recommendation information
State operational ton of second recommendation information in the terminal.
Speculate same initiator to the operational ton of known recommendation information by the initiator of access request data
The operational ton of two recommendation informations, when there are currently no actual services flow in fact, determines the operational ton of the second recommendation information.
In step 106, according to first recommendation information and the comparison result of the operational ton of the second recommendation information, it is determined that
Test result.
The operational ton of recommendation information can reflect matching degree of the recommendation information to user's request, the recommendation of quantification
The operational ton of information is (i.e. the arrival rate of recommendation information is higher) higher, then illustrate that the recommendation information is got over and meet user's request, and this is pushed away
The recommendation effect for recommending information is more preferable.
As service traffics constantly enter aol server 20, aol server 20 forms substantial amounts of first recommendation information,
Server 30 also forms same amount of second recommendation information under isochrone, is pushed away by comparing these first recommendation informations and second
Recommend the operational ton of information, it may be determined that the height of the arrival rate of the first recommendation information and the second recommendation information.In the present embodiment,
If the operational ton of the second recommendation information is lower, determine that test result is：The effect of first recommendation information is more excellent；If second pushes away
The operational ton for recommending information is higher, determines that test result is：The effect of second recommendation information is more excellent.
In the present embodiment, during generating and recommending the first recommendation information and the second recommendation information, aol server
20 and line under server 30 produce resource consumption.The content of this resource consumption and the two recommendation informations, initiator are homogeneous
Close.Then step 106 can further be adjusted to：
The consumed resource of the first recommendation information of operational ton and unit operation amount according to first recommendation information, really
The resource consumption total amount of fixed first recommendation information.
The consumed resource of the second recommendation information of operational ton and unit operation amount according to second recommendation information, really
The resource consumption total amount of fixed second recommendation information.
Judge whether the resource consumption total amount of second recommendation information is more than the resource consumption total amount of the first recommendation information.
If it is not, determining that test result is：The effect of first recommendation information is more excellent；
If so, determining that test result is：The effect of second recommendation information is more excellent.
The resource consumption that the recommendation information of unit operation amount the is brought i.e. recommendation information is operated to related service every time
Resource consumption produced by device.By introducing the consumed resource that is brought of unit operation amount on the basis of operational ton, can be with
The two multiplication is obtained the resource consumption total amount of each recommendation information, the recommendation of the more complete recommendation information of assessment second is realized
Effect.
In actual applications, the first recommendation information and the second recommendation information may each be advertisement, can be by advertisement recommendation
The resource consumption monetization and hypostazation of breath, the consumed resource of the recommendation information of unit operation amount can be advertisement quilt every time
During operation, initiator pays the advertising expense of service provider, then can obtain the advertisement after big magnitude recommendation information is operated
Take total value, specific aim assesses the effect of commercial paper information recommendation accordingly.
To sum up, the method for testing of the information recommendation algorithm that the embodiment of the present application is provided, is pushed away by the first information on line
Recommend after algorithm process access request data obtains the first recommendation information, obtain operational ton of first recommendation information in terminal；Again
The copy data consistent with access request data are processed by the second information recommendation algorithm under line to obtain the second recommendation
Breath, then determines the operation of the second recommendation information by the initiator of access request data to the operational ton of known recommendation information
Amount, the operational ton finally according to the first recommendation information and the second recommendation information determines test result.Second information recommendation algorithm is not
On service line, it is ensured that while test accuracy, also solve probably due to the recommendation effect of the second proposed algorithm it is poor and
Cause the problem of the wasting of resources of service providing end.
Fig. 3 is the flow chart of another method of testing according to an exemplary embodiment.The method of testing can be answered
With on online lower server 30.As shown in figure 3, method of testing includes step 201 to 206.
In step 201, the copy data from aol server are received, the copy data are by copying the line
The access request data gained that upper server is received from terminal.
In step 202., receive and the access request data is utilized by the aol server, based on the first letter on line
Cease operational ton of the first recommendation information obtained by proposed algorithm in the terminal.
In step 203, using the copy data, the second recommendation is obtained based on the second information recommendation algorithm under line
Breath.
Step 204 may include steps of 241 to 243.
In step 241, according to the copy data, enquiry of historical data storehouse judges to whether there is in historical data base
To the operational ton of target recommendation information, the target recommendation information is that type is pushed away with second to the initiator of the access request data
Recommendation information known to information identical is recommended, if so, into step 242, if it is not, into step 243.
In step 242, using the initiator of the access request data for target recommendation information operational ton as this
Operational ton of the initiator to known recommendation information.
In step 243, whether the default user operational ton for judging the initiator of the access request data is empty, if
It is no, into step 244, if so, into step 245.
In step 244, using the default user operational ton of the initiator of the access request data as the initiator couple
The operational ton of known recommendation information.
In step 245, the default group body operational ton of the initiator of the access request data as the initiator to
Know the operational ton of recommendation information.
It is worth noting that, in other embodiments of the invention, after step 243 is entered, optional step 244 and step
Any one is used as subsequent step in 245.Not as defined in the present embodiment, step 244 is must simultaneously have after step 243
With step 245, will not be described here.
Fig. 4 is a kind of Organization Chart of the historical data base according to an exemplary embodiment.As shown in figure 4, in one kind
In exemplary scenario, historical data base is in the form of hierarchical tree, and orlop is initiator to different types of known recommendation information
Operational ton；Upper strata is the default user operational ton of the initiator again, and the default user operational ton is the user to all types
Known recommendation information operational ton average；The superiors are the default group body operational tons of the initiator, and the default group gymnastics is made
Measure the average of the operational ton for all known initiators to known recommendation information.
In operational ton of the execution step 204 determination initiator to known recommendation information, first from historical data base most
Whether low layer starts, confirm the initiator to carrying out operation to target recommendation information using the identity information of initiator.True
Whether the fixed initiator did not carry out operation to target recommendation information, then continue, toward upper strata, to determine the initiator to any class
The known recommendation information of type carried out operation, it is determined that carried out operation in the presence of to any type of recommendation information, with the user
To the average of the operational ton of all types of known recommendation informations user's operational ton by default, the default user operational ton is made
It is the initiator to the operational ton of known recommendation information.It is determined that the initiator is not carried out to any kind of known recommendation information
Operation is crossed, is then continued up, with all initiators to the average of the operational ton of known recommendation information colony's operational ton by default,
Using the default group body operational ton as the initiator to the operational ton of known recommendation information.
In step 205, the initiator according to the access request data determines institute to the operational ton of known recommendation information
State operational ton of second recommendation information in the terminal.
In step 206, according to first recommendation information and the comparison result of the operational ton of the second recommendation information, it is determined that
Test result.
Fig. 5 is the flow chart of another method of testing according to an exemplary embodiment.The method of testing can be answered
With on online lower server 30.As shown in figure 5, method of testing includes step 301 to 308：
In step 301, the copy data from aol server are received, the copy data are by copying the line
The access request data gained that upper server is received from terminal.
In step 302, receive and the access request data is utilized by the aol server, based on the first letter on line
Cease operational ton of the first recommendation information obtained by proposed algorithm in the terminal.
In step 303, using the copy data, the second recommendation is obtained based on the second information recommendation algorithm under line
Breath.
In step 304, judge whether second recommendation information is identical with the first recommendation information, if so, into step
305, if it is not, into step 306.
When the second recommendation information is identical with the first recommendation information, the operational ton of the first recommendation information can be directly used,
It is predicted without the operational ton again to the second recommendation information.
In step 305, using first recommendation information the terminal operational ton as second recommendation information
In the operational ton of the terminal.
Within step 306, according to the copy data, enquiry of historical data storehouse determines the hair of the access request data
Play operational ton of the side to known recommendation information.
It is worth noting that, in other embodiments of the invention, after step 304 is entered, optional step 305 and step
Any one is used as subsequent step in 306.Not as defined in the present embodiment, step 305 is must simultaneously have after step 304
With step 306, will not be described here.
In step 307, the initiator according to the access request data determines institute to the operational ton of known recommendation information
State operational ton of second recommendation information in the terminal.
In step 308, according to first recommendation information and the comparison result of the operational ton of the second recommendation information, it is determined that
Test result.
Fig. 6 is a kind of module map of the test device according to an exemplary embodiment.The test device can be used for
Online on lower server 30, in the present embodiment, test device can include：
Data reception module 401, for receiving the copy data from aol server, the copy data are by copying
The access request data gained that aol server described in shellfish is received from terminal；
First operational ton receiver module 402, for receiving by the aol server using the access request data, base
The operational ton of the first recommendation information obtained by the first information proposed algorithm on line in the terminal；
Information recommendation module 403, for utilizing the copy data, the is obtained based on the second information recommendation algorithm under line
Two recommendation informations；
Known operation amount enquiry module 404, for according to the copy data, enquiry of historical data storehouse to determine the visit
Ask the operational ton of the initiator to known recommendation information of request data；
Second operational ton determining module 405, for the initiator according to the access request data to known recommendation information
Operational ton, determine operational ton of second recommendation information in the terminal；
Test result determining module 406, for the operational ton according to first recommendation information and the second recommendation information
Comparison result, determines test result.
In one embodiment, the copy data are copied in real time from terminal by copying code in aol server
The access request data gained for receiving.
In one embodiment, the access request data includes：Initiator's identity information, request environmental information and treat please
Seek object information.
In one embodiment, first recommendation information and the second recommendation information are advertising message, the advertising message
It is exposure series advertisements or display series advertisements；
When the advertising message is for exposure series advertisements, the operational ton is light exposure；
When the advertising message is to click on series advertisements, the operational ton is click volume.
In one embodiment, first recommendation information and the second recommendation information are respectively provided with consumed resource；
The test result determining module 406, specifically for：
The consumed resource of the first recommendation information of operational ton and unit operation amount according to first recommendation information, really
The resource consumption total amount of fixed first recommendation information；
The consumed resource of the second recommendation information of operational ton and unit operation amount according to second recommendation information, really
The resource consumption total amount of fixed second recommendation information；
Judge whether the resource consumption total amount of second recommendation information is more than the resource consumption total amount of the first recommendation information；
If it is not, determining that test result is：The effect of first recommendation information is more excellent；
If so, determining that test result is：The effect of second recommendation information is more excellent.
In one embodiment it is known that operational ton enquiry module 404, specifically for：
According to the copy data, enquiry of historical data storehouse is judged in historical data base with the presence or absence of the access request
To the operational ton of target recommendation information, the target recommendation information is type and the second recommendation information identical to the initiator of data
Known recommendation information；
If so, using the initiator of the access request data to the operational ton of target recommendation information as the initiator to
Know the operational ton of recommendation information；
If it is not, using the default user operational ton of the initiator of the access request data as the initiator to known recommendation
The operational ton of information.
In one embodiment, the default user operational ton is the user to the equal of the operational ton of all known recommendation informations
Value.
In one embodiment, the known operation amount enquiry module 404, is additionally operable to：
Determine do not exist the operational ton of the initiator to target recommendation information of the access request data in historical data base
Afterwards, before the operational ton using default user operational ton as the initiator to known recommendation information, judge that the access please
Whether the default user operational ton for asking the initiator of data is empty；
If it is not, using default user operational ton as the initiator to the operational ton of known recommendation information；
If so, using the default group body operational ton of the initiator of the access request data as the initiator to known recommendation
The operational ton of information.
In one embodiment, the default group body operational ton is operational ton of all known initiators to known recommendation information
Average.
To sum up, the test device of the information recommendation algorithm that the embodiment of the present application is provided, is pushed away by the first information on line
Recommend after algorithm process access request data obtains the first recommendation information, obtain operational ton of first recommendation information in terminal；Again
The copy data consistent with access request data are processed by the second information recommendation algorithm under line to obtain the second recommendation
Breath, then determines the operation of the second recommendation information by the initiator of access request data to the operational ton of known recommendation information
Amount, the operational ton finally according to the first recommendation information and the second recommendation information determines test result.Second information recommendation algorithm is not
On service line, it is ensured that while test accuracy, also solve probably due to the recommendation effect of the second proposed algorithm it is poor and
Cause the problem of the wasting of resources of service providing end.
Fig. 7 is the module map of another test device according to an exemplary embodiment.In embodiment illustrated in fig. 6
On the basis of, test device also includes information comparison module 407 and operational ton equivalent module 408.
Information comparison module 407, for judging whether second recommendation information is identical with the first recommendation information.
Operational ton equivalent module 408, for when second recommendation information is identical with the first recommendation information, with described
One recommendation information the terminal operational ton as second recommendation information the terminal operational ton.In operational ton etc.
When starting with module 408, it is known that the operational ton determining module 405 of operational ton enquiry module 404 and second does not start.It is appreciated that
In this case, the operational ton and operational ton of the first recommendation information that the first operational ton receiver module 402 is received are equal to
The operational ton of the second recommendation information that module 408 is determined is sent to test result determining module 406 and compares.
The known operation amount enquiry module 404, specifically for：
Only when second recommendation information and the first recommendation information are differed, with first recommendation information at the end
The operational ton at end as second recommendation information the terminal operational ton.Start in known operation amount enquiry module 404
When, the second operational ton determining module 405 just starts.It is appreciated that in this case, the first operational ton receiver module 402 connects
The behaviour of the second recommendation information that the operational ton of the first recommendation information for receiving and the second operational ton determining module 405 are determined
Test result determining module 406 is sent to as amount to compare.
Meanwhile, when the quantity of the targeted recommendation information of test device is more, test result determining module 406 can be from
Second operational ton determining module 405 and operational ton equivalent module 408 receive the operational ton of the second recommendation information, are grasped with from first
The operational ton of the first recommendation information that work amount receiver module 402 is received is compared, and be will not be described here.
Fig. 8 is a kind of block diagram of the electronic equipment suitable for equipment test according to an exemplary embodiment.For example,
Electronic equipment 500 may be provided in a server.Reference picture 8, electronic equipment 500 includes processing assembly 510, its further bag
One or more processors, and the memory resource as representated by memory 520 are included, can be by processing assembly 510 for storing
Execution instruction, such as application program.The application program stored in memory 520 can include one or more every
One corresponds to one group of module of instruction.The processing assembly 510 is configured as：
The copy data from aol server are received, the copy data are by copying the aol server from end
The access request data gained that termination is received；
Receive and the access request data is utilized by the aol server, based on the first information proposed algorithm institute on line
Operational ton of the first recommendation information for obtaining in the terminal；
Using the copy data, the second recommendation information is obtained based on the second information recommendation algorithm under line；
According to the copy data, enquiry of historical data storehouse determines that the initiator of the access request data pushes away to known
Recommend the operational ton of information；
Initiator according to the access request data determines second recommendation to the operational ton of known recommendation information
Cease the operational ton in the terminal；
According to first recommendation information and the comparison result of the operational ton of the second recommendation information, test result is determined.
Electronic equipment 500 can also include a power supply module 530, and power supply module 530 is configured as performing electronic equipment
500 power management, a wired or wireless network interface 540 is configured as electronic equipment 500 being connected to network, and one
Input and output (I/O) interface 550.Electronic equipment 500 can operate the operating system in memory 520 based on storage, for example
Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or similar.
In the exemplary embodiment, a kind of non-transitorycomputer readable storage medium including instructing, example are additionally provided
Such as include the memory 520 of instruction, above-mentioned instruction can be performed to complete the above method by the processing assembly 510 of electronic equipment 500.
For example, the non-transitorycomputer readable storage medium can be ROM, random access memory (RAM), CD-ROM, tape,
Floppy disk and optical data storage devices etc..
Each embodiment in this specification is described by the way of progressive, identical similar portion between each embodiment
Divide mutually referring to what each embodiment was stressed is the difference with other embodiment.Especially for system reality
Apply for example, because it is substantially similar to embodiment of the method, so description is fairly simple, related part is referring to embodiment of the method
Part explanation.
Embodiments herein is the foregoing is only, the application is not limited to.For those skilled in the art
For, the application can have various modifications and variations.It is all any modifications made within spirit herein and principle, equivalent
Replace, improve etc., within the scope of should be included in claims hereof.
Claims (12)
1. a kind of method of testing of information recommendation, it is characterised in that including：
The copy data from aol server are received, the copy data are connect from terminal by copying the aol server
The access request data gained for receiving；
Receive and the access request data is utilized by the aol server, obtained by the first information proposed algorithm on line
The first recommendation information the terminal operational ton；
Using the copy data, the second recommendation information is obtained based on the second information recommendation algorithm under line；
According to the copy data, enquiry of historical data storehouse determines the initiator of the access request data to known recommendation
The operational ton of breath；
Initiator according to the access request data determines that second recommendation information exists to the operational ton of known recommendation information
The operational ton of the terminal；
According to first recommendation information and the comparison result of the operational ton of the second recommendation information, test result is determined.
2. method of testing as claimed in claim 1, it is characterised in that utilize the copy data, based on the second letter under line
Breath proposed algorithm is obtained after the second recommendation information, and according to the copy data, before enquiry of historical data storehouse, methods described is also
Including：
Confirm that second recommendation information is differed with the first recommendation information；
And, methods described also includes：
If it is judged that second recommendation information is identical with the first recommendation information, with first recommendation information in the terminal
Operational ton as second recommendation information the terminal operational ton.
3. method of testing as claimed in claim 1, it is characterised in that according to the copy data, enquiry of historical data storehouse, really
The initiator of the fixed access request data specifically includes to the operational ton of known recommendation information：
According to the copy data, enquiry of historical data storehouse is judged in historical data base with the presence or absence of the access request data
Initiator to the operational ton of target recommendation information, the target recommendation information be type with known to the second recommendation information identical
Recommendation information；
If so, being pushed away to known as the initiator to the operational ton of target recommendation information using the initiator of the access request data
Recommend the operational ton of information；
If it is not, using the default user operational ton of the initiator of the access request data as the initiator to known recommendation information
Operational ton；The default user operational ton is average of the user to the operational ton of all known recommendation informations.
4. method of testing as claimed in claim 3, it is characterised in that determine in historical data base in the absence of the access request
After the initiator of data is to the operational ton of target recommendation information, pushed away to known as the initiator using default user operational ton
Before recommending the operational ton of information, the method for testing also includes：
The default user operational ton for confirming the initiator of the access request data is not sky；
And, methods described also includes：
If judging, the default user operational ton of the initiator of the access request data is sky, with the access request data
The default group body operational ton of initiator is as the initiator to the operational ton of known recommendation information；The default group body operational ton is
The average of all known operational tons of the initiator to known recommendation information.
5. method of testing as claimed in claim 1, it is characterised in that first recommendation information and the second recommendation information are
Advertising message, the advertising message is exposure series advertisements or display series advertisements；
When the advertising message is for exposure series advertisements, the operational ton is light exposure；
When the advertising message is to click on series advertisements, the operational ton is click volume.
6. method of testing as claimed in claim 5, it is characterised in that first recommendation information and the second recommendation information have
There is consumed resource；
According to first recommendation information and the comparison result of the operational ton of the second recommendation information, test result is determined, specific bag
Include：
Operational ton and the consumed resource of the first recommendation information of unit operation amount according to first recommendation information, determine institute
State the resource consumption total amount of the first recommendation information；
Operational ton and the consumed resource of the second recommendation information of unit operation amount according to second recommendation information, determine institute
State the resource consumption total amount of the second recommendation information；
Judge whether the resource consumption total amount of second recommendation information is more than the resource consumption total amount of the first recommendation information；
If it is not, determining that test result is：The effect of first recommendation information is more excellent；
If so, determining that test result is：The effect of second recommendation information is more excellent.
7. a kind of test device of information recommendation, it is characterised in that including：
Data reception module, for receiving the copy data from aol server, the copy data are described by copy
The access request data gained that aol server is received from terminal；
First operational ton receiver module, for receiving by the aol server using the access request data, based on line
First information proposed algorithm obtained by the first recommendation information the terminal operational ton；
Information recommendation module, for utilizing the copy data, the second recommendation is obtained based on the second information recommendation algorithm under line
Information；
Known operation amount enquiry module, for according to the copy data, enquiry of historical data storehouse to determine the access request number
According to initiator to the operational ton of known recommendation information；
Second operational ton determining module, the operation for the initiator according to the access request data to known recommendation information
Amount, determines operational ton of second recommendation information in the terminal；
Test result determining module, for the comparison knot according to first recommendation information and the operational ton of the second recommendation information
Really, test result is determined.
8. test device as claimed in claim 7, it is characterised in that the test device also includes：
Information comparison module, for judging whether second recommendation information is identical with the first recommendation information；
Operational ton equivalent module, for when second recommendation information is identical with the first recommendation information, recommending with described first
Information the terminal operational ton as second recommendation information the terminal operational ton；
The known operation amount enquiry module, specifically for：
Only when second recommendation information and the first recommendation information are differed, according to the copy data, enquiry of historical data
Storehouse, determines the operational ton of the initiator to known recommendation information of the access request data.
9. test device as claimed in claim 7, it is characterised in that the known operation amount enquiry module, specifically for：
According to the copy data, enquiry of historical data storehouse is judged in historical data base with the presence or absence of the access request data
Initiator to the operational ton of target recommendation information, the target recommendation information be type with known to the second recommendation information identical
Recommendation information；
If so, being pushed away to known as the initiator to the operational ton of target recommendation information using the initiator of the access request data
Recommend the operational ton of information；
If it is not, using the default user operational ton of the initiator of the access request data as the initiator to known recommendation information
Operational ton；The default user operational ton is average of the user to the operational ton of all known recommendation informations.
10. test device as claimed in claim 9, it is characterised in that the known operation amount enquiry module, is additionally operable to：
After determining the interior initiator in the absence of the access request data of historical data base to the operational ton of target recommendation information,
Before operational ton using default user operational ton as the initiator to known recommendation information, the access request data is judged
The default user operational ton of initiator whether be empty；
If it is not, using default user operational ton as the initiator to the operational ton of known recommendation information；
If so, using the default group body operational ton of the initiator of the access request data as the initiator to known recommendation information
Operational ton；The default group body operational ton is the average of all known operational tons of the initiator to known recommendation information.
11. test devices as claimed in claim 7, it is characterised in that first recommendation information and the second recommendation information are equal
With consumed resource；
The test result determining module, specifically for：
Operational ton and the consumed resource of the first recommendation information of unit operation amount according to first recommendation information, determine institute
State the resource consumption total amount of the first recommendation information；
Operational ton and the consumed resource of the second recommendation information of unit operation amount according to second recommendation information, determine institute
State the resource consumption total amount of the second recommendation information；
Judge whether the resource consumption total amount of second recommendation information is more than the resource consumption total amount of the first recommendation information；
If it is not, determining that test result is：The effect of first recommendation information is more excellent；
If so, determining that test result is：The effect of second recommendation information is more excellent.
12. a kind of electronic equipment, it is characterised in that including：
Processor；
Memory for storing processor-executable instruction；
The processor is configured as：
The copy data from aol server are received, the copy data are connect from terminal by copying the aol server
The access request data gained for receiving；
Receive and the access request data is utilized by the aol server, obtained by the first information proposed algorithm on line
The first recommendation information the terminal operational ton；
Using the copy data, the second recommendation information is obtained based on the second information recommendation algorithm under line；
According to the copy data, enquiry of historical data storehouse determines the initiator of the access request data to known recommendation
The operational ton of breath；
Initiator according to the access request data determines that second recommendation information exists to the operational ton of known recommendation information
The operational ton of the terminal；
According to first recommendation information and the comparison result of the operational ton of the second recommendation information, test result is determined.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710024224.XA CN106817296B (en) | 2017-01-12 | 2017-01-12 | Information recommendation test method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710024224.XA CN106817296B (en) | 2017-01-12 | 2017-01-12 | Information recommendation test method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106817296A true CN106817296A (en) | 2017-06-09 |
| CN106817296B CN106817296B (en) | 2020-04-14 |
Family
ID=59110849
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710024224.XA Active CN106817296B (en) | 2017-01-12 | 2017-01-12 | Information recommendation test method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106817296B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110610377A (en) * | 2019-08-09 | 2019-12-24 | 微梦创科网络科技（中国）有限公司 | Advertisement effect testing method and system |
| CN111190801A (en) * | 2019-12-19 | 2020-05-22 | 广州华多网络科技有限公司 | Recommendation system testing method and device and electronic equipment |
| CN113129061A (en) * | 2021-04-20 | 2021-07-16 | 微梦创科网络科技（中国）有限公司 | Advertisement index checking method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103559303A (en) * | 2013-11-15 | 2014-02-05 | 南京大学 | Evaluation and selection method for data mining algorithm |
| US20150012852A1 (en) * | 2013-07-08 | 2015-01-08 | Kobo Incorporated | User interface tool for planning an ab type of test |
| CN105488107A (en) * | 2015-11-20 | 2016-04-13 | 天津大学 | Offline evaluation method for recommendation system |
| CN105608121A (en) * | 2015-12-14 | 2016-05-25 | 东软集团股份有限公司 | Personalized recommendation method and apparatus |
| CN105630946A (en) * | 2015-12-23 | 2016-06-01 | 百度在线网络技术（北京）有限公司 | Big data based field cross recommendation method and apparatus |
| CN106027329A (en) * | 2016-05-16 | 2016-10-12 | 乐视控股（北京）有限公司 | Push service testing method and device |
| CN106157151A (en) * | 2016-06-29 | 2016-11-23 | 厦门趣处网络科技有限公司 | A kind of information aggregation method based on user interest, system |
-
2017
- 2017-01-12 CN CN201710024224.XA patent/CN106817296B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150012852A1 (en) * | 2013-07-08 | 2015-01-08 | Kobo Incorporated | User interface tool for planning an ab type of test |
| CN103559303A (en) * | 2013-11-15 | 2014-02-05 | 南京大学 | Evaluation and selection method for data mining algorithm |
| CN105488107A (en) * | 2015-11-20 | 2016-04-13 | 天津大学 | Offline evaluation method for recommendation system |
| CN105608121A (en) * | 2015-12-14 | 2016-05-25 | 东软集团股份有限公司 | Personalized recommendation method and apparatus |
| CN105630946A (en) * | 2015-12-23 | 2016-06-01 | 百度在线网络技术（北京）有限公司 | Big data based field cross recommendation method and apparatus |
| CN106027329A (en) * | 2016-05-16 | 2016-10-12 | 乐视控股（北京）有限公司 | Push service testing method and device |
| CN106157151A (en) * | 2016-06-29 | 2016-11-23 | 厦门趣处网络科技有限公司 | A kind of information aggregation method based on user interest, system |
Non-Patent Citations (2)
| Title |
|---|
| 朱郁筱: "推荐系统评价指标综述", 《电子科技大学学报》 * |
| 符永顺: "基于机器学习的移动应用推荐系统的研究与实现", 《中国优秀硕士学位论文全文数据库（电子期刊）》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110610377A (en) * | 2019-08-09 | 2019-12-24 | 微梦创科网络科技（中国）有限公司 | Advertisement effect testing method and system |
| CN111190801A (en) * | 2019-12-19 | 2020-05-22 | 广州华多网络科技有限公司 | Recommendation system testing method and device and electronic equipment |
| CN113129061A (en) * | 2021-04-20 | 2021-07-16 | 微梦创科网络科技（中国）有限公司 | Advertisement index checking method and system |
| CN113129061B (en) * | 2021-04-20 | 2023-07-18 | 微梦创科网络科技（中国）有限公司 | Advertisement index verification method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106817296B (en) | 2020-04-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109145204B (en) | Portrait label generation and use method and system | |
| US20220414255A1 (en) | User consent framework | |
| CN106528432A (en) | Construction method and apparatus for test scene data, and buried point test method | |
| US20170366917A1 (en) | Method for logging in to application, server, terminal, and nonvolatile computer readable storage medium | |
| US20200110622A1 (en) | Mobile application activity detector | |
| KR20180089564A (en) | Automatic verification of advertiser identifier in advertisements | |
| US9865008B2 (en) | Determining a configuration of a content item display environment | |
| CN102203765A (en) | Uniquely identifying network-distributed devices without explicitly provided device or user identifying information | |
| US8984151B1 (en) | Content developer abuse detection | |
| CN107832409B (en) | Accessing location-based content | |
| CN107026832A (en) | Account logon method, equipment and server | |
| CN106469385A (en) | user invitation method and device | |
| US20190130440A1 (en) | Method and system for detecting fraudulent advertisement activity | |
| US20140278926A1 (en) | System and method for attribution of mobile advertisements related to mobile applications | |
| WO2014026429A1 (en) | Method and device for data determining in thermodynamic chart | |
| US20140278927A1 (en) | System and method for attribution of mobile advertisements related to mobile applications | |
| CN114095567B (en) | Data access request processing method and device, computer equipment and medium | |
| CN108023867A (en) | Method, gateway server and the filtering server of mobile terminal web advertisement filtering | |
| CN107315833A (en) | Method and apparatus of the retrieval with downloading based on application program | |
| CN113971243A (en) | Data processing method, system, equipment and storage medium applied to questionnaire survey | |
| CN111008059A (en) | Control method and device for popup window display, terminal and storage medium | |
| CN106817296A (en) | The method of testing of information recommendation, device and electronic equipment | |
| CN112351350A (en) | Content display method, device, system, equipment and storage medium | |
| CN106796695A (en) | Using the conversion and identification installed | |
| CN113900907A (en) | Mapping construction method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |