CN113312554A - Method and device for evaluating recommendation system, electronic equipment and medium - Google Patents

Method and device for evaluating recommendation system, electronic equipment and medium Download PDF

Info

Publication number
CN113312554A
CN113312554A CN202110662575.XA CN202110662575A CN113312554A CN 113312554 A CN113312554 A CN 113312554A CN 202110662575 A CN202110662575 A CN 202110662575A CN 113312554 A CN113312554 A CN 113312554A
Authority
CN
China
Prior art keywords
content
target object
candidate
recommendation system
evaluated
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110662575.XA
Other languages
Chinese (zh)
Other versions
CN113312554B (en
Inventor
耿林
陈洋
李洪岩
秦涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Priority to CN202110662575.XA priority Critical patent/CN113312554B/en
Publication of CN113312554A publication Critical patent/CN113312554A/en
Application granted granted Critical
Publication of CN113312554B publication Critical patent/CN113312554B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9535—Search customisation based on user profiles and personalisation

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The disclosure provides a method and a device for evaluating a recommendation system, electronic equipment and a medium, relates to the technical field of data processing, and particularly relates to an intelligent recommendation technology. The implementation scheme is as follows: determining at least one content to be evaluated of the target object, wherein the at least one content to be evaluated is historical recommended content presented to the target object by a recommendation system; acquiring evaluation data of a target object aiming at least one content to be evaluated; and evaluating the recommendation effect of the recommendation system according to the evaluation data.

Description

Method and device for evaluating recommendation system, electronic equipment and medium
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to an intelligent recommendation technology, and in particular, to a method and apparatus for evaluating a recommendation system, an electronic device, a computer-readable storage medium, and a computer program product.
Background
The recommendation system is used for screening out the content which is possibly interested by the user from the mass data and pushing the content to the user. At present, recommendation systems have been widely applied to a variety of scenes such as news information recommendation, commodity recommendation, audio and video recommendation, advertisement delivery, social friend recommendation, and the like. With the development of information technology, the data volume of the recommendation system is increasing day by day, and the adopted recommendation algorithm is more complex and diversified, so that the recommendation system faces a huge challenge in providing personalized recommendation service for users, and the interested contents may not be accurately recommended to the users. In order to optimize the recommendation system and enable the recommendation system to better provide personalized recommendation service to users, the recommendation effect of the recommendation system needs to be evaluated.
The approaches described in this section are not necessarily approaches that have been previously conceived or pursued. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise indicated, the problems mentioned in this section should not be considered as having been acknowledged in any prior art.
Disclosure of Invention
The present disclosure provides a method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product for evaluating a recommendation system.
According to an aspect of the present disclosure, there is provided a method for rating a recommendation system, including: determining at least one content to be evaluated of a target object, wherein the at least one content to be evaluated is historical recommended content presented to the target object by the recommending system; obtaining evaluation data of the target object aiming at the at least one content to be evaluated; and evaluating the recommendation effect of the recommendation system according to the evaluation data.
According to another aspect of the present disclosure, there is provided an apparatus for evaluating a recommendation system, including: the content determination module is configured to determine at least one content to be evaluated of a target object, wherein the at least one content to be evaluated is historical recommended content presented to the target object by the recommendation system; the data acquisition module is configured to acquire evaluation data of the target object for the at least one content to be evaluated through the evaluation interface; and an evaluation module configured to evaluate a recommendation effect of the recommendation system according to the evaluation data.
According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor. The memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method for rating recommendation systems described above.
According to another aspect of the present disclosure, a non-transitory computer-readable storage medium having computer instructions stored thereon is provided. The computer instructions are for causing a computer to perform the above-described method for evaluating a recommendation system.
According to another aspect of the disclosure, a computer program product is provided, comprising a computer program. The computer program, when executed by a processor, implements the above-described method for rating a recommendation system.
According to one or more embodiments of the present disclosure, the recommendation effect of the recommendation system is evaluated according to evaluation data of a user (i.e., a target object) on historical recommended content (i.e., content to be evaluated) presented thereto by the recommendation system. The evaluation data is real experience feedback of the user to the historical recommended content, the recommendation effect of the recommendation system is evaluated according to the evaluation data, and the accuracy of the evaluation result of the recommendation system can be improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the embodiments and, together with the description, serve to explain the exemplary implementations of the embodiments. The illustrated embodiments are for purposes of illustration only and do not limit the scope of the claims. Throughout the drawings, identical reference numbers designate similar, but not necessarily identical, elements.
FIG. 1 illustrates a schematic diagram of an exemplary system in which various methods described herein may be implemented, according to an embodiment of the present disclosure;
FIG. 2 shows a flow diagram of a method for rating a recommendation system according to an embodiment of the present disclosure;
3A, 3B illustrate schematic diagrams of exemplary content presentation interfaces, according to embodiments of the present disclosure;
4A, 4B illustrate schematic diagrams of exemplary data interfaces according to embodiments of the present disclosure;
FIG. 5 shows a block diagram of an apparatus for rating recommendation systems according to an embodiment of the present disclosure; and
FIG. 6 illustrates a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, the timing relationship, or the importance relationship of the elements, and such terms are used only to distinguish one element from another. In some examples, a first element and a second element may refer to the same instance of the element, and in some cases, based on the context, they may also refer to different instances.
The terminology used in the description of the various described examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the elements may be one or more. Furthermore, the term "and/or" as used in this disclosure is intended to encompass any and all possible combinations of the listed items.
Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
Fig. 1 illustrates a schematic diagram of an exemplary system 100 in which various methods and apparatus described herein may be implemented in accordance with embodiments of the present disclosure. Referring to fig. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. Client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.
In embodiments of the present disclosure, the server 120 may run one or more services or software applications that enable the execution of methods for rating recommendation systems.
In some embodiments, the server 120 may also provide other services or software applications that may include non-virtual environments and virtual environments. In certain embodiments, these services may be provided as web-based services or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and/or 106 under a software as a service (SaaS) model.
In the configuration shown in fig. 1, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or a combination thereof, which may be executed by one or more processors. A user operating a client device 101, 102, 103, 104, 105, and/or 106 may, in turn, utilize one or more client applications to interact with the server 120 to take advantage of the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from system 100. Accordingly, fig. 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.
The user may use client devices 101, 102, 103, 104, 105, and/or 106 to rate historical recommended content presented to them by the recommendation system. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although fig. 1 depicts only six client devices, those skilled in the art will appreciate that any number and type of client devices may be supported by the present disclosure.
Client devices 101, 102, 103, 104, 105, and/or 106 may include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and so forth. These computer devices may run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux, or Linux-like operating systems (e.g., Google Chrome OS); or include various Mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular telephones, smart phones, tablets, Personal Digital Assistants (PDAs), and the like. Wearable devices may include head mounted displays and other devices. The gaming system may include a variety of handheld gaming devices, internet-enabled gaming devices, and the like. The client device is capable of executing a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications), Short Message Service (SMS) applications, and may use a variety of communication protocols.
Network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP/IP, SNA, IPX, etc. By way of example only, one or more networks 110 may be a Local Area Network (LAN), an ethernet-based network, a token ring, a Wide Area Network (WAN), the internet, a virtual network, a Virtual Private Network (VPN), an intranet, an extranet, a Public Switched Telephone Network (PSTN), an infrared network, a wireless network (e.g., bluetooth, WIFI), and/or any combination of these and/or other networks.
The server 120 may include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-end servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and/or combination. The server 120 may include one or more virtual machines running a virtual operating system, or other computing architecture involving virtualization (e.g., one or more flexible pools of logical storage that may be virtualized to maintain virtual storage for the server). In various embodiments, the server 120 may run one or more services or software applications that provide the functionality described below.
The computing units in server 120 may run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and/or middle tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, and the like.
In some implementations, the server 120 may include one or more applications to analyze and consolidate data feeds and/or event updates received from users of the client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and/or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.
In some embodiments, the server 120 may be a server of a distributed system, or a server incorporating a blockchain. The server 120 may also be a cloud server, or a smart cloud computing server or a smart cloud host with artificial intelligence technology. The cloud Server is a host product in a cloud computing service system, and is used for solving the defects of high management difficulty and weak service expansibility in the traditional physical host and Virtual Private Server (VPS) service.
The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The data store 130 may reside in various locations. For example, the data store used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data store 130 may be of different types. In certain embodiments, the data store used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to the command.
In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the application may be different types of databases, such as key-value stores, object stores, or regular stores supported by a file system.
The system 100 of fig. 1 may be configured and operated in various ways to enable application of the various methods and apparatus described in accordance with the present disclosure.
For purposes of the disclosed embodiments, in the example of fig. 1, client devices 101, 102, 103, 104, 105, and 106 may include therein a client application for content browsing, through which a user may browse content. The content browsed by the user can be news information, audio and video, commodity information and the like, and correspondingly, the client application can be news information application, audio and video entertainment application, shopping application and the like. The client application may exist in the client device in a number of ways. For example, the client application may be an application program that needs to be downloaded and installed before running, a website that can be accessed through a browser, a light-weight applet that runs in a host application, and the like.
The server 120 may be a server corresponding to a client application for content browsing in a client device, corresponding to the client application. A service program may be included in the server 120, and the service program may provide a content browsing service to the user based on the content information (including the title, the arrangement, the body, the author, the type, the interaction situation (e.g., approval, comment, forwarding, etc.) of the content) stored in the database 130. Further, a recommendation system is included in the service program, and the recommendation system is capable of providing a personalized recommendation service to the user, determining content (i.e., recommended content) that may be of interest to the user from the stored plurality of pieces of content according to relevant information (e.g., attribute information, behavior information, etc.) of the user, and presenting part or all of the determined plurality of pieces of recommended content to the user. Accordingly, the user can browse recommended contents recommended thereto by the recommendation system through the client application.
In some cases, the recommendation effect of the recommendation system can be evaluated, and the recommendation system is optimized according to the evaluation result, so that the recommendation system can better provide personalized recommendation services for users. In the related art, the recommendation effect of the recommendation system is generally evaluated according to technical indexes such as the number of clicks of a user on recommended content, a click-to-spread ratio (i.e., a ratio of the number of clicks to the number of presentations), and browsing duration. Since the quality of the recommended content is uneven, there may be a case where the title is excessively exaggerated or distorted (i.e., colloquially called "title party"), the title does not coincide with the text, and so on, and therefore the behavior of clicking, browsing for a long time, and so on of the recommended content by the user does not mean that the user is satisfied with the recommended content. Technical indexes such as click times, point-to-area ratios, browsing duration and the like cannot accurately reflect real experience and subjective feeling of a user on recommended contents, so that evaluation results of a recommendation system obtained according to the technical indexes are not accurate enough, and confidence is low.
In order to accurately evaluate the recommendation effect of the recommendation system, in an embodiment of the present disclosure, the server 120 may execute the method 200 for evaluating the recommendation system, acquire evaluation data of historical recommendation content viewed by the user for the user, and evaluate the recommendation effect of the recommendation system according to the evaluation data. The evaluation data is real experience feedback of the user on the historical recommended content, the recommendation effect of the recommendation system is evaluated according to the evaluation data, the accuracy of the evaluation result of the recommendation system can be improved, and clear guidance is provided for the optimization direction of the recommendation system.
Further, the server 120 may optimize the recommendation system according to the evaluation result obtained by executing the method for evaluating the recommendation system of the embodiment of the present disclosure. For the optimized recommendation system, the method for evaluating the recommendation system according to the embodiment of the present disclosure may be used again to evaluate it, and the method may be optimized again based on the evaluation result. The evaluation and optimization process can be executed repeatedly, so that the recommendation effect of the recommendation system can be continuously improved, and the recommendation system can better provide personalized recommendation service for users.
FIG. 2 shows a flow diagram of a method 200 for rating a recommendation system according to an embodiment of the present disclosure. The method 200 may be performed at a server (e.g., the server 120 shown in fig. 1), that is, the execution subject of each step of the method 200 may be the server 120 shown in fig. 1. It will be appreciated that in some embodiments, method 200 may also be performed at a client device (e.g., client devices 101, 102, 103, 104, 105, and 106 shown in fig. 1). Further, the client device may upload the evaluation result of the recommendation system obtained by executing the method 200 to the server.
As shown in fig. 2, the method 200 includes: step 210, determining at least one content to be evaluated of the target object, wherein the content to be evaluated is historical recommended content presented to the target object by a recommendation system; step 220, obtaining evaluation data of the target object aiming at the at least one content to be evaluated; and a step 230 of evaluating the recommendation effect of the recommendation system according to the evaluation data.
According to the embodiment of the present disclosure, the recommendation effect of the recommendation system is evaluated according to the evaluation data of the user (i.e., the target object) on the history recommended content (i.e., the content to be evaluated) presented thereto by the recommendation system. The evaluation data is real experience feedback of the user on the historical recommended content, the recommendation effect of the recommendation system is evaluated according to the evaluation data, the accuracy of the evaluation result of the recommendation system can be improved, and clear guidance is provided for the optimization direction of the recommendation system.
The various steps of method 200 are described in detail below.
In step 210, at least one content to be evaluated of the target object is determined, and the content to be evaluated is historical recommended content presented to the target object by the recommendation system.
In an embodiment of the present disclosure, the target object refers to an actual user participating in the rating recommendation system, which may be selected from a plurality of users using the recommendation system. Hereinafter, unless otherwise specified, a user who uses the recommendation system is referred to as a "candidate object", and a user who participates in the rating recommendation system, which is selected from a plurality of users who use the recommendation system, is referred to as a "target object". It should be understood that there may be one or more target objects, and that the set of target objects (including one or more target objects) is a subset of the set of candidate objects (including multiple candidate objects).
According to some embodiments, the method 200 may further comprise a step 240 for determining the target object, the step 240 comprising: obtaining object attributes and liveness attributes of a plurality of candidate objects using a recommendation system; and determining a target object from the plurality of candidate objects according to the object attribute and the liveness attribute.
Object attributes refer to characteristics of the candidate itself, including but not limited to the gender, age, region of the candidate, and the like.
The activity attribute refers to the activity level of the candidate object on the recommendation system, which may be determined, for example, based on the length of time and/or frequency with which the candidate object is used with the recommendation system. The longer and more frequently (i.e., more frequently) the candidate uses the recommendation system, the greater the value of its liveness attribute. As described above, the recommendation system is typically part of a service program corresponding to the client application. In some embodiments, the duration and frequency of usage of the client application by the candidate may be taken as the duration and frequency of usage of the recommendation system by the candidate.
It should be noted that, in the embodiments of the present disclosure, the acquisition, storage, and use of the object attribute and the activity attribute of the related candidate object all meet the regulations of related legal regulations, and do not violate the customs of the public order. The object attributes, liveness attributes of candidate objects (i.e., users) are obtained, stored, and used based on the authorization and consent of the users. And the object attribute and the activity attribute are subjected to desensitization processing (namely anonymization processing) in the processes of acquisition, storage and use.
The use experience of the recommendation system by the user (candidate) has a significant positive correlation with the activity attribute. In the embodiment of the disclosure, the target object is determined from the plurality of candidate objects according to the object attribute and the activity attribute of the candidate object, so that the composition of the target object can be closer to the real user group of the recommendation system, and the authenticity and the accuracy of the evaluation result of the recommendation system are improved.
According to some embodiments, the target object may be further determined from the plurality of candidate objects according to the following steps 242-246: step 242, sampling a plurality of candidate objects according to the object attributes to obtain a first object set; step 244, determining a second object set based on the first object set according to the activity degree attributes, wherein the distribution situation of the activity degree attributes of the candidate objects in the second object set is consistent with the distribution situation of the activity degree attributes of the candidate objects; and step 246, taking the candidate objects in the second object set as target objects.
According to some embodiments, step 242 further comprises: dividing the plurality of candidate objects into a plurality of object groups according to object attributes, wherein each object group comprises at least one candidate object with the same object attribute; and according to the number of the candidate objects included in each object group in the plurality of object groups, hierarchically sampling the candidate objects in the plurality of object groups.
Generally, in the hierarchical sampling process, the number of candidate objects to be extracted from each object group may be determined first. Specifically, the number of candidate objects extracted from the ith object group is (n)i*n)/∑niWherein n isiIs the number of candidate objects included in the ith object group, n is the total number of target objects, Σ niIs the total number of candidates. Subsequently, a corresponding number of candidate objects is randomly extracted from each object group, i.e. from the ith object group (n)i*n)/∑niAnd (4) a candidate object.
For example, the object attributes include gender (male/female), age (teenager/young/middle-aged/old), and region (first-line city/second-line city/third-line city/fourth-line city/fifth-line city), and accordingly, a plurality of object candidates may be divided into 2 x 4 x 5 or 40 object groups in total, such as (male, teenager, first-line city), (female, teenager, first-line city), (male, young, second-line city), (female, young, second-line city), and the like, and each object group includes n object candidates in number1，n2，n3，…，n40Total number of candidates
Figure BDA0003115946770000091
Setting the total number of target objects to n, it is necessary to randomly extract (n) from the i-th (i ═ 1, 2, 3, …, 40) object groupiN)/N candidates.
By dividing a plurality of candidate objects into a plurality of object groups and hierarchically sampling the candidate objects in the plurality of object groups according to the number of candidate objects included in each object group, the object attribute configuration of the target object can be made to coincide with the object attribute configurations of the plurality of candidate objects.
According to some embodiments, step 244 further comprises: determining a first activity attribute and a second activity attribute, wherein the proportion of the candidate objects with the first activity attribute in the first object set is greater than the proportion of the candidate objects with the first activity attribute in the plurality of candidate objects, and the proportion of the candidate objects with the second activity attribute in the first object set is less than the proportion of the candidate objects with the second activity attribute in the plurality of candidate objects; removing a first candidate object having a first liveness attribute from the first set of objects; and adding a second candidate object with a second activity attribute to the first object set, wherein the object attribute of the second candidate object is the same as the object attribute of the first candidate object.
The first candidate object and the second candidate object may be randomly selected or selected according to a certain rule. The object attributes of the second candidate object are the same as those of the first candidate object, so that the object attribute distribution of the candidate objects in the first object set is not changed (consistent with the object attributes of the plurality of candidate objects all the time) after the first candidate object is removed from the first object set and the second candidate object is added to the first object set.
The step of removing the first candidate object from the first object set and adding the second candidate object thereto may be repeatedly performed for a plurality of times until the distribution of the activity attributes of the candidate objects in the first object set is consistent with the distribution of the activity attributes of the plurality of candidate objects, that is, the second object set is obtained.
For example, liveness attributes include highly active, moderately active, and lightly active. The proportions of the highly active, moderately active and slightly active candidate objects in the first object set are respectively 0.4, 0.3 and 0.3, and the proportions of the highly active, moderately active and slightly active candidate objects in the plurality of candidate objects are respectively 0.2, 0.5 and 0.3. Highly active is a first activity attribute because the proportion 0.4 of highly active candidate objects in the first set of objects is greater than the proportion 0.2 of highly active candidate objects in the plurality of candidate objects; since the proportion 0.3 of the moderately active candidate object in the first set of objects is less than the proportion 0.5 of the moderately active candidate object in the plurality of candidate objects, moderate activity is the second activity attribute. Accordingly, a highly active candidate (i.e., a first candidate) may be randomly removed from the first set of objects, the object attribute of which may be, for example, (male, juvenile, first-line city); and randomly selecting a candidate object with moderate activity (namely a second candidate object) with the same object attribute (male, juvenile, first-line city) from the candidate objects which do not belong to the first object set at present, and adding the candidate object into the first object set. The steps of removing the highly active first candidate object from the first object set and adding the moderately active second candidate object thereto may be repeatedly executed for a plurality of times until the distribution of the activity attributes of the candidate objects in the first object set is consistent with the distribution of the activity attributes of the plurality of candidate objects, that is, the proportions of the highly active, moderately active, and lightly active candidate objects are 0.2, 0.5, and 0.3. And the first object set which is consistent with the distribution situation of the activity attributes of the candidate objects is the second object set.
Based on the target object determined in step 240, at least one content to be evaluated of the target object may be further determined.
The content to be evaluated is historical recommended content presented to the target object by the recommending system. From the perspective of the target object, the content to be evaluated is history recommended content browsed by the target object.
As described above, the recommendation system may determine, from the stored pieces of content, content that may be of interest to the target object, and present some or all of the determined pieces of recommended content to the target object. It should be understood that the above-described act of "presenting" may not be performed by the recommendation system itself, but rather by the recommendation system instructing the display of the client device.
For example, the recommendation system determines 100 pieces of recommended content for the target object a from the stored pieces of content, and sorts the 100 pieces of recommended content in order from high to low as the degree of possible interest of the user. The target object may initiate a recommendation request through a client application in the client device to request a quantity of recommended content to be obtained. The recommendation system returns a corresponding amount of recommended content to the client device in response to the recommendation request, and presents the recommended content to the target object by a display of the client device. FIG. 3A illustrates a schematic diagram of an exemplary content presentation interface 300A presented on a client device, in accordance with embodiments of the present disclosure. As shown in fig. 3, four recommended contents, i.e., recommended content 1-recommended content 4, are presented in the interface 300A. The target object may initiate the recommendation request by clicking (e.g., clicking on a preset area or control in interface 300A), sliding (e.g., sliding up in interface 300A), and the like. The recommendation system further returns a certain amount of recommended content, such as recommended content 5, recommended content 6, to the client device in response to the recommendation request, and presents the recommended content 5, recommended content 6 to the target object through the display of the client device. The presentation interface 300B of the recommended content 5 and the recommended content 6 is shown in fig. 3B.
According to some embodiments, at least one content to be evaluated of the target object may be obtained according to the following steps: acquiring a browsing history screenshot of a target object; performing Optical Character Recognition (OCR) on the browsing history screenshot to determine recommended content included in the browsing history screenshot; and using the recommended content as the at least one content to be evaluated.
The browsing history screenshot of the target object may be, for example, the screenshot of the target object during the browsing of the content. Typically, only the title of each recommended content and the background image of the title are included in the browsing history screenshot. According to some embodiments, after the titles of the recommended contents in the browsing history screenshot are identified through the OCR technology, information such as authors, fields, texts and images corresponding to the recommended contents can be further acquired from the database according to the titles of the recommended contents, and reference is provided for a subsequent evaluation recommendation system.
According to other embodiments, at least one content to be evaluated of the target object may also be obtained according to the following steps: acquiring an access log of a target object; and taking the recommended content in the access log as the at least one content to be evaluated.
The access log may be, for example, an access log of the target object to the recommendation system or an access log of the target object to a client application served by the recommendation system. The access log records the contents that the target object browses once. Through analyzing the access log, the recommended contents which are browsed by the target object once can be determined and taken as at least one content to be evaluated of the target object.
It should be noted that, in the above-described embodiment, the access log of the target object is acquired, stored, and used based on the authorization and approval of the user. The acquisition, storage, use and the like of the access log of the target object all conform to the regulations of related laws and regulations and do not violate the good customs of the public order.
In step 220, evaluation data of the target object for the at least one content to be evaluated may be obtained.
According to some embodiments, step 220 further comprises: providing a data interface for evaluating the at least one content to be evaluated to a target object; and acquiring evaluation data of the target object aiming at the at least one content to be evaluated through the data interface. In this embodiment, the data interface is generated for the content to be evaluated, and may be used to accurately obtain the evaluation data of the content to be evaluated by the target object, without obtaining the evaluation of other irrelevant content (for example, non-recommended content, recommended content not presented to the target object, and the like) by the target object, so that the efficiency of obtaining the evaluation data is improved, and the efficiency of the evaluation recommendation system is further improved.
According to some embodiments, the evaluation data includes, but is not limited to, satisfaction of the target object for each of the at least one content to be evaluated and reason information corresponding to the satisfaction. The satisfaction of the target object with the content to be evaluated can be represented by a set of preset numerical scores (e.g., an integer of 1-5), and the higher the satisfaction of the target object with the content to be evaluated, the larger the numerical score.
Fig. 4A shows a schematic diagram of an exemplary data interface 400A, according to an embodiment of the present disclosure, the data interface 400A is used for acquiring evaluation data of a target object to-be-evaluated content 1.
As shown in fig. 4A, the top of the data interface 400A shows basic information of the content to be rated 1, including a title 402 of the content to be rated 1 and maps 404, 406, and 408 in the content to be rated 1. The text box 410 is used to obtain a verification code (CAPTCHA) to distinguish whether the target object is a real user or a computer program. The verification code may specifically be the first three words of the title of the content to be evaluated.
The evaluation data that the data interface 400A can acquire includes the satisfaction of the target object with the content 1 to be evaluated (corresponding to the problem 1 in fig. 4A) and the cause information corresponding to the satisfaction (corresponding to the problems 2 and 3 in fig. 4A).
For problem 1, the data interface 400A shows a plurality of satisfaction options, and the target object may submit its satisfaction of the content to be evaluated 1 by checking the radio box 412 corresponding to the satisfaction option. For problem 2, the data interface 400A shows a plurality of satisfactory reason options, and the target object can submit the reason information that it is satisfactory to the content 1 to be evaluated by checking the check box 414 corresponding to the reason options. For problem 3, the data interface 400A shows a plurality of unsatisfied reason options, and the target object can submit the information of the dissatisfaction reason of the content to be evaluated 1 by checking the check box 416 corresponding to the reason options.
Fig. 4B is a schematic diagram of another exemplary data interface 400B according to an embodiment of the present disclosure, where the data interface 400B is configured to further obtain evaluation data of other aspects of the content to be evaluated 1 from the target object on the basis of the data interface 400A. Data interface 400B and data interface 400A may be the same data interface, i.e., both belong to the same page. For example, the target object may get the data interface 400B shown in FIG. 4B by performing a slide-up operation in the data interface 400A shown in FIG. 4A.
As shown in fig. 4B, the top of the data interface 400B shows, similarly to fig. 4A, a title 402, arrangement 404, 406, and 408 of the content 1 to be evaluated, and a text box 410 for acquiring a verification code.
The rating data that the data interface 400B can acquire includes the type of the content 1 to be rated (corresponding to the question 4 in fig. 4B), which the target object considers, the domain to which it belongs (corresponding to the question 5 in fig. 4B), and the frequency with which the target object browses similar content (corresponding to the question 6 in fig. 4B).
For question 4, the data interface 400B shows a plurality of type options, and the target object may submit the type of the content 1 to be evaluated, which the target object considers, by checking the radio box 418 corresponding to the type options. For question 5, the data interface 400B shows two drop-down boxes 420, and the target object can select the domain to which the content 1 to be evaluated belongs by clicking on the drop-down boxes 420. For question 6, the data interface 400B shows a number of frequency options, and the target object may submit its frequency of seeing the same or similar content as the content 1 to be evaluated by checking the radio box 422 corresponding to the frequency option.
In step 230, the recommendation effect of the recommendation system may be evaluated according to the evaluation data acquired in step 220.
The evaluation data obtained in step 220 may be abnormal, for example, the evaluation data may be freely input by the target object, and such evaluation data may not express the real feeling of the target object to the evaluated content. According to some embodiments, abnormal data in the evaluation data can be determined according to preset abnormal judgment conditions, the abnormal data are eliminated, and the recommendation effect of the recommendation system is evaluated according to the rest evaluation data, so that the accuracy of the evaluation result of the recommendation system is ensured.
The preset abnormality determination condition may be, for example, whether the time taken for the target object to input the evaluation data is abnormal, whether the evaluation of all the contents to be evaluated by the target object is the same, whether the evaluation data has a logical error, or the like. If the time taken by the target object to input the evaluation data is short (for example, shorter than a preset time threshold), or the evaluation data is the same for all the contents to be evaluated (for example, the contents to be evaluated have the same satisfaction or the same reason information), or the evaluation data is obviously wrong (for example, the true type of a certain content to be evaluated is "image-text", and the type selected by the target object is "short video"), the evaluation data is judged to be abnormal data.
After the abnormal data in the evaluation data is eliminated, the recommendation effect of the recommendation system can be evaluated according to the rest evaluation data. According to some embodiments, the overall satisfaction of each target object to the recommendation system may be determined separately according to the corresponding evaluation data; and determining the recommendation effect of the recommendation system according to the overall satisfaction of each target object.
For example, the recommendation effect S of the recommendation system can be calculated according to the following steps:
first, the overall satisfaction s of the target object i to the recommendation system is calculatedi：
Figure BDA0003115946770000141
Wherein m isiThe number of contents to be evaluated, c, of the target object iijAnd the satisfaction degree of the target object i to the jth content to be evaluated is obtained.
Calculating the recommendation effect S of the recommendation system according to the overall satisfaction degree of each target object to the recommendation system:
Figure BDA0003115946770000151
where n is the number of target objects.
According to some embodiments, the recommendation effect of the recommendation system for different types of content can be evaluated according to the evaluation data of the target objects for each content to be evaluated, for example, the recommendation effect of the recommendation system for the types of content such as graphics, short video, live broadcast, advertisement and the like can be evaluated, so that the content type with poor recommendation effect can be optimized in a targeted manner.
Fig. 5 shows a block diagram of an apparatus 500 for rating a recommendation system according to an embodiment of the present disclosure. As shown in fig. 5, the apparatus 500 includes a content determination module 510, a data acquisition module 520, and an evaluation module 530.
The content determination module 510 may be configured to determine at least one content to be rated of a target object, the at least one content to be rated being historical recommended content that the recommendation system presents to the target object.
The data obtaining module 520 may be configured to obtain rating data of the target object for the at least one content to be rated.
The evaluation module 530 may be configured to evaluate the recommendation effect of the recommendation system according to the evaluation data.
According to the embodiment of the present disclosure, the recommendation effect of the recommendation system is evaluated according to the evaluation data of the user (i.e., the target object) on the history recommended content (i.e., the content to be evaluated) presented thereto by the recommendation system. The evaluation data is real experience feedback of the user on the historical recommended content, the recommendation effect of the recommendation system is evaluated according to the evaluation data, the accuracy of the evaluation result of the recommendation system can be improved, and clear guidance is provided for the optimization direction of the recommendation system.
According to some embodiments, the apparatus 500 further comprises an object determination module. The object determination module further comprises an attribute acquisition unit and an object determination unit, wherein the attribute acquisition unit may be configured to acquire object attributes and activity attributes of a plurality of candidate objects using the recommendation system; the object determination unit may be configured to determine the target object from the plurality of candidate objects based on the object attribute and the liveness attribute.
According to some embodiments, the liveness attribute is determined according to a duration and/or frequency of use of the recommendation system by the candidate object.
According to some embodiments, the object determination unit may be further configured to: sampling the candidate objects according to the object attributes to obtain a first object set; the second object determination unit may be configured to determine a second object set based on the first object set according to the activity attribute, wherein a distribution of the activity attribute of the candidate objects in the second object set is consistent with a distribution of the activity attribute of the candidate objects; and taking the candidate object in the second object set as the target object.
According to some embodiments, the object determination unit may be further configured to: dividing the plurality of candidate objects into a plurality of object groups according to the object attributes, wherein each object group comprises at least one candidate object with the same object attribute; and according to the number of the candidate objects included in each object group in the plurality of object groups, hierarchically sampling the candidate objects in the plurality of object groups.
According to some embodiments, the object determination unit may be further configured to: determining a first activity attribute and a second activity attribute, wherein the proportion of the candidate objects with the first activity attribute in the first object set is greater than the proportion of the candidate objects with the first activity attribute in the plurality of candidate objects, and the proportion of the candidate objects with the second activity attribute in the first object set is less than the proportion of the candidate objects with the second activity attribute in the plurality of candidate objects; removing a first candidate object having the first liveness attribute from the first set of objects; and adding a second candidate object having the second liveness attribute to the first set of objects, wherein the object attributes of the second candidate object are the same as the object attributes of the first candidate object.
According to some embodiments, the content determination module 510 may be further configured to: acquiring a browsing history screenshot of a target object; carrying out optical character recognition on the browsing history screenshot to determine recommended content included in the browsing history screenshot; and using the recommended content as the at least one content to be evaluated.
According to some embodiments, the content determination module 510 may be further configured to: acquiring an access log of the target object; and taking the recommended content in the access log as the at least one content to be evaluated.
According to some embodiments, the data acquisition module 520 may be further configured to: providing a data interface for rating the at least one content to be rated to the target object; and acquiring evaluation data of the target object aiming at the at least one content to be evaluated through the data interface.
According to some embodiments, the evaluation data includes a satisfaction degree of the target object for each of the at least one content to be evaluated and reason information corresponding to the satisfaction degree.
According to some embodiments, the target object is multiple, and the evaluation module 530 may be further configured to: respectively determining the overall satisfaction degree of each target object to the recommendation system according to the corresponding evaluation data; and determining the recommendation effect of the recommendation system according to the overall satisfaction of each target object.
According to some embodiments, the apparatus 500 further comprises a data cleansing module. The data cleansing module may be configured to: determining abnormal data in the evaluation data according to a preset abnormal judgment condition; and rejecting the abnormal data.
It should be understood that the various modules of the apparatus 500 shown in fig. 5 may correspond to the various steps in the method 200 described with reference to fig. 2. Thus, the operations, features and advantages described above with respect to the method 200 are equally applicable to the apparatus 500 and the modules comprised thereby. Certain operations, features and advantages may not be described in detail herein for the sake of brevity.
Although specific functionality is discussed above with reference to particular modules, it should be noted that the functionality of the various modules discussed herein may be divided into multiple modules and/or at least some of the functionality of multiple modules may be combined into a single module. For example, the content determination module 510 and the data acquisition module 520 described above may be combined into a single module in some embodiments.
It should also be appreciated that various techniques may be described herein in the general context of software, hardware elements, or program modules. The various modules described above with respect to fig. 5 may be implemented in hardware or in hardware in combination with software and/or firmware. For example, the modules may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, the modules may be implemented as hardware logic/circuitry. For example, in some embodiments, one or more of the content determination module 510, the data acquisition module 520, and the evaluation module 530 may be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip (which includes one or more components of a Processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, Digital Signal Processor (DSP), etc.), memory, one or more communication interfaces, and/or other circuitry), and may optionally execute received program code and/or include embedded firmware to perform functions.
According to an embodiment of the present disclosure, there is also provided an electronic device, a readable storage medium, and a computer program product.
Referring to fig. 6, a block diagram of a structure of an electronic device 600, which may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. Electronic device is intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 6, the apparatus 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The calculation unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
A number of components in the device 600 are connected to the I/O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the device 600, and the input unit 606 may receive input numeric or character information and generate key signal inputs related to user settings and/or function controls of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and/or a remote control. Output unit 607 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakersA device, a video/audio output terminal, a vibrator, and/or a printer. The storage unit 608 may include, but is not limited to, a magnetic disk, an optical disk. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and/or a chipset, such as bluetoothTMDevices, 1302.11 devices, WiFi devices, WiMax devices, cellular communication devices, and/or the like.
The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The computing unit 601 performs the various methods and processes described above, such as the method 200 described above. For example, in some embodiments, the method 200 may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the method 200 in any other suitable manner (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be performed in parallel, sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.
Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it is to be understood that the methods, systems, and apparatus described above are merely exemplary embodiments or examples and that the scope of the present disclosure is not limited by these embodiments or examples, but only by the claims as issued and their equivalents. Various elements in the embodiments or examples may be omitted or may be replaced with equivalents thereof. Further, the steps may be performed in an order different from that described in the present disclosure. Further, various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced with equivalent elements that appear after the present disclosure.

Claims (16)

1. A method for rating a recommendation system, comprising:
determining at least one content to be evaluated of a target object, wherein the at least one content to be evaluated is historical recommended content presented to the target object by the recommending system;
obtaining evaluation data of the target object aiming at the at least one content to be evaluated; and
and evaluating the recommendation effect of the recommendation system according to the evaluation data.
2. The method of claim 1, further comprising:
obtaining object attributes and liveness attributes of a plurality of candidate objects using the recommendation system; and
determining the target object from the plurality of candidate objects according to the object attributes and the liveness attributes.
3. The method of claim 2, wherein the liveness attribute is determined as a function of a length of time and/or a frequency of use of the recommendation system by the candidate object.
4. The method of claim 2 or 3, wherein the determining the target object from the plurality of candidate objects comprises:
sampling the candidate objects according to the object attributes to obtain a first object set;
determining a second object set based on the first object set according to the activity degree attributes, wherein the distribution of the activity degree attributes of the candidate objects in the second object set is consistent with the distribution of the activity degree attributes of the candidate objects; and
and taking the candidate object in the second object set as the target object.
5. The method of claim 4, wherein said sampling the plurality of candidate objects comprises:
dividing the plurality of candidate objects into a plurality of object groups according to the object attributes, wherein each object group comprises at least one candidate object with the same object attribute; and
and according to the number of the candidate objects included in each object group in the plurality of object groups, hierarchically sampling the candidate objects in the plurality of object groups.
6. The method of claim 4 or 5, wherein the determining a second set of objects based on the first set of objects comprises:
determining a first activity attribute and a second activity attribute, wherein the proportion of the candidate objects with the first activity attribute in the first object set is greater than the proportion of the candidate objects with the first activity attribute in the plurality of candidate objects, and the proportion of the candidate objects with the second activity attribute in the first object set is less than the proportion of the candidate objects with the second activity attribute in the plurality of candidate objects;
removing a first candidate object having the first liveness attribute from the first set of objects; and
adding a second candidate object having the second liveness attribute to the first set of objects, wherein the object attributes of the second candidate object are the same as the object attributes of the first candidate object.
7. The method of any of claims 1-6, wherein the determining at least one content to be rated for a target object comprises:
acquiring a browsing history screenshot of a target object;
carrying out optical character recognition on the browsing history screenshot to determine recommended content included in the browsing history screenshot; and
and taking the recommended content as the at least one content to be evaluated.
8. The method of any of claims 1-6, wherein the determining at least one content to be rated for a target object comprises:
acquiring an access log of the target object; and
and taking the recommended content in the access log as the at least one content to be evaluated.
9. The method according to any one of claims 1 to 8, wherein the obtaining of rating data of the target object for the at least one content to be rated comprises:
providing a data interface for rating the at least one content to be rated to the target object; and
and obtaining evaluation data of the target object aiming at the at least one content to be evaluated through the data interface.
10. The method according to any one of claims 1 to 9, wherein the rating data includes a satisfaction level of the target object for each of the at least one content to be rated and reason information corresponding to the satisfaction level.
11. The method according to claim 10, wherein there are a plurality of the target objects, and the evaluating the recommendation effect of the recommendation system according to the evaluation data includes:
respectively determining the overall satisfaction degree of each target object to the recommendation system according to the corresponding evaluation data; and
and determining the recommendation effect of the recommendation system according to the overall satisfaction of each target object.
12. The method according to any one of claims 1-11, further comprising:
determining abnormal data in the evaluation data according to a preset abnormal judgment condition; and
and rejecting the abnormal data.
13. An apparatus for rating a recommendation system, comprising:
the content determination module is configured to determine at least one content to be evaluated of a target object, wherein the at least one content to be evaluated is historical recommended content presented to the target object by the recommendation system;
the data acquisition module is configured to acquire evaluation data of the target object for the at least one content to be evaluated; and
an evaluation module configured to evaluate a recommendation effect of the recommendation system according to the evaluation data.
14. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.
15. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-12.
16. A computer program product comprising a computer program, wherein the computer program realizes the method of any one of claims 1-12 when executed by a processor.
CN202110662575.XA 2021-06-15 2021-06-15 Method and device for evaluating recommendation system, electronic equipment and medium Active CN113312554B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110662575.XA CN113312554B (en) 2021-06-15 2021-06-15 Method and device for evaluating recommendation system, electronic equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110662575.XA CN113312554B (en) 2021-06-15 2021-06-15 Method and device for evaluating recommendation system, electronic equipment and medium

Publications (2)

Publication Number Publication Date
CN113312554A true CN113312554A (en) 2021-08-27
CN113312554B CN113312554B (en) 2023-11-03

Family

ID=77378808

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110662575.XA Active CN113312554B (en) 2021-06-15 2021-06-15 Method and device for evaluating recommendation system, electronic equipment and medium

Country Status (1)

Country Link
CN (1) CN113312554B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116402249A (en) * 2023-03-06 2023-07-07 贝壳找房(北京)科技有限公司 Recommendation system overflow effect evaluation method, recommendation system overflow effect evaluation device, storage medium and program product

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107729542A (en) * 2017-10-31 2018-02-23 咪咕音乐有限公司 A kind of information methods of marking and device and storage medium
WO2018130220A1 (en) * 2017-01-16 2018-07-19 广州市动景计算机科技有限公司 Message pushing method and device, and programmable device
CN108319611A (en) * 2017-01-17 2018-07-24 腾讯科技（深圳）有限公司 The methods of sampling and sampling apparatus
CN108804506A (en) * 2018-04-13 2018-11-13 北京猫眼文化传媒有限公司 A kind of the recommendation method, apparatus and electronic equipment of information
CN110297975A (en) * 2019-06-26 2019-10-01 北京百度网讯科技有限公司 Appraisal procedure, device, electronic equipment and the storage medium of Generalization bounds
CN112817870A (en) * 2021-02-26 2021-05-18 北京小米移动软件有限公司 Software testing method, device and medium
CN112925978A (en) * 2021-02-26 2021-06-08 北京百度网讯科技有限公司 Recommendation system evaluation method and device, electronic equipment and storage medium

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018130220A1 (en) * 2017-01-16 2018-07-19 广州市动景计算机科技有限公司 Message pushing method and device, and programmable device
CN108319611A (en) * 2017-01-17 2018-07-24 腾讯科技（深圳）有限公司 The methods of sampling and sampling apparatus
CN107729542A (en) * 2017-10-31 2018-02-23 咪咕音乐有限公司 A kind of information methods of marking and device and storage medium
CN108804506A (en) * 2018-04-13 2018-11-13 北京猫眼文化传媒有限公司 A kind of the recommendation method, apparatus and electronic equipment of information
CN110297975A (en) * 2019-06-26 2019-10-01 北京百度网讯科技有限公司 Appraisal procedure, device, electronic equipment and the storage medium of Generalization bounds
CN112817870A (en) * 2021-02-26 2021-05-18 北京小米移动软件有限公司 Software testing method, device and medium
CN112925978A (en) * 2021-02-26 2021-06-08 北京百度网讯科技有限公司 Recommendation system evaluation method and device, electronic equipment and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116402249A (en) * 2023-03-06 2023-07-07 贝壳找房(北京)科技有限公司 Recommendation system overflow effect evaluation method, recommendation system overflow effect evaluation device, storage medium and program product
CN116402249B (en) * 2023-03-06 2024-02-23 贝壳找房(北京)科技有限公司 Recommendation system overflow effect evaluation method, recommendation system overflow effect evaluation equipment and storage medium

Also Published As

Publication number Publication date
CN113312554B (en) 2023-11-03

Similar Documents

Publication Publication Date Title
US11449682B2 (en) Adjusting chatbot conversation to user personality and mood
CN110442712B (en) Risk determination method, risk determination device, server and text examination system
CN112579909A (en) Object recommendation method and device, computer equipment and medium
US11314825B2 (en) Machine-learning based personalization
CN113411645B (en) Information recommendation method and device, electronic equipment and medium
CN114791982B (en) Object recommendation method and device
CN113312554B (en) Method and device for evaluating recommendation system, electronic equipment and medium
US20140052842A1 (en) Measuring problems from social media discussions
CN116501960B (en) Content retrieval method, device, equipment and medium
CN115269989B (en) Object recommendation method, device, electronic equipment and storage medium
CN116304335A (en) Object recommendation method, user preference identification method and device
CN113596011B (en) Flow identification method and device, computing device and medium
CN113076395B (en) Semantic model training and search display method, device, equipment and storage medium
CN114238745A (en) Method and device for providing search result, electronic equipment and medium
CN114880580A (en) Information recommendation method and device, electronic equipment and medium
CN114896061A (en) Training method of computing resource control model, and computing resource control method and device
CN114281990A (en) Document classification method and device, electronic equipment and medium
CN113312511A (en) Method, apparatus, device and computer-readable storage medium for recommending content
CN112906387A (en) Risk content identification method, apparatus, device, medium, and computer program product
CN115033782B (en) Object recommendation method, training method, device and equipment of machine learning model
CN116883181B (en) Financial service pushing method based on user portrait, storage medium and server
CN115809364B (en) Object recommendation method and model training method
CN113420227B (en) Training method of click rate estimation model, click rate estimation method and device
CN113963234B (en) Data annotation processing method, device, electronic equipment and medium
CN116405551B (en) Social platform-based data pushing method and system and cloud platform

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant