CN111210290A

CN111210290A - Sorting method, sorting device and computer-readable storage medium

Info

Publication number: CN111210290A
Application number: CN201811387892.XA
Authority: CN
Inventors: 刘逸博; 李勇; 姚亚飞; 严严; 包勇军
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2018-11-21
Filing date: 2018-11-21
Publication date: 2020-05-29

Abstract

The invention discloses a sorting method, a sorting device and a computer readable storage medium, and relates to the field of data processing. The sorting method comprises the following steps: calculating a first ranking index of each candidate object according to the user operation data and the value index data of the candidate objects; performing first sequencing on the alternative objects by adopting a first sequencing index; acquiring partial alternative objects in the first sorting result as second sorting objects; and adopting a pre-trained sorting model to re-sort the second sorted objects so as to recommend the second sorted objects to the user according to the sequence of the second sorting result. When the ranking model is adopted for second ranking, the ranked objects are not all the alternative objects but part of the alternative objects, so that the calculation pressure of the ranking model is reduced, and the recommendation efficiency is improved. In addition, the number of objects browsed by the user in a short time is limited, so that the use experience of the user is not influenced by the scheme of the invention.

Description

Sorting method, sorting device and computer-readable storage medium

Technical Field

The present invention relates to the field of data processing, and in particular, to a sorting method, apparatus, and computer-readable storage medium.

Background

With the continuous development of internet-based e-commerce technology, the number of goods displayed on e-commerce websites has increased explosively. The real-time shopping interest of the user can be found out accurately in time according to the behavior characteristics of the user, the shopping experience of the user can be greatly optimized, the advertisement clicking income can be brought to an e-commerce website, and the shopping transaction amount is greatly increased.

At present, the industrial-level personalized recommendation system mainly comprises two stages of recalling and sorting. During the recall phase, items that may be of interest to the user may be recalled based on various features, among others. In the ranking stage, the user's historical behavior may be combined into features and then input into a large-scale neural network that has been trained. In the stage, various behaviors of the user are combined and analyzed, each candidate commodity is scored, and the commodity with high score is selected to be preferentially recommended to the user.

Disclosure of Invention

The inventor analyzes the related technology to find that the sorting stage can sort the commodities recalled in the recalling stage more accurately. However, because the model is complex and the number of recalled commodities is huge, the time consumption for sorting is high at present, and the recommendation efficiency is reduced.

The embodiment of the invention aims to solve the technical problem that: how to improve the recommendation efficiency.

According to a first aspect of some embodiments of the present invention, there is provided a sorting method comprising: calculating a first ranking index of each candidate object according to the user operation data and the value index data of the candidate objects; performing first sequencing on the alternative objects by adopting a first sequencing index; acquiring partial alternative objects in the first sorting result as second sorting objects; and adopting a pre-trained sorting model to re-sort the second sorted objects so as to recommend the second sorted objects to the user according to the sequence of the second sorting result.

In some embodiments, a preset number of objects with the highest first ranking index in the first ranking result is obtained as the second ranking object.

In some embodiments, calculating the first ranking indicator for each candidate based on the user operation data and the value index data for the candidate comprises: and calculating the first ranking index of each candidate object by adopting a preset formula, wherein the independent variable of the preset formula is determined according to the user operation data and the value index data.

In some embodiments, the first ranking indicator is calculated using the following formula:

wherein R1 is the first ranking index of the candidate, n is the number of clicks of the candidate, m is the number of times the candidate is exposed, α is the adjustment coefficient and 0 ≦ α ≦ 1, bid is the advertisement bid value of the candidate, and gmv is the deal amount of the candidate.

In some embodiments, the sorting method further comprises: and determining the number of the second sequencing objects according to the preset feedback time and the computing power of the sequencing model.

In some embodiments, the sorting method further comprises: acquiring user operation data from at least one of a user click log and a user order log; value index data is obtained from at least one of a user order log, an advertisement bidding system.

In some embodiments, the user operation data comprises one or more of click data, browse data, favorite data, purchase data of each alternative object; the value index data includes one or more of advertising bid information, deal information for the subject.

According to a second aspect of some embodiments of the present invention, there is provided a sorting apparatus comprising: the first ranking index calculation module is configured to calculate a first ranking index of each candidate object according to the user operation data and the value index data of the candidate objects; the first ordering module is configured to adopt a first ordering index to carry out first ordering on the candidate objects; the second sequencing object acquisition module is configured to acquire part of the alternative objects in the first sequencing result as second sequencing objects; and the second sequencing module is configured to perform second sequencing on the second sequencing objects by adopting a pre-trained sequencing model so as to recommend the second sequencing objects to the user according to the sequence of the second sequencing result.

In some embodiments, the second ranking object obtaining module is further configured to obtain, as the second ranking object, a preset number of objects with the highest first ranking index in the first ranking result.

In some embodiments, the first ranking indicator calculating module is further configured to calculate the first ranking indicator of each candidate using a preset formula, wherein the independent variable of the preset formula is determined according to the user operation data and the value index data.

In some embodiments, the first ranking indicator calculation module is further configured to calculate the first ranking indicator using the formula:

In some embodiments, the sorting apparatus further comprises: and the second sequencing object quantity determining module is configured to determine the quantity of the second sequencing objects according to the preset feedback time and the computing capacity of the sequencing model.

In some embodiments, the sorting apparatus further comprises: the data acquisition module is configured to acquire user operation data from at least one of a user click log and a user order log; value index data is obtained from at least one of a user order log, an advertisement bidding system.

According to a third aspect of some embodiments of the present invention, there is provided a sorting apparatus comprising: a memory; and a processor coupled to the memory, the processor configured to perform any of the foregoing ordering methods based on instructions stored in the memory.

According to a fourth aspect of some embodiments of the present invention, there is provided a computer readable storage medium having a computer program stored thereon, wherein the program when executed by a processor implements any of the aforementioned sorting methods.

Some embodiments of the above invention have the following advantages or benefits: when the ranking model is adopted for second ranking, the ranked objects are not all the alternative objects but part of the alternative objects, so that the calculation pressure of the ranking model is reduced, and the recommendation efficiency is improved. In addition, the number of objects browsed by the user in a short time is limited, so that the use experience of the user is not influenced by the scheme of the invention.

Other features of the present invention and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

FIG. 1 is a flow diagram illustrating a sorting method according to some embodiments of the invention.

FIG. 2 is a flow diagram of a ranking model training method according to some embodiments of the invention.

FIG. 3 is a flow diagram of a recommendation method according to some embodiments of the invention.

FIG. 4 is a schematic diagram of a sequencing device according to some embodiments of the present invention.

Fig. 5 is a schematic view of an application scenario of a sorting apparatus according to some embodiments of the present invention.

Fig. 6 is a schematic structural diagram of a sorting apparatus according to other embodiments of the present invention.

Fig. 7 is a schematic diagram of a sorting apparatus according to further embodiments of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

FIG. 1 is a flow diagram illustrating a sorting method according to some embodiments of the invention. As shown in fig. 1, the sorting method of this embodiment includes steps S102 to S108.

In step S102, a first ranking index of each candidate is calculated from the user operation data and the value index data of the candidates.

The candidate may be selected according to a specific application scenario. For example, in a search scenario of a user, a candidate object may be an object having a search term entered by the user; in a promotional activity scenario, the candidate objects may be objects participating in a promotional activity; for a recommended location in the user's shopping cart page, the candidate object may be an object that is related to an object in the user's shopping cart; for the recommended position in the user's home page, the candidate object may be an object related to an object recently browsed, purchased by the user. The object in the present invention may refer to a commodity, or may refer to other items for recommendation, such as articles, advertisements, and the like.

The user operation data reflects the interest degree of the user in the alternative objects, and comprises one or more of click data, browse data, collection data and purchase data of the user for each alternative object. The data may be quantity or rate, such as exposure, number of clicks, click rate, and the like. In some embodiments, the user operation data may be obtained from at least one of a user click log, a user order log.

In some embodiments, the user operation data may be operation data of the user for a recommended page or layout, for example, operation data of the user in a home page, a shopping cart page, or a recommended page of an application or a website. Since the user already has a clear click or purchase intention in the search page, the recommendation plays a minor role. Therefore, the recommendation accuracy can be further improved by acquiring the operation data of the user aiming at the recommended page or layout.

The value index data reflects the revenue that the candidate can bring, either historical revenue or future revenue. The value index data may include, for example, one or more of advertising bid information, deal information for the subject. In some embodiments, the value index data may be obtained from at least one of a user order log, an ad bidding system.

In step S104, the candidates are first sorted using the first sorting index.

In step S106, a part of the candidates in the result of the first ranking is acquired as a second ranking object.

The specific selection of which candidates to perform the second ordering may be determined according to a specific application scenario. In some embodiments, a preset number of objects with the highest first ranking index in the first ranking result may be obtained as the second ranking object.

In step S108, the second ranking is performed again on the second ranked objects by using the pre-trained ranking model, so that the second ranked objects are recommended to the user according to the order of the second ranking results. For example, the data of the second sort object may be sent to a terminal of the user for presentation.

The pre-trained ranking model may be, for example, a pre-trained neural network model. In some embodiments, the object characteristics of the second ranking object and the user characteristics of the user to be recommended may be ranked together. For example, the above features may be input into the ranking model to obtain an output value, and the second ranking object may be ranked according to the size of the output value.

In some embodiments, the number of second sort objects may be determined according to a preset feedback time and a computational power of the sort model. For example, if the time required for the sorting system to respond is 400ms, the time occupied for performing the second sorting on different numbers of objects may be tested in advance, and the number that occupies a time shorter than and closest to 400ms may be selected as the number of the second sorted objects. Therefore, the number of recommended objects can be increased as much as possible under the condition of meeting the user experience.

By the method of the embodiment, when the ranking model is used for second ranking, the ranked objects are not all the candidate objects but part of the candidate objects, so that the calculation pressure of the ranking model is reduced, and the recommendation efficiency is improved. In addition, the number of objects browsed by the user in a short time is limited, so that the use experience of the user is not influenced by the scheme of the invention.

In some embodiments, the computational complexity for performing the first ordering is less than the computational complexity for performing the second ordering. Namely, the coarse sorting is adopted to sort all the candidate objects to screen out the commodities to be recommended, and then the commodities to be recommended are sorted finely to enable the recommendation result to be more accurate. For example, the first ranking index of each candidate object may be calculated using a preset formula, wherein the independent variable of the preset formula is determined according to the user operation data and the value index data.

The formula is adopted for calculation to carry out coarse sorting, namely first sorting, so that sorting with smaller calculation complexity can be carried out on all the candidate objects, and sorting with higher calculation complexity and more accuracy is carried out on part of the candidate objects as recommendation results, so that the recommendation efficiency is improved.

A calculation method of the first ranking index is exemplarily described below with reference to formula (1).

In formula (1), R1 is the first ranking index of the candidate, n is the number of clicks of the candidate, m is the number of times the candidate is exposed, α is the adjustment factor, and 0 ≦ α ≦ 1, bid is the advertisement bid value of the candidate, gmv is the deal amount of the candidate.

The click rate of the alternative object is embodied,

the click conversion rate of the alternative object is embodied.

May be used to estimate the revenue of the advertisement,

α can be used to adjust the ratio of advertising revenue to the amount of revenue.

In some embodiments, the ranking model may be trained in advance. An embodiment of the ranking model training method of the present invention is described below with reference to FIG. 2.

FIG. 2 is a flow diagram of a ranking model training method according to some embodiments of the invention. As shown in fig. 2, the ranking model training method of this embodiment includes steps S202 to S204.

In step S202, training data is obtained, each piece of training data includes features of an exposed object and features of a user to which the exposed object relates, and a flag value of the training data is determined according to whether the user clicks the object. For example, the object clicked by the user may be labeled as 1 and the object not clicked may be labeled as 0.

In step S204, the neural network model is trained using the training data to obtain a ranking model. The output of the sorting model is the probability of the user clicking the object corresponding to the input data, so that the objects can be sorted according to the output value, and the objects which are more likely to be clicked by the user can be displayed preferentially.

In some embodiments, a user may make multiple requests on the same page, and the present invention illustratively provides two approaches.

One way to do this is to re-execute the method of the embodiment of fig. 1 once in response to each request by the user. After a new request is made by a user, the operation of the user becomes historical data after the last request, so that the selection of the alternative object and the first sequencing index are changed, and the whole process can be executed again.

Another embodiment is to divide the candidate objects into a plurality of batches in advance, perform the second sorting on the objects of each batch in response to each request of the user, and return the sorting result. An embodiment of the ranking method of the present invention is described below with reference to fig. 3.

FIG. 3 is a flow diagram of a recommendation method according to some embodiments of the invention. As shown in fig. 3, the recommendation method of this embodiment includes steps S302 to S312.

In step S302, in response to a request from a user, a first ranking index of each candidate is calculated from user operation data and value index data of the candidates.

In step S304, the candidate objects are first sorted by using the first sorting index, and a first sorting result is obtained.

In step S306, the results of the first sorting are divided into a plurality of batches in order.

In step S308, the objects of the nth batch are re-ranked again by using the pre-trained ranking model, where the initial value of n is 1.

In step S310, the second ranked objects are recommended to the user in the order of the second ranking results.

In step S312, in response to the user' S request again, let n +1, and return to step S308.

Therefore, the alternative objects can be subjected to rough sorting at one time, fine sorting is performed in batches according to the request of the user, the sorting result of the batch is fed back, and the recommendation efficiency is improved.

An embodiment of the sorting apparatus of the invention is described below with reference to fig. 4.

FIG. 4 is a schematic diagram of a sequencing device according to some embodiments of the present invention. As shown in fig. 4, the sorting apparatus 40 of this embodiment includes: a first ranking index calculation module 410 configured to calculate a first ranking index of each candidate according to the user operation data and the value index data of the candidate; a first ranking module 420 configured to perform a first ranking on the candidates using a first ranking index; a second ranking object obtaining module 430, configured to obtain part of the candidate objects in the result of the first ranking as a second ranking object; and the second sorting module 440 is configured to perform the second sorting on the second sorted objects by using the pre-trained sorting model, so as to recommend the second sorted objects to the user according to the order of the second sorting result.

In some embodiments, the second ranked object obtaining module 430 is further configured to obtain a preset number of objects with the highest first ranking index in the first ranked result as the second ranked object.

In some embodiments, the first ranking indicator calculating module 410 is further configured to calculate the first ranking indicator of each candidate using a preset formula, wherein the independent variable of the preset formula is determined according to the user operation data and the value index data.

In some embodiments, the first ranking indicator calculation module 410 is further configured to calculate the first ranking indicator using the following formula:

In some embodiments, the sorting apparatus 40 further comprises: a second sort object number determination module 450 configured to determine the number of second sort objects according to a preset feedback time and a calculation capability of the sort model.

In some embodiments, the sorting apparatus 40 further comprises: a data obtaining module 460 configured to obtain user operation data from at least one of a user click log and a user order log; value index data is obtained from at least one of a user order log, an advertisement bidding system.

An application scenario embodiment of the sorting apparatus of the present invention is described below with reference to fig. 5.

Fig. 5 is a schematic view of an application scenario of a sorting apparatus according to some embodiments of the present invention. As shown in FIG. 5, the sorting apparatus 50 obtains a set of advertised items from the advertised item recall system 51 as candidate items, and reads data from the user click log 52, the advertiser bid system 53, and the user order log 54 to calculate a first sorting index. After the sorting calculation, the sorting device 50 pushes the sorted commodities to the client 55 for display. The user behavior collected by the client 55 is fed back to the user click log 52 and the user order log 54 for storage.

Fig. 6 is a schematic structural diagram of a sorting apparatus according to other embodiments of the present invention. As shown in fig. 6, the sorting apparatus 60 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610, the processor 620 being configured to perform the sorting method of any of the previous embodiments based on instructions stored in the memory 610.

Memory 610 may include, for example, system memory, fixed non-volatile storage media, and the like. The system memory stores, for example, an operating system, an application program, a Boot Loader (Boot Loader), and other programs.

Fig. 7 is a schematic diagram of a sorting apparatus according to further embodiments of the present invention. As shown in fig. 7, the sorting apparatus 70 of this embodiment includes: the memory 710 and the processor 720 may further include an input/output interface 730, a network interface 740, a storage interface 750, and the like. These

interfaces

730, 740, 750, as well as the memory 710 and the processor 720, may be connected, for example, by a bus 760. The input/output interface 730 provides a connection interface for input/output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 740 provides a connection interface for various networking devices. The storage interface 750 provides a connection interface for external storage devices such as an SD card and a usb disk.

An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, wherein the program is configured to implement any one of the foregoing sorting methods when executed by a processor.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims

1. A method of sorting, comprising:

calculating a first ranking index of each candidate object according to the user operation data and the value index data of the candidate objects;

performing first sequencing on the alternative objects by adopting a first sequencing index;

acquiring partial alternative objects in the first sorting result as second sorting objects;

and adopting a pre-trained sorting model to re-sort the second sorted objects so as to recommend the second sorted objects to the user according to the sequence of the second sorting result.

2. The ranking method according to claim 1, wherein a preset number of objects with the highest first ranking index in the first ranking result is obtained as the second ranking object.

3. The ranking method according to claim 1, wherein calculating a first ranking indicator for each candidate based on user operation data and value index data for the candidate comprises:

and calculating a first ranking index of each candidate object by adopting a preset formula, wherein the independent variable of the preset formula is determined according to the user operation data and the value index data.

4. The ranking method according to claim 3, wherein the first ranking indicator is calculated using the formula:

5. The sorting method of claim 1, further comprising:

and determining the number of the second sequencing objects according to the preset feedback time and the computing power of the sequencing model.

6. The sequencing method of any of claims 1-5, further comprising:

acquiring user operation data from at least one of a user click log and a user order log;

value index data is obtained from at least one of a user order log, an advertisement bidding system.

7. The sorting method according to claim 1,

the user operation data comprises one or more of click data, browsing data, collection data and purchase data of each alternative object;

the value index data includes one or more of advertising bid information, deal information for the subject.

8. A sequencing apparatus, comprising:

the first ranking index calculation module is configured to calculate a first ranking index of each candidate object according to the user operation data and the value index data of the candidate objects;

the first ordering module is configured to adopt a first ordering index to carry out first ordering on the candidate objects;

the second sequencing object acquisition module is configured to acquire part of the alternative objects in the first sequencing result as second sequencing objects;

and the second sequencing module is configured to perform second sequencing on the second sequencing objects by adopting a pre-trained sequencing model so as to recommend the second sequencing objects to the user according to the sequence of the second sequencing result.

9. A sequencing apparatus, comprising:

a memory; and

a processor coupled to the memory, the processor configured to perform the method of sorting of any of claims 1-7 based on instructions stored in the memory.

10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the sorting method according to any one of claims 1 to 7.