CN113112148B

CN113112148B - Evaluation method for evaluation result of recommendation system model and electronic equipment

Info

Publication number: CN113112148B
Application number: CN202110384899.1A
Authority: CN
Inventors: 李凯元; 冯志超; 陈昱硕; 侯宇蓬; 王鹏飞; 赵鑫
Original assignee: Beijing University of Posts and Telecommunications
Current assignee: Beijing University of Posts and Telecommunications
Priority date: 2021-04-09
Filing date: 2021-04-09
Publication date: 2022-08-05
Anticipated expiration: 2041-04-09
Also published as: CN113112148A

Abstract

The disclosure provides an evaluation method for a recommendation system model evaluation result and electronic equipment. The method is applied to a PyTorch recommendation system framework, and a two-dimensional fraction tensor of a positive sample tensor in front of a negative sample tensor is obtained by obtaining a one-dimensional fraction tensor of each user to a candidate commodity in a single batch and performing corresponding rearrangement processing according to an evaluation mode; generating out-of-order arrangement of column indexes of the two-dimensional fraction tensor to obtain a rearranged two-dimensional fraction tensor; calculating the original index tensor of the top-k commodity of each user in a single batch in parallel; the steps are circulated to obtain the original indexes of the top-k commodities of all the users; calculating Boolean index tensors of intersections of all positive sample tensors and the top-k commodities of all users in original indexes of the top-k commodities of all users; and calculating the Boolean index tensor and all positive sample tensors to obtain the values of all relevant top-k indexes. The evaluation is accurate, and the running speed and the running space are considered.

Description

Evaluation method for evaluation result of recommendation system model and electronic equipment

Technical Field

The disclosure relates to the technical field of intelligent recommendation, in particular to an evaluation method for a recommendation system model evaluation result and electronic equipment.

Background

Personalized recommendation systems are now becoming indispensable and ubiquitous systems for connecting users and items in various scenarios, such as multimedia recommendations, news recommendations, merchandise recommendations, etc. The algorithms of the existing recommendation system mainly comprise two types, one type is a non-deep learning algorithm based on rules, and the other type is a deep learning algorithm based on a neural network. The PyTorch is a deep learning framework and occupies a main position in a personalized recommendation system, but the existing PyTorch recommendation system framework has the risk of failure of an evaluation result, so that the evaluation of the evaluation result of the PyTorch recommendation system framework is very necessary. However, the efficiency of the current evaluation stage is not high enough, and the evaluation process is time-consuming. Therefore, an effective method for evaluating the evaluation result of the recommended system model suitable for pytorre is needed.

Disclosure of Invention

In view of this, an object of the present disclosure is to provide an evaluation method for recommending a system model evaluation result and an electronic device.

Based on the above purpose, the present disclosure provides an evaluation method for an evaluation result of a recommendation system model, which is applied to a PyTorch recommendation system framework, and the method includes:

s100, obtaining a recommendation system model evaluation result, obtaining a one-dimensional score tensor of each user on the candidate commodity in a single batch, and performing corresponding rearrangement processing according to an evaluation mode to obtain a two-dimensional score tensor; in the two-dimensional fractional tensor, a positive sample tensor of each user is positioned in front of a negative sample tensor;

s200, generating disordered arrangement of the column indexes of the two-dimensional fraction tensor, and rearranging the columns of the two-dimensional fraction tensor by utilizing the disordered arrangement of the column indexes to obtain a rearranged two-dimensional fraction tensor;

s300, according to the rearranged two-dimensional fraction tensor, an index corresponding to the top-k commodity of each user is calculated in parallel, the out-of-order arrangement of the column indexes is indexed by the index, and an original index tensor of the top-k commodity of each user in a single batch is obtained;

s400, the steps S100 to S300 are circulated to obtain the original indexes of the top-k commodities of each user in all batches, and the original indexes of the top-k commodities of all the users are obtained by splicing the top-k commodities according to lines; calculating Boolean index tensors of intersections of all positive sample tensors and the top-k commodities of all users in original indexes of the top-k commodities of all users;

and S500, calculating the Boolean index tensor and all positive sample tensors to obtain values of all relevant top-k indexes.

In some embodiments, the performing the corresponding rearrangement processing according to the evaluation mode to obtain the two-dimensional fraction tensor specifically includes:

if the evaluation mode is negative sampling, negative sampling rearrangement is carried out;

and if the evaluation mode is full sorting, performing full sorting rearrangement.

In some embodiments, the negative sample reordering comprises:

dividing the one-dimensional fraction tensor of each user on the candidate commodities in a single batch according to the number of the candidate commodities of each user;

and filling the division result to obtain a two-dimensional fraction tensor.

In some embodiments, the full-order reordering comprises:

filling scores of the commodities which are interacted historically in the candidate commodities;

and recombining the one-dimensional fraction tensor to obtain a two-dimensional fraction tensor.

In some embodiments, the obtaining the rearranged two-dimensional fractional tensor specifically includes:

randomly generating a bijection table by the two-dimensional fractional tensor, wherein the bijection table comprises a column index and random arrangement of the column index;

and rearranging the columns of the two-dimensional fraction tensor by utilizing the random arrangement of the column indexes in the bijection table to obtain a rearranged two-dimensional fraction tensor.

In some embodiments, the obtaining the original index of the top-k commodity of each user in a single batch specifically includes:

acquiring indexes of the first k commodities in the rearranged two-dimensional fraction tensor;

and accessing the column indexes in the bijection table by using the indexes of the first k commodities to obtain the original index of the top-k commodity of each user in a single batch.

In some embodiments, said computing the boolean index tensor of the intersection of all positive sample tensors and all users 'top-k commodities in the original index of all users' top-k commodities specifically comprises:

comparing the original indexes of top-k commodities of all users with the numerical value of the positive sample tensor of the user;

if the value is less than the preset value, the filling value is 1; if it is greater, the padding value is 0.

In some embodiments, the obtaining of the values of all relevant top-k indicators specifically includes:

according to the index type of the single integral corresponding index, performing corresponding highly parallel processing on the Boolean index tensor and all positive sample tensors to obtain a processed tensor;

performing function processing on the processed tensor to obtain a value of a single integral corresponding index;

and circulating the processing of the single overall corresponding indexes to obtain the values of all the overall corresponding indexes.

In some embodiments, the obtaining the processed tensor specifically includes:

when the index is the recall rate, accumulating and summing the Boolean index tensor according to rows;

dividing the result of summation by the tensor of the positive sample of the user according to the line to obtain a processed tensor;

when the index is the precision, accumulating and summing the Boolean index tensor according to rows;

generating a one-dimensional tensor with the length of top-k according to the result of the summation;

and dividing the result of summation by the one-dimensional tensor according to the row to obtain the processed tensor.

An embodiment of the present disclosure further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the method according to any one of the foregoing embodiments is implemented.

From the above, the method for evaluating the evaluation result of the recommendation system model provided by the disclosure is applied to a PyTorch recommendation system framework, the one-dimensional score tensor of each user on the candidate commodity in a single batch is obtained by obtaining the evaluation result of the recommendation system model, and the two-dimensional score tensor is obtained by performing corresponding rearrangement processing according to the evaluation mode; in the two-dimensional fractional tensor, a positive sample tensor of each user is positioned in front of a negative sample tensor; generating a disordered array of the column indexes of the two-dimensional fractional tensor, and rearranging the columns of the two-dimensional fractional tensor by utilizing the disordered array of the column indexes to obtain a rearranged two-dimensional fractional tensor; according to the rearranged two-dimensional fraction tensor, an index corresponding to the top-k commodity of each user is calculated in parallel, the out-of-order arrangement of the column indexes is indexed by the index, and an original index tensor of the top-k commodity of each user in a single batch is obtained; the steps are circulated to obtain the original indexes of the top-k commodities of each user in all batches, and the original indexes of the top-k commodities of all the users are obtained by splicing the original indexes according to lines; calculating Boolean index tensors of intersections of all positive sample tensors and the top-k commodities of all users in original indexes of the top-k commodities of all users; and calculating the Boolean index tensor and all positive sample tensors to obtain the values of all relevant top-k indexes. The front k commodities can be randomly selected when the scores of the plurality of commodities and the positive sample tensor are the same, and the accuracy of the evaluation result is effectively improved. Meanwhile, the method is also suitable for various evaluation scenes; and the complexity of time and space can be reduced, and the evaluation efficiency is higher. In addition, the method can be conveniently transplanted to other programs, and has good application prospect.

Drawings

In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings needed to be used in the description of the embodiments or related technologies are briefly introduced below, and it is obvious that the drawings in the following description are only embodiments of the present disclosure, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

Fig. 1 is a schematic flow chart of a method for evaluating a recommendation system model evaluation result according to an embodiment of the present disclosure;

FIG. 2 is a schematic diagram illustrating results of top-3 index acquisition performed by 6 users and 5 commodities in a full-sequence manner according to the embodiment of the present disclosure;

FIG. 3 is a schematic diagram of a result of sampling a top-2 index by performing 1:2 negative sampling on 4 users and 5 commodities according to an embodiment of the present disclosure;

FIG. 4 is an overall flowchart of an evaluation method for an evaluation result of a recommendation system model according to an embodiment of the present disclosure;

fig. 5 is a schematic view of an electronic device of an embodiment of the disclosure.

Detailed Description

For the purpose of promoting a better understanding of the objects, aspects and advantages of the present disclosure, reference is made to the following detailed description taken in conjunction with the accompanying drawings.

It is to be noted that technical terms or scientific terms used in the embodiments of the present disclosure should have a general meaning as understood by those having ordinary skill in the art to which the present disclosure belongs, unless otherwise defined. The use of the terms "comprising" or "including" and the like in the embodiments of the present disclosure is intended to mean that the elements or items listed before the term cover the elements or items listed after the term and their equivalents, without excluding other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

Referring to fig. 1 and fig. 4, an embodiment of the present disclosure provides an evaluation method for an evaluation result of a recommendation system model, which is applied to a PyTorch recommendation system framework, and the method includes:

s400, the steps S100 to S300 are circulated to obtain the original indexes of the top-k commodities of each user in all batches, and the original indexes are combined to obtain the original indexes of the top-k commodities of all users; calculating Boolean index tensors of intersections of all positive sample tensors and the top-k commodities of all users in original indexes of the top-k commodities of all users;

In some embodiments, in step S100, the recommended system model evaluation result may be obtained by directly obtaining an output result of the model. The recommendation system model evaluation result comprises evaluation results of users in all batches, namely the evaluation results of all users. The evaluation result comprises a one-dimensional score tensor of each user on the candidate commodity. In the two-dimensional fractional tensor, the rows represent users and the columns represent goods. In the columns, the positive sample tensors are all located in front of the negative sample tensors. By the step S100, the two-dimensional fraction tensors of different evaluation modes can be unified.

In an application scenario, the one-dimensional score tensor for each user on a candidate good in a single batch may be batch _ scores, which may be understood as that batch _ scores includes scores of all candidate goods of users in a batch. The resulting two-dimensional fractional tensor can be batch _ users × n _ items; wherein, batch _ users represents the number of all users in a single batch, and n _ items represents the maximum number of candidate commodities owned by the users in the single batch.

And the evaluation mode refers to an evaluation mode adopted for generating the evaluation result of the recommendation system model. The evaluation mode may include negative sampling and full sorting. The negative sampling refers to sampling from commodities which have not been interacted by a user according to a certain positive and negative sample proportion. For example, 1: 100, i.e. for 1 positive sample, it matches 100 negative samples, and there are 101k candidate items in this case, where the first k are positive samples and the last 100k are negative samples. The full ranking means that for one user, if the number of positive samples to be evaluated is k, the number of historical interactive commodities is m, and the total commodity set comprises n commodities, then the commodities which are not interacted by the user in the total commodity set are all candidate commodities, so that the total number of the candidate commodities is n-m.

In some embodiments, the negative sample reordering comprises:

and filling the division result to obtain a two-dimensional fraction tensor.

In an application scenario, a one-dimensional score tensor of each user on a candidate commodity in a single batch can be divided specifically by calling a torch. Specifically, the result of the division can be filled by calling a torch, nn, rn, utils, pad _ sequence function of the pytorque. Wherein the fill value is-np.inf, which is negative infinity. Through the filling value filling, the final ranking can not be influenced by the filling part, so that the sorting problem caused by inconsistent quantity of candidate commodities of different users is avoided.

In some embodiments, the full-order reordering comprises:

and recombining the one-dimensional fraction tensor, and performing displacement processing to obtain the two-dimensional fraction tensor.

In an application scenario, scores of historically interacted commodities in candidate commodities can be filled through assignment operation, and the filling value is-np. The padding value is negative infinity. Through the filling value filling, recommendation of historical interactive commodities can be avoided. The one-dimensional fraction tensor can be reorganized by calling the view function of PyTorch. And the shifting process can enable the positive sample tensor to be arranged in front of the negative sample tensor through the assignment operation of the shifting type.

In the application scenario, in step S200, the rank-out is arranged as rand _ index; the two-dimensional fraction tensor is reordered to rand _ score.

In an application scenario, in a bijective table, the column index x may be 0, 1.. and n _ items, and y is a random arrangement of the n column indices, which may be generated by torch. Index _ select can be used to rearrange the columns of the two-dimensional fractional tensor.

In some embodiments, in step S300, the obtaining the original index of the top-k product of each user in a single batch specifically includes:

and accessing the column indexes in the bijection table by using the indexes of the first k commodities to obtain the original index tensor of the top-k commodities of each user in a single batch.

In an application scenario, the index idx of the first k commodities can be obtained by calling the torch. And obtaining the index of the top-k commodity on the column index x by using the index idx as the index for accessing the column index x in the bijective table, and obtaining the original index tensor of the top-k commodity of each user in a single batch. The original index tensor is the two-dimensional index tensor of batch _ users × top-k.

Through steps S200 to S300, the positions of the positive samples do not have the particularity of always being ahead of the negative samples, and when the same-score commodity problem is encountered by using the sorting algorithm such as torch, the first k commodities can be randomly selected. Therefore, the failure of the evaluation result caused by the fact that the exposure probability of the positive sample is easily and artificially increased in the model test under the negative sampling condition when a plurality of commodities have the same scores as the positive sample is avoided.

In some embodiments, in step S400, the obtaining the original indexes of the top-k commodities of all the users specifically includes:

the original indexes of the top-k commodities of each user in all batches are spliced according to rows.

In the application scenario, the original index of all users' top-k commodities is an index of n _ users × k. In an embodiment of 6 users and 5 commodities, the index of n _ users × k obtained after splicing is the two-dimensional tensor a in fig. 2. In the embodiment of 4 users and 5 commodities, the index of n _ users × k obtained after splicing is the two-dimensional tensor a in fig. three.

The calculating the boolean index tensor of the intersection of all positive sample tensors and all user top-k commodities in the original index of all user top-k commodities specifically includes:

comparing the original indexes of top-k commodities of all users with the number of positive sample tensors of the users;

This step may be understood as calculating the intersection of the positive sample tensor and the top-k commodities of all users by comparing the number of the original indexes of the top-k commodities of all users with the number of the positive sample tensor, and the boolean index tensor in the original indexes of the top-k commodities of all users. In the application scenario, the original index of all users' top-k commodities is the index of n _ users k. In the 6 user, 5 commodity embodiment, the positive sample tensor is tensor b in fig. 2, and the resulting boolean index tensor is C in fig. 2. In the 4 user, 5 commodity embodiment, the positive sample tensor is tensor b in fig. 3, and the resulting boolean index tensor is C in fig. 3. The broadcasting characteristic of PyTorch is utilized in the step, because the position of each row of positive samples in the two-dimensional fraction tensor is always in the front row, after the index of top-k commodities on the row index x is obtained, only the index of the front k commodities is needed to be judged whether to be smaller than the number of the positive samples, namely the positive samples appear in the front k positions, the positions are filled with 1, otherwise, the Boolean index tensor is obtained by filling with 0.

In some embodiments, step S500 may be implemented by setting K to be the set of top-K that the user needs to calculate, and setting m to be the maximum value of K. Wherein K is a top-K set which needs to be calculated by a user. And is

And when the number 1 is less than or equal to i is less than or equal to m, the order evaluation index under all top-i conditions is calculated in parallel on the GPU by using the Boolean index pos _ index obtained in the step S400 and the set positive sample tensor pos _ len _ list.

The obtaining of the values of all relevant top-k indexes specifically includes:

and circulating the processing of the corresponding indexes of the single whole to obtain the values of the corresponding indexes of all the whole.

In some embodiments, the obtaining the processed tensor specifically includes:

In an application scenario, by using the boolean index tensor and the positive sample tensor, various highly parallel tensor operations defined by PyTorch and based on CUDA can be used to calculate the relevant top-k index. When the index is Recall (Recall), and Recall is calculated, a tensor Z can be obtained by accumulating and summing the tensor C by rows by a torch. When the index is Precision (Precision), when Precision is calculated, a tensor Z can be obtained by accumulating and summing the tensor C by a torch.

In an application scenario, the processed tensor can be processed through a torch.

The method provided by the embodiment of the disclosure can adapt to most evaluation scenes through an integrated evaluation flow. And under the condition of negative sampling and full-sequencing evaluation modes, the problem of invalid recommendation caused by the fact that the scores of a plurality of candidate commodities are the same as the scores of the positive samples can be solved. Meanwhile, various highly parallel operations such as top-k calculation and the like on the GPU realized by PyTorch are fully utilized, the time complexity is greatly reduced, and the problems that the evaluation process is time-consuming and the efficiency is low due to the fact that only a serial evaluation method is adopted or only some simple parallel skills (multithreading and C + + expansion) are used are solved. Meanwhile, by means of calculating the top-k commodities of the users in batches, the candidate commodities of all the users are prevented from being stored at the same time, and space complexity is effectively reduced. The structure realized by the method is easy to expand, new sequencing indexes are easy to increase, the running speed and the occupied space are considered, and the time and the space are saved. Easy to understand, run stably, and can be conveniently transplanted to other programs.

It should be noted that the method of the embodiments of the present disclosure may be executed by a single device, such as a computer or a server. The method of the embodiment can also be applied to a distributed scene and completed by the mutual cooperation of a plurality of devices. In such a distributed scenario, one of the devices may only perform one or more steps of the method of the embodiments of the present disclosure, and the devices may interact with each other to complete the method.

It should be noted that the above describes some embodiments of the disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments described above and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

Based on the same inventive concept, corresponding to the method of any embodiment described above, the present disclosure further provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and operable on the processor, where the processor executes the program to implement the method for evaluating the recommended system model evaluation result according to any embodiment described above.

Fig. 5 is a schematic diagram illustrating a more specific hardware structure of an electronic device according to this embodiment, where the electronic device may include: a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040, and a bus 1050. Wherein the processor 1010, memory 1020, input/output interface 1030, and communication interface 1040 are communicatively coupled to each other within the device via bus 1050.

The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present disclosure.

The Memory 1020 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 1020 may store an operating system and other application programs, and when the technical solution provided by the embodiments of the present specification is implemented by software or firmware, the relevant program codes are stored in the memory 1020 and called to be executed by the processor 1010.

The input/output interface 1030 is used for connecting an input/output module to input and output information. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.

The communication interface 1040 is used for connecting a communication module (not shown in the drawings) to implement communication interaction between the present device and other devices. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).

Bus 1050 includes a path that transfers information between various components of the device, such as processor 1010, memory 1020, input/output interface 1030, and communication interface 1040.

It should be noted that although the above-mentioned device only shows the processor 1010, the memory 1020, the input/output interface 1030, the communication interface 1040 and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will appreciate that the above-described apparatus may also include only those components necessary to implement the embodiments of the present description, and not necessarily all of the components shown in the figures.

The electronic device of the above embodiment is used for implementing the corresponding method for evaluating the recommendation system model evaluation result in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.

Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method for evaluating a recommendation system model evaluation result according to any of the above-mentioned embodiments.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, for storing information may be implemented in any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method for evaluating the recommendation system model evaluation result according to any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which are not described herein again.

Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the idea of the present disclosure, also technical features in the above embodiments or in different embodiments may be combined, steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of brevity.

In addition, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown in the provided figures for simplicity of illustration and discussion, and so as not to obscure the embodiments of the disclosure. Furthermore, devices may be shown in block diagram form in order to avoid obscuring embodiments of the present disclosure, and this also takes into account the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the embodiments of the present disclosure are to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the disclosure, it should be apparent to one skilled in the art that the embodiments of the disclosure can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.

While the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic ram (dram)) may use the discussed embodiments.

The disclosed embodiments are intended to embrace all such alternatives, modifications and variances which fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalents, improvements, and the like that may be made within the spirit and principles of the embodiments of the disclosure are intended to be included within the scope of the disclosure.

Claims

1. An evaluation method for an evaluation result of a recommendation system model is applied to a PyTorch recommendation system framework, and is characterized by comprising the following steps:

s400, looping the steps S100 to S300 to obtain the original indexes of the top-k commodities of each user in all batches, and splicing the original indexes according to lines to obtain the original indexes of the top-k commodities of all users; calculating Boolean index tensors of intersections of all positive sample tensors and the top-k commodities of all users in original indexes of the top-k commodities of all users;

2. The evaluation method according to claim 1, wherein the performing the corresponding rearrangement processing according to the evaluation mode to obtain the two-dimensional fractional tensor specifically comprises:

3. Method of evaluation according to claim 2, characterized in that the negative sample rearrangement comprises:

and filling the division result to obtain a two-dimensional fraction tensor.

4. The evaluation method according to claim 2, wherein the full rank shuffling comprises:

5. The evaluation method according to claim 1, wherein the deriving a rearranged two-dimensional fractional tensor specifically comprises:

6. The evaluation method according to claim 5, wherein the obtaining an original index of top-k goods for each user within a single batch specifically comprises:

obtaining indexes of the first k commodities in the rearranged two-dimensional fraction tensor;

7. The evaluating method according to claim 1, wherein the calculating the boolean index tensor of the intersection of all positive sample tensors and all user top-k commodities in the original index of all user top-k commodities specifically comprises:

8. Evaluation method according to claim 1, wherein said obtaining values of all relevant top-k indicators comprises in particular:

9. An evaluation method according to claim 8, wherein the deriving the processed tensor specifically comprises:

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of claims 1 to 9 when executing the program.