CN116795848A

CN116795848A - List data processing method and related equipment

Info

Publication number: CN116795848A
Application number: CN202210267994.8A
Authority: CN
Inventors: 朱斌; 张李阳; 许宏业; 许志豪; 王杏婷
Original assignee: Beijing Zitiao Network Technology Co Ltd
Current assignee: Beijing Zitiao Network Technology Co Ltd
Priority date: 2022-03-17
Filing date: 2022-03-17
Publication date: 2023-09-22

Abstract

The disclosure provides a list data processing method and related equipment. The method comprises the following steps: receiving a data updating request sent by a first terminal, and acquiring a target object and a score to be updated corresponding to the target object based on the data updating request; updating the first data set based on the target object and the score to be updated to obtain the target object and a target score corresponding to the target object; and updating the second data set based on the target object and the target score to obtain a list data list. The list data processing method and the related equipment can support the score storage of a large number of objects, and only a small number of objects focused by users are ranked, so that the method and the related equipment have the characteristics of high performance, low delay and high stability.

Description

List data processing method and related equipment

Technical Field

The disclosure relates to the technical field of communication, and in particular relates to a list data processing method and related equipment.

Background

Many applications are provided with a list function. The list can be used for ranking the list objects according to the index value of one or more dimensions as ranking rules.

The list function can be realized through a database, for example, a list object and a corresponding score can be stored as one piece of data in the database, and a ranking effect can be realized by using a database query function; alternatively, however, the database stability is poor and cannot resist very high query rate per second (QPS).

The list function can be realized through an ordered set (zset) of remote dictionary services (Redis), but the list storage length realized by using zset is limited, and when the number of list objects is excessive, the problem of service blockage caused by a large key of Redis can be solved.

Disclosure of Invention

Accordingly, an objective of the present disclosure is to provide a method and related device for processing list data.

Based on the above objects, the present disclosure provides a method for processing list data, the method comprising:

receiving a data updating request sent by a first terminal, and acquiring a target object and a score to be updated corresponding to the target object based on the data updating request;

updating the first data set based on the target object and the score to be updated to obtain the target object and a target score corresponding to the target object; the first data set is used for storing a plurality of objects and scores;

updating the second data set based on the target object and the target score to obtain list data; the second data set includes at least some of the plurality of objects and scores, the at least some of the objects and scores ordered in a preset rule.

Optionally, the updating the first data set based on the target object and the score to be updated to obtain the target object and the target score corresponding to the target object includes:

Setting the score of the target object in the first data set as the score to be updated in response to the target object belonging to the first data set, and/or modifying the score of the object to be ordered in the first data set based on the score to be updated to obtain the target object and the target score;

and in response to the target object not belonging to the first data set, inserting the target object and the score to be updated into the first data set to obtain the target object and the target score.

Optionally, the target score includes a primary score and a secondary score, the primary score is obtained based on the score to be updated, and the secondary score is obtained based on an update time in the data update request.

Optionally, the second data set includes a list object and a list score corresponding to the list object; the method further comprises the steps of:

and responding to the fact that the first grade of the target score is the same as the first grade of the list score, and ranking the target object and the list object based on the second grade of the target score and the second grade of the list score.

Optionally, the updating the second data set based on the target object and the target score to obtain the list data includes:

Updating the scores of the target objects in the second data set based on the target scores and sorting in response to the target objects belonging to the second data set;

and calling a join command of the second data set to join the target object and the target score to the second data set and sort in response to the target object not belonging to the second data set.

Optionally, after the inserting the target object and the target score into the second data set and sorting, the method further includes:

and deleting the objects and scores exceeding the list length in the second data set in response to the number of the objects in the second data set being greater than the preset list length.

Optionally, the method further comprises:

and receiving a list query request sent by a second terminal, and sending at least part of list data in the second data set to the second terminal based on the list query request.

Optionally, the method further comprises:

receiving a data deleting request sent by a third terminal, and acquiring an object to be deleted based on the data deleting request;

deleting the same objects and scores as the objects to be deleted in the second data set based on the objects to be deleted; or deleting the same objects and scores as the objects to be deleted in the first data set and the second data set based on the objects to be deleted.

Optionally, the method further comprises:

acquiring objects and scores of the changes in the first data set and the second data set within a preset time period;

comparing the changed objects and scores in the first data set with the changed objects and scores in the second data set;

responsive to the changed objects and scores in the first data set being different from the changed objects and scores in the second data set, the objects and scores in the second data set are modified based on the changed objects and scores in the first data set.

Optionally, the method further comprises:

generating an update request message for each update to store in a message queue in response to the frequency of updating the second data set based on the target object and the target score being greater than a preset value; the update request message is used for marking the target object;

and processing the update request message at a preset frequency, and updating the second data set based on the target object marked by the update request message and the target score of the target object in the first data set.

Optionally, the method further comprises:

Acquiring a list name, a sub list and a list period of the target object based on the data updating request;

and updating the first data set and the second data set based on the list names, sub-list and list period splicing information of the target objects.

Optionally, the list period includes a sliding period; the method further comprises the steps of:

acquiring the sliding speed of a list as a sub-list period;

updating the third data set based on the target object and the target score to obtain sub-list data in each sub-list period;

and obtaining the list data based on the outdated sub-list data in the second data set and the third data set.

The present disclosure also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing a method as claimed in any one of the preceding claims when executing the program.

The present disclosure also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform any of the methods described above.

The present disclosure also provides a computer program product comprising a computer-readable storage medium storing instructions that, when executed, cause at least one central processor unit of a computing device to perform a method according to any one of the preceding claims.

As can be seen from the foregoing, according to the method for processing list data and the related device, the scoring information of the objects is stored through the first data set, and the second data set including at least part of the objects and the scores in the first data set, which are ordered according to the preset rule, is updated based on the target object to be updated each time and the score to be updated, so as to obtain list data; thus, when receiving the list inquiry request of the user, the method can quickly acquire list data and return the list data to the user; the method can support scoring storage of a large number of objects, can sort only a small number of objects focused by users, and has the characteristics of high performance, low delay and high stability.

Drawings

In order to more clearly illustrate the technical solutions of the present disclosure or related art, the drawings required for the embodiments or related art description will be briefly described below, and it is apparent that the drawings in the following description are only embodiments of the present disclosure, and other drawings may be obtained according to these drawings without inventive effort to those of ordinary skill in the art.

Fig. 1 is a schematic diagram of a hardware structure of an exemplary electronic device provided in an embodiment of the disclosure;

FIG. 2 is a flowchart illustrating a method for processing list data according to an embodiment of the present disclosure;

fig. 3 is a schematic diagram of target scoring provided in an embodiment of the present disclosure.

Detailed Description

For the purposes of promoting an understanding of the principles and advantages of the disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the same.

It should be noted that unless otherwise defined, technical or scientific terms used in the embodiments of the present disclosure should be given the ordinary meaning as understood by one of ordinary skill in the art to which the present disclosure pertains. The terms "first," "second," and the like, as used in embodiments of the present disclosure, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that elements or items preceding the word are included in the element or item listed after the word and equivalents thereof, but does not exclude other elements or items. The terms "connected" or "connected," and the like, are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", etc. are used merely to indicate relative positional relationships, which may also be changed when the absolute position of the object to be described is changed.

As described in the background, the list function can be realized through the database in the prior art. For example, the list object and the corresponding score are stored as a piece of data in a mysql database, and each time a list query request is received, the query instruction select by of the mysql database is used to order and thus return list data. Because each query list needs to be ranked once by using the query instruction selector by, the ranking speed is low. Particularly, when the data volume of the list object stored in the mysql database is large, the ranking speed is slower, so that the ranking request with large data volume cannot be satisfied.

An ordered set (zset) of remote dictionary services (dis) may enable ranking of list objects. However, the list storage length implemented using zset is limited, and can only meet the ranking requirement of a small number of list objects (e.g., less than 5000). When the number of the list objects is large (for example, tens of thousands), the problem of Redis large key can occur, so that congestion of list service is caused.

In view of this, an embodiment of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements a method for processing list data when executing the program, so as to implement quick ranking of list data, improve ranking speed when there are more data, ensure stability of list services, and avoid congestion of list services.

Fig. 1 shows a schematic hardware structure of an exemplary electronic device 100 provided in this embodiment, where the device may include: a processor 102, a memory 104, an input/output interface 106, a communication interface 108, and a bus 110. Wherein the processor 102, the memory 104, the input/output interface 106, and the communication interface 108 are communicatively coupled to each other within the device via a bus 110.

The processor 102 may be a central processing unit (Central Processing Unit, CPU), an image processor, a neural Network Processor (NPU), a Microcontroller (MCU), a programmable logic device, a Digital Signal Processor (DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits. The processor 102 may be used to perform functions related to the techniques described in this disclosure. In some embodiments, processor 102 may also include multiple processors integrated as a single logical component.

The memory 104 may be configured to store data (e.g., instructions, computer code, etc.). As shown in fig. 1, the data stored by the memory may include program instructions (e.g., program instructions for implementing the method of implementing the present disclosure for interactive photographing) as well as data to be processed (e.g., the memory may store a document of a logic system design, etc.). The processor 102 may also access program instructions and data stored in the memory and execute the program instructions to perform operations on the data to be processed. The memory 104 may include volatile storage or nonvolatile storage. In some embodiments, memory 104 may include Random Access Memory (RAM), read Only Memory (ROM), optical disks, magnetic disks, hard disks, solid State Disks (SSD), flash memory, memory sticks, and the like.

The input/output interface 106 is used to connect the input/output module to connect the apparatus 100 with one or more peripheral devices to thereby enable information input and output. The input/output module may be configured as a component in the device 100 (not shown in the figures) or may be external to the device 100 to provide corresponding functionality. The peripheral devices may include input devices such as keyboards, mice, touch screens, microphones, various types of sensors, and so forth. In some embodiments, as shown in FIG. 1, the input device may also include a camera 112 for capturing images of the user. The peripheral device may also include a display screen (or display panel/display) 114, speakers, vibrators, indicator lights, etc. output devices. In some embodiments, the display 114 may provide a presentation screen for presenting images of the user and template images.

The communication interface 108 is used to connect a communication module (not shown in the figure) to enable communication interaction of the present device 100 with other devices. The communication module may implement communication through a wired manner (such as USB, network cable, etc.), or may implement communication through a wireless manner (such as mobile network, WIFI, bluetooth, etc.). It is to be understood that the communication manner is not limited to the specific examples described above. In some embodiments, the communication interface 108 may include any combination of any number of Network Interface Controllers (NICs), radio frequency modules, receivers, modems, routers, gateways, adapters, cellular network chips, etc.

Bus 110 includes a path to transfer information between components of the device (e.g., processor 102, memory 104, input/output interface 106, and communication interface 108). Bus 110 may include an internal bus (e.g., processor-memory bus), an external bus (USB port, PCI-E bus), and so forth.

It should be noted that although the above-described device only shows the processor 102, the memory 104, the input/output interface 106, the communication interface 108, and the bus 110, the device may include other components necessary to achieve proper operation in a particular implementation. Furthermore, it will be understood by those skilled in the art that the above-described apparatus may include only the components necessary to implement the embodiments of the present description, and not all the components shown in the drawings.

Wherein the electronic device 100 may be a server. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), basic cloud computing services such as big data and artificial intelligence platforms, but is not limited thereto.

The implementation of the list data processing method provided in the embodiments of the present disclosure is further described below with reference to the electronic device 100 shown in fig. 1.

First, a list service is created on the electronic device 100 that includes a first data set (CounterStore) and a second data set (RankStore). The first data set is used for storing a plurality of unordered or partially ordered objects and scores, the second data set is used for storing at least part of the objects and scores in the first data set, and the at least part of the objects and scores are ordered according to a preset rule. The configuration file for the list service is stored in a distributed Transaction (TCC).

After the list service is established, the data in the list service can be updated and queried based on the request of the user.

In some embodiments, the electronic device 100 receives a data update request sent by a terminal, and obtains a target object and a score to be updated corresponding to the target object based on the data update request. The terminal can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a wearable device or the like.

For example, in a live scene, a user may give a gift to a host through a terminal, and the terminal may generate a corresponding data update request based on the gift given by the user and transmit to the electronic device 100. In this embodiment, the target object includes account information of the user, and the score to be updated is a score (score) obtained based on the gift.

For example, in a game scenario, a user may acquire a game score through a terminal, and the terminal may generate a corresponding data update request based on the game score acquired by the user and transmit the data update request to the electronic device 100. In this embodiment, the target object includes account information of the user, and the score to be updated is a score (score) obtained based on the game score.

The electronic device 100 obtains and parses the data update request, thereby obtaining a target object and a score to be updated corresponding to the target object.

After obtaining the target object and the score to be updated, the electronic device 100 first calculates a current list period, and then updates the first data set based on the target object and the score to be updated to obtain the target object and the target score corresponding to the target object.

In this embodiment, the first data set is an unordered data set, so none of the plurality of objects and scores stored in the first data set are ordered. Therefore, when the first data set is updated, the target object and the score to be updated can be directly inserted into the first data set or the scores of the target objects existing in the first data set are updated based on the score to be updated, and the ranking is not needed when the first data set is updated, so that the ranking function is not affected even if the Query Per Second (QPS) rate is high.

In this embodiment, the target object and the target score thereof obtained based on each data update request are stored in the first data set, so that a complete list can be completely reproduced based on the stored data in the first data set. For example, when a list has fifty thousand members, the first data set stores data of fifty thousand objects and their scores.

Optionally, in this embodiment, the storage of the target object and the target score thereof may adopt a key-value (key-value) storage manner, where information of the target object, the object, and the like is a key, and information of the target score, the score, and the like is a value. Meanwhile, the electronic device 100 obtains information such as a list name (RankName), a sub list (SubKey), a list Period (Period) and the like, to which the target object belongs, based on the data update request, so as to facilitate generation of subsequent list data. The list name (RankName) is a unique mark created by the user and used for marking the list. The sub-list (sub-key) is used for realizing mutual isolation of different data, for example, for a regional active user list, there are sub-lists of sub-key= "a province" and sub-key= "B province", and the data of the two sub-keys are completely independent. For the Period of the list (Period), the current period= (current time-list creation time)/scroll time +1, for example, the creation time of the list a is zero point No. 10, the scroll Period is 1 day, then the Period No. 10 is 1, and the Period No. 11 is 2; the data and the ranking of each period are not mutually influenced and are independently calculated. The scroll time may include an hour list, a day list, zhou Bang, a month list, etc., and the user may also customize the scroll time.

In some embodiments, the target object and its target score may be stored in different storage manners based on the difference in the attributes of the target object and its target score. That is, the objects in the first data set and their scores may be stored in different storage manners, thereby improving storage performance. For example, because the redis string has good storage performance, and the data stored by using the redis string can be automatically cleared after expiration without occupying storage space, the redis string can be used for some list data (such as an hour list) which can be expired; however, the cost of the byteSQL storage is lower, so that for some data (such as a total list) which cannot expire, the byteSQL storage can be adopted; mysql is costly to store, but has better storage stability and security, so mysql can be used for some important list data.

In some embodiments, when there is no target object in the first data set, that is, the target object does not belong to the first data set, the target object and the score to be updated are directly inserted into the first data set, and the score to be updated of the target object is directly used as the target score of the target object, so that the target object and the target score are obtained.

In some embodiments, the score of the target object in the first data set may be updated when the target object is present in the first data set, i.e. the target object belongs to the first data set. In this embodiment, two updating methods are provided, including a settanklist command and an updatenRanklist command.

The settanklist command refers to setting the score of the target object in the first data set directly as the score to be updated, so as to obtain the target object and the target score. For example, assuming that the score to be updated of the target object memberA is 20, the score of memberA in the first data set is 30, and after updating the first data set through the settanklist, the score of memberA in the first data set is 20.

The updateRankList command refers to modifying the scores of the objects to be sorted in the first data set based on the scores to be updated and a preset calculation rule. For example, assuming that the score to be updated of the target object memberA is 20, the score of the memberA in the first data set is 30, the preset calculation rule is accumulation, and after the first data set is updated through the settanklist, the score of the memberA in the first data set is 50.

In the above embodiment, when the target object does not exist in the first data set, the target object and the score to be updated thereof are directly inserted into the first data set; when the target object exists in the first data set, two different commands, namely a SetRankList command and/or an UpdateRankList command, are provided to update the scores of the target objects in the first data set, so that the same target object only stores one piece of information in the first data set, and repeated storage of the scores of the same target object is avoided.

In this embodiment, even if the number of users with a list is extremely large, all the target objects and the target scores thereof may be stored in the first data set, and each target object stored in the first data set may appear in the second data set and be acquired by the user when the preset ranking rule of the second data set is satisfied.

Optionally, in this embodiment, in order to ensure idempotency of the update interface, that is, to ensure consistency of the update result during multiple update operations, a unique constraint condition (uniqueKey) may be set in parameters of the update interface, so that the list service sets a redislock (key is a RankName: subKey: period: uniqueKey) for a preset duration (for example, 10 minutes) before updating the first data set, so that it is ensured that the same uniqueKey as the same object as the list is updated only once within 10 minutes.

After the first data set is updated, the electronic device 100 updates the second data set based on the target object and the target score, and obtains the list data.

In this embodiment, the second data set includes at least some of the plurality of objects and scores, that is, each object and score thereof stored in the second data set is from the first data set; meanwhile, the second data set is a ordered data set, so that at least part of the objects and scores stored in the second data set are data ordered according to a preset rule. Meanwhile, since the second data set is updated based on the target object and the target score in the first data set, and the second data set has a preset list length, at least part of the objects and scores in the first data set are sorted according to a preset rule. Wherein, the list length rank limit is determined when the list is created, if the length is set to 1000, the objects with the rank lower than 1000 will not be ranked.

In some embodiments, the electronic device 100 can update the second data set based on the target object and the target score to obtain updated list data.

Wherein when there is a target object in the second data set, i.e. the target object belongs to the second data set, the score of the target object in the second data set may be updated and reordered based on the updated data. And when no target object exists in the second data set, namely the target object does not belong to the second data set, calling a joining command of the second data set to directly join the target object and the target score into the second data set, and reordering based on the updated data.

In some embodiments, the second data set may be a redis ordered data set stored in a rediszset storage manner. Accordingly, the electronic device 100 may call the add command zadd in zset to add the target object and the target score to the redis ordered data set and implement ranking, so that the plurality of objects and scores stored in the second data set may be ranked according to a preset rule order based on the plurality of objects and scores to provide the terminal with the list data. Alternatively, other ordering methods, such as bubble ordering, may be invoked to implement ordering of the data in the second data set, which is not limited in this embodiment.

In some embodiments, if the number of objects in the second data set is greater than the preset list length after the target objects and the target scores are directly added to the second data set, deleting the objects and the scores exceeding the list length in the second data set, so as to avoid that the number of the objects in actual ranking is greater than the preset list length. For example, when storing in the rediszset storage mode, the objects with the smallest scores can be deleted when the scores are sorted from small to large.

In this embodiment, the target object and the target score are stored in the second data set by adopting the splicing information (splice) of the list name (rank name), the sub list (sub key) and the list Period (Period), so that the data isolation of different sub lists is ensured; and when the list data is scrolled to the next period, the data in the current period is automatically changed.

In one particular embodiment, the list name is a regional list and the scroll period is 1 day. The list name (RankName) is rank_area, the subkey= "M" is M list, three list members are in the list, the membrane score is 30, the membrane score is 20, and the membrane score is 10. Assuming that it is the first day, the period is 1. The keys and values of the three list members in the first data set are as follows:

Key	Value
		rank_area:sichuan:1:memberA	30
rank_area:sichuan:1:memberB	20
		rank_area:sichuan:1:memberC	10

in the second data set, zset key of subkey= "M" is rank_area: M:1, and three data zsets (memberC: 10, memberb:20, memberC: 30) in order are stored.

In some embodiments, ranking of the list data in the second data set is achieved in a two-level ranking manner. Thus, the target scores stored in the first and second data sets comprise a primary score obtained based on the score to be updated and a secondary score obtained based on the update time in the data update request.

In this embodiment, a partial number of bits of the target score is determined based on the first-order score, and another partial number of bits of the target score is determined based on the second-order score.

The first-order score is obtained after updating the target object based on the score to be updated, that is, the first-order score is used for representing the score after updating the target object based on the score to be updated.

The secondary score is obtained based on an update time of the data update request, for example, a time stamp of the update time is obtained according to a generation time for recording the data update request or a reception time for the electronic device 100 to receive the data update request, and the secondary score is determined based on the time stamp; the second fraction is used for representing the generation time of the data update request or the fraction corresponding to the receiving time of the data update request, and the second fraction is generated according to the time sequence.

In some embodiments, the target score may be represented in binary, octal, decimal, hexadecimal, or the like. Taking the target score as an example, an N-bit binary representation is used, as shown in fig. 3, to mark the N-bit binary from right to left as the 1 st bit, the 2 nd bit … … ith bit, and the i+1st bit … … nth bit. Taking the 1 st to i th bits as a second grade fraction to represent the update time in the data update request, and taking the i+1 th to N th bits as a first grade fraction to represent the score to be updated; alternatively, bits 1 to i may be used as the first order fraction and bits i+1 to N may be used as the second order fraction, which is not limited in this embodiment.

Alternatively, the number of bits of the primary fraction and the number of bits of the secondary fraction may be the same, i.e., i=n/2; alternatively, the number of bits of the primary fraction and the number of bits of the secondary fraction may be different, which is not limited in this embodiment.

In one particular embodiment, the target score employs a 64-bit binary data store, the first 32-bit binary of which is used to store a primary score, i.e., a score that is directly obtained based on the score to be updated, such as a score that is directly obtained based on game credits, gifts, and the like; the 32-bit binary is then used to store the secondary score, which may be obtained based on the update time in the data update request, e.g. a timestamp may be obtained based on the generation time of the data update request in the terminal, which timestamp is stored in the secondary score. Thus, when sorting is performed in the second data set, even if the primary scores of different objects are the same, sorting can be performed based on the secondary scores, so that the problem that sorting cannot be realized by two target objects with the same target score can be avoided.

For example, assuming that the second data set includes a list object and a list score corresponding to the list object, and the list score also includes a first score and a second score, if the first score of the target score is the same as the first score of the list score, the target object and the list object are ranked based on the second score of the target score and the second score of the list score. When the secondary fraction is used to characterize a timestamp, the ordering may be based on the scoring generation times of the two objects.

After obtaining the list data, the electronic device 100 may receive a list query request sent by the second terminal, and send at least part of the list data in the second data set to the terminal based on the list query request. The second terminal may be the same as or different from the first terminal device, which is not limited in this embodiment.

In this embodiment, the user queries the list data through the second terminal, the second terminal may generate a list query request based on the query of the user and send the list query request to the electronic device 100, the electronic device 100 receives and parses the list query request, obtains the ranking of the list interval or the ranking of a certain object queried by the user, and obtains a corresponding result based on the list data in the second data set and returns the result to the second terminal.

When the query of the user is ranking in the list section, for example, the ranking and the score of the top 20 members in the query list, the electronic device 100 may acquire the names, the ranking and the score of the top 20 members in the list data by using the instruction for acquiring the object information of the specified section in the ordered set, and return the names, the ranking and the score to the second terminal. For example, when the redis storage data is adopted in the second data set, list data of a corresponding section can be obtained based on the zrange command in combination with count and offset, and returned to the second terminal.

When the query of the user is the ranking of a certain object, for example, the ranking and the score of the 10 th member in the query list, the electronic device 100 may acquire the name, the ranking and the score of the 10 th member in the list data by adopting the instruction for acquiring the specified object information and return the name, the ranking and the score to the second terminal. For example, when the second data set adopts redis storage data, list data of the corresponding object can be obtained based on the zrank command and returned to the second terminal. Optionally, in order to reduce the number of queries of the zrank command, the electronic device 100 may first determine whether the object is stored in the first data set, if yes, continue to obtain the list data of the corresponding object based on the zrank command, otherwise, indicate that the object is not in the current list, and need not to invoke the query of the zrank command.

Optionally, the list service may also include a fourth data set (DataStore) for storing other information of the object, such as comments, etc. Optionally, the data in the fourth data set may be stored using an abase storage system.

In the above embodiments, the list service may be run in the form of a remote procedure call service (RPC). For example, the user sends a data update request to a third party server through the first terminal, and the third party server forwards the data update request to the electronic device 100 to update the list data. Meanwhile, the user sends the list query request to the third-party server through the second terminal, the third-party server forwards the list query request to the electronic device 100, and after the electronic device 100 obtains at least part of the list data in the second data set based on the list query request and sends the list data to the third-party server, the third-party server forwards at least part of the list data in the second data set to the second terminal.

In this embodiment, the call to the list service includes operations such as creation of the list, update, modification, and deletion of the list data, which can provide corresponding interfaces to the third party server, and the third party server can call the corresponding interfaces to implement operations such as creation of the list, update, modification, and deletion of the list data.

In this embodiment, the list service provides the third party platform with an automatic update task configuration, i.e., the user can configure the automatic update task, set eventName of MQ messages to be consumed, field conversion rules (including which fields need to be used in the message), filtering rules (e.g., only messages with a field > 0), update rules (which list is updated, which fields are subject, which fields are scored). After the task is online, the list service can update the list according to the automatic consumption MQ, and the user only needs to develop the code for inquiring the list part, so that the service access cost of the user can be reduced.

In some embodiments, the electronic device 100 can also delete information for certain objects in the list data based on the user's request. Specifically, the electronic device 100 may receive a data deletion request sent by the third terminal, and acquire the object to be deleted based on the data deletion request; then, the electronic device 100 deletes the object and score identical to the object to be deleted in the second data set based on the object to be deleted; or deleting the object and the score which are the same as the object to be deleted in the first data set and the second data set based on the object to be deleted.

In this embodiment, when a user wants to delete the ranking of some objects in a list, for example, in a live scene, an object memberB located on the list leaves a live room, and then the information of the memberB in the live room needs to be deleted. The electronic device 100 receives the data deletion request sent by the third terminal, analyzes the data deletion request to obtain the object to be deleted, namely the memberB, and then deletes the ranking and the score of the memberB from the list data, thereby ensuring the real-time update of the list data.

In this embodiment, two deletion modes are provided, including a removentdelette command and a DeleteRankList command.

The removetnotdelete command is to delete the same object and score as the object to be deleted in the second data set based on the object to be deleted memberB, without deleting the information of the object to be deleted memberB in the first data set. For example, on-line user lists in a live scene are ranked according to the live gift amount for users in a certain live room. The 10 th gift with value a is sent by the memberB, and then the memberB temporarily leaves the living room, at this time, the electronic device 100 calls the removentdelet command to delete the gift from the list in the living room, so that the memberB is removed from the list. But the score of memberB is also stored in the first data set. In this live broadcast, the memberB enters the live broadcast room again, and the electronic device 100 invokes the updatenranklist command to set the memberB on the list of the live broadcast room for ranking again, where the score of the memberB is still a previous a, instead of 0. The present disclosure can provide better services than a list implemented using only Redis zset that cannot do the RemovNotDelete command.

Deleterank list is to delete the same objects and scores as the objects to be deleted in the first data set and the second data set based on the objects to be deleted, namely, the scores of the objects to be deleted are completely deleted from the list service, and the scores are recalculated when updating next time.

In some embodiments, the electronic device 100 may further obtain the object and the score that change in the first data set and the second data set within a preset period of time; comparing the subject and score of the change in the first data set with the subject and score of the change in the second data set; and in response to the object and score of the first data set being different from the object and score of the second data set, modifying the object and score of the second data set based on the object and score of the first data set, thereby avoiding the difference of scores of the same object in the second data set and the first data set caused by the failure of updating the data in the second data set, ensuring the consistency of scores of the same object in the second data set and the first data set, and ensuring the accuracy of the list data.

In this embodiment, each time the data in the first data set is updated, including inserting the target object and the score to be updated into the first data set, modifying the score of the target object in the first data set based on the score to be updated, and deleting the object and the score to be deleted in the first data set, the electronic device 100 generates an action flow data message, for example, a kafka message, where the action flow data message records the data updated at this time, including a list name, an object, a score, a timestamp, and the like. And the action flow data message may be stored in a database, such as a data warehouse tool (hive), to facilitate updating of detail information for subsequent query data.

For an object memberA, since it stores a score in both the first data set and the second data set, there may be a problem in that the scores in the first data set and the second data set are not consistent. For example, when updating the score, there may be a failure to update the second data set after updating the first data set, resulting in inconsistent scores in the first data set and the second data set. Thus, a consistency check is required. Specifically, every first preset time (for example, 10 minutes), the electronic device 100 may acquire the object and the score that change in the first data set and the second data set in the preset time period based on the action stream data message stored in the database, and compare the object and the score, if the object and the score are the same, repair is not required; and if the scores are different, updating the scores in the second data set based on the scores in the first data set, so as to ensure the data consistency.

Optionally, in this embodiment, when the objects and scores of the first data set and the second data set that change are obtained based on the action flow data message, there may be multiple score updates for one object, where it is not necessary to perform consistency check on each update of the object, but only on the result after the last update, so that the action flow data message may be processed based on a deduplication command (for example, a flink command), so that the repeated objects and scores are eliminated, and the number of consistency checks is reduced.

In some embodiments, since the update of the list data in the second data set is based on the RankName-subKey dimension, all update requests in this dimension update the second data set. Some updated QPS of the list are relatively high, and easily cause the problem of Redis hot key, for example, if some total list does not use subkeys and the updated QPS is 5 ten thousand, then the updated QPS of the list rank store zset is also 5 ten thousand, and the list becomes a Redis hot key.

In this case, asynchronous update may be used to avoid Redis hot key problem. Specifically, in response to the frequency of updating the second data set based on the target object and the target score being greater than a preset value, generating an update request message for each update and storing the update request message in a message queue; the update request message is used for marking the target object; and processing the update request message at a preset frequency, and updating the second data set based on the target object marked by the update request message and the target score of the target object in the first data set.

In this embodiment, the frequency of updating the second data set may be the update request number of times per second of all target objects or a certain target object in the second data set, that is, the update QPS, when the update frequency is greater than a preset value (for example, QPS is greater than 5 ten thousand), an asynchronous update switch of all target objects or a specific target object with an update frequency greater than a preset value may be turned on, when the asynchronous update switch is turned on, the objects in the first data set are still updated in real time, and when the second data set is updated based on the objects and scores in the first data set, the real-time update is not performed, so that the problem of dis hot key is avoided.

For example, the update request times per second, or update QPS, for each target object may be calculated, e.g., aggregate statistics may be aggregated using a flink task. For QPS exceeding a preset value (e.g., twenty thousand), an asynchronous update switch of the target object is opened. Thus, when updating the second data set based on the target object, the update is discarded because the asynchronous update switch of the target object is open, and an update request message is generated for the update and stored in a Message Queue (MQ), which is consumed by the list service at a fixed and lower QPS.

And closing an asynchronous update switch after the frequency of updating the second data set based on the target object and the target score is smaller than or equal to a preset value, and updating the first data set and the second data set in real time. The unconsumed MQ messages during this period may continue to be consumed until the MQ messages are consumed. And when the MQ message is consumed each time, updating the second data set by adopting a target score corresponding to the target object in the first data set based on the target object marked in the MQ message, thereby avoiding the Redis hot key problem and ensuring the consistency of the second data set, the object and the score in the first data set.

In some embodiments, the list service of the present disclosure may support a list that scrolls according to a period, that is, the period of the list is a scroll period, and after the scroll list reaches the next period, the data of the previous period is not used, and the data is restarted to be accumulated.

In other embodiments, the list service of the present disclosure may also support a sliding list, i.e., the list period may also include a sliding period. When the list period is a sliding period, the electronic device 100 obtains the sliding speed of the list as a sub-list period; updating the third data set based on the target object and the target score to obtain sub-list data in each sub-list period; and obtaining the list data based on the outdated sub-list data in the second data set and the third data set.

The sliding list includes a sliding window (sliding window) and a sliding speed (sliding speed), for example, for the last 7 days of XXX list, the sliding window is 7 days, and the sliding speed is 1 day.

In this embodiment, when the list period is a sliding period, the electronic device 100 first obtains the sliding speed of the list as a sub-list period, and additionally sets a third data set for recording updated data in each sub-list period, that is, the target object and the target score. If the sliding list is the first request in the sub-list period, the sliding list filling is triggered, the object of the list in the previous period is used, the data of the object in the sub-list period needing to be eliminated is subtracted, and then the new list is inserted.

For example, a recent 3-day interactive list, the sliding window is 3 days, and the sliding speed is 1 day. On the first day, subject a increased by 10 points and subject B increased by 20 points; the following day, subject a increased by 20 points and subject C increased by 50 points; on the third day, subject D increased by 10 points; then the current data is as follows:

days (period)	Small list data	Normal list data
			First day	A:10,B:20	(B:20,A:10)
The next day	A:20,C:50	(C:50,A:30,B20)
			Third day	D:10	(C:50,A:30,B20,D:10)

On the fourth day, the user queries the list for the first time, triggers the sliding list filling, uses the list data of the last period to subtract the small list data to be eliminated, namely uses the data of the third day (C: 50, A:30, B20, D: 10), subtracts the small list data of the first day (the sliding window size is 3 days, and only the data of the second, third and fourth days should be contained on the fourth day, so that the data of the first day is eliminated), and obtains the list data of the fourth day (C: 50, A:20, D: 10).

According to the electronic device disclosed by the embodiment of the disclosure, scoring information of all objects is stored through a first data set, and at least part of the objects and scoring second data sets which are ordered according to a preset rule in the first data set are updated based on target objects updated each time and scores to be updated, so that list data are obtained; thus, when receiving the list inquiry request of the user, the method can quickly acquire list data and return the list data to the user; the method can support scoring storage of a large number of objects, only orders the objects focused by a small number of users, has the characteristics of high performance, low delay and high stability, can support 20 thousands of updated QPS (quick service PS) and 100 thousands of inquired QPS by a single list, has read-write delay within 50ms, and maintains daily stability SLA at 99.99%.

Based on the same inventive concept, the embodiment of the disclosure also provides a list data processing method. As shown in fig. 2, the list service includes a first data set and a second data set; the method comprises the following steps:

step S101, a data update request sent by a first terminal is received, and a target object and a score to be updated corresponding to the target object are obtained based on the data update request.

Step S103, updating the first data set based on the target object and the score to be updated to obtain the target object and the target score corresponding to the target object; the first data set is for storing a plurality of objects and scores that are unordered or partially ordered.

Step S105, updating the second data set based on the target object and the target score to obtain list data; the second data set includes at least a portion of the objects and scores in the first data set ordered by a preset rule.

In this embodiment, scoring information of all the objects that are not ranked or are partially ranked is stored through the first data set, and partial objects and scores in the first data set are ranked through the second data set, so that list data is obtained; thus, when receiving the list inquiry request of the user, the method can quickly acquire list data and return the list data to the user; the method can support scoring storage of a large number of objects, can sort only a small number of objects focused by users, and has the characteristics of high performance, low delay and high stability.

Optionally, after step S105, the method further includes:

step S107, a list query request sent by a second terminal is received, and at least part of list data in the second data set is sent to the terminal based on the list query request.

In this embodiment, when the target object does not exist in the first data set, the target object and the score to be updated thereof are directly inserted into the first data set; when the target object exists in the first data set, the scores of the target objects in the first data set can be directly updated, so that only one piece of information is stored in the first data set by the same target object, and repeated storage of the scores of the same target object is avoided.

The secondary score is obtained based on an update time of the data update request, for example, a time stamp of the update time is obtained according to a generation time for recording the data update request or a reception time for the electronic device 100 to receive the data update request, and the secondary score is determined based on the time stamp; the second fraction is used for representing the generation time of the data update request or the fraction corresponding to the receiving time of the data update request, and the second fraction is generated according to the time sequence. Optionally, the second data set includes a list object and a list score corresponding to the list object; the method further comprises the steps of:

In this embodiment, when sorting is performed in the second data set, even if the primary scores of different objects are the same, sorting can be performed based on the secondary scores, so that the problem that sorting cannot be achieved for two target objects with the same target score can be avoided.

in response to the target object belonging to the second data set, the scores of the target objects in the second data set are updated and ranked based on the target scores, step 201.

Step S205, in response to the target object not belonging to the second data set, invoking a join command of the second data set to join the target object and the target score into the second data set and sorting.

In this embodiment, the adding command zadd in the second data set, i.e., the redis ordered data set, may be invoked to add the target object and the target score to the redis ordered data set and implement ranking, so that the objects and the scores stored in the second data set may be ranked according to a preset rule order based on the objects and the scores, so as to provide the terminal with the list data.

Optionally, after the inserting the target object and the target score into the second data set and sorting, the method further includes: and deleting the objects exceeding the list length and scoring in the second data set in response to the number of the objects in the second data set being greater than the preset list length, so as to avoid that the number of the objects in actual ranking is greater than the preset list length.

Optionally, the method further comprises:

deleting the object and the score which are the same as the object to be deleted in the second data set based on the object to be deleted; or deleting the object and the score which are the same as the object to be deleted in the first data set and the second data set based on the object to be deleted.

Optionally, the method further comprises:

acquiring the object and the score which change in the first data set and the second data set within a preset time period;

comparing the subject and score of the change in the first data set with the subject and score of the change in the second data set;

And in response to the object and score of the first data set being different from the object and score of the second data set, modifying the object and score of the second data set based on the object and score of the first data set, thereby avoiding the difference of scores of the same object in the second data set and the first data set caused by the failure of updating the data in the second data set, ensuring the consistency of scores of the same object in the second data set and the first data set, and ensuring the accuracy of the list data.

Optionally, the method further comprises:

generating an update request message for each update to store in a message queue in response to the frequency of updating the second data set based on the target object and the target score being greater than a preset value; the update request message is used for marking the target object; and consuming the update request message at a preset frequency, and updating the second data set based on the target object marked by the update request message and the target score of the target object in the first data set, so that the Redis hot key problem can be avoided, and the consistency of the second data set and the objects and scores in the first data set is ensured.

Optionally, the method further comprises:

acquiring the sliding speed of a list as a sub-list period;

and obtaining the list data based on the outdated sub-list data in the second data set and the third data set. It should be noted that the method of the embodiments of the present disclosure may be performed by a single device, such as a computer or a server. The method of the embodiment can also be applied to a distributed scene, and is completed by mutually matching a plurality of devices. In the case of such a distributed scenario, one of the devices may perform only one or more steps of the methods of embodiments of the present disclosure, the devices interacting with each other to accomplish the methods.

It should be noted that the foregoing describes some embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments described above and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Based on the same inventive concept, corresponding to any of the above-described embodiments of the method, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method as described in any of the above embodiments.

The computer readable media of the present embodiments, including both permanent and non-permanent, removable and non-removable media, may be used to implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device.

The storage medium of the foregoing embodiments stores computer instructions for causing the computer to perform the method of any of the foregoing embodiments, and has the advantages of the corresponding method embodiments, which are not described herein.

Based on the same inventive concept, the present disclosure also provides a computer program product, corresponding to the method of any of the embodiments described above, comprising a computer program. In some embodiments, the computer program is executable by one or more processors to cause the processors to perform the described methods. Corresponding to the execution bodies corresponding to the steps in the embodiments of the method, the processor executing the corresponding step may belong to the corresponding execution body.

The computer program product of the above embodiment is configured to cause a processor to perform the method of any of the above embodiments, and has the advantages of the corresponding method embodiments, which are not described herein.

Those of ordinary skill in the art will appreciate that: the discussion of any of the embodiments above is merely exemplary and is not intended to suggest that the scope of the disclosure, including the claims, is limited to these examples; the technical features of the above embodiments or in the different embodiments may also be combined under the idea of the present disclosure, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in details for the sake of brevity.

Additionally, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown within the provided figures, in order to simplify the illustration and discussion, and so as not to obscure the embodiments of the present disclosure. Furthermore, the devices may be shown in block diagram form in order to avoid obscuring the embodiments of the present disclosure, and this also accounts for the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform on which the embodiments of the present disclosure are to be implemented (i.e., such specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the disclosure, it should be apparent to one skilled in the art that embodiments of the disclosure can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative in nature and not as restrictive.

While the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of those embodiments will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

The disclosed embodiments are intended to embrace all such alternatives, modifications and variances which fall within the broad scope of the appended claims. Accordingly, any omissions, modifications, equivalents, improvements, and the like, which are within the spirit and principles of the embodiments of the disclosure, are intended to be included within the scope of the disclosure.

Claims

1. A method for processing list data, the method comprising:

2. The method of claim 1, wherein updating the first data set based on the target object and the score to be updated to obtain the target object and a target score corresponding to the target object comprises:

3. The method of claim 2, wherein the target score comprises a primary score and a secondary score, the primary score obtained based on the score to be updated, the secondary score obtained based on an update time in a data update request.

4. The method recited in claim 3, wherein the second data set includes a list object and a list score corresponding to the list object; the method further comprises the steps of:

5. The method of claim 1, wherein the updating the second data set based on the target object and target score to obtain list data comprises:

6. The method of claim 5, wherein after inserting the target object and target score into the second data set and ordering, further comprising:

7. The method as recited in claim 1, further comprising:

and receiving the data sent by the second terminal and sending the data to the second terminal.

8. The method as recited in claim 1, further comprising:

9. The method according to any one of claims 1-8, further comprising:

10. The method according to any one of claims 1-8, further comprising:

11. The method as recited in claim 1, further comprising:

12. The method of claim 11, wherein the list period comprises a slide period; the method further comprises the steps of:

acquiring the sliding speed of a list as a sub-list period;

13. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 12 when the program is executed by the processor.

14. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1 to 12.

15. A computer program product comprising a computer-readable storage medium storing instructions that, when executed, cause at least one central processor unit of a computing device to perform the method of any one of claims 1 to 12.