CN113986936A

CN113986936A - Data processing method and device, electronic equipment and storage medium

Info

Publication number: CN113986936A
Application number: CN202111224338.1A
Authority: CN
Inventors: 林明明
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2021-10-19
Filing date: 2021-10-19
Publication date: 2022-01-28

Abstract

The embodiment of the application discloses a data processing method, a data processing device, electronic equipment and a storage medium, wherein the method comprises the following steps: responding to a data updating operation of a user, and determining a characteristic identifier and an evaluation index of an element indicated by the data updating operation; the evaluation index represents the sequence of elements in the element set, and the feature identifier of each element in one element set is the same as that of the element set; adding the evaluation index of the element indicated by the updating operation to the data storage unit matched with the characteristic identifier; if the reading instruction is detected, identifying the reading instruction to determine a target feature identifier; determining a target element set corresponding to the target feature identifier and a target data storage unit for storing evaluation indexes of all elements in the target element set; and reading the target evaluation index in the target data storage unit according to a reading strategy determined based on the size of the evaluation index. The problem of data concurrency conflicts caused when multiple data update operations occur simultaneously is solved.

Description

Data processing method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.

Background

For user-oriented applications, in a scenario of a high-concurrency large flow field, a high-performance cache component is usually used to handle high-frequency read-write operations.

Common cache components are Redis and Memcache, which are k-v memory databases, and keys and values are in one-to-one correspondence. When an application is operated, no judgment is made when k-v data is written, so that mutual overlapping phenomena exist, and thus, a conflict problem exists for data writing in a concurrent environment.

In the related art, the following problems generally exist in the solution to this problem: in the scheme of the distributed lock, all concurrent threads compete for the same lock, a performance bottleneck can occur, extra resources are occupied, and the distributed lock cannot be used in a production environment; in the scheme of the message queue, the parallel-to-serial operation causes the data updating operation to be slow, so that the cached data is not timely enough; in the scheme of using a transaction mechanism in Redis, the realization logic is complex, errors are easy to occur, and the performance is not high; the problem cannot be solved essentially in the scheme of replacing other cache components or changing Redis source codes. Therefore, how to solve the problem of data collision in a concurrent environment is particularly important.

Disclosure of Invention

The embodiment of the application provides a data processing method and device, electronic equipment and a storage medium, and further solves the problem of data conflict in a concurrent environment.

In a first aspect, an embodiment of the present application provides a data processing method, including:

responding to a data updating operation of a user, and determining a characteristic identifier and an evaluation index of an element indicated by the data updating operation; the evaluation index characterizes the sequence of the elements in element sets, and the feature identifier of each element in one element set is the same as that of the element set;

adding the evaluation index of the element indicated by the data updating operation to a data storage unit matched with the characteristic identifier in a database; wherein, the database comprises at least one data storage unit; each data storage unit stores evaluation indexes of each element in one element set;

if a reading instruction is detected, identifying the reading instruction to determine a target feature identifier;

determining a target element set corresponding to the target feature identifier, and a target data storage unit for storing evaluation indexes of each element in the target element set;

and reading the target evaluation index in the target data storage unit according to a reading strategy determined based on the size of the evaluation index.

In some exemplary embodiments, if the read policy determined based on the size of the evaluation index is a read-maximum evaluation index, the reading and the evaluation index in the target data storage unit include:

and reading the maximum evaluation index in the target data storage unit.

In some exemplary embodiments, the reading the maximum evaluation index in the target data storage unit includes:

sending an element reading command to the database, wherein the element reading command comprises a reading start character and a reading end character, and the reading start character and the reading end character are determined according to the maximum evaluation index in the target data storage unit;

and reading the maximum evaluation index according to the reading start character and the reading end character.

In some exemplary embodiments, the adding, to the data storage unit matching the feature identifier, the evaluation index of the element indicated by the data update operation includes:

sending an element adding command to the database, wherein the element adding command comprises an evaluation index of the element indicated by the data updating operation and a characteristic identifier of the element indicated by the data updating operation;

determining a data storage unit matched with the characteristic identification of the element indicated by the data updating operation;

and adding the evaluation index of the element indicated by the data updating operation to the data storage unit matched with the characteristic identifier.

In some exemplary embodiments, before reading the target evaluation index in the target data storage unit according to the reading policy determined based on the size of the evaluation index, the method further includes:

for each data storage unit, eliminating other evaluation indexes except the maximum evaluation index in the data storage unit;

the reading strategy determined based on the size of the evaluation index is the evaluation index in the target data storage unit after the elimination operation is performed;

reading the target evaluation index in the target data storage unit according to a reading strategy determined based on the size of the evaluation index, including:

and reading the rest evaluation indexes in the target data storage unit after the elimination operation.

In some exemplary embodiments, the rejecting, for each data storage unit, other evaluation indexes except the maximum evaluation index in the data storage unit includes:

for each data store, sending an element filtering command to the data store by sending the element filtering command to the database; wherein the element filter command includes a filter start and a filter stop, the filter start determined from a minimum rating measure in the data storage unit and the filter stop determined from a next largest rating measure in the data storage unit;

and applying the filtering start character and the filtering stop character to remove other evaluation indexes except the maximum evaluation index in the data storage unit.

In some exemplary embodiments, after reading the target evaluation index in the target data storage unit, the method further includes:

and displaying the target elements corresponding to the target evaluation indexes.

In a second aspect, an embodiment of the present application provides a data processing apparatus, including:

a first determination unit configured to perform a data update operation in response to a user, and determine a feature identification and an evaluation index of an element indicated by the data update operation; the evaluation index characterizes the sequence of the elements in element sets, and the feature identifier of each element in one element set is the same as that of the element set;

an adding unit configured to perform addition of an evaluation index of an element indicated by the data updating operation to a data storage unit matched with the feature identifier in a database; wherein, the database comprises at least one data storage unit; each data storage unit stores evaluation indexes of each element in one element set;

the instruction identification unit is configured to identify the reading instruction to determine a target characteristic identifier when the reading instruction is detected;

a second determination unit configured to perform determination of a target element set corresponding to the target feature identifier, and a target data storage unit that stores evaluation indexes of respective elements in the target element set;

a reading unit configured to execute reading of the target evaluation index in the target data storage unit in accordance with a reading policy determined based on a size of the evaluation index.

In some exemplary embodiments, if the read policy determined based on the size of the evaluation index is a read-maximum evaluation index, the reading unit is specifically configured to perform:

and reading the maximum evaluation index in the target data storage unit.

In some exemplary embodiments, the reading unit is specifically configured to perform:

and reading the maximum evaluation index in the target data storage unit according to the reading start character and the reading end character.

In some exemplary embodiments, the adding unit is configured to perform:

In some exemplary embodiments, the method further includes a filtering unit, before reading the target evaluation index in the target data storage unit according to a reading policy determined based on the size of the evaluation index, the filtering unit is configured to perform:

In some exemplary embodiments, the filtering unit is specifically configured to perform:

In some exemplary embodiments, the method further includes a presentation unit, and after reading the target evaluation index in the target data storage unit, the presentation unit is configured to perform:

In a third aspect, an embodiment of the present application further provides an electronic device, including:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement any of the methods as provided in the first aspect of the application.

In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, where instructions, when executed by a processor of an electronic device, enable the electronic device to perform any one of the methods as provided in the first aspect of the present application.

In a fifth aspect, an embodiment of the present application provides a computer program product comprising computer programs/instructions which, when executed by a processor, implement any of the methods as provided in the first aspect of the present application.

The technical scheme provided by the embodiment of the application at least has the following beneficial effects:

determining a characteristic identifier and an evaluation index of an element indicated by user operation while responding to the user operation; the evaluation indexes can represent the sequence of the elements in the element sets, and the feature identifiers of the elements in one element set are the same as the feature identifiers of the element sets, so that the sequence of the target elements in the element sets corresponding to the feature identifiers can be determined through data updating operation of a user, and then the evaluation indexes of the elements indicated by the updating operation are added into the data storage unit matched with the feature identifiers in the database, and therefore the data storage unit is updated. When the reading command is detected, the reading command is identified so as to determine a target feature identifier, and further a target element set corresponding to the target feature identifier is determined, and a target data storage unit is used for storing evaluation indexes of all elements in the target element set; and reading the evaluation index in the target data storage unit according to a reading strategy determined based on the size of the evaluation index. In this way, since the reading strategy is determined based on the evaluation index, even if a plurality of data updating operations exist simultaneously in a short time, the evaluation index can be read by taking the size of the evaluation index as the reading criterion, and the problem of data conflict in a concurrent environment is solved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic view of an application scenario of a data processing method according to an embodiment of the present application;

fig. 2 is an application scenario diagram of a data processing method according to an embodiment of the present application;

fig. 3 is a flowchart of a data processing method according to an embodiment of the present application;

FIG. 4 is a schematic diagram of an element structure provided in an embodiment of the present application;

FIG. 5 is a schematic diagram of another element structure provided in an embodiment of the present application;

FIG. 6 is a schematic diagram of another element structure provided in an embodiment of the present application;

FIG. 7 is a schematic diagram of another element structure provided in an embodiment of the present application;

fig. 8 is an application scenario diagram of a data processing method according to an embodiment of the present application;

fig. 9 is a block diagram of a data processing apparatus according to an embodiment of the present application;

fig. 10 is a schematic view of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions of the present application better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

Hereinafter, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

(1) In the embodiments of the present application, the term "plurality" means two or more, and other terms are similar thereto.

(2) "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

(3) A server serving the terminal, the contents of the service such as providing resources to the terminal, storing terminal data; the server is corresponding to the application program installed on the terminal and is matched with the application program on the terminal to run.

(4) The terminal device may refer to an APP (Application) of a software class, or may refer to a client. The system is provided with a visual display interface and can interact with a user; is corresponding to the server, and provides local service for the client. For software applications, except some applications that are only run locally, the software applications are generally installed on a common client terminal and need to be run in cooperation with a server terminal. After the internet has developed, more common applications include e-mail clients for e-mail receiving and sending, and instant messaging clients. For such applications, a corresponding server and a corresponding service program are required in the network to provide corresponding services, such as database services, configuration parameter services, and the like, so that a specific communication connection needs to be established between the client terminal and the server terminal to ensure the normal operation of the application program.

(5) Redis an open-source, network-enabled, memory-based, distributed, optionally persistent key-value pair storage database written using ANSI C (American National Standards Institute C, a standard promulgated by the American National Standards Institute for C language).

(6) Memcache is an open-source, high-performance, distributed, memory-based key-value pair database.

(7) Zset is a data structure, ordered set, in Redis.

(8) Rpc (Remote Procedure Call), refers to a node in a network environment requesting a service provided by another node.

(9) Qps (Queries Per Second), the number of requests Per Second that can be processed.

(10) High concurrency: environments where a large number of network requests are received at the same time or in a very short time.

(11) Distributed: a system of computer nodes communicating over a network and coordinated to perform a common task.

(12) Transaction: a sequence of database operations that access and potentially manipulate various data items, either all or none, is an indivisible unit of work.

(13) Idempotent: the effect of any multiple execution is the same as the effect of one execution.

(14) And (3) synchronization: after the request is submitted, the execution result is returned to the user after the operation is completely executed.

(15) Asynchronization: after the request is submitted, the execution result is not waited for, and the submitting party cannot perceive the execution result.

In a specific practical process, for user-oriented applications, in a scenario of a high-concurrency large flow field, a high-performance cache component is usually used to handle high-frequency read-write operations.

Taking a scene of a video APP (Application program) as an example, at a certain entry of the video APP, for a series that a user watches, the latest watched episode number needs to be recorded and displayed on an APP page. For example, if the user watches 1 st, 2 nd, 5 th, 6 th, 8 th episode of a series, the number of the latest watched episodes is 8; if the user does not see episodes in the normal order of episode, but sees

episodes

3, 1, 5, 7, 2 in sequence, then the number of episodes most recently viewed is 7, i.e., the maximum value. Since the value needs to be frequently read and displayed on the front-end page, which requires the server interface to respond quickly, this part of data is put into a high-performance memory database (cache), and each read/write (query/update of the latest set number of users) is an operation cache.

The commonly used cache components in the industry are Redis and Memcache which are k-v memory databases, keys and values are in one-to-one correspondence, and the structure is simple and easy to use.

When watching a series play on an App, the number of episodes is switched by sliding up and down, a user is not necessarily interested in all episodes, the user may continuously slide a page, and the sliding speed may be high, so that a server interface is frequently requested, an update command is continuously sent to a Redis cache, and a concurrent conflict exists in the update operation of the latest number of episodes of the same series: if the user clicks on set 7 and set 8 at the same time in a short time, two commands are sent to the Redis server, and the latter command may arrive before the former command due to the uncertainty of network transmission, so the Redis will write set id 8 to value first, and then update the late set id 7 to value, so that, unlike the expected result, which is the id expected to be the largest recorded in the Redis, the id 7 will cover the value of id 8, and when the user refreshes the APP page, the user will see the wrong data, and see that the latest set (the largest set) viewed by the user is set 7 rather than set 8. Referring to fig. 1, the latest set viewed by oneself is presented to the user.

The essential reason for the problem is that Redis does not judge when writing k-v data, so that the phenomenon of mutual coverage exists, the same Memcache caches exist, the same is the commonality of the k-v memory database, and the conflict problem exists for data writing in a concurrent environment.

In addition to the application scenarios mentioned above, there are many common functions and scenarios that can suffer from this problem. In the related art, there are probably the following solutions:

(1) distributed lock: before each update request reaches the Redis server, a global lock needs to be acquired first, and the thread which has preempted the lock can set the set $ userId _ $ tvId $ episodeId, otherwise, the lock needs to be contended after another thread releases the lock. The advantages are that: the method can solve the problem of critical resources and avoid the phenomenon that values of the same key in Redis are mutually covered. The disadvantages are as follows: because all concurrent threads compete for the same lock, a performance bottleneck occurs, and additional resources are occupied, which cannot be used in a production environment.

(2) Message queue: all update requests are serialized by using a message queue component, and are queued to be processed one by one in sequence, so that only one update request operates the Redis server at the same time. The advantages are that: because the request is processed in a serialization way, the concurrency phenomenon does not exist, and the problem of data collision does not occur. The disadvantages are as follows: originally, parallel requests are changed into serial requests after manual intervention, and in serial logic, before a set command is sent to Redis, value of key needs to be obtained by using a get command for carrying out prepositive judgment (whether the current set id is larger than the set id in Redis or not) each time, so that updating operation is necessarily slowed down, cache data is not timely enough, and the old latest set number id is always seen on an APP page.

(3) Using the transaction mechanism in Redis: and (3) using a watch-multi-exec mechanism in Redis, and carrying out watch on the key each time set operation is carried out, so that the key is prevented from being changed. The advantages are that: this scheme can reduce the probability that values of a key are overlaid with each other to some extent. The disadvantages are as follows: the transaction in Redis is not a real transaction, and in extreme cases, a phenomenon of concurrent conflict still occurs, and the implementation is logically complex, easy to make mistakes, and low in performance (because the whole process contains several commands, there are several times of Rpc round trips).

(4) Swap with other cache components or modify the Redis source code. The disadvantages are as follows: other cache components such as the Memcache have similar defects, and the problem of data collision in a concurrent environment cannot be avoided. As for the change of the Redis source code, the set command of the Redis source code supports the prepositive judgment, the problem can be solved theoretically, but research personnel is required to be quite familiar with the Redis source code, the function of other commands can not be influenced after the change, and the realization is difficult.

Therefore, the application provides a data processing method by using a special data structure Zset in Redis and using characteristics of Zset order and Redis atomicity execution commands, and the data processing method is specifically realized as follows: responding to a data updating operation of a user (the user clicks a certain set), and determining a characteristic identifier (Zhang III, West notes) and an evaluation index (ID is 8) of an element (such as the 8 th set) indicated by the data updating operation; the evaluation index represents the sequence of elements in the element set, and the feature identifiers of all the element sets are different; adding the evaluation index of the element indicated by the data updating operation to a data storage unit matched with the characteristic identifier in the database; the database comprises at least one data storage unit; if the reading instruction is detected, identifying the reading instruction to determine a target feature identifier; determining a target element set corresponding to the target feature identifier, and a target data storage unit for storing evaluation indexes of all elements in the target element set; and reading the target evaluation index in the target data storage unit according to a reading strategy determined based on the size of the evaluation index.

After introducing the design concept of the embodiment of the present application, some simple descriptions are provided below for application scenarios to which the technical solution of the embodiment of the present application can be applied, and it should be noted that the application scenarios described below are only used for describing the embodiment of the present application and are not limited. In specific implementation, the technical scheme provided by the embodiment of the application can be flexibly applied according to actual needs.

Fig. 2 is a schematic view of an application scenario of the data processing method according to the embodiment of the present application. The application scenario includes a plurality of terminal devices 201 (including terminal device 201-1, terminal device 201-2, … … terminal device 201-n), server 202. The terminal device 201 and the server 202 are connected through a wireless or wired network, and the terminal device 201 includes but is not limited to an electronic device such as a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, and a smart television. The server 202 may be a server, a server cluster composed of several servers, or a cloud computing center. The server 202 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, middleware service, a domain name service, a security service, a CDN, a big data and artificial intelligence platform, and the like.

The server 202 responds to the data updating operation of the user and determines the feature identification and the evaluation index of the element indicated by the data updating operation; adding the evaluation index of the element indicated by the data updating operation to a data storage unit matched with the characteristic identifier in the database; if the reading instruction is detected, identifying the reading instruction to determine a target feature identifier; determining a target element set corresponding to the target feature identifier, and a target data storage unit for storing evaluation indexes of all elements in the target element set; according to a reading strategy determined based on the size of the evaluation index, reading the target evaluation index in the target data storage unit, and displaying the read target evaluation index and the corresponding target element through the terminal device, for example, the server sends the read evaluation index and the read target element to the terminal device 201-1 according to a reading instruction of the user 201, and displays the read evaluation index and the read target element through the terminal device 201-1.

Of course, the method provided in the embodiment of the present application is not limited to be used in the application scenario shown in fig. 1, and may also be used in other possible application scenarios, and the embodiment of the present application is not limited. The functions that can be implemented by each device in the application scenario shown in fig. 1 will be described in the following method embodiments, and will not be described in detail herein.

To further illustrate the technical solutions provided by the embodiments of the present application, the following detailed description is made with reference to the accompanying drawings and the detailed description. Although the embodiments of the present application provide method steps as shown in the following embodiments or figures, more or fewer steps may be included in the method based on conventional or non-inventive efforts. In steps where no necessary causal relationship exists logically, the order of execution of the steps is not limited to that provided by the embodiments of the present application.

The following describes the technical solution provided in the embodiment of the present application with reference to the application scenario shown in fig. 1.

Referring to fig. 3, an embodiment of the present application provides a data processing method, including the following steps:

s301, responding to data updating operation of a user, and determining feature identification and evaluation indexes of elements indicated by the data updating operation; the evaluation index represents the sequence of the target elements in the element set, and the feature identifier of each element in one element set is the same as the feature identifier of the element set.

S302, adding the evaluation index of the element indicated by the updating operation to a data storage unit matched with the feature identifier in the database; the database comprises at least one data storage unit; each data storage unit stores therein evaluation indexes of respective elements in one element set.

And S303, if the reading instruction is detected, identifying the reading instruction to determine the target feature identifier.

S304, determining a target element set corresponding to the target feature identifier, and storing a target data storage unit of the evaluation index of each element in the target element set.

S305, reading the target evaluation index in the target data storage unit according to the reading strategy determined according to the size of the evaluation index.

In the embodiment of the application, when the user operation is responded, the feature identification and the evaluation index of the element indicated by the user operation are determined; the evaluation index can represent the sequence of the elements in the element set, and the feature identifier of each element in one element set is the same as the feature identifier of the element set, so that the sequence of the target element in the element set corresponding to the feature identifier can be determined through data updating operation of a user. Adding the evaluation indexes of the elements indicated by the updating operation into a data storage unit matched with the characteristic identifier in the database, so as to update the data storage unit, identifying a reading instruction to determine a target characteristic identifier when a reading command is detected, further determining a target element set corresponding to the target characteristic identifier, and storing the evaluation indexes of the elements in the target element set; and reading the evaluation index in the target data storage unit according to a reading strategy determined based on the size of the evaluation index. In this way, since the reading strategy is determined based on the evaluation index, even if a plurality of data updating operations exist simultaneously in a short time, the evaluation index can be read by taking the size of the evaluation index as the reading criterion, and the problem of data conflict in a concurrent environment is solved.

Referring to S301, in the embodiment of the present application, data processing is mainly implemented in a Zset structure of a Redis database. Still taking the above video APP as an example, each episode of a tv series is called an element, and the whole tv series composed of multiple episodes of tv series is called an element set. The characteristic identification of each element in each element set is the same as that of the element set, and the characteristic identifications of different element sets are different. The evaluation index is, for example, the episode number of the tv series, and the feature identifier of the element may be a user identifier and a tv series identifier. Thus, for example, zhang three + drama a corresponds to one element set, and lie four + drama B corresponds to another element set.

Specifically, the scenario of the data update operation of the user is that the user clicks the icon of episode 8 of the series a three times to indicate that the user wants to watch episode 8, and before that, the database stores episodes of series a three times clicked watched series a, such as episode 1 to episode 7. In practical applications, the user may not watch continuously, and in this case, the 1 st, 2 nd, 3 rd and 6 th sets are stored in the database.

In this step, from the perspective of the server, data update operations of different users can be responded to simultaneously. From the perspective of the user, the same user clicks set 7 and set 8 simultaneously in a short time, and the server also responds to the two data updating operations of the user.

Taking user zhang san as an example, since analyzing the operation of zhang san by the user can determine which set, which point data, and who the target element indicated by the operation is operated, the feature of the target element is identified as zhang san + drama a, and the target element is set 8.

Referring to S302, since the server needs to process requests of a large number of users, the database generally includes a plurality of data storage units, each data storage unit stores evaluation indexes of respective elements in an element set, and feature identifiers of the elements stored in each storage unit are the same, for example, a tv series a watched in three is in one data storage unit, a tv series B watched in four is in another data storage unit, and the respective data storage units form a database.

And after responding to the data updating operation of the user, adding the evaluation index of the element indicated by the data updating operation to the data storage unit matched with the characteristic identifier in the database. For example, if the feature identifier of the user is "three + drama a", the 8 th-set evaluation index is added to the data storage unit corresponding to the "three + drama a".

In an actual application process, the adding process of the elements can be specifically realized by the following modes:

sending an element adding command to the database, wherein the element adding command comprises an evaluation index of the element indicated by the data updating operation and a characteristic identifier of the element indicated by the data updating operation; determining a data storage unit matched with the characteristic identification of the element indicated by the data updating operation; and adding the evaluation index of the element indicated by the data updating operation to the data storage unit matched with the characteristic identifier.

Illustratively, the format of the element add command is zadd $ key $ score $ member, where zadd is the key and the function characterizing the command is element add; $ key is a feature identifier that characterizes the set of elements to which the element belongs, such as the set of episodes of which television show which user watches; score is an evaluation index, the evaluation index represents the sequence of each element in the element set, for example, the earlier the sequence in the set is, the smaller the evaluation index is, for example, a weight or a score is set for each set, the weight or the score is positively correlated with the sequence of the elements in the set, and the later the sequence is, the larger the evaluation index is, and in the actual application process, the evaluation index can be directly represented by the set number. The number is the unique identifier of the element, and in this example, can be represented by a set number, and can also be represented by other identifiers. The corresponding relation between the unique identifier of the element and the evaluation index of the element can be stored in the database in advance, so that $ number in the adding command is an optional item and can be set according to the actual situation in the actual application process. In addition, since the server needs to determine whether the addition is successful, the manner of sending the element addition command may be a synchronous manner.

In this way, the data storage unit to which the element indicated by the data updating operation belongs can be determined by the feature identifier of the element, and the evaluation index of the element generated by the corresponding user trigger is added to the data storage unit, so that the evaluation index of each element can be referred to in the subsequent data reading.

Referring to S303, if a read command is detected, the read command is recognized to determine the target signature. For example, the reading instruction may be generated specifically by, for example, generating the reading instruction without any operation on the current interface within a preset time length; alternatively, the user double-clicks on the currently displayed page, and a read instruction may be generated. In both of the two manners, it indicates that the user has a need, and at this time, the server may recognize the read instruction to determine the target feature identifier, so that it may be known which user operates on which data unit, for example, the determined target feature identifier is zhang three + westernist.

Referring to S304, since the evaluation indexes of the respective elements in one element set are stored in each data storage unit; each element set has a feature identifier, and the feature identifier of the element set is the same as the feature identifier of each element in the set. Therefore, after the target feature identifier is determined, a target element set corresponding to the target feature identifier may be determined according to a correspondence relationship between the feature identifier and the element set, and a target data storage unit storing the evaluation index of each element in the target element set may be determined according to a correspondence relationship between the target element set and the data storage unit.

Referring to S305, the target evaluation index in the target data storage unit is read according to a read policy determined based on the size of the evaluation index.

In the data reading process, a reading strategy can be determined based on the size of the evaluation index, and the target evaluation index in the target data storage unit is read according to the reading strategy. In a specific example, if the evaluation index is the episode number of the drama and the target evaluation index is the 9 th episode, the target evaluation index is read, and then the target element (for example, the content of the 9 th episode) corresponding to the target evaluation index is displayed to the user.

In an actual application process, the read strategy determined based on the size of the evaluation index is the maximum evaluation index read, and in this case, may be the maximum evaluation index in the read target data storage unit. In this way, regardless of how many evaluation indexes are in the corresponding target data storage unit, the latest element can be presented to the user as long as the largest evaluation index is read.

Alternatively, the reading process may be implemented as follows: sending an element reading command to a database, wherein the element reading command comprises a reading start character and a reading end character, and the reading start character and the reading end character are determined according to the maximum evaluation index in the target data storage unit; and reading the maximum evaluation index in the target data storage unit according to the reading start character and the reading end character.

For example, the format of the element read command may be zrange $ key $ start $ stop, where zrange is the key and the function characterizing the command is element read; $ key is a feature identifier, and represents the attribute of the element set to which the element to be read belongs; and $ start and $ stop are read identifiers. In this example, $ start takes evaluation index-1 as the read start symbol and $ stop takes evaluation index-1 as the read stop symbol, so that reading the largest evaluation index in the corresponding target data storage unit can be achieved by this command.

In this way, by sending the element reading command to the database, the function of reading the maximum evaluation index can be configured in the element reading command, and the reading of the maximum evaluation index is realized.

The above operation can realize the reading of the maximum evaluation index, but in order to save the memory, the evaluation indexes of the unnecessary elements can be deleted, and only the maximum evaluation index is reserved. The delete operation may occur after the add and before the read. Specifically, for each data storage unit, eliminating other evaluation indexes except the maximum evaluation index in the data storage unit; at this time, since only the maximum evaluation index remains in the corresponding data storage unit, the read strategy determined based on the size of the evaluation index is the evaluation index in the target data storage unit after the thinning operation is performed. Therefore, in the reading process, in the target data storage unit after the removing operation, the remaining evaluation indexes in the target data storage unit are read, and the read evaluation indexes are the maximum evaluation indexes. In addition, since the server does not need to determine whether the deletion is successful, the manner in which the element addition command is sent may be an asynchronous manner.

Therefore, only the maximum evaluation index is reserved through the removing operation, so that the memory is saved, and the problem of data concurrency conflict is solved.

Specifically, element deletion can be realized by: for each data storage unit, sending an element filtering command to the data storage unit by sending the element filtering command to the database; wherein the element filtering command comprises a filtering start character and a filtering end character, the filtering start character is determined according to the minimum evaluation index in the data storage unit, and the filtering end character is determined according to the second largest evaluation index in the data storage unit; and applying the filtering start character and the filtering stop character to remove other evaluation indexes except the maximum evaluation index in the data storage unit.

Illustratively, the format of the element filtering command is zremrangbyrank $ key $ start $ stop, where zremrangbyrank is the key and the function characterizing the command is element filtering; $ key is a feature identifier, which represents the attribute of the element set to which the element to be filtered belongs; $ start and $ stop are filter identifiers. In this example, $ start takes evaluation index 0 as the filter start symbol, and $ stop takes evaluation index-2 as the filter stop symbol, so that the effect of filtering the smallest evaluation index to the next largest element index can be achieved, leaving only the largest evaluation index.

In this way, by sending the element filtering command to the database, the function of deleting the evaluation index except the largest evaluation index can be configured in the element filtering command, and the filtering of each evaluation index is realized.

In addition, even if the deletion process fails or a read command is detected before deletion, the maximum evaluation index can be read by the data reading mode in the embodiment of the application, and the problem of data concurrent conflict can be solved.

In order to improve the display effect, after reading the target evaluation index in the target data storage unit, the method further comprises the following steps: and displaying the target elements corresponding to the target evaluation indexes. In this way, still taking the example of a tv series, after the target evaluation index (set 9) is determined, the target element (video picture of set 9) corresponding to set 9 may be displayed to the user. Therefore, the user can obtain the target elements and the display effect in time, and the user experience is better.

This is illustrated below with an example of a complete flow chart, with reference to fig. 8.

S801, responding to data updating operation of a user, and determining feature identification and evaluation index of elements indicated by the data updating operation.

The evaluation index represents the sequence of the target elements in the element set, and the feature identifier of each element in one element set is the same as the feature identifier of the element set.

S802, sending an element adding command to the database.

The element adding command comprises an evaluation index of the element indicated by the data updating operation and a characteristic identification of the element indicated by the data updating operation.

S803, a data storage unit matched with the feature identification of the element indicated by the data updating operation is determined.

And S804, adding the evaluation index of the element indicated by the data updating operation to the data storage unit matched with the characteristic identifier.

S805, for each data storage unit, sending an element filtering command to the data storage unit by sending the element filtering command to the database.

Wherein the element filter command includes a filter start and a filter stop, the filter start being determined according to a minimum rating index in the data storage unit, and the filter stop being determined according to a second largest rating index in the data storage unit.

And S806, applying the filtering start character and the filtering stop character, and eliminating other evaluation indexes except the maximum evaluation index in the data storage unit.

S807, if the reading instruction is detected, the reading instruction is identified to determine the target characteristic mark.

S808, determining a target element set corresponding to the target feature identifier, and storing a target data storage unit of the evaluation index of each element in the target element set.

And S809, sending an element reading command to the database.

The element reading command comprises a reading start character and a reading end character, and the reading start character and the reading end character are determined according to the maximum evaluation index in the target data storage unit;

and S810, reading the maximum evaluation index in the target data storage unit according to the reading start character and the reading end character.

It should be noted that fig. 8 is only an exemplary illustration of the case where the reading strategy determined based on the size of the evaluation index is the evaluation index with the maximum reading, and the case of other reading strategies may refer to the relevant expressions in the embodiment, which is not shown in fig. 8.

In order to make the technical solution of the present application easier to understand, a general structure of zset and a specific implementation process of the embodiments of the present application are described below with a specific example.

First, the general structure of Zset is as follows:

zset is an ordered set in which different elements are stored in order. Each element contains two attributes: score (evaluation index) which may be a score/weight for ranking, and memer (element identification) which is a value or id of an element for uniquely identifying an element. Referring to FIG. 4, there is shown a general structure of Zset containing 5 elements with score of 100 ~ 500 respectively. In this example, the evaluation indexes are 100, 200, 300, 400, and 500, respectively; the number is 1, 2, 3, 4 and 5 respectively.

In the example applied to a tv show, score and member may be both expressed by using the set number id for convenience of expression, that is, both the evaluation index and the element identification of the 8 th set are 8.

Secondly, the specific implementation process of the embodiment of the application is as follows:

step 1: in the k-v structure, the key structure is user id (zhang) + series id (also in a bead grid), represented by $ userId _ $ tvId in the command, and uniquely identifies the user in the memory. value uses zset, where score and memer both use set number id. When a user clicks a certain video set, the back-end server sends a set $ userId _ $ tvId $ episodeId command to the Redis server through the preposed logic check, and the command is used for updating the latest set id of the user into the value corresponding to the Redis key. When updating is needed, corresponding to a service scene, namely when a user watches a certain episode of a series, the following two steps are carried out:

(1) the server synchronously sends a command of zadd $ userId _ $ tvId 11 to the Redis (the user watched the first episode of the series) (the format of the zadd command is zadd $ key $ score $ memer, so the first 1 is score, the second 1 is memer, all using the value of the episode number id).

(2) If (1) the execution is successful, a command of zremrangibbyrank $ userId _ $ tvId 0-2 is asynchronously sent to Redis (the format of the zremrangibbyrank command is zremrangibbyrank $ key $ start $ stop), which means that elements from the first name to the second last name in the Zset of $ userId _ $ tvId are deleted; and if the execution of the step (1) fails, no operation is carried out, and an error prompt is directly returned to the upper layer.

In this example, assuming that Zset is empty before, after the above steps (1) and (2) are successfully performed, the Zset structure of $ userId _ $ tvId in Redis as shown in fig. 5:

step 2: elements from the first to the second last are deleted, since in this example there is only one element in the Zset, there is no change in the Zset structure after the delete command is executed.

Applying this example to analyze how concurrency conflicts are prevented, assuming the user clicks on set 2 and set 3 at approximately the same time, and both requests arrive at the server at approximately the same time, then the server performs step (1) at the same time, sending zadd $ userId _ $ tvId 22 and zadd $ userId _ $ tvId 33 to Redis, when zset is structured as in fig. 6:

and the two also execute step (2) almost simultaneously, and respectively send zremrangebylank $ userId _ $ tvId 0-2 commands to the Redis server, wherein the zset structure after the first command is executed is as shown in FIG. 7. The same second command executes without any change in the Zset structure after execution because there is only one element left in Zset.

These two operations guarantee that, even in a concurrent environment, the last element in Zset is the one with the largest set id at any time, i.e. our target element, and therefore, when reading (for APP page presentation), it is necessary to fetch the last element in Zset using the command zrange $ userId _ $ tvId-1-1 (the format of zrange command is zrange $ key $ start $ stop). Thus, assuming that step (2) of the above updating steps is not executed or fails to execute, there will be more than one element in Zset, but only the one with the largest set number id, i.e. the one arranged at the end, so that the last element can be correctly fetched by using the command zrange $ userId _ $ tvId-1-1, no matter how many elements are contained in Zset.

In addition, the element deletion operation has the advantage that even if Zset is allowed to exist for more than one element in a short time, unnecessary elements are deleted in order to save memory. The asynchronous rather than synchronous approach is used because whether the deletion was successful does not affect the read result (even if unsuccessful, the element still exists in Zset does not affect the result of the zrange command of the read operation), so the asynchrony is used to improve the performance of the write interface, avoiding the time consuming waiting for the result synchronously. At the same time, the command also has the capability of fault-tolerant correction, so that even if the last zremrangebylank deletion operation is unsuccessful, all elements left over before are deleted once next time, and thus the result is idempotent.

In summary, after using the Zset data structure and the above algorithm steps, under the concurrent environment of tens of thousands of stages of Qps on line, the problem of data collision does not occur any more in the cache, and meanwhile, the performance of the read-write interface is normal, the phenomenon of decline does not occur, and the memory consumption of the Redis server is also normal. On the premise of meeting the requirements of product functions and performances, the problem of concurrency conflict is solved.

Fig. 9 is a block diagram illustrating a data processing apparatus according to an exemplary embodiment, and referring to fig. 7, the apparatus includes a first determination unit 91, an adding unit 92, an instruction identifying unit 93, a second determination unit 94, and a reading unit 95.

A first determination unit 91 configured to perform a data update operation in response to a user, and determine a feature identification and an evaluation index of an element indicated by the data update operation; the evaluation index represents the sequence of elements in the element set, and the feature identifier of each element in one element set is the same as the feature identifier of the element set;

an adding unit 92 configured to perform addition of the evaluation index of the element indicated by the data update operation to the data storage unit in the database that matches the feature identification; the database comprises at least one data storage unit; each data storage unit stores evaluation indexes of each element in one element set;

an instruction recognition unit 93 configured to perform, upon detection of the read instruction, recognizing the read instruction to determine the target feature identifier;

a second determination unit 94 configured to perform determination of a target element set corresponding to the target feature identifier, and a target data storage unit that stores evaluation indexes of respective elements in the target element set;

a reading unit 95 configured to execute reading of the target evaluation index in the target data storage unit in accordance with a reading policy determined based on the size of the evaluation index.

In some exemplary embodiments, if the read policy determined based on the size of the evaluation index is the evaluation index with the largest read, the reading unit 95 is specifically configured to perform:

and reading the maximum evaluation index in the target data storage unit.

In some exemplary embodiments, the reading unit 95 is specifically configured to perform:

sending an element reading command to a database, wherein the element reading command comprises a reading start character and a reading end character, and the reading start character and the reading end character are determined according to the maximum evaluation index in the target data storage unit;

In some exemplary embodiments, the adding unit 92 is configured to perform:

In some exemplary embodiments, the apparatus further includes a filtering unit, and before reading the target evaluation index in the target data storage unit according to a reading policy determined based on a size of the evaluation index, the filtering unit is configured to perform:

the reading strategy determined based on the size of the evaluation index is used for reading the evaluation index in the target data storage unit subjected to the rejection operation;

reading the target evaluation index in the target data storage unit according to a reading strategy determined based on the size of the evaluation index, comprising:

and reading the remaining evaluation indexes in the target data storage unit after the removing operation.

In some exemplary embodiments, the filtration unit is specifically configured to perform:

for each data storage unit, sending an element filtering command to the data storage unit by sending the element filtering command to the database; wherein the element filtering command includes a filtering start character and a filtering end character, the filtering start character is determined according to the minimum evaluation index in the data storage unit, and the filtering end character is determined according to the second largest evaluation index in the data storage unit;

The data processing device and the data processing method provided by the embodiment of the application adopt the same inventive concept, can obtain the same beneficial effects, and are not described again.

Having described the data processing method and apparatus of the exemplary embodiments of the present application, an electronic device according to another exemplary embodiment of the present application is next described.

As will be appreciated by one skilled in the art, aspects of the present application may be embodied as a system, method or program product. Accordingly, various aspects of the present application may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.

In some possible implementations, an electronic device according to the present application may include at least one processor, and at least one memory. Wherein the memory stores program code which, when executed by the processor, causes the processor to perform the steps of the data processing method according to various exemplary embodiments of the present application described above in the present specification. For example, the processor may perform steps as in a data processing method.

The electronic apparatus 130 according to this embodiment of the present application is described below with reference to fig. 10. The electronic device 130 shown in fig. 10 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 10, the electronic device 130 is represented in the form of a general electronic device. The components of the electronic device 130 may include, but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 that connects the various system components (including the memory 132 and the processor 131).

Bus 133 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

The memory 132 may include readable media in the form of volatile memory, such as Random Access Memory (RAM)1321 and/or cache memory 1322, and may further include Read Only Memory (ROM) 1323.

Memory 132 may also include a program/utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.

The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), with one or more devices that enable a user to interact with the electronic device 130, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 130 to communicate with one or more other electronic devices. Such communication may occur via input/output (I/O) interfaces 135. Also, the electronic device 130 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 136. As shown, network adapter 136 communicates with other modules for electronic device 130 over bus 133. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

In an exemplary embodiment, a computer-readable storage medium comprising instructions, such as the memory 132 comprising instructions, executable by the processor 131 to perform the above-described method is also provided. Alternatively, the computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

In an exemplary embodiment, there is also provided a computer program product comprising computer programs/instructions which, when executed by the processor 131, implement any of the data processing methods as provided herein.

In exemplary embodiments, aspects of a data processing method provided herein may also be implemented in the form of a program product including program code for causing a computer device to perform the steps of a data processing method according to various exemplary embodiments of the present application described above in this specification when the program product is run on the computer device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The program product for image scaling of embodiments of the present application may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the consumer electronic device, partly on the consumer electronic device, as a stand-alone software package, partly on the consumer electronic device and partly on a remote electronic device, or entirely on the remote electronic device or server. In the case of remote electronic devices, the remote electronic devices may be connected to the consumer electronic device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external electronic device (e.g., through the internet using an internet service provider).

It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, the features and functions of two or more units described above may be embodied in one unit, according to embodiments of the application. Conversely, the features and functions of one unit described above may be further divided into embodiments by a plurality of units.

Further, while the operations of the methods of the present application are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable image scaling apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable image scaling apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable image scaling apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable image scaling device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer implemented process such that the instructions which execute on the computer or other programmable device provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A data processing method, comprising:

2. The method of claim 1, wherein if the read policy determined based on the size of the evaluation index is a read-largest evaluation index, the reading of the evaluation index in the target data storage unit comprises:

and reading the maximum evaluation index in the target data storage unit.

3. The method of claim 2, wherein reading the maximum evaluation index in the target data storage unit comprises:

4. The method according to claim 1, wherein the adding the evaluation index of the element indicated by the data updating operation to the data storage unit matched with the feature identifier comprises:

5. The method of claim 1, wherein before reading the target evaluation indicator in the target data storage unit according to the reading policy determined based on the size of the evaluation indicator, further comprising:

6. The method according to claim 5, wherein the eliminating, for each data storage unit, the other evaluation indexes except the maximum evaluation index in the data storage unit comprises:

7. The method according to any one of claims 1-6, wherein after reading the target evaluation index in the target data storage unit, further comprising:

8. A data processing apparatus, comprising:

9. An electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the data processing method of any one of claims 1 to 7.

10. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the data processing method of any of claims 1 to 7.