CN113329078A

CN113329078A - Data storage method and device

Info

Publication number: CN113329078A
Application number: CN202110586766.2A
Authority: CN
Inventors: 林明明
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2021-05-27
Filing date: 2021-05-27
Publication date: 2021-08-31
Anticipated expiration: 2041-05-27
Also published as: CN113329078B

Abstract

The present disclosure provides a data storage method and apparatus. The data storage method comprises the following steps: receiving data request information to be inserted, wherein the data request information comprises data identification information of the data request information to be inserted, and the data identification information is used as a first attribute corresponding to the data request information to be inserted; determining a second attribute according to the data request information to be inserted, wherein the second attribute comprises at least two data attribute information of the data request information to be inserted, and the at least two data attribute information is used for determining a data storage sequence; and inserting the data request information to be inserted into the corresponding position in the queue according to the first attribute and the second attribute.

Description

Data storage method and device

Technical Field

The present disclosure relates to the field of internet technologies, and in particular, to a method and an apparatus for data storage.

Background

With the rapid development of internet and computer technologies, in many common application scenarios (e.g., video special effect/transcoding processing in the video field, and transaction reconciliation service in the payment financial field), data storage and reading are time-consuming operations, which cannot return a data processing result instantly, and require a certain time to process, so that a server needs to queue data requests (e.g., data requests from a client of a user) in a certain order, and needs to design a fair queuing algorithm and implement a queue with good queuing/dequeuing performance. Meanwhile, in order to realize the robustness of the system and more reliable fairness, the queue also needs to support the re-queuing of elements, namely, the elements can be put in the correct position instead of the tail of the queue when re-enqueuing after dequeuing.

In the related art, the queues used for storing data are generally ordinary queues using the principle of first-in first-out, for example, message queue services provided by open source components Kafka, RabbitMQ, rocktmq, etc., or list data structures of memory database Redis used, or custom queues, and these queues have the following disadvantages: each enqueue element can only be arranged at the tail of the queue and cannot be inserted into the correct position in the queue, which can cause that the elements with higher priority cannot be consumed by dequeuing in time; or the data can be correctly inserted into the correct position in the queue, but the enqueue/dequeue performance is poor, for example, the priority queue RPriorityQueue in the open source component Redisson needs to lock the whole queue for each operation, and the performance is reduced due to the occurrence of a large amount of lock waiting in a high-concurrency environment, so that an application scenario with high performance requirement cannot be met.

Therefore, none of the existing solutions can overcome the above disadvantages.

Disclosure of Invention

The present disclosure provides a method and an apparatus for data storage, so as to solve at least the problems of the related art that the enqueue/dequeue performance for a queue is poor, the queue cannot be inserted into a correct position when re-enqueuing, and also not solve any of the problems. The technical scheme of the disclosure is as follows:

according to a first aspect of the embodiments of the present disclosure, there is provided a data storage method, including: receiving data request information to be inserted, wherein the data request information comprises data identification information of the data request information to be inserted, and the data identification information is used as a first attribute corresponding to the data request information to be inserted; determining a second attribute according to the data request information to be inserted, wherein the second attribute comprises at least two data attribute information of the data request information to be inserted, and the at least two data attribute information is used for determining a data storage sequence; and inserting the data request information to be inserted into the corresponding position in the queue according to the first attribute and the second attribute.

Optionally, a sum of digits of values corresponding to the at least two kinds of data attribute information may be less than or equal to a maximum precision digit of the value of the second attribute.

Optionally, the step of inserting the data request information to be inserted into the corresponding position in the queue may include: comparing the value of the second attribute of the data request information to be inserted with the value of the second attribute of the data request information stored in the queue; and inserting the data request information to be inserted into the corresponding position in the queue according to the comparison result, wherein the value of the second attribute of the data request information to be inserted is larger than the value of the second attribute of the previous stored data request information and smaller than the value of the second attribute of the next stored data request information in the sequence from the head of the queue to the tail of the queue.

Optionally, when the value of the second attribute corresponding to the data request information to be inserted is the same as the value of the second attribute corresponding to at least another data request information in the queue, the data request information to be inserted may be inserted into the corresponding position in the queue in the order from small to large of the value of the first attribute.

Alternatively, the queue may be a zset data structure, with the first attribute corresponding to memer in the zset data structure and the second attribute corresponding to score in the zset data structure.

Alternatively, the at least two kinds of data attribute information may be sequentially stored as the number of bits of the value of the second attribute in accordance with the priority.

Optionally, the at least two data attribute information may be three data attribute information, which are request priority information, data status information, and request time information, respectively.

Alternatively, the request priority information may be higher in priority than the data state information, and the data state information may be higher in priority than the request time information.

Alternatively, the data request information may be request information of a user for a video, and the at least two types of data attribute information may be three types of data attribute information, which are user attribute priority information, video status information, and request time information, respectively.

Optionally, the step of determining the second attribute may comprise: and determining values corresponding to the request priority information, the data state information and the request time information as values of a second attribute, wherein the values corresponding to the request priority information, the data state information and the request time information are sequentially stored as the digits of the values of the second attribute from front to back according to the digits, the request priority information has higher priority than the data state information, and the data state information has higher priority than the request time information.

Optionally, the step of inserting the data request information to be inserted into the corresponding position in the queue may include: comparing the value of the second attribute of the data request information to be inserted with the value of the second attribute of the data request information stored in the queue; and inserting the data request information to be inserted into the corresponding position in the queue according to the comparison result, wherein the comparison step is executed in sequence according to the sequence of the corresponding values of the request priority information, the data state information and the request time information.

Alternatively, the data storage method may be applied to a server, the server is connected to a client in communication, and the step of receiving the data request information may include: data request information is received from a client.

Optionally, the data storage method may further include: receiving data reading request information from a client; and sending the data stored in the queue head in the queue to the server so as to realize the consumption of the data stored in the queue head.

According to a second aspect of embodiments of the present disclosure, there is provided a data storage device comprising: a receiving module configured to: receiving data request information to be inserted, wherein the data request information comprises data identification information of the data request information to be inserted, and the data identification information is used as a first attribute corresponding to the data request information to be inserted; a determination module configured to: determining a second attribute according to the data request information, wherein the second attribute comprises at least two data attribute information of the data request information to be inserted, and the at least two data attribute information is used for determining a data storage sequence; a storage module configured to: and inserting the data request information to be inserted into the corresponding position in the queue according to the first attribute and the second attribute.

Optionally, the operation of the storage module inserting the data request information to be inserted into the corresponding position in the queue may include: comparing the value of the second attribute of the data request information to be inserted with the value of the second attribute of the data request information stored in the queue; and inserting the data request information to be inserted into the corresponding position in the queue according to the comparison result, wherein the value of the second attribute of the data request information to be inserted is larger than the value of the second attribute of the previous stored data request information and smaller than the value of the second attribute of the next stored data request information in the sequence from the head of the queue to the tail of the queue.

Optionally, the operation of the determination module determining the second attribute may comprise: and determining values corresponding to the request priority information, the data state information and the request time information as values of a second attribute, wherein the values corresponding to the request priority information, the data state information and the request time information are sequentially stored as the digits of the values of the second attribute from front to back according to the digits, the request priority information has higher priority than the data state information, and the data state information has higher priority than the request time information.

Optionally, the operation of the storage module inserting the data request information to be inserted into the corresponding position in the queue may include: comparing the value of the second attribute of the data request information to be inserted with the value of the second attribute of the data request information stored in the queue; and inserting the data request information to be inserted into the corresponding position in the queue according to the comparison result, wherein the comparison operation is executed in sequence according to the sequence of the corresponding values of the request priority information, the data state information and the request time information.

Alternatively, the storage device may be installed in a server, the server is connected to a client in a communication manner, and the operation of receiving the data request information to be inserted by the receiving module may include: data request information is received from a client.

Optionally, the receiving module may be further configured to: receiving data reading request information from a client; the storage device further includes a transmission module configured to: and sending the data stored in the queue head in the queue to the server so as to realize the consumption of the data stored in the queue head.

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the method as described above.

According to a fourth aspect of embodiments of the present disclosure, there is provided a storage medium having instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the method as described above.

According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product, instructions of which are executed by at least one processor in an electronic device to perform the method as described above.

The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects: by inserting the data request information into the queue by using the data attribute information and the data identification information, the data can be efficiently stored and read, the data which fails to be read can be reinserted into the correct position in the queue for storage, and the storage sequence of the data can be sequenced by using various types of data attributes, so that the user experience is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.

FIG. 1 is a schematic diagram of an exemplary system architecture to which exemplary embodiments of the present disclosure may be applied;

FIG. 2 is a flowchart illustrating a data storage method according to an exemplary embodiment of the present disclosure;

FIG. 3 is a schematic diagram illustrating an example of a data storage method according to an exemplary embodiment of the present disclosure;

FIG. 4 is a block diagram illustrating a data storage device according to an exemplary embodiment of the present disclosure;

fig. 5 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

Detailed Description

In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The embodiments described in the following examples do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

In this case, the expression "at least one of the items" in the present disclosure means a case where three types of parallel expressions "any one of the items", "a combination of any plural ones of the items", and "the entirety of the items" are included. For example, "include at least one of a and B" includes the following three cases in parallel: (1) comprises A; (2) comprises B; (3) including a and B. For another example, "at least one of the first step and the second step is performed", which means that the following three cases are juxtaposed: (1) executing the step one; (2) executing the step two; (3) and executing the step one and the step two.

The embodiment of the disclosure provides a data storage method and a data storage device, and the data storage method can be applied to an application environment as shown in fig. 1.

As mentioned in the background of the present disclosure, in the related art, in many application scenarios, a server needs to queue data requests (e.g., data requests from a client of a user) in a certain order, and a fair queuing algorithm and a queue implementation with good enqueue/dequeue performance need to be designed. Meanwhile, in order to realize the robustness of the system and more reliable fairness, the queue also needs to support the re-queuing of elements, namely, the elements can be put in the correct position instead of the tail of the queue when re-enqueuing after dequeuing.

The fairness mentioned above is mainly embodied in that it is first served, i.e. the request (e.g. a user's request for video) initiated first within a certain time frame (e.g. within one or ten seconds) should be processed first, time being the first element. However, there are other factors besides time, for example, the processing progress of the user video, in which case, the video which is estimated to be less time-consuming or has undergone the preprocessing should be processed more preferentially, because the sum of waiting time of the subsequent videos is reduced, and the overall processing efficiency is higher; meanwhile, for a system supporting a value added service, the priority of a predetermined category user (e.g., a member user, a payment user, etc.) should be raised, and a request of the predetermined category user should be processed at a more advanced position. Therefore, the queuing algorithm is influenced by a plurality of factors (also referred to as a plurality of dimensions), and needs to be considered together.

Based on this, according to an exemplary embodiment of the present disclosure, the present disclosure provides a queue-based data storage method and apparatus. Because the server side adopts a distributed deployment mode (for example, multi-place multi-machine room multi-instance deployment) in an actual application scene, the data request queue provided by the present disclosure satisfies the distributed characteristic, that is, the queue provided by the present disclosure is a distributed queue which is independent of service, has a consistent state with data, and conforms to idempotent. By utilizing the distributed queue disclosed by the invention, the storage and reading of data can be efficiently realized, the data which fails to be read can be reinserted into the correct position in the queue, and the storage sequence of the data can be sequenced by utilizing various types of data attributes. Further, optionally, in terms of processing of video request information, by using the distributed queue of the present disclosure, processing of video request information can be efficiently and orderly implemented, so that a client can obtain processed result information in a short time.

Hereinafter, a data storage method and apparatus according to an exemplary embodiment of the present disclosure will be described in detail with reference to fig. 1 to 5.

Fig. 1 is a schematic diagram of an exemplary system architecture 100 in which exemplary embodiments of the present disclosure may be applied.

The application environment (e.g., system architecture 100 of fig. 1) of the exemplary embodiments of the present disclosure may include one or more clients and servers, with the clients and servers being communicatively connected in some manner to enable data interaction with the servers.

The client may be installed on a terminal device (e.g.,

terminal devices

101, 102, 103 in fig. 1). As an example, as shown in fig. 1, the system architecture 100 may include

terminal devices

101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few. A user may use the

terminal devices

101, 102, 103 to interact with the server 105 over the network 104 to receive or send messages (e.g., video data upload requests, video data acquisition requests), and the like. Various communication client applications, such as a video rendering application, a video recording application, a video compression application, a video and audio editing application, an instant messaging tool, a mailbox client, social platform software, and the like, may be installed on the

terminal devices

101, 102, 103. The

terminal apparatuses

101, 102, and 103 may be hardware or software. When the

terminal devices

101, 102, 103 are hardware, they may be various electronic devices having a display screen and capable of rendering and editing videos, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, Personal Digital Assistants (PDAs), Augmented Reality (AR)/Virtual Reality (VR) devices, wearable devices, and any other electronic products capable of human-computer interaction with a user through one or more of a keyboard, a touch pad, a touch screen, a remote controller, a voice interaction device, a handwriting device, and the like. When the

terminal device

101, 102, 103 is software, it may be installed in the electronic devices listed above, it may be implemented as a plurality of software or software modules (for example, to provide distributed services), or it may be implemented as a single software or software module. And is not particularly limited herein.

The

terminal apparatuses

101, 102, 103 may be mounted with an image pickup device (e.g., a camera) to pick up video data, and further, the

terminal apparatuses

101, 102, 103 may also be mounted with a component (e.g., a speaker) for converting an electric signal into sound to play sound, and may also be mounted with a device (e.g., a microphone) for converting an analog audio signal into a digital audio signal to pick up sound.

The

terminal devices

101, 102, 103 may perform acquisition of video data using an image acquisition apparatus mounted thereon and acquisition of audio data using an audio acquisition apparatus mounted thereon, and the

terminal devices

101, 102, 103 may encode, store and transmit the acquired video data and audio data and may decode and play encoded video and audio received from another terminal device or from the server 105.

The server 105 may be a server that provides various services, such as a background server that provides support for a video rendering-type application, a video recording-type application, a video compression-type application, a video editing-type application, and the like installed on the

terminal devices

101, 102, 103, or a storage server that stores encoded video and audio uploaded by the

terminal devices

101, 102, 103 and can transmit the stored encoded video and audio to the

terminal devices

101, 102, 103 in response to a request of the

terminal devices

101, 102, 103.

The server may be hardware or software. When the server is hardware, it may be implemented as a distributed server cluster formed by multiple servers, or may be implemented as a single server. When the server is software, it may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services), or as a single piece of software or software module. In addition, the server can also be a cloud computing service center. And is not particularly limited herein.

It should be noted that the data storage method provided by the embodiment of the present application is generally executed by the server 105, and accordingly, the data storage device is generally disposed in the server 105. In addition, optionally, the data storage method provided in the embodiment of the present application may also be executed by the

terminal devices

101, 102, and 103, and accordingly, the data storage device may also be disposed in the

terminal devices

101, 102, and 103.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation, and the disclosure is not limited thereto.

Fig. 2 is a flowchart illustrating a data storage method 200 according to an exemplary embodiment of the present disclosure.

The data storage method 200 according to an exemplary embodiment of the present disclosure may be applied to a server as well as a client. Alternatively, when the data storage method 200 is applicable to a server, the operation of reading the stored data may be performed at a client.

As shown in fig. 2, according to an exemplary embodiment of the present disclosure, in step S201, data request information to be inserted is received, wherein the data request information includes data identification information of the data request information to be inserted as a first attribute.

Alternatively, the value corresponding to the data identification information may be a decimal value. It should be understood that the value corresponding to the data identification information may also be other forms of numerical values, such as octal and hexadecimal values, etc., according to the actual use requirement.

Alternatively, the data request information may be request information of a user for a video, and the data identification information corresponding to the request information may be a video id (identification code of the video), where the video id may be expressed as a decimal number.

Alternatively, when the data storage method 200 is applied to a server and the server is in communication connection with a client, the step of receiving data request information may include: the server receives data request information from the client.

According to an exemplary embodiment of the present disclosure, in step S202, a second attribute is determined according to the data request information to be inserted, wherein the second attribute includes at least two types of data attribute information of the data request information to be inserted for determining a data storage order.

According to an exemplary embodiment of the present disclosure, a sum of digits of values corresponding to at least two kinds of data attribute information may be less than or equal to a maximum precision digit of the value of the second attribute. Alternatively, the maximum precision digits may be 15 bits. Alternatively, at least two kinds of data attribute information may be sequentially stored as the number of bits of the value of the second attribute in accordance with the priority.

Alternatively, the at least two kinds of data attribute information may be three kinds of data attribute information, which are request priority information, data status information, and request time information, respectively. Alternatively, the priority order of the request priority information, the data status information, and the request time information may be from high to low.

According to an exemplary embodiment of the present disclosure, in the case where the at least two kinds of data attribute information are request priority information, data status information, and request time information, the determining of the second attribute may include: and determining values corresponding to the request priority information, the data state information and the request time information as values of a second attribute, wherein the values corresponding to the request priority information, the data state information and the request time information are sequentially stored as the digits of the values of the second attribute from front to back according to the digits, the request priority information has higher priority than the data state information, and the data state information has higher priority than the request time information.

Alternatively, in the case where the data request information is request information of a user for a video, the at least two types of data attribute information may be three types of data attribute information, which are user attribute priority information, video status information, and request time information, respectively. Alternatively, the priority order of the user attribute priority information, the video status information, and the request time information may be high to low.

According to an exemplary embodiment of the present disclosure, in step S203, data request information to be inserted is inserted into a corresponding position in the queue according to the first attribute and the second attribute.

Alternatively, the values corresponding to the at least two kinds of data attribute information may be decimal values. It should be understood that the values corresponding to the at least two data attribute information may also be other forms of values, such as octal and hexadecimal values, according to the actual usage requirement.

According to an exemplary embodiment of the present disclosure, the step of inserting the data request information to be inserted into the corresponding position in the queue may include: and inserting the data request information to be inserted into the corresponding positions in the queue according to the sequence from the small to the large of the value of the second attribute from the head of the queue to the tail of the queue.

According to an exemplary embodiment of the present disclosure, optionally, the step of inserting the data request information to be inserted into the corresponding position in the queue may include: comparing the value of the second attribute of the data request information to be inserted with the value of the second attribute of the data request information stored in the queue; and inserting the data request information to be inserted into the corresponding position in the queue according to the comparison result, wherein the value of the second attribute of the data request information to be inserted is larger than the value of the second attribute of the previous stored data request information and smaller than the value of the second attribute of the next stored data request information in the sequence from the head of the queue to the tail of the queue.

Further, in accordance with an exemplary embodiment of the present disclosure, in the case that the at least two kinds of data attribute information are request priority information, data status information, and request time information, the inserting of the data request information to be inserted into the corresponding position in the queue may include: comparing the value of the second attribute of the data request information to be inserted with the value of the second attribute of the data request information stored in the queue; and inserting the data request information to be inserted into the corresponding position in the queue according to the comparison result, wherein the comparison step is executed in sequence according to the sequence of the corresponding values of the request priority information, the data state information and the request time information.

Further, in accordance with an exemplary embodiment of the present disclosure, in the case where the at least two types of data attribute information are request priority information, data status information, and request time information, when comparing values of the second attribute of the two elements in the queue, a value of a number of bits corresponding to the request priority information among the values of the second attribute of the two elements may be compared first. Specifically, if the values of the bits corresponding to the request priority information are the same, the values of the bits corresponding to the data state information and the request time information are sequentially compared; if the values of the number of bits corresponding to the request priority information are different, the comparison result of the sizes of the values of the second attributes of the two elements can be directly determined without comparing the values of the data state information and the request time information. The above exemplary embodiments according to the present disclosure are advantageous in that: the information with high priority (here, the request priority information has higher priority than the data state information, and the data state information has higher priority than the request time information) is used for comparison first, so that the comparison result can be obtained quickly, and the efficiency of processing the related data is further improved.

According to an exemplary embodiment of the present disclosure, when a value of a second attribute corresponding to data request information to be inserted is the same as a value of a second attribute corresponding to at least another data request information in the queue, the data request information to be inserted may be inserted into corresponding positions in the queue in an order from a smaller value to a larger value of the first attribute.

Optionally, in a case that the data storage method 200 is applied to a server and the server is in communication connection with a client, when a read operation needs to be performed on stored data, the data storage method 200 may further include: receiving data reading request information from a client; and sending the data stored in the queue head in the queue to the server so as to realize the consumption of the data stored in the queue head.

The data storage and read-related operations are specifically illustrated below with reference to fig. 3, but the aspects of the present disclosure are not limited by this example. Fig. 3 is a schematic diagram illustrating an example of a data storage method according to an exemplary embodiment of the present disclosure.

As shown in fig. 3, the data stored in the queue shown in fig. 3 is request information of a user for a video and the queue is implemented particularly with a zset data structure. Specifically, two attributes involved in the queue are respectively represented by means and score, where a value corresponding to means represents an id (e.g., video id in fig. 3) uniquely identifying the request information, and a value corresponding to score represents at least two kinds of data attribute information (for determining the position of the request information in the queue) of the request information of the user for the video. Optionally, the score may be stored in a persistent storage system (e.g., without limitation, mysql/rocksdb) for use when re-enqueueing after dequeueing). The data storage process is described in detail below:

referring to fig. 3, head represents the head of the queue, tail represents the tail of the queue, and member and score represent two attributes of elements in the queue. Referring to diagram (a) in fig. 3, the value of member of each element in the queue represents video id (6, 3, 9, 5, 8, respectively, from left to right), and the value of score represents the weight for sorting, the value of weight increasing sequentially from left to right, the smaller the weight closer to the head of the queue and the higher the priority, the earlier it is dequeued for consumption (i.e., data processing).

When a new element (i.e., request information of a user for a video to be inserted) is enqueued with reference to diagram (b) in fig. 3, assuming that the video id of the request information is 12 and the calculated score is 100301 (the manner of calculation is described in detail later), the element is inserted into the 4 th position in the queue by a predetermined command (e.g., zadd $ queue _ name 10030112), and the elements after the original 4 th position are all moved backward by one position, and the new queue is as shown in diagram (b).

Here, after the element is inserted into the queue, the element can be dequeued before the two elements whose video id is 5 and 8 when dequeued, i.e., the video request will be processed preferentially because the value of score of the element is smaller than the latter two, with higher priority.

The following describes the operation procedure of inserting a queue. Referring to diagrams (c), (d), (e), and (f) of fig. 3, when dequeuing, the element at the head of the queue (i.e., the element with video id 6) (hereinafter referred to as the first element) is dequeued first (as shown in diagram (c) of fig. 3), but the consumption of the dequeued first element fails (i.e., is not processed completely) (e.g., various factors that may cause failure, such as data processing error, network packet loss, etc., occur), at this time, the first element should be reinserted into the queue. Specifically, the score corresponding to the video id is looked up in the above persistent storage system, and then the first element is reinserted into the correct position in the queue according to the size of the score by a predetermined command (e.g., zadd $ queue _ name 1001006) (see diagram (d)) instead of the position of the tail of the queue.

Further, as an example, if the second element corresponding to video id 3 has been dequeued and consumed before the first element is inserted into the queue, the position of the first element after being inserted into the queue at this time is as shown in diagram (e). Here, the elements in the queue are still sorted from small to large according to the score value, and the problem of the reinsertion position error of the first element due to the dequeue of the second element does not occur.

Further, as an example, if the second element corresponding to video id 3 has been dequeued and consumed before the first element is inserted into the queue and the third element corresponding to video id 20 has been inserted into the queue, the position after the first element is inserted into the queue at this time is as shown in diagram (f), and since the value of the score of the first element is greater than the value of the score of the third element and less than the value of the score of the fourth element corresponding to video id 9, the first element is inserted into the position between the third element and the fourth element. Here, the order of the elements in the queue is still sorted from small to large according to the score value, and the problem of a reinsertion position error of the first element due to dequeue of the second element and enqueue of the third element does not occur.

In the above manner, after the queue is enqueued/dequeued for multiple times by multiple elements, the sequence of the queue can still be maintained, and the enqueue/dequeue performance is high, for example, the enqueue/dequeue completion can be completed within a microsecond time, and almost no delay is perceived, which can improve the efficiency of the whole video request processing process in terms of video request processing (described in detail later).

According to an exemplary embodiment of the present disclosure, the manner in which score is determined (i.e., the value of score is calculated) is specifically illustrated. In an example, the data attribute information of the request information of the user for the video includes three kinds, i.e., user priority, video status, request time. The score value is a floating point type number with a precision of 15 decimal digits beyond which accurate data storage and calculation cannot be performed, which affects the result of the sorting, and therefore, in this example, 15 decimal digits are used to represent the three information.

Specifically, the 15-bit number in score is set as follows: 1) the first two digits correspond to the value of the user priority, for example, the first class of users is 90, the second class of users is 80, the third class of users is 70, and the like, the smaller the value is, the higher the priority is, the earlier the dequeue is; 2) bit 3 corresponds to the value of the video state, for example, 9 indicates the initialized state, 8 indicates the state that the preprocessing has been performed, 7 indicates the state that the transcoding is completed and the subsequent logic such as rendering can be performed, etc.; 3) the 12-bit value of the 4 th to 15 th bits corresponds to the value of the request time and is expressed by a millisecond-level time stamp, and the millisecond-level time stamp has the advantage that the accuracy of the request time can be improved, so that the accuracy of data storage and processing corresponding to the information is improved.

Here, the millisecond-level time stamp has a relatively large span range, and therefore needs to be stored in a large number of bits. The actual length of the millisecond timestamp is 13 bits, only 12 bits can be used for storing the millisecond timestamp, and the 1 st bit in the 13 bits is directly removed. Here, the reason for this processing is that since all the 1 st bits of the millisecond-level time stamp are 1 in the next several decades, the relevant system can operate correctly even if it is applied at least in several decades without explicit storage.

As an example, the value of score is specifically illustrated, and assuming that the request information of the video from the user is an initialized video request of a general user, and the request time is 2021-3-2012: 00:00:123, the value of score is as follows: 909616212800123. specifically, the first two digits 90 represent the average user, the 3 rd digit 9 represents the initialization of the video, and the second 616212800123 with the default first digit 1 is 1616212800123 to represent the millisecond-level time stamp corresponding to 2021-3-2012: 00:00: 123. By using the 15-bit number in the above manner, the upper limit of the accuracy of score is utilized to the maximum extent, and the sorting of multidimensional data is realized.

It should be understood that the above examples are by way of illustration only and not by way of limitation. For example, in practical applications, the score value may also represent more (e.g., four, five) or less (e.g., two) data attribute information. For example, with regard to the use of precision digits for score, the data attribute information may not be stored using all of the digits.

Through the mode, the data request information is stored by utilizing the ordered queue, the data can be efficiently stored and read, the data which fails to be read can be reinserted into the correct position in the queue, the storage sequence of the data can be sequenced by utilizing various types of data attributes, and therefore the user experience is improved.

Fig. 4 is a block diagram illustrating a data storage device 400 according to an exemplary embodiment of the present disclosure.

As shown in fig. 4, the data storage device 400 includes a receiving module 401, a determining module 402, and a storing module 403. Specifically, the receiving module 401 is configured to receive data request information to be inserted, where the data request information includes data identification information of the data request information to be inserted as the first attribute. The determining module 402 is configured to determine a second attribute according to the data request information, wherein the second attribute includes at least two kinds of data attribute information of the data request information to be inserted for determining a data storage order. The storage module 406 is configured to insert data request information to be inserted into corresponding locations in the queue according to the first attribute and the second attribute.

According to an exemplary embodiment of the present disclosure, a sum of digits of values corresponding to at least two kinds of data attribute information may be less than or equal to a maximum precision digit of the value of the second attribute. Alternatively, the maximum precision digits may be 15 bits. Alternatively, at least two kinds of data attribute information may be sequentially stored as the number of bits of the value of the second attribute in accordance with the priority. Optionally, the value corresponding to the data identification information and the values corresponding to the at least two types of data attribute information may be both decimal values.

In accordance with an exemplary embodiment of the present disclosure, in case that the at least two kinds of data attribute information are request priority information, data status information, and request time information, the determining module 402 may determine the second attribute including: and determining values corresponding to the request priority information, the data state information and the request time information as values of a second attribute, wherein the values corresponding to the request priority information, the data state information and the request time information are sequentially stored as the digits of the values of the second attribute from front to back according to the digits, the request priority information has higher priority than the data state information, and the data state information has higher priority than the request time information.

Alternatively, the data request information may be request information of a user for a video, and the at least two types of data attribute information may be three types of data attribute information, which are user attribute priority information, video status information, and request time information, respectively. Alternatively, the priority order of the user attribute priority information, the video status information, and the request time information may be high to low.

According to an exemplary embodiment of the present disclosure, the operation of the storage module 403 to insert the data request information to be inserted into the corresponding position in the queue may include: and inserting the data request information to be inserted into the corresponding positions in the queue according to the sequence from the small to the large of the value of the second attribute from the head of the queue to the tail of the queue.

According to an exemplary embodiment of the present disclosure, optionally, the operation of the storage module 403 to insert the data request information to be inserted into the corresponding position in the queue may include: comparing the value of the second attribute of the data request information to be inserted with the value of the second attribute of the data request information stored in the queue; and inserting the data request information to be inserted into the corresponding position in the queue according to the comparison result, wherein the value of the second attribute of the data request information to be inserted is larger than the value of the second attribute of the previous stored data request information and smaller than the value of the second attribute of the next stored data request information in the sequence from the head of the queue to the tail of the queue.

Further, in accordance with an exemplary embodiment of the present disclosure, in the case that the at least two types of data attribute information are request priority information, data status information, and request time information, the operation of the storage module 403 inserting the data request information to be inserted into the corresponding position in the queue may include: comparing the value of the second attribute of the data request information to be inserted with the value of the second attribute of the data request information stored in the queue; and inserting the data request information to be inserted into the corresponding position in the queue according to the comparison result, wherein the comparison operation is executed in sequence according to the sequence of the corresponding values of the request priority information, the data state information and the request time information.

Further, in accordance with an exemplary embodiment of the present disclosure, in the case that the at least two types of data attribute information are request priority information, data status information, and request time information, the storage module 403 may compare the value of the number of bits corresponding to the request priority information among the values of the second attribute of the two elements when comparing the values of the second attribute of the two elements in the queue. Specifically, if the values of the bits corresponding to the request priority information are the same, the values of the bits corresponding to the data state information and the request time information are sequentially compared; if the values of the number of bits corresponding to the request priority information are different, the comparison result of the sizes of the values of the second attributes of the two elements can be directly determined without comparing the values of the data state information and the request time information. The above exemplary embodiments according to the present disclosure are advantageous in that: the information with high priority (here, the request priority information has higher priority than the data state information, and the data state information has higher priority than the request time information) is used for comparison first, so that the comparison result can be obtained quickly, and the efficiency of processing the related data is further improved.

According to an exemplary embodiment of the present disclosure, the storage apparatus 400 may be installed in a server, the server is connected to a client in a communication manner, and the operation of the receiving module 401 receiving the data request information to be inserted may include: data request information is received from a client. Optionally, the receiving module 401 may be further configured to: data read request information is received from a client. The storage apparatus 400 further includes a sending module configured to send data stored at a head of a queue in the queue to the server to enable consumption of the data stored at the head of the queue.

According to an example embodiment of the present disclosure, a queue may be a zset data structure, with a first attribute corresponding to memer in the zset data structure and a second attribute corresponding to score in the zset data structure.

The processing specifically performed by each of the above-described modules and the related processing information have been described above with reference to fig. 2, and the description is not repeated here.

According to the data storage device, the data request information is stored by the ordered queue, the data can be efficiently stored and read, the data which fails to be read can be reinserted into the correct position in the queue, the storage sequence of the data can be sequenced by the data attributes of multiple types, and therefore user experience is improved.

Furthermore, in accordance with exemplary embodiments of the present disclosure, where a queue employs a zset data structure, the queue of the present disclosure relies only on the zset data structure in redis, except for the use to a persistent storage system (e.g., mysql), with only one write operation per enqueue/dequeue, without the risk of data inconsistency due to multiple write operations. In contrast, in the related art, depending on other structures (e.g., list of redis), a double write operation is performed each time a write operation is performed, and since the redis does not support a transaction, the operation in the related art has a great risk of data inconsistency, and cannot be applied to an on-line system (e.g., a video production related application system) with a high concurrent flow rate. The queue of the present disclosure may be well applied in high concurrent high flow online systems.

In addition, as an example, the data storage method and apparatus of the present disclosure have a prominent effect in practical applications, for example, in a function of, for example, a snapshot app treasure box, the video request is processed (enqueued/dequeued, etc.) by using the queue of the present disclosure, and under a high concurrent request load on line, the video request of the user is processed efficiently and orderly, the time consumption of the related processing interface is low, for example, the time consumption of the related processing interface corresponding to the video list acquired by the user after uploading the video is within 15 ms.

That is to say, the user can in time obtain the progress result of video relevant processing, has avoided the urgent situation of waiting for the nothing, has promoted user experience well.

In addition, the data storage method and device according to the present disclosure also exhibit excellent effects in terms of stability of a device (e.g., server) used for the data storage, smooth processing of, for example, video requests, and low memory usage.

In addition, when the device provided in the above embodiment implements the functions thereof, only the division of the above functional modules is illustrated, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to implement all or part of the functions described above.

Fig. 5 is a block diagram illustrating an electronic device 500 according to an exemplary embodiment of the present disclosure, the electronic device 500 may include a memory 510 and a processor 520, the memory 510 having stored therein a set of computer-executable instructions that, when executed by the processor, perform a data storage method according to an embodiment of the present disclosure.

By way of example, the electronic device may be a PC computer, tablet device, personal digital assistant, smartphone, or other device capable of executing the set of instructions described above. The electronic device need not be a single electronic device, but can be any collection of devices or circuits that can execute the above instructions (or sets of instructions) either individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that interfaces with local or remote (e.g., via wireless transmission).

In an electronic device, a processor may include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a programmable logic device, a special-purpose processor system, a microcontroller, or a microprocessor. By way of example, and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, and the like.

The processor may execute instructions or code stored in the memory, which may also store data. The instructions and data may also be transmitted or received over a network via a network interface device, which may employ any known transmission protocol.

The memory may be integral to the processor, e.g., RAM or flash memory disposed within an integrated circuit microprocessor or the like. Further, the memory may comprise a stand-alone device, such as an external disk drive, storage array, or any other storage device usable by a database system. The memory and the processor may be operatively coupled or may communicate with each other, such as through an I/O port, a network connection, etc., so that the processor can read files stored in the memory.

In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device may be connected to each other via a bus and/or a network.

According to an embodiment of the present disclosure, there may also be provided a computer-readable storage medium, for example, the memory 520 including instructions executable by the processor 510 of the electronic device 500 to perform the above-mentioned video quality improvement model training method or video quality improvement method. Alternatively, the computer-readable storage medium may be: read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD + R, CD-RW, CD + RW, DVD-ROM, DVD-R, DVD + R, DVD-RW, DVD + RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or compact disc memory, Hard Disk Drive (HDD), solid-state drive (SSD), card-type memory (such as a multimedia card, a Secure Digital (SD) card or a extreme digital (XD) card), magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a magnetic tape, a magneto-optical data storage device, a hard disk, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, A solid state disk, and any other device configured to store and provide a computer program and any associated data, data files, and data structures to a processor or computer in a non-transitory manner such that the processor or computer can execute the computer program. The computer program in the computer-readable storage medium described above can be run in an environment deployed in a computer apparatus, such as a client, a host, a proxy device, a server, and the like, and further, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by one or more processors or computers.

In exemplary embodiments of the present disclosure, there may also be provided a computer program product comprising computer programs/instructions which, when executed by a processor, implement the method for data storage according to exemplary embodiments of the present disclosure.

According to the method and the device for data storage, the ordered queue is used for storing the data request information, the data can be efficiently stored and read, the data which fails to be read can be reinserted into the correct position in the queue, the storage sequence of the data can be sequenced by using various types of data attributes, and therefore user experience is improved.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data storage method, characterized in that the data storage method comprises:

receiving data request information to be inserted, wherein the data request information comprises data identification information of the data request information to be inserted, and the data identification information is used as a first attribute corresponding to the data request information to be inserted;

determining a second attribute according to the data request information to be inserted, wherein the second attribute comprises at least two data attribute information of the data request information to be inserted, and the at least two data attribute information is used for determining a data storage sequence;

and inserting the data request information to be inserted into the corresponding position in the queue according to the first attribute and the second attribute.

2. The data storage method of claim 1, wherein a sum of digits of values corresponding to the at least two kinds of data attribute information is less than or equal to a maximum precision digit of the value of the second attribute.

3. The data storage method of claim 1, wherein the step of inserting the data request information to be inserted into the corresponding position in the queue comprises:

comparing the value of the second attribute of the data request information to be inserted with the value of the second attribute of the data request information stored in the queue;

according to the comparison result, inserting the data request information to be inserted into the corresponding position in the queue,

and according to the sequence from the head of the queue to the tail of the queue, the value of the second attribute of the data request information to be inserted is greater than the value of the second attribute of the previous stored data request information and less than the value of the second attribute of the next stored data request information.

4. The data storage method according to claim 3, wherein when the value of the second attribute corresponding to the data request information to be inserted is the same as the value of the second attribute corresponding to at least another data request information in the queue, the data request information to be inserted is inserted into the corresponding position in the queue in the order from small to large of the value of the first attribute.

5. The data storage method of claim 1, wherein the queue is a zset data structure, the first attribute corresponds to memer in the zset data structure, and the second attribute corresponds to score in the zset data structure.

6. The data storage method of claim 1, wherein the at least two kinds of data attribute information are sequentially stored as the number of bits of the value of the second attribute in accordance with the priority.

7. A data storage device, characterized in that the data storage device comprises:

a receiving module configured to: receiving data request information to be inserted, wherein the data request information comprises data identification information of the data request information to be inserted, and the data identification information is used as a first attribute corresponding to the data request information to be inserted;

a determination module configured to: determining a second attribute according to the data request information, wherein the second attribute comprises at least two data attribute information of the data request information to be inserted, and the at least two data attribute information is used for determining a data storage sequence;

a storage module configured to: and inserting the data request information to be inserted into the corresponding position in the queue according to the first attribute and the second attribute.

8. An electronic device, comprising:

at least one processor;

at least one memory storing computer-executable instructions,

wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 6.

9. A storage medium having instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the method of any of claims 1 to 6.

10. A computer program product, characterized in that instructions in the computer program product are executed by at least one processor in an electronic device to perform the method according to any of claims 1 to 6.