CN110427394B - Data operation method and device - Google Patents
Data operation method and device Download PDFInfo
- Publication number
- CN110427394B CN110427394B CN201910729086.4A CN201910729086A CN110427394B CN 110427394 B CN110427394 B CN 110427394B CN 201910729086 A CN201910729086 A CN 201910729086A CN 110427394 B CN110427394 B CN 110427394B
- Authority
- CN
- China
- Prior art keywords
- data
- cache
- release time
- bucket
- cache bucket
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
Abstract
The embodiment of the disclosure provides a data operation method and device, wherein after a server receives an operation instruction, such as a data storage instruction, a data release instruction or a data query instruction, according to release time indicated by the operation instruction, a cache bucket is determined from a cache bucket set and corresponding operation is performed. In the operation process, the corresponding cache barrel is determined according to the release time, so that data storage, data release or data query is performed, all data do not need to be traversed, the workload is small, the CPU occupancy rate is low, and the data operation speed is high.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of Redis databases, in particular to a data operation method and device.
Background
Remote dictionary server (Redis) is one of the most widely used memory unstructured databases at present, and Redis is an open-source, network-supported, memory-based key-value database. The data types supported by Redis to be stored comprise a string (string), a queue (list), a set (set), an ordered set (zset), a hash type (hash), and the like.
In the Redis zset technology, a server adopts a distributed cache structure, one server is provided with a plurality of memories, and each memory can be an ordered set and can also be called a cache bucket. When any cache bucket stores data, the position of the data to be stored in the cache bucket is determined according to the release moment of the data to be stored, the data to be stored is stored to the corresponding position, and the data in the cache bucket is an ordered data queue substantially. And in the process of releasing the data, releasing the data in the cache bucket according to the release time.
In the Redis zset-based data operation process, when the data volume in the cache bucket is large, for example, tens of millions of data are stored, at this time, if the cache bucket needs to be operated, for example, a certain data in the cache bucket is queried, and the current time does not reach the release time of the data to be queried, the cache bucket needs to be traversed, the workload is large, the CPU occupancy rate is high, so that the data is difficult to query, and even the phenomenon of query failure occurs.
Disclosure of Invention
The embodiment of the disclosure provides a data operation method and device, data are stored in cache buckets according to release time, so that when data are queried, the cache buckets are determined according to the release time of the data to be queried, and then the data are queried, and the method and device are small in workload and high in query efficiency.
In a first aspect, an embodiment of the present disclosure provides a data operation method, including:
receiving a data storage instruction, wherein the data storage instruction is used for indicating a first release time of data to be stored;
determining a storage cache bucket from a cache bucket set according to the first release time, wherein different cache buckets in the cache bucket set correspond to different release time periods;
and storing the data to be stored to the storage cache bucket.
In a second aspect, an embodiment of the present disclosure provides a data operating apparatus, including:
the device comprises a receiving module, a storage module and a processing module, wherein the receiving module is used for receiving a data storage instruction, and the data storage instruction is used for indicating a first release time of data to be stored;
the processing module is used for determining a storage cache bucket from a cache bucket set according to the first release time, wherein different cache buckets in the cache bucket set correspond to different release time periods;
and the storage module is used for storing the data to be stored to the storage cache bucket.
In a third aspect, the present disclosure provides a data operating apparatus, including a processor, a memory, and a computer program stored in the memory and executable on the processor, where the processor executes the program to implement the method according to the first aspect or the various possible implementations of the first aspect.
In a fourth aspect, embodiments of the present disclosure provide a storage medium having stored therein instructions that, when executed on a server, cause the server to perform a method as set forth in the first aspect or the various possible implementations of the first aspect.
In a fifth aspect, embodiments of the present disclosure provide a computer program product, which when run on a server, causes the server to perform the method as described above for the first aspect or the various possible implementations of the first aspect.
According to the data operation method and device provided by the embodiment of the disclosure, after the server receives the operation instruction, for example, after receiving the data storage instruction, the data release instruction or the data query instruction, the cache bucket is determined from the cache bucket set according to the release time indicated by the operation instruction, and corresponding operation is performed. In the operation process, the corresponding cache barrel is determined according to the release time, so that data storage, data release or data query is performed, all data do not need to be traversed, the workload is small, the CPU occupancy rate is low, and the data operation speed is high.
Drawings
In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present disclosure, and for those skilled in the art, other drawings can be obtained according to the drawings without inventive exercise.
FIG. 1 is a schematic diagram of an operating environment of a data operation method according to an embodiment of the present disclosure;
FIG. 2 is a flow chart of a method of data manipulation provided by an embodiment of the present disclosure;
FIG. 3 is a flow chart of another method of data manipulation provided by embodiments of the present disclosure;
FIG. 4 is a flow chart of yet another method of data manipulation provided by an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of a data operation device according to an embodiment of the present disclosure;
FIG. 6 is a schematic structural diagram of another data manipulation device according to an embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of another data manipulation device according to an embodiment of the present disclosure.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
Redis is a key-value based storage system, and is more and more widely applied due to higher read-write performance. The data types supported by Redis to be stored comprise a string (string), a queue (list), a set (set), an ordered set (zset), a hash type (hash), and the like. The method comprises the steps that distributed storage is adopted in a Redis zset-based technology, a plurality of storages correspond to one server, different storages of the server are regarded as different cache buckets, ordered data are stored in each cache bucket according to the release time of the data, each cache corresponds to a key-value pair, the memory space occupied by the value corresponding to the key is large, the key is called a big key (big key), and common big keys are divided into a big key of a character string type and a big key of a non-character string type.
In the data storage technology based on Redis zset, data in a cache bucket is an ordered data queue, each element (member) in the cache has a unique score (score) which is correspondingly bound with the element, and when data insertion, deletion and other operations are carried out on the cache bucket, the data are sorted according to the size of the score and the like, and then the operations of the insertion, the deletion and the like are carried out. A common playground for cache buckets includes: storing data, returning data, and releasing data. When the data is stored, the position of the data to be stored in the cache bucket is determined according to the score of the data to be stored, and then the data to be stored is stored to the corresponding position. After inserting data, for the stored data in the cache bucket, the score of the stored data needs to be updated to ensure that the stored data is in the correct location. The returned data refers to data in a designated score interval in the returned cache bucket, and can realize the functions of sorting and fetching the first few data, and the function is also obtained based on the characteristic that the data in the cache bucket is sorted according to the order of score values from small to small. Releasing data refers to releasing one or more data in the cache bucket. Wherein score can be obtained from the release time, etc., e.g., the later the release time, the higher score.
According to the above, it can be seen that: in the first technology based on Redis zset, data to be stored has a time attribute, and the time attribute is used for indicating the time of releasing the data to be stored after the data to be stored is stored in a cache bucket; second, operations on the cache bucket include storing data, returning data, and releasing data. When the amount of data in the cache bucket is large, such as tens of millions of data are stored, at this time, if the cache bucket is operated, such as storage operation, return operation or query operation, the position to be operated needs to be determined from the whole cache bucket, the workload is large, the CPU occupancy rate is high, and the phenomena of operation failure and server crash can occur in severe cases.
In view of this, the embodiments of the present disclosure provide a data operation method, in which data is stored in cache buckets according to release time, that is, a large cache bucket is divided into a plurality of small cache buckets, and when the large cache bucket is operated, the small cache bucket that needs to be operated is determined and operated, so that workload is small, and CPU occupancy is low.
First, concepts related to the embodiments of the present disclosure will be described in detail.
Redis: is an open source (BSD licensed), memory-stored data structure server that can be used as a database, cache, and message queue agent that supports data types such as strings, hash tables, lists, sets, ordered sets, bitmaps, etc. Built-in copy, Lua script, LRU eviction, transaction, and different levels of disk persistence functionality, while providing high availability through Redis sentinels (Sentinel), and automatic partitioning through Redis Cluster.
Zset, referred to as ordered set in Redis, may also be referred to as cache bucket in this application. In the prior art, a server with distributed storage has a plurality of storages, and one storage can be understood as one zset. Compared with the prior art, in the embodiment of the present disclosure, one memory may be divided into a plurality of cache buckets, and different cache buckets correspond to different release time periods, where the release time period is independent of a time point when data is stored in the cache bucket and is only related to a time when data is released from the cache bucket, so that: the cache bucket in the prior art is a large cache bucket, and the cache bucket in the embodiment of the disclosure is a small cache bucket. If no specific description is given below, the buffer bucket average value divides the memory into any of the plurality of buffer buckets.
Fig. 1 is a schematic operating environment diagram of a data operation method according to an embodiment of the present disclosure. Referring to fig. 1, the operating environment includes an electronic device 10 and a server 20, and a network connection is established between the electronic device 10 and the server 20, where the network connection may be a wired connection, a wireless communication link, a fiber optic cable, or the like. A user may use the electronic device 10 to interact with the server 20 to receive or send messages or the like. Various APPs, such as a shopping APP, a web browsing APP, a search APP, an instant messaging tool, social platform software, and the like, may be installed on the electronic device 10. The electronic device 10 includes, but is not limited to, a smart phone, a tablet computer, an e-book reader, a laptop portable computer, a desktop computer, and the like.
The server 20 may be a server 20 capable of providing various services, for example, a background management server 20 capable of providing support for a shopping network browsed by a user using the electronic device 10, and may query the received product information and feed back the query result to the electronic device 10. In the embodiment of the present disclosure, the server 20 is a server 20 provided with a Redis database, which may also be referred to as a Redis server 20, the data operation method described in the embodiment of the present disclosure is generally executed by the server 20, and accordingly, the data operation device is generally disposed on the server 20.
It should be noted that the number of the electronic devices 10 and the servers 20 shown in fig. 1 is only illustrative, and in actual implementation, any number of electronic devices and servers may be provided according to implementation requirements.
Next, a recommendation method according to an embodiment of the present disclosure will be described in detail based on fig. 1. For example, please refer to fig. 2.
Fig. 2 is a flowchart of a data operation method provided in an embodiment of the present disclosure. The embodiment of the present disclosure describes the data operation method in detail from the perspective of data storage, and the embodiment of the present disclosure includes:
101. receiving a data storage instruction, wherein the data storage instruction is used for indicating a first release time of data to be stored.
Generally, access to a server provided with a Redis database mainly includes data storage, data query, and data release.
For data storage, when the electronic equipment needs to write data into the server, the release time of the data to be stored is carried in a data storage instruction and sent to the server; accordingly, the server receives the data storage instruction.
102. And determining a storage cache bucket from a cache bucket set according to the first release time, wherein different cache buckets in the cache bucket set correspond to different release time periods.
Illustratively, there are many cache buckets in the cache bucket set, and different cache buckets correspond to different release periods. The release time period can be set by self according to actual needs, for example, a cache bucket can be set according to one release time period every 5 minutes; as another example, 12: the time period of 00-13: 00 is a cache bucket. After the cache buckets are set, the unique identifiers of the cache buckets do not need to be recorded, the unique identifiers can also be called as small keys of the cache buckets, and the unique identifiers can be calculated through a hash algorithm without extra recording or maintenance. For a specific cache bucket, the data stored in the cache bucket may be sorted in the bucket according to the release time, so as to realize time-phased output of the data in one cache bucket. Or, if the release time period corresponding to the cache bucket is short enough, a random sequence sorting mode and the like can be considered, and at this time, when releasing the data in the bucket, the data can be released at one time.
After receiving the data storage instruction, the server determines the release time period of the first release time, and takes the cache bucket corresponding to the release time period as a storage cache bucket. For example, there are 24 cache buckets in the cache bucket set, the unique identifiers of the 24 cache buckets are 1, 2 and … … 24, respectively, and correspond to different hours in 24 hours in a day, the first release time is 15:00, obviously, since the release period of the 15 th cache bucket is 15:00, the first release time falls into the release period of the 15 th cache bucket, and therefore, the lower 15 cache buckets are taken as storage cache buckets. Obviously, if data of 24 hours a day is stored in one large cache bucket, the data amount is quite huge, in the embodiment of the disclosure, the data of 24 hours a day is stored in 24 small cache buckets, and the data amount in each small cache bucket is relatively small, so that the power amount of server operation can be reduced, the CPU occupancy rate is low, and the operation speed is high.
103. And storing the data to be stored to the storage cache bucket.
In this step, the data to be stored is stored into the storage cache bucket randomly or according to certain data, such as the sequence of the release time, regardless of the sequence, and then the data and the data stored in the storage cache bucket are sorted, so that the data in the storage cache bucket is ordered.
According to the data operation method provided by the embodiment of the disclosure, after a server receives a data storage instruction, a storage cache bucket is determined from a cache bucket set according to a first release time indicated by the data storage instruction, and data to be stored is stored in the storage cache bucket. Wherein different cache buckets in the cache bucket set correspond to different release time periods. In the process, when the server stores data, the storage cache barrel is determined according to the first release time so as to store the data to be stored, the workload is small, the CPU occupancy rate is low, and the data storage speed is high.
In the above embodiment, data stored in any cache bucket in the cache bucket set is ordered data, and when the server receives a data storage instruction, it is highly probable that data is already stored in the cache bucket corresponding to the first release time indicated by the data storage instruction, and at this time, the server determines the position of the data to be stored in the cache bucket according to the first release time of the data to be stored and the release time of the data already stored in the cache bucket; and storing the data to be stored to the corresponding position of the storage cache bucket.
For example, data in one cache bucket is likely not to be stored at one time, that is, when data to be stored is stored, the data is already stored in the cache bucket, at this time, the data needs to be sorted according to the data to be stored and the release time of the stored data in the cache bucket, so as to determine the position of the data to be stored in the cache bucket, and then, the data to be stored is stored according to the position. For example, assuming that the first release time is 15:30, the release time period of the storage cache bucket corresponding to the first release time is 15: 00-16: 00. After receiving a data storage instruction, the server finds that the data stored in the storage cache bucket has been stored, the release time of the stored data is 15:20, and if the release time is in the order from morning to evening, the data to be stored is stored behind the stored data; and if the release time is in the order from late to early, storing the data to be stored before the stored data.
In the above embodiment, a cache bucket in a cache bucket set may not cover all release time periods, after receiving a data storage instruction, a server determines whether a first release time indicated by the data storage instruction falls into a release time period corresponding to a first cache bucket in the cache bucket set, and if the first release time falls into the release time period corresponding to the first cache bucket, the first cache bucket is used as the storage cache bucket, where the first cache bucket is any one of the cache buckets in the cache bucket set; if the first release time does not fall into the release time periods corresponding to all the cache buckets in the cache bucket set, constructing a second cache bucket in the cache bucket set, and taking the second cache bucket as the storage cache bucket.
Illustratively, after obtaining the first release time, the server determines whether the first release time falls into a release time period corresponding to an existing cache bucket in the cache bucket set, if so, the existing cache bucket is used as a storage cache bucket and stores data, otherwise, a new cache bucket is created, and the data to be stored is stored into the new cache bucket. For example, assume that there are two cache buckets in the cache bucket set, corresponding to a release period of 12: 00-13: 00 and a release period of 15: 00-17: 00, respectively. If the first release time is 14:00, the server newly constructs a cache bucket in the cache bucket set, and the release time period corresponding to the cache bucket is 13: 00-15: 00.
In addition, in the above embodiment, in the process of storing data, if the data to be stored is more, the amount of data in the storage cache bucket is huge after the data to be stored is stored in the storage cache bucket. At this time, the storage cache bucket may be considered to be split into a plurality of sub-cache buckets. How to disassemble the molecule cache bucket will be described in detail below.
In a feasible implementation manner, after the server stores the data to be stored in the storage cache bucket, it is further determined whether the data amount of the stored data in the storage cache bucket exceeds a threshold value; if the data volume of the stored data in the storage cache bucket exceeds a threshold value, dividing the storage cache bucket into at least two sub cache buckets, wherein different sub cache buckets correspond to different release time periods, and the sum of the release time periods corresponding to the sub cache buckets is the release time period corresponding to the storage cache bucket.
For example, if 10000 pieces of data are stored in the storage cache bucket after the data to be stored is stored in the storage cache bucket, and the corresponding release time period is 15:00 to 16:00, the storage cache bucket may be divided into two sub-cache buckets, the release time periods are 15:00 to 15:30 and 15:30 to 16:00, and then the data are stored in the corresponding sub-cache buckets according to the release time of the data in the storage cache bucket.
Fig. 3 is a flowchart of another data manipulation method provided by the embodiments of the present disclosure. The embodiment of the present disclosure describes the data operation method in detail from the perspective of data release, and the embodiment of the present disclosure includes:
201. and receiving a data release instruction, wherein the data release instruction is used for indicating a second release time.
Generally, access to a server provided with a Redis database mainly includes data storage, data query, and data release.
202. And determining a released cache bucket from the cache bucket set according to the second release time, wherein the second release time falls into a release time period corresponding to the released cache bucket.
For the description of the cache bucket set and the cache bucket, refer to step 102 above, and the description is omitted here.
Illustratively, there are many cache buckets in the cache bucket set, and different cache buckets correspond to different release periods. And after receiving the data release instruction, the server determines the release time period of the second release time, and takes the cache bucket corresponding to the release time period as a release cache bucket. For example, there are 24 cache buckets in the cache bucket set, the unique identifiers of the 24 cache buckets are 1, 2 and … … 24 respectively, the unique identifiers correspond to different hours in 24 hours in a day, the second release time is 15:00, obviously, since the release time period of the 15 th cache bucket is 15:00, the second release time falls into the release time period of the 15 th cache bucket, and therefore, the lower 15 cache buckets are taken as release cache buckets.
203. And when the current time reaches the second release time, sequentially releasing the data in the release cache bucket according to the release time of each stored data in the release cache bucket.
Illustratively, the server monitors the time, and when the current time reaches a second release time, the data in the release cache bucket is released. In the releasing process, all data in the cache bucket can be released at one time. In addition, the release time of the data stored in the same cache bucket is different. For example, although the cache bucket corresponding to the second release time is the 15 th cache bucket, the data in the released cache bucket is not released at one time, but is released according to the sequence of release.
According to the data operation method provided by the embodiment of the disclosure, after the server receives the data release instruction, the server determines to release the cache bucket from the cache bucket set according to the second release time indicated by the data release instruction, and releases the data in the released cache bucket when the current time reaches the second release time. In the process, when the server releases the data, the cache bucket is determined to be released according to the second release time, so that the data release is carried out, the workload is small, the CPU occupancy rate is low, and the data release speed is high.
Fig. 4 is a flowchart of another data manipulation method provided by the embodiments of the present disclosure. The embodiment of the present disclosure describes the data operation method in detail from the perspective of data query, and the embodiment of the present disclosure includes:
301. and receiving a data query instruction, wherein the data query quality is used for indicating a third release time.
Generally, access to a server provided with a Redis database mainly includes data storage, data query, and data release.
For data query, when a user needs to query data stored in the server, a data query instruction carrying a third release time is sent to the server through the electronic equipment; accordingly, the server receives the data query quality instruction.
302. And determining a query cache bucket from the cache bucket set according to the third release time, wherein the third release time falls into a release time period corresponding to the query cache bucket.
For the description of the cache bucket set and the cache bucket, refer to step 102 above, and the description is omitted here.
Illustratively, there are many cache buckets in the cache bucket set, and different cache buckets correspond to different release periods. And after receiving the data query instruction, the server determines the release time period of the third release time, and takes the cache bucket corresponding to the release time period as the query cache bucket. For example, there are 24 cache buckets in the cache bucket set, the unique identifiers of the 24 cache buckets are 1, 2 and … … 24, respectively, and correspond to different hours in 24 hours in a day, the third release time is 15:00, and since the release period of the 15 th cache bucket is 15:00, the third release time falls into the release period of the 15 th cache bucket, the lower 15 cache buckets are taken as release cache buckets. Obviously, if data of 24 hours a day is stored in one large cache bucket, the data volume is quite large, in the embodiment of the present disclosure, the data of 24 hours a day is stored in 24 small cache buckets, the data volume in each small cache bucket is relatively small, when querying data, it is not necessary to traverse all cache buckets, but only a release period corresponding to the third release time is found, and the cache bucket can be determined and data query can be performed according to the release period. Therefore, the amount of power for operating the server can be reduced, so that the CPU occupancy is low and the operation speed is high.
303. And querying the stored data in the query cache bucket.
For example, the server may query the data in the cache bucket at any point in time before the data in the cache bucket is not released.
According to the data operation method provided by the embodiment of the disclosure, after the server receives the data query instruction, the server determines the query cache bucket from the cache bucket set according to the third release time indicated by the data query instruction, and queries the data in the query cache bucket. In the process, when the server inquires data, the inquiry cache barrel is determined according to the third release time so as to inquire the data, the workload is small, the CPU occupancy rate is low, and the data release speed is high.
It should be noted that, although the above embodiments of fig. 2, fig. 3 and fig. 4 are described separately for data storage, data release and data query. However, the data storage, the data release and the data query are not independent, and in practice, the data storage, the data release and the data release may be performed simultaneously or sequentially, and the embodiments of the present disclosure are not limited.
The following are embodiments of the disclosed apparatus that may be used to perform embodiments of the disclosed methods. For details not disclosed in the embodiments of the apparatus of the present disclosure, refer to the embodiments of the method of the present disclosure.
Fig. 5 is a schematic structural diagram of a data operation device according to an embodiment of the present disclosure. The data operation device according to the present embodiment may be a server, or may be a chip applied to a server. The data manipulation device may be used to perform the functions of the server in the above embodiments. As shown in fig. 5, the data manipulation device 100 may include:
a receiving module 11, configured to receive a data storage instruction, where the data storage instruction is used to indicate a first release time of data to be stored;
a processing module 12, configured to determine a storage cache bucket from a cache bucket set according to the first release time, where different cache buckets in the cache bucket set correspond to different release time periods;
and the storage module 13 is configured to store the data to be stored in the storage cache bucket.
In one possible design, the memory module 13 is used for
Determining the position of the data to be stored in the storage cache bucket according to the first release time of the data to be stored and the release time of the stored data in the storage cache bucket; and storing the data to be stored to the position of the storage cache bucket.
In one possible design, the processing module 12 is used for
Judging whether the first release time falls into a release time period corresponding to a first cache bucket in the cache bucket set, and if the first release time falls into the release time period corresponding to the first cache bucket, using the first cache bucket as the storage cache bucket, wherein the first cache bucket is any one cache bucket in the cache bucket set; if the first release time does not fall into the release time periods corresponding to all the cache buckets in the cache bucket set, constructing a second cache bucket in the cache bucket set, and taking the second cache bucket as the storage cache bucket.
Fig. 6 is a schematic structural diagram of another data operation apparatus according to an embodiment of the present disclosure. The data operating device 100 provided in this embodiment further includes, on the basis of the foregoing fig. 5: the release module 14, the receiving module 11, further configured to receive a data release instruction, where the data release instruction is used to indicate a second release time;
the processing module 12 is further configured to determine, according to the second release time, a release cache bucket from the cache bucket set, where the second release time falls within a release time period corresponding to the release cache bucket;
the releasing module 14 is configured to sequentially release the data in the release cache bucket according to the release time of each stored data in the release cache bucket when the current time reaches the second release time.
Referring to fig. 6 again, the data operation apparatus 100 further includes: the query module 15, the receiving module 11, is further configured to receive a data query instruction, where the data query quality is used to indicate a third release time;
the processing module 12 is further configured to determine an inquiry cache bucket from the cache bucket set according to the third release time, where the third release time falls into a release time period corresponding to the inquiry cache bucket;
the query module 15 is configured to query the data stored in the query cache bucket.
In one possible design, the cache buckets in the set of cache buckets are different or the same size;
the lengths of the release time periods corresponding to the cache buckets in the cache bucket set are the same or different.
In a possible design, the processing module 12 is further configured to determine whether the data amount of the stored data in the storage cache bucket exceeds a threshold after the storage module 13 stores the data to be stored in the storage cache bucket;
if the data volume of the stored data in the storage cache bucket exceeds a threshold value, dividing the storage cache bucket into at least two sub cache buckets, wherein different sub cache buckets correspond to different release time periods, and the sum of the release time periods corresponding to the sub cache buckets is the release time period corresponding to the storage cache bucket.
Fig. 7 is a schematic structural diagram of another data manipulation device according to an embodiment of the present disclosure. As shown in fig. 7, the data manipulation device 200 includes:
at least one processor 21 and memory 22;
the memory 22 stores computer-executable instructions;
the at least one processor 21 executes computer-executable instructions stored by the memory 22 to cause the at least one processor 21 to perform the data manipulation method described above.
Optionally, the data manipulation device 200 further comprises a communication component 23. The processor 21, the memory 22, and the communication unit 23 may be connected by a bus 24.
The embodiment of the present disclosure also provides a storage medium, in which computer execution instructions are stored, and when the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the data operation method described above.
The embodiment of the disclosure also provides a computer program product, which when running on a server, causes the server to execute the data operation method.
In the above embodiments, it should be understood that the described apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules is only one logical division, and other divisions may be realized in practice, for example, a plurality of modules may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present disclosure may be integrated into one processing unit, or each module may exist alone physically, or two or more modules are integrated into one unit. The unit formed by the modules can be realized in a hardware form, and can also be realized in a form of hardware and a software functional unit.
The integrated module implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions to cause a server (which may be a personal computer, a server, or a network device, etc.) or a processor (english: processor) to execute some steps of the method according to the embodiments of the present disclosure.
It should be understood that the processor may be a Central Processing Unit (CPU), other general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present disclosure may be embodied directly in a hardware processor, or in a combination of the two.
The memory may comprise a high-speed RAM memory, and may further comprise a non-volatile storage NVM, such as at least one disk memory, and may also be a usb disk, a removable hard disk, a read-only memory, a magnetic or optical disk, etc.
The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the buses in the figures of the present disclosure are not limited to only one bus or one type of bus.
The storage medium may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks. A storage media may be any available media that can be operated by a general purpose or special purpose computer.
An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be integral to the processor. The processor and the storage medium may reside in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the storage medium may reside as discrete components in a terminal or server.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the present application are all or partially generated when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium upon which a computer can operate or a data storage device including one or more available media integrated servers, data centers, and the like. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The term "plurality" herein means two or more. The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship; in the formula, the character "/" indicates that the preceding and following related objects are in a relationship of "division".
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present disclosure, and not for limiting the same; while the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present disclosure.
Claims (8)
1. A method of data manipulation, comprising:
receiving a data storage instruction, wherein the data storage instruction is used for indicating a first release time of data to be stored;
determining a storage cache bucket from a cache bucket set according to the first release time, wherein different cache buckets in the cache bucket set correspond to different release time periods;
storing the data to be stored to the storage cache bucket;
after the storing the data to be stored to the storage cache bucket, the method further includes:
judging whether the data quantity of the stored data in the storage cache bucket exceeds a threshold value or not;
if the data volume of the stored data in the storage cache bucket exceeds a threshold value, dividing the storage cache bucket into at least two sub cache buckets, wherein different sub cache buckets correspond to different release time periods, and the sum of the release time periods corresponding to the sub cache buckets is the release time period corresponding to the storage cache bucket.
2. The method of claim 1, wherein storing the data to be stored to the storage cache bucket comprises:
determining the position of the data to be stored in the storage cache bucket according to the first release time of the data to be stored and the release time of the stored data in the storage cache bucket;
and storing the data to be stored to the position of the storage cache bucket.
3. The method of claim 1, wherein determining a storage cache bucket from a set of cache buckets according to the first release time comprises:
judging whether the first release time falls into a release time period corresponding to a first cache bucket in the cache bucket set, and if the first release time falls into the release time period corresponding to the first cache bucket, using the first cache bucket as the storage cache bucket, wherein the first cache bucket is any one cache bucket in the cache bucket set;
if the first release time does not fall into the release time periods corresponding to all the cache buckets in the cache bucket set, constructing a second cache bucket in the cache bucket set, and taking the second cache bucket as the storage cache bucket.
4. The method according to any one of claims 1 to 3, further comprising:
receiving a data release instruction, wherein the data release instruction is used for indicating a second release time;
determining a released cache bucket from the cache bucket set according to the second release time, wherein the second release time falls into a release time period corresponding to the released cache bucket;
and when the current time reaches the second release time, sequentially releasing the data in the release cache bucket according to the release time of each data in the release cache bucket.
5. The method according to any one of claims 1 to 3, further comprising:
receiving a data query instruction, wherein the data query instruction is used for indicating a third release time;
determining an inquiry cache bucket from the cache bucket set according to the third release time, wherein the third release time falls into a release time period corresponding to the inquiry cache bucket;
and querying the stored data in the query cache bucket.
6. A data manipulation device, comprising:
the device comprises a receiving module, a storage module and a processing module, wherein the receiving module is used for receiving a data storage instruction, and the data storage instruction is used for indicating a first release time of data to be stored;
the processing module is used for determining a storage cache bucket from a cache bucket set according to the first release time, wherein different cache buckets in the cache bucket set correspond to different release time periods;
the storage module is used for storing the data to be stored to the storage cache barrel;
the processing module is used for judging whether the data volume of the stored data in the storage cache bucket exceeds a threshold value or not after the storage module stores the data to be stored in the storage cache bucket;
if the data volume of the stored data in the storage cache bucket exceeds a threshold value, dividing the storage cache bucket into at least two sub cache buckets, wherein different sub cache buckets correspond to different release time periods, and the sum of the release time periods corresponding to the sub cache buckets is the release time period corresponding to the storage cache bucket.
7. A server comprising a processor, a memory and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of the preceding claims 1-5 when executing the program.
8. A storage medium having stored therein instructions which, when run on a server, cause the server to perform the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910729086.4A CN110427394B (en) | 2019-08-08 | 2019-08-08 | Data operation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910729086.4A CN110427394B (en) | 2019-08-08 | 2019-08-08 | Data operation method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110427394A CN110427394A (en) | 2019-11-08 |
| CN110427394B true CN110427394B (en) | 2022-02-25 |
Family
ID=68414977
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910729086.4A Active CN110427394B (en) | 2019-08-08 | 2019-08-08 | Data operation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110427394B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113688062B (en) * | 2020-05-18 | 2022-08-26 | 北京市商汤科技开发有限公司 | Method for storing data and related product |
| CN111913977B (en) * | 2020-08-19 | 2023-10-20 | 上海莉莉丝网络科技有限公司 | Data processing method, device and medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105653198A (en) * | 2014-11-13 | 2016-06-08 | 杭州迪普科技有限公司 | Data processing method and device |
| CN106202485A (en) * | 2016-07-18 | 2016-12-07 | 乐视控股（北京）有限公司 | Data manipulation method and system |
| CN107092628A (en) * | 2017-01-10 | 2017-08-25 | 口碑控股有限公司 | The treating method and apparatus of time series data |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10067876B2 (en) * | 2017-01-09 | 2018-09-04 | Splunk, Inc. | Pre-fetching data from buckets in remote storage for a cache |
| US10356115B2 (en) * | 2017-03-31 | 2019-07-16 | Level 3 Communications, Llc | Creating aggregate network flow time series in network anomaly detection systems |
-
2019
- 2019-08-08 CN CN201910729086.4A patent/CN110427394B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105653198A (en) * | 2014-11-13 | 2016-06-08 | 杭州迪普科技有限公司 | Data processing method and device |
| CN106202485A (en) * | 2016-07-18 | 2016-12-07 | 乐视控股（北京）有限公司 | Data manipulation method and system |
| CN107092628A (en) * | 2017-01-10 | 2017-08-25 | 口碑控股有限公司 | The treating method and apparatus of time series data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110427394A (en) | 2019-11-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109947668B (en) | Method and device for storing data | |
| US9223564B2 (en) | Update systems responsive to ongoing processing at a storage system | |
| CN108829344A (en) | Date storage method, device and storage medium | |
| CN111247518A (en) | Database sharding | |
| CN107092628B (en) | Time series data processing method and device | |
| CN110427394B (en) | Data operation method and device | |
| US11593268B2 (en) | Method, electronic device and computer program product for managing cache | |
| CN109697019B (en) | Data writing method and system based on FAT file system | |
| US11237761B2 (en) | Management of multiple physical function nonvolatile memory devices | |
| CN111400334A (en) | Data processing method, data processing device, storage medium and electronic device | |
| US11093389B2 (en) | Method, apparatus, and computer program product for managing storage system | |
| CN111930305A (en) | Data storage method and device, storage medium and electronic device | |
| US11080239B2 (en) | Key value store using generation markers | |
| CN111143113A (en) | Method, electronic device and computer program product for copying metadata | |
| CN117009389A (en) | Data caching method, device, electronic equipment and readable storage medium | |
| CN113132241B (en) | ACL template dynamic configuration method and device | |
| CN113986921A (en) | Blacklist query method, system, electronic equipment and storage medium | |
| CN113742131B (en) | Method, electronic device and computer program product for storage management | |
| CN110427525B (en) | Data access method and device | |
| CN111666045A (en) | Data processing method, system, equipment and storage medium based on Git system | |
| CN108959517B (en) | File management method and device and electronic equipment | |
| CN108733678B (en) | Data searching method, device and related equipment | |
| CN111858590A (en) | Storage system metadata organization method, system, terminal and storage medium | |
| CN110968267A (en) | Data management method, device, server and system | |
| CN113806249B (en) | Object storage sequence lifting method, device, terminal and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |