CN115374117A

CN115374117A - Data processing method and device, readable storage medium and electronic equipment

Info

Publication number: CN115374117A
Application number: CN202211014402.8A
Authority: CN
Inventors: 徐泉清; 方伯阳; 王国平; 聂铁铮; 杨传辉; 申德荣; 寇月
Original assignee: Beijing Oceanbase Technology Co Ltd
Current assignee: Beijing Oceanbase Technology Co Ltd
Priority date: 2022-08-23
Filing date: 2022-08-23
Publication date: 2022-11-22

Abstract

The specification discloses a data processing method, a data processing device, a readable storage medium and electronic equipment, wherein key value pairs corresponding to keys to be processed carried in a data processing request are inquired from a hot data table and a cold data table as specified key value pairs based on a pre-deployed hot data table and a pre-deployed cold data table, target key value pairs processed by the specified key value pairs are determined, and the target key value pairs are stored in the hot data table according to storage positions of the specified key value pairs in the hot data table and the cold data table. The cold data table and the hot data table in the method only store the latest processing results corresponding to each key, so that when data query is carried out, only the specified key value pair corresponding to the key to be processed needs to be queried in the hot data table and the cold data table, the condition that a large number of invalid key value pairs need to be traversed when data query is carried out is avoided, and the data query and data processing efficiency is improved.

Description

Data processing method and device, readable storage medium and electronic equipment

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a data processing method and apparatus, a readable storage medium, and an electronic device.

Background

At present, with the development of computer technology, the need of deep service integration and the growing concern of people on private data, databases are increasingly applied to people's lives for service processing, and how to ensure the data processing efficiency also becomes one of the problems to be solved in the field of databases.

The present specification provides a data processing method, in which, after data to be processed is received, how to perform data processing and improving data processing efficiency becomes a problem to be solved.

Disclosure of Invention

The present specification provides a data processing method, an apparatus, a readable storage medium, and an electronic device, so as to partially solve the above problems in the prior art.

The technical scheme adopted by the specification is as follows:

the present specification provides a data processing method, which is applied to a database, where a hot data table and a cold data table are pre-deployed in the database, and the method includes:

receiving a data processing request, wherein the data processing request carries a key to be processed;

according to the key to be processed, inquiring a key value pair corresponding to the key to be processed from the hot data table and the cold data table to serve as an appointed key value pair;

processing the appointed key value pair according to the data processing request to obtain a target key value pair;

if the specified key-value pair is stored in the hot data table, updating the specified key-value pair in the hot data table according to the target key-value pair, and when the target key-value pair meets a preset cooling condition, moving the target key-value pair from the hot data table into the cold data table;

if the specified key-value pair is stored in the cold data table, deleting the specified key-value pair in the cold data table, adding the target key-value pair to the hot data table, and when the target key-value pair meets a preset cooling condition, moving the target key-value pair from the hot data table to the cold data table.

The present specification provides a data processing apparatus applied to a database, in which a hot data table and a cold data table are pre-deployed, including:

the device comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving a data processing request which carries keys to be processed;

the determining module is used for inquiring the key value pair corresponding to the key to be processed from the hot data table and the cold data table according to the key to be processed, and the key value pair is used as a specified key value pair;

the processing module is used for processing the appointed key value pair according to the data processing request to obtain a target key value pair;

the first storage module is used for updating the appointed key-value pair in the hot data table according to the target key-value pair if the appointed key-value pair is stored in the hot data table, and moving the target key-value pair from the hot data table to the cold data table when the target key-value pair meets a preset cooling condition;

and the second storage module is used for deleting the appointed key-value pair in the cold data table, adding the target key-value pair into the hot data table and moving the target key-value pair from the hot data table into the cold data table when the target key-value pair meets a preset cooling condition if the appointed key-value pair is stored in the cold data table.

The present specification provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the above-described data processing method.

The present specification provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the above data processing method when executing the program.

The technical scheme adopted by the specification can achieve the following beneficial effects:

and based on a hot data table and a cold data table which are deployed in advance, inquiring a key value pair corresponding to a key to be processed carried in the data processing request from the hot data table and the cold data table to be used as an appointed key value pair, determining a target key value pair processed by the appointed key value pair, and storing the target key value pair in the hot data table according to the storage position of the appointed key value pair in the hot data table and the cold data table.

The cold data table and the hot data table in the method only store the latest processing results corresponding to each key, so that when data query is carried out, only the appointed key value pair corresponding to the key to be processed needs to be queried in the hot data table and the cold data table, the condition that a large number of invalid key value pairs need to be traversed when data query is carried out at present is avoided, and the data query and data processing efficiency is improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the specification and are incorporated in and constitute a part of this specification, illustrate embodiments of the specification and together with the description serve to explain the specification and not to limit the specification in a non-limiting sense. On the attachment

In the figure:

FIG. 1 is a schematic flow chart of a data processing method provided herein;

FIG. 2 is a schematic flow chart of a data processing method provided herein;

FIG. 3 is a data processing apparatus provided herein;

fig. 4 is a schematic diagram of an electronic device corresponding to fig. 1 provided in the present specification.

Detailed Description

To make the objects, technical solutions and advantages of the present specification clearer and more complete, the technical solutions of the present specification will be described in detail and completely with reference to the specific embodiments of the present specification and the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present disclosure, and not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present specification without making any creative effort belong to the protection scope of the present specification.

The technical solutions provided by the embodiments of the present description are described in detail below with reference to the accompanying drawings.

Fig. 1 is a schematic flow chart of a data processing method provided in this specification, and specifically includes the following steps:

s100: receiving a data processing request, wherein the data processing request carries a key to be processed.

Generally, in the field of data processing, processing data in a database generally requires three stages: a data query stage, a data processing stage and a data storage stage. That is, data is first queried and then processed, and the data is stored after being processed, so as to avoid that corresponding data cannot be queried subsequently. When the database stores data, the data storage form is usually a key value pair, that is, based on a corresponding key, a value corresponding to the key can be found.

Based on this, the key corresponding to the key-value pair that needs to be queried may be determined in order to determine an accurate key-value pair.

Specifically, the data processing request may be determined first. The data processing request may be a request statement sent by the electronic device, or may be a query statement. The electronic device may be a server, a terminal, an intelligent device, and the like, and the form of the data processing request may be set as required, which is not limited in this specification.

Then, the data processing request can be analyzed to obtain the key carried in the data processing request.

Finally, the determined key can be used as a key to be processed. The key to be processed is a key corresponding to the key value pair which needs to be subjected to data processing.

S102: and according to the key to be processed, inquiring a key value pair corresponding to the key to be processed from the hot data table and the cold data table to serve as an appointed key value pair.

Different from the prior art that the specified key-value pairs are directly inquired from the memory table, the efficiency of data inquiry and data processing is low due to the large number of invalid key-value pairs in the memory table. The present specification provides a new data processing method, where a hot data table and a cold data table are pre-deployed in a database, after a data processing request is received, based on a key to be processed carried in the data processing request, a key value pair corresponding to the key to be processed is queried from the hot data table and the cold data table, and the key value pair is used as an assigned key value pair, and then data processing is performed on the assigned key value pair according to the data processing request, so as to determine a target key value pair. And storing the target key-value pair in the hot data table according to the storage positions of the specified key-value pair in the hot data table and the cold data table. Wherein for each key-value pair in the hot data table, when the key-value pair satisfies the cooling condition, the key-value pair is moved into the cold data table.

Compared with the existing data processing method, the method is based on the cold data table and the hot data table which only store the latest value corresponding to each key, the latest value corresponding to each key is separated from the historical key value corresponding to each key, when data is inquired, the appointed key value corresponding to the key to be processed is only required to be inquired in the hot data table and the cold data table, the situation that a large number of invalid key value pairs need to be traversed when data is inquired at present is avoided, and the data inquiry and data processing efficiency is improved.

Based on the above brief description of the data processing method provided in this specification, the data processing method provided in this specification can be applied to a relational database and a database, for example, a key-value database, and the data processing method can be implemented by processing data by the database according to a received data processing request and storing the processed result in a hot data table and a cold data table.

In this way, since the most recent processing results corresponding to the key values are stored in the hot data table and the cold data table, the database that performs data processing based on the data processing method provided in the present specification can be searched for a specific key value pair based on only the key values stored in the hot data table and the cold data table.

Specifically, the database may query, according to the key to be processed determined in step S100, the key value pair corresponding to the key to be processed from the hot data table and the cold data table according to the key corresponding to each key value pair stored in the hot data table and the cold data table, and use the key value pair as the designated key value pair.

Further, since the processing time of a key-value pair stored in the hot data table is late, if a certain key-value pair is hot data, the probability of the key-value pair in the hot data table is higher. The database may then first determine the specified key-value pair from the hot data table and then query the specified key-value pair from the cold data table without querying the specified key-value pair from the hot data table.

Specifically, the database may query, according to the key to be processed determined in step S100 and the key corresponding to each key value pair stored in the thermal data table, the key value pair corresponding to the key to be processed from the thermal data table.

If so, the database may use the queried key-value pair as the specified key-value pair.

If not, the database can inquire the key value pair corresponding to the key to be processed from the cold data table according to the key corresponding to each key value pair in the key to be processed and the cold data table.

Similarly, if the key-value pair corresponding to the key to be processed is searched from the cold data table, the key-value pair is taken as the designated key-value pair.

If the specified key-value pair is not searched from the cold data table, the database can search the key-value pair corresponding to the key to be processed from the data stored in the persistent manner according to the key corresponding to each key-value pair in the key to be processed and the data stored in the persistent manner.

Of course, if the data is not queried from the database, the database may return a prompt message according to the data processing request, where the prompt message is used to prompt that the data processing request has an exception. The database can also create the appointed key value pair, assign the appointed key value pair to be null, and return the null value as a processing result based on the data processing request. Specifically, how to process the specified key value pairs is not queried from the database, and the processing result is returned and can be set as required, which is not limited in this specification.

S104: and processing the appointed key value pair according to the data processing request to obtain a target key value pair.

In one or more embodiments provided herein, as previously described, processing data in a database typically requires three phases: in the data query stage, the data processing stage and the data storage stage, after the specified key value pair needing data processing is queried, the specified key value pair needs to be processed.

Based on this, after determining the designated key-value pair, the database may process the designated key-value pair according to the data processing request.

Specifically, the data processing request is a processing request for performing operations such as adding, deleting, modifying, and searching on data in the database.

Thus, after the designated key-value pair is determined, the database may process the value of the designated key-value pair according to the execution data carried in the data processing request, and combine the value after data processing with the key to be processed to obtain the target key-value pair.

Of course, when the value after data processing is combined with the key to be processed, the database may also assign a new physical storage address to the value after data processing based on the physical storage address corresponding to the key to be processed.

S106: if the specified key-value pair is stored in the hot data table, updating the specified key-value pair in the hot data table according to the target key-value pair, and when the target key-value pair meets a preset cooling condition, moving the target key-value pair from the hot data table to the cold data table.

In one or more embodiments provided in this specification, as described above, the core idea of this solution is to distinguish the latest value corresponding to each key from the historical key-value pair corresponding to each key based on the hot data table and the cold data table, that is, only the latest key-value pair corresponding to each key is stored in the hot data table and the cold data table.

And for each key in the database, if the key is a key to be processed, determining that the target key-value pair is the latest key-value pair corresponding to the key, and then after determining the target key-value pair, adding the target key-value pair to the hot data table by the database.

In the hot data table and the cold data table, only the latest key value pair corresponding to each key is stored. Thus, a distinction is made between the steps that the database needs to perform when the specified key-value pair is stored in the hot data table and the steps that the database needs to perform when the key-value pair is stored in the cold data table, and therefore the database can perform subsequent steps based on where the specified key-value pair is stored in the database.

Specifically, the database may determine whether the thermal data table stores the specified key-value pair according to the key of the target key-value pair and the key corresponding to each key-value pair stored in the thermal data table.

If so, the database may update the specified key-value pair in the hot data table according to the target key-value pair. That is, the data corresponding to the designated key-value pair is updated to the data corresponding to the target key-value pair.

Thus, a target key-value pair may be stored in the hot data table.

The reason for setting the hot data table is that: the hot data is stored in the hot data table, and when data query is carried out, the data is firstly obtained from the hot data table, so that the condition that I/0 overhead is large due to excessive disk access times when data query is carried out and the hot data is subjected to persistent storage is avoided.

Therefore, when data is persistently stored, the data in the hot data table cannot be persistently stored in the disk, and the storage resources in the memory are limited, so that the amount of data stored in the hot data table cannot be too much, otherwise, the data in the hot data table is too much, which may cause frequent occupancy of the memory, and further cause frequent persistence.

Based on this, for each key-value pair stored in the hot data table, the key-value pair is moved into the cold data table when the key-value pair satisfies a preset cooling condition.

Specifically, for each key-value pair, the attribute information of the key-value pair includes at least one of a determination time of the key-value pair, a storage space occupied by storing the key-value pair, and an access frequency of the key-value pair. The cooling conditions may then be: and determining at least one of the earliest time, the largest occupied storage space and the least access times.

Of course, for each key-value pair, the database may further determine a heat corresponding to the key-value pair based on the attribute information of the key-value pair, and determine whether the key-value pair satisfies a preset cooling condition based on the determined heat, and then the cooling condition may also be: and the key value pair with the heat degree lower than the preset heat degree threshold value meets the preset cooling condition.

S108: if the specified key-value pair is stored in the cold data table, deleting the specified key-value pair in the cold data table, adding the target key-value pair to the hot data table, and when the target key-value pair meets a preset cooling condition, moving the target key-value pair from the hot data table to the cold data table.

In one or more embodiments provided herein, if the cold data table has a specified key-value pair stored therein, the database may update the hot data table and the cold data table according to the target key-value pair.

Specifically, when the query result of the key to be processed in the hot data table is empty, the database may determine whether the specified key-value pair is stored in the cold data table according to the key corresponding to the target key-value pair and the key corresponding to each key-value pair stored in the cold data table.

If so, the database may delete the designated key-value pair from the cold data table and add the target key-value pair to the hot data table to implement the step of storing the latest processing result in the hot data table.

Likewise, for a target key-value pair stored in the hot data table, the target key-value pair may be moved into the cold data table when the target key-value pair satisfies a predetermined cooling condition.

Wherein, because each key-value pair stored in the thermal data table is a target key-value pair corresponding to each key, the database can determine whether the target key-value pair satisfies the cooling condition after updating the specified key-value pair to the target key-value pair.

Of course, in step S106 and step S108, the storage locations of the designated key value pair in the hot data table and the cold data table are determined based on the matching between the key value pair stored in the to-be-processed key and the hot data table and the key value pair stored in the cold data table, which may take too long, and in step S102, when the designated key value pair is determined, the storage location corresponding to the designated key value pair may also be determined at the same time, so in step S106 and step S108, the corresponding steps may be directly performed based on the storage location of the designated key value pair.

Further, the hot data table and the cold data table are disposed in the memory, and the computing resources and the storage resources in the memory are limited, so that if the data is processed in the memory all the time, the security of the data cannot be guaranteed, and the data execution efficiency is also affected.

Based on the above, when the data in the hot data table and the data in the cold data table reach a preset third threshold, the database can perform persistent storage on the data in the memory.

Then, the database may determine whether a sum of the data amounts in the hot data table and the cold data table is greater than a preset third threshold, and if so, the database may perform persistent storage on the cold data table. If not, the database can also continue to execute the tasks corresponding to the corresponding data processing requests based on the cold data table and the hot data table.

Of course, since the storage resources corresponding to the hot data table are limited, that is, the length of the hot data table is fixed, and the length of the cold data table is not limited, when determining whether to perform persistent storage based on the hot data table and the cold data table, it may also be determined whether the data amount of the cold data table exceeds a preset fourth threshold value only according to the data amount of the cold data table and the preset fourth threshold value, and if so, the database may perform persistent storage on the cold data table. If not, the database can also continue to execute the corresponding tasks corresponding to the data processing requests based on the cold data table and the hot data table.

Based on the data processing method shown in fig. 1, based on a hot data table and a cold data table which are deployed in advance, a key-value pair corresponding to a key to be processed carried in the data processing request is inquired from the hot data table and the cold data table to serve as an appointed key-value pair, a target key-value pair after the appointed key-value pair processing is determined, and the target key-value pair is stored in the hot data table according to storage positions of the appointed key-value pair in the hot data table and the cold data table. The cold data table and the hot data table in the method only store the latest processing results corresponding to the keys, so that when data query is carried out, only the appointed key value pair corresponding to the key to be processed needs to be queried in the hot data table and the cold data table, the condition that a large number of invalid key value pairs need to be traversed when data query is carried out is avoided, and the data query and data processing efficiency is improved.

In one or more embodiments provided in this specification, in order to avoid affecting the data storage format of the LSM tree architecture itself, the hot data table and the cold data table in this specification may be pre-deployed in a memory table.

As mentioned above, the cold data table and the hot data table store key value pairs consisting of the latest values corresponding to each key and each key in the memory table. Therefore, when the memory table is made persistent, only the hot data table and the cold data table may be stored. However, if the hot data is processed in the disk, the frequent access of the hot data causes high computing resources required for I/O, which reduces the query efficiency, so the database can store only the cold data table. That is, the above-mentioned persisting the cold data table essentially persists the memory table.

Therefore, when the data in the memory table reaches a preset first threshold value, the database can perform persistent storage on the data in the cold data table.

In addition, because the cold data table and the hot data table are arranged in the memory table, if one data is repeatedly stored in the memory table and the cold data table, the resource utilization rate is low, and therefore, only the logical relation corresponding to the key value pair can be stored in the cold data table and the hot data table.

Thus, when the data in the memory table reaches the preset first threshold, the database may determine, according to each key-value pair in the cold data table, the storage address in the external storage corresponding to each key-value pair and each key-value pair from the memory table, that is, for each key-value pair, the storage address in the external storage of the key-value pair is stored in the memory table.

The database may then persist each key-value pair in the cold data table to the external memory based on the determined memory address in the external memory. The external memory may be a non-volatile memory, such as a hard Disk, a Solid State Disk (SSD), or the like. The storage form of each key value pair in the external memory may be a Sorted String Table (SST) file form.

Meanwhile, the cold data table and the hot data table only store the key value pairs and the logical corresponding relation between the data of the key value pairs, but the actual storage positions, metadata and the like of the data corresponding to the key value pairs and the key value pairs still exist in the memory table, so that the database can move the data in the hot data table to the newly determined memory table while the data in the cold data table is persisted, and the subsequent query is facilitated.

Further, if only the logical relationship corresponding to the key-value pair is stored in the hot data table and the cold data table, the database may store the target key-value pair in the memory table after determining the target key-value pair.

Specifically, the database may process the target data based on the data processing request, and store the target key value pair obtained by the processing in the memory table.

The data processing request may be a processing request for performing operations such as adding, deleting, modifying, and searching on data in the database. If the data processing request is a processing request of an operation such as an add operation, a change operation and the like, the result of the data processing task can be directly stored, and if the data processing request is a processing request of a delete operation, the database can store the value corresponding to the data as null or as an identifier for representing that the key value pair is deleted. If the type of the data processing request is query, the database may store the key-value pair directly as a new key-value pair in the memory table.

Furthermore, in step S104, when the target key-value pair is added to the hot data table, a situation that the space in the hot data table is full may also occur, so the database may further preset a second threshold, and when the database needs to add the target key-value pair to the hot data table, it is determined whether the data amount in the hot data table reaches the preset second threshold.

If so, the structural database may determine, from the hot data table, a key-value pair satisfying a preset cooling condition according to the preset cooling condition, as a cooling key-value pair, and move the cooling key-value pair into the cold data table.

The cooling condition may be the earliest determined time, the lowest heat, etc. The degree of heat is determined based on the number of times the key values are accessed, the access time, and the like. The cooling conditions are similar to those in step S106 described above.

Furthermore, the storage space required for storing each key-value pair in the database may be different, so that when storing the target key-value pair in the hot data table, it can be determined whether the target key-value pair can be directly added to the hot data table based on the free storage space in the hot data table.

In particular, the database may determine a storage resource required to store the target key-value pair as a first resource and determine a free storage resource in the hot data table as a second resource.

The database may then compare the first resource to the second resource.

If the first resource is higher than the second resource, the database may select, according to the storage resource occupied by each key-value pair stored in the thermal data table, a preset cooling condition and the first resource, a key-value pair satisfying the cooling condition from among the key-value pairs included in the thermal data table as a cooling key-value pair.

If the first resource is not higher than the second resource, the database may directly add the target key-value pair to the hot data table.

Of course, for each key-value pair stored in the hot data table, the database may move the key-value pair into the hot data table when the key-value pair satisfies a preset unfreezing condition.

Specifically, the database may determine, for each key-value pair in the cold data table, attribute information corresponding to the key-value pair, where the attribute information includes at least one of processing time, access times corresponding to the key-value pair, and storage space required to store the key-value pair.

Then, the database can determine the heat degree corresponding to each key value pair according to the attribute information corresponding to each key value pair, and then judge whether the key value pair meeting the unfreezing condition is included according to the heat degree corresponding to each key value pair. Wherein the thawing condition may be that the heat is higher than a preset heat threshold.

If so, the database may move the key-value pairs satisfying the predetermined thawing condition into a hot data table.

Meanwhile, the database can also judge whether to move the key value pair meeting the unfreezing condition into the hot data table or not based on the heat of the key value pair meeting the unfreezing condition and the heat corresponding to each key value pair in the hot data table.

Certainly, a hot data form may also be preset in the database, where the hot data form is a key value pair that is determined according to a historical access record and is frequently accessed, and thus, for each key value pair in the cold data form, if a key corresponding to the key value pair exists in the preset hot data form, it may be considered that the key value pair meets a thawing condition and needs to be moved into the hot data form.

Since the processing time corresponding to the target key value pair corresponding to the key to be processed is the latest, the target key value pair satisfies a preset thawing condition when the processing time is set as the thawing condition.

The thawing conditions at least comprise: at least one of the latest processing time, the minimum occupied storage space and the maximum access times.

Based on the same idea, the present specification further provides a flow diagram of a data processing method, as shown in fig. 2.

Fig. 2 is a schematic flow chart of the data processing method provided in this specification, where when the data amount in the memory table reaches a preset first threshold, the database may use the original memory table as a read-only memory table, determine a new memory table based on each key value pair in the hot data table, and then persistently store data in the cold data table in the read-only memory table, so as to discard the history record corresponding to each key value pair, avoid storing a large number of history records corresponding to each key value pair in the database, and improve query efficiency.

And the original memory table is used as a read-only memory table, and a new memory table is determined based on each key value pair in the thermal data table, so that data processing can be performed based on the newly generated memory table while the original memory table is subjected to persistence, and the efficiency of service execution is improved.

Based on the same idea, the present specification also provides a data processing apparatus, as shown in fig. 3.

Fig. 3 is a schematic structural diagram of a data processing apparatus provided in this specification, which is applied to a database, where a memory table is pre-deployed in the database, and the memory table includes a hot data table and a cold data table, where:

a receiving module 200, configured to receive a data processing request, where the data processing request carries a key to be processed.

A determining module 202, configured to query, according to the key to be processed, a key-value pair corresponding to the key to be processed from the hot data table and the cold data table, as an assigned key-value pair.

And the processing module 204 is configured to process the specified key-value pair according to the data processing request to obtain a target key-value pair.

A first storage module 206, configured to update the specified key-value pair in the hot data table according to the target key-value pair if the specified key-value pair is stored in the hot data table, and move the target key-value pair from the hot data table to the cold data table when the target key-value pair meets a preset cooling condition.

A second storage module 208, configured to delete the designated key-value pair in the cold data table and add the target key-value pair to the hot data table if the designated key-value pair is stored in the cold data table, and move the target key-value pair from the hot data table to the cold data table when the target key-value pair meets a preset cooling condition.

Optionally, the hot data table and the cold data table are disposed in a memory table, and the second storage module 208 is configured to, when the data amount in the memory table reaches a preset first threshold, perform persistent storage on each key value pair in the cold data table, generate a new memory table according to each key value pair in the hot data table, and perform a data processing request according to the new memory table.

Optionally, the determining module 202 is configured to determine a key to be processed carried in the data processing request, query, according to the key to be processed and the key corresponding to each key-value pair stored in the hot data table, an assigned key-value pair corresponding to the key to be processed from the hot data table, and query, if the query result is empty, the assigned key-value pair corresponding to the key to be processed from the cold data table according to the key corresponding to each key-value pair stored in the key to be processed and the key corresponding to each key-value pair stored in the cold data table.

Optionally, the second storage module 208 is configured to determine whether the data amount in the hot data table reaches a preset second threshold, and if yes, determine, according to a preset cooling condition, a key-value pair that satisfies the cooling condition in the hot data table, as a cooling key-value pair, and move the cooling key-value pair into the cold data table.

Optionally, the storage resources occupied by storing each key value pair are not completely the same; the second storage module 208 is configured to determine a storage resource required to store the target key value pair as a first resource, determine a free storage resource in the hot data table as a second resource, and if the first resource is higher than the second resource, determine, according to the storage resource occupied by each key value pair stored in the hot data table, a preset cooling condition and the first resource, a key value pair satisfying the cooling condition from among the key value pairs included in the hot data table, as a cooling key value pair, and move the cooling key value pair into the cold data table.

Optionally, the second storing module 208 is configured to, for each key-value pair stored in the cold data table, move the key-value pair into the hot data table when the key-value pair meets a preset thawing condition.

Optionally, the hot data table and the cold data table store logical relationships of the key values, the memory table stores the key values and storage addresses of the key values in the external storage, and the second storage module 208 is configured to determine, according to the key values in the cold data table, storage addresses in the external storage corresponding to the key values and the key values in the memory table, and perform persistent storage on the key values according to the determined storage addresses in the external storage.

The present specification also provides a computer-readable storage medium storing a computer program operable to execute the data processing method provided in fig. 1 above.

This specification also provides a schematic block diagram of the electronic device shown in fig. 4. As shown in fig. 4, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs the computer program to implement the data processing method described in fig. 1. Of course, besides the software implementation, this specification does not exclude other implementations, such as logic devices or combination of software and hardware, and so on, that is, the execution subject of the following processing flow is not limited to each logic unit, and may be hardware or logic devices.

In the 90 s of the 20 th century, improvements in a technology could clearly distinguish between improvements in hardware (e.g., improvements in circuit structures such as diodes, transistors, switches, etc.) and improvements in software (improvements in process flow). However, as technology advances, many of today's process flow improvements have been seen as direct improvements in hardware circuit architecture. Designers almost always obtain the corresponding hardware circuit structure by programming an improved method flow into the hardware circuit. Thus, it cannot be said that an improvement in the process flow cannot be realized by hardware physical modules. For example, a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA), is an integrated circuit whose Logic functions are determined by programming the Device by a user. A digital system is "integrated" on a PLD by the designer's own programming without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing an Integrated Circuit chip, such Programming is often implemented by "logic compiler" software, which is similar to a software compiler used in program development, but the original code before compiling is also written in a specific Programming Language, which is called Hardware Description Language (HDL), and the HDL is not only one kind but many kinds, such as abll (Advanced boot Expression Language), AHDL (alternate hard Description Language), traffic, CUPL (computer universal Programming Language), HDCal (Java hard Description Language), lava, lola, HDL, PALASM, software, rhydl (Hardware Description Language), and vhul-Language (vhyg-Language), which is currently used in the field. It will also be apparent to those skilled in the art that hardware circuitry that implements the logical method flows can be readily obtained by merely slightly programming the method flows into an integrated circuit using the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include, but are not limited to, the following microcontrollers: ARC625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic for the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may thus be considered a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.

The systems, apparatuses, modules or units described in the above embodiments may be specifically implemented by a computer chip or an entity, or implemented by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functions of the various elements may be implemented in the same one or more software and/or hardware implementations of the present description.

As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The description has been presented with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the description. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of other like elements in a process, method, article, or apparatus comprising the element.

As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.

This description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, key value pairs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The above description is only an example of the present specification, and is not intended to limit the present specification. Various modifications and alterations to this description will become apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification should be included in the scope of the claims of the present specification.

Claims

1. A data processing method is applied to a database, wherein a hot data table and a cold data table are deployed in advance in the database, and the method comprises the following steps:

2. The method of claim 1, the hot data table and the cold data table being disposed in a memory table;

the method further comprises the following steps:

when the data amount in the memory table reaches a preset first threshold value, performing persistent storage on each key value pair in the cold data table, generating a new memory table according to each key value pair in the hot data table, and executing a data processing request according to the new memory table.

3. The method according to claim 1, wherein the step of querying the key-value pair corresponding to the key to be processed from the hot data table and the cold data table as the designated key-value pair specifically comprises:

determining a key to be processed carried in the data processing request;

according to the key corresponding to each key value pair stored in the key to be processed and the hot data table, inquiring the appointed key value pair corresponding to the key to be processed from the hot data table;

and if the query result is null, querying the appointed key value pair corresponding to the key to be processed from the cold data table according to the key corresponding to the key value pair stored in the key to be processed and the cold data table.

4. The method of claim 1, prior to adding the target key-value pair to the hot data table, the method further comprising:

judging whether the data volume in the thermal data table reaches a preset second threshold value or not;

if yes, determining the key value pair which meets the cooling condition in the hot data table according to a preset cooling condition to serve as the cooling key value pair, and moving the cooling key value pair into the cold data table.

5. The method of claim 1, wherein storing each key-value pair occupies a non-identical storage resource;

before adding the target key-value pair to the hot data table, the method further comprises:

determining a storage resource required for storing the target key-value pair as a first resource, and determining a free storage resource in the hot data table as a second resource;

if the first resource is higher than the second resource, determining key value pairs meeting the cooling conditions from the key value pairs contained in the thermal data table as cooling key value pairs according to the storage resources respectively occupied by the key value pairs stored in the thermal data table, preset cooling conditions and the first resource;

moving the cooling key-value pair into the cold data table.

6. The method of claim 1, further comprising:

and for each key-value pair stored in the cold data table, when the key-value pair meets a preset unfreezing condition, moving the key-value pair into the hot data table.

7. The method of claim 2, the hot data table and the cold data table storing logical relationships of key-value pairs, the memory table storing key-value pairs and storage addresses of the key-value pairs in an external memory;

according to each key value pair in the cold data table and the memory table, performing persistent storage on each key value pair in the cold data table, specifically comprising:

determining the key value pairs and storage addresses of the key values in an external storage respectively corresponding to the key values from the memory table according to the key value pairs in the cold data table;

and according to the determined storage address in the external memory, performing persistent storage on each key value pair.

8. A data processing device is applied to a database, wherein a hot data table and a cold data table are deployed in advance in the database, and the device comprises:

the device comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving a data processing request which carries a key to be processed;

the determining module is used for inquiring a key value pair corresponding to the key to be processed from the hot data table and the cold data table according to the key to be processed to serve as an appointed key value pair;

9. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method of any one of the preceding claims 1 to 7.

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any of the preceding claims 1 to 7 when executing the program.