CN117992547A - Data processing method and device, electronic device and storage medium - Google Patents
Data processing method and device, electronic device and storage medium Download PDFInfo
- Publication number
- CN117992547A CN117992547A CN202410199887.5A CN202410199887A CN117992547A CN 117992547 A CN117992547 A CN 117992547A CN 202410199887 A CN202410199887 A CN 202410199887A CN 117992547 A CN117992547 A CN 117992547A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- heat
- sequence
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/957—Browsing optimisation, e.g. caching or content distillation
- G06F16/9574—Browsing optimisation, e.g. caching or content distillation of access to content, e.g. by caching
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
One or more embodiments of the present disclosure provide a data processing method and apparatus, an electronic device, and a storage medium, where the method includes: correspondingly determining heat values of a plurality of data in each node in the data access process of each node in a distributed data system, wherein the heat values are used for representing the access probability of the data; and loading partial data in the data heat sequence of each node in the distributed data system to a cache space corresponding to the node, wherein the data heat sequence is a sequence formed by a plurality of data in the node from high to low according to heat values. The method can load the data with higher access probability into the cache space in the data access process, thereby improving the hit rate of the data in each node when being accessed, namely improving the hit rate of the data in the distributed data system when being accessed, and improving the service quality of the distributed data system for external supply.
Description
Technical Field
One or more embodiments of the present disclosure relate to the field of database technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.
Background
Today, the development of the internet and informatization is rapid, and the generation of data is explosively increasing, so that the requirements for databases and management thereof are increasing. Data in a database needs to be loaded into a cache space (i.e., a memory) to be accessed, and the cache space of the database is often smaller or even far smaller than the persistence space of the database, i.e., only part of the data in the database can be simultaneously loaded into the cache space. If some data is accessed in the cache space just as it is, then it is accessed directly without a load, which may be referred to as a hit.
In the related art, the distributed data system often performs processes such as capacity expansion, capacity shrinkage, and data equalization, which may cause data transfer between different nodes of the distributed data system, that is, the data in each node may change frequently. The hit rate when data within the distributed data system is accessed is low, resulting in poor quality of service provided outside the distributed data system.
Disclosure of Invention
In view of this, one or more embodiments of the present disclosure provide a data processing method and apparatus, an electronic device, and a storage medium.
In order to achieve the above object, one or more embodiments of the present disclosure provide the following technical solutions:
according to a first aspect of one or more embodiments of the present specification, there is provided a data processing method, the method comprising:
Correspondingly determining heat values of a plurality of data in each node in the data access process of each node in a distributed data system, wherein the heat values are used for representing the access probability of the data;
And loading partial data in the data heat sequence of each node in the distributed data system to a cache space corresponding to the node, wherein the data heat sequence is a sequence formed by a plurality of data in the node from high to low according to heat values.
In one embodiment of the present disclosure, during the data access process of each node in the distributed data system, the determining the heat value of the plurality of data in each node correspondingly includes:
and determining the heat value of the data according to the latest access time and/or access frequency of the data in the node.
In one embodiment of the present disclosure, the determining the heat value of the data according to the latest access time and access frequency of the data in the node includes:
And carrying out weighted summation on the latest access time and the access frequency of the data in the node to obtain the heat value of the data.
In one embodiment of the present specification, the method further comprises:
Determining that the weight of the latest access time of the data is a first weight and the weight of the access frequency of the data is a second weight in response to the occupied space of the data being smaller than a space threshold, wherein the first weight is greater than the second weight;
And determining that the weight of the latest access time of the data is a third weight and the weight of the access frequency of the data is a fourth weight in response to the occupied space of the data is not smaller than a space threshold, wherein the third weight is smaller than the fourth weight.
In one embodiment of the present disclosure, during the data access process of each node in the distributed data system, the determining the heat value of the plurality of data in each node correspondingly includes:
And correspondingly determining the heat value of N data in each node in the data access process of each node in the distributed data system, wherein the occupied space of the N data is larger than the corresponding cache space of the node, and N is an integer larger than or equal to 1.
In one embodiment of the present disclosure, during the data access process of each node in the distributed data system, the determining the heat value of the plurality of data in each node correspondingly includes:
In response to data of any node within the distributed data system being accessed, a heat value of a plurality of data within the node is determined.
In one embodiment of the present disclosure, the loading the first part of data in the data heat sequence of each node in the distributed data system into the cache space corresponding to the node includes:
and loading the M data at the forefront in the data heat sequence of the node into a buffer space corresponding to the node, wherein the occupied space of the M data is smaller than or equal to the effective space of the buffer space, and M is an integer larger than or equal to 1.
In one embodiment of the present specification, the method further comprises at least one of:
responding to the transfer or deletion of target data of any node in the distributed data system, and updating the data popularity sequence of the node according to the popularity value of the target data;
and responding to the writing of target data into any node in the distributed data system, and updating the data heat sequence of the node according to the heat value of the target data.
In one embodiment of the present specification, the method further comprises:
Responding to the change of the M data which are the forefront in the data heat sequence of the node in updating, and adjusting the data in the buffer space corresponding to the node according to the updated data heat sequence.
In one embodiment of the present specification, the method further comprises:
And storing the data heat sequence of each node in the distributed data system into a disk space corresponding to the node.
According to a second aspect of one or more embodiments of the present specification, there is provided a data processing apparatus, the apparatus comprising:
The system comprises a heat determining module, a heat determining module and a data processing module, wherein the heat determining module is used for determining heat values of a plurality of data in each node in the data access process of each node in a distributed data system, and the heat values are used for representing the access probability of the data;
And the data caching module is used for loading partial data in the front in the data heat sequence of each node in the distributed data system into a cache space corresponding to the node, wherein the data heat sequence is a sequence formed by a plurality of data in the node from high to low according to heat values.
In one embodiment of the present specification, the heat determining module is configured to:
and determining the heat value of the data according to the latest access time and/or access frequency of the data in the node.
In one embodiment of the present disclosure, the heat determining module is configured to, when determining a heat value of the data according to a latest access time and an access frequency of the data in the node,:
And carrying out weighted summation on the latest access time and the access frequency of the data in the node to obtain the heat value of the data.
In one embodiment of the present specification, the apparatus further includes a weight module for:
Determining that the weight of the latest access time of the data is a first weight and the weight of the access frequency of the data is a second weight in response to the occupied space of the data being smaller than a space threshold, wherein the first weight is greater than the second weight;
And determining that the weight of the latest access time of the data is a third weight and the weight of the access frequency of the data is a fourth weight in response to the occupied space of the data is not smaller than a space threshold, wherein the third weight is smaller than the fourth weight.
In one embodiment of the present specification, the heat determining module is configured to:
And correspondingly determining the heat value of N data in each node in the data access process of each node in the distributed data system, wherein the occupied space of the N data is larger than the corresponding cache space of the node, and N is an integer larger than or equal to 1.
In one embodiment of the present specification, the heat determining module is configured to:
In response to data of any node within the distributed data system being accessed, a heat value of a plurality of data within the node is determined.
In one embodiment of the present specification, the data caching module is configured to:
and loading the M data at the forefront in the data heat sequence of the node into a buffer space corresponding to the node, wherein the occupied space of the M data is smaller than or equal to the effective space of the buffer space, and M is an integer larger than or equal to 1.
In one embodiment of the present specification, the apparatus further comprises an update module for at least one of:
responding to the transfer or deletion of target data of any node in the distributed data system, and updating the data popularity sequence of the node according to the popularity value of the target data;
and responding to the writing of target data into any node in the distributed data system, and updating the data heat sequence of the node according to the heat value of the target data.
In one embodiment of the present specification, the apparatus further comprises an adjustment module for:
Responding to the change of the M data which are the forefront in the data heat sequence of the node in updating, and adjusting the data in the buffer space corresponding to the node according to the updated data heat sequence.
In one embodiment of the present specification, the apparatus further comprises a persistence module for:
And storing the data heat sequence of each node in the distributed data system into a disk space corresponding to the node.
According to a fourth aspect of one or more embodiments of the present specification, there is provided an electronic device comprising:
A processor;
a memory for storing processor-executable instructions;
Wherein the processor implements the method of the first aspect by executing the executable instructions.
According to a fifth aspect of one or more embodiments of the present description, there is provided a computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method according to the first aspect.
The technical scheme provided by the embodiment of the specification can comprise the following beneficial effects:
According to the data processing method provided by the embodiment of the specification, in the data access process of each node in the distributed data system, the heat value of a plurality of data in each node can be determined, and partial data in the data heat sequence of each node in the distributed data system, which is the front part of the data, is loaded into the cache space corresponding to the node. Because the heat value of the data characterizes the access probability of the data, the method can load the data with higher access probability into the cache space in the data access process, thereby improving the hit rate of the data in each node when being accessed, namely improving the hit rate of the data in the distributed data system when being accessed, and improving the service quality of the distributed data system to the outside.
Drawings
Fig. 1 is a flow chart of a data processing method according to an exemplary embodiment.
FIG. 2 is a schematic diagram of an update process of a data heat sequence provided by an exemplary embodiment.
Fig. 3 is a schematic diagram of a capacity expansion process of a distributed data system according to an exemplary embodiment.
Fig. 4 is a schematic diagram of a capacity reduction process of a distributed data system according to an exemplary embodiment.
Fig. 5 is a schematic diagram of an apparatus according to an exemplary embodiment.
Fig. 6 is a block diagram of a data processing apparatus provided in yet another exemplary embodiment.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of the present specification. Rather, they are merely examples of apparatus and methods consistent with aspects of one or more embodiments of the present description as detailed in the accompanying claims.
It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than described in this specification. Furthermore, individual steps described in this specification, in other embodiments, may be described as being split into multiple steps; while various steps described in this specification may be combined into a single step in other embodiments.
Today, the development of the internet and informatization is rapid, and the generation of data is explosively increasing, so that the requirements for databases and management thereof are increasing. Data in a database needs to be loaded into a cache space (i.e., a memory) to be accessed, and the cache space of the database is often smaller or even far smaller than the persistence space of the database, i.e., only part of the data in the database can be simultaneously loaded into the cache space. If some data is accessed in the cache space just as it is, then it is accessed directly without a load, which may be referred to as a hit.
In the related art, the distributed data system often performs processes such as capacity expansion, capacity shrinkage, and data equalization, which may cause data transfer between different nodes of the distributed data system, that is, the data in each node may change frequently. The hit rate when data within the distributed data system is accessed is low, resulting in poor quality of service provided outside the distributed data system.
In view of this, in a first aspect, at least one embodiment of the present disclosure provides a data processing method, where data with a higher access probability in each node in a distributed data system may be selected in real time and loaded into a cache space, so as to improve a hit rate when data in the distributed data system is accessed, and especially, when data transfer such as capacity expansion, capacity shrinkage, data equalization and the like occurs in the distributed data system (that is, data is transferred between different nodes), the hit rate when the data in the distributed data system is accessed can still be ensured.
The method may be applied, for example, to a database management system, i.e. a database management system of a distributed data system.
First, a brief description of a distributed data system will be presented. The distributed data system includes a plurality of nodes, each of which may be a separate storage unit, which may be a physical server, or a virtual server, etc. Each node in the distributed data system is configured with a corresponding buffer space, and the buffer space may be a part of the storage space of the node itself, or a part of the storage space of a specific storage unit in the distributed data system (i.e. the specific storage unit provides an independent buffer space for each node). If the cache space is a part of the storage space of the node, the storage space except the cache space in the node is a persistence space; if the cache space is a part of the storage space of a specific storage unit in the distributed data system, all the storage spaces in the nodes are persistent spaces; wherein, the difference between the persistence space and the cache space is: the data stored in the persistent space cannot disappear due to the factors of downtime, power failure and the like, and the data stored in the cache space can disappear due to the factors of downtime, power failure and the like. If a particular storage unit within the distributed data system provides a cache space for each node, the database management system may also be installed on that particular storage unit.
Distributed data systems often perform operations such as capacity expansion, capacity contraction, load balancing, and replacement of failed nodes. When the capacity is reduced, the node number of the distributed data system is reduced, and the data of the reduced nodes are transferred to one or more other target nodes. During capacity expansion, data of one or more source nodes are migrated to one or more target nodes. The system needs to migrate data among nodes when performing operations such as load balancing or newly added data copy.
Referring to fig. 1, a flow of the data processing method is shown in an exemplary manner, and includes steps S101 to S102.
In step S101, during data access of each node in the distributed data system, a heat value of a plurality of data in each node is determined, where the heat value is used to characterize an access probability of the data.
The data access of the nodes in the distributed data system can comprise data reading, data writing, data transferring, data deleting and the like. For example, the database management system responds to the instruction of the upper layer application to read the data of a certain node in the distributed data system, wherein the data belongs to data reading; for another example, the database management system responds to the instruction of the upper layer application to write data to a certain node in the distributed data system, wherein the data belongs to data writing; for another example, the database management system responds to the instruction of the upper layer application to transfer the data of one node in the distributed data system to other nodes belonging to the data transfer; for another example, the database management system responds to instructions of an upper layer application to delete data of a node in the distributed data system belonging to the data deletion. It should be understood that data access also includes other forms, which are merely examples and do not constitute a formal limitation on data access.
Wherein the heat value of the data may be characterized by a data last access time, a data access frequency, etc. I.e. the step may determine the hotness value of the data in the node based on the latest access time and/or access frequency of the data.
For example, if the heat value of the data is determined according to the latest access time of the data in the node, the heat value is higher as the latest access time is later (i.e., the time interval from the current time is smaller), and the heat value is lower as the latest access time is earlier (i.e., the time interval from the current time is larger).
Further exemplary, if the popularity value of the data is determined according to the access frequency of the data in the node, the popularity value is higher as the access frequency is higher, and the popularity value is lower as the access frequency is lower. It should be appreciated that the frequency of access to the data within the node may be determined by the number of accesses within a preset time period prior to the current time.
Also exemplary, if the heat value of the data is determined based on the most recent access time and access frequency of the data within the node. The latest access time and access frequency of the data in the node can be weighted and summed to obtain the heat value of the data. Before the latest access time and the access frequency are weighted and summed, the latest access time and the access frequency can be unified to a certain value representing the data access heat, and then the weighted and summed are carried out according to the weights of the latest access time and the access frequency, and the sum of the weights of the latest access time and the access frequency can be 1.
For example, the reciprocal of a proportional value (the proportional value can be rounded) of the interval between the latest access time and the current time and the preset duration is taken as a value for representing the data access heat; and taking the proportional value of the access frequency and the preset times as a value representing the data access heat.
For example, the weight of the latest access time and the weight of the access frequency may be set in advance. Or the weight of the latest access time and the weight of the access frequency corresponding to each data can be respectively determined according to the information such as the occupied space (namely commonly called data memory, which can be represented by the bit number of the data) of each data. For example: determining that the weight of the latest access time of the data is a first weight and the weight of the access frequency of the data is a second weight in response to the occupied space of the data being smaller than a space threshold, wherein the first weight is greater than the second weight; and determining that the weight of the latest access time of the data is a third weight and the weight of the access frequency of the data is a fourth weight in response to the occupied space of the data is not smaller than a space threshold, wherein the third weight is smaller than the fourth weight. Namely, when the occupied space of the data is smaller, the weight of the latest access time is larger than the weight of the access frequency; when the occupied space of the data is large, the weight of the latest access time is smaller than the weight of the access frequency.
The relative concept has guiding significance for selecting partial data to be loaded into the cache space. Alternatively, after determining the heat value of the data in the node, a data heat sequence (or referred to as a heat map) of the node may be determined, that is, a sequence in which a plurality of data in the node are formed from high to low according to the heat value.
The data heat sequence may comprise a sequence of all data within a node from high to low according to heat values. In this case, this step may correspondingly determine the heat value of each data in each node during the data access process of each node in the distributed data system.
Or the data heat sequence may comprise a sequence of N data (i.e., partial data) within a node from high to low according to heat values. In this case, the present step may correspondingly determine the heat value of N data in each node during the data access process of each node in the distributed data system. It should be understood that the occupied space of the N data may be greater than the buffer space corresponding to the node, where N is an integer greater than or equal to 1; that is, the data volume in the data heat sequence determined and maintained for each node is larger than, even far larger than, the data volume which can be cached in the corresponding cache space of the node, so that when part or all of the data in the cache space is deleted or transferred, part of the data with higher access probability can be quickly selected from the rest data in the node and loaded into the cache space, and the hit rate of the data of the node is improved.
It should be appreciated that over time, both the most recent access time and access frequency of the data within the node will change, but the relative relationship of the heat values between the data will not change if the data within the node is not accessed. As described above, the relative relationship between the heat values of the data in the nodes has guiding significance for selecting part of the data to be loaded into the cache space, so that the heat values of a plurality of data in the nodes can be determined in response to the data of any node in the distributed data system being accessed. That is, when data in a certain node is accessed, the relative relationship between the heat values of the data in the node is changed, so that the heat value of the data in the node is redetermined to update the relative relationship between the heat values of the data, such as a data heat sequence.
It should be appreciated that the heat value of the data in each node in the distributed data system and the data heat sequence of the node may be saved to the cache space of the node for reading. Of course, the data heat sequence of each node in the distributed data system can be stored (i.e. persisted) into the disk space (i.e. persisted space) corresponding to the node, so as to avoid the heat value and the disappearance of the data heat sequence in the cache space due to downtime, power failure and the like, and thus, after the node is restored to normal, data can be directly loaded into the cache space corresponding to each node according to the data heat sequence in the disk space, so that the hit rate of data access of each node after downtime and power failure is improved. For example, the data hotness value and the data hotness sequence may be updated in real-time within the cache space; and updated in disk space at preset intervals.
Referring to fig. 2, a schematic diagram of adjusting a data heat sequence of a node according to step S101 is shown in an image. As can be seen from the figure, the data in the primary-copy node can be processed according to the traffic access flow, for example, the data in the cache space is dropped to the local storage, or the data in the local storage is loaded into the cache space; and the data heat sequence is updated according to the service access flow; and finally, synchronizing the updating result of the data heat sequence in the master copy node to each slave copy node.
In step S102, the data of the first part in the data heat sequence of each node in the distributed data system is loaded into the buffer space corresponding to the node, where the data heat sequence is a sequence formed by a plurality of data in the node from high to low according to the heat value.
For example, for each node in the distributed data system, loading the first M data in the data heat sequence of the node into a buffer space corresponding to the node, where the occupied space of the M data is smaller than or equal to the effective space of the buffer space, and M is an integer greater than or equal to 1. It should be understood that the effective space of the buffer space may be all or part of the buffer space, for example, the occupied ratio preset for the buffer space.
Since the first part of data in the data popularity sequence of the node is loaded into the buffer space corresponding to the node in this step, the data loaded into the buffer space corresponding to the node can be synchronously updated when the data popularity sequence of the node is updated (for example, updated in step S101 or updated for other reasons). In other words, when the data in a certain node is accessed, the relative relationship of the heat value among the data in the node is changed, so that the heat value of the data in the node is redetermined to update the relative relationship of the heat value among the data, such as a data heat sequence; and then updating the data loaded into the buffer space corresponding to the node according to the updated data heat sequence, namely loading the part of data in the front part of the updated data heat sequence of the node into the buffer space corresponding to the node.
For example, the a data of the a node is relatively forward in the data heat sequence, so that the a data is loaded into the cache space corresponding to the a node; the a data of the a node is deleted by the database management system in response to the instruction of the upper layer application, so that one data with the highest heat value in the data which is not loaded into the cache space by the a node can be supplemented into the cache space.
According to the data processing method provided by the embodiment of the specification, in the data access process of each node in the distributed data system, the heat value of a plurality of data in each node can be determined, and partial data in the data heat sequence of each node in the distributed data system, which is the front part of the data, is loaded into the cache space corresponding to the node. Because the heat value of the data characterizes the access probability of the data, the method can load the data with higher access probability into the cache space in the data access process, thereby improving the hit rate of the data in each node when being accessed, namely improving the hit rate of the data in the distributed data system when being accessed, and improving the service quality of the distributed data system to the outside.
As described above, in the data access process of the node in the distributed data system, the embodiment of the present disclosure can update the heat value of the data in the node, the data heat sequence of the node, and the data in the cache space corresponding to the node. The operations of capacity expansion, capacity shrinkage, data equalization, data copy addition and the like of the distributed data system all involve data access of at least one node, so that the data in the cache space corresponding to the involved node can be adjusted when the operations of capacity expansion, capacity shrinkage, data equalization, data copy addition and the like of the distributed data system occur, and the higher hit rate of the involved node data when accessed is ensured.
When the distributed data system is subjected to operations such as capacity expansion, capacity shrinkage, data equalization and the like, data transfer, deletion, writing and the like are mainly involved. The heat value of the data determined by the method can be used as an attribute of the data to be transferred, deleted, written and the like along with the data. The data heat sequence of the node may be maintained during the above operations in accordance with the following alternative examples.
Alternative example 1: and in response to the transfer or deletion of target data of any node in the distributed data system, updating the data popularity sequence of the node according to the popularity value of the target data. Optionally, deleting the target data in the data heat sequence of the node.
For example, when the distributed data system expands, new nodes are added on the basis of original nodes, and part or all of original nodes transfer part of data to the new nodes, so that the data in the original nodes are transferred. For another example, when the distributed data system is scaled, some nodes in all nodes are reserved, and data in other nodes are transferred to reserved nodes, so that data in unreserved nodes can be transferred.
Further, in response to the change of the first M data in the data heat sequence of the node in updating, data in the buffer space corresponding to the node is adjusted according to the updated data heat sequence, where M is an upper limit of data amount loaded into the buffer space corresponding to the node. For example, if the data with the highest heat value in the data heat sequence of the node is deleted in the update, the data with the highest heat value in the data which is not loaded in the cache space by the node may be supplemented into the cache space.
Alternative example 2: and responding to the writing of target data into any node in the distributed data system, and updating the data heat sequence of the node according to the heat value of the target data. Optionally, the target data is added to the data popularity sequence of the node according to the popularity value of the target node.
For example, when the distributed data system expands, new nodes are added on the basis of original nodes, and part or all of original nodes transfer part of data to the new nodes, so that data in the new nodes are written. For another example, when the distributed data system is scaled, some nodes in all nodes are reserved, and data in other nodes are transferred to the reserved nodes, so that the reserved nodes have data written.
Further, in response to the change of the first M data in the data heat sequence of the node in updating, data in the buffer space corresponding to the node is adjusted according to the updated data heat sequence, where M is an upper limit of data amount loaded into the buffer space corresponding to the node. For example, if the target data is added to the M-th position of the data heat sequence of the node, the data with the lowest heat value in the data in the cache space corresponding to the node may be deleted, and the target data may be loaded into the cache space.
The effect of performing the method when a capacity expansion of the distributed data system occurs will be described in detail with reference to fig. 3.
Fig. 3 shows a heat map (i.e., a data heat sequence) of each node before expansion and a heat map (i.e., a data heat sequence) of each node after expansion, wherein the heat map of the node before expansion comprises cached data and uncached data, and the heat map of the node after expansion comprises cached data, new cached data and uncached data.
In the capacity expansion process in fig. 3, a node C is newly added on the basis of the original node a and the original node B, so that the node a and the node B both transfer part of data to the node C, wherein the data reserved by the node a is called a data set 1, the data transferred to the node C is called a data set 2, the data reserved by the node B is called a data set 3, and the data transferred to the node C is called a data set 4.
As can be seen from fig. 3, since the data set 2 in the node a is transferred to the node C, the cached data in the data set 2 is also transferred, and the space occupied by the cached data in the data set 2 in the node a is released, and since the method determines the heat value of each data in the data set 1 in real time, the partial data with the front heat value in the uncached data before the expansion of the data set 1 can be loaded into the cache space of the node a, so that the partial data with the higher heat value in the rest data can be loaded into the cache space immediately after the transfer of the data set 2, thereby improving the utilization rate of the cache space and the hit rate of data access.
As can be seen from fig. 3, since the data set 4 in the node B is transferred to the node C, the cached data in the data set 4 is also transferred, and the space occupied by the cached data in the data set 2 in the node B is released, and since the method determines the heat value of each data in the data set 3 in real time, the partial data with the front heat value in the uncached data before the expansion of the data set 3 can be loaded into the cache space of the node B, so that the partial data with the higher heat value in the rest data can be loaded into the cache space immediately after the transfer of the data set 4, thereby improving the utilization rate of the cache space and the hit rate of data access.
As can be seen from fig. 3, when the data set 2 in the node a and the data set 4 in the node B are transferred to the node C, the heat map of the data set 2 (i.e. the heat values of the data in the data set 2 are in a sequence consisting of high to low), and the heat map of the data set 4 (i.e. the heat values of the data in the data set 4 are in a sequence consisting of high to low), are also transferred to the node C, so that part of the data originally loaded in the data set 2 into the cache space of the data set a and part of the data originally loaded in the data set 4 into the cache space of the data set B can be directly loaded into the cache space of the node C; if the cache space still has free space, partial data with the front heat value can be further loaded into the cache space of the node C according to the heat maps of the data sets 2 and 4 so as to fully utilize the cache space. Therefore, in the transfer process of the data sets 2 and 4, the heat maps are synchronously transferred, so that data with higher heat values can be loaded immediately after the data sets 2 and 4 are transferred, and the utilization rate of the cache space and the hit rate of data access are improved.
The effect of performing the method when a distributed data system is scaled is described in detail below with reference to fig. 4.
Fig. 4 shows a heat map of each node before the capacity reduction (i.e., a data heat sequence) and a heat map of each node after the capacity reduction (i.e., a data heat sequence), wherein the heat map of the node before the capacity reduction comprises old cache data and uncached data, and the heat map of the node after the capacity reduction comprises old cache data, new obsolete cache data (i.e., data transferred from a cache space to a disk in the old cache data) and uncached data.
In the capacity reduction process in fig. 4, the original nodes D and E are reduced to node D, so that node E transfers all data to node D, where the data in node D before capacity reduction is referred to as data set 5, and the data in node E before capacity reduction is referred to as data set 6. It should be appreciated that the pre-scaling dataset 5, 6 occupies the cache space of node D, E, whereas the post-scaling dataset 5, 6 occupies only the cache space of node D, i.e. the cache space is also synchronously scaled down during the scaling process.
As can be seen from fig. 4, the method determines the heat value of each data in the data sets 5 and 6 in real time, and the heat value of each data in the data set 6 is also transferred to the node D at any time in the process of transferring the data set 6 to the node D, so that all the data in the data sets 5 and 6 can be reordered according to the heat value, and part of the data with higher heat value in the ordering result is loaded into the cache space of the node D, then part of the old cache data in the data set 5 is eliminated to form new elimination cache data, and part of the old cache data in the data set 6 is eliminated to form new elimination cache data, so that part of the data with higher heat value in all the data (i.e. the data sets 5 and 6) can be immediately reserved in the cache space after the node D, E is contracted, thereby improving the utilization rate of the cache space and the hit rate of data access.
Therefore, in the transferring process of the data set 6, the heat map is synchronously transferred, so that data with higher heat value can be loaded immediately after the data sets 5 and 6 are concentrated to the node D, and the utilization rate of the cache space and the hit rate of data access are improved. It is supposed that if the data set 6 is merged into the node D, the hit rate when the data in the data set 6 is accessed is 0 if the data in the cache space is not adjusted according to the method.
Fig. 5 is a schematic block diagram of an apparatus according to an exemplary embodiment. Referring to fig. 5, at the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile storage 510, although other tasks may be performed by the device. One or more embodiments of the present description may be implemented in a software-based manner, such as by the processor 502 reading a corresponding computer program from the non-volatile storage 510 into the memory 508 and then running. Of course, in addition to software implementation, one or more embodiments of the present disclosure do not exclude other implementation manners, such as a logic device or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or a logic device.
Referring to fig. 6, the data processing apparatus may be applied to the device shown in fig. 5 to implement the technical solution of the present specification. The device comprises:
A heat determining module 601, configured to determine a heat value of a plurality of data in each node during a data access process of each node in the distributed data system, where the heat value is used to characterize an access probability of the data;
And the data caching module 602 is configured to load a part of data in the data heat sequence of each node in the distributed data system, where the data heat sequence is a sequence formed by a plurality of data in the node from high to low according to a heat value, into a cache space corresponding to the node.
In one embodiment of the present specification, the heat determining module is configured to:
and determining the heat value of the data according to the latest access time and/or access frequency of the data in the node.
In one embodiment of the present disclosure, the heat determining module is configured to, when determining a heat value of the data according to a latest access time and an access frequency of the data in the node,:
And carrying out weighted summation on the latest access time and the access frequency of the data in the node to obtain the heat value of the data.
In one embodiment of the present specification, the apparatus further includes a weight module for:
Determining that the weight of the latest access time of the data is a first weight and the weight of the access frequency of the data is a second weight in response to the occupied space of the data being smaller than a space threshold, wherein the first weight is greater than the second weight;
And determining that the weight of the latest access time of the data is a third weight and the weight of the access frequency of the data is a fourth weight in response to the occupied space of the data is not smaller than a space threshold, wherein the third weight is smaller than the fourth weight.
In one embodiment of the present specification, the heat determining module is configured to:
And correspondingly determining the heat value of N data in each node in the data access process of each node in the distributed data system, wherein the occupied space of the N data is larger than the corresponding cache space of the node, and N is an integer larger than or equal to 1.
In one embodiment of the present specification, the heat determining module is configured to:
In response to data of any node within the distributed data system being accessed, a heat value of a plurality of data within the node is determined.
In one embodiment of the present specification, the data caching module is configured to:
and loading the M data at the forefront in the data heat sequence of the node into a buffer space corresponding to the node, wherein the occupied space of the M data is smaller than or equal to the effective space of the buffer space, and M is an integer larger than or equal to 1.
In one embodiment of the present specification, the apparatus further comprises an update module for at least one of:
responding to the transfer or deletion of target data of any node in the distributed data system, and updating the data popularity sequence of the node according to the popularity value of the target data;
and responding to the writing of target data into any node in the distributed data system, and updating the data heat sequence of the node according to the heat value of the target data.
In one embodiment of the present specification, the apparatus further comprises an adjustment module for:
Responding to the change of the M data which are the forefront in the data heat sequence of the node in updating, and adjusting the data in the buffer space corresponding to the node according to the updated data heat sequence.
In one embodiment of the present specification, the apparatus further comprises a persistence module for:
And storing the data heat sequence of each node in the distributed data system into a disk space corresponding to the node.
One or more embodiments of the present specification also provide a computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method according to the first aspect.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or a combination of any of these devices.
In a typical configuration, a computer includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, read only compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by the computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
The terminology used in the one or more embodiments of the specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the specification. As used in this specification, one or more embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of related data is required to comply with the relevant laws and regulations and standards of the relevant country and region, and is provided with corresponding operation entries for the user to select authorization or rejection.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.
It should be understood that although the terms first, second, third, etc. may be used in one or more embodiments of the present description to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "in response to a determination" depending on the context.
The foregoing description of the preferred embodiment(s) is (are) merely intended to illustrate the embodiment(s) of the present invention, and it is not intended to limit the embodiment(s) of the present invention to the particular embodiment(s) described.
Claims (13)
1. A method of data processing, the method comprising:
Correspondingly determining heat values of a plurality of data in each node in the data access process of each node in a distributed data system, wherein the heat values are used for representing the access probability of the data;
And loading partial data in the data heat sequence of each node in the distributed data system to a cache space corresponding to the node, wherein the data heat sequence is a sequence formed by a plurality of data in the node from high to low according to heat values.
2. The data processing method according to claim 1, wherein the determining the heat value of the plurality of data in each node correspondingly in the data access process of each node in the distributed data system includes:
and determining the heat value of the data according to the latest access time and/or access frequency of the data in the node.
3. The data processing method according to claim 2, wherein the determining the heat value of the data according to the latest access time and access frequency of the data in the node comprises:
And carrying out weighted summation on the latest access time and the access frequency of the data in the node to obtain the heat value of the data.
4. A data processing method according to claim 3, the method further comprising:
Determining that the weight of the latest access time of the data is a first weight and the weight of the access frequency of the data is a second weight in response to the occupied space of the data being smaller than a space threshold, wherein the first weight is greater than the second weight;
And determining that the weight of the latest access time of the data is a third weight and the weight of the access frequency of the data is a fourth weight in response to the occupied space of the data is not smaller than a space threshold, wherein the third weight is smaller than the fourth weight.
5. The data processing method according to any one of claims 1 to 4, wherein during the data access process of each node in the distributed data system, determining the heat value of the plurality of data in each node correspondingly includes:
And correspondingly determining the heat value of N data in each node in the data access process of each node in the distributed data system, wherein the occupied space of the N data is larger than the corresponding cache space of the node, and N is an integer larger than or equal to 1.
6. The data processing method according to any one of claims 1 to 4, wherein during the data access process of each node in the distributed data system, determining the heat value of the plurality of data in each node correspondingly includes:
In response to data of any node within the distributed data system being accessed, a heat value of a plurality of data within the node is determined.
7. The data processing method according to claim 1, wherein loading the first part of the data in the data heat sequence of each node in the distributed data system into the cache space corresponding to the node includes:
and loading the M data at the forefront in the data heat sequence of the node into a buffer space corresponding to the node, wherein the occupied space of the M data is smaller than or equal to the effective space of the buffer space, and M is an integer larger than or equal to 1.
8. The data processing method of claim 7, the method further comprising at least one of:
responding to the transfer or deletion of target data of any node in the distributed data system, and updating the data popularity sequence of the node according to the popularity value of the target data;
and responding to the writing of target data into any node in the distributed data system, and updating the data heat sequence of the node according to the heat value of the target data.
9. The data processing method of claim 8, the method further comprising:
Responding to the change of the M data which are the forefront in the data heat sequence of the node in updating, and adjusting the data in the buffer space corresponding to the node according to the updated data heat sequence.
10. The data processing method of claim 1, the method further comprising:
And storing the data heat sequence of each node in the distributed data system into a disk space corresponding to the node.
11. A data processing apparatus, the apparatus comprising:
The system comprises a heat determining module, a heat determining module and a data processing module, wherein the heat determining module is used for determining heat values of a plurality of data in each node in the data access process of each node in a distributed data system, and the heat values are used for representing the access probability of the data;
And the data caching module is used for loading partial data in the front in the data heat sequence of each node in the distributed data system into a cache space corresponding to the node, wherein the data heat sequence is a sequence formed by a plurality of data in the node from high to low according to heat values.
12. An electronic device, comprising:
A processor;
a memory for storing processor-executable instructions;
Wherein the processor is configured to implement the method of any of claims 1-10 by executing the executable instructions.
13. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method of any of claims 1-10.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410199887.5A CN117992547A (en) | 2024-02-22 | 2024-02-22 | Data processing method and device, electronic device and storage medium |
| PCT/CN2025/077837 WO2025176121A1 (en) | 2024-02-22 | 2025-02-18 | Data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410199887.5A CN117992547A (en) | 2024-02-22 | 2024-02-22 | Data processing method and device, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117992547A true CN117992547A (en) | 2024-05-07 |
Family
ID=90888541
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410199887.5A Pending CN117992547A (en) | 2024-02-22 | 2024-02-22 | Data processing method and device, electronic device and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN117992547A (en) |
| WO (1) | WO2025176121A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120144064A (en) * | 2025-05-13 | 2025-06-13 | 苏州元脑智能科技有限公司 | Metadata management method, device, equipment, medium and product |
| WO2025176121A1 (en) * | 2024-02-22 | 2025-08-28 | 北京奥星贝斯科技有限公司 | Data processing |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN121071190B (en) * | 2025-11-07 | 2026-02-06 | 杭州悦数科技有限公司 | A method and system for managing adjacency lists in graph databases based on dynamic weights |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111176560B (en) * | 2019-12-17 | 2022-02-18 | 腾讯科技（深圳）有限公司 | A cache management method, device, computer equipment and storage medium |
| CN116974955A (en) * | 2022-04-22 | 2023-10-31 | 中兴通讯股份有限公司 | Persistent memory file data caching method, system, device and storage medium |
| CN117472279A (en) * | 2023-11-11 | 2024-01-30 | 大熊集团有限公司 | Data cache node management method |
| CN117992547A (en) * | 2024-02-22 | 2024-05-07 | 北京奥星贝斯科技有限公司 | Data processing method and device, electronic device and storage medium |
-
2024
- 2024-02-22 CN CN202410199887.5A patent/CN117992547A/en active Pending
-
2025
- 2025-02-18 WO PCT/CN2025/077837 patent/WO2025176121A1/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2025176121A1 (en) * | 2024-02-22 | 2025-08-28 | 北京奥星贝斯科技有限公司 | Data processing |
| CN120144064A (en) * | 2025-05-13 | 2025-06-13 | 苏州元脑智能科技有限公司 | Metadata management method, device, equipment, medium and product |
| CN120144064B (en) * | 2025-05-13 | 2025-07-25 | 苏州元脑智能科技有限公司 | Metadata management method, device, equipment, medium and product |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2025176121A1 (en) | 2025-08-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11372568B2 (en) | System and method for storing and accessing blockchain data | |
| CN108009008B (en) | Data processing method and system and electronic equipment | |
| US20200257450A1 (en) | Data hierarchical storage and hierarchical query method and apparatus | |
| CN112181902B (en) | Database storage method and device and electronic equipment | |
| US20130290636A1 (en) | Managing memory | |
| WO2025176121A1 (en) | Data processing | |
| CN116795878B (en) | Data processing method and device, electronic equipment and medium | |
| CN119473162A (en) | Cache management method and device, cache, electronic device and storage medium | |
| JP2023127069A (en) | Information processing device and memory access control method | |
| CN114461681A (en) | Cache content management method and device, electronic equipment and storage medium | |
| CN115268777B (en) | A storage node expansion method and device | |
| CN109002400A (en) | A kind of perception of content type Computer Cache management system and method | |
| CN113806355A (en) | Implementation method, database, server and medium of distributed database redistribution | |
| CN116974955A (en) | Persistent memory file data caching method, system, device and storage medium | |
| US11429311B1 (en) | Method and system for managing requests in a distributed system | |
| CN119396888A (en) | Data caching method and device, electronic device and storage medium | |
| CN113204520A (en) | Remote sensing data rapid concurrent read-write method based on distributed file system | |
| CN119166062A (en) | Partition management method and device with short write-stop time, electronic device and storage medium | |
| CN115048457B (en) | Method, device and storage medium for processing state data based on block chain | |
| CN118132600A (en) | Data processing method and device, electronic device and storage medium | |
| CN119357087B (en) | Cache replacement method, device, equipment and storage medium | |
| CN113282585A (en) | Report calculation method, device, equipment and medium | |
| CN119669198B (en) | Data management methods and apparatus, electronic devices and storage media | |
| CN119127728B (en) | Data access method, device, electronic equipment and storage medium | |
| CN115809140B (en) | Memory management method, system, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |