CN115203195A

CN115203195A - Data table heat distinguishing method and device and related equipment

Info

Publication number: CN115203195A
Application number: CN202110389324.9A
Authority: CN
Inventors: 季振峰
Original assignee: Huawei Cloud Computing Technologies Co Ltd
Current assignee: Huawei Cloud Computing Technologies Co Ltd
Priority date: 2021-04-12
Filing date: 2021-04-12
Publication date: 2022-10-18
Also published as: WO2022217987A1

Abstract

The application provides a data table heat degree distinguishing method, a data table heat degree distinguishing device and related equipment, wherein the method comprises the following steps: the service node acquires a second data table associated with the first data table from the storage node, then acquires the associated heat degrees of the first data table and the second data table according to the second data table, and determines the heat degree of the first data table according to the associated heat degrees of the first data table and the second data table after acquiring the associated heat degrees of the first data table and the second data table, wherein the associated heat degrees of the first data table and the second data table are acquired according to the inherent heat degree of the second data table and the association relation of the first data table and the second data table. The method can improve the distinguishing accuracy of the heat degree of the data table.

Description

Data table heat distinguishing method and device and related equipment

Technical Field

The present application relates to the field of big data, and in particular, to a method and an apparatus for distinguishing heat of a data table, and a related device.

Background

In the big data era, data is explosively increased, and the number of data tables is becoming more and more huge as the data is increased. In order to improve the usage efficiency of the data tables, it is necessary to distinguish the heat degrees of a large number of data tables and manage the large number of data tables according to the heat degrees of the data tables, for example, to clean the data tables with low heat degrees or set the data tables with high heat degrees at the top.

However, the existing data table heat degree distinguishing method has the problem of low accuracy of distinguishing the data table heat degree.

Disclosure of Invention

The application provides a data table heat degree distinguishing method, a data table heat degree distinguishing device and related equipment, and distinguishing accuracy of the data table heat degree can be improved.

In a first aspect, a method for distinguishing heat of a data table is provided, the method comprising:

the service node acquires a second data table associated with the first data table from a storage node, wherein the storage node stores a plurality of data tables;

the service node acquires the associated heat of the first data table and the second data table, wherein the associated heat of the first data table and the second data table is acquired according to the inherent heat of the second data table and the association relation of the first data table and the second data table, and the inherent heat of the second data table is the heat generated by calling the second data table;

and the service node determines the heat degree of the first data table according to the association heat degree of the first data table and the second data table.

In the above scheme, when the heat degree of the first data table is calculated, the heat degree brought by the second data table having a correlation relationship with the first data table for the first data table, that is, the correlation heat degree of the first data table and the second data table is introduced, so that the calculated heat degree of the first data table can be improved more accurately, and the heat degrees of the plurality of data tables can be better distinguished under the condition that the heat degrees of the plurality of data tables are obtained.

In one possible implementation, the obtaining, by the service node, the second data table associated with the first data table from the storage node includes:

the service node acquires the second data table having a data blood relationship with the first data table from the storage node, wherein the data blood relationship represents that the second data table is obtained by calculation according to the first data table, or the first data table is obtained by calculation according to the second data table;

the step of acquiring, by the service node, the association heat of the first data table and the second data table includes:

and the service node calculates the association heat of the first data table and the second data table according to the data blood relationship of the first data table and the second data table.

the service node acquires the second data table having a primary foreign key association relationship with the first data table from the storage node, wherein the primary foreign key association relationship indicates that one or more fields in the first data table are referred to as a primary key of the second data table, or one or more fields in the second data table are referred to as a primary key of the first data table;

and the service node calculates the association heat of the first data table and the second data table according to the main foreign key association relationship of the first data table and the second data table.

In a possible implementation manner, the determining, by the service node, the heat degree of the first data table according to the associated heat degrees of the first data table and the second data table includes:

and the service node determines the heat degree of the first data table according to the inherent heat degree of the first data table and the associated heat degree of the first data table and the second data table, wherein the inherent heat degree of the first data table is the heat degree generated by calling the first data table.

In one possible implementation, the method further includes:

the service node calculates the heat degree of the plurality of data tables;

and the service node deletes the data table with the heat degree smaller than a first preset threshold value from the storage node according to the calculation result.

In the scheme, the service node deletes the data table with low heat from the storage node according to the calculation result, so that the storage space can be saved.

In one possible implementation, the method further includes:

the service node calculates the heat degree of the plurality of data tables;

and the service node adjusts the position of the data table with the heat degree larger than the second preset threshold value in the plurality of data tables on the display interface to the front of the data table with the heat degree smaller than the second preset threshold value according to the calculation result.

In the scheme, the service node adjusts the position of the data table with high heat degree on the display interface to the front of the data table with low heat degree, so that a user can conveniently and quickly check the data table with high heat degree.

In one possible implementation, the method further includes:

the service node calculates the heat degrees of the plurality of data tables;

and the service node migrates the data table with the heat degree smaller than a third preset threshold value to a first storage device according to the calculation result, wherein the storage performance of the first storage device is lower than that of the storage node.

In the above scheme, the service node migrates the data table with low heat to the first storage device with storage performance lower than that of the storage node, so that the situation that the data table with low heat continuously occupies the storage node resource is avoided, and a subsequent user can find the data table from the first storage device when needing to check the data table.

In one possible implementation, the method further includes:

the service node calculates the heat degrees of the plurality of data tables;

and the service node migrates the data table with the heat degree larger than a fourth preset threshold value to a second storage device according to the calculation result, wherein the storage performance of the second storage device is higher than that of the storage node.

In the above scheme, the service node migrates the data table with high heat to the second storage device with higher storage performance than the storage node, so that the efficiency of operating data in the data table with high heat can be improved, and the storage safety of the data table with high heat can be improved.

In a second aspect, an apparatus for distinguishing heat of a data table is provided, where the apparatus is applied to a service node, and the apparatus includes:

the acquisition module is used for acquiring a second data table associated with the first data table from a storage node, and the storage node stores a plurality of data tables;

the processing module is configured to obtain an associated heat degree of the first data table and the second data table, where the associated heat degree of the first data table and the second data table is obtained according to an inherent heat degree of the second data table and an association relationship between the first data table and the second data table, and the inherent heat degree of the second data table is a heat degree generated by calling the second data table;

the processing module is used for determining the heat degree of the first data table according to the associated heat degree of the first data table and the second data table.

In a possible implementation manner, the obtaining module is specifically configured to:

acquiring the second data table having a data blood relationship with the first data table from the storage node, wherein the data blood relationship indicates that the second data table is obtained by calculation according to the first data table, or the first data table is obtained by calculation according to the second data table;

the processing module is specifically configured to:

and calculating the association heat of the first data table and the second data table according to the data blood relationship of the first data table and the second data table.

obtaining the second data table having a primary foreign key association relationship with the first data table from the storage node, wherein the primary foreign key association relationship indicates that one or more fields in the first data table are referenced as primary keys of the second data table, or one or more fields in the second data table are referenced as primary keys of the first data table;

the processing module is specifically configured to:

and calculating the association heat of the first data table and the second data table according to the association relation of the main external key of the first data table and the second data table.

In a possible implementation manner, the processing module is specifically configured to:

and determining the heat degree of the first data table according to the inherent heat degree of the first data table and the associated heat degree of the first data table and the second data table, wherein the inherent heat degree of the first data table is the heat degree generated by calling the first data table.

In a possible implementation manner, the processing module is further configured to:

calculating the heat degree of the plurality of data tables;

and deleting the data table with the heat degree smaller than a first preset threshold value from the storage node according to the calculation result.

calculating the heat degree of the plurality of data tables;

and adjusting the position of the data table with the heat degree larger than the second preset threshold value in the plurality of data tables on the display interface to the front of the data table with the heat degree smaller than the second preset threshold value according to the calculation result.

In one possible implementation manner, the processing module is further configured to:

calculating the heat degrees of the plurality of data tables;

and migrating the data table with the heat degree smaller than a third preset threshold value to a first storage device and migrating the data table with the heat degree larger than a fourth preset threshold value to a second storage device according to the calculation result, wherein the storage performance of the first storage device is lower than that of the storage nodes, and the storage performance of the second storage device is higher than that of the storage nodes.

In a third aspect, there is provided a non-transitory computer readable storage medium having stored thereon computer readable instructions which, when executed, perform a method as described in the first aspect above or any specific implementation of the first aspect.

In a fourth aspect, a computer program product is provided, comprising a computer program which, when read and executed by a cluster of computer devices, causes the cluster of computer devices to perform the method as described in the first aspect or any specific implementation manner of the first aspect.

In a fifth aspect, a cluster of computing devices is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device to perform the method as described in the first aspect above or any specific implementation of the first aspect.

In one possible implementation, the cluster of computing devices includes a computing device including a processor and a memory; the processor is configured to execute the instructions stored in the memory to cause the computing device to perform the method as provided by the first aspect or any possible implementation manner of the first aspect.

In one possible implementation, the cluster of computing devices includes at least two computing devices, each computing device including a processor and a memory; the processors of the at least two computing devices are configured to execute instructions stored in the memories of the at least two computing devices to cause the cluster of computing devices to perform a method as provided by the first aspect or any possible implementation manner of the first aspect.

Drawings

Fig. 1 is a schematic diagram of an application scenario related to an embodiment of the present application;

FIG. 2 is a diagram illustrating a data relationship according to an embodiment of the present application;

FIG. 3 is a diagram illustrating a main foreign key association according to an embodiment of the present disclosure;

FIG. 4 is a flowchart illustrating a method for distinguishing heat of a data table according to an embodiment of the present application;

FIG. 5 is a diagram illustrating a data relationship of a first data table according to an embodiment of the present application;

FIG. 6 is a schematic flow chart diagram illustrating another method for distinguishing heat of a data table according to an embodiment of the present application;

fig. 7 is a schematic diagram of a primary foreign key association relationship of a first data table according to an embodiment of the present application;

FIG. 8 is a block diagram of a data processing system according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of a computing device cluster according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of a computing device according to an embodiment of the present application.

Detailed Description

The technical solution in the present application will be described below with reference to the accompanying drawings.

The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

In the embodiments of the present application, "at least one" means one or more, "a plurality" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a alone, A and B together, and B alone, wherein A and B may be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of the singular or plural items. For example, at least one (one) of a, b, or c, may represent: a. b, c, a-b, a-c, b-c or a-b-c, wherein a, b and c can be single or multiple.

Any embodiment or design described herein as "exemplary" or "e.g.," is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "such as" is intended to present concepts related in a concrete fashion.

In order to facilitate understanding of the embodiments of the present application, the concepts, terms, and the like related to the embodiments of the present application will be described below.

(1) The transaction data (transactional data), which may also be referred to as transaction data, business data, etc., describes internal or external events or transaction records during the operation of the organization business, such as sales orders, call records, etc.

(2) The data popularity is large and indicates that the data is high in popularity, indicates that the data is high in probability of being accessed within a period of time from the current beginning, and indicates that the data is low in popularity and indicates that the data is low in probability of being accessed within a period of time from the current beginning.

(3) The data table heat degree is used for reflecting a numerical value of the attention degree of the data table, the numerical value indicates the possibility that the data table is accessed within a period of time from the current beginning, the data table heat degree is large and indicates that the attention degree of the data table is high and indicates that the possibility that the data table is accessed within the period of time from the current beginning is high, the data table heat degree is small and indicates that the attention degree of the data table is low and indicates that the possibility that the data table is accessed within the period of time from the current beginning is low.

(4) The inherent heat of the data table, the heat generated for the data table itself to be called, may be determined according to the number of times of being called (also referred to as the number of times of use or the number of times of access) of the data table, and generally, the inherent heat of the data table is equal to the number of times of being called of the data table, where the number of times of being called of the data table includes the number of times of querying (select) data, the number of times of adding (insert) data, the number of times of deleting (delete) data, the number of times of modifying (update) data, and the like, that is, the number of times of being called of the data table = the number of times of querying data in the data table + the number of times of adding data in the data table + the number of times of deleting data in the data table + the number of times of modifying data in the data table, and in the case that the calling operation of the data table further includes other operations, the number of being called of the data table further includes other operations in the data table.

The following provides a brief description of an application scenario related to an embodiment of the present application.

With the rapid development of internet technology, users of various website platforms rapidly grow, the amount of data to be processed also grows exponentially, the types of data are various, the data are very complex, and the complex data are fused, converted and circulated to generate new data to be converged into a data ocean. It can be understood that tens of thousands or even millions of data tables are deposited in the process of rapid data growth, as shown in fig. 1, some tables in a large number of data tables stored by a storage node are temporary tables or Chen Jiubiao, which are rarely called and should be cleaned, and some tables are frequently called and should be emphasized to improve the utilization efficiency of the data tables and save storage resources. Therefore, managing a large number of data tables has become one of the important issues of concern for each enterprise. The division of the popularity of the large amount of data tables is a key ring for the enterprise to manage the large amount of data tables.

At present, when a large number of data tables are managed, the following two methods are generally adopted to distinguish the heat of the data tables: (1) Based on the data creation time, (2) based on the inherent heat of the data table, wherein,

specifically, it is assumed that a storage node stores a transaction data table a and a transaction data table B, data in the transaction data table a is created in the last year, data in the transaction data table B is created one year ago, after a service node acquires the transaction data table a and the transaction data table B from the storage node, the creation time of the data in the transaction data table a and the creation time of the data in the transaction data table B are acquired and compared, when it is judged that most or all of the creation time of the data in the transaction data table a is later than the creation time of the data in the transaction data table B, it is determined that the heat of the transaction data table a is greater than the heat of the transaction data table B, otherwise, it is determined that the heat of the transaction data table a is less than the heat of the transaction data table B.

It can be understood that in a practical application scenario, the following situations are very likely to exist: although the creation time of the data in the transaction data table a is mostly or entirely later than the creation time of the data in the transaction data table B, the data in the transaction data table B is more important than the data in the transaction data table a, and the call of the transaction data table B is more frequent than the call of the transaction data table a, that is, the heat of the transaction data table B is actually greater than that of the transaction data table a.

Under the possible existing condition, the service node distinguishes the heat of the two transaction data tables according to the method based on the data creation time, and the obtained heat distinguishing result is obviously inaccurate and is not consistent with the actual application scene.

The data table-based inherent heat method is mainly used for distinguishing the heat of a webpage data table (i.e. a table mainly including webpage data (such as articles, pictures, videos and the like published on webpages)), specifically, assuming that a storage node stores a webpage data table a and a webpage data table B, after the storage node acquires the webpage data table a and the webpage data table B, the service node acquires and compares the inherent heat of the webpage data table a and the inherent heat of the webpage data table B, and when the inherent heat of the webpage data table a is judged to be greater than the inherent heat of the webpage data table B, it is determined that the heat of the webpage data table a is greater than the heat of the webpage data table B, otherwise, it is determined that the heat of the webpage data table a is less than the heat of the webpage data table B.

It can be understood that in a practical application scenario, the following situations are very likely to exist: although the inherent heat of the web page data table a is greater than the inherent heat of the web page data table B, the data in the web page data table a is created in the last year, and the data in the web page data table B is created a year ago, wherein the creation time of the web page data can be understood as the time when the web page data is published on the web page.

Under the possible existing condition, the service node distinguishes the heat of the two webpage data tables according to the method based on the inherent heat of the data tables, and the obtained heat distinguishing result is obviously inaccurate and is not consistent with the actual application scene.

It can be seen that the two methods have the problems that the accuracy of distinguishing the heat degrees of the data tables is low, and the method is not consistent with the practical application scene.

In order to solve the above problems, embodiments of the present application provide a method, an apparatus, and a related device for distinguishing a heat degree of a data table, which can improve the distinguishing accuracy of the heat degree of the data table and better conform to an actual application scenario.

Before introducing the method, the apparatus, and the related device for distinguishing the heat of the data table provided in the embodiment of the present application, concepts such as an association relationship of the data table, an association heat of the data table, and the like, and an acquisition process of an inherent heat of the data table related to the embodiment of the present application are introduced.

(1) The data table association relationship specifically includes a data consanguinity relationship and a main foreign key association relationship, wherein,

data lineage relationships, which may also be referred to as data lineage relationships, data provenance relationships, data lineage relationships, and the like, refer to a relationship that may be formed between data sheets during their creation, fusion, transformation, circulation to extinction. As shown in fig. 2, it is assumed that the original data is stored in the data table 1, after calculating part or all of the original data in the data table 1, an intermediate table 2 including intermediate data (i.e., calculated part or all of the original data) is obtained, and after calculating the intermediate data in the intermediate table 2 from the data table 1, a data table 3 including final data is formed, and at this time, the data link from the data table 1 to the data table 2 to the data table 3 represents the data relationship of the three tables. Specifically, data table 1 and data table 2 may be said to have a direct blood-related relationship, data table 2 and data table 3 may be said to have a direct blood-related relationship, and data table 1 and data table 3 may be said to have an indirect blood-related relationship. By analyzing the data blood-related relation between the data tables, the migration circulation of the data tables can be clearly understood, and a basis is provided for evaluation of the value of the data tables and management of the data tables.

In the case where the data bloodline relationships among data table 1, data table 2, and data table 3 are as shown in fig. 2, it can be seen that data table 2 directly depends on data table 1, and data table 3 directly depends on data table 2 and indirectly depends on data table 1. It can be understood that if the data in the data table 1 for calculating the data table 2 and the data table 3 is accessed, it means that the data table 2 and the data table 3 are indirectly accessed, that is, the data table 1 has an effect of improving the heat degree of the data table 2 and the heat degree of the data table 3 to some extent; if the data from the data table 1 in the data table 2 is accessed, it indicates that the data table 1 and the data table 3 are indirectly accessed, that is, the data table 2 has an effect of improving the heat of the data table 1 and the heat of the data table 3 to a certain extent; if the data from the data table 2 in the data table 3 is accessed, it means that the data table 1 and the data table 2 are indirectly accessed, that is, the data table 3 has an effect of improving the heat of the data table 1 and the heat of the data table 2 to some extent.

Therefore, it can be understood that, when determining the heat degree of the data table 1, the heat degree of the data table 2, and the heat degree of the data table 3, if the heat degree of each data table, which is increased by other data tables having data relationship (including direct blood relationship and indirect blood relationship) with the data table, is also taken into consideration in addition to the inherent heat degree of each data table, the determined heat degree of each data table is more accurate, and the importance of each data table can be more emphasized.

Primary key-foreign key relationship (primary key-foreign key relationship), which defines an association relationship between two tables in a relational database, as shown in fig. 3, one or more fields A1 in data table 1 are referred to as primary keys of data table 2', and at this time, field A1 in data table 1 is referred to as a foreign key pointing to data table 2', and data table 1 and data table 2' have a primary foreign key association relationship.

As shown in fig. 3, the primary key of the data table 2 'is also referred to as the primary key of the data table 3', in this case, the data table 1 and the data table 3 'are also referred to as having a primary external key association relationship, and for distinguishing and describing, the primary external key association relationship between the data table 1 and the data table 2' and the primary external key association relationship between the data table 2 'and the data table 3' are referred to as a direct primary external key association relationship, and the primary external key association relationship between the data table 2 'and the data table 3' is referred to as an indirect primary external key association relationship.

In the case that the primary foreign key association relationship among data table 1, data table 2', and data table 3' is as shown in fig. 3, it can be seen that data table 2' is directly dependent on data table 1, and data table 3' is directly dependent on data table 2', and is indirectly dependent on data table 1. It can be understood that if the field A1 in the data table 1 is accessed, it indicates that the data table 2 and the data table 3 are indirectly accessed, that is, the data table 1 has an effect of increasing the popularity of the data table 2 and the popularity of the data table 3 to some extent; if the primary key of the data table 2 'is accessed, it indicates that the data table 1 and the data table 3' are indirectly accessed, that is, the data table 2 'has an effect of improving the heat degree of the data table 1 and the data table 3' to a certain extent; if the primary key of the data table 3 'is accessed, it means that the data table 1 and the data table 2' are indirectly accessed, that is, the data table 3 'has an effect of increasing the hotness of the data table 1 and the data table 2' to some extent.

Therefore, it can be understood that, when determining the heat degree of the data table 1, the heat degree of the data table 2', and the heat degree of the data table 3', if the heat degree of each data table, which is increased by other data tables having a main foreign key association relationship (including a direct main foreign key association relationship and an indirect main foreign key association relationship) with the data table, is taken into consideration in addition to the inherent heat degree of each data table, the determined heat degree of each data table is more accurate, and the importance of each data table can be better highlighted.

(2) The associated heat degree of the data table refers to the heat degree brought by the associated data table to the associated data table, such as the heat degree increased by the data table 2 and/or the data table 3 having a data relationship with the data table 1, the heat degree increased by the data table 2 'and/or the data table 3' having a main foreign key relationship with the data table 1, and the like.

(3) The acquisition process of the inherent heat of the data table comprises the following steps:

taking the inherent heat of the service node acquiring the data table 1 as an example, the process includes, but is not limited to, the following steps:

a1, the service node acquires the log information of the data operation of the data table 1 from the storage node, and acquires the information of the data operation of the data table 1 according to the log information of the data operation of the data table 1.

The log information of the data operation of the data table 1 indicates log information about the data operation performed by the user, which is automatically recorded by the storage node when the user performs the data operation on the data table 1, and the log information includes information about the data operation performed by the user on the data table 1, such as the type of the data operation performed on the data table 1 (e.g., deleting data, adding data, etc.) and the time when the data operation is performed on the data table 1, so that the information about the data operation on the data table 1 can be obtained according to the log information about the data operation on the data table 1.

In a specific implementation, the service node may obtain log information of data table 1 in a preset time period from the storage node, and then obtain information of the data operation of data table 1 in the preset time period according to the log information, for example, the service node may obtain log information of data table 1 in 2020, and then obtain information of the data operation of data table 1 in 2020 according to the log information of data table 1 in 2020.

And A2, the service node determines the called times of the data table 1 according to the data operation information of the data table 1.

Specifically, the number of times of querying data in the data table 1, the number of times of adding data in the data table 1, the number of times of deleting data in the data table 1, the number of times of modifying data in the data table 1, and the like may be counted according to the information of the data operation of the data table 1, and then the number of times of being called of the data table 1 may be determined by summing the numbers.

And A3, determining the inherent heat of the data table 1 according to the called times of the data table 1.

In a specific embodiment, the intrinsic heat of data table 1 = the number of times data table 1 is called.

In the method, the apparatus, and the related device for distinguishing the heat of the data table provided in the embodiment of the present application, a service node may obtain a second data table associated with a first data table from a storage node, then obtain an association heat of the first data table and the second data table according to an association relationship between the first data table and the second data table and an inherent heat of the second data table, and determine the heat of the first data table according to the association heat after obtaining the association heat of the first data table and the second data table, where the association relationship between the first data table and the second data table includes one or more of a data blood relationship and a main foreign key association relationship.

A data table heat degree distinguishing method provided in the embodiment of the present application is described in more detail below with reference to fig. 4, and as shown in fig. 4, the data table heat degree distinguishing method provided in the embodiment of the present application includes, but is not limited to, the following steps:

s101, the service node acquires a first data table and a second data table having a data blood relationship with the first data table from the storage node.

The storage node stores a plurality of data tables, and the first data table may be any one or more of the plurality of data tables stored by the storage node. The plurality of data tables stored by the storage node may be various types of tables such as a transaction data table, a web page data table, and the like, and the plurality of data tables may be tables in databases such as an sql server database, an oracle database, and the like, and may also be tables which are temporarily created by a user and do not belong to any database, which is not specifically limited herein.

As can be seen from the above description of the data relationship, the data relationship between the first data table and the second data table means that the second data table is calculated according to the first data table, and/or the first data table is calculated according to the second data table. Specifically, the data relationship between the first data table and the second data table may be a direct relationship or an indirect relationship, which is not limited herein.

In a specific implementation, after acquiring the first data table, the service node may acquire, from the storage node, a second data table having a data blood relationship with the first data table through a data warehouse tool (e.g., hive) or an sql statement, where hive is a data warehouse tool based on Hadoop, and is used to perform data extraction, transformation, and loading, which is a mechanism that may store, query, and analyze large-scale data stored in Hadoop.

It should be noted that, the service node obtains the second data table having a data lineage relationship with the first data table from the storage node through a data warehouse tool or an sql statement, which is merely an example and should not be considered as a specific limitation. In specific implementation, the service node may also obtain, through other manners, a second data table having a data consanguinity relationship with the first data table, for example, a second data table having a data consanguinity relationship with the first data table is searched by a manual reading code, the service node receives a manually input name of the second data table having a data consanguinity relationship with the first data table, and then obtains the second data table according to the manually input name of the second data table.

S102, the service node acquires the inherent heat H of the first data table ₀ 。

Wherein the inherent heat H of the first data table ₀ The heat generated for the first data table itself to be called.

S103, the service node calculates the associated heat H of the first data table and the second data table according to the data blood relationship of the first data table and the second data table and the inherent heat of the second data table ₁ 。

The inherent heat of the second data table is the heat generated when the second data table is called.

Specifically, after acquiring the second data table having a data blood relationship with the first data table, the service node may determine a blood relationship weight corresponding to the second data table according to the data blood relationship between the first data table and the second data table, calculate an inherent heat of the second data table, and then calculate an associated heat H of the first data table and the second data table according to the blood relationship weight corresponding to the second data table and the inherent heat of the second data table ₁ 。

For example, as shown in FIG. 5, assume that there are two second data tables having a data blood relationship with the first data table, namely data table A and data table B, wherein the second data table A has a direct blood relationship with the first data table, and the second data table B has an indirect blood relationship with the first data table, and assume that the intrinsic heat of the second data table A is H _0,A The intrinsic heat of the second data table B is H _0,B The second data table A corresponds to a blood relationship weight W _A The second data table B corresponds to a blood relationship weight of W _B If yes, the service node obtains the associated heat H of the first data table A, B and the second data table A, B ₁ Comprises the following steps:

H ₁ ＝W _A *H _0,A +W _B *H _0,B

wherein, W _A And W _B Are numbers greater than 0 and less than 1, preferably W, considering that the second data table A has a direct blood relationship with the first data table, the second data table B has an indirect blood relationship with the first data table, and the second data table A has a closer relationship with the first data table _A Greater than W _B 。

S104, the service node according to the inherent heat H of the first data table ₀ And the associated heat degree H of the first data table and the second data table ₁ And determining the heat degree H of the first data table.

In specific examples of the present application, H = H ₀ +H ₁ 。

It should be noted that, for the sake of simplicity, the inherent heat H of the first data table is not set forth in the embodiments of the present application ₀ The obtaining process of the inherent heat of the second data table and the obtaining process of the inherent heat of the second data table are described in an expanded manner, and reference may be specifically made to the obtaining process of the inherent heat of the data table 1 described above, which is not described herein again.

Referring to fig. 6, fig. 6 is a schematic flow chart of another data table heat distinguishing method provided in the embodiment of the present application, and as shown in fig. 6, the data table heat distinguishing method provided in the embodiment of the present application includes, but is not limited to, the following steps:

s201, the service node acquires a first data table and a second data table which has a main foreign key association relation with the first data table from the storage node.

As can be seen from the above description of the primary foreign key association relationship, having a primary foreign key association relationship between the first data table and the second data table means that one or more fields in the first data table are referenced as primary keys of the second data table, and/or one or more fields in the second data table are referenced as primary keys of the first data table.

In a specific implementation, after acquiring the first data table, the service node may acquire, from the storage node, a second data table having a main foreign key association relationship with the first data table through a data warehouse tool or an sql statement.

It should be noted that, the service node obtains the second data table, in which the first data table has the main foreign key association relationship, from the storage node through a data warehouse tool or an sql statement, which is only an example. In a specific implementation, the service node may also obtain, through other manners, a second data table having a main foreign key association relationship with the first data table, for example, a manual reading code is used to search for the second data table having the main foreign key association relationship with the first data table, the service node receives a manually input name of the second data table having the main foreign key association relationship with the first data table, and then obtains the second data table according to the manually input name of the second data table.

S202, the service node acquires the inherent heat H of the first data table ₀ 。

S203, the service node associates the relationship and the second external key according to the first data table and the second data tableThe inherent heat degree of the data table is calculated, and the associated heat degree H of the first data table and the second data table is calculated ₁ 。

Specifically, after acquiring the second data table having a primary external key association relationship with the first data table, the service node may determine an association weight corresponding to the second data table according to the primary external key association relationship between the first data table and the second data table, calculate an inherent heat of the second data table, and then calculate an association heat H of the first data table and the second data table according to the association weight corresponding to the second data table and the inherent heat of the second data table ₁ 。

For example, as shown in fig. 7, two second data tables having a primary foreign key association relationship with the first data table are assumed, namely, a data table C having a direct primary foreign key association relationship with the first data table and a data table D having an indirect primary foreign key association relationship with the first data table, and the inherent heat of the second data table C is assumed to be H _0,C The intrinsic heat of the second data table D is H _0,D The second data table C corresponds to an association weight W _C The second data table D has an associated weight W _D If so, the service node obtains the associated heat H of the first data table C, D ₁ Comprises the following steps:

H ₁ ＝W _C *H _0,C +W _D *H _0,D

wherein, W _C And W _D Are numbers greater than 0 and less than 1, preferably W, considering that the second data table C has a direct primary foreign key association with the first data table, the second data table D has an indirect primary foreign key association with the first data table, and the second data table C has a closer relationship with the first data table _C Greater than W _D 。

S204, the service node according to the inherent heat H of the first data table ₀ And the associated heat degree H of the first data table and the second data table ₁ And determining the heat degree H of the first data table.

In specific examples of the present application, H = H ₀ +H ₁ 。

As can be appreciated, the slave storage node is at the service nodeWhen a second data table having an association relation with the first data table is obtained, if not only the second data table having a data blood relationship with the first data table but also the second data table having a main foreign key association relation with the first data table is obtained, the association heat H of the first data table and the second data table obtained by the service node through calculation ₁ The method not only comprises the heat brought by the second data table having data consanguinity relation with the first data table, but also comprises the heat brought by the second data table having main foreign key association relation with the first data table.

Continuing with the example of fig. 5 and 7 as mentioned above, assuming that the first data table has both the data relationship shown in fig. 5 and the primary foreign key association relationship shown in fig. 7, the service node obtains the association heat H of the first data table and the second data table ₁ Comprises the following steps:

H ₁ ＝W _A *H _0,A +W _B *H _0,B +W _C *H _0,C +W _D *H _0,D

the heat degree H of the first data table = the inherent heat degree H of the first data table ₀ + the associated heat degree H of the first data table and the second data table ₁ Therefore, it can be understood that the association heat H in the first data table and the second data table ₁ Under the condition that the heat brought by the second data table having the data blood relationship with the first data table is included, and the heat brought by the second data table having the main foreign key association relationship with the first data table is also included, the heat H of the first data table calculated by the service node does not only include the heat brought by the second data table having the data blood relationship with the first data table, but also includes the heat brought by the second data table having the main foreign key association relationship with the first data table.

It can be understood that the service node may obtain the heat degrees of the plurality of data tables according to the data table heat degree distinguishing method provided above, and under the condition that the service node obtains the heat degrees of the plurality of data tables, the service node may distinguish which data tables have higher heat degrees and which data tables have lower heat degrees from the plurality of data tables, thereby achieving the purpose of managing the plurality of data tables.

In a possible embodiment, after acquiring the heat degrees of the multiple data tables, the service node may delete the data tables with the heat degrees smaller than the first preset threshold from the storage node according to the heat degrees of the multiple data tables, so as to save the storage space.

In a possible embodiment, after obtaining the heat degrees of the plurality of data tables, the service node may adjust, according to the heat degrees of the plurality of data tables, the position of the data table of which the heat degree is greater than the second preset threshold value in the plurality of data tables on the display interface to the front of the data table of which the heat degree is less than the second preset threshold value, that is, adjust the position of the data table of which the heat degree is greater than the second preset threshold value on the display interface to a position which is more convenient for a user to view, so that the user can conveniently and quickly view the data table of which the heat degree is large.

In a possible embodiment, after obtaining the heat degrees of the plurality of data tables, the service node may further migrate the data table with the heat degree smaller than a third preset threshold to the first storage device, and migrate the data table with the heat degree greater than a fourth preset threshold to the second storage device, where the storage performance of the first storage device is lower than that of the storage node, and the storage performance of the second storage device is higher than that of the storage node.

The first preset threshold, the second preset threshold, the third preset threshold, and the fourth preset threshold may be set according to actual conditions, and are not specifically limited herein.

It can be understood that the service node migrates the data table with low heat to the first storage device with lower storage performance than the storage node, which can not only prevent the data table with low heat from continuously occupying the storage node resources, but also can be found from the first storage device when a subsequent user needs to check the data table; the service node migrates the data table with high heat to the second storage device with higher storage performance than the storage node, so that the efficiency of operating data in the data table with high heat can be improved, and the storage safety of the data table with high heat can be improved.

As can be seen from the foregoing embodiments, the method for distinguishing heat of data tables according to the embodiments of the present application introduces the heat of the first data table into the first data table when determining the heat H of the first data tableThe second data table with the association relation brings the heat degree of the first data table, namely the association heat degree H of the first data table and the second data table ₁ The calculated heat degree H of the first data table can be more accurate, the data table more accords with the actual application scene, and the heat degrees of the data tables can be better distinguished under the condition that the heat degrees of the data tables are obtained.

The foregoing has set forth in detail a method for distinguishing heat of a data table according to an embodiment of the present application, and based on the same inventive concept, the following provides a device for distinguishing heat of a data table according to an embodiment of the present application.

Referring to fig. 8, fig. 8 is a schematic structural diagram of a data processing system 10 provided in an embodiment of the present application, where the data processing system 10 includes a data table popularity distinguishing apparatus 1100 provided in the embodiment of the present application, and the data table popularity distinguishing apparatus 1100 includes: an obtaining module 1101 and a processing module 1102, the data table heat distinguishing apparatus 1100 may be integrated into a service node 110 in the data processing system 10, and the data processing system 10 may further include a storage node 120, a first storage apparatus 130, and a second storage apparatus 140 in addition to the service node 110, wherein,

the storage node 120 stores a plurality of data tables;

an obtaining module 1101, configured to obtain a second data table associated with the first data table from the storage node 120;

a processing module 1102, configured to obtain an association heat H of the first data table and the second data table ₁ Wherein the correlation heat degree H of the first data table and the second data table ₁ Obtaining the inherent heat of the second data table and the incidence relation between the first data table and the second data table, wherein the inherent heat of the second data table is the heat generated by calling the second data table;

a processing module 1102, configured to determine a heat degree H associated with the first data table and the second data table ₁ The heat H of the first data table is determined.

In a possible embodiment, the obtaining module 1101 is specifically configured to:

obtaining a second data table having a data relationship with the first data table from the storage node 120, wherein the data relationship indicates that the second data table is obtained by calculation according to the first data table, and/or the first data table is obtained by calculation according to the second data table;

the processing module 1102 is specifically configured to:

calculating the associated heat H of the first data table and the second data table according to the data blood relationship of the first data table and the second data table ₁ 。

obtaining a second data table having a primary foreign key association with the first data table from the storage node 120, wherein the primary foreign key association indicates that one or more fields in the first data table are referenced as a primary key of the second data table and/or one or more fields in the second data table are referenced as a primary key of the first data table;

the processing module 1102 is specifically configured to:

calculating the association heat H of the first data table and the second data table according to the association relation of the main external key of the first data table and the second data table ₁ 。

In a possible embodiment, the processing module 1102 is specifically configured to:

according to the inherent heat H of the first data table ₀ And the associated heat degree H of the first data table and the second data table ₁ Determining the heat degree H of the first data table, wherein the inherent heat degree H of the first data table ₀ The generated heat is called for the first data table.

In a possible embodiment, the processing module 1102 is further configured to:

calculating the heat degree of a plurality of data tables;

and deleting the data table with the heat degree smaller than the first preset threshold value from the storage node 120 according to the calculation result.

In a possible embodiment, the processing module 1102 is further configured to:

and migrating the data table with the heat degree smaller than the third preset threshold value to the first storage device 130 and migrating the data table with the heat degree larger than the fourth preset threshold value to the second storage device 140 according to the calculation result, wherein the performance of the first storage device 130 is lower than that of the storage node 120, and the performance of the second storage device 140 is higher than that of the storage node 120.

The first preset threshold, the second preset threshold, the third preset threshold, and the fourth preset threshold may be set according to actual conditions, and are not limited specifically here.

Specifically, the specific implementation of the data table heat degree distinguishing device 1100 in the data processing system 10 to perform various operations may refer to the description in the relevant content in the data table heat degree distinguishing method embodiment, and for the sake of brevity of the description, no further description is given here.

It should be understood that the data processing system 10 and the data table heat distinguishing apparatus 1100 are only one example provided by the embodiments of the present application, and that the data processing system 10 and the data table heat distinguishing apparatus 1100 may have more or less components than those shown in fig. 8, may combine two or more components, or may have different configurations of components to implement.

The embodiment of the present application further provides a computing device cluster 20, where the computing device cluster 20 may be used to deploy the data processing system 10 shown in fig. 8, and specifically may be used to deploy the data table heat degree distinguishing device 1100 in the data processing system 10 shown in fig. 8, so as to execute the data table heat degree distinguishing method provided in the embodiment of the present application. As shown in fig. 9, the computing device cluster 20 includes at least one computing device 200.

Specifically, in the case where the computing device cluster 20 includes only one computing device 200, all of the modules in the data processing system 10 shown in fig. 8 may be deployed in the one computing device 200: service node 110, storage node 120, first storage 130, and second storage 140.

Where the computing device cluster 20 includes a plurality of computing devices 200, each computing device 200 in the plurality of computing devices 200 may be used to deploy some of the modules in the data processing system 10 shown in FIG. 8, or two or more computing devices 200 in the plurality of computing devices 200 may be used in common to deploy one or more of the modules in the data processing system 10 shown in FIG. 8.

By way of example, assuming that the plurality of computing devices 200 includes a computing device 200A and a computing device 200B, the computing device 200A may be configured to deploy the service node 110 and the storage node 120, and the computing device 200B may be configured to deploy the first storage apparatus 130 and the second storage apparatus 140, or the computing device 200A and the computing device 200B may be collectively configured to deploy the service node 110, for example, the obtaining module 1101 in the data table heat degree distinguishing apparatus 1100 is deployed on the computing device 200A, the processing module 1102 in the data table heat degree distinguishing apparatus 1100 is deployed on the computing device 200B, the computing device 200A is further configured to deploy the storage node, and the computing device 200B is further configured to deploy the first storage apparatus 130 and the second storage apparatus 140; assuming that the plurality of computing devices 200 includes computing devices 200A, 200B, 200C, and 200D, computing device 200A may be used to deploy service node 110, computing device 200B may be used to deploy storage node 120, computing device 200C may be used to deploy first storage 130, and computing device 200D may be used to deploy second storage 140.

In a specific implementation, all of the at least one computing device 200 included in the computing device cluster 20 may be a terminal device, or all of the computing device may be a cloud server, or a part of the computing device may be a cloud server, and a part of the computing device is a terminal device, which is not specifically limited herein.

More specifically, each computing device 200 in the computing device cluster 20 may include a processor, a memory, a communication interface, and the like, the memory in one or more computing devices 200 in the computing device cluster 20 may store the same codes (which may also be referred to as instructions or program instructions, and the like) for executing the data table thermal distinguishing method provided by the embodiment of the present application, the processor may read the codes from the memory and execute the codes to implement the data table thermal distinguishing method provided by the embodiment of the present application, and the communication interface may be used to implement communication between each computing device 200 and other devices.

In some possible implementations, each computing device 200 in the computing device cluster 20 may also communicate with other device connections over a network. Wherein the network may be a wide area network or a local area network, etc.

The computing device 200 provided by the embodiment of the present application and equipped with the data table heat degree distinguishing apparatus 1100 is described in detail below with reference to fig. 10.

Referring to fig. 10, a computing device 200 with a data table heat distinguishing apparatus 1100 deployed therein includes: a processor 210, a memory 220, and a communication interface 230, wherein the processor 210, the memory 220, and the communication interface 230 may be connected to each other through a bus 240. Wherein, the first and the second end of the pipe are connected with each other,

processor 210 may read code stored in memory 220 to perform some or all of the steps of the method for distinguishing hot spots of a data sheet performed by data sheet hot spot distinguishing apparatus 1100 in the above-described embodiments of the present application in cooperation with communication interface 230.

The processor 210 may have various specific implementation forms, for example, the processor 210 may be a Central Processing Unit (CPU) or a Graphics Processing Unit (GPU), and the processor 210 may also be a single-core processor or a multi-core processor. The processor 210 may be a combination of a CPU and a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof. The processor 210 may also be implemented as a single logic device with built-in processing logic, such as an FPGA or a Digital Signal Processor (DSP).

The memory 220 may store code as well as data. Wherein the code comprises: the code of the acquisition module 1101 and the code of the processing module 1102, etcThe data includes: intrinsic heat H of first data table ₀ Inherent heat of the second data table, and associated heat H of the first data table and the second data table ₁ And so on.

In practical applications, the memory 220 may be a non-volatile memory, such as a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash memory. Memory 220 may also be volatile memory, which may be Random Access Memory (RAM), that acts as external cache memory.

The communication interface 230 may be a wired interface (e.g., an ethernet interface) or a wireless interface (e.g., a cellular network interface or using a wireless local area network interface) for communicating with other computing nodes or devices. When the communication interface 230 is a wired interface, the communication interface 230 may adopt a protocol family over a transmission control protocol/internet protocol (TCP/IP), such as a Remote Function Call (RFC) protocol, a Simple Object Access Protocol (SOAP) protocol, a Simple Network Management Protocol (SNMP) protocol, a Common Object Request Broker Architecture (CORBA) protocol, a distributed protocol, and the like.

The bus 240 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 240 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 10, but this is not intended to represent only one bus or type of bus.

The above-mentioned method used by the computing device 200 to execute the method in the above-mentioned data table heat distinguishing method embodiment belongs to the same concept as the above-mentioned method embodiment, and its specific implementation process is described in the above-mentioned method embodiment, and is not described here again.

It should be understood that computing device 200 is only one example provided for the embodiments of the present application and that computing device 200 may have more or fewer components than shown in FIG. 10, may combine two or more components, or may have a different configuration implementation of components.

The present application further provides a non-transitory computer-readable storage medium, in which codes are stored, and when the codes are executed on a processor, the non-transitory computer-readable storage medium may implement part or all of the steps of the data table heat distinguishing method described in the foregoing embodiments.

It can be understood that as the number of data tables becomes larger and larger, a large number of databases, data systems, etc. may also appear, and enterprises need to manage a large number of data tables, as well as a large number of databases and data systems. Therefore, differentiating the heat of a large number of databases is bound to become a key ring for managing the large number of databases by an enterprise, and differentiating the heat of a large number of data systems is bound to become a key ring for managing the large number of data systems by the enterprise. The idea of the data table heat distinguishing method, device and related equipment provided by the application can be applied to not only management of a large number of data tables, but also distinguishing of heat of a large number of databases and a large number of data systems.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the above embodiments, all or part may be implemented by software, hardware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product may include code. When the computer program product is read and executed by a computer, part or all of the steps of the data table heat degree distinguishing method described in the above method embodiments may be implemented. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), optical medium, or semiconductor medium, among others.

The steps in the method of the embodiment of the application can be sequentially adjusted, combined or deleted according to actual needs; the units in the device of the embodiment of the application can be divided, combined or deleted according to actual needs.

The foregoing detailed description of the embodiments of the present application has been presented to illustrate the principles and implementations of the present application, and the above description of the embodiments is only provided to help understand the method and the core concept of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. A method for heat differentiation of a data table, the method comprising:

the service node acquires the associated heat degrees of the first data table and the second data table, wherein the associated heat degrees of the first data table and the second data table are acquired according to the inherent heat degree of the second data table and the association relation of the first data table and the second data table, and the inherent heat degree of the second data table is the heat degree generated by calling the second data table;

and the service node determines the heat degree of the first data table according to the associated heat degree of the first data table and the second data table.

2. The method of claim 1, wherein the service node obtaining a second data table associated with the first data table from the storage node, comprises:

the service node acquires the second data table having a data blood relationship with the first data table from the storage node, wherein the data blood relationship indicates that the second data table is obtained by calculation according to the first data table, or the first data table is obtained by calculation according to the second data table;

3. The method of claim 1, wherein the service node obtaining a second data table associated with the first data table from the storage node, comprises:

4. The method according to any one of claims 1 to 3, wherein the determining, by the service node, the heat degree of the first data table according to the associated heat degrees of the first data table and the second data table comprises:

5. The method according to any one of claims 1 to 4, further comprising:

the service node calculates the heat degree of the plurality of data tables;

6. The method according to any one of claims 1 to 5, further comprising:

the service node calculates the heat degrees of the plurality of data tables;

7. The method of any of claims 1 to 6, further comprising:

the service node calculates the heat degrees of the plurality of data tables;

8. The method of any one of claims 1 to 7, further comprising:

the service node calculates the heat degree of the plurality of data tables;

9. An apparatus for differentiating heat of data table, the apparatus being applied to a service node, the apparatus comprising:

the processing module is configured to acquire the associated heat of the first data table and the second data table, where the associated heat of the first data table and the second data table is acquired according to the inherent heat of the second data table and the association relationship between the first data table and the second data table, and the inherent heat of the second data table is the heat generated by calling the second data table;

10. The apparatus of claim 9,

the acquisition module is specifically configured to:

the processing module is specifically configured to:

11. The apparatus of claim 9,

the acquisition module is specifically configured to:

the processing module is specifically configured to:

12. The apparatus according to any one of claims 9 to 11, wherein the processing module is specifically configured to:

determining the heat degree of the first data table according to the inherent heat degree of the first data table and the associated heat degree of the first data table and the second data table, wherein the inherent heat degree of the first data table is the heat degree generated by calling the first data table.

13. The apparatus of any of claims 9 to 12, wherein the processing module is further configured to:

calculating the heat degree of the plurality of data tables;

14. The apparatus according to any one of claims 9 to 13, wherein the processing module is further configured to:

calculating the heat degree of the plurality of data tables;

15. The apparatus of any of claims 9 to 14, wherein the processing module is further configured to:

calculating the heat degree of the plurality of data tables;

16. A non-transitory computer-readable storage medium storing computer-readable instructions which, when executed, perform the method of any one of claims 1 to 8.

17. A cluster of computing devices comprising at least one computing device, each computing device comprising a processor and a memory;

the processor of the at least one computing device is to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any of claims 1-8.