CN113010514B - Thermal loading method and device - Google Patents
Thermal loading method and device Download PDFInfo
- Publication number
- CN113010514B CN113010514B CN202110224453.2A CN202110224453A CN113010514B CN 113010514 B CN113010514 B CN 113010514B CN 202110224453 A CN202110224453 A CN 202110224453A CN 113010514 B CN113010514 B CN 113010514B
- Authority
- CN
- China
- Prior art keywords
- loading
- data table
- data
- hot
- memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000011068 loading method Methods 0.000 title claims abstract description 406
- 238000000034 method Methods 0.000 claims abstract description 31
- 238000004590 computer program Methods 0.000 claims description 13
- 238000012827 research and development Methods 0.000 abstract description 10
- 230000007246 mechanism Effects 0.000 description 15
- 230000006870 function Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 9
- 239000000872 buffer Substances 0.000 description 8
- 238000012545 processing Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000007547 defect Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 230000009191 jumping Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a heat loading method and a heat loading device, which are suitable for the technical field of cloud computing, wherein the method comprises the following steps: acquiring loading information of loading data to be subjected to hot loading; if the hot loading identifier is determined to be the memory identifier, inquiring the memory according to the data table list to acquire loading data corresponding to the current data table list for loading; if the hot loading identifier is determined to be a distributed cache identifier, inquiring a distributed cache according to the data table list to acquire loading data corresponding to the current data table list for loading; and the loading data corresponding to the current data table list is not queried, and the loading data corresponding to the current data table list is read from the database according to the loading information so as to carry out hot loading. The invention automatically loads the data into the memory or the distributed cache to realize intelligent automatic hierarchical hot loading of the data, does not need a system developer to add codes or configuration information, improves the research and development efficiency, reduces the system risk and improves the balance of resource investment and performance.
Description
Technical Field
The invention relates to a data processing technology, in particular to a hot loading method and a hot loading device.
Background
In the prior art, general hot loading loads data into a memory or a distributed cache, newly added data table hot loading is complicated by writing codes or adding configuration information to realize hierarchical hot loading, small table hot loading into the memory cannot be automatically realized, large table hot loading into the distributed cache is realized, and the input-output ratio of resources and performance is improved.
Disclosure of Invention
In order to overcome at least one defect in the existing heat loading technology, the invention provides a heat loading method, which comprises the following steps:
acquiring loading information of loading data to be subjected to hot loading; wherein the loading information includes: the hot loading identification comprises a data table list and a hot loading identification, wherein the hot loading identification comprises the following components: a memory identifier and a distributed cache identifier;
if the hot loading identifier is determined to be a memory identifier, inquiring a memory according to the data table list to acquire loading data corresponding to the current data table list so as to carry out hot loading; if the hot loading identifier is determined to be a distributed cache identifier, inquiring a distributed cache according to the data table list to acquire loading data corresponding to the current data table list so as to carry out hot loading;
and the memory or the distributed cache does not inquire the loading data corresponding to the current data table list, and the loading data corresponding to the current data table list is read from the database according to the loading information so as to carry out hot loading.
In the embodiment of the present invention, the loading information further includes: the data type, the preset loading business rule and the data table occupation space threshold;
the loading business rule comprises the following steps: a preset access frequency threshold value and a data type loading strategy.
In the embodiment of the present invention, the obtaining loading information of loading data to be subjected to hot loading includes:
acquiring the data list and the data types from a storage device preset by a user;
and acquiring preset loading business rules from the loading configuration file by using the container.
In the embodiment of the present invention, the reading, according to the loading information, loading data corresponding to a current data table list from a database to perform hot loading includes:
reading loading data corresponding to the current data table list from the database according to the data table list;
determining the access frequency of the data table in the data table list to the database and the occupied space of the data table;
determining that the counted access frequency is greater than a preset access frequency threshold value and the occupied space of the data table is greater than a preset space threshold value, and loading corresponding data of the current data table into a distributed cache; and if the statistical access frequency is determined to be greater than the preset access frequency threshold and the occupied space of the data table is not greater than the preset space threshold, loading the corresponding data of the current data table into the memory.
In the embodiment of the present invention, the reading, according to the loading information, loading data corresponding to a current data table list from a database to perform hot loading includes:
determining a query data table according to the data type and a preset data type loading strategy;
and reading loading data corresponding to the current data table list from the database according to the determined query data table to carry out hot loading.
Meanwhile, the invention also provides a heat loading device, which comprises:
the loading information acquisition module is used for acquiring loading information of loading data to be subjected to hot loading; wherein the loading information includes: a data table list;
the hot loading module is used for determining that the hot loading identifier is a memory identifier, and inquiring a memory according to the data table list to acquire loading data corresponding to the current data table list so as to carry out hot loading; if the hot loading identifier is determined to be a distributed cache identifier, inquiring a distributed cache according to the data table list to acquire loading data corresponding to the current data table list so as to carry out hot loading;
and the asynchronous hot loading module is used for reading the loading data corresponding to the current data table list from the database according to the loading information so as to carry out hot loading, wherein the loading data corresponding to the current data table list is not queried in the memory or the distributed cache.
In an embodiment of the present invention, the loading information obtaining module includes:
the acquisition unit is used for acquiring the data list and the data types from a storage device preset by a user;
and the container loading unit is used for acquiring preset loading business rules from the loading configuration file by using the container.
In an embodiment of the present invention, the asynchronous hot loading module includes:
the database reading unit is used for reading the loading data corresponding to the current data table list from the database according to the data table list;
the statistics unit is used for counting the access frequency of the data table in the data table list to the database;
the cache unit is used for determining that the counted access frequency is greater than a preset access frequency threshold value, and the occupied space of the data table is greater than a preset space threshold value, and loading corresponding data of the current data table into the distributed cache; and if the statistical access frequency is determined to be greater than the preset access frequency threshold and the occupied space of the data table is not greater than the preset space threshold, loading the corresponding data of the current data table into the memory.
In an embodiment of the present invention, the asynchronous hot loading module includes:
the query table determining unit is used for determining a query data table according to the data type and a preset data type loading strategy;
and the loading unit is used for reading the loading data corresponding to the current data table list from the database according to the determined query data table so as to carry out hot loading.
The invention also provides a computer device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the computer program to realize the method.
Meanwhile, the invention also provides a computer readable storage medium which stores a computer program for executing the method.
The invention provides a hot loading method and a hot loading device, which are characterized in that loading information of loading data to be subjected to hot loading is obtained; if the hot loading identifier is determined to be a memory identifier, inquiring a memory according to the data table list to acquire loading data corresponding to the current data table list so as to carry out hot loading; if the hot loading identifier is determined to be a distributed cache identifier, inquiring a distributed cache according to the data table list to acquire loading data corresponding to the current data table list so as to carry out hot loading; and the memory or the distributed cache does not inquire the loading data corresponding to the current data table list, and the loading data corresponding to the current data table list is read from the database according to the loading information so as to carry out hot loading. Based on the mechanism, a data table list corresponding to an application and data type and table record size information are obtained and loaded into a memory, and data which are accessed frequently by transactions are automatically loaded into the memory or a distributed cache by adopting different strategies according to the data type and the table space occupation to realize intelligent automatic hierarchical hot loading of the data. The data table is newly added, the system can realize intelligent automatic hierarchical hot loading by self-adapting identification data, small tables are loaded into the memory, and large tables are loaded into the distributed cache, so that system developers do not need to add codes or configuration information, the research and development efficiency is improved, the system risk is reduced, and the balance of resource investment and performance is improved.
The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular description of preferred embodiments, as illustrated in the accompanying drawings.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method of thermal loading provided by the present invention;
FIG. 2 is a flowchart of a method for obtaining a data table list for a container start of an intelligent automatic hot loading mechanism according to an embodiment of the present invention;
FIG. 3 is a flow chart of transaction triggering automatic thermal loading of an intelligent automatic thermal loading mechanism according to an embodiment of the present invention;
FIG. 4 is a flow chart of asynchronous automatic thermal loading of an intelligent automatic thermal loading mechanism provided in an embodiment of the present invention;
FIG. 5 is a block diagram of a thermal loading apparatus provided by the present invention;
FIG. 6 is a block diagram of an embodiment of the present invention;
FIG. 7 is a block diagram of an embodiment of the present invention;
FIG. 8 is a block diagram of an embodiment of the present invention;
fig. 9 is a schematic diagram of an embodiment of an electronic device provided in an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the prior art, the hot loading of the data into the memory is realized by writing codes or adding configuration information to the newly added data table, so that the hot loading is complicated, and whether the newly added data table needs to be hot loaded or not can not be flexibly and automatically identified. In this regard, the present invention provides a method for loading heat, as shown in fig. 1, the method provided by the present invention includes:
step S1, loading information of loading data to be subjected to hot loading is obtained; wherein the loading information includes: the hot loading identification comprises a data table list and a hot loading identification, wherein the hot loading identification comprises the following components: a memory identifier and a distributed cache identifier;
step S2, determining that the hot loading identifier is a memory identifier, and inquiring a memory according to the data table list to acquire loading data corresponding to the current data table list so as to perform hot loading; if the hot loading identifier is determined to be a distributed cache identifier, inquiring a distributed cache according to the data table list to acquire loading data corresponding to the current data table list so as to carry out hot loading;
and step S3, the memory or the distributed cache does not inquire the loading data corresponding to the current data table list, and the loading data corresponding to the current data table list is read from the database according to the loading information so as to carry out hot loading.
According to the hot loading method provided by the invention, loading information of loading data to be subjected to hot loading is obtained; according to the hot loading identification in the recorded information, inquiring a memory or a distributed cache to acquire loading data corresponding to the current data table list for loading, and according to the loading information, reading the loading data corresponding to the current data table list from a database to perform hot loading, wherein the loading data corresponding to the current data table list is not inquired in the memory or the distributed cache.
According to the hot loading method provided by the embodiment of the invention, the data which is frequently accessed by transactions is automatically loaded into the memory or the distributed cache by adopting different strategies according to the data type and the table space occupation according to the data table list corresponding to the application and the data type and the table record size information, so that the intelligent automatic hierarchical hot loading of the data is realized. The data table is newly added, the system can realize intelligent automatic hierarchical hot loading by self-adapting identification data, small tables are loaded into the memory, and large tables are loaded into the distributed cache, so that system developers do not need to add codes or configuration information, the research and development efficiency is improved, the system risk is reduced, and the balance of resource investment and performance is improved. The method overcomes the defect that the hot loading of the newly added data table in the prior art is realized by writing codes or adding configuration information, and overcomes the defect that whether the newly added data table needs hot loading or not cannot be flexibly and automatically identified in the prior art.
Further, in the embodiment of the present invention, the loading information further includes: data type and preset loading business rule;
the loading business rule comprises the following steps: a preset access frequency threshold value and a data type loading strategy.
According to the hot loading method provided by the invention, the data which is accessed frequently by transaction is automatically loaded into the memory or the distributed cache by adopting different strategies according to the data type in the loading information and the preset loading business rule. The data table is newly added, the system can realize intelligent automatic hot loading by self-adapting identification data, and system developers are not required to add codes or configuration information, so that the research and development efficiency is improved, and the system risk is reduced.
Further, in the embodiment of the present invention, the step S1 of obtaining loading information of loading data to be hot loaded includes:
acquiring the data list and the data types from a storage device preset by a user;
and acquiring preset loading business rules from the loading configuration file by using the container.
Specifically, in an embodiment of the present invention, the method for starting the intelligent automatic thermal loading mechanism in the container to obtain the configuration information of the thermal loading specifically includes:
acquiring a data table and a data type list corresponding to the application from a storage device preset by a user; in one embodiment of the invention, the metadata and the software resource information management system are called as a storage device preset by a user;
loading the acquired data table and the data type list into a memory;
loading service rules such as access frequency, intelligent data type loading strategy and the like preset in the configuration file;
the container is successfully started.
The method comprises the steps of starting a container to obtain information to be subjected to hot loading, loading an obtained data table and a data type list into a memory, loading service rules such as preset access frequency, intelligent data type loading strategies and the like in a configuration file, obtaining loading data from a buffer pool and a database according to the configured service rules, and carrying out hot loading, thereby overcoming the defect that the hot loading of a newly added data table in the prior art is realized by writing codes or adding configuration information, and automatically loading data accessed frequently by transactions into the buffer pool according to the data types by adopting different strategies. According to the preset access frequency and data type intelligent loading strategy in the business rule, the newly added data table system can realize intelligent automatic hot loading by self-adaptively identifying data, and system developers are not required to add codes or configuration information, so that the research and development efficiency is improved and the system risk is reduced.
In the embodiment of the present invention, in step S3, the reading, according to the loading information, loading data corresponding to a current data table list from a database to perform hot loading includes:
reading loading data corresponding to the current data table list from the database according to the data table list;
determining the access frequency of the data table in the data table list to the database and the occupied space of the data table;
determining that the counted access frequency is greater than a preset access frequency threshold value and the occupied space of the data table is greater than a preset space threshold value, and loading corresponding data of the current data table into a distributed cache; and if the statistical access frequency is determined to be greater than the preset access frequency threshold and the occupied space of the data table is not greater than the preset space threshold, loading the corresponding data of the current data table into the memory.
Specifically, in the embodiment of the invention, when data is queried in a transaction, the system accumulates the access frequency in the distributed cache according to query SQL and minutes; judging whether the data is hot loaded or not according to the table data hot loading identification; if the identification is the memory, inquiring the memory to acquire inquired data; if the identification is the distributed cache, querying the distributed cache to acquire queried data, and if the queried data is not found, executing a data unloaded branch; if the identification is not loaded, the database is queried to acquire data, and if the access frequency reaches a threshold value, an asynchronous thread is started to trigger an automatic hot loading process. The automatic hot loading process obtains the data type and the table record size of the data table according to the name of the data table, and adopts different loading strategies according to the data type, the table record size and the table space: if the data table is a parameter table, acquiring the record number of the parameter table, calculating the table space occupation, if the space does not exceed a threshold value, loading the whole table into a memory, and if the space does exceed the threshold value, loading the whole table into a distributed cache; if the data table is a service table, only the data meeting the conditions is screened and loaded into the distributed cache. The hot loading identification of the table data is arranged in the memory, and the hot loading identification of the table data is distinguished from the hot loading of the data into the memory or the distributed cache.
In the embodiment of the present invention, the reading, according to the loading information, loading data corresponding to a current data table list from a database to perform hot loading includes:
determining a query data table according to the data type and a preset data type loading strategy;
and reading loading data corresponding to the current data table list from the database according to the determined query data table to carry out hot loading.
Specifically, in the embodiment of the present invention, the determined query data table includes: service table and parameter table.
In this embodiment, the table list and the data type information loaded in the memory are obtained.
Judging the data type corresponding to the data table, and inquiring all parameter data in the parameter table if the data type is the parameter table; and if the data type is service data, inquiring the data meeting the conditions in the service table.
And according to the statistics of the minutes, the access frequency of the data table to the database is obtained, and if the access frequency is larger than a preset threshold value, the sql is used as a key value, and the searched data is used as a value and is stored into a buffer pool.
The embodiment of the invention provides an intelligent automatic hot loading mechanism, based on the mechanism, a system automatically presets a storage device such as metadata and a software resource information management system, obtains a data table list corresponding to an application, data types and table record size information to be loaded into a memory, and automatically loads data frequently accessed by transactions into the memory or a distributed cache by adopting different strategies according to the data types and the table space occupation to realize intelligent automatic hierarchical hot loading of the data. The data table is newly added, the system can realize intelligent automatic hierarchical hot loading by self-adapting identification data, small tables are loaded into the memory, and large tables are loaded into the distributed cache, so that system developers do not need to add codes or configuration information, the research and development efficiency is improved, the system risk is reduced, and the balance of resource investment and performance is improved.
The technical scheme of the invention is further described in detail below.
In this embodiment, when the container is started, the system automatically acquires a data table list corresponding to the application from the metadata and software resource information management system, and loads the data type and table record size information into the memory.
When inquiring data, the system accumulates access frequency in the distributed cache according to the query SQL and minutes; judging whether the data is hot loaded or not according to the table data hot loading identification; if the identification is the memory, inquiring the memory to acquire inquired data; if the identification is the distributed cache, querying the distributed cache to acquire queried data, and if the queried data is not found, executing a data unloaded branch; if the identification is not loaded, the database is queried to acquire data, and if the access frequency reaches a threshold value, an asynchronous thread is started to trigger an automatic hot loading process. The automatic hot loading process obtains the data type and the table record size of the data table according to the name of the data table, and adopts different loading strategies according to the data type, the table record size and the table space: if the data table is a parameter table, acquiring the record number of the parameter table, calculating the table space occupation, if the space does not exceed a threshold value, loading the whole table into a memory, and if the space does exceed the threshold value, loading the whole table into a distributed cache; if the data table is a service table, only the data meeting the conditions is screened and loaded into the distributed cache. The hot loading identification of the table data is arranged in the memory, and the hot loading identification of the table data is distinguished from the hot loading of the data into the memory or the distributed cache.
According to the automatic hot loading mechanism provided by the embodiment, the system automatically acquires a data table list corresponding to the application and data type and table record size information from the metadata and software resource information management system, loads the data which are frequently accessed by transactions into the memory or the distributed cache by adopting different strategies according to the data type and the table space occupation, and achieves intelligent automatic hierarchical hot loading of the data. The data table is newly added, the system can realize intelligent automatic hierarchical hot loading by self-adapting identification data, small tables are loaded into the memory, and large tables are loaded into the distributed cache, so that system developers do not need to add codes or configuration information, the research and development efficiency is improved, the system risk is reduced, and the balance of resource investment and performance is improved.
As shown in fig. 2, a flowchart for acquiring a data table list at the start of a container by using an intelligent automatic hot loading mechanism according to an embodiment of the present invention includes:
s001: the container starts loading configuration information.
S002: and calling the metadata and software resource information management system to acquire a data table, a data type and table record size information corresponding to the application.
S003: and loading the acquired data table, the data type and the table record size information into a memory.
S004: and loading service rules such as access frequency, data types, table space intelligent loading strategies and the like preset in the configuration file.
S005: the container is successfully started.
In the embodiment of the invention, the preset business rules such as the access frequency, the data type, the table space intelligent loading strategy and the like are stored in the container loading configuration file, and the preset loading business rules are acquired from the loading configuration file by using the container, so that the defect that in the prior art, the hot loading of the newly-added data table is realized by writing codes or adding configuration information, the method is complicated, and whether the newly-added data table needs hot loading or not can not be flexibly and automatically identified is overcome.
As shown in fig. 3, in this embodiment, an intelligent automatic thermal loading mechanism transaction triggering automatic thermal loading flow chart is provided, which includes:
s101: a transaction request is received.
S102: and calling an intelligent automatic hot loading module to query data.
S103: the SQL statement is identified to obtain the queried data table and key value.
S104: the access frequency is counted in the distributed cache according to minutes.
S105: inquiring the hot loading identification of the memory acquisition table data, if the memory is the hot loading identification, jumping to S106; if the distributed cache is the distributed cache, the step S107 is skipped; otherwise, S109 is adjusted.
S106: and returning the data which is loaded in the memory.
S107: if the distributed cache finds data, the step S108 is skipped; otherwise, the process goes to S109.
S108: returning hot loaded data in distributed caches
S109: and directly inquiring the data table to acquire data.
S110: if the access frequency reaches the threshold, the process goes to S109; otherwise, the process goes to S111.
S111: and starting an asynchronous thread to perform automatic hot loading processing.
S112: the transaction continues to be processed.
FIG. 4 is a flowchart of implementing asynchronous automatic hot loading of an intelligent automatic hot loading mechanism according to an embodiment of the present invention, which includes:
s201: the thread acquires the loaded list of the table in the memory, the data type and the table record size information.
S202: judging the data type corresponding to the data table, and jumping to S203 if the data type is the parameter table; the data type is traffic data and the process goes to S204.
S203: and inquiring the record number of the parameter table in the database.
S204: calculating the occupied space of the table according to the record size and the record number of the table
S205: if the tablespace exceeds the threshold, otherwise jump to S206; yes, step S207 is skipped.
S206: and inquiring and loading all data of the parameter table into a memory, wherein the inquired data is value by taking sql as a key value.
S207: and querying and loading all data of the parameter table into a distributed cache, wherein the sql is used as a key value, and the queried data is a value.
S208: and data meeting the conditions in the query service table is loaded to the distributed cache.
S209: the hot loading identification of the table data is arranged in the memory, and the hot loading identification of the table data is distinguished from the hot loading of the data into the memory or the distributed cache.
The embodiment of the invention provides an intelligent automatic hot loading mechanism, wherein a system automatically acquires a data table list corresponding to an application from a metadata and software resource information management system, loads data types and table record size information into a memory, and automatically loads data frequently accessed by transactions into the memory or a distributed cache by adopting different strategies according to the data types and the table space occupation to realize intelligent automatic hierarchical hot loading of the data. The data table is newly added, the system can realize intelligent automatic hierarchical hot loading by self-adapting identification data, small tables are loaded into the memory, and large tables are loaded into the distributed cache, so that system developers do not need to add codes or configuration information, the research and development efficiency is improved, the system risk is reduced, and the balance of resource investment and performance is improved.
The present invention also provides a heat loading apparatus, as shown in fig. 5, comprising:
a loading information obtaining module 501, configured to obtain loading information of loading data to be subjected to hot loading; wherein the loading information includes: a data table list;
the hot loading module 502 is configured to determine that the hot loading identifier is a memory identifier, and query a memory according to the data table list to obtain loading data corresponding to the current data table list for loading; if the hot loading identifier is determined to be a distributed cache identifier, inquiring a distributed cache according to the data table list to acquire loading data corresponding to the current data table list for loading;
and the asynchronous hot loading module 503 is configured to, in the memory or the distributed cache, not query the loading data corresponding to the current data table list, and read the loading data corresponding to the current data table list from the database according to the loading information to perform hot loading.
In the embodiment of the invention, the data type and the preset loading business rule;
the loading business rule comprises the following steps: a preset access frequency threshold value and a data type loading strategy.
In an embodiment of the present invention, as shown in fig. 6, the loading information obtaining module 501 includes:
an obtaining unit 5011, configured to obtain the data table list and the data type from a storage device preset by a user;
the container loading unit 5012 is configured to obtain a preset loading business rule from the loading configuration file by using a container.
In an embodiment of the present invention, as shown in fig. 7, the asynchronous hot loading module 503 includes:
a database reading unit 5031, configured to read, from the database, the loading data corresponding to the current data table list according to the data table list;
a statistics unit 5032, configured to count the access frequency of the data table in the data table list to the database;
the caching unit 5033 is configured to determine that the counted access frequency is greater than a preset access frequency threshold, and the occupied space of the data table is greater than a preset space threshold, and load the corresponding data of the current data table into the distributed cache; and if the statistical access frequency is determined to be greater than the preset access frequency threshold and the occupied space of the data table is not greater than the preset space threshold, loading the corresponding data of the current data table into the memory.
In an embodiment of the present invention, as shown in fig. 8, the asynchronous hot loading module 503 further includes:
a lookup table determining unit 5034, configured to determine a lookup data table according to the data type and a preset data type loading policy;
the loading unit 5035 is configured to read, from the database, loading data corresponding to the current data table list according to the determined query data table, so as to perform hot loading.
The implementation manner of the heat loading device provided by the present invention will be clear to those skilled in the art from the foregoing description of the embodiments, and will not be described in detail herein.
It should be noted that, the heat loading method and the heat loading device provided by the invention are applicable to the technical field of cloud computing, realize heat loading, and also can be used in the financial field or other fields.
The present embodiment also provides an electronic device, which may be a desktop computer, a tablet computer, a mobile terminal, or the like, and the present embodiment is not limited thereto. In this embodiment, the electronic device may refer to the embodiments of the foregoing method and apparatus, and the content thereof is incorporated herein, and the repetition is not repeated.
Fig. 9 is a schematic block diagram of a system configuration of an electronic device 600 according to an embodiment of the present invention. As shown in fig. 9, the electronic device 600 may include a central processor 100 and a memory 140; memory 140 is coupled to central processor 100. Notably, the diagram is exemplary; other types of structures may also be used in addition to or in place of the structures to implement telecommunications functions or other functions.
In one embodiment, the hot load function may be integrated into the central processor 100. Wherein the central processor 100 may be configured to control as follows:
acquiring loading information of loading data to be subjected to hot loading; wherein the loading information includes: the hot loading identification comprises a data table list and a hot loading identification, wherein the hot loading identification comprises the following components: a memory identifier and a distributed cache identifier;
if the hot loading identifier is determined to be a memory identifier, inquiring a memory according to the data table list to acquire loading data corresponding to the current data table list for loading; if the hot loading identifier is determined to be a distributed cache identifier, inquiring a distributed cache according to the data table list to acquire loading data corresponding to the current data table list for loading;
and the memory or the distributed cache does not inquire the loading data corresponding to the current data table list, and the loading data corresponding to the current data table list is read from the database according to the loading information so as to carry out hot loading.
In the embodiment of the present invention, the loading information further includes: data type and preset loading business rule;
the loading business rule comprises the following steps: the access frequency threshold value, the data type loading strategy and the data table occupation space threshold value are preset.
Acquiring the data list and the data types from a storage device preset by a user;
and acquiring preset loading business rules from the loading configuration file by using the container.
In the embodiment of the present invention, the reading, according to the loading information, loading data corresponding to a current data table list from a database to perform hot loading includes:
reading loading data corresponding to the current data table list from the database according to the data table list;
determining the access frequency of the data table in the data table list to the database and the occupied space of the data table;
determining that the counted access frequency is greater than a preset access frequency threshold value and the occupied space of the data table is greater than a preset space threshold value, and loading corresponding data of the current data table into a distributed cache; and if the statistical access frequency is determined to be greater than the preset access frequency threshold and the occupied space of the data table is not greater than the preset space threshold, loading the corresponding data of the current data table into the memory.
In the embodiment of the present invention, the reading, according to the loading information, loading data corresponding to a current data table list from a database to perform hot loading includes:
determining a query data table according to the data type and a preset data type loading strategy;
and reading loading data corresponding to the current data table list from the database according to the determined query data table to carry out hot loading.
In another embodiment, the heat loading device may be configured separately from the central processor 100, for example, the heat loading device may be configured as a chip connected to the central processor 100, and the heat loading function is implemented by control of the central processor.
The electronic equipment provided by the embodiment of the invention realizes an intelligent automatic hot loading mechanism, based on the mechanism, the system automatically loads the data which is frequently accessed by transactions into a buffer pool by adopting different strategies according to the data type from the data table list corresponding to the acquired application and the data type loaded into the memory. The data table is newly added, the system can realize intelligent automatic hot loading by self-adapting identification data, and system developers are not required to add codes or configuration information, so that the research and development efficiency is improved, and the system risk is reduced.
As shown in fig. 9, the electronic device 600 may further include: a communication module 110, an input unit 120, an audio processing unit 130, a display 160, a power supply 170. It is noted that the electronic device 600 need not include all of the components shown in fig. 9; in addition, the electronic device 600 may further include components not shown in fig. 9, to which reference is made to the related art.
As shown in fig. 9, the central processor 100, sometimes also referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, which central processor 100 receives inputs and controls the operation of the various components of the electronic device 600.
The memory 140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information about failure may be stored, and a program for executing the information may be stored. And the central processor 100 can execute the program stored in the memory 140 to realize information storage or processing, etc.
The input unit 120 provides an input to the central processor 100. The input unit 120 is, for example, a key or a touch input device. The power supply 170 is used to provide power to the electronic device 600. The display 160 is used for displaying display objects such as images and characters. The display may be, for example, but not limited to, an LCD display.
The memory 140 may be a solid state memory such as Read Only Memory (ROM), random Access Memory (RAM), SIM card, or the like. But also a memory which holds information even when powered down, can be selectively erased and provided with further data, an example of which is sometimes referred to as EPROM or the like. Memory 140 may also be some other type of device. Memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application/function storage 142, the application/function storage 142 for storing application programs and function programs or a flow for executing operations of the electronic device 600 by the central processor 100.
The memory 140 may also include a data store 143, the data store 143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by the electronic device. The driver storage 144 of the memory 140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, address book applications, etc.).
The communication module 110 is a transmitter/receiver 110 that transmits and receives signals via an antenna 111. A communication module (transmitter/receiver) 110 is coupled to the central processor 100 to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, etc., may be provided in the same electronic device. The communication module (transmitter/receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and to receive audio input from the microphone 132 to implement usual telecommunication functions. The audio processor 130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 130 is also coupled to the central processor 100 so that sound can be recorded locally through the microphone 132 and so that sound stored locally can be played through the speaker 131.
The embodiment of the present invention also provides a computer-readable program, wherein the program, when executed in an electronic device, causes the computer to execute the heat loading method as described in the above embodiment in the electronic device.
The embodiment of the present invention also provides a storage medium storing a computer-readable program, wherein the computer-readable program causes a computer to execute the thermal loading described in the above embodiment in an electronic device.
The intelligent automatic hot loading mechanism is realized by the embodiment of the invention, the system automatically acquires a data table list corresponding to the application and data type and table record size information from the metadata and software resource information management system, loads the data which is accessed frequently by transaction into the memory or the distributed cache by adopting different strategies according to the data type and the table space occupation, and realizes intelligent automatic hierarchical hot loading of the data. The data table is newly added, the system can realize intelligent automatic hierarchical hot loading by self-adapting identification data, small tables are loaded into the memory, and large tables are loaded into the distributed cache, so that system developers do not need to add codes or configuration information, the research and development efficiency is improved, the system risk is reduced, and the balance of resource investment and performance is improved.
Preferred embodiments of the present invention are described above with reference to the accompanying drawings. The many features and advantages of the embodiments are apparent from the detailed specification, and thus, it is intended by the appended claims to cover all such features and advantages of the embodiments which fall within the true spirit and scope thereof. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the embodiments of the invention to the exact construction and operation illustrated and described, and accordingly, all suitable modifications and equivalents may be resorted to, falling within the scope thereof.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principles and embodiments of the present invention have been described in detail with reference to specific examples, which are provided to facilitate understanding of the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (6)
1. A method of thermal loading, said method comprising:
acquiring loading information of loading data to be subjected to hot loading; wherein the loading information includes: the hot loading identification comprises a data table list and a hot loading identification, wherein the hot loading identification comprises the following components: memory identification and distributed cache identification; the loading information further comprises: data type and preset loading business rule; the loading business rule comprises the following steps: the method comprises the steps of presetting an access frequency threshold, a data type loading strategy and a data table occupation space threshold;
if the hot loading identifier is determined to be a memory identifier, inquiring a memory according to the data table list to acquire loading data corresponding to the current data table list for loading; if the hot loading identifier is determined to be a distributed cache identifier, inquiring a distributed cache according to the data table list to acquire loading data corresponding to the current data table list for loading;
the memory or the distributed cache does not inquire the loading data corresponding to the current data table list, and the loading data corresponding to the current data table list is read from the database according to the loading information so as to carry out hot loading;
the obtaining loading information of the loading data to be subjected to hot loading comprises the following steps:
acquiring the data list and the data types from a storage device preset by a user;
acquiring preset loading business rules from a loading configuration file by using a container;
the step of reading the loading data corresponding to the current data table list from the database according to the loading information to perform hot loading comprises the following steps:
reading loading data corresponding to the current data table list from the database according to the data table list;
determining the access frequency of the data table in the data table list to the database and the occupied space of the data table;
determining that the counted access frequency is greater than a preset access frequency threshold value and the occupied space of the data table is greater than a preset space threshold value, and loading corresponding data of the current data table into a distributed cache; and if the statistical access frequency is determined to be greater than the preset access frequency threshold and the occupied space of the data table is not greater than the preset space threshold, loading the corresponding data of the current data table into the memory.
2. The method of claim 1, wherein the step of reading the loading data corresponding to the current data table list from the database according to the loading information to perform the hot loading comprises:
determining a query data table according to the data type and a preset data type loading strategy;
and reading loading data corresponding to the current data table list from the database according to the determined query data table to carry out hot loading.
3. A heat loading apparatus, said apparatus comprising:
the loading information acquisition module is used for acquiring loading information of loading data to be subjected to hot loading; wherein the loading information includes: the hot loading identification comprises a data table list and a hot loading identification, wherein the hot loading identification comprises the following components: memory identification and distributed cache identification; the loading information further comprises: data type and preset loading business rule; the loading business rule comprises the following steps: the method comprises the steps of presetting an access frequency threshold, a data type loading strategy and a data table occupation space threshold;
the hot loading module is used for determining that the hot loading identifier is a memory identifier, and inquiring a memory according to the data table list to acquire loading data corresponding to the current data table list for loading; if the hot loading identifier is determined to be a distributed cache identifier, inquiring a distributed cache according to the data table list to acquire loading data corresponding to the current data table list for loading;
the asynchronous hot loading module is used for reading the loading data corresponding to the current data table list from the database according to the loading information so as to carry out hot loading, wherein the loading data corresponding to the current data table list is not queried in the memory or the distributed cache;
the loading information acquisition module comprises:
the acquisition unit is used for acquiring the data list and the data types from a storage device preset by a user;
the container loading unit is used for acquiring preset loading business rules from the loading configuration file by using the container;
the asynchronous hot loading module comprises:
the database reading unit is used for reading the loading data corresponding to the current data table list from the database according to the data table list;
the statistics unit is used for counting the access frequency of the data table in the data table list to the database;
the cache unit is used for determining that the counted access frequency is greater than a preset access frequency threshold value, and the occupied space of the data table is greater than a preset space threshold value, and loading corresponding data of the current data table into the distributed cache; and if the statistical access frequency is determined to be greater than the preset access frequency threshold and the occupied space of the data table is not greater than the preset space threshold, loading the corresponding data of the current data table into the memory.
4. A thermal loading apparatus as defined in claim 3, wherein said asynchronous thermal loading module comprises:
the query table determining unit is used for determining a query data table according to the data type and a preset data type loading strategy;
and the loading unit is used for reading the loading data corresponding to the current data table list from the database according to the determined query data table so as to carry out hot loading.
5. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1 to 2 when executing the computer program.
6. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program which, when executed by a processor, causes the processor to carry out the method steps of any of claims 1 to 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110224453.2A CN113010514B (en) | 2021-03-01 | 2021-03-01 | Thermal loading method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110224453.2A CN113010514B (en) | 2021-03-01 | 2021-03-01 | Thermal loading method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113010514A CN113010514A (en) | 2021-06-22 |
| CN113010514B true CN113010514B (en) | 2024-02-20 |
Family
ID=76386884
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110224453.2A Active CN113010514B (en) | 2021-03-01 | 2021-03-01 | Thermal loading method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113010514B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114840140A (en) * | 2022-03-28 | 2022-08-02 | 阿里巴巴（中国）有限公司 | On-cloud data caching method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183389A (en) * | 2015-09-15 | 2015-12-23 | 北京金山安全软件有限公司 | Data hierarchical management method and device and electronic equipment |
| CN107102896A (en) * | 2016-02-23 | 2017-08-29 | 阿里巴巴集团控股有限公司 | A kind of operating method of multi-level buffer, device and electronic equipment |
| CN109726191A (en) * | 2018-12-12 | 2019-05-07 | 中国联合网络通信集团有限公司 | A kind of processing method and system across company-data, storage medium |
| CN111858676A (en) * | 2020-07-24 | 2020-10-30 | 政采云有限公司 | Data processing method and device |
| CN112015674A (en) * | 2020-08-27 | 2020-12-01 | 平安科技（深圳）有限公司 | Cache resource access method and device based on multiple hierarchies and computer equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106557486A (en) * | 2015-09-25 | 2017-04-05 | 阿里巴巴集团控股有限公司 | A kind of storage method and device of data |
-
2021
- 2021-03-01 CN CN202110224453.2A patent/CN113010514B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183389A (en) * | 2015-09-15 | 2015-12-23 | 北京金山安全软件有限公司 | Data hierarchical management method and device and electronic equipment |
| CN107102896A (en) * | 2016-02-23 | 2017-08-29 | 阿里巴巴集团控股有限公司 | A kind of operating method of multi-level buffer, device and electronic equipment |
| CN109726191A (en) * | 2018-12-12 | 2019-05-07 | 中国联合网络通信集团有限公司 | A kind of processing method and system across company-data, storage medium |
| CN111858676A (en) * | 2020-07-24 | 2020-10-30 | 政采云有限公司 | Data processing method and device |
| CN112015674A (en) * | 2020-08-27 | 2020-12-01 | 平安科技（深圳）有限公司 | Cache resource access method and device based on multiple hierarchies and computer equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113010514A (en) | 2021-06-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9201810B2 (en) | Memory page eviction priority in mobile computing devices | |
| CN110018902B (en) | Memory processing method and device, electronic equipment and computer readable storage medium | |
| CN109992523B (en) | Memory processing method and device, electronic equipment and computer readable storage medium | |
| CN108572789B (en) | Disk storage method and device, message pushing method and device and electronic equipment | |
| CN110018900B (en) | Memory processing method and device, electronic equipment and computer readable storage medium | |
| CN109240607B (en) | File reading method and device | |
| US8244213B2 (en) | Communication device and method for managing service sharing | |
| CN113010514B (en) | Thermal loading method and device | |
| CN115391356A (en) | Data processing method, device, equipment, medium and computer program product | |
| CN108647102B (en) | Service request processing method and device of heterogeneous system and electronic equipment | |
| CN114416178A (en) | Data access method, device and non-transitory computer readable storage medium | |
| CN104426926A (en) | Processing method and apparatus for regularly issued data | |
| CN115129791A (en) | Data compression storage method, device and equipment | |
| CN114428589B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN110780820A (en) | Method and device for determining continuous storage space, electronic equipment and storage medium | |
| CN107682892B (en) | Flow control method and device, computer equipment and computer readable storage medium | |
| CN114089912B (en) | Data processing method and device based on message middleware and storage medium | |
| CN101819589B (en) | Method and device for controlling file to be input into/output from cache | |
| CN113010513A (en) | Hot loading method and device | |
| CN110837499A (en) | Data access processing method and device, electronic equipment and storage medium | |
| CN110866003B (en) | Index value number estimation method and device and electronic equipment | |
| CN111444225B (en) | Universal index analysis method and device | |
| CN112612865A (en) | Document storage method and device based on elastic search | |
| CN113626396B (en) | Cluster detection method and device, electronic equipment, storage medium and product | |
| CN111177157B (en) | Account blacklist detection method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |