CN117349326A - Static data query method and system - Google Patents

Static data query method and system Download PDF

Info

Publication number
CN117349326A
CN117349326A CN202311210749.4A CN202311210749A CN117349326A CN 117349326 A CN117349326 A CN 117349326A CN 202311210749 A CN202311210749 A CN 202311210749A CN 117349326 A CN117349326 A CN 117349326A
Authority
CN
China
Prior art keywords
page
data
value
storage
index
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311210749.4A
Other languages
Chinese (zh)
Inventor
李文峰
李旭升
林学博
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Birui Data Technology Co ltd
Original Assignee
Beijing Birui Data Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Birui Data Technology Co ltd filed Critical Beijing Birui Data Technology Co ltd
Priority to CN202311210749.4A priority Critical patent/CN117349326A/en
Publication of CN117349326A publication Critical patent/CN117349326A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24553—Query execution of query operations
    • G06F16/24554—Unary operations; Data partitioning operations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2308—Concurrency control
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a static data query method and a system, which are applied to the fields of database technology and data visualization. The method comprises the following steps: reading static data from a persistent storage space, performing page storage on the static data according to a page data storage structure, counting, and constructing and initializing a static data query optimization model according to page values, page numbers, index page numbers and index values of all data pages formed after the page storage; based on the static data query optimization model, obtaining and returning result data from the data page according to the analysis result of the static data query request, and updating the page value and the index value according to the result data; and retraining the static data query optimization model according to the updated page value and the updated index value to obtain an optimized static data query optimization model, and dynamically optimizing all the data pages, so that the efficiency and performance of static data query are improved.

Description

Static data query method and system
Technical Field
The present application relates to the field of database technology and data visualization, and more particularly, to a static data query method and system.
Background
The static data is a static data set which is mainly used as control or reference in the running process, the data generally does not change along with the running, namely, the memory does not change for a long time, and the static data is mainly characterized by large, bounded and durable storage, and the static data is generally applied to the fields of historical bill inquiry, consumption record inquiry, economic growth trend analysis and prediction, user portraits, accurate marketing and the like.
In the prior art, the static data is processed mainly in a processing mode based on a traditional database and a processing mode based on a cache. When processing large-scale static data based on a traditional database, a hard disk (including a magnetic disk and a solid state disk) is mainly used as a physical medium, the processing mechanism is separated from storage and processing, namely, the large-scale static data is stored on the magnetic disk (or the solid state disk), when a user needs to inquire, the data is loaded into a memory in batches for processing, after one batch of data processing is finished, the memory space is released, the next batch of data is loaded for processing, and the inquiry result is fed back to the user until all the data are processed. When the processing mode processes large-scale static data, the disk is required to be frequently accessed to perform interactive read-write operation of the data, on one hand, the operation of the disk for reading and writing the data needs to be performed by mechanical movement of a magnetic head, on the other hand, the operation is influenced by time of system call (usually completed through CPU interruption and limited by CPU clock cycles), when the data volume is large, the operation is frequent and complex, the generated delay becomes obvious, the delay can linearly increase along with the increase of the data volume, and the inquiry performance is low and the response speed is slower. Although solid state disks with faster read and write operations are used to replace magnetic disks later, the improvement in query efficiency is also a fly-over.
In order to improve the query efficiency of the large-scale static data, a buffer pool is increased on the basis of the processing based on the traditional database, namely, a second mode for processing the large-scale static in the prior art: processing is performed based on the cache. When the static data is processed, a part of data in the large-scale static data set is firstly loaded into a buffer (namely a buffer pool), then the buffer is loaded into a memory, and the released buffer space is used for synchronously and dynamically loading the data from the large-scale static data set. Because the storage space of the cache is larger than the memory space, and the speed of read-write operation is faster than that of a magnetic disk and a hard disk, the processing mode based on the traditional database has a certain improvement on the processing efficiency. But the method is also limited by a processing mechanism that data is firstly stored on a disk or a hard disk and then loaded into a cache and a memory, and the method also needs frequent read-write operation during processing, so that the inquiry performance is still lower, the response speed is slower, the delay linearly grows along with the increase of the data quantity, and the delay is also obvious when the data scale is larger.
In summary, in the prior art, two ways of static data processing are to store and then load data, that is, data is stored on a hard disk first, and then is loaded into a memory in batches for processing during inquiry, after one batch of data processing is completed, the memory is released, and then the next batch of data is loaded, so that the read-write operation of the disk is frequently performed. Thus, large delay exists in large-scale static data processing, and the data query efficiency is difficult to improve all the time. Therefore, how to improve the data query efficiency of static data is a technical problem to be solved at present.
Based on the above, a new method and system are necessary to be introduced, the static data is paged and stored at high speed, and by constructing a static data query optimization model, the static data is dynamically optimized in real time according to the frequency of static data reading and accessing, so as to realize static data query optimization and acceleration, and solve the technical problem of low static data query efficiency caused by frequent data reading and writing and disk I/O bottleneck in the prior art, thereby dynamically optimizing the query performance of large-scale static data according to the requirement of query service and improving the efficiency and performance of static data query.
Disclosure of Invention
Aiming at the technical problems, the invention provides a static data query method and a system, which are used for carrying out paging high-speed storage on static data, carrying out dynamic optimization processing on the static data according to the frequency of static data reading and accessing by constructing a static data query optimization model, and realizing static data query optimization and acceleration so as to solve the technical problem of low static data query efficiency caused by frequent data reading and writing and disk I/O bottleneck in the prior art, thereby dynamically optimizing the query performance of large-scale static data according to the requirement of query service and improving the efficiency and performance of static data query.
The invention provides a static data query method, which comprises the following steps:
s101, reading static data from a persistent storage space, performing paging storage on the static data according to a page data storage structure, and counting page values, page numbers, index page numbers and index values of all data pages formed after the paging storage; s102, constructing and initializing a static data query optimization model according to the page value, the page number, the index page number and the index value of each data page; s103, receiving a static data query request based on the static data query optimization model, acquiring and returning result data from the data page according to the analysis result of the static data query request, and updating the page value and the index value according to the result data; s104, retraining the static data query optimization model according to the updated page value and the updated index value to obtain an optimized static data query optimization model, and dynamically optimizing all the data pages;
wherein the page data storage structure comprises: page name, page value, page field name, page parameters, page attributes, page index, storage domain address, storage domain size, dirty page identification; the page parameters include: field value, page size, data type, data length; the page attributes include: page creation time, page release status, data control, disk storage address.
As described above, in S101, the step of storing the page specifically includes: s1011, based on reading a data block containing static data from a persistent storage space and column names of a data table, acquiring the static data from the data block; s1012, according to the column names and the page data storage structure, dividing the static data into the data pages according to the page sizes and a preset storage mode, and adding the page names, the page field names, the page parameters and the page attributes; s1023, determining the storage domain size of the data page storage according to the page size, distributing the storage domain address, and storing the data page into a high-speed storage according to the storage domain address; s1024, initially setting the page value of the data page to be 1, identifying the dirty page to be null, creating the page index based on field names contained in all the data pages and page names of the data pages, and generating an index page corresponding to the data page; s1025, counting page values, page numbers, index page numbers and index values of all the data pages in real time; the preset storage mode comprises a column storage mode and a row storage mode, and the high-speed storage comprises a memory and a cache; and storing index data of the index page according to the preset storage mode.
As described above, in S102, the step of constructing and initializing the static data query optimization model includes: s1021, summing page values of all the data pages, namely, the sum of the page values of all the data pages is:s1022, according to the sum of page values of all said data pages +.>And constructing and initializing a static data query optimization model according to the page number, the index page number and the index value.
As described above, in step S102, the step of constructing and initializing the static data query optimization model further includes: setting the number Sp of pages, the number of index pages as Sindex, the index value as Dindex, the page name as x and the value of the static data query optimization model of each data page as F (x) The static data query optimization model isWherein n is an integer greater than or equal to 1.
As described above, the step of S102 includes: s1031, receiving the static data query request from an application end, analyzing the static data query request to obtain the analysis result, and selecting the page index according to the analysis result, wherein the analysis result comprises the page name, the page field name and the field value; s1032, traversing whether the result data exists in the high-speed storage according to the page index, the page name, the page field name and the field value, acquiring the result data according to the traversing result, returning the result data to the application end, and updating the page value and the index value.
As described above, the specific steps of S1032 are as follows: if the result data exists in the high-speed storage, 1) determining a storage domain address of the data page according to the page index, the page name, the page field name and the field value; 2) Obtaining the result data from the data page in the high-speed storage by using the storage domain address, and adding 1 to the page value and the index value respectively; if the result data does not exist in the high-speed storage, 1) determining a disk storage address according to the page name, the page field name and the field value; 2) Sending the disk storage address to a persistent storage space, determining a data block containing the result data, loading the data block to the high-speed storage, paging and storing the static data, and creating a corresponding page index; 3) Determining a storage domain address of the data page according to the corresponding page index, the page name, the page field name and the field value; 4) And acquiring the result data from the data page in the high-speed storage by using the storage domain address, and respectively adding 2 to the page value and the index value.
As described above, the specific steps of 104 include: s1041, training the static data query optimization model according to the updated page value and the updated index value to obtain the optimized static data query optimization model; s1042, calculating and obtaining the value of the static data query optimization model of each data page as F according to the optimized static data query optimization model (x) The method comprises the steps of carrying out a first treatment on the surface of the S1043 according toThe value of the static data query optimization model of each data page is F (x) The model optimization value is preset, and dynamic optimization processing is carried out on all the data pages; the preset model optimization value is set according to the requirements of the static data query service.
As described above, the specific steps of S1043 are as follows: if the value of the static data query optimization model of the data page is F (x) And if the data page is smaller than the preset model optimization value, writing the data page into the persistent storage from the high-speed storage, and deleting the data page from the high-speed storage.
Correspondingly, the invention also provides a static data query system, which comprises a paging storage module, a model construction module, a request processing module and a model optimization module;
the page storage module is used for reading static data from the persistent storage space, carrying out page storage on the static data according to the page data storage structure, and counting page values, page quantity, index page numbers and index values of all data pages formed after the page storage; the model construction module is used for constructing and initializing a static data query optimization model according to the page value, the page number, the index page number and the index value of each data page; the request processing module is used for receiving a static data query request based on the static data query optimization model, acquiring and returning result data from the data page according to the analysis result of the static data query request, and updating the page value and the index value according to the result data; the model optimization module is used for retraining the static data query optimization model according to the updated page value and the updated index value to obtain an optimized static data query optimization model, and dynamically optimizing all the data pages;
wherein the page data storage structure comprises: page name, page value, page field name, page parameters, page attributes, page index, storage domain address, storage domain size, dirty page identification; the page parameters include: field value, page size, data type, data length; the page attributes include: page creation time, page release status, data control, disk storage address. .
By applying the technical scheme, the invention realizes the paging high-speed storage of the static data, and by constructing the static data query optimization model, the static data is dynamically optimized in real time according to the frequency of static data reading and accessing, and the static data query optimization and acceleration are realized, so that the technical problem of low static data query efficiency caused by frequent data reading and writing and disk I/O bottlenecks in the prior art is solved, the query performance of large-scale static data is dynamically optimized according to the requirement of query service, and the efficiency and performance of static data query are improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 shows a flow diagram of a static data query method according to an embodiment of the present invention;
fig. 2 shows a schematic structural diagram of a static data query system according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
The invention provides a static data query method, as shown in figure 1, which comprises the following steps:
s101, reading static data from a persistent storage space, performing paging storage on the static data according to a page data storage structure, and counting page values, page numbers, index page numbers and index values of all data pages formed after the paging storage.
S102, constructing and initializing a static data query optimization model according to the page value, the page number, the index page number and the index value of each data page.
S103, based on the static data query optimization model, receiving a static data query request, acquiring and returning result data from the data page according to the analysis result of the static data query request, and updating the page value and the index value according to the result data.
S104, retraining the static data query optimization model according to the updated page value and the updated index value to obtain an optimized static data query optimization model, and carrying out dynamic optimization processing on all the data pages.
Wherein the page data storage structure comprises: page name, page value, page field name, page parameters, page attributes, page index, storage domain address, storage domain size, dirty page identification; the page parameters include: field value, page size, data type, data length; the page attributes include: page creation time, page release status, data control, disk storage address.
In this embodiment, the step of paging storage specifically includes:
s1011, based on reading a data block containing static data from a persistent storage space and column names of a data table, acquiring the static data from the data block;
s1012, according to the column names and the page data storage structure, dividing the static data into the data pages according to the page sizes and a preset storage mode, and adding the page names, the page field names, the page parameters and the page attributes;
s1023, determining the storage domain size of the data page storage according to the page size, distributing the storage domain address, and storing the data page into a high-speed storage according to the storage domain address;
s1024, initially setting the page value of the data page to be 1, identifying the dirty page to be null, creating the page index based on field names contained in all the data pages and page names of the data pages, and generating an index page corresponding to the data page;
s1025, counting page values, page numbers, index page numbers and index values of all the data pages in real time;
wherein,
the preset storage mode comprises a column storage mode and a row storage mode, and the high-speed storage comprises a memory and a cache;
and storing index data of the index page according to the preset storage mode.
In this embodiment, in S102, the step of constructing and initializing the static data query optimization model includes:
s1021, summing page values of all the data pages, namely, the sum of the page values of all the data pages is:
s1022, according to the sum of page values of all the data pagesAnd constructing and initializing a static data query optimization model according to the page number, the index page number and the index value.
In this embodiment, in step S102, the step of constructing and initializing the static data query optimization model further includes:
setting the number Sp of pages, the number of index pages as Sindex, the index value as Dindex, the page name as x and the value of the static data query optimization model of each data page as F (x) The static data query optimization model is
Wherein n is an integer greater than or equal to 1.
In this embodiment, the step of S102 includes:
s1031, receiving the static data query request from an application end, analyzing the static data query request to obtain the analysis result, and selecting the page index according to the analysis result, wherein the analysis result comprises the page name, the page field name and the field value;
s1032, traversing whether the result data exists in the high-speed storage according to the page index, the page name, the page field name and the field value, acquiring the result data according to the traversing result, returning the result data to the application end, and updating the page value and the index value.
In this embodiment, the specific steps of S1032 are as follows:
if the result data exists in the high-speed storage, then
1) Determining a storage domain address of the data page according to the page index, the page name, the page field name and the field value;
2) Obtaining the result data from the data page in the high-speed storage by using the storage domain address, and adding 1 to the page value and the index value respectively;
if the result data does not exist in the high-speed storage, then
1) Determining a disk storage address according to the page name, the page field name and the field value;
2) Sending the disk storage address to a persistent storage space, determining a data block containing the result data, loading the data block to the high-speed storage, paging and storing the static data, and creating a corresponding page index;
3) Determining a storage domain address of the data page according to the corresponding page index, the page name, the page field name and the field value;
4) And acquiring the result data from the data page in the high-speed storage by using the storage domain address, and respectively adding 2 to the page value and the index value.
In this embodiment, the specific steps of 104 include:
s1041, training the static data query optimization model according to the updated page value and the updated index value to obtain the optimized static data query optimization model;
s1042, calculating and obtaining the value of the static data query optimization model of each data page as F according to the optimized static data query optimization model (x) ；
S1043, according to the static data query optimization model of each data page, the value of the static data query optimization model is F (x) The model optimization value is preset, and dynamic optimization processing is carried out on all the data pages;
the preset model optimization value is set according to the requirements of the static data query service.
In this embodiment, the specific steps of S1043 are as follows:
if the value of the static data query optimization model of the data page is F (x) And if the data page is smaller than the preset model optimization value, writing the data page into the persistent storage from the high-speed storage, and deleting the data page from the high-speed storage.
By applying the technical scheme, static data are read from the persistent storage space, paging storage is carried out on the static data according to the page data storage structure, and page values, page quantity, index page numbers and index values of all data pages formed after the paging storage are counted; constructing and initializing a static data query optimization model according to the page value, the page number, the index page number and the index value of each data page; based on the static data query optimization model, receiving a static data query request, acquiring and returning result data from the data page according to an analysis result of the static data query request, and updating the page value and the index value according to the result data; according to the updated page value and the updated index value, retraining the static data query optimization model to obtain an optimized static data query optimization model, and dynamically optimizing all the data pages, so that the static data is paged and stored at high speed, and the static data is dynamically optimized according to the reading and accessing frequency of the static data by constructing the static data query optimization model, and the static data query optimization and acceleration are realized, so that the technical problem of low static data query efficiency caused by frequent reading and writing of data and disk I/O bottlenecks in the prior art is solved, the query performance of large-scale static data is dynamically optimized according to the requirement of query service, and the efficiency and performance of static data query are improved.
Corresponding to one of the static data query methods in the embodiments of the present invention, the present invention also discloses a static data query system, as shown in fig. 2, where the system includes a paging storage module, a model construction module, a request processing module, and a model optimization module;
wherein,
the paging storage module is used for reading static data from the persistent storage space, paging storage is carried out on the static data according to the page data storage structure, and page values, page quantity, index page numbers and index values of all data pages formed after the paging storage are counted;
the model construction module is used for constructing and initializing a static data query optimization model according to the page value, the page number, the index page number and the index value of each data page;
the request processing module is used for receiving a static data query request based on the static data query optimization model, acquiring and returning result data from the data page according to the analysis result of the static data query request, and updating the page value and the index value according to the result data;
the model optimization module is used for retraining the static data query optimization model according to the updated page value and the updated index value to obtain an optimized static data query optimization model, and dynamically optimizing all the data pages;
wherein,
the page data storage structure includes: page name, page value, page field name, page parameters, page attributes, page index, storage domain address, storage domain size, dirty page identification;
the page parameters include: field value, page size, data type, data length;
the page attributes include: page creation time, page release status, data control, disk storage address.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims (9)

1. A static data query method, wherein the method is applied to a big data visualization data processing platform, and the method comprises the following steps:
s101, reading static data from a persistent storage space, performing paging storage on the static data according to a page data storage structure, and counting page values, page numbers, index page numbers and index values of all data pages formed after the paging storage;
s102, constructing and initializing a static data query optimization model according to the page value, the page number, the index page number and the index value of each data page;
s103, receiving a static data query request based on the static data query optimization model, acquiring and returning result data from the data page according to the analysis result of the static data query request, and updating the page value and the index value according to the result data;
s104, retraining the static data query optimization model according to the updated page value and the updated index value to obtain an optimized static data query optimization model, and dynamically optimizing all the data pages;
wherein,
the page data storage structure includes: page name, page value, page field name, page parameters, page attributes, page index, storage domain address, storage domain size, dirty page identification;
the page parameters include: field value, page size, data type, data length;
the page attributes include: page creation time, page release status, data control, disk storage address.
2. The method as recited in claim 1, wherein in S101, the step of paging the memory is specifically:
s1011, based on reading a data block containing static data from a persistent storage space and column names of a data table, acquiring the static data from the data block;
s1012, according to the column names and the page data storage structure, dividing the static data into the data pages according to the page sizes and a preset storage mode, and adding the page names, the page field names, the page parameters and the page attributes;
s1023, determining the storage domain size of the data page storage according to the page size, distributing the storage domain address, and storing the data page into a high-speed storage according to the storage domain address;
s1024, initially setting the page value of the data page to be 1, identifying the dirty page to be null, creating the page index based on field names contained in all the data pages and page names of the data pages, and generating an index page corresponding to the data page;
s1025, counting page values, page numbers, index page numbers and index values of all the data pages in real time;
wherein,
the preset storage mode comprises a column storage mode and a row storage mode, and the high-speed storage comprises a memory and a cache;
and storing index data of the index page according to the preset storage mode.
3. The method of claim 1, wherein in S102, the step of constructing and initializing a static data query optimization model comprises:
s1021, summing page values of all the data pages, namely, the sum of the page values of all the data pages is:
s1022, according to the sum of page values of all the data pagesAnd constructing and initializing a static data query optimization model according to the page number, the index page number and the index value.
4. The method of claim 1, wherein in step S102, the step of constructing and initializing a static data query optimization model further comprises:
setting the number Sp of pages, the number of index pages as Sindex, the index value as Dindex, the page name as x and the value of the static data query optimization model of each data page as F (x) The static data query optimization model is
Wherein n is an integer greater than or equal to 1.
5. The method of claim 1, wherein the step of S102 comprises:
s1031, receiving the static data query request from an application end, analyzing the static data query request to obtain the analysis result, and selecting the page index according to the analysis result, wherein the analysis result comprises the page name, the page field name and the field value;
s1032, traversing whether the result data exists in the high-speed storage according to the page index, the page name, the page field name and the field value, acquiring the result data according to the traversing result, returning the result data to the application end, and updating the page value and the index value.
6. The method according to claim 5, wherein the specific steps of S1032 are as follows:
if the result data exists in the high-speed storage, then
1) Determining a storage domain address of the data page according to the page index, the page name, the page field name and the field value;
2) Obtaining the result data from the data page in the high-speed storage by using the storage domain address, and adding 1 to the page value and the index value respectively;
if the result data does not exist in the high-speed storage, then
1) Determining a disk storage address according to the page name, the page field name and the field value;
2) Sending the disk storage address to a persistent storage space, determining a data block containing the result data, loading the data block to the high-speed storage, paging and storing the static data, and creating a corresponding page index;
3) Determining a storage domain address of the data page according to the corresponding page index, the page name, the page field name and the field value;
4) And acquiring the result data from the data page in the high-speed storage by using the storage domain address, and respectively adding 2 to the page value and the index value.
7. The method of claim 1, wherein the step of 104 comprises:
s1041, training the static data query optimization model according to the updated page value and the updated index value to obtain the optimized static data query optimization model;
s1042, calculating and obtaining the value of the static data query optimization model of each data page as F according to the optimized static data query optimization model (x) ；
S1043, according to the static data query optimization model of each data page, the value of the static data query optimization model is F (x) The model optimization value is preset, and dynamic optimization processing is carried out on all the data pages;
the preset model optimization value is set according to the requirements of the static data query service.
8. The method of claim 7, wherein the specific steps of S1043 are:
if the value of the static data query optimization model of the data page is F (x) And if the data page is smaller than the preset model optimization value, writing the data page into the persistent storage from the high-speed storage, and deleting the data page from the high-speed storage.
9. A system for implementing the static data query method of claim 1, wherein the system comprises a paged storage module, a model building module, a request processing module and a model optimization module;
wherein,
the paging storage module is used for reading static data from the persistent storage space, paging storage is carried out on the static data according to the page data storage structure, and page values, page quantity, index page numbers and index values of all data pages formed after the paging storage are counted;
the model construction module is used for constructing and initializing a static data query optimization model according to the page value, the page number, the index page number and the index value of each data page;
the request processing module is used for receiving a static data query request based on the static data query optimization model, acquiring and returning result data from the data page according to the analysis result of the static data query request, and updating the page value and the index value according to the result data;
the model optimization module is used for retraining the static data query optimization model according to the updated page value and the updated index value to obtain an optimized static data query optimization model, and dynamically optimizing all the data pages;
wherein,
the page data storage structure includes: page name, page value, page field name, page parameters, page attributes, page index, storage domain address, storage domain size, dirty page identification;
the page parameters include: field value, page size, data type, data length;
the page attributes include: page creation time, page release status, data control, disk storage address.
CN202311210749.4A 2023-09-19 2023-09-19 Static data query method and system Pending CN117349326A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311210749.4A CN117349326A (en) 2023-09-19 2023-09-19 Static data query method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311210749.4A CN117349326A (en) 2023-09-19 2023-09-19 Static data query method and system

Publications (1)

Publication Number Publication Date
CN117349326A true CN117349326A (en) 2024-01-05

Family

ID=89370183

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311210749.4A Pending CN117349326A (en) 2023-09-19 2023-09-19 Static data query method and system

Country Status (1)

Country Link
CN (1) CN117349326A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117971905A (en) * 2024-04-01 2024-05-03 华能曲阜热电有限公司 Caching and indexing method for real-time statistics of historical data of industrial production process

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117971905A (en) * 2024-04-01 2024-05-03 华能曲阜热电有限公司 Caching and indexing method for real-time statistics of historical data of industrial production process
CN117971905B (en) * 2024-04-01 2024-06-11 华能曲阜热电有限公司 Caching and indexing method for real-time statistics of historical data of industrial production process

Similar Documents

Publication Publication Date Title
JP6639420B2 (en) Method for flash-optimized data layout, apparatus for flash-optimized storage, and computer program
US8108355B2 (en) Providing a partially sorted index
US9736270B2 (en) Automated client/server operation partitioning
US20090210445A1 (en) Method and system for optimizing data access in a database using multi-class objects
JP5460486B2 (en) Apparatus and method for sorting data
CN107391544B (en) Processing method, device and equipment of column type storage data and computer storage medium
CN111881135A (en) Data aggregation method, device, equipment and computer readable storage medium
CN108139872A (en) A kind of buffer memory management method, cache controller and computer system
CN117349326A (en) Static data query method and system
CN113268457B (en) Self-adaptive learning index method and system supporting efficient writing
CN111159176A (en) Method and system for storing and reading mass stream data
CN106897311B (en) Database batch updating method, data reduction log generating method and storage device
CN106201918B (en) A kind of method and system based on big data quantity and extensive caching quick release
CN114489475A (en) Distributed storage system and data storage method thereof
JP2020144737A (en) Information processor and task management program
CN111444154B (en) Method for efficiently storing log files in single chip microcomputer system
CN110874360A (en) Ordered queue caching method and device based on fixed capacity
CN108897618B (en) Resource allocation method based on task perception under heterogeneous memory architecture
CN106909623A (en) A kind of data set and date storage method of supporting efficient mass data to analyze and retrieve
US6584518B1 (en) Cycle saving technique for managing linked lists
US11055622B2 (en) Data storage apparatus with selective adaptive predictive behavior
US11487731B2 (en) Read iterator for pre-fetching nodes of a B-tree into memory
CN107193487B (en) A kind of random access recognition methods and system
CN113487026B (en) Method and system for efficiently reading data by IO node in graph calculation
Agullo et al. A preliminary out-of-core extension of a parallel multifrontal solver

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination