CN118132553A - Pre-aggregate storage table creation method and device, electronic equipment and storage medium - Google Patents
Pre-aggregate storage table creation method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN118132553A CN118132553A CN202410139594.8A CN202410139594A CN118132553A CN 118132553 A CN118132553 A CN 118132553A CN 202410139594 A CN202410139594 A CN 202410139594A CN 118132553 A CN118132553 A CN 118132553A
- Authority
- CN
- China
- Prior art keywords
- data
- time
- aggregation
- type
- storage table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000002776 aggregation Effects 0.000 claims abstract description 312
- 238000004220 aggregation Methods 0.000 claims abstract description 312
- 238000005192 partition Methods 0.000 claims description 78
- 238000004590 computer program Methods 0.000 claims description 22
- 230000011218 segmentation Effects 0.000 claims description 18
- 238000012512 characterization method Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 abstract description 2
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000000638 solvent extraction Methods 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to the technical field of data processing, and discloses a method and a device for creating a pre-aggregate storage table, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring the data type and the data identification of each to-be-processed table data in an original storage table; acquiring index data of each non-secondary aggregation type in the original storage table according to the data type and the data identification of each table data to be processed; acquiring a pre-aggregation storage type of index data of each non-secondary aggregation type; the original storage type is replaced by the pre-aggregation storage type of index data of each non-secondary aggregation type in the original storage table, and a first pre-aggregation storage table is created. The application can replace index data of a non-secondary aggregation type in the original storage table with index data of a secondary aggregation type, so that the storage line number of the created pre-aggregation storage table is obviously reduced, the data volume of the pre-aggregation storage table is reduced, and the query performance of the pre-aggregation storage table is improved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method and apparatus for creating a pre-aggregated storage table, an electronic device, and a storage medium.
Background
In the prior art, when the DAU index is faced, the user_id is mostly taken as a dimension, so that when in visualization, the user_id is subjected to count (distinct user _id) to carry out de-duplication summation, thereby obtaining an accurate result, and as the number of the dimension base of the user_id is large, the number of the final aggregation table is obviously increased, so that the number of combinations of the dimension is exponentially increased, thereby causing the data volume of the aggregation table to be overlarge, leading to the obvious reduction of the query performance aiming at the aggregation table, and directly influencing the data analysis application and the data query.
Disclosure of Invention
The present application is directed to solving at least one of the technical problems existing in the related art. Therefore, the embodiment of the application provides a method, a device, electronic equipment and a storage medium for creating a pre-aggregate storage table, which can reduce the data volume of the pre-aggregate storage table and improve the query performance of the pre-aggregate storage table.
In a first aspect, an embodiment of the present application provides a method for creating a pre-aggregate storage table, including:
Acquiring the data type and the data identification of each to-be-processed table data in an original storage table; the data identification represents whether the table data is index data; the data type characterization table data is of a secondary aggregation type;
Acquiring index data of each non-secondary aggregation type in the original storage table according to the data type and the data identification of each to-be-processed table data;
Acquiring a pre-aggregation storage type of index data of each non-secondary aggregation type;
And replacing the original storage type of the index data of each non-secondary aggregation type with the pre-aggregation storage type of the index data of each non-secondary aggregation type in the original storage table, and creating a first pre-aggregation storage table.
In an alternative embodiment, replacing the original storage type of each index data of the non-secondary aggregation type with the pre-aggregation storage type of each index data of the non-secondary aggregation type in the original storage table to obtain a first pre-aggregation storage table, and then further including:
Acquiring a time partition strategy of the first pre-aggregate storage table; the time partition policy characterizes a time granularity of partition management of the first pre-aggregate storage table;
Creating time field table data according to the time partition strategy;
And inserting the time field table data into the first pre-aggregation storage table, and creating a second pre-aggregation storage table.
In an alternative embodiment, the inserting the time field table data into the first pre-aggregate storage table, after creating the second pre-aggregate storage table, further includes:
acquiring data to be stored; the data to be stored at least comprises time attribute information;
storing the data to be stored into a time partition corresponding to the second pre-aggregation storage table according to the time partition strategy and the time attribute information in the data to be stored to obtain a pre-aggregation data table;
and storing the pre-aggregation data table into a pre-aggregation database.
In an alternative embodiment, after storing the pre-aggregate data table in the time sequence database, the method further includes:
responding to a data query instruction, and acquiring a time sequence in the data query instruction;
Performing time sequence segmentation on the time sequence according to the time partition strategy to obtain a plurality of time subsequences;
And inquiring a pre-aggregated data table in the pre-aggregated database according to the plurality of time subsequences to obtain a target inquiring result of the data inquiring instruction.
In an optional embodiment, querying the pre-aggregated data table in the pre-aggregated database according to the plurality of time subsequences to obtain the target query result of the data query instruction, including:
Inquiring a pre-aggregation data table in the pre-aggregation database through a first time subsequence to obtain a first pre-aggregation inquiry result;
Starting from the second time sub-sequence, judging whether the starting time of the current time sub-sequence is the dividing time or not through a preset time sequence segmentation logic, and obtaining a judging result;
Inquiring a pre-aggregation data table in the pre-aggregation database based on the judging result until all time subsequences are traversed, and obtaining a second pre-aggregation inquiring result;
and determining the first pre-aggregated query result and the second pre-aggregated query result as target query results of the data query instruction.
In an alternative embodiment, starting from the second time sub-sequence, determining whether the start time of the current time sub-sequence is the split time by the preset time sequence splitting logic, to obtain a determination result, including:
If the starting time of the current time sub-sequence is the dividing time through the preset time sequence dividing logic from the second time sub-sequence, a first judging result is generated;
If the starting time of the current time sub-sequence is not the dividing time through the preset time sequence dividing logic from the second time sub-sequence, generating a second judging result;
The first judgment result and the second judgment result are different judgment results.
In an optional embodiment, querying the pre-aggregate data table in the pre-aggregate database based on the determination result until all the time sub-sequences are traversed, to obtain a second pre-aggregate query result, including:
if the judging result is the first judging result, inquiring a pre-aggregation data table in the pre-aggregation database through the second time subsequence until all the time subsequences are traversed, and obtaining a second pre-aggregation inquiring result; or alternatively, the first and second heat exchangers may be,
And if the judging result is the second judging result, shifting the starting time of the current time sub-sequence by a preset time length to obtain a third time sub-sequence, and inquiring a pre-aggregation data table in the pre-aggregation database through the third time sub-sequence until all the time sub-sequences are traversed to obtain a second pre-aggregation inquiring result.
In a second aspect, an embodiment of the present application provides a device for creating a pre-aggregate storage table, including:
the first acquisition module is used for acquiring the data type and the data identification of each to-be-processed table data in the original storage table; the data identification represents whether the table data is index data; the data type characterization table data is of a secondary aggregation type;
the second acquisition module is used for acquiring index data of each non-secondary aggregation type in the original storage table according to the data type and the data identification of each table data to be processed; the index data of the non-secondary aggregation type is index data of the non-secondary aggregation type;
the third acquisition module is used for acquiring the pre-aggregation storage type of the index data of each non-secondary aggregation type;
and the storage table creation module is used for replacing the original storage type of the index data of each non-secondary aggregation type with the pre-aggregation storage type of the index data of each non-secondary aggregation type in the original storage table to create a first pre-aggregation storage table.
In a third aspect, embodiments of the present application further provide an electronic device, including a memory storing a plurality of computer programs; the processor loads a computer program from the memory to perform any of the pre-aggregate storage table creation methods provided by the embodiments of the present application.
In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, where a plurality of computer programs are stored, where the computer programs are adapted to be loaded by a processor to execute any one of the methods for creating a pre-aggregate storage table provided by the embodiments of the present application.
In a fifth aspect, an embodiment of the present application further provides a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, implements any one of the pre-aggregate storage table creating methods provided in the embodiment of the present application.
According to the embodiment of the application, index data of a non-secondary aggregation type in the original storage table can be replaced by index data of a secondary aggregation type, namely, the non-accumulated index data is converted into accumulated index data, so that the storage line number of the created pre-aggregation storage table is obviously reduced, the data volume of the pre-aggregation storage table is reduced, and the query performance of the pre-aggregation storage table is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a method for creating a pre-aggregate storage table according to an embodiment of the present application;
FIG. 2 is a second flow chart of a method for creating a pre-aggregate memory table according to an embodiment of the present application;
FIG. 3 is a third flow chart of a method for creating a pre-aggregate memory table according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a structure of a pre-aggregate storage table creating apparatus provided in an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to fall within the scope of the application. Meanwhile, in the description of the embodiments of the present application, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance. Thus, features defining "first", "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "plurality" is two or more, unless explicitly defined otherwise.
The embodiment of the application provides a method and a device for creating a pre-aggregate storage table, electronic equipment and a storage medium. Specifically, the embodiment of the present application will be described from the viewpoint of a pre-aggregate storage table creation apparatus that may be integrated in an electronic device in particular, that is, the method of pre-aggregate storage table creation of the embodiment of the present application may be executed by the electronic device. Optionally, the electronic device includes a terminal device. The terminal device may be a mobile phone, a tablet computer, a notebook computer, a game console, a personal computer (Personal Computer, PC), or an intelligent interactive tablet. Optionally, the electronic device includes a server, which may be a stand alone server, or may be a server network or a server cluster including, but not limited to, a computer, a network host, a single network server, a network server set, or a cloud server formed by servers. Wherein the Cloud server is composed of a large number of computers or web servers based on Cloud Computing (Cloud Computing).
The following description of the embodiments is not intended to limit the preferred embodiments. Although a logical order is depicted in the flowchart, in some cases the steps shown or described may be performed in an order different than depicted in the figures.
In the scene of a storage table, when the DAU index is faced, the user_id is mostly taken as a dimension, so that when the user_id is visualized, the user_id is subjected to count (distinct user _id) to carry out de-duplication summation, an accurate result is obtained, the number of final aggregation table rows is obviously increased due to the fact that the number of dimension bases of the user_id is very large, the number of combinations of the dimensions is exponentially increased, the data volume of the aggregation table is excessively large, the query performance of the aggregation table is obviously reduced, and the data analysis application and the data query are directly affected.
In order to solve the problems in the prior art, the embodiment of the application acquires the data type and the data identification of each to-be-processed table data in the original storage table; acquiring index data of each non-secondary aggregation type in the original storage table according to the data type and the data identification of each table data to be processed; acquiring a pre-aggregation storage type of index data of each non-secondary aggregation type; the method comprises the steps of replacing the original storage type of index data of each non-secondary aggregation type with the pre-aggregation storage type of index data of each non-secondary aggregation type in the original storage table, and creating a first pre-aggregation storage table. Therefore, the embodiment of the application can replace index data of a non-secondary aggregation type in the original storage table with index data of a secondary aggregation type, namely, the non-accumulated index data is converted into accumulated index data, so that the storage line number of the created pre-aggregation storage table is obviously reduced, the data volume of the pre-aggregation storage table is reduced, and the query performance of the pre-aggregation storage table is improved.
Alternatively, the execution body of the embodiment of the present application may describe the pre-aggregate storage table creating device. Referring to fig. 1, fig. 1 is a schematic flow chart of a method for creating a pre-aggregate storage table according to an embodiment of the present application.
In an embodiment, the specific flow of the method for creating a pre-aggregate storage table according to the embodiment of the present application may include the following steps 10 to 40:
And step 10, acquiring the data type and the data identification of each piece of data of the to-be-processed table in the original storage table.
In the embodiment of the application, the data identifier characterizes whether the table data is index data, if the data is index data, the data identifier marks the index data identifier, and if the data is dimension data, the data identifier marks the dimension data identifier, so that the index data and the dimension data in the storage table can be distinguished through the data identifier. The data type characterizes whether the table data is a secondary aggregatable type, and the secondary aggregatable type and the non-secondary aggregatable type in the storage table can be distinguished by the data type.
Therefore, the pre-aggregate storage table creating device can acquire the data type and the data identification of each piece of table data to be processed in the original storage table, and distinguish each piece of table data to be processed in the original storage table according to the data type and the data identification.
And step 20, acquiring index data of each non-secondary aggregation type in the original storage table according to the data type and the data identification of each table data to be processed.
In the embodiment of the application, the index data of the non-secondary aggregation type is index data of the non-secondary aggregation type.
Optionally, the pre-aggregate storage table creating device screens each piece of table data to be processed in the original storage table through the data identifier to obtain index data in the original storage table, that is, filters the piece of table data to be processed with the data identifier being the dimension identifier to obtain the index data in the original storage table.
Further, the pre-aggregation storage table creating device performs secondary screening according to the data types of the index data in the original storage table to obtain index data of a non-secondary aggregation type in the original storage table, namely, the index data of the secondary aggregation type in the index data of the original storage table is removed to obtain index data of the non-secondary aggregation type in the original storage table.
And step 30, acquiring the pre-aggregation storage type of the index data of each non-secondary aggregation type.
Optionally, the pre-aggregate storage table creating means acquires a pre-aggregate storage type of the index data of each non-quadratic aggregate type, and in an embodiment, the index data of the non-quadratic aggregate type includes index data uv of the non-quadratic aggregate type and index data avg of the non-quadratic aggregate type, the pre-aggregate storage type of the index data uv of the non-quadratic aggregate type is bitmap_build (xx), and the pre-aggregate storage type of the index data avg of the non-quadratic aggregate type is sum (x)/count (x).
Step 40, replacing the original storage type of each index data of the non-secondary aggregation type with the pre-aggregation storage type of each index data of the non-secondary aggregation type in the original storage table, and creating a first pre-aggregation storage table.
Optionally, the pre-aggregate storage table creating means acquires an original storage type of the index data of each non-quadratic aggregate type in the original storage table, and replaces the original storage type of the index data of each non-quadratic aggregate type with the pre-aggregate storage type of the index data of each non-quadratic aggregate type in the original storage table, creating the first pre-aggregate storage table. In an embodiment, the index data of the non-secondary aggregation type includes index data uv of the non-secondary aggregation type and index data avg of the non-secondary aggregation type, and both the index data uv of the non-secondary aggregation type and the index data avg of the non-secondary aggregation type are non-accumulative index data, so that conversion is required to be performed on the index data uv of the non-secondary aggregation type and the index data avg of the non-secondary aggregation type, and the index data uv of the non-secondary aggregation type and the index data avg of the non-secondary aggregation type are converted into an accumulative pre-aggregation storage mode.
The table data to be processed included in the original storage table are: pending table data country, pending table data uv, pending table data pv, and pending table data avg_value.
The method comprises the steps of determining the data identification and the data type of the data to be processed, wherein the country of the data to be processed is dimension data identification, the uv of the data to be processed, the pv of the data to be processed and the avg_value of the data to be processed are index data identification, so that the uv of the data to be processed and the avg_value of the data to be processed are of a non-secondary aggregation type, and the pv of the data to be processed are of a secondary aggregation type, and therefore the uv of the data to be processed and the avg_value of the data to be processed are determined to be index data of the non-secondary aggregation type in an original storage table through the data identification and the data type.
The original storage type of the to-be-processed table data uv in the original storage table is count (DISITINCT USER _id), the de-duplication count value of the user_id, the original storage type of the to-be-processed table data avg_value is avg (value), and the average value of the value is the average value of the value.
The pre-aggregate storage type of the acquired to-be-processed table data uv is bitmap_build (xx), and is changed into bitmap storage, and is changed into an accumulated storage mode. And the pre-aggregate storage type of the avg_value of the table data to be processed is sum (value)/count (value), the molecular denominator is split, and the split sum and count are accumulated. Therefore, the original storage type count (DISITINCT USER _id) of the pending table data uv is replaced with the pre-aggregate storage type bitmap_build (xx), the original storage type avg (value) of the pending table data avg_value is replaced with the pre-aggregate storage type sum (value)/count (value), and the created first pre-aggregate storage table is shown in table 1.
Table 1 first pre-aggregate storage table
According to the embodiment of the application, index data of a non-secondary aggregation type in the original storage table can be replaced by index data of a secondary aggregation type, namely, the non-accumulated index data is converted into accumulated index data, so that the storage line number of the created pre-aggregation storage table is obviously reduced, the data volume of the pre-aggregation storage table is reduced, and the query performance of the pre-aggregation storage table is improved.
In an alternative embodiment, referring to fig. 2, fig. 2 is a second flowchart of a method for creating a pre-aggregate storage table according to an embodiment of the present application, which specifically includes steps 50 to 70:
step 50, obtaining a time partition strategy of a first pre-aggregate storage table;
Step 60, creating time field table data according to the time partition strategy;
step 70, inserting the time field table data into the first pre-aggregate storage table, and creating a second pre-aggregate storage table.
In the embodiment of the application, the time partition strategy characterizes the time granularity of the partition management of the first pre-aggregate storage table, namely, the time granularity can be divided into days, weeks, months, quarters and the like through the time partition strategy.
Optionally, the pre-aggregate storage table creating means acquires a time partition policy of the first pre-aggregate storage table, and creates the time field table data according to the time partition policy.
In one embodiment, the time partitioning policy may be a day partitioning policy, i.e., the first pre-aggregate storage table is managed in a number of days partition, thus creating day time field table data. The time partition policy may be a week partition policy, i.e., the first pre-aggregated storage table is managed in weeks partitions, thus creating week time field table data. The time partition policy may be a month partition policy, i.e., the first pre-aggregated storage table is managed in month partitions, thus creating the montage time field table data.
Further, in the embodiment of the present application, the time partition policy is a day partition policy, so that the created time field table data is day, and therefore, the time field table data day is inserted into the first pre-aggregate storage table, and a second pre-aggregate storage table is created, where the second pre-aggregate storage table is shown in table 2.
Table 2 second pre-aggregate storage table
| country | uv | avg_value | day |
| data_1 | data_2 | data_3 | data_4 |
| data_5 | data_6 | data_7 | data_8 |
| data_9 | data_10 | data_11 | data_12 |
According to the embodiment of the application, the time field table data is created according to the time partition strategy, so that the pre-aggregation storage table can be maintained in a time period, the pre-aggregation storage table is inquired in the time period, and the inquiry performance of the pre-aggregation storage table is improved. Meanwhile, the data can be scheduled to the pre-aggregation storage table in a time period, so that the scheduling cost of the pre-aggregation storage table is reduced.
In an alternative embodiment, after the second pre-aggregate storage table is created, the data to be stored may be stored by using the second pre-aggregate storage table, which specifically includes steps a to c:
step a, obtaining data to be stored;
step b, according to the time partition strategy and the time attribute information in the data to be stored, storing the data to be stored into a time partition corresponding to the second pre-aggregate storage table to obtain a pre-aggregate data table;
And c, storing the pre-aggregation data table into a pre-aggregation database.
Optionally, the pre-aggregate storage table creating device acquires data to be stored, where the data to be stored at least includes time attribute information. Data to be stored such as { count: guangzhou; uv:1000; day:2023-12-25}, { country: shanghai; uv:1500; day:2023-12-25}, and the like.
Further, the pre-aggregation storage table creating device determines a time partition corresponding to the data to be stored in the second pre-aggregation storage table according to the time partition strategy and the time attribute information in the data to be stored, and stores the data to be stored in the time partition corresponding to the second pre-aggregation storage table to obtain the pre-aggregation data table.
In one embodiment, the time partition policy is a day partition policy, and the data to be stored includes: data 1{ count to be stored: guangzhou; uv:1000; day:2023-12-25}, data 2{ count to be stored: guangzhou; uv:1000; day:2023-12-26}, data 3{ count to be stored: shanghai; uv:1500; day:2023-12-25}, data to be stored 4{ count }: beijing; uv:2000; day:2023-12-25}, the resulting pre-aggregate data table is shown in table 3.
Table 3 pre-aggregate data table
Further, the pre-aggregation storage table creating device stores the pre-aggregation data table to a pre-aggregation database, and a user can query data in the pre-aggregation data table through the pre-aggregation database.
According to the embodiment of the application, the time field table data is created according to the time partition strategy, so that the pre-aggregation storage table can be maintained in a time period, the pre-aggregation storage table is inquired in the time period, and the inquiry performance of the pre-aggregation storage table is improved. Meanwhile, the data can be scheduled to the pre-aggregation storage table in a time period, so that the scheduling cost of the pre-aggregation storage table is reduced.
In an alternative embodiment, referring to fig. 3, fig. 3 is a third flowchart of a method for creating a pre-aggregate storage table according to an embodiment of the present application, which specifically includes steps 80 to 100:
Step 80, responding to the data query instruction, and acquiring a time sequence in the data query instruction;
step 90, performing time sequence segmentation on the time sequence according to a time partition strategy to obtain a plurality of time subsequences;
And step 100, inquiring the pre-aggregation data table in the pre-aggregation database according to the plurality of time subsequences to obtain a target inquiry result of the data inquiry instruction.
Optionally, when the user needs to pre-aggregate the data in the data table, the user may initiate a data query instruction to the pre-aggregate storage table creating device, where the data query instruction needs to carry a time sequence of query data, such as a data query instruction { xx of 21 st 25 th 12 th 21 st 2023 in guangzhou. Thus, the pre-aggregate storage table creating means acquires the time series in the data query instruction in response to the data query instruction.
Further, in order to improve the data query efficiency, it is necessary to split a long time sequence into short time sequences, and therefore, the pre-aggregate storage table creating device performs time sequence splitting on the time sequences according to a time partition policy, so as to obtain a plurality of time sub-sequences.
In an embodiment, the time partition strategy is a week partition strategy, and the time sequence in the data query instruction is split with 7 days as a period to obtain a plurality of time sub-sequences.
Further, the pre-aggregate storage table creating device queries the pre-aggregate data table in the pre-aggregate database according to the plurality of time sub-sequences to obtain the target query result of the data query instruction, which is specifically described in steps 1001 to 1004.
According to the embodiment of the application, the time sequence is segmented according to the time partitioning strategy, and the pre-aggregation data table in the pre-aggregation database is queried through a plurality of time subsequences, so that the data query efficiency is improved.
In an alternative embodiment, the descriptions of steps 1001 through 1004 are as follows:
Step 1001, inquiring a pre-aggregation data table in a pre-aggregation database through a first time subsequence to obtain a first pre-aggregation inquiry result;
step 1002, starting from the second time sub-sequence, judging whether the starting time of the current time sub-sequence is a division time or not through a preset time sequence segmentation logic, and obtaining a judgment result;
step 1003, inquiring a pre-aggregation data table in the pre-aggregation database based on the judging result until all time subsequences are traversed, and obtaining a second pre-aggregation inquiring result;
step 1004, determining the first pre-aggregate query result and the second pre-aggregate query result as target query results of the data query instruction.
Optionally, the pre-aggregation storage table creating device queries the pre-aggregation data table in the pre-aggregation database through the first time subsequence to obtain a first pre-aggregation query result.
Further, when the time sequence splitting is performed, there may be a splitting error or an error, so the pre-aggregate storage table creating device starts from the second time sub-sequence, and determines whether the start time of the current time sub-sequence is the splitting time through the preset time sequence splitting logic, thereby obtaining a determination result.
In an embodiment, the time partition policy is a week number (7 days) partition policy, the starting time of the first time sub-sequence is obtained, the starting time of the current time sub-sequence and the starting time of the first time sub-sequence are differentiated, whether the difference is equal to 7 is determined, whether the starting time of the current time sub-sequence is the partition time is determined, and a judgment result is obtained, so that the preset time sequence segmentation logic can immediately determine whether the time difference between the starting time of the current time sub-sequence and the starting time of the last time sub-sequence is equal to the number of days, weeks, months or seasons in the time partition policy.
Further, the pre-aggregation storage table creating device queries the pre-aggregation data table in the pre-aggregation database based on the judging result until the second pre-aggregation query result is obtained after all the time subsequences are traversed, and the first pre-aggregation query result and the second pre-aggregation query result are determined to be target query results of the data query instruction.
According to the embodiment of the application, whether the starting time of the current time sub-sequence is the judgment result of the division time is judged through the preset time sequence segmentation logic to perform data query, so that the repetition of data query is avoided, and the accuracy of the data query is improved.
In an alternative embodiment, steps 10021 to 10022 are described as follows:
Step 10021, if starting from the second time sub-sequence, determining that the starting time of the current time sub-sequence is the dividing time through the preset time sequence dividing logic, generating a first determination result;
In step 10022, if the starting time of the current time sub-sequence is not the divided time, a second determination result is generated.
Optionally, if starting from the second time sub-sequence, determining that the starting time of the current time sub-sequence is the splitting time through a preset time sequence splitting logic, and determining that the time sequence splitting is correct by the pre-aggregate storage table creating device, so as to generate a first determination result. Further, if starting from the second time sub-sequence, the starting time of the current time sub-sequence is judged to be not the dividing time through the preset time sequence segmentation logic, and the pre-aggregation storage table creation device determines that the time sequence splitting is incorrect, and generates a second judgment result.
Therefore, in the embodiment of the application, the first judgment result is a judgment result of correct time sequence splitting, and the second judgment result is a judgment result of incorrect time sequence splitting.
According to the embodiment of the application, whether the starting time of the current time sub-sequence is the judgment result of the division time is judged through the preset time sequence segmentation logic to perform data query, so that the repetition of data query is avoided, and the accuracy of the data query is improved.
In an alternative embodiment, the descriptions of steps 10031 to 10032 are as follows:
step 10031, if the judgment result is the first judgment result, inquiring the pre-aggregation data table in the pre-aggregation database through the second time subsequence until all the time subsequences are traversed, and obtaining a second pre-aggregation inquiry result; or alternatively, the first and second heat exchangers may be,
If the judgment result is the second judgment result, step 10032, shifting the start time of the current time sub-sequence by a preset time length to obtain a third time sub-sequence, and querying the pre-aggregation data table in the pre-aggregation database through the third time sub-sequence until all the time sub-sequences are traversed to obtain a second pre-aggregation query result.
Optionally, if the judgment result is the first judgment result and the real-time sequence splitting is correct, the pre-aggregate storage table creating device queries the pre-aggregate data table in the pre-aggregate database through the second time subsequence until all the time subsequences are traversed, and then a second pre-aggregate query result is obtained.
Further, if the judgment result is the second judgment result and the time sequence splitting is incorrect, the pre-aggregate storage table creating device calculates the time difference between the starting time of the current time sub-sequence, the starting time of the last time sub-sequence and the number of days, weeks, months or quarters in the time partition strategy, and determines the time difference as the preset time length of the current time sub-sequence.
In one embodiment, the time partition strategy is a number of weeks, so the time difference = the start time of the current time sub-sequence-the start time of the last time sub-sequence- (7-1).
Further, if the time difference is smaller than 0, the pre-aggregation storage table creating device shifts the starting time of the current time sub-sequence backwards by a preset time length to obtain a third time sub-sequence. When the time difference is greater than 0, the pre-aggregation storage table creating device shifts the starting time of the current time sub-sequence forward by a preset time length to obtain a third time sub-sequence. In one embodiment, the time partition strategy is a number of weeks, the start time of the current time sub-sequence is 2023-12-25, the start time of the last time sub-sequence is 2023-12-18, and thus the time difference is (2023-12-25) - (2023-12-18) - (7-1) =1, greater than 0, and thus the start time of the current time sub-sequence is shifted forward by 1, i.e., the third time sub-sequence is 2023-12-24.
Further, the pre-aggregation storage table creating device queries the pre-aggregation data table in the pre-aggregation database through the third time subsequence until all the time subsequences are traversed, and a second pre-aggregation query result is obtained.
The embodiment of the application avoids data query errors and improves the accuracy of data query by shifting the initial time of the incorrectly split current time sub-sequence by the preset time length.
The pre-aggregate storage table creating device provided by the embodiment of the present application will be described below, and the pre-aggregate storage table creating device described below and the pre-aggregate storage table creating method described above may be referred to correspondingly to each other.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a pre-aggregate storage table creating apparatus provided in an embodiment of the present application, the pre-aggregate storage table creating apparatus may include:
A first obtaining module 401, configured to obtain a data type and a data identifier of each table data to be processed in the original storage table; the data identification represents whether the table data is index data; the data type characterization table data is of a secondary aggregation type;
A second obtaining module 402, configured to obtain, according to the data type and the data identifier of each to-be-processed table data, index data of each non-secondary aggregation type in the original storage table; the index data of the non-secondary aggregation type is index data of the non-secondary aggregation type;
a third obtaining module 403, configured to obtain a pre-aggregate storage type of the index data of each non-secondary aggregate type;
a storage table creating module 404, configured to replace the original storage type of each index data of the non-secondary-polymerizable type with the pre-polymerized storage type of each index data of the non-secondary-polymerizable type in the original storage table, and create a first pre-polymerized storage table.
In an alternative embodiment, the pre-aggregate storage table creating means is further for:
Acquiring a time partition strategy of the first pre-aggregate storage table; the time partition policy characterizes a time granularity of partition management of the first pre-aggregate storage table;
Creating time field table data according to the time partition strategy;
And inserting the time field table data into the first pre-aggregation storage table, and creating a second pre-aggregation storage table.
In an alternative embodiment, the pre-aggregate storage table creating means is further for:
acquiring data to be stored; the data to be stored at least comprises time attribute information;
storing the data to be stored into a time partition corresponding to the second pre-aggregation storage table according to the time partition strategy and the time attribute information in the data to be stored to obtain a pre-aggregation data table;
and storing the pre-aggregation data table into a pre-aggregation database.
In an alternative embodiment, the pre-aggregate storage table creating means is further for:
responding to a data query instruction, and acquiring a time sequence in the data query instruction;
Performing time sequence segmentation on the time sequence according to the time partition strategy to obtain a plurality of time subsequences;
And inquiring a pre-aggregated data table in the pre-aggregated database according to the plurality of time subsequences to obtain a target inquiring result of the data inquiring instruction.
In an alternative embodiment, the pre-aggregate storage table creating means is further for:
Inquiring a pre-aggregation data table in the pre-aggregation database through a first time subsequence to obtain a first pre-aggregation inquiry result;
Starting from the second time sub-sequence, judging whether the starting time of the current time sub-sequence is the dividing time or not through a preset time sequence segmentation logic, and obtaining a judging result;
Inquiring a pre-aggregation data table in the pre-aggregation database based on the judging result until all time subsequences are traversed, and obtaining a second pre-aggregation inquiring result;
and determining the first pre-aggregated query result and the second pre-aggregated query result as target query results of the data query instruction.
In an alternative embodiment, the pre-aggregate storage table creating means is further for:
If the starting time of the current time sub-sequence is the dividing time through the preset time sequence dividing logic from the second time sub-sequence, a first judging result is generated;
If the starting time of the current time sub-sequence is not the dividing time through the preset time sequence dividing logic from the second time sub-sequence, generating a second judging result;
The first judgment result and the second judgment result are different judgment results.
In an alternative embodiment, the pre-aggregate storage table creating means is further for:
if the judging result is the first judging result, inquiring a pre-aggregation data table in the pre-aggregation database through the second time subsequence until all the time subsequences are traversed, and obtaining a second pre-aggregation inquiring result; or alternatively, the first and second heat exchangers may be,
If the judging result is the second judging result, shifting the starting time of the current time sub-sequence by a preset time length to obtain a third time sub-sequence, and inquiring a pre-aggregation data table in the pre-aggregation database through the third time sub-sequence until all the time sub-sequences are traversed to obtain a second pre-aggregation inquiring result
According to the embodiment of the application, index data of a non-secondary aggregation type in the original storage table can be replaced by index data of a secondary aggregation type, namely, the non-accumulated index data is converted into accumulated index data, so that the storage line number of the created pre-aggregation storage table is obviously reduced, the data volume of the pre-aggregation storage table is reduced, and the query performance of the pre-aggregation storage table is improved.
The specific embodiment of the pre-aggregate storage table creating device provided by the application is basically the same as each embodiment of the pre-aggregate storage table creating method, and is not described herein.
Optionally, as shown in fig. 5, fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device may include: processor 510, communication interface (Communication Interface) 520, memory 530, and communication bus 540, wherein processor 510, communication interface 520, memory 530 complete communication with each other through communication bus 540. Processor 510 may call a computer program in memory 530 to perform the steps of the pre-aggregate storage table creation method, including:
Acquiring the data type and the data identification of each to-be-processed table data in an original storage table; the data identification represents whether the table data is index data; the data type characterization table data is of a secondary aggregation type;
Acquiring index data of each non-secondary aggregation type in the original storage table according to the data type and the data identification of each to-be-processed table data;
Acquiring a pre-aggregation storage type of index data of each non-secondary aggregation type;
And replacing the original storage type of the index data of each non-secondary aggregation type with the pre-aggregation storage type of the index data of each non-secondary aggregation type in the original storage table, and creating a first pre-aggregation storage table.
In an alternative embodiment, replacing the original storage type of each index data of the non-secondary aggregation type with the pre-aggregation storage type of each index data of the non-secondary aggregation type in the original storage table to obtain a first pre-aggregation storage table, and then further including:
Acquiring a time partition strategy of the first pre-aggregate storage table; the time partition policy characterizes a time granularity of partition management of the first pre-aggregate storage table;
Creating time field table data according to the time partition strategy;
And inserting the time field table data into the first pre-aggregation storage table, and creating a second pre-aggregation storage table.
In an alternative embodiment, the inserting the time field table data into the first pre-aggregate storage table, after creating the second pre-aggregate storage table, further includes:
acquiring data to be stored; the data to be stored at least comprises time attribute information;
storing the data to be stored into a time partition corresponding to the second pre-aggregation storage table according to the time partition strategy and the time attribute information in the data to be stored to obtain a pre-aggregation data table;
and storing the pre-aggregation data table into a pre-aggregation database.
In an alternative embodiment, after storing the pre-aggregate data table in the time sequence database, the method further includes:
responding to a data query instruction, and acquiring a time sequence in the data query instruction;
Performing time sequence segmentation on the time sequence according to the time partition strategy to obtain a plurality of time subsequences;
And inquiring a pre-aggregated data table in the pre-aggregated database according to the plurality of time subsequences to obtain a target inquiring result of the data inquiring instruction.
In an optional embodiment, querying the pre-aggregated data table in the pre-aggregated database according to the plurality of time subsequences to obtain the target query result of the data query instruction, including:
Inquiring a pre-aggregation data table in the pre-aggregation database through a first time subsequence to obtain a first pre-aggregation inquiry result;
Starting from the second time sub-sequence, judging whether the starting time of the current time sub-sequence is the dividing time or not through a preset time sequence segmentation logic, and obtaining a judging result;
Inquiring a pre-aggregation data table in the pre-aggregation database based on the judging result until all time subsequences are traversed, and obtaining a second pre-aggregation inquiring result;
and determining the first pre-aggregated query result and the second pre-aggregated query result as target query results of the data query instruction.
In an alternative embodiment, starting from the second time sub-sequence, determining whether the start time of the current time sub-sequence is the split time by the preset time sequence splitting logic, to obtain a determination result, including:
If the starting time of the current time sub-sequence is the dividing time through the preset time sequence dividing logic from the second time sub-sequence, a first judging result is generated;
If the starting time of the current time sub-sequence is not the dividing time through the preset time sequence dividing logic from the second time sub-sequence, generating a second judging result;
The first judgment result and the second judgment result are different judgment results.
In an optional embodiment, querying the pre-aggregate data table in the pre-aggregate database based on the determination result until all the time sub-sequences are traversed, to obtain a second pre-aggregate query result, including:
if the judging result is the first judging result, inquiring a pre-aggregation data table in the pre-aggregation database through the second time subsequence until all the time subsequences are traversed, and obtaining a second pre-aggregation inquiring result; or alternatively, the first and second heat exchangers may be,
And if the judging result is the second judging result, shifting the starting time of the current time sub-sequence by a preset time length to obtain a third time sub-sequence, and inquiring a pre-aggregation data table in the pre-aggregation database through the third time sub-sequence until all the time sub-sequences are traversed to obtain a second pre-aggregation inquiring result.
Furthermore, the logic computer program in the memory 530 may be implemented in the form of a software functional unit and may be stored in a computer readable storage medium when sold or used as a separate product. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in the form of a software product stored in a storage medium, comprising a number of computer programs for causing a computer device (which may be a personal computer, a server, a network device, etc.) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, embodiments of the present application further provide a non-transitory computer readable storage medium, including a computer program, where the computer program may be stored on the non-transitory computer readable storage medium, and when the computer program is executed by a processor, the computer program may be capable of executing the steps of the method for creating a pre-aggregate storage table provided in the foregoing embodiments, where the method includes:
Acquiring the data type and the data identification of each to-be-processed table data in an original storage table; the data identification represents whether the table data is index data; the data type characterization table data is of a secondary aggregation type;
Acquiring index data of each non-secondary aggregation type in the original storage table according to the data type and the data identification of each to-be-processed table data;
Acquiring a pre-aggregation storage type of index data of each non-secondary aggregation type;
And replacing the original storage type of the index data of each non-secondary aggregation type with the pre-aggregation storage type of the index data of each non-secondary aggregation type in the original storage table, and creating a first pre-aggregation storage table.
In an alternative embodiment, replacing the original storage type of each index data of the non-secondary aggregation type with the pre-aggregation storage type of each index data of the non-secondary aggregation type in the original storage table to obtain a first pre-aggregation storage table, and then further including:
Acquiring a time partition strategy of the first pre-aggregate storage table; the time partition policy characterizes a time granularity of partition management of the first pre-aggregate storage table;
Creating time field table data according to the time partition strategy;
And inserting the time field table data into the first pre-aggregation storage table, and creating a second pre-aggregation storage table.
In an alternative embodiment, the inserting the time field table data into the first pre-aggregate storage table, after creating the second pre-aggregate storage table, further includes:
acquiring data to be stored; the data to be stored at least comprises time attribute information;
storing the data to be stored into a time partition corresponding to the second pre-aggregation storage table according to the time partition strategy and the time attribute information in the data to be stored to obtain a pre-aggregation data table;
and storing the pre-aggregation data table into a pre-aggregation database.
In an alternative embodiment, after storing the pre-aggregate data table in the time sequence database, the method further includes:
responding to a data query instruction, and acquiring a time sequence in the data query instruction;
Performing time sequence segmentation on the time sequence according to the time partition strategy to obtain a plurality of time subsequences;
And inquiring a pre-aggregated data table in the pre-aggregated database according to the plurality of time subsequences to obtain a target inquiring result of the data inquiring instruction.
In an optional embodiment, querying the pre-aggregated data table in the pre-aggregated database according to the plurality of time subsequences to obtain the target query result of the data query instruction, including:
Inquiring a pre-aggregation data table in the pre-aggregation database through a first time subsequence to obtain a first pre-aggregation inquiry result;
Starting from the second time sub-sequence, judging whether the starting time of the current time sub-sequence is the dividing time or not through a preset time sequence segmentation logic, and obtaining a judging result;
Inquiring a pre-aggregation data table in the pre-aggregation database based on the judging result until all time subsequences are traversed, and obtaining a second pre-aggregation inquiring result;
and determining the first pre-aggregated query result and the second pre-aggregated query result as target query results of the data query instruction.
In an alternative embodiment, starting from the second time sub-sequence, determining whether the start time of the current time sub-sequence is the split time by the preset time sequence splitting logic, to obtain a determination result, including:
If the starting time of the current time sub-sequence is the dividing time through the preset time sequence dividing logic from the second time sub-sequence, a first judging result is generated;
If the starting time of the current time sub-sequence is not the dividing time through the preset time sequence dividing logic from the second time sub-sequence, generating a second judging result;
The first judgment result and the second judgment result are different judgment results.
In an optional embodiment, querying the pre-aggregate data table in the pre-aggregate database based on the determination result until all the time sub-sequences are traversed, to obtain a second pre-aggregate query result, including:
if the judging result is the first judging result, inquiring a pre-aggregation data table in the pre-aggregation database through the second time subsequence until all the time subsequences are traversed, and obtaining a second pre-aggregation inquiring result; or alternatively, the first and second heat exchangers may be,
And if the judging result is the second judging result, shifting the starting time of the current time sub-sequence by a preset time length to obtain a third time sub-sequence, and inquiring a pre-aggregation data table in the pre-aggregation database through the third time sub-sequence until all the time sub-sequences are traversed to obtain a second pre-aggregation inquiring result.
In still another aspect, an embodiment of the present application further provides a computer product, where the computer product includes a computer program, where the computer program may be stored on the computer product, and when the computer program is executed by a processor, the computer is capable of executing the steps of the method for creating a pre-aggregate storage table provided in the foregoing embodiments, where the method includes:
Acquiring the data type and the data identification of each to-be-processed table data in an original storage table; the data identification represents whether the table data is index data; the data type characterization table data is of a secondary aggregation type;
Acquiring index data of each non-secondary aggregation type in the original storage table according to the data type and the data identification of each to-be-processed table data;
Acquiring a pre-aggregation storage type of index data of each non-secondary aggregation type;
And replacing the original storage type of the index data of each non-secondary aggregation type with the pre-aggregation storage type of the index data of each non-secondary aggregation type in the original storage table, and creating a first pre-aggregation storage table.
In an alternative embodiment, replacing the original storage type of each index data of the non-secondary aggregation type with the pre-aggregation storage type of each index data of the non-secondary aggregation type in the original storage table to obtain a first pre-aggregation storage table, and then further including:
Acquiring a time partition strategy of the first pre-aggregate storage table; the time partition policy characterizes a time granularity of partition management of the first pre-aggregate storage table;
Creating time field table data according to the time partition strategy;
And inserting the time field table data into the first pre-aggregation storage table, and creating a second pre-aggregation storage table.
In an alternative embodiment, the inserting the time field table data into the first pre-aggregate storage table, after creating the second pre-aggregate storage table, further includes:
acquiring data to be stored; the data to be stored at least comprises time attribute information;
storing the data to be stored into a time partition corresponding to the second pre-aggregation storage table according to the time partition strategy and the time attribute information in the data to be stored to obtain a pre-aggregation data table;
and storing the pre-aggregation data table into a pre-aggregation database.
In an alternative embodiment, after storing the pre-aggregate data table in the time sequence database, the method further includes:
responding to a data query instruction, and acquiring a time sequence in the data query instruction;
Performing time sequence segmentation on the time sequence according to the time partition strategy to obtain a plurality of time subsequences;
And inquiring a pre-aggregated data table in the pre-aggregated database according to the plurality of time subsequences to obtain a target inquiring result of the data inquiring instruction.
In an optional embodiment, querying the pre-aggregated data table in the pre-aggregated database according to the plurality of time subsequences to obtain the target query result of the data query instruction, including:
Inquiring a pre-aggregation data table in the pre-aggregation database through a first time subsequence to obtain a first pre-aggregation inquiry result;
Starting from the second time sub-sequence, judging whether the starting time of the current time sub-sequence is the dividing time or not through a preset time sequence segmentation logic, and obtaining a judging result;
Inquiring a pre-aggregation data table in the pre-aggregation database based on the judging result until all time subsequences are traversed, and obtaining a second pre-aggregation inquiring result;
and determining the first pre-aggregated query result and the second pre-aggregated query result as target query results of the data query instruction.
In an alternative embodiment, starting from the second time sub-sequence, determining whether the start time of the current time sub-sequence is the split time by the preset time sequence splitting logic, to obtain a determination result, including:
If the starting time of the current time sub-sequence is the dividing time through the preset time sequence dividing logic from the second time sub-sequence, a first judging result is generated;
If the starting time of the current time sub-sequence is not the dividing time through the preset time sequence dividing logic from the second time sub-sequence, generating a second judging result;
The first judgment result and the second judgment result are different judgment results.
In an optional embodiment, querying the pre-aggregate data table in the pre-aggregate database based on the determination result until all the time sub-sequences are traversed, to obtain a second pre-aggregate query result, including:
if the judging result is the first judging result, inquiring a pre-aggregation data table in the pre-aggregation database through the second time subsequence until all the time subsequences are traversed, and obtaining a second pre-aggregation inquiring result; or alternatively, the first and second heat exchangers may be,
And if the judging result is the second judging result, shifting the starting time of the current time sub-sequence by a preset time length to obtain a third time sub-sequence, and inquiring a pre-aggregation data table in the pre-aggregation database through the third time sub-sequence until all the time sub-sequences are traversed to obtain a second pre-aggregation inquiring result.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the above technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., comprising a number of computer programs for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims (10)
1. A method of creating a pre-aggregate storage table, comprising:
Acquiring the data type and the data identification of each to-be-processed table data in an original storage table; the data identification represents whether the table data is index data; the data type characterization table data is of a secondary aggregation type;
Acquiring index data of each non-secondary aggregation type in the original storage table according to the data type and the data identification of each to-be-processed table data;
Acquiring a pre-aggregation storage type of index data of each non-secondary aggregation type;
And replacing the original storage type of the index data of each non-secondary aggregation type with the pre-aggregation storage type of the index data of each non-secondary aggregation type in the original storage table, and creating a first pre-aggregation storage table.
2. The method for creating a pre-aggregate storage table according to claim 1, wherein replacing the original storage type of each index data of the non-secondary aggregate type with the pre-aggregate storage type of each index data of the non-secondary aggregate type in the original storage table, after obtaining the first pre-aggregate storage table, further comprises:
Acquiring a time partition strategy of the first pre-aggregate storage table; the time partition policy characterizes a time granularity of partition management of the first pre-aggregate storage table;
Creating time field table data according to the time partition strategy;
And inserting the time field table data into the first pre-aggregation storage table, and creating a second pre-aggregation storage table.
3. The method for creating a pre-aggregate storage table according to claim 2, wherein said inserting said time field table data into said first pre-aggregate storage table, after creating a second pre-aggregate storage table, further comprises:
acquiring data to be stored; the data to be stored at least comprises time attribute information;
storing the data to be stored into a time partition corresponding to the second pre-aggregation storage table according to the time partition strategy and the time attribute information in the data to be stored to obtain a pre-aggregation data table;
and storing the pre-aggregation data table into a pre-aggregation database.
4. The method for creating a pre-aggregate storage table according to claim 3, further comprising, after storing the pre-aggregate data table in a time series database:
responding to a data query instruction, and acquiring a time sequence in the data query instruction;
Performing time sequence segmentation on the time sequence according to the time partition strategy to obtain a plurality of time subsequences;
And inquiring a pre-aggregated data table in the pre-aggregated database according to the plurality of time subsequences to obtain a target inquiring result of the data inquiring instruction.
5. The method for creating a pre-aggregate storage table according to claim 4, wherein the querying the pre-aggregate data table in the pre-aggregate database according to the plurality of time sub-sequences to obtain the target query result of the data query instruction includes:
Inquiring a pre-aggregation data table in the pre-aggregation database through a first time subsequence to obtain a first pre-aggregation inquiry result;
Starting from the second time sub-sequence, judging whether the starting time of the current time sub-sequence is the dividing time or not through a preset time sequence segmentation logic, and obtaining a judging result;
Inquiring a pre-aggregation data table in the pre-aggregation database based on the judging result until all time subsequences are traversed, and obtaining a second pre-aggregation inquiring result;
and determining the first pre-aggregated query result and the second pre-aggregated query result as target query results of the data query instruction.
6. The method for creating a pre-aggregate storage table according to claim 5, wherein the determining whether the start time of the current time sub-sequence is the split time from the second time sub-sequence by the preset time sequence splitting logic, to obtain the determination result, includes:
If the starting time of the current time sub-sequence is the dividing time through the preset time sequence dividing logic from the second time sub-sequence, a first judging result is generated;
If the starting time of the current time sub-sequence is not the dividing time through the preset time sequence dividing logic from the second time sub-sequence, generating a second judging result;
The first judgment result and the second judgment result are different judgment results.
7. The method for creating the pre-aggregate storage table according to claim 6, wherein the querying the pre-aggregate data table in the pre-aggregate database based on the determination result until all the time sub-sequences are traversed, to obtain a second pre-aggregate query result, includes:
if the judging result is the first judging result, inquiring a pre-aggregation data table in the pre-aggregation database through the second time subsequence until all the time subsequences are traversed, and obtaining a second pre-aggregation inquiring result; or alternatively, the first and second heat exchangers may be,
And if the judging result is the second judging result, shifting the starting time of the current time sub-sequence by a preset time length to obtain a third time sub-sequence, and inquiring a pre-aggregation data table in the pre-aggregation database through the third time sub-sequence until all the time sub-sequences are traversed to obtain a second pre-aggregation inquiring result.
8. A pre-aggregate storage table creation apparatus, comprising:
the first acquisition module is used for acquiring the data type and the data identification of each to-be-processed table data in the original storage table; the data identification represents whether the table data is index data; the data type characterization table data is of a secondary aggregation type;
the second acquisition module is used for acquiring index data of each non-secondary aggregation type in the original storage table according to the data type and the data identification of each table data to be processed; the index data of the non-secondary aggregation type is index data of the non-secondary aggregation type;
the third acquisition module is used for acquiring the pre-aggregation storage type of the index data of each non-secondary aggregation type;
and the storage table creation module is used for replacing the original storage type of the index data of each non-secondary aggregation type with the pre-aggregation storage type of the index data of each non-secondary aggregation type in the original storage table to create a first pre-aggregation storage table.
9. An electronic device comprising a processor and a memory, the memory storing a plurality of computer programs; the processor loads a computer program from the memory to perform the pre-aggregate storage table creation method of any of claims 1 to 7.
10. A computer readable storage medium, characterized in that it stores a plurality of computer programs adapted to be loaded by a processor for executing the pre-aggregate storage table creation method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410139594.8A CN118132553A (en) | 2024-01-31 | 2024-01-31 | Pre-aggregate storage table creation method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410139594.8A CN118132553A (en) | 2024-01-31 | 2024-01-31 | Pre-aggregate storage table creation method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118132553A true CN118132553A (en) | 2024-06-04 |
Family
ID=91229447
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410139594.8A Pending CN118132553A (en) | 2024-01-31 | 2024-01-31 | Pre-aggregate storage table creation method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118132553A (en) |
-
2024
- 2024-01-31 CN CN202410139594.8A patent/CN118132553A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8112421B2 (en) | Query selection for effectively learning ranking functions | |
| CN112287182A (en) | Graph data storage and processing method and device and computer storage medium | |
| US7478083B2 (en) | Method and system for estimating cardinality in a database system | |
| CN110797085B (en) | Method, system, equipment and storage medium for inquiring gene data | |
| CN107239541B (en) | Cost estimation method and device | |
| CN111064776B (en) | Block generation method in block chain, accounting node and storage medium | |
| CN111159130A (en) | Small file merging method and electronic equipment | |
| CN110222046B (en) | List data processing method, device, server and storage medium | |
| CN110162540B (en) | Block chain account book data query method, electronic device and storage medium | |
| CN108549688B (en) | Data operation optimization method, device, equipment and storage medium | |
| US20140214826A1 (en) | Ranking method and system | |
| CN112579608A (en) | Case data query method, system, device and computer readable storage medium | |
| CN118132553A (en) | Pre-aggregate storage table creation method and device, electronic equipment and storage medium | |
| CN110765125B (en) | Method and device for storing data | |
| CN116595044A (en) | Optimization method, storage medium and equipment for database selectivity calculation | |
| CN106446080B (en) | Data query method, query service equipment, client equipment and data system | |
| CN107203550B (en) | Data processing method and database server | |
| CN114691630A (en) | Smart supply chain big data sharing method and system | |
| CN106933933B (en) | Data table information processing method and device | |
| CN107066247B (en) | Patch query method and device | |
| CN115952172B (en) | Data matching method and device based on database temporary table | |
| CN117555968B (en) | Data processing method, device, equipment and storage medium | |
| CN117271562B (en) | Data acquisition processing method and system | |
| US8423532B1 (en) | Managing data indexed by a search engine | |
| CN115827930B (en) | Data query optimization method, system and device for graph database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |