CN116401254A - Unified storage method and device for index result data - Google Patents
Unified storage method and device for index result data Download PDFInfo
- Publication number
- CN116401254A CN116401254A CN202310413526.1A CN202310413526A CN116401254A CN 116401254 A CN116401254 A CN 116401254A CN 202310413526 A CN202310413526 A CN 202310413526A CN 116401254 A CN116401254 A CN 116401254A
- Authority
- CN
- China
- Prior art keywords
- data
- index
- column
- dimension
- unique
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 230000008859 change Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 8
- 238000005192 partition Methods 0.000 claims 2
- 238000000638 solvent extraction Methods 0.000 claims 2
- 238000007726 management method Methods 0.000 description 17
- 238000013500 data storage Methods 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 6
- 239000012634 fragment Substances 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 101000773153 Homo sapiens Thioredoxin-like protein 4A Proteins 0.000 description 3
- 102100030272 Thioredoxin-like protein 4A Human genes 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 101100063432 Caenorhabditis elegans dim-1 gene Proteins 0.000 description 1
- 101000773151 Homo sapiens Thioredoxin-like protein 4B Proteins 0.000 description 1
- 102100030273 Thioredoxin-like protein 4B Human genes 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000036316 preload Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Theoretical Computer Science (AREA)
- Strategic Management (AREA)
- Educational Administration (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- Development Economics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Game Theory and Decision Science (AREA)
- Software Systems (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a unified storage method of index result data, which generates an elastic data table, comprising the following steps: generating a general dimension column, an index column and a dimension column unique to the index; the unique dimension column of each index means that each index has unique dimension besides the universal dimension, and the unique dimension column of each index has corresponding data columns in the table; the dimension column unique to the index stores unique data in index result data; the elastic data table is stored in a storage medium. By adopting the technical scheme, the technical problems that in the prior art, the number of data tables is large, management is difficult, data redundancy is realized, data cannot be directly inquired, the number of access is complex and the like can be solved.
Description
Technical Field
The application relates to the field of index result data storage, in particular to a method and a device for uniformly storing index result data.
Background
Under the current strong market competition environment, enterprises continuously strengthen internal fine management, gradually build a digital management system with indexes as cores, stare at the core index data of enterprise management, and build and continuously perfect a comprehensive and unified index system of the enterprises. Meanwhile, the problem of enterprise index management is continuously amplified, and particularly, the storage and inquiry of massive index data in a big data environment provide great challenges for data teams and technical teams.
In order to avoid influencing the normal operation of the existing business systems, enterprises generally construct a theme-level or enterprise-level data warehouse based on a relational database or a large data platform, extract data in each business system into a plurality of bins, and perform hierarchical calculation and storage. When data analysis of a service application scene is proposed, a service analysis team analyzes related indexes and analysis dimensions of the data, a data development team performs data modeling, writes data ETL scripts and configures timing tasks to perform data calculation, and finally index data meeting service requirements is stored in a corresponding data table.
Under the condition of meeting the current data demand of the service, the index data storage scheme can meet the performance requirement of data query, and can rapidly respond to the continuously-changing service demand under the condition of not affecting the existing service. In response to complex business data requirements, data development teams typically employ the following methods to store index data.
The first technical scheme is to store index data in different data tables according to subjects and analysis scenes. Dealer analysis-summary data table (daily granularity) designed as shown in table 1 below contains three indices: the three indexes comprise month, province area, dealer and activity five analysis dimensions under the analysis scene. The data table will be timed by day statistics.
Table 1: dealer analysis-summary data sheet (day granularity) example
The technical scheme has the advantages that the technical scheme is easy to understand, can adapt to different databases and large data platforms, can directly meet the access requirement of upper-layer business, is also a common scheme of a traditional data warehouse, and has mature modeling technology and modeling specification matched with the scheme. But the problem of this solution is also highlighted in the case of rapid changes in services in the digitization era. Firstly, in the digital age, the data analysis requirement is rapidly increased, so that more and more data tables in a data warehouse are caused, and management is difficult. Secondly, the maintenance cost of the data table caused by the rapid change of the service is high, and other services are easily affected. The data table is redesigned in order not to affect the existing service, and the problems of data redundancy and data inconsistency are easily caused. Finally, the increasing number of data tables increases the demands for storage resources and computing resources, resulting in a rapid increase in the operational and maintenance costs of the enterprise.
The second technical scheme is to design a set of general index result data storage scheme based on a high table. The technical scheme comprises an index result data table and a plurality of configuration information tables. In the index result data table, besides fixed dimensions including date, province area, region and the like, N dimensions including DIM1, DIM2 … … DIMN and the like are reserved additionally, and different indexes may correspond to different dimensions and dimension numbers. In the index result table, the same reserved dimension field (for example, DIM 1) may correspond to different dimensions of different indexes. For example, in the data record of the index of the number of the manufacturer, the DIM1 dimension corresponds to the activity dimension, and in the data record of the sales index, the DIM1 dimension corresponds to the store. Since the index values are all of a numeric type, the values of all of the indices may be stored in the same column, such as the index value column defined in the table.
Table 2: index result data representation example
In addition to the index result data table, a corresponding dimension information table, an index dimension relation table and the like need to be designed. The dimension information table contains information such as dimension type, dimension name, dimension English name, dimension description, dimension classification, dimension state and the like. The index dimension relation table is used for associating indexes with dimensions and defining dimension columns in the index result data table in detail.
Table 3: dimension information representation example
Table 4: index dimension relation representation example
The technical scheme based on the high table has the advantages that all index result data can be stored uniformly, new indexes can be added continuously along with the change of the service, the index result data storage table does not need to be designed for many times, each index can be calculated independently, and the consistency of the index caliber and the index result can be ensured. The technical scheme can also be implemented by using a traditional relational database or a big data platform in a floor mode. However, the problem is also obvious that the index result data cannot be directly inquired, the access is complex, and the upper business application and the BI report tool cannot directly read the data. When the data is needed to be used, the needed data is extracted from the index result data storage scheme according to the requirement of the service on the index data, and the data meeting the requirement is created.
In the first technical scheme, the problems of more data tables, difficult management, data redundancy, inconsistent data, high resource consumption and the like exist; the second technical scheme cannot directly query data, has complex access, needs to perform data extraction for multiple times, and the like, so the inventor considers that a method for uniformly storing index result data is necessary.
Disclosure of Invention
The embodiment of the application provides a method and a device for uniformly storing index result data, which are used for solving the technical problems of more data tables, difficult management, data redundancy, incapability of directly inquiring data, complex number of access and the like in the prior art.
The embodiment of the application provides a unified storage method of index result data, which generates an elastic data table, comprising the following steps:
generating a general dimension column, an index column and a dimension column unique to the index;
the universal dimension refers to the dimension of all indexes, and the column name and the data type of the universal dimension column are fixed and cannot change along with the change of the indexes;
the index column refers to a column for storing index result data, and each index corresponds to one data column and is used for storing the index result data;
the unique dimension column of each index means that each index has unique dimension besides the universal dimension, and the unique dimension column of each index has corresponding data columns in the table; the dimension column unique to the index stores unique data in index result data;
the elastic data table is stored in a storage medium.
Further, the elastic data table supports transverse expansion and longitudinal expansion, wherein the transverse expansion refers to dynamically increasing data columns of the elastic data table when data of index columns are written, and the longitudinal expansion refers to dynamically dividing all index data into different data files by the elastic data table, automatically dividing the files when the index data are written, and writing different index data into different data files.
Further, all the index result data are stored in the elastic data table in a unified way, the index result data with the same dimension in the index result data are stored in the same data column of the elastic data table, and the index result data with different dimensions in the index result data are stored in independent data columns.
Furthermore, the method for uniformly storing the index result data is used for storing enterprise operation management data.
Further, the storage medium is a volatile storage medium or a nonvolatile storage medium.
The embodiment of the application also provides a device for uniformly storing the index result data, which comprises:
the data table generating module generates an elastic data table, which comprises:
generating a general dimension column, an index column and a dimension column unique to the index;
the universal dimension refers to the dimension of all indexes, and the column name and the data type of the universal dimension column are fixed and cannot change along with the change of the indexes;
the index column refers to a column for storing index result data, and each index corresponds to one data column and is used for storing the index result data;
the unique dimension column of each index means that each index has unique dimension besides the universal dimension, and the unique dimension column of each index has corresponding data columns in the table; the dimension column unique to the index stores unique data in index result data;
the storage stores the elastic data table in a storage medium.
Further, the elastic data table supports transverse expansion and longitudinal expansion, wherein the transverse expansion refers to dynamically increasing data columns of the elastic data table when data of index columns are written, and the longitudinal expansion refers to dynamically dividing all index data into different data files by the elastic data table, automatically dividing the files when the index data are written, and writing different index data into different data files.
Further, the storage medium is a volatile storage medium or a nonvolatile storage medium.
The embodiment of the application also provides computer equipment, which comprises a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor realizes the steps of the method when executing the computer program.
Embodiments of the present application also provide a computer readable medium having non-volatile program code executable by a processor, the program code causing the processor to perform the aforementioned method.
The embodiment provided by the application has at least the following beneficial effects:
the invention provides a unified storage method of index result data, which generates an elastic data table, comprising the following steps: generating a general dimension column, an index column and a dimension column unique to the index; the universal dimension refers to the dimension of all indexes, and the column name and the data type of the universal dimension column are fixed and cannot change along with the change of the indexes; the index column refers to a column for storing index result data, and each index corresponds to one data column and is used for storing the index result data; the unique dimension column of each index means that each index has unique dimension except the universal dimension, and the unique dimension column of each index has a corresponding data column; the dimension column unique to the index stores unique data in index result data; the elastic data table is stored in a storage medium. By adopting the technical scheme, the technical problems that in the prior art, the number of data tables is large, management is difficult, data redundancy is realized, data cannot be directly inquired, the number of access is complex and the like can be solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:
FIG. 1 is a flowchart illustrating a method for unified storage of index result data according to an embodiment of the present application;
fig. 2 is a schematic diagram of a query module for unified storage of index result data according to an embodiment of the present application;
fig. 3 is a query flow chart for unified storage of index result data according to an embodiment of the present application.
Detailed Description
For the purposes, technical solutions and advantages of the present application, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
Referring to fig. 1, the present application provides a method for unified storage of index result data, which generates an elastic data table, including:
generating a general dimension column, an index column and a dimension column unique to the index;
the universal dimension refers to the dimension of all indexes, and the column name and the data type of the universal dimension column are fixed and cannot change along with the change of the indexes;
the index column refers to a column for storing index result data, and each index corresponds to one data column and is used for storing the index result data;
the unique dimension column of each index means that each index has unique dimension besides the universal dimension, and the unique dimension column of each index has corresponding data columns in the table; the dimension column unique to the index stores unique data in index result data;
the elastic data table is stored in a storage medium.
In order to overcome the defects of the prior art, the invention provides a unified storage method of index result data, and provides a distributed data storage and query method, device and system.
The method for uniformly storing the index result data is that the result data of all indexes can be uniformly stored in an elastic data table by designing the elastic data table, the indexes with the same dimension are stored in the same data column of the elastic data table, and different dimensions among the indexes are respectively stored in independent data columns, wherein the different columns correspond to different dimensions of different indexes. The results of the elastic data table are shown in table 5 below:
table 5: elastic data Table structural example
Table 5 consists of 3 types of columns: a general dimension column, an index column, and a dimension column unique to the index. The universal dimension column refers to the dimension of all indexes, such as time, index name, index code, index unit, index version and the like, and the column names and the data types of the columns are fixed and cannot be changed according to different indexes. The index column refers to a column for storing index values, each index has a specific data column for storing the values of the index, and the data type of the column is associated with the index. The unique dimension column of the index means that each index has other specific dimensions besides the general dimension, and the dimensions have corresponding data columns. Taking the data of rows 3 and 4 in table 5 as an example, row 3 represents a data record with index code C01 and index name a, the record represents data time of 1 st 2010, version V1.0, the unit is human, the value of the index is stored in the i1 data column, and the data record also comprises two dimensions D1 and D2, and the values of the two dimensions are D1 and D2 respectively. And row 4 represents the data of index B stored in the i2 data column, which contains dN dimension whose value is dN. It can be seen that the index a and the index B are different indices, and the values of the two indices are stored in different data columns, including partially identical dimensions, and also including different dimensions unique to each other, but may be stored in a unified elastic data table.
The data tables of conventional database technology require an initial table construction operation and define all columns and column types at the time of the table construction. The elastic data table in the invention does not need to be subjected to table building operation, the structure of the table is defined during data writing, and dynamic expansion is supported. The expansion here includes both transverse expansion and longitudinal expansion. The lateral expansion means that the data columns of the elastic data table can be dynamically increased when the index result data is written. The longitudinal expansion means that the elastic data table dynamically divides all index result data into different data files, the index result data is automatically divided into files when written, and the different index result data is written into different data files.
Data warehouses built based on conventional databases or large data platforms are not capable of supporting the above-described flexible data tables. In order to realize the floor-type elastic data table, the invention provides a system and a storage engine device for supporting distributed data storage and query, which technically adopt a distributed architecture, and each node module can support high availability of service and comprises a management node module, a data node module and a query node module, as shown in figure 2.
The management node module performs unified management and data scheduling on the whole system, including metadata information and data fragmentation information of a management elastic data table, data distribution conditions of the management data node module and the like. The management node module realizes automatic switching of the active and standby modes by means of the Zookeeper, and ensures that the service of the management node module is high and available.
The query node module provides data query request service to the outside and supports two connection modes of HTTP and JDBC for query. The query node module caches information of the elastic data table in the whole system, including table structure information, data slicing maintained by each data node module and the like. The query node module intelligently splits the query request into a plurality of sub-queries, distributes the sub-queries to the optimal data node module, combines and summarizes the results returned by each sub-query, and returns the final results to the request terminal.
The data node module is used for storing the data file of the elastic data table and providing the query service of the data. After receiving the sub-query forwarded by the query node module, the data node module finds out the data fragments meeting the query condition according to the query condition, reads the data records meeting the condition from the data file respectively by using multithreading, then combines and gathers a plurality of data from different data fragments, and finally returns the query result of the sub-query to the query node module.
Index result data is typically obtained by performing final statistics based on data that has been processed multiple times. In order to write the index result data more conveniently, the storage engine device can start a data writing task in a data node module, wherein the task comprises an SQL statement for calculating the index result, the data node module is connected to a source database in a JDBC mode, sends the SQL statement to the source database, the source database finishes the calculation of the index result, and waits for the source database to return a query result. After the source database returns the query result, the data node module receives the returned index result data and writes the index result data into the data slicing file. When data is written into the data slicing files, the data is divided according to a strategy, wherein the strategy refers to classifying the indexes according to the theme or service classification of the indexes, the indexes of the same theme or service classification are stored in the same data slicing, in one data slicing, the approximate average distribution can be carried out according to the record number of the index result data, and a batch of index result data is written into different data files of the same data slicing. Each data file contains a value of index result data and metadata information such as a data column stored in the data file. The data node module preloads metadata information in each data fragment file.
As described above, after the request end initiates the query request, the query node first receives the query, splits the query into a plurality of sub-queries, distributes the sub-queries to a plurality of data node modules, the data node reads data records from different data slices respectively, returns the combined and summarized data to the query node module, and the query node module further combines and summarizes the data returned by the plurality of data node modules, and finally returns the final calculation result to the request end. The specific flow is shown in the following figure 3:
the request end is connected to the query node module in an HTTP or JDBC mode, and after a query request is initiated, the system processes the query request as follows:
1. inquiring the node module: receiving a query request initiated by a request end;
2. inquiring the node module: analyzing information such as a data table requested in a query request, a requested index and the like, finding out data fragments where data meeting the conditions are located and data node modules where the data fragments are located according to the information of the elastic data table in the cached system, and selecting the most suitable data node module according to the information such as loads;
3. inquiring the node module: based on the analysis result of the last step, splitting the query request into a plurality of sub queries, and increasing proper query conditions to reduce the data volume processed by the data node module;
4. inquiring the node module: distributing sub-queries to a plurality of most appropriate data node modules respectively;
5. and the data node module is used for: receiving a sub-query request distributed by a query node module;
6. and the data node module is used for: analyzing a data table requested in the sub-query request, and judging data fragments in which data is likely to be in based on the request condition and the preloaded metadata information;
7. and the data node module is used for: using a multithreading technology to perform parallel processing, and reading data meeting the conditions from each data slicing file;
8. and the data node module is used for: merging/summarizing data obtained from different data slices in a data node module;
9. and the data node module is used for: returning the local merging/summarizing result to the query node module;
10. inquiring the node module: after the query node module distributes sub-queries to the data node modules, waiting for each data node module to finish the sub-query processing, and receiving the local merging/summarizing results returned by each data node module;
11. inquiring the node module: merging/summarizing the data results returned by the plurality of data node modules after receiving the local results;
12. inquiring the node module: and returning the final query result to the request end.
The embodiment of the application also provides a device for uniformly storing the index result data, which comprises:
the data table generating module generates an elastic data table, which comprises:
generating a general dimension column, an index column and a dimension column unique to the index;
the universal dimension refers to the dimension of all indexes, and the column name and the data type of the universal dimension column are fixed and cannot change along with the change of the indexes;
the index column refers to a column for storing index result data, and each index corresponds to one data column and is used for storing the index result data;
the unique dimension column of each index means that each index has unique dimension except the universal dimension, and the unique dimension column of each index has a corresponding data column; the dimension column unique to the index stores unique data in index result data;
the storage stores the elastic data table in a storage medium.
Further, the elastic data table supports transverse expansion and longitudinal expansion, wherein the transverse expansion refers to dynamically increasing data columns of the elastic data table when data of index columns are written, and the longitudinal expansion refers to dynamically dividing all index data into different data files by the elastic data table, automatically dividing the files when the index data are written, and writing different index data into different data files.
Further, all the index result data are stored in the elastic data table in a unified way, the index result data with the same dimension in the index result data are stored in the same data column of the elastic data table, and the index result data with different dimensions in the index result data are stored in independent data columns.
Furthermore, the method for uniformly storing the index result data is used for storing enterprise operation management data.
Further, the storage medium is a volatile storage medium or a nonvolatile storage medium.
The embodiment of the application also provides computer equipment, which comprises a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor realizes the steps of the method when executing the computer program.
Embodiments of the present application also provide a computer readable medium having non-volatile program code executable by a processor, the program code causing the processor to perform the aforementioned method.
It should be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.
Claims (10)
1. A method for unified storage of index result data, characterized in that an elastic data table is generated, comprising:
generating a general dimension column, an index column and a dimension column unique to the index;
the universal dimension refers to the dimension of all indexes, and the column name and the data type of the universal dimension column are fixed and cannot change along with the change of the indexes;
the index column refers to a column for storing index result data, and each index corresponds to one data column and is used for storing the index result data;
the unique dimension column of each index means that each index has unique dimension besides the universal dimension, and the unique dimension column of each index has corresponding data columns in the table; the dimension column unique to the index stores unique data in index result data;
the elastic data table is stored in a storage medium.
2. The method for unified storage of index result data according to claim 1, wherein the elastic data table supports lateral expansion and longitudinal expansion, the lateral expansion means that data columns of the elastic data table are dynamically increased when data of index columns are written, the longitudinal expansion means that the elastic data table dynamically partitions all index data into different data files, file partitioning is automatically performed when index data are written, and different index data are written into different data files.
3. The method of unified storage of index result data according to claim 1, wherein all index result data are unified stored in the elastic data table, index result data of the same dimension in the index result data are stored in the same data column of the elastic data table, and index result data of different dimensions in the index result data are stored in separate data columns.
4. The method for unified storage of index result data according to claim 1, wherein the method for unified storage of index result data is used for storing enterprise management data.
5. The method for unified storage of index result data according to claim 1, wherein the storage medium is a volatile storage medium or a nonvolatile storage medium.
6. An apparatus for unified storage of index result data, comprising:
the data table generating module generates an elastic data table, and comprises a general dimension column, an index column and a dimension column unique to the index;
the universal dimension refers to the dimension of all indexes, and the column name and the data type of the universal dimension column are fixed and cannot change along with the change of the indexes;
the index column refers to a column for storing index result data, and each index corresponds to one data column and is used for storing the index result data;
the unique dimension column of each index means that each index comprises unique dimensions besides general dimensions, and the unique dimensions of the indexes are listed in a table with data columns corresponding to the indexes; the dimension column unique to the index stores unique data in index result data;
the storage stores the elastic data table in a storage medium.
7. The device for unified storage of index result data according to claim 6, wherein the elastic data table supports lateral expansion and longitudinal expansion, the lateral expansion means that data columns of the elastic data table are dynamically increased when data of index columns are written, the longitudinal expansion means that the elastic data table dynamically partitions all index data into different data files, file partitioning is automatically performed when index data are written, and different index data are written into different data files.
8. The apparatus for unified storage of index result data according to claim 6, wherein the storage medium is a volatile storage medium or a nonvolatile storage medium.
9. A computer device comprising a memory, a processor, the memory having stored therein a computer program executable on the processor, characterized in that the processor, when executing the computer program, implements the steps of the method of any of the preceding claims 1 to 5.
10. A computer readable medium having non-volatile program code executable by a processor, the program code causing the processor to perform the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310413526.1A CN116401254A (en) | 2023-04-17 | 2023-04-17 | Unified storage method and device for index result data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310413526.1A CN116401254A (en) | 2023-04-17 | 2023-04-17 | Unified storage method and device for index result data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116401254A true CN116401254A (en) | 2023-07-07 |
Family
ID=87012202
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310413526.1A Pending CN116401254A (en) | 2023-04-17 | 2023-04-17 | Unified storage method and device for index result data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116401254A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8639650B1 (en) * | 2003-06-25 | 2014-01-28 | Susan Pierpoint Gill | Profile-responsive system for information exchange in human- and device-adaptive query-response networks for task and crowd management, distributed collaboration and data integration |
| CN108776690A (en) * | 2018-06-05 | 2018-11-09 | 上海孚典智能科技有限公司 | The method of HDFS Distribution and Centralization blended data storage systems based on separated layer handling |
| CN110162515A (en) * | 2019-04-30 | 2019-08-23 | 中国科学院深圳先进技术研究院 | A kind of uncoupled elastic data warehouse schema |
| CN110489409A (en) * | 2018-09-14 | 2019-11-22 | 北京数聚鑫云信息技术有限公司 | A kind of method and device of dynamic expansion data field |
| CN113791729A (en) * | 2021-08-11 | 2021-12-14 | 合肥先进产业研究院 | Dynamic capacity expansion method for monitoring data storage of internet of things equipment |
| CN114357088A (en) * | 2021-12-14 | 2022-04-15 | 中核武汉核电运行技术股份有限公司 | Nuclear power industry data warehouse system |
| CN115129768A (en) * | 2022-05-25 | 2022-09-30 | 网易（杭州）网络有限公司 | Node capacity expansion method of distributed search engine |
| CN115577028A (en) * | 2022-10-09 | 2023-01-06 | 齐鲁工业大学 | ETL tool construction method based on module programmable extension |
| CN115905239A (en) * | 2022-12-16 | 2023-04-04 | 中盈优创资讯科技有限公司 | Method and device for realizing highly-multiplexed performance index data retrieval |
-
2023
- 2023-04-17 CN CN202310413526.1A patent/CN116401254A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8639650B1 (en) * | 2003-06-25 | 2014-01-28 | Susan Pierpoint Gill | Profile-responsive system for information exchange in human- and device-adaptive query-response networks for task and crowd management, distributed collaboration and data integration |
| CN108776690A (en) * | 2018-06-05 | 2018-11-09 | 上海孚典智能科技有限公司 | The method of HDFS Distribution and Centralization blended data storage systems based on separated layer handling |
| CN110489409A (en) * | 2018-09-14 | 2019-11-22 | 北京数聚鑫云信息技术有限公司 | A kind of method and device of dynamic expansion data field |
| CN110162515A (en) * | 2019-04-30 | 2019-08-23 | 中国科学院深圳先进技术研究院 | A kind of uncoupled elastic data warehouse schema |
| CN113791729A (en) * | 2021-08-11 | 2021-12-14 | 合肥先进产业研究院 | Dynamic capacity expansion method for monitoring data storage of internet of things equipment |
| CN114357088A (en) * | 2021-12-14 | 2022-04-15 | 中核武汉核电运行技术股份有限公司 | Nuclear power industry data warehouse system |
| CN115129768A (en) * | 2022-05-25 | 2022-09-30 | 网易（杭州）网络有限公司 | Node capacity expansion method of distributed search engine |
| CN115577028A (en) * | 2022-10-09 | 2023-01-06 | 齐鲁工业大学 | ETL tool construction method based on module programmable extension |
| CN115905239A (en) * | 2022-12-16 | 2023-04-04 | 中盈优创资讯科技有限公司 | Method and device for realizing highly-multiplexed performance index data retrieval |
Non-Patent Citations (1)
| Title |
|---|
| 滕爱国;单新文;王鹏飞;陶晔波;闾龙;顾玉皎;: "基于Hadoop平台电力数据服务匹配查询性能研究", 计算机技术与发展, no. 09 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112534396B (en) | Diary watch in database system | |
| JP7130600B2 (en) | Implementing semi-structured data as first-class database elements | |
| US11030186B2 (en) | Incremental refresh of a materialized view | |
| US7917512B2 (en) | Method for automated design of range partitioned tables for relational databases | |
| US7174345B2 (en) | Methods and systems for auto-partitioning of schema objects | |
| US8543596B1 (en) | Assigning blocks of a file of a distributed file system to processing units of a parallel database management system | |
| US7949687B1 (en) | Relational database system having overlapping partitions | |
| US9418092B2 (en) | Index selection in a multi-system database management system | |
| WO2017170459A1 (en) | Method, program, and system for automatic discovery of relationship between fields in environment where different types of data sources coexist | |
| Zhan et al. | AnalyticDB: real-time OLAP database system at Alibaba cloud | |
| JP6159908B1 (en) | Method, program, and system for automatic discovery of relationships between fields in a heterogeneous data source mixed environment | |
| US8214408B2 (en) | Method, database system and computer program for joining temporal database tables | |
| JPWO2017170459A6 (en) | Method, program, and system for automatic discovery of relationships between fields in a heterogeneous data source mixed environment | |
| El Alami et al. | Supply of a key value database redis in-memory by data from a relational database | |
| CN111078705A (en) | Spark platform based data index establishing method and data query method | |
| CN116541427B (en) | Data query method, device, equipment and storage medium | |
| US9141632B1 (en) | Selecting a compression technique | |
| D’silva et al. | Secondary indexing techniques for key-value stores: Two rings to rule them all | |
| US10810219B2 (en) | Top-k projection | |
| CN116401254A (en) | Unified storage method and device for index result data | |
| US8290935B1 (en) | Method and system for optimizing database system queries | |
| CN112000666B (en) | Database management system of facing array | |
| Munir et al. | ATUN-HL: Auto tuning of hybrid layouts using workload and data characteristics | |
| US20040236724A1 (en) | Searching element-based document descriptions in a database | |
| US11853298B2 (en) | Data storage and data retrieval methods and devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |