CN103425772A - Method for searching massive data with multi-dimensional information - Google Patents

Abstract

A massive data query method with multi-dimensional information relates to the field of data mining. Load the dimensional information of massive data with multi-dimensional information; load massive data; use online data analysis OLAP method to query massive data. A method for querying massive data with multi-dimensional information in the present invention organizes the massive data with multi-dimensional information through the method of dimension coding, uses the method of data block storage to simplify the addressing of data blocks, and uses intermediate variables (i.e. analysis paths) In this way, the transformation of dimension level can be quickly realized, and the data selection based on the data block selection method can be used for data screening, and only the actual participating data can be calculated and processed.

Description

A massive data query method with multi-dimensional information

technical field

The invention relates to the field of data mining, in particular to a massive data query method with multi-dimensional information.

Background technique

With the advent of the era of big data, it poses great challenges to traditional data analysis fields such as traditional data management and query. In order to deal with the challenges brought by massive data, the MapReduce programming model and distributed file system are widely used in academia and industry to deal with this challenge. OLAP (On-Line Analytical Processing) is a very important analysis means and method in the field of traditional data analysis. In the field of big data, new requirements are also put forward for OLAP analysis.

OLAP can be divided into three types according to its implementation methods: ROLAP (Relational OLAP Relational Online Data Analysis Processing), MOLAP (Multidimensional OLAP Multidimensional Online Analytical Processing) and HOLAP (Hybrid OLAP Hybrid Online Analytical Processing). Among them, ROLAP uses relational tables to store dimensional information and fact data, MOLAP uses multidimensional data structures to store dimensional information and fact data, and HOLAP is called hybrid OLAP, which combines ROLAP and MOLAP technologies. No matter what kind of OLAP it is, it needs the support of storage and computing platforms, especially in the environment of big data. In order to solve the many challenges brought by big data, many new technologies have emerged in academia and industry, such as distributed file system, NoSQL (Not Only Structured Query Language) database system, MapReduce programming model and Related optimization methods, these techniques are widely used in big data analysis.

There are two commonly used OLAP optimization methods in the big data environment: optimizing OLAP performance by using the results of precomputation and condensing data cubes, and optimizing OLAP performance by optimizing storage structures and algorithms. However, the former will generate a large amount of data and cannot be applied to a massive data environment, while the latter's optimization measures are mostly based on ROLAP, which does not improve the performance of OLAP qualitatively. Some studies have proposed the SPAJG-OLAP subset in OLAP queries, and studied the optimization strategies and implementation technologies of massive parallel processing frameworks for massive data in terms of storage, query, data distribution, network transmission, and distributed caching. This research optimizes ROLAP performance based on parallel database technology, and achieves the purpose of accelerating OLAP by optimizing OLAP query and storage. However, because ROLAP is based on relational database technology, a large number of connection operations will be generated. When the amount of data is very large, its optimization The effect is not obvious.

As far as distributed OLAP systems are concerned, some Hadoop-based cloud database systems, such as Hive, HadoopDB, HBase, etc., all support OLAP. In the current field of distributed OLAP for massive data, methods such as data indexing and fragmentation are widely used to optimize ROLAP. However, ROLAP needs to adopt a relational model and resource-consuming connection operations. When the amount of data increases, the optimization effect of indexing and fragmentation drops sharply. In addition, there is a method to optimize ROLAP by querying conditions. But also due to the unavoidable connection operation, its optimization effect is not very obvious. MOLAP stores data as a data cube, but it needs to manage and optimize the dimensions, resulting in no authoritative reports on the current research on MOLAP and the system.

Contents of the invention

In view of the shortcomings of the existing invention, the purpose of the present invention is to provide a massive data query method with multi-dimensional information, so as to achieve the purpose of data query and aggregation calculation in the massive data environment.

The technical solution of the present invention is achieved in this way: a massive data query method with multi-dimensional information comprises the following steps:

Step 1: Load the dimensional information of massive data with multi-dimensional information, specifically including the following steps:

Step 1.1: Identify the dimensional information of the massive data, and judge whether each dimensional information of the massive data satisfies the following three constraints at the same time:

Constraint 1: A dimension consists of one and only one dimension level, that is, a dimension is a total order relationship composed of all dimension levels;

Constraint 2: In any dimension level of a dimension, only one dimension attribute is included, and the dimension attribute contains several dimension values;

Constraint 3: In the dimension value tree composed of all dimension values, sibling nodes contain the same number of child nodes;

If all are satisfied, go to step 1.3, otherwise go to step 1.2;

Step 1.2: Process the dimension information so that each dimension forms a dimension value tree structure conforming to the constraints. The processing process is as follows:

Constraint 1: If there are multiple dimension levels, discard the dimension levels as needed, and only keep one dimension level;

Constraint 2: If a dimension level contains multiple dimension attributes, discard the dimension attributes as needed, and only keep one dimension attribute;

For constraint 3: if the number of child nodes contained in the sibling nodes is different, add a null value to make the number of child nodes of the sibling nodes the same;

Step 1.3: Encoding dimension information;

For the dimension values of each dimension level in the dimension value tree, encode them sequentially in decimal numbers from left to right. When all the dimension values have corresponding encodings, the encoding work ends;

Step 1.4: store the encoding of the dimension information;

For any dimension information of massive data, store the name of each dimension level and the number of sibling nodes in this level, and finally form a file of all dimension information of massive data, which is stored in the distributed file system;

Step 2: Load massive data;

Step 2.1: The user establishes the corresponding relationship between the actual meaning of the dimension information of the massive data and the coding of the dimension information according to the needs, that is, to express any piece of data in the massive data with the coding of the dimension information;

Step 2.2: All the most fine-grained multi-dimensional massive data form a data cube structure, and any piece of data in the massive data is used as a cell in the data cube, and the information of the cell includes: the coordinates of the cell within the cube, and The fact data value represented by the cell; where the coordinates of the cell are expressed as:

<encoding of information in one dimension, encoding of information in another dimension, ..., encoding of information in the last dimension>;

The finest granularity mentioned refers to: the data pointed to by the lowest dimension level;

Step 2.3: Cut the data cube:

According to the user's query requirements, under the condition of ensuring the shortest query time, the data cube is cut to form a data block, and the side length of the data block is determined;

Step 2.4: Encoding the data block divided in step 2.3, the method is: divide the coordinates of any cell in the data block by the side length of the data block, and round up the obtained data to obtain the value as the encoding of the data block;

Step 2.5: The data block cut in step 2.3 is stored in the distributed file system, and the code established in step 2.4 is used as the name of the data block file;

Step 3: Use online data analysis OLAP method to query massive data;

Step 3.1: The user sets query conditions, including:

Query target: refers to determine which data cube to query, that is, the target cube;

Query scope: in the established query target, which part of the data is queried;

The dimension information of the result: refers to the dimension information of the result data cube;

Aggregation method: the operation of aggregating the data within the query range;

Step 3.2: Determine whether the query conditions set in step 3.1 meet the following constraints:

Constraint 1: The query target already exists, and the query range should be less than or equal to the data range of the query target;

Constraint 2: The number of dimensions of the result data cube should be less than or equal to the number of dimensions of the query target;

Constraint 3: The lowest dimension level of any dimension of the result data cube should be higher than the lowest dimension level of the corresponding dimension of the query target;

Constraint 4: Aggregation methods must be distributed or algebraic;

If constraint 1, constraint 2 and constraint 4 are satisfied at the same time, then step 3.3 is performed;

If constraints 1 to 4 are satisfied at the same time, execute step 3.4;

If any of the above conditions are not met, the query fails and ends;

Step 3.3: Convert the query target, find the upper-level cube of the current query target, and judge whether the upper-level cube satisfies constraint 3, if not, continue to query the upper-level cube of the upper-level cube, if it still cannot If it is satisfied, the query fails and the query process ends; if the upper-level cube that satisfies constraint 3 is found, replace the cube with the target cube;

Step 3.4: Coarse screening of data: determine the range of the minimum data block required for query according to the query range;

Step 3.5: Fine screening of data; scan the data block files screened in step 3.4, and filter all the cells in the data block according to the query range, if the cells are in the query range, perform step 3.6; otherwise, discard the cell;

Step 3.6: Use dimension coding to change the dimension level of the cell, compare the dimension information of the result data cube with the dimension information of the target cube, determine the changed dimension information, and modify the coordinates representing the dimension on the cell coordinates;

Step 3.7: For cells with the same coordinates, aggregate the fact data values in the cells according to the set aggregation method;

Step 3.8: The data gathered in step 3.7 forms a result data cube, returns the information of the result data cube to the user, and stores the result cube as a new data cube so that it can be used as the query target of the next round of query .

Beneficial effects of the present invention: a massive data query method with multi-dimensional information in the present invention has the following advantages:

1. Organize massive data with multi-dimensional information through the method of dimensional coding

2. The method of storing data in blocks simplifies the addressing of data blocks.

3. Use dimension encoding to quickly realize dimension-level conversion.

4. Through the data screening based on the method of data block selection, only the actual participating data is calculated and processed.

Description of drawings

Fig. 1 is a schematic diagram of a dimension structure in a marine scientific data set according to an embodiment of the present invention;

Fig. 2 is a schematic diagram of time dimension encoding in an embodiment of the present invention;

Fig. 3 is a schematic diagram of area dimension coding in an embodiment of the present invention;

FIG. 4 is a schematic diagram of depth-dimensional coding in an embodiment of the present invention;

FIG. 5 is a schematic diagram of the time dimension of an embodiment of the present invention that does not meet constraint 1;

Fig. 6 is a schematic diagram of a multi-dimensional massive data cube structure in an embodiment of the present invention;

Fig. 7 is a block schematic diagram of a multi-dimensional mass data cube in an embodiment of the present invention;

Fig. 8 is a schematic diagram of coarse screening of data according to an embodiment of the present invention;

Fig. 9 is a schematic diagram of a method for realizing massive data query by MapReduce in an embodiment of the present invention;

Fig. 10 is a schematic diagram of changing dimension coding in an embodiment of the present invention;

Fig. 11 is an example diagram of changing dimension coding in MapReduce in an embodiment of the present invention;

FIG. 12 is a flowchart of a massive data query method with multi-dimensional information according to an embodiment of the present invention.

Detailed ways

Hereinafter, the present invention will be further described in detail by taking certain marine scientific data as an example, in conjunction with the accompanying drawings and specific implementation methods.

This embodiment takes certain marine scientific data as the research object, and the marine data is massive data with three-dimensional information. Including: Time Dimension, Area Dimension and Depth Dimension, which are represented by Time, Area and Depth respectively. The dimension structure is shown in Figure 1. Among them, Time has four levels: Year, Month, Day, and Slot. Among them, Slot refers to a test in three time periods of a day, and its dimension values are "morning", "afternoon" and "evening", respectively, and its dimension value tree structure is shown in Figure 2. In Figure 2, if there are 3 years of data (January 1, 2010 to December 31, 2012), there are three corresponding dimension values at the Year dimension level, namely 2010, 2011, and 2012. On the Month dimension level, there are 36 dimension values corresponding to January 2010, February 2010, ..., December 2012. On the Day dimension level, we assume that each month has 31 day, it corresponds to 1116 dimension values, which are January 1, 2010, January 2, 2010, ..., December 31, 2012, and there are 3348 dimensions at the Slot dimension level The values are the morning of January 1, 2010, the noon of January 1, 2010, ..., and the afternoon of December 31, 2012. Area has 7 dimension levels: 1°, 1/2°, 1/4°, 1/8°, 1/16°, 1/32° and 1/64°. Among them, 1° refers to a square area composed of 1° longitude and 1° latitude. The earth’s surface can be divided into 360×180 1° square areas, and its dimension value tree structure is shown in Figure 3. In Figure 3, the data of the 2° area is included, and the 1° dimension level contains 2 dimension values, which are 1° and 2° respectively, and the 1/2° dimension level contains 4 dimension values, which are 1/ 2°, 1°, 3/2°, 2°, and so on, including 128 dimension values at the 1/64° dimension level, which are 1/64°, 1/32°, ..... ., 2°. Depth has three dimensional levels: 100m, 50m, and 1Om. Among them, 100m refers to "the depth of the ocean is divided at intervals of 100 meters", and its dimensional value tree structure is shown in Figure 4. In Figure 4, if the water depth is 500m, at the dimension level of 100m, there are 5 dimension values corresponding to 100m, 200m, ..., 500m, and at the dimension level of 50m, there are 10 values Dimension values are 50m, 100m, ..., 500m, respectively, and at the dimension level of 1Om, there are 50 dimension values, respectively 1Om, 20m, ..., 500m. In the cube of the ocean dataset, there is only one fact data in each cell, that is, the temperature data of the ocean, and the data type of this data is a double-precision floating-point number.

In this embodiment, the method for querying the above-mentioned marine data with 3-dimensional information, the process of which is shown in Figure 12, includes the following steps:

Step 1: Load the dimensional information of massive data with 3-dimensional information, specifically including the following steps:

Step 1.1: Identify the dimensional information of the massive data, and judge whether each dimensional information of the massive data satisfies the following three constraints at the same time:

Constraint 1: A dimension consists of one and only one dimension level, that is, a dimension is a total order relationship composed of all dimension levels.

In this embodiment, for marine data, if the time dimension includes the dimension level of week (for example, the morning of Friday, January 1, 2010), since the month cannot contain a complete week (for example, January 1, 2010 day is the beginning of January, but this day is Friday, not the beginning of a complete week), so two dimension levels will be generated, which are year (year)-week (week)-day (day)-time (slot ), year (year)-month (month)-day (day)-time (slot), this dimension does not meet the constraints, and its schematic diagram is shown in Figure 5. The total order relationship defined in this embodiment, for the time dimension of marine data, should satisfy the year containing season, season containing month, month containing day, and day containing time. If such a relationship is not satisfied, the constraint is not satisfied.

Constraint 2: In any dimension level of a dimension, only one dimension attribute is included, and the dimension attribute contains several dimension values.

Consider a hypothetical city dimension whose dimension level is: province-city-district (for example, Liaoning Province-Shenyang City-Heping District). At the city dimension level, it contains two dimension attributes, which are city name and city area code (for example: Shenyang/024). This dimension does not conform to the constraints.

Constraint 3: In the paired dimension value tree composed of all dimension values, sibling nodes contain the same number of child nodes;

In this embodiment, for marine data, in real cases, each month contains different numbers of days in the time dimension (for example, January 2010 contains 31 days, February 2010 contains 28 days, etc.), Reflected in the dimension value tree, the months in the same year are sibling nodes, for example, January 2010 and February 2010 are sibling nodes, and their parent node is 2010. Then for January 2010, the number of child nodes is 31 (including 31 days in January 2010), and for February 2010, the number of child nodes is 28 (including 28 days in February 2010) . Then the time dimension does not fit the constraint.

If satisfied, go to step 1.3, otherwise go to step 1.2;

Step 1.2: Process the dimension information so that each dimension forms a dimension value tree structure conforming to the constraints. The processing process is as follows:

Constraint 1: If there are multiple dimension levels, discard the dimension levels as needed, and only keep one dimension level;

For example, for the time dimension of Friday, January 1, 2010, since there are two dimension levels, the level of the week dimension needs to be discarded. Finally, the time dimension as shown in Figure 1 is formed.

Constraint 2: If a dimension level includes multiple dimension attributes, discard the dimension attributes as needed, and only keep one dimension attribute;

In a hypothetical city dimension, there are 2 dimension attributes in the dimension level of city, city name and city number. In order to comply with constraint 2, the dimension attribute city name is deleted, and only the city number is retained, so that the city dimension complies with constraint 2.

Constraint 3: If sibling nodes contain different numbers of child nodes, add a null value to make the number of child nodes of sibling nodes the same.

For example, in this embodiment, it is assumed that the time dimension information of marine data is: there are 12 months in a year, and each month has 31 fixed days. In this assumption, there is a situation that does not match the actual situation. There is data on February 31, 2010, but this date does not exist. In this embodiment, the time dimension is revised by setting a null value. For example, if it is a leap year, set the 29th to 31st as null Value, if it is a non-leap year, set the 30th-31st to a null value. For months with only 30 days, set the 31st to null. The final dimension value tree of the time dimension is shown in Figure 2. In Figure 2, the dimension value tree structure of the time dimension is shown. It is fixed that there are 31 days in each month. If there are less than 31 days, use a null value to make up.

Step 1.3: Encoding dimension information;

For the dimension values of each level in the dimension value tree, encode them sequentially in decimal numbers from left to right. When all the dimension values have corresponding encodings, the encoding work ends;

In this embodiment, the result after encoding the time dimension is shown in FIG. 2 . For the time dimension, it does not conform to the assumption of the dimension in this patent, so each month is fixed at 31 days, and if there are less than 31 days, a null value is used to make up. There are actually only 28 days in February 2010, so use 3 null values to make up 31 days. In Figure 2, only one null value is displayed, which is February 31, 2010. In this embodiment, the marine scientific data set contains 3 years of ocean water temperature data, so there are 3 dimensional values at the year level, coded from left to right, and the codes are 0, 1, 2; There are 36 dimension values at the level (from January 2010 to December 2013), and their codes are 0, 1, ..., 35; there are 1116 dimension values at the daily dimension level, and the codes are 0 , 1, ..., 1115; there are 3348 dimension values at the sub-dimensional level, and the codes are 0, 1, ..., 3347 respectively. The encoding of the region dimension is shown in Figure 3. In this embodiment, the marine scientific data set contains data involving a 2° square area, and contains 2 dimension values at the 1° dimension level, which are coded as 0 and 1, and so on, at the lowest dimension level (1/ 64°), there are 128 dimension values in total, coded as 0, 1, ..., 127 respectively. The encoding of the depth dimension is shown in Figure 4. In this embodiment, the marine scientific data set contains data involving a layer depth of 500 meters. There are 5 dimension values at the 100m dimension level, and their codes are 0, 1, ..., 4; there are 10 dimension values at the 50m dimension level, and their codes are 0, 1, ... ..., 9; there are 50 dimension values at the 10m dimension level, and their codes are 0, 1, ..., 49 respectively.

Step 1.4: store the encoding of the dimension information;

For any dimension information of massive data, store the name of each dimension level and the number of sibling nodes in this level, and finally form a file of all dimension information of massive data, which is stored in the distributed file system.

In this embodiment, the encoded result of the time dimension is shown in FIG. 2 . In the dimension value tree of the time dimension, the year dimension level contains 3 dimension values in total, and the 3 dimension values are sibling nodes to each other, and its parent node is ALL, which means all data, and the number of sibling nodes is 3; the month level contains 36 Nodes in the same year are sibling nodes, and their parent nodes are nodes of that year. For example, under 2010, there are 12 child nodes from January 2010 to December 2010, and the number of sibling nodes under the level The number is 12; the daily level contains a total of 1116 nodes, among which the nodes under the same month are sibling nodes, and their parent nodes are the nodes of this month. There are 31 nodes on the 31st of the month, so the number of sibling nodes in this level is 31; similarly, in the sub-level, the number of sibling nodes is 3. In this embodiment, an XML file is used to store the time dimension, and the storage file is shown in Table 1. The file contains the name of the time dimension Time, the basic name of each dimension and the number of sibling nodes, for example, for the year dimension level, the name is Year, and the number of sibling nodes is 3.

Table 1 is the time dimension stored in XML form

Step 2: Load massive data;

Step 2.1: The user establishes the corresponding relationship between the actual meaning of the dimension information of the massive data and the coding of the dimension information according to the needs, that is, to express any piece of data in the massive data with the coding of the dimension information;

The translator of dimension coding and dimension value meaning is implemented by users according to their own needs. For example, in this embodiment, on the morning of January 1, 2010 on the time dimension, the code at the sub-dimensional level can be obtained through the translator as 0, and 1/64° on the area dimension can be obtained through the translator at 1/ The code on the 64° dimension level is 0. The commonly used implementation methods of translators are as follows: when the amount of data is large, a database is used for storage and search. For example, create a table in the database, and there are only two columns in the table, which respectively store the meaning of the dimension value and the corresponding code of the dimension value; when the number of dimension values is small, use distributed memory for storage and search. For example, create a hash table in distributed memory to store the meaning of the dimension value and the corresponding encoding of the dimension value; when the encoding and the dimension value have a mathematical correspondence, the encoding is used for direct calculation. For example, for the depth dimension, it can be obtained by dividing the current layer depth by the layer depth division step of the current dimension level and then rounding. For example: the coding of 45m layer depth at 10m dimension level is 45/10 rounded to 4, and 4 is the coding of 45m layer depth at 10m dimension level.

Step 2.2: All the most fine-grained multi-dimensional massive data form a data cube structure, and any piece of data in the massive data is used as a cell in the data cube, and the information of the cell includes: the coordinates of the cell within the cube, and The fact data value represented by the cell; where the coordinates of the cell are expressed as:

<encoding of information in one dimension, encoding of information in another dimension, ..., encoding of information in the last dimension>;

The finest granularity mentioned refers to: the data pointed to by the lowest dimension level;

In this embodiment, the multi-dimensional mass data forms a data cube structure, as shown in FIG. 6 . Any piece of data in the mass data is used as a cell in the data cube, and the information of the cell includes: the coordinates of the cell within the cube, and the factual data value represented by the cell. For ocean data: on the morning of January 1, 2010, the temperature data at 10m water depth in the 1/64° area is 18°C, this piece of ocean data is saved as a cell in the data cube in the distributed file system The information of the unit, including the coordinates <0, 0, 0>, where the first 0 indicates the encoding on the time dimension, that is, the morning of January 1, 2010 is encoded as 0 at the sub-level, and the second 0 indicates the area Dimensional encoding, that is, 1/64° is encoded as 0 at the 1/64° dimension level, and the third 0 indicates that the encoding on the depth dimension is 0, that is, the encoding of 10m at the 10m dimension level is 0, and the fact data 18°C.

Step 2.3: Cut the data cube:

According to the user's query requirements, under the condition of ensuring the shortest query time, the data cube is cut to form a data block, and the side length of the data block is determined;

User's query requirements: Refers to the user's frequently used query conditions and the probability of their occurrence. Taking the user's query requirements as the object will have the following advantages for us: to ensure query efficiency, the system can quickly respond to user queries; to ensure the balance between system parallelism and scheduling costs.

In this embodiment, the method for querying massive data is implemented using the MapReduce programming model. The input of MapReduce in the massive data query method is a data block file, and the performance of the entire query method is closely related to the block size. The smaller the value of the block file, the better the parallelism and the smaller the number of actual calculations, but at this time the scheduling cost becomes higher, so how to determine the value of the block file in a compromise will become very critical. Since it is difficult to enumerate all query conditions and their occurrence probabilities, random sampling is used in this embodiment to extract some query conditions and their occurrence probabilities. In addition, in this embodiment, the value of the block file size also considers the operating environment of the query method. The massive data query method in this embodiment is realized by using MapReduce, so the value of the block file size needs to consider some characteristics of MapReduce, such as file addressing time and data processing time. Table 2 lists the definitions of related symbols, where λ _i is a variable, T and N _a are calculation results, and others are known constants.

Table 2 Definition of related symbols

The average number N _a of blocks hit by a query can be obtained through the occurrence probability of each query condition, as shown in formula 1.

After considering some influencing factors related to MapReduce, the average time consumed by an OLAP operation can be obtained, as shown in formula 2.

The λ _i when T takes the minimum value can be calculated through Formula 2, and thus the block size can be obtained.

In this embodiment, it is assumed that the calculated values of the blocks in each dimension are 6, 4, and 5 respectively, that is, the side lengths of the blocks are 558, 32, and 10 respectively, that is, the step length in the time dimension is 558 for division, the area dimension is divided according to the step size of 32, and the depth dimension is divided according to the step size of 10. The final division result is shown in Figure 7. In Fig. 7, the smaller square grid is a unit cell, and the larger square grid is a data block (such as data block 5, data block 11, data block 18, data block 19, data block 20, data block 21, data block 22, data block 23, data block 42, data block 43 and data block 44). For the convenience of drawing, each data block in the figure only contains 9 cells, but in actual operation, the number of squares contained in the data block is 558*32*10, which is 178560. In this embodiment, 6*4*5=120 data blocks are included.

When there is a situation where the number of block values on the calculated dimension cannot be divisible, for example, assuming that the number of values of the blocks calculated in this embodiment on each dimension is 7, 4, and 5, then in Divisibility cannot be performed on the time dimension, that is, when 3348/7 is not an integer, the division step on this dimension is obtained by rounding up, that is, the division step on the time dimension is 479 at this time. At this time, for the data block, it needs to be supplemented by adding some null values.

Step 2.4: Encode the data block divided in step 2.3. The method is: divide the coordinates of any cell in the data block by the side length of the data block, and round up the obtained data as the code of the data block.

From a logical point of view, a data block can be regarded as a small cube, which contains a part of the cells in the data cube, and a data cube can also be regarded as composed of data blocks. In this implementation, suppose there is a piece of data that the seawater temperature in the 1/32° square area at a depth of 40m in the morning of June 1, 2012 is 15°C, and its corresponding coordinates are <2697, 1, 4>, if block The side lengths of are 558, 32, and 10 respectively, and the coordinates of the data block they belong to are <4, 0, 0>. Among them, 4 is obtained by dividing the code 2697 of the piece of data in the time dimension by the length of the data block in the time dimension 558, and rounding up; the first 0 is the code 1 of the piece of data in the region dimension Divided by the length of the data block in the region dimension of 32; the second 0 is obtained by dividing the code 4 of the data in the depth dimension by the length of the data block in the depth dimension of 10.

Step 2.5: The data block cut in step 2.3 is stored in the distributed file system, and the code established in step 2.4 is used as the name of the data block file.

In order to store the data block and the cells in the data block in the distributed file system, this embodiment first serializes its encoding. Logically, the data structure of "cube and its cells" or "cube and its blocks" can be compared to the data structure of "multidimensional array and its elements". Physically, a block is a cubic storage unit. After the cells in the block are linearized, the block can be stored as an independent file. In order to facilitate addressing, both blocks and cells need to support linearization and delinearization operations, and this operation is consistent with the linearization and delinearization operations of multidimensional arrays. Assume that there is an n-dimensional array, whose dimension scale is denoted as _{<A 1 , A 2 ,...An>} , and the coordinates of the element X in the multidimensional array in the multidimensional space are denoted as (X ₁ , X ₂ ,..., X _n ), the coordinates after linearization are marked as index(X), then the linearization method is shown in formula 3, and the de-linearization method is shown in formula 4.

index(X)＝(...((X _n ×A _n-1 +X _n-1 )×A _n-2 +...+X ₃ )×A ₂ +X ₂ )×A ₁ +X ₁ (3)

temp ₁ = index

...

X _n =temp _n %A _n

The method of linearization and delinearization of cell and data block encoding is the same as the method of linearization and delinearization of the above-mentioned multidimensional array. When linearizing and delinearizing a data block, the dimension scale refers to the number of values in each dimension of the data block. When linearizing and delinearizing a cell, the dimension scale refers to each dimension of the data cube The length of , that is, the number of dimension values in the lowest dimension level of each dimension that makes up the data cube.

X₃＝4％128＝4，则最终反线性化后的坐标为<2679，1，4>。Assuming that there is data on the morning of June 1, 2012, the temperature of the seawater at a depth of 40m in the 1/32° square area is 15°C, the corresponding coordinates are <2697, 1, 4>, and the side lengths of the data cube are 3348, 128, 50, of which 3348 is the lowest level of the time dimension, the number of dimension values contained in the sub-dimensional level, 128 is the lowest dimension level of the region dimension, the number of dimension values contained in the 1/64° dimension level, and 50 is the lowest dimension of the depth dimension Hierarchy, the number of dimension values contained in the 1Om dimension hierarchies. Directly using the formula (3) can calculate the linearized coordinate of the piece of data as (4*128+1)*3348+2697=1720221. Formula (4) is used for delinearization, temp ₁ =1720221 during initialization; X ₁ =1720221%3348=2679, ; X ₂ =513%128=1,

X ₃ =4%128=4, then the final delinearized coordinates are <2679, 1, 4>.

X₂＝0％128＝0，

X₃＝0％128＝0则最终反线性化后的该块的坐标为<4，0，0>。Assume that there is a data block whose coordinates are <4, 0, 0>, where the values of the block in each dimension are 6, 4, and 5 respectively. Then the coordinates of this block after linearization are ((0*4+0)*6)+4=4; formula (4) is used for delinearization, and temp ₁ =4 during initialization; X ₁ =4%3348=4 ,

X ₂ =0% 128 =0,

X ₃ =0% 128 =0, then the coordinates of the block after final delinearization are <4, 0, 0>.

If the coordinates of the currently loaded data block are <4, 0, 0>, the coordinate range of the cells in the block is from <2232, 0, 0> to <2769, 31, 9>. The starting range 2232 is the starting range encoded in the time dimension, which is calculated by 4*558. The end range 2769 in time is calculated from 2232+558-1. If the currently loaded data coordinates are <2769, 31, 9>, the corresponding data is the seawater temperature at a depth of 90m in the 32/64° square area on the morning of June 24, 2012. Assuming that the value of the data is 20°C, the newly generated record is <3961241, 20>, where 3961241=(((9*128)+31)*3348+557). Add a record <3961241, 20> in the data block file. At this time, the entire block file is imported, and its file name is the coordinate of the serialized data block, which is 4.

According to the above method, all the block files are loaded. Name the currently loaded data cube as the original cube, and the data loading work is now complete.

Step 3: Use online data analysis OLAP method to query massive data;

Step 3.1: The user sets query conditions, including:

Query target: It refers to determining which data cube to query, that is, the target cube.

For an initial query, the data cube formed by massive data is called the original data cube. After the initial query, the next level cube of the original data cube will be generated. After multiple queries, it will also be The lower-level cube that generates the lower-level cube, or another lower-level cube of the original cube, these newly generated data cubes can be selected as the basis for the next query.

Query scope: In the established query target, which part of the data is queried.

The client gives the range of data to be analyzed. Through the user-defined translation method, the actual meaning of the dimension value in the data range is translated into the coding information of the dimension value. If the user wants to query the temperature data in the original cube from 1m in the afternoon of April 20, 2012 in the 4/64° square area to the 90m in the 32/64° square area in the morning of June 25, 2012, the corresponding data code after translation The range is <2570, 4, 0> to <2769, 31, 9>.

The dimension information of the result: refers to the dimension information of the result data cube.

If we want to view the average daily sea temperature of all data within the selected data range, the dimension information of the result data is as follows: time dimension, year-month-day; area dimension is the same as the target cube; depth dimension is the same as the target cube .

Aggregation method: the operation of aggregating the data within the query range

For example, if the user wants to query the water depth of 1m in the 4/64° square area in the afternoon of April 20, 2012 to 90m in the 32/64° square area in the morning of June 25, 2012, the daily average in this range , then the aggregation method is to find the average. Commonly used aggregation methods are: summation, maximum value, minimum value, etc.

Step 3.2: Determine whether the query conditions set in step 3.1 meet the following constraints:

Constraint 1: The query target already exists, and the query range should be less than or equal to the data range of the query target;

The query target specified by the user must already exist in the system. For example, in the system initialization phase, the user loaded all the massive data and named it the original cube. If the user specifies other cubes during the initial query, constraint 1 is not met. If the data loaded by the user is data from 2010 to 2012, 2° square area, 500m layer depth and is queried in the original cube, but the query content involves the data of 2013, the query range is greater than the range of the query target , which does not meet constraint 1. In fact, for multidimensional data, any one dimension that is not within the data range of the query target does not meet constraint 1.

Constraint 2: The number of dimensions of the result data cube should be less than or equal to the number of dimensions of the query target;

If the query target given by the user is the original cube, the dimensions of the original cube are time dimension, region dimension, and layer depth dimension, but the result data cube set by the user involves other dimensions, such as the recording personnel dimension, which is not suitable at this time. Constraint 2. Constraint 2 is met when the user only specifies the time dimension but does not specify the area dimension and layer depth dimension. At this time, it means that the user does not change the granularity of the region dimension and layer depth dimension or that the result data cube is consistent with the query target on the region dimension and layer depth dimension.

Constraint 3: The lowest dimension level of any dimension of the result data cube should be higher than the lowest dimension level of the corresponding dimension of the query target;

In this implementation, the result data cube and the query target do not change in the region dimension and layer depth dimension, only in time, the dimension level of the result data cube is year-month-day, and the dimension level of the original data cube is year -month-day-times. It means that the resulting data cube is one level higher than the query target in the time dimension. This is in compliance with constraint 3. If we assume that the dimension information of the query cube and the result data cube are swapped, it means that the result data cube is one level lower than the query target in the time dimension, which does not meet constraint 3.

Constraint 4: Aggregation methods must be distributed or algebraic;

An aggregate function is distributed if it can be computed in a distributed manner as follows: Suppose the data is divided into n sets, and the computation of the function on each part yields an aggregate value. If applying a function to n aggregated values yields the same result as applying the function to all data, then the function can be used for distributed computation. Common distributed aggregation methods are: summation, maximum value, minimum value, counting, etc.

A function is algebraic if it can be computed by an algebraic function with several parameters, each of which can be found using a distributed aggregate function. Common algebraic aggregation methods are: averaging.

Using the mean as the aggregation method in this embodiment is consistent with constraint 4. Constraint 4 is not met if median is used as the aggregation method.

If constraint 1, constraint 2 and constraint 4 are satisfied at the same time, then step 3.3 is performed;

If constraints 1 to 4 are satisfied at the same time, execute step 3.4;

If any of the above conditions are not met, the query fails and ends;

Step 3.3: Convert the query target, find the upper-level cube of the current query target, and judge whether the upper-level cube satisfies constraint 3, if not, continue to query the upper-level cube of the upper-level cube, if it still cannot If it is satisfied, the query fails and the query process ends; if the upper-level cube that satisfies constraint 3 is found, replace the cube with the target cube;

If constraint 3 is not satisfied, the target cube can be converted to satisfy constraint 3. However, if the query cannot satisfy constraint 3 through the conversion method, the query fails.

If there is cubic one, each dimension level is: year-month-day-time, 1°-1/2°-1/4°-1/8°-1/16°-1/32°-1/64° , 1OOm-50m-1Om; cubic two, each dimension level is: year-month-day, 1°-1/2°-1/4°-1/8°-1/16°-1/32°-1 /64°, 1OOm-50m-1Om; cubic three, year-month-day-time, 1°-1/2°-1/4°-1/8°-1/16°-1/32°-1 /64°, 100m-50m-10m, and we guarantee that in terms of data range, cube 1 includes cube 2, and cube 2 includes cube 3. We assume that the cube 2 is obtained from the query of the cube 1. At this time, we plan to get the cube 3 through the query, and the query target is the cube 2. But this query does not meet constraint 3, then the conversion of the target cube is performed. Cube 2 is obtained by establishing a query on Cube 1, then we say that Cube 1 is the upper cube of Cube 2. Then we transform the target cube into cube 1. At this time, the query becomes the result data cube to be cube 3, and the target cube is cube 1. Check constraint 3 again, and if it is satisfied, the conversion is successful. If constraint 3 cannot be satisfied until the target data cube is the original cube after conversion, the query fails.

Step 3.4: Coarse screening of data: determine the range of the minimum data block required for query according to the query range;

The schematic diagram of the data coarse screening process is shown in Figure 8. In this implementation, we will query the original cube, and the data range is the temperature of 1m in the 4/64° square area in the afternoon of April 20, 2012 to the water depth of 90m in the 32/64° square area in the morning of June 25, 2012 data, the corresponding coding range is <2570, 4, 0> to <2769, 31, 9>. It is reflected in Figure 8, the coordinates of point B are <2570, 4, 0>, and the coordinates of point H are <2769, 31, 9>, then the cube ABCD-EFGH is the data range to be queried. In order to determine the range of the smallest data block, we convert the coordinates of point B and point H into the coordinates of the block where they are located, and the conversion method is the same as step 2.4. In this embodiment, the side lengths of the data blocks are 558, 32, 10. After conversion, the range of the smallest data block obtained is <4, 0, 0> to <4, 0, 0>. Therefore, in this embodiment, only the data of one block file needs to be scanned. Reflected in Figure 8, the corresponding A'B'C'D'-E'F'G'H' is the data contained in the block <4, 0, 0>. In this embodiment, a total of 6*4*5=120 data blocks are included. Then, through the method of data coarse screening, we only process the data that needs to be queried, thus removing the other 119 data blocks from participating in the calculation.

Step 3.5: Fine screening of data; scan the data block files screened in step 3.4, and filter all the cells in the data block according to the query range, if the cells are in the query range, perform step 3.6; otherwise, discard the cell;

In this embodiment, the method of querying multi-dimensional information mass data is realized by using the MapReduce programming model. MapReduce consists of four parts: InputFormatter, Mapper, Reducer, and OutputFormatter, which correspond to the four steps of refining data, changing dimension levels, and aggregating and outputting result sets. Its operation process is shown in Figure 9.

In this embodiment, the coordinates of the data blocks that pass the coarse data screening are <4, 0, 0>, and the corresponding data block name is 4, where 4 is linearized from the data block coordinates. InputFormatter reads data at the file level, so InputFormatter reads the data block file that has passed the rough screening, that is, data block file 4. InputFormatter reads all the records in data block 4 in turn, each record is in <key, value> format, where key is the linearized coordinate of the currently read cell, and value is the data in the current cell value. Afterwards, the InputFormatter delinearizes the key, and then obtains the coordinates of the data in the data cube, and compares it with the query range. If it meets the query range, the data is kept, otherwise, the data is discarded. For example, InputFormatter reads a piece of data as <3961241, 20>, and delinearizes the linearized coordinate 3961241 to obtain the coordinate <2570, 4, 0>, where the range of query data in this embodiment is <2570 , 4, 0> to <2769, 31, 9>, then the piece of data meets the query range, and enters the next step to change the cell dimension level. Otherwise, the data is discarded.

Step 3.6: Change the dimension level of the cell, compare the dimension information of the result data cube with the dimension information of the target cube, determine the changed dimension information, and modify the coordinates representing the dimension on the cell coordinates;

FIG. 9 shows the whole process of the massive data query method implemented based on MapReduce in this embodiment. If we abstract the records in the block file as <key, value>, the key is the coordinate of the cell after linearization, and the value refers to the value of the data in the cell. Read the required block file through InputFormatter, deserialize the key, and the obtained data format is <(a ₁ , a ₂ ,..., a _n ), value> where (a ₁ , a ₂ , .. ., a _n ) are the actual coordinates of the cell. After passing the check and query conditions, all qualified data will enter the Mapper to change the dimension level. In Mapper, the data changes from <(a ₁ , a ₂ ,..., a _n ), value> to <(b ₁ , b ₂ ,..., b _n ), value> where (b ₁ , b ₂ ,...,b _n ) are the coordinates of the cell after changing the dimension level. Finally, the coordinates will be linearized and passed to the Reducer for processing. The output of all Mappers is in the format of <index, value>. The value value will never change during processing. In the Reducer, the records of the same index will enter the same Reducer for processing, and the Reducer uses the specified aggregation method to aggregate all the values and output them to the OutputFormatter. Finally OutputFormatter is responsible for the generation of chunk files. Changing the dimension level of cell coordinates in Mapper is a very important operation during the entire calculation process.

In order to change the dimension coding of the cell, we introduce the concept of analysis path in this embodiment. Figure 10 shows a schematic diagram of changing the dimension hierarchy, where the dimension hierarchy is year-month-day. There are 4 dimension values at the year dimension level, which are 2010, 2011, 2012, and 2013, and there are 48 values at the monthly dimension level, which are January 2010, February 2010, ..., 2013 December; at the daily dimension level, we fix 31 days in each month, so there are 1488 values in total. At this point, we take the dimension value of February 2, 2010 as an example. If we currently know that the dimension code on February 2, 2010 is 2, and we need to change the dimension value to the month level at this time, that is to calculate the dimension code in February 2010. For the convenience of calculation, the concept of analysis path is introduced in this embodiment. At the same time, for the sake of brief description, we set the dimension value of February 2, 2010 as <2010 ¹ , 2 ² , 2 ³ >, where 2010 ¹ represents the first time in the time dimension In one dimension level, that is, in the year dimension level, the value of the dimension value is 2010, and the same is true for several other items. At the same time, the position of the dimension value among the sibling nodes (the position is counted from left to right, starting from 0) is called the analysis path of the dimension value. For example, for <2010 ¹ , 2 ² , 2 ³ >, its analysis path is <0, 1, 1>, and 2010 is a sibling node with all other dimension values at the year dimension level, and is at the beginning dimension value, so Its analysis path is 0. In the monthly dimension level, February 2010 and the dimension values of other months in 2010 are sibling nodes, and its parent node is 2010, so it is in the second dimension value, but due to the analysis The path is counted from 0, and its analysis path is 1. The calculation method of the analysis path on February 2, 2010 is the same as the calculation method of the analysis path on February 2010. It is currently known that <2010 ¹ , 2 ² , 2 ³ > is encoded as 32, and the encoding of <2010 ¹ , 2 ² > needs to be obtained. In this embodiment, the conversion relationship between the introduction of the analysis path and the encoding is shown in formula (5) and formula (6). Among them, the code is recorded as code(), for example, code(2010 ¹ ) is the code for 2010, that is, code(2010 ¹ )=0; the analysis path is recorded as order(), for example, order() is the analysis path for 2010, that is, order( 2010 ¹ ) = 0. |l _i | indicates the number of sibling nodes of any dimension value in the i-th dimension level. For example, in the year dimension level |l ₁ |=4, each year is a sibling node; in the month dimension level |l ₂ |=12, each month in the same year is a sibling node; in the day dimension level| l ₃ |=31, days in the same month are sibling nodes.

code(v ⁱ )＝(...((0+order(v ¹ ))×|l ₂ |+order(v ² ))×|l ₃ |+order(v ³ )...)×|l _i |+order(v ⁱ )

(5)

temp _i ＝code(v ⁱ )

...

order(v ¹ )=temp ₁ %|l ₁ |

，order(2010¹，2²)＝1％12＝1，order(2010¹)＝0％4＝0。从而求得对于<2010¹，2²，2³>而言分析路径为<0，1，1>，那么显然的由于节点的相对位置不会发生改变，所以对于<2010¹，2²>而言分析路径为<0，1>。通过公式(5)，计算得到code(2010¹，2²)＝(0+0)*48+1＝1。从而求得对于2010年2月而言，编码为2。至此完成了编码2010年2月2日向2010年2月的转化。Currently known code(2010 ¹ , 2 ² 2 ³ )=32. According to formula (6), then temp ₃ =32, order(2010 ¹ , 2 ² 2 ³ )=32%31=1,

, order(2010 ¹ , 2 ² )=1%12=1, order(2010 ¹ )=0%4=0. Therefore, the analysis path for <2010 ¹ , 2 ² , 2 ³ > is <0, 1, 1>, then obviously since the relative position of the nodes will not change, so for <2010 ¹ , 2 ² > and The language analysis path is <0, 1>. Through formula (5), code(2010 ¹ , 2 ² )=(0+0)*48+1=1 is calculated. Thus, it is obtained that for February 2010, the code is 2. So far, the conversion from February 2, 2010 to February 2010 has been completed.

In this embodiment, the query method for massive data is realized by using MapReduce, and the key to the realization is the updating of cell coordinates and the aggregation operation of metrics in cells. For ease of description, we use a 2-dimensional cube to illustrate the process. Figure 11 shows the change process of the cell coordinates and their values during the execution of the query method, where d ₁ and d ₂ are two dimensions, which are divided into Makes 4 servings. For d ₁ , its dimensional level is l ₁ ¹ -l ₂ ¹ -l ₃ ¹ , for d ₂ , its dimensional level is l ₁ ² -l ₂ ² -l ₃ ² -l ₄ ² and its corresponding The parameters are shown in Table 3. |l| refers to the number of sibling nodes of any node in dimension level l.

Table 3 shows the relevant parameters for OLAP algorithm execution

Assume that the coordinate of the processed unit is 126, the fact data is 67, and the coordinate of the corresponding block is 6. If the unit meets the query conditions, it will be handed over to Mapper for processing, and the input is <126, 67>. In Mapper, first 126 will be delinearized to get (10, 6). If in the query process, the corresponding new dimension level is l ₁ ² , l ₂ ² . The two-dimensional codes are updated respectively by Formula 5 and Formula 6, and the result is (1, 0), and the index obtained after linearization is 1 again. Finally, the output result of the Mapper is <1, 67>. When the codes of all input data are changed, this step ends.

Step 3.7: For cells with the same coordinates, aggregate the fact data values in the cells according to the set aggregation method;

Continue to take Figure 11 as an example. If other Mappers output the following data <1, 57>, <1, 43>, then the input data of the Reducer in Figure 11 is <1, {67, 57, 43}). Assuming that in this embodiment, summation is used as the aggregation method, the output of the Reducer is <1, 167>. The output of the above Reducer is stored in the form of records in the chunk file generated by the OutputFormatter.

Step 3.8: The data gathered in step 3.7 forms a result data cube, returns the information of the result data cube to the user, and stores the result cube as a new data cube so that it can be used as the query target of the next round of query .

After all MapReduce tasks are completed, return success information to the client. And store the result cube as a new data cube, so that it can be used as the query target of the next round of query.

Although the specific embodiments of the present invention have been described above, those skilled in the art should understand that these are only examples, and various changes or modifications can be made to these embodiments without departing from the principles and principles of the present invention. substance. The scope of the invention is limited only by the appended claims.

Claims (2)

1. A massive data query method with multidimensional information, characterized in that: comprise the following steps: Step 1: Load the dimensional information of massive data with multi-dimensional information, specifically including the following steps: Step 1.1: Identify the dimensional information of the massive data, and judge whether each dimensional information of the massive data satisfies the following three constraints at the same time: Constraint 1: A dimension consists of one and only one dimension level, that is, a dimension is a total order relationship composed of all dimension levels; Constraint 2: In any dimension level of a dimension, only one dimension attribute is included, and the dimension attribute contains several dimension values; Constraint 3: In the paired dimension value tree composed of all dimension values, sibling nodes contain the same number of child nodes; If satisfied, go to step 1.3, otherwise go to step 1.2; Step 1.2: Process the dimension information so that each dimension forms a dimension value tree structure conforming to the constraints. The processing process is as follows: Constraint 1: If there are multiple dimension levels, discard the dimension levels as needed, and only keep one dimension level; Constraint 2: If a dimension level includes multiple dimension attributes, discard the dimension attributes as needed, and only keep one dimension attribute; For constraint 3: if the number of child nodes contained in the sibling nodes is different, add a null value to make the number of child nodes of the sibling nodes the same; Step 1.3: Encoding dimension information; For the dimension values of each level in the dimension value tree, encode them sequentially in decimal numbers from left to right. When all the dimension values have corresponding encodings, the encoding work ends; Step 1.4: store the encoding of the dimension information; For any dimension information of massive data, store the name of each dimension level and the number of sibling nodes in this level, and finally form a file of all dimension information of massive data, which is stored in the distributed file system; Step 2: Load massive data; Step 2.1: The user establishes the corresponding relationship between the actual meaning of the dimension information of the massive data and the coding of the dimension information according to the needs, that is, to express any piece of data in the massive data with the coding of the dimension information; Step 2.2: All the most fine-grained multi-dimensional massive data form a data cube structure, and any piece of data in the massive data is used as a cell in the data cube, and the information of the cell includes: the coordinates of the cell within the cube, and The fact data value represented by the cell; where the coordinates of the cell are expressed as: <encoding of information in one dimension, encoding of information in another dimension, ..., encoding of information in the last dimension>; Step 2.3: Cut the data cube: According to the user's query requirements, under the condition of ensuring the shortest query time, the data cube is cut to form a data block, and the side length of the data block is determined; Step 2.4: Encoding the data block divided in step 2.3, the method is: divide the coordinates of any cell in the data block by the side length of the data block, and round up the obtained data to obtain the value as the encoding of the data block; Step 2.5: The data block cut in step 2.3 is stored in the distributed file system, and the code established in step 2.4 is used as the name of the data block file; Step 3: Use online data analysis OLAP method to query massive data; Step 3.1: The user sets query conditions, including: Query target: refers to determine which data cube to query, that is, the target cube; Query scope: in the established query target, which part of the data is queried; The dimension information of the result: refers to the dimension information of the result data cube; Aggregation method: the operation of aggregating the data within the query range; Step 3.2: Determine whether the query conditions set in step 3.1 meet the following constraints: Constraint 1: The query target already exists, and the query range should be less than or equal to the data range of the query target; Constraint 2: The number of dimensions of the result data cube should be less than or equal to the number of dimensions of the query target; Constraint 3: The lowest dimension level of any dimension of the result data cube should be higher than the lowest dimension level of the corresponding dimension of the query target; Constraint 4: Aggregation methods must be distributed or algebraic; If constraint 1, constraint 2 and constraint 4 are satisfied at the same time, then step 3.3 is performed; If constraints 1 to 4 are satisfied at the same time, execute step 3.4; If any of the above conditions are not met, the query fails and ends; Step 3.3: Convert the query target, find the upper-level cube of the current query target, and judge whether the upper-level cube satisfies constraint 3, if not, continue to query the upper-level cube of the upper-level cube, if it still cannot If it is satisfied, the query fails and the query process ends; if the upper-level cube that satisfies constraint 3 is found, replace the cube with the target cube; Step 3.4: Coarse screening of data: determine the range of the minimum data block required for query according to the query range; Step 3.5: Fine screening of data; scan the data block files filtered in step 3.4, and filter all the cells in the data block according to the query range, if the cells are in the query range, perform step 3.6; otherwise, discard the cell; Step 3.6: Change the dimension level of the cell, compare the dimension information of the result data cube with the dimension information of the target cube, determine the changed dimension information, and modify the coordinates representing the dimension on the cell coordinates; Step 3.7: For cells with the same coordinates, aggregate the fact data values in the cells according to the set aggregation method; Step 3.8: The data gathered in step 3.7 forms a result data cube, returns the information of the result data cube to the user, and stores the result cube as a new data cube so that it can be used as the query target of the next round of query . 2. The massive data query method with multi-dimensional information according to claim 1, characterized in that: the finest granularity in step 2.2 refers to the data pointed to by the lowest dimensional level.

Images