CN114595215A - Data processing method, device, electronic device and storage medium - Google Patents

Abstract

The invention provides a data processing method, a data processing device, electronic equipment and a storage medium. The data processing method comprises the following steps: firstly, grouping a plurality of data to be processed according to data dimensions to obtain a data set of each dimension; secondly, performing aggregation duplicate removal processing on the data sets of all dimensions through an aggregation function to obtain bitmap arrays corresponding to the data sets of all dimensions; then, acquiring a base number value of a bitmap array corresponding to each dimension data set; and finally, determining the data size of each dimension data set after the duplication removal according to the base number value of the bitmap array. In the steps of the method, the data to be processed is converted into the Bitmap array (Bitmap), and the fast duplicate removal processing operation can be realized on the data under different dimensions through the characteristic of small storage space of the Bitmap array, so that the efficiency of data statistical query is improved.

Description

Data processing method, device, electronic device and storage medium

technical field

The present invention relates to the technical field of data processing, and in particular, to a data processing method, device, electronic device and storage medium.

Background technique

With the rapid development of information technology, the storage capacity of information data continues to increase, and the demand for data management and analysis also spreads across various application scenarios. In the process of data management and analysis, it is usually necessary to perform deduplication statistics on data objects, such as obtaining data such as the number of UVs (independent visitors).

As the amount of data increases, the amount of calculation in query statistics will also increase, and the requirements for computer resources such as CPU, memory, and network IO will become higher and higher, and the processing speed will become slower and slower.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned problems and deficiencies in the prior art, the purpose of the present invention is to provide a data processing method, apparatus, electronic device and storage medium, which can realize the rapid deduplication processing of data.

To achieve the above object, the present invention first provides a data processing method, comprising:

Group a plurality of data to be processed according to the data dimension to obtain a data set of each dimension;

The data sets of each dimension are aggregated and deduplicated through the aggregation function to obtain the bitmap array corresponding to the data sets of each dimension;

Get the cardinality value of the bitmap array corresponding to each dimension dataset;

Determine the amount of data after deduplication of each dimension data set according to the cardinality value of the bitmap array.

Optionally, the step of acquiring the cardinality value of the bitmap array corresponding to each dimension dataset includes:

Obtain the compressed bitmap array according to the bitmap array, and the storage space occupied by the compressed bitmap array is smaller than the storage space occupied by the bitmap array;

Obtain the corresponding base value of the bitmap array according to the compressed bitmap array.

Optionally, the data set of each dimension includes multiple data groups; according to the compressed bitmap array corresponding to each dimension data set, the step of obtaining the corresponding bitmap array cardinality value includes:

Obtain the cardinality value of each data group according to the compressed bitmap array of each data group;

According to the cardinality value of each data set, determine the bitmap array cardinality value corresponding to each dimension data set.

Optionally, the data dimension includes at least a first grouping field and a second grouping field; the steps of grouping a plurality of data to be processed according to the data dimension and obtaining a data set of each dimension include:

Get the grouping field for each data to be processed;

According to the grouping field of the data to be processed, the data to be processed is divided into multiple data sets, and the multiple data sets include a first data set corresponding to the first grouping field and a second data set corresponding to the second grouping field.

Optionally, both the first grouping field and the second grouping field contain multiple data groups; the step of determining the deduplicated data volume of each dimension data set according to the cardinality value of the bitmap array includes:

According to the cardinality value of the bitmap array corresponding to the data in the first grouping field, determine the data volume of each data group in the first grouping field after deduplication;

According to the cardinality value of the bitmap array corresponding to the data in the second grouping field, the deduplicated data amount of each data group in the second grouping field is determined.

The present invention also provides a data processing device, comprising:

The data grouping module is used to group a plurality of data to be processed according to the data dimension, and obtain the data set of each dimension;

The aggregation deduplication module is used to aggregate and deduplicate the data sets of each dimension through the aggregation function, and obtain the bitmap array corresponding to the data set of each dimension;

The cardinality acquisition module is used to obtain the cardinality value of the bitmap array corresponding to each dimension dataset;

The data determination module is used to determine the deduplicated data volume of each dimension data set according to the cardinality value of the bitmap array.

Optionally, the cardinality acquisition module includes:

The compressed bitmap obtaining module is used to obtain the compressed bitmap array according to the bitmap array, and the storage space occupied by the compressed bitmap array is smaller than the storage space occupied by the bitmap array;

The bitmap cardinality obtaining module is used to obtain the corresponding bitmap array cardinality value according to the compressed bitmap array.

Optionally, the data set of each dimension includes multiple data groups, and the bitmap cardinality obtaining module includes:

The cardinality value acquisition module is used to obtain the cardinality value of each data group according to the compressed bitmap array of each data group;

The cardinality value determination module is used for determining the cardinality value of the bitmap array corresponding to each dimension data set according to the cardinality value of each data group.

The present invention also provides an electronic device, comprising a storage medium and a processor, wherein the storage medium stores a computer program, and the processor implements the steps of any one of the data processing methods described above when the computer program is executed.

The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of any one of the above data processing methods.

Compared with the prior art, the beneficial effects of the present invention include: firstly, a plurality of data to be processed are grouped according to the data dimension to obtain a data set of each dimension; secondly, the data set of each dimension is aggregated and deduplicated by an aggregation function. Process to obtain the bitmap array corresponding to each dimension dataset; then obtain the cardinality value of the bitmap array corresponding to each dimension dataset; finally, determine the deduplicated data volume of each dimension dataset according to the cardinality value of the bitmap array. In the above method steps of the present invention, the data to be processed is converted into a bitmap array (Bitmap), and by virtue of the small storage space of the bitmap array, the data in different dimensions can be quickly deduplicated and processed, thereby improving data statistics. query efficiency.

Description of drawings

In order to more clearly illustrate the embodiments or technical solutions in the prior art, the following briefly introduces the accompanying drawings that are used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only for invention. In some embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

FIG. 1 is a flowchart 1 of a data processing method according to an embodiment of the present invention;

2 is a second flowchart of a data processing method according to an embodiment of the present invention;

3 is a flowchart 3 of a data processing method according to an embodiment of the present invention;

4 is a frame diagram of a data processing apparatus according to an embodiment of the present invention;

5 is a frame diagram of a cardinality acquisition module according to an embodiment of the present invention;

6 is a frame diagram of a bitmap cardinality acquisition module according to an embodiment of the present invention;

FIG. 7 is a structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, in this embodiment of the present invention, the terms "first", "second", etc. are only used for the purpose of distinguishing and describing, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. In the embodiments of the present invention, "such as", "example" and "such as" are used to mean "serving as an example, illustration or illustration". Any embodiments of the present invention described as "such as," "examples," and "such as" are not necessarily to be construed as preferred or advantageous over other embodiments. The following description is presented to enable any person skilled in the art to make and use the present invention. In the following description, details are set forth for the purpose of explanation. It will be understood by one of ordinary skill in the art that the present invention may be practiced without the use of these specific details. In other instances, well-known structures and procedures have not been described in detail so as not to obscure the description of the present invention with unnecessary detail. Thus, the present invention is not intended to be limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and features disclosed herein.

An embodiment of the present invention provides a data processing method, as shown in FIG. 1 , including step 100, step 200, step 300 and step 400. The data processing method is specifically as follows:

Step 100: Group a plurality of data to be processed according to the data dimension to obtain a data set of each dimension. The data dimension includes at least a first grouping field and a second grouping field, and both the first grouping field and the second grouping field include multiple data groups.

In an embodiment, step 100 may specifically include the following steps:

First, get the packet word for each data to be processed.

Then, according to the grouping field of the data to be processed, the data to be processed is divided into multiple data sets, and the multiple data sets include a first data set corresponding to the first grouping field and a second data set corresponding to the second grouping field. For example, when querying user statistics, the first grouping field is country, the second grouping field is age, the first data set includes each country name or code, the ID or name of each user, and the second data set includes each age segment , the ID or name of each user.

The dimension levels of the first grouping field and the second grouping field may be the same or different.

For example, the dimension level of the first grouping field is higher than the dimension level of the second grouping field. The first grouping field is country, which includes multiple data groups such as China, the United States, the United Kingdom, and Japan, and the second grouping field is province, city and county. Including Beijing, Wuhan, Hunan Province, Hokkaido, Yamagata, Fukushima, Arizona and other provinces, cities and county-level data groups.

For example, the dimension level of the first grouping field is the same as the dimension level of the second grouping field. The first grouping field is age, which includes 18-year-old, 25-year-old, 30-year-old, 40-year-old and other age-segmented data groups; the second grouping field The field is education, which includes data groups such as elementary school, junior high school, high school, and university.

In one embodiment, the data dimension may further include a third grouping field, a fourth grouping field, or more grouping fields, which are not exhaustive here.

Step 200: Perform aggregation and deduplication processing on the data sets of each dimension through an aggregation function to obtain a bitmap array corresponding to the data sets of each dimension. For example, the first dimension data set is the data set corresponding to the first grouping field, and the second dimension data set is the data set corresponding to the second grouping field.

Among them, the aggregation functions include the rbmcreate32 function and the rbmcardinality32 function, which are functions extended and implemented in the Impala execution engine for aggregation and deduplication. Impala is an MPP (massively parallel processing) SQL (Structured Query Language) query engine for processing large amounts of data stored in Hadoop (distributed systems infrastructure) clusters, it is a C++ and Java written of open source software. SQL is a database query and programming language used to access data and query, update, and manage relational database systems.

The basic idea of a bitmap array is to use a bit to mark the Value corresponding to an element, and the Key is the element. Since the unit of Bit is used to store data, the storage space can be greatly saved in terms of storage space.

For example, a data set is [2, 3, 5, 8], and the corresponding Bitmap array is [001101001]. In the data set, 2 corresponds to the position where the index (index) is 2 in the Bitmap array, 3 corresponds to the position where the index (index) is 3 in the Bitmap array, 5 corresponds to the position where the index (index) is 5 in the Bitmap array, and 8 corresponds to The position of index (index) in the Bitmap array is 8. In the Bitmap array, there are only 0 and 1, 0 means that there is no data at the index position, and 1 means that there is data at the index position. In the Bitmap array [001101001], counting from left to right, 0 in the first position means data 0 does not exist, 0 in the second position means data 1 does not exist, 1 in the third position means data 2 exists, A 1 in the four positions means that data 3 exists, and so on for the rest.

Step 300: Obtain the cardinality value of the bitmap array corresponding to each dimension data set. Among them, the cardinality value of the bitmap array is the number of "1" in the bitmap array. For example, in the bitmap array [001101001], the number of 1s is 4, so the base value of the bitmap array is 4.

Because the same data has the same index in the Bitmap array, the corresponding index position in the Bitmap array is only represented as 1. Therefore, in the process of constructing the Bitmap array, the original data collection is also deduplicated. For example, the data set [2, 3, 5, 5, 8, 2] converted into a bitmap array is also [001101001].

In an embodiment, as shown in FIG. 2 , step 300 specifically includes step 310 and step 320, wherein:

Step 310: Obtain a compressed bitmap array according to the bitmap array, where the storage space occupied by the compressed bitmap array is smaller than the storage space occupied by the bitmap array.

Specifically, a compressed bitmap (Roaring Bitmap) occupies less storage space than a general bitmap. In RoaringBitmap, 32-bit integers are divided into 2 to 16 blocks. The first 16 bits of any 32-bit integer determine which block to put in, and the last 16 bits are the contents of this block. For example, 0xFFFF0000 and 0xFFFF0001, the first 16 bits are FFFF, indicating that these two numbers should be placed in one block. The last 16 bits are 0 and 1, respectively. It's fine to store only 0s and 1s in this block, no need to store full integers. In this way, the Bitmap is efficiently compressed, so that the space occupied by the bitmap array is much smaller.

Step 320: Obtain the corresponding base value of the bitmap array according to the compressed bitmap array. As shown in Figure 3, it specifically includes steps 321 and 322, wherein the data set of each dimension includes multiple data groups, and the steps are as follows:

Step 321: Obtain the cardinality value of each data group according to the compressed bitmap array of each data group.

Step 322: Determine the cardinality value of the bitmap array corresponding to each dimension data set according to the cardinality value of each data set.

For example, in the task of counting the number of uvs (independent visitors) by nationality and province, in the second dimension data set, the data groups include Hokkaido, Yamagata, Iwate, Fukushima, Akita, and Aomori, and the corresponding base values are respectively 18, 31, 23, 54, 42, 37. The corresponding base values for the data sets Arizona, Alabama, and Illinois are 8, 9, and 7, respectively.

Step 400: Determine the deduplicated data volume of each dimension data set according to the cardinality value of the bitmap array. Specifically include the following steps:

First, according to the cardinality value of the bitmap array corresponding to the data in the first grouping field, the data amount of each data group in the first grouping field after deduplication is determined. For example, in the task of counting UVs by nationality and province, the first grouping field includes Japan and the United States. The base value corresponding to Japan is 205, and the base value of the United States is 24. Then the number of UVs in Japan is 205, and the number of UVs in the United States is 205. is 24.

Then, according to the cardinality value of the bitmap array corresponding to the data in the second grouping field, the deduplicated data amount of each data group in the second grouping field is determined. For example, in the second grouping field, the data groups corresponding to Japan are Hokkaido, Yamagata, Iwate, Fukushima, Akita, and Aomori, and the corresponding base values are 18, 31, 23, 54, 42, and 37 respectively. , the uvs of Hokkaido, Yamagata, Iwate, Fukushima, Akita, and Aomori are 18, 31, 23, 54, 42, and 37, respectively. In the US data set, the base values corresponding to Arizona, Alabama, and Illinois are 8, 9, and 7, respectively. 8, 9, 7.

It should be noted that, in some cases, a single data may exist in multiple different grouping fields. For example, a user has dual nationality, then when counting the number of UVs by nationality, it is necessary to count the entire data in order to de-emphasize Data coincidence under the first grouping field.

The embodiment of the present invention introduces the calculation method of Bitmap into the Impala engine. In order to accurately count the number of user_ids in each grouping field, it is necessary to save the user_id details of each grouping field, and the minimum unit of information retained in the computer is bit.

This embodiment extends and implements the rbmcreate32 function and the rbmcardinality32 function in the Impala execution engine. The rbmcreate32 function can construct a corresponding bitmap for any attribute, and at the same time, the bitmap can be passed upward for aggregate calculation of deduplication values.

As shown in the following table, under the two dimensions of pege and title, the respective uv data are counted.

page title user_id waimai gaifan 101 waimai kuaican 102 xiaoxiang qiezi 102 xiaoxiang kuaican 101

The first dimension is title, the second dimension is pege, and the respective user uv indicators under the two dimensions of title and page are calculated at the same time. The rbmcreate32 function and the rbmcardinality32 function can be calculated by the following SQL:

selectpage,title,rbmcardinality32(uv_bitmap)over(partition by page),rbmcardinality32(uv_bitmap)over(partition by title)

from(select page,title,rbmcreate32(distinct user_id)as uv_bitmap fromtable group by page,title).

In the embodiment of the present invention, a plurality of data to be processed are firstly grouped according to the data dimensions to obtain a data set of each dimension; secondly, an aggregation function is used to perform aggregation and deduplication processing on the data set of each dimension to obtain the bit corresponding to the data set of each dimension. image array; then obtain the cardinality value of the bitmap array corresponding to each dimension dataset; finally, determine the deduplicated data volume of each dimension dataset according to the cardinality value of the bitmap array. In the above method steps of the present invention, the data to be processed is converted into a bitmap array (Bitmap), and by virtue of the small storage space of the bitmap array, data processing such as fast deduplication, sorting, and query can be realized for data in different dimensions operation, which improves the efficiency of data query and statistics.

An embodiment of the present invention provides a data processing apparatus, as shown in FIG. 4 , including a data grouping module 500 , an aggregation deduplication module 600 , a cardinality obtaining module 700 and a data determining module 800 .

The data grouping module 500 is configured to group a plurality of data to be processed according to the data dimension to obtain a data set of each dimension.

The aggregation deduplication module 600 is configured to perform aggregation and deduplication processing on the data sets of each dimension through an aggregation function to obtain a bitmap array corresponding to the data sets of each dimension.

The cardinality obtaining module 700 is configured to obtain the cardinality value of the bitmap array corresponding to each dimension data set.

Wherein, as shown in FIG. 5 , the cardinality obtaining module 700 specifically includes a compressed bitmap obtaining module 710 and a bitmap cardinality obtaining module 720 . The compressed bitmap obtaining module 710 is configured to obtain a compressed bitmap array according to the bitmap array, and the storage space occupied by the compressed bitmap array is smaller than the storage space occupied by the bitmap array. The bitmap cardinality obtaining module 720 is configured to obtain the corresponding bitmap array cardinality value according to the compressed bitmap array.

Further, the data set of each dimension includes multiple data groups. As shown in FIG. 6 , the bitmap cardinality obtaining module 720 includes a cardinality value obtaining module 721 and a cardinality value determining module 722 . The cardinality value obtaining module 721 is configured to obtain the cardinality value of each data group according to the compressed bitmap array of each data group. The cardinality value determining module 722 is configured to determine the cardinality value of the bitmap array corresponding to each dimension data set according to the cardinality value of each data set.

The data determination module 800 is configured to determine the deduplicated data volume of each dimension data set according to the cardinality value of the bitmap array.

The present invention is the data processing device of the embodiment. First, a plurality of data to be processed are grouped according to the data dimensions to obtain data sets of each dimension; The bitmap array corresponding to the dimension dataset; then, the cardinality value of the bitmap array corresponding to each dimension dataset is obtained; finally, the deduplicated data volume of each dimension dataset is determined according to the cardinality value of the bitmap array. In the above method steps of the present invention, the data to be processed is converted into a bitmap array (Bitmap), and by virtue of the small storage space of the bitmap array, data processing such as fast deduplication, sorting, and query can be realized for data in different dimensions operation, which improves the efficiency of data query and statistics.

An embodiment of the present invention provides an electronic device, as shown in FIG. 7 , including a storage medium and a processor, the storage medium stores a computer program, and the processor implements any one of the data processing methods provided in the above embodiments when the computer program is executed. step.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions (computer programs), or by controlling relevant hardware through instructions (computer programs), and the instructions can be stored in computer-readable storage medium, and loaded and executed by a processor. To this end, the storage medium of the electronic device in the embodiment of the present invention stores a plurality of instructions, and the instructions can be loaded by the processor to execute the steps of any embodiment of the device control method provided in the embodiment of the present invention.

In the electronic device of this embodiment, firstly, a plurality of data to be processed are grouped according to the data dimension to obtain a data set of each dimension; secondly, the data set of each dimension is aggregated and deduplicated by an aggregation function to obtain a data set of each dimension The corresponding bitmap array; then, the cardinality value of the bitmap array corresponding to each dimension dataset is obtained; finally, the deduplication data amount of each dimension dataset is determined according to the cardinality value of the bitmap array. In the above method steps of the present invention, the data to be processed is converted into a bitmap array (Bitmap), and by virtue of the small storage space of the bitmap array, data processing such as fast deduplication, sorting, and query can be realized for data in different dimensions operation, which improves the efficiency of data query and statistics.

Embodiments of the present invention further provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of any of the data processing methods provided in the foregoing embodiments.

As shown in the figure, the storage medium and the processor are directly or indirectly electrically connected to realize data transmission or interaction. For example, these elements may be electrically connected to each other by one or more communication buses or signal lines, such as may be connected by a bus. The storage medium stores computer-executed instructions for implementing the data access control method, including at least one software function module that can be stored in the storage medium in the form of software or firmware, and the processor runs the software programs and modules stored in the storage medium. Execute various functional applications and data processing. The storage medium may be, but not limited to, a random access storage medium (Random Access Memory, referred to as: RAM), a read-only storage medium (ReadOnly Memory, referred to as: ROM), a programmable read-only storage medium (Programmable Read-Only Memory, referred to as: PROM), An erasable read-only storage medium (ErasableProgrammableRead-OnlyMemory, referred to as: EPROM), an electrically erasable read-only storage medium (ElectricErasableProgrammableRead-OnlyMemory, referred to as: EEPROM), and the like. The storage medium is used to store the program, and the processor executes the program after receiving the execution instruction. Further, the software programs and modules in the above-mentioned storage medium may also include an operating system, which may include various software components and/or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and Can communicate with various hardware or software components to provide an operating environment for other software components. The processor may be an integrated circuit chip with signal processing capability. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (NetworkProcessor, NP for short). Each method, step and logic flow block diagram disclosed in this embodiment can be realized or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

Since the instructions stored in the storage medium can execute the steps in any of the data processing method embodiments provided by the embodiments of the present invention, it is possible to implement any of the data processing methods provided by the embodiments of the present invention. For the beneficial effects, refer to the foregoing embodiments for details, which will not be repeated here.

The above description is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited to this. Substitutions should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (10)

1. a data processing method, is characterized in that, comprises: Group a plurality of data to be processed according to the data dimension to obtain a data set of each dimension; Perform aggregation and deduplication processing on the data sets of each dimension through an aggregation function to obtain a bitmap array corresponding to the data sets of each dimension; obtaining the cardinality value of the bitmap array corresponding to each dimension dataset; The deduplicated data amount of each dimension data set is determined according to the cardinality value of the bitmap array. 2. The data processing method according to claim 1, wherein the step of acquiring the cardinality value of the bitmap array corresponding to each dimension data set comprises: Obtain a compressed bitmap array according to the bitmap array, where the storage space occupied by the compressed bitmap array is smaller than the storage space occupied by the bitmap array; According to the compressed bitmap array corresponding to each dimension data set, the corresponding base value of the bitmap array is obtained. 3. The data processing method according to claim 2, wherein the data set of each dimension comprises multiple data groups; and the corresponding bitmap is obtained according to the compressed bitmap array corresponding to the data set of each dimension Steps for array radix values, including: Obtain the cardinality value of each data group according to the compressed bitmap array of each data group; According to the cardinality value of each data set, the bitmap array cardinality value corresponding to each dimension data set is determined. 4 . The data processing method according to claim 1 , wherein the data dimension includes at least a first grouping field and a second grouping field; the data to be processed is grouped according to the data dimension to obtain each Steps for dimensional datasets, including: Get the grouping field for each data to be processed; According to the grouping field of the data to be processed, the data to be processed is divided into multiple data sets, and the multiple data sets include a first data set corresponding to the first grouping field and a second data corresponding to the second grouping field set. 5. The data processing method according to claim 4, wherein the first grouping field and the second grouping field both contain multiple data groups; Describe the steps of the data volume after deduplication of each dimension dataset, including: According to the cardinality value of the bitmap array corresponding to the data in the first grouping field, determine the data volume after deduplication of each data group in the first grouping field; According to the cardinality value of the bitmap array corresponding to the data in the second grouping field, the deduplicated data amount of each data group in the second grouping field is determined. 6. A data processing device, comprising: The data grouping module is used to group a plurality of data to be processed according to the data dimension to obtain the data set of each dimension; an aggregation deduplication module, configured to perform aggregation deduplication processing on the data sets of each dimension through an aggregation function, to obtain bitmap arrays corresponding to the data sets of each dimension; a cardinality obtaining module, used for obtaining the cardinality value of the bitmap array corresponding to each dimension data set; A data determination module, configured to determine the deduplicated data amount of each dimension data set according to the cardinality value of the bitmap array. 7. The data processing device according to claim 6, wherein the cardinality acquisition module comprises: a compressed bitmap obtaining module, configured to obtain a compressed bitmap array according to the bitmap array, where the storage space occupied by the compressed bitmap array is smaller than the storage space occupied by the bitmap array; The bitmap cardinality obtaining module is configured to obtain the corresponding bitmap array cardinality value according to the compressed bitmap array. 8. The data processing device according to claim 7, wherein the data set of each dimension comprises a plurality of data groups, and the bitmap cardinality obtaining module comprises: a cardinality value obtaining module, used for obtaining the cardinality value of each data group according to the compressed bitmap array of each data group; The cardinality value determination module is configured to determine the cardinality value of the bitmap array corresponding to each dimension data set according to the cardinality value of each data set. 9. An electronic device, comprising a storage medium and a processor, wherein the storage medium stores a computer program, wherein the processor implements the computer program described in any one of claims 1 to 5 when the processor executes the computer program. The steps of a data processing method. 10. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 5 are implemented.

Images