CN113157695A

CN113157695A - Data processing method and device, readable medium and electronic equipment

Info

Publication number: CN113157695A
Application number: CN202110336511.0A
Authority: CN
Inventors: 王石冲; 王航宇; 罗梦瑶; 汪鹏; 丁春雷; 宋骞; 于佳萍
Original assignee: Beijing ByteDance Network Technology Co Ltd
Current assignee: Beijing ByteDance Network Technology Co Ltd
Priority date: 2021-03-29
Filing date: 2021-03-29
Publication date: 2021-07-23
Anticipated expiration: 2041-03-29
Also published as: CN113157695B

Abstract

The present disclosure relates to a data processing method, apparatus, readable medium and electronic device, including: acquiring import data, wherein the import data comprises a user ID and label data corresponding to the user ID; allocating a target data node for the user ID; and storing the imported data in a first bitmap respectively corresponding to each target data node through each label in the label data, wherein the first bitmap comprises a mapping table, the first 32 bits of data in 64-bit integer data are used as a key word of the mapping table, the second 32 bits of data are used as values of the key word, and the values of the key word are stored in a second bitmap supporting 32-bit integer data. Therefore, a data processing system for carrying out insight analysis or crowd selection on users can be established, compared with a bitmap which only supports 32-bit integer, the first bitmap which supports 64-bit integer data enlarges the scale of users which can be supported by the system, can realize data processing of billions or even billions of users, and improves the processing capacity of the system.

Description

Data processing method and device, readable medium and electronic equipment

Technical Field

The present disclosure relates to the field of data processing, and in particular, to a data processing method and apparatus, a readable medium, and an electronic device.

Background

Crowd insight is a very important function. The method can help the user to know the designated crowd more deeply and carefully. While crowd insight analysis usually requires querying a large amount of tagged user data, there are many existing implementations, such as Spark offline reading, an ElasticSearch-based approach, and a Bitmap-based approach. The Spark offline reading mode has a certain speed problem, even if the data volume is not large, quick query response cannot be achieved, the mode based on the elastic search cannot support the requirement of user data with a large number of labels, the mode based on the Bitmap is faster than the former two response speeds, the query effect on the user data with a large number of labels is better, but only supports 32-bit integer ID types, and under the condition that an integer ID is used for representing users, the number of supported tagged users is limited to a certain extent, and the user data with the large number of users cannot be processed.

Disclosure of Invention

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

In a first aspect, the present disclosure provides a data processing method, the method comprising:

acquiring import data, wherein the import data comprises a user ID and label data corresponding to the user ID;

allocating a target data node to the user ID;

storing the imported data by using first bitmaps respectively corresponding to the tags in the tag data in the target data nodes, wherein the first bitmaps can store 64-bit long integer data;

the first bitmap comprises a mapping table, the first 32 bits of data in the 64-bit integer data are used as a key word of the mapping table, the second 32 bits of data are used as a value of the key word, and the value of the key word is stored in a second bitmap supporting 32-bit integer data.

In a second aspect, the present disclosure provides a data processing apparatus, the apparatus comprising:

the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring import data, and the import data comprises a user ID and label data corresponding to the user ID;

the fragmentation module is used for distributing target data nodes for the user ID;

the processing module is used for storing the imported data through corresponding first bitmaps in the target data nodes through each label in the label data, and 64-bit long-length integer data can be stored in the first bitmaps;

In a third aspect, the present disclosure provides a computer readable medium having stored thereon a computer program which, when executed by a processing apparatus, performs the steps of the method of the first aspect.

In a fourth aspect, the present disclosure provides an electronic device comprising:

a storage device having a computer program stored thereon;

processing means for executing said computer program in said storage means to carry out the steps of the method of the first aspect.

Through the technical scheme, the data processing system for performing hole-finding analysis or crowd selection on the user can be established according to the label data of the user, compared with a bitmap which only supports 32-bit integer, the first bitmap which supports 64-bit integer data enlarges the scale of the user which can be supported by the system, can realize the data processing of billions or even billions of users, greatly simplifies the process of processing the label data under the condition of overlarge scale of the user, and improves the processing capacity of the data processing system.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale. In the drawings:

fig. 1 is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure.

Fig. 2 is a flowchart illustrating a data processing method according to still another exemplary embodiment of the present disclosure.

Fig. 3 is a flowchart illustrating a data processing method according to yet another exemplary embodiment of the present disclosure.

Fig. 4 is a flowchart illustrating a data processing method according to yet another exemplary embodiment of the present disclosure.

Fig. 5 is a flowchart illustrating a data processing method according to yet another exemplary embodiment of the present disclosure.

Fig. 6 is a block diagram illustrating a structure of a data processing apparatus according to an exemplary embodiment of the present disclosure.

Fig. 7 is a block diagram illustrating a structure of a data processing apparatus according to still another exemplary embodiment of the present disclosure.

Fig. 8 is a block diagram illustrating a structure of a data processing apparatus according to still another exemplary embodiment of the present disclosure.

FIG. 9 shows a schematic structural diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and examples of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.

It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.

It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.

The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.

Fig. 1 is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure. As shown in fig. 1, the method includes steps 101 to 103.

In step 101, import data is obtained, where the import data includes a user ID and tag data corresponding to the user ID. The tag data may be any attribute identifier of the user corresponding to the user ID, and the obtaining method may be any one or more of the attribute identifiers given by other evaluation systems, actively filled by the user, obtained by friend evaluation of the user, and the like. The manner of acquiring the tag data is not limited in the present application. The tag data may include a plurality of tags, each of which may include, for example, a tag ID, a tag value, etc.

In step 102, a target data node is assigned to the user ID. The data node is a work node for storing and reading the user ID and the tag data corresponding to the user ID, and the number of data nodes may be plural when the user ID is large. When the target data node is allocated, the target data node may be selected according to the amount of data stored in the plurality of data nodes, for example, when the tag data corresponding to the user ID is more, the plurality of target data nodes are allocated to the user ID, so as to ensure data balance on each data node; or, the target data node may be determined according to a tag included in tag data corresponding to the user ID, so that the same tag data is stored in the same data node; in addition, all the tag data corresponding to the same user ID can be stored in the same target data node, so that when a tag of a certain user ID is queried, only the data in the same data node is acquired and calculated (local Join), and the situation that a Shuffle operation is triggered is avoided.

In step 103, storing the imported data in a first bitmap corresponding to each tag in the tag data in the target data node, wherein the first bitmap can store 64-bit long integer data; the first bitmap comprises a mapping table, the first 32 bits of data in the 64-bit integer data are used as a key word of the mapping table, the second 32 bits of data are used as a value of the key word, and the value of the key word is stored in a second bitmap supporting the 32-bit integer data.

After the target data node is determined, the imported data can be stored according to the tag data corresponding to the user ID. In the target data node, each label corresponds to an independent first bitmap (bitmap), and the first bitmap contains all user IDs corresponding to the label in the target data node, that is, a label-user ID data modeling mode is adopted.

The first bitmap can store 64-bit integer data, that is, the user ID supporting the highest 64-bit integer data, so as to support billion or even billion-level user scale, and is implemented by means of the mapping table. When the two 64-bit first bitmaps are subjected to set calculation (for example, merging and complementing), the key keys of the mapping tables on both sides may be traversed first, and then the merging and complementing result between the corresponding values is filled as a new value, so that the result of the same 64-bit first bitmap obtained after the two 64-bit first bitmaps are subjected to merging and complementing set calculation may be obtained.

Fig. 2 is a flowchart illustrating a data processing method according to still another exemplary embodiment of the present disclosure. As shown in fig. 2, the method further comprises step 201.

In step 201, a first hash calculation is performed on the user ID to obtain a first ID, a remainder obtained by dividing the first ID by a total number of data nodes is determined, and the target data node allocated to the user ID is determined according to the remainder.

That is, in the process of step 102, an allocation method is adopted in which all the tag data corresponding to the same user ID are stored in the same target data node, specifically, a hash calculation method is used to implement the allocation method.

In a possible case, each data node may correspond to a sequence number, for example, in a case that the total number of the data nodes is N, the sequence number of each data node may be M, where M ∈ (0, N-1), when the target data node is determined according to the remainder, the remainder may be directly determined as the sequence number of the target data node.

Because the same user ID uses the output value obtained by the same hash algorithm, namely the first ID is fixed and random, the remainder obtained by dividing the first ID by the total number of the data nodes can be uniformly distributed in the range of (0, N-1), thereby not only ensuring that the label data of the same user ID is only distributed to the same data nodes, but also ensuring that all the user IDs are relatively uniformly distributed in each data node, and ensuring the balance of the number of users on each data node.

Fig. 3 is a flowchart illustrating a data processing method according to yet another exemplary embodiment of the present disclosure. As shown in fig. 3, the method further comprises step 301.

In step 301, a second hash calculation is performed on the user ID to obtain a second ID, and a target partition in the target data node to which the user ID is to be allocated is determined according to the second ID.

In step 302, the import data is stored in the target partition according to the corresponding first bitmap in each tag in the tag data.

That is, after the unique target data node to which the user ID is to be allocated is determined, the partition (Shard) in which the user ID is specifically located in the target data node may be determined according to the user ID, so that not only can the tag data of one user ID be guaranteed to be stored in the same data node, but also the tag data of one user ID can be guaranteed to be stored in the same partition. In this way, the data stored in the first bitmap between each partition in each data node does not have an intersection, and each partition can be simultaneously processed in parallel in the query process, so that the parallel capability of the computing engine is fully utilized, and the processing efficiency of the data processing system is further improved.

Fig. 4 is a flowchart illustrating a data processing method according to yet another exemplary embodiment of the present disclosure. As shown in fig. 4, the method further includes step 401 and step 402.

In step 401, determining a target encoding range according to the number of the user IDs stored in the target data node, and re-encoding the user IDs into a second ID, where the second ID is within the target encoding range, and storing a one-to-one mapping relationship between the user IDs and the second ID, where the second ID is 64-bit long integer data, and the target encoding range is 2 of the first 32-bit data, which is the same as the first 32-bit data³²A number of adjacent codes.

In step 402, the second ID is imported into the first bitmap corresponding to each tag. And under the condition that the target partition is determined, the first bitmap corresponding to each label is the first bitmap in the target partition.

The number of user IDs stored in the target data node is less than 2³²In each case, the target coding range can be determined to be 0-2³²In the process ofThe number of the stored user IDs of the target data node is more than 2³²And less than 2.2³²In the case of (2), the target encoding range may be determined to be 2³²～2·2³²And so on.

In order to ensure that the second IDs encoded by the different user IDs stored in the different data nodes do not overlap when there are a plurality of data nodes, the target encoding range that can be allocated to each data node may be divided on the basis of the second IDs. For example, the data node 0 is divided into 0 ~ 2 &2 in advance³²The coding range of (1) is divided into 2 and 2 for the data node³²～4·2³²And in the same way, each data node determines the current target coding range according to the number of the stored user IDs, and the user IDs are recoded. Alternatively, the coding range may be divided in real time according to the number of user IDs actually stored in each data node, for example, each data node may be first allocated 2 in the sequence of the sequence number of the data node³²Adjacent codes are obtained, and the target coding range of the Mth data node is M.2³²～(M+1)·2³²Then, the number of the user IDs stored in each data node is monitored, and if the number of the first stored user IDs reaches 2³²If the data node is the data node X, the next target coding range of the data node X can be N.2³²～(N+1)·2³²And N is the total number of data nodes.

After re-encoding the user ID into the second ID in the target encoding range, a one-to-one mapping relationship therebetween may be stored as index data in the data processing system to be introduced into a memory to assist decoding when decoding is required.

Since the second ID in each target encoding range is 2 which is the same as the first 32 bits of data³²And the adjacent codes ensure that the second ID after being recoded can be controlled in a fixed interval by recoding even if the distribution of the user IDs is scattered, so that the storage space occupied by the first bitmap corresponding to each label on the target data node is largerSmall; in addition, the first 32 bits of data of all the second IDs in each data node can be concentrated on a part of the key, and the number of times of traversal operations that need to be performed on the key when performing calculations such as intersection and complementation on the data in the data node is greatly reduced, and the intersection and complementation operations that need to be performed on the first bitmap are also greatly reduced.

In another possible embodiment, the determining the target encoding range according to the number of user IDs stored in the target data node may be further performed by the following method.

Determining the target coding range as (a.N + M). 2³²～(a·N+M+1)·2³²Wherein, in the step (A),

s is the number of user IDs already stored in the target data node,

and N is the total number of the data nodes, and M is the serial number of the target data node.

That is, each data node is respectively allocated with 2 in advance according to the sequence number sequence of the data node³²Adjacent codes, when the number of user IDs stored in any data node reaches 2³²Then, the next target coding range allocated to the data node is still determined according to the sequence number sequence of the data node.

In a possible implementation manner, before performing step 103, and storing the imported data by using the first bitmaps respectively corresponding to the tags in the tag data and in the target data node, the method may further include: judging whether the label in the label data has the corresponding first bitmap in the target data node or not; if the label in the label data does not have the corresponding first bitmap in the target data node, creating the first bitmap corresponding to the label in the label data in the target data node. If the label in the label data has a corresponding first bitmap in the target data node, the imported data can be directly stored.

If the target partition is determined, it is determined whether a corresponding first bitmap exists in the target partition of the target data node for a tag in the tag data, and if not, a first bitmap corresponding to the tag in the tag data is created so as to store the tag data.

Fig. 5 is a flowchart illustrating a data processing method according to yet another exemplary embodiment of the present disclosure. As shown in fig. 5, the method further includes step 501 and step 502.

In step 501, a query instruction is received, where the query instruction includes a plurality of sub-query conditions and a logic operation instruction between the plurality of sub-query conditions.

In step 502, a plurality of groups of user data are respectively obtained according to the plurality of sub-query conditions, and the plurality of groups of user data are calculated according to the logic operation instruction to obtain target data.

The query instruction may be a query instruction obtained by indirect judgment according to a query requirement input by a user, or a query instruction directly input by the user and including a plurality of sub-query conditions and a logic operation instruction between the plurality of sub-query conditions. For example, the query instruction may be entered by the following custom function bitmapCount:

select bitmapCount(expression)(idx_column，bitmap_column)from ((${index_sql_1})union all(${index_sql_2})union all(${index_sql_3}))，

the expression is a logical operation expression among the sub-query conditions, and the supported logical operations may be and/or unequal, where the number of the sub-query condition is used to represent each sub-query condition, idx _ column is the number of the sub-query conditions, and bitmap _ column is a bitmap (first bitmap) to be queried, and $ { index _ sql _1}, $ { index _ sql _2} and $ { index _ sql _3} are query statements of three sub-query conditions, respectively.

An exemplary query instruction given according to the above-described custom function may be as follows:

a select bitmap count ('(1&2) |3') (idx, bitmap) from (select 1as idx, bitmap or (bitmap) from database, table whose value is 'male' group by value) non-aggregate all (select 2as idx, bitmap or (bitmap) from database, table whose value in ('shanghai', 'beijing') group by value) non-aggregate (select 3as idx, bitmap) from database.

The query instruction means that all men in Shanghai or Beijing, or people older than 25, are obtained. Here, the default _ value is a numerical label value in the first bitmap, the value is a string label value in the first bitmap, and operators (for example, in, >, <, >, hasAny, hasAll, and the like) used when constructing the query sentence may be different according to different label value types and query types.

By the method for separating the data acquisition logic and the calculation logic in the query instruction, each sub-query instruction can be executed concurrently, and then the logic operation is carried out according to the logic operation instruction at the final stage, so that the parallel capability of the data processing system is more fully utilized, and the query performance is fully improved.

Fig. 6 is a block diagram illustrating a structure of a data processing apparatus according to an exemplary embodiment of the present disclosure. As shown in fig. 6, the apparatus includes: an obtaining module 10, configured to obtain import data, where the import data includes a user ID and tag data corresponding to the user ID; a fragmentation module 20, configured to allocate a target data node to the user ID; a processing module 30, configured to store the imported data through first bitmaps corresponding to respective tags in the tag data in the target data node, where the first bitmaps are capable of storing 64-bit long-form data; the first bitmap comprises a mapping table, the first 32 bits of data in the 64-bit long integer data are used as key words of the mapping table, the second 32 bits of data are used as values of the key words, and the values of the key words are stored in a second bitmap supporting the 32-bit integer data.

In a possible embodiment, the slicing module 20 is further configured to: and performing first hash calculation on the user ID to obtain a first ID, determining a remainder obtained by dividing the first ID by the total number of the data nodes, and determining the target data node distributed for the user ID according to the remainder.

In a possible embodiment, the slicing module 20 is further configured to: performing second hash calculation on the user ID to obtain a second ID, and determining a target partition in the target data node to which the user ID is to be distributed according to the second ID; the processing module 30 is further configured to: and storing the imported data by using the first bitmaps respectively corresponding to the labels in the label data in the target partition.

In a possible implementation, the processing module 30 comprises: a first processing sub-module, configured to determine a target encoding range according to the number of user IDs stored in the target data node, and re-encode the user IDs into second IDs, where the second IDs are within the target encoding range, and store a one-to-one mapping relationship between the user IDs and the second IDs, where the second IDs are 64-bit long integer data, and the target encoding range is 2 bits of the first 32 bits of data that are the same as each other³²A number of adjacent codes; and the second processing submodule is used for importing the second ID into the first bitmap corresponding to each label.

In one possible embodiment, the method comprisesThe first processing sub-module is further configured to determine the target encoding range by: determining the target coding range as (a.N + M). 2³²～ (a·N+M+1)·2³²Wherein, in the step (A),

s is the number of user IDs already stored in the target data node,

the operation symbol is rounded down, N is the total number of data nodes, and M is the serial number of the target data node.

Fig. 7 is a block diagram illustrating a structure of a data processing apparatus according to still another exemplary embodiment of the present disclosure. As shown in fig. 7, the apparatus further includes: a determining module 40, configured to determine whether the tag in the tag data has the corresponding first bitmap in the target data node; a creating module 50, configured to create the first bitmap corresponding to the tag in the tag data in the target data node if the tag in the tag data does not have the corresponding first bitmap in the target data node.

Fig. 8 is a block diagram illustrating a structure of a data processing apparatus according to still another exemplary embodiment of the present disclosure. As shown in fig. 8, the apparatus further includes: a receiving module 60, configured to receive a query instruction, where the query instruction includes multiple sub-query conditions and a logic operation instruction between the multiple sub-query conditions; and the query module 70 is configured to obtain multiple groups of user data according to the multiple sub-query conditions, and calculate the multiple groups of user data according to the logic operation instruction to obtain target data.

Referring now to FIG. 9, shown is a schematic diagram of an electronic device 900 suitable for use in implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 9 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 9, the electronic device 900 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 901 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage means 908 into a Random Access Memory (RAM) 903. In the RAM 903, various programs and data necessary for the operation of the electronic apparatus 900 are also stored. The processing apparatus 901, the ROM 902, and the RAM 903 are connected to each other by a bus 904. An input/output (I/O) interface 905 is also connected to bus 904.

Generally, the following devices may be connected to the I/O interface 905: input devices 906 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 907 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 908 including, for example, magnetic tape, hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device 900 to communicate wirelessly or by wire with other devices to exchange data. While fig. 9 illustrates an electronic device 900 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such embodiments, the computer program may be downloaded and installed from a network through the communication device 909, or installed from the storage device 908, or installed from the ROM 902. The computer program performs the above-described functions defined in the methods of the embodiments of the present disclosure when executed by the processing apparatus 901.

It should be noted that the computer readable medium mentioned in the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring import data, wherein the import data comprises a user ID and label data corresponding to the user ID; allocating a target data node for the user ID; storing the imported data by using first bitmaps respectively corresponding to the tags in the tag data in the target data nodes, wherein the first bitmaps can store 64-bit long integer data; the first bitmap comprises a mapping table, the first 32 bits of data in the 64-bit integer data are used as a key word of the mapping table, the second 32 bits of data are used as a value of the key word, and the value of the key word is stored in a second bitmap supporting the 32-bit integer data.

Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present disclosure may be implemented by software or hardware. The name of a module does not in some cases constitute a limitation of the module itself, and for example, an acquisition module may also be described as a "module for acquiring import data".

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

Example 1 provides, in accordance with one or more embodiments of the present disclosure, a data processing method, the method including: acquiring import data, wherein the import data comprises a user ID and label data corresponding to the user ID; allocating a target data node to the user ID; storing the imported data by using first bitmaps respectively corresponding to the tags in the tag data in the target data nodes, wherein the first bitmaps can store 64-bit long integer data; the first bitmap comprises a mapping table, the first 32 bits of data in the 64-bit integer data are used as a key word of the mapping table, the second 32 bits of data are used as a value of the key word, and the value of the key word is stored in a second bitmap supporting the 32-bit integer data.

Example 2 provides the method of example 1, the assigning a target data node for the user ID comprising: and performing first hash calculation on the user ID to obtain a first ID, determining a remainder obtained by dividing the first ID by the total number of the data nodes, and determining the target data node distributed for the user ID according to the remainder.

Example 3 provides the method of example 2, the assigning a target data node for the user ID further comprising, in accordance with one or more embodiments of the present disclosure: performing second hash calculation on the user ID to obtain a second ID, and determining a target partition in the target data node to which the user ID is to be distributed according to the second ID; the storing the imported data through the first bitmaps respectively corresponding to the labels in the label data and in the target data node includes: and storing the imported data through the first bitmaps respectively corresponding to the labels in the label data in the target partition.

Example 4 provides the method of example 1, and the storing the imported data by the first bitmap corresponding to each tag in the tag data in the target data node according to each tag in the tag data includes: determining a target coding range according to the number of user IDs stored in the target data node, and re-coding the user IDs into second IDs, wherein the second IDs are in the target coding range, the one-to-one mapping relation between the user IDs and the second IDs is stored, the second IDs are 64-bit long-shape data, and the target coding range is 2 with the same first 32-bit data³²A number of adjacent codes; and importing the second ID into the first bitmap corresponding to each label.

Example 5 provides the method of example 4, wherein determining the target encoding range according to the number of user IDs already stored in the target data node comprises: determining the target coding range as (a.N + M). 2³²～(a·N+M+1)·2³²Wherein, in the step (A),

s is the number of user IDs already stored in the target data node,

and N is the total number of data nodes, and M is the serial number of the target data node.

Example 6 provides the method of example 1, wherein before storing the imported data in the target data node via the first bitmap corresponding to each of the tags in the tag data, the method further includes: judging whether the label in the label data has the corresponding first bitmap in the target data node or not; if the label in the label data does not have the corresponding first bitmap in the target data node, creating the first bitmap corresponding to the label in the label data in the target data node.

Example 7 provides the method of example 1, further comprising, in accordance with one or more embodiments of the present disclosure: receiving a query instruction, wherein the query instruction comprises a plurality of sub-query conditions and a logic operation instruction between the plurality of sub-query conditions; and respectively acquiring multiple groups of user data according to the multiple sub-query conditions, and calculating the multiple groups of user data according to the logic operation instruction to obtain target data.

Example 8 provides, in accordance with one or more embodiments of the present disclosure, a data processing apparatus, characterized in that the apparatus comprises: the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring import data, and the import data comprises a user ID and label data corresponding to the user ID; the fragmentation module is used for distributing target data nodes for the user ID; the processing module is used for storing the imported data through corresponding first bitmaps in the target data nodes through each label in the label data, and 64-bit long-form data can be stored in the first bitmaps; the first bitmap comprises a mapping table, the first 32 bits of data in the 64-bit integer data are used as a key word of the mapping table, the second 32 bits of data are used as a value of the key word, and the value of the key word is stored in a second bitmap supporting 32-bit integer data.

Example 9 provides a computer readable medium having stored thereon a computer program that, when executed by a processing apparatus, implements the steps of the method of any of examples 1-7, in accordance with one or more embodiments of the present disclosure.

Example 10 provides, in accordance with one or more embodiments of the present disclosure, an electronic device comprising: a storage device having a computer program stored thereon; processing means for executing the computer program in the storage means to carry out the steps of the method of any of examples 1-7.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present disclosure.

Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With regard to the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

Claims

1. A method of data processing, the method comprising:

allocating a target data node to the user ID;

the first bitmap comprises a mapping table, the first 32 bits of data in the 64-bit integer data are used as a key word of the mapping table, the second 32 bits of data are used as a value of the key word, and the value of the key word is stored in a second bitmap supporting the 32-bit integer data.

2. The method of claim 1, wherein the assigning the user ID to the target data node comprises:

and performing first hash calculation on the user ID to obtain a first ID, determining a remainder obtained by dividing the first ID by the total number of the data nodes, and determining the target data node distributed for the user ID according to the remainder.

3. The method of claim 2, wherein the assigning the user ID to the target data node further comprises:

performing second hash calculation on the user ID to obtain a second ID, and determining a target partition in the target data node to which the user ID is to be distributed according to the second ID;

the storing the imported data through the first bitmaps respectively corresponding to the tags in the tag data in the target data node includes:

and storing the imported data through the first bitmaps respectively corresponding to the labels in the label data in the target partition.

4. The method according to claim 1, wherein the storing the imported data with the first bitmap corresponding to each of the labels in the label data in the target data node comprises:

determining a target coding range according to the number of user IDs stored in the target data node, and re-coding the user IDs into second IDs, wherein the second IDs are in the target coding range, the one-to-one mapping relation between the user IDs and the second IDs is stored, the second IDs are 64-bit long integer data, and the target coding range is 2 with the same first 32-bit data³²A number of adjacent codes;

and importing the second ID into the first bitmap corresponding to each label.

5. The method of claim 4, wherein determining a target encoding range according to the number of user IDs stored in the target data node comprises:

s is the number of user IDs already stored in the target data node,

6. The method according to claim 1, wherein before storing the imported data via the respective corresponding first bitmap in the target data node via the respective label in the label data, the method further comprises:

judging whether the label in the label data has the corresponding first bitmap in the target data node or not;

if the label in the label data does not have the corresponding first bitmap in the target data node, creating the first bitmap corresponding to the label in the label data in the target data node.

7. The method of claim 1, further comprising:

receiving a query instruction, wherein the query instruction comprises a plurality of sub-query conditions and a logic operation instruction between the plurality of sub-query conditions;

and respectively acquiring multiple groups of user data according to the multiple sub-query conditions, and calculating the multiple groups of user data according to the logic operation instruction to obtain target data.

8. A data processing apparatus, characterized in that the apparatus comprises:

the processing module is used for storing the imported data through corresponding first bitmaps in the target data nodes through each label in the label data, and 64-bit long-form data can be stored in the first bitmaps;

9. A computer-readable medium, on which a computer program is stored, characterized in that the program, when being executed by processing means, carries out the steps of the method of any one of claims 1 to 7.

10. An electronic device, comprising:

a storage device having a computer program stored thereon;

processing means for executing the computer program in the storage means to carry out the steps of the method according to any one of claims 1 to 7.