TWI746511B - Data table connection method and device - Google Patents

Abstract

The invention provides a data table connection method and device. The method includes: receiving the data table connection task, the data table connection task instructs to connect the first data table and the second data table according to the connection conditions; according to the connection conditions, load the data records in the second data table to the distributed system Read the data record in the first data table as the current data record, determine the target node from at least two nodes according to the connection condition corresponding to the current data record, and read the first data record stored on the target node 2. The data record in the data table is used as the target data record; the current data record and the target data record are connected. The invention can reduce the computing resources consumed by the data table connection operation.

Description

Data table connection method and device

The present invention relates to the technical field of database, in particular to a method and device for connecting data tables.

With the development of the Internet, data has shown explosive growth, and data structures have begun to diversify. Data contains more and more information. Data warehouses play a huge role in this context. With the advent of the era of big data, data warehouses have turned into a decentralized structure to meet the explosive growth of computing and storage requirements. Distributed data warehouses generally use row storage and save data in the form of files. Therefore, the use of distributed data warehouses can improve the storage and computing performance of large data.

In the process of querying distributed data warehouses, it is often necessary to perform join calculations between data tables. When processing Join calculations between data tables in the prior art, generally, all the data tables to be joined are first shuffled and sorted through MapReduce, and then the data tables that have been sorted are merged on the Reducer side. operate. Shuffle sorting actually refers to the process of partitioning the data tables to be joined on the Map side according to the Join conditions and assigning them to different Reducer sides.

In a typical "star" Join scenario, assuming that the data table to be joined includes a main table and n auxiliary tables, and the main table contains M data records, then when performing Join calculations on the main table and n auxiliary tables, shuffle The total amount of data that needs to be processed for sorting includes the amount of data that needs to be processed by the shuffle main table, that is, the amount of data that needs to be processed by M*n and shuffle n auxiliary tables, which consumes a lot of computing resources.

Various aspects of the present invention provide a data table connection method and device, which are used to reduce the computing resources consumed by the data table connection operation.

An aspect of the present invention provides a data table connection method, including: receiving a data table connection task, the data table connection task instructs to perform a connection operation on the first data table and the second data table according to the connection condition; Condition, load the data records in the second data table to at least two nodes in the distributed system; read the data records in the first data table as current data records, and correspond to the current data records The connection condition of the target node is determined from the at least two nodes, and the data record in the second data table stored on the target node is read as the target data record; the current data record and the The target data record is connected.

Another aspect of the present invention provides a data table connection device, including: a receiving module for receiving data table connection tasks, the data table connection The connection task instruction connects the first data table and the second data table according to the connection conditions; the loading module is used to load the data records in the second data table to the distributed system according to the connection conditions The reading module is used to read the data record in the first data table as the current data record, according to the connection condition corresponding to the current data record, from the at least two nodes Determine the target node, and read the data record in the second data table stored on the target node as the target data record; the connection module is used to connect the current data record and the target data record .

In the present invention, when processing the data table connection task, first load the data records in the second data table to at least two nodes in the distributed system according to the connection conditions therein, and then directly read the first data record. Data records in the data table, and according to the connection conditions corresponding to the data records in the first data table read, read the data records in the second data table needed from the corresponding node, and then compare the read data records The data records in the two data tables are connected. It can be seen that the present invention only needs to distribute the second data table to different nodes according to the connection conditions, and does not need to distribute the first data table to different nodes, which reduces the amount of data that needs to be processed for shuffle sorting, and is beneficial to reduce connection operations. The computing resources consumed.

S101, S102, S103, S104‧‧‧Method steps

21‧‧‧Control Node

22‧‧‧Scheduling Node

23‧‧‧Compute Node

31‧‧‧Receiving Module

32‧‧‧Load module

33‧‧‧Reading Module

34‧‧‧Connecting module

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following A brief introduction will be made to the embodiments or the accompanying drawings needed in the description of the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. Under the premise of creative work, other drawings may be obtained based on these drawings.

Fig. 1 is a schematic flow chart of a data table connection method provided by an embodiment of the present invention; Fig. 2 is a schematic diagram of the architecture of a distributed system provided by another embodiment of the present invention; Fig. 3 is a data table connection provided by another embodiment of the present invention Schematic diagram of the structure of the device; FIG. 4 is a schematic diagram of the structure of a data table connection device provided by another embodiment of the present invention.

In order to make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

In the process of querying distributed data warehouses, it is often necessary to perform join calculations between data tables. In the prior art, when processing Join operations between data tables, since the data tables to be joined are relatively large, they are generally It first shuffles all the data tables to be joined through MapReduce, and then merges the ordered data tables on the Reducer side. Shuffle sorting actually refers to the process of partitioning the data tables to be joined on the Map side according to the Join conditions and assigning them to different Reducer sides. Due to the need to shuffle all the data tables to be joined, more computing resources are consumed.

In view of the above technical problems, the present invention provides a solution, that is, by distributing and storing the second data table on multiple nodes, it becomes a distributed cache. When the first data table is processed, the remote node is obtained through the network. The data records in the second data table stored on the above, so as to perform a distributed hash map join (Hash map Join), so that there is no need to shuffle the main table, which can save the computing resources consumed by shuffle the first data table .

FIG. 1 is a schematic flowchart of a data table connection method provided by an embodiment of the present invention. As shown in Figure 1, the method includes:

101. Receive a data table connection task, where the data table connection task instructs to perform a connection operation on the first data table and the second data table according to the connection conditions.

102. According to the above connection conditions, load the data records in the second data table to at least two nodes in the distributed system.

103. Read the data record in the first data table as the current data record, determine the target node from at least two nodes according to the connection conditions corresponding to the current data record, and read the second data table stored on the target node The data record serves as the target data record.

104. Connect the current data record and target data record do.

This embodiment provides a data table connection method, which can be executed by a data table connection device to perform a Join operation between data tables while minimizing the consumption of computing resources. The method provided in this embodiment is applicable to a distributed system, where different machines in the distributed system can be used as a node respectively. This embodiment does not limit the implementation architecture of the distributed system, for example, it may be but not limited to the MapReduce architecture.

When a Join operation between data tables is required, the data table connection task can be sent to the data table connection device; the data table connection device receives the data table connection task. The data table connection task instructs to perform Join processing on the first data table and the second data table according to the connection conditions. The first data table and the second data table here are actually data tables to be connected.

In specific implementation, the data table connection task carries information such as connection conditions, the identification of the first data table, the identification of the second data table, the storage location of the first data table, and the storage location of the second data table. Among them, the data table connection device can analyze the data table connection task to obtain connection conditions, the identification of the first data table, the identification of the second data table, the storage location of the first data table, and the storage location of the second data table, etc. Information, and according to the identification of the first data table and the identification of the second data table to determine the data table that needs to be joined. In addition, the first data can be read according to the storage location of the first data table and the storage location of the second data table Table and second data table.

In a practical application, the first data table can be used as the main table, and the second data table can be implemented as the auxiliary table. Among them, the number of auxiliary tables can be one or more.

After the data table connection device receives the data table connection task, it can learn that it is necessary to perform a Join operation on the first data table and the second data table according to the connection conditions. After that, before performing the Join operation, first load the data records in the second data table to at least two nodes in the distributed system according to the connection conditions to realize distributed storage.

Preferably, the data records in the second data table located on each of the at least two nodes have a data amount less than the memory limit of a single node, that is, they are distributed to each of the at least two nodes The data records in the second data table can all be placed in the storage space (preferably memory) of the corresponding node.

In an optional embodiment, the above connection condition includes at least one target key required for the connection, and the target key here is actually a key in a key-value pair. Based on this, the data table connection device can specifically perform a hash operation on each target key in the at least one target key to obtain the hash value of each target key; according to the hash value of each target key and the above used to store the second data table The number of at least two nodes in the data record of, determine the node corresponding to each target key; load the data record corresponding to each target key in the second data table to the node corresponding to each target key.

Further, the data table connection device may use the hash value of each target key to modulate the number of at least two nodes used for storing data records in the second data table, and determine the node corresponding to each target key according to the result of the modulus. Specifically, the node represented by the result of the modulus can be used as the node corresponding to the target key. or, The data table connection device can be used to store the second data table according to the above The data records the number of at least two nodes and the number of target keys, and each target key is equally divided into each node. In the process of equalization, the target key with similar hash value can be divided according to the hash value of each target key. To the same node. The similar hash values here may mean that the difference between the hash values is less than a preset threshold, but it is not limited to this.

Further, in the process of loading the data records in the second data table to at least two nodes, specifically, the data records in the second data table may be loaded into the memories of at least two nodes. The data records in the second data table are stored in the memory of the node, which can be read at any time, and the reading speed is faster, which is beneficial to improve the efficiency of the Join operation.

It is worth noting that, preferably, the data records in the second data table can be loaded into the memory of the above at least two nodes, but it is not limited to the memory, and it can also be a solid state drive (Solid State Drive) of the node. Drives, SSD) or other storage media.

In an optional embodiment, before loading the data records in the second data table to at least two nodes in the distributed system according to the connection conditions, it can be determined whether the data volume of the second data table is greater than that of a single node If the judgment result is yes, that is, the amount of data in the second data table is greater than the memory limit of a single node, which means that the data records in the second data table cannot all be placed in the memory of a single node, so According to the connection conditions, the data records in the second data table can be loaded into at least two nodes, so that the data records in the second data table distributed on each node can all be placed in the memory of the corresponding node , Realize decentralized storage. Simply put, the data records in the second data table distributed to each node, its The amount of data is less than the memory limit of a single node.

If the above judgment result is no, that is, the amount of data in the second data table is less than or equal to the memory limit of a single node, which means that all the data records in the second data table can be placed in the memory of a single node, which is better Yes, all the data records of the second data table can be stored in the memory of a single node, thereby saving the shuffle sorting of the data records of the second data table and saving computing resources.

The above loading the data records in the second data table to at least two nodes in the distributed system, which is equivalent to turning the second data table into multiple small tables, and each small table can be all in the memory of the corresponding node It can be put down to form a distributed KV storage, which makes it possible to do distributed Hash map Join without the need to do Sort Merge Join. To do distributed Hash map Join, you don’t need to sort the data records in the first data table, you can directly read the data records in the first data table, and correspond to the data records in the first data table read. The connection condition is to read the required data records in the second data table from the corresponding node, and then perform the Join operation on the data records in the read two data tables.

Among them, the difference between the distributed Hash map Join in this embodiment and the existing Hash map Join is that when processing the first data table, instead of looking up data records in the second data table in the local memory, it uses the Internet. To obtain the data records in the second data table stored on the remote node.

Specifically, after the data records in the second data table are loaded into at least two nodes in the distributed system, the data table connection device can go to the storage location of the first data table to read the data records in the first data table, Will read The data record of is used as the current data record. According to the connection conditions corresponding to the current data record, the target node is determined from the above two nodes. The target node here refers to the second data table required for the Join operation with the current data record The node where the data record is located, and then read the data record in the second data table stored on the target node as the target data record, where the target data record refers to the second data table required for the Join operation with the current data record Data records.

After reading the current data record and the target data record required for the Join operation with the current data record, perform the Join operation on the current data record and the target data record. Since how to perform the Join operation on the current data record and the target data record is not the focus of the present invention, it will not be described in detail here, and the processing flow of the Join operation in the prior art can be referred to.

In an alternative embodiment, considering that there may be target data records required for the Join operation with the current data record in the local cache of the data table connection device, based on this, according to the connection conditions corresponding to the current data record, from Before determining the target node among at least two nodes, and reading the data record in the second data table stored on the target node as the target data record, it can be judged whether the target data exists in the local cache according to the connection condition corresponding to the current data record Record, if the result of the judgment is no, execute the target node from at least two nodes according to the connection condition corresponding to the current data record, and read the data record in the second data table stored on the target node as the target data record Operation; if the judgment result is yes, the target data record can be obtained from the local cache, so that the target data record can be obtained more quickly, and the network resources consumed to obtain the target data record can be saved Source, improve the efficiency of Join operation.

Further, the connection condition corresponding to the current data record may be a target key, and an implementation manner of determining the target node from at least two nodes according to the connection condition corresponding to the current data record includes: hashing the target key corresponding to the current data record Perform operations to obtain the hash value of the target key corresponding to the current data record; and determine the node corresponding to the target key corresponding to the current data record as the target node according to the hash value of the target key corresponding to the current data record and the number of at least two nodes mentioned above.

Furthermore, in the process of determining to fetch target data from a certain node through the target key, if there are multiple target keys, batch operations can be performed, which can give full play to the advantages of a distributed system and improve processing performance.

It can be seen from the above analysis that in this embodiment, when processing the data table connection task, first load the data records in the second data table to at least two nodes according to the connection conditions, which is equivalent to becoming a distributed KV storage (that is, there will be a distributed hash table), so that you do not need to do Sort Merge Join, so that you can do distributed Hash map Join, that is, you do not need to sort the data records in the first data, but can be read directly Take the data records in the first data table, and read the required data records in the second data table from the corresponding node according to the connection conditions corresponding to the data records in the first data table read, and then read The data records in the obtained two data tables perform the Join operation. It can be seen that in this embodiment, only the second data table needs to be distributed to different nodes according to the connection conditions, and the first data table does not need to be distributed to different nodes, which reduces the amount of data that needs to be processed for shuffle sorting, and is beneficial to reduce connections. The computing resources consumed by the operation.

The following compares the computing resources consumed by Sort Merge Join and distributed Hash map Join to illustrate the advantages brought by the technical solution of the present invention.

Suppose that the main table is A, and its data size is 100T. Suppose there are two auxiliary tables B and C, the data size of auxiliary table B is 10G, and the data size of auxiliary table C is 100G.

If the existing Sort Merge Join is used, the shuffle sorting stage requires a sorting process for the main table A and auxiliary table B, and a sorting process for the main table A and auxiliary table C. Each sorting process includes reading through the network IO The data table is sorted by the CPU, so the resource consumption of each sorting process includes: the CPU occupied by the sorting and the network IO occupied by the table reading. For ease of description, the amount of processed data represents resource consumption. Here, considering that the amount of data processed by CPU sorting is the amount of data read through network IO, a piece of data is used to represent the resource for each sorting process. Consumption, the total resource consumption required in the shuffle sorting stage is: (100T+10G)+(100T+100G)=2 * 100T+10G+100G.

If the distributed Hash map Join of the present invention is used, the auxiliary table B needs to be distributed to different nodes in the shuffle sorting phase, and auxiliary table C needs to be distributed to different nodes. Each time a certain table is distributed to different nodes, including Reading data tables through network IO and sorting through CPU, so the resource consumption of distributing a table to different nodes also includes: CPU occupied by sorting and network IO occupied by table reading. For ease of description, the amount of processed data represents resource consumption. Here, considering that the amount of data processed by the CPU sorting is also the amount of data read through the network IO, the amount of data is To represent the resource consumption of each sorting process, the total resource consumption required for the shuffle sorting stage is: 10G+100G.

It can be seen from the above that, because the technical solution of the present invention only needs to distribute the second data table to different nodes according to the connection conditions, it is not necessary to distribute the first data table to different nodes, which reduces the amount of data that needs to be processed for shuffle sorting, which is beneficial Reduce the computing resources consumed by connection operations.

FIG. 2 is a schematic diagram of the architecture of a distributed system provided by another embodiment of the present invention. As shown in FIG. 2, the distributed system includes: a control node 21, a scheduling node 22, and at least two computing nodes 23. Further, as shown in FIG. 2, the computing node 23 at least includes a cache module and a processing module.

It is worth noting that the distributed system shown in FIG. 2 is only an example and is not limited to this. For example, the scheduling node 22 in FIG. 2 can be omitted to obtain a simpler distributed system.

The technical scheme of the present invention will be described in detail below based on the distributed system shown in FIG. 2.

The control node 21 is responsible for receiving the data table connection task, and learns from the data table connection task that it is necessary to perform a connection operation on the first data table and the second data table according to the connection conditions.

The control node 21 can send scheduling instructions to the scheduling node 22 according to the data table connection task, and control the scheduling node 22 to schedule the computing nodes 23 available in the distributed system. The scheduling node 22 specifically receives the scheduling instruction of the control node 21, and schedules the computing node 23 in the distributed system according to the scheduling instruction.

In the process of scheduling computing nodes 23 in the decentralized system described above, The control node 21 provides the computing node 23 with the configuration file required for subsequent loading of the data records in the second data table through the scheduling node 22. The configuration file records the identifier of the second data table, the storage location, and the data to be loaded. Recorded identification information, etc.

A loading process is deployed on each computing node 23 in the distributed system, and the loading process mainly loads the data records in the second data table into the cache module according to the configuration file. Specifically, the scheduling module 22 starts the loading process on each computing node 23. The loading process reads the corresponding data record in the second data table from the corresponding storage location according to the configuration file, and loads the data record read by it. Into the cache module. It is worth noting that the second data table can be stored in a space outside the distributed system, but it is not limited to this.

When all the load processes on the computing nodes 23 finish the load operation and enter the listening port state, a load end instruction is returned to the control node 21 through the scheduling node 22. According to the load end instruction, the control node 21 can learn that each computing node 23 has loaded the data records in the second data table into the cache module.

The control node 21 sends a start instruction to the scheduling node 22, so that the scheduling node 22 starts the processing process on each computing node 23. A processing process is deployed on each computing node 23. The processing process is mainly used to read the data record in the first data table as the current data record, and determine the second corresponding to the key according to the key corresponding to the read current data record. The computing node 23 where the data record in the data table is located, reads the data record in the second data table stored on the determined computing node 23 as the target data record, and performs Join on the read current data record and the target data record operate. value It should be noted that the first data table can be stored in a space outside the distributed system, but it is not limited to this.

Optionally, in a specific implementation manner, the foregoing computing nodes 23 may be implemented in a server/user manner. For example, the cache module of each computing node 23 can be implemented as a cache server (CacheService), and the cache server also includes a cache manager (CacheManager), and each cache module corresponds to a cache node (CacheNode ); Correspondingly, the processing module of each computing node 23 is implemented as a cache client (CacheClient).

Specifically, CacheManager coordinates and manages all CacheNodes. CacheNode is responsible for loading data into memory and providing services. Optionally, the second data table can be stored and managed in the form of a shard file. The purpose of using the shard file is because in the event of a failure (failover), once the CacheNode restarts, only the shard file needs to be read again, making the processing relatively simple.

CacheClient accesses CacheService, performs hash calculation on the key, and reads data from one of the CacheNodes based on the calculation result. In addition, there should be a part of local cache in CacheClient. Usually, cache algorithms such as Least Recently Used (LRU) are used to save some of the data that has been read in the local cache, so that CacheClient can take priority from The required data is read in the local cache. If the required data is read in the local cache, the operation of reading data from the CacheNode through the network can be saved, which is beneficial to improve efficiency and save resources.

It can be seen from the above analysis that when processing the data table connection task in this embodiment, first, according to the connection conditions, load the data records in the second data table to at least two nodes to realize distributed storage, so that it can be directly read. Take the data records in the first data table, and read the required data records in the second data table from the corresponding node according to the connection conditions corresponding to the data records in the first data table read, and then read The data records in the obtained two data tables are joined to realize distributed Hash map Join. It can be seen that in this embodiment, only the second data table needs to be distributed to different nodes according to the connection conditions, and the first data table does not need to be distributed to different nodes, which reduces the amount of data that needs to be processed for shuffle sorting, and is beneficial to reduce connections. The computing resources consumed by the operation.

It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described sequence of actions. Because according to the present invention, some steps can be performed in other order or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the related actions and modules are not necessarily required by the present invention.

In the above-mentioned embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in an embodiment, reference may be made to related descriptions of other embodiments.

FIG. 3 is a schematic structural diagram of a data table connection device provided by another embodiment of the present invention. As shown in FIG. 3, the device includes: a receiving module 31, a loading module 32, a reading module 33, and a connecting module 34.

The receiving module 31 is configured to receive a data table connection task, which instructs to perform a connection operation on the first data table and the second data table according to the connection conditions.

The loading module 32 is used for loading the data records in the second data table to at least two nodes in the distributed system according to the connection conditions.

The reading module 33 is used to read the data record in the first data table as the current data record, determine the target node from at least two nodes according to the connection condition corresponding to the current data record, and read the data stored on the target node The data record in the second data table is used as the target data record.

The connection module 34 is used to connect the current data record and the target data record.

Preferably, the data records in the second data table located on each of the at least two nodes have a data amount less than the memory limit of a single node, that is, they are distributed to each of the at least two nodes The data records in the second data table can all be placed in the storage space (preferably memory) of the corresponding node.

Furthermore, as shown in FIG. 4, the device further includes: a first judgment module 35.

The first judgment module 35 is used to judge whether the amount of data in the second data table is greater than the memory limit of a single node, and when the judgment result is yes, trigger the load module 32 to execute the second data table according to the connection condition The data records are loaded into the operation of at least two nodes in the distributed system.

Furthermore, as shown in FIG. 4, the device further includes: a second judgment module 36.

The second judgment module 36 is used for judging whether the target data record exists in the local cache according to the connection condition corresponding to the current data record, and triggering the reading module 33 to execute the correspondence based on the current data record when the judgment result is no The operation of determining the target node from at least two nodes, and reading the data record in the second data table stored on the target node as the target data record.

In an optional embodiment, the above-mentioned connection condition includes at least one target key required for connection. The target key here is actually the key in the key-value pair.

Based on the foregoing, the loading module 32 is specifically configured to: perform a hash operation on each target key in the at least one target key to obtain the hash value of each target key; according to the hash value of each target key and the at least two target keys The number of nodes determines the node corresponding to each target key; the data records corresponding to each target key in the second data table are respectively loaded onto the node corresponding to each target key.

In an alternative embodiment, the loading module 32 is specifically configured to load the data records in the second data table into the memory of the at least two nodes according to the connection condition. The data records in the second data table are stored in the memory of the node, which can be read at any time, and the reading speed is faster, which is beneficial to improve the efficiency of the Join operation.

It is worth noting that, preferably, the data records in the second data table can be loaded into the memory of the at least two nodes mentioned above, but it is not limited to the memory, and it can also be the SSD or other storage media of the node. .

The data table connection device provided in this embodiment is processing data table connection During the task, first load the data records in the second data table to at least two nodes in the distributed system according to the connection conditions, which is equivalent to becoming a distributed KV storage, so there is no need to do Sort Merge Join makes it possible to do distributed Hash map Join, that is, it is not necessary to sort the data records in the first data, but can directly read the data records in the first data table, and according to the first data read The data records in the table correspond to the connection conditions, the data records in the second data table required are read from the corresponding node, and then the data records in the two data tables read are connected. It can be seen that with the data table connection device provided in this embodiment, only the second data table needs to be distributed to different nodes according to the connection conditions, and the first data table does not need to be distributed to different nodes, which reduces the need for shuffle sorting. The amount of data is conducive to reducing the computing resources consumed by the connection operation.

Those skilled in the art can clearly understand that, for the convenience and conciseness of the description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided by the present invention, it should be understood that the disclosed system, device, and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or elements may be combined or may be Integrate into another system, or some features can be ignored or not implemented. Another point, the mutual coupling or direct coupling shown or discussed Or the communication connection may be indirect coupling or communication connection of devices or units through some interfaces, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. . Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be realized either in the form of hardware, or in the form of hardware plus software functional units.

The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer readable storage medium. The above-mentioned software functional unit is stored in a storage medium, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute the various embodiments of the present invention Part of the method. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disks or optical disks, etc., which can store program codes Media.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: However, the technical solutions described in the foregoing embodiments can be modified, or some of the technical features can be equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and spirit of the technical solutions of the embodiments of the present invention. Scope.

Claims (10)

A data table connection method for a distributed system, characterized in that the method includes: in the query process of the distributed data warehouse, receiving a data table connection task, the data table connection task instructs to compare the first data table and the first data table according to the connection conditions. Two data tables are connected; according to the connection condition, the data records in the second data table are loaded to at least two nodes in the distributed system to realize distributed storage; and the data in the first data table is read The data record is used as the current data record. According to the connection condition corresponding to the current data record, the target node is determined from the at least two nodes, and the data record in the second data table stored on the target node is read as the target data record ; And the connection operation of the current data record and the target data record. The method according to item 1 of the scope of patent application, wherein, before loading the data records in the second data table to at least two nodes in the distributed system according to the connection condition, the method includes: judging the first data record Whether the data volume of the second data table is greater than the memory limit of a single node; and if the judgment result is yes, execute according to the connection condition, load the data records in the second data table into the distributed system at least two Operations on the node. The method according to item 1 of the scope of patent application, wherein the target node is determined from the at least two nodes according to the connection condition corresponding to the current data record, and the second data stored on the target node is read Before the data record in the table is used as the target data record, it includes: judging whether the target data record exists in the local cache according to the connection condition corresponding to the current data record; and if the judgment result is no, execute the corresponding corresponding to the current data record The operation of determining the target node from the at least two nodes, and reading the data record in the second data table stored on the target node as the target data record. The method according to item 1 of the scope of patent application, wherein the connection condition includes at least one target key required for connection; the data record in the second data table is loaded into the distributed system according to the connection condition At least two nodes in the at least two nodes, including: performing a hash operation on each target key in the at least one target key to obtain the hash value of each target key; according to the hash value of each target key and the number of the at least two nodes, Determine the node corresponding to each target key; and load the data record corresponding to each target key in the second data table to the node corresponding to each target key. The method according to any one of items 1 to 4 of the scope of patent application, wherein the data records in the second data table are loaded to at least two nodes in the distributed system according to the connection condition, It includes: according to the connection condition, loading the data records in the second data table into the memory of the at least two nodes. A data table connection device of a distributed system, characterized in that the device includes: The receiving module is used to receive the data table connection task in the query process of the distributed data warehouse. The data table connection task instructs the connection operation of the first data table and the second data table according to the connection conditions; loading the module, According to the connection condition, load the data records in the second data table to at least two nodes in the distributed system to realize distributed storage; the reading module is used to read the first data table The data record in the current data record is used as the current data record, the target node is determined from the at least two nodes according to the connection condition corresponding to the current data record, and the data record in the second data table stored on the target node is read as the target Data record; and a connection module for connecting the current data record and the target data record. The device according to item 6 of the scope of patent application, further comprising: a first judgment module for judging whether the amount of data in the second data table is greater than the memory limit of a single node, and when the judgment result is yes The loading module is triggered to perform the operation of loading the data records in the second data table to at least two nodes in the distributed system according to the connection condition. The device according to item 6 of the scope of patent application, further comprising: a second judgment module for judging whether the target data record exists in the local cache according to the connection condition corresponding to the current data record, and when judging If the result is no, trigger the reading module to execute the connection condition corresponding to the current data record to determine the target node from the at least two nodes, and The operation of reading the data record in the second data table stored on the target node as the target data record. The device according to item 6 of the scope of patent application, wherein the connection condition includes at least one target key required for connection; the loading module is specifically used to: hash each target key of the at least one target key respectively Perform operations to obtain the hash value of each target key; determine the node corresponding to each target key according to the hash value of each target key and the number of the at least two nodes; and the data corresponding to each target key in the second data table The records are respectively loaded on the nodes corresponding to each target key. The device according to any one of items 6 to 9 of the scope of patent application, wherein the loading module is specifically used to: load the data records in the second data table into the at least two data records according to the connection condition. In the memory of each node.

Images