CN110399394A - The data traversal method, apparatus and calculate node of calculate node - Google Patents

Abstract

Present disclose provides a kind of data traversal methods of calculate node.The data are stored in multiple data fragmentations, which comprises obtain multiple burst informations, the corresponding data fragmentation of each burst information；And multiple burst informations are concurrently based on at least one other calculate node and execute reading-traversing operation, to traverse the data for reading the corresponding data fragmentation storage of multiple burst informations, wherein, different calculate nodes traversal reads the data of different data fragmentation storages.The disclosure additionally provides the data traversal device and calculate node of a kind of calculate node.

Description

The data traversal method, apparatus and calculate node of calculate node

Technical field

This disclosure relates to field of computer technology, more particularly, to a kind of data traversal method, apparatus of calculate node And calculate node.

Background technique

With the fast development of computer technology, in order to meet the data such as internet, Streaming Media, high-performance calculation, the energy Intensive, high concurrent access industry requirement, cluster-based storage mode are come into being.

During realizing disclosure design, at least there are the following problems in the prior art for inventor's discovery: tradition meter Calculate cluster to traverse the data stored in storage cluster by the following method: calling data-storage system traversal method first is read In the data stored in storage cluster to the memory or disk of ETL server, then by data distribution to each calculate node.But In the method, meter can not be played due to only supporting the sequence of single thread to read for storage clusters such as Redis, MySQL The concurrent advantage of cluster multinode is calculated, therefore time-consuming for data traversal；Furthermore data needs carry out secondary transmission on network, Limited speed is traversed in the network bandwidth and I O process ability of ETL server, therefore data traversal speed is slow.

Summary of the invention

In view of this, present disclose provides the data traversal method, apparatus and calculate node of a kind of calculate node.

The first aspect of the disclosure provides a kind of data traversal method of calculate node, and the data are stored in multiple numbers According in fragment, this method comprises: obtaining multiple burst informations, each burst information corresponds to a data fragmentation；And with extremely Few other calculate nodes are concurrently based on multiple burst informations and execute reading-traversing operation, read multiple fragments with traversal The data of the corresponding data fragmentation storage of information, wherein different calculate node traversals reads different data fragmentation storages Data.

In accordance with an embodiment of the present disclosure, the above method further includes that preset memory space is written in multiple burst informations；On Stating reading-traversing operation includes: the burst information read in multiple burst informations from preset memory space；And it is based on The burst information of reading, traversal read the data of the corresponding data fragmentation storage of the burst information.

In accordance with an embodiment of the present disclosure, above-mentioned multiple data fragmentations are stored on multiple memory nodes of storage cluster, often A memory node is stored with one or more data fragmentations, the above method further include: obtains the memory node of multiple data fragmentations Information；It includes: based on preset allocation rule and the memory node that preset memory space then, which is written, in multiple burst informations Information, multiple burst informations are evenly distributed, so that the burst information of the different data fragment with identical memory node information It is alternatively arranged；And evenly distributed multiple burst informations are sequentially written in preset memory space；It is above-mentioned from preset storage It includes: sequentially to read one from evenly distributed multiple burst informations that a burst information in multiple burst informations is read in space A burst information.

In accordance with an embodiment of the present disclosure, above-mentioned reading-traversing operation further include: more being had read from preset memory space In the case where a burst information in a burst information, the burst information of reading is set to the second shape by first state State；And in the case where traversal has read the data of the corresponding data fragmentation storage of burst information, by burst information by second State is set to the third state；Wherein, burst information first state indicate burst information can by calculate node or at least one Other calculate nodes are read；Second state of burst information indicates that burst information has been read, and can not be read again；Fragment letter The third state of breath indicates that the data of the corresponding data fragmentation storage of burst information have been traversed and has read that the burst information cannot It is enough to be read again.

In accordance with an embodiment of the present disclosure, the above method further include: judge to have read multiple fragments from preset memory space Whether other calculate nodes of a burst information in information are malfunctioning node；And other calculate nodes be malfunctioning node In the case where, the burst information which reads is set to first state by the second state, for calculate node or except this At least one other calculate node outside malfunctioning node is read.

In accordance with an embodiment of the present disclosure, the above method further include: judge whether multiple burst informations are in the third state； In the case where there is the burst information for being not at the third state, returning and rejudging after the preset time of interval；And multiple In the case that burst information is in the third state, the traversal of the data is completed.

In accordance with an embodiment of the present disclosure, above-mentioned calculate node is by calculate node and at least one other calculate node using pre- If election regulation elect to obtain；And/or it is greater than calculate node and at least one other in the quantity of multiple burst informations In the case where the node total number of calculate node, executed by calculate node and/or at least one other calculate node circulation above-mentioned Reading-traversing operation, so that the data of the corresponding data fragmentation storage of multiple burst informations are traversed reading.

The second aspect of the disclosure provides a kind of data traversal method of calculate node, and the data are stored in multiple numbers According in fragment, this method comprises: being concurrently based on multiple burst informations at least one other calculate node executes reading-traversals Operation, to traverse the data for reading the corresponding data fragmentation storage of multiple burst informations, wherein multiple burst informations are by least one A calculate node in other a calculate nodes obtains in advance, the corresponding data fragmentation of each burst information, different Calculate node traversal reads the data of different data fragmentation storages.

In accordance with an embodiment of the present disclosure, above-mentioned multiple burst informations are by a calculating at least one other calculate node Node is written after obtaining in advance in preset memory space；Above-mentioned reading-traversing operation includes: to read from preset memory space A burst information in multiple burst informations；And the burst information based on reading, traversal read the corresponding number of burst information The data stored according to fragment.

In accordance with an embodiment of the present disclosure, above-mentioned reading-traversing operation further include: more being had read from preset memory space In the case where a burst information in a burst information, the burst information of reading is set to the second state by first state；With And in the case where traversal has read the data of the corresponding data fragmentation storage of burst information, burst information is set by the second state For the third state；Wherein, the first state of burst information indicates that burst information can be by calculate node or at least one other meter Operator node is read；Second state of burst information indicates that burst information has been read, and can not be read again；The of burst information Three condition indicates that the data of burst information corresponding data fragmentation storage have been traversed and read, the burst information can not again by It reads.

In accordance with an embodiment of the present disclosure, the above-mentioned calculate node for obtaining multiple burst informations in advance is by calculate node and at least One other calculate node elects to obtain using preset election regulation；And/or it is greater than in the quantity of multiple burst informations In the case where the node total number of calculate node and at least one other calculate node, calculate node and/or at least one other meter Operator node circulation executes above-mentioned reading-traversing operation, so that the data quilt of the corresponding data fragmentation storage of multiple burst informations Traversal is read.

The third aspect of the disclosure provides a kind of data traversal device of calculate node, and the data are stored in multiple numbers According in fragment, which includes: acquisition module, obtains multiple burst informations, the corresponding data fragmentation of each burst information； And first processing module, reading-traversing operation is executed based on multiple burst informations, wherein the data traversal of calculate node fills It sets and is concurrently based on multiple burst informations execution reading-traversals behaviour with the data traversal device of at least one other calculate node Make, to traverse the data for reading the corresponding data fragmentation storage of multiple burst informations, the data traversal dress of different calculate nodes Set the data that traversal reads different data fragmentation storages.

In accordance with an embodiment of the present disclosure, above-mentioned apparatus further includes write operation module, pre- for multiple burst informations to be written If memory space；Above-mentioned first processing module includes the first burst information reading unit and the first data traversal reading unit, First burst information reading unit is used to read a burst information in multiple burst informations from preset memory space；First Data traversal reading unit is used for the burst information based on reading, and traversal reads the corresponding data fragmentation storage of burst information Data.

In accordance with an embodiment of the present disclosure, above-mentioned multiple data fragmentations are stored on multiple memory nodes of storage cluster, often A memory node is stored with one or more data fragmentations；Above-mentioned acquisition module is also used to obtain multiple data fragmentations Memory node information；Above-mentioned write operation module includes burst information arrangement units and write operation unit, and burst information arrangement is single Member is evenly distributed by multiple burst informations for being based on preset allocation rule and the memory node information, so as to have phase Burst information with the different data fragment of memory node information is alternatively arranged；Write operation unit is used for will be evenly distributed multiple Burst information is sequentially written in preset memory space；Above-mentioned first burst information reading unit is believed from evenly distributed multiple fragments A burst information is sequentially read in breath.

In accordance with an embodiment of the present disclosure, above-mentioned first processing module further includes the first changing unit, and the first changing unit is used In having read from preset memory space a burst information in multiple burst informations in the first burst information reading unit In the case of, the burst information of reading is set to the second state by first state；And it is traversed in the first data traversal reading unit In the case where the data for having read the corresponding data fragmentation storage of burst information, burst information is set to third shape by the second state State；Wherein, burst information first state indicate burst information can by the data traversal device of calculate node or at least one The data traversal device of other calculate nodes is read；Second state of burst information indicates that burst information has been read, can not It is read again；The third state of burst information indicates that the data of the corresponding data fragmentation storage of burst information have been traversed reading Complete, which can not be read again.

In accordance with an embodiment of the present disclosure, above-mentioned first processing module further includes breakdown judge unit, for judging from default Memory space have read a burst information in multiple burst informations other calculate nodes data traversal device whether For failed equipment；Above-mentioned first changing unit is also used to the case where the data traversal device of other calculate nodes is failed equipment Under, the burst information that the data traversal device of the malfunctioning node is read is set to first state by the second state, for calculating section The data traversal device of point or the data traversal device of at least one other calculate node in addition to the malfunctioning node are read.

In accordance with an embodiment of the present disclosure, above-mentioned first processing module further includes state judging unit, for judging multiple points Whether piece information is in the third state；And in the case where there is the burst information for being not at the third state, when interval is default Between after return rejudge；And in the case where multiple burst informations are in the third state, complete the traversal of the data.

In accordance with an embodiment of the present disclosure, the data traversal device of above-mentioned calculate node by calculate node data traversal device And the data traversal device of at least one other calculate node elects to obtain using preset election regulation；And/or more The quantity of a burst information is greater than the data traversal device of calculate node and the data traversal of at least one other calculate node In the case where the regulating device sum of device, data traversal device and/or at least one other calculate node by calculate node Data traversal device circulation executes above-mentioned reading-traversing operation, so that the number of the corresponding data fragmentation storage of multiple burst informations According to being traversed reading.

The fourth aspect of the disclosure provides a kind of data traversal device of calculate node, and the data are stored in multiple numbers According in fragment, which includes Second processing module, for executing reading-traversing operation, Duo Gefen based on multiple burst informations Piece information is obtained in advance by the data traversal device of a calculate node at least one other calculate node, each fragment Information corresponds to a data fragmentation.Wherein, the data traversal device of the calculate node and at least one other calculate node Data traversal device is concurrently based on multiple burst informations and executes above-mentioned reading-traversing operation, reads multiple fragment letters with traversal The data of corresponding data fragmentation storage are ceased, the data traversal device traversal of different calculate nodes reads different data fragmentations The data of storage.

In accordance with an embodiment of the present disclosure, above-mentioned multiple burst informations are by a calculating at least one other calculate node The data traversal device of node is written in preset memory space after obtaining in advance；Above-mentioned Second processing module includes the second fragment Information reading unit and the second data traversal reading unit, the second burst information reading unit are used to read from preset memory space Take a burst information in multiple burst informations；Second data traversal reading unit is used for the burst information based on reading, time Go through the data for reading the corresponding data fragmentation storage of burst information.

In accordance with an embodiment of the present disclosure, above-mentioned Second processing module further includes the second changing unit, in the second fragment In the case that Information reading unit has read a burst information in multiple burst informations from preset memory space, it will read Burst information the second state is set to by first state；And burst information has been read in the second data traversal reading unit traversal In the case where the data of corresponding data fragmentation storage, burst information is set to the third state by the second state；Wherein, fragment is believed The first state of breath indicates that burst information can be by the data traversal device of calculate node or at least one other calculate node Data traversal device is read；Second state of burst information indicates that burst information has been read, and can not be read again；Fragment letter The third state of breath indicates that the data of the corresponding data fragmentation storage of burst information have been traversed and has read that the burst information cannot It is enough to be read again.

In accordance with an embodiment of the present disclosure, the data traversal device of the above-mentioned calculate node for obtaining multiple burst informations in advance by The data traversal device of calculate node and the data traversal device of at least one other calculate node use preset election regulation Election obtains；And/or the quantity of multiple burst informations be greater than calculate node data traversal device and at least one its In the case where the device sum of the data traversal device of his calculate node, the data traversal device of calculate node and/or at least The data traversal device circulation of one other calculate node executes above-mentioned reading-traversing operation, so that multiple burst informations are corresponding Data fragmentation storage data be traversed reading.

5th aspect of the disclosure provides a kind of calculate node, comprising: one or more processors；Storage device is used In the one or more programs of storage, wherein when one or more programs are executed by one or more processors respectively, so that one A or multiple processors execute the data traversal method and disclosure second aspect for the calculate node that disclosure first aspect provides The data traversal method of the calculate node of offer.

6th aspect of the disclosure provides a kind of computer-readable medium, is stored thereon with executable instruction, the instruction Processor is set to execute the data traversal method and the disclosure of the calculate node that disclosure first aspect provides when being executed by processor The data traversal method for the calculate node that second aspect provides.

In accordance with an embodiment of the present disclosure, data present in the ergodic process of data can be at least avoided to need by secondary Transmission and can only the defect that reads of single thread sequence, and data traversal speed therefore can be improved, reduces the time-consuming of data traversal.

Detailed description of the invention

By referring to the drawings to the description of the embodiment of the present disclosure, the above-mentioned and other purposes of the disclosure, feature and Advantage will be apparent from, in the accompanying drawings:

Fig. 1 diagrammatically illustrates the data traversal method, apparatus according to the calculate node of the embodiment of the present disclosure and calculates section The system architecture of point；

Fig. 2A diagrammatically illustrates the method flow diagram of the data traversal method according to the calculate node of the embodiment of the present disclosure；

Fig. 2 B diagrammatically illustrates the method stream of the data traversal method of the calculate node according to another embodiment of the disclosure Cheng Tu；

Fig. 2 C diagrammatically illustrates reading-traversing operation flow chart according to the embodiment of the present disclosure；

Fig. 2 D diagrammatically illustrates the operational flowchart with reference to Fig. 2 B operation S230 described；

Fig. 3 diagrammatically illustrates reading-traversing operation flow chart according to another embodiment of the disclosure；

Fig. 4 diagrammatically illustrates the method flow of the data traversal method of the calculate node according to another embodiment of the disclosure Figure；

Fig. 5 diagrammatically illustrates the method flow of the data traversal method of the calculate node according to another embodiment of the disclosure Figure；

Fig. 6 A~Fig. 6 B diagrammatically illustrates reading-traversing operation flow chart according to another embodiment of the disclosure；

Fig. 7 A~Fig. 7 B diagrammatically illustrates the structure of the data traversal device according to the calculate node of the embodiment of the present disclosure Block diagram；

Fig. 8 A~Fig. 8 D diagrammatically illustrates the structural block diagram of the first processing module according to the embodiment of the present disclosure；

Fig. 9 diagrammatically illustrates the structural block diagram of the write operation module according to the embodiment of the present disclosure；

Figure 10 A~Figure 10 B diagrammatically illustrates the structural block diagram of the Second processing module according to the embodiment of the present disclosure；

Figure 11 diagrammatically illustrates the block diagram for being adapted for carrying out method as described above according to the embodiment of the present disclosure.

Specific embodiment

Hereinafter, will be described with reference to the accompanying drawings embodiment of the disclosure.However, it should be understood that these descriptions are only exemplary , and it is not intended to limit the scope of the present disclosure.In the following detailed description, to elaborate many specific thin convenient for explaining Section is to provide the comprehensive understanding to the embodiment of the present disclosure.It may be evident, however, that one or more embodiments are not having these specific thin It can also be carried out in the case where section.In addition, in the following description, descriptions of well-known structures and technologies are omitted, to avoid Unnecessarily obscure the concept of the disclosure.

Term as used herein is not intended to limit the disclosure just for the sake of description specific embodiment.It uses herein The terms "include", "comprise" etc. show the presence of the feature, step, operation and/or component, but it is not excluded that in the presence of Or add other one or more features, step, operation or component.

There are all terms (including technical and scientific term) as used herein those skilled in the art to be generally understood Meaning, unless otherwise defined.It should be noted that term used herein should be interpreted that with consistent with the context of this specification Meaning, without that should be explained with idealization or excessively mechanical mode.

It, in general should be according to this using statement as " at least one in A, B and C etc. " is similar to Field technical staff is generally understood the meaning of the statement to make an explanation (for example, " system at least one in A, B and C " Should include but is not limited to individually with A, individually with B, individually with C, with A and B, with A and C, have B and C, and/or System etc. with A, B, C).Using statement as " at least one in A, B or C etc. " is similar to, generally come Saying be generally understood the meaning of the statement according to those skilled in the art to make an explanation (for example, " having in A, B or C at least One system " should include but is not limited to individually with A, individually with B, individually with C, with A and B, have A and C, have B and C, and/or the system with A, B, C etc.).It should also be understood by those skilled in the art that substantially arbitrarily indicating two or more The adversative conjunction and/or phrase of optional project shall be construed as either in specification, claims or attached drawing A possibility that giving including one of these projects, either one or two projects of these projects.For example, phrase " A or B " should A possibility that being understood to include " A " or " B " or " A and B ".

Embodiment of the disclosure provides a kind of data traversal method of calculate node, and the data are stored in multiple data In fragment, this method comprises: multiple burst informations are obtained, the corresponding data fragmentation of each burst information；And at least One other calculate node is concurrently based on multiple burst informations and executes reading-traversing operation, reads multiple fragment letters with traversal Cease the data of corresponding data fragmentation storage, wherein different calculate node traversals reads the number of different data fragmentation storages According to.

Embodiment of the disclosure additionally provides the data traversal method of another calculate node, and the data are stored in multiple In data fragmentation, this method comprises: being concurrently based on multiple burst informations at least one other calculate node executes reading-time Operation is gone through, to traverse the data for reading the corresponding data fragmentation storage of multiple burst informations, wherein multiple burst informations are by least A calculate node in one other calculate node obtains in advance, the corresponding data fragmentation of each burst information, different Calculate node traversal read the data of different data fragmentation storages.

Fig. 1 diagrammatically illustrates the data traversal method, apparatus according to the calculate node of the embodiment of the present disclosure and calculates section The system architecture 100 of point.It should be noted that be only the example that can apply the system architecture of the embodiment of the present disclosure shown in Fig. 1, To help skilled in the art to understand the technology contents of the disclosure, but it is not meant to that the embodiment of the present disclosure may not be usable for it His equipment, system, environment or scene.

As shown in Figure 1, according to this embodiment may include storage cluster and computing cluster, storage in system architecture 100 Cluster includes memory node 101,102,103,104, and computing cluster includes calculate node 105,106,107.Storage cluster and meter It calculates by network connection between cluster, network may include various connection types, such as wired, wireless communication link or optical fiber Cable etc., and composition distribution or parallel computing cluster are mutually connected to each other between each calculate node.

Calculate node 105,106,107 can be interacted by network with memory node 101,102,103,104, to storage Node carries out the operation such as write-in, reading or deletion of data.

The data stored in storage cluster for example can be used data fragmentation storage strategy and be stored, and data fragmentation stores plan Slightly refer to data according to specific slicing algorithm, be divided into multiple data fragmentations, and these data fragmentations are respectively stored into not It is stored on same memory node.So that memory space is not limited by storage capacity of single node, stored by increase and decrease The Number dynamics of node adjust memory space, and support concurrent reading and writing data.

Traditionally, the reading of data is carried out by the way that ETL server is arranged between storage cluster and computing cluster, specifically Are as follows: calling data-storage system traversal method is read in the data to ETL server memory or disk in storage cluster first, Again by each calculate node of data distribution to computing cluster, the reading of data is completed.In the process, since storage cluster provides Default traversal method only support single thread sequence to read, therefore the concurrent advantage of multinode that cluster can not be played；Furthermore data Need first to read ETL server, then be distributed to each node of computing cluster, thus data needs carried out on network it is secondary Transmission, reduces data read rates；And the reading rate of entire data read process is limited to the network bandwidth of ETL server With I O process ability, and the processing speed of ETL is slower, and if in data traversal reading process, ETL server fail, It then will lead to data traversal failure.

In view of problem above, as shown in Figure 1, embodiment of the disclosure passes through preset election regulation in computing cluster The calculate node 105 that election has obtained coordinated management effect obtains the burst information of each data fragmentation in storage cluster, and passes through Distributed coordination service architecture and other calculate nodes realize information sharing, then each calculate node 105 in computing cluster, 106, 107 by reading shared burst information, can be directly to data fragmentation corresponding with the burst information of reading in storage cluster The data of storage carry out traversal reading.And different burst informations can be read in multiple calculate nodes, concurrently to corresponding data The data of fragment storage carry out traversal reading, therefore can effectively improve data traversal speed, and theoretically traversal speed is to pass " MIN (data fragmentation quantity, the calculate node quantity) " times of system traversal method, for example, there are 10 data fragmentations at one In the computing cluster environment of storage cluster and 10 nodes, data traversal time-consuming is 1/10th of conventional method.

It should be noted that the data traversal method of calculate node provided by the embodiment of the present disclosure generally can be by calculating Node 105,106,107 executes.Correspondingly, the data traversal device of calculate node provided by the embodiment of the present disclosure generally can be with It is set in calculate node 105,106,107.

It should be understood that the number of memory node and calculate node in Fig. 1 and the type of various connection relationships are only Schematically.According to needs are realized, any number of memory node and calculate node and any type of connection can have Relationship.

Fig. 2A diagrammatically illustrates the method flow diagram of the data traversal method according to the calculate node of the embodiment of the present disclosure.

As shown in Figure 2 A, an aspect of this disclosure provides a kind of data traversal method of calculate node, this method packet Include operation S210~S220.

In operation S210, multiple burst informations are obtained, the corresponding data fragmentation of each burst information.

Above-mentioned data are stored in multiple data fragmentations, and each data fragmentation is corresponding with a burst information, with unique Corresponding data fragment.

In accordance with an embodiment of the present disclosure, in accordance with an embodiment of the present disclosure, which belongs to one in computing cluster Calculate node, such as can be the calculate node 105 described with reference to Fig. 1.It further include in addition to the calculate node in computing cluster At least one other calculate node, such as the calculate node 106,107 described with reference to Fig. 1.Obtain the calculating of multiple burst informations Node for example can be elected to obtain by preset election regulation by all calculate nodes in computing cluster, for convenience of describing, with Calling the calculate node that the election obtains in the following text is management node.Wherein, the embodiment of the present disclosure does not carry out the preset election regulation It limits, FastLeaderElection algorithm etc. for example can be used in preset election regulation.

In accordance with an embodiment of the present disclosure, computing cluster initialize for the first time or before the management node elected occur Failure, any calculate node in computing cluster find that management node is not present or not when attempting to communicate with management node When available, triggering election, so that election obtains new management node.

In accordance with an embodiment of the present disclosure, any one calculate node in computing cluster all can provide interface and mention for client Traversal task is handed over, ergodic condition is carried in the traversal task, at least one other calculate node is receiving client submission Traversal task when, by the communication with management node, according to the traversal task, initiate traversal request to management node.Then should Management node is when obtaining burst information, based on traversal request (including ergodic condition), from multiple numbers of storage cluster storage Burst information is obtained according to fragment.

In accordance with an embodiment of the present disclosure, the traversal task of client submission is received in the interface that multiple calculate nodes provide When, which can create multiple tasks node after receiving multiple traversal tasks in distributed coordination service architecture, Each traversal task is corresponding with a task node, then the ergodic condition that the management node is carried according to multiple traversal tasks is certainly more A data fragmentation reads burst information, and by the burst information of return according to the matching with the task of traversal, respectively will with it is each The matched multiple burst informations of traversal task are sequentially written in the child node of each task node, then multiple task node can be with It is arranged in task node catalogue, each task node includes multiple child nodes again, and each child node is written with a burst information. The embodiment of the present disclosure is not defined distributed coordination service architecture, for example, the distributed coordination service architecture can be Zookeeper (abbreviation zk) distributed coordination service etc., to realize the information sharing in computing cluster between multiple calculate nodes.

In operation S220, the multiple burst information is concurrently based on at least one other calculate node and executes reading- Traversing operation.

In accordance with an embodiment of the present disclosure, reading-traversing operation is specifically as follows: reading one point in multiple burst informations Piece information, then traversal reads the data of the corresponding data fragmentation storage of burst information.

Wherein, at least one other calculate node is are as follows: in addition to the calculating section of acquisition burst information in computing cluster Other calculate nodes outside point.Since the calculate node and at least one other calculate node can be based on the fragment letters obtained Breath is performed in parallel reading-traversing operation, to traverse the data for reading the corresponding data fragmentation storage of multiple burst informations, because This, the scheme of the embodiment of the present disclosure can be realized the concurrently traversal of data, greatly improve the speed of data traversal.Furthermore by Burst information is based in calculate node and traverses reading data directly from storage cluster, avoids data by ETL server Secondary transfer, therefore can be further improved the speed of data traversal.

Fig. 2 B diagrammatically illustrates the method stream of the data traversal method of the calculate node according to another embodiment of the disclosure Cheng Tu.

As shown in Figure 2 B, according to the embodiment of the present disclosure, the data traversal method of the calculate node of Fig. 2A description is being operated Between S210~operation S220, it may also include S230.

In operation S230, preset memory space is written into the multiple burst information.

In accordance with an embodiment of the present disclosure, the calculate node for obtaining multiple burst informations multiple burst informations can be written in advance If memory space, the preset memory space for example can be the communal space, for each calculate node in computing cluster Access.

Specifically, which can be for example the memory space that distributed coordination service architecture provides, from And realize the information sharing in computing cluster between each calculate node, then each calculate node in computing cluster can certainly should Burst information is read in memory space, is read with the traversal that the burst information based on reading carries out data in corresponding data fragment. For example, burst information is written in the child node in the task node in Zookeeper by management node, then to single traversal When task is traversed, each calculate node in computing cluster can the child node of self-corresponding task node read fragment letter Breath.

Fig. 2 C diagrammatically illustrates reading-traversing operation flow chart according to the embodiment of the present disclosure.

As shown in Figure 2 C, the case where preset memory space is written in burst information by the calculate node for obtaining burst information Under, reading-traversing operation correspondingly includes operation S221~S222.

In operation S221, a burst information in multiple burst informations is read from preset memory space；It is operating S222, the burst information based on reading, traversal read the data of the corresponding data fragmentation storage of burst information.

In accordance with an embodiment of the present disclosure, of calculate node can be used by being less than in computing cluster in the quantity of multiple burst informations In the case where number (node total number of management node and at least one other calculate node), multiple calculate nodes in computing cluster (equal to the quantity of multiple burst informations) can be completed via primary above-mentioned reading-traversing operation to multiple burst informations pair The traversal of the data for the data fragmentation storage answered is read.Certainly, the embodiment of the present disclosure does not limit specific traversal reading process, Such as it can also be grasped by the part calculate node (less than the quantity of burst information) in multiple calculate nodes via multiple reading-traversal The circulation of work, so that multiple burst informations are read, and the data of corresponding data fragmentation storage are traversed reading.

In accordance with an embodiment of the present disclosure, of calculate node can be used by being greater than in computing cluster in the quantity of multiple burst informations In the case where number (node total number of management node and at least one other calculate node), the management node and at least one other Calculate node then recycles execution reading-traversing operation.Specifically, in wherein calculate node task section from Zookeeper Number after the child node of point has read the burst information of child node write-in, i.e., to the storage of the burst information corresponding data fragmentation According to carrying out traversal reading；After completing traversal and reading, which is then returned from other sons of task node in Zookeeper Node (other child nodes not being read) reads the burst information of other child nodes write-in, and to corresponding data fragmentation The data of storage carry out traversal reading, and also circulation executes reading-traversing operation to other calculate nodes in this way, until The burst information of all child nodes write-in of task node was read in Zookeeper, to guarantee multiple burst informations pair The data for the data fragmentation storage answered are traversed reading.

In accordance with an embodiment of the present disclosure, if there are when multiple tasks node in Zookeeper, available meter in computing cluster Operator node then successively traverses the traversal task of multiple tasks node, is written in the child node to a task node The data of burst information corresponding data fragmentation storage complete after traversal reads, then read the child node of next task node and write The burst information entered is traversed with the traversal task to next task node.

Fig. 2 D diagrammatically illustrates the operational flowchart with reference to Fig. 2 B operation S230 described.

In accordance with an embodiment of the present disclosure, multiple data fragmentations are storable on multiple memory nodes of storage cluster, and every A memory node is stored with one or more data fragmentations.

Then, it can also be operation S210 ' with reference to the operation S210 in Fig. 2A, obtain multiple burst informations and multiple data point The memory node information of piece, the memory node information can indicate which the corresponding data fragmentation of certain burst information is stored in and deposits Store up node.

As shown in Figure 2 D, operation S230 may include operation S231 and operation S232.

In operation S231, it is based on preset allocation rule and memory node information, multiple burst informations are evenly distributed, with It is alternatively arranged the burst information of the different data fragment with identical memory node information；It, will be evenly distributed in operation S232 Multiple burst informations be sequentially written in preset memory space.

Specifically, the principle of the preset allocation rule are as follows: make the fragment of the data fragmentation in same memory node Information is separated by setting in arrangement queue, preferably makes separated by a distance farthest as far as possible.The preset allocation rule for example can be Strategy is simply evenly distributed below: burst information is organized into one two according to memory node where its corresponding data fragmentation The burst information of dimension group, the data fragmentation with identical memory node information is located at same row, has different memory nodes letter The burst information of the data fragmentation of breath is located at different rows.Such as multiple burst informations can be arranged as to a burst information list, It, can be since the first row first row of two-dimensional array, successively from two dimension then when carrying out evenly distributed to multiple burst informations Every a line of first row takes a burst information to be placed on the end of list in array；The burst information of first row in two-dimensional array After having arranged, then since the burst information of the first row secondary series, a burst information successively is taken from every a line of secondary series It is placed on the end of list, and so on, it is up to arranging all burst informations in 2-D data to burst information list Only.

For there are in the case where multiple traversal tasks, then being carried out uniformly to multiple burst informations of each traversal task Arrangement, so that the burst information of the different data fragment in each traversal task with identical memory node information is alternatively arranged.

Then in operation S232, the multiple burst informations arranged in burst information list are sequentially written in preset deposit in order Space is stored up, for Zookeeper distributed coordination service architecture, then by multiple burst informations according to suitable in burst information list Sequence is sequentially written in the child node under task node.And for there are in the case where multiple tasks node, then burst information being arranged Burst information in table is written according to corresponding relationship in the child node of corresponding task node, in different burst information lists Burst information is written in the child node of different task nodes.

In reading-traversing operation, a burst information in multiple burst informations is read from preset memory space, it can Think and sequentially reads a burst information from evenly distributed multiple burst informations.Specifically, it is as pressed from task node Putting in order for child node is successively read burst information.

The embodiment of the present disclosure can make the multiple data fragmentations for being stored in same memory node corresponding by above-mentioned setting Position of the burst information in task node it is farthest as far as possible, then, can when reading the burst information in preset memory space The different burst informations for avoiding multiple calculate nodes from reading are the burst information of the different data fragment of same memory node storage The case where, therefore, it can avoid multiple calculating when the data stored to the corresponding data fragmentation of burst information carry out traversal reading The case where node accesses to same memory node is read so as to avoid traversing caused by single memory node overload The lower defect of speed, to improve the efficiency of data traversal as far as possible.

In order to guarantee the stability of data storage, some calculate node is avoided in storage cluster to break down caused part Data fragmentation can be saved more parts, then when obtaining burst information, unavoidably by loss of data on different memory nodes Meeting there are identical burst informations.

Therefore, when carrying out evenly distributed to multiple burst informations of acquisition, duplicate checking can be carried out, to guarantee to deposit with difference Multiple identical burst informations of storage nodal information only save one after evenly distributed.Multiple burst informations can be also arranged as One burst information set, due to gathering not reproducible characteristic, first be only arranged in multiple same slice information Fragment burst information can be put into burst information set.

By the above-mentioned means, can guarantee the stability of storing data in storage cluster, also can avoid in subsequent traversal In reading process, the repetition for the identical data for being stored in different memory nodes is traversed and is read, so as to avoid unnecessary volume Overhead improves the efficiency of computing cluster ergodic data.

Fig. 3 diagrammatically illustrates reading-traversing operation flow chart according to another embodiment of the disclosure.

As shown in figure 3, reading-the traversing operation other than operating S221~S222, further includes operation S223 and operation S224.Wherein, operation S223 is executed between operation S221 and operation S222, and operation S224 is held after having executed operation S223 Row.

In operation S223, the burst information of reading is set to the second state by first state；Operation S224, time It goes through in the case where having read, the burst information is set to the third state by the second state.

In accordance with an embodiment of the present disclosure, after the calculate node obtains multiple burst informations, by multiple burst information When preset memory space is written, multiple burst information is set to original state i.e. first state, to indicate multiple fragment Information can be read by any calculate node (calculate node or at least one other calculate node) in computing cluster；Fragment letter Second state of breath then indicates that the burst information is read by any one calculate node in computing cluster, can not be counted again The calculate node calculated in cluster is read；The third state of burst information indicates that the corresponding data fragmentation of the burst information stores Data, which have been traversed, have been read, and the burst information can not be read by the calculate node in computing cluster again.

By the above-mentioned setting to burst information state, the corresponding number of any burst information in multiple burst informations can avoid Traversal is repeated by computing cluster according to the data of fragment storage to read, and can avoid the case where part burst information is not read, from And under the premise of guaranteeing to read data integrity, the time-consuming of data traversal can be reduced as far as possible.

Fig. 4 diagrammatically illustrates the method flow of the data traversal method of the calculate node according to another embodiment of the disclosure Figure.

As shown in figure 4, this method includes operation S410~S440.

In operation S410, the state of multiple burst informations of preset memory space storage is obtained；In operation S420, judgement Whether multiple burst informations are in the third state；In the case where there is the burst information for being not at the third state, behaviour is executed Make S430, returns to execution operation S410 after being spaced preset time, judgement is re-started with the state to multiple burst informations；Institute It states in the case that multiple burst informations are in the third state, executes operation S440, complete the traversal of the data.

In accordance with an embodiment of the present disclosure, it is an operating procedure that aforesaid operations S410 and operation S420 is also combinable, directly Judge whether multiple burst informations are in the third state, without obtaining the state of multiple burst informations in advance, but to pre- If memory space storage multiple burst informations judge one by one in order, if judgement discovery be not at the third state fragment letter Breath then interrupts judgement, after being spaced preset time, then since at the burst information that the discovery is not at the third state, rejudges With the presence or absence of the burst information for being not at the third state, until not finding to be not at the burst information of the third state, explanation Multiple burst informations are in the third state, then terminate judgement, complete the traversal of data.

Pass through aforesaid operations, it can be ensured that the data of the corresponding data fragmentation storage of multiple burst informations are traversed reading It takes, avoids the omission to partial data, ensure that the integrality of reading data.

Fig. 5 diagrammatically illustrates the method flow of the data traversal method of the calculate node according to another embodiment of the disclosure Figure.

As shown in figure 5, this method other than operating S210~S220, further includes operation S510 and operation S520.

In operation S510, judge to have read a burst information in multiple burst informations from the preset memory space Other calculate nodes whether be malfunctioning node；In the case where other malfunctioning nodes are malfunctioning node, operation S520 is executed, it will The burst information that the malfunctioning node is read is set to first state by the second state.

In accordance with an embodiment of the present disclosure, it when calculate node reads burst information from preset memory space, such as can incite somebody to action At the storage location for the burst information that the reading is written in its identification information, and calculate node is carried out with preset memory space When initial interaction, a transient node, the identification of the transient node and the calculate node can be created in preset memory space Information matches；When the calculate node and preset memory space interact end or calculate node failure cause cannot When interaction, which will disappear.Therefore, when judging whether read the calculate node of burst information is malfunctioning node, It can first check whether the burst information is in the second state, if so, judging that preset memory space whether there is and the fragment again The transient node that the identification information for the calculate node being written at the storage location of information matches, and if it exists, then illustrate the meter Operator node works normally, and if it does not exist, then illustrates that the calculate node does not traverse and has read the corresponding data point of the burst information The data of piece, and the calculate node is malfunctioning node.

In the case where other calculate nodes are malfunctioning node, it is responsible for by management node by other calculate nodes reading Burst information is set to first state by the second state, so that the management node or other normal calculate nodes are read.

In accordance with an embodiment of the present disclosure, it is monitored by the judgement of above-mentioned malfunctioning node, can effectively avoid malfunctioning node time Go through data failure, and other nodes be because that can not obtain corresponding burst information, thus this partial data can not be traversed Defect greatly improves the high availability of computing cluster, further ensures the integrality of data traversal reading.

Another aspect of the present disclosure additionally provides a kind of data traversal method of calculate node.

In accordance with an embodiment of the present disclosure, the data are stored in multiple data fragmentations, the data traversal method include with Lower operation: multiple burst informations are concurrently based on at least one other calculate node and execute reading-traversing operation, are read with traversing Take the data of the corresponding data fragmentation storage of multiple burst informations.

In accordance with an embodiment of the present disclosure, the calculate node is the calculate node in computing cluster, which can be with For with reference to the calculate node 106,107 in Fig. 1；It can also be at least one other calculate node described with reference to Fig. 2A~Fig. 2 B In calculate node.

In accordance with an embodiment of the present disclosure, multiple burst informations are by other calculate nodes of at least one of computing cluster A calculate node obtain in advance, which for example can be for reference to the meter in Fig. 1 Operator node 105, can also for reference to Fig. 2A~with reference to Fig. 5 describe calculate node data traversal method in calculate node.

In accordance with an embodiment of the present disclosure, the calculate node at least one other calculate node for obtaining burst information is It is elected by computing cluster (calculate node and at least one other calculate node) using preset election regulation.The disclosure Embodiment is not defined the preset election regulation, and FastLeaderElection for example can be used in preset election regulation Algorithm etc..

In accordance with an embodiment of the present disclosure, computing cluster initialize for the first time or before the management node elected occur Failure, any calculate node in computing cluster find that management node is not present or not when attempting to communicate with management node When available, triggering election, so that election obtains new management node.

In accordance with an embodiment of the present disclosure, the corresponding data fragmentation of each burst information.And the calculate node with When at least one other calculate node traversal reads data, the data fragmentation that is different that different calculate node traversals is read is deposited The data of storage.

The data traversal method of the calculate node of the embodiment of the present disclosure, due to at least one other calculate node concurrently Reading-traversing operation is executed, therefore the speed of data traversal can greatly be improved；Furthermore due to the calculate node according to point Piece information can the data of directly self-corresponding data fragmentation storage carry out traversal reading, therefore avoid data by ETL service The secondary transfer of device, can be further improved the speed of data traversal.

In accordance with an embodiment of the present disclosure, multiple burst informations are by the calculating section at least one other calculate node Point is written in preset memory space after obtaining in advance, therefore can save for the calculate node and at least one other calculating Point is directly read, and is read out-traversing operation with the burst information according to reading.

Fig. 6 A~Fig. 6 B diagrammatically illustrates reading-traversing operation flow chart according to another embodiment of the disclosure.

In accordance with an embodiment of the present disclosure, as shown in Figure 6A, above-mentioned reading-traversing operation includes operation S611~operation S612.In operation S611, a burst information in multiple burst informations is read from preset memory space；S612 is being operated, Burst information based on reading, traversal read the data of the corresponding data fragmentation storage of burst information.According to the implementation of the disclosure Example, the reading-traversing operation are similar to reading-traversing operation with reference to Fig. 2 C description, and details are not described herein.

In accordance with an embodiment of the present disclosure, as shown in Figure 6B, above-mentioned reading-traversing operation for example may also include operation S613 and Operate S614.In the case where having read a burst information in multiple burst informations from preset memory space, behaviour is executed Make S613, the burst information of reading is set to the second state by first state；And to have read burst information corresponding in traversal In the case where the data of data fragmentation storage, operation S614 is executed, burst information is set to the third state by the second state.Its In, burst information is when being written preset memory space by a calculate node at least one other calculate node, i.e. quilt It is set to first state, to indicate that burst information can be read by calculate node or at least one other calculate node；Burst information The second state indicate that the burst information has been read, can not be read again；The third state of burst information indicates the fragment The data of the corresponding data fragmentation storage of information, which have been traversed, have been read, which can not be read again.According to this public affairs The embodiment opened, the reading-traversing operation are similar to reading-traversing operation with reference to Fig. 3 description, and details are not described herein.

In accordance with an embodiment of the present disclosure, of calculate node can be used by being less than in computing cluster in the quantity of multiple burst informations In the case where number (node total number of calculate node and at least one other calculate node), multiple calculate nodes in computing cluster (equal to the quantity of multiple burst informations) can be completed via primary above-mentioned reading-traversing operation to multiple burst informations pair The traversal of the data for the data fragmentation storage answered is read.Certainly, the embodiment of the present disclosure does not limit specific traversal reading process, Such as it can also be grasped by the part calculate node (less than the quantity of burst information) in multiple calculate nodes via multiple reading-traversal The circulation of work, so that multiple burst informations are read, and the data of corresponding data fragmentation storage are traversed reading.

In accordance with an embodiment of the present disclosure, of calculate node can be used by being greater than in computing cluster in the quantity of multiple burst informations In the case where number (node total number of calculate node and at least one other calculate node), the calculate node and at least one other Calculate node then recycles execution reading-traversing operation.Specifically, in wherein calculate node task section from Zookeeper Number after the child node of point has read the burst information of child node write-in, i.e., to the storage of the burst information corresponding data fragmentation According to carrying out traversal reading；After completing traversal and reading, which is then returned from other sons of task node in Zookeeper Node (other child nodes not being read) reads the burst information of other child nodes write-in, and to corresponding data fragmentation The data of storage carry out traversal reading, and also circulation executes reading-traversing operation to other calculate nodes in this way, until The burst information of all child nodes write-in of task node was read in Zookeeper, so that multiple burst informations are corresponding Data fragmentation storage data be traversed reading.

Fig. 7 A~Fig. 7 B diagrammatically illustrates the structure of the data traversal device according to the calculate node of the embodiment of the present disclosure Block diagram.

As shown in Figure 7 A, another aspect of the present disclosure proposes a kind of data traversal device of calculate node, the device 700 packet It includes and obtains module 710 and first processing module 720.

Wherein, the data are stored in multiple data fragmentations.

Wherein, module 710 is obtained for obtaining multiple burst informations, each burst information corresponds to a data fragmentation. First processing module 720 is used to execute reading-traversing operation based on multiple burst informations.

Wherein, the data traversal device of the calculate node can be with the data traversal device of at least one other calculate node Reading-traversing operation concurrently is executed based on multiple burst informations, the corresponding data fragmentation of multiple burst informations is read with traversal The data of storage.And the data traversal device traversal of different calculate nodes reads the data of different data fragmentation storages.It can With understanding, the data traversal device of the calculate node be can be set in the calculate node.

In accordance with an embodiment of the present disclosure, the acquisition module 710 and first processing module 720 are respectively used to execute with reference to Fig. 2A The operation S210 and operation S220 of description, details are not described herein.

In accordance with an embodiment of the present disclosure, as shown in Figure 7 B, the data traversal device 700 of the calculate node, which may also include, writes behaviour Make module 730.

Wherein, preset storage is written in multiple burst informations which is used to will acquire the acquisition of module 710 Space.In accordance with an embodiment of the present disclosure, which for example can be used for executing the operation S230 described with reference to Fig. 2 B, Details are not described herein.

Fig. 8 A~Fig. 8 D diagrammatically illustrates the structural block diagram of the first processing module according to the embodiment of the present disclosure.

As shown in Figure 8 A, first processing module 720 may include the first burst information reading unit 721 and the first data time Go through reading unit 722.

First burst information reading unit 721 is used to execute the read operation in reading-traversing operation, i.e., deposits from preset Read a burst information in multiple burst informations in storage space；First data traversal reading unit 722 is for executing reading-time The traversing operation in operation, the i.e. burst information based on reading are gone through, traversal reads the corresponding data fragmentation of burst information and deposits The data of storage.In accordance with an embodiment of the present disclosure, the first burst information reading unit 721 and the first data traversal reading unit 722 can be used for executing respectively with reference to Fig. 2 C operation S221 described and operation S222, and details are not described herein.

In accordance with an embodiment of the present disclosure, as shown in Figure 8 B, first processing module 720 can also include the first changing unit 723。

First changing unit 723 in the first burst information reading unit 721 from preset memory space for having read In the case where a burst information in multiple burst informations, the burst information of reading is set to the second state by first state； And the case where the first data traversal reading unit 722 traversal has read the data of the corresponding data fragmentation storage of burst information Under, burst information is set to the third state by the second state.Wherein, burst information be written by write operation module 730 it is presetting When memory space, it is set to first state, to indicate that the burst information can by the data traversal device of calculate node or at least The data traversal device of one other calculate node is read；Second state of burst information indicates that the burst information has been read, It can not be read again；The third state of burst information indicate the data of the burst information corresponding data fragmentation storage by time It goes through and has read, which can not be read again.In accordance with an embodiment of the present disclosure, which can be used for The operation S223 and operation S224 with reference to Fig. 3 description are executed, details are not described herein.

In accordance with an embodiment of the present disclosure, as shown in Figure 8 C, which for example may also include breakdown judge list Member 724.

Breakdown judge unit 724 is for judging to have read one in multiple burst informations from the preset memory space Whether the data traversal device of other calculate nodes of burst information is failed equipment.In the data traversal of other calculate nodes In the case that device is failed equipment point, burst information that above-mentioned first changing unit 723 is also used to read the failed equipment First state is set to by the second state, the data traversal device for calculate node or the data traversal device except the malfunctioning node The data traversal device of outer at least one other calculate node is read.In accordance with an embodiment of the present disclosure, the breakdown judge unit 724 for example can be used for executing the operation S510 described with reference to Fig. 5, and the first changing unit 723 for example can also be used to execute with reference to Fig. 5 The operation S520 of description, details are not described herein.

In accordance with an embodiment of the present disclosure, as in fig. 8d, which it is single for example to may also include state judgement Member 725.

The state judging unit 725 exists not for judging whether multiple burst informations are in the third state In the case where burst information in the third state, returns and rejudge after the preset time of interval, and in multiple burst informations In the case where being in the third state, the traversal of data is completed.In accordance with an embodiment of the present disclosure, state judging unit 725 Operation S410~operation S440 as described in can be used for executing with reference to Fig. 4, details are not described herein.

In accordance with an embodiment of the present disclosure, above-mentioned multiple data fragmentations are stored on multiple memory nodes of storage cluster, Each memory node is stored with one or more data fragmentations.

In accordance with an embodiment of the present disclosure, the memory node information that module 710 is also used to obtain multiple data fragmentations is obtained.Example Such as, which can also be used to execute the operation S210 ' with reference to Fig. 2 B description, and details are not described herein.

Fig. 9 diagrammatically illustrates the structural block diagram of the write operation module according to the embodiment of the present disclosure.

As shown in figure 9, write operation module 730 may include burst information arrangement units 731 and write operation unit 732.

Burst information arrangement units 731 are used to be based on preset allocation rule and memory node information, and multiple fragments are believed Cease it is evenly distributed so that the burst information of different data fragment with identical memory node information is alternatively arranged；Write operation list Member 732 is for being sequentially written in preset memory space for evenly distributed multiple burst informations.Then the first burst information reads single Member 721 sequentially reads a burst information from evenly distributed multiple burst informations.In accordance with an embodiment of the present disclosure, the fragment Information arrangement unit 731 and write operation unit 732 are respectively used to execute the operation S231 and operation S232, In with reference to Fig. 2 D description This is repeated no more.

In accordance with an embodiment of the present disclosure, the data traversal device of calculate node may be disposed in the calculate node, the meter The data traversal device (hereinafter referred to as management node) of operator node is by the data traversal of each calculate node in computing cluster Device (the data traversal device of the calculate node and the data traversal device of at least one other calculate node) is using default Election regulation elect to obtain.The embodiment of the present disclosure is not defined the preset election regulation, preset election regulation example FastLeaderElection algorithm such as can be used.

In accordance with an embodiment of the present disclosure, computing cluster initialize for the first time or before the number of management node elected When breaking down according to traversal device, the data traversal device of any calculate node in computing cluster is being attempted to save with management When discovery management node is not present or is unavailable when point communication, triggering election, so that election obtains new management node.

In accordance with an embodiment of the present disclosure, calculate node can be used by being less than in computing cluster in the quantity of multiple burst informations Data traversal number of devices (the data traversal device of the calculate node and the data traversal dress of at least one other calculate node The device sum set) in the case where, the data traversal device of multiple calculate nodes is (equal to multiple burst informations in computing cluster Quantity) via primary above-mentioned reading-traversing operation, the number to the storage of multiple burst informations corresponding data fragmentation can be completed According to traversal read.Certainly, the embodiment of the present disclosure does not limit specific traversal reading process, such as can also be saved by multiple calculating Data traversal device (less than the quantity of burst information) following via multiple reading-traversing operation of part calculate node in point Ring, so that multiple burst informations are read, and the data of corresponding data fragmentation storage are traversed reading.

In accordance with an embodiment of the present disclosure, the number of calculate node can be used by being greater than in computing cluster in the quantity of multiple burst informations According to number (the data traversal device of the calculate node and the data traversal of at least one other calculate node of traversal device The device sum of device) in the case where, the data traversal device of the calculate node and the data of at least one other calculate node Traversal device then recycles execution reading-traversing operation.Specifically, the data traversal device of a wherein calculate node from It is after the child node of task node has read the burst information of child node write-in in Zookeeper, i.e., corresponding to the burst information Data fragmentation storage data carry out traversal reading；After completing traversal and reading, the data traversal device of the calculate node is then Other child nodes (other child nodes not being read) for being returned from task node in Zookeeper read other child nodes The burst information of write-in, and traversal reading, the data time of other calculate nodes are carried out to the data of corresponding data fragmentation storage Also circulation executes reading-traversing operation to calendar device in this way, until all child nodes of task node are written in Zookeeper Burst information be read so that the data of multiple burst informations corresponding data fragmentation storage are traversed reading.

Acquisition module 710 according to an embodiment of the present disclosure, first processing module 720,730, first points of write operation module Piece Information reading unit 721, the first data traversal reading unit 722, the first changing unit 723, breakdown judge unit 724, shape It is any number of or in which any number of in state judging unit 725, burst information arrangement units 731 and write operation unit 732 At least partly function can be realized in a module.According in the module of the embodiment of the present disclosure, submodule, unit, subelement Any one or more can be split into multiple modules to realize.According to the module of the embodiment of the present disclosure, submodule, list Member, any one or more in subelement can at least be implemented partly as hardware circuit, such as field-programmable gate array Arrange (FPGA), programmable logic array (PLA), system on chip, the system on substrate, the system in encapsulation, specific integrated circuit (ASIC), it or can be realized by carrying out hardware or the firmware of any other rational method that is integrated or encapsulating to circuit, Or it several appropriately combined is realized with any one in three kinds of software, hardware and firmware implementations or with wherein any. Alternatively, can be at least by partly according to one or more of the module of the embodiment of the present disclosure, submodule, unit, subelement It is embodied as computer program module, when the computer program module is run, corresponding function can be executed.

For example, obtaining module 710, first processing module 720, write operation module 730, the first burst information reading unit 721, the first data traversal reading unit 722, the first changing unit 723, breakdown judge unit 724, state judging unit 725, Any number of may be incorporated in a module in burst information arrangement units 731 and write operation unit 732 realize, Huo Zheqi In any one module can be split into multiple modules.Alternatively, at least portion of one or more modules in these modules Point function can be combined at least partly function of other modules, and be realized in a module.According to the implementation of the disclosure Example obtains module 710, first processing module 720, write operation module 730, the first burst information reading unit 721, the first data Traverse reading unit 722, the first changing unit 723, breakdown judge unit 724, state judging unit 725, burst information arrangement At least one of unit 731 and write operation unit 732 can at least be implemented partly as hardware circuit, such as scene can compile Journey gate array (FPGA), programmable logic array (PLA), system on chip, the system on substrate, the system in encapsulation, dedicated collection At circuit (ASIC), or can by circuit carry out the hardware such as any other rational method that is integrated or encapsulating or firmware come Realize, or with any one in three kinds of software, hardware and firmware implementations or with wherein it is any it is several it is appropriately combined come It realizes.Alternatively, obtain module 710, first processing module 720, write operation module 730, the first burst information reading unit 721, First data traversal reading unit 722, the first changing unit 723, breakdown judge unit 724, state judging unit 725, fragment At least one of information arrangement unit 731 and write operation unit 732 can at least be implemented partly as computer program mould Block can execute corresponding function when the computer program module is run.

Figure 10 A~Figure 10 B diagrammatically illustrates the structural block diagram of the Second processing module according to the embodiment of the present disclosure.

Another aspect of the present disclosure proposes a kind of data traversal device of calculate node.As shown in Figure 10 A, which includes Second processing module 1010.

The Second processing module 1010 is used to execute reading-traversing operation, multiple fragment letter based on multiple burst informations Breath is obtained in advance by the data traversal device of a calculate node at least one other calculate node, each burst information A corresponding data fragmentation.Wherein, the data traversal device of the calculate node can be with the number of at least one other calculate node Multiple burst informations are concurrently based on according to traversal device and execute reading-traversing operation, and it is corresponding to read multiple burst informations with traversal Data fragmentation storage data, the data traversal device traversal of different calculate nodes reads different data fragmentation storages Data.

In accordance with an embodiment of the present disclosure, multiple burst informations are by a calculate node at least one other calculate node Data traversal device obtain in advance after be written in preset memory space.A meter at least one other calculate node The data traversal device (hereinafter referred to as management node) of operator node is the data traversal dress of each calculate node in computing cluster (the data traversal device of the calculate node and the data traversal device of at least one other calculate node) is set using preset Election regulation elects to obtain.The embodiment of the present disclosure is not defined the preset election regulation, and preset election regulation is for example FastLeaderElection algorithm etc. can be used.

In accordance with an embodiment of the present disclosure, computing cluster initialize for the first time or before the management node elected occur When failure, the data traversal device of any calculate node in computing cluster is being attempted to fill with the data traversal of management node When finding that management node is not present or is unavailable when setting communication, triggering election, so that election obtains new management node.

In accordance with an embodiment of the present disclosure, as shown in Figure 10 A, which reads including the second burst information Unit 1011 and the second data traversal reading unit 1012.

Second burst information reading unit 1011 is used to read one in multiple burst informations from preset memory space Burst information, the second data traversal reading unit 1012 are used for the burst information based on reading, and it is corresponding that traversal reads burst information Data fragmentation storage data.In accordance with an embodiment of the present disclosure, the second burst information reading unit 1011 and the second data time Going through reading unit 1012 can be used for executing respectively with reference to Fig. 6 A operation S611 described and operation S612, and details are not described herein.

In accordance with an embodiment of the present disclosure, as shown in Figure 10 B, Second processing module 1010 can also include the second changing unit 1013。

Second changing unit 1013 in the second burst information reading unit 1011 from preset memory space for having read In the case where a burst information in multiple burst informations, the burst information of reading is set to the second shape by first state State；And the data of the corresponding data fragmentation storage of burst information have been read in the second data traversal reading unit 1012 traversal In the case of, burst information is set to the third state by the second state.Wherein, burst information is in the data traversal dress by management node It sets and is set to first state when preset memory space is written, to indicate that the burst information can be by the data traversal of calculate node Device or the data traversal device of at least one other calculate node are read；Second state of burst information indicates the burst information It has been be read that, can not be read again；The third state of burst information indicates the corresponding data fragmentation storage of the burst information Data, which have been traversed, have been read, which can not be read again.In accordance with an embodiment of the present disclosure, second changing unit Such as can be used for executing with reference to Fig. 6 B operation S613 described and operation S614, details are not described herein.

In accordance with an embodiment of the present disclosure, the number of calculate node can be used by being less than in computing cluster in the quantity of multiple burst informations According to number (the data traversal device of the calculate node and the data traversal dress of at least one other calculate node of traversal device The device sum set) in the case where, the data traversal device of multiple calculate nodes is (equal to multiple burst informations in computing cluster Quantity) via primary above-mentioned reading-traversing operation, the number to the storage of multiple burst informations corresponding data fragmentation can be completed According to traversal read.Certainly, the embodiment of the present disclosure does not limit specific traversal reading process, such as can also be saved by multiple calculating Data traversal device (less than the quantity of burst information) following via multiple reading-traversing operation of part calculate node in point Ring, so that multiple burst informations are read, and the data of corresponding data fragmentation storage are traversed reading.

In accordance with an embodiment of the present disclosure, calculate node can be used by being greater than in computing cluster in the quantity of multiple burst informations Number (the data traversal device of the calculate node and the data time of at least one other calculate node of data traversal device The device sum of calendar device) in the case where, the data traversal device of the calculate node and the number of at least one other calculate node Execution reading-traversing operation is then recycled according to traversal device.Specifically, in wherein calculate node task from Zookeeper After the child node of node has read the burst information of child node write-in, i.e., to the storage of the burst information corresponding data fragmentation Data carry out traversal reading；After completing traversal and reading, the data traversal device of the calculate node is then returned from Zookeeper Other child nodes (other child nodes not being read) of task node read the burst information of other child nodes write-in, and Traversal reading is carried out to the data of corresponding data fragmentation storage, the data traversal device of other calculate nodes also follows in this way Ring executes reading-traversing operation, until the burst information of all child nodes write-in of task node is read in Zookeeper It crosses, so that the data of the corresponding data fragmentation storage of multiple burst informations are traversed reading.

Second processing module 1010 according to an embodiment of the present disclosure, write operation module 730, the second burst information read single It is any number of or in which any number of in first 1011, second data traversal reading unit 1012 and the second changing unit 1013 At least partly function can be realized in a module.According to the module of the embodiment of the present disclosure, submodule, unit, subelement In any one or more can be split into multiple modules to realize.According to the module of the embodiment of the present disclosure, submodule, list Member, any one or more in subelement can at least be implemented partly as hardware circuit, such as field-programmable gate array Arrange (FPGA), programmable logic array (PLA), system on chip, the system on substrate, the system in encapsulation, specific integrated circuit (ASIC), it or can be realized by carrying out hardware or the firmware of any other rational method that is integrated or encapsulating to circuit, Or it several appropriately combined is realized with any one in three kinds of software, hardware and firmware implementations or with wherein any. Alternatively, can be at least by partly according to one or more of the module of the embodiment of the present disclosure, submodule, unit, subelement It is embodied as computer program module, when the computer program module is run, corresponding function can be executed.

For example, Second processing module 1010, the second burst information reading unit 1011, the second data traversal reading unit 1012 and second any number of in changing unit 1013 may be incorporated in a module realize or it is therein any one Module can be split into multiple modules.Alternatively, at least partly function of one or more modules in these modules can be with At least partly function of other modules combines, and realizes in a module.In accordance with an embodiment of the present disclosure, second processing mould In block 1010, the second burst information reading unit 1011, the second data traversal reading unit 1012 and the second changing unit 1013 At least one can at least be implemented partly as hardware circuit, such as field programmable gate array (FPGA), programmable patrol Array (PLA), system on chip, the system on substrate, the system in encapsulation, specific integrated circuit (ASIC) are collected, or can be passed through The hardware such as any other rational method that is integrated or encapsulating or firmware are carried out to realize to circuit, or with software, hardware and Any one in three kinds of implementations of firmware several appropriately combined is realized with wherein any.Alternatively, Second processing module 1010, in the second burst information reading unit 1011, the second data traversal reading unit 1012 and the second changing unit 1013 At least one can at least be implemented partly as computer program module, can be with when the computer program module is run Execute corresponding function.

Figure 11 diagrammatically illustrates the block diagram for being adapted for carrying out method as described above according to the embodiment of the present disclosure.Figure 11 The calculate node shown is only an example, should not function to the embodiment of the present disclosure and use scope bring any restrictions.

It as shown in figure 11, include processor 1101 according to the calculate node of the embodiment of the present disclosure 1100, it can be according to depositing It stores up the program in read-only memory (ROM) 1102 or is loaded into random access storage device (RAM) from storage section 1108 Program in 1103 and execute various movements appropriate and processing.Processor 1101 for example may include general purpose microprocessor (example Such as CPU), instruction set processor and/or related chip group and/or special microprocessor (for example, specific integrated circuit (ASIC)), Etc..Processor 1101 can also include the onboard processing device for caching purposes.Processor 1101 may include for executing According to single treatment unit either multiple processing units of the different movements of the method flow of the embodiment of the present disclosure.

In RAM 1103, it is stored with calculate node 1100 and operates required various programs and data.Processor 1101, ROM 1102 and RAM 1103 is connected with each other by bus 1104.Processor 1101 is by executing ROM 1102 and/or RAM Program in 1103 executes the various operations of the method flow according to the embodiment of the present disclosure.It is noted that described program can also To be stored in one or more memories in addition to ROM 1102 and RAM 1103.Processor 1101 can also pass through execution The program in one or more of memories is stored in execute the various operations of the method flow according to the embodiment of the present disclosure.

In accordance with an embodiment of the present disclosure, calculate node 1100 can also include input/output (I/O) interface 1105, input/ Output (I/O) interface 1105 is also connected to bus 1104.Calculate node 1100 can also include be connected to I/O interface 1105 with It is one or more in lower component: the importation 1106 including keyboard, mouse etc.；Including such as cathode-ray tube (CRT), liquid The output par, c 1107 of crystal display (LCD) etc. and loudspeaker etc.；Storage section 1108 including hard disk etc.；And including all The communications portion 1109 of such as network interface card of LAN card, modem.Communications portion 1109 via such as internet net Network executes communication process.Driver 1110 is also connected to I/O interface 1105 as needed.Detachable media 1111, such as disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on as needed on driver 1110, in order to from the meter read thereon Calculation machine program is mounted into storage section 1108 as needed.

In accordance with an embodiment of the present disclosure, computer software journey may be implemented as according to the method flow of the embodiment of the present disclosure Sequence.For example, embodiment of the disclosure includes a kind of computer program product comprising carry meter on a computer-readable medium Calculation machine program, the computer program include the program code for method shown in execution flow chart.In such embodiments, The computer program can be downloaded and installed from network by communications portion 1109, and/or from 1111 quilt of detachable media Installation.When the computer program is executed by processor 1101, the above-mentioned function of limiting in the system of the embodiment of the present disclosure is executed. In accordance with an embodiment of the present disclosure, system as described above, unit, module, unit etc. can pass through computer program module To realize.

The disclosure additionally provides a kind of computer-readable medium, which, which can be in above-described embodiment, retouches Included in the equipment/device/system stated；It is also possible to individualism, and without in the supplying equipment/device/system.On It states computer-readable medium and carries one or more program, when said one or multiple programs are performed, realize root According to the method flow of the embodiment of the present disclosure.

In accordance with an embodiment of the present disclosure, computer-readable medium can be computer-readable signal media or computer can Read storage medium either the two any combination.Computer readable storage medium for example can be --- but it is unlimited In system, device or the device of --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor, or any above combination.It calculates The more specific example of machine readable storage medium storing program for executing can include but is not limited to: have the electrical connection, portable of one or more conducting wires Formula computer disk, hard disk, random access storage device (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device or The above-mentioned any appropriate combination of person.In the disclosure, computer readable storage medium can be it is any include or storage program Tangible medium, which can be commanded execution system, device or device use or in connection.And in this public affairs In opening, computer-readable signal media may include in a base band or as carrier wave a part propagate data-signal, In carry computer-readable program code.The data-signal of this propagation can take various forms, including but not limited to Electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be computer-readable Any computer-readable medium other than storage medium, the computer-readable medium can send, propagate or transmit for by Instruction execution system, device or device use or program in connection.The journey for including on computer-readable medium Sequence code can transmit with any suitable medium, including but not limited to: wireless, wired, optical cable, radiofrequency signal etc., or Above-mentioned any appropriate combination.

For example, in accordance with an embodiment of the present disclosure, computer-readable medium may include above-described ROM 1102 and/or One or more memories other than RAM 1103 and/or ROM 1102 and RAM 1103.

Flow chart and block diagram in attached drawing are illustrated according to the system of the various embodiments of the disclosure, method and computer journey The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation A part of one module, program segment or code of table, a part of above-mentioned module, program segment or code include one or more Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical On can be basically executed in parallel, they can also be executed in the opposite order sometimes, and this depends on the function involved.Also it wants It is noted that the combination of each box in block diagram or flow chart and the box in block diagram or flow chart, can use and execute rule The dedicated hardware based systems of fixed functions or operations is realized, or can use the group of specialized hardware and computer instruction It closes to realize.

It will be understood by those skilled in the art that the feature recorded in each embodiment and/or claim of the disclosure can To carry out multiple combinations or/or combination, even if such combination or combination are not expressly recited in the disclosure.Particularly, In In the case where not departing from disclosure spirit or teaching, the feature recorded in each embodiment and/or claim of the disclosure can To carry out multiple combinations and/or combination.All these combinations and/or combination each fall within the scope of the present disclosure.

Embodiment of the disclosure is described above.But the purpose that these embodiments are merely to illustrate that, and It is not intended to limit the scope of the present disclosure.Although respectively describing each embodiment above, but it is not intended that each reality Use cannot be advantageously combined by applying the measure in example.The scope of the present disclosure is defined by the appended claims and the equivalents thereof.It does not take off From the scope of the present disclosure, those skilled in the art can make a variety of alternatives and modifications, these alternatives and modifications should all fall in this Within scope of disclosure.

Claims (24)

1. a kind of data traversal method of calculate node, the data are stored in multiple data fragmentations, which comprises

Multiple burst informations are obtained, the corresponding data fragmentation of each burst information；And

The multiple burst information is concurrently based on at least one other calculate node and executes reading-traversing operation, with traversal The data of the corresponding data fragmentation storage of the multiple burst information are read,

Wherein, different calculate nodes traversal reads the data of different data fragmentation storages.

2. according to the method described in claim 1, further include:

Preset memory space is written into the multiple burst information；

Reading-the traversing operation includes:

A burst information in multiple burst informations is read from the preset memory space；And

The burst information based on reading, traversal read the data of the corresponding data fragmentation storage of the burst information.

3. according to the method described in claim 2, wherein, multiple data fragmentations are stored in multiple storage sections of storage cluster On point, each memory node is stored with one or more data fragmentations, the method also includes:

Obtain the memory node information of multiple data fragmentations；

Preset memory space, which is written, in the multiple burst information includes:

It is based on preset allocation rule and the memory node information, the multiple burst information is evenly distributed, so as to have The burst information of the different data fragment of identical memory node information is alternatively arranged；And

Evenly distributed the multiple burst information is sequentially written in preset memory space；

Reading a burst information in multiple burst informations from the preset memory space includes:

A burst information is sequentially read from evenly distributed the multiple burst information.

4. according to the method described in claim 2, wherein, the reading-traversing operation further include:

In the case where having read a burst information in multiple burst informations from the preset memory space, by reading The burst information is set to the second state by first state；And

In the case where traversal has read the data that the corresponding data fragmentation of the burst information stores, by the burst information by Second state is set to the third state；

Wherein, the first state of the burst information indicates that the burst information can be by the calculate node or at least one its He reads calculate node；Second state of the burst information indicates that the burst information has been read, and can not be read again； The third state of the burst information indicates that the data of the burst information corresponding data fragmentation storage have been traversed and read, The burst information can not be read again.

5. according to the method described in claim 4, further include:

Judge other calculate nodes that a burst information in multiple burst informations is had read from the preset memory space It whether is malfunctioning node；And

In the case where other described calculate nodes are malfunctioning node, the burst information which is read is by second State is set to first state, and at least one other calculate node for the calculate node or in addition to the malfunctioning node is read.

6. according to the method described in claim 4, further include:

Judge whether the multiple burst information is in the third state；

In the case where there is the burst information for being not at the third state, returning and rejudging after the preset time of interval；

In the case where the multiple burst information is in the third state, the traversal of the data is completed.

7. according to the method described in claim 1, wherein:

The calculate node is elected by the calculate node and at least one other calculate node using preset election regulation It arrives；And/or

It is greater than the node total number of the calculate node and at least one other calculate node in the quantity of the multiple burst information In the case where, the reading-traversing operation is executed by the calculate node and/or at least one other calculate node circulation, with The data of the corresponding data fragmentation storage of the multiple burst information are made to be traversed reading.

8. a kind of data traversal method of calculate node, the data are stored in multiple data fragmentations, which comprises

Multiple burst informations are concurrently based on at least one other calculate node and execute reading-traversing operation, are read with traversing The data of the corresponding data fragmentation storage of the multiple burst information,

Wherein, the multiple burst information is obtained in advance by a calculate node at least one other calculate node, each The corresponding data fragmentation of a burst information, different calculate node traversals read the data of different data fragmentation storages.

9. according to the method described in claim 8, wherein:

The multiple burst information is written default after being obtained in advance by a calculate node at least one other calculate node Memory space in；

Reading-the traversing operation includes:

A burst information in multiple burst informations is read from the preset memory space；And

The burst information based on reading, traversal read the data of the corresponding data fragmentation storage of the burst information.

10. according to the method described in claim 9, wherein, the reading-traversing operation further include:

In the case where having read a burst information in multiple burst informations from the preset memory space, by reading The burst information is set to the second state by first state；And

In the case where traversal has read the data that the corresponding data fragmentation of the burst information stores, by the burst information by Second state is set to the third state；

Wherein, the first state of the burst information indicates that the burst information can be by the calculate node or at least one its He reads calculate node；Second state of the burst information indicates that the burst information has been read, and can not be read again； The third state of the burst information indicates that the data of the burst information corresponding data fragmentation storage have been traversed and read, The burst information can not be read again.

11. according to the method described in claim 8, wherein:

The calculate node for obtaining the multiple burst information in advance is adopted by the calculate node and at least one other calculate node It elects to obtain with preset election regulation；And/or

It is greater than the node total number of the calculate node and at least one other calculate node in the quantity of the multiple burst information In the case where, the calculate node and/or at least one other calculate node circulation execute the reading-traversing operation, so that The data of the corresponding data fragmentation storage of the multiple burst information are traversed reading.

12. a kind of data traversal device of calculate node, the data are stored in multiple data fragmentations, described device includes:

Module is obtained, multiple burst informations are obtained, the corresponding data fragmentation of each burst information；And

First processing module executes reading-traversing operation based on the multiple burst information,

Wherein, the data traversal device of the data traversal device of the calculate node and at least one other calculate node is concurrently Reading-the traversing operation is executed based on the multiple burst information, the corresponding number of the multiple burst information is read with traversal According to the data that fragment stores, the data traversal device traversal of different calculate nodes reads the number of different data fragmentation storages According to.

13. device according to claim 12, further includes:

Preset memory space is written in the multiple burst information by write operation module；

The first processing module includes:

First burst information reading unit reads the letter of a fragment in multiple burst informations from the preset memory space Breath；And

First data traversal reading unit, the burst information based on reading, traversal read the corresponding number of the burst information The data stored according to fragment.

14. device according to claim 13, in which:

Multiple data fragmentations are stored on multiple memory nodes of storage cluster, and each memory node is stored with one or more A data fragmentation；

The acquisition module obtains the memory node information of multiple data fragmentations；

The write operation module includes:

Burst information arrangement units are based on preset allocation rule and the memory node information, by the multiple burst information It is evenly distributed, so that the burst information of the different data fragment with identical memory node information is alternatively arranged；And

Evenly distributed the multiple burst information is sequentially written in preset memory space by write operation unit；

The first burst information reading unit sequentially reads a fragment letter from evenly distributed the multiple burst information Breath.

15. device according to claim 13, wherein the first processing module further includes the first changing unit:

One in multiple burst informations is had read from the preset memory space in the first burst information reading unit In the case where burst information, the burst information of reading is set to the second state by first state；And

The data of the corresponding data fragmentation storage of the burst information have been read in the first data traversal reading unit traversal In the case where, the burst information is set to the third state by the second state；

Wherein, the first state of the burst information indicates that the burst information can be filled by the data traversal of the calculate node It sets or the data traversal device of at least one other calculate node is read；Second state of the burst information indicates the fragment Information has been read, and can not be read again；The third state of the burst information indicates the corresponding data of the burst information The data of fragment storage, which have been traversed, have been read, which can not be read again.

16. device according to claim 15, wherein the first processing module further include:

Breakdown judge unit judges to have read a burst information in multiple burst informations from the preset memory space Whether the data traversal device of other calculate nodes is failed equipment；

First changing unit, in the case where the data traversal device of other calculate nodes is failed equipment, by this The burst information that failed equipment is read is set to first state by the second state, so that the data traversal of the calculate node fills It sets or the data traversal device of at least one other calculate node in addition to the malfunctioning node is read.

17. device according to claim 15, wherein the first processing module further includes state judging unit:

Judge whether the multiple burst information is in the third state；

In the case where there is the burst information for being not at the third state, returning and rejudging after the preset time of interval；

In the case where the multiple burst information is in the third state, the traversal of the data is completed.

18. device according to claim 12, in which:

Data traversal device and at least one other calculating of the data traversal device of the calculate node by the calculate node The data traversal device of node elects to obtain using preset election regulation；And/or

In data traversal device of the quantity of the multiple burst information greater than the calculate node and at least one other calculating In the case where the device sum of the data traversal device of node, by the data traversal device of the calculate node and/or at least one The data traversal device circulation of other a calculate nodes executes the reading-traversing operation, so that the multiple burst information pair The data for the data fragmentation storage answered are traversed reading.

19. a kind of data traversal device of calculate node, the data are stored in multiple data fragmentations, described device includes:

Second processing module executes reading-traversing operation based on multiple burst informations, and the multiple burst information is by least one The data traversal device of a calculate node in other calculate nodes obtains in advance, the corresponding data of each burst information Fragment,

Wherein, the data traversal device of the data traversal device of the calculate node and at least one other calculate node is concurrently Reading-the traversing operation is executed based on the multiple burst information, the corresponding number of the multiple burst information is read with traversal According to the data that fragment stores, the data traversal device traversal of different calculate nodes reads the number of different data fragmentation storages According to.

20. device according to claim 19, in which:

The multiple burst information is preparatory by the data traversal device of a calculate node at least one other calculate node It is written after acquisition in preset memory space；

The Second processing module includes:

Second burst information reading unit reads the letter of a fragment in multiple burst informations from the preset memory space Breath；And

Second data traversal reading unit, the burst information based on reading, traversal read the corresponding number of the burst information The data stored according to fragment.

21. device according to claim 20, wherein the Second processing module further includes the second changing unit:

One in multiple burst informations is had read from the preset memory space in the second burst information reading unit In the case where burst information, the burst information of reading is set to the second state by first state；And

The data of the corresponding data fragmentation storage of the burst information have been read in the second data traversal reading unit traversal In the case where, the burst information is set to the third state by the second state,

Wherein, the first state of the burst information indicates that the burst information can be filled by the data traversal of the calculate node It sets or the data traversal device of at least one other calculate node is read；Second state of the burst information indicates the fragment Information has been read, and can not be read again；The third state of the burst information indicates the corresponding data of the burst information The data of fragment storage, which have been traversed, have been read, which can not be read again.

22. device according to claim 19, in which:

The data traversal device for obtaining the calculate node of the multiple burst information in advance, by the data traversal of the calculate node Device and the data traversal device of at least one other calculate node elect to obtain using preset election regulation；And/or

In data traversal device of the quantity of the multiple burst information greater than the calculate node and at least one other calculating In the case where the device sum of the data traversal device of node, the data traversal device of the calculate node and/or at least one The data traversal device circulation of other a calculate nodes executes the reading-traversing operation, so that the multiple burst information pair The data for the data fragmentation storage answered are traversed reading.

23. a kind of calculate node, comprising:

One or more processors；

Storage device, for storing one or more programs,

Wherein, when one or more of programs respectively by one or more of processors execute when so that it is one or Multiple processors execute method described in any one according to claim 1~11.

24. a kind of computer-readable medium, is stored thereon with executable instruction, which hold processor when being executed by processor Method described in row claim 1~11 any one.