CN110399394A - The data traversal method, apparatus and calculate node of calculate node - Google Patents
The data traversal method, apparatus and calculate node of calculate node Download PDFInfo
- Publication number
- CN110399394A CN110399394A CN201810340710.7A CN201810340710A CN110399394A CN 110399394 A CN110399394 A CN 110399394A CN 201810340710 A CN201810340710 A CN 201810340710A CN 110399394 A CN110399394 A CN 110399394A
- Authority
- CN
- China
- Prior art keywords
- data
- burst information
- traversal
- calculate node
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 94
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 147
- 238000003860 storage Methods 0.000 claims abstract description 143
- 238000013467 fragmentation Methods 0.000 claims abstract description 118
- 230000014759 maintenance of location Effects 0.000 claims abstract description 12
- 230000015654 memory Effects 0.000 claims description 124
- 239000012634 fragment Substances 0.000 claims description 55
- 238000012545 processing Methods 0.000 claims description 40
- 230000033228 biological regulation Effects 0.000 claims description 21
- 230000015556 catabolic process Effects 0.000 claims description 9
- 238000007726 management method Methods 0.000 description 31
- 230000006870 function Effects 0.000 description 17
- 238000010586 diagram Methods 0.000 description 16
- 238000004590 computer program Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 11
- 230000006854 communication Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 5
- 230000006399 behavior Effects 0.000 description 4
- 238000005538 encapsulation Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 239000000758 substrate Substances 0.000 description 4
- 208000032370 Secondary transmission Diseases 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000001052 transient effect Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 2
- 230000005291 magnetic effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 239000013078 crystal Substances 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000000151 deposition Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 239000007788 liquid Substances 0.000 description 1
- 238000011017 operating method Methods 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Present disclose provides a kind of data traversal methods of calculate node.The data are stored in multiple data fragmentations, which comprises obtain multiple burst informations, the corresponding data fragmentation of each burst information;And multiple burst informations are concurrently based on at least one other calculate node and execute reading-traversing operation, to traverse the data for reading the corresponding data fragmentation storage of multiple burst informations, wherein, different calculate nodes traversal reads the data of different data fragmentation storages.The disclosure additionally provides the data traversal device and calculate node of a kind of calculate node.
Description
Technical field
This disclosure relates to field of computer technology, more particularly, to a kind of data traversal method, apparatus of calculate node
And calculate node.
Background technique
With the fast development of computer technology, in order to meet the data such as internet, Streaming Media, high-performance calculation, the energy
Intensive, high concurrent access industry requirement, cluster-based storage mode are come into being.
During realizing disclosure design, at least there are the following problems in the prior art for inventor's discovery: tradition meter
Calculate cluster to traverse the data stored in storage cluster by the following method: calling data-storage system traversal method first is read
In the data stored in storage cluster to the memory or disk of ETL server, then by data distribution to each calculate node.But
In the method, meter can not be played due to only supporting the sequence of single thread to read for storage clusters such as Redis, MySQL
The concurrent advantage of cluster multinode is calculated, therefore time-consuming for data traversal;Furthermore data needs carry out secondary transmission on network,
Limited speed is traversed in the network bandwidth and I O process ability of ETL server, therefore data traversal speed is slow.
Summary of the invention
In view of this, present disclose provides the data traversal method, apparatus and calculate node of a kind of calculate node.
The first aspect of the disclosure provides a kind of data traversal method of calculate node, and the data are stored in multiple numbers
According in fragment, this method comprises: obtaining multiple burst informations, each burst information corresponds to a data fragmentation;And with extremely
Few other calculate nodes are concurrently based on multiple burst informations and execute reading-traversing operation, read multiple fragments with traversal
The data of the corresponding data fragmentation storage of information, wherein different calculate node traversals reads different data fragmentation storages
Data.
In accordance with an embodiment of the present disclosure, the above method further includes that preset memory space is written in multiple burst informations;On
Stating reading-traversing operation includes: the burst information read in multiple burst informations from preset memory space;And it is based on
The burst information of reading, traversal read the data of the corresponding data fragmentation storage of the burst information.
In accordance with an embodiment of the present disclosure, above-mentioned multiple data fragmentations are stored on multiple memory nodes of storage cluster, often
A memory node is stored with one or more data fragmentations, the above method further include: obtains the memory node of multiple data fragmentations
Information;It includes: based on preset allocation rule and the memory node that preset memory space then, which is written, in multiple burst informations
Information, multiple burst informations are evenly distributed, so that the burst information of the different data fragment with identical memory node information
It is alternatively arranged;And evenly distributed multiple burst informations are sequentially written in preset memory space;It is above-mentioned from preset storage
It includes: sequentially to read one from evenly distributed multiple burst informations that a burst information in multiple burst informations is read in space
A burst information.
In accordance with an embodiment of the present disclosure, above-mentioned reading-traversing operation further include: more being had read from preset memory space
In the case where a burst information in a burst information, the burst information of reading is set to the second shape by first state
State;And in the case where traversal has read the data of the corresponding data fragmentation storage of burst information, by burst information by second
State is set to the third state;Wherein, burst information first state indicate burst information can by calculate node or at least one
Other calculate nodes are read;Second state of burst information indicates that burst information has been read, and can not be read again;Fragment letter
The third state of breath indicates that the data of the corresponding data fragmentation storage of burst information have been traversed and has read that the burst information cannot
It is enough to be read again.
In accordance with an embodiment of the present disclosure, the above method further include: judge to have read multiple fragments from preset memory space
Whether other calculate nodes of a burst information in information are malfunctioning node;And other calculate nodes be malfunctioning node
In the case where, the burst information which reads is set to first state by the second state, for calculate node or except this
At least one other calculate node outside malfunctioning node is read.
In accordance with an embodiment of the present disclosure, the above method further include: judge whether multiple burst informations are in the third state;
In the case where there is the burst information for being not at the third state, returning and rejudging after the preset time of interval;And multiple
In the case that burst information is in the third state, the traversal of the data is completed.
In accordance with an embodiment of the present disclosure, above-mentioned calculate node is by calculate node and at least one other calculate node using pre-
If election regulation elect to obtain;And/or it is greater than calculate node and at least one other in the quantity of multiple burst informations
In the case where the node total number of calculate node, executed by calculate node and/or at least one other calculate node circulation above-mentioned
Reading-traversing operation, so that the data of the corresponding data fragmentation storage of multiple burst informations are traversed reading.
The second aspect of the disclosure provides a kind of data traversal method of calculate node, and the data are stored in multiple numbers
According in fragment, this method comprises: being concurrently based on multiple burst informations at least one other calculate node executes reading-traversals
Operation, to traverse the data for reading the corresponding data fragmentation storage of multiple burst informations, wherein multiple burst informations are by least one
A calculate node in other a calculate nodes obtains in advance, the corresponding data fragmentation of each burst information, different
Calculate node traversal reads the data of different data fragmentation storages.
In accordance with an embodiment of the present disclosure, above-mentioned multiple burst informations are by a calculating at least one other calculate node
Node is written after obtaining in advance in preset memory space;Above-mentioned reading-traversing operation includes: to read from preset memory space
A burst information in multiple burst informations;And the burst information based on reading, traversal read the corresponding number of burst information
The data stored according to fragment.
In accordance with an embodiment of the present disclosure, above-mentioned reading-traversing operation further include: more being had read from preset memory space
In the case where a burst information in a burst information, the burst information of reading is set to the second state by first state;With
And in the case where traversal has read the data of the corresponding data fragmentation storage of burst information, burst information is set by the second state
For the third state;Wherein, the first state of burst information indicates that burst information can be by calculate node or at least one other meter
Operator node is read;Second state of burst information indicates that burst information has been read, and can not be read again;The of burst information
Three condition indicates that the data of burst information corresponding data fragmentation storage have been traversed and read, the burst information can not again by
It reads.
In accordance with an embodiment of the present disclosure, the above-mentioned calculate node for obtaining multiple burst informations in advance is by calculate node and at least
One other calculate node elects to obtain using preset election regulation;And/or it is greater than in the quantity of multiple burst informations
In the case where the node total number of calculate node and at least one other calculate node, calculate node and/or at least one other meter
Operator node circulation executes above-mentioned reading-traversing operation, so that the data quilt of the corresponding data fragmentation storage of multiple burst informations
Traversal is read.
The third aspect of the disclosure provides a kind of data traversal device of calculate node, and the data are stored in multiple numbers
According in fragment, which includes: acquisition module, obtains multiple burst informations, the corresponding data fragmentation of each burst information;
And first processing module, reading-traversing operation is executed based on multiple burst informations, wherein the data traversal of calculate node fills
It sets and is concurrently based on multiple burst informations execution reading-traversals behaviour with the data traversal device of at least one other calculate node
Make, to traverse the data for reading the corresponding data fragmentation storage of multiple burst informations, the data traversal dress of different calculate nodes
Set the data that traversal reads different data fragmentation storages.
In accordance with an embodiment of the present disclosure, above-mentioned apparatus further includes write operation module, pre- for multiple burst informations to be written
If memory space;Above-mentioned first processing module includes the first burst information reading unit and the first data traversal reading unit,
First burst information reading unit is used to read a burst information in multiple burst informations from preset memory space;First
Data traversal reading unit is used for the burst information based on reading, and traversal reads the corresponding data fragmentation storage of burst information
Data.
In accordance with an embodiment of the present disclosure, above-mentioned multiple data fragmentations are stored on multiple memory nodes of storage cluster, often
A memory node is stored with one or more data fragmentations;Above-mentioned acquisition module is also used to obtain multiple data fragmentations
Memory node information;Above-mentioned write operation module includes burst information arrangement units and write operation unit, and burst information arrangement is single
Member is evenly distributed by multiple burst informations for being based on preset allocation rule and the memory node information, so as to have phase
Burst information with the different data fragment of memory node information is alternatively arranged;Write operation unit is used for will be evenly distributed multiple
Burst information is sequentially written in preset memory space;Above-mentioned first burst information reading unit is believed from evenly distributed multiple fragments
A burst information is sequentially read in breath.
In accordance with an embodiment of the present disclosure, above-mentioned first processing module further includes the first changing unit, and the first changing unit is used
In having read from preset memory space a burst information in multiple burst informations in the first burst information reading unit
In the case of, the burst information of reading is set to the second state by first state;And it is traversed in the first data traversal reading unit
In the case where the data for having read the corresponding data fragmentation storage of burst information, burst information is set to third shape by the second state
State;Wherein, burst information first state indicate burst information can by the data traversal device of calculate node or at least one
The data traversal device of other calculate nodes is read;Second state of burst information indicates that burst information has been read, can not
It is read again;The third state of burst information indicates that the data of the corresponding data fragmentation storage of burst information have been traversed reading
Complete, which can not be read again.
In accordance with an embodiment of the present disclosure, above-mentioned first processing module further includes breakdown judge unit, for judging from default
Memory space have read a burst information in multiple burst informations other calculate nodes data traversal device whether
For failed equipment;Above-mentioned first changing unit is also used to the case where the data traversal device of other calculate nodes is failed equipment
Under, the burst information that the data traversal device of the malfunctioning node is read is set to first state by the second state, for calculating section
The data traversal device of point or the data traversal device of at least one other calculate node in addition to the malfunctioning node are read.
In accordance with an embodiment of the present disclosure, above-mentioned first processing module further includes state judging unit, for judging multiple points
Whether piece information is in the third state;And in the case where there is the burst information for being not at the third state, when interval is default
Between after return rejudge;And in the case where multiple burst informations are in the third state, complete the traversal of the data.
In accordance with an embodiment of the present disclosure, the data traversal device of above-mentioned calculate node by calculate node data traversal device
And the data traversal device of at least one other calculate node elects to obtain using preset election regulation;And/or more
The quantity of a burst information is greater than the data traversal device of calculate node and the data traversal of at least one other calculate node
In the case where the regulating device sum of device, data traversal device and/or at least one other calculate node by calculate node
Data traversal device circulation executes above-mentioned reading-traversing operation, so that the number of the corresponding data fragmentation storage of multiple burst informations
According to being traversed reading.
The fourth aspect of the disclosure provides a kind of data traversal device of calculate node, and the data are stored in multiple numbers
According in fragment, which includes Second processing module, for executing reading-traversing operation, Duo Gefen based on multiple burst informations
Piece information is obtained in advance by the data traversal device of a calculate node at least one other calculate node, each fragment
Information corresponds to a data fragmentation.Wherein, the data traversal device of the calculate node and at least one other calculate node
Data traversal device is concurrently based on multiple burst informations and executes above-mentioned reading-traversing operation, reads multiple fragment letters with traversal
The data of corresponding data fragmentation storage are ceased, the data traversal device traversal of different calculate nodes reads different data fragmentations
The data of storage.
In accordance with an embodiment of the present disclosure, above-mentioned multiple burst informations are by a calculating at least one other calculate node
The data traversal device of node is written in preset memory space after obtaining in advance;Above-mentioned Second processing module includes the second fragment
Information reading unit and the second data traversal reading unit, the second burst information reading unit are used to read from preset memory space
Take a burst information in multiple burst informations;Second data traversal reading unit is used for the burst information based on reading, time
Go through the data for reading the corresponding data fragmentation storage of burst information.
In accordance with an embodiment of the present disclosure, above-mentioned Second processing module further includes the second changing unit, in the second fragment
In the case that Information reading unit has read a burst information in multiple burst informations from preset memory space, it will read
Burst information the second state is set to by first state;And burst information has been read in the second data traversal reading unit traversal
In the case where the data of corresponding data fragmentation storage, burst information is set to the third state by the second state;Wherein, fragment is believed
The first state of breath indicates that burst information can be by the data traversal device of calculate node or at least one other calculate node
Data traversal device is read;Second state of burst information indicates that burst information has been read, and can not be read again;Fragment letter
The third state of breath indicates that the data of the corresponding data fragmentation storage of burst information have been traversed and has read that the burst information cannot
It is enough to be read again.
In accordance with an embodiment of the present disclosure, the data traversal device of the above-mentioned calculate node for obtaining multiple burst informations in advance by
The data traversal device of calculate node and the data traversal device of at least one other calculate node use preset election regulation
Election obtains;And/or the quantity of multiple burst informations be greater than calculate node data traversal device and at least one its
In the case where the device sum of the data traversal device of his calculate node, the data traversal device of calculate node and/or at least
The data traversal device circulation of one other calculate node executes above-mentioned reading-traversing operation, so that multiple burst informations are corresponding
Data fragmentation storage data be traversed reading.
5th aspect of the disclosure provides a kind of calculate node, comprising: one or more processors;Storage device is used
In the one or more programs of storage, wherein when one or more programs are executed by one or more processors respectively, so that one
A or multiple processors execute the data traversal method and disclosure second aspect for the calculate node that disclosure first aspect provides
The data traversal method of the calculate node of offer.
6th aspect of the disclosure provides a kind of computer-readable medium, is stored thereon with executable instruction, the instruction
Processor is set to execute the data traversal method and the disclosure of the calculate node that disclosure first aspect provides when being executed by processor
The data traversal method for the calculate node that second aspect provides.
In accordance with an embodiment of the present disclosure, data present in the ergodic process of data can be at least avoided to need by secondary
Transmission and can only the defect that reads of single thread sequence, and data traversal speed therefore can be improved, reduces the time-consuming of data traversal.
Detailed description of the invention
By referring to the drawings to the description of the embodiment of the present disclosure, the above-mentioned and other purposes of the disclosure, feature and
Advantage will be apparent from, in the accompanying drawings:
Fig. 1 diagrammatically illustrates the data traversal method, apparatus according to the calculate node of the embodiment of the present disclosure and calculates section
The system architecture of point;
Fig. 2A diagrammatically illustrates the method flow diagram of the data traversal method according to the calculate node of the embodiment of the present disclosure;
Fig. 2 B diagrammatically illustrates the method stream of the data traversal method of the calculate node according to another embodiment of the disclosure
Cheng Tu;
Fig. 2 C diagrammatically illustrates reading-traversing operation flow chart according to the embodiment of the present disclosure;
Fig. 2 D diagrammatically illustrates the operational flowchart with reference to Fig. 2 B operation S230 described;
Fig. 3 diagrammatically illustrates reading-traversing operation flow chart according to another embodiment of the disclosure;
Fig. 4 diagrammatically illustrates the method flow of the data traversal method of the calculate node according to another embodiment of the disclosure
Figure;
Fig. 5 diagrammatically illustrates the method flow of the data traversal method of the calculate node according to another embodiment of the disclosure
Figure;
Fig. 6 A~Fig. 6 B diagrammatically illustrates reading-traversing operation flow chart according to another embodiment of the disclosure;
Fig. 7 A~Fig. 7 B diagrammatically illustrates the structure of the data traversal device according to the calculate node of the embodiment of the present disclosure
Block diagram;
Fig. 8 A~Fig. 8 D diagrammatically illustrates the structural block diagram of the first processing module according to the embodiment of the present disclosure;
Fig. 9 diagrammatically illustrates the structural block diagram of the write operation module according to the embodiment of the present disclosure;
Figure 10 A~Figure 10 B diagrammatically illustrates the structural block diagram of the Second processing module according to the embodiment of the present disclosure;
Figure 11 diagrammatically illustrates the block diagram for being adapted for carrying out method as described above according to the embodiment of the present disclosure.
Specific embodiment
Hereinafter, will be described with reference to the accompanying drawings embodiment of the disclosure.However, it should be understood that these descriptions are only exemplary
, and it is not intended to limit the scope of the present disclosure.In the following detailed description, to elaborate many specific thin convenient for explaining
Section is to provide the comprehensive understanding to the embodiment of the present disclosure.It may be evident, however, that one or more embodiments are not having these specific thin
It can also be carried out in the case where section.In addition, in the following description, descriptions of well-known structures and technologies are omitted, to avoid
Unnecessarily obscure the concept of the disclosure.
Term as used herein is not intended to limit the disclosure just for the sake of description specific embodiment.It uses herein
The terms "include", "comprise" etc. show the presence of the feature, step, operation and/or component, but it is not excluded that in the presence of
Or add other one or more features, step, operation or component.
There are all terms (including technical and scientific term) as used herein those skilled in the art to be generally understood
Meaning, unless otherwise defined.It should be noted that term used herein should be interpreted that with consistent with the context of this specification
Meaning, without that should be explained with idealization or excessively mechanical mode.
It, in general should be according to this using statement as " at least one in A, B and C etc. " is similar to
Field technical staff is generally understood the meaning of the statement to make an explanation (for example, " system at least one in A, B and C "
Should include but is not limited to individually with A, individually with B, individually with C, with A and B, with A and C, have B and C, and/or
System etc. with A, B, C).Using statement as " at least one in A, B or C etc. " is similar to, generally come
Saying be generally understood the meaning of the statement according to those skilled in the art to make an explanation (for example, " having in A, B or C at least
One system " should include but is not limited to individually with A, individually with B, individually with C, with A and B, have A and C, have
B and C, and/or the system with A, B, C etc.).It should also be understood by those skilled in the art that substantially arbitrarily indicating two or more
The adversative conjunction and/or phrase of optional project shall be construed as either in specification, claims or attached drawing
A possibility that giving including one of these projects, either one or two projects of these projects.For example, phrase " A or B " should
A possibility that being understood to include " A " or " B " or " A and B ".
Embodiment of the disclosure provides a kind of data traversal method of calculate node, and the data are stored in multiple data
In fragment, this method comprises: multiple burst informations are obtained, the corresponding data fragmentation of each burst information;And at least
One other calculate node is concurrently based on multiple burst informations and executes reading-traversing operation, reads multiple fragment letters with traversal
Cease the data of corresponding data fragmentation storage, wherein different calculate node traversals reads the number of different data fragmentation storages
According to.
Embodiment of the disclosure additionally provides the data traversal method of another calculate node, and the data are stored in multiple
In data fragmentation, this method comprises: being concurrently based on multiple burst informations at least one other calculate node executes reading-time
Operation is gone through, to traverse the data for reading the corresponding data fragmentation storage of multiple burst informations, wherein multiple burst informations are by least
A calculate node in one other calculate node obtains in advance, the corresponding data fragmentation of each burst information, different
Calculate node traversal read the data of different data fragmentation storages.
Fig. 1 diagrammatically illustrates the data traversal method, apparatus according to the calculate node of the embodiment of the present disclosure and calculates section
The system architecture 100 of point.It should be noted that be only the example that can apply the system architecture of the embodiment of the present disclosure shown in Fig. 1,
To help skilled in the art to understand the technology contents of the disclosure, but it is not meant to that the embodiment of the present disclosure may not be usable for it
His equipment, system, environment or scene.
As shown in Figure 1, according to this embodiment may include storage cluster and computing cluster, storage in system architecture 100
Cluster includes memory node 101,102,103,104, and computing cluster includes calculate node 105,106,107.Storage cluster and meter
It calculates by network connection between cluster, network may include various connection types, such as wired, wireless communication link or optical fiber
Cable etc., and composition distribution or parallel computing cluster are mutually connected to each other between each calculate node.
Calculate node 105,106,107 can be interacted by network with memory node 101,102,103,104, to storage
Node carries out the operation such as write-in, reading or deletion of data.
The data stored in storage cluster for example can be used data fragmentation storage strategy and be stored, and data fragmentation stores plan
Slightly refer to data according to specific slicing algorithm, be divided into multiple data fragmentations, and these data fragmentations are respectively stored into not
It is stored on same memory node.So that memory space is not limited by storage capacity of single node, stored by increase and decrease
The Number dynamics of node adjust memory space, and support concurrent reading and writing data.
Traditionally, the reading of data is carried out by the way that ETL server is arranged between storage cluster and computing cluster, specifically
Are as follows: calling data-storage system traversal method is read in the data to ETL server memory or disk in storage cluster first,
Again by each calculate node of data distribution to computing cluster, the reading of data is completed.In the process, since storage cluster provides
Default traversal method only support single thread sequence to read, therefore the concurrent advantage of multinode that cluster can not be played;Furthermore data
Need first to read ETL server, then be distributed to each node of computing cluster, thus data needs carried out on network it is secondary
Transmission, reduces data read rates;And the reading rate of entire data read process is limited to the network bandwidth of ETL server
With I O process ability, and the processing speed of ETL is slower, and if in data traversal reading process, ETL server fail,
It then will lead to data traversal failure.
In view of problem above, as shown in Figure 1, embodiment of the disclosure passes through preset election regulation in computing cluster
The calculate node 105 that election has obtained coordinated management effect obtains the burst information of each data fragmentation in storage cluster, and passes through
Distributed coordination service architecture and other calculate nodes realize information sharing, then each calculate node 105 in computing cluster, 106,
107 by reading shared burst information, can be directly to data fragmentation corresponding with the burst information of reading in storage cluster
The data of storage carry out traversal reading.And different burst informations can be read in multiple calculate nodes, concurrently to corresponding data
The data of fragment storage carry out traversal reading, therefore can effectively improve data traversal speed, and theoretically traversal speed is to pass
" MIN (data fragmentation quantity, the calculate node quantity) " times of system traversal method, for example, there are 10 data fragmentations at one
In the computing cluster environment of storage cluster and 10 nodes, data traversal time-consuming is 1/10th of conventional method.
It should be noted that the data traversal method of calculate node provided by the embodiment of the present disclosure generally can be by calculating
Node 105,106,107 executes.Correspondingly, the data traversal device of calculate node provided by the embodiment of the present disclosure generally can be with
It is set in calculate node 105,106,107.
It should be understood that the number of memory node and calculate node in Fig. 1 and the type of various connection relationships are only
Schematically.According to needs are realized, any number of memory node and calculate node and any type of connection can have
Relationship.
Fig. 2A diagrammatically illustrates the method flow diagram of the data traversal method according to the calculate node of the embodiment of the present disclosure.
As shown in Figure 2 A, an aspect of this disclosure provides a kind of data traversal method of calculate node, this method packet
Include operation S210~S220.
In operation S210, multiple burst informations are obtained, the corresponding data fragmentation of each burst information.
Above-mentioned data are stored in multiple data fragmentations, and each data fragmentation is corresponding with a burst information, with unique
Corresponding data fragment.
In accordance with an embodiment of the present disclosure, in accordance with an embodiment of the present disclosure, which belongs to one in computing cluster
Calculate node, such as can be the calculate node 105 described with reference to Fig. 1.It further include in addition to the calculate node in computing cluster
At least one other calculate node, such as the calculate node 106,107 described with reference to Fig. 1.Obtain the calculating of multiple burst informations
Node for example can be elected to obtain by preset election regulation by all calculate nodes in computing cluster, for convenience of describing, with
Calling the calculate node that the election obtains in the following text is management node.Wherein, the embodiment of the present disclosure does not carry out the preset election regulation
It limits, FastLeaderElection algorithm etc. for example can be used in preset election regulation.
In accordance with an embodiment of the present disclosure, computing cluster initialize for the first time or before the management node elected occur
Failure, any calculate node in computing cluster find that management node is not present or not when attempting to communicate with management node
When available, triggering election, so that election obtains new management node.
In accordance with an embodiment of the present disclosure, any one calculate node in computing cluster all can provide interface and mention for client
Traversal task is handed over, ergodic condition is carried in the traversal task, at least one other calculate node is receiving client submission
Traversal task when, by the communication with management node, according to the traversal task, initiate traversal request to management node.Then should
Management node is when obtaining burst information, based on traversal request (including ergodic condition), from multiple numbers of storage cluster storage
Burst information is obtained according to fragment.
In accordance with an embodiment of the present disclosure, the traversal task of client submission is received in the interface that multiple calculate nodes provide
When, which can create multiple tasks node after receiving multiple traversal tasks in distributed coordination service architecture,
Each traversal task is corresponding with a task node, then the ergodic condition that the management node is carried according to multiple traversal tasks is certainly more
A data fragmentation reads burst information, and by the burst information of return according to the matching with the task of traversal, respectively will with it is each
The matched multiple burst informations of traversal task are sequentially written in the child node of each task node, then multiple task node can be with
It is arranged in task node catalogue, each task node includes multiple child nodes again, and each child node is written with a burst information.
The embodiment of the present disclosure is not defined distributed coordination service architecture, for example, the distributed coordination service architecture can be
Zookeeper (abbreviation zk) distributed coordination service etc., to realize the information sharing in computing cluster between multiple calculate nodes.
In operation S220, the multiple burst information is concurrently based on at least one other calculate node and executes reading-
Traversing operation.
In accordance with an embodiment of the present disclosure, reading-traversing operation is specifically as follows: reading one point in multiple burst informations
Piece information, then traversal reads the data of the corresponding data fragmentation storage of burst information.
Wherein, at least one other calculate node is are as follows: in addition to the calculating section of acquisition burst information in computing cluster
Other calculate nodes outside point.Since the calculate node and at least one other calculate node can be based on the fragment letters obtained
Breath is performed in parallel reading-traversing operation, to traverse the data for reading the corresponding data fragmentation storage of multiple burst informations, because
This, the scheme of the embodiment of the present disclosure can be realized the concurrently traversal of data, greatly improve the speed of data traversal.Furthermore by
Burst information is based in calculate node and traverses reading data directly from storage cluster, avoids data by ETL server
Secondary transfer, therefore can be further improved the speed of data traversal.
Fig. 2 B diagrammatically illustrates the method stream of the data traversal method of the calculate node according to another embodiment of the disclosure
Cheng Tu.
As shown in Figure 2 B, according to the embodiment of the present disclosure, the data traversal method of the calculate node of Fig. 2A description is being operated
Between S210~operation S220, it may also include S230.
In operation S230, preset memory space is written into the multiple burst information.
In accordance with an embodiment of the present disclosure, the calculate node for obtaining multiple burst informations multiple burst informations can be written in advance
If memory space, the preset memory space for example can be the communal space, for each calculate node in computing cluster
Access.
Specifically, which can be for example the memory space that distributed coordination service architecture provides, from
And realize the information sharing in computing cluster between each calculate node, then each calculate node in computing cluster can certainly should
Burst information is read in memory space, is read with the traversal that the burst information based on reading carries out data in corresponding data fragment.
For example, burst information is written in the child node in the task node in Zookeeper by management node, then to single traversal
When task is traversed, each calculate node in computing cluster can the child node of self-corresponding task node read fragment letter
Breath.
Fig. 2 C diagrammatically illustrates reading-traversing operation flow chart according to the embodiment of the present disclosure.
As shown in Figure 2 C, the case where preset memory space is written in burst information by the calculate node for obtaining burst information
Under, reading-traversing operation correspondingly includes operation S221~S222.
In operation S221, a burst information in multiple burst informations is read from preset memory space;It is operating
S222, the burst information based on reading, traversal read the data of the corresponding data fragmentation storage of burst information.
In accordance with an embodiment of the present disclosure, of calculate node can be used by being less than in computing cluster in the quantity of multiple burst informations
In the case where number (node total number of management node and at least one other calculate node), multiple calculate nodes in computing cluster
(equal to the quantity of multiple burst informations) can be completed via primary above-mentioned reading-traversing operation to multiple burst informations pair
The traversal of the data for the data fragmentation storage answered is read.Certainly, the embodiment of the present disclosure does not limit specific traversal reading process,
Such as it can also be grasped by the part calculate node (less than the quantity of burst information) in multiple calculate nodes via multiple reading-traversal
The circulation of work, so that multiple burst informations are read, and the data of corresponding data fragmentation storage are traversed reading.
In accordance with an embodiment of the present disclosure, of calculate node can be used by being greater than in computing cluster in the quantity of multiple burst informations
In the case where number (node total number of management node and at least one other calculate node), the management node and at least one other
Calculate node then recycles execution reading-traversing operation.Specifically, in wherein calculate node task section from Zookeeper
Number after the child node of point has read the burst information of child node write-in, i.e., to the storage of the burst information corresponding data fragmentation
According to carrying out traversal reading;After completing traversal and reading, which is then returned from other sons of task node in Zookeeper
Node (other child nodes not being read) reads the burst information of other child nodes write-in, and to corresponding data fragmentation
The data of storage carry out traversal reading, and also circulation executes reading-traversing operation to other calculate nodes in this way, until
The burst information of all child nodes write-in of task node was read in Zookeeper, to guarantee multiple burst informations pair
The data for the data fragmentation storage answered are traversed reading.
In accordance with an embodiment of the present disclosure, if there are when multiple tasks node in Zookeeper, available meter in computing cluster
Operator node then successively traverses the traversal task of multiple tasks node, is written in the child node to a task node
The data of burst information corresponding data fragmentation storage complete after traversal reads, then read the child node of next task node and write
The burst information entered is traversed with the traversal task to next task node.
Fig. 2 D diagrammatically illustrates the operational flowchart with reference to Fig. 2 B operation S230 described.
In accordance with an embodiment of the present disclosure, multiple data fragmentations are storable on multiple memory nodes of storage cluster, and every
A memory node is stored with one or more data fragmentations.
Then, it can also be operation S210 ' with reference to the operation S210 in Fig. 2A, obtain multiple burst informations and multiple data point
The memory node information of piece, the memory node information can indicate which the corresponding data fragmentation of certain burst information is stored in and deposits
Store up node.
As shown in Figure 2 D, operation S230 may include operation S231 and operation S232.
In operation S231, it is based on preset allocation rule and memory node information, multiple burst informations are evenly distributed, with
It is alternatively arranged the burst information of the different data fragment with identical memory node information;It, will be evenly distributed in operation S232
Multiple burst informations be sequentially written in preset memory space.
Specifically, the principle of the preset allocation rule are as follows: make the fragment of the data fragmentation in same memory node
Information is separated by setting in arrangement queue, preferably makes separated by a distance farthest as far as possible.The preset allocation rule for example can be
Strategy is simply evenly distributed below: burst information is organized into one two according to memory node where its corresponding data fragmentation
The burst information of dimension group, the data fragmentation with identical memory node information is located at same row, has different memory nodes letter
The burst information of the data fragmentation of breath is located at different rows.Such as multiple burst informations can be arranged as to a burst information list,
It, can be since the first row first row of two-dimensional array, successively from two dimension then when carrying out evenly distributed to multiple burst informations
Every a line of first row takes a burst information to be placed on the end of list in array;The burst information of first row in two-dimensional array
After having arranged, then since the burst information of the first row secondary series, a burst information successively is taken from every a line of secondary series
It is placed on the end of list, and so on, it is up to arranging all burst informations in 2-D data to burst information list
Only.
For there are in the case where multiple traversal tasks, then being carried out uniformly to multiple burst informations of each traversal task
Arrangement, so that the burst information of the different data fragment in each traversal task with identical memory node information is alternatively arranged.
Then in operation S232, the multiple burst informations arranged in burst information list are sequentially written in preset deposit in order
Space is stored up, for Zookeeper distributed coordination service architecture, then by multiple burst informations according to suitable in burst information list
Sequence is sequentially written in the child node under task node.And for there are in the case where multiple tasks node, then burst information being arranged
Burst information in table is written according to corresponding relationship in the child node of corresponding task node, in different burst information lists
Burst information is written in the child node of different task nodes.
In reading-traversing operation, a burst information in multiple burst informations is read from preset memory space, it can
Think and sequentially reads a burst information from evenly distributed multiple burst informations.Specifically, it is as pressed from task node
Putting in order for child node is successively read burst information.
The embodiment of the present disclosure can make the multiple data fragmentations for being stored in same memory node corresponding by above-mentioned setting
Position of the burst information in task node it is farthest as far as possible, then, can when reading the burst information in preset memory space
The different burst informations for avoiding multiple calculate nodes from reading are the burst information of the different data fragment of same memory node storage
The case where, therefore, it can avoid multiple calculating when the data stored to the corresponding data fragmentation of burst information carry out traversal reading
The case where node accesses to same memory node is read so as to avoid traversing caused by single memory node overload
The lower defect of speed, to improve the efficiency of data traversal as far as possible.
In order to guarantee the stability of data storage, some calculate node is avoided in storage cluster to break down caused part
Data fragmentation can be saved more parts, then when obtaining burst information, unavoidably by loss of data on different memory nodes
Meeting there are identical burst informations.
Therefore, when carrying out evenly distributed to multiple burst informations of acquisition, duplicate checking can be carried out, to guarantee to deposit with difference
Multiple identical burst informations of storage nodal information only save one after evenly distributed.Multiple burst informations can be also arranged as
One burst information set, due to gathering not reproducible characteristic, first be only arranged in multiple same slice information
Fragment burst information can be put into burst information set.
By the above-mentioned means, can guarantee the stability of storing data in storage cluster, also can avoid in subsequent traversal
In reading process, the repetition for the identical data for being stored in different memory nodes is traversed and is read, so as to avoid unnecessary volume
Overhead improves the efficiency of computing cluster ergodic data.
Fig. 3 diagrammatically illustrates reading-traversing operation flow chart according to another embodiment of the disclosure.
As shown in figure 3, reading-the traversing operation other than operating S221~S222, further includes operation S223 and operation
S224.Wherein, operation S223 is executed between operation S221 and operation S222, and operation S224 is held after having executed operation S223
Row.
In operation S223, the burst information of reading is set to the second state by first state;Operation S224, time
It goes through in the case where having read, the burst information is set to the third state by the second state.
In accordance with an embodiment of the present disclosure, after the calculate node obtains multiple burst informations, by multiple burst information
When preset memory space is written, multiple burst information is set to original state i.e. first state, to indicate multiple fragment
Information can be read by any calculate node (calculate node or at least one other calculate node) in computing cluster;Fragment letter
Second state of breath then indicates that the burst information is read by any one calculate node in computing cluster, can not be counted again
The calculate node calculated in cluster is read;The third state of burst information indicates that the corresponding data fragmentation of the burst information stores
Data, which have been traversed, have been read, and the burst information can not be read by the calculate node in computing cluster again.
By the above-mentioned setting to burst information state, the corresponding number of any burst information in multiple burst informations can avoid
Traversal is repeated by computing cluster according to the data of fragment storage to read, and can avoid the case where part burst information is not read, from
And under the premise of guaranteeing to read data integrity, the time-consuming of data traversal can be reduced as far as possible.
Fig. 4 diagrammatically illustrates the method flow of the data traversal method of the calculate node according to another embodiment of the disclosure
Figure.
As shown in figure 4, this method includes operation S410~S440.
In operation S410, the state of multiple burst informations of preset memory space storage is obtained;In operation S420, judgement
Whether multiple burst informations are in the third state;In the case where there is the burst information for being not at the third state, behaviour is executed
Make S430, returns to execution operation S410 after being spaced preset time, judgement is re-started with the state to multiple burst informations;Institute
It states in the case that multiple burst informations are in the third state, executes operation S440, complete the traversal of the data.
In accordance with an embodiment of the present disclosure, it is an operating procedure that aforesaid operations S410 and operation S420 is also combinable, directly
Judge whether multiple burst informations are in the third state, without obtaining the state of multiple burst informations in advance, but to pre-
If memory space storage multiple burst informations judge one by one in order, if judgement discovery be not at the third state fragment letter
Breath then interrupts judgement, after being spaced preset time, then since at the burst information that the discovery is not at the third state, rejudges
With the presence or absence of the burst information for being not at the third state, until not finding to be not at the burst information of the third state, explanation
Multiple burst informations are in the third state, then terminate judgement, complete the traversal of data.
Pass through aforesaid operations, it can be ensured that the data of the corresponding data fragmentation storage of multiple burst informations are traversed reading
It takes, avoids the omission to partial data, ensure that the integrality of reading data.
Fig. 5 diagrammatically illustrates the method flow of the data traversal method of the calculate node according to another embodiment of the disclosure
Figure.
As shown in figure 5, this method other than operating S210~S220, further includes operation S510 and operation S520.
In operation S510, judge to have read a burst information in multiple burst informations from the preset memory space
Other calculate nodes whether be malfunctioning node;In the case where other malfunctioning nodes are malfunctioning node, operation S520 is executed, it will
The burst information that the malfunctioning node is read is set to first state by the second state.
In accordance with an embodiment of the present disclosure, it when calculate node reads burst information from preset memory space, such as can incite somebody to action
At the storage location for the burst information that the reading is written in its identification information, and calculate node is carried out with preset memory space
When initial interaction, a transient node, the identification of the transient node and the calculate node can be created in preset memory space
Information matches;When the calculate node and preset memory space interact end or calculate node failure cause cannot
When interaction, which will disappear.Therefore, when judging whether read the calculate node of burst information is malfunctioning node,
It can first check whether the burst information is in the second state, if so, judging that preset memory space whether there is and the fragment again
The transient node that the identification information for the calculate node being written at the storage location of information matches, and if it exists, then illustrate the meter
Operator node works normally, and if it does not exist, then illustrates that the calculate node does not traverse and has read the corresponding data point of the burst information
The data of piece, and the calculate node is malfunctioning node.
In the case where other calculate nodes are malfunctioning node, it is responsible for by management node by other calculate nodes reading
Burst information is set to first state by the second state, so that the management node or other normal calculate nodes are read.
In accordance with an embodiment of the present disclosure, it is monitored by the judgement of above-mentioned malfunctioning node, can effectively avoid malfunctioning node time
Go through data failure, and other nodes be because that can not obtain corresponding burst information, thus this partial data can not be traversed
Defect greatly improves the high availability of computing cluster, further ensures the integrality of data traversal reading.
Another aspect of the present disclosure additionally provides a kind of data traversal method of calculate node.
In accordance with an embodiment of the present disclosure, the data are stored in multiple data fragmentations, the data traversal method include with
Lower operation: multiple burst informations are concurrently based on at least one other calculate node and execute reading-traversing operation, are read with traversing
Take the data of the corresponding data fragmentation storage of multiple burst informations.
In accordance with an embodiment of the present disclosure, the calculate node is the calculate node in computing cluster, which can be with
For with reference to the calculate node 106,107 in Fig. 1;It can also be at least one other calculate node described with reference to Fig. 2A~Fig. 2 B
In calculate node.
In accordance with an embodiment of the present disclosure, multiple burst informations are by other calculate nodes of at least one of computing cluster
A calculate node obtain in advance, which for example can be for reference to the meter in Fig. 1
Operator node 105, can also for reference to Fig. 2A~with reference to Fig. 5 describe calculate node data traversal method in calculate node.
In accordance with an embodiment of the present disclosure, the calculate node at least one other calculate node for obtaining burst information is
It is elected by computing cluster (calculate node and at least one other calculate node) using preset election regulation.The disclosure
Embodiment is not defined the preset election regulation, and FastLeaderElection for example can be used in preset election regulation
Algorithm etc..
In accordance with an embodiment of the present disclosure, computing cluster initialize for the first time or before the management node elected occur
Failure, any calculate node in computing cluster find that management node is not present or not when attempting to communicate with management node
When available, triggering election, so that election obtains new management node.
In accordance with an embodiment of the present disclosure, the corresponding data fragmentation of each burst information.And the calculate node with
When at least one other calculate node traversal reads data, the data fragmentation that is different that different calculate node traversals is read is deposited
The data of storage.
The data traversal method of the calculate node of the embodiment of the present disclosure, due to at least one other calculate node concurrently
Reading-traversing operation is executed, therefore the speed of data traversal can greatly be improved;Furthermore due to the calculate node according to point
Piece information can the data of directly self-corresponding data fragmentation storage carry out traversal reading, therefore avoid data by ETL service
The secondary transfer of device, can be further improved the speed of data traversal.
In accordance with an embodiment of the present disclosure, multiple burst informations are by the calculating section at least one other calculate node
Point is written in preset memory space after obtaining in advance, therefore can save for the calculate node and at least one other calculating
Point is directly read, and is read out-traversing operation with the burst information according to reading.
Fig. 6 A~Fig. 6 B diagrammatically illustrates reading-traversing operation flow chart according to another embodiment of the disclosure.
In accordance with an embodiment of the present disclosure, as shown in Figure 6A, above-mentioned reading-traversing operation includes operation S611~operation
S612.In operation S611, a burst information in multiple burst informations is read from preset memory space;S612 is being operated,
Burst information based on reading, traversal read the data of the corresponding data fragmentation storage of burst information.According to the implementation of the disclosure
Example, the reading-traversing operation are similar to reading-traversing operation with reference to Fig. 2 C description, and details are not described herein.
In accordance with an embodiment of the present disclosure, as shown in Figure 6B, above-mentioned reading-traversing operation for example may also include operation S613 and
Operate S614.In the case where having read a burst information in multiple burst informations from preset memory space, behaviour is executed
Make S613, the burst information of reading is set to the second state by first state;And to have read burst information corresponding in traversal
In the case where the data of data fragmentation storage, operation S614 is executed, burst information is set to the third state by the second state.Its
In, burst information is when being written preset memory space by a calculate node at least one other calculate node, i.e. quilt
It is set to first state, to indicate that burst information can be read by calculate node or at least one other calculate node;Burst information
The second state indicate that the burst information has been read, can not be read again;The third state of burst information indicates the fragment
The data of the corresponding data fragmentation storage of information, which have been traversed, have been read, which can not be read again.According to this public affairs
The embodiment opened, the reading-traversing operation are similar to reading-traversing operation with reference to Fig. 3 description, and details are not described herein.
In accordance with an embodiment of the present disclosure, of calculate node can be used by being less than in computing cluster in the quantity of multiple burst informations
In the case where number (node total number of calculate node and at least one other calculate node), multiple calculate nodes in computing cluster
(equal to the quantity of multiple burst informations) can be completed via primary above-mentioned reading-traversing operation to multiple burst informations pair
The traversal of the data for the data fragmentation storage answered is read.Certainly, the embodiment of the present disclosure does not limit specific traversal reading process,
Such as it can also be grasped by the part calculate node (less than the quantity of burst information) in multiple calculate nodes via multiple reading-traversal
The circulation of work, so that multiple burst informations are read, and the data of corresponding data fragmentation storage are traversed reading.
In accordance with an embodiment of the present disclosure, of calculate node can be used by being greater than in computing cluster in the quantity of multiple burst informations
In the case where number (node total number of calculate node and at least one other calculate node), the calculate node and at least one other
Calculate node then recycles execution reading-traversing operation.Specifically, in wherein calculate node task section from Zookeeper
Number after the child node of point has read the burst information of child node write-in, i.e., to the storage of the burst information corresponding data fragmentation
According to carrying out traversal reading;After completing traversal and reading, which is then returned from other sons of task node in Zookeeper
Node (other child nodes not being read) reads the burst information of other child nodes write-in, and to corresponding data fragmentation
The data of storage carry out traversal reading, and also circulation executes reading-traversing operation to other calculate nodes in this way, until
The burst information of all child nodes write-in of task node was read in Zookeeper, so that multiple burst informations are corresponding
Data fragmentation storage data be traversed reading.
Fig. 7 A~Fig. 7 B diagrammatically illustrates the structure of the data traversal device according to the calculate node of the embodiment of the present disclosure
Block diagram.
As shown in Figure 7 A, another aspect of the present disclosure proposes a kind of data traversal device of calculate node, the device 700 packet
It includes and obtains module 710 and first processing module 720.
Wherein, the data are stored in multiple data fragmentations.
Wherein, module 710 is obtained for obtaining multiple burst informations, each burst information corresponds to a data fragmentation.
First processing module 720 is used to execute reading-traversing operation based on multiple burst informations.
Wherein, the data traversal device of the calculate node can be with the data traversal device of at least one other calculate node
Reading-traversing operation concurrently is executed based on multiple burst informations, the corresponding data fragmentation of multiple burst informations is read with traversal
The data of storage.And the data traversal device traversal of different calculate nodes reads the data of different data fragmentation storages.It can
With understanding, the data traversal device of the calculate node be can be set in the calculate node.
In accordance with an embodiment of the present disclosure, the acquisition module 710 and first processing module 720 are respectively used to execute with reference to Fig. 2A
The operation S210 and operation S220 of description, details are not described herein.
In accordance with an embodiment of the present disclosure, as shown in Figure 7 B, the data traversal device 700 of the calculate node, which may also include, writes behaviour
Make module 730.
Wherein, preset storage is written in multiple burst informations which is used to will acquire the acquisition of module 710
Space.In accordance with an embodiment of the present disclosure, which for example can be used for executing the operation S230 described with reference to Fig. 2 B,
Details are not described herein.
Fig. 8 A~Fig. 8 D diagrammatically illustrates the structural block diagram of the first processing module according to the embodiment of the present disclosure.
As shown in Figure 8 A, first processing module 720 may include the first burst information reading unit 721 and the first data time
Go through reading unit 722.
First burst information reading unit 721 is used to execute the read operation in reading-traversing operation, i.e., deposits from preset
Read a burst information in multiple burst informations in storage space;First data traversal reading unit 722 is for executing reading-time
The traversing operation in operation, the i.e. burst information based on reading are gone through, traversal reads the corresponding data fragmentation of burst information and deposits
The data of storage.In accordance with an embodiment of the present disclosure, the first burst information reading unit 721 and the first data traversal reading unit
722 can be used for executing respectively with reference to Fig. 2 C operation S221 described and operation S222, and details are not described herein.
In accordance with an embodiment of the present disclosure, as shown in Figure 8 B, first processing module 720 can also include the first changing unit
723。
First changing unit 723 in the first burst information reading unit 721 from preset memory space for having read
In the case where a burst information in multiple burst informations, the burst information of reading is set to the second state by first state;
And the case where the first data traversal reading unit 722 traversal has read the data of the corresponding data fragmentation storage of burst information
Under, burst information is set to the third state by the second state.Wherein, burst information be written by write operation module 730 it is presetting
When memory space, it is set to first state, to indicate that the burst information can by the data traversal device of calculate node or at least
The data traversal device of one other calculate node is read;Second state of burst information indicates that the burst information has been read,
It can not be read again;The third state of burst information indicate the data of the burst information corresponding data fragmentation storage by time
It goes through and has read, which can not be read again.In accordance with an embodiment of the present disclosure, which can be used for
The operation S223 and operation S224 with reference to Fig. 3 description are executed, details are not described herein.
In accordance with an embodiment of the present disclosure, as shown in Figure 8 C, which for example may also include breakdown judge list
Member 724.
Breakdown judge unit 724 is for judging to have read one in multiple burst informations from the preset memory space
Whether the data traversal device of other calculate nodes of burst information is failed equipment.In the data traversal of other calculate nodes
In the case that device is failed equipment point, burst information that above-mentioned first changing unit 723 is also used to read the failed equipment
First state is set to by the second state, the data traversal device for calculate node or the data traversal device except the malfunctioning node
The data traversal device of outer at least one other calculate node is read.In accordance with an embodiment of the present disclosure, the breakdown judge unit
724 for example can be used for executing the operation S510 described with reference to Fig. 5, and the first changing unit 723 for example can also be used to execute with reference to Fig. 5
The operation S520 of description, details are not described herein.
In accordance with an embodiment of the present disclosure, as in fig. 8d, which it is single for example to may also include state judgement
Member 725.
The state judging unit 725 exists not for judging whether multiple burst informations are in the third state
In the case where burst information in the third state, returns and rejudge after the preset time of interval, and in multiple burst informations
In the case where being in the third state, the traversal of data is completed.In accordance with an embodiment of the present disclosure, state judging unit 725
Operation S410~operation S440 as described in can be used for executing with reference to Fig. 4, details are not described herein.
In accordance with an embodiment of the present disclosure, above-mentioned multiple data fragmentations are stored on multiple memory nodes of storage cluster,
Each memory node is stored with one or more data fragmentations.
In accordance with an embodiment of the present disclosure, the memory node information that module 710 is also used to obtain multiple data fragmentations is obtained.Example
Such as, which can also be used to execute the operation S210 ' with reference to Fig. 2 B description, and details are not described herein.
Fig. 9 diagrammatically illustrates the structural block diagram of the write operation module according to the embodiment of the present disclosure.
As shown in figure 9, write operation module 730 may include burst information arrangement units 731 and write operation unit 732.
Burst information arrangement units 731 are used to be based on preset allocation rule and memory node information, and multiple fragments are believed
Cease it is evenly distributed so that the burst information of different data fragment with identical memory node information is alternatively arranged;Write operation list
Member 732 is for being sequentially written in preset memory space for evenly distributed multiple burst informations.Then the first burst information reads single
Member 721 sequentially reads a burst information from evenly distributed multiple burst informations.In accordance with an embodiment of the present disclosure, the fragment
Information arrangement unit 731 and write operation unit 732 are respectively used to execute the operation S231 and operation S232, In with reference to Fig. 2 D description
This is repeated no more.
In accordance with an embodiment of the present disclosure, the data traversal device of calculate node may be disposed in the calculate node, the meter
The data traversal device (hereinafter referred to as management node) of operator node is by the data traversal of each calculate node in computing cluster
Device (the data traversal device of the calculate node and the data traversal device of at least one other calculate node) is using default
Election regulation elect to obtain.The embodiment of the present disclosure is not defined the preset election regulation, preset election regulation example
FastLeaderElection algorithm such as can be used.
In accordance with an embodiment of the present disclosure, computing cluster initialize for the first time or before the number of management node elected
When breaking down according to traversal device, the data traversal device of any calculate node in computing cluster is being attempted to save with management
When discovery management node is not present or is unavailable when point communication, triggering election, so that election obtains new management node.
In accordance with an embodiment of the present disclosure, calculate node can be used by being less than in computing cluster in the quantity of multiple burst informations
Data traversal number of devices (the data traversal device of the calculate node and the data traversal dress of at least one other calculate node
The device sum set) in the case where, the data traversal device of multiple calculate nodes is (equal to multiple burst informations in computing cluster
Quantity) via primary above-mentioned reading-traversing operation, the number to the storage of multiple burst informations corresponding data fragmentation can be completed
According to traversal read.Certainly, the embodiment of the present disclosure does not limit specific traversal reading process, such as can also be saved by multiple calculating
Data traversal device (less than the quantity of burst information) following via multiple reading-traversing operation of part calculate node in point
Ring, so that multiple burst informations are read, and the data of corresponding data fragmentation storage are traversed reading.
In accordance with an embodiment of the present disclosure, the number of calculate node can be used by being greater than in computing cluster in the quantity of multiple burst informations
According to number (the data traversal device of the calculate node and the data traversal of at least one other calculate node of traversal device
The device sum of device) in the case where, the data traversal device of the calculate node and the data of at least one other calculate node
Traversal device then recycles execution reading-traversing operation.Specifically, the data traversal device of a wherein calculate node from
It is after the child node of task node has read the burst information of child node write-in in Zookeeper, i.e., corresponding to the burst information
Data fragmentation storage data carry out traversal reading;After completing traversal and reading, the data traversal device of the calculate node is then
Other child nodes (other child nodes not being read) for being returned from task node in Zookeeper read other child nodes
The burst information of write-in, and traversal reading, the data time of other calculate nodes are carried out to the data of corresponding data fragmentation storage
Also circulation executes reading-traversing operation to calendar device in this way, until all child nodes of task node are written in Zookeeper
Burst information be read so that the data of multiple burst informations corresponding data fragmentation storage are traversed reading.
Acquisition module 710 according to an embodiment of the present disclosure, first processing module 720,730, first points of write operation module
Piece Information reading unit 721, the first data traversal reading unit 722, the first changing unit 723, breakdown judge unit 724, shape
It is any number of or in which any number of in state judging unit 725, burst information arrangement units 731 and write operation unit 732
At least partly function can be realized in a module.According in the module of the embodiment of the present disclosure, submodule, unit, subelement
Any one or more can be split into multiple modules to realize.According to the module of the embodiment of the present disclosure, submodule, list
Member, any one or more in subelement can at least be implemented partly as hardware circuit, such as field-programmable gate array
Arrange (FPGA), programmable logic array (PLA), system on chip, the system on substrate, the system in encapsulation, specific integrated circuit
(ASIC), it or can be realized by carrying out hardware or the firmware of any other rational method that is integrated or encapsulating to circuit,
Or it several appropriately combined is realized with any one in three kinds of software, hardware and firmware implementations or with wherein any.
Alternatively, can be at least by partly according to one or more of the module of the embodiment of the present disclosure, submodule, unit, subelement
It is embodied as computer program module, when the computer program module is run, corresponding function can be executed.
For example, obtaining module 710, first processing module 720, write operation module 730, the first burst information reading unit
721, the first data traversal reading unit 722, the first changing unit 723, breakdown judge unit 724, state judging unit 725,
Any number of may be incorporated in a module in burst information arrangement units 731 and write operation unit 732 realize, Huo Zheqi
In any one module can be split into multiple modules.Alternatively, at least portion of one or more modules in these modules
Point function can be combined at least partly function of other modules, and be realized in a module.According to the implementation of the disclosure
Example obtains module 710, first processing module 720, write operation module 730, the first burst information reading unit 721, the first data
Traverse reading unit 722, the first changing unit 723, breakdown judge unit 724, state judging unit 725, burst information arrangement
At least one of unit 731 and write operation unit 732 can at least be implemented partly as hardware circuit, such as scene can compile
Journey gate array (FPGA), programmable logic array (PLA), system on chip, the system on substrate, the system in encapsulation, dedicated collection
At circuit (ASIC), or can by circuit carry out the hardware such as any other rational method that is integrated or encapsulating or firmware come
Realize, or with any one in three kinds of software, hardware and firmware implementations or with wherein it is any it is several it is appropriately combined come
It realizes.Alternatively, obtain module 710, first processing module 720, write operation module 730, the first burst information reading unit 721,
First data traversal reading unit 722, the first changing unit 723, breakdown judge unit 724, state judging unit 725, fragment
At least one of information arrangement unit 731 and write operation unit 732 can at least be implemented partly as computer program mould
Block can execute corresponding function when the computer program module is run.
Figure 10 A~Figure 10 B diagrammatically illustrates the structural block diagram of the Second processing module according to the embodiment of the present disclosure.
Another aspect of the present disclosure proposes a kind of data traversal device of calculate node.As shown in Figure 10 A, which includes
Second processing module 1010.
The Second processing module 1010 is used to execute reading-traversing operation, multiple fragment letter based on multiple burst informations
Breath is obtained in advance by the data traversal device of a calculate node at least one other calculate node, each burst information
A corresponding data fragmentation.Wherein, the data traversal device of the calculate node can be with the number of at least one other calculate node
Multiple burst informations are concurrently based on according to traversal device and execute reading-traversing operation, and it is corresponding to read multiple burst informations with traversal
Data fragmentation storage data, the data traversal device traversal of different calculate nodes reads different data fragmentation storages
Data.
In accordance with an embodiment of the present disclosure, multiple burst informations are by a calculate node at least one other calculate node
Data traversal device obtain in advance after be written in preset memory space.A meter at least one other calculate node
The data traversal device (hereinafter referred to as management node) of operator node is the data traversal dress of each calculate node in computing cluster
(the data traversal device of the calculate node and the data traversal device of at least one other calculate node) is set using preset
Election regulation elects to obtain.The embodiment of the present disclosure is not defined the preset election regulation, and preset election regulation is for example
FastLeaderElection algorithm etc. can be used.
In accordance with an embodiment of the present disclosure, computing cluster initialize for the first time or before the management node elected occur
When failure, the data traversal device of any calculate node in computing cluster is being attempted to fill with the data traversal of management node
When finding that management node is not present or is unavailable when setting communication, triggering election, so that election obtains new management node.
In accordance with an embodiment of the present disclosure, as shown in Figure 10 A, which reads including the second burst information
Unit 1011 and the second data traversal reading unit 1012.
Second burst information reading unit 1011 is used to read one in multiple burst informations from preset memory space
Burst information, the second data traversal reading unit 1012 are used for the burst information based on reading, and it is corresponding that traversal reads burst information
Data fragmentation storage data.In accordance with an embodiment of the present disclosure, the second burst information reading unit 1011 and the second data time
Going through reading unit 1012 can be used for executing respectively with reference to Fig. 6 A operation S611 described and operation S612, and details are not described herein.
In accordance with an embodiment of the present disclosure, as shown in Figure 10 B, Second processing module 1010 can also include the second changing unit
1013。
Second changing unit 1013 in the second burst information reading unit 1011 from preset memory space for having read
In the case where a burst information in multiple burst informations, the burst information of reading is set to the second shape by first state
State;And the data of the corresponding data fragmentation storage of burst information have been read in the second data traversal reading unit 1012 traversal
In the case of, burst information is set to the third state by the second state.Wherein, burst information is in the data traversal dress by management node
It sets and is set to first state when preset memory space is written, to indicate that the burst information can be by the data traversal of calculate node
Device or the data traversal device of at least one other calculate node are read;Second state of burst information indicates the burst information
It has been be read that, can not be read again;The third state of burst information indicates the corresponding data fragmentation storage of the burst information
Data, which have been traversed, have been read, which can not be read again.In accordance with an embodiment of the present disclosure, second changing unit
Such as can be used for executing with reference to Fig. 6 B operation S613 described and operation S614, details are not described herein.
In accordance with an embodiment of the present disclosure, the number of calculate node can be used by being less than in computing cluster in the quantity of multiple burst informations
According to number (the data traversal device of the calculate node and the data traversal dress of at least one other calculate node of traversal device
The device sum set) in the case where, the data traversal device of multiple calculate nodes is (equal to multiple burst informations in computing cluster
Quantity) via primary above-mentioned reading-traversing operation, the number to the storage of multiple burst informations corresponding data fragmentation can be completed
According to traversal read.Certainly, the embodiment of the present disclosure does not limit specific traversal reading process, such as can also be saved by multiple calculating
Data traversal device (less than the quantity of burst information) following via multiple reading-traversing operation of part calculate node in point
Ring, so that multiple burst informations are read, and the data of corresponding data fragmentation storage are traversed reading.
In accordance with an embodiment of the present disclosure, calculate node can be used by being greater than in computing cluster in the quantity of multiple burst informations
Number (the data traversal device of the calculate node and the data time of at least one other calculate node of data traversal device
The device sum of calendar device) in the case where, the data traversal device of the calculate node and the number of at least one other calculate node
Execution reading-traversing operation is then recycled according to traversal device.Specifically, in wherein calculate node task from Zookeeper
After the child node of node has read the burst information of child node write-in, i.e., to the storage of the burst information corresponding data fragmentation
Data carry out traversal reading;After completing traversal and reading, the data traversal device of the calculate node is then returned from Zookeeper
Other child nodes (other child nodes not being read) of task node read the burst information of other child nodes write-in, and
Traversal reading is carried out to the data of corresponding data fragmentation storage, the data traversal device of other calculate nodes also follows in this way
Ring executes reading-traversing operation, until the burst information of all child nodes write-in of task node is read in Zookeeper
It crosses, so that the data of the corresponding data fragmentation storage of multiple burst informations are traversed reading.
Second processing module 1010 according to an embodiment of the present disclosure, write operation module 730, the second burst information read single
It is any number of or in which any number of in first 1011, second data traversal reading unit 1012 and the second changing unit 1013
At least partly function can be realized in a module.According to the module of the embodiment of the present disclosure, submodule, unit, subelement
In any one or more can be split into multiple modules to realize.According to the module of the embodiment of the present disclosure, submodule, list
Member, any one or more in subelement can at least be implemented partly as hardware circuit, such as field-programmable gate array
Arrange (FPGA), programmable logic array (PLA), system on chip, the system on substrate, the system in encapsulation, specific integrated circuit
(ASIC), it or can be realized by carrying out hardware or the firmware of any other rational method that is integrated or encapsulating to circuit,
Or it several appropriately combined is realized with any one in three kinds of software, hardware and firmware implementations or with wherein any.
Alternatively, can be at least by partly according to one or more of the module of the embodiment of the present disclosure, submodule, unit, subelement
It is embodied as computer program module, when the computer program module is run, corresponding function can be executed.
For example, Second processing module 1010, the second burst information reading unit 1011, the second data traversal reading unit
1012 and second any number of in changing unit 1013 may be incorporated in a module realize or it is therein any one
Module can be split into multiple modules.Alternatively, at least partly function of one or more modules in these modules can be with
At least partly function of other modules combines, and realizes in a module.In accordance with an embodiment of the present disclosure, second processing mould
In block 1010, the second burst information reading unit 1011, the second data traversal reading unit 1012 and the second changing unit 1013
At least one can at least be implemented partly as hardware circuit, such as field programmable gate array (FPGA), programmable patrol
Array (PLA), system on chip, the system on substrate, the system in encapsulation, specific integrated circuit (ASIC) are collected, or can be passed through
The hardware such as any other rational method that is integrated or encapsulating or firmware are carried out to realize to circuit, or with software, hardware and
Any one in three kinds of implementations of firmware several appropriately combined is realized with wherein any.Alternatively, Second processing module
1010, in the second burst information reading unit 1011, the second data traversal reading unit 1012 and the second changing unit 1013
At least one can at least be implemented partly as computer program module, can be with when the computer program module is run
Execute corresponding function.
Figure 11 diagrammatically illustrates the block diagram for being adapted for carrying out method as described above according to the embodiment of the present disclosure.Figure 11
The calculate node shown is only an example, should not function to the embodiment of the present disclosure and use scope bring any restrictions.
It as shown in figure 11, include processor 1101 according to the calculate node of the embodiment of the present disclosure 1100, it can be according to depositing
It stores up the program in read-only memory (ROM) 1102 or is loaded into random access storage device (RAM) from storage section 1108
Program in 1103 and execute various movements appropriate and processing.Processor 1101 for example may include general purpose microprocessor (example
Such as CPU), instruction set processor and/or related chip group and/or special microprocessor (for example, specific integrated circuit (ASIC)),
Etc..Processor 1101 can also include the onboard processing device for caching purposes.Processor 1101 may include for executing
According to single treatment unit either multiple processing units of the different movements of the method flow of the embodiment of the present disclosure.
In RAM 1103, it is stored with calculate node 1100 and operates required various programs and data.Processor 1101,
ROM 1102 and RAM 1103 is connected with each other by bus 1104.Processor 1101 is by executing ROM 1102 and/or RAM
Program in 1103 executes the various operations of the method flow according to the embodiment of the present disclosure.It is noted that described program can also
To be stored in one or more memories in addition to ROM 1102 and RAM 1103.Processor 1101 can also pass through execution
The program in one or more of memories is stored in execute the various operations of the method flow according to the embodiment of the present disclosure.
In accordance with an embodiment of the present disclosure, calculate node 1100 can also include input/output (I/O) interface 1105, input/
Output (I/O) interface 1105 is also connected to bus 1104.Calculate node 1100 can also include be connected to I/O interface 1105 with
It is one or more in lower component: the importation 1106 including keyboard, mouse etc.;Including such as cathode-ray tube (CRT), liquid
The output par, c 1107 of crystal display (LCD) etc. and loudspeaker etc.;Storage section 1108 including hard disk etc.;And including all
The communications portion 1109 of such as network interface card of LAN card, modem.Communications portion 1109 via such as internet net
Network executes communication process.Driver 1110 is also connected to I/O interface 1105 as needed.Detachable media 1111, such as disk,
CD, magneto-optic disk, semiconductor memory etc. are mounted on as needed on driver 1110, in order to from the meter read thereon
Calculation machine program is mounted into storage section 1108 as needed.
In accordance with an embodiment of the present disclosure, computer software journey may be implemented as according to the method flow of the embodiment of the present disclosure
Sequence.For example, embodiment of the disclosure includes a kind of computer program product comprising carry meter on a computer-readable medium
Calculation machine program, the computer program include the program code for method shown in execution flow chart.In such embodiments,
The computer program can be downloaded and installed from network by communications portion 1109, and/or from 1111 quilt of detachable media
Installation.When the computer program is executed by processor 1101, the above-mentioned function of limiting in the system of the embodiment of the present disclosure is executed.
In accordance with an embodiment of the present disclosure, system as described above, unit, module, unit etc. can pass through computer program module
To realize.
The disclosure additionally provides a kind of computer-readable medium, which, which can be in above-described embodiment, retouches
Included in the equipment/device/system stated;It is also possible to individualism, and without in the supplying equipment/device/system.On
It states computer-readable medium and carries one or more program, when said one or multiple programs are performed, realize root
According to the method flow of the embodiment of the present disclosure.
In accordance with an embodiment of the present disclosure, computer-readable medium can be computer-readable signal media or computer can
Read storage medium either the two any combination.Computer readable storage medium for example can be --- but it is unlimited
In system, device or the device of --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor, or any above combination.It calculates
The more specific example of machine readable storage medium storing program for executing can include but is not limited to: have the electrical connection, portable of one or more conducting wires
Formula computer disk, hard disk, random access storage device (RAM), read-only memory (ROM), erasable programmable read only memory
(EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device or
The above-mentioned any appropriate combination of person.In the disclosure, computer readable storage medium can be it is any include or storage program
Tangible medium, which can be commanded execution system, device or device use or in connection.And in this public affairs
In opening, computer-readable signal media may include in a base band or as carrier wave a part propagate data-signal,
In carry computer-readable program code.The data-signal of this propagation can take various forms, including but not limited to
Electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be computer-readable
Any computer-readable medium other than storage medium, the computer-readable medium can send, propagate or transmit for by
Instruction execution system, device or device use or program in connection.The journey for including on computer-readable medium
Sequence code can transmit with any suitable medium, including but not limited to: wireless, wired, optical cable, radiofrequency signal etc., or
Above-mentioned any appropriate combination.
For example, in accordance with an embodiment of the present disclosure, computer-readable medium may include above-described ROM 1102 and/or
One or more memories other than RAM 1103 and/or ROM 1102 and RAM 1103.
Flow chart and block diagram in attached drawing are illustrated according to the system of the various embodiments of the disclosure, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part of one module, program segment or code of table, a part of above-mentioned module, program segment or code include one or more
Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box
The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical
On can be basically executed in parallel, they can also be executed in the opposite order sometimes, and this depends on the function involved.Also it wants
It is noted that the combination of each box in block diagram or flow chart and the box in block diagram or flow chart, can use and execute rule
The dedicated hardware based systems of fixed functions or operations is realized, or can use the group of specialized hardware and computer instruction
It closes to realize.
It will be understood by those skilled in the art that the feature recorded in each embodiment and/or claim of the disclosure can
To carry out multiple combinations or/or combination, even if such combination or combination are not expressly recited in the disclosure.Particularly, In
In the case where not departing from disclosure spirit or teaching, the feature recorded in each embodiment and/or claim of the disclosure can
To carry out multiple combinations and/or combination.All these combinations and/or combination each fall within the scope of the present disclosure.
Embodiment of the disclosure is described above.But the purpose that these embodiments are merely to illustrate that, and
It is not intended to limit the scope of the present disclosure.Although respectively describing each embodiment above, but it is not intended that each reality
Use cannot be advantageously combined by applying the measure in example.The scope of the present disclosure is defined by the appended claims and the equivalents thereof.It does not take off
From the scope of the present disclosure, those skilled in the art can make a variety of alternatives and modifications, these alternatives and modifications should all fall in this
Within scope of disclosure.
Claims (24)
1. a kind of data traversal method of calculate node, the data are stored in multiple data fragmentations, which comprises
Multiple burst informations are obtained, the corresponding data fragmentation of each burst information;And
The multiple burst information is concurrently based on at least one other calculate node and executes reading-traversing operation, with traversal
The data of the corresponding data fragmentation storage of the multiple burst information are read,
Wherein, different calculate nodes traversal reads the data of different data fragmentation storages.
2. according to the method described in claim 1, further include:
Preset memory space is written into the multiple burst information;
Reading-the traversing operation includes:
A burst information in multiple burst informations is read from the preset memory space;And
The burst information based on reading, traversal read the data of the corresponding data fragmentation storage of the burst information.
3. according to the method described in claim 2, wherein, multiple data fragmentations are stored in multiple storage sections of storage cluster
On point, each memory node is stored with one or more data fragmentations, the method also includes:
Obtain the memory node information of multiple data fragmentations;
Preset memory space, which is written, in the multiple burst information includes:
It is based on preset allocation rule and the memory node information, the multiple burst information is evenly distributed, so as to have
The burst information of the different data fragment of identical memory node information is alternatively arranged;And
Evenly distributed the multiple burst information is sequentially written in preset memory space;
Reading a burst information in multiple burst informations from the preset memory space includes:
A burst information is sequentially read from evenly distributed the multiple burst information.
4. according to the method described in claim 2, wherein, the reading-traversing operation further include:
In the case where having read a burst information in multiple burst informations from the preset memory space, by reading
The burst information is set to the second state by first state;And
In the case where traversal has read the data that the corresponding data fragmentation of the burst information stores, by the burst information by
Second state is set to the third state;
Wherein, the first state of the burst information indicates that the burst information can be by the calculate node or at least one its
He reads calculate node;Second state of the burst information indicates that the burst information has been read, and can not be read again;
The third state of the burst information indicates that the data of the burst information corresponding data fragmentation storage have been traversed and read,
The burst information can not be read again.
5. according to the method described in claim 4, further include:
Judge other calculate nodes that a burst information in multiple burst informations is had read from the preset memory space
It whether is malfunctioning node;And
In the case where other described calculate nodes are malfunctioning node, the burst information which is read is by second
State is set to first state, and at least one other calculate node for the calculate node or in addition to the malfunctioning node is read.
6. according to the method described in claim 4, further include:
Judge whether the multiple burst information is in the third state;
In the case where there is the burst information for being not at the third state, returning and rejudging after the preset time of interval;
In the case where the multiple burst information is in the third state, the traversal of the data is completed.
7. according to the method described in claim 1, wherein:
The calculate node is elected by the calculate node and at least one other calculate node using preset election regulation
It arrives;And/or
It is greater than the node total number of the calculate node and at least one other calculate node in the quantity of the multiple burst information
In the case where, the reading-traversing operation is executed by the calculate node and/or at least one other calculate node circulation, with
The data of the corresponding data fragmentation storage of the multiple burst information are made to be traversed reading.
8. a kind of data traversal method of calculate node, the data are stored in multiple data fragmentations, which comprises
Multiple burst informations are concurrently based on at least one other calculate node and execute reading-traversing operation, are read with traversing
The data of the corresponding data fragmentation storage of the multiple burst information,
Wherein, the multiple burst information is obtained in advance by a calculate node at least one other calculate node, each
The corresponding data fragmentation of a burst information, different calculate node traversals read the data of different data fragmentation storages.
9. according to the method described in claim 8, wherein:
The multiple burst information is written default after being obtained in advance by a calculate node at least one other calculate node
Memory space in;
Reading-the traversing operation includes:
A burst information in multiple burst informations is read from the preset memory space;And
The burst information based on reading, traversal read the data of the corresponding data fragmentation storage of the burst information.
10. according to the method described in claim 9, wherein, the reading-traversing operation further include:
In the case where having read a burst information in multiple burst informations from the preset memory space, by reading
The burst information is set to the second state by first state;And
In the case where traversal has read the data that the corresponding data fragmentation of the burst information stores, by the burst information by
Second state is set to the third state;
Wherein, the first state of the burst information indicates that the burst information can be by the calculate node or at least one its
He reads calculate node;Second state of the burst information indicates that the burst information has been read, and can not be read again;
The third state of the burst information indicates that the data of the burst information corresponding data fragmentation storage have been traversed and read,
The burst information can not be read again.
11. according to the method described in claim 8, wherein:
The calculate node for obtaining the multiple burst information in advance is adopted by the calculate node and at least one other calculate node
It elects to obtain with preset election regulation;And/or
It is greater than the node total number of the calculate node and at least one other calculate node in the quantity of the multiple burst information
In the case where, the calculate node and/or at least one other calculate node circulation execute the reading-traversing operation, so that
The data of the corresponding data fragmentation storage of the multiple burst information are traversed reading.
12. a kind of data traversal device of calculate node, the data are stored in multiple data fragmentations, described device includes:
Module is obtained, multiple burst informations are obtained, the corresponding data fragmentation of each burst information;And
First processing module executes reading-traversing operation based on the multiple burst information,
Wherein, the data traversal device of the data traversal device of the calculate node and at least one other calculate node is concurrently
Reading-the traversing operation is executed based on the multiple burst information, the corresponding number of the multiple burst information is read with traversal
According to the data that fragment stores, the data traversal device traversal of different calculate nodes reads the number of different data fragmentation storages
According to.
13. device according to claim 12, further includes:
Preset memory space is written in the multiple burst information by write operation module;
The first processing module includes:
First burst information reading unit reads the letter of a fragment in multiple burst informations from the preset memory space
Breath;And
First data traversal reading unit, the burst information based on reading, traversal read the corresponding number of the burst information
The data stored according to fragment.
14. device according to claim 13, in which:
Multiple data fragmentations are stored on multiple memory nodes of storage cluster, and each memory node is stored with one or more
A data fragmentation;
The acquisition module obtains the memory node information of multiple data fragmentations;
The write operation module includes:
Burst information arrangement units are based on preset allocation rule and the memory node information, by the multiple burst information
It is evenly distributed, so that the burst information of the different data fragment with identical memory node information is alternatively arranged;And
Evenly distributed the multiple burst information is sequentially written in preset memory space by write operation unit;
The first burst information reading unit sequentially reads a fragment letter from evenly distributed the multiple burst information
Breath.
15. device according to claim 13, wherein the first processing module further includes the first changing unit:
One in multiple burst informations is had read from the preset memory space in the first burst information reading unit
In the case where burst information, the burst information of reading is set to the second state by first state;And
The data of the corresponding data fragmentation storage of the burst information have been read in the first data traversal reading unit traversal
In the case where, the burst information is set to the third state by the second state;
Wherein, the first state of the burst information indicates that the burst information can be filled by the data traversal of the calculate node
It sets or the data traversal device of at least one other calculate node is read;Second state of the burst information indicates the fragment
Information has been read, and can not be read again;The third state of the burst information indicates the corresponding data of the burst information
The data of fragment storage, which have been traversed, have been read, which can not be read again.
16. device according to claim 15, wherein the first processing module further include:
Breakdown judge unit judges to have read a burst information in multiple burst informations from the preset memory space
Whether the data traversal device of other calculate nodes is failed equipment;
First changing unit, in the case where the data traversal device of other calculate nodes is failed equipment, by this
The burst information that failed equipment is read is set to first state by the second state, so that the data traversal of the calculate node fills
It sets or the data traversal device of at least one other calculate node in addition to the malfunctioning node is read.
17. device according to claim 15, wherein the first processing module further includes state judging unit:
Judge whether the multiple burst information is in the third state;
In the case where there is the burst information for being not at the third state, returning and rejudging after the preset time of interval;
In the case where the multiple burst information is in the third state, the traversal of the data is completed.
18. device according to claim 12, in which:
Data traversal device and at least one other calculating of the data traversal device of the calculate node by the calculate node
The data traversal device of node elects to obtain using preset election regulation;And/or
In data traversal device of the quantity of the multiple burst information greater than the calculate node and at least one other calculating
In the case where the device sum of the data traversal device of node, by the data traversal device of the calculate node and/or at least one
The data traversal device circulation of other a calculate nodes executes the reading-traversing operation, so that the multiple burst information pair
The data for the data fragmentation storage answered are traversed reading.
19. a kind of data traversal device of calculate node, the data are stored in multiple data fragmentations, described device includes:
Second processing module executes reading-traversing operation based on multiple burst informations, and the multiple burst information is by least one
The data traversal device of a calculate node in other calculate nodes obtains in advance, the corresponding data of each burst information
Fragment,
Wherein, the data traversal device of the data traversal device of the calculate node and at least one other calculate node is concurrently
Reading-the traversing operation is executed based on the multiple burst information, the corresponding number of the multiple burst information is read with traversal
According to the data that fragment stores, the data traversal device traversal of different calculate nodes reads the number of different data fragmentation storages
According to.
20. device according to claim 19, in which:
The multiple burst information is preparatory by the data traversal device of a calculate node at least one other calculate node
It is written after acquisition in preset memory space;
The Second processing module includes:
Second burst information reading unit reads the letter of a fragment in multiple burst informations from the preset memory space
Breath;And
Second data traversal reading unit, the burst information based on reading, traversal read the corresponding number of the burst information
The data stored according to fragment.
21. device according to claim 20, wherein the Second processing module further includes the second changing unit:
One in multiple burst informations is had read from the preset memory space in the second burst information reading unit
In the case where burst information, the burst information of reading is set to the second state by first state;And
The data of the corresponding data fragmentation storage of the burst information have been read in the second data traversal reading unit traversal
In the case where, the burst information is set to the third state by the second state,
Wherein, the first state of the burst information indicates that the burst information can be filled by the data traversal of the calculate node
It sets or the data traversal device of at least one other calculate node is read;Second state of the burst information indicates the fragment
Information has been read, and can not be read again;The third state of the burst information indicates the corresponding data of the burst information
The data of fragment storage, which have been traversed, have been read, which can not be read again.
22. device according to claim 19, in which:
The data traversal device for obtaining the calculate node of the multiple burst information in advance, by the data traversal of the calculate node
Device and the data traversal device of at least one other calculate node elect to obtain using preset election regulation;And/or
In data traversal device of the quantity of the multiple burst information greater than the calculate node and at least one other calculating
In the case where the device sum of the data traversal device of node, the data traversal device of the calculate node and/or at least one
The data traversal device circulation of other a calculate nodes executes the reading-traversing operation, so that the multiple burst information pair
The data for the data fragmentation storage answered are traversed reading.
23. a kind of calculate node, comprising:
One or more processors;
Storage device, for storing one or more programs,
Wherein, when one or more of programs respectively by one or more of processors execute when so that it is one or
Multiple processors execute method described in any one according to claim 1~11.
24. a kind of computer-readable medium, is stored thereon with executable instruction, which hold processor when being executed by processor
Method described in row claim 1~11 any one.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810340710.7A CN110399394A (en) | 2018-04-16 | 2018-04-16 | The data traversal method, apparatus and calculate node of calculate node |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810340710.7A CN110399394A (en) | 2018-04-16 | 2018-04-16 | The data traversal method, apparatus and calculate node of calculate node |
Publications (1)
Publication Number | Publication Date |
---|---|
CN110399394A true CN110399394A (en) | 2019-11-01 |
Family
ID=68319527
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201810340710.7A Pending CN110399394A (en) | 2018-04-16 | 2018-04-16 | The data traversal method, apparatus and calculate node of calculate node |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110399394A (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110968265A (en) * | 2019-11-05 | 2020-04-07 | 北京字节跳动网络技术有限公司 | Fragmentation expansion method and device and electronic equipment |
CN112541041A (en) * | 2020-12-23 | 2021-03-23 | 平安普惠企业管理有限公司 | Data processing method, device, server and storage medium |
CN112988343A (en) * | 2021-02-05 | 2021-06-18 | 开店宝科技集团有限公司 | Batch data fragmentation method, system, computer equipment and storage medium |
CN117951184A (en) * | 2024-03-26 | 2024-04-30 | 天津南大通用数据技术股份有限公司 | Result set paging return method and system based on distributed database |
Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102801784A (en) * | 2012-07-03 | 2012-11-28 | 华为技术有限公司 | Distributed type data storing method and equipment |
CN103138981A (en) * | 2011-11-30 | 2013-06-05 | 中国移动通信集团公司 | Method and device for social network service analysis |
CN103714142A (en) * | 2013-12-25 | 2014-04-09 | 乐视网信息技术(北京)股份有限公司 | Data search method and device |
US20150188978A1 (en) * | 2013-12-30 | 2015-07-02 | Microsoft Corporation | Hierarchical organization for scale-out cluster |
CN105630808A (en) * | 2014-10-31 | 2016-06-01 | 北京奇虎科技有限公司 | Distributed file system based file reading and writing method and node server |
CN105740068A (en) * | 2016-01-27 | 2016-07-06 | 中国科学院计算技术研究所 | Big data platform oriented and memory data locality based scheduling method and system |
CN106527981A (en) * | 2016-10-31 | 2017-03-22 | 华中科技大学 | Configuration-based data fragmentation method for adaptive distributed storage system |
CN106933868A (en) * | 2015-12-30 | 2017-07-07 | 阿里巴巴集团控股有限公司 | A kind of method and data server for adjusting data fragmentation distribution |
CN107193857A (en) * | 2016-03-15 | 2017-09-22 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus traveled through for database |
US20180039673A1 (en) * | 2016-08-05 | 2018-02-08 | International Business Machines Corporation | Distributed graph databases that facilitate streaming data insertion and low latency graph queries |
CN107870829A (en) * | 2016-09-24 | 2018-04-03 | 华为技术有限公司 | A kind of distributed data restoration methods, server, relevant device and system |
-
2018
- 2018-04-16 CN CN201810340710.7A patent/CN110399394A/en active Pending
Patent Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103138981A (en) * | 2011-11-30 | 2013-06-05 | 中国移动通信集团公司 | Method and device for social network service analysis |
CN102801784A (en) * | 2012-07-03 | 2012-11-28 | 华为技术有限公司 | Distributed type data storing method and equipment |
CN103714142A (en) * | 2013-12-25 | 2014-04-09 | 乐视网信息技术(北京)股份有限公司 | Data search method and device |
US20150188978A1 (en) * | 2013-12-30 | 2015-07-02 | Microsoft Corporation | Hierarchical organization for scale-out cluster |
CN105630808A (en) * | 2014-10-31 | 2016-06-01 | 北京奇虎科技有限公司 | Distributed file system based file reading and writing method and node server |
CN106933868A (en) * | 2015-12-30 | 2017-07-07 | 阿里巴巴集团控股有限公司 | A kind of method and data server for adjusting data fragmentation distribution |
CN105740068A (en) * | 2016-01-27 | 2016-07-06 | 中国科学院计算技术研究所 | Big data platform oriented and memory data locality based scheduling method and system |
CN107193857A (en) * | 2016-03-15 | 2017-09-22 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus traveled through for database |
US20180039673A1 (en) * | 2016-08-05 | 2018-02-08 | International Business Machines Corporation | Distributed graph databases that facilitate streaming data insertion and low latency graph queries |
CN107870829A (en) * | 2016-09-24 | 2018-04-03 | 华为技术有限公司 | A kind of distributed data restoration methods, server, relevant device and system |
CN106527981A (en) * | 2016-10-31 | 2017-03-22 | 华中科技大学 | Configuration-based data fragmentation method for adaptive distributed storage system |
Non-Patent Citations (2)
Title |
---|
JULIA WARNKE 等: "" An energy-aware bioinformatics application for assembling short reads in high performance computing systems"", 2012 INTERNATIONAL CONFERENCE ON HIGH PERFORMANCE COMPUTING & SIMULATION (HPCS) * |
宋杰 等: ""大数据分析的分布式MOLAP技术"", 《软件学报》, vol. 25, no. 4 * |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110968265A (en) * | 2019-11-05 | 2020-04-07 | 北京字节跳动网络技术有限公司 | Fragmentation expansion method and device and electronic equipment |
CN110968265B (en) * | 2019-11-05 | 2023-08-08 | 北京字节跳动网络技术有限公司 | Method and device for expanding capacity by fragments and electronic equipment |
CN112541041A (en) * | 2020-12-23 | 2021-03-23 | 平安普惠企业管理有限公司 | Data processing method, device, server and storage medium |
CN112988343A (en) * | 2021-02-05 | 2021-06-18 | 开店宝科技集团有限公司 | Batch data fragmentation method, system, computer equipment and storage medium |
CN117951184A (en) * | 2024-03-26 | 2024-04-30 | 天津南大通用数据技术股份有限公司 | Result set paging return method and system based on distributed database |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20210365199A1 (en) | Technologies for coordinating disaggregated accelerator device resources | |
CN110399394A (en) | The data traversal method, apparatus and calculate node of calculate node | |
US8984085B2 (en) | Apparatus and method for controlling distributed memory cluster | |
US20210255915A1 (en) | Cloud-based scale-up system composition | |
CN103970520B (en) | Method for managing resource, device and architecture system in MapReduce frameworks | |
US20130212594A1 (en) | Method of optimizing performance of hierarchical multi-core processor and multi-core processor system for performing the method | |
CN109032796B (en) | Data processing method and device | |
CN108874528A (en) | Distributed task scheduling storage system and distributed task scheduling storage/read method | |
US9811287B2 (en) | High-performance hash joins using memory with extensive internal parallelism | |
WO2019001319A1 (en) | Quasi-agentless cloud resource management | |
CN109697120A (en) | Method, electronic equipment for application migration | |
CN109828790B (en) | Data processing method and system based on Shenwei heterogeneous many-core processor | |
CN107273195A (en) | A kind of batch processing method of big data, device and computer system | |
US20240160488A1 (en) | Dynamic microservices allocation mechanism | |
CN107357630A (en) | A kind of method, apparatus and storage medium for realizing that virtual machine is synchronous | |
CN110213338A (en) | A kind of clustering acceleration calculating method and system based on cryptographic calculation | |
CN110162410A (en) | A kind of message treatment method and device | |
CN109597810A (en) | A kind of task cutting method, device, medium and electronic equipment | |
WO2016085642A1 (en) | Generating approximate usage measurements for shared cache memory systems | |
US9690728B1 (en) | Burst buffer appliance comprising multiple virtual machines | |
CN104750614B (en) | Method and apparatus for managing memory | |
CN110399393A (en) | Data processing method, device, medium and electronic equipment | |
EP4198728A1 (en) | Method and apparatus to perform workload management in a disaggregated computing system | |
CN112052104A (en) | Message queue management method based on multi-computer-room realization and electronic equipment | |
CN116339989A (en) | Mixed part server, resource management method and device of mixed part server |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |