CN105824810B - A kind of Hadoop group system and data processing method - Google Patents
A kind of Hadoop group system and data processing method Download PDFInfo
- Publication number
- CN105824810B CN105824810B CN201510001684.1A CN201510001684A CN105824810B CN 105824810 B CN105824810 B CN 105824810B CN 201510001684 A CN201510001684 A CN 201510001684A CN 105824810 B CN105824810 B CN 105824810B
- Authority
- CN
- China
- Prior art keywords
- node
- data
- host
- service request
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The embodiment of the invention discloses a kind of Hadoop group system and data processing methods；The system comprises: gateway node, at least two host nodes and at least two are from node；The gateway node after the service request for receiving client, broadcasts the service request at least two host node by broadcasting link, and will receive the host node of response signaling at first as the first host node；It is also used to establish the connection between the client and first host node；The host node judges whether to meet the service request according to the resource situation of itself after receiving the service request that the gateway node is broadcasted；After determining that the resource situation of itself meets the service request, Xiang Suoshu gateway node sends response signaling；It is that the service request of the client distributes the slave node of meet demand, instruction is described to execute the service request from node after being also used to establish connection by the gateway node and the client.
Description
Technical field
The present invention relates to distributed file system fields, and in particular to a kind of Hadoop group system and data processing side
Method.
Background technique
The Hadoop framework framework important as one kind of cloud computing is used widely.Hadoop is by distributed field system
It unites (HDFS, Hadoop Distribute File System) and two major parts of MapReduce forms.Wherein, HDFS master
Storage is provided for mass data；And MapReduce is mainly used for calculating mass data.Fig. 1 is distributed field system
The configuration diagram of system；As shown in Figure 1, including: client (Client) 11, namenode in distributed file system
(NameNode) 12 and multiple back end (Datanode) 13.Distributed file system, will be literary by deblocking to be stored
Part block is stored on some back end (Datanode) 13, and multiple backups of blocks of files dispersion is stored in other differences
Back end (Datanode) 13, with improve data storage reliability.And the distributed intelligence about blocks of files is then saved
The distributing position of blocks of files is inquired when so as to the access of client (Client) 11 in single namenode (NameNode) 12.
In this case, 12 single-point situation of namenode (NameNode) is brought the high availability of distributed file system unknowable
Hidden danger.
And MapReduce mainly provides distributed computing services.MapReduce structure by JobTracker and
TaskTracker two parts composition.Wherein, JobTracker is equivalent to namenode shown in Fig. 1 (NameNode) 12
Function, TaskTracker are equivalent to the function of back end shown in Fig. 1 (Datanode) 13.JobTracker is mainly used
It in the service request of response Client, sends the computation requests decomposition of user at each TaskTracker and executes calculating, and will
Calculated result, which summarizes, returns to Client.In MapReduce, JobTracker is similarly single-point situation, a large amount of when occurring
When the computation requests of Client, the bottleneck of system integrity service certainly will be become.
Summary of the invention
To solve existing technical problem, the embodiment of the present invention provides a kind of Hadoop group system and data processing
Method can disperse the pressure of single host node, the safety of lifting system.
In order to achieve the above objectives, the technical solution of the embodiment of the present invention is achieved in that
The embodiment of the invention provides a kind of Hadoop group system, the system comprises: gateway nodes, at least two main
Node and at least two is from node；Wherein,
The gateway node after the service request for receiving client, is broadcasted the service by broadcasting link and is asked
It asks at least two host node, and the host node of response signaling will be received at first as the first host node；It is also used to establish
Connection between the client and first host node；
The host node, after receiving the service request that the gateway node is broadcasted, according to the resource of itself
Situation judges whether to meet the service request；After determining that the resource situation of itself meets the service request, Xiang Suoshu gateway
Node sends response signaling；It is the client after being also used to establish connection by the gateway node and the client
Service request distributes the slave node of meet demand, and instruction is described to execute the service request from node；
It is described from node, execute the service request for the instruction based on the host node.
In above scheme, the service request is data storage request；Correspondingly,
The host node, after receiving the data storage request that the gateway node is broadcasted, according to storage
Metadata information judges whether the residual memory space of the slave node of self-management meets the data storage request；Determine itself
After the residual memory space of the slave node of management meets the data storage request, Xiang Suoshu gateway node sends response signaling；
Wherein, the metadata information includes data from the storage information in node；Be also used to by the gateway node with it is described
After client establishes connection, meet the slave node of the data storage request for data storage request distribution, will it is described from
The information of node is sent to client by the gateway node；So that the client is carried out according to described from the information of node
Data storage；
It is described from node, sent out for receiving and storing the client by data transmission link by the gateway node
The data sent.
In above scheme, the service request is data removal request；Correspondingly,
The host node retrieves storage after receiving the data removal request that the gateway node is broadcasted
Metadata information, judges whether the data store in the slave node of self-management；Determine the data in self-management
After storing in node, Xiang Suoshu gateway node sends response signaling；Wherein, the metadata information includes data from node
In storage information and the metadata information other host nodes backup information；Be also used to by the gateway node with
After the client establishes connection, data described in the slave knot removal of the data are stored with by broadcasting link instruction, and
Indicate that other host nodes for the backup information for being stored with the metadata information update the metadata information of itself storage；
It is described from node, delete the data for the instruction based on the host node.
In above scheme, the service request is data computation requests；Correspondingly,
The host node retrieves storage after receiving the data computation requests that the gateway node is broadcasted
Metadata information, judges whether the data store in the slave node of self-management；Determine the data in self-management
After storing in node, Xiang Suoshu gateway node sends response signaling；Wherein, the metadata information includes data from node
In storage information；After being also used to establish connection by the gateway node and the client, calculated to the data are met
Data computation requests described in the slave node broadcasts of request, and broadcast instruction and receive the slave node of response signaling at first as calculating section
Point；It is also used to the calculated result that the calculate node is sent being sent to the client by the gateway node.
It is described from node, for executing calculating operation according to the data computation requests, obtaining as after the calculate node
Obtain calculated result；The calculated result is broadcasted to the host node.
It is described from node in above scheme, it is also used to obtain the storage condition of itself by preset rules, by the storage feelings
Condition is broadcasted to the host node；
The host node, be also used to receive it is described after the storage condition of node broadcasts, determine it is described from node be from
After the slave node of barrel reason, the metadata information that is updated storage according to the storage condition；Wherein, the metadata information includes
Data are from the storage information in node；It is also used to broadcast the metadata information to other host nodes, so that storage is
Other host nodes for stating metadata information backup update the metadata information backup.
The embodiment of the invention also provides a kind of data processing method, the data processing method is applied to include gateway section
Point, at least two host nodes and at least two are from the Hadoop group system of node；The described method includes:
After the gateway node receives the service request of client, the service request is broadcasted to institute by broadcasting link
State at least two host nodes；
After the host node receives the service request, judged whether to meet the service according to the resource situation of itself
Request；After determining that the resource situation of itself meets the service request, Xiang Suoshu gateway node sends response signaling；
The gateway node will receive the host node of response signaling as the first host node at first, establish the client with
Connection between first host node；
First host node is that the service request of the client distributes the slave node of meet demand, and instruction is described from section
Point executes the service request.
In above scheme, the service request is data storage request；Then the host node receives the service request
Afterwards, judged whether to meet the service request according to the resource situation of itself；Determine that the resource situation of itself meets the service
After request, Xiang Suoshu gateway node sends response signaling, comprising:
After the host node receives the data storage request, self-management is judged according to the metadata information of storage
Whether meet the data storage request from the residual memory space of node；Determine that the remaining storage of the slave node of self-management is empty
Between meet the data storage request after, Xiang Suoshu gateway node send response signaling；Wherein, the metadata information includes number
According to from the storage information in node；
Correspondingly, first host node is that the service request of the client distributes the slave node of meet demand, instruction
It is described to execute the service request from node, comprising:
First host node be the client data storage request distribution meet the data storage request from
The information from node is sent to client by the gateway node by node so that the client according to it is described from
The information of node carries out data storage；
It is described to receive and store what the client was sent by the gateway node by data transmission link from node
Data.
In above scheme, the service request is data removal request；Then the host node receives the service request
Afterwards, judged whether to meet the service request according to the resource situation of itself；Determine that the resource situation of itself meets the service
After request, Xiang Suoshu gateway node sends response signaling, comprising:
After the host node receives the data removal request, the metadata information of storage is retrieved, judges the data
Whether stored in the slave node of self-management；After determining that the data store in the slave node of self-management, to the net
Artis sends response signaling；Wherein, the metadata information includes data from the storage information and the member in node
Backup information of the data information in other host nodes；
Correspondingly, first host node is that the service request of the client distributes the slave node of meet demand, instruction
It is described to execute the service request from node, comprising:
First host node is stored with data described in the slave knot removal of the data by broadcasting link instruction, and
Indicate that other host nodes for the backup information for being stored with the metadata information update the metadata information of itself storage；
It is described that the data are deleted based on the instruction of the host node from node.
In above scheme, the service request is data computation requests；Then the host node receives the service request
Afterwards, judged whether to meet the service request according to the resource situation of itself；Determine that the resource situation of itself meets the service
After request, Xiang Suoshu gateway node sends response signaling, comprising:
After the host node receives the data computation requests, the metadata information of storage is retrieved, judges the data
Whether stored in the slave node of self-management；After determining that the data store in the slave node of self-management, to the net
Artis sends response signaling；Wherein, the metadata information includes data from the storage information in node；
Correspondingly, first host node is that the service request of the client distributes the slave node of meet demand, instruction
It is described to execute the service request from node, comprising:
First host node is and wide to data computation requests described in the slave node broadcasts for meeting the data computation requests
It broadcasts instruction and receives the slave node of response signaling at first as calculate node；
Data computation requests described in the calculate node execute calculating operation, obtain calculated result；It is wide to the host node
Broadcast the calculated result.
In above scheme, the method also includes: the storage condition that itself is obtained from node by preset rules, by institute
Storage condition is stated to broadcast to the host node；
The host node receive it is described after the storage condition of node broadcasts, determine it is described from node be self-management
After node, the metadata information that is updated storage according to the storage condition；Wherein, the metadata information include data from
Storage information in node；
The host node broadcasts the metadata information to other host nodes, so as to be stored with the metadata information
Other host nodes of backup information update the metadata information.
Hadoop group system and data processing method provided in an embodiment of the present invention, the system comprises: gateway node,
At least two host nodes and at least two are from node；In this way, having dispersed original single host node by least two host nodes
Pressure, avoid single host node and the abnormal hidden danger to make system be unable to run, the safety of lifting system occur；And
Host node can need linear expansion according to system.On the other hand, pass through relatively independent broadcast inside Hadoop group system
Link is connected with data transmission link, is kept apart data and broadcast signaling by gateway node, avoids signaling broadcast logarithm
According to the influence of transmission, the data-handling capacity of each node in system is enhanced, improves the data throughout of system.
Detailed description of the invention
Fig. 1 is the configuration diagram of distributed file system；
Fig. 2 is the composed structure schematic diagram of the Hadoop group system of the embodiment of the present invention；
Fig. 3 is the flow diagram of the data processing method of the embodiment of the present invention；
Fig. 4 is the first detailed process schematic diagram of the data processing method of the embodiment of the present invention；
Fig. 5 is the second detailed process schematic diagram of the data processing method of the embodiment of the present invention；
Fig. 6 is the third detailed process schematic diagram of the data processing method of the embodiment of the present invention.
Specific embodiment
In various embodiments of the present invention, it in order to avoid the prior art problem, proposes a kind of based on broadcast chain
The Hadoop group system of more host nodes of road and data transmission link.Host node therein is equivalent to the namenode in HDFS
(NameNode) JobTracker or in MapReduce；From node be equivalent to back end (Datanode) in HDFS or
TaskTracker in MapReduce.The quantity of host node is at least two, has dispersed the pressure of original single host node,
It avoids single host node and the abnormal hidden danger to make system be unable to run, the safety of lifting system occurs；And host node
Linear expansion can be needed according to system.On the other hand, inside Hadoop group system by relatively independent broadcasting link and
Data transmission link connection avoids signaling and broadcasts the influence transmitted to data, enhances the data processing of each node in system
Ability improves the data throughout of system.
With reference to the accompanying drawing and specific embodiment the present invention is described in further detail.
Embodiment one
The embodiment of the invention provides a kind of Hadoop group systems.Fig. 2 is the Hadoop cluster system of the embodiment of the present invention
The composed structure schematic diagram of system；As shown in Fig. 2, the system comprises: gateway node 24, at least two host nodes 22 and at least two
It is a from node 23；The every two from node 23 of the gateway node 24, at least two host node 22 and described at least two
Broadcasting link and data transmission link are established between node respectively；Wherein,
The gateway node 24 after the service request for receiving client 21, broadcasts the clothes by broadcasting link
Business is requested at least two host node 22, and the host node 22 that will receive response signaling at first is used as the first host node 22；
It is also used to establish the connection between the client 21 and first host node 22；
The host node 22, after receiving the service request that the gateway node 24 is broadcasted, according to itself
Resource situation judges whether to meet the service request；After determining that the resource situation of itself meets the service request, Xiang Suoshu
Gateway node 24 sends response signaling；After being also used to establish connection with the client 21 by the gateway node 24, for institute
The slave node 23 of the service request distribution meet demand of client 21 is stated, instruction is described to execute the service request from node 23；
It is described from node 23, execute the service request for the instruction based on the host node 22.
In the present embodiment, establishing inside the Hadoop group system has broadcasting link and data transmission link；Wherein,
The data transmission link is specially the data transmission of one-to-one connection object, i.e. a node is to another node-node transmission number
According to；The broadcasting link is specially one-to-many broadcasting link, i.e. a node broadcasts signaling, and multiple nodes can receive described
Signaling.In the present embodiment, the gateway node 24 is opposite with signaling by data for distinguishing transmission data or signalling
Keep apart, enables and the host node 22 and described more effectively handle data service from node 23.
Wherein, the gateway node 24 receives the service request of client 21 by data transmission link, passes through broadcast
Service request described in link broadcast is to all host nodes 22；All host nodes 22 respectively judge whether according to the resource situation of itself
Meet the service request；After determining that the resource situation of itself meets the service request, by the broadcast signaling to described
Gateway node 24 sends response signaling；The gateway node 24 regard the host node 22 for receiving response signaling at first as the first main section
Point 22, is sent to the client 21 by data transmission link for the information of first host node 22, so that the client
Connection is established with first host node 22 in end 21；First host node 22 is in the confirmation broadcast for receiving the gateway node 24
It is the slave node 23 of the service request distribution meet demand of the client 21, and by described in broadcasting link instruction after signaling
The service request is executed from node 23.
Specifically, the service request of the client 21 may include a variety of service requests, below with specific service request
It is described in detail for type.
In the first embodiment, the service request is data storage request；Correspondingly,
The gateway node 24 after the data storage request for receiving client 21, broadcasts institute by broadcasting link
Data storage request is stated at least two host node 22, and the host node 22 that will receive response signaling at first is main as first
Node 22；It is also used to establish the connection between the client 21 and first host node 22；
The host node 22, after receiving the data storage request that the gateway node 24 is broadcasted, according to depositing
The metadata information of storage judges whether the residual memory space of the slave node 23 of self-management meets the data storage request；Really
Determine the slave node 23 of self-management residual memory space meet the data storage request after, Xiang Suoshu gateway node 24 send
Response signaling；Wherein, the metadata information includes data from the storage location information in node 23；It is also used to by described
After gateway node 24 and the client 21 establish connection, meet the data storage request for data storage request distribution
Slave node 23, the information from node 23 is sent to client 21 by the gateway node 24；So that the client
End 21 carries out data storage from the information of node 23 according to described；
It is described from node 23, pass through the gateway section for receiving and storing the client 21 by data transmission link
The data that point 24 is sent.
Specifically, the gateway node 24 receives the data storage request of client 21 by data transmission link, lead to
It crosses broadcasting link and broadcasts the data storage request to all host nodes 22；It is stored with metadata information in all host nodes 22,
All data storage conditions from node 23 that host node 22 manages store in the metadata information, specifically, described
Metadata information includes data from the storage location information in node 23；All host nodes 22 are according to the metadata information of storage
Judge whether the residual memory space of the slave node 23 of self-management meets the data storage request；Determine self-management from
After the residual memory space of node 23 meets the data storage request, according to the busy degree of itself, at once or delay passes through
The broadcast signaling sends response signaling to the gateway node 24；The gateway node 24 will receive the master of response signaling at first
Node 22 is used as the first host node 22, and the information of first host node 22 is sent to the client by data transmission link
End 21, so that the client 21 and first host node 22 establish connection；First host node 22 is receiving the net
It is that the data storage request of the client 21 distributes the slave node 23 of meet demand after the confirmation broadcast signaling of artis 24,
The residual memory space of the slave node 23 distributed is greater than the capacity of data to be stored；First host node 22 will be described from section
23 information of point are sent to the gateway node 24 by broadcasting link, and will by data transmission link by the gateway node 24
It is described to be sent to client 21 from 23 information of node, so that the client 21 passes through system according to the information from node 23
Interior data transmission link carries out data storage；After the completion of pending data storage, the client 21 is sent out to the gateway node 24
Storage is sent to complete message；The storage is completed message and is broadcasted by broadcast signaling to the described first main section by the gateway node 24
Point 22；The metadata information that first host node 22 updates storage, and by updated metadata information to as backup
Second host node, 22 broadcast transmission；Second host node 22 again by the metadata information to it is another as backup third master
22 broadcast transmission of node；After the completion of the third host node 22 saves, signaling is completed in the broadcast of the second host node of Xiang Suoshu 22；Institute
It states the second host node 22 and broadcasts completion signaling to first host node 22；First host node 22 is to the gateway node 24
Signaling is completed in broadcast, is sent completely message to the client 21 by data transmission link from the gateway node 24, so far
Confirm that data storage is completed.
In the second embodiment, the service request is data removal request；Correspondingly,
The gateway node 24 after the data removal request for receiving client 21, broadcasts institute by broadcasting link
Data removal request is stated at least two host node 22, and the host node 22 that will receive response signaling at first is main as first
Node 22；It is also used to establish the connection between the client 21 and first host node 22；
The host node 22, after receiving the data removal request that the gateway node 24 is broadcasted, retrieval is deposited
The metadata information of storage, judges whether the data store in the slave node 23 of self-management；Determine the data at itself
After storing in the slave node 23 of management, Xiang Suoshu gateway node 24 sends response signaling；Wherein, the metadata information includes number
According to from node 23 storage location information and the metadata information other host nodes 22 backup information；It is also used to
After establishing connection with the client 21 by the gateway node 24, by broadcasting link instruction be stored with the data from
Node 23 deletes the data, and instruction is stored with other host nodes 22 update of the backup information of the metadata information certainly
The metadata information of body storage；
It is described from node 23, delete the data for the instruction based on the host node 22.
Specifically, the gateway node 24 receives the data removal request of client 21 by data transmission link, lead to
It crosses broadcasting link and broadcasts the data removal request to all host nodes 22；It is stored with metadata information in all host nodes 22,
Host node 22 manage it is all from the data storage condition of node 23 and the backup instances of other host nodes 22 in the member
Stored in data information, specifically, the metadata information include data from node 23 storage location information and institute
Metadata information is stated in the backup information of other host nodes 22；The metadata information of all retrieval of host node 22 storages, judges institute
State whether data store in the slave node 23 of self-management；Determine that the data store in the slave node 23 of self-management
Afterwards, according to the busy degree of itself, at once or delay passes through the broadcast signaling and sends response signaling to the gateway node 24；
The gateway node 24 regard the host node 22 for receiving response signaling at first as the first host node 22, by first host node 22
Information the client 21 is sent to by data transmission link so that the client 21 is built with first host node 22
Vertical connection；First host node 22 after the confirmation broadcast signaling for receiving the gateway node 24, by broadcasting link to its
He sends signaling by host node 22, so that other host nodes 22 for being stored with the metadata information backup delete the metadata letter
Information relevant to the data of the requirement deletion of client 21 in breath backup；Other described host nodes 22 are broadcasted by broadcasting link
After signaling confirmation message is deleted, first host node 22 passes through the slave node that broadcasting link is managed to first host node 22
23 broadcast signalings require the data deleted to delete the client 21, if not executing and appointing without the data from node 23
What is operated；If deleting the data there are the data from node 23, and response signaling is broadcasted to institute by broadcasting link
The first host node 22 is stated to determine that data have been deleted；First host node 22 broadcasts to the gateway node 24 and completes signaling,
Message is sent completely to the client 21 by data transmission link from the gateway node 24, so far confirms that data have been deleted
At.
In the third embodiment, the service request is data computation requests；Correspondingly,
The gateway node 24 after the data removal request for receiving client 21, broadcasts institute by broadcasting link
Data removal request is stated at least two host node 22, and the host node 22 that will receive response signaling at first is main as first
Node 22；It is also used to establish the connection between the client 21 and first host node 22；
The host node 22, after receiving the data computation requests that the gateway node 24 is broadcasted, retrieval is deposited
The metadata information of storage, judges whether the data store in the slave node 23 of self-management；Determine the data at itself
After storing in the slave node 23 of management, Xiang Suoshu gateway node 24 sends response signaling；Wherein, the metadata information includes number
According to from the storage location information in node 23；It is also used to establish connection with the client 21 by the gateway node 24
Afterwards, the data computation requests are broadcasted to the slave node 23 for meeting the data computation requests, and broadcasts instruction and receives sound at first
Answer the slave node 23 of signaling as calculate node；The calculated result for being also used to send the calculate node passes through the gateway section
Point 24 is sent to the client 21.
It is described after being used as the calculate node, to execute calculating operation according to the data computation requests from node 23,
Obtain calculated result；The calculated result is broadcasted to the host node 22.
Specifically, the gateway node 24 receives the data computation requests of client 21 by data transmission link, lead to
It crosses broadcasting link and broadcasts the data computation requests to all host nodes 22；It is stored with metadata information in all host nodes 22,
All data storage conditions from node 23 that host node 22 manages store in the metadata information, specifically, described
Metadata information includes data from the storage location information in node 23；The metadata letter of all retrieval of host node 22 storages
Breath, judges whether the data store in the slave node 23 of self-management；Determine the data in the slave node of self-management
After storing in 23, according to the busy degree of itself, at once or delay is sent by the broadcast signaling to the gateway node 24
Response signaling；The host node 22 that the gateway node 24 will receive response signaling at first is used as the first host node 22, by described the
The information of one host node 22 is sent to the client 21 by data transmission link, so that the client 21 and described first
Host node 22 establishes connection；First host node 22 is after the confirmation broadcast signaling for receiving the gateway node 24, by wide
It broadcasts chain road direction and is stored with all from 23 broadcast data computation requests of node of the data；It is described to be stored with all of the data
The data computation requests are received from node 23, response signaling is broadcasted by broadcasting link；First host node 22 will most
The slave node 23 of response signaling is first received as calculate node, Xiang Suoshu calculate node broadcast acknowledgements signaling；The calculate node
Data calculating is carried out according to the data computation requests of the client 21, obtains calculated result, the calculated result is passed through wide
Link broadcast is broadcast to first host node 22；First host node 22 broadcasts the calculating knot to the gateway node 24
Fruit sends the calculated result to the client 21 by data transmission link from the gateway node 24, so far confirms number
It is completed according to calculating.
It is described from node 23 based on above embodiment, it is also used to obtain the storage condition of itself by preset rules, by institute
Storage condition is stated to broadcast to the host node 22；
The host node 22, is also used to receive described after the storage condition that node 23 is broadcasted, and determines described from node
After 23 is the slave nodes 23 of self-management, the metadata information that is updated storage according to the storage condition；Wherein, the metadata
Information includes data from the storage location information in node 23；It is also used to broadcast the metadata information to other host nodes
22, so that other host nodes 22 for being stored with the metadata information backup update the metadata information backup.
Wherein, the preset rules can be clocking discipline, such as described obtain itself from the daily fixed time of node 23
Storage condition, the storage condition is broadcasted to the host node 22；Or the preset rules can be timer rule,
Such as the timer setting time is n hours, i.e., the described storage condition for obtaining itself every described n hours from node 23,
The storage condition is broadcasted to the host node 22.
In the present embodiment, not only the data of client are stored in multiple from node, different from node appearance to avoid one
Often so as to cause loss of data the case where；And the metadata information in host node also backs up in other multiple host nodes, with
It avoids a host node from exception occur to cause confusion so as to cause the distribution and management of system resource.
Using the technical solution of the embodiment of the present invention, a kind of how main section based on broadcasting link and data transmission link is proposed
The Hadoop group system of point.Host node therein is equivalent in the namenode in HDFS (NameNode) or MapReduce
JobTracker；From the back end (Datanode) or MapReduce that node is equivalent in HDFS
TaskTracker.The quantity of host node is at least two, has dispersed the pressure of original single host node, has avoided single master
Node occurs abnormal to make the hidden danger that system is unable to run, the safety of lifting system；And host node can be according to system
Need linear expansion.On the other hand, pass through relatively independent broadcasting link and data transmission link inside Hadoop group system
Connection avoids signaling and broadcasts the influence transmitted to data, enhances the data-handling capacity of each node in system, improves and is
The data throughout of system.
Embodiment two
The embodiment of the invention also provides a kind of data processing method, the data method is applied to described in embodiment one
In Hadoop group system.Hadoop group system includes: gateway node, at least two host nodes and at least two from node；
Fig. 3 is the flow diagram of the data processing method of the embodiment of the present invention；As shown in figure 3, the data processing method includes:
Step 301: after the gateway node receives the service request of client, the service being broadcasted by broadcasting link
It requests at least two host node.
In the present embodiment, the service request may include it is a variety of, be specifically as follows data storage request, data deletion ask
It asks or data computation requests.Correspondingly, described broadcast the service request at least two host node by broadcasting link,
It include: that the data storage request, data removal request or data computation requests are broadcasted to described at least two by broadcasting link
A host node.
Step 302: after the host node receives the service request, judging whether to meet according to the resource situation of itself
The service request；After determining that the resource situation of itself meets the service request, Xiang Suoshu gateway node sends response signaling.
In the present embodiment, after the host node determines that the resource situation of itself meets the service request, according to itself
Busy situation at once or postpones to send response signaling to the gateway node.
Specifically, when the service request is data storage request, after the host node receives the service request,
Judged whether to meet the service request according to the resource situation of itself；Determine that the resource situation of itself meets the service request
Afterwards, Xiang Suoshu gateway node sends response signaling, comprising:
After the host node receives the data storage request, self-management is judged according to the metadata information of storage
Whether meet the data storage request from the residual memory space of node；Determine that the remaining storage of the slave node of self-management is empty
Between meet the data storage request after, Xiang Suoshu gateway node send response signaling；Wherein, the metadata information includes number
According to from the storage information in node.
When the service request is data removal request, after the host node receives the service request, according to certainly
The resource situation of body judges whether to meet the service request；After determining that the resource situation of itself meets the service request, to
The gateway node sends response signaling, comprising:
After the host node receives the data removal request, the metadata information of storage is retrieved, judges the data
Whether stored in the slave node of self-management；After determining that the data store in the slave node of self-management, to the net
Artis sends response signaling；Wherein, the metadata information includes data from the storage information and the member in node
Backup information of the data information in other host nodes.
When the service request is data computation requests, after the host node receives the service request, according to certainly
The resource situation of body judges whether to meet the service request；After determining that the resource situation of itself meets the service request, to
The gateway node sends response signaling, comprising:
After the host node receives the data computation requests, the metadata information of storage is retrieved, judges the data
Whether stored in the slave node of self-management；After determining that the data store in the slave node of self-management, to the net
Artis sends response signaling；Wherein, the metadata information includes data from the storage information in node.
Step 303: the gateway node will receive the host node of response signaling as the first host node, described in foundation at first
Connection between client and first host node.
Step 304: first host node is that the service request of the client distributes the slave node of meet demand, instruction
It is described to execute the service request from node.
Specifically, first host node is the clothes of the client when the service request is data storage request
The slave node of business request distribution meet demand, instruction is described to execute the service request from node, comprising:
First host node be the client data storage request distribution meet the data storage request from
The information from node is sent to client by the gateway node by node so that the client according to it is described from
The information of node carries out data storage；
It is described to receive and store what the client was sent by the gateway node by data transmission link from node
Data.
When the service request is data removal request, first host node is the service request point of the client
Slave node with meet demand, instruction is described to execute the service request from node, comprising:
First host node is stored with data described in the slave knot removal of the data by broadcasting link instruction, and
Indicate that other host nodes for the backup information for being stored with the metadata information update the metadata information of itself storage；
It is described that the data are deleted based on the instruction of the host node from node.
When the service request is data computation requests, first host node is the service request point of the client
Slave node with meet demand, instruction is described to execute the service request from node, comprising:
First host node is and wide to data computation requests described in the slave node broadcasts for meeting the data computation requests
It broadcasts instruction and receives the slave node of response signaling at first as calculate node；
Data computation requests described in the calculate node execute calculating operation, obtain calculated result；It is wide to the host node
Broadcast the calculated result.
Another embodiment of the data processing method according to an embodiment of the present invention, the method also includes: it is described from
Node is obtained the storage condition of itself by preset rules, and the storage condition is broadcasted to the host node；
The host node receive it is described after the storage condition of node broadcasts, determine it is described from node be self-management
After node, the metadata information that is updated storage according to the storage condition；Wherein, the metadata information include data from
Storage information in node；
The host node broadcasts the metadata information to other host nodes, so as to be stored with the metadata information
Other host nodes of backup information update the metadata information.
Wherein, the preset rules can be clocking discipline, such as described obtain itself from the daily fixed time of node
Storage condition broadcasts the storage condition to the host node；Or the preset rules can be timer rule, such as
The timer setting time is n hours, i.e., the described storage condition for obtaining itself every described n hours from node, will be described
Storage condition is broadcasted to the host node.
It will be appreciated by those skilled in the art that the detailed description of the data processing method of the embodiment of the present invention, before can refer to
It states the associated description of Hadoop group system and understands.
Embodiment three is deleted respectively with data storage, data to embodiment five and data are revised as example to the embodiment of the present invention
It is described in further detail.
Embodiment three
The embodiment of the invention provides a kind of data processing methods.Fig. 4 is the data processing method of the embodiment of the present invention
First detailed process schematic diagram；As shown in figure 4, the data processing method includes:
Step 1001: client issues data storage request, and the data storage request includes file name information and appearance
Measure size requirement.
Here, the client sends the data storage request by data transmission link.
Step 1002: gateway node is after receiving the data storage request, by the data storage request to broadcast letter
The form of order is sent to each host node；Host node 1, host node 2 and host node 3 are only listed in Fig. 4, are not limited to the main section of above three
Point.
Step 1003: each host node according in the metadata information being locally stored respectively from the residual memory space feelings of node
Condition judges whether there is the slave node for meeting the data storage request.When judging result is to ask without meeting the data storage
When the slave node asked, then not respond；Host node 3 as shown in Figure 4 be without the slave node for meeting the data storage request,
It does not reply then.
Step 1004: when judging result is in the presence of the slave node for meeting the data storage request, host node is according to certainly
Body busy degree at once or postpones to respond confirmation (Ack) signaling to client.The Ack signaling information includes meeting storage need
The confirmation asked.
Here, the host node responds confirmation signaling and is sent to the client by the gateway node；Specifically, institute
It states host node and the confirmation signaling is sent to the gateway node by broadcasting link；The gateway node passes through data chain
Client described in road direction sends the confirmation signaling.
Step 1005: client selects the host node responded at first, establishes session (Session) connection.Remaining host node
Request signal is established due to not receiving Session in a period of time, then abandons the connection request for waiting client.In Fig. 4
The shown host node responded at first is host node 1, then the client is established Session with the host node 1 and connect；And main section
Point 2 establishes request signal due to not receiving Session, abandons the connection request for waiting client.Here, the host node 1
As the first host node.
Wherein, the Session connection procedure of establishing includes: that the client is asked to host node transmission Session foundation
Signal is sought, after receiving the confirmation response message that the host node is sent, the Session connection, which is established, to be completed.
Step 1006: host node 1 will be sent to client from nodal information.Wherein, it is described from nodal information to meet
State the slave nodal information of data storage requirement；It is described from nodal information include from the mark of node, address etc..
Step 1007: client starts to carry out data storage.
Here, the client passes through data transmission link first the data is sent to gateway node；The gateway
Node passes through data transmission link again and is sent to the data described to be stored from node.
Step 1008: after storing process is finished, client informs host node 1, and confirmation data storage is completed.
Here, the client first will confirm that information is sent to the gateway node, the net by data transmission link
Artis passes through broadcasting link again and sends confirmation message to the host node 1.
Step 1009: host node will confirm that information unloading storage to the second host node, and the second host node will confirm that information unloading
To third host node.Rear to after preservation, signaling is completed in reverse response, traces back to the first host node,
Step 1010: after confirmation saves, host node 1 responds client, and confirmation storing process finishes.
Here, the host node 1 sends confirmation message to the gateway node by broadcasting link；The gateway node is again
The confirmation message is sent to the client by data transmission link.
Example IV
The embodiment of the invention also provides a kind of data processing methods.Fig. 5 is the data processing method of the embodiment of the present invention
The second detailed process schematic diagram；As shown in figure 5, the data processing method includes:
Step 1101: client end sends data removal request.
Here, the client sends the data removal request by data transmission link.The data removal request
In include the information such as data name to be deleted.
Step 1102: after gateway node receives the data removal request, by the data removal request with the forms of broadcasting
It is sent to each host node；Host node 1, host node 2 and host node 3 are only listed in Fig. 5, are not limited to above three host node.
Step 1103: after each host node receives the data removal request, checking that itself whether there is comprising the data
Metadata information.It if it does not exist include the metadata information of the data, then it is without any processing；It is main as shown in Figure 5
The metadata information comprising the data is not present in node 3, then does not reply.
Step 1104: if it exists including the metadata information of the data, then host node is according to itself busy degree, at once
Or delay is attempted to respond confirmation (Ack) signaling to client.The Ack signaling information includes the presence of first number comprising the data
It is believed that the confirmation of breath.
Here, the host node responds confirmation signaling and is sent to the client by the gateway node；Specifically, institute
It states host node and the confirmation signaling is sent to the gateway node by broadcasting link；The gateway node passes through data chain
Client described in road direction sends the confirmation signaling.
Step 1105: client selects the host node responded at first as node is executed, and establishes session (Session) even
It connects.Remaining host node establishes request signal due to not receiving Session in a period of time, then abandons the connection for waiting client
Request.The host node responded at first as shown in Figure 5 is that host node 1, the then client and the host node 1 are established
Session connection；And host node 2 establishes request signal due to not receiving Session, abandons that the connection of client is waited to ask
It asks.Here, the host node 1 is used as the first host node.
Wherein, the Session connection procedure of establishing includes: that the client is asked to host node transmission Session foundation
Signal is sought, after receiving the confirmation response message that the host node is sent, the Session connection, which is established, to be completed.
Step 1106: host node 1 is according to the metadata information of storage, and notice stores this metadata in the form of broadcast signaling
The backup information of the metadata information is cleared up in remaining host node of information backup information, request；Cleaning first number
It is believed that the backup information of breath is used to delete the data deleted in the backup information of the metadata information with the client request
Relevant information.Remaining described host node host node 2 as shown in Figure 5.The host node 2 completes the metadata letter in cleaning
After the backup information of breath, confirmation is sent to the host node 1 in the form of broadcast signaling and deletes information.
Step 1107: after the host node 2 replys confirmation deletion information with the forms of broadcasting, host node 1 is to all from node
Information is deleted in broadcast.It include data name to be deleted in the deletion information.
Step 1108: all data stored from nodal test itself, the data, then delete the data if it exists,
And sent in the form of broadcast signaling to host node 1 and delete confirmation message, confirmation data are deleted；The data if it does not exist, then
Any response is not done.
Step 1109: after host node confirms that all data have been cleaned, responding client, confirmation data deletion finishes.
Here, the host node 1 sends confirmation message to the gateway node by broadcasting link；The gateway node is again
The confirmation message is sent to the client by data transmission link.
Embodiment five
The embodiment of the invention also provides a kind of data processing methods.Fig. 6 is the data processing method of the embodiment of the present invention
Third detailed process schematic diagram；As shown in fig. 6, the data processing method includes:
Step 1201: the data computation requests that client is sent.
Here, the client sends the data computation requests by data transmission link.
Step 1202: after gateway node receives the data computation requests, by the data computation requests with the forms of broadcasting
It is sent to each host node；Host node 1, host node 2 and host node 3 are only listed in Fig. 6, are not limited to above three host node.
Step 1203: all host nodes check that itself whether there is comprising institute after receiving the data computation requests
State the metadata information of data.It if it does not exist include the metadata information of the data, then it is without any processing；Such as institute in Fig. 6
The metadata information comprising the data is not present in the host node 3 shown, then does not reply.
Step 1204: if it exists including the metadata information of the data, then host node is according to itself busy degree, at once
Or postpone to respond confirmation (Ack) signaling to client.The Ack signaling information includes in the presence of the metadata letter comprising the data
The confirmation of breath.
Here, the host node responds confirmation signaling and is sent to the client by the gateway node；Specifically, institute
It states host node and the confirmation signaling is sent to the gateway node by broadcasting link；The gateway node passes through data chain
Client described in road direction sends the confirmation signaling.
Step 1205: client selects the host node responded at first as Job management node (i.e. execution node).
Step 1206: the host node 1 is sent in the form of broadcast signaling to all slave nodes for being stored with the data
Data computation requests.
Step 1207: receiving the slave node of the computation requests, according to itself busy degree, at once or delay is sent really
Recognize signaling.Host node 1 then selects the slave node for receiving confirmation signaling at first as calculate node, to institute in the form of broadcast signaling
It states from node and sends confirmation signal.
Step 1208: it is described from node carry out calculate obtain calculated result, to the host node 1 in the form of broadcast signaling
Return to the calculated result.
Step 1209: the calculated result is sent to client by the host node 1.
Here, the host node 1 sends the calculated result to the gateway node by broadcasting link；The gateway section
Point sends the calculated result to the client by data transmission link again.
It should be understood by those skilled in the art that, the embodiment of the present invention can provide as method, system or computer program
Product.Therefore, the shape of hardware embodiment, software implementation or embodiment combining software and hardware aspects can be used in the present invention
Formula.Moreover, the present invention, which can be used, can use storage in the computer that one or more wherein includes computer usable program code
The form for the computer program product implemented on medium (including but not limited to magnetic disk storage and optical memory etc.).
The present invention be referring to according to the method for the embodiment of the present invention, the flow chart of system and computer program product and/or
Block diagram describes.It should be understood that each process that can be realized by computer program instructions in flowchart and/or the block diagram and/or
The combination of process and/or box in box and flowchart and/or the block diagram.It can provide these computer program instructions to arrive
General purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices processor to generate one
Machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for realizing flowing
The device for the function of being specified in journey figure one process or multiple processes and/or block diagrams one box or multiple boxes.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
The foregoing is only a preferred embodiment of the present invention, is not intended to limit the scope of the present invention.
Claims (10)
1. a kind of Hadoop group system, which is characterized in that the system comprises: gateway nodes, at least two host nodes and extremely
Lack two from node；Wherein,
The gateway node after the service request for receiving client, broadcasts the service request extremely by broadcasting link
At least two host node, and the host node of response signaling will be received at first as the first host node；It is also used to establish described
Connection between client and first host node；It is also used to distinguish transmission data or signalling；
The host node, after receiving the service request that the gateway node is broadcasted, according to the resource situation of itself
Judge whether to meet the service request；After determining that the resource situation of itself meets the service request, Xiang Suoshu gateway node
Send response signaling；It is the service of the client after being also used to establish connection by the gateway node and the client
The slave node of request distribution meet demand, instruction is described to execute the service request from node；
It is described from node, execute the service request for the instruction based on the host node.
2. system according to claim 1, which is characterized in that the service request is data storage request；Correspondingly,
The host node, after receiving the data storage request that the gateway node is broadcasted, according to first number of storage
It is believed that breath judges whether the residual memory space of the slave node of self-management meets the data storage request；Determine self-management
Slave node residual memory space meet the data storage request after, Xiang Suoshu gateway node send response signaling；Wherein,
The metadata information includes data from the storage information in node；It is also used to through the gateway node and the client
After establishing connection, meet the slave node of the data storage request for data storage request distribution, by described from node
Information is sent to client by the gateway node；So that the client is deposited according to described from the information of node progress data
Storage；
It is described from node, for receiving and storing what the client was sent by the gateway node by data transmission link
Data.
3. system according to claim 1, which is characterized in that the service request is data removal request；Correspondingly, institute
Host node is stated, after the data removal request for receiving the gateway node broadcast, retrieves the metadata information of storage,
Judge whether the data store in the slave node of self-management；Determine that the data store in the slave node of self-management
Afterwards, Xiang Suoshu gateway node sends response signaling；Wherein, the metadata information includes data from the storage information in node
And the metadata information is in the backup information of other host nodes；It is also used to build by the gateway node and the client
After vertical connection, data described in the slave knot removal of the data are stored with by broadcasting link instruction, and instruction storage is
Other host nodes for stating the backup information of metadata information update the metadata information of itself storage；
It is described from node, delete the data for the instruction based on the host node.
4. system according to claim 1, which is characterized in that the service request is data computation requests；Correspondingly,
The host node retrieves first number of storage after receiving the data computation requests that the gateway node is broadcasted
It is believed that breath, judges whether the data store in the slave node of self-management；Determine the data in the slave section of self-management
After storing in point, Xiang Suoshu gateway node sends response signaling；Wherein, the metadata information includes data from node
Store information；After being also used to establish connection by the gateway node and the client, to meeting the data computation requests
Slave node broadcasts described in data computation requests, and broadcast instruction and receive the slave node of response signaling at first as calculate node；
It is also used to the calculated result that the calculate node is sent being sent to the client by the gateway node；
It is described from node, for executing calculating operation according to the data computation requests, being counted as after the calculate node
Calculate result；The calculated result is broadcasted to the host node.
5. system according to any one of claims 1 to 4, which is characterized in that it is described from node, it is also used to by preset rules
The storage condition for obtaining itself broadcasts the storage condition to the host node；
The host node, be also used to receive it is described after the storage condition of node broadcasts, determine it is described from node be from barrel
After the slave node of reason, the metadata information that is updated storage according to the storage condition；Wherein, the metadata information includes data
From the storage information in node；It is also used to broadcast the metadata information to other host nodes, so as to be stored with the member
Other host nodes of data information backup update the metadata information backup.
6. a kind of data processing method, which is characterized in that the data processing method is applied to include gateway node, at least two
Host node and at least two is from the Hadoop group system of node；The described method includes:
After the gateway node receives the service request of client, by broadcasting link broadcast the service request to it is described extremely
Few two host nodes；
After the host node receives the service request, judge whether that meeting the service asks according to the resource situation of itself
It asks；After determining that the resource situation of itself meets the service request, Xiang Suoshu gateway node sends response signaling；
The gateway node will receive the host node of response signaling as the first host node at first, establish the client with it is described
Connection between first host node；It is also used to distinguish transmission data or signalling；
First host node is that the service request of the client distributes the slave node of meet demand, and instruction is described to hold from node
The row service request.
7. according to the method described in claim 6, it is characterized in that, the service request is data storage request；The then master
After node receives the service request, judged whether to meet the service request according to the resource situation of itself；Determine itself
Resource situation meet the service request after, Xiang Suoshu gateway node send response signaling, comprising:
After the host node receives the data storage request, the slave section of self-management is judged according to the metadata information of storage
Whether the residual memory space of point meets the data storage request；Determine that the residual memory space of the slave node of self-management is full
After the foot data storage request, Xiang Suoshu gateway node sends response signaling；Wherein, the metadata information includes that data exist
From the storage information in node；Correspondingly, the service request that first host node is the client distributes meet demand
From node, instruction is described to execute the service request from node, comprising:
First host node is that the data storage request distribution of the client meets the slave node of the data storage request,
The information from node is sent to client by the gateway node, so that the client is according to described from node
Information carries out data storage；
It is described that the data that the client is sent by the gateway node are received and stored by data transmission link from node.
8. according to the method described in claim 6, it is characterized in that, the service request is data removal request；The then master
After node receives the service request, judged whether to meet the service request according to the resource situation of itself；Determine itself
Resource situation meet the service request after, Xiang Suoshu gateway node send response signaling, comprising:
After the host node receives the data removal request, the metadata information of storage is retrieved, whether judges the data
It is stored in the slave node of self-management；After determining that the data store in the slave node of self-management, Xiang Suoshu gateway section
Point sends response signaling；Wherein, the metadata information includes data from the storage information and the metadata in node
Backup information of the information in other host nodes；Correspondingly, the service request that first host node is the client is distributed completely
The slave node of sufficient demand, instruction is described to execute the service request from node, comprising:
First host node is stored with data described in the slave knot removal of the data, and instruction by broadcasting link instruction
Other host nodes for being stored with the backup information of the metadata information update the metadata information of itself storage；
It is described that the data are deleted based on the instruction of the host node from node.
9. according to the method described in claim 6, it is characterized in that, the service request is data computation requests；The then master
After node receives the service request, judged whether to meet the service request according to the resource situation of itself；Determine itself
Resource situation meet the service request after, Xiang Suoshu gateway node send response signaling, comprising:
After the host node receives the data computation requests, the metadata information of storage is retrieved, whether judges the data
It is stored in the slave node of self-management；After determining that the data store in the slave node of self-management, Xiang Suoshu gateway section
Point sends response signaling；Wherein, the metadata information includes data from the storage information in node；Correspondingly, described
One host node is that the service request of the client distributes the slave node of meet demand, and instruction is described to execute the service from node
Request, comprising:
First host node is broadcasted and is referred to data computation requests described in the slave node broadcasts for meeting the data computation requests
Show the slave node for receiving response signaling at first as calculate node；
Data computation requests described in the calculate node execute calculating operation, obtain calculated result；Institute is broadcasted to the host node
State calculated result.
10. according to the described in any item methods of claim 6 to 9, which is characterized in that the method also includes: it is described from node
The storage condition that itself is obtained by preset rules broadcasts the storage condition to the host node；
The host node receives described after the storage condition of node broadcasts, determines that described is the slave section of self-management from node
After point, the metadata information that is updated storage according to the storage condition；Wherein, the metadata information includes data from node
In storage information；
The host node broadcasts the metadata information to other host nodes, so as to be stored with the backup of the metadata information
Other host nodes of information update the metadata information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510001684.1A CN105824810B (en) | 2015-01-04 | 2015-01-04 | A kind of Hadoop group system and data processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510001684.1A CN105824810B (en) | 2015-01-04 | 2015-01-04 | A kind of Hadoop group system and data processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105824810A CN105824810A (en) | 2016-08-03 |
| CN105824810B true CN105824810B (en) | 2019-04-23 |
Family
ID=56986812
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510001684.1A Active CN105824810B (en) | 2015-01-04 | 2015-01-04 | A kind of Hadoop group system and data processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105824810B (en) |
Families Citing this family (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106447775A (en) * | 2016-09-18 | 2017-02-22 | 电子科技大学 | Cloud-computing-based large scale curved surface reconstruction system |
| CN106533738B (en) * | 2016-10-20 | 2019-09-10 | 中国民生银行股份有限公司 | The methods, devices and systems of distributed batch processing |
| CN109829076B (en) * | 2017-08-22 | 2021-08-06 | 上海策赢网络科技有限公司 | Method and equipment for generating block chain |
| CN108121514B (en) * | 2017-12-29 | 2021-05-25 | 北京奇虎科技有限公司 | Meta information updating method and device, computing equipment and computer storage medium |
| CN108304555A (en) * | 2018-02-05 | 2018-07-20 | 丁武轩 | Distributed maps data processing method |
| CN108449401A (en) * | 2018-03-12 | 2018-08-24 | 厦门益东智能科技有限公司 | A kind of calculation power sharing method and system based on block chain technology |
| CN108829787B (en) * | 2018-05-31 | 2022-06-17 | 郑州云海信息技术有限公司 | Metadata distributed system |
| CN109271243B (en) * | 2018-08-31 | 2021-09-17 | 郑州云海信息技术有限公司 | Cluster task management system |
| CN109639773B (en) * | 2018-11-26 | 2021-04-13 | 中国船舶重工集团公司第七一六研究所 | Dynamically constructed distributed data cluster control system and method thereof |
| CN110096358A (en) * | 2019-04-11 | 2019-08-06 | 上海交通大学 | Chain drive remote center distributed storage and distributed computing method |
| CN110311981A (en) * | 2019-07-05 | 2019-10-08 | 吉旗（成都）科技有限公司 | Branch gateway communication method and device, terminal, communication device based on Transmission Control Protocol |
| CN112394944A (en) * | 2019-08-13 | 2021-02-23 | 阿里巴巴集团控股有限公司 | Distributed development method, device, storage medium and computer equipment |
| CN110968762B (en) * | 2019-12-05 | 2023-07-18 | 北京天融信网络安全技术有限公司 | Adjustment method and device for retrieval |
| CN112565349B (en) * | 2020-11-18 | 2022-09-09 | 华帝股份有限公司 | Wireless communication method based on central range hood and related equipment |
| CN112506955B (en) * | 2020-12-10 | 2021-09-21 | 星环信息科技(上海)股份有限公司 | Query processing method, computer equipment and storage medium |
| CN113328920B (en) * | 2021-08-04 | 2021-10-29 | 成都飞机工业（集团）有限责任公司 | Method for collecting and storing equipment data |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102572810A (en) * | 2011-12-26 | 2012-07-11 | 南京邮电大学 | Method for automatically configuring addresses of large-scale wireless sensor network based on multi-token |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7155526B2 (en) * | 2002-06-19 | 2006-12-26 | Azaire Networks, Inc. | Method and system for transparently and securely interconnecting a WLAN radio access network into a GPRS/GSM core network |
-
2015
- 2015-01-04 CN CN201510001684.1A patent/CN105824810B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102572810A (en) * | 2011-12-26 | 2012-07-11 | 南京邮电大学 | Method for automatically configuring addresses of large-scale wireless sensor network based on multi-token |
Non-Patent Citations (1)
| Title |
|---|
| "基于Hadoop的电信BSS大数据平台建设研究";陈娜等;《电信科学》;20130331(第3期);期刊第3.2节，图3 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105824810A (en) | 2016-08-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105824810B (en) | A kind of Hadoop group system and data processing method | |
| CN103841170B (en) | A kind of cloud storage method and server | |
| CN110365748B (en) | Service data processing method and device, storage medium and electronic device | |
| CN107666525B (en) | Method and device for IP allocation of cluster container | |
| US9888048B1 (en) | Supporting millions of parallel light weight data streams in a distributed system | |
| CN113169952A (en) | Container cloud management system based on block chain technology | |
| CN106712981B (en) | Node change notification method and device | |
| US20110271135A1 (en) | Data management method and node apparatus | |
| CN109379448B (en) | File distributed deployment method and device, electronic equipment and storage medium | |
| CN108804465B (en) | Method and system for data migration of distributed cache database | |
| CN110209492A (en) | A kind of data processing method and device | |
| JP2012018449A (en) | Snapshot acquisition processing program, snapshot acquisition processing method, snapshot participant computer, and snap shot coordinator computer | |
| JP2018503159A (en) | Method, apparatus, and system for executing distributed transaction resources | |
| JP2017530453A (en) | Equal sharing of system resources in workflow execution | |
| US9614646B2 (en) | Method and system for robust message retransmission | |
| CN113742135A (en) | Data backup method and device and computer readable storage medium | |
| US10216593B2 (en) | Distributed processing system for use in application migration | |
| CN109271367A (en) | Distributed file system multinode snapshot rollback method and system | |
| CN104426968B (en) | Data managing method and device | |
| CN110751458A (en) | Business approval method, device and system | |
| CN111311254A (en) | Service processing method, device and system based on block chain | |
| US10817512B2 (en) | Standing queries in memory | |
| CN110083504A (en) | The running state monitoring method and device of distributed task scheduling | |
| CN113347238A (en) | Message partitioning method, system, device and storage medium based on block chain | |
| US9607068B2 (en) | Parallel truncation point management for log based replication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |