CN104468638B - A kind of distributed data processing method and system - Google Patents
A kind of distributed data processing method and system Download PDFInfo
- Publication number
- CN104468638B CN104468638B CN201310415547.3A CN201310415547A CN104468638B CN 104468638 B CN104468638 B CN 104468638B CN 201310415547 A CN201310415547 A CN 201310415547A CN 104468638 B CN104468638 B CN 104468638B
- Authority
- CN
- China
- Prior art keywords
- client
- heartbeat request
- task
- server
- pending data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Abstract
The present invention relates to a kind of distributed data processing method and system, wherein, server is that the meeting its disposal ability of the task is distributed and sent to the client for meeting treatment conditions；After client reception task, task is handled to obtain card address and pending data address；The client loads plug-in unit according to card address, and pending data is taken out according to pending data address；Client handles data after pending data is handled with the plug-in unit.When increasing the processing to new type pending data, because client handles pending data by the way of call plug-in unit, therefore a plug-in unit for handling the new type pending data need to only be developed, and plug-in unit is put into plugin library and can be called by different clients, without large-scale updating client, therefore, the present invention has the advantages of to processing new type pending data strong adaptability.
Description
Technical field
The present invention relates to a kind of data processing method and system, specifically a kind of client and server end operation
Distributed data processing method and system.
Background technology
So-called distributed data processing is exactly by different location or with difference in functionality or possesses different pieces of information
Multiple stage computers are connected with communication network, under unified management control, complete the calculating of information handling task in phase
Machine data processing method.Distributed data processing is introduced in detachable file data processing procedure, i.e., by single computer
The data of processing carry out concurrent collaboration process by multiple stage computers, can provide strong skill for the processing of substantial amounts of file data
Art guarantee.When to being handled using mass data file, frequent picture compression is especially related to, file format turns
The performance of special consumption calculations machine such as change, the advantage of distributed data processing will be particularly evident.
A kind of method of distributed system and distributed system processing task flow is disclosed in the prior art, and the patent is to appointing
When business stream is handled, for each different executable task in task flow, server can be respectively that its distribution is different
Client, be respectively processed by different clients, so as to realize the distributed treatment to task flow.Although the patent energy
Good data processing performance and Modes of Sharing Resources are enough provided, but when needing the data to new type to handle, by
Can not complete the processing of the new type data in the existing data processing method of client, thus just need in time to client and
Server is changed on a large scale, such as the server end controller of modification distributed treatment program, and newly-increased client
Program is realized, so realization to new type data processing will become extremely complex and difficult.When newly increasing client, need
To give each newly-increased client that plug-in unit, the poor expandability of whole distributed data processing system are installed one by one.
The content of the invention
Therefore, the technical problems to be solved by the invention be to overcome in the prior art due to plug-in unit be arranged on client and
A kind of the problem of caused whole distributed file system poor expandability, there is provided the strong client and server end of scalability
The distributed data processing method and system of operation.
In order to solve the above technical problems, the present invention provides a kind of distributed data processing method of server end operation, bag
Include following steps：
The server obtains the resource information of client in real time；
The server is determined for compliance with the task of the client process ability according to the resource information；
The server is that the meeting its disposal ability of the task is distributed and sent to the client；
Wherein, the address comprising pending data, plugin library address and the processing pending data need in the task
The process instruction type wanted；The plug-in unit of plug and play is stored with the plugin library.
A kind of distributed data processing method of server end operation, in addition to the step of following renewal plugin library：
New plug-in unit is stored into the plugin library；
Process instruction type corresponding to new plug-in unit is stored into the server.
A kind of distributed data processing method of server end operation, the server is the visitor for meeting task treatment conditions
In the step of family end is distributed and sends the meeting the client process ability of the task, the task also includes the shape of pending data
State, the state of the pending data further comprise client-side information and label information, wherein,
The client-side information is the information for the client that the pending data is distributed to；
The information that the label information is removed for the mark pending data.
A kind of distributed data processing method of server end operation, the server obtain the resource letter of client in real time
The step of breath, further comprises：
The server receives the heartbeat request that client is sent with cycle T, wherein, the heartbeat request is registration first
Heartbeat request, non-heartbeat request first are task heartbeat request, and the task heartbeat request includes the resource of the client
The state of information and pending data；
The server is that each client establishes a message queue for depositing the client heartbeat request；
The server reads the heartbeat request positioned at the message queue stem, and the heartbeat request after reading is from institute
State in message queue and fall out；When the heartbeat request that the server is read is registers heartbeat request, then to sending the note
The client of volume heartbeat request is registered；Otherwise it is transferred in next step；
The server carries out the resource information that parsing obtains the client to the heartbeat request.
A kind of distributed data processing method of server end operation, the server parse to the heartbeat request
Afterwards, the state of the pending data of the client is obtained, and according to the state of the pending data in the server
The information of task be updated.
A kind of distributed data processing method of server end operation, the server reads first positioned at the message queue
The step of heartbeat request in portion, further comprises：
The server is that a heartbeat request in each message queue positioned at stem distributes a thread；
The thread reads the heartbeat request for being located at stem in the message queue；
The step of server carries out to the heartbeat request and parses the resource information for obtaining the client is further wrapped
Include：
The thread parses to the heartbeat request of taking-up；
The thread judges to send the whether available free resource of client of the heartbeat request.
A kind of distributed data processing method of server end operation, the thread judge to send the client of the heartbeat request
End whether available free resource the step of further comprise：
The thread is analyzed the heartbeat request after parsing；
The thread judge in the client described in heartbeat request whether available free CPU core, as available free
CPU core, then judge the client resource free time.
A kind of distributed data processing method of client operation, comprises the following steps：
The task that the client the reception server is sent, and the task is parsed with obtaining pending data
Location, plugin library address and process instruction type, and obtain being used to locate according to the plugin library address and process instruction type that parse
Manage the card address of the pending data；Wherein, the plug-in unit of plug and play is stored with the plugin library；
The client loads the plug-in unit according to the card address；
The client takes out the pending data according to the pending data address；
The client handles data after the pending data is handled with the plug-in unit of loading.
A kind of distributed data processing method of client operation, the client by resource information and are waited to locate with cycle T
The state of reason data is sent to the server in a manner of task heartbeat request；And the client is sending task first
The step of also including transmission registration heartbeat request before heartbeat request, send and appoint first described in the registration heartbeat request and transmission
The time interval for heartbeat request of being engaged in is T.
A kind of distributed data processing system of server end operation, including resource information acquisition device, task processing bar
Part judgment means and task allocation apparatus, wherein,
The resource information acquisition device, for obtaining the resource information of client in real time；
The task treatment conditions judgment means, for being determined for compliance with the client process energy according to the resource information
The task of power；
The task allocation apparatus, meet the task of its disposal ability for distributing and sending for the client；
Wherein, the address comprising pending data, plugin library address and the processing pending data need in the task
The process instruction type wanted；The plug-in unit of plug and play is stored with the plugin library.
A kind of distributed data processing system of server end operation, in addition to updating device, the updating device enter one
Step includes plugin library updating device and process instruction type of storage, wherein,
The plugin library updating device, for new plug-in unit to be stored into the plugin library；
The process instruction type of storage, it is described for the process instruction type storage corresponding to new plug-in unit to be arrived
In server.
A kind of distributed data processing system of server end operation, the task that the task allocation apparatus is distributed
Also including the state of pending data, the state of the pending data further comprises client-side information and label information, its
In,
The client-side information is the information of the client of pending data distribution；
The information that the label information is removed for the mark pending data.
A kind of distributed data processing system of server end operation, in addition to heartbeat request reception device, queue are established
Device, heartbeat request reading device and heartbeat request resolver, wherein,
The heartbeat request reception device, the heartbeat request sent for receiving client with cycle T, wherein, institute first
Heartbeat request is stated as registration heartbeat request, non-heartbeat request first is task heartbeat request, and the task heartbeat request includes
The resource information of the client and the state of pending data；
Device is established in the queue, for establishing a message queue for depositing itself heartbeat request for each client；
The heartbeat request reading device, for reading the heartbeat request positioned at the message queue stem, read
Heartbeat request afterwards is fallen out from the message queue；When the heartbeat request that the server is read is registration heartbeat request
When, then the client for sending the registration heartbeat request is registered；
The heartbeat request resolver, should for carrying out parsing acquisition to the task heartbeat request in the heartbeat request
The resource information of client.
A kind of distributed data processing system of server end operation, in addition to task updating device,
The task updating device, for after the heartbeat request device parses to the heartbeat request, obtaining
The state of the pending data of the client, and according to the state of the pending data to the task in the server
Information is updated.
A kind of distributed data processing system of server end operation, the heartbeat request reading device further comprise line
Journey distributor and thread reading device, wherein,
The thread distributor, for distributing a line to be located at a heartbeat request of stem in each message queue
Journey；
The thread reading device, it is located at the institute of stem in the message queue for being read by the thread of distribution
State heartbeat request；
The heartbeat request device further comprises heartbeat request resolver and idling-resource judgment means, wherein,
The heartbeat request resolver, for being parsed using the thread to the heartbeat request of taking-up；
Whether the idling-resource judgment means, the client for judging to send the heartbeat request using the thread have
Idling-resource.
A kind of distributed data processing system of server end operation, the idling-resource judgment means further comprise the heart
Requirement analysis device and CPU core judgment means are jumped, wherein,
The heartbeat request analytical equipment, for being analyzed using the thread the heartbeat request after parsing；
The CPU core judgment means, it is in the client described in heartbeat request for being judged using the thread
No available free CPU core, such as available free CPU core, then judge the client resource free time.
A kind of distributed data processing system of client operation, including task resolver, loading device, pending number
According to data storage device after reading device, data processing equipment and processing, wherein,
The task resolver, being sent for the reception server for task, and the task is parsed and treated
Processing data address, plugin library address and process instruction type, and according to the plugin library address and process instruction type parsed
Obtain the card address for handling the pending data；Wherein, the plug-in unit of plug and play is stored with the plugin library；
The loading device, for loading the plug-in unit according to the card address；
The pending data reading device, for taking out the pending data according to the pending data address；
The data processing equipment, for handling number after the pending data is handled with the plug-in unit of loading
According to.
A kind of distributed data processing system of client operation, in addition to heartbeat request dispensing device,
The heartbeat request dispensing device, for making the client with cycle T by the resource information and pending number
According to state the server is sent in a manner of task heartbeat request；And the client is sending task heartbeat first
The step of also including sending registration heartbeat request before request, send the registration heartbeat request and the task heart first described in sending
The time interval for jumping request is T.
The above-mentioned technical proposal of the present invention has advantages below compared with prior art：
1st, the distributed data processing method and system of a kind of client and server end operation of the present invention；Its
In, in the distributed data processing method and system of server end operation, server obtains the resource information of client in real time, and
The task of the client process ability is determined for compliance with according to the resource information, the server is that the client is distributed simultaneously
Send the meeting its disposal ability of the task；Wherein, address, plugin library address and the processing of pending data are included in the task
The process instruction type that the pending data needs；The plug-in unit of plug and play is stored with the plugin library；Such server
The address of the plug-in unit needed in the task of distribution containing processing pending data, without single respectively in advance during client process task
Data processing can be completed by solely installing corresponding plug-in unit, without installing plug-in unit in client one by one again；Correspondingly, client is run
Distributed data processing method and system in, the just address containing plugin library in the task that client is got, pass through calling
Plug-in unit in plugin library can complete the processing to distributed data.When increasing new client, during the client process task
The plug-in unit of calling plug and play is only needed, without installing any plug-in unit, therefore, distributed data processing method of the invention
The advantages of there is scalability and strong adaptability with system.
2. a kind of distributed data processing method and system of the present invention, can also be updated to plugin library, when
When adding the pending data of new type, it is only necessary to the plug-in unit of processing this type pending data is developed, and this is new
Plug-in unit is supplemented in plugin library, and each client can call new plug-in unit to handle when handling new task, it is possible to achieve to new type
Data carry out distributed data processing, therefore the advantages of with system change small, strong adaptability.
3. a kind of distributed data processing method and system of the present invention, sent using client to the server
The mode of heartbeat request carries out data processing, because the server does not have to send invitation to the client, therefore in institute
When stating distributed system and expanding or shrinking client terminal quantity, without making what is invited on server to client in the server
Parameter configuration, there is the advantages of client terminal quantity configuration is flexible, running efficiency of system is high.
A kind of 4. distributed data processing method and system of the present invention, wherein the server please for each heartbeat
A heartbeat request in queue positioned at stem is asked to distribute a thread, i.e., described distributed system is using multithreading to multiple institutes
The heartbeat request for stating client transmission carries out concurrent operations, avoids multiple heartbeat request queues and grabs thread, thus improves
System resource service efficiency.
5. a kind of distributed data processing method and system of the present invention, wherein, the status information of pending data
Show whether the pending data has been removed and has distributed, be removed, the thread is in the task of packing with regard to pending
Data mode be taken away distribution be just not repeated read distribution, avoid and the pending data repeated, from
The aspect of resource allocation avoids the generation of distributed system resource deadlock, is developed skill for system normal operation and supports and protect
Barrier.
A kind of 6. distributed data processing method and system of the present invention, wherein the server passes through heartbeat request
To obtain the CPU core free time number of the client, so as to judge the whether available free resource of the client, so as to avoid
Cause the congestion of client because server distributes excessive task to client, reduce task response-time, improve
The treatment effeciency of distributed system.
Brief description of the drawings
In order that present disclosure is more likely to be clearly understood, specific embodiment and combination below according to the present invention
Accompanying drawing, the present invention is further detailed explanation, wherein
Fig. 1 is a kind of flow chart of the distributed data processing method of client operation of one embodiment of the invention；
Fig. 2 is a kind of flow chart of the distributed data processing method of server end operation of one embodiment of the invention；
Fig. 3 is a kind of structural representation of distributed data processing system of one embodiment of the invention；
Fig. 4 is a kind of schematic diagram on heartbeat request of distributed data processing method of one embodiment of the invention；
Fig. 5 is a kind of structural representation of the distributed data processing system of client operation of one embodiment of the invention
Figure；
Fig. 6 is a kind of structural representation of the distributed data processing system of server end operation of one embodiment of the invention
Figure.
Embodiment
The embodiment of the present invention is described in detail below in conjunction with accompanying drawing.It should be appreciated that this place is retouched
The embodiment stated is merely to illustrate and explain the present invention, and is not intended to limit the invention.
Embodiment 1
As a kind of distributed data processing method of server end operation of the present invention, as shown in Fig. 2 including such as
Lower step：
The server obtains the resource information of client in real time.
The server is determined for compliance with the task of the client process ability according to the resource information.
The server is that the meeting its disposal ability of the task is distributed and sent to the client.
Wherein, the address comprising pending data, plugin library address and the processing pending data need in the task
The process instruction type wanted.The plug-in unit of plug and play is stored with the plugin library.
In the distributed data processing method of the server operation of the present invention, the pending number of processing is contained in the task of distribution
According to the address of the plug-in unit of needs, data can be completed without individually install corresponding plug-in unit in advance during client process task
Processing, the realization to new type data processing need to only be uniformly processed in server so that whole data processing becomes very
It is simple and easy.
Embodiment 2
As one embodiment of the present of invention, on the basis of embodiment 1, the distributed data of the server end operation
Processing method, in addition to the step of following renewal plugin library：
New plug-in unit is stored into the plugin library.
Process instruction type corresponding to new plug-in unit is stored into the server.
Version corresponding to new plug-in unit is stored into the server.Corresponding, the task can also include
Plug-in version information, including the concrete form of the task of plug-in version information are as follows：＜ pending datas address=" specific address "
Plugin library address=" specific address " plug-in version information=" specific version information " pending instruction type=" particular type " ＞.
In server during normal operation, in the event of the data processing of new type, now, it is thus only necessary to which renewal is inserted
Part storehouse, the data processing of new type is supplemented in existing plugin library, the task of the server-assignment with plug-in unit
In contain the address of new plug-in unit, during the client process task, it is only necessary to according to the new plug-in unit call by location this new insert
Part.
Embodiment 3
As one embodiment of the present of invention, on the basis of above-described embodiment 1 or embodiment 2, the server is symbol
In the step of client of conjunction task treatment conditions is distributed and sends the meeting the client process ability of the task, the task is also
State including pending data, the state of the pending data further comprise client-side information and label information, wherein,
The client-side information is the information of the client of pending data distribution；The label information is mark
Remember the information that the pending data is removed.
The server all illustrates the pending number when distribution new task takes pending data with the label information
According to being removed, it is impossible to taken away distribution again again；Can be effectively less or even same pending data be avoided to be assigned to different appoint
In business so be duplicatedly distributed processing caused by wasting of resources phenomenon generation.Moreover, avoided from the aspect of resource allocation
The generation of distributed system resource deadlock, developed skill for system normal operation and support and ensure.
Embodiment 4
As one embodiment of the present of invention, on the basis of upper embodiment 3, as shown in figure 4, the server obtains in real time
The step of resource information of client, further comprises：
The server receives the heartbeat request that client is sent with cycle T, wherein, the heartbeat request is registration first
Heartbeat request, non-heartbeat request first are task heartbeat request, and the task heartbeat request includes the resource of the client
The state of information and pending data.
The server is that each client establishes a message queue for depositing the client heartbeat request；
The server reads the heartbeat request positioned at the message queue stem, and the heartbeat request after reading is from institute
State in message queue and fall out；When the heartbeat request that the server is read is registers heartbeat request, then to sending the note
The client of volume heartbeat request is registered.Otherwise it is transferred in next step.
The server carries out the resource information that parsing obtains the client to the heartbeat request.
Server is that each client establishes a message queue for depositing the client heartbeat request, and according to advanced elder generation
The order gone out sequentially reads the heartbeat request positioned at the message queue head of the queue, and any one heartbeat request is not omitted in guarantee.Moreover,
The independent management of the heartbeat request sent to different clients is realized, it is clear.
Embodiment 5
As one embodiment of the present of invention, on the basis of upper embodiment 4, the server is carried out to the heartbeat request
After parsing, the state of the pending data of the client is obtained, and according to the state of the pending data to the service
The information of task in device is updated.
So server can obtain the process of each client process task at any time, be easy to implement to each client task
Tracking and management.
Embodiment 6
As one embodiment of the present of invention, on the basis of upper embodiment 5, the server, which is read, is located at the message team
The step of heartbeat request of row stem, further comprises：
The server is that a heartbeat request in each message queue positioned at stem distributes a thread.
The thread reads the heartbeat request for being located at stem in the message queue.
The step of server carries out to the heartbeat request and parses the resource information for obtaining the client is further wrapped
Include：
The thread parses to the heartbeat request of taking-up.
The thread judges to send the whether available free resource of client of the heartbeat request.
The server is that a heartbeat request in each heartbeat request queue positioned at stem distributes a thread, i.e. institute
The concurrent operations that the heartbeat request that distributed system is sent using multithreading to multiple clients is read out and parsed are stated,
Avoid multiple heartbeat request queues and grab thread, improve system resource service efficiency.
Certainly, the other embodiment as the present invention, the server can also be that multiple message queues distribute a line
Journey, but so just need to be handled according to timeslice rotation in multiple message queues, multiple message queue processing can not be held parallel
OK, therefore response speed is slower.
Embodiment 7
As one embodiment of the present of invention, on the basis of upper embodiment 6, the thread judges to send the heartbeat request
Client whether available free resource the step of further comprise：
The thread is analyzed the heartbeat request after parsing.
The thread judge in the client described in heartbeat request whether available free CPU core, as available free
CPU core, then judge the client resource free time.
The server obtains the CPU core free time number of the client by heartbeat request, so as to judge
The whether available free resource of client is stated, causes client because server distributes excessive task to client so as to avoid
Congestion, reduce task response-time, improve the treatment effeciency of distributed system.
Correspondingly, the present invention provides a kind of distributed data processing system of client and server end operation.
Embodiment 8
The distributed data processing method of a kind of client operation of one embodiment of the invention, as shown in figure 1, including such as
Lower step：
The task that the client the reception server is sent, and the task is parsed with obtaining pending data
Location, plugin library address and process instruction type, and obtain being used to locate according to the plugin library address and process instruction type that parse
Manage the card address of the pending data.Wherein, the plug-in unit of plug and play is stored with the plugin library.The process instruction class
Type is the pending data type and handles the corresponding relation of the plug-in unit of the type pending data, and the client is according to slotting
Part storehouse address accesses the plugin library, is found out further according to the process instruction type corresponding with the pending data type
Plug-in unit.The plug-in unit of so-called plug and play refers to can load the plug-in unit used without installation.The task includes, and ＜ is pending
Data address=" specific address " plugin library address=" specific address " pending instruction type=" particular type " ＞.
The client loads the plug-in unit according to the card address.
The client takes out the pending data according to the pending data address.
The client handles data after the pending data is handled with the plug-in unit of loading.
Involved pending data and the plugin library can be stored in distributed data processing system in the task
Optional position, including server and client side.Preferably, as shown in figure 3, the pending data is stored in
On the server of the distributed data processing system.The plugin library is stored on the data file server.General visitor
Data also are stored in the specified location of the data file server after the processing that the family end execution task obtains.Therefore, with
The task that the corresponding client the reception server of preferred embodiment is sent, it should also include performing knot for store tasks
The address of the specified location of fruit, the task include, and ＜ pending datas address=" specific address " plugin library address=" specifically
Location " pending instruction type=" particular type " specified location address=" specific address " ＞.
The distributed data processing method of a kind of client operation of the present invention, because the client is using calling
The mode of plug-in unit handles pending data, therefore only needs exploitation one to be used for the plug-in unit for handling particular type pending data
Be put into plugin library can, without installing plug-in unit one by one on the client on a large scale；Especially when increasing client, without
Any plug-in unit is installed in client, thus the distributed data processing method of client of the present invention operation have scalability by force and
The advantages of strong adaptability.
As the other embodiment of the present invention, wait to locate according to taking out the pending data address in the client
After the step of managing data, it can also include:The client feeds back to service after task is finished, by task action result
The step of device.Or the step of also including resource information is sent to the client into server.
Embodiment 9
As one embodiment of the present of invention, on the basis of embodiment 8, as shown in figure 4, the client is with cycle T
The state of the resource information and pending data is sent to the server in a manner of task heartbeat request；It is and described
The step of client also includes sending registration heartbeat request before task heartbeat request first is sent, send the registration heartbeat
The time interval of task heartbeat request is T first described in request and transmission.The task heartbeat request is sent in the form of following, ＜
State=" particular state " ＞ of resource information=" specific resource information " pending data.The registration following form of heartbeat request
Send, ＜ resource informations=" specific resource information " ＞.As a kind of preferred embodiment, the task heartbeat request and the registration
Also include heartbeat request type in heartbeat request, wherein, the task heartbeat request is sent in the form of following, ＜ heartbeat request classes
State=" particular state " ＞ of type=" task heartbeat request type " resource information=" specific resource information " pending data, institute
State the following form of registration heartbeat request to send, ＜ heartbeat requests type=" registration heartbeat request type " resource information=" specific money
Source information " ＞.
Cycle T is set using following,<MsgCircle>T</MsgCircle>, when the cycle is 5s, specific setting is as follows：
<MsgCircle>5</MsgCircle>。
The distributed data processing method of a kind of client operation of the present invention, using client to the server
The mode for sending heartbeat request carries out data communication, certainly as other embodiment of the invention, except institute in the present embodiment
The method of the transmission heartbeat request of use, can also be by the way of the three-way handshake in traditional computer network, i.e., described clothes
Business device first sends invitation to client, and client is response to the information of server client current state, and server is again to client
The mode of transmission task is held, equally, the distribution and transmission of task can be realized.But the mode of three-way handshake, in the number of client
, it is necessary to the parameter configuration invited to client be changed on the server, with respect to the heartbeat request ratio in the present embodiment during amount change
It is more complicated.Relative to the communication modes of three-way handshake, the mode of the transmission heartbeat request in the present embodiment, because client is spontaneous
Carrying out, the server in actual distribution formula data handling system does not have to send invitation to the client, therefore in the distribution
When formula system expands or shrinks client terminal quantity, it only need to directly increase and decrease client, be related to without making in the server
The parameter configuration that server is invited to client, therefore with the advantages of client terminal quantity configuration is flexible, running efficiency of system is high.
Embodiment 10
The distributed data processing system that a kind of server end as one embodiment of the invention is run, as shown in fig. 6,
Including resource information acquisition device, task treatment conditions judgment means and task allocation apparatus.Wherein
The resource information acquisition device, for obtaining the resource information of client in real time.
The task treatment conditions judgment means, for judging whether the client meets task according to the resource information
Treatment conditions, and the task allocation apparatus work is triggered when the client meets task distributive condition.
The task allocation apparatus, meet for distributing and sending to meet the client of task distributive condition at its
The task of reason ability, meanwhile, the information of the task is stored in the server.
Wherein, the address comprising pending data, plugin library address and the processing pending data need in the task
The process instruction type wanted；The plug-in unit of plug and play is stored with the plugin library.
In the distributed data processing system of the server operation of the present invention, the pending number of processing is contained in the task of distribution
According to the address of the plug-in unit of needs, data can be completed without individually install corresponding plug-in unit in advance during client process task
Processing, the realization to new type data processing need to only be uniformly processed in server so that whole data processing becomes very
It is simple and easy.
Embodiment 11
The distributed data processing system that a kind of server end as one embodiment of the invention is run, in above-mentioned implementation
On the basis of example 10, in addition to updating device, the updating device further comprise plugin library updating device and process instruction class
Type storage device, wherein,
The plugin library updating device, for new plug-in unit to be stored into the plugin library.
The process instruction type of storage, it is described for the process instruction type storage corresponding to new plug-in unit to be arrived
In server.
In server during normal operation, in the event of the data processing of new type, now, it is thus only necessary to which renewal is inserted
Part storehouse, the data processing of new type is supplemented in existing plugin library, the task of the server-assignment with plug-in unit
In contain the address of new plug-in unit, during the client process task, it is only necessary to according to the new plug-in unit call by location this new insert
Part.
Embodiment 12
The distributed data processing system that a kind of server end as one embodiment of the invention is run, in above-mentioned implementation
On the basis of example 10 or embodiment 11, the task that the task allocation apparatus is distributed also includes the shape of pending data
State, the state of the pending data further comprise client-side information and label information, wherein,
The client-side information is the information of the client of pending data distribution.
The information that the label information is removed for the mark pending data.
The server all illustrates the pending number when distribution new task takes pending data with the label information
According to being removed, it is impossible to taken away distribution again again.Can be effectively less or even same pending data be avoided to be assigned to different appoint
In business so be duplicatedly distributed processing caused by wasting of resources phenomenon generation.
Embodiment 13
The distributed data processing system that a kind of server end as one embodiment of the invention is run, in above-mentioned implementation
On the basis of example 12, in addition to heartbeat request reception device, queue establish device, heartbeat request reading device and heartbeat request solution
Analysis apparatus, wherein,
The heartbeat request reception device, the heartbeat request sent for receiving client with cycle T, wherein, institute first
Heartbeat request is stated as registration heartbeat request, non-heartbeat request first is task heartbeat request, and the task heartbeat request includes
The resource information of the client and the state of pending data.
Device is established in the queue, for establishing a message queue for depositing itself heartbeat request for each client.
The heartbeat request reading device, for reading the heartbeat request positioned at the message queue stem, read
Heartbeat request afterwards is fallen out from the message queue；When the heartbeat request that the server is read is registration heartbeat request
When, then the client for sending the registration heartbeat request is registered.
The heartbeat request resolver, should for carrying out parsing acquisition to the task heartbeat request in the heartbeat request
The resource information of client.
Server is that each client establishes a message queue for depositing the client heartbeat request, and according to advanced elder generation
The order gone out sequentially reads the heartbeat request positioned at the message queue head of the queue, and any one heartbeat request is not omitted in guarantee.Moreover,
The independent management of the heartbeat request sent to different clients is realized, it is clear.
Embodiment 14
The distributed data processing system that a kind of server end as one embodiment of the invention is run, in above-mentioned implementation
On the basis of example 13, in addition to task updating device, the task updating device, in the heartbeat request device to described
After heartbeat request is parsed, the state of the pending data of the client is obtained, and according to the shape of the pending data
State is updated to the information of the task in the server.
So server can obtain the process of each client process task at any time, be easy to implement to each client task
Tracking and management.
Embodiment 15
The distributed data processing system that a kind of server end as one embodiment of the invention is run, in above-mentioned implementation
On the basis of example 14, the heartbeat request reading device further comprises thread distributor and thread reading device, wherein,
The thread distributor, for distributing a line to be located at a heartbeat request of stem in each message queue
Journey.
The thread reading device, it is located at the institute of stem in the message queue for being read by the thread of distribution
State heartbeat request.
The heartbeat request device further comprises heartbeat request resolver and idling-resource judgment means, wherein,
The heartbeat request resolver, for being parsed using the thread to the heartbeat request of taking-up.
Whether the idling-resource judgment means, the client for judging to send the heartbeat request using the thread have
Idling-resource.
The server is that a heartbeat request in each heartbeat request queue positioned at stem distributes a thread, i.e. institute
The concurrent operations that the heartbeat request that distributed system is sent using multithreading to multiple clients is read out and parsed are stated,
Avoid multiple heartbeat request queues and grab thread, improve system resource service efficiency.
Embodiment 16
The distributed data processing system that a kind of server end as one embodiment of the invention is run, in above-mentioned implementation
On the basis of example 15, the idling-resource judgment means further comprise heartbeat request analytical equipment and CPU core judgment means,
Wherein,
The heartbeat request analytical equipment, for being analyzed using the thread the heartbeat request after parsing.
The CPU core judgment means, it is in the client described in heartbeat request for being judged using the thread
No available free CPU core, such as available free CPU core, then judge the client resource free time.
The server obtains the CPU core free time number of the client by heartbeat request, so as to judge
The whether available free resource of client is stated, causes client because server distributes excessive task to client so as to avoid
Congestion, reduce task response-time, improve the treatment effeciency of distributed system.
Embodiment 17
As a kind of distributed data processing system of client operation of one specific embodiment of the present invention, such as Fig. 5 institutes
Show, including data storage after task resolver, loading device, pending data reading device, data processing equipment and processing
Device.Wherein,
The task resolver, being sent for the reception server for task, and the task is parsed and treated
Processing data address, plugin library address and process instruction type, and according to the plugin library address and process instruction type parsed
Obtain the card address for handling the pending data；Wherein, the plug-in unit of plug and play is stored with the plugin library.
The loading device, for loading the plug-in unit according to the card address.
The pending data reading device, for taking out the pending data according to the pending data address.
The data processing equipment, for handling number after the pending data is handled with the plug-in unit of loading
According to.
Data storage device after the processing, data output after the processing to specified location is stored.
The distributed data processing system of a kind of client operation of the present invention, because the client is using calling
The mode of plug-in unit handles pending data, thus only need by for handle the plug-in unit of particular type pending data be put into it is slotting
Part storehouse, without installing plug-in unit one by one on the client on a large scale.Especially when increasing client, without on the client
Any plug-in unit is installed, therefore the distributed data processing method of client of the present invention operation has scalability strong and strong adaptability
The advantages of.
As other embodiments of the invention, the distributed data processing system of client operation also includes resource information and sent
Device, the resource information dispensing device are used to the resource information of the client being sent to server.
Embodiment 18
As a kind of distributed data processing system of client operation of one specific embodiment of the present invention, in above-mentioned reality
On the basis of applying example 17, the resource information dispensing device is heartbeat request dispensing device,
The heartbeat request dispensing device, for making the client with cycle T by the resource information and pending number
According to state the server is sent in a manner of task heartbeat request；And the client is sending task heartbeat first
The step of also including sending registration heartbeat request before request, send the registration heartbeat request and the task heart first described in sending
The time interval for jumping request is T.
The distributed data processing system of a kind of client operation of the present invention, using client to the server
The mode for sending heartbeat request carries out data communication, and certainly as other embodiment of the invention, the client can also
Only existed using other communication modes, such as passive response formula, that is, the active inquisitorial that server end is initiated, the client
Data are just carried out when receiving inquiry message to report；Relative to the communication modes of passive response formula, the transmission heartbeat in the present embodiment
The mode of request, due to the spontaneous progress of client, the server in actual distribution formula data handling system does not have to the client
End sends invitation, therefore when the distributed system expands or shrinks client terminal quantity, only need to directly increase and decrease client,
Without making the parameter configuration for being related to server and being invited to client in the server, therefore with client terminal quantity configuration spirit
Living, the advantages of running efficiency of system is high.
Obviously, above-described embodiment is only intended to clearly illustrate example, and is not the restriction to embodiment.It is right
For those of ordinary skill in the art, can also make on the basis of the above description it is other it is various forms of change or
Change.There is no necessity and possibility to exhaust all the enbodiments.And the obvious change thus extended out or
Among changing still in the protection domain of the invention.
Claims (18)
- A kind of 1. distributed data processing method of server end operation, it is characterised in that comprise the following steps：The server obtains the resource information of client in real time；The server is determined for compliance with the task of the client process ability according to the resource information；The server is that the meeting its disposal ability of the task is distributed and sent to the client；Wherein, include what the address, plugin library address and the processing pending data of pending data needed in the task Process instruction type；The plug-in unit of plug and play is stored with the plugin library, to cause the client according to the plugin library Address and the process instruction type obtain the card address for handling the pending data, are added according to the card address The plug-in unit is carried, the pending data is taken out according to the pending data address, with the plug-in unit of loading to handle State data after pending data is handled.
- 2. the distributed data processing method of a kind of server end operation according to claim 1, it is characterised in that also wrap The step of including following renewal plugin library：New plug-in unit is stored into the plugin library；Process instruction type corresponding to new plug-in unit is stored into the server.
- A kind of 3. distributed data processing method of server end operation according to claim 1 or 2, it is characterised in that The server is that the client for meeting task treatment conditions is distributed and sends the step for the meeting the client process ability of the task In rapid, the task also includes the state of pending data, and the state of the pending data further comprises client-side information And label information, wherein,The client-side information is the information for the client that the pending data is distributed to；The information that the label information is removed for the mark pending data.
- 4. the distributed data processing method of a kind of server end operation according to claim 3, it is characterised in that described Server obtains the step of resource information of client and further comprised in real time：The server receives the heartbeat request that client is sent with cycle T, wherein, the heartbeat request is registration heartbeat first Request, non-heartbeat request first is task heartbeat request, and the task heartbeat request includes the resource information of the client With the state of pending data；The server is that each client establishes a message queue for depositing the client heartbeat request；The server reads the heartbeat request positioned at the message queue stem, and the heartbeat request after reading disappears from described Fallen out in breath queue；When the heartbeat request that the server is read is registers heartbeat request, then to sending the registration heart The client for jumping request is registered；Otherwise it is transferred in next step；The server carries out the resource information that parsing obtains the client to the heartbeat request.
- 5. the distributed data processing method of a kind of server end operation according to claim 4, it is characterised in that described After server parses to the heartbeat request, the state of the pending data of the client is obtained, and treat according to described The state of processing data is updated to the information of the task in the server.
- 6. the distributed data processing method of a kind of server end operation according to claim 5, it is characterised in that described Server is read to be further comprised positioned at the step of heartbeat request of the message queue stem：The server is that a heartbeat request in each message queue positioned at stem distributes a thread；The thread reads the heartbeat request for being located at stem in the message queue；The step of server carries out to the heartbeat request and parses the resource information for obtaining the client further comprises：The thread parses to the heartbeat request of taking-up；The thread judges to send the whether available free resource of client of the heartbeat request.
- 7. the distributed data processing method of a kind of server end operation according to claim 6, it is characterised in that described Thread judge send the heartbeat request client whether available free resource the step of further comprise：The thread is analyzed the heartbeat request after parsing；The thread judge in the client described in heartbeat request whether available free CPU core, as in available free CPU Core, then judge the client resource free time.
- A kind of 8. distributed data processing method of client operation, it is characterised in that comprise the following steps：The task that the client the reception server is sent, and the task is parsed to obtain pending data address, inserted Part storehouse address and process instruction type, and obtain treating for handling this according to the plugin library address and process instruction type that parse The card address of processing data；Wherein, the plug-in unit of plug and play is stored with the plugin library；The client loads the plug-in unit according to the card address；The client takes out the pending data according to the pending data address；The client handles data after the pending data is handled with the plug-in unit of loading.
- A kind of 9. distributed data processing method of client operation according to claim 8, it is characterised in that the visitor The state of resource information and pending data in a manner of task heartbeat request is sent to the server by family end by cycle T； And the client send also include sending registration heartbeat request before task heartbeat request first the step of, described in transmission Register heartbeat request and the time interval of task heartbeat request first described in sending is T.
- A kind of 10. distributed data processing system of server end operation, it is characterised in that including resource information acquisition device, Task treatment conditions judgment means and task allocation apparatus, wherein,The resource information acquisition device, for obtaining the resource information of client in real time；The task treatment conditions judgment means, for being determined for compliance with the client process ability according to the resource information Task；The task allocation apparatus, meet the task of its disposal ability for distributing and sending for the client；Wherein, include what the address, plugin library address and the processing pending data of pending data needed in the task Process instruction type；The plug-in unit of plug and play is stored with the plugin library, to cause the client according to the plugin library Address and the process instruction type obtain the card address for handling the pending data, are added according to the card address The plug-in unit is carried, the pending data is taken out according to the pending data address, with the plug-in unit of loading to handle State data after pending data is handled.
- 11. the distributed data processing system of a kind of server end operation according to claim 10, it is characterised in that also Including updating device, the updating device further comprises plugin library updating device and process instruction type of storage, wherein,The plugin library updating device, for new plug-in unit to be stored into the plugin library；The process instruction type of storage, for the process instruction type storage corresponding to new plug-in unit to be arrived into the service In device.
- 12. a kind of distributed data processing system of server end operation according to claim 10 or 11, its feature exist In the task that the task allocation apparatus is distributed also includes the state of pending data, the shape of the pending data State further comprises client-side information and label information, wherein,The client-side information is the information of the client of pending data distribution；The information that the label information is removed for the mark pending data.
- 13. the distributed data processing system of a kind of server end operation according to claim 12, it is characterised in that also Device, heartbeat request reading device and heartbeat request resolver are established including heartbeat request reception device, queue, wherein,The heartbeat request reception device, the heartbeat request sent for receiving client with cycle T, wherein, the heart first Request is jumped as registration heartbeat request, non-heartbeat request first is task heartbeat request, and the task heartbeat request includes described The resource information of client and the state of pending data；Device is established in the queue, for establishing a message queue for depositing itself heartbeat request for each client；The heartbeat request reading device, for reading the heartbeat request positioned at the message queue stem, after reading Heartbeat request is fallen out from the message queue；When the heartbeat request that the server is read is registers heartbeat request, Then the client for sending the registration heartbeat request is registered；The heartbeat request resolver, the client is obtained for carrying out parsing to the task heartbeat request in the heartbeat request The resource information at end.
- 14. the distributed data processing system of a kind of server end operation according to claim 13, it is characterised in that also Including task updating device,The task updating device, it is described for after the heartbeat request device parses to the heartbeat request, obtaining The state of the pending data of client, and the information according to the state of the pending data to the task in the server It is updated.
- A kind of 15. distributed data processing system of server end operation according to claim 14, it is characterised in that institute State heartbeat request reading device and further comprise thread distributor and thread reading device, wherein,The thread distributor, for distributing a thread to be located at a heartbeat request of stem in each message queue；The thread reading device, it is located at the heart of stem in the message queue for being read by the thread of distribution Jump request；The heartbeat request device further comprises heartbeat request resolver and idling-resource judgment means, wherein,The heartbeat request resolver, for being parsed using the thread to the heartbeat request of taking-up；The idling-resource judgment means, for judging whether the client for sending the heartbeat request is available free using the thread Resource.
- A kind of 16. distributed data processing system of server end operation according to claim 15, it is characterised in that institute State idling-resource judgment means and further comprise heartbeat request analytical equipment and CPU core judgment means, wherein,The heartbeat request analytical equipment, for being analyzed using the thread the heartbeat request after parsing；The CPU core judgment means, for judging whether have in the client described in heartbeat request using the thread Idle CPU core, such as available free CPU core, then judge the client resource free time.
- 17. a kind of distributed data processing system of client operation, it is characterised in that including task resolver, loading dress Put, data storage device after pending data reading device, data processing equipment and processing, wherein,The task resolver, being sent for the reception server for task, and the task is parsed to obtain pending Data address, plugin library address and process instruction type, and obtained according to the plugin library address and process instruction type parsed For handling the card address of the pending data；Wherein, the plug-in unit of plug and play is stored with the plugin library；The loading device, for loading the plug-in unit according to the card address；The pending data reading device, for taking out the pending data according to the pending data address；The data processing equipment, for handling data after the pending data is handled with the plug-in unit of loading.
- 18. the distributed data processing system of a kind of client operation according to claim 17, it is characterised in that also wrap Heartbeat request dispensing device is included,The heartbeat request dispensing device, for making the client with cycle T by resource information and the state of pending data The server is sent in a manner of task heartbeat request；And the client is before task heartbeat request first is sent The step of also including sending registration heartbeat request, send the registration heartbeat request and task heartbeat request first described in sending Time interval is T.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310415547.3A CN104468638B (en) | 2013-09-12 | 2013-09-12 | A kind of distributed data processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310415547.3A CN104468638B (en) | 2013-09-12 | 2013-09-12 | A kind of distributed data processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104468638A CN104468638A (en) | 2015-03-25 |
| CN104468638B true CN104468638B (en) | 2018-04-03 |
Family
ID=52914009
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310415547.3A Expired - Fee Related CN104468638B (en) | 2013-09-12 | 2013-09-12 | A kind of distributed data processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104468638B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105138598B (en) * | 2015-08-05 | 2018-11-09 | 深圳联友科技有限公司 | The method and system of remote timing task |
| CN105847356A (en) * | 2016-03-23 | 2016-08-10 | 上海爱数信息技术股份有限公司 | Communication system, electronic device, data processing method and system |
| CN107239328B (en) * | 2016-03-29 | 2020-06-09 | 上海普兰金融服务有限公司 | Task allocation method and device |
| CN108205470A (en) * | 2016-12-20 | 2018-06-26 | 北京奇虎科技有限公司 | A kind of distribution ad data calculating task management system and method |
| CN109445870A (en) * | 2018-09-28 | 2019-03-08 | 浙江乾冠信息安全研究院有限公司 | A kind of data processing method, electronic equipment and storage medium |
| CN109614167B (en) * | 2018-12-07 | 2023-10-20 | 杭州数澜科技有限公司 | Method and system for managing plug-ins |
| CN109639840A (en) * | 2019-02-25 | 2019-04-16 | 网宿科技股份有限公司 | A kind of data processing method and edge calculations system based on edge calculations |
| CN111858418B (en) * | 2019-04-30 | 2023-04-07 | 华为技术有限公司 | Memory communication method and device based on remote direct memory access RDMA |
| CN114513500A (en) * | 2022-01-19 | 2022-05-17 | 统信软件技术有限公司 | Method, device and system for batch installation of browser plug-ins and computing device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101853179A (en) * | 2010-05-10 | 2010-10-06 | 深圳市极限网络科技有限公司 | Universal distributed dynamic operation technology for executing task decomposition based on plug-in unit |
| CN102075358A (en) * | 2010-12-31 | 2011-05-25 | 网宿科技股份有限公司 | System and method for distributing and deploying content of large-scale server cluster |
| CN102480512A (en) * | 2010-11-29 | 2012-05-30 | 国际商业机器公司 | Method and device for expanding processing capacity at server end |
| CN103095758A (en) * | 2011-11-01 | 2013-05-08 | 北大方正集团有限公司 | Distributed file system and file data processing method thereof |
-
2013
- 2013-09-12 CN CN201310415547.3A patent/CN104468638B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101853179A (en) * | 2010-05-10 | 2010-10-06 | 深圳市极限网络科技有限公司 | Universal distributed dynamic operation technology for executing task decomposition based on plug-in unit |
| CN102480512A (en) * | 2010-11-29 | 2012-05-30 | 国际商业机器公司 | Method and device for expanding processing capacity at server end |
| CN102075358A (en) * | 2010-12-31 | 2011-05-25 | 网宿科技股份有限公司 | System and method for distributing and deploying content of large-scale server cluster |
| CN103095758A (en) * | 2011-11-01 | 2013-05-08 | 北大方正集团有限公司 | Distributed file system and file data processing method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104468638A (en) | 2015-03-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104468638B (en) | A kind of distributed data processing method and system | |
| CN104750543B (en) | Thread creation method, service request processing method and relevant device | |
| CN103118076B (en) | Upgraded server cluster system and load balancing method thereof | |
| CN107172187B (en) | A kind of SiteServer LBS and method | |
| CN109194538A (en) | Test method, device, server and storage medium based on distributed coordination | |
| AU2016298207B2 (en) | Background job processing framework | |
| CN103078759B (en) | The management method and device of calculate node, system | |
| US10498817B1 (en) | Performance tuning in distributed computing systems | |
| CN107491346A (en) | A kind of task processing method of application, apparatus and system | |
| CN109726004B (en) | Data processing method and device | |
| EP3837604A1 (en) | In situ triggered function as a service within a service mesh | |
| CN111930525B (en) | GPU resource use method, electronic device and computer readable medium | |
| EP2659653A1 (en) | Method and apparatus for providing multi-level distributed computations | |
| KR20160013884A (en) | Updating software components through online stores | |
| CN107688495A (en) | The method and apparatus of dispatch processor | |
| CN113220431B (en) | Cross-cloud distributed data task scheduling method, device and storage medium | |
| CN104243617A (en) | Task scheduling method and system facing mixed load in heterogeneous cluster | |
| CN103023980A (en) | Method and system for processing user service request by cloud platform | |
| WO2017000601A1 (en) | Software compiling method and apparatus | |
| CN102316156A (en) | Method for distributing and processing dynamically extensible task | |
| CN109491788A (en) | A kind of virtual platform implementation of load balancing and device | |
| CN102790715B (en) | A kind of resource allocation system based on the Internet and method | |
| CN103905484B (en) | Handle the method and device of hypertext transfer protocol requests | |
| KR20130085713A (en) | A server, an arithmatic processing method and a system thereof | |
| CN104317635B (en) | Dynamic resource scheduling method and system under hybrid task |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 100871, Beijing, Haidian District Cheng Fu Road 298, founder building, 9 floor Patentee after: PEKING UNIVERSITY FOUNDER GROUP Co.,Ltd. Patentee after: FOUNDER APABI TECHNOLOGY Ltd. Patentee after: PKU FOUNDER INFORMATION INDUSTRY GROUP CO.,LTD. Address before: 100871, Beijing, Haidian District Cheng Fu Road 298, founder building, 9 floor Patentee before: PEKING UNIVERSITY FOUNDER GROUP Co.,Ltd. Patentee before: FOUNDER APABI TECHNOLOGY Ltd. Patentee before: FOUNDER INFORMATION INDUSTRY HOLDINGS Co.,Ltd. |
|
| CP01 | Change in the name or title of a patent holder | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220919 Address after: 3007, Hengqin international financial center building, No. 58, Huajin street, Hengqin new area, Zhuhai, Guangdong 519031 Patentee after: New founder holdings development Co.,Ltd. Patentee after: FOUNDER APABI TECHNOLOGY Ltd. Address before: 100871, Beijing, Haidian District Cheng Fu Road 298, founder building, 9 floor Patentee before: PEKING UNIVERSITY FOUNDER GROUP Co.,Ltd. Patentee before: FOUNDER APABI TECHNOLOGY Ltd. Patentee before: PKU FOUNDER INFORMATION INDUSTRY GROUP CO.,LTD. |
|
| TR01 | Transfer of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180403 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |