CN107743246A - Task processing method, system and data handling system - Google Patents
Task processing method, system and data handling system Download PDFInfo
- Publication number
- CN107743246A CN107743246A CN201710060155.8A CN201710060155A CN107743246A CN 107743246 A CN107743246 A CN 107743246A CN 201710060155 A CN201710060155 A CN 201710060155A CN 107743246 A CN107743246 A CN 107743246A
- Authority
- CN
- China
- Prior art keywords
- processing
- node
- data
- task
- central node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000012545 processing Methods 0.000 claims abstract description 222
- 238000003860 storage Methods 0.000 claims abstract description 36
- 238000000034 method Methods 0.000 claims abstract description 27
- 238000005194 fractionation Methods 0.000 claims abstract description 11
- 238000009826 distribution Methods 0.000 claims description 15
- 235000013399 edible fruits Nutrition 0.000 claims 2
- 230000005540 biological transmission Effects 0.000 description 11
- 238000005516 engineering process Methods 0.000 description 6
- 238000012546 transfer Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000000151 deposition Methods 0.000 description 2
- 230000004888 barrier function Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 230000008713 feedback mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234336—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
- H04N21/64723—Monitoring of network processes or resources, e.g. monitoring of network load
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Security & Cryptography (AREA)
- Multi Processors (AREA)
Abstract
The embodiments of the invention provide a kind of task processing method, system and data handling system, task processing system, which includes central node and multiple processing nodes, method, to be included：Central node obtains waiting task；The central node obtains pending data corresponding with the waiting task from the storage system；The pending data is split as multiple subdatas by the central node；The multiple subdata is distributed at least one processing node and carries out data processing by the central node；The central node merges the subdata after each processing node processing.The setting that the present invention passes through central node, processing node is allowd to obtain data directly at the central node, simultaneously as central node has carried out data fractionation, therefore each processing node need to only obtain the data of processing needed for oneself at central node, greatly reduce network burden.
Description
Technical field
The present invention relates to distributed computing technology field, more particularly to task processing method, system and data processing system
System.
Background technology
With the development of science and technology, distributed computing technology is also widely applied.Such as：In video code conversion field, often
The transcoding task of one video is distributed into multiple processing nodes and carries out transcoding.
When existing distributed computing technology carries out task processing, each node that handles is required for obtaining from storage system respectively
Complete pending data is taken, then therefrom selected part data are handled, and this undoubtedly adds network burden.When certain processing
When node is to data processing failure, it is necessary to another processing node obtains complete pending data from storage system again,
Equally add network burden.
How to solve the technical problem is still one technical barrier urgently to be resolved hurrily of those skilled in the art.
The content of the invention
The purpose of the embodiment of the present invention is to provide a kind of task processing method, system and data handling system, to realize
Reduce network burden.Concrete technical scheme is as follows：
A kind of task processing method, applied in the task processing system communicated to connect with storage system, at the task
Reason system, which includes central node and multiple processing nodes, methods described, to be included：
The central node obtains waiting task；
The central node obtains pending data corresponding with the waiting task from the storage system；
The pending data is split as multiple subdatas by the central node；
The multiple subdata is distributed at least one processing node and carries out data processing by the central node；
The central node merges the subdata after each processing node processing.
Optionally, methods described also includes：
The data obtained after merging are sent to the storage system by the central node.
Optionally, the pending data is split as multiple subdatas by the central node, including：
The central node obtains the working condition of each processing node, and executable task is determined according to the working condition
Handle number of nodes；
The pending data is split as the subdata of the processing number of nodes by the central node.
Optionally, after the processing number of nodes that executable task is determined according to the working condition, the center
The pending data is split as multiple subdatas by node, in addition to：
Determine the disposal ability of the processing node of each executable task；
The pending data is split as the subdata of the processing number of nodes by the central node, including：
The central node splits the pending data according to the disposal ability of the processing node of each executable task
For the subdata of the processing number of nodes, the place of the data volume of each subdata with the processing node of an executable task
Reason ability matches.
Optionally, the multiple subdata is distributed at least one processing node in the central node and enters line number
After processing, methods described also includes：
The central node obtains the result that each processing node carries out data processing；
The result is removed the place by the central node for the data distribution handled by the processing node of failure
Manage result and carry out data processing for other processing nodes outside the processing node of failure.
A kind of task processing system, communicated to connect with storage system, the task processing system includes central node and more
Individual processing node, the central node include：Task obtaining unit, data acquiring unit, data split cells, data distribution list
Member and data combination unit,
The task obtaining unit, for obtaining waiting task；
The data acquiring unit, it is corresponding with the waiting task pending for being obtained from the storage system
Data；
The data split cells, for the pending data to be split as into multiple subdatas；
The data allocation unit, enter line number for the multiple subdata to be distributed at least one processing node
According to processing；
The data combination unit, for the subdata after each processing node processing to be merged.
Optionally, the data split cells, including：Node state obtains subelement and splits subelement,
The node state obtains subelement, for obtaining the working condition of each processing node, according to the working condition
It is determined that the processing number of nodes of executable task；
The fractionation subelement, for the pending data to be split as to the subdata of the processing number of nodes.
Optionally, the data split cells, in addition to：Disposal ability determination subelement, in the node state
After obtaining the processing number of nodes that subelement determines executable task according to the working condition, the place of each executable task is determined
Manage the disposal ability of node；
The fractionation subelement, is specifically used for：According to the disposal ability of the processing node of each executable task, treated described
Processing data is split as the subdata of the processing number of nodes, the data volume of each subdata with executable task
The disposal ability of processing node matches.
Optionally, the central node also includes：Result obtaining unit and reallocation unit,
The result obtaining unit, for distributing to the multiple subdata at least in the data allocation unit
After one processing node carries out data processing, the result that each processing node carries out data processing is obtained；
The reallocation unit, for the result to be removed for the data distribution handled by the processing node of failure
The result carries out data processing for other processing nodes outside the processing node of failure.
A kind of data handling system, including：Storage system, task distribution system and multiple tasks processing system, described
Processing system of being engaged in is the above-mentioned task processing system of any one,
The task distribution system is used to distribute waiting task for each task processing system；
The storage system is used to store pending data corresponding with the waiting task.
Task processing method, system and data handling system provided in an embodiment of the present invention, during task processing system includes
Centre node and multiple processing nodes, method include：Central node obtains waiting task；The central node stores system from described
Pending data corresponding with the waiting task is obtained in system；The pending data is split as more by the central node
Individual subdata；The multiple subdata is distributed at least one processing node and carries out data processing by the central node；
The central node merges the subdata after each processing node processing.Setting of the invention by central node,
Processing node is allowd to obtain data directly at the central node, simultaneously as central node has carried out data fractionation, because
This each processing node need to only obtain the data of processing needed for oneself at central node, greatly reduce network burden.Together
When, because data transmission times greatly reduce between node and storage system so that data transmission procedure by unstable networks because
The probability that element influences is smaller, improves the stability of data transfer.
Certainly, any product or method for implementing the present invention must be not necessarily required to reach all the above excellent simultaneously
Point.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
There is the required accompanying drawing used in technology description to be briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with
Other accompanying drawings are obtained according to these accompanying drawings.
Fig. 1 is a kind of structural representation of task processing system provided in an embodiment of the present invention；
Fig. 2 is a kind of flow chart of task processing method provided in an embodiment of the present invention；
Fig. 3 is the flow chart of another task processing method provided in an embodiment of the present invention；
Fig. 4 is the flow chart of another task processing method provided in an embodiment of the present invention；
Fig. 5 is the flow chart of another task processing method provided in an embodiment of the present invention；
Fig. 6 is a kind of structural representation of the central location of task processing system provided in an embodiment of the present invention；
Fig. 7 is a kind of structural representation of data handling system provided in an embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, rather than whole embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained every other under the premise of creative work is not made
Embodiment, belong to the scope of protection of the invention.
As shown in figure 1, a kind of task processing method provided in an embodiment of the present invention, communicates applied to storage system 001
In the task processing system 002 of connection, the task processing system 002 includes central node 100 and multiple processing nodes 200,
As shown in Fig. 2 this method can include：
S100, the central node obtain waiting task；
Optionally, the task processing system 002 communicated to connect with storage system 001 can have multiple, each task processing
Central node 100 and multiple processing nodes 200 in system 002 can be respectively positioned in default geographic area or each task processing
The distance between central node 100 and multiple processing nodes 200 in system 002 are less than the first pre-determined distance.
So, can of the present invention fully utilizes the system of various regions, and each local system is both configured into one
Individual task processing system.
The present invention can be multiplexed existing hierarchical distributed networks such as content distributing network, without resetting equipment again,
Save equipment and bandwidth cost.Meanwhile pass through division of the present invention to task processing system so that largely set in certain region
It is standby that a task processing system can be used as to be used to perform task, improve the speed of tasks carrying so that existing layering point
There is cloth network stronger mass data to support and more excellent set expandability.
S200, the central node 100 are obtained from treat corresponding with the waiting task from the storage system 001
Manage data；
Wherein, waiting task can be video code conversion task, and pending data can be video file.
Wherein, storage system 001 and the distance of task processing system 002 can be more than the second pre-determined distance, it is possible to understand that
, for cost consideration, storage system 001 is often not arranged in the same area with task processing system.In this case, originally
The central node 100 of invention from storage system 001 after pending data is obtained, it is possible to is distributed after directly splitting to each
Node 200 is handled, pending data is obtained from storage system 001 respectively without each processing node 200, reduces the data of transmission
Amount.
The pending data is split as multiple subdatas by S300, the central node 100；
The multiple subdata is distributed at least one processing node 200 and carried out by S400, the central node 100
Data processing；
S500, the central node 100 by it is each it is described processing node 200 handle after subdata merge.
In other embodiments of the present invention, can also include after step S500：
The data obtained after merging are sent to the storage system 001 by the central node 100.
A kind of task processing method provided in an embodiment of the present invention, passes through the setting of central node so that processing node can
To obtain data directly at the central node, simultaneously as central node has carried out data fractionation, therefore each processing node is only
The data of processing needed for oneself need to be obtained at central node, greatly reduce network burden.Simultaneously as node and depositing
Data transmission times greatly reduce between storage system so that the probability that data transmission procedure is influenceed by unstable networks factor is more
It is small, improve the stability of data transfer.
As shown in figure 3, in another task processing method provided in an embodiment of the present invention, step S300 shown in Fig. 1 can be with
Including：
S310, the central node 100 obtain the working condition of each processing node 200, are determined according to the working condition
The processing number of nodes of executable task；
Specifically, when central node 100 can not obtain the working condition of certain processing node, then central node 100 determines
The processing node is the processing node of not executable task.Wherein, central node 100 can not obtain the work shape of certain processing node
The situation of state have it is a variety of, such as：The network connection of central node 100 and the processing node is gone wrong, or, the processing node is delayed
Machine.
Wherein, handling the working condition of node 200 can include：Handle load information of node 200 etc..Optionally, when
When handling the load of node 200 less than default load, it may be determined that the processing node 200 is the processing node of executable task.
The pending data is split as the subdata of the processing number of nodes by S320, the central node 100.
So, the processing node of each executable task can be handled a subdata.
On the basis of embodiment illustrated in fig. 3, as shown in figure 4, another task processing method provided in an embodiment of the present invention,
It can include：
S100, the central node 100 obtain waiting task；
S200, the central node 100 are obtained from treat corresponding with the waiting task from the storage system 001
Manage data；
S310, the central node 100 obtain the working condition of each processing node 200, are determined according to the working condition
The processing number of nodes of executable task；
S311, the disposal ability for handling node for determining each executable task；
Wherein it is possible to according to the physical device parameter of each processing node (such as：CPU parameters) determine to handle the processing energy of node
Power, the disposal ability of processing node can also be determined according to the loading condition of processing node.
S321, the central node 100, will be described pending according to the disposal ability of the processing node of each executable task
Data are split as the subdata of the processing number of nodes, the processing of the data volume of each subdata with an executable task
The disposal ability of node matches.
Step S321 is a kind of step S320 specific implementation.
The multiple subdata is distributed at least one processing node 200 and carried out by S400, the central node 100
Data processing, wherein, the data volume and the disposal ability of the processing node for distributing to the subdata of processing node match；
By step S321 and step S400 allow for central node distribute to it is each processing node subdata data volume
Disposal ability with handling node matches, and is advantageous to improve data-handling efficiency.
S500, the central node 100 by it is each it is described processing node 200 handle after subdata merge.
, can be with after step S400 as shown in figure 5, another task processing method provided in an embodiment of the present invention
Including：
S410, the central node 100 obtain the result that each processing node 200 carries out data processing, are locating
Reason result performs step S500 when being successfully, otherwise performs step S420；
S420, the central node 100 give the result for the data distribution handled by the processing node of failure
Other processing nodes in addition to the result is the processing node of failure carry out data processing.
After data are reassigned to other processing nodes by step S420, step S410 can receive other processing nodes
Result and continue to judge whether result is success.Just ensure that data are successfully processed by this feedback mechanism,
The reliability of system is improved, improves the concurrently execution amount of task, improves the data throughput capabilities of system.
Task processing method provided in an embodiment of the present invention, when handling node processing data failure, another processing node
Without obtain data again from storage system, but the subdata of processing failure before is sent to directly by central node another
Node is handled, the data volume of transmission is greatly reduced, reduces network burden.Simultaneously as node and storage system it
Between data transmission times greatly reduce so that the probability that data transmission procedure is influenceed by unstable networks factor is smaller, improves
The stability of data transfer.
Corresponding with above method embodiment, the embodiment of the present invention additionally provides a kind of task processing system.
Incorporated by reference to Fig. 1 and Fig. 6, a kind of task processing system provided in an embodiment of the present invention, with the communication link of storage system 001
Connect, the task processing system includes central node 100 and multiple processing nodes 200, and the central node 100 includes：Task
Obtaining unit 810, data acquiring unit 820, data split cells 830, data allocation unit 840 and data combination unit 850,
The task obtaining unit 810, for obtaining waiting task；
The data acquiring unit 820, it is corresponding with the waiting task for being obtained from the storage system 001
Pending data；
The data split cells 830, for the pending data to be split as into multiple subdatas；
The data allocation unit 840, for the multiple subdata to be distributed at least one processing node 200
Carry out data processing；
The data combination unit 850, merged for the subdata after each processing node 200 is handled.
A kind of task processing system provided in an embodiment of the present invention, passes through the setting of central node so that processing node can
To obtain data directly at the central node, simultaneously as central node has carried out data fractionation, therefore each processing node is only
The data of processing needed for oneself need to be obtained at central node, greatly reduce network burden.Simultaneously as node and depositing
Data transmission times greatly reduce between storage system so that the probability that data transmission procedure is influenceed by unstable networks factor is more
It is small, improve the stability of data transfer.
In other embodiments of the present invention, the central node 100 can also include：Data transmission unit, for that will close
The data obtained after and are sent to the storage system 001.
Wherein, the data split cells 830, can include：Node state obtains subelement and splits subelement,
The node state obtains subelement, for obtaining the working condition of each processing node 200, according to the work shape
State determines the processing number of nodes of executable task；
The fractionation subelement, for the pending data to be split as to the subdata of the processing number of nodes.
Further, the data split cells 830, can also include：Disposal ability determination subelement, in the section
After dotted state obtains the processing number of nodes that subelement determines executable task according to the working condition, each executable is determined
The disposal ability of the processing node of business；
The fractionation subelement, is specifically used for：According to the disposal ability of the processing node of each executable task, treated described
Processing data is split as the subdata of the processing number of nodes, the data volume of each subdata with executable task
The disposal ability of processing node matches.
In an alternative embodiment of the invention, the central node 100 can also include：Result obtaining unit and divide again
With unit,
The result obtaining unit, for distributing to the multiple subdata in the data allocation unit 840
After at least one processing node carries out data processing, the result that each processing node carries out data processing is obtained；
The reallocation unit, for the result to be removed for the data distribution handled by the processing node of failure
The result carries out data processing for other processing nodes outside the processing node of failure.
As shown in fig. 7, the embodiment of the present invention additionally provides a kind of data handling system, can include：Storage system 001,
Task distribution system 003 and multiple tasks processing system 002, the task processing system 002 are what embodiment illustrated in fig. 6 provided
Task processing system,
The task distribution system 003 is used to distribute waiting task for each task processing system；
The storage system 001 is used to store pending data corresponding with the waiting task.
Specifically, task distribution system 003 can be according to the network condition, Regional Distribution, processing of each task processing system
Number of nodes, processing node computing capability, loading condition etc. distribute waiting task for each task processing system.
It should be noted that herein, such as first and second or the like relational terms are used merely to a reality
Body or operation make a distinction with another entity or operation, and not necessarily require or imply and deposited between these entities or operation
In any this actual relation or order.Moreover, term " comprising ", "comprising" or its any other variant are intended to
Nonexcludability includes, so that process, method, article or equipment including a series of elements not only will including those
Element, but also the other element including being not expressly set out, or it is this process, method, article or equipment also to include
Intrinsic key element.In the absence of more restrictions, the key element limited by sentence "including a ...", it is not excluded that
Other identical element also be present in process, method, article or equipment including the key element.
Each embodiment in this specification is described by the way of related, identical similar portion between each embodiment
Divide mutually referring to what each embodiment stressed is the difference with other embodiment.It is real especially for system
For applying example, because it is substantially similar to embodiment of the method, so description is fairly simple, related part is referring to embodiment of the method
Part explanation.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is all
Any modification, equivalent substitution and improvements made within the spirit and principles in the present invention etc., are all contained in protection scope of the present invention
It is interior.
Claims (10)
- A kind of 1. task processing method, it is characterised in that applied in the task processing system communicated to connect with storage system, institute State task processing system includes including central node and multiple processing nodes, methods described：The central node obtains waiting task；The central node obtains pending data corresponding with the waiting task from the storage system；The pending data is split as multiple subdatas by the central node；The multiple subdata is distributed at least one processing node and carries out data processing by the central node；The central node merges the subdata after each processing node processing.
- 2. according to the method for claim 1, it is characterised in that methods described also includes：The data obtained after merging are sent to the storage system by the central node.
- 3. according to the method for claim 1, it is characterised in that the pending data is split as more by the central node Individual subdata, including：The central node obtains the working condition of each processing node, and the processing of executable task is determined according to the working condition Number of nodes；The pending data is split as the subdata of the processing number of nodes by the central node.
- 4. according to the method for claim 3, it is characterised in that executable task is determined according to the working condition described Processing number of nodes after, the pending data is split as multiple subdatas by the central node, in addition to：Determine the disposal ability of the processing node of each executable task；The pending data is split as the subdata of the processing number of nodes by the central node, including：The pending data is split as institute by the central node according to the disposal ability of the processing node of each executable task State the subdata of processing number of nodes, the processing energy of the data volume of each subdata with the processing node of an executable task Power matches.
- 5. method according to any one of claim 1 to 4, it is characterised in that will be the multiple in the central node After subdata distributes at least one processing node progress data processing, methods described also includes：The central node obtains the result that each processing node carries out data processing；The central node gives the result except the processing is tied for the data distribution handled by the processing node of failure Fruit carries out data processing for other processing nodes outside the processing node of failure.
- 6. a kind of task processing system, it is characterised in that communicated to connect with storage system, the task processing system includes center Node and multiple processing nodes, the central node include：Task obtaining unit, data acquiring unit, data split cells, number According to allocation unit and data combination unit,The task obtaining unit, for obtaining waiting task；The data acquiring unit, for obtaining pending number corresponding with the waiting task from the storage system According to；The data split cells, for the pending data to be split as into multiple subdatas；The data allocation unit, carried out for the multiple subdata to be distributed at least one processing node at data Reason；The data combination unit, for the subdata after each processing node processing to be merged.
- 7. system according to claim 6, it is characterised in that the data split cells, including：Node state obtains son Unit and fractionation subelement,The node state obtains subelement, for obtaining the working condition of each processing node, is determined according to the working condition The processing number of nodes of executable task；The fractionation subelement, for the pending data to be split as to the subdata of the processing number of nodes.
- 8. system according to claim 7, it is characterised in that the data split cells, in addition to：Disposal ability determines Subelement, for obtaining processing nodes of the subelement according to the executable task of working condition determination in the node state After amount, the disposal ability of the processing node of each executable task is determined；The fractionation subelement, is specifically used for：, will be described pending according to the disposal ability of the processing node of each executable task Data are split as the subdata of the processing number of nodes, the processing of the data volume of each subdata with an executable task The disposal ability of node matches.
- 9. the system according to any one of claim 6 to 8, it is characterised in that the central node also includes：Processing knot Fruit obtaining unit and reallocation unit,The result obtaining unit, it is at least one for distributing to the multiple subdata in the data allocation unit After the processing node carries out data processing, the result that each processing node carries out data processing is obtained；The reallocation unit, for the result to be given except described for the data distribution handled by the processing node of failure Result carries out data processing for other processing nodes outside the processing node of failure.
- A kind of 10. data handling system, it is characterised in that including：Storage system, task distribution system and multiple tasks processing system System, the task processing system are the task processing system any one of claim 6 to 9,The task distribution system is used to distribute waiting task for each task processing system；The storage system is used to store pending data corresponding with the waiting task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710060155.8A CN107743246A (en) | 2017-01-24 | 2017-01-24 | Task processing method, system and data handling system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710060155.8A CN107743246A (en) | 2017-01-24 | 2017-01-24 | Task processing method, system and data handling system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107743246A true CN107743246A (en) | 2018-02-27 |
Family
ID=61235032
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710060155.8A Pending CN107743246A (en) | 2017-01-24 | 2017-01-24 | Task processing method, system and data handling system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107743246A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109214132A (en) * | 2018-10-30 | 2019-01-15 | 中国运载火箭技术研究院 | A kind of big Throughput Asynchronous task processing system of non-coupled streaming towards LVC emulation |
| CN109918204A (en) * | 2018-03-22 | 2019-06-21 | 南京地平线机器人技术有限公司 | Data processing system and method |
| CN116743752A (en) * | 2023-08-11 | 2023-09-12 | 山东恒宇电子有限公司 | System for realizing data processing load balance by distributed network communication |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH11242658A (en) * | 1998-02-25 | 1999-09-07 | Digital Vision Laboratories:Kk | Distributed processing method, network system, and node device |
| US20090259713A1 (en) * | 2001-02-24 | 2009-10-15 | International Business Machines Corporation | Novel massively parallel supercomputer |
| CN103188161A (en) * | 2011-12-30 | 2013-07-03 | 中国移动通信集团公司 | Method and system of distributed data loading scheduling |
| CN103716381A (en) * | 2013-12-12 | 2014-04-09 | 华为技术有限公司 | Control method and management node of distributed system |
| CN104102475A (en) * | 2013-04-11 | 2014-10-15 | 腾讯科技（深圳）有限公司 | Method, device and system for processing distributed type parallel tasks |
| CN104378665A (en) * | 2014-11-24 | 2015-02-25 | 深圳市天威视讯股份有限公司 | Distributed transcoding system and method based on digital television |
| CN105740063A (en) * | 2014-12-08 | 2016-07-06 | 杭州华为数字技术有限公司 | Data processing method and apparatus |
-
2017
- 2017-01-24 CN CN201710060155.8A patent/CN107743246A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH11242658A (en) * | 1998-02-25 | 1999-09-07 | Digital Vision Laboratories:Kk | Distributed processing method, network system, and node device |
| US20090259713A1 (en) * | 2001-02-24 | 2009-10-15 | International Business Machines Corporation | Novel massively parallel supercomputer |
| CN103188161A (en) * | 2011-12-30 | 2013-07-03 | 中国移动通信集团公司 | Method and system of distributed data loading scheduling |
| CN104102475A (en) * | 2013-04-11 | 2014-10-15 | 腾讯科技（深圳）有限公司 | Method, device and system for processing distributed type parallel tasks |
| CN103716381A (en) * | 2013-12-12 | 2014-04-09 | 华为技术有限公司 | Control method and management node of distributed system |
| CN104378665A (en) * | 2014-11-24 | 2015-02-25 | 深圳市天威视讯股份有限公司 | Distributed transcoding system and method based on digital television |
| CN105740063A (en) * | 2014-12-08 | 2016-07-06 | 杭州华为数字技术有限公司 | Data processing method and apparatus |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109918204A (en) * | 2018-03-22 | 2019-06-21 | 南京地平线机器人技术有限公司 | Data processing system and method |
| CN109214132A (en) * | 2018-10-30 | 2019-01-15 | 中国运载火箭技术研究院 | A kind of big Throughput Asynchronous task processing system of non-coupled streaming towards LVC emulation |
| CN109214132B (en) * | 2018-10-30 | 2023-06-30 | 中国运载火箭技术研究院 | LVC simulation-oriented uncoupled streaming type large-flux asynchronous task processing system |
| CN116743752A (en) * | 2023-08-11 | 2023-09-12 | 山东恒宇电子有限公司 | System for realizing data processing load balance by distributed network communication |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104731647B (en) | Task processing method and system | |
| CN104699542B (en) | Task processing method and system | |
| CN109146447A (en) | The transparent sharding method of block chain, apparatus and system | |
| CN106790112B (en) | Node operating system integrating lightweight block chains and data updating method | |
| TWI751139B (en) | Business processing system, business processing method and business update method | |
| CN108052398A (en) | A kind of feedback method of order processing state and polymerization payment platform | |
| CN106790660A (en) | A kind of dispositions method and device for realizing distributed memory system | |
| CN108833163A (en) | Creation method, device, computer equipment and the storage medium of Linux virtual server | |
| CN107743246A (en) | Task processing method, system and data handling system | |
| CN105704458A (en) | Container-technology-based video monitoring cloud service platform realization method and system | |
| CN101447891B (en) | Service model self-adapting system and method thereof | |
| CN103294728B (en) | A kind of data processing method and system | |
| CN104202351B (en) | Distributed virtual safety device and the repeater system based on stream using virtual machine | |
| CN103366117B (en) | A kind of viral restorative procedure of infection type and system | |
| CN106610871A (en) | Cloud operating system architecture | |
| CN104618253B (en) | A kind of transmitting message treating method and apparatus of dynamic change | |
| CN110213116B (en) | A kind of network system, method and the communication equipment of network service centralized processing | |
| CN107707592A (en) | Task processing method, node and content distributing network | |
| CN106453143A (en) | Bandwidth setting method, device and system | |
| CN112187864A (en) | Load balancing method and device, storage medium and electronic equipment | |
| CN105786447A (en) | Method and apparatus for processing data by server and server | |
| CN1588921A (en) | Data processing system and data processing method based on multilayer structure | |
| CN107305570A (en) | A kind of data retrieval method and system | |
| CN106959886A (en) | A kind of deploying virtual machine control platform and application method based on cloud computing | |
| CN109951551A (en) | A kind of container mirror image management system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 550001 Gui'an New Area High-end Equipment Industry Park in the Southern Park Applicant after: Guizhou Baishan cloud Polytron Technologies Inc Address before: 100015 5 floor, block E, 201 IT tower, electronic city, 10 Jiuxianqiao Road, Chaoyang District, Beijing. Applicant before: Guizhou white cloud Technology Co., Ltd. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180227 |