CN106713944A - Method and apparatus for processing streaming data task - Google Patents
Method and apparatus for processing streaming data task Download PDFInfo
- Publication number
- CN106713944A CN106713944A CN201611263882.6A CN201611263882A CN106713944A CN 106713944 A CN106713944 A CN 106713944A CN 201611263882 A CN201611263882 A CN 201611263882A CN 106713944 A CN106713944 A CN 106713944A
- Authority
- CN
- China
- Prior art keywords
- flow data
- node
- task
- treatment
- subtask
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 266
- 238000012545 processing Methods 0.000 title claims abstract description 53
- 230000008569 process Effects 0.000 claims abstract description 212
- 230000005540 biological transmission Effects 0.000 claims description 30
- 238000003672 processing method Methods 0.000 claims description 9
- 238000007726 management method Methods 0.000 description 59
- 230000002159 abnormal effect Effects 0.000 description 39
- 238000012544 monitoring process Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 7
- 230000007246 mechanism Effects 0.000 description 5
- 241001269238 Data Species 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001035 drying Methods 0.000 description 1
- 238000005194 fractionation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 241000894007 species Species 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234336—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
- H04N21/2405—Monitoring of the internal components or processes of the server, e.g. server load
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
- H04N21/2407—Monitoring of transmitted content, e.g. distribution time, number of downloads
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
- H04N21/64723—Monitoring of network processes or resources, e.g. monitoring of network load
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Security & Cryptography (AREA)
- General Engineering & Computer Science (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a method and apparatus for processing a streaming data task. The method comprises the following steps: splitting the streaming data task into a plurality of streaming data sub-tasks; allocating the plurality of streaming data sub-tasks to at least one target processing node; calling the target processing nodes to operate at least one processing process to process the plurality of streaming data sub-tasks, wherein one called processing process processes one or more streaming data sub-tasks; and receiving processing results of the streaming data sub-tasks fed back by target processing nodes, and collecting the processing results to obtain a processing result of the streaming data task. The plurality of processing processes do not interfere each other in the execution process, thereby avoiding the situation that the whole node cannot work normally in the case of a problem of one thread in the background art, and effectively guaranteeing the normal work of the target processing nodes.
Description
Technical field
The present invention relates to communication technical field, and in particular to the processing method and a kind of flow data of a kind of flow data task
The processing unit of task.
Background technology
To meet user to watching the different demands of the definition of video, the video stream data that video record end pushes is entered
Row transcoding treatment, the video stream data for obtaining multiple resolution ratio is selected for user.
During current transcoded video flow data, transcoding server is processed multiple transcodings using built-in multiple threads and appointed simultaneously
The video stream data of gained is pushed to default memory space by business, one transcoding task of each thread process, transcoding after terminating.
Although existing method realizes the transcoding treatment of video stream data, there is following defect：In transcoding server
The multiple threads put are interrelated, and when a certain thread process transcoding task goes wrong, such as thread is stuck or collapses, and can cause
Whole transcoding server cannot normal work, so as to influence the carrying out of other transcoding tasks.
The content of the invention
In view of the above problems, it is proposed that the present invention so as to provide one kind overcome above mentioned problem or at least in part solve on
State the processing method of the flow data task of problem and the processing unit of corresponding flow data task.
According to one aspect of the present invention, there is provided a kind of processing method of flow data task, including：
Flow data task is split into multiple flow data subtasks；
Multiple flow data subtasks are distributed at least one target and processes node；
Call each target treatment node to run at least one treatment progress and process the multiple flow data subtask, wherein,
One treatment progress for being called processes one or more flow data subtasks；
Each target treatment node is received for the result of flow data subtask feedback, is collected and is obtained the flow data times
The result of business.
Alternatively, described distribute at least one target treatment node multiple flow data subtasks includes：
The load condition of each treatment node is obtained from management node；
Choose load condition and meet at least one target treatment node for imposing a condition, and section is processed to the target of each selection
Point issues at least one flow data subtask.
Alternatively, each treatment node one task queue of correspondence, is issued to described to the target treatment node of each selection
Before a few flow data subtask, methods described also includes：
Obtain the task queue that each target processes node；
The sequence of flow data subtask to be issued in each task queue is set, makes the treatment of each flow data subtask
The difference of time is less than given threshold.
Alternatively, the sequence for setting flow data subtask to be issued in each task queue, makes each flow data
The difference of the process time of subtask includes less than given threshold：
Existing flow data subtask in the task queue of node is processed according to each target, each fluxion to be issued is predicted
According to the process time of subtask；
Determine process time target flow data subtask the latest；
Process time according to the target flow data subtask arranges the treatment of each flow data subtask to be issued
Time.
Alternatively, the task queue for obtaining each target treatment node includes：
Task queue is obtained from the managing process of each target treatment node.
Alternatively, it is described flow data task is split as multiple flow data subtasks before, methods described also includes：
Receive the flow data process instruction for carrying flow data address；
By accessing the flow data address, pending flow data is downloaded.
Alternatively, the flow data task is transcoding task, described that flow data task is split as into multiple flow data times
Business includes：
According to various transcoded formats that the flow data process instruction is carried, it is right that the flow data process task is split as
Answer multiple flow data subtasks of various transcoded formats.
Alternatively, the flow data task is transcoding task, described that flow data task is split as into multiple flow data times
Business includes：
Extract the service identification that the flow data process instruction is carried；
Search and be directed to the default various transcoded formats of the service identification；
According to the various transcoded formats searched, the flow data process task is split as multiple flow data subtasks.
Alternatively, methods described also includes：
Monitor the processing progress of each treatment progress；
The processed corresponding flow data in flow data subtask is pushed into storage server, for the transmission of flow data
The client end of playing back of node or flow data is called.
Alternatively, methods described also includes：
The treatment that reception processing process reports to managing process notifies, and is treatment in current goal treatment node extremely
Redistribute treatment progress in abnormal flow data subtask.
Alternatively, methods described also includes：
Receive current goal treatment node and report to the treatment notice extremely of management node, and be in other treatment nodes
Process abnormal flow data subtask and redistribute a target treatment node；
A treatment progress for calling the target redistributed to process node is carried out to the abnormal flow data subtask for the treatment of
Treatment.
According to another aspect of the present invention, there is provided a kind of processing unit of flow data task, including：
Task splits module, for flow data task to be split into multiple flow data subtasks；
Subtask distribute module, node is processed for multiple flow data subtasks to be distributed at least one target；
Treatment progress calling module is described more for calling each target treatment node to run the treatment of at least one treatment progress
Individual flow data subtask, wherein, a treatment progress for being called processes one or more flow data subtasks；
Result receiver module, for receiving treatment knot of each target treatment node for flow data subtask feedback
Really, the result for obtaining the flow data task is collected.
Alternatively, the subtask distribute module includes：
Load condition acquisition submodule, the load condition for obtaining each treatment node from management node；
Target processes node selection submodule, and at least one target for imposing a condition treatment is met for choosing load condition
Node, and issue at least one flow data subtask to the target treatment node of each selection.
Alternatively, described device also includes：
Task queue acquisition module, for issuing at least one flow data to the target treatment node of each selection described
Before task, the task queue that each target processes node, each treatment node one task queue of correspondence are obtained；
Sequence sets submodule, for setting the sequence of flow data subtask to be issued in each task queue, makes each
The difference of the process time of individual flow data subtask is less than given threshold.
Alternatively, the sequence sets submodule and includes：
Process time predicting unit, appoints for existing flow data in the task queue that node is processed according to each target
Business, predicts the process time of each flow data subtask to be issued；
Target flow data subtask determining unit, for determining process time target flow data subtask the latest；
Process time arranging unit, arranges each to wait to issue for the process time according to the target flow data subtask
Flow data subtask process time.
Alternatively, the task queue acquisition module, appoints specifically for being obtained from the managing process of each target treatment node
Business queue.
Alternatively, described device also includes：
Process instruction receiver module, for it is described flow data task is split as multiple flow data subtasks before, connect
Receive the flow data process instruction for carrying flow data address；
Flow data download module, for by accessing the flow data address, downloading pending flow data.
Alternatively, the task splits module, specifically for the various transcodings carried according to the flow data process instruction
Form, the flow data process task is split as multiple flow data subtasks of the various transcoded formats of correspondence, the flow data
Task is transcoding task.
Alternatively, the task splits module and includes：
Service identification extracting sub-module, for extracting the service identification that the flow data process instruction is carried, the fluxion
It is transcoding task according to task；
Transcoded format searches submodule, and the default various transcoded formats of the service identification are directed to for searching；
Task divides submodule, for according to the various transcoded formats searched, the flow data process task being split
It is multiple flow data subtasks.
Alternatively, described device also includes：
Processing progress monitoring module, the processing progress for monitoring each treatment progress；
Flow data pushing module, for the processed corresponding flow data in flow data subtask to be pushed into storage service
Device, so that the transmission node of flow data or the client end of playing back of flow data are called.
Alternatively, described device also includes：
First notifies receiver module, and the treatment for reporting to managing process for reception processing process is notified extremely；
Treatment progress redistributes module, for being the abnormal flow data subtask for the treatment of in current goal treatment node
Redistribute treatment progress.
Alternatively, described device also includes：
Second notify receiver module, for receive current goal treatment node report to management node treatment it is extremely logical
Know；
Target treatment node redistributes module, for being the abnormal flow data subtask for the treatment of in other treatment nodes
Redistribute a target treatment node；
Target processes node calling module, for calling the target redistributed to process a treatment progress of node to place
Abnormal flow data subtask is managed to be processed.
According to the embodiment of the present invention, treatment progress in invocation target treatment node to multiple flow data subtasks at
Reason, because multiple treatment progress are independent of each other in the process of implementation, therefore in avoiding background technology, a thread goes wrong
Cause whole node cannot normal work the occurrence of, be effectively ensured target process node normal work.Meanwhile, respectively
The result of flow data subtask and collect the flow data task for obtaining result can as Mission Monitor, son appoint
It is engaged in the foundation of distribution, so that perfect treatment mechanism.
The embodiment of the present invention is in the built-in managing process of each transmission node and is provided with management node, using managing process and
The treatment of management node stream data subtask is managed, when treatment progress occur treatment it is abnormal when, managing process or management
Node can be that treatment progress or treatment node are redistributed in the abnormal flow data subtask for the treatment of, so as to ensure that flow data task
Processing procedure smoothly complete.Because abnormal reparation the in processing procedure is completed by managing process and management node,
Without manually operating, therefore substantial amounts of task can be simultaneously processed, improve task treatment effeciency, shorten the treatment of flow data
The stand-by period of operation.
The embodiment of the present invention monitors the load condition of each treatment node using management node, each by what is extracted from management node
The load condition of node is processed as foundation, is multiple flow data subtask distribution object transmission nodes, so as to ensure multiple places
Manage the load balancing of node.
Described above is only the general introduction of technical solution of the present invention, in order to better understand technological means of the invention,
And can be practiced according to the content of specification, and in order to allow the above and other objects of the present invention, feature and advantage can
Become apparent, below especially exemplified by specific embodiment of the invention.
Brief description of the drawings
By reading the detailed description of hereafter preferred embodiment, various other advantages and benefit is common for this area
Technical staff will be clear understanding.Accompanying drawing is only used for showing the purpose of preferred embodiment, and is not considered as to the present invention
Limitation.And in whole accompanying drawing, identical part is denoted by the same reference numerals.In the accompanying drawings：
The step of Fig. 1 shows a kind of processing method of according to embodiments of the present invention one flow data task flow is illustrated
Figure；
The step of Fig. 2 shows a kind of processing method of according to embodiments of the present invention two flow data task flow is illustrated
Figure；
Fig. 3 shows a kind of structured flowchart of the processing unit of according to embodiments of the present invention three flow data task；
Fig. 4 shows a kind of structured flowchart of the processing unit of according to embodiments of the present invention four flow data task.
Specific embodiment
The exemplary embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although showing the disclosure in accompanying drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
Limited.Conversely, there is provided these embodiments are able to be best understood from the disclosure, and can be by the scope of the present disclosure
Complete conveys to those skilled in the art.
Reference picture 1, show according to embodiments of the present invention one a kind of flow data task processing method the step of flow
Schematic diagram, specifically may include steps of：
Step 101, flow data task is split into multiple flow data subtasks.
Method described in the embodiment of the present invention is applied to the scene that there is the flow data transmitted with streaming manner, such as flows
Transcoded data scene, Stream Data Transmission scene etc..Correspondingly, flow data task can circulate for video stream transcoding task, audio
Code task, video flowing transformation task, audio stream transformation task etc., flow data task carries corresponding flow data.
Performing the system of embodiment of the present invention methods described includes management node and multiple treatment nodes, in each treatment node
Multiple treatment progress are equipped with, wherein, management node is used for the reception and distribution of flow data task, and for multiple treatment section
The ruuning situation of point is managed；Treatment progress is used to process the flow data subtask after dividing.At a time a certain treatment
Process processes a subtask, and each treatment progress can be according to task allocation order, successively to distributing to the treatment progress
Multiple subtasks are processed.
Management node and multiple treatment nodes can for content distributing network (Content Delivery Networ k,
CDN the node in), it is possible to use content distributing network is passed to the flow data task between management node and multiple treatment nodes
It is defeated.
Management node can be split into multiple flow data subtasks after flow data task is received, to use treatment
Treatment progress stream data subtask in node is processed.
Can be divided according to the species stream data task of flow data task, for example, when flow data task is video
Circulation code task, task is indicated the video stream transcoding Cheng Chaoqing of 1080P forms, three kinds of video flowings of high definition and SD, at this moment managed
Be divided into for video stream transcoding task the flow data of 1080P forms be transcoded into super clear stream data by reason node, by 1080P forms
Flow data is transcoded into high definition flow data and the flow data of 1080P forms is transcoded into three subtasks of SD flow data；Work as stream
When data task is Stream Data Transmission task, the flow data that flow data task is carried can be divided into multiple subflows by management node
Data, each flow data subtask is hop flow data.
Step 102, by multiple flow data subtasks distribute at least one target process node.
In the embodiment of the present invention, flow data subtask completes stream data task by treatment node processing, management node
After fractionation, at least one target treatment node can be selected from multiple treatment node, afterwards appoint multiple flow datas of gained
Business distribution to the target of selection processes node.It can be one or many to distribute to the flow data subtask of a certain target treatment node
It is individual, for example, when flow data subtask is three, three flow data subtasks being distributed to three different targets and processing section
Point, or two flow data subtasks are distributed to an object transmission node, by a remaining flow data subtask distribute to
Another object transmission node.
Can be flow data subtask distribution target treatment node according to many factors, specifically can be according to each treatment section
One or more selection target transmission section in the information such as loading condition, the geographical location information of management node region of point
Point.
Step 103, call each target treatment node to run at least one treatment progress and process the multiple flow data to appoint
Business.
After each object transmission node receives flow data subtask, can be using built-in treatment progress stream data at
Reason.One or more treatment progress can be selected to process flow data subtask according to the rest processing capacity of each treatment progress.
When a certain object transmission node receives multiple flow data subtasks, multiple treatment progress can be called to fluxion
Processed according to subtask, each treatment progress can process one or more flow data subtasks；Can also call at one
Reason process is processed multiple flow data subtasks.
Step 104, each target treatment node of reception collect and obtain described for the result of flow data subtask feedback
The result of flow data task.
In the embodiment of the present invention, target treatment node can obtain the result of flow data subtask and be fed back to
Management node.Result can be presupposed information, for example, the default result when flow data subtask processes completion is 1,
When flow data subtask processes exception, result is 2.
Object transmission node can in several ways determine the result of flow data subtask, for example, object transmission
Node can be detected periodically to treatment progress, so that it is determined that the result of each flow data subtask；Or, work as flow data
When subtask treatment is completed, treatment progress can send the instruction or information for indicating treatment to complete, if object transmission node exists
Instruction or information that the instruction treatment for the treatment of progress transmission is completed are not received in preset duration, then judges flow data subtask
Result is abnormal for treatment, otherwise for treatment is normal.
Further management node can converge to the result of the flow data subtask of each target treatment node feeding back
Always, the result of flow data task is obtained, the disposition of flow data task can be determined according to feedback result.
According to the embodiment of the present invention, treatment progress in invocation target treatment node to multiple flow data subtasks at
Reason, because multiple treatment progress are independent of each other in the process of implementation, therefore in avoiding background technology, a thread goes wrong
Cause whole node cannot normal work the occurrence of, be effectively ensured target process node normal work.Meanwhile, respectively
The result of flow data subtask and collect the flow data task for obtaining result can as Mission Monitor, son appoint
It is engaged in the foundation of distribution, so that perfect treatment mechanism.
Reference picture 2, show according to embodiments of the present invention two a kind of flow data task processing method the step of flow
Schematic diagram, specifically may include steps of：
Step 201, flow data task is split into multiple flow data subtasks.
Method described in the embodiment of the present invention is applied to the scene that there is the flow data transmitted with streaming manner, such as flows
Transcoded data scene, Stream Data Transmission scene etc..
When flow data transcoding scene is applied to, flow data task is transcoding task.Can in advance limit and allow at flow data
Reason instruction be that transcoding instructs the various transcoded formats of carrying, the step of flow data task at this moment is split as into multiple flow data subtasks
Can include：According to various transcoded formats that flow data process instruction is carried, flow data process task is split as correspondence various
Multiple flow data subtasks of transcoded format.For example, flow data process instruction is to be transcoded into the video stream data of 1080P forms
Two kinds of forms of 720P and 480P, carry two kinds of transcoded formats of 720P and 480P, and management node can refer to according to being processed according to flow data
The two kinds of transcoded formats for carrying are made, the video stream data to 1080P forms carries out transcoding, obtains two kinds of forms of 720P and 480P
Video stream data.
Can also in advance in management node storage service mark and the corresponding relation of transcoded format, while limiting at flow data
Reason instruction is that transcoding instruction carries service identification, wherein, service identification can indicate concrete application, can be for Apply Names
AppName, service identification can be one-to-many relation with the corresponding relation of transcoded format, at this moment be split as flow data task
The step of multiple flow data subtasks, can include：The service identification that flow data process instruction is carried is extracted, is searched and is directed to the industry
The default various transcoded formats of business mark, according to the various transcoded formats searched, multiple are split as by flow data process task
Flow data subtask.
, it is necessary to obtain flow data before flow data task is split as into multiple flow data subtasks.Pre- transcoding in practice
The business end of flow data can allow flow data process instruction to carry flow data when flow data process instruction is sent to management node
Flow data address, after management node receives the flow data address of flow data, flow data can be obtained by following steps：
The flow data process instruction for carrying flow data address is received, by accessing the flow data address, pending flow data is downloaded.
For example, the flow data address that the flow data process instruction that management node is received is carried is * * */AppName/SN,
Wherein AppName is service identification, and SN is client identification, and management node can be by accessing flow data address acquisition fluxion
According to, and according to the service identification and the corresponding relation of transcoded format for prestoring, it is determined that required various transcoded formats, enter one
Walk according to the various transcoded formats for determining, flow data task is split into multiple flow data subtasks, each flow data subtask
A kind of transcoded format of correspondence.
Step 202, by multiple flow data subtasks distribute at least one target process node.
Management node is used to be managed multiple treatment node, manages the loading condition of each treatment node, each treatment section
Ruuning situation for the treatment of progress etc. in point, management node can record the loading condition of each treatment node and carry out ruuning situation
And constantly update.
Management node can send the target treatment section for multiple flow data subtasks to scheduling in the embodiment of the present invention
The distribution request of point, scheduling can be multiple flow data subtask distribution target treatment sections according to the load condition of each treatment node
Point, can specifically select the treatment node of light load to process node as target, to ensure the load of multiple treatment nodes
Equilibrium, the number of the target treatment node of distribution can be one or more, and the target treatment node that then scheduling will be distributed is anti-
It is fed to management node.
Because record has the loading condition of multiple treatment nodes in management node, therefore this step appoints multiple flow datas
The step of business distribution processes node at least one target can include：The load shape of each treatment node is obtained from management node
State, chooses load condition and meets at least one target treatment node for imposing a condition, and under the target treatment node of each selection
Send out at least one flow data subtask.Wherein, imposing a condition can include various restriction contents, and (N is for just for such as N of most lightly loaded
Integer) individual treatment node, the treatment node of the number less than M (M is positive integer) of waiting task etc..One target processes node
One or more flow data subtasks can be distributed.
Each treatment node one task queue of correspondence, have recorded distribution to the treatment in the embodiment of the present invention in task queue
The Queue sequence of multiple flow data subtasks of node.The embodiment of the present invention is being determined at least one target according to load condition
After reason node, to make the treatment end time of multiple flow data subtasks close or identical, can be to the target of each selection
Before reason node issues at least one flow data subtask, the Queue sequence to flow data subtask in each treatment node is adjusted
It is whole.Specifically, the step of Queue sequence to flow data subtask in each treatment node is adjusted can include：Obtain each mesh
The task queue of mark treatment node；The sequence of flow data subtask to be issued in each task queue is set, makes each fluxion
Given threshold is less than according to the difference of the process time of subtask, wherein given threshold can be according to being actually configured.
The sequence that flow data subtask to be issued is set in each task queue described further, makes each flow data
The step of difference of the process time of task is less than given threshold can include：In task queue according to each target treatment node
Existing flow data subtask, predicts the process time of each flow data subtask to be issued；Determine process time the latest
Target flow data subtask；Process time according to the target flow data subtask arranges each flow data subtask to be issued
Process time.
Specifically, it is determined that behind process time target flow data subtask the latest, will distribute to other treatment nodes
The sequence of flow data subtask is adjusted backward, if other flow data subtask treatment terminate and do not arrive in target treatment node
The process time of arrangement, then stream data subtask is processed again after wait intervals.
The embodiment of the present invention is provided with managing process in each transmission node, and managing process is used for what multiple treatment was carried out
Ruuning situation is monitored and manages, and the disposition of flow data subtask is fed back into management node.Each target treatment section
The task queue of point can be arranged in managing process, and correspondingly obtaining the step of each target processes the task queue of node can be with
Including：Task queue is obtained from the managing process of each target treatment node.
The task queue of managing process record can include the multiple tasks subqueue set for multiple treatment progress, from
A certain task subqueue understands the number of the subtask for waiting corresponding treatment progress treatment.It is determined that process time mesh the latest
Behind mark flow data subtask, can the sequence of stream data subtask be entered in a certain task subqueue according to process time the latest
Row adjustment, it is also possible to a suitable task subqueue is selected according to process time the latest, flow data subtask is sorted to this
Business subqueue is last.
Step 203, call each target treatment node to run at least one treatment progress and process the multiple flow data to appoint
Business, wherein, a treatment progress for being called processes one or more flow data subtasks.
Step 204, each target treatment node of reception collect and obtain described for the result of flow data subtask feedback
The result of flow data task.
Step 205, the processing progress for monitoring each treatment progress, by the processed corresponding fluxion in flow data subtask
According to storage server is pushed to, so that the transmission node of flow data or the client end of playing back of flow data are called.
In the embodiment of the present invention, managing process is built-in with each treatment node, it is possible to use managing process monitoring treatment section
The processing progress of multiple treatment progress, if monitor the treatment of flow data subtask terminated, can notify that target is processed in point
The corresponding flow data in flow data subtask that node will have been processed pushes to storage server.Storage server can be default
, can be that flow data process instruction is indicated, can be scheduling is distributed from default multiple storage servers, the present invention
It is not limited herein.
After storage server receives the flow data of push, if receiving the transmission node or client end of playing back of flow data
Call request, then by Stream Data Transmission to transmission node and client end of playing back are for transmission or play.
In the embodiment of the present invention, it is preferable that methods described also includes：
The treatment that reception processing process reports to managing process notifies, and is treatment in current goal treatment node extremely
Redistribute treatment progress in abnormal flow data subtask.
The embodiment of the present invention is entered using the managing process being built in treatment node to the ruuning situation of multiple treatment progress
Row monitoring and management.When treatment progress occurs abnormal for the treatment of certain stream data subtask, such as treatment progress it is stuck or
Collapse, treatment progress can report that treatment is abnormal to be notified to managing process, managing process after receiving that treatment is abnormal and notifying, meeting
Ruuning situation, load condition of other treatment progress in target treatment node etc. is checked and analyzed, further basis
Analysis result selects other treatment progress of operational excellence and light load, and the abnormal flow data subtask for the treatment of is carried out again
Treatment, so as to ensure that smoothly completing for the flow data subtask.
In addition to the mode for notifying to managing process except the treatment of above-mentioned treatment progress active reporting is abnormal, managing process can be according to
The ruuning situation of prefixed time interval regular check treatment progress, it is different to process if detecting treatment progress occurs exception
Redistribute treatment progress in normal flow data subtask.
In the embodiment of the present invention, it is preferable that methods described also includes：
Receive current goal treatment node and report to the treatment notice extremely of management node, and be in other treatment nodes
Process abnormal flow data subtask and redistribute a target treatment node；
A treatment progress for calling the target redistributed to process node is carried out to the abnormal flow data subtask for the treatment of
Treatment.
In the embodiment of the present invention, the managing process in target treatment node is receiving the treatment exception that treatment progress is reported
After notice, if it is determined that heavier loads or load are full for all treatment progress in target treatment node, then managing process
The abnormal notice for the treatment of can be reported to management node.It is extremely logical that management node receives the treatment that a certain target treatment node reports
Can be the abnormal stream for the treatment of in other treatment nodes according to the loading condition of other treatment nodes or other factors after knowing
A target treatment node is redistributed in data subtask, and the flow data subtask for processing abnormal is sent to the mesh redistributed
Mark treatment node is processed.
After newly assigned target treatment node receives the flow data subtask, a treatment progress pair for calling it internal
The subtask is processed.Specifically, each treatment progress can be checked and analyzed to the managing process in target treatment node
Situation, selects operational excellence and the treatment progress of light load processes the subtask.
Embodiment of the present invention methods described be can apply in the scene of video stream transcoding, it is possible to use the management of setting is entered
Journey and management node are managed to video stream transcoding process, when treatment progress occur treatment it is abnormal when, managing process or management
Node can be that treatment progress or treatment node are redistributed in the abnormal flow data transcoding subtask for the treatment of, so as to ensure that video flowing
Transcoding task is smoothly completed.Because abnormal reparation the in processing procedure is completed by managing process and management node, nothing
Need to manually operate, therefore can simultaneously process substantial amounts of transcoding task, meet transcoding demand.
According to the embodiment of the present invention, treatment progress in invocation target treatment node to multiple flow data subtasks at
Reason, because multiple treatment progress are independent of each other in the process of implementation, therefore in avoiding background technology, a thread goes wrong
Cause whole node cannot normal work the occurrence of, be effectively ensured target process node normal work.Meanwhile, respectively
The result of flow data subtask and collect the flow data task for obtaining result can as Mission Monitor, son appoint
It is engaged in the foundation of distribution, so that perfect treatment mechanism.
The embodiment of the present invention is in the built-in managing process of each transmission node and is provided with management node, using managing process and
The treatment of management node stream data subtask is managed, when treatment progress occur treatment it is abnormal when, managing process or management
Node can be that treatment progress or treatment node are redistributed in the abnormal flow data subtask for the treatment of, so as to ensure that flow data task
Processing procedure smoothly complete.Because abnormal reparation the in processing procedure is completed by managing process and management node,
Without manually operating, therefore substantial amounts of task can be simultaneously processed, improve task treatment effeciency, shorten the treatment of flow data
The stand-by period of operation.
The embodiment of the present invention monitors the load condition of each treatment node using management node, each by what is extracted from management node
The load condition of node is processed as foundation, is multiple flow data subtask distribution object transmission nodes, so as to ensure multiple places
Manage the load balancing of node.
Fig. 3 shows a kind of structured flowchart of the processing unit of according to embodiments of the present invention three flow data task.In Fig. 3
The processing unit of flow data task include：
Task splits module 301, for flow data task to be split into multiple flow data subtasks.
Subtask distribute module 302, node is processed for multiple flow data subtasks to be distributed at least one target.
Treatment progress calling module 303, for calling each target treatment node to run at least one treatment progress treatment institute
Multiple flow data subtasks are stated, wherein, a treatment progress for being called processes one or more flow data subtasks.
Result receiver module 304, for receiving treatment of each target treatment node for flow data subtask feedback
As a result, the result for obtaining the flow data task is collected.
According to the embodiment of the present invention, treatment progress in invocation target treatment node to multiple flow data subtasks at
Reason, because multiple treatment progress are independent of each other in the process of implementation, therefore in avoiding background technology, a thread goes wrong
Cause whole node cannot normal work the occurrence of, be effectively ensured target process node normal work.Meanwhile, respectively
The result of flow data subtask and collect the flow data task for obtaining result can as Mission Monitor, son appoint
It is engaged in the foundation of distribution, so that perfect treatment mechanism.
Fig. 4 shows a kind of structured flowchart of the processing unit of according to embodiments of the present invention four flow data task.In Fig. 4
The processing unit of flow data task include：
Task splits module 401, for flow data task to be split into multiple flow data subtasks.
Subtask distribute module 402, node is processed for multiple flow data subtasks to be distributed at least one target.
Treatment progress calling module 403, for calling each target treatment node to run at least one treatment progress treatment institute
Multiple flow data subtasks are stated, wherein, a treatment progress for being called processes one or more flow data subtasks.
Result receiver module 404, for receiving treatment of each target treatment node for flow data subtask feedback
As a result, the result for obtaining the flow data task is collected.
Processing progress monitoring module 405, the processing progress for monitoring each treatment progress.
Flow data pushing module 406, for the processed corresponding flow data in flow data subtask to be pushed into storage
Server, so that the transmission node of flow data or the client end of playing back of flow data are called.
In the embodiment of the present invention, it is preferable that the subtask distribute module includes：
Load condition acquisition submodule, the load condition for obtaining each treatment node from management node；
Target processes node selection submodule, and at least one target for imposing a condition treatment is met for choosing load condition
Node, and issue at least one flow data subtask to the target treatment node of each selection.
In the embodiment of the present invention, it is preferable that described device also includes：
Task queue acquisition module, for issuing at least one flow data to the target treatment node of each selection described
Before task, the task queue that each target processes node, each treatment node one task queue of correspondence are obtained；
Sequence sets submodule, for setting the sequence of flow data subtask to be issued in each task queue, makes each
The difference of the process time of individual flow data subtask is less than given threshold.
In the embodiment of the present invention, it is preferable that the sequence sets submodule to be included：
Process time predicting unit, appoints for existing flow data in the task queue that node is processed according to each target
Business, predicts the process time of each flow data subtask to be issued；
Target flow data subtask determining unit, for determining process time target flow data subtask the latest；
Process time arranging unit, arranges each to wait to issue for the process time according to the target flow data subtask
Flow data subtask process time.
In the embodiment of the present invention, it is preferable that the task queue acquisition module, specifically for processing node from each target
Managing process obtains task queue.
In the embodiment of the present invention, it is preferable that described device also includes：
Process instruction receiver module, for it is described flow data task is split as multiple flow data subtasks before, connect
Receive the flow data process instruction for carrying flow data address；
Flow data download module, for by accessing the flow data address, downloading pending flow data.
In the embodiment of the present invention, it is preferable that the task splits module, specifically for according to the flow data process instruction
The various transcoded formats for carrying, multiple flow datas that the flow data process task is split as the various transcoded formats of correspondence is appointed
Business, the flow data task is transcoding task.
In the embodiment of the present invention, it is preferable that the task splits module to be included：
Service identification extracting sub-module, for extracting the service identification that the flow data process instruction is carried, the fluxion
It is transcoding task according to task；
Transcoded format searches submodule, and the default various transcoded formats of the service identification are directed to for searching；
Task divides submodule, for according to the various transcoded formats searched, the flow data process task being split
It is multiple flow data subtasks.
In the embodiment of the present invention, it is preferable that described device also includes：
First notifies receiver module, and the treatment for reporting to managing process for reception processing process is notified extremely；
Treatment progress redistributes module, for being the abnormal flow data subtask for the treatment of in current goal treatment node
Redistribute treatment progress.
In the embodiment of the present invention, it is preferable that described device also includes：
Second notify receiver module, for receive current goal treatment node report to management node treatment it is extremely logical
Know；
Target treatment node redistributes module, for being the abnormal flow data subtask for the treatment of in other treatment nodes
Redistribute a target treatment node；
Target processes node calling module, for calling the target redistributed to process a treatment progress of node to place
Abnormal flow data subtask is managed to be processed.
According to the embodiment of the present invention, treatment progress in invocation target treatment node to multiple flow data subtasks at
Reason, because multiple treatment progress are independent of each other in the process of implementation, therefore in avoiding background technology, a thread goes wrong
Cause whole node cannot normal work the occurrence of, be effectively ensured target process node normal work.Meanwhile, respectively
The result of flow data subtask and collect the flow data task for obtaining result can as Mission Monitor, son appoint
It is engaged in the foundation of distribution, so that perfect treatment mechanism.
The embodiment of the present invention is in the built-in managing process of each transmission node and is provided with management node, using managing process and
The treatment of management node stream data subtask is managed, when treatment progress occur treatment it is abnormal when, managing process or management
Node can be that treatment progress or treatment node are redistributed in the abnormal flow data subtask for the treatment of, so as to ensure that flow data task
Processing procedure smoothly complete.Because abnormal reparation the in processing procedure is completed by managing process and management node,
Without manually operating, therefore substantial amounts of task can be simultaneously processed, improve task treatment effeciency, shorten the treatment of flow data
The stand-by period of operation.
The embodiment of the present invention monitors the load condition of each treatment node using management node, each by what is extracted from management node
The load condition of node is processed as foundation, is multiple flow data subtask distribution object transmission nodes, so as to ensure multiple places
Manage the load balancing of node.
Algorithm and display be not inherently related to any certain computer, virtual system or miscellaneous equipment provided herein.
Various general-purpose systems can also be used together with based on teaching in this.As described above, construct required by this kind of system
Structure be obvious.Additionally, the present invention is not also directed to any certain programmed language.It is understood that, it is possible to use it is various
Programming language realizes the content of invention described herein, and the description done to language-specific above is to disclose this hair
Bright preferred forms.
In specification mentioned herein, numerous specific details are set forth.It is to be appreciated, however, that implementation of the invention
Example can be put into practice in the case of without these details.In some instances, known method, structure is not been shown in detail
And technology, so as not to obscure the understanding of this description.
Similarly, it will be appreciated that in order to simplify one or more that the disclosure and helping understands in each inventive aspect, exist
Above to the description of exemplary embodiment of the invention in, each feature of the invention is grouped together into single implementation sometimes
In example, figure or descriptions thereof.However, the method for the disclosure should be construed to reflect following intention：I.e. required guarantor
The application claims of shield features more more than the feature being expressly recited in each claim.More precisely, such as following
Claims reflect as, inventive aspect is all features less than single embodiment disclosed above.Therefore,
Thus the claims for following specific embodiment are expressly incorporated in the specific embodiment, and wherein each claim is in itself
All as separate embodiments of the invention.
Those skilled in the art are appreciated that can be carried out adaptively to the module in the equipment in embodiment
Change and they are arranged in one or more equipment different from the embodiment.Can be the module or list in embodiment
Unit or component be combined into a module or unit or component, and can be divided into addition multiple submodule or subelement or
Sub-component.In addition at least some in such feature and/or process or unit exclude each other, can use any
Combine to all features disclosed in this specification (including adjoint claim, summary and accompanying drawing) and so disclosed appoint
Where all processes or unit of method or equipment are combined.Unless expressly stated otherwise, this specification (including adjoint power
Profit is required, summary and accompanying drawing) disclosed in each feature can the alternative features of or similar purpose identical, equivalent by offer carry out generation
Replace.
Although additionally, it will be appreciated by those of skill in the art that some embodiments described herein include other embodiments
In included some features rather than further feature, but the combination of the feature of different embodiments means in of the invention
Within the scope of and form different embodiments.For example, in the following claims, embodiment required for protection is appointed
One of meaning mode can be used in any combination.
All parts embodiment of the invention can be realized with hardware, or be run with one or more processor
Software module realize, or with combinations thereof realize.It will be understood by those of skill in the art that can use in practice
Microprocessor or digital signal processor (DSP) are come in the processing equipment for realizing flow data task according to embodiments of the present invention
Some or all parts some or all functions.The present invention is also implemented as performing side as described herein
Some or all equipment or program of device (for example, computer program and computer program product) of method.It is such
Realize that program of the invention can be stored on a computer-readable medium, or there can be the shape of one or more signal
Formula.Such signal can be downloaded from internet website and obtained, or be provided on carrier signal, or with any other shape
Formula is provided.
It should be noted that above-described embodiment the present invention will be described rather than limiting the invention, and ability
Field technique personnel can design alternative embodiment without departing from the scope of the appended claims.In the claims,
Any reference symbol being located between bracket should not be configured to limitations on claims.Word "comprising" is not excluded the presence of not
Element listed in the claims or step.Word "a" or "an" before element is not excluded the presence of as multiple
Element.The present invention can come real by means of the hardware for including some different elements and by means of properly programmed computer
It is existing.If in the unit claim for listing equipment for drying, several in these devices can be by same hardware branch
To embody.The use of word first, second, and third does not indicate that any order.These words can be explained and run after fame
Claim.
The invention discloses A1, a kind of processing method of flow data task, including：
Flow data task is split into multiple flow data subtasks；
Multiple flow data subtasks are distributed at least one target and processes node；
Call each target treatment node to run at least one treatment progress and process the multiple flow data subtask, wherein,
One treatment progress for being called processes one or more flow data subtasks；
Each target treatment node is received for the result of flow data subtask feedback, is collected and is obtained the flow data times
The result of business.
A2, the method according to A1, wherein, it is described to distribute to the treatment of at least one target multiple flow data subtasks
Node includes：
The load condition of each treatment node is obtained from management node；
Choose load condition and meet at least one target treatment node for imposing a condition, and section is processed to the target of each selection
Point issues at least one flow data subtask.
A3, the method according to A1, wherein, each treatment node one task queue of correspondence, described to each selection
Before target treatment node issues at least one flow data subtask, methods described also includes：
Obtain the task queue that each target processes node；
The sequence of flow data subtask to be issued in each task queue is set, makes the treatment of each flow data subtask
The difference of time is less than given threshold.
A4, the method according to A3, wherein, it is described that flow data subtask to be issued is set in each task queue
Sequence, make the difference of the process time of each flow data subtask includes less than given threshold：
Existing flow data subtask in the task queue of node is processed according to each target, each fluxion to be issued is predicted
According to the process time of subtask；
Determine process time target flow data subtask the latest；
Process time according to the target flow data subtask arranges the treatment of each flow data subtask to be issued
Time.
A5, the method according to A3, wherein, the task queue for obtaining each target treatment node includes：
Task queue is obtained from the managing process of each target treatment node.
A6, the method according to A1, wherein, it is described by flow data task be split as multiple flow data subtasks it
Before, methods described also includes：
Receive the flow data process instruction for carrying flow data address；
By accessing the flow data address, pending flow data is downloaded.
A7, the method according to A6, wherein, the flow data task is transcoding task, described to tear flow data task open
Being divided into multiple flow data subtasks includes：
According to various transcoded formats that the flow data process instruction is carried, it is right that the flow data process task is split as
Answer multiple flow data subtasks of various transcoded formats.
A8, the method according to A6, wherein, the flow data task is transcoding task, described to tear flow data task open
Being divided into multiple flow data subtasks includes：
Extract the service identification that the flow data process instruction is carried；
Search and be directed to the default various transcoded formats of the service identification；
According to the various transcoded formats searched, the flow data process task is split as multiple flow data subtasks.
A9, the method according to A1, wherein, methods described also includes：
Monitor the processing progress of each treatment progress；
The processed corresponding flow data in flow data subtask is pushed into storage server, for the transmission of flow data
The client end of playing back of node or flow data is called.
A10, the method according to A1, wherein, methods described also includes：
The treatment that reception processing process reports to managing process notifies, and is treatment in current goal treatment node extremely
Redistribute treatment progress in abnormal flow data subtask.
A11, the method according to A10, wherein, methods described also includes：
Receive current goal treatment node and report to the treatment notice extremely of management node, and be in other treatment nodes
Process abnormal flow data subtask and redistribute a target treatment node；
A treatment progress for calling the target redistributed to process node is carried out to the abnormal flow data subtask for the treatment of
Treatment.
The invention also discloses B12, a kind of processing unit of flow data task, including：
Task splits module, for flow data task to be split into multiple flow data subtasks；
Subtask distribute module, node is processed for multiple flow data subtasks to be distributed at least one target；
Treatment progress calling module is described more for calling each target treatment node to run the treatment of at least one treatment progress
Individual flow data subtask, wherein, a treatment progress for being called processes one or more flow data subtasks；
Result receiver module, for receiving treatment knot of each target treatment node for flow data subtask feedback
Really, the result for obtaining the flow data task is collected.
B13, the device according to B12, wherein, the subtask distribute module includes：
Load condition acquisition submodule, the load condition for obtaining each treatment node from management node；
Target processes node selection submodule, and at least one target for imposing a condition treatment is met for choosing load condition
Node, and issue at least one flow data subtask to the target treatment node of each selection.
B14, the device according to B12, wherein, described device also includes：
Task queue acquisition module, for issuing at least one flow data to the target treatment node of each selection described
Before task, the task queue that each target processes node, each treatment node one task queue of correspondence are obtained；
Sequence sets submodule, for setting the sequence of flow data subtask to be issued in each task queue, makes each
The difference of the process time of individual flow data subtask is less than given threshold.
B15, the device according to B14, wherein, the sequence sets submodule to be included：
Process time predicting unit, appoints for existing flow data in the task queue that node is processed according to each target
Business, predicts the process time of each flow data subtask to be issued；
Target flow data subtask determining unit, for determining process time target flow data subtask the latest；
Process time arranging unit, arranges each to wait to issue for the process time according to the target flow data subtask
Flow data subtask process time.
B16, the device according to B14, wherein, the task queue acquisition module, specifically for being processed from each target
The managing process of node obtains task queue.
B17, the device according to B12, wherein, described device also includes：
Process instruction receiver module, for it is described flow data task is split as multiple flow data subtasks before, connect
Receive the flow data process instruction for carrying flow data address；
Flow data download module, for by accessing the flow data address, downloading pending flow data.
B18, the device according to B17, wherein, the task splits module, at according to the flow data
Various transcoded formats that reason instruction is carried, the flow data process task is split as multiple fluxions of the various transcoded formats of correspondence
According to subtask, the flow data task is transcoding task.
B19, the device according to B17, wherein, the task splits module to be included：
Service identification extracting sub-module, for extracting the service identification that the flow data process instruction is carried, the fluxion
It is transcoding task according to task；
Transcoded format searches submodule, and the default various transcoded formats of the service identification are directed to for searching；
Task divides submodule, for according to the various transcoded formats searched, the flow data process task being split
It is multiple flow data subtasks.
B20, the device according to B12, wherein, described device also includes：
Processing progress monitoring module, the processing progress for monitoring each treatment progress；
Flow data pushing module, for the processed corresponding flow data in flow data subtask to be pushed into storage service
Device, so that the transmission node of flow data or the client end of playing back of flow data are called.
B21, the device according to B12, wherein, described device also includes：
First notifies receiver module, and the treatment for reporting to managing process for reception processing process is notified extremely；
Treatment progress redistributes module, for being the abnormal flow data subtask for the treatment of in current goal treatment node
Redistribute treatment progress.
B22, the device according to B21, wherein, described device also includes：
Second notify receiver module, for receive current goal treatment node report to management node treatment it is extremely logical
Know；
Target treatment node redistributes module, for being the abnormal flow data subtask for the treatment of in other treatment nodes
Redistribute a target treatment node；
Target processes node calling module, for calling the target redistributed to process a treatment progress of node to place
Abnormal flow data subtask is managed to be processed.
Claims (10)
1. a kind of processing method of flow data task, including：
Flow data task is split into multiple flow data subtasks；
Multiple flow data subtasks are distributed at least one target and processes node；
Call each target treatment node to run at least one treatment progress and process the multiple flow data subtask, wherein, one
The treatment progress for being called processes one or more flow data subtasks；
Each target treatment node is received for the result of flow data subtask feedback, is collected and is obtained the flow data task
Result.
2. method according to claim 1, wherein, it is described to distribute at least one target multiple flow data subtasks
Reason node includes：
The load condition of each treatment node is obtained from management node；
Choose load condition and meet at least one target treatment node for imposing a condition, and under the target treatment node of each selection
Send out at least one flow data subtask.
3. method according to claim 1, wherein, each treatment node one task queue of correspondence, described to each selection
Target treatment node issue at least one flow data subtask before, methods described also includes：
Obtain the task queue that each target processes node；
The sequence of flow data subtask to be issued in each task queue is set, makes the process time of each flow data subtask
Difference be less than given threshold.
4. method according to claim 3, wherein, it is described that flow data subtask to be issued is set in each task queue
Sequence, make the difference of the process time of each flow data subtask includes less than given threshold：
Existing flow data subtask in the task queue of node is processed according to each target, each flow data to be issued is predicted
The process time of task；
Determine process time target flow data subtask the latest；
Process time according to the target flow data subtask arranges the process time of each flow data subtask to be issued.
5. method according to claim 3, wherein, the task queue for obtaining each target treatment node includes：
Task queue is obtained from the managing process of each target treatment node.
6. method according to claim 1, wherein, it is described by flow data task be split as multiple flow data subtasks it
Before, methods described also includes：
Receive the flow data process instruction for carrying flow data address；
By accessing the flow data address, pending flow data is downloaded.
7. method according to claim 6, wherein, the flow data task is transcoding task, described by flow data task
Being split as multiple flow data subtasks includes：
According to various transcoded formats that the flow data process instruction is carried, the flow data process task is split as correspondence each
Plant multiple flow data subtasks of transcoded format.
8. method according to claim 6, wherein, the flow data task is transcoding task, described by flow data task
Being split as multiple flow data subtasks includes：
Extract the service identification that the flow data process instruction is carried；
Search and be directed to the default various transcoded formats of the service identification；
According to the various transcoded formats searched, the flow data process task is split as multiple flow data subtasks.
9. method according to claim 1, wherein, methods described also includes：
Monitor the processing progress of each treatment progress；
The processed corresponding flow data in flow data subtask is pushed into storage server, for the transmission node of flow data
Or the client end of playing back of flow data is called.
10. a kind of processing unit of flow data task, including：
Task splits module, for flow data task to be split into multiple flow data subtasks；
Subtask distribute module, node is processed for multiple flow data subtasks to be distributed at least one target；
Treatment progress calling module, the multiple stream is processed for calling each target treatment node to run at least one treatment progress
Data subtask, wherein, a treatment progress for being called processes one or more flow data subtasks；
Result receiver module, for receiving each target treatment node for the result of flow data subtask feedback, converges
Collection obtains the result of the flow data task.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611263882.6A CN106713944A (en) | 2016-12-30 | 2016-12-30 | Method and apparatus for processing streaming data task |
| PCT/CN2017/119880 WO2018121738A1 (en) | 2016-12-30 | 2017-12-29 | Method and apparatus for processing streaming data task |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611263882.6A CN106713944A (en) | 2016-12-30 | 2016-12-30 | Method and apparatus for processing streaming data task |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106713944A true CN106713944A (en) | 2017-05-24 |
Family
ID=58905629
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611263882.6A Pending CN106713944A (en) | 2016-12-30 | 2016-12-30 | Method and apparatus for processing streaming data task |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106713944A (en) |
| WO (1) | WO2018121738A1 (en) |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107231437A (en) * | 2017-07-18 | 2017-10-03 | 郑州云海信息技术有限公司 | A kind of task backup management method and device |
| WO2018121738A1 (en) * | 2016-12-30 | 2018-07-05 | 北京奇虎科技有限公司 | Method and apparatus for processing streaming data task |
| CN109086152A (en) * | 2017-06-14 | 2018-12-25 | 杭州海康威视数字技术股份有限公司 | Picture analyzing method, apparatus and system, computer equipment and storage medium |
| CN109086125A (en) * | 2017-06-14 | 2018-12-25 | 杭州海康威视数字技术股份有限公司 | Picture analyzing method, apparatus and system, computer equipment and storage medium |
| CN109522138A (en) * | 2018-11-14 | 2019-03-26 | 北京中电普华信息技术有限公司 | A kind of processing method and system of distributed stream data |
| CN109783210A (en) * | 2018-12-13 | 2019-05-21 | 平安科技（深圳）有限公司 | Multi-task processing method, device, computer equipment and storage medium |
| CN109861966A (en) * | 2018-12-06 | 2019-06-07 | 北京达佳互联信息技术有限公司 | Method, apparatus, terminal and the storage medium of processing status event |
| CN110704551A (en) * | 2018-06-21 | 2020-01-17 | 中兴通讯股份有限公司 | Data processing method, device, equipment and computer readable storage medium |
| CN111190707A (en) * | 2019-08-02 | 2020-05-22 | 腾讯科技（深圳）有限公司 | Data processing method and device |
| CN111212015A (en) * | 2018-11-21 | 2020-05-29 | 华为技术有限公司 | Service cooperative processing method and device |
| CN111355689A (en) * | 2018-12-21 | 2020-06-30 | 中兴通讯股份有限公司 | Stream data processing method and device |
| CN111629013A (en) * | 2019-02-27 | 2020-09-04 | 北京奇虎科技有限公司 | Method and device for managing service node and node network |
| CN109254833B (en) * | 2017-07-12 | 2020-10-09 | 杭州海康威视数字技术股份有限公司 | Picture analysis method, device and system and computer equipment |
| CN111880918A (en) * | 2020-07-28 | 2020-11-03 | 南京市城市与交通规划设计研究院股份有限公司 | Road network front end rendering method and device and electronic equipment |
| CN111917812A (en) * | 2019-05-10 | 2020-11-10 | 北京百度网讯科技有限公司 | Data transmission control method, device, equipment and storage medium |
| CN112130972A (en) * | 2020-08-24 | 2020-12-25 | 北京思特奇信息技术股份有限公司 | Task processing method and system |
| CN112395085A (en) * | 2020-11-05 | 2021-02-23 | 深圳市中博科创信息技术有限公司 | HDFS-based distributed relational database scheduling method |
| CN113535320A (en) * | 2020-04-14 | 2021-10-22 | 深信服科技股份有限公司 | Data access method, device, equipment and storage medium |
| CN114697705A (en) * | 2020-12-29 | 2022-07-01 | 深圳云天励飞技术股份有限公司 | Video stream object processing method and device, video stream processing system and electronic equipment |
Families Citing this family (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109324887B (en) * | 2018-09-28 | 2024-05-10 | 中国平安财产保险股份有限公司 | Auditing data parallel processing method and system |
| CN111090502B (en) * | 2018-10-24 | 2024-05-17 | 阿里巴巴集团控股有限公司 | Stream data task scheduling method and device |
| CN109214132B (en) * | 2018-10-30 | 2023-06-30 | 中国运载火箭技术研究院 | LVC simulation-oriented uncoupled streaming type large-flux asynchronous task processing system |
| CN112650578A (en) * | 2019-10-12 | 2021-04-13 | 阿里巴巴集团控股有限公司 | Data processing method, scheduling platform, video storage platform and video processing engine |
| CN110795244B (en) * | 2019-10-24 | 2022-07-08 | 浙江大华技术股份有限公司 | Task allocation method, device, equipment and medium |
| CN111049900B (en) * | 2019-12-11 | 2022-07-01 | 中移物联网有限公司 | Internet of things flow calculation scheduling method and device and electronic equipment |
| CN111258724B (en) * | 2020-01-14 | 2024-02-06 | 平安银行股份有限公司 | Data processing method, device, equipment and storage medium of distributed system |
| US11706097B2 (en) * | 2020-01-23 | 2023-07-18 | Novnet Computing System Tech Co., Ltd. | Task processing method applied to network topology, electronic device and storage medium |
| CN113840161B (en) * | 2020-06-23 | 2023-07-25 | 龙芯中科技术股份有限公司 | Streaming media transmission method, receiving method, device, electronic equipment and storage medium |
| CN111818159B (en) * | 2020-07-08 | 2024-04-05 | 腾讯科技（深圳）有限公司 | Management method, device, equipment and storage medium of data processing node |
| CN113051049A (en) * | 2020-07-10 | 2021-06-29 | 北京迈格威科技有限公司 | Task scheduling system, method, electronic device and readable storage medium |
| CN113238837B (en) * | 2020-07-10 | 2022-12-27 | 北京旷视科技有限公司 | Computing flow chart construction method, computing efficiency optimization method, computing efficiency construction device and electronic equipment |
| CN112351299B (en) * | 2020-11-05 | 2022-11-04 | 山东亚华电子股份有限公司 | Propaganda and education broadcasting method and device |
| CN112817724A (en) * | 2021-02-05 | 2021-05-18 | 苏州互方得信息科技有限公司 | Task allocation method capable of dynamically arranging sequence |
| CN113254194B (en) * | 2021-03-29 | 2023-03-28 | 河海大学 | Real-time GNSS data processing method and system |
| CN113179304B (en) * | 2021-04-22 | 2022-10-28 | 平安消费金融有限公司 | Message issuing method, system, device and storage medium |
| CN113434310B (en) * | 2021-06-23 | 2024-03-22 | 深圳博瑞晶芯科技有限公司 | Multithreading task allocation method, device, equipment and storage medium |
| CN114780214B (en) * | 2022-04-01 | 2024-01-09 | 中国电信股份有限公司 | Task processing method, device, system and equipment |
| CN115080241A (en) * | 2022-06-30 | 2022-09-20 | 支付宝(杭州)信息技术有限公司 | Data processing method and device |
| CN115988092B (en) * | 2023-02-08 | 2023-06-23 | 天翼云科技有限公司 | Image processing method, device, equipment, medium and product |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102123279A (en) * | 2010-12-28 | 2011-07-13 | 乐视网信息技术（北京）股份有限公司 | Distributed real-time transcoding method and system |
| CN103309946A (en) * | 2013-05-16 | 2013-09-18 | 华为技术有限公司 | Method, device and system for processing multimedia file |
| CN103458270A (en) * | 2013-09-09 | 2013-12-18 | 深圳市融创天下科技股份有限公司 | Video transcoding system and method |
| CN104469370A (en) * | 2013-09-17 | 2015-03-25 | 中国普天信息产业股份有限公司 | Video transcode method and device |
| CN105451031A (en) * | 2015-11-18 | 2016-03-30 | 腾讯科技（深圳）有限公司 | Video transcoding method and system thereof |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004289295A (en) * | 2003-03-19 | 2004-10-14 | Fujitsu Ltd | Data processing system, data processor, and data processing method |
| CN104244025A (en) * | 2013-06-09 | 2014-12-24 | 北京同步科技有限公司 | Cluster transcoding system and method thereof |
| CN104935952B (en) * | 2014-03-17 | 2018-06-19 | 无锡天脉聚源传媒科技有限公司 | A kind of video transcoding method and system |
| CN104158909B (en) * | 2014-08-29 | 2015-11-25 | 北京阅联信息技术有限公司 | A kind of disributed media processing method and system thereof |
| CN106713944A (en) * | 2016-12-30 | 2017-05-24 | 北京奇虎科技有限公司 | Method and apparatus for processing streaming data task |
-
2016
- 2016-12-30 CN CN201611263882.6A patent/CN106713944A/en active Pending
-
2017
- 2017-12-29 WO PCT/CN2017/119880 patent/WO2018121738A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102123279A (en) * | 2010-12-28 | 2011-07-13 | 乐视网信息技术（北京）股份有限公司 | Distributed real-time transcoding method and system |
| CN103309946A (en) * | 2013-05-16 | 2013-09-18 | 华为技术有限公司 | Method, device and system for processing multimedia file |
| CN103458270A (en) * | 2013-09-09 | 2013-12-18 | 深圳市融创天下科技股份有限公司 | Video transcoding system and method |
| CN104469370A (en) * | 2013-09-17 | 2015-03-25 | 中国普天信息产业股份有限公司 | Video transcode method and device |
| CN105451031A (en) * | 2015-11-18 | 2016-03-30 | 腾讯科技（深圳）有限公司 | Video transcoding method and system thereof |
Cited By (28)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018121738A1 (en) * | 2016-12-30 | 2018-07-05 | 北京奇虎科技有限公司 | Method and apparatus for processing streaming data task |
| CN109086152A (en) * | 2017-06-14 | 2018-12-25 | 杭州海康威视数字技术股份有限公司 | Picture analyzing method, apparatus and system, computer equipment and storage medium |
| CN109086125A (en) * | 2017-06-14 | 2018-12-25 | 杭州海康威视数字技术股份有限公司 | Picture analyzing method, apparatus and system, computer equipment and storage medium |
| CN109086152B (en) * | 2017-06-14 | 2021-12-14 | 杭州海康威视数字技术股份有限公司 | Picture analysis method, device and system, computer equipment and storage medium |
| CN109086125B (en) * | 2017-06-14 | 2021-01-22 | 杭州海康威视数字技术股份有限公司 | Picture analysis method, device and system, computer equipment and storage medium |
| CN109254833B (en) * | 2017-07-12 | 2020-10-09 | 杭州海康威视数字技术股份有限公司 | Picture analysis method, device and system and computer equipment |
| CN107231437A (en) * | 2017-07-18 | 2017-10-03 | 郑州云海信息技术有限公司 | A kind of task backup management method and device |
| CN110704551A (en) * | 2018-06-21 | 2020-01-17 | 中兴通讯股份有限公司 | Data processing method, device, equipment and computer readable storage medium |
| CN110704551B (en) * | 2018-06-21 | 2023-02-17 | 中兴通讯股份有限公司 | Data processing method, device, equipment and computer readable storage medium |
| CN109522138A (en) * | 2018-11-14 | 2019-03-26 | 北京中电普华信息技术有限公司 | A kind of processing method and system of distributed stream data |
| CN111212015A (en) * | 2018-11-21 | 2020-05-29 | 华为技术有限公司 | Service cooperative processing method and device |
| CN109861966A (en) * | 2018-12-06 | 2019-06-07 | 北京达佳互联信息技术有限公司 | Method, apparatus, terminal and the storage medium of processing status event |
| CN109783210B (en) * | 2018-12-13 | 2023-10-20 | 平安科技（深圳）有限公司 | Multitasking method, apparatus, computer device and storage medium |
| CN109783210A (en) * | 2018-12-13 | 2019-05-21 | 平安科技（深圳）有限公司 | Multi-task processing method, device, computer equipment and storage medium |
| CN111355689A (en) * | 2018-12-21 | 2020-06-30 | 中兴通讯股份有限公司 | Stream data processing method and device |
| CN111355689B (en) * | 2018-12-21 | 2022-04-22 | 金篆信科有限责任公司 | Stream data processing method and device |
| CN111629013A (en) * | 2019-02-27 | 2020-09-04 | 北京奇虎科技有限公司 | Method and device for managing service node and node network |
| CN111917812A (en) * | 2019-05-10 | 2020-11-10 | 北京百度网讯科技有限公司 | Data transmission control method, device, equipment and storage medium |
| CN111917812B (en) * | 2019-05-10 | 2022-09-30 | 北京百度网讯科技有限公司 | Data transmission control method, device, equipment and storage medium |
| CN111190707A (en) * | 2019-08-02 | 2020-05-22 | 腾讯科技（深圳）有限公司 | Data processing method and device |
| CN113535320A (en) * | 2020-04-14 | 2021-10-22 | 深信服科技股份有限公司 | Data access method, device, equipment and storage medium |
| CN113535320B (en) * | 2020-04-14 | 2024-02-23 | 深信服科技股份有限公司 | Data access method, device, equipment and storage medium |
| CN111880918A (en) * | 2020-07-28 | 2020-11-03 | 南京市城市与交通规划设计研究院股份有限公司 | Road network front end rendering method and device and electronic equipment |
| CN112130972A (en) * | 2020-08-24 | 2020-12-25 | 北京思特奇信息技术股份有限公司 | Task processing method and system |
| CN112395085A (en) * | 2020-11-05 | 2021-02-23 | 深圳市中博科创信息技术有限公司 | HDFS-based distributed relational database scheduling method |
| CN112395085B (en) * | 2020-11-05 | 2022-10-25 | 深圳市中博科创信息技术有限公司 | HDFS-based distributed relational database scheduling method |
| CN114697705A (en) * | 2020-12-29 | 2022-07-01 | 深圳云天励飞技术股份有限公司 | Video stream object processing method and device, video stream processing system and electronic equipment |
| CN114697705B (en) * | 2020-12-29 | 2024-03-22 | 深圳云天励飞技术股份有限公司 | Video stream object processing method and device, video stream processing system and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2018121738A1 (en) | 2018-07-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106713944A (en) | Method and apparatus for processing streaming data task | |
| CN106227596A (en) | Mission Monitor method and apparatus for task scheduling server | |
| CN111275369B (en) | Method and device for realizing planned warehouse crossing based on sorting by sub-sowing | |
| CN107026900B (en) | Shooting task allocation method and device | |
| US20070025381A1 (en) | Method and apparatus for allocating processing in a network | |
| US11656906B2 (en) | Method and system for coordinated service placement in multiple clouds | |
| US20180176657A1 (en) | Social delivery of iptv streams | |
| US9020225B2 (en) | Medical image processing system | |
| CN106357738A (en) | Automatic arrangement method for server cluster and device as well as system thereof | |
| US20130144931A1 (en) | Candidate set solver with user advice | |
| US20150347509A1 (en) | Optimizing performance in cep systems via cpu affinity | |
| CN108234242A (en) | A kind of method for testing pressure and device based on stream | |
| EP2570922A1 (en) | Method and system for managing an elastic server farm | |
| CN110287022A (en) | A kind of scheduling node selection method, device, storage medium and server | |
| CN106790549A (en) | A kind of data-updating method and device | |
| EP3021521A1 (en) | A method and system for scaling, telecommunications network and computer program product | |
| CN110413527A (en) | Test macro, test method, electronic equipment and computer readable storage medium | |
| CN109951320A (en) | A kind of expansible multi layer monitoing frame and its monitoring method of facing cloud platform | |
| CN108228327A (en) | A kind for the treatment of method and apparatus of task | |
| US20160119626A1 (en) | System to dispatch video decoding to dedicated hardware resources | |
| US20210374144A1 (en) | System for embedding stream processing execution in a database | |
| CN114153609A (en) | Resource control method and device, electronic equipment and computer readable storage medium | |
| CN112416472B (en) | Customer service distribution method and device, electronic equipment and medium | |
| CN111014048B (en) | Method and device for controlling distribution equipment to distribute goods | |
| US20140047454A1 (en) | Load balancing in an sap system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170524 |