CN105808585A - Stream data processing method and apparatus - Google Patents
Stream data processing method and apparatus Download PDFInfo
- Publication number
- CN105808585A CN105808585A CN201410849709.9A CN201410849709A CN105808585A CN 105808585 A CN105808585 A CN 105808585A CN 201410849709 A CN201410849709 A CN 201410849709A CN 105808585 A CN105808585 A CN 105808585A
- Authority
- CN
- China
- Prior art keywords
- stream data
- thread
- copy
- data
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Embodiments of the invention provide a stream data processing method and apparatus. The method specifically comprises the steps of receiving stream data for a current process copy, wherein the process copy is a calculation entity with independent calculation threads; processing the stream data by utilizing the threads of the current process copy, wherein the threads are in one-to-one correspondence with sources of the stream data; and transmitting the processed stream data to a next process copy by utilizing the threads of the current process copy, and processing the processed stream data by the next process copy until the stream data is processed by a final process copy. According to the embodiments of the invention, the stream data can be concurrently processed, so that the processing efficiency is improved.
Description
Technical field
The present invention relates to networking technology area, particularly relate to processing method and the device of a kind of stream data.
Background technology
In recent years, computer technology and Internet technology obtain swift and violent development, the business of network and application and enrich gradually, and network has become as a part indispensable in people's daily life.Meanwhile, along with the continuous expansion of network size so that the equipment that NMS manages gets more and more, it is necessary to the data volume of process is also increasing, and therefore, this just it is also proposed higher requirement to designing and developing of NMS.
Traditional NMS, according to 2～3 processes of function classifying and dividing, by CORBA (CommonObjectRequestBrokerArchitecture, Common Object Request Broker Architecture) middleware communication between process, work in coordination with the process of stream data.Further, it is also possible to disperse the process of stream data physically by disposing multiple servers, reach to increase the purpose of entire throughput.
But, above-mentioned traditional NMS uses the mode of serial arithmetic inside process, and this computing mode can cause that treatment effeciency is low；Such as, the consuming time of execution process A and process B is all 1 hour, when there being a data to arrive, first this data is sent into process A, and after process A process completes this data, then this data is sent into process B, then the processing procedure of next data is still so, process A or process B is performing computing in synchronization only one of which, if there being 100 data etc. pending, then and total needs consuming time 200 hours.
Although additionally, also there is indivedual processing procedure to use thread pool to carry out computing, but owing to alerting the forward-backward correlation of stream data so that thread pool can not improve treatment effeciency；Such as, there is alarm data D, E, F sequentially, wherein D and F has operational association, then must analyze its associating with D when processing F.Therefore, alarm stream data must strictly in accordance with time sequencing when processing, and the random concurrent processing of thread pool can cause out of order problem, and on thread pool, increase order is then as good as with serial.So, thread pool is only applicable to uncorrelated data.
Summary of the invention
Embodiment of the present invention technical problem to be solved is to provide processing method and the device of a kind of stream data, it is possible to streaming data concurrent processing, the efficiency that raising processes.
In order to solve the problems referred to above, the invention discloses the processing method of a kind of stream data, including:
Stream data is received for active procedure copy；Wherein, described process copy is the computing entity possessing operation independent thread；
Utilize the thread that described active procedure copies that described stream data is processed；Wherein, the source one_to_one corresponding of described thread and described stream data；
Utilize the thread that described active procedure copies that the stream data after process passes to next process copy, next process copy the stream data after described process is processed, until described stream data is complete by last process copy process.
Preferably, described stream data is processed by the described thread utilizing described active procedure to copy, including:
Utilize described active procedure copy that the source of described stream data is identified；
Whether the thread that the source of the described stream data that judgement identifies is corresponding creates, if creating, then described stream data adds the buffer queue end of described thread；If not creating, then create corresponding thread for the source of described stream data and start this thread, and being the buffer queue that described thread creation is corresponding, described stream data is added the buffer queue end that described thread is corresponding；
Described thread is utilized to take out described stream data from the buffer queue of its correspondence and process.
Preferably, the stream data after processing is passed to one or several that transfer mode corresponding to next process copy include in the following manner by the described thread utilizing described active procedure to copy: the transfer mode of transfer mode, the transfer mode of text and JMS message that local interface calls.
Preferably, described for active procedure copy reception stream data, including:
Copy for active procedure and receive stream data according to pre-set sequence successively.
According to another aspect of the present invention, it is provided that the process device of a kind of stream data, including:
Receiver module, for receiving stream data for active procedure copy；Wherein, described process copy is the computing entity possessing operation independent thread；
Processing module, for utilizing the thread that described active procedure copies that described stream data is processed；Wherein, the source one_to_one corresponding of described thread and described stream data；And
Transmission module, copy for utilizing the thread that described active procedure copies that the stream data after process is passed to next process, by next process copy, the stream data after described process is processed, until described stream data is complete by last process copy process.
Preferably, described processing module, including:
Recognition unit, for utilizing described active procedure copy that the source of described stream data is identified；
Judging unit, for judging whether thread corresponding to the source of the described stream data identified creates；
Creating unit, for when the thread that the source not creating described stream data is corresponding, the source for described stream data creates corresponding thread and starts this thread；
Join the team unit, for described stream data being added the buffer queue end of described thread；
Processing unit, for utilizing described thread take out described stream data from the buffer queue of its correspondence and process.
Preferably, the stream data after processing is passed to one or several that transfer mode corresponding to next process copy include in the following manner by the described thread utilizing described active procedure to copy: the transfer mode of transfer mode, the transfer mode of text and JMS message that local interface calls.
Preferably, described receiver module, including:
Receive unit, receive stream data successively according to pre-set sequence for copying for active procedure.
Compared with prior art, the embodiment of the present invention includes advantages below:
The embodiment of the present invention is in the process processing stream data, for process, several independent process copies are set, decoupling between being copied by process realizes the concurrent processing of interprocedual, therefore can improve the treatment effeciency of stream data, solve the problem adopting the mode of serial arithmetic to cause inefficiency in prior art；And, the mode that the embodiment of the present invention is distinguished according further to data source header processes stream data, it is capable of the stream data concurrent processing to multiple sources of multiple thread, and the source one_to_one corresponding of thread and the stream data in active procedure copy, each thread only processes the stream data in corresponding source, and then the stream data achieving same source can according to pre-set sequence (such as d1, d2, ..., the order of dn) enter same thread concurrent processing, the random concurrent processing solving thread pool can cause out of order problem, further increase the treatment effeciency of stream data.
Accompanying drawing explanation
Fig. 1 illustrates the method step flow chart that a kind of stream data of the present invention processes；
Fig. 2 illustrates the flow chart of steps that described stream data is processed by a kind of thread utilizing described active procedure to copy of the present invention；
Fig. 3 illustrates the structural representation of the process system of a kind of stream data of the present invention；
Fig. 4 illustrates the flow chart of the processing method of a kind of stream data of the present invention with data views；
Fig. 5 illustrates the flow chart of the processing method of a kind of stream data of the present invention with process visual angle；
Fig. 6 illustrates the structured flowchart processing device of a kind of stream data of the present invention.
Detailed description of the invention
Understandable for enabling the above-mentioned purpose of the present invention, feature and advantage to become apparent from, below in conjunction with the drawings and specific embodiments, the present invention is further detailed explanation.
Embodiment one
With reference to Fig. 1, it is shown that the method step flow chart that a kind of stream data of the present invention processes, specifically may include steps of:
Step 101, for active procedure copy receive stream data；Wherein, described process copy is the computing entity possessing operation independent thread；
The processing procedure of existing stream data adopts the method for serial arithmetic, namely after data have been processed by process A, give process B process again, cause that treatment effeciency is low, the present invention proposes and arranges several independent process copies for process, and is realized the executed in parallel of interprocedual by the decoupling between process copies.
For example, it is possible to for process A by data source header 1,2,3 ... n arranges independent process copy (namely possessing the computing entity of operation independent thread) A1, A2 ..., An, A1 only process the data of data source 1, and A2 only processes the data of data source 2.nullWherein，A1+A2+ ...+An realizes the work of process A，Process B is also decomposed into independent process copy B1、B2、…、Bn，Processing 100 stream data (d1 from data source 1、d2、…、D100) time，In process A, the process copy A1 in corresponding data source 1 is first to d1 process，D1 is given next process after completing and copies B1 process by process，Simultaneously，Process copy A1 may also begin to d2 process，Process B is passed to when current data has been processed by process A，While the reason current data of process B beginning, process A has begun to process next data，If performing the consuming time of process A and process B is all 1 hour，Then 100 total needs consuming time of stream data 101 hours，Compared to existing technology，Improve the efficiency that stream data processes.
Step 102, utilize the thread that described active procedure copies that described stream data is processed；Wherein, the source one_to_one corresponding of described thread and described stream data；
In a particular application, owing to alerting the forward-backward correlation of stream data, make thread pool can not improve treatment effeciency, in order to solve this problem, the mode that the present invention proposes according to data source header is distinguished carrys out concurrent processing stream data, specifically, utilize the thread in process copy that the stream data from same source is processed, to realize the concurrent processing of the stream data in multiple source.
Such as, in the processing procedure of active procedure copy, corresponding thread independent mutually can be created for the source of each stream data, such as thread T1, thread T2, thread T3 etc., wherein, thread T1 only processes the data from source S1, and thread T2 only processes the data from source S2, and thread T3 only processes the data from source S3.Specific to NMS scenario, source is specifically as follows an equipment, a collection point etc., for instance when monitoring the warning information of 100 equipment, an equipment is just considered as a source.
So, the source one_to_one corresponding of thread and stream data in active procedure copy, the stream data in each thread process corresponding source with it, such that it is able to realize the stream data concurrent processing to multiple sources of multiple thread.
In a particular application, operationally can there is multiple process copy according to the source difference of stream data in process.Process copy and corresponding same data source header, receive the data arrived in same source and perform the work entity of the service logic identical with affiliated process, possesses separate threads, therefore be independent in computing, multiple copy identical points with process are that service logic is identical, and difference is that the data processed are from different pieces of information source.Such as: " data loading process ", the service logic of this process is to preserve data to data base, for there is process copy first from the data of A point, the data coming from B point are existed process copy second, the fixing service code being performed by the thread in copy first to preserve of the data that A point produces, B point is in like manner, first and second is all preserve the logic to data base but the data processed originate from different pieces of information source), multiple processes must operate in multiple copies of single process in the middle of a process that (many processes multicopy shares a process.Common form is: totally 13 processes, and front 7 process synthesises are A class process, and rear 5 processes consist of B class process, and A class process operates on 10 servers, totally 10 processes；B class process operates on 3 servers, and totally 3 processes, namely the relation between process and process is very flexible.But the single working copy of process operationally must comprise a worker thread.
Step 103, utilize thread that described active procedure copies that the stream data after processing is passed to next process copy, next process copy the stream data after described process is processed, until described stream data is complete by last process copy process.
Such as aforementioned exemplary, first, process copy A1 is to data d1 process, and namely the thread in process copy A1 is to data d1 process, after process completes, data d1 is passed to process copy B1 by this thread, same, after data d1 process is completed by process copy B1, again data d1 is continued going down, until data d1 is disposed by all processes, now, namely complete the process to data d1 of all processes.
In one preferred embodiment of the invention, receive the step of stream data for active procedure copy, specifically may include that
Copy for active procedure and receive stream data according to pre-set sequence successively.
Must strictly in accordance with time sequencing during due to alarm stream data process, the random concurrent processing of thread pool can cause out of order problem, therefore, the present invention proposes to distinguish according to source streaming data, and further, can copy according to pre-set sequence (such as d1 for active procedure, d2, ..., the order of dn) receive stream data successively, thread corresponding with source in being copied by active procedure carries out the process of stream data, such that it is able to realize related stream data is concurrently processed, the random concurrent processing solving thread pool can cause out of order problem, make the related stream data can also concurrent processing, thus improving the treatment effeciency of stream data.
To sum up, the embodiment of the present invention is in the process processing stream data, decoupling between being copied by process realizes the concurrent processing of interprocedual, such that it is able to improve the treatment effeciency of stream data, solves the problem adopting the mode of serial arithmetic to cause inefficiency in prior art；The mode according to data source header differentiation that also proposes is to process stream data, such that it is able to realize the stream data concurrent processing to multiple sources of multiple thread, and make the source one_to_one corresponding of the thread in active procedure copy and stream data, each thread only processes the stream data in corresponding source, and then the stream data achieving same source can according to pre-set sequence (such as d1, d2, ..., the order of dn) enter same thread concurrent processing, the random concurrent processing solving thread pool can cause out of order problem, further increase the treatment effeciency of stream data.
In another preferred embodiment of the invention, the stream data after processing is passed to one or several that transfer mode corresponding to next process copy specifically can include in the following manner by the described thread utilizing described active procedure to copy: the transfer mode of transfer mode, the transfer mode of text and JMS message that local interface calls.
Wherein, the transfer mode that local interface calls has fast response time, and handling capacity is unrestricted, saves the feature of I/O；
The transfer mode response speed of text is slow, although handling capacity is limited by disk but will not become bottleneck, furthermore, it is possible to striding course, and file possesses natural archive, observability；
JMS is the application programming interfaces of one group of standard, it is possible to be used for accessing multiple message server.JMS message response is the slowest also at Millisecond, and handling capacity is by restriction in many ways such as the performance of middleware server, the network bandwidth etc., furthermore, it is possible to cross-server, based on IP agreement cross-region, and free queue middleware provides multiple congestion policies and management instrument.
In the specific implementation, it is possible to take any of which mode process copy between transmission stream data, even take these three hybrid mode can also, this present invention is not particularly limited.
Between process of the present invention copy, the mode of data transmission is diversified, except the transfer mode (cross-thread) that most basic local interface calls, also has the transfer mode (striding course) of text, more there is the transfer mode (cross-server) of JMS message, the flexible distributed of each process computation is ensure that by three kinds of transfer modes, there is provided basis of software for system architecture flexibly, be adapted to various application scenarios.
The situation above process including multiple thread is introduced, what deserves to be explained is, process is also configured to single example thread mode (being limited in one process unique), special requirement can be tackled, such as, consider during relevant database that efficiency needs to adopt batch warehouse-in to be accomplished by collecting all data to processing together.
Wherein, " unique in one process " refers to: no matter the source of stream data exists how many, certain process in-process only exists a process copy, certainly only exists a worker thread accordingly, and when multiple stream data source, process was create multiple processes copy originally.
Such as, because speed is promoted very big by the batch processing of SQL, substantially reduce the number the compiling and interpreting consumption of SQL with batch processing, decrease network transmission, so batch processing should be used when the SQL of big data quantity performs as far as possible.So, the single within the acceptable range batch of SQL quantity comprised is the bigger the better, therefore, if the SQL execution efficiency together that the data in multiple stream data sources produce is the highest, this " persistence process " preferably in-process be unique (not uniquely being because being not desired to produce extra I O cost in system-wide to lower efficiency on the contrary), so the singleton pattern of process creates.
Embodiment two
The method that the stream data of the present embodiment processes, on the basis of above-described embodiment one, can also include following optional technical scheme further.
Referring to Fig. 2, it is shown that the flow chart of steps that described stream data is processed by a kind of thread utilizing described active procedure to copy of the present invention, specifically may include that
Step 201, utilize described active procedure copy that the source of described stream data is identified；
In a kind of application example of the present invention, it is possible to use the source of described stream data is identified by adapter, further, it is also possible to utilize adapter to complete input and the output services of thread streaming data.Specifically, it is possible to use input adapter completes the reception of streaming data, and the source of this stream data is identified, and the source information that will identify that be marked in data as adeditive attribute.Further, after certain data has been processed by thread, it is also possible to utilize o adapter that this data exports next process copy.
Being appreciated that those skilled in the art according to practical situation, can adopt various recognition methods to carry out the identification in source of described stream data, concrete recognition methods is not any limitation as by the embodiment of the present invention.
Whether the thread that the source of the described stream data that step 202, judgement identify is corresponding creates, if creating, then described stream data adds the buffer queue end of described thread；If not creating, then create corresponding thread for the source of described stream data and start this thread, and being the buffer queue that described thread creation is corresponding, described stream data is added the buffer queue end that described thread is corresponding；
Specifically, it is assumed that the source of the stream data d1 identified is S1, and the thread T1 corresponding with S1 creates, then d1 can add the buffer queue end of thread T1；If active procedure copy does not also set up the thread T1 corresponding with S1, then can create thread T1 and start T1, and creating corresponding buffer queue for T1, d1 is added this buffer queue end.
Step 203, described thread is utilized to take out described stream data from the buffer queue of its correspondence and process.
The stream data in the corresponding source respectively of each thread in active procedure copy, each thread takes out stream data from respective buffer queue and processes so that the stream data in multiple sources can concurrent processing.Then changeable as number of concurrent, maximum number of concurrent can be source total quantity during the subdividable minimum particle size of data source.
For making those skilled in the art be more fully understood that the present invention, with reference to Fig. 3, illustrating the structural representation of the process system of a kind of stream data of the present invention, this process system specifically can include consisting of key element: process copy P, the source S of stream data, thread T, adapter, data D and transfer mode；
Wherein, process copy P: processing procedure entirety is decomposed and obtains, it is possible to several independent process copies, the P1 in Fig. 3, P2 ..., Pn are set for process.
The source S of stream data: when there is multiple data source in system, is divided into multiple data source header according to different sources, S1, S2, the S3 in Fig. 3.
In an application example of the present invention, acquisition server is responsible for gathering stream data, if collection point is a lot, it is necessary to 3 station servers go correspondence, is responsible for 10～20 collection points for every, and the thing that these acquisition servers do is all the process gathering data.So, referring to Fig. 3, acquisition server can be Server_1-1, Server_1-2, Server_1-3, and collection point is the source of stream data, and the process of collection point corresponding data can be copied P1 by process and complete.
Thread T: i.e. computing thread, concrete operation method depends on its affiliated process, the fixing source from specific stream data of the input data of thread, has different threads according to different sources corresponding with it.
Adapter: the ingredient of process, is responsible for the input of streaming data and output services, the identification work in the source of streaming data, and between responsible processing procedure, different between process copy transfer mode.
Specifically, in adapter, the identification maneuver in the source of stream data can be reserved groove, developer can develop different recognition methodss, this is not particularly limited by the present invention, adapter can be marked in data as adeditive attribute after being identified in the source of stream data, and adapter belongs to process.
Specifically, adapter is divided into input adapter and o adapter two kinds as the ingredient of process, create when procedure initialization and enter active state, if the words of input adapter, the recognition methods being defined according to adapter when data arrive, identifies data source header and data is distributed in the middle of the buffer queue of corresponding process copy；If o adapter, after the output of the process of reception copy, after data being packed, it is transferred to the input adapter of next process copy.
Data D: from the real data in certain source, D1-x represent the x-th data from source 1.
Transfer mode: aforementioned three kinds of transfer modes can optional one, and mix and match can give play to the feature of different transfer mode.
As can be seen from Figure 3 the syntagmatic between process, source and server three, framework can accomplish that process operationally physically can be separated.Specifically, Server_1-1, Server_1-2 and Server_1-3 have run begin (i.e. data source) and first process copy P1, Server_1-1, Server_1-2 and Server_1-3 have belonged to same group of server.Server_2 and Server_3 has then taken on remaining P2～Pn process copy.
For the processing method of the stream data of the clearer description present invention, it is described in detail with the processing procedure of two aspect streaming data of data views and process visual angle separately below.
Referring to Fig. 4, illustrate the flow chart of the processing method of a kind of stream data of the present invention with data views, for the processing procedure of data D1-1, specifically may comprise steps of:
Step 401, source S1 produce data D1-1 (data that sequence is 1 that namely source S1 produces)；
Step 402, data D1-1 call arrival process copy P1 by this locality；
Step 403, data D1-1, through the process of input adapter, identify source S1；
Step 404, for the data creation new thread T1-1 (namely the thread T1-1 in process copy P1 processes from the data of S1) from S1, start T1-1；
Step 405, for T1-1 create buffer queue, D1-1 is put into queue end；
Step 406, T1-1 receive D1-1 from buffer queue, complete the calculation process process of P1 self；
Treated D1-1 is passed to next process copy P2 via o adapter by step 407, T1-1；
Specifically, it is possible to adopt local interface to call, the mode of any one or combination in any in text and JMS message these three transfer mode carries out the transmission of data between process copy.
Step 408, data D1-1 arrival process copy P2；
Step 409, data D1-1, through the process of input adapter, identify source S1；
Step 410, for the data creation new thread T2-1 (namely the thread T2-1 in process copy P2 processes from the data of S1) from S1, start T2-1；
Step 411, for T2-1 create buffer queue, D1-1 is put into queue end；
Step 412, T2-1 receive D1-1 from buffer queue, complete the calculation process process of P2 self；
Treated D1-1 is passed to next process copy P3 via o adapter by step 413, T2-1；
Step 414, and so forth, until Pn completes the processing procedure to D1-1；
Step 415, data D1-1 process terminate.
From data views, before stream data arrives, it is ready that different processes has formed pipeline, and data, from entering first process, through the process in turn of some processes, are finally completed all working.
Referring to Fig. 5, illustrate the flow chart of the processing method of a kind of stream data of the present invention with process visual angle, specifically may comprise steps of:
Step 501, process copy P1 carries out initialization operation；
Specifically, it is possible to include loading every parameter preset, the preliminary preparations such as teledata are downloaded.
Step 502, establishment input adapter, o adapter also initialize；
Step 503, input adapter wait that called or pending file increases or waits that JMS message arrives；
Step 504, the data D1-1 from data source S1 detected；
Step 505, create for the computing thread T1-1 of data source S1 and start；
Step 506, for T1-1 create buffer queue, D1-1 is put into queue end；
Step 507, T1-1 get latest data D1-1 from buffer queue, perform computing；
The D1-1 processed is transferred to next process through o adapter and copies P2 by step 508, T1-1；
Specifically, it is possible to adopt local interface to call, the mode of any one or combination in any in text and JMS message these three transfer mode.
Step 509, T1-1 continuously attempt to take latest data；
Step 510, T1-1 get latest data D1-2, perform computing；
……
With process visual angle, being loaded in internal memory from program, through initializing and a series of preparations, initially enter data decryptor state, after data arrive, perform the function logic of process copy self, and pass data to next process copy.
In a kind of application example of the present invention, it is provided that the definition of following procedural interface:
Procedural interface provides code skeleton, developer to inherit this interface for developer, realizes Virtual Function as requested and can use dual concurrent distributed arithmetic function provided by the invention.
publicabstractclassSuperFactory<Material,Product>{
protectedabstractList<Product>process(Materialt)throwsException；
protectedabstractSuperFactory<Material,Product>newInstance()；
publicvoidadd(Materialcontent){…}
}
The parameter that interface can be arranged has:
Daemon: whether new thread is created as daemon thread
Two kinds of threads of Java, user thread and daemon thread, daemon thread is the thread of low priority, and when only existing daemon thread in process, process automatically exits from.
Be set to the purpose of daemon thread is enable process exiting and exit thus avoiding the generation of zombie process with user thread herein.User thread is important thread, expendable exception occurs when user thread exits then representation program, now needs process actively to exit, in order to carry out follow-up automatically restarting the outside recovery measures such as process.
Name: thread name
Concurrent: whether concurrent processing (single example thread/multithreading)
BlockOut: go out whether team blocks
BlockIn: join the team and whether block
InWay: input mode (calls/file/JMS)
OutWay: the way of output (calls/file/JMS)
In a particular application, this abstract class SuperFactory need to only be inherited during developer's compiling procedure code.
This abstract class achieves adaption function, buffer queue, worker thread, singleton pattern.
This type of SuperFactory is the abstract class that all processes need to be inherited when realizing, and process inherent logic need to cover process method, calls and only occurs in after begin (i.e. data source) produces data, data source carries thread dispatching add method.
Example is as follows: call in main method.
The embodiment of the present invention provides various computing modes and parameter to arrange for the interface of process, to adapt to the needs of each process, such as high speed document buffer memory, congestion policies and queuing policy.Framework is responsible for the management of whole life cycle, adapter input and output, and each process only need to pay close attention to inherent logic.
Each process also needs to other aids except input data, if high speed document buffer memory is the one of instrument, independent of core frame, by providing file cache basic framework, it is possible to accomplish that simple exploitation can complete a high speed local cache Tong Bu with master library for inquiry.
When arranging and can reach to need and wait, thread block being saved computing by block policy, some needs the logic periodically judged can pass through to arrange the purpose that non-obstruction reach the cycle and perform.
Reach, by arranging priority queuing, the effect that prioritized data is jumped the queue, most high latency is set simultaneously and avoids unlimited Holding Problem.
In addition, the computing of the embodiment of the present invention carries out in process, continuous print process can divide physically be combined into process group (according to naturally understanding and operation strength, I/O intensity), process group according to data source division can corresponding multiple servers, thus realizing that calculating process is modular to be assembled.
Device embodiment
With reference to Fig. 6, it is shown that the structured flowchart processing device of a kind of stream data of the present invention, specifically may include that
Receiver module 610, for receiving stream data for active procedure copy；Wherein, described process copy is the computing entity possessing operation independent thread；
Processing module 620, for utilizing the thread that described active procedure copies that described stream data is processed；Wherein, the source one_to_one corresponding of described thread and described stream data；
Transmission module 630, copy for utilizing the thread that described active procedure copies that the stream data after process is passed to next process, by next process copy, the stream data after described process is processed, until described stream data is complete by last process copy process.
In one preferred embodiment of the invention, described processing module 620, specifically may include that
Recognition unit, for utilizing described active procedure copy that the source of described stream data is identified；
Judging unit, for judging whether thread corresponding to the source of the described stream data identified creates；
Creating unit, for when the thread that the source not creating described stream data is corresponding, the source for described stream data creates corresponding thread and starts this thread；
Join the team unit, for described stream data being added the buffer queue end of described thread；
Processing unit, for utilizing described thread take out described stream data from the buffer queue of its correspondence and process.
In another preferred embodiment of the invention, the stream data after processing is passed to one or several that transfer mode corresponding to next process copy specifically can include in the following manner by the described thread utilizing described active procedure to copy: the transfer mode of transfer mode, the transfer mode of text and JMS message that local interface calls.
In another preferred embodiment of the present invention, described receiver module, specifically may include that
Receive unit, receive stream data successively according to pre-set sequence for copying for active procedure.
For device embodiment, due to itself and embodiment of the method basic simlarity, so what describe is fairly simple, relevant part illustrates referring to the part of embodiment of the method.
Each embodiment in this specification all adopts the mode gone forward one by one to describe, and what each embodiment stressed is the difference with other embodiments, between each embodiment identical similar part mutually referring to.
Those skilled in the art are it should be appreciated that the embodiment of the embodiment of the present invention can be provided as method, device or computer program.Therefore, the embodiment of the present invention can adopt the form of complete hardware embodiment, complete software implementation or the embodiment in conjunction with software and hardware aspect.And, the embodiment of the present invention can adopt the form at one or more upper computer programs implemented of computer-usable storage medium (including but not limited to disk memory, CD-ROM, optical memory etc.) wherein including computer usable program code.
The embodiment of the present invention is that flow chart and/or block diagram with reference to method according to embodiments of the present invention, terminal unit (system) and computer program describe.It should be understood that can by the combination of the flow process in each flow process in computer program instructions flowchart and/or block diagram and/or square frame and flow chart and/or block diagram and/or square frame.These computer program instructions can be provided to produce a machine to the processor of general purpose computer, special-purpose computer, Embedded Processor or other programmable data processing terminal equipment so that the instruction performed by the processor of computer or other programmable data processing terminal equipment is produced for realizing the device of function specified in one flow process of flow chart or multiple flow process and/or one square frame of block diagram or multiple square frame.
These computer program instructions may be alternatively stored in and can guide in the computer-readable memory that computer or other programmable data processing terminal equipment work in a specific way, the instruction making to be stored in this computer-readable memory produces to include the manufacture of command device, and this command device realizes the function specified in one flow process of flow chart or multiple flow process and/or one square frame of block diagram or multiple square frame.
These computer program instructions also can be loaded on computer or other programmable data processing terminal equipment, make on computer or other programmable terminal equipment, to perform sequence of operations step to produce computer implemented process, thus the instruction performed on computer or other programmable terminal equipment provides for realizing the step of function specified in one flow process of flow chart or multiple flow process and/or one square frame of block diagram or multiple square frame.
Although having been described for the preferred embodiment of the embodiment of the present invention, but those skilled in the art are once know basic creative concept, then these embodiments can be made other change and amendment.So, claims are intended to be construed to include preferred embodiment and fall into all changes and the amendment of range of embodiment of the invention.
Finally, it can further be stated that, in this article, the relational terms of such as first and second or the like is used merely to separate an entity or operation with another entity or operating space, and not necessarily requires or imply the relation that there is any this reality between these entities or operation or sequentially.And, term " includes ", " comprising " or its any other variant are intended to comprising of nonexcludability, so that include the process of a series of key element, method, article or terminal unit not only include those key elements, but also include other key elements being not expressly set out, or also include the key element intrinsic for this process, method, article or terminal unit.When there is no more restriction, statement " including ... " key element limited, it is not excluded that there is also other identical element in including the process of described key element, method, article or terminal unit.
The method and apparatus above a kind of stream data provided by the present invention processed, it is described in detail, principles of the invention and embodiment are set forth by specific case used herein, and the explanation of above example is only intended to help to understand method and the core concept thereof of the present invention；Simultaneously for one of ordinary skill in the art, according to the thought of the present invention, all will change in specific embodiments and applications, in sum, this specification content should not be construed as limitation of the present invention.
Claims (8)
1. the processing method of a stream data, it is characterised in that described method includes:
Stream data is received for active procedure copy；Wherein, described process copy is the computing entity possessing operation independent thread；
Utilize the thread that described active procedure copies that described stream data is processed；Wherein, the source one_to_one corresponding of described thread and described stream data；
Utilize the thread that described active procedure copies that the stream data after process passes to next process copy, next process copy the stream data after described process is processed, until described stream data is complete by last process copy process.
2. the method for claim 1, it is characterised in that described stream data is processed by the described thread utilizing described active procedure to copy, including:
Utilize described active procedure copy that the source of described stream data is identified；
Whether the thread that the source of the described stream data that judgement identifies is corresponding creates, if creating, then described stream data adds the buffer queue end of described thread；If not creating, then create corresponding thread for the source of described stream data and start this thread, and being the buffer queue that described thread creation is corresponding, described stream data is added the buffer queue end that described thread is corresponding；
Described thread is utilized to take out described stream data from the buffer queue of its correspondence and process.
3. the method for claim 1, it is characterized in that, the stream data after processing is passed to one or several that transfer mode corresponding to next process copy include in the following manner by the described thread utilizing described active procedure to copy: the transfer mode of transfer mode, the transfer mode of text and JMS message that local interface calls.
4. the method for claim 1, it is characterised in that described for active procedure copy reception stream data, including:
Copy for active procedure and receive stream data according to pre-set sequence successively.
5. the process device of a stream data, it is characterised in that described device includes:
Receiver module, for receiving stream data for active procedure copy；Wherein, described process copy is the computing entity possessing operation independent thread；
Processing module, for utilizing the thread that described active procedure copies that described stream data is processed；Wherein, the source one_to_one corresponding of described thread and described stream data；And
Transmission module, copy for utilizing the thread that described active procedure copies that the stream data after process is passed to next process, by next process copy, the stream data after described process is processed, until described stream data is complete by last process copy process.
6. device as claimed in claim 5, it is characterised in that described processing module, including:
Recognition unit, for utilizing described active procedure copy that the source of described stream data is identified；
Judging unit, for judging whether thread corresponding to the source of the described stream data identified creates；
Creating unit, for when the thread that the source not creating described stream data is corresponding, the source for described stream data creates corresponding thread and starts this thread；
Join the team unit, for described stream data being added the buffer queue end of described thread；
Processing unit, for utilizing described thread take out described stream data from the buffer queue of its correspondence and process.
7. device as claimed in claim 5, it is characterized in that, the stream data after processing is passed to one or several that transfer mode corresponding to next process copy include in the following manner by the described thread utilizing described active procedure to copy: the transfer mode of transfer mode, the transfer mode of text and JMS message that local interface calls.
8. device as claimed in claim 5, it is characterised in that described receiver module, including:
Receive unit, receive stream data successively according to pre-set sequence for copying for active procedure.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410849709.9A CN105808585B (en) | 2014-12-29 | 2014-12-29 | Method and device for processing streaming data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410849709.9A CN105808585B (en) | 2014-12-29 | 2014-12-29 | Method and device for processing streaming data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105808585A true CN105808585A (en) | 2016-07-27 |
| CN105808585B CN105808585B (en) | 2020-01-24 |
Family
ID=56421474
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410849709.9A Active CN105808585B (en) | 2014-12-29 | 2014-12-29 | Method and device for processing streaming data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105808585B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110609852A (en) * | 2019-07-16 | 2019-12-24 | 招联消费金融有限公司 | Streaming data processing method and device, computer equipment and storage medium |
| CN111290744A (en) * | 2020-01-22 | 2020-06-16 | 北京百度网讯科技有限公司 | Stream computing job processing method, stream computing system and electronic device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6476826B1 (en) * | 2000-08-22 | 2002-11-05 | Vastvideo, Inc. | Integrated system and method for processing video |
| CN101694665A (en) * | 2009-10-27 | 2010-04-14 | 中兴通讯股份有限公司 | Method and device for data query of heterogeneous data source |
| CN103838552A (en) * | 2014-03-18 | 2014-06-04 | 北京邮电大学 | System and method for processing multi-core parallel assembly line signals of 4G broadband communication system |
-
2014
- 2014-12-29 CN CN201410849709.9A patent/CN105808585B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6476826B1 (en) * | 2000-08-22 | 2002-11-05 | Vastvideo, Inc. | Integrated system and method for processing video |
| CN101694665A (en) * | 2009-10-27 | 2010-04-14 | 中兴通讯股份有限公司 | Method and device for data query of heterogeneous data source |
| CN103838552A (en) * | 2014-03-18 | 2014-06-04 | 北京邮电大学 | System and method for processing multi-core parallel assembly line signals of 4G broadband communication system |
Non-Patent Citations (1)
| Title |
|---|
| 何逸越: "拼接式等离子显示系统音视频获取处理和网络传输系统的研制", 《万方数据库 浙江大学硕士学位论文》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110609852A (en) * | 2019-07-16 | 2019-12-24 | 招联消费金融有限公司 | Streaming data processing method and device, computer equipment and storage medium |
| CN111290744A (en) * | 2020-01-22 | 2020-06-16 | 北京百度网讯科技有限公司 | Stream computing job processing method, stream computing system and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105808585B (en) | 2020-01-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ben-Nun et al. | Groute: An asynchronous multi-GPU programming model for irregular computations | |
| US9589069B2 (en) | Platform for continuous graph update and computation | |
| CN110908788B (en) | Spark Streaming based data processing method and device, computer equipment and storage medium | |
| ES2765415T3 (en) | Microservices-based data processing apparatus, method and program | |
| CN106933669B (en) | Apparatus and method for data processing | |
| US20140130059A1 (en) | Lattice Computing | |
| US10580190B2 (en) | Graph based heterogeneous parallel processing system | |
| US9262223B2 (en) | Lazy initialization of operator graph in a stream computing application | |
| CN104094232A (en) | Processing element management in a streaming data system | |
| CN106649391B (en) | Method and device for processing graph data | |
| US11550621B2 (en) | Distributable and customizable load-balancing of data-associated computation via partitions and virtual processes | |
| EP3226133A1 (en) | Task scheduling and resource provisioning system and method | |
| JP2014525640A (en) | Expansion of parallel processing development environment | |
| CN107729138B (en) | Method and device for analyzing high-performance distributed vector space data | |
| CN111932257B (en) | Block chain parallelization processing method and device | |
| CN110945481B (en) | Method for executing a tuple graph program across a network | |
| CN107204998B (en) | Method and device for processing data | |
| Roth et al. | Pgx. d/async: A scalable distributed graph pattern matching engine | |
| Liu et al. | Optimizing shuffle in wide-area data analytics | |
| TWI689946B (en) | System of type inference for tuple graph programs and method of executing a tuple graph program across a network | |
| CN105808585A (en) | Stream data processing method and apparatus | |
| EP3011456B1 (en) | Sorted event monitoring by context partition | |
| CN115361382B (en) | Data processing method, device, equipment and storage medium based on data group | |
| CN110247818A (en) | A kind of data monitoring method, device, storage medium and server | |
| US20210243137A1 (en) | Multiplexed resource allocation architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |