CN104778042B - A kind of stream data processing method based on event stream processing and plug-in type Development Framework - Google Patents
A kind of stream data processing method based on event stream processing and plug-in type Development Framework Download PDFInfo
- Publication number
- CN104778042B CN104778042B CN201510146187.0A CN201510146187A CN104778042B CN 104778042 B CN104778042 B CN 104778042B CN 201510146187 A CN201510146187 A CN 201510146187A CN 104778042 B CN104778042 B CN 104778042B
- Authority
- CN
- China
- Prior art keywords
- event
- data
- plug
- unit
- outfield
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Stored Programmes (AREA)
Abstract
The invention discloses a kind of stream data processing method based on event stream processing and plug-in type Development Framework, including：Step 1, plug-in unit host program is created；Step 2, the interface specification between plug-in unit host program and plug-in unit is defined；Step 3, the data acquisition protocol for different outfield awareness apparatus generates corresponding data acquisition plug-in unit；Step 4, event stream processing engine is created；Step 5, it is event stream processing engine definitions time window or length window；Step 6, the data acquisition protocol for different outfield awareness apparatus realizes corresponding data processing rule；Step 7, definition and registered events；Step 8, it is the event of registration is associated with the various data processing rules realized in step 6 respectively；Step 9, event handling condition is set；Step 10, source data queue and object queue are created；Step 11, consuming thread is created；Step 12, target data is exported into destination；Step 13, finger daemon is created.
Description
Technical field
At the real-time data acquisition that sensing equipment is perceived the present invention relates to outfields such as electric power, traffic, energy-conserving and environment-protective, new energy
Reason method, particularly a kind of stream data processing method based on event stream processing and plug-in type Development Framework.
Background technology
At present when carrying out the data acquisition process of outfield awareness apparatus, different outfields perception are generally docked simultaneously and are set
It is standby.Due to different awareness apparatus, its communication protocol is different, and the even difference having is very big, and traditional way is typically all to be directed to
The data acquisition of different outfield awareness apparatus generally carries out special exploitation.
The data volume gathered when sensing equipment is more is than larger, and the real-time of data processing is difficult to ensure that.Even if using
The temporary cache that message-oriented middleware carries out data can also have multiple treatment progress contention message-oriented middlewares, now message
Middleware becomes the bottleneck of real-time lifting.
The data acquisition treatment method of traditional outfield awareness apparatus is often coupling in the collection, processing and analysis of data
Together, and then it result in the decline of data processing overall performance.
The content of the invention
Goal of the invention：The technical problems to be solved by the invention are to be based in view of the shortcomings of the prior art there is provided one kind
The stream data processing method of event stream processing and plug-in type Development Framework.
In order to solve the above-mentioned technical problem, the invention discloses a kind of based on event stream processing and plug-in type Development Framework
Stream data processing method, comprises the following steps：
Step 1, plug-in unit host program is created；
Step 2, the interface specification between plug-in unit host program and plug-in unit is defined；
Step 3, according to the interface specification in step 2, for the data acquisition protocol of a variety of outfield awareness apparatus,
The generation data acquisition plug-in unit corresponding with various data acquisition protocols；
Step 4, event stream processing engine is created based on Esper frameworks, as real-time event driver framework, when in flow of event
When having the generation of event treatment conditions, event stream processing engine triggering custom action；
Step 5, it is event stream processing engine definitions time window or length window, time window defines certain time scope
Data in interior or any one period, length window defines the data of several nearest events；
Step 6, the data acquisition protocol for a variety of outfield awareness apparatus is realized and each data acquisition protocol phase
Corresponding data processing rule；
Step 7, definition and registered events, are defined outside various events, each class for a variety of outfield awareness apparatus
Field awareness apparatus is defined as an event, and by various event registrations into event stream processing engine；
Step 8, it is the event registered in step 7 is associated with the various data processing rules realized in step 6 respectively, use
The custom action of correlation can be triggered automatically when corresponding event occurs；
Step 9, event handling condition is set；
Step 10, source data queue PackageMessageQueue is created, a variety of outfield awareness apparatus hairs are deposited
The initial data brought, and object queue RecordMessageQueue, for depositing the target data after decoding process；
Step 11, source data consuming thread ConsumePackage and target data consuming thread are created
ConsumeRecord：
(1) source data consuming thread ConsumePackage is constantly from source data queue PackageMessageQueue
Middle taking-up data, and the data are packaged into according to the type of data by the event object of corresponding types, and be sent at flow of event
Engine is managed, triggering corresponding data processing rule when reaching the event handling condition that step 9 is set is handled, after processing
Obtained data are put into the object queue RecordMessageQueue in step 10；
(2) target data consuming thread ConsumeRecord is constantly from object queue RecordMessageQueue
Take out target data；
Step 12, the target data that step 11 is obtained is exported into destination, such as can export to database, file or
In message-oriented middleware；
Step 13, finger daemon is created, finger daemon is responsible for the working condition of detecting event flow data engine processing, works as stream
The working condition of data processor is actively closed flow data processing routine and restarted when exception occur or being stopped.
In the present invention, further, the plug-in unit host program described in step 1 includes following functions：
(1a) registered plug-ins：Plug-in unit is registered in host program before the use, for positioning plug-in unit path；Only
Plug-in unit path could be correctly found after succeeding in registration；
(2) anti-registered plug-ins：Counter to plug-in unit it can be registered when host program no longer needs to use the plug-in unit,
So that it no longer turns into a part for host program；
(3) plug-in unit is enabled：Chartered plug-in unit can be enabled；
(4) plug-in unit is disabled：When not using some plug-in unit, the plug-in unit being currently running can be disabled.
The function of data acquisition plug-in unit in step 3 can be realized by dynamic link library technology.
The establishment process of event stream processing engine described in step 4 is as follows：
(4a) pre-defines the corresponding event of various awareness apparatus according to various outfield awareness apparatus, then by various things
Part type is registered in configuration surroundings, generates related configuration object；
(4b) is used as the container environment of event handling to configure object as parameter establishment event stream processing engine instance；
(4c) creates or imported event handling rule；
(4d) creates monitor object, and monitor object is associated with event handling rule；
(4e) event stream processing engine create event stream processing performing environment object, receive flow of event, in engine with
Event handling condition carries out logic matching；
(4f) performing environment object is responsible for monitoring all events come, when event matches event handling bar set in advance
Corresponding monitor is triggered during part outfield awareness apparatus event progress is handled and obtain target data.
Data processing rule described in step 6, which refers to, to be received to the data of outfield awareness apparatus and according to awareness apparatus
Acquisition protocols decoding, decoded data are handled according to user's request.
Event described in step 7 is a triple<Id, attrs, time>, wherein id is unique mark of the event
Know；Attrs represents the attribute of the event, the different attribute of the data packaging produced for different outfield awareness apparatus
A variety of outfield awareness apparatus are identified, event stream processing engine is belonged to by attrs by event using attrs attributes
Property selects corresponding data processing rule；Time represents the time that event occurs, and identifies a variety of outfield awareness apparatus
Data acquisition time, and be used as a condition entry of event stream processing.
The present invention is more flexibly more increased by the way that event stream processing and plug-in type Development Framework are combined there is provided a kind of
The real-time streaming data processing method of effect.Compared with prior art, have an advantageous effect in that：
(1) plug-in type Development Framework is incorporated into Data processing by the present invention so that various outfield awareness apparatus are transmitted
The adaptation of agreement is completed by way of loading plug-in unit, more flexibly, and autgmentability is strong.
(2) present invention uses the mechanism based on event stream processing, data processing is carried out based on flow of event, the number to be analyzed
According to the abstract input and the processing of first registers for being event, according to event after event stream processing engine receives dependent event
Model obtains the result of event so that data-handling efficiency is higher.
By the mechanism for introducing event stream processing so that traditional data processing platform really possesses the spy of flow data processing
Levy, no longer the storage of focused data, but the data of user's care are got while data flow through event stream processing engine.
(3) handled due to converting data to event, can be according to pre-defined time window or length window
Statistical analysis is carried out to event information, so as to quickly find that the abnormal data collected (such as becomes big, diminished suddenly suddenly
Etc.) and responded, operation system is accomplished from control self feed back.
(4) by introducing finger daemon, the management work of process itself is completed, when occurring being stopped or working condition
Flow data processing software can be restarted in the case of exception, it is ensured that flow data processing software " never shutting down ".
Brief description of the drawings
The present invention is done with reference to the accompanying drawings and detailed description and further illustrated, of the invention is above-mentioned
And/or otherwise advantage will become apparent.
Fig. 1 is the implementation process of the stream data processing method of the invention based on event stream processing and plug-in type Development Framework.
Fig. 2 is the architecture of typical plug-in type Development Framework.
Fig. 3 is the flow chart that host program manages plug-in unit.
Fig. 4 is the establishment process of event stream processing engine.
Fig. 5 is the implementation procedure of event stream processing engine.
Fig. 6 is the processing window schematic diagram of event stream processing engine.
Fig. 7 is the class figure of some Sensor Events.
Embodiment
Fig. 1 is the implementation process of the stream data processing method based on event stream processing and plug-in type Development Framework, wherein needing
Real-time data acquisition processing is carried out to a variety of outfield awareness apparatus 1, equipment 2..., equipment N, data acquisition plug-in unit is born
The data acquisition of various different sensing equipments is blamed, different outfields are docked according to different outfield awareness apparatus Data Transport Protocols
The corresponding relation of awareness apparatus, gathered data, data acquisition plug-in unit and equipment by configuration information interface read configuration file come
Obtain；The initial data collected is put into source data queue, and consuming thread A constantly takes out from source data queue
Data and the event object that data are packaged into corresponding types according to the type of data, are sent to event stream processing engine, event
Stream process engine can match different event handling rules according to a variety of event types and event is handled, and obtain
Target data is simultaneously put into object queue, and consuming thread B is responsible for taking out target data from object queue and inputs to export
Module, export module exports to target data according to no usage scenario different data formats.
Step 1, plug-in unit host program is created, host is the environment for carrying plug-in component operation, and most base is provided for the operation of plug-in unit
This service, while receiving the service that plug-in unit is provided.
Plug-in part technology is a kind of component technology in itself, and the technology is simpler than component technologys such as com, is constructing locally applied system
It is more prone to realize during system, by needing summation function to be divided software in the design and R＆D process of software, makes journey
Sequence is divided into two major parts：Host program and plug-in unit.The functional requirement on basis is designed in host program, in addition, host
Program also provides the interface with plug-in unit, corresponding plug-in unit is carried out data exchange according to certain rule, so as to realize one
A little functions, Fig. 2 is the architecture of typical plug-in type Development Framework；And plug-in unit is then the group for realizing partial function one by one
Part, the function of software is adjusted by increasing and decreasing plug-in unit or modification plug-in unit built-in function, because plug-in type is relatively independent part,
Edlin can independently be entered.So as to realize the extension of software function and update.
The major function of host program is that plug-in unit is managed, and major function includes：
(1) registered plug-ins：Plug-in unit must be registered in host program before the use, ability after only succeeding in registration
Correctly find plug-in unit path.
(2) anti-registered plug-ins：Counter to plug-in unit it can be registered when host program no longer needs to use the plug-in unit,
So that it no longer turns into a part for host program.
(3) plug-in unit is enabled：Chartered plug-in unit can be enabled.
(4) plug-in unit is disabled：When not using some plug-in unit, the plug-in unit being currently running can be disabled.
Fig. 3 is the flow chart that host program manages plug-in unit, when it is new plug-in unit that host program, which finds the plug-in unit currently loaded,
Initialization operation is carried out to the plug-in unit, plug-in registration into host frame and is then started into plug-in unit；When host program currently adds
Plug-in unit can first be disabled first by having been loaded before the plug-in unit of load, and then plug-in unit is updated, after the completion of update of plug-in again
Start plug-in unit.
Step 2, the interface specification between plug-in unit host program and plug-in unit is defined.
Host program and plug-in unit can be combined together work, it is necessary to have set of rule and agreement to make this of separate sources
A little Program Coordination runnings.As a rule the interface specification between host program and plug-in unit all by the way of abstraction interface come real
It is existing.
(1) interface between data acquisition plug-in unit and host program：
| Module is acted on | ● unified data acquisition interface is provided |
| Module characteristic | Abstract class, for being inherited |
| Class name (object name) | Communicater |
| Primary interface | Virtual void run_collect (void)=0； |
By this interface, the collection plug-in unit of various sensors can be linked into host program, according to the class of sensor
Type calls corresponding plug-in unit.
(2) interface between decoder plug-in unit and host program：
| Module is acted on | ● unified decoding interface is provided |
| Module characteristic | Abstract class, for being inherited |
| Class name (object name) | Decoder |
| Primary interface | Virtual intdecode_package (ACE_Message_Block*＆mb)=0； |
By this interface, the sensor raw data that collection comes can be decoded, according to the type of sensor
Call corresponding plug-in unit.
Step 3, according to the interface specification in step 2, the agreement for different outfield awareness apparatus data acquisitions is different,
Such as agreement 1, agreement 2 ..., agreement N, by realizing that various acquisition protocols generate various data acquisition plug-in units, such as gather and insert
Part 1, collection plug-in unit 2 ..., collection plug-in unit N.
Different outfield awareness apparatus and its corresponding data acquisition protocol is exemplified below：
Collection plug-in unit is used for being adapted to the collecting work of various different outfield awareness apparatus, according to specific data acquisition protocol
Exploitation, is mounted in host program by the interface between data acquisition plug-in unit and host program.
The concrete function for gathering plug-in unit is that the data that various outfield awareness apparatus are sent are obtained in communication process.Plug-in unit is needed
To be loaded by host program and be managed by host program and controlled, can dynamic insertion into host program, strengthen original program
Function.The realization of plug-in unit typically has two schemes：The Component Object Model and dynamic link library.Dynamic link is used in the present invention
Storehouse technology realizes feature card, and dynamic link library (DLL) is the executable software module of certain function, and itself can not independently be transported
OK, can only output variable, function or class, by other can the programs (host program) of independent operatings call its internal work(
Energy.
Step 4, event stream processing engine is created based on Esper frameworks, Esper is an event stream processing framework, works as thing
When having event condition generation in part stream, event stream processing engine can trigger custom action.
Fig. 4 is the establishment process of event stream processing engine, is comprised the following steps：
(1) processing of event stream processing engine is in units of event, so being needed in event stream processing engine in advance
Various events are defined, the corresponding event of various awareness apparatus are pre-defined according to various outfield awareness apparatus in the present invention,
Then various event types are registered in configuration surroundings, generate related configuration object；
(2) to configure object as parameter establishment event stream processing engine instance, it is used as the container environment of event handling；
(3) event handling rule is created or imports, event handling rule is for various different outfield awareness apparatus
The method that data are handled, different outfield awareness apparatus data processing method method is different, (for example, being adopted based on the fixed cycle
Under the scene for collecting device data, the processing rule of event is exactly a task of timer, such as calculates the ammeter received in 5 minutes
The average value of data, the peak value of ammeter data received in nearest one day etc.), the good post-registration of rule creation draws to event stream processing
In holding up.
(4) monitor object is created, and monitor object is associated with event handling rule.
(5) event stream processing engine creates the performing environment object of event stream processing, flow of event is received, with thing in engine
Part treatment conditions carry out logic matching.
(6) performing environment object is responsible for monitoring all events come, when event matches condition set in advance with regard to energy
Enough trigger corresponding monitor outfield awareness apparatus event progress is handled and obtain target data, target data is flushed in time
Caching, and it is persisted to database.Fig. 5 is the implementation procedure of event stream processing engine, and event stream processing engine receives event
Stream carries out condition coupling to event, and difference can be distributed to according to different event types when condition meets set in advance require
Event handling rule event is handled.
Step 5, it is event stream processing engine definitions time window or length window, time window is used for defining certain time
In the range of or certain period in data handled；Length window is used for defining at the data of nearest how many events
Reason.
Fig. 6 is the processing window schematic diagram of event stream processing engine, by defining length window and event window, Neng Gougen
Statistical analysis is carried out to event information according to pre-defined time window or length window, collected so as to quick discovery
Abnormal data (such as become big suddenly, diminish suddenly etc.) and responded, operation system is accomplished from control self feed back.
Step 6, to agreement 1, agreement 2 ..., agreement N, realize the data processing method of each agreement, data processing rule 1,
Data processing rule 2 ..., data processing rule N.
The outfield received perception sensing data is decoded and carries out correlation by the major function of data processing rule
Processing.
Step 7, definition and registered events, are defined outside various events, each class for a variety of outfield awareness apparatus
Awareness apparatus is defined as an event, such as sensing equipment 1, sensing equipment 2 ..., sensing equipment N define thing respectively
Part 1, event 2 ..., event N, and various event types are registered in event stream processing engine.
Event is a triple E=(id, attrs, time), and wherein id is the mark of the event, and each event has
One unique mark, attrs represents the attribute of the event, is packaged into for the data that different outfield awareness apparatus is produced
The event of different attribute, is used for being identified a variety of outfield awareness apparatus, event stream processing using this property value
Engine selects corresponding processing rule by this property value, and time represents the time that event occurs, the collection of mark data
Time, can as event stream processing a condition entry.
The event object of public class device//some equipment
{
String DeviceName；// device name
intEventId；// event id
public String getDeviceName( )
{returnDeviceName；}
publicintgetEventId( )
{returnEventId}
}
Epimere code illustrates a simplest device events object, is assigned for different equipment DeviceName
For different marks, EventId is used for the id of identified event, and each event is owned by different No. id.
Step 8, it is the event registered in step 7 is related to the various data rule methods realized in preceding step 6 respectively
Connection.The custom action of correlation can be triggered when occurring so as to corresponding event automatically.
Step 9, event handling condition is set.
Step 10, creating source data queue PackageMessageQueue, (each outfield awareness apparatus of storage is sended over
Initial data) and object queue RecordMessageQueue (being used for depositing the target data after decoding process).
Step 11, consuming thread is created.Create source event consuming thread ConsumePackage and target data consumption line
Journey ConsumeRecord.
(1) source event consuming thread ConsumePackage can be constantly from source data queue
Data are taken out in PackageMessageQueue, and data are packaged into according to the type of data the event object of corresponding types,
And event stream processing engine is sent to, can trigger corresponding data processing rule when reaching certain predefined condition is carried out
Processing, the data obtained after processing are put into the object queue RecordMessageQueue in step 10.
(2) target data consuming thread ConsumeRecord can be constantly from object queue RecordMessageQueue
Middle taking-up data.
Step 12, the target data that step 11 is obtained can need to export to different destinations according to business, for example, will
Data export to database, data are exported to file, data are exported into message-oriented middleware.
Step 13, finger daemon is created, finger daemon is responsible for heartbeat and the working condition of detection stream data processing process, when
The working condition of flow data treatment progress can actively close residual process and start newly to enter when exception occur or being stopped
Journey.
Finger daemon mainly has 3 functions：(1) stream data processing routine is managed, including it is carried out to start/close
Closed operation；(2) current operating state of flow data processing routine is obtained；(3) the heartbeat letter of flow data processing routine can be obtained
Breath.
Embodiment
Below by Windows run exemplified by explanation.
Step 1, host program CollectService is started；
Step 2, the interface specification between plug-in unit host program and plug-in unit is defined；Specific interface specification is as follows：
(1) interface between data acquisition plug-in unit and host program：
| Module is acted on | ● unified data acquisition interface is provided |
| Module characteristic | Abstract class, for being inherited |
| Class name (object name) | Communicater |
| Primary interface | Virtual void run_collect (void)=0； |
(2) interface between decoder plug-in unit and host program：
| Module is acted on | ● unified decoding interface is provided |
| Module characteristic | Abstract class, for being inherited |
| Class name (object name) | Decoder |
| Primary interface | Virtual intdecode_package (ACE_Message_Block*＆mb)=0； |
Step 3, the information that host program is provided according to configuration load-on module determines that this time operation needs the data loaded to adopt
Collecting plug-in unit is：Eda9033.dll, Eda9033.dll, Eda9033.dll, wherein Eda9033.dll plug-in units use NODBUS-
RTU/DLT645-1997/DLT645-2007 agreements；Eda9033A.dll plug-in units use DL/T645-97 agreements；Ex8-33.dll
Plug-in unit uses DLT645-1997/DLT645-2007 agreements；Ex8.dll plug-in units use MODBUS-RTU agreements.Call successively each
Run_collect () method of individual plug-in unit, starts collection service；The initial data of sensor to collecting calls correspondence
The decode_package (ACE_Message_Block*＆mb) of sensing equipment carries out the decoding of initial data.
Step 4, event stream processing engine is created based on Esper frameworks, is used as the container environment of event stream processing.
Step 5, it is event stream processing engine definitions time window or length window, time window or length window are events
The condition of stream process, event stream processing engine read in event queue all event ids according to this condition and to event at
Reason, each event is a triple, including (ammeter id, ammeter and its correspondence collector attribute attrs, current time
time).Attrs represents the attribute of the event, including collector IP (ip), port numbers (port), volume of the ammeter in collector
Number (collectid), decoder number (decodeid), ammeter data (data) etc..Such as if necessary to feel to some outfield
Know that the equipment data event of nearest one day carries out statistical analysis, it is necessary to which time window is set to one day.
Step 6, according to different event type setting event handling rules (corresponding with the type of outfield awareness apparatus),
Event handling rule is realized by EPL language.Such as calculate the average value of the daily ammeter data of some outfield awareness apparatus：
selectavg(data)fromSensorEventevery1day.
Some sensor data event (SensorEvent) nearest one day ammeter data (Data) of above-mentioned Rule
Average value；
Step 7, definition and registered events, are defined outside various events, each class for a variety of outfield awareness apparatus
Field awareness apparatus is defined as an event, and by various event registrations into event stream processing engine；
Event exists in the form of classes, and the attribute of event is corresponding member variable in class.Fig. 7 is some Sensor Events
Class figure.Wherein Id is event id, represents the uniqueness (perceiving sensing equipment type association with outfield) of event；
CollectTime is the raw data acquisition time；SourceIp is the IP address of the sensing equipment of data source；SourcePort
For the port of the sensing equipment of data source；CollectId is the ID of corresponding collection plug-in unit, and DecodeId is corresponding decoding
The ID of plug-in unit, Data are the ammeter data that actual acquisition is obtained.
Step 8, it is the event registered in step 7 is associated with the various data processing rules in step 6 respectively, for phase
Automatic triggering related custom action when the event answered occurs；
Step 9, event handling condition is set；
The treatment conditions of event are exactly the value of some member variable in event class, such as the event in step 7, such as
Fruit will set the time window of event handling to be one day, then need CollectTime's when processing rule is write
Span is set in the range of nearest one day；If necessary to handle some IP data collected, it is only necessary to
IP is set to corresponding numerical value.
Step 10, source data queue PackageMessageQueue is created, is set for depositing a variety of outfields perception
The initial data that preparation is brought, and object queue RecordMessageQueue, for depositing the target after decoding process
Data；
Step 11, source data consuming thread ConsumePackage and target data consuming thread ConsumeRecord is created：
(11a) source data consuming thread ConsumePackage is constantly from source data queue PackageMessageQueu
Data are taken out in e, and the data are packaged into according to the type of data by the event object of corresponding types, and are sent at flow of event
Manage engine.
| Module is acted on | ● unified event package attaching mouthful is provided |
| Module characteristic | Abstract class, for being inherited |
| Class name (object name) | EventWrap |
| Primary interface | Virtual intEvent_Wrap (ACE_Message_Block*＆mb)=0； |
ConsumePackage is by interface Event_Wrap (ACE_Message_Block*＆mb) by the original collected
Beginning sensing data is by calling the constructed fuction of event class to create event instance, so that initial data packaging is specific for some
Event.
Corresponding data processing rule is triggered when reaching the event handling condition that step 9 is set to be handled, and is handled
The data obtained afterwards are put into the object queue RecordMessageQueue in step 10；
(11b) target data consuming thread ConsumeRecord is constantly from object queue RecordMessageQueue
Middle taking-up target data；
Step 12, the target data that step 11 is obtained is exported into destination；
Step 13, finger daemon is created, finger daemon is responsible for the working condition of detection stream data processor, works as flow data
The working condition of processing routine is actively closed flow data processing routine and restarted when exception occur or being stopped.
It is specific real the invention provides a kind of stream data processing method based on event stream processing and plug-in type Development Framework
Now the method and approach of the technical scheme are a lot, and described above is only the preferred embodiment of the present invention, it is noted that for this
For the those of ordinary skill of technical field, under the premise without departing from the principles of the invention, some improvement and profit can also be made
Decorations, these improvements and modifications also should be regarded as protection scope of the present invention.Each part being not known in the present embodiment can use
Prior art is realized.
Claims (7)
1. a kind of stream data processing method based on event stream processing and plug-in type Development Framework, it is characterised in that including following
Step：
Step 1, plug-in unit host program is created；
Step 2, the interface specification between plug-in unit host program and plug-in unit is defined；
Step 3, according to the interface specification in step 2, for the data acquisition protocol of a variety of outfield awareness apparatus, generation
The data acquisition plug-in unit corresponding with various data acquisition protocols；
Step 4, event stream processing engine is created based on Esper frameworks, as real-time event driver framework, when busy in flow of event
When part treatment conditions occur, event stream processing engine triggering custom action；
Step 5, be event stream processing engine definitions time window or length window, time window define in the range of certain time or
Data in any one period of person, length window defines the data of several nearest events；
Step 6, realized according to the data acquisition protocol of a variety of outfield awareness apparatus corresponding with each data acquisition protocol
Data processing rule；
Step 7, definition and registered events, various events, each class outfield sense are defined for a variety of outfield awareness apparatus
It is an event to know device definition, and by various event registrations into event stream processing engine；
Step 8, it is the event registered in step 7 is associated with the various data processing rules in step 6 respectively, for corresponding
Automatic triggering related custom action when event occurs；
Step 9, event handling condition is set；
Step 10, source data queue PackageMessageQueue is created, for depositing a variety of outfield awareness apparatus hairs
The initial data brought, and object queue RecordMessageQueue, for depositing the target data after decoding process；
Step 11, source data consuming thread ConsumePackage and target data consuming thread ConsumeRecord is created：
(11a) source data consuming thread ConsumePackage is constantly from source data queue PackageMessageQueue
Data are taken out, and the data are packaged into according to the type of data by the event object of corresponding types, and are sent to event stream processing
Engine, triggers corresponding data processing rule and is handled when reaching the event handling condition that step 9 is set, after processing
To data be put into object queue RecordMessageQueue in step 10；
(11b) target data consuming thread ConsumeRecord constantly takes from object queue RecordMessageQueue
Go out target data；
Step 12, the target data that step 11 is obtained is exported into destination；
Step 13, finger daemon is created, finger daemon is responsible for monitoring the working condition of flow data processing routine, when flow data processing
The working condition of program is actively closed flow data processing routine and restarted when exception occur or being stopped.
2. a kind of stream data processing method based on event stream processing and plug-in type Development Framework as claimed in claim 1, its
It is characterised by, the plug-in unit host program described in step 1 includes following functions：
(1a) registered plug-ins：Plug-in unit is registered in host program before the use, for positioning plug-in unit path；
(1b) anti-registered plug-ins：It is counter to plug-in unit when host program no longer needs to use the plug-in unit to be registered, make it not
Turn into a part for host program again；
(1c) enables plug-in unit：Enable chartered plug-in unit；
(1d) disables plug-in unit：When not using some plug-in unit, the plug-in unit being currently running is disabled.
3. a kind of stream data processing method based on event stream processing and plug-in type Development Framework as claimed in claim 2, its
It is characterised by, the data acquisition plug-in unit described in step 3 is used to obtain various in communication process by dynamic link library technology
The data that outfield awareness apparatus is sent.
4. a kind of stream data processing method based on event stream processing and plug-in type Development Framework as claimed in claim 3, its
It is characterised by, the establishment process of the event stream processing engine described in step 4 is as follows：
(4a) pre-defines the corresponding event of various awareness apparatus according to various outfield awareness apparatus, then by various event classes
Type is registered in configuration surroundings, generates related configuration object；
(4b) is used as the container environment of event handling to configure object as parameter establishment event stream processing engine instance；
(4c) creates or imported event handling rule；
(4d) creates monitor object, and monitor object is associated with event handling rule；
(4e) event stream processing engine creates the performing environment object of event stream processing, flow of event is received, with event in engine
Treatment conditions carry out logic matching；
(4f) performing environment object is responsible for monitoring all events come, when event matches event handling condition set in advance
Trigger corresponding monitor outfield awareness apparatus event progress is handled and obtain target data.
5. a kind of stream data processing method based on event stream processing and plug-in type Development Framework as claimed in claim 4, its
It is characterised by, the data processing rule described in step 6, which refers to, to be received to the data of outfield awareness apparatus and set according to perception
Decoded data are handled by standby acquisition protocols decoding according to user's request.
6. a kind of stream data processing method based on event stream processing and plug-in type Development Framework as claimed in claim 5, its
It is characterised by, the event described in step 7 is a triple<Id, attrs, time>, wherein id is unique mark of the event
Know；Attrs represents the attribute of the event, the different attribute of the data packaging produced for different outfield awareness apparatus
A variety of outfield awareness apparatus are identified, event stream processing engine is belonged to by attrs by event using attrs attributes
Property selects corresponding data processing rule；Time represents the time that event occurs, and identifies a variety of outfield awareness apparatus
Data acquisition time, and be used as a condition entry of event stream processing.
7. a kind of stream data processing method based on event stream processing and plug-in type Development Framework as claimed in claim 6, its
It is characterised by, in step 12, the target data that step 11 is obtained is exported into database, file or message-oriented middleware.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510146187.0A CN104778042B (en) | 2015-03-30 | 2015-03-30 | A kind of stream data processing method based on event stream processing and plug-in type Development Framework |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510146187.0A CN104778042B (en) | 2015-03-30 | 2015-03-30 | A kind of stream data processing method based on event stream processing and plug-in type Development Framework |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104778042A CN104778042A (en) | 2015-07-15 |
| CN104778042B true CN104778042B (en) | 2017-09-19 |
Family
ID=53619528
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510146187.0A Active CN104778042B (en) | 2015-03-30 | 2015-03-30 | A kind of stream data processing method based on event stream processing and plug-in type Development Framework |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104778042B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106095535B (en) * | 2016-06-08 | 2019-11-08 | 东华大学 | A kind of thread management system for supporting Data Stream Processing under multi-core platform |
| CN108614820B (en) * | 2016-12-09 | 2021-01-15 | 腾讯科技（深圳）有限公司 | Method and device for realizing streaming source data analysis |
| CN107133324B (en) * | 2017-05-04 | 2020-03-20 | 浪潮云信息技术有限公司 | Processing engine and method for streaming data based on domestic CPU and operating system |
| CN107368334B (en) * | 2017-07-24 | 2020-08-28 | 北京奇艺世纪科技有限公司 | Business plug-in interaction system and method |
| CN107545186B (en) * | 2017-09-06 | 2020-03-17 | 杭州安恒信息技术股份有限公司 | Method, device and system for rapidly solving problem of engine non-work |
| CN110492907B (en) * | 2018-05-15 | 2023-01-06 | 华为技术有限公司 | Method and device for establishing archive for ammeter |
| CN109344170B (en) * | 2018-09-04 | 2022-04-12 | 创新先进技术有限公司 | Stream data processing method, system, electronic device and readable storage medium |
| CN109284098A (en) * | 2018-09-25 | 2019-01-29 | 四川长虹电器股份有限公司 | Method based on RxJava building event bus RxBus |
| CN110415518B (en) * | 2019-07-25 | 2020-11-24 | 民航成都信息技术有限公司 | Passenger flow and transport capacity monitoring and early warning method, device, equipment and medium |
| CN110457250B (en) * | 2019-08-29 | 2023-04-07 | 苏州德姆斯信息技术有限公司 | Embedded data acquisition processing terminal and data acquisition processing method |
| CN114691235B (en) * | 2022-03-22 | 2024-03-22 | 上海智殷自动化科技有限公司 | Method for supporting free interaction of data between plugins |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101685466A (en) * | 2009-07-22 | 2010-03-31 | 中兴通讯股份有限公司 | Event handling method and event handling equipment |
| CN102955841A (en) * | 2011-08-15 | 2013-03-06 | 德商赛克公司 | Systems and/or methods for forecasting future behavior of event streams in complex event processing (cep) environments |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120311562A1 (en) * | 2011-06-01 | 2012-12-06 | Yanlin Wang | Extendable event processing |
-
2015
- 2015-03-30 CN CN201510146187.0A patent/CN104778042B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101685466A (en) * | 2009-07-22 | 2010-03-31 | 中兴通讯股份有限公司 | Event handling method and event handling equipment |
| CN102955841A (en) * | 2011-08-15 | 2013-03-06 | 德商赛克公司 | Systems and/or methods for forecasting future behavior of event streams in complex event processing (cep) environments |
Non-Patent Citations (2)
| Title |
|---|
| 一种基于事件流处理引擎的企业服务总线平台;张潇毅，邓波，丁鲲;《解放军理工大学学报(自然科学版)》;20100228;第11卷(第2期);第178-183页 * |
| 插件技术及其应用;姜昌华;《计算机应用与软件》;20031031(第10期);第10-11页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104778042A (en) | 2015-07-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104778042B (en) | A kind of stream data processing method based on event stream processing and plug-in type Development Framework | |
| RU2419986C2 (en) | Combining multiline protocol accesses | |
| US8730819B2 (en) | Flexible network measurement | |
| EP2810412B1 (en) | Systems and methods for extracting structured application data from a communications link | |
| CN108200086B (en) | High-speed network data packet filtering device | |
| CN109831318A (en) | A kind of system, method and server obtaining network topology | |
| KR20170020311A (en) | Wireless sensor network | |
| CN110401624A (en) | The detection method and system of source net G system mutual message exception | |
| WO2013063950A1 (en) | Inspection method and system of multimode communication device | |
| CN109639754A (en) | A kind of implementation method of dispatching of power netwoks gateway Data Audit | |
| CN104852939B (en) | A kind of method and system of deployment ability interface | |
| US8305908B2 (en) | System analysis method, system analysis apparatus, and computer readable storage medium storing system analysis program | |
| CN110912782B (en) | Data acquisition method, device and storage medium | |
| CN112818064A (en) | Multi-system data synchronization method, device, equipment and storage medium | |
| US20080162690A1 (en) | Application Management System | |
| CN113364624B (en) | Mixed cloud flow acquisition method and system based on edge computing | |
| WO2021004197A1 (en) | Method and device for checking electric energy data, and system | |
| CN113691460B (en) | Data transmission method, device, equipment and storage medium based on load balancing | |
| CN109413001A (en) | The method and device of safeguard protection is carried out to the interaction data in cloud computing system | |
| CN104202328B (en) | A kind of method, configuration module and the subscription end of subscription GOOSE/SMV messages | |
| US7961614B2 (en) | Information processing device, information processing method, and recording medium for reducing consumption of memory capacity | |
| US20180324150A1 (en) | Integrated pcs functional competency assessment | |
| CN103731345B (en) | A kind of network message processing method and equipment | |
| CN101510843B (en) | Method for real time separation of P2P flow based on NetFlow flow | |
| CN105337797A (en) | Data capturing method of network protocol of complex electronic information system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 210000 Tianyuan East Road, Jiangning District, Nanjing, Jiangsu Province, No. 1 Patentee after: Zhong Tong clothing consulting and Design Research Institute Co., Ltd. Address before: 210000 Tianyuan East Road, Jiangning District, Nanjing, Jiangsu Province, No. 1 Patentee before: Jiangsu Posts & Telecommunications Planning and Designing Institute Co., Ltd. |
|
| CP01 | Change in the name or title of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: The Jianye District of Nanjing City, Jiangsu province 210000 Jiangdong Street No. 58 Patentee after: CHINA INFORMATION CONSULTING & DESIGNING INSTITUTE Co.,Ltd. Address before: 210000 No. 1 East Tianyuan Road, Jiangning District, Jiangsu, Nanjing Patentee before: CHINA INFORMATION CONSULTING & DESIGNING INSTITUTE Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder |